From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 574F1158086 for ; Tue, 4 Jan 2022 22:58:45 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id EA6992BC071; Tue, 4 Jan 2022 22:58:39 +0000 (UTC) Received: from smtp.gentoo.org (woodpecker.gentoo.org [140.211.166.183]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 474D92BC020 for ; Tue, 4 Jan 2022 22:58:39 +0000 (UTC) From: Sam James To: gentoo-dev@lists.gentoo.org Cc: qa@gentoo.org, Sam James Subject: [gentoo-dev] [PATCH] check-reqs.eclass: clamp MAKEOPTS for memory/RAM usage Date: Tue, 4 Jan 2022 22:58:17 +0000 Message-Id: <20220104225817.2341950-1-sam@gentoo.org> X-Mailer: git-send-email 2.34.1 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Archives-Salt: 4035f17a-efc1-4587-a8d1-043940e393a9 X-Archives-Hash: 2c399e67d1445d73ff467e5e27782e26 Crank down MAKEOPTS jobs if MAKEOPTS="-jN" is too high for the amount of RAM available (uses amount declared as needed in the ebuild). Typically should be ~2GB per job. Bug: https://bugs.gentoo.org/570534 Signed-off-by: Sam James --- eclass/check-reqs.eclass | 42 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 39 insertions(+), 3 deletions(-) diff --git a/eclass/check-reqs.eclass b/eclass/check-reqs.eclass index 2130e2e349141..8c4adc8b4f121 100644 --- a/eclass/check-reqs.eclass +++ b/eclass/check-reqs.eclass @@ -43,6 +43,8 @@ case ${EAPI} in *) die "${ECLASS}: EAPI=${EAPI:-0} is not supported" ;; esac +inherit multiprocessing + EXPORT_FUNCTIONS pkg_pretend pkg_setup if [[ ! ${_CHECK_REQS_ECLASS} ]]; then @@ -53,6 +55,13 @@ _CHECK_REQS_ECLASS=1 # @DESCRIPTION: # How much RAM is needed? Eg.: CHECKREQS_MEMORY=15M +# @ECLASS-VARIABLE: CHECKREQS_MEMORY_MANGLE_JOBS +# @USER_VARIABLE +# @DESCRIPTION: +# Allow packages to reduce the number of multiprocessing (e.g. make, ninja) jobs +# to lower memory usage. +: ${CHECKREQS_MEMORY_MANGLE_JOBS=yes} + # @ECLASS-VARIABLE: CHECKREQS_DISK_BUILD # @DEFAULT_UNSET # @DESCRIPTION: @@ -346,9 +355,36 @@ _check-reqs_memory() { eend 0 else eend 1 - _check-reqs_unsatisfied \ - ${size} \ - "RAM" + + # Has the user allowed us to mangle their MAKEOPTS? + if [[ ${CHECKREQS_MEMORY_MANGLE_JOBS} == "yes" ]] ; then + local jobs=$(makeopts_jobs) + + local estimated_max_memory=$((${actual_memory}/$(_check-reqs_get_kibibytes 1G))) + if [[ $((jobs*2)) -gt ${estimated_max_memory} ]] ; then + # Number of jobs exceeds RAM/2GB, so clamp it. + local new_jobs=$(($(_check-reqs_get_number ${estimated_max_memory}G)*10/20)) + + # This might _still_ be too big on small machines. Give up in such cases. + # (Users can still set the do nothing variable which is independent of this.) + if [[ $((new_jobs*2)) -gt ${estimated_max_memory} ]] ; then + _check-reqs_unsatisfied \ + ${size} \ + "RAM" + else + # The clamped jobs seem to be enough to satisfy the check-reqs requirement from the ebuild. + ewarn "Clamping MAKEOPTS jobs to -j${new_jobs} to reduce memory usage" + ewarn "Compiler jobs may use around ~2GB each: https://wiki.gentoo.org/wiki/MAKEOPTS" + ewarn "To disable this, set CHECKREQS_MEMORY_MANGLE_JOBS=no." + + MAKEOPTS+=" -j${new_jobs}" + fi + fi + else + _check-reqs_unsatisfied \ + ${size} \ + "RAM" + fi fi else eend 1 -- 2.34.1