From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 28024158089 for ; Mon, 18 Sep 2023 18:49:23 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 3A1152BC0D0; Mon, 18 Sep 2023 18:49:18 +0000 (UTC) Received: from mail-ot1-f47.google.com (mail-ot1-f47.google.com [209.85.210.47]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 5143B2BC020 for ; Mon, 18 Sep 2023 18:49:17 +0000 (UTC) Received: by mail-ot1-f47.google.com with SMTP id 46e09a7af769-6c09f1f4610so2775197a34.0 for ; Mon, 18 Sep 2023 11:49:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695062956; x=1695667756; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dx4TntmVBcUMJjGN8vs1kGNdgW331wUez0ivURt6Ivs=; b=ajtbrfOlHnLNYouZVAOlEX+n85WrV6wLq2Q8ZOnjeVEht4oaQ3x+W+UDb4FF5Dmvc6 /Q10U5R8zZCSvUhPYaUFhxVG0EJitQHf0+Q2sFqygK85Zscf/arEeB+LJwRInp3gflKP DzB0rOZsXgyqkO4oSuT3JzLNRJIB3Wnqr5Tv+ip6hkDW6T5gqBhvImBHkKa1ANCdYv5m Ly5D3pDZPExS2UCivZGsg5p/HX4P5dwbLVb2YAL5ywUkuupNocQy7RclRjvtc41t8VBL 4MWU4Jx7Y5H2vSYrNT748WOn5h1nS73Zv8Dypy8XutR8SvRapxBh/nPzqE9wOv4tE8Au J9/g== X-Gm-Message-State: AOJu0Yxd4Y3Gi368aWjkHivx6iBn2oBcnI76+x7Erd4SwHd59A9toWZC PksQ6dFcREb7moNmuNlr2JPXtrCQGUpk+lN6cZZyphnb X-Google-Smtp-Source: AGHT+IHYj2nGyp+AE7Yr8qr74qPOjSKn4AurxMEdZWqCSu7GyJyvazeMULC8NA96CERSM25ROZooynqmI8pFotIVVv4= X-Received: by 2002:a05:6358:63a2:b0:134:d030:e5da with SMTP id k34-20020a05635863a200b00134d030e5damr7757916rwh.21.1695062956273; Mon, 18 Sep 2023 11:49:16 -0700 (PDT) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 References: <4531844.LvFx2qVVIh@wstn> <8292274.T7Z3S40VBb@wstn> <1983362.usQuhbGJ8B@wstn> In-Reply-To: From: Rich Freeman Date: Mon, 18 Sep 2023 14:49:09 -0400 Message-ID: Subject: Re: [gentoo-user] Controlling emerges To: gentoo-user@lists.gentoo.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Archives-Salt: 889ff1e4-33b5-4182-b1d4-e1cd53ea5dbb X-Archives-Hash: 2aeec717d8f3842fcbb594cf085b8c55 On Mon, Sep 18, 2023 at 12:13=E2=80=AFPM Alan McKinnon wrote: > > Whether you just let emerge do it's thing or try get it to do big package= s on their own, everything is still going to use the same number of cpu cyc= les overall and you will save nothing. That is true of CPU, but not RAM. The problem with large parallel builds is that for 95% of packages they're fine, and for a few packages they'll eat up all the RAM in the system until the OOM killer kicks in, or the system just goes into a swap storm (which can cause panics with some less-than-perfect kernel drivers). I'm not aware of any simple solutions. I do have some packages set to just build with a small number of jobs, but that won't prevent other packages from being built alongside them. Usually that is enough though. It is just frustrating to watch a package take all day to build because I can't use more than -j2 or so without running out of RAM, usually just at one step of the build process. I can't see anybody bothering with this, but in theory packages could have a variable to hint at the max RAM consumed per job, and the max number of jobs it will run. Then the package manager could take the lesser of -j and the max jobs the package can run, multiply it by the RAM requirement, and compare that to available memory (or have a setting to limit max RAM). Basically treat RAM as a resource and let the package manager reduce -j to manage it if necessary. Hmm, I guess a workaround would be to set ulimits on the portage user so that emerge is killed before RAM use gets too out of hand. That won't help complete builds, but it would at least keep it from killing the system. --=20 Rich