From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 6CFEF158089 for ; Mon, 18 Sep 2023 13:03:28 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 4C32A2BC087; Mon, 18 Sep 2023 13:02:57 +0000 (UTC) Received: from mail-oi1-f182.google.com (mail-oi1-f182.google.com [209.85.167.182]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 01D532BC077 for ; Mon, 18 Sep 2023 13:02:56 +0000 (UTC) Received: by mail-oi1-f182.google.com with SMTP id 5614622812f47-3a889485117so523431b6e.1 for ; Mon, 18 Sep 2023 06:02:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695042176; x=1695646976; h=in-reply-to:reply-to:from:references:to:content-language:subject :user-agent:mime-version:date:message-id:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=BMu08iVxmodLP8RX6r3V6kyQCEY3z12J8A/l7Ex5s0E=; b=pSz/dQ1yneToJy/eE78Z//a5GFc9SLqLDoPLG6FaWY4nmccs9GWL10ym9FBZn0D0ey /HgOr/YooyDYuYH88z4fOGjQDL4zbGHQy9wfNAqqk+cur1nov85iI3gG9S1y0DXMbdr6 fg0chijdbrwzqEbnpPidB/5l3FRtdgG087OuEWWvowG16d+MnIuGdAa8NNW0FroHSNYG 5ZTE2maxuMjLnLZlDiILMGZvYL29mzlSFovflMS10AMUK6oLUETfJ2n0s/QXfhdzvjQN feGY+T2i+GZ+FlI5mHkU00Ukt7GeVQ+b9vKu2b0nOU0oEWl76Hs9Sd85TII5/cUboHVh SXQg== X-Gm-Message-State: AOJu0YxVJJEM3vyNuHqqOZe+w4Z6+dg1aN/jFaAgKLD70GPW0g4QukBB cnXhnSOFRI3rJ24ZgINdtcasezOm7jl8KEOqgnx0/g== X-Google-Smtp-Source: AGHT+IHUdYl/yhfRxXqhz86YIJQ7zNC1+fcvrR0HDroYUvAPaSIbhyu2Q4d0GMwMZmuopXRiaMKqgw== X-Received: by 2002:a05:6870:391f:b0:1d0:e372:6cf8 with SMTP id b31-20020a056870391f00b001d0e3726cf8mr9141339oap.2.1695042174174; Mon, 18 Sep 2023 06:02:54 -0700 (PDT) Received: from [192.168.1.18] (c-73-238-129-126.hsd1.ma.comcast.net. [73.238.129.126]) by smtp.gmail.com with ESMTPSA id e2-20020a0cf342000000b0064f4f14aecesm625548qvm.24.2023.09.18.06.02.53 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 Sep 2023 06:02:53 -0700 (PDT) Content-Type: multipart/alternative; boundary="------------G4eznZekFLgWONcTKP4Lhntu" Message-ID: <73f47a82-4e46-44fe-a604-c3b8b0fa8b53@users.sourceforge.net> Date: Mon, 18 Sep 2023 08:59:03 -0400 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [gentoo-user] Controlling emerges Content-Language: en-US-large To: gentoo-user@lists.gentoo.org References: <4531844.LvFx2qVVIh@wstn> From: Jack In-Reply-To: <4531844.LvFx2qVVIh@wstn> X-Archives-Salt: 4686f830-d837-4991-8a4a-d23cde1422fb X-Archives-Hash: a3d2ad28f5fc24ccb6b0b3b3d71439c9 This is a multi-part message in MIME format. --------------G4eznZekFLgWONcTKP4Lhntu Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 9/18/23 08:00, Peter Humphrey wrote: > Hello list, > > We've had a few discussions here on how to balance the parameters to emerge > to make the most of the resources available. Here's another idea: > > One the one hand, big jobs should be able to use the maximum CPU > performance and RAM capacity, but on the other we don't want to flood the > system. > > Therefore, I think it would be useful to be able to specify in env and > package.env that a job should be run on its own - if any other emerge jobs are > scheduled, wait until they're finished. Combine that with a specific MAKEOPTS, > and we'd have a more flexible deployment of resouces. > > Is this feasible? What have I not thought of? I've had exactly the same thought for some time now.  My guess is that it is theoretically possible to add some USE flag or ENV var for portage to recognize, but I don't know the portage internals well enough to guess how much effort it would be.  Given that portage orders ebuilds in a single emerge session based on some dependency graph, that seems like a good place to put the necessary hooks. As a starting point, one option might be to create a special/magic ebuild and make it a dependency of those jobs that need to be run alone, and have something about it that won't run if anything else is still running.  But, I don't know if those pre-checks (such as checking for enough RAM and/or disk space) can be run at build time and not just at portage startup time.  The other possible problem with that approach would be to be sure that ebuild gets run separately for each other ebuild that depends on it - not all of them depending on it being run once.Also, those blocking ebuilds have work so that if several of them are queued (and running their "wait for everything else to finish" scripts - exactly one of them needs to start. I don't know if those pre-check scripts count as running before or within the ebuild itself. --------------G4eznZekFLgWONcTKP4Lhntu Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 9/18/23 08:00, Peter Humphrey wrote:
Hello list,

We've had a few discussions here on how to balance the parameters to emerge 
to make the most of the resources available. Here's another idea:

One the one hand, big jobs should be able to use the maximum CPU 
performance and RAM capacity, but on the other we don't want to flood the 
system.

Therefore, I think it would be useful to be able to specify in env and 
package.env that a job should be run on its own - if any other emerge jobs are 
scheduled, wait until they're finished. Combine that with a specific MAKEOPTS, 
and we'd have a more flexible deployment of resouces.

Is this feasible? What have I not thought of?

I've had exactly the same thought for some time now.  My guess is that it is theoretically possible to add some USE flag or ENV var for portage to recognize, but I don't know the portage internals well enough to guess how much effort it would be.  Given that portage orders ebuilds in a single emerge session based on some dependency graph, that seems like a good place to put the necessary hooks.

As a starting point, one option might be to create a special/magic ebuild and make it a dependency of those jobs that need to be run alone, and have something about it that won't run if anything else is still running.  But, I don't know if those pre-checks (such as checking for enough RAM and/or disk space) can be run at build time and not just at portage startup time.  The other possible problem with that approach would be to be sure that ebuild gets run separately for each other ebuild that depends on it - not all of them depending on it being run once. Also, those blocking ebuilds have work so that if several of them are queued (and running their "wait for everything else to finish" scripts - exactly one of them needs to start. I don't know if those pre-check scripts count as running before or within the ebuild itself.

--------------G4eznZekFLgWONcTKP4Lhntu--