From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 2BBA0158089 for ; Mon, 18 Sep 2023 18:18:11 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 24B3F2BC0CD; Mon, 18 Sep 2023 18:18:07 +0000 (UTC) Received: from mail-yw1-x112c.google.com (mail-yw1-x112c.google.com [IPv6:2607:f8b0:4864:20::112c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 9E1342BC016 for ; Mon, 18 Sep 2023 18:18:06 +0000 (UTC) Received: by mail-yw1-x112c.google.com with SMTP id 00721157ae682-579de633419so46432867b3.3 for ; Mon, 18 Sep 2023 11:18:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1695061086; x=1695665886; darn=lists.gentoo.org; h=in-reply-to:mime-version:user-agent:date:message-id:autocrypt :openpgp:from:references:to:subject:from:to:cc:subject:date :message-id:reply-to; bh=mTgxAfqMiOH2ueQEomZQq/zN5vXhkxGQ5bbksEz8RGM=; b=Y6eQIWQIHJSbj2YfVf9oZha7+vdAeyw54CLHsuRWS5LHsiUhop98kuEipGdZZoqj9G L7NkhFOztvuh3H42aYHchv68RLYDGrj5nEnaypTyjJIWhjZfvWIunnuOyczaZKvTrQxx Aw4xn9tKNAR5+YHmtqVtMSH/UZJYo0fAgSxnkAum9poqQ+TaOJWvwIP0R6N/3JivLfe8 KGvP5A0EoXvQZ3CJA7N+IOTALhKWdqjbu4efwPzPU0an48CMS4XOc/MdXLPx22srj1q+ ITcrye+a6MnT2PYSaz3di7ZWFU9dGEaoqhwveiSLBPT78Nc34Ia5aBP0vuxoiZ3GKZ3q 3Zvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695061086; x=1695665886; h=in-reply-to:mime-version:user-agent:date:message-id:autocrypt :openpgp:from:references:to:subject:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mTgxAfqMiOH2ueQEomZQq/zN5vXhkxGQ5bbksEz8RGM=; b=pK12lx2pKYLEUKe7n527uU4Zt1ij1bFtwYZtBVDjMNR/Xb5R6MH9znISKw6RQjTouZ ll9k4P8paXryVxEhBh8V6PwMVDiBsKLOmvoDZJAHxlp4+Vz24TfJVvlo11WJ+59m93bu GWGGguzNSqZArfilE2H7UuXzulBZmGUTQtIvYlarNriWbkTIqspbRDtLS+djYCTGITGT Iz9D3dqoMETnqQxF6OkalarvJEVvKdw3zD3VsP3xUGEJWiFEUjISPtfqtVgOMD6qSU/L ZoA8aU4Pi7espbzBOIQRGmW1J1zeWmBFj3ki3yi44tYB3YFOvyN9abWBkAVHH4H9cws5 U7Lg== X-Gm-Message-State: AOJu0Yx+Zuwz7LU0IDPkFkMKGbAVgb5m0V2c/6j/h6GHqpIc885Oa++n GvdjTnEtdJZqxQVAeiec4aNs8PgWqNQ= X-Google-Smtp-Source: AGHT+IGVnTBuKvspSNvhIO7EC9el0jcH8okm3FyJB9oCvXOe1jCbPCIUGAgj/EPEIoXTcUrSuXIiPA== X-Received: by 2002:a81:a113:0:b0:59b:ee58:67fb with SMTP id y19-20020a81a113000000b0059bee5867fbmr9809531ywg.10.1695061086019; Mon, 18 Sep 2023 11:18:06 -0700 (PDT) Received: from [10.7.7.3] ([195.181.171.248]) by smtp.gmail.com with ESMTPSA id e3-20020a81dd03000000b00589a1dc0809sm2697359ywn.120.2023.09.18.11.18.05 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 18 Sep 2023 11:18:05 -0700 (PDT) Subject: Re: [gentoo-user] Controlling emerges To: gentoo-user@lists.gentoo.org References: <4531844.LvFx2qVVIh@wstn> <8292274.T7Z3S40VBb@wstn> <1983362.usQuhbGJ8B@wstn> From: Dale Openpgp: preference=signencrypt Autocrypt: addr=rdalek1967@gmail.com; prefer-encrypt=mutual; keydata= mQINBGFSciYBEADcEGMyJBSuavKO/XKUVvgkxck7Nl8Iuu8N2lcnRji/rSKg5c1Acix1ll9i oW8JBCHwvn0+Xy60BvEsqcup3YSHw5STl/bR1ePEehtnYrg8FdjdS91+B805RfnKMm69rFVI wLSBHQrSG1yxHd8CloWoEdhmVtP24buajbh114bgXd9ahtpZrCVMrWdWYUg2mEXguGV5uNAh Rf8SWxDNc79w24JxsV34a8niMUYMjzWr0rafIbzk732X38vGjVMLo/2mMpkbp9mPp++LHoY+ 0Pet8zxxdXPJSCd475kza1AD+hhSyBZXB9yknYWgyY3cZe1rGmooJSi2KX4QxO7npwLThcO1 be6KKRkd35+Fi/a1BzVOHsZMiK/gcwxEFoMd27gir4ehaeHJfFXl+65w4hj0EsOZSxrJrm2C R50g5By2czSKP1bADEygFNpIJj51AR+wM88NImG2RPtlT2maYBzazvF05g65cdHXGp1C7W5P wwwKU2DgABB2t7N7z5A69LnryBRw4zUYDRRYLTYlBlYgg+xILm2c0OrBdxJgLJa7JE50Eo25 d3PFwt9J0gYvqy6sPFLl9So0sDg9zm0hKQtXOP5kgropUFGrNoJI+mjwF4rYLRBVzZwNAvlO OhEvHubBo3mEllv4x+FeptwXZxlk7gUsdqI8AxnFB8K9wi6FVQARAQABtBtEYWxlIDxyZGFs ZWsxOTY3QGdtYWlsLmNvbT6JAk4EEwEIADgCGyMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AW IQQSG1h01ruv/WNXc3Q3RqOgiQH1GwUCYVJy8gAKCRA3RqOgiQH1G+waEACeTZCt77jnRAmQ AV7otKuZekDWiLi3Eig8tj5ZJiCNSYA/hIxzmexRP0GMqjitcXK1iGwWcvMzzvIq30GAjIfB 4BR38cnXbtBa6fNewiT7QaZe/Hn6yBRldXNQypzbHy+/o27bUEy+oX4rE7etUgEHQAjuw7xz XFWg4tH1/KJvsOVY5upnWc5LdxYhsuQ3dQD4b22GsK0pOBDfb9PiirYM8eGKvrVuq4E/c75z lDDFhINl18lNZ9D0ZFL3IkTjHsAAqFH9uhnnEB8CWdHbBewPEfRaOhBUYWZ3Q8uTkmDgZT8q D9jlvLEdw7Nh2ApdxoepnI/4D+ql2Gr4DtH7SEPydr5gcf1Qr/2bXRb1hAYnIVcbncs/Bm3Z bkRKPVWMfE3Fusa+p5hMzixk0YysMaTHlc7mYRYAEZGnPMXnmcCbetwARU7A0yz1M1kCMOAQ Lsz8KH5kv3cRenMB6SFfjND2JfAK61H5TtnPq3L8noS2ZykRYxq9Nm3X64O1tJojIKBoZFr8 AwYNCvqC6puUyGMuzHPh7jPof8glfrrEKIYUvNPGMDoVX3IGetxh/9l6NcxgFA4JGoR+LS3C zmeNrwlllAe3OEUfKoWVQ+pagpSdM+8hHolaSda4Ys66Z3fCR4ZvcTqfhTAVskpqdXa4isAk 7vTcXu3L499ttywEp7rJTbkCDQRhUnImARAAncUdVhmtRr59zqpTUppKroQYlzR0jv8oa7DG K4gakTAT2N7evnI9wpssmzyVk8VEiLzhnFQ/Ol3FRt6hZCXDJt0clyHOyTfvz/MNFttWuZTc mLpSvmRR6VRjAH+Tz3Eam2xUw3PGuH97BcXQ3NnX3msv1UDxtxxBu6e2YrdeOhrCUSgzokcJ 98ChUNy934cgepPybAI12lSWqVFQ1aG7jExZfiUk+333fPSDbpKoZbTW5YJLXbycmW/C1IWL qYQyNjRWKaGoJtUWFhhmNiOQct7n90aKivNVPavmN+UQ9LlMaINtf9T6XCzLfogCFsulDCDJ 0yNQLDTurHaB4E71xoctgXmLLq9z1RQ0W2XiVAAOZQj6K3+d0AOUjDhCQ2QW8dUSq0ckkZXV DKVJOGS8Nhf2eIWIqRnP3AcUiiaiFGqUaVUmUAZ6h/oJmgghEu/1S+pcuUKU5i69+XCZ3hH2 Jzwzbf7K+FAIkOhCfHncF8i1N1pk00pOVykNnqHTfFo3qFusHt0ZWgXVnnn4pYdXqZNoDhvF BRE5Vm4k/k96Pw8HRx6Os6eFSRrlqGzRgqsu86FekxusXB9UGv4lJhtU/J+8MRWsh22K718s DbQnABicGKFz1qQlWvcf59oTByhLINJCBt1WXl+TzJDXepr3QSkqmK41dO9Hob97C9dMiK8A EQEAAYkCNgQYAQgAIAIbDBYhBBIbWHTWu6/9Y1dzdDdGo6CJAfUbBQJhUnLyAAoJEDdGo6CJ AfUbVHIQAKSWw620vPhR3A/njU2z77F3z/Jk+HTKdE3fIyWSWdkYN7CBFL0NguOMP30WZ+qE sJhZu7T5hf251MwQUUt27xlfnKYOmQs7CqONlXuXlGZI6WufrUjxNcVz+5gJsqvUWuuJWsgg sDmE92IBnfG/f81fPHWQyfr/SF4wYDMyoFp5xCCQpp1zB63iuFvvrhxBkEHzmbRtVDOhl0Xp BVEDR1w3QRACw9QJD/KM05Czv9JNQYlwinWO/OaQ9cMlUpKLgswUPg9IZ5vucxScfuAUA5uC B1jlAQ8ZPlVukBmbEv5RGOv+lpuEbA3YDMVtEeH4YMFbjt/+vH3Cr2vTbp5JlpByLburJEH0 WXZLUawEfUsZvVwpOuJK75vaa2HYXee+Cb3iCIzwfIfctdlqzUcbGRczlRNM59hpvj4z29Gh 3kAxVHItAYq54ikxQ9l4hQ8s9sLYPbX/WtcBxNX8crBSw0FLnmzGleVEtBHyqtt5CLzQNgrj GYWl1vKDUmRPw1CdZ1c+fMN9CY11jOM5B5ZnqZWfDeVYO2iJ5SuvTycChexCb8WYn1bdCBIo bBtga2RBXbVt4Mh9E4owsszefn51MwfjXxB20Fc5k3GU1AVpTCMs3ayYCzo0b2pvEvdjtDcA CYLEFPWgaFX9iQAM/CDfKvTtvgGWpqtCL2raq/mQoJEU Message-ID: <723cb763-a72e-223e-e2a7-b643f180cb5f@gmail.com> Date: Mon, 18 Sep 2023 13:18:04 -0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0 SeaMonkey/2.53.17 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------C4F5E2469D56F5E27FAF20D3" X-Archives-Salt: a5403d65-186c-491e-9d5d-49a3ba73761e X-Archives-Hash: 2d193c4fa6a20e88d48859b19366320c This is a multi-part message in MIME format. --------------C4F5E2469D56F5E27FAF20D3 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Alan McKinnon wrote: > > > On Mon, Sep 18, 2023 at 6:03 PM Peter Humphrey > wrote: > > On Monday, 18 September 2023 14:48:46 BST Alan McKinnon wrote: > > On Mon, Sep 18, 2023 at 3:44 PM Peter Humphrey > > > > > > wrote: > > > It may be less complex than you think, Jack. I envisage a > package being > > > marked > > > as solitary, and when portage reaches that package, it waits > until all > > > current > > > jobs have finished, then it starts the solitary package with the > > > environment > > > specified for it, and it doesn't start the next one until that > one has > > > finished. > > > The dependency calculation shouldn't need to be changed. > > > > > > It seems simple the way I see it. > > > > How does that improve emerge performance overall? > > By allocating all the system resources to huge packages while not > flooding the > system with lesser ones. For example, I can set -j20 for > webkit-gtk today > without overflowing the 64GB RAM, and still have 4 CPU threads > available to > other tasks. The change I've proposed should make the whole > operation more > efficient overall and take less time. > > As things stand today, I have to make do with -j12 or so, wasting > time and > resources. I have load-average set at 32, so if I were to set -j20 > generally > I'd run out of RAM in no time. I've had many instances of packages > failing to > compile in a large update, but going just fine on their own; and > I've had > mysterious operational errors resulting, I suspect, from otherwise > undetected > miscompilation. > > Previous threads have more detail of what I've tried already. > > > I did read all those but no matter how you move things around you > still have only X resources available all the time. > Whether you just let emerge do it's thing or try get it to do big > packages on their own, everything is still going to use the same > number of cpu cycles overall and you will save nothing. > > If webkit-gtk is the only big package, have you considered: > > emerge -1v webkit-gtk && emerge -avuND @world? > > > What you have is not a portage problem. It is a orthodox parallelism > problem, and I think you are thinking your constraint is unique in the > work - it isn't. > With parallelism, trying to fiddle single nodes to improve things > overall never really works out. > > Just my $0.02 > > > Alan > > -- > Alan McKinnon > alan dot mckinnon at gmail dot com I have to admit, I wish I could tell emerge to compile certain packages on their own as well.  LOo, that qtweb package and a few others.  Sometimes they end up naturally compiling on their own but sometimes, I end up with LOo, Seamonkey or Firefox, or that qtweb package trying to compile at the same time in some combination.  Sometimes, all four hit at once.  It's bad enough when it is just two of them but when they all hit, it causes problems.  It would be nice if we could set up a list that tells emerge to emerge only one at a time just like we tell it not to use tmpfs for certain builds.  While just emerging them first might work, it also limits emerge to just doing that package instead of the whole update.  It also could have dependencies that also want a lot of resources.  I don't know about most people but I run my updates while I sleep.  Having the option to set that up would be nice.  It's not like packages are getting any smaller either.  This is a growing problem.  I have no idea how to do this but I do like the idea.  Dale :-)  :-)  --------------C4F5E2469D56F5E27FAF20D3 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
Alan McKinnon wrote:


On Mon, Sep 18, 2023 at 6:03 PM Peter Humphrey <peter@prh.myzen.co.uk> wrote:
On Monday, 18 September 2023 14:48:46 BST Alan McKinnon wrote:
> On Mon, Sep 18, 2023 at 3:44 PM Peter Humphrey <peter@prh.myzen.co.uk>
>
> wrote:
> > It may be less complex than you think, Jack. I envisage a package being
> > marked
> > as solitary, and when portage reaches that package, it waits until all
> > current
> > jobs have finished, then it starts the solitary package with the
> > environment
> > specified for it, and it doesn't start the next one until that one has
> > finished.
> > The dependency calculation shouldn't need to be changed.
> >
> > It seems simple the way I see it.
>
> How does that improve emerge performance overall?

By allocating all the system resources to huge packages while not flooding the
system with lesser ones. For example, I can set -j20 for webkit-gtk today
without overflowing the 64GB RAM, and still have 4 CPU threads available to
other tasks. The change I've proposed should make the whole operation more
efficient overall and take less time.

As things stand today, I have to make do with -j12 or so, wasting time and
resources. I have load-average set at 32, so if I were to set -j20 generally
I'd run out of RAM in no time. I've had many instances of packages failing to
compile in a large update, but going just fine on their own; and I've had
mysterious operational errors resulting, I suspect, from otherwise undetected
miscompilation.

Previous threads have more detail of what I've tried already.


I did read all those but no matter how you move things around you still have only X resources available all the time.
Whether you just let emerge do it's thing or try get it to do big packages on their own, everything is still going to use the same number of cpu cycles overall and you will save nothing.

If webkit-gtk is the only big package, have you considered:

emerge -1v webkit-gtk && emerge -avuND @world?

--
Alan McKinnon
alan dot mckinnon at gmail dot com


I have to admit, I wish I could tell emerge to compile certain packages on their own as well.  LOo, that qtweb package and a few others.  Sometimes they end up naturally compiling on their own but sometimes, I end up with LOo, Seamonkey or Firefox, or that qtweb package trying to compile at the same time in some combination.  Sometimes, all four hit at once.  It's bad enough when it is just two of them but when they all hit, it causes problems.  It would be nice if we could set up a list that tells emerge to emerge only one at a time just like we tell it not to use tmpfs for certain builds. 

While just emerging them first might work, it also limits emerge to just doing that package instead of the whole update.  It also could have dependencies that also want a lot of resources.  I don't know about most people but I run my updates while I sleep.  Having the option to set that up would be nice.  It's not like packages are getting any smaller either.  This is a growing problem. 

I have no idea how to do this but I do like the idea. 

Dale

:-)  :-) 
--------------C4F5E2469D56F5E27FAF20D3--