Re: [gentoo-user] Controlling emerges

[Dale <rdalek1967@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3761 bytes --]

Alan McKinnon wrote:
>
>
> On Mon, Sep 18, 2023 at 6:03 PM Peter Humphrey <peter@prh.myzen.co.uk
> <mailto:peter@prh.myzen.co.uk>> wrote:
>
>     On Monday, 18 September 2023 14:48:46 BST Alan McKinnon wrote:
>     > On Mon, Sep 18, 2023 at 3:44 PM Peter Humphrey
>     <peter@prh.myzen.co.uk <mailto:peter@prh.myzen.co.uk>>
>     >
>     > wrote:
>     > > It may be less complex than you think, Jack. I envisage a
>     package being
>     > > marked
>     > > as solitary, and when portage reaches that package, it waits
>     until all
>     > > current
>     > > jobs have finished, then it starts the solitary package with the
>     > > environment
>     > > specified for it, and it doesn't start the next one until that
>     one has
>     > > finished.
>     > > The dependency calculation shouldn't need to be changed.
>     > >
>     > > It seems simple the way I see it.
>     >
>     > How does that improve emerge performance overall?
>
>     By allocating all the system resources to huge packages while not
>     flooding the
>     system with lesser ones. For example, I can set -j20 for
>     webkit-gtk today
>     without overflowing the 64GB RAM, and still have 4 CPU threads
>     available to
>     other tasks. The change I've proposed should make the whole
>     operation more
>     efficient overall and take less time.
>
>     As things stand today, I have to make do with -j12 or so, wasting
>     time and
>     resources. I have load-average set at 32, so if I were to set -j20
>     generally
>     I'd run out of RAM in no time. I've had many instances of packages
>     failing to
>     compile in a large update, but going just fine on their own; and
>     I've had
>     mysterious operational errors resulting, I suspect, from otherwise
>     undetected
>     miscompilation.
>
>     Previous threads have more detail of what I've tried already.
>
>
> I did read all those but no matter how you move things around you
> still have only X resources available all the time.
> Whether you just let emerge do it's thing or try get it to do big
> packages on their own, everything is still going to use the same
> number of cpu cycles overall and you will save nothing.
>
> If webkit-gtk is the only big package, have you considered:
>
> emerge -1v webkit-gtk && emerge -avuND @world?
>
>
> What you have is not a portage problem. It is a orthodox parallelism
> problem, and I think you are thinking your constraint is unique in the
> work - it isn't.
> With parallelism, trying to fiddle single nodes to improve things
> overall never really works out.
>
> Just my $0.02
>
>
> Alan
>
> -- 
> Alan McKinnon
> alan dot mckinnon at gmail dot com


I have to admit, I wish I could tell emerge to compile certain packages
on their own as well.  LOo, that qtweb package and a few others. 
Sometimes they end up naturally compiling on their own but sometimes, I
end up with LOo, Seamonkey or Firefox, or that qtweb package trying to
compile at the same time in some combination.  Sometimes, all four hit
at once.  It's bad enough when it is just two of them but when they all
hit, it causes problems.  It would be nice if we could set up a list
that tells emerge to emerge only one at a time just like we tell it not
to use tmpfs for certain builds. 

While just emerging them first might work, it also limits emerge to just
doing that package instead of the whole update.  It also could have
dependencies that also want a lot of resources.  I don't know about most
people but I run my updates while I sleep.  Having the option to set
that up would be nice.  It's not like packages are getting any smaller
either.  This is a growing problem. 

I have no idea how to do this but I do like the idea. 

Dale

:-)  :-) 

[-- Attachment #2: Type: text/html, Size: 7010 bytes --]