[gentoo-user] Compile large packages as last package

[gentoo-user] Compile large packages as last package

[Ramon Fischer]

[-- Attachment #1.1: Type: text/plain, Size: 733 bytes --]

Hello list,

I would like to tag or be able to prioritise (not via "nice" or 
"renice") large packages.

Currently, one system is compiling package 245 of 279 and nothing else. 
"qlop --running --verbose --time" shows me, that it is "qtwebengine", 
which will compile for a few hours, blocking all other packages.

I set "MAKEOPTS" AND "EMERGE_DEFAULT_OPTS" like so:

    MAKEOPTS="--jobs 8 --load-average 7.2"
    EMERGE_DEFAULT_OPTS="--jobs 8 --load-average 7.2"

Is there any way to tell "portage", that packages like "qtwebengine", 
"(ungoogled-)chromium", "firefox" and so on are always compiled as last 
package?

Regards
-Ramon

-- 
GPG public key: 5983 98DA 5F4D A464 38FD CF87 155B E264 13E6 99BF



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [gentoo-user] Compile large packages as last package

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 578 bytes --]

On Sat, 14 Aug 2021 21:20:23 +0200, Ramon Fischer wrote:

> Is there any way to tell "portage", that packages like "qtwebengine", 
> "(ungoogled-)chromium", "firefox" and so on are always compiled as last 
> package?

You can use the --exclude option for emerge. Maybe a script that runs
emerge @world with --exclude then again without?

While I haven't tried it with a script, I do use --exclude when chromium
is in the list because my laptop only has 8GB RAM, which means chromium
takes ages.



-- 
Neil Bothwick

Isn't 'Criminal Lawyer' rather redundant?

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[gentoo-user] Re: Compile large packages as last package

[Nikos Chantziaras]

On 14/08/2021 22:20, Ramon Fischer wrote:
> Is there any way to tell "portage", that packages like "qtwebengine", 
> "(ungoogled-)chromium", "firefox" and so on are always compiled as last 
> package?

The simplest way is to exclude those packages in the first update, and 
then allow them in the second:

emerge -uDU @world --exclude "qtwebengine firefox chromium" && emerge 
-uDU @world

The dependency tracker of portage will of course also exclude packages 
that depend on the excluded packages, unless they themselves have 
updates pending. In that case, they *might* get built twice; once 
against the current version of the excluded packages, and then perhaps 
again on the second run, if there's rebuild triggers involved.

Most of the time though, you won't run into cases of redundant rebuilds. 
Rebuild triggers are not very common.

Re: [gentoo-user] Re: Compile large packages as last package

[Ramon Fischer]

[-- Attachment #1.1: Type: text/plain, Size: 2212 bytes --]

Thank you for your ideas!

I was actually hoping for a neat hack with "/etc/portage/env/" and 
"/etc/portage/package.env/", where you can set environment variables.

I will try out the following solution:

    $ < "/etc/portage/package.env/no_tmpfs.conf
    # custom - 20181121 - rfischer: list packages, which are too big for
    tmpfs
    #app-emulation/qemu-kv no_tmpfs.conf
    #app-office/libreoffice no_tmpfs.conf
    #dev-java/icedtea no_tmpfs.conf
    #dev-lang/ghc no_tmpfs.conf
    #dev-lang/rust no_tmpfs.conf
    #mail-client/thunderbird no_tmpfs.conf
    #sci-libs/tensorflow no_tmpfs.conf
    #sys-devel/gcc no_tmpfs.conf
    #www-client/firefox no_tmpfs.conf
    #www-client/ungoogled-chromium no_tmpfs.conf #throttle_make_emerge.conf

    $ < "${HOME}/bin/update.sh"
    [...]
    large_package_list=$(/bin/grep --extended-regexp --only-matching
    "[a-z]+-[a-z]+\/[-0-9a-zA-Z]+" "/etc/portage/package.env/no_tmpfs.conf")
    [...]
    /usr/bin/emerge --ask --update --deep --newuse --tree --verbose
    --exclude="${large_package_list//$'\n'/ }" @world
    /usr/bin/emerge --ask --update --deep --newuse --tree --verbose @world

-Ramon

On 15/08/2021 17:48, Nikos Chantziaras wrote:
> On 14/08/2021 22:20, Ramon Fischer wrote:
>> Is there any way to tell "portage", that packages like "qtwebengine", 
>> "(ungoogled-)chromium", "firefox" and so on are always compiled as 
>> last package?
>
> The simplest way is to exclude those packages in the first update, and 
> then allow them in the second:
>
> emerge -uDU @world --exclude "qtwebengine firefox chromium" && emerge 
> -uDU @world
>
> The dependency tracker of portage will of course also exclude packages 
> that depend on the excluded packages, unless they themselves have 
> updates pending. In that case, they *might* get built twice; once 
> against the current version of the excluded packages, and then perhaps 
> again on the second run, if there's rebuild triggers involved.
>
> Most of the time though, you won't run into cases of redundant 
> rebuilds. Rebuild triggers are not very common.
>
>

-- 
GPG public key: 5983 98DA 5F4D A464 38FD CF87 155B E264 13E6 99BF



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: [gentoo-user] Re: Compile large packages as last package

[David M. Fellows]

>Thank you for your ideas!
>
>I was actually hoping for a neat hack with "/etc/portage/env/" and 
>"/etc/portage/package.env/", where you can set environment variables.

Did you look at example 2 in
https://wiki.gentoo.org/wiki//etc/portage/package.env ?

It seems to address your problem.
DaveF
>
>I will try out the following solution:
>
>    $ < "/etc/portage/package.env/no_tmpfs.conf
>    # custom - 20181121 - rfischer: list packages, which are too big for
>    tmpfs
>    #app-emulation/qemu-kv no_tmpfs.conf
>    #app-office/libreoffice no_tmpfs.conf
>    #dev-java/icedtea no_tmpfs.conf
>    #dev-lang/ghc no_tmpfs.conf
>    #dev-lang/rust no_tmpfs.conf
>    #mail-client/thunderbird no_tmpfs.conf
>    #sci-libs/tensorflow no_tmpfs.conf
>    #sys-devel/gcc no_tmpfs.conf
>    #www-client/firefox no_tmpfs.conf
>    #www-client/ungoogled-chromium no_tmpfs.conf #throttle_make_emerge.conf
>
>    $ < "${HOME}/bin/update.sh"
>    [...]
>    large_package_list=$(/bin/grep --extended-regexp --only-matching
>    "[a-z]+-[a-z]+\/[-0-9a-zA-Z]+" "/etc/portage/package.env/no_tmpfs.conf")
>    [...]
>    /usr/bin/emerge --ask --update --deep --newuse --tree --verbose
>    --exclude="${large_package_list//$'\n'/ }" @world
>    /usr/bin/emerge --ask --update --deep --newuse --tree --verbose @world
>
>-Ramon
>
>On 15/08/2021 17:48, Nikos Chantziaras wrote:
>> On 14/08/2021 22:20, Ramon Fischer wrote:
>>> Is there any way to tell "portage", that packages like "qtwebengine", 
>>> "(ungoogled-)chromium", "firefox" and so on are always compiled as 
>>> last package?
>>
>> The simplest way is to exclude those packages in the first update, and 
>> then allow them in the second:
>>
>> emerge -uDU @world --exclude "qtwebengine firefox chromium" && emerge 
>> -uDU @world
>>
>> The dependency tracker of portage will of course also exclude packages 
>> that depend on the excluded packages, unless they themselves have 
>> updates pending. In that case, they *might* get built twice; once 
>> against the current version of the excluded packages, and then perhaps 
>> again on the second run, if there's rebuild triggers involved.
>>
>> Most of the time though, you won't run into cases of redundant 
>> rebuilds. Rebuild triggers are not very common.
>>
>>
>
>-- 
>GPG public key: 5983 98DA 5F4D A464 38FD CF87 155B E264 13E6 99BF
>
>
>
>>> application/pgp-signature attachment, name=OpenPGP_signature

Re: [gentoo-user] Re: Compile large packages as last package [SOLVED]

[Ramon Fischer]

[-- Attachment #1.1: Type: text/plain, Size: 3560 bytes --]

Thank you for the hint, but I am very aware of this.

As the subject says, I want to compile large packages as last package, 
since it sometimes happens, that a large package may get compiled as 
package 245 of 300 for example; blocking other small packages.

I just re-used "/etc/portage/package.env/no_tmpfs.conf", since it 
already contains a list of large packages, which most likely need a long 
time to compile.

I should have been more elaborate in my last e-mail:

    I was actually hoping for a neat hack, where you can tag or
    prioritise (not nice or renice) packages in "/etc/portage/env/" or
    "/etc/portage/package.env/" to compile them as last packages.

The solution with "--exclude" is working well so far; I had no redundant 
rebuilds. :)

Maybe I need to tweak it later on, if rebuilds occur frequently, but 
this for another time.

-Ramon

On 08/09/2021 17:24, David M. Fellows wrote:
>> Thank you for your ideas!
>>
>> I was actually hoping for a neat hack with "/etc/portage/env/" and
>> "/etc/portage/package.env/", where you can set environment variables.
> Did you look at example 2 in
> https://wiki.gentoo.org/wiki//etc/portage/package.env ?
>
> It seems to address your problem.
> DaveF
>> I will try out the following solution:
>>
>>     $ < "/etc/portage/package.env/no_tmpfs.conf
>>     # custom - 20181121 - rfischer: list packages, which are too big for
>>     tmpfs
>>     #app-emulation/qemu-kv no_tmpfs.conf
>>     #app-office/libreoffice no_tmpfs.conf
>>     #dev-java/icedtea no_tmpfs.conf
>>     #dev-lang/ghc no_tmpfs.conf
>>     #dev-lang/rust no_tmpfs.conf
>>     #mail-client/thunderbird no_tmpfs.conf
>>     #sci-libs/tensorflow no_tmpfs.conf
>>     #sys-devel/gcc no_tmpfs.conf
>>     #www-client/firefox no_tmpfs.conf
>>     #www-client/ungoogled-chromium no_tmpfs.conf #throttle_make_emerge.conf
>>
>>     $ < "${HOME}/bin/update.sh"
>>     [...]
>>     large_package_list=$(/bin/grep --extended-regexp --only-matching
>>     "[a-z]+-[a-z]+\/[-0-9a-zA-Z]+" "/etc/portage/package.env/no_tmpfs.conf")
>>     [...]
>>     /usr/bin/emerge --ask --update --deep --newuse --tree --verbose
>>     --exclude="${large_package_list//$'\n'/ }" @world
>>     /usr/bin/emerge --ask --update --deep --newuse --tree --verbose @world
>>
>> -Ramon
>>
>> On 15/08/2021 17:48, Nikos Chantziaras wrote:
>>> On 14/08/2021 22:20, Ramon Fischer wrote:
>>>> Is there any way to tell "portage", that packages like "qtwebengine",
>>>> "(ungoogled-)chromium", "firefox" and so on are always compiled as
>>>> last package?
>>> The simplest way is to exclude those packages in the first update, and
>>> then allow them in the second:
>>>
>>> emerge -uDU @world --exclude "qtwebengine firefox chromium" && emerge
>>> -uDU @world
>>>
>>> The dependency tracker of portage will of course also exclude packages
>>> that depend on the excluded packages, unless they themselves have
>>> updates pending. In that case, they *might* get built twice; once
>>> against the current version of the excluded packages, and then perhaps
>>> again on the second run, if there's rebuild triggers involved.
>>>
>>> Most of the time though, you won't run into cases of redundant
>>> rebuilds. Rebuild triggers are not very common.
>>>
>>>
>> -- 
>> GPG public key: 5983 98DA 5F4D A464 38FD CF87 155B E264 13E6 99BF
>>
>>
>>
>>>> application/pgp-signature attachment, name=OpenPGP_signature

-- 
GPG public key: 5983 98DA 5F4D A464 38FD CF87 155B E264 13E6 99BF



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]