[gentoo-user] emerge --emptytree : how to ?

[gentoo-user] emerge --emptytree : how to ?

[Helmut Jarausch]

Hi,
I think I'm in need of doing
emerge --emptytree ...
for the first time.
Can I do it on a running Gentoo system?
I expected it will take several days to complete.
This poses some problems to me.
First, I have to shut down my machine overnight.
Second, I haven't made good experience with --keep-going
nor with --resume.
What can I do if 'emerge -e ...' fails by itself or because I have to
shut my machine down?

Many thanks for some hints,
Helmut

Re: [gentoo-user] emerge --emptytree : how to ?

[Dale]

Helmut Jarausch wrote:
> Hi,
> I think I'm in need of doing 
> emerge --emptytree ...
> for the first time.
> Can I do it on a running Gentoo system?
> I expected it will take several days to complete.
> This poses some problems to me.
> First, I have to shut down my machine overnight.
> Second, I haven't made good experience with --keep-going
> nor with --resume.
> What can I do if 'emerge -e ...' fails by itself or because I have to
> shut my machine down?
>
> Many thanks for some hints,
> Helmut
>
>


I do this on occasion when some update makes things go weird.  I'll look
for anyone else having the issue and if not, then I do a emerge -e world
to see if it helps. 

First, I have entries in make.conf to help make it so that it doesn't
affect what I'm doing.  I use ionice, -j and friends to do that.  I also
use --keep-going as well.  Generally, I can't tell it is doing
anything.  There is a few packages that it slows things down for a few
minutes.  It doesn't do it for the whole compile process, just a few
minutes of it.  Firefox, Libreoffice is two that I recall. I'll post
some of my make.conf items below. 

Second, you can skip certain programs, large ones for example.  You can
for example add this:  --exclude libreoffice  That will let it skip
libreoffice but keep in mind, some dependencies may be skipped as well,
if nothing else depends on them.  I haven't tested that but that's my
thinking.  Maybe someone else has more ideas on that. 

Third, I'm almost certain --resume works even after a reboot.  Just keep
in mind, if it was in the middle of a package compile, it likely will
start over from scratch.  That's my experience at least. 

Some of my make.conf entries.  You may not need all of these so edit out
what you don't want or change values if you need to.  I have a four core
CPU. 

FEATURES="-usersync -userpriv -usersandbox buildpkg sandbox parallel-fetch"

MAKEOPTS="-j5"

EMERGE_DEFAULT_OPTS="--with-bdeps y --backtrack=100 --keep-going -v -j5
--quiet-build=n -1 --unordered-display"

PORTAGE_NICENESS=5

PORTAGE_IONICE_COMMAND="ionice -c 3 -p \${PID}"

As for the command I use, emerge -ea world.  If you have to shutdown for
a while, once you reboot, try emerge --resume and see if it works.  It
should.  I've done it before but its been a good while back.

Hope that helps.

Dale

:-)  :-) 

Re: [gentoo-user] [OT] emerge default opts

[Michael Orlitzky]

On 10/14/2017 07:25 AM, Dale wrote:
> 
> EMERGE_DEFAULT_OPTS="--with-bdeps y ...

You probably don't need this one any more... you'll have to consult a
lawyer to figure out what `man emerge` says on the matter, but IIRC, the
--with-bdeps-auto flag (enabled by default) now does the sensible thing,
by default, for both updates and depcleans.

Re: [gentoo-user] [OT] emerge default opts

[Dale]

Michael Orlitzky wrote:
> On 10/14/2017 07:25 AM, Dale wrote:
>> EMERGE_DEFAULT_OPTS="--with-bdeps y ...
> You probably don't need this one any more... you'll have to consult a
> lawyer to figure out what `man emerge` says on the matter, but IIRC, the
> --with-bdeps-auto flag (enabled by default) now does the sensible thing,
> by default, for both updates and depcleans.
>
>


My make.conf is old.  Some of the entries there could be over a decade
old.  I also have commented out a lot of really old stuff, just to see
if it does anything.   I'll go check man emerge and friends and then see
if google does anything useful.  We all know how google is.  ;-)

While at it.  Is there a tool that tells when USE flags in make.conf is
either no longer used or doesn't even exist anymore?  I googled that a
while back and didn't find anything.  I don't think the eix tools does
that either.  It does for the other /etc/portage/ files but I don't
think it touches make.conf. 

Thanks for the heads up. 

Dale

:-)  :-) 

Re: [gentoo-user] [OT] emerge default opts

[Michael Orlitzky]

On 10/14/2017 09:30 PM, Dale wrote:
> 
> While at it.  Is there a tool that tells when USE flags in make.conf is
> either no longer used or doesn't even exist anymore? 

I don't know of one. It doesn't *sound* hard, but you would have to
consider local use flags, flags from overlays, USE_EXPAND flags,
wildcards, USE_ORDER, etc. -- so maybe it's actually hard/slow to do it.

I found this feature request,

  https://github.com/vaeth/eix/issues/38

and I guess that confirms that it's harder than it looks. Checking for
nonexistent flags would be easier than checking for redundant flags
because the latter depends on your package manager configuration.

Re: [gentoo-user] [OT] emerge default opts

[Dale]

Michael Orlitzky wrote:
> On 10/14/2017 09:30 PM, Dale wrote:
>> While at it.  Is there a tool that tells when USE flags in make.conf is
>> either no longer used or doesn't even exist anymore? 
> I don't know of one. It doesn't *sound* hard, but you would have to
> consider local use flags, flags from overlays, USE_EXPAND flags,
> wildcards, USE_ORDER, etc. -- so maybe it's actually hard/slow to do it.
>
> I found this feature request,
>
>   https://github.com/vaeth/eix/issues/38
>
> and I guess that confirms that it's harder than it looks. Checking for
> nonexistent flags would be easier than checking for redundant flags
> because the latter depends on your package manager configuration.
>
>


I was thinking it may be harder than one thinks since I don't know of a
way to do it but have seen some who want something that does it.  I'd be
happy if it just told me what USE flags no longer exist at all.  My USE
line in make.conf is quite lengthy.  I'm certain a lot of it could be
gone.  Some may not have existed for years. 

Maybe one day.  Just maybe. 

Thanks

Dale

:-)  :-) 

Re: [gentoo-user] [OT] emerge default opts

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 751 bytes --]

On Sun, 15 Oct 2017 09:46:26 -0500, Dale wrote:

> I was thinking it may be harder than one thinks since I don't know of a
> way to do it but have seen some who want something that does it.  I'd be
> happy if it just told me what USE flags no longer exist at all.  My USE
> line in make.conf is quite lengthy.  I'm certain a lot of it could be
> gone.  Some may not have existed for years. 

How about something like this?

source /etc/portage/make.conf
for flag in $USE; do
    grep -q '^${flag} ' $PORTDIR/profile/use.desc || \
      grep -q ':${flag} ' $PORTDIR/profile/use.local.desc || \
      echo "${flag} not found in use.desc or use.local.desc"
fi


-- 
Neil Bothwick

OPERATOR ERROR: Nyah, Nyah, Nyah, Nyah, Nyah!

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] [OT] emerge default opts

[Alan McKinnon]

On 15/10/2017 16:31, Michael Orlitzky wrote:
> On 10/14/2017 09:30 PM, Dale wrote:
>>
>> While at it.  Is there a tool that tells when USE flags in make.conf is
>> either no longer used or doesn't even exist anymore? 
> 
> I don't know of one. It doesn't *sound* hard, but you would have to
> consider local use flags, flags from overlays, USE_EXPAND flags,
> wildcards, USE_ORDER, etc. -- so maybe it's actually hard/slow to do it.
> 
> I found this feature request,
> 
>   https://github.com/vaeth/eix/issues/38
> 
> and I guess that confirms that it's harder than it looks. Checking for
> nonexistent flags would be easier than checking for redundant flags
> because the latter depends on your package manager configuration.
> 

There is a suitable tool. It's called grep, copious use of.
A suitably complex solution for the complexity of the problem!

-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] [OT] emerge default opts

[Rich Freeman]

On Sun, Oct 15, 2017 at 11:47 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> On 15/10/2017 16:31, Michael Orlitzky wrote:
>> On 10/14/2017 09:30 PM, Dale wrote:
>>>
>>> While at it.  Is there a tool that tells when USE flags in make.conf is
>>> either no longer used or doesn't even exist anymore?
>>
>> I don't know of one. It doesn't *sound* hard, but you would have to
>> consider local use flags, flags from overlays, USE_EXPAND flags,
>> wildcards, USE_ORDER, etc. -- so maybe it's actually hard/slow to do it.
>>
>> I found this feature request,
>>
>>   https://github.com/vaeth/eix/issues/38
>>
>> and I guess that confirms that it's harder than it looks. Checking for
>> nonexistent flags would be easier than checking for redundant flags
>> because the latter depends on your package manager configuration.
>>
>
> There is a suitable tool. It's called grep, copious use of.
> A suitably complex solution for the complexity of the problem!
>

Or you could just use portpeek...

-- 
Rich

Re: [gentoo-user] [OT] emerge default opts

[Bill Kenworthy]

On 16/10/17 04:39, Rich Freeman wrote:
> On Sun, Oct 15, 2017 at 11:47 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
>> On 15/10/2017 16:31, Michael Orlitzky wrote:
>>> On 10/14/2017 09:30 PM, Dale wrote:
>>>>
>>>> While at it.  Is there a tool that tells when USE flags in make.conf is
>>>> either no longer used or doesn't even exist anymore?
>>>
>>> I don't know of one. It doesn't *sound* hard, but you would have to
>>> consider local use flags, flags from overlays, USE_EXPAND flags,
>>> wildcards, USE_ORDER, etc. -- so maybe it's actually hard/slow to do it.
>>>
>>> I found this feature request,
>>>
>>>   https://github.com/vaeth/eix/issues/38
>>>
>>> and I guess that confirms that it's harder than it looks. Checking for
>>> nonexistent flags would be easier than checking for redundant flags
>>> because the latter depends on your package manager configuration.
>>>
>>
>> There is a suitable tool. It's called grep, copious use of.
>> A suitably complex solution for the complexity of the problem!
>>
> 
> Or you could just use portpeek...
> 


moriah ~ # euse -i flag
global use flags (searching: flag)
************************************************************
no matching entries found

local use flags (searching: flag)
************************************************************
no matching entries found
moriah ~ #

BillK

Re: [gentoo-user] [OT] emerge default opts

[Michael Orlitzky]

On 10/15/2017 04:39 PM, Rich Freeman wrote:
>>> On 10/14/2017 09:30 PM, Dale wrote:
>>>>
>>>> While at it.  Is there a tool that tells when USE flags in make.conf is
>>>> either no longer used or doesn't even exist anymore?
>>>
> 
> Or you could just use portpeek...
> 

portpeek doesn't check make.conf...

On the other hand, now would be a good time for me to clean all of the
positive USE flags out of my make.conf.

Re: [gentoo-user] emerge --emptytree : how to ?

[Walter Dnes]

On Sat, Oct 14, 2017 at 06:25:35AM -0500, Dale wrote

> Some of my make.conf entries.  You may not need all of these so edit out
> what you don't want or change values if you need to.  I have a four core
> CPU. 
> 
> FEATURES="-usersync -userpriv -usersandbox buildpkg sandbox parallel-fetch"
> 
> MAKEOPTS="-j5"

  There is some controversy over setting MAKEOPTS=${number of threads}
possibly being better than MAKEOPTS=${number of threads} + 1
https://blogs.gentoo.org/ago/2013/01/14/makeopts-jcore-1-is-not-the-best-optimization/

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [gentoo-user] emerge --emptytree : how to ?

[R0b0t1]

On Sun, Oct 15, 2017 at 8:51 PM, Walter Dnes <waltdnes@waltdnes.org> wrote:
> On Sat, Oct 14, 2017 at 06:25:35AM -0500, Dale wrote
>
>> Some of my make.conf entries.  You may not need all of these so edit out
>> what you don't want or change values if you need to.  I have a four core
>> CPU.
>>
>> FEATURES="-usersync -userpriv -usersandbox buildpkg sandbox parallel-fetch"
>>
>> MAKEOPTS="-j5"
>
>   There is some controversy over setting MAKEOPTS=${number of threads}
> possibly being better than MAKEOPTS=${number of threads} + 1
> https://blogs.gentoo.org/ago/2013/01/14/makeopts-jcore-1-is-not-the-best-optimization/
>

I received the same results when I tested it myself. On modern servers
especially, RAM access speed seems to be matched to core processing
capability very well.

Strangely, adding to the build thread count doesn't seem to help when
disk IO is the bottleneck. It is conceivable that it could but in
practice the location of build data and files seems to be disperse
enough that there are no great access optimizations, and each read
blocks individually.

Cheers,
     R0b0t1

Re: [gentoo-user] emerge --emptytree : how to ?

[Dale]

Walter Dnes wrote:
> On Sat, Oct 14, 2017 at 06:25:35AM -0500, Dale wrote
>
>> Some of my make.conf entries.  You may not need all of these so edit out
>> what you don't want or change values if you need to.  I have a four core
>> CPU. 
>>
>> FEATURES="-usersync -userpriv -usersandbox buildpkg sandbox parallel-fetch"
>>
>> MAKEOPTS="-j5"
>   There is some controversy over setting MAKEOPTS=${number of threads}
> possibly being better than MAKEOPTS=${number of threads} + 1
> https://blogs.gentoo.org/ago/2013/01/14/makeopts-jcore-1-is-not-the-best-optimization/
>


I tried different settings before that one and settled on the one that
works best.  It may not work on someone else's system but it does fine
here.  One thing about most of my settings, they've been tested pretty
well and work well on this system at least. 

Since I have a four core system, I usually set everything to number of
cores plus one which should lead to a always busy CPU.  So far, it has
worked out that way for the most part.  If I had enough memory, I might
could up that but it could lead to another problem I had before lowering
it to that number.  Compiles that fail.  It's been a while but at one
point, some packages had to have -j 1 to compile.  That is rare nowadays
I guess but since the settings I have works for me, I'll likely leave
them like they are. 

The one thing I did have to change recently, not compiling some large
packages on tmpfs.  If it was just one package I have enough memory. 
However, sometimes it would be two and sometimes even three that were
large.  The main culprits were Seamonkey, Firefox and Libreoffice.  I
think I added a couple other large ones in just to be sure but other
than that, these settings have been around for a good long while. 

I might add, I've read that blog before.  I also read the comments where
others had different results and pointed out some issues with the blog
points.  The blogger used kdelibs as a test case.  Thing is, I compile
more than just kdelibs here.  Most of the time I start my emerges before
I go to sleep.  I put Gkrellm where I can see it and I watch what the
CPUs are doing.  Generally speaking, the CPU cores stay busy the whole
time.  There are a few times that it is not at 100% but generally it
is.  If anything, since it does have times where it isn't at 100%, I may
need to up that number by one to see if that keeps it more busy.  Thing
is, in the past, it didn't help any. 

Thanks for the info. 

Dale

:-)  :-)