[gentoo-user] How long does "Verifying /usr/portage" take?

[gentoo-user] How long does "Verifying /usr/portage" take?

[Grant Edwards]

Now that the public key stuff is working again (knock on wood), I'm
curious if it's usual for an emerge --sync to take 10-15 minutes
longer than it used due to the "Verifying /usr/portage" step.

On some systems (with fewer packages installed) it only takes a minute
or less. But, on my "main" desktop system it takes 10-15 minutes every
time.  During the verify step, the emerge process is only using about
5% of the CPU, and my system is running 80% or more idle.

-- 
Grant Edwards               grant.b.edwards        Yow! TONY RANDALL!  Is YOUR
                                  at               life a PATIO of FUN??
                              gmail.com            

Re: [gentoo-user] How long does "Verifying /usr/portage" take?

[Dale]

Grant Edwards wrote:
> Now that the public key stuff is working again (knock on wood), I'm
> curious if it's usual for an emerge --sync to take 10-15 minutes
> longer than it used due to the "Verifying /usr/portage" step.
>
> On some systems (with fewer packages installed) it only takes a minute
> or less. But, on my "main" desktop system it takes 10-15 minutes every
> time.  During the verify step, the emerge process is only using about
> 5% of the CPU, and my system is running 80% or more idle.
>


I haven't timed mine yet but that sounds about like mine here.  I'm not
sure what the bottleneck is but I have a four core AMD CPU running at
3.2GHz with 16GBs of ram and SATA spinning rust drives.  While I'm glad
to have the added security measures, it does add a significant amount of
time to the update process, the tree not the compile part.  We all know
the compile part can get big.  lol 

I guess like everything else, we'll just have to get used to it.  People
will hack a ham sandwich if they can and can get something from it. 
That would be mustard on mine.  Some may like Mayo, which is fine too. 
;-) 

Dale

:-)  :-) 

[gentoo-user] Re: How long does "Verifying /usr/portage" take?

[Grant Edwards]

On 2018-07-06, Dale <rdalek1967@gmail.com> wrote:

> I haven't timed mine yet but that sounds about like mine here.  I'm not
> sure what the bottleneck is but I have a four core AMD CPU running at
> 3.2GHz with 16GBs of ram and SATA spinning rust drives.  While I'm glad
> to have the added security measures, it does add a significant amount of
> time to the update process, the tree not the compile part.  We all know
> the compile part can get big.  lol 

Yea, it sounds a bit stupid to whine about an extra 15 minutes doing a
"sync" now that the build time for chromium is measured in days on a
not-that-old machine. :)

-- 
Grant Edwards               grant.b.edwards        Yow! One FISHWICH coming
                                  at               up!!
                              gmail.com            

Re: [gentoo-user] How long does "Verifying /usr/portage" take?

[R0b0t1]

On Fri, Jul 6, 2018 at 2:16 PM, Dale <rdalek1967@gmail.com> wrote:
> Grant Edwards wrote:
>> Now that the public key stuff is working again (knock on wood), I'm
>> curious if it's usual for an emerge --sync to take 10-15 minutes
>> longer than it used due to the "Verifying /usr/portage" step.
>>
>> On some systems (with fewer packages installed) it only takes a minute
>> or less. But, on my "main" desktop system it takes 10-15 minutes every
>> time.  During the verify step, the emerge process is only using about
>> 5% of the CPU, and my system is running 80% or more idle.
>>
>
>
> I haven't timed mine yet but that sounds about like mine here.  I'm not
> sure what the bottleneck is but I have a four core AMD CPU running at
> 3.2GHz with 16GBs of ram and SATA spinning rust drives.  While I'm glad
> to have the added security measures, it does add a significant amount of
> time to the update process, the tree not the compile part.  We all know
> the compile part can get big.  lol
>
> I guess like everything else, we'll just have to get used to it.  People
> will hack a ham sandwich if they can and can get something from it.
> That would be mustard on mine.  Some may like Mayo, which is fine too.
> ;-)
>

Run a program with `strace -c` to get statistics on time spent in
system calls. It will be disk IO.

Re: [gentoo-user] How long does "Verifying /usr/portage" take?

[Dale]

R0b0t1 wrote:
> On Fri, Jul 6, 2018 at 2:16 PM, Dale <rdalek1967@gmail.com> wrote:
>> Grant Edwards wrote:
>>> Now that the public key stuff is working again (knock on wood), I'm
>>> curious if it's usual for an emerge --sync to take 10-15 minutes
>>> longer than it used due to the "Verifying /usr/portage" step.
>>>
>>> On some systems (with fewer packages installed) it only takes a minute
>>> or less. But, on my "main" desktop system it takes 10-15 minutes every
>>> time.  During the verify step, the emerge process is only using about
>>> 5% of the CPU, and my system is running 80% or more idle.
>>>
>>
>> I haven't timed mine yet but that sounds about like mine here.  I'm not
>> sure what the bottleneck is but I have a four core AMD CPU running at
>> 3.2GHz with 16GBs of ram and SATA spinning rust drives.  While I'm glad
>> to have the added security measures, it does add a significant amount of
>> time to the update process, the tree not the compile part.  We all know
>> the compile part can get big.  lol
>>
>> I guess like everything else, we'll just have to get used to it.  People
>> will hack a ham sandwich if they can and can get something from it.
>> That would be mustard on mine.  Some may like Mayo, which is fine too.
>> ;-)
>>
> Run a program with `strace -c` to get statistics on time spent in
> system calls. It will be disk IO.
>
>


I was thinking my drive light was on a lot during that time but wasn't
sure.  I used to go to the kitchen and get something to drink and a
snack and it be ready when I came back.  I guess now I can cook a light
meal and come back and it be ready.  Maybe I will lose a few pounds
because of this, looking for something positive in this besides the
obvious security improvements.  :? 

Either way, it takes longer but given the status of hackers, we really
need this.  It seems github sort of shined a light on what can happen
even if it is noticed pretty quick. 

Dale

:-)  :-) 

Re: [gentoo-user] How long does "Verifying /usr/portage" take?

[Rich Freeman]

On Fri, Jul 6, 2018 at 3:02 PM Grant Edwards <grant.b.edwards@gmail.com> wrote:
>
> Now that the public key stuff is working again (knock on wood), I'm
> curious if it's usual for an emerge --sync to take 10-15 minutes
> longer than it used due to the "Verifying /usr/portage" step.
>

Again, the sync mechanisms are different, but I note that the git
verify is nearly instant.

Of course, the only thing being fed to gpg in the git case is the git
commit record itself, which is about 10 lines of text.  That record
contains a content hash of the tree record, which in turn references
content hashes of every directory inside, and so on.  So, with git
most of the hash validation is happening constantly just by virtue of
everything being content-hashed, and the only extra layer with the gpg
signature is to sign the top level of the whole tree.

Now, on the flip side, some of those git operations might take time
since it is stating files for things like git status/etc.  I've never
noticed an issue, but I'm also on an SSD and tend to always have a
warm cache when I'm using it.  emerge shouldn't need to trigger any
git operations except when syncing.

--
Rich



--
Rich

[gentoo-user] Re: How long does "Verifying /usr/portage" take?

[Grant Edwards]

On 2018-07-06, Grant Edwards <grant.b.edwards@gmail.com> wrote:

> Now that the public key stuff is working again (knock on wood), I'm
> curious if it's usual for an emerge --sync to take 10-15 minutes
> longer than it used due to the "Verifying /usr/portage" step.
>
> On some systems (with fewer packages installed) it only takes a minute
> or less. But, on my "main" desktop system it takes 10-15 minutes every
> time.

I cleared out /usr/portage/distfiles, and the verify time dropped to
about 10 seconds.  I should probably do that more often...

-- 
Grant Edwards               grant.b.edwards        Yow! Hello, GORRY-O!!
                                  at               I'm a GENIUS from HARVARD!!
                              gmail.com            

Re: [gentoo-user] Re: How long does "Verifying /usr/portage" take?

[Mick]

[-- Attachment #1: Type: text/plain, Size: 843 bytes --]

On Friday, 6 July 2018 21:43:35 BST Grant Edwards wrote:
> On 2018-07-06, Grant Edwards <grant.b.edwards@gmail.com> wrote:
> > Now that the public key stuff is working again (knock on wood), I'm
> > curious if it's usual for an emerge --sync to take 10-15 minutes
> > longer than it used due to the "Verifying /usr/portage" step.
> > 
> > On some systems (with fewer packages installed) it only takes a minute
> > or less. But, on my "main" desktop system it takes 10-15 minutes every
> > time.
> 
> I cleared out /usr/portage/distfiles, and the verify time dropped to
> about 10 seconds.  I should probably do that more often...

This is odd.  Why would a verification of portage include the distfiles, when 
the latter are checked before they are unpacked as a package is being emerged.  
It doesn't make sense to me.  :-/

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Re: How long does "Verifying /usr/portage" take?

[Rich Freeman]

On Fri, Jul 6, 2018 at 4:43 PM Grant Edwards <grant.b.edwards@gmail.com> wrote:
>
> On 2018-07-06, Grant Edwards <grant.b.edwards@gmail.com> wrote:
>
> > Now that the public key stuff is working again (knock on wood), I'm
> > curious if it's usual for an emerge --sync to take 10-15 minutes
> > longer than it used due to the "Verifying /usr/portage" step.
> >
> > On some systems (with fewer packages installed) it only takes a minute
> > or less. But, on my "main" desktop system it takes 10-15 minutes every
> > time.
>
> I cleared out /usr/portage/distfiles, and the verify time dropped to
> about 10 seconds.  I should probably do that more often...
>

Assuming it is reproducible it is probably a bug.

That said, I always move distfiles to someplace like /var/cache.  I
guess /usr/portage should probably be in there as well, though I would
not mix my distfiles with my repository for a number of reasons.  I
think it is just inertia preserving the current situation as I can't
imagine anybody involved in portage/council/etc really would design it
this way today.

You can tweak this in make.conf with DISTDIR=...

-- 
Rich

Re: [gentoo-user] Re: How long does "Verifying /usr/portage" take?

[Dale]

Rich Freeman wrote:
> On Fri, Jul 6, 2018 at 4:43 PM Grant Edwards <grant.b.edwards@gmail.com> wrote:
>> On 2018-07-06, Grant Edwards <grant.b.edwards@gmail.com> wrote:
>>
>>> Now that the public key stuff is working again (knock on wood), I'm
>>> curious if it's usual for an emerge --sync to take 10-15 minutes
>>> longer than it used due to the "Verifying /usr/portage" step.
>>>
>>> On some systems (with fewer packages installed) it only takes a minute
>>> or less. But, on my "main" desktop system it takes 10-15 minutes every
>>> time.
>> I cleared out /usr/portage/distfiles, and the verify time dropped to
>> about 10 seconds.  I should probably do that more often...
>>
> Assuming it is reproducible it is probably a bug.
>
> That said, I always move distfiles to someplace like /var/cache.  I
> guess /usr/portage should probably be in there as well, though I would
> not mix my distfiles with my repository for a number of reasons.  I
> think it is just inertia preserving the current situation as I can't
> imagine anybody involved in portage/council/etc really would design it
> this way today.
>
> You can tweak this in make.conf with DISTDIR=...
>


I set mine up this way and it seems to work OK.  It's sort of along the
lines of yours. 

root@fireball / # ls /var/cache/portage/ -al
total 160
drwxr-xr-x   5 root    root      4096 Dec 20  2012 .
drwxr-xr-x  13 root    root      4096 Jul  4 03:26 ..
drwxrwxr-x   3 portage portage 143360 Jul  2 20:42 distfiles
drwxr-xr-x 103 portage portage   4096 Jul  3 00:01 packages
drwxr-xr-x 171 portage portage   4096 Jul  2 18:22 tree
root@fireball / # 


If anyone wants to duplicate this, this is the relevant parts of make.conf:


DISTDIR="/var/cache/portage/distfiles/"
PKGDIR="/var/cache/portage/packages"
PORTDIR="/var/cache/portage/tree" 


After I did the move, I think someone came up with a better place but to
be blunt, I just didn't feel like moving it all again.  It's not like
portage really cares where it is anyway. 

Dale

:-)  :-) 

[gentoo-user] Re: How long does "Verifying /usr/portage" take?

[Grant Edwards]

On 2018-07-06, Grant Edwards <grant.b.edwards@gmail.com> wrote:
> On 2018-07-06, Grant Edwards <grant.b.edwards@gmail.com> wrote:
>
>> Now that the public key stuff is working again (knock on wood), I'm
>> curious if it's usual for an emerge --sync to take 10-15 minutes
>> longer than it used due to the "Verifying /usr/portage" step.
>>
>> On some systems (with fewer packages installed) it only takes a minute
>> or less. But, on my "main" desktop system it takes 10-15 minutes every
>> time.
>
> I cleared out /usr/portage/distfiles, and the verify time dropped to
> about 10 seconds.  I should probably do that more often...

FWIW, I think that change was due to a warm buffer cache.  The next
few times I did a sync the verify was back to the usual 10+ minutes
(with /usr/portage/distfiles still empty).

-- 
Grant Edwards               grant.b.edwards        Yow! With YOU, I can be
                                  at               MYSELF ...  We don't NEED
                              gmail.com            Dan Rather ...