[gentoo-user] Mammoth emerge ...

[gentoo-user] Mammoth emerge ...

[Mick]

[-- Attachment #1: Type: text/plain, Size: 1242 bytes --]

OK, I know my laptop is quite old, or at least Intel thinks so, but emerging 
chromium is now taking *much* longer than it ever did:

     Tue Apr 24 11:55:49 2018 >>> www-client/chromium-66.0.3359.117
       merge time: 1 day, 16 minutes and 28 seconds.

I'm currently emerging chromium-66.0.3359.139, which I will be surprised if it 
takes less time.  Six months ago or so I went through a patch where a chromium 
emerge would chew up all of my 4G RAM and then start thrashing the swap 
partition (4.2G) endlessly.  At some point I had to set up a secondary swap 
space, because chromium would fail to emerge when swap ran out.  In addition I 
started limiting the number of jobs (max and average) to stop it consuming all 
my memory.

Since a couple of versions ago I no longer needed to use any of these manual 
interventions.  Now chromium emerges without consuming all my RAM and hardly 
any swap either.  However, as you can see above it takes more than a day(!) to 
complete.  In a comparison a quad-core AMD which compiles wholly in RAM, 
emerged the last version in  8 hours, 53 minutes and 41 seconds.

Is there anything I can do with the existing laptop and its limited resources 
to speed up chromium's emerge?

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Mammoth emerge ...

[Peter Humphrey]

On Thursday, 3 May 2018 15:58:44 BST Mick wrote:

> Is there anything I can do with the existing laptop and its limited
> resources to speed up chromium's emerge?

All I can suggest is to build a package in a chroot on a speedier machine and 
transfer it to the laptop. That's how I cope with a very slow 32-bit single-
core Atom box.

-- 
Regards,
Peter.

Re: [gentoo-user] Mammoth emerge ...

[Wols Lists]

On 03/05/18 16:22, Peter Humphrey wrote:
> On Thursday, 3 May 2018 15:58:44 BST Mick wrote:
> 
>> Is there anything I can do with the existing laptop and its limited
>> resources to speed up chromium's emerge?
> 
> All I can suggest is to build a package in a chroot on a speedier machine and 
> transfer it to the laptop. That's how I cope with a very slow 32-bit single-
> core Atom box.
> 
Is distcc still there? I set that up ages ago, and when it was working
it worked great. Trouble is, I think it got messed up in an emerge and I
never set it up again.

Cheers,
Wol

Re: [gentoo-user] Mammoth emerge ...

[Mick]

[-- Attachment #1: Type: text/plain, Size: 2528 bytes --]

On Sunday, 6 May 2018 11:18:40 BST Wols Lists wrote:
> On 03/05/18 16:22, Peter Humphrey wrote:
> > On Thursday, 3 May 2018 15:58:44 BST Mick wrote:
> >> Is there anything I can do with the existing laptop and its limited
> >> resources to speed up chromium's emerge?
> > 
> > All I can suggest is to build a package in a chroot on a speedier machine
> > and transfer it to the laptop. That's how I cope with a very slow 32-bit
> > single- core Atom box.
> 
> Is distcc still there? I set that up ages ago, and when it was working
> it worked great. Trouble is, I think it got messed up in an emerge and I
> never set it up again.
> 
> Cheers,
> Wol

Yes, it's still there - see below - although I've never used it.  For really 
slow PCs I emerge everything within a chroot on a fast PC and then rsync all 
binpkg over to the slow PC and emerge them locally.

 eix -l distcc
* sec-policy/selinux-distcc
     Available versions:  
       ~    2.20170204-r1
            2.20170204-r2
            2.20170204-r3
            2.20170204-r4
            2.20170805-r2
            2.20170805-r3
       ~    2.20170805-r4
            2.20180114-r1
       ~    2.20180114-r2
       **   9999
     Homepage:            https://wiki.gentoo.org/wiki/Project:SELinux
     Description:         SELinux policy for distcc

* sys-devel/distcc
     Available versions:  
            3.1-r10   ^t	[avahi gtk hardened ipv6 selinux xinetd 
PYTHON_TARGETS="python2_7"]	["python_targets_python2_7"]
            3.2_rc1-r4 ^t	[crossdev gnome gssapi gtk hardened ipv6 selinux 
xinetd zeroconf PYTHON_TARGETS="python2_7"]	["python_targets_python2_7"]
       ~    3.2_rc1-r5 ^t	[gnome gssapi gtk hardened ipv6 selinux xinetd 
zeroconf PYTHON_TARGETS="python2_7"]	["python_targets_python2_7"]
       [M]~ 3.3       ^t	[gnome gssapi gtk hardened ipv6 selinux xinetd 
zeroconf PYTHON_SINGLE_TARGET="python3_5 python3_6" PYTHON_TARGETS="python3_5 
python3_6"]	["^^ ( python_single_target_python3_5 
python_single_target_python3_6 ) python_single_target_python3_5? ( 
python_targets_python3_5 ) python_single_target_python3_6? ( 
python_targets_python3_6 )"] Michał Górny <mgorny@gentoo.org> (20 Mar 2018) 
Poorly tested version bump followed by a series of quick hacks that do not 
make it any more working. Bug #651030.
     Homepage:            http://distcc.org/
     Description:         Distribute compilation of C code across several 
machines on a network

Found 2 matches

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[gentoo-user] Re: Mammoth emerge ...

[Grant Edwards]

On 2018-05-03, Mick <michaelkintzios@gmail.com> wrote:

> OK, I know my laptop is quite old, or at least Intel thinks so, but emerging 
> chromium is now taking *much* longer than it ever did:

A while back I accidentally broke the CPU throttling on my laptop, and
it was always running at 1/4 speed.  [That effected all emerges.]  I'd
start by doing a "grep MHz /proc/cpuinfo" during the build to verify
that all the CPUs have open throttles.

That said, I've noticed that Chromium in particular is taking a lot
longer than it used to...

-- 
Grant Edwards               grant.b.edwards        Yow! I have a very good
                                  at               DENTAL PLAN.  Thank you.
                              gmail.com            

Re: [gentoo-user] Re: Mammoth emerge ...

[Mick]

[-- Attachment #1: Type: text/plain, Size: 2877 bytes --]

On Thursday, 3 May 2018 16:41:55 BST Grant Edwards wrote:
> On 2018-05-03, Mick <michaelkintzios@gmail.com> wrote:
> > OK, I know my laptop is quite old, or at least Intel thinks so, but
> > emerging
> > chromium is now taking *much* longer than it ever did:
> A while back I accidentally broke the CPU throttling on my laptop, and
> it was always running at 1/4 speed.  [That effected all emerges.]  I'd
> start by doing a "grep MHz /proc/cpuinfo" during the build to verify
> that all the CPUs have open throttles.
> 
> That said, I've noticed that Chromium in particular is taking a lot
> longer than it used to...

Thank you Peter and Grant for your replies.  I have once or twice used a 
faster machine to build a Chromium binary and then copied over to the slow 
laptop, but it is a bit of a faff.  I may have to do this in the future, 
although with the landscape changing (GPZ patches, gcc upgrade, etc.) the 
build times are changing all the time.  They may never reduce again, in which 
case it will be an option to consider.

I don't think I have an issue with CPU throttling.  i7z shows the CPU with 
turbo regularly goes up to 2.6GHz:

=================================
Cpu speed from cpuinfo 1595.00Mhz
cpuinfo might be wrong if cpufreq is enabled. To guess correctly try 
estimating via tsc
Linux's inbuilt cpu_khz code emulated now
True Frequency (without accounting Turbo) 1595 MHz
  CPU Multiplier 12x || Bus clock frequency (BCLK) 132.92 MHz

Socket [0] - [physical cores=4, logical cores=8, max online cores ever=4]
  TURBO ENABLED on 4 Cores, Hyper Threading ON
  Max Frequency without considering Turbo 1727.92 MHz (132.92 x [13])
  Max TURBO Multiplier (if Enabled) with 1/2/3/4 Cores is  21x/18x/13x/13x
  Real Current Frequency 2244.06 MHz [132.92 x 16.88] (Max of below)
        Core [core-id]  :Actual Freq (Mult.)	  C0%   Halt(C1)%  C3 %   C6 %   
C7 %  Temp	 VCore
        Core 1 [0]:	  1454.95 (10.95x)	4.76    11.2    54.6    29.9	   0    
69	0.0000
        Core 2 [1]:	  1634.49 (12.30x)	3.09     4.5    52.6    39.7	   0    
69	0.0000
        Core 3 [2]:	  2244.06 (16.88x)	99.5	   0	   0       0	   0    72	
0.0000
        Core 4 [3]:	  2235.68 (16.82x)	99.4       0       0       0	   0    
69	0.0000



C0 = Processor running without halting
C1 = Processor running with halts (States >C0 are power saver modes with cores 
idling)
C3 = Cores running with PLL turned off and core cache turned off
C6, C7 = Everything in C3 + core state saved to last level cache, C7 is deeper 
than C6
  Above values in table are in percentage over the last 1 sec
[core-id] refers to core-id number in /proc/cpuinfo
'Garbage Values' message printed when garbage values are read
=============================================================

/proc/cpuinfo shows a number of cores regularly @1600MHz while syslog does not 
report any problems.

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Re: Mammoth emerge ...

[Corbin Bird]

.
Chromium switched to 'clang++ v5.x' as its primary compiler.
Why?
The Chromium devs are using 'c++' features supported in gcc v8+.
.
So ... first compile run is with 'gcc' ... then Chromium is re-compiled
with 'clang++'.
That is what I am seeing ( console && log wise ).
2 Compile runs ... twice the time.
.
No gold linker setup on my system.
Just how is 'clang++' supposed to work with 'ld.bfd'?
.
As far as I can tell, all optimization depending on '-march= / -mtune= '
is still discarded, as well.
( clang / clang ++, does not seem to accept the '-march= / -mtune= / -O2
/ -pipe' switches either. )
.
Corbin

Re: [gentoo-user] Re: Mammoth emerge ...

[Mick]

[-- Attachment #1: Type: text/plain, Size: 951 bytes --]

On Thursday, 3 May 2018 18:00:45 BST Corbin Bird wrote:
> .
> Chromium switched to 'clang++ v5.x' as its primary compiler.
> Why?
> The Chromium devs are using 'c++' features supported in gcc v8+.
> .
> So ... first compile run is with 'gcc' ... then Chromium is re-compiled
> with 'clang++'.
> That is what I am seeing ( console && log wise ).
> 2 Compile runs ... twice the time.
> .
> No gold linker setup on my system.
> Just how is 'clang++' supposed to work with 'ld.bfd'?
> .
> As far as I can tell, all optimization depending on '-march= / -mtune= '
> is still discarded, as well.
> ( clang / clang ++, does not seem to accept the '-march= / -mtune= / -O2
> / -pipe' switches either. )
> .
> Corbin

Thanks Corbin, it makes sense.  Back in November build time jumped from 8 to 
20 hours on my system.  It's only gone further downhill since.  :-(

I will give USE="jumbo-build" a spin later to see what improvement I may get.

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Re: Mammoth emerge ...

[Mick]

[-- Attachment #1: Type: text/plain, Size: 795 bytes --]

On Thursday, 3 May 2018 18:22:01 BST Mick wrote:

> I will give USE="jumbo-build" a spin later to see what improvement I may
> get.


I rebuilt chromium with USE="jumbo-build" with amazing results:

     Fri May  4 12:37:06 2018 >>> www-client/chromium-66.0.3359.139
       merge time: 1 day, 41 minutes and 56 seconds.

     Sat May  5 18:28:05 2018 >>> www-client/chromium-66.0.3359.139
       merge time: 9 hours, 20 minutes and 1 second.


Inevitably the first build took place to a large extent overnight when the PC 
was not in use.  Otherwise during day time the PC saw similar usage, light 
browsing, emails.  The above result shows more than a 60% shorter emerge time!

I noticed slightly more swap was being used (c. 100M or so).

Thanks Neil for suggesting it.  :-)

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Re: Mammoth emerge ...

[Neil Bothwick]

On 3 May 2018 16:41:55 BST, Grant Edwards <grant.b.edwards@gmail.com> wrote:
>On 2018-05-03, Mick <michaelkintzios@gmail.com> wrote:
>
>> OK, I know my laptop is quite old, or at least Intel thinks so, but
>emerging 
>> chromium is now taking *much* longer than it ever did:
>
>A while back I accidentally broke the CPU throttling on my laptop, and
>it was always running at 1/4 speed.  [That effected all emerges.]  I'd
>start by doing a "grep MHz /proc/cpuinfo" during the build to verify
>that all the CPUs have open throttles.
>
>That said, I've noticed that Chromium in particular is taking a lot
>longer than it used to...

I had similar issues with swapping on an 8GB laptop.. Limiting the numbers of jobs helped but using jumbo-build more than halved the build time .

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [gentoo-user] Re: Mammoth emerge ...

[Mick]

[-- Attachment #1: Type: text/plain, Size: 1128 bytes --]

On Thursday, 3 May 2018 17:57:20 BST Neil Bothwick wrote:
> On 3 May 2018 16:41:55 BST, Grant Edwards <grant.b.edwards@gmail.com> wrote:
> >On 2018-05-03, Mick <michaelkintzios@gmail.com> wrote:
> >> OK, I know my laptop is quite old, or at least Intel thinks so, but
> >
> >emerging
> >
> >> chromium is now taking *much* longer than it ever did:
> >A while back I accidentally broke the CPU throttling on my laptop, and
> >it was always running at 1/4 speed.  [That effected all emerges.]  I'd
> >start by doing a "grep MHz /proc/cpuinfo" during the build to verify
> >that all the CPUs have open throttles.
> >
> >That said, I've noticed that Chromium in particular is taking a lot
> >longer than it used to...
> 
> I had similar issues with swapping on an 8GB laptop.. Limiting the numbers
> of jobs helped but using jumbo-build more than halved the build time .

Interesting ... I had made a mental note to try out USE="jumbo-build".  I'm 
not sure what pressure it will place on my limited resources, but the proof of 
the pudding ... 

I will give it a spin tomorrow, or whenever this emerge completes.

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]