[gentoo-user] Kernel compiles ... monitoring

[gentoo-user] Kernel compiles ... monitoring

[Harry]

How can one tell how far along a kernel compile is?  I can see the modules being
built in /var/log/genkernel.log 
   (Aside: Please, no hysteria about using genkernel)

But I'd like to know of some way to guesstimate how much of the process is
completed.  Is there a list the compile has generated and is following somewhere
under /usr/src/linux? Or some other way of knowing where the compile is in terms
of percentage completed?

[gentoo-user] Re: Kernel compiles ... monitoring

[Harry]

Harry <hputnam3 <at> gmail.com> writes:

> 
> How can one tell how far along a kernel compile is?  I can see the 
> modules being built in /var/log/genkernel.log 
>    (Aside: Please, no hysteria about using genkernel)
> 
> But I'd like to know of some way to guesstimate how much of the process is
> completed.  Is there a list the compile has generated and is following 
> somewhere under /usr/src/linux? Or some other way of knowing where the 
> compile is in terms of percentage completed?

Just reporting a further bit on this:
You might think that tailing the log output and grepping for the module names
in .config would be one way, and I suppose it is, but no directly because the
compile does not appear to follow .config all the time.

Seems to follow closely when there modules in a row, but then jumps to whole
different sections... so still not very helpful

Re: [gentoo-user] Re: Kernel compiles ... monitoring

[Jonas de Buhr]

hi harry!

don't take my word for it, but i don't think there is an easy solution
to this :)

cmake creates makefiles which show the percentage completed in a build
step. maybe you can look into how it does that and apply that to the
kernel?

/jonas

>Harry <hputnam3 <at> gmail.com> writes:
>
>> 
>> How can one tell how far along a kernel compile is?  I can see the 
>> modules being built in /var/log/genkernel.log 
>>    (Aside: Please, no hysteria about using genkernel)
>> 
>> But I'd like to know of some way to guesstimate how much of the
>> process is completed.  Is there a list the compile has generated and
>> is following somewhere under /usr/src/linux? Or some other way of
>> knowing where the compile is in terms of percentage completed?
>
>Just reporting a further bit on this:
>You might think that tailing the log output and grepping for the
>module names in .config would be one way, and I suppose it is, but no
>directly because the compile does not appear to follow .config all the
>time.
>
>Seems to follow closely when there modules in a row, but then jumps to
>whole different sections... so still not very helpful
>
>
>
>
>

Re: [gentoo-user] Kernel compiles ... monitoring

[Paul Hartman]

On Mon, Sep 26, 2011 at 10:59 AM, Harry <hputnam3@gmail.com> wrote:
> How can one tell how far along a kernel compile is?  I can see the modules being
> built in /var/log/genkernel.log
>   (Aside: Please, no hysteria about using genkernel)
>
> But I'd like to know of some way to guesstimate how much of the process is
> completed.  Is there a list the compile has generated and is following somewhere
> under /usr/src/linux? Or some other way of knowing where the compile is in terms
> of percentage completed?

All I can think of is: time it. Maybe you can create a wrapper script
to time it, record times (for successful builds only) and measure
progress based on estimated time remaining. Kind of like what genlop
does with emerge logs.

[gentoo-user] Re: Kernel compiles ... monitoring

[Harry]

Paul Hartman <paul.hartman+gentoo@gmail.com> writes:

> On Mon, Sep 26, 2011 at 10:59 AM, Harry <hputnam3@gmail.com> wrote:
>> How can one tell how far along a kernel compile is?  I can see the
>> modules being
>> built in /var/log/genkernel.log
>>   (Aside: Please, no hysteria about using genkernel)
>>
>> But I'd like to know of some way to guesstimate how much of the
>> process is completed.  Is there a list the compile has generated
>> and is following somewhere under /usr/src/linux? Or some other way
>> of knowing where the compile is in terms of percentage completed?
>
> All I can think of is: time it. Maybe you can create a wrapper
> script to time it, record times (for successful builds only) and
> measure progress based on estimated time remaining. Kind of like
> what genlop does with emerge logs.

Sounds like there is no real way unless as you say.  I've been
wrestling with kernel build after kernel build trying to get a new
install booted.  Many failures led me to finally resorting to
genkernel... but you may know already that is a full day of compile at
least.

But I'm seeing really massive times even on the trimmed down kernels
where I've set only known things I need.

This is happening in a chroot from sysrescueCD on an older P4 with 2G
ram.  But my god, I'm seeing hrs and hrs of compile time go by even on
a lean mean menuconfig produced .config.

Is this pretty normal?

Re: [gentoo-user] Re: Kernel compiles ... monitoring

[Brennan Shacklett]

[-- Attachment #1: Type: text/plain, Size: 164 bytes --]

>Is this pretty normal?

I would say no. I have a p3 system with 512 MB of ram that can make a
.config in less than a minute... and the kernel takes about an hour.

[-- Attachment #2: Type: text/html, Size: 181 bytes --]

Re: [gentoo-user] Re: Kernel compiles ... monitoring

[Alan McKinnon]

On Mon, 26 Sep 2011 18:04:24 -0500
Harry <hputnam3@gmail.com> wrote:

> Paul Hartman <paul.hartman+gentoo@gmail.com> writes:
> 
> > On Mon, Sep 26, 2011 at 10:59 AM, Harry <hputnam3@gmail.com> wrote:
> >> How can one tell how far along a kernel compile is?  I can see the
> >> modules being
> >> built in /var/log/genkernel.log
> >>   (Aside: Please, no hysteria about using genkernel)
> >>
> >> But I'd like to know of some way to guesstimate how much of the
> >> process is completed.  Is there a list the compile has generated
> >> and is following somewhere under /usr/src/linux? Or some other way
> >> of knowing where the compile is in terms of percentage completed?
> >
> > All I can think of is: time it. Maybe you can create a wrapper
> > script to time it, record times (for successful builds only) and
> > measure progress based on estimated time remaining. Kind of like
> > what genlop does with emerge logs.
> 
> Sounds like there is no real way unless as you say.  I've been
> wrestling with kernel build after kernel build trying to get a new
> install booted.  Many failures led me to finally resorting to
> genkernel... but you may know already that is a full day of compile at
> least.
> 
> But I'm seeing really massive times even on the trimmed down kernels
> where I've set only known things I need.
> 
> This is happening in a chroot from sysrescueCD on an older P4 with 2G
> ram.  But my god, I'm seeing hrs and hrs of compile time go by even on
> a lean mean menuconfig produced .config.
> 
> Is this pretty normal?

No, that's very abnormal even for a P4. Something is badly wrong.

I get this in a VM with a clean set of kernel sources:

# genlop -t gentoo-sources
 * sys-kernel/gentoo-sources

     Sun Sep  4 23:08:52 2011 >>> sys-kernel/gentoo-sources-2.6.39-r3
       merge time: 6 minutes and 48 seconds.

The VM host is a flashy Samsung knock-off of an Apple Air with a nice
cpu (Intel(R) Core(TM) i5-2537M CPU @ 1.40GHz) and SSD.

The guest that did the compile is 1 cpu, 1G RAM and /var/tmp/portage
on the SSD. It's VirtualBox and while the compile was running I had 3
other VMs running and one of them was Windows7. Kernel compiles should
be quick, that same VM takes 90 minutes to build gcc.


My old ancient desktop was sort-of comparable to your P4, with IDE
disks and 1.5G RAM. Kernel compiles there took about 30-40 minutes.
sysrescueCD *shouldn't* make a difference as the entire chroot fs I
presume will be on hard disk.

What do your basics performance tools like top and friends say? See
what swap usage looks like outside the chroot while the compile is
running - is it thrashing? What speed are you getting for the hard disk
from hdparm -t -T?

How long does it take to build other big packages like gcc, glibc, 
maybe even binutils?


-- 
Alan McKinnnon
alan.mckinnon@gmail.com

[gentoo-user] Re: Kernel compiles ... monitoring

[Harry Putnam]

Alan McKinnon <alan.mckinnon@gmail.com> writes:

> What do your basics performance tools like top and friends say? See
> what swap usage looks like outside the chroot while the compile is
> running - is it thrashing? What speed are you getting for the hard disk
> from hdparm -t -T?

Top shows CC taking about 50% or in that vacinity... nothing else of
note is running.  But man I've been at this for 3 days, or so.
Created at least 6 different kernels and none will get me
booted... either I get a panic and root cannot be mounted, or the
screen goes black shortly after grub screen... and nothing more
happens.

I've tried 3.0.4 and backed up to 2.6.39* ... all this on a machine
that has run gentoo for several yrs.  And I'm not a complete greenhorn
either. 

There must be some other source of problem I don't know about is all I
can think...

I'm on the verge of copying an OS off the livecd and using that
kernel... this is just ridiculous.

Re: [gentoo-user] Re: Kernel compiles ... monitoring

[Michael Mol]

On Mon, Sep 26, 2011 at 9:42 PM, Harry Putnam <reader@newsguy.com> wrote:
> Alan McKinnon <alan.mckinnon@gmail.com> writes:
>
>> What do your basics performance tools like top and friends say? See
>> what swap usage looks like outside the chroot while the compile is
>> running - is it thrashing? What speed are you getting for the hard disk
>> from hdparm -t -T?
>
> Top shows CC taking about 50% or in that vacinity... nothing else of
> note is running.  But man I've been at this for 3 days, or so.

There's just no way your system is behaving appropriately.

Have you ever used the tool 'htop'? It's quite handy for visualizing
CPU usage, including CPUs waiting for, e.g. disk I/O.

Also, when running "make bzImage" and "make modules", be sure to
supply a -j option.

On my system (quad-core phenom 9650) I measured the sweet spot as
being at -j8 while building mplayer on a tmpfs mount. Any higher or
lower led to an increase in time required for the jobs to complete.

Also, you might tail -f /var/log/syslog, or poke dmesg from time to
time, and run some smart tests; your disk might be on its last legs.
-- 
:wq

Re: [gentoo-user] Re: Kernel compiles ... monitoring

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 1200 bytes --]

On Sep 27, 2011 8:47 AM, "Harry Putnam" <reader@newsguy.com> wrote:
>
> Alan McKinnon <alan.mckinnon@gmail.com> writes:
>
> > What do your basics performance tools like top and friends say? See
> > what swap usage looks like outside the chroot while the compile is
> > running - is it thrashing? What speed are you getting for the hard disk
> > from hdparm -t -T?
>
> Top shows CC taking about 50% or in that vacinity... nothing else of
> note is running.  But man I've been at this for 3 days, or so.
> Created at least 6 different kernels and none will get me
> booted... either I get a panic and root cannot be mounted, or the
> screen goes black shortly after grub screen... and nothing more
> happens.
>
> I've tried 3.0.4 and backed up to 2.6.39* ... all this on a machine
> that has run gentoo for several yrs.  And I'm not a complete greenhorn
> either.
>
> There must be some other source of problem I don't know about is all I
> can think...
>
> I'm on the verge of copying an OS off the livecd and using that
> kernel... this is just ridiculous.
>

Weird.

A thought occurred: could there be failing hardware? Start with testing your
RAM, then your hard disk, then everything else.

Rgds,

[-- Attachment #2: Type: text/html, Size: 1582 bytes --]

Re: [gentoo-user] Re: Kernel compiles ... monitoring

[Paul Hartman]

On Mon, Sep 26, 2011 at 9:01 PM, Pandu Poluan <pandu@poluan.info> wrote:
>
> On Sep 27, 2011 8:47 AM, "Harry Putnam" <reader@newsguy.com> wrote:
>>
>> Alan McKinnon <alan.mckinnon@gmail.com> writes:
>>
>> > What do your basics performance tools like top and friends say? See
>> > what swap usage looks like outside the chroot while the compile is
>> > running - is it thrashing? What speed are you getting for the hard disk
>> > from hdparm -t -T?
>>
>> Top shows CC taking about 50% or in that vacinity... nothing else of
>> note is running.  But man I've been at this for 3 days, or so.
>> Created at least 6 different kernels and none will get me
>> booted... either I get a panic and root cannot be mounted, or the
>> screen goes black shortly after grub screen... and nothing more
>> happens.
>>
>> I've tried 3.0.4 and backed up to 2.6.39* ... all this on a machine
>> that has run gentoo for several yrs.  And I'm not a complete greenhorn
>> either.
>>
>> There must be some other source of problem I don't know about is all I
>> can think...
>>
>> I'm on the verge of copying an OS off the livecd and using that
>> kernel... this is just ridiculous.
>>
>
> Weird.
>
> A thought occurred: could there be failing hardware? Start with testing your
> RAM, then your hard disk, then everything else.

With speed that slow it reminds me of CPU cache gone bad

Re: [gentoo-user] Re: Kernel compiles ... monitoring

[Jonas de Buhr]

>Top shows CC taking about 50% or in that vacinity... nothing else of
>note is running.  But man I've been at this for 3 days, or so.
>Created at least 6 different kernels and none will get me
>booted... either I get a panic and root cannot be mounted, or the
>screen goes black shortly after grub screen... and nothing more
>happens.
>
>I've tried 3.0.4 and backed up to 2.6.39* ... all this on a machine
>that has run gentoo for several yrs.  And I'm not a complete greenhorn
>either. 
>
>There must be some other source of problem I don't know about is all I
>can think...

hardware problem? try memtest86 and a HD check with a (boot cd-)tool
from the vendor.

Re: [gentoo-user] Re: Kernel compiles ... monitoring

[Michael Orlitzky]

On 09/26/11 19:04, Harry wrote:
> 
> Sounds like there is no real way unless as you say.  I've been
> wrestling with kernel build after kernel build trying to get a new
> install booted.  Many failures led me to finally resorting to
> genkernel... but you may know already that is a full day of compile at
> least.
> 
> But I'm seeing really massive times even on the trimmed down kernels
> where I've set only known things I need.
> 
> This is happening in a chroot from sysrescueCD on an older P4 with 2G
> ram.  But my god, I'm seeing hrs and hrs of compile time go by even on
> a lean mean menuconfig produced .config.
> 
> Is this pretty normal?
> 

Don't forget, you have to pass "-j $JOBS" to make manually when building
the kernel. Using -j2 should speed *any* machine up. The optimal number
depends on your machine though.

Re: [gentoo-user] Re: Kernel compiles ... monitoring

[Paul Hartman]

On Mon, Sep 26, 2011 at 6:04 PM, Harry <hputnam3@gmail.com> wrote:
> Sounds like there is no real way unless as you say.  I've been
> wrestling with kernel build after kernel build trying to get a new
> install booted.  Many failures led me to finally resorting to
> genkernel... but you may know already that is a full day of compile at
> least.
>
> But I'm seeing really massive times even on the trimmed down kernels
> where I've set only known things I need.
>
> This is happening in a chroot from sysrescueCD on an older P4 with 2G
> ram.  But my god, I'm seeing hrs and hrs of compile time go by even on
> a lean mean menuconfig produced .config.
>
> Is this pretty normal?

No, I don't think that's normal... for that CPU it should be something
like 10-15 minutes maybe.