[gentoo-amd64] System load during emerge

[gentoo-amd64] System load during emerge

[Peter Humphrey]

I'm watching the emerge of mozilla-firefox with top (I've just set the java 
USE flag so that I can use some astronomy Web sites), and I'm puzzled. This 
is a dual-Opteron box with 4G memory, and I have it set up with 6G /tmp 
mounted on tmpfs. Top shows 0k of used swap, so I'm confident that I'm not 
doing any swapping. It should show me a total of up to 200% when the system 
is fully loaded.

But top shows 49 - 50% wa, which I understand is I/O waiting time (even with 
the X term rolled up), cc1plus is showing only single-digit percent CPU, 
and the System and User CPU % are in the teens. I can get X to use 30-odd % 
by leaving the terminal on display, or 1% by rolling it up; that seems not 
to affect the other figures. No other user processes are running; only the 
background processes started by init and so on. What is the machine doing? 
Why does it not work harder at its allotted task?

-- 
Rgds
Peter
-- 
gentoo-amd64@gentoo.org mailing list

[gentoo-amd64] Re: System load during emerge

[Duncan]

Peter Humphrey <prh@gotadsl.co.uk> posted
200703041151.37846.prh@gotadsl.co.uk, excerpted below, on  Sun, 04 Mar
2007 11:51:37 +0000:

> I'm watching the emerge of mozilla-firefox with top (I've just set the
> java USE flag so that I can use some astronomy Web sites), and I'm
> puzzled. This is a dual-Opteron box with 4G memory, and I have it set up
> with 6G /tmp mounted on tmpfs. Top shows 0k of used swap, so I'm
> confident that I'm not doing any swapping. It should show me a total of
> up to 200% when the system is fully loaded.
> 
> But top shows 49 - 50% wa, which I understand is I/O waiting time (even
> with the X term rolled up), cc1plus is showing only single-digit percent
> CPU, and the System and User CPU % are in the teens. I can get X to use
> 30-odd % by leaving the terminal on display, or 1% by rolling it up;
> that seems not to affect the other figures. No other user processes are
> running; only the background processes started by init and so on. What
> is the machine doing? Why does it not work harder at its allotted task?

Ouch!

There's a couple things to check.  First, you say /tmp is on tmpfs and in 
memory (no swap), but you don't mention where you have $PORTAGE_TMPDIR 
pointed.  That's set in make.conf and defaults to /var/tmp.  Here, /var/
tmp is a symlink pointing to /tmp so it doesn't really matter, but I have 
$PORTAGE_TMPDIR set to /tmp anyway.  You may also wish to tweak 
$PORTAGE_TMPFS (see the documentation in make.conf.example) and if you 
have FEATURES=buildpkg set, $PKG_TMPDIR (which I don't see documentation 
for ATM, but...).  If /tmp is tmpfs and it's of reasonable size (as yours 
is), there's little reason not to have all of them set to /tmp.

Second, for general disk access speed (IOW, not portage specific), if 
you've not checked it, emerge hdparm if necessary and see if your hard 
drives are using DMA.  If they aren't, you'll have MUCH slower disk i/o.  
Normally, if you are using the correct drivers, DMA should be activated 
by default, but it's not uncommon for folks to be using the generic 
drivers or some other specific hardware drivers that don't fully match 
the disk controller chipset they actually have, thus forcing the kernel 
to use compatibility mode.  Less commonly, a driver won't activate DMA 
even if it's the right one, or will, but at a lower speed (say UDMA66 
rather than UDMA133) than both the drive and the chipset can handle.  Of 
course, don't attempt to activate a higher UDMA mode than both drive and 
chipset support, or you'll risk data corruption.

Of course, also note that depending on the ebuild stage and how much of 
what it is accessing you already have cached, portage does do some non-
tmp I/O access.  In particular, it has to read the portage tree to 
discover dependencies, then distdir to see if the tarball is there (and 
if not fetch it), then untar it to $PORTAGE_TMPDIR and read and apply any 
patches.  At the beginning of the compile step it does the configure, 
which accesses and tests for all sorts of system binaries, so until those 
are cached (generally after the first thing using a configure script is 
emerged), there will be some disk I/O reading those.  If you have CCACHE 
set up, of course it must write the compiles to that as it goes, but 
unless your drives aren't using DMA, that shouldn't be a big issue.  
Finally, at the package (if FEATURES=buildpkg) and qmerge steps, it'll 
write to the pkg dir and to the general system as it installs.

There's also portage logging and associated log variables.  Again, this 
shouldn't be a big deal if you've got DMA on your drives, but it might be 
if you don't.

Finally, if it's writing anything to NFS mounts or the like over the 
network, or if you have distcc active, it's possible the wait is on the 
network, not on your local hard drives, of course depending on exactly 
how you are configured.


-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: System load during emerge

[Peter Humphrey]

On Sunday 04 March 2007 16:09:03 Duncan wrote:
> Peter Humphrey <prh@gotadsl.co.uk> posted
> 200703041151.37846.prh@gotadsl.co.uk, excerpted below, on  Sun, 04 Mar
>
> > What is the machine doing?
>
> There's a couple things to check.  First, you say /tmp is on tmpfs and in
> memory (no swap), but you don't mention where you have $PORTAGE_TMPDIR
> pointed. 

It's not long since we discussed this. It's set to /tmp, and so is 
PKG_TMPDIR.

> Second, for general disk access speed (IOW, not portage specific), if
> you've not checked it, emerge hdparm if necessary and see if your hard
> drives are using DMA.  If they aren't, you'll have MUCH slower disk i/o.

Hdparm is installed and does set DMA on. The only disk access I can foresee 
portage doing is to write its log file. Could that really consume as much 
time as I'm seeing? Seems unlikely to me.

[Later...] I've now found that if I run "hdparm -d1 /dev/sda" from an X 
terminal I get "HDIO_SET_DMA failed: Inappropriate ioctl for device", but 
when /etc/init.d/hdparm is run during boot I see [ok] for each drive. Looks 
like I've got some exploring to do.

> Of course, also note that depending on the ebuild stage and how much of
> what it is accessing you already have cached, portage does do some non-
> tmp I/O access. 

As I said, I was watching firefox being compiled. All the preliminary stages 
had been completed by then. All the source code should have been in memory, 
but I can't say the same for any other libraries. On the other hand, just 
how much time is needed to load all the development libraries needed for 
this compilation?

> There's also portage logging and associated log variables.  Again, this
> shouldn't be a big deal if you've got DMA on your drives, but it might be
> if you don't.

Yes, that's the only thing I can think of.

> Finally, if it's writing anything to NFS mounts or the like over the
> network, or if you have distcc active, it's possible the wait is on the
> network, not on your local hard drives, of course depending on exactly
> how you are configured.

I did play with distcc but found it unreliable on a large package, so I've 
removed it from make.conf. I don't use NFS or any other networked 
operations that I can think of.

Thanks for the suggestions.

-- 
Rgds
Peter
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: System load during emerge

[dustin]

On Sun, Mar 04, 2007 at 05:35:16PM +0000, Peter Humphrey wrote:
> As I said, I was watching firefox being compiled. All the preliminary stages 
> had been completed by then. All the source code should have been in memory, 
> but I can't say the same for any other libraries. On the other hand, just 
> how much time is needed to load all the development libraries needed for 
> this compilation?

Not necessarily - gcc pulls in quite a range of system headers, which
may be cached by the kernel less aggressively than the other stuff
you're keeping in RAM.

I'd recommend running iostat (in the sysstat ebuild) to see at least
what kind of IO is going on here.  You may also want to experiment with
remounting e.g., /usr/include on a tmpfs.

Dustin
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] System load during emerge

[Thomas Rösner]

Hi,

Peter Humphrey schrieb:
> I'm watching the emerge of mozilla-firefox with top (I've just set the java 
> USE flag so that I can use some astronomy Web sites), and I'm puzzled. This 
> is a dual-Opteron box with 4G memory, and I have it set up with 6G /tmp 
> mounted on tmpfs. Top shows 0k of used swap, so I'm confident that I'm not 
> doing any swapping. It should show me a total of up to 200% when the system 
> is fully loaded.
>
> But top shows 49 - 50% wa, which I understand is I/O waiting time (even with 
> the X term rolled up), cc1plus is showing only single-digit percent CPU, 
> and the System and User CPU % are in the teens. I can get X to use 30-odd % 
> by leaving the terminal on display, or 1% by rolling it up; that seems not 
> to affect the other figures. No other user processes are running; only the 
> background processes started by init and so on. What is the machine doing? 
> Why does it not work harder at its allotted task?

What are your MAKEOPTS set to? Are you sure firefox can be built in 
parallel? Depending on how long lived the g++ processes are, top showing 
low CPU percentage for them is quite normal.

If nothing helps, try compiling inside a screen session.

Regards,
    Thomas

-- 
gentoo-amd64@gentoo.org mailing list

[gentoo-amd64] Re: System load during emerge

[Bernhard Auzinger]

> [Later...] I've now found that if I run "hdparm -d1 /dev/sda" from an X
> terminal I get "HDIO_SET_DMA failed: Inappropriate ioctl for device", but
> when /etc/init.d/hdparm is run during boot I see [ok] for each drive. Looks
> like I've got some exploring to do.

As far as I know "hdparm -d1" is not available, because sata drives work with 
libata (within the scsi subsystem) which uses dma for all drives.

rgds
Bernhard
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] System load during emerge

[Peter Humphrey]

On Sunday 04 March 2007 18:29:55 Thomas Rösner wrote:
> What are your MAKEOPTS set to?

-j5, to give one active and one waiting on each CPU, with one just in case.

> Are you sure firefox can be built in parallel?

I can't say I am. It does appear to launch a batch of compilations together, 
but then, later, top only shows one compiler instance at a time.

> Depending on how long lived the g++ processes are, top showing 
> low CPU percentage for them is quite normal.

I don't understand the connection with long life.

> If nothing helps, try compiling inside a screen session.

Hmm. I haven't used screen yet, not having had a need so far as I know.

Thanks for the ideas.

-- 
Rgds
Peter Humphrey
Linux Counter 5290, Aug 93
--
gentoo-amd64@gentoo.org mailing list