[gentoo-user] Advice on system monitoring

[gentoo-user] Advice on system monitoring

[Michael Mol]

I haven't yet needed to do this kind of system monitoring, so I'm very
much a newbie here.

Let's start with that dual-xeon box I was using to benchmark "emerge
-e @world", figure I'm looking for how better to tune my MAKEOPTS and
EMERGE_DEFAULT_OPTS variables, and assume I'd like to get more
information about the following factors:

* What was the 1m, 5m 15m load averages?
* What were the similar averages for CPU spent in user time, system
time and I/O wait?
* What was network usage like? (I have a caching proxy server on the
network, so even if distfiles are lost on-system, well, a cache hit
transfers at up to around 50MB/s. It'd be better, except for read
performance limitations on the router box, and write performance
limitations on the local machine)
* What was the temperature of each CPU core, RAM module and hard
drive? (Not so relevant for improving system performance, but still of
interest.)

I'd like to have a web interface I could navigate to which would show
graphs of these counters.

-- 
:wq

Re: [gentoo-user] Advice on system monitoring

[Stéphane Guedon]

[-- Attachment #1: Type: Text/Plain, Size: 1636 bytes --]

On Monday 05 December 2011 07:29:34 Michael Mol wrote:
> I haven't yet needed to do this kind of system monitoring, so I'm very
> much a newbie here.
> 
> Let's start with that dual-xeon box I was using to benchmark "emerge
> -e @world", figure I'm looking for how better to tune my MAKEOPTS and
> EMERGE_DEFAULT_OPTS variables, and assume I'd like to get more
> information about the following factors:
> 
> * What was the 1m, 5m 15m load averages?
> * What were the similar averages for CPU spent in user time, system
> time and I/O wait?
> * What was network usage like? (I have a caching proxy server on the
> network, so even if distfiles are lost on-system, well, a cache hit
> transfers at up to around 50MB/s. It'd be better, except for read
> performance limitations on the router box, and write performance
> limitations on the local machine)
> * What was the temperature of each CPU core, RAM module and hard
> drive? (Not so relevant for improving system performance, but still of
> interest.)
> 
> I'd like to have a web interface I could navigate to which would show
> graphs of these counters.

There are many web interface for that. You should look at munin, rrdtool, 
nagios, this kind of stuff.
I have set my own.
Have a look there : https://www.22decembre.eu/status/ (I have setup my own 
certificate authority for ssl).
If you need help, don't hesitate to contact me ! But you may find also better 
help around !

See you...

-- 
Stéphane Guedon
http://www.22decembre.eu/
http://lectures.22decembre.eu/
carte de visite : http://www.22decembre.eu/downloads/Stephane-Guedon.vcf

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

[gentoo-user] Re: Advice on system monitoring

[James]

Michael Mol <mikemol <at> gmail.com> writes:


> Let's start with that dual-xeon box I was using to benchmark "emerge
> -e @world", figure I'm looking for how better to tune my MAKEOPTS and
> EMERGE_DEFAULT_OPTS variables, and assume I'd like to get more
> information about the following factors:

Complex and never finished, imho.....


> * What was the 1m, 5m 15m load averages?
> * What were the similar averages for CPU spent in user time, system
> time and I/O wait?

sys-process/iotop

> * What was network usage like? (I have a caching proxy server on the
> network

Lots of different tools to look at network performance:

wireshark,  (look around /usr/portage/net-analyzer)


> so even if distfiles are lost on-system, well, a cache hit
> transfers at up to around 50MB/s. It'd be better, except for read
> performance limitations on the router box, and write performance
> limitations on the local machine)


bonnie++ (or bonnie)


> * What was the temperature of each CPU core, RAM module and hard
> drive? (Not so relevant for improving system performance, but still of
> interest.)

app-admin/hddtemp (for drives)

dunno on individual cpu cores...

> I'd like to have a web interface I could navigate to which would show
> graphs of these counters.


Now all of that in one gui tool?  Do post back when you get it working,
as I'd like to use it too!!!!!

hth,
James

Re: [gentoo-user] Advice on system monitoring

[kashani]

On 12/4/2011 10:29 PM, Michael Mol wrote:
> I haven't yet needed to do this kind of system monitoring, so I'm very
> much a newbie here.
>
> Let's start with that dual-xeon box I was using to benchmark "emerge
> -e @world", figure I'm looking for how better to tune my MAKEOPTS and
> EMERGE_DEFAULT_OPTS variables, and assume I'd like to get more
> information about the following factors:
>
> * What was the 1m, 5m 15m load averages?
> * What were the similar averages for CPU spent in user time, system
> time and I/O wait?
> * What was network usage like? (I have a caching proxy server on the
> network, so even if distfiles are lost on-system, well, a cache hit
> transfers at up to around 50MB/s. It'd be better, except for read
> performance limitations on the router box, and write performance
> limitations on the local machine)
> * What was the temperature of each CPU core, RAM module and hard
> drive? (Not so relevant for improving system performance, but still of
> interest.)
>
> I'd like to have a web interface I could navigate to which would show
> graphs of these counters.
>

Collectd might be interesting to you. It can collect all of these and 
write them out to rrd files. The frontend cgi script is a little lame, 
but you can try some of the other frontends. The emerge flags are ... 
extensive.

http://collectd.org/

kashani

Re: [gentoo-user] Re: Advice on system monitoring

[Michael Mol]

On Mon, Dec 5, 2011 at 12:01 PM, James <wireless@tampabay.rr.com> wrote:
> Michael Mol <mikemol <at> gmail.com> writes:
>> Let's start with that dual-xeon box I was using to benchmark "emerge
>> -e @world", figure I'm looking for how better to tune my MAKEOPTS and
>> EMERGE_DEFAULT_OPTS variables, and assume I'd like to get more
>> information about the following factors:
>
> Complex and never finished, imho.....
>
>
>> * What was the 1m, 5m 15m load averages?
>> * What were the similar averages for CPU spent in user time, system
>> time and I/O wait?
>
> sys-process/iotop
>
>> * What was network usage like? (I have a caching proxy server on the
>> network
>
> Lots of different tools to look at network performance:
>
> wireshark,  (look around /usr/portage/net-analyzer)
>
>
>> so even if distfiles are lost on-system, well, a cache hit
>> transfers at up to around 50MB/s. It'd be better, except for read
>> performance limitations on the router box, and write performance
>> limitations on the local machine)
>
>
> bonnie++ (or bonnie)
>
>
>> * What was the temperature of each CPU core, RAM module and hard
>> drive? (Not so relevant for improving system performance, but still of
>> interest.)
>
> app-admin/hddtemp (for drives)
>
> dunno on individual cpu cores...
>
>> I'd like to have a web interface I could navigate to which would show
>> graphs of these counters.
>
>
> Now all of that in one gui tool?  Do post back when you get it working,
> as I'd like to use it too!!!!!

The approach I'd like to take is to have all the monitoring set up,
launch emerge -e @world, and see what's going on around (and just
prior to) stalls and CPU waste. I'm defining a stall as where my
operating load falls below my number of CPU cores, and I'm defining
CPU waste as CPU time spent anywhere but 'user'. I'd like to look at
graphs of the metrics from over the course of the emerge.

My chief thought is this: I have both 'make' and 'emerge' trying to
reach a specific load average, which means that this particular
dynamic system is going to have feedback as they go back and forth. I
expect that I'll want to duck one of them under the other, but I don't
know which one yet, and I don't know how far.

I should also look to see if pbzip2 supports load awareness. Having
eight cores suddenly start churning through BWT blocks is great if
your load average is something like 0.24, but not so great if it
launches your load average up to around 12.

-- 
:wq