[gentoo-server] Forthcoming 2.6.13 timer frequency changes & server performance

[gentoo-server] Forthcoming 2.6.13 timer frequency changes & server performance

[Kerin Millar]

Hi folks.

The forthcoming kernel will be introducing a rather visible (and
potentially important) change. Firstly, it will provide an option to
select the default timer frequency. Secondly, the default frequency
will now be 250HZ. For the record, the value used to be 100HZ and was
rather drastically changed to 1000HZ for 2.6. It will now be possible
to choose a HZ value of either 100, 250 or 1000.

The most interesting aspect of the timer is that it affects scheduling
granularity. That is, the minimum possible time for which a given
process is allowed to run before interruption. This is commonly known
as a "jiffy" and can be calculated by dividing 1000 by the HZ value.
So, the length of a jiffy in 2.6 at present is 1ms. In general, a
longer jiffy results in more throughput but with more latency whereas
a shorter jiffy results in less throughput (due to the overhead from
frequent interruption) but with a reduction in latency. In my view,
this is rather interesting in terms of the impact this can have on
server performance. The help text for the new kernel configuration
option explains further:

- "Allows the configuration of the timer frequency. It is customary to
have the timer interrupt run at 1000 HZ but 100 HZ may be more
beneficial for servers and NUMA systems that do not need to have a
fast response for user interaction and that may experience bus
contention and cacheline bounces as a result of timer interrupts. Note
that the timer interrupt occurs on each processor in an SMP
environment leading to NR_CPUS * HZ number of timer interrupts per
second."

The specific help text for the 100HZ option is as follows:

- "100 HZ is a typical choice for servers, SMP and NUMA systems with
lots of processors that may show reduced performance if too many timer
interrupts are occurring."

For 250HZ:

- "250 HZ is a good compromise choice allowing server performance
while also showing good interactive responsiveness even on SMP and
NUMA systems."

For 1000HZ:

- "1000 HZ is the preferred choice for desktop systems and other
systems requiring fast interactive responses to events."

Another interesting point that is not mentioned above is that reducing
the timer frequency can significantly help to reduce time drift
(particularly on "real" SMP systems where this effect may be even more
pronounced). The reason that I am posting is twofold:

* To notify everyone that this change is coming
* To provide a means for anyone interested to test the ramifications
of this newly acquired flexibility using a 2.6.12 kernel prior to the
eventual release of 2.6.13.

To that end I have grabbed the specific patch that adds the
configuration option which can be found here:

  http://www.recruit2recruit.net/kerframil/2.6.12-i386-selectable-hz.patch

This patch is taken directly from upstream (the option itself can be
found at the bottom of the "Processor type and features" menu).
Interestingly, the change has drawn attention to a few areas in the
kernel where the methods used for timing (such as delay loops) are not
ideal. So I have gone through all of the patches committed to the
mainline tree since 2.6.12.3 and collected the ones that are relevant
then aggregated them into one patch (the second one applies cleanly
against a gentoo-sources tree):

  http://www.recruit2recruit.net/kerframil/2.6.12-timing-fixes-rollup.patch
  http://www.recruit2recruit.net/kerframil/2.6.12-gentoo-r7-timing-fixes-rollup.patch

The individual patches are all very small and non-intrusive by nature.
They are certainly not critical in order to be able to change the
timer frequency but I recommend that they be used. For the curious,
some notes on the contents of the patch can be found here here:
http://www.recruit2recruit.net/kerframil/NOTES-2.6.12-timing-fixes-rollup

What I would like is for anyone who is interested to put an alternate
value (100 or 250) to the test (if they are in a position to be able
to do so) and to determine whether there is any clear performance
improvement with their workload. I switched my system (Compaq Proliant
ML370) to 100HZ 2 days ago and can at least confirm that it is stable
although I have not yet had the opportunity to perform any comparative
benchmarks. If in doubt, then 250 might be a good value to try as,
after all, that is going to be the default - at least for x86 ;)

Finally, this topic has generated a nice little flame war over on the
LKML (actually, it's quite an interesting thread if a little confusing
at points):

http://kerneltrap.org/node/5430
http://lkml.org/lkml/2005/7/8/259

Cheers,

--Kerin Millar

-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Forthcoming 2.6.13 timer frequency changes & server performance

[Bastian Balthazar Bux]

Kerin Millar wrote:
> Hi folks.
> 
> The forthcoming kernel will be introducing a rather visible (and
> potentially important) change. Firstly, it will provide an option to
> select the default timer frequency. Secondly, the default frequency
> will now be 250HZ. For the record, the value used to be 100HZ and was
> rather drastically changed to 1000HZ for 2.6. It will now be possible
> to choose a HZ value of either 100, 250 or 1000.
> 
> The most interesting aspect of the timer is that it affects scheduling
> granularity. That is, the minimum possible time for which a given
> process is allowed to run before interruption. This is commonly known
> as a "jiffy" and can be calculated by dividing 1000 by the HZ value.
> So, the length of a jiffy in 2.6 at present is 1ms. In general, a
> longer jiffy results in more throughput but with more latency whereas
> a shorter jiffy results in less throughput (due to the overhead from
> frequent interruption) but with a reduction in latency. In my view,
> this is rather interesting in terms of the impact this can have on
> server performance. The help text for the new kernel configuration
> option explains further:
> 
> - "Allows the configuration of the timer frequency. It is customary to
> have the timer interrupt run at 1000 HZ but 100 HZ may be more
> beneficial for servers and NUMA systems that do not need to have a
> fast response for user interaction and that may experience bus
> contention and cacheline bounces as a result of timer interrupts. Note
> that the timer interrupt occurs on each processor in an SMP
> environment leading to NR_CPUS * HZ number of timer interrupts per
> second."
> 
> The specific help text for the 100HZ option is as follows:
> 
> - "100 HZ is a typical choice for servers, SMP and NUMA systems with
> lots of processors that may show reduced performance if too many timer
> interrupts are occurring."
> 
> For 250HZ:
> 
> - "250 HZ is a good compromise choice allowing server performance
> while also showing good interactive responsiveness even on SMP and
> NUMA systems."
> 
> For 1000HZ:
> 
> - "1000 HZ is the preferred choice for desktop systems and other
> systems requiring fast interactive responses to events."
> 
> Another interesting point that is not mentioned above is that reducing
> the timer frequency can significantly help to reduce time drift
> (particularly on "real" SMP systems where this effect may be even more
> pronounced). The reason that I am posting is twofold:
> 
> * To notify everyone that this change is coming
> * To provide a means for anyone interested to test the ramifications
> of this newly acquired flexibility using a 2.6.12 kernel prior to the
> eventual release of 2.6.13.
> 
> To that end I have grabbed the specific patch that adds the
> configuration option which can be found here:
> 
>   http://www.recruit2recruit.net/kerframil/2.6.12-i386-selectable-hz.patch
> 
> This patch is taken directly from upstream (the option itself can be
> found at the bottom of the "Processor type and features" menu).
> Interestingly, the change has drawn attention to a few areas in the
> kernel where the methods used for timing (such as delay loops) are not
> ideal. So I have gone through all of the patches committed to the
> mainline tree since 2.6.12.3 and collected the ones that are relevant
> then aggregated them into one patch (the second one applies cleanly
> against a gentoo-sources tree):
> 
>   http://www.recruit2recruit.net/kerframil/2.6.12-timing-fixes-rollup.patch
>   http://www.recruit2recruit.net/kerframil/2.6.12-gentoo-r7-timing-fixes-rollup.patch
> 
> The individual patches are all very small and non-intrusive by nature.
> They are certainly not critical in order to be able to change the
> timer frequency but I recommend that they be used. For the curious,
> some notes on the contents of the patch can be found here here:
> http://www.recruit2recruit.net/kerframil/NOTES-2.6.12-timing-fixes-rollup
> 
> What I would like is for anyone who is interested to put an alternate
> value (100 or 250) to the test (if they are in a position to be able
> to do so) and to determine whether there is any clear performance
> improvement with their workload. I switched my system (Compaq Proliant
> ML370) to 100HZ 2 days ago and can at least confirm that it is stable
> although I have not yet had the opportunity to perform any comparative
> benchmarks. If in doubt, then 250 might be a good value to try as,
> after all, that is going to be the default - at least for x86 ;)
> 
> Finally, this topic has generated a nice little flame war over on the
> LKML (actually, it's quite an interesting thread if a little confusing
> at points):
> 
> http://kerneltrap.org/node/5430
> http://lkml.org/lkml/2005/7/8/259
> 
> Cheers,
> 
> --Kerin Millar
> 

Good idea to publicize this.

Another thing that should be mentioned is that the timer frequency for
2.4 kernels was(/is?) fixed to 100 Hz.

-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Forthcoming 2.6.13 timer frequency changes & server performance

[Maik Musall]

On Wed, Aug 03, 2005 at 01:58:41PM +0200, Bastian Balthazar Bux wrote:
> Kerin Millar wrote:
> > For the record, the value used to be 100HZ and was
> > rather drastically changed to 1000HZ for 2.6.
[...]
> 
> Another thing that should be mentioned is that the timer frequency for
> 2.4 kernels was(/is?) fixed to 100 Hz.

As you see, he did.

BTW, it's not necessary to repeat the whole message just to add one
sentence.  We all have the original. http://learn.to/quote

Regards
-- 
Maik Musall <maik@musall.de>
GPG public key 0x856861EB (keyserver: wwwkeys.de.pgp.net)
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Forthcoming 2.6.13 timer frequency changes & server performance

[Bastian Balthazar Bux]

Maik Musall wrote:
> On Wed, Aug 03, 2005 at 01:58:41PM +0200, Bastian Balthazar Bux wrote:
> 
>>Kerin Millar wrote:
>>
>>>For the record, the value used to be 100HZ and was
>>>rather drastically changed to 1000HZ for 2.6.
> 
> [...]
> 
>>Another thing that should be mentioned is that the timer frequency for
>>2.4 kernels was(/is?) fixed to 100 Hz.
> 
> 
> As you see, he did.
> 
> BTW, it's not necessary to repeat the whole message just to add one
> sentence.  We all have the original. http://learn.to/quote
> 
> Regards

I'll reread his message as first ;-)

-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Forthcoming 2.6.13 timer frequency changes & server performance

[peter.gantner]

Thanks for collecting the patches. Very handy.

Quoth Kerin Millar (on Wednesday, the 3rd of August):

> Hi folks.
>
> Another interesting point that is not mentioned above is that reducing
> the timer frequency can significantly help to reduce time drift
> (particularly on "real" SMP systems where this effect may be even more
> pronounced). The reason that I am posting is twofold:

What do you mean with "real" SMP?
Is that real as opposed to fake==Hyperthreading, or real as in "more 
than a handful of nodes"?

>  http://www.recruit2recruit.net/kerframil/2.6.12-timing-fixes-rollup.patch
>  http://www.recruit2recruit.net/kerframil/2.6.12-gentoo-r7-timing-fixes-rollup.patch

strangely enough, the gentoo patch applies cleanly to my 2.6.12.3 tree, 
the other does not.

> What I would like is for anyone who is interested to put an alternate
> value (100 or 250) to the test (if they are in a position to be able
> to do so) and to determine whether there is any clear performance
> improvement with their workload.

Will report from dual-AthlonXP 1800 system soon.

Cheers,
 	Peter G.
-- 
"I do not think the way you think I think."
     -- Kai, last of the Brunnen G
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Forthcoming 2.6.13 timer frequency changes & server performance

[Kerin Millar]

On 8/3/05, peter.gantner@stud.uni-graz.at
<peter.gantner@stud.uni-graz.at> wrote:
> 
> Thanks for collecting the patches. Very handy.

No problem. Interestingly, applying this "rollup" patch prevented
2.6.12-ck4 from crashing my rather nondescript desktop machine during
boot. Con Kolivas changed HZ from 1000 to 864 in this patchset and it
has been causing problems (I do not think that 864HZ is a convenient
value in practice). I'm not exactly certain as to how the crash is
rectified by the changes in this patch though.

> What do you mean with "real" SMP?
> Is that real as opposed to fake==Hyperthreading, or real as in "more
> than a handful of nodes"?

Hyperthreading does not count as far as I'm aware so yes, I
specifically meant that. But obviously the problem may be far worse
with a greater number of nodes.

> strangely enough, the gentoo patch applies cleanly to my 2.6.12.3 tree,
> the other does not.

Sorry about that; the name is a little misleading. The principal
difference between the patch that applies against a stock 2.6.12 and
the "gentoo" one is that the gentoo tree incorporates a patch that
modifies the tpm.h header. This patch was actually added in the
regular stable tree so really the patch that you are using ought to be
called "2.6.12.3-timing-fixes-rollup.patch".

> Will report from dual-AthlonXP 1800 system soon.

Thanks. Also, I forgot to mention this (and so does the help text for
that matter) but reducing HZ also reduces power consumption. In
particular, this may be observed on laptops. Laptops can be strange
beasts and reducing HZ can apparently solve other strange problems
with some machines such as audible noises when entering/leaving
power-saving states and interference with the sound output.

Cheers,

--Kerin Millar

-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Forthcoming 2.6.13 timer frequency changes & server performance

[peter.gantner]

Quoth peter.gantner@stud.uni-graz.at (on Wednesday, the 3rd of August):

>> http: //www.recruit2recruit.net/kerframil/2.6.12-timing-fixes-rollup.patch
>> http: //www.recruit2recruit.net/kerframil/2.6.12-gentoo-r7-timing-fixes-rollup.patch
>
>>  What I would like is for anyone who is interested to put an alternate
>>  value (100 or 250) to the test (if they are in a position to be able
>>  to do so) and to determine whether there is any clear performance
>>  improvement with their workload.
>
> Will report from dual-AthlonXP 1800 system soon.

Well, I've been running an 2.6.12.3 and .4 kernel with your patches on 
this machine, both compiled with HZ=250;

I have noticed no problems with multimedia apps (amarok, mplayer, 
tvtime) at all.

ntpd doesn't complain about timing problems either. I haven't tried 
without running it, but from past expierience timedrift without is 
enourmuous regardless of kernel version used.

Interactivity feels a bit faster, but that might be my imagination...

I am running without any pre-empt BTW, because from hearsay I know that 
might even decrease performance on SMP.

If you would like me to do any specific benchmarking let me know, I'll 
be happy to run a few.

Greets,
 	Peter G.

-- 
"I do not think the way you think I think."
     -- Kai, last of the Brunnen G
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Forthcoming 2.6.13 timer frequency changes & server performance

[Eduardo Tongson]

> The forthcoming kernel will be introducing a rather visible (and
> potentially important) change. Firstly, it will provide an option to
> select the default timer frequency. Secondly, the default frequency
> will now be 250HZ. For the record, the value used to be 100HZ and was
> rather drastically changed to 1000HZ for 2.6. It will now be possible
> to choose a HZ value of either 100, 250 or 1000.
> 

Although a non-issue to servers(?),
the change affects sleep call latencies.
<http://leaf.dragonflybsd.org/~ed/misc/nanosleep/>
a linux 2.6 @ 100hz is the same as netbsd @ 100hz


--ed 
Eduardo Tongson

-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] Forthcoming 2.6.13 timer frequency changes & server performance

[kashani]

Eduardo Tongson wrote:
> Although a non-issue to servers(?),
> the change affects sleep call latencies.
> <http://leaf.dragonflybsd.org/~ed/misc/nanosleep/>
> a linux 2.6 @ 100hz is the same as netbsd @ 100hz

It's my understanding that 100hz is best for servers under load. Less 
context switches whereas moving towards 1000 gives you a more responsive 
  system due to the smaller time slices.

http://kerneltrap.org/node/5411

kashani
-- 
gentoo-server@gentoo.org mailing list