[gentoo-user] system lag with gentoo-sources-2.6.35-r2

[gentoo-user] system lag with gentoo-sources-2.6.35-r2

[Alan Warren]

[-- Attachment #1: Type: text/plain, Size: 857 bytes --]

Hello,

I am having some system performance issues with this kernel release. I have
a SMP machine (dual xeon nehalem 8 core / 16 threads) with 24gb non-ecc
memory.

On occasion (seems random so far) my system feels like a Pentium II trying
to cope with Vista. For example, I was in the middle of tar'ing a semi-large
file and noticed all of my apps came to a crawl. Scrolling in firefox,
typing in the terminal, or trying to navigate in my file manager resulted in
breif "pauses" that came in waves. On one occasion my system froze
completely and I had to manually reset the machine. (that was with
2.6.35-r1)

I didn't activate anything "new" in this kernel release that I don't
normally activate. ie, no cpuidle driver

Is there a proper venu for debugging such matters, or should I just wait for
this kernel to go prime-time?

Thanks for your time,
Alan

[-- Attachment #2: Type: text/html, Size: 934 bytes --]

Re: [gentoo-user] system lag with gentoo-sources-2.6.35-r2

[Robert Bridge]

On Sun, Aug 22, 2010 at 8:01 PM, Alan Warren <bluemoonshine@gmail.com> wrote:
> Is there a proper venu for debugging such matters, or should I just wait for
> this kernel to go prime-time?

Can you reliably reproduce the problem? If so, and you have a kernel
that works git-bisect should allow you to pinpoint the offending
commit.

By the sounds of it though, this could be related to the problems
Linux has when under heavy I/O, in which case you best bet would be to
look to the upstream git as there are supposed to be fixes in it.

Cheers,
RobbieAB.

Re: [gentoo-user] system lag with gentoo-sources-2.6.35-r2

[Alan Warren]

[-- Attachment #1: Type: text/plain, Size: 1360 bytes --]

Thanks,

I'm not very savvy when it comes to working with the kernel beyond
using the normal stable cut gentoo provides. I'll research git-bisect
and see if I can't figure this out though.

I think you are correct though. It does seem to only happen while the system
is under heavy I/O.

I've never experienced anything like this in previous versions of the linux
kernel,
and resorting back to gentoo-sources-2.6.34 fixes the issue completely.

If there are I/O fixes upstream, then I am assuming you are referring to a
cut
that is more recent then gentoo-sources-2.6.35-r2 that the Gentoo devs have
yet to provide their patches to?  ( I see vanilla sources has 2.6.35 .3)

Thanks again,
Alan


On Sun, Aug 22, 2010 at 2:14 PM, Robert Bridge <robert@robbieab.com> wrote:

> On Sun, Aug 22, 2010 at 8:01 PM, Alan Warren <bluemoonshine@gmail.com>
> wrote:
> > Is there a proper venu for debugging such matters, or should I just wait
> for
> > this kernel to go prime-time?
>
> Can you reliably reproduce the problem? If so, and you have a kernel
> that works git-bisect should allow you to pinpoint the offending
> commit.
>
> By the sounds of it though, this could be related to the problems
> Linux has when under heavy I/O, in which case you best bet would be to
> look to the upstream git as there are supposed to be fixes in it.
>
> Cheers,
> RobbieAB.
>
>

[-- Attachment #2: Type: text/html, Size: 1822 bytes --]

Re: [gentoo-user] system lag with gentoo-sources-2.6.35-r2

[Robert Bridge]

On Sun, Aug 22, 2010 at 8:35 PM, Alan Warren <bluemoonshine@gmail.com> wrote:
> I think you are correct though. It does seem to only happen while the system
> is under heavy I/O.
>
> I've never experienced anything like this in previous versions of the linux
> kernel,
> and resorting back to gentoo-sources-2.6.34 fixes the issue completely.
>
> If there are I/O fixes upstream, then I am assuming you are referring to a
> cut
> that is more recent then gentoo-sources-2.6.35-r2 that the Gentoo devs have
> yet to provide their patches to?  ( I see vanilla sources has 2.6.35 .3)

Well, the fix is in the line for 2.6.36 IIRC, so wouldn't be in an
2.6.35 kernel.

That said, the problem supposedly being fixed goes back well before
2.6.34, so if that kernel works, it suggests that it is a different
issue you are hitting. However... If there was a FF update, that could
be triggering the bug, as FF3.5+ are pretty stinky for I/O levels.

Re: [gentoo-user] system lag with gentoo-sources-2.6.35-r2

[Peter Humphrey]

On Sunday 22 August 2010 21:04:56 Robert Bridge wrote:

> Well, the fix is in the line for 2.6.36 IIRC, so wouldn't be in an
> 2.6.35 kernel.
> 
> That said, the problem supposedly being fixed goes back well before
> 2.6.34, so if that kernel works, it suggests that it is a different
> issue you are hitting.

Alan's problem sounds like the one I've been having through several 
kernel versions - the way I describe the symptom is that the system 
"lurches" through its work. It's an i5 box with 4 GB simple RAM (4 
cores, 8 threads).

I did mention this here a few weeks ago, but when I discovered that 
killing BOINC, or reducing its allowed CPU load, seemed to cure it I 
took no further action.

> However... If there was a FF update, that could be triggering the bug,
> as FF3.5+ are pretty stinky for I/O levels.

You may be onto something there. My genlop history of firefox goes back 
to January, when 3.5.6 was installed together with sys-kernel/gentoo-
sources-2.6.32-r1. I don't remember when I first noticed the lurching 
though.

-- 
Rgds
Peter.          Linux Counter 5290, 1994-04-23.

Re: [gentoo-user] system lag with gentoo-sources-2.6.35-r2

[Paul Hartman]

On Sun, Aug 22, 2010 at 2:01 PM, Alan Warren <bluemoonshine@gmail.com> wrote:
> Hello,
>
> I am having some system performance issues with this kernel release. I have
> a SMP machine (dual xeon nehalem 8 core / 16 threads) with 24gb non-ecc
> memory.
>
> On occasion (seems random so far) my system feels like a Pentium II trying
> to cope with Vista. For example, I was in the middle of tar'ing a semi-large
> file and noticed all of my apps came to a crawl. Scrolling in firefox,
> typing in the terminal, or trying to navigate in my file manager resulted in
> breif "pauses" that came in waves. On one occasion my system froze
> completely and I had to manually reset the machine. (that was with
> 2.6.35-r1)
>
> I didn't activate anything "new" in this kernel release that I don't
> normally activate. ie, no cpuidle driver

Sounds exactly like the same problem I was having. I posted about it
recently on this list. Downgrading to the latest 2.6.34 made
everything work normally again.

> Is there a proper venu for debugging such matters, or should I just wait for
> this kernel to go prime-time?

Well, 2.6.35 is already prime-time as far as it has been released and
is not RC...

The proper way to debug would be to do a git bisect of the kernel.org
sources until you find the exact patch that broke things. I haven't
had time to attempt such a feat yet, so I'm just waiting patiently for
someone else to figure it out instead. :)

Re: [gentoo-user] system lag with gentoo-sources-2.6.35-r2

[Mark Knecht]

On Sun, Aug 22, 2010 at 12:01 PM, Alan Warren <bluemoonshine@gmail.com> wrote:
> Hello,
>
> I am having some system performance issues with this kernel release. I have
> a SMP machine (dual xeon nehalem 8 core / 16 threads) with 24gb non-ecc
> memory.
>
> On occasion (seems random so far) my system feels like a Pentium II trying
> to cope with Vista. For example, I was in the middle of tar'ing a semi-large
> file and noticed all of my apps came to a crawl. Scrolling in firefox,
> typing in the terminal, or trying to navigate in my file manager resulted in
> breif "pauses" that came in waves. On one occasion my system froze
> completely and I had to manually reset the machine. (that was with
> 2.6.35-r1)
>
> I didn't activate anything "new" in this kernel release that I don't
> normally activate. ie, no cpuidle driver
>
> Is there a proper venu for debugging such matters, or should I just wait for
> this kernel to go prime-time?
>
> Thanks for your time,
> Alan
>

Hi Alan,
   Sorry for the problems. I've seen them also in the recent past. In
my case it was on new hardware so I couldn't say it was due to a
specific kernel release.

1) What happens when you watch top while doing the tar? Do you by any
chance see large wait times in top? (Hit '1' to watch all CPUs) If so
the problem could well be how the kernel is dealing with writing data
back to the hard drive. I had this problem with the WD Green drives.
When I changed to WD RAID Edition drives (1/2 the storage for 30% more
money) the problems disappeared.

2) If it's not the drive issue then there is a kernel option called (I
think) RCU_CPU_STALL_DETECTION (or something like that. If you turn it
on I may generate a trace of what's keeping a core busy to long.
Mileage will vary.

Good luck,
Mark

Re: [gentoo-user] system lag with gentoo-sources-2.6.35-r2

[Alan Warren]

[-- Attachment #1: Type: text/plain, Size: 2501 bytes --]

Thanks Mark, I'll look into that config option, and try again with top open.

In this case I was doing a home backup to a 1TB WD Caviar black formatted
as ext3.

I also have a raid0 with 2 other non-WD sata drives, and a single WD
velociraptor I can test
with.

It also doesn't sound too far off that FF could be the culprit (mentioned
above), as I have it
open all the time, and so far it's been the first place I've noticed these
hiccups. That could
be coincidence though, as I've pretty much always got it open and these
hiccups are
system wide.

On Mon, Aug 23, 2010 at 10:03 AM, Mark Knecht <markknecht@gmail.com> wrote:

> On Sun, Aug 22, 2010 at 12:01 PM, Alan Warren <bluemoonshine@gmail.com>
> wrote:
> > Hello,
> >
> > I am having some system performance issues with this kernel release. I
> have
> > a SMP machine (dual xeon nehalem 8 core / 16 threads) with 24gb non-ecc
> > memory.
> >
> > On occasion (seems random so far) my system feels like a Pentium II
> trying
> > to cope with Vista. For example, I was in the middle of tar'ing a
> semi-large
> > file and noticed all of my apps came to a crawl. Scrolling in firefox,
> > typing in the terminal, or trying to navigate in my file manager resulted
> in
> > breif "pauses" that came in waves. On one occasion my system froze
> > completely and I had to manually reset the machine. (that was with
> > 2.6.35-r1)
> >
> > I didn't activate anything "new" in this kernel release that I don't
> > normally activate. ie, no cpuidle driver
> >
> > Is there a proper venu for debugging such matters, or should I just wait
> for
> > this kernel to go prime-time?
> >
> > Thanks for your time,
> > Alan
> >
>
> Hi Alan,
>   Sorry for the problems. I've seen them also in the recent past. In
> my case it was on new hardware so I couldn't say it was due to a
> specific kernel release.
>
> 1) What happens when you watch top while doing the tar? Do you by any
> chance see large wait times in top? (Hit '1' to watch all CPUs) If so
> the problem could well be how the kernel is dealing with writing data
> back to the hard drive. I had this problem with the WD Green drives.
> When I changed to WD RAID Edition drives (1/2 the storage for 30% more
> money) the problems disappeared.
>
> 2) If it's not the drive issue then there is a kernel option called (I
> think) RCU_CPU_STALL_DETECTION (or something like that. If you turn it
> on I may generate a trace of what's keeping a core busy to long.
> Mileage will vary.
>
> Good luck,
> Mark
>
>

[-- Attachment #2: Type: text/html, Size: 3201 bytes --]

Re: [gentoo-user] system lag with gentoo-sources-2.6.35-r2

[Chen Huan]

[-- Attachment #1: Type: text/plain, Size: 2729 bytes --]

I am the same problems with 2.6.35, now I downgrade to 2.6.34-r6, it is
normal till now

2010/8/23 Alan Warren <bluemoonshine@gmail.com>

> Thanks Mark, I'll look into that config option, and try again with top
> open.
>
> In this case I was doing a home backup to a 1TB WD Caviar black formatted
> as ext3.
>
> I also have a raid0 with 2 other non-WD sata drives, and a single WD
> velociraptor I can test
> with.
>
> It also doesn't sound too far off that FF could be the culprit (mentioned
> above), as I have it
> open all the time, and so far it's been the first place I've noticed these
> hiccups. That could
> be coincidence though, as I've pretty much always got it open and these
> hiccups are
> system wide.
>
>
> On Mon, Aug 23, 2010 at 10:03 AM, Mark Knecht <markknecht@gmail.com>wrote:
>
>> On Sun, Aug 22, 2010 at 12:01 PM, Alan Warren <bluemoonshine@gmail.com>
>> wrote:
>> > Hello,
>> >
>> > I am having some system performance issues with this kernel release. I
>> have
>> > a SMP machine (dual xeon nehalem 8 core / 16 threads) with 24gb non-ecc
>> > memory.
>> >
>> > On occasion (seems random so far) my system feels like a Pentium II
>> trying
>> > to cope with Vista. For example, I was in the middle of tar'ing a
>> semi-large
>> > file and noticed all of my apps came to a crawl. Scrolling in firefox,
>> > typing in the terminal, or trying to navigate in my file manager
>> resulted in
>> > breif "pauses" that came in waves. On one occasion my system froze
>> > completely and I had to manually reset the machine. (that was with
>> > 2.6.35-r1)
>> >
>> > I didn't activate anything "new" in this kernel release that I don't
>> > normally activate. ie, no cpuidle driver
>> >
>> > Is there a proper venu for debugging such matters, or should I just wait
>> for
>> > this kernel to go prime-time?
>> >
>> > Thanks for your time,
>> > Alan
>> >
>>
>> Hi Alan,
>>   Sorry for the problems. I've seen them also in the recent past. In
>> my case it was on new hardware so I couldn't say it was due to a
>> specific kernel release.
>>
>> 1) What happens when you watch top while doing the tar? Do you by any
>> chance see large wait times in top? (Hit '1' to watch all CPUs) If so
>> the problem could well be how the kernel is dealing with writing data
>> back to the hard drive. I had this problem with the WD Green drives.
>> When I changed to WD RAID Edition drives (1/2 the storage for 30% more
>> money) the problems disappeared.
>>
>> 2) If it's not the drive issue then there is a kernel option called (I
>> think) RCU_CPU_STALL_DETECTION (or something like that. If you turn it
>> on I may generate a trace of what's keeping a core busy to long.
>> Mileage will vary.
>>
>> Good luck,
>> Mark
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 3685 bytes --]