[gentoo-embedded] lockups with hardened kernel

[gentoo-embedded] lockups with hardened kernel

[Natanael Copa]

I got a report from someone who experienced lockups when using
gentoo-embedded built system (2.6.14-hardened-r1)

He managed to reproduce it on 2 different computers by doing this:

install openssh

get an ip address using dhcpcd

# create a network stream
ssh root@otherbox "dd if=/dev/urandom" > /dev/null

wait a while
read something from cdrom.

Now the box locks up:

> Box locked up - no kernel panic, no messages, keyboard's locked up -
> no more activity on the net interface lights.

When he uses dropbear & udhcpcd it works?

> Looks like it may be openssh or dhcpcd related. ?!
> dropbear & udhcoc doing the same thing has been running for more than
> an hour, no problems.

I have not been able to reproduce it.

I wonder if anyone here have experienced something similar?

Only lockups I have experienced have been lockups after some time of no
activity at all. Booting with "noacpi" fixes that. (but i still dont
know why it happens)

The kernel config used are here:
http://tanael.org/Config-2.6.14-hardened-r1


-- 
Natanael Copa

-- 
gentoo-embedded@gentoo.org mailing list

Re: [gentoo-embedded] lockups with hardened kernel

[Mike Frysinger]

On Wednesday 04 January 2006 03:52, Natanael Copa wrote:
> I got a report from someone who experienced lockups when using
> gentoo-embedded built system (2.6.14-hardened-r1)

this is a hardened kernel issue rather than a userland issue ... you'd be 
better off e-mailing the gentoo-hardened list
-mike
-- 
gentoo-embedded@gentoo.org mailing list

Re: [gentoo-embedded] lockups with hardened kernel

[Natanael Copa]

On lør, 2006-01-07 at 00:44 -0500, Mike Frysinger wrote:
> On Wednesday 04 January 2006 03:52, Natanael Copa wrote:
> > I got a report from someone who experienced lockups when using
> > gentoo-embedded built system (2.6.14-hardened-r1)
> 
> this is a hardened kernel issue rather than a userland issue ... you'd be 
> better off e-mailing the gentoo-hardened list

Probably, but I think I found the reason. Whenever I build the kernel
with "optimize for size" it hangs after an hour with intense net load
and io load (I made a loop that dd if=/dev/cdrom)

When I disabled "Configure standard kernel features (for small
systems)" (gcc -Os) the problem magically went away.

So I think its a bug in gcc version 3.4.4 (Gentoo Hardened 3.4.4-r1,
ssp-3.4.4-1.0, pie-8.7.8)

Makes me nervous because I have built everything with -Os.

I can probably just report it on bugzilla directly?

> -mike

-- 
gentoo-embedded@gentoo.org mailing list

Re: [gentoo-embedded] lockups with hardened kernel

[Mike Frysinger]

On Saturday 07 January 2006 02:30, Natanael Copa wrote:
> On lør, 2006-01-07 at 00:44 -0500, Mike Frysinger wrote:
> > On Wednesday 04 January 2006 03:52, Natanael Copa wrote:
> > > I got a report from someone who experienced lockups when using
> > > gentoo-embedded built system (2.6.14-hardened-r1)
> >
> > this is a hardened kernel issue rather than a userland issue ... you'd be
> > better off e-mailing the gentoo-hardened list
>
> Probably, but I think I found the reason. Whenever I build the kernel
> with "optimize for size" it hangs after an hour with intense net load
> and io load (I made a loop that dd if=/dev/cdrom)
>
> When I disabled "Configure standard kernel features (for small
> systems)" (gcc -Os) the problem magically went away.

the kernel can be pretty sensitive to optimizations

> So I think its a bug in gcc version 3.4.4 (Gentoo Hardened 3.4.4-r1,
> ssp-3.4.4-1.0, pie-8.7.8)
>
> Makes me nervous because I have built everything with -Os.
>
> I can probably just report it on bugzilla directly?

yes, but you probably wont get much feedback :/
-mike

-- 
gentoo-embedded@gentoo.org mailing list

Re: [gentoo-embedded] lockups with hardened kernel

[Peter S. Mazinger]

On Sat, 7 Jan 2006, Mike Frysinger wrote:

> On Saturday 07 January 2006 02:30, Natanael Copa wrote:
> > On lør, 2006-01-07 at 00:44 -0500, Mike Frysinger wrote:
> > > On Wednesday 04 January 2006 03:52, Natanael Copa wrote:
> > > > I got a report from someone who experienced lockups when using
> > > > gentoo-embedded built system (2.6.14-hardened-r1)
> > >
> > > this is a hardened kernel issue rather than a userland issue ... you'd be
> > > better off e-mailing the gentoo-hardened list
> >
> > Probably, but I think I found the reason. Whenever I build the kernel
> > with "optimize for size" it hangs after an hour with intense net load
> > and io load (I made a loop that dd if=/dev/cdrom)
> >
> > When I disabled "Configure standard kernel features (for small
> > systems)" (gcc -Os) the problem magically went away.
> 
> the kernel can be pretty sensitive to optimizations

if kernel is built w/ gcc-3.4.4 and -Os (at least I had that for 2.4.x 
kernels) it "optimizes away" memcmp and the kernels have gotten a patch 
to overcome this, the same problem does not occur w/ gcc-3.3.x.

> > So I think its a bug in gcc version 3.4.4 (Gentoo Hardened 3.4.4-r1,
> > ssp-3.4.4-1.0, pie-8.7.8)
> >
> > Makes me nervous because I have built everything with -Os.

I have built everything w/ -Os and do not have trouble w/ it

Peter

-- 
Peter S. Mazinger <ps dot m at gmx dot net>           ID: 0xA5F059F2
Key fingerprint = 92A4 31E1 56BC 3D5A 2D08  BB6E C389 975E A5F0 59F2

-- 
gentoo-embedded@gentoo.org mailing list

Re: [gentoo-embedded] lockups with hardened kernel

[Douglas Campos]

[-- Attachment #1: Type: text/plain, Size: 2088 bytes --]

I think that we can have some problems with 3.4.4-r1

all python modules like pygresql, psycopg et all won't work, with
uclibc-0.9.27-r1, 0.9.28
when we'd with gcc-3.4.3-20050110 it worked well

python -m pgdb -i returns

python: cannot resolve 'error_message'

Perhaps is the size optimization that is doing something nasty to some
binaries?

HEEEEELP!

Cheers

Douglas Campos

On 1/9/06, Peter S. Mazinger <ps.m@gmx.net> wrote:
>
> On Sat, 7 Jan 2006, Mike Frysinger wrote:
>
> > On Saturday 07 January 2006 02:30, Natanael Copa wrote:
> > > On lør, 2006-01-07 at 00:44 -0500, Mike Frysinger wrote:
> > > > On Wednesday 04 January 2006 03:52, Natanael Copa wrote:
> > > > > I got a report from someone who experienced lockups when using
> > > > > gentoo-embedded built system (2.6.14-hardened-r1)
> > > >
> > > > this is a hardened kernel issue rather than a userland issue ...
> you'd be
> > > > better off e-mailing the gentoo-hardened list
> > >
> > > Probably, but I think I found the reason. Whenever I build the kernel
> > > with "optimize for size" it hangs after an hour with intense net load
> > > and io load (I made a loop that dd if=/dev/cdrom)
> > >
> > > When I disabled "Configure standard kernel features (for small
> > > systems)" (gcc -Os) the problem magically went away.
> >
> > the kernel can be pretty sensitive to optimizations
>
> if kernel is built w/ gcc-3.4.4 and -Os (at least I had that for 2.4.x
> kernels) it "optimizes away" memcmp and the kernels have gotten a patch
> to overcome this, the same problem does not occur w/ gcc-3.3.x.
>
> > > So I think its a bug in gcc version 3.4.4 (Gentoo Hardened 3.4.4-r1,
> > > ssp-3.4.4-1.0, pie-8.7.8)
> > >
> > > Makes me nervous because I have built everything with -Os.
>
> I have built everything w/ -Os and do not have trouble w/ it
>
> Peter
>
> --
> Peter S. Mazinger <ps dot m at gmx dot net>           ID: 0xA5F059F2
> Key fingerprint = 92A4 31E1 56BC 3D5A 2D08  BB6E C389 975E A5F0 59F2
>
> --
> gentoo-embedded@gentoo.org mailing list
>
>

[-- Attachment #2: Type: text/html, Size: 2853 bytes --]

Re: [gentoo-embedded] lockups with hardened kernel

[Natanael Copa]

On man, 2006-01-09 at 17:20 +0100, Peter S. Mazinger wrote:
> On Sat, 7 Jan 2006, Mike Frysinger wrote:
> 
> > On Saturday 07 January 2006 02:30, Natanael Copa wrote:
> > > On lør, 2006-01-07 at 00:44 -0500, Mike Frysinger wrote:
> > > > On Wednesday 04 January 2006 03:52, Natanael Copa wrote:
> > > > > I got a report from someone who experienced lockups when using
> > > > > gentoo-embedded built system (2.6.14-hardened-r1)

> > > When I disabled "Configure standard kernel features (for small
> > > systems)" (gcc -Os) the problem magically went away.
> > 
> > the kernel can be pretty sensitive to optimizations
> 
> if kernel is built w/ gcc-3.4.4 and -Os (at least I had that for 2.4.x 
> kernels) it "optimizes away" memcmp and the kernels have gotten a patch 
> to overcome this, the same problem does not occur w/ gcc-3.3.x.
> 
> > > So I think its a bug in gcc version 3.4.4 (Gentoo Hardened 3.4.4-r1,
> > > ssp-3.4.4-1.0, pie-8.7.8)
> > >
> > > Makes me nervous because I have built everything with -Os.
> 
> I have built everything w/ -Os and do not have trouble w/ it

I have researched this and found out that it is not the -Os itself that
causes this, but it is something else under "Configure standard kernel
features (for small systems)". I'm not sure of exactly what but I know
at least that it is not "Optimize for size". I have turned all options
under "Configure standard kernel features (for small systems)" except
"Load all symbols for debugging/kksymoops" (including optimize for size
(-Os)) and done the netload test all night. This morning the computer
was still up.  When I have an unstable kernel it hangs after an hour or
two.

I have tried to flip the "futex support" and that did not make any
difference (anyone who know how this affects uclibc based apps btw? Do I
need futex support?)

I think I have narrowed down the problem to one of those:
* BUG() support
* Enable full-sized data structures for core
* Enable eventpoll support

But exactly what, I dont know. If anyone here have had any problems with
disabling one of those, I'm very interested to hear.

Lots of bla bla here, I just thought it could be good for you guys to
know that turning off of those might make your kernel unstable, while
-Os should be ok with current gcc.


-- 
Natanael Copa

-- 
gentoo-embedded@gentoo.org mailing list

Re: [gentoo-embedded] lockups with hardened kernel

[Douglas Campos]

[-- Attachment #1: Type: text/plain, Size: 2801 bytes --]

I've found that using gcc-3.4.4-r1 made a lot of python modules going with
"unresolved symbols", even when they compiled shamelessly

I switched to gcc-3.3.6 and then everything went right!

Just some food for thought.

Cheers

Douglas


On 1/13/06, Natanael Copa <mlists@tanael.org> wrote:
>
> On man, 2006-01-09 at 17:20 +0100, Peter S. Mazinger wrote:
> > On Sat, 7 Jan 2006, Mike Frysinger wrote:
> >
> > > On Saturday 07 January 2006 02:30, Natanael Copa wrote:
> > > > On lør, 2006-01-07 at 00:44 -0500, Mike Frysinger wrote:
> > > > > On Wednesday 04 January 2006 03:52, Natanael Copa wrote:
> > > > > > I got a report from someone who experienced lockups when using
> > > > > > gentoo-embedded built system (2.6.14-hardened-r1)
>
> > > > When I disabled "Configure standard kernel features (for small
> > > > systems)" (gcc -Os) the problem magically went away.
> > >
> > > the kernel can be pretty sensitive to optimizations
> >
> > if kernel is built w/ gcc-3.4.4 and -Os (at least I had that for 2.4.x
> > kernels) it "optimizes away" memcmp and the kernels have gotten a patch
> > to overcome this, the same problem does not occur w/ gcc-3.3.x.
> >
> > > > So I think its a bug in gcc version 3.4.4 (Gentoo Hardened 3.4.4-r1,
> > > > ssp-3.4.4-1.0, pie-8.7.8)
> > > >
> > > > Makes me nervous because I have built everything with -Os.
> >
> > I have built everything w/ -Os and do not have trouble w/ it
>
> I have researched this and found out that it is not the -Os itself that
> causes this, but it is something else under "Configure standard kernel
> features (for small systems)". I'm not sure of exactly what but I know
> at least that it is not "Optimize for size". I have turned all options
> under "Configure standard kernel features (for small systems)" except
> "Load all symbols for debugging/kksymoops" (including optimize for size
> (-Os)) and done the netload test all night. This morning the computer
> was still up.  When I have an unstable kernel it hangs after an hour or
> two.
>
> I have tried to flip the "futex support" and that did not make any
> difference (anyone who know how this affects uclibc based apps btw? Do I
> need futex support?)
>
> I think I have narrowed down the problem to one of those:
> * BUG() support
> * Enable full-sized data structures for core
> * Enable eventpoll support
>
> But exactly what, I dont know. If anyone here have had any problems with
> disabling one of those, I'm very interested to hear.
>
> Lots of bla bla here, I just thought it could be good for you guys to
> know that turning off of those might make your kernel unstable, while
> -Os should be ok with current gcc.
>
>
> --
> Natanael Copa
>
> --
> gentoo-embedded@gentoo.org mailing list
>
>

[-- Attachment #2: Type: text/html, Size: 3503 bytes --]

Re: [gentoo-embedded] lockups with hardened kernel

[Dan C]

Douglas Campos wrote:
> I've found that using gcc-3.4.4-r1 made a lot of python modules going
> with "unresolved symbols", even when they compiled shamelessly
>  
> I switched to gcc-3.3.6 and then everything went right!
>  
> Just some food for thought.
>  
> Cheers
>  
> Douglas

This is very interesting. I am trying to test out whether this will fix
my problem ("unresolved symbol" in libsvn Python module) by emerging
gcc-3.3.6 but it's not happy.

I am using softfloat (my host type is armeb-softfloat-linux-uclibc), and
have successfully emerged gcc-3.4.4* in the past, but in gcc-3.3.6 I
(eventually) get a whole bunch of errors like:

/usr/armeb-softfloat-linux-uclibc/bin/ld: ERROR: ./crtbeginS.o uses FPA
instructions, whereas libgcc_s.so.1 does not
/usr/armeb-softfloat-linux-uclibc/bin/ld: ERROR: ./crtbeginS.o uses
hardware FP, whereas libgcc_s.so.1 uses software FP
/usr/armeb-softfloat-linux-uclibc/bin/ld: failed to merge target
specific data of file ./crtbeginS.o
...

Maybe this is a problem with the gcc-3.3.6 ebuild? Is there something I
can set in my CFLAGS for the gcc emerge that will tell it to use softfloat?

(All these problems have made me start to regret fooling around with
bizarre embedded arches ...! :-D)

-- 
Dan C <djc@djc.id.au>
-- 
gentoo-embedded@gentoo.org mailing list

Re: [gentoo-embedded] lockups with hardened kernel

[Dan C]

Dan C wrote:
> I am using softfloat (my host type is armeb-softfloat-linux-uclibc), and
> have successfully emerged gcc-3.4.4* in the past, but in gcc-3.3.6 I
> (eventually) get a whole bunch of errors like:
> 
> [snip]

Okay never mind, it looks like it's not possible (or rather, too
difficult for me!) to use softfloat with gcc-3.3.*. I will just use a
non-softfloat alternate root to try out gcc-3.3.6.

-- 
Dan C <djc@djc.id.au>
-- 
gentoo-embedded@gentoo.org mailing list

Re: [gentoo-embedded] lockups with hardened kernel

[Dan C]

Douglas Campos wrote:
> I've found that using gcc-3.4.4-r1 made a lot of python modules going
> with "unresolved symbols", even when they compiled shamelessly
>  
> I switched to gcc-3.3.6 and then everything went right!

Well I have had no such luck. :-(

I set up an alternate root and switched it to gcc-3.3.6, emerged
Subversion, but still got an unresolved symbol error with the Python
bindings. I updated to python 2.4 (since the stage only has 2.3), which
successfully compiled but now python2.4 itself dies with

 /usr/bin/python: can't resolve symbol 'dl_iterate_phdr'

I guess for some reason, gcc, python, and uclibc are not playing nicely
together, at least not on my system, and I lack the necessary skills to
figure out exactly what's going on ...

I'm really starting to wish that there were a big-endian ARM stage
available that uses regular old glibc, I expect that would eliminate all
these problems ...

-- 
Dan C <djc@djc.id.au>
-- 
gentoo-embedded@gentoo.org mailing list

Re: [gentoo-embedded] lockups with hardened kernel

[Douglas Campos]

[-- Attachment #1: Type: text/plain, Size: 1165 bytes --]

No luck too... Just reboot then go crying aloud

On 1/16/06, Dan C <djc@djc.id.au> wrote:
>
> Douglas Campos wrote:
> > I've found that using gcc-3.4.4-r1 made a lot of python modules going
> > with "unresolved symbols", even when they compiled shamelessly
> >
> > I switched to gcc-3.3.6 and then everything went right!
>
> Well I have had no such luck. :-(
>
> I set up an alternate root and switched it to gcc-3.3.6, emerged
> Subversion, but still got an unresolved symbol error with the Python
> bindings. I updated to python 2.4 (since the stage only has 2.3), which
> successfully compiled but now python2.4 itself dies with
>
> /usr/bin/python: can't resolve symbol 'dl_iterate_phdr'
>
> I guess for some reason, gcc, python, and uclibc are not playing nicely
> together, at least not on my system, and I lack the necessary skills to
> figure out exactly what's going on ...
>
> I'm really starting to wish that there were a big-endian ARM stage
> available that uses regular old glibc, I expect that would eliminate all
> these problems ...
>
> --
> Dan C <djc@djc.id.au>
> --
> gentoo-embedded@gentoo.org mailing list
>
>

[-- Attachment #2: Type: text/html, Size: 1565 bytes --]