[gentoo-hardened] PAX bug?

[gentoo-hardened] PAX bug?

[Alex Efros]

Hi!

Today I found server nearly unresponsible (loadavg around 30, ssh type speed
around few chars per second). It looks like nearly all processes (very
different ones) eat each 3-5% CPU, with top's report about 95% CPU spend
in "system" (i.e. not "user" or "wait"). At a glance it looks like kernel
issue, so I checked kernel error log and found this, reported few hours ago:

2010-05-12_03:51:29.90675 kern.err: PAX: refcount overflow detected in: fastcgi:32201, uid/euid: 1067/1067
2010-05-12_03:51:29.93807 kern.err: PAX: refcount overflow occured at: iret_exc+0x1d3e/0x4565
2010-05-12_03:51:29.93813 kern.warn:
2010-05-12_03:51:29.94129 kern.warn: Pid: 32201, comm: fastcgi Tainted: G        W  (2.6.28-hardened-r9 #1) ProLiant DL140 G3
2010-05-12_03:51:29.94137 kern.warn: EIP: 0060:[<c06d95ee>] EFLAGS: 00000a96 CPU: 1
2010-05-12_03:51:29.94140 kern.warn: EIP is at iret_exc+0x1d3e/0x4565
2010-05-12_03:51:29.94143 kern.warn: EAX: 00000004 EBX: f756b080 ECX: df037a54 EDX: 00000070
2010-05-12_03:51:29.94145 kern.warn: ESI: f6ce1c00 EDI: df03781c EBP: df0377fc ESP: df0377a8
2010-05-12_03:51:29.94147 kern.warn:  DS: 0068 ES: 0068 FS: 00d8 GS: 0033 SS: 0068
2010-05-12_03:51:29.94150 <0>Process fastcgi (pid: 32201, ti=df036000 task=c3c46e10 task.ti=df036000)
2010-05-12_03:51:29.94152 <0>Stack:
2010-05-12_03:51:29.94154 kern.warn:  c04d35fb 00000000 00000000 00000000 00000000 df0377fc c04d39f7 00000000
2010-05-12_03:51:29.94156 <0> 000001f8 00000000 df037a54 df037940 00000004 00000001 000001f8 00000000
2010-05-12_03:51:29.94159 <0> 00000000 00000070 00000000 df037a80 effd8380 df0379d8 c04df769 00000070

The fastcgi process mentioned in report is perl script (but it uses C
libraries libev and libadns for accessing epoll(2) and doing async dns
resolving (it's event-based fastcgi implementation, not usual forked one).

Server was rebooted, now everything is fine. Server software is nearly
up-to-date x86 Gentoo (last update was 2-3 weeks ago), kernel is
sys-kernel/hardened-sources-2.6.28-r9.

-- 
			WBR, Alex.

Re: [gentoo-hardened] PAX bug?

[Javier Juan Martínez Cabezón]

Why do you think is a PaX bug? It seems that PaX REFCOUNT is doing his
homeworks.

Maybe I'm wrong (to the boss, please correct me) but seems that the
bug is in the perl fastcgi script.

The wrong fix to this is disabling PaX_REFCOUNT in your .config that
is nothing mode than disabling a security PaX feature.



2010/5/12 Alex Efros <powerman@powerman.name>:
> Hi!
>
> Today I found server nearly unresponsible (loadavg around 30, ssh type speed
> around few chars per second). It looks like nearly all processes (very
> different ones) eat each 3-5% CPU, with top's report about 95% CPU spend
> in "system" (i.e. not "user" or "wait"). At a glance it looks like kernel
> issue, so I checked kernel error log and found this, reported few hours ago:
>
> 2010-05-12_03:51:29.90675 kern.err: PAX: refcount overflow detected in: fastcgi:32201, uid/euid: 1067/1067
> 2010-05-12_03:51:29.93807 kern.err: PAX: refcount overflow occured at: iret_exc+0x1d3e/0x4565
> 2010-05-12_03:51:29.93813 kern.warn:
> 2010-05-12_03:51:29.94129 kern.warn: Pid: 32201, comm: fastcgi Tainted: G        W  (2.6.28-hardened-r9 #1) ProLiant DL140 G3
> 2010-05-12_03:51:29.94137 kern.warn: EIP: 0060:[<c06d95ee>] EFLAGS: 00000a96 CPU: 1
> 2010-05-12_03:51:29.94140 kern.warn: EIP is at iret_exc+0x1d3e/0x4565
> 2010-05-12_03:51:29.94143 kern.warn: EAX: 00000004 EBX: f756b080 ECX: df037a54 EDX: 00000070
> 2010-05-12_03:51:29.94145 kern.warn: ESI: f6ce1c00 EDI: df03781c EBP: df0377fc ESP: df0377a8
> 2010-05-12_03:51:29.94147 kern.warn:  DS: 0068 ES: 0068 FS: 00d8 GS: 0033 SS: 0068
> 2010-05-12_03:51:29.94150 <0>Process fastcgi (pid: 32201, ti=df036000 task=c3c46e10 task.ti=df036000)
> 2010-05-12_03:51:29.94152 <0>Stack:
> 2010-05-12_03:51:29.94154 kern.warn:  c04d35fb 00000000 00000000 00000000 00000000 df0377fc c04d39f7 00000000
> 2010-05-12_03:51:29.94156 <0> 000001f8 00000000 df037a54 df037940 00000004 00000001 000001f8 00000000
> 2010-05-12_03:51:29.94159 <0> 00000000 00000070 00000000 df037a80 effd8380 df0379d8 c04df769 00000070
>
> The fastcgi process mentioned in report is perl script (but it uses C
> libraries libev and libadns for accessing epoll(2) and doing async dns
> resolving (it's event-based fastcgi implementation, not usual forked one).
>
> Server was rebooted, now everything is fine. Server software is nearly
> up-to-date x86 Gentoo (last update was 2-3 weeks ago), kernel is
> sys-kernel/hardened-sources-2.6.28-r9.
>
> --
>                        WBR, Alex.
>
>

Re: [gentoo-hardened] PAX bug?

[Javier Juan Martínez Cabezón]

PD: Or probably in the main kernel too

El día 13 de mayo de 2010 21:10,
> Maybe I'm wrong (to the boss, please correct me) but seems that the
> bug is in the perl fastcgi script.

Re: [gentoo-hardened] PAX bug?

[Alex Efros]

Hi!

On Thu, May 13, 2010 at 09:10:47PM +0200, Javier Juan Martínez Cabezón wrote:
> Why do you think is a PaX bug? It seems that PaX REFCOUNT is doing his
> homeworks.

I've no idea is it PaX bug - that's why subject line is "PaX bug?" instead
of "PaX bug!!!". :)

> Maybe I'm wrong (to the boss, please correct me) but seems that the
> bug is in the perl fastcgi script.

It's hard for me to imagine a way to implement perl script which will
trigger anything in PaX, and especially things like refcount overflow..

> PD: Or probably in the main kernel too

Maybe. In any case it's likely bug somewhere in kernel, and not in perl -
because _any_ possible perl script shouldn't be able to put server in such
unusable state.

-- 
			WBR, Alex.

Re: [gentoo-hardened] PAX bug?

[pageexec]

On 13 May 2010 at 0:55, Alex Efros wrote:

> Server was rebooted, now everything is fine. Server software is nearly
> up-to-date x86 Gentoo (last update was 2-3 weeks ago), kernel is
> sys-kernel/hardened-sources-2.6.28-r9.

i'd need the vmlinux image to tell for sure but it's most likely a false positive
that has been fixed since in later kernels, so please try to use something we actually
support (.32 or .33, soon .34), not .28.

Re: [gentoo-hardened] PAX bug?

[David Sommerseth]

On 15/05/10 02:15, pageexec@freemail.hu wrote:
> On 13 May 2010 at 0:55, Alex Efros wrote:
> 
>> Server was rebooted, now everything is fine. Server software is nearly
>> up-to-date x86 Gentoo (last update was 2-3 weeks ago), kernel is
>> sys-kernel/hardened-sources-2.6.28-r9.
> 
> i'd need the vmlinux image to tell for sure but it's most likely a false positive
> that has been fixed since in later kernels, so please try to use something we actually
> support (.32 or .33, soon .34), not .28.

I'm sorry for probably being quite upset now.  But in regards to the
supported version, that sounds like utter non-sense to me.  I just
updated the portage tree ... and this is what is available there:

$ find /usr/portage/sys-kernel/hardened-sources
/usr/portage/sys-kernel/hardened-sources
/usr/portage/sys-kernel/hardened-sources/metadata.xml
/usr/portage/sys-kernel/hardened-sources/Manifest
/usr/portage/sys-kernel/hardened-sources/ChangeLog
/usr/portage/sys-kernel/hardened-sources/hardened-sources-2.6.25-r13.ebuild
/usr/portage/sys-kernel/hardened-sources/hardened-sources-2.6.26-r9.ebuild
/usr/portage/sys-kernel/hardened-sources/hardened-sources-2.6.28-r9.ebuild
/usr/portage/sys-kernel/hardened-sources/hardened-sources-2.6.29.ebuild
/usr/portage/sys-kernel/hardened-sources/hardened-sources-2.6.28-r7.ebuild

I simply do not see that .32, .33 nor any .34 kernels have been made
available nor marked as stable.  And I am not using a fixed mirror in
/etc/make.conf.

This is once again a repetition of the a similar discussion a few weeks
ago, where several of us raised the concern about an outdated hardened
stable kernel.

I might have missed some obvious information, but I cannot see anywhere
among the hardened project documentation pages that we should expect to
find the hardened kernels anywhere else.

<http://www.gentoo.org/doc/en/?catid=project>
<http://www.gentoo.org/proj/en/hardened/>

We *seriously* need to get this clarified now.  There might even be a
lot of users who don't visit the hardened IRC channel or the mailing
list - and they are most probably running a .28-r9 kernel, which is the
latest stable kernel - at least how I can understand it ... this
situation is making me uncomfortable and quite worried now!


kind regards,

David Sommerseth

Re: [gentoo-hardened] PAX bug?

[pageexec]

On 15 May 2010 at 10:46, David Sommerseth wrote:

> On 15/05/10 02:15, pageexec@freemail.hu wrote:
> > i'd need the vmlinux image to tell for sure but it's most likely a false positive
> > that has been fixed since in later kernels, so please try to use something we actually
> > support (.32 or .33, soon .34), not .28.
> 
> I'm sorry for probably being quite upset now.  But in regards to the
> supported version, that sounds like utter non-sense to me.  I just
> updated the portage tree ... and this is what is available there:

i think you misunderstood me. when i talked about support, i wasn't talking
about hardened-gentoo, i was talking about PaX/grsecurity only (myself being
the upstream ;). and yes, that means that if the hardened gentoo maintainers
cannot provide anything newer than .28/.29 themselves for whatever reasons,
then you'll have to roll your own kernels if you expect *us* (PaX/grsec devs)
to support you. this is simply because we have only so much time and desire
for backporting fixes to older kernels (and it's always been the case, we never
supported more than one kernel version before, .32 stable support is a new thing
for us).

> We *seriously* need to get this clarified now.  There might even be a
> lot of users who don't visit the hardened IRC channel or the mailing
> list - and they are most probably running a .28-r9 kernel, which is the
> latest stable kernel - at least how I can understand it ... this
> situation is making me uncomfortable and quite worried now!

i certainly agree that .28 stable is not a good idea when the rest of the world
has decided on at least .32 for a longer term stable kernel, but i'm not the guy
to have this discussion with, you should ask the hardened maintainers, if there's
anyone still left active.

Re: [gentoo-hardened] PAX bug?

[Constantine Kardaris]

[-- Attachment #1: Type: text/plain, Size: 2636 bytes --]

add "anarchy" overlay
http://git.overlays.gentoo.org/gitweb/?p=dev/anarchy.git;a=tree;f=sys-kernel/hardened-sources;h=398bb516fac60966b6f83d9335f4b530cc0e7406;hb=HEAD

On Sat, May 15, 2010 at 11:46 AM, David Sommerseth <
gentoo.list@topphemmelig.net> wrote:

> On 15/05/10 02:15, pageexec@freemail.hu wrote:
> > On 13 May 2010 at 0:55, Alex Efros wrote:
> >
> >> Server was rebooted, now everything is fine. Server software is nearly
> >> up-to-date x86 Gentoo (last update was 2-3 weeks ago), kernel is
> >> sys-kernel/hardened-sources-2.6.28-r9.
> >
> > i'd need the vmlinux image to tell for sure but it's most likely a false
> positive
> > that has been fixed since in later kernels, so please try to use
> something we actually
> > support (.32 or .33, soon .34), not .28.
>
> I'm sorry for probably being quite upset now.  But in regards to the
> supported version, that sounds like utter non-sense to me.  I just
> updated the portage tree ... and this is what is available there:
>
> $ find /usr/portage/sys-kernel/hardened-sources
> /usr/portage/sys-kernel/hardened-sources
> /usr/portage/sys-kernel/hardened-sources/metadata.xml
> /usr/portage/sys-kernel/hardened-sources/Manifest
> /usr/portage/sys-kernel/hardened-sources/ChangeLog
> /usr/portage/sys-kernel/hardened-sources/hardened-sources-2.6.25-r13.ebuild
> /usr/portage/sys-kernel/hardened-sources/hardened-sources-2.6.26-r9.ebuild
> /usr/portage/sys-kernel/hardened-sources/hardened-sources-2.6.28-r9.ebuild
> /usr/portage/sys-kernel/hardened-sources/hardened-sources-2.6.29.ebuild
> /usr/portage/sys-kernel/hardened-sources/hardened-sources-2.6.28-r7.ebuild
>
> I simply do not see that .32, .33 nor any .34 kernels have been made
> available nor marked as stable.  And I am not using a fixed mirror in
> /etc/make.conf.
>
> This is once again a repetition of the a similar discussion a few weeks
> ago, where several of us raised the concern about an outdated hardened
> stable kernel.
>
> I might have missed some obvious information, but I cannot see anywhere
> among the hardened project documentation pages that we should expect to
> find the hardened kernels anywhere else.
>
> <http://www.gentoo.org/doc/en/?catid=project>
> <http://www.gentoo.org/proj/en/hardened/>
>
> We *seriously* need to get this clarified now.  There might even be a
> lot of users who don't visit the hardened IRC channel or the mailing
> list - and they are most probably running a .28-r9 kernel, which is the
> latest stable kernel - at least how I can understand it ... this
> situation is making me uncomfortable and quite worried now!
>
>
> kind regards,
>
> David Sommerseth
>
>

[-- Attachment #2: Type: text/html, Size: 3588 bytes --]

Re: [gentoo-hardened] PAX bug?

["Tóth Attila"]

or:
http://lmgtfy.com/?q=hardened-sources-2.6.33
-- 
dr Tóth Attila, Radiológus, 06-20-825-8057, 06-30-5962-962
Attila Toth MD, Radiologist, +36-20-825-8057, +36-30-5962-962

2010.Május 15.(Szo) 11:37 időpontban Constantine Kardaris ezt írta:
> add "anarchy" overlay
> http://git.overlays.gentoo.org/gitweb/?p=dev/anarchy.git;a=tree;f=sys-kernel/hardened-sources;h=398bb516fac60966b6f83d9335f4b530cc0e7406;hb=HEAD
>
> On Sat, May 15, 2010 at 11:46 AM, David Sommerseth <
> gentoo.list@topphemmelig.net> wrote:
>
>> On 15/05/10 02:15, pageexec@freemail.hu wrote:
>> > On 13 May 2010 at 0:55, Alex Efros wrote:
>> >
>> >> Server was rebooted, now everything is fine. Server software is
>> nearly
>> >> up-to-date x86 Gentoo (last update was 2-3 weeks ago), kernel is
>> >> sys-kernel/hardened-sources-2.6.28-r9.
>> >
>> > i'd need the vmlinux image to tell for sure but it's most likely a
>> false
>> positive
>> > that has been fixed since in later kernels, so please try to use
>> something we actually
>> > support (.32 or .33, soon .34), not .28.
>>
>> I'm sorry for probably being quite upset now.  But in regards to the
>> supported version, that sounds like utter non-sense to me.  I just
>> updated the portage tree ... and this is what is available there:
>>
>> $ find /usr/portage/sys-kernel/hardened-sources
>> /usr/portage/sys-kernel/hardened-sources
>> /usr/portage/sys-kernel/hardened-sources/metadata.xml
>> /usr/portage/sys-kernel/hardened-sources/Manifest
>> /usr/portage/sys-kernel/hardened-sources/ChangeLog
>> /usr/portage/sys-kernel/hardened-sources/hardened-sources-2.6.25-r13.ebuild
>> /usr/portage/sys-kernel/hardened-sources/hardened-sources-2.6.26-r9.ebuild
>> /usr/portage/sys-kernel/hardened-sources/hardened-sources-2.6.28-r9.ebuild
>> /usr/portage/sys-kernel/hardened-sources/hardened-sources-2.6.29.ebuild
>> /usr/portage/sys-kernel/hardened-sources/hardened-sources-2.6.28-r7.ebuild
>>
>> I simply do not see that .32, .33 nor any .34 kernels have been made
>> available nor marked as stable.  And I am not using a fixed mirror in
>> /etc/make.conf.
>>
>> This is once again a repetition of the a similar discussion a few weeks
>> ago, where several of us raised the concern about an outdated hardened
>> stable kernel.
>>
>> I might have missed some obvious information, but I cannot see anywhere
>> among the hardened project documentation pages that we should expect to
>> find the hardened kernels anywhere else.
>>
>> <http://www.gentoo.org/doc/en/?catid=project>
>> <http://www.gentoo.org/proj/en/hardened/>
>>
>> We *seriously* need to get this clarified now.  There might even be a
>> lot of users who don't visit the hardened IRC channel or the mailing
>> list - and they are most probably running a .28-r9 kernel, which is the
>> latest stable kernel - at least how I can understand it ... this
>> situation is making me uncomfortable and quite worried now!
>>
>>
>> kind regards,
>>
>> David Sommerseth
>>
>>
>

[gentoo-hardened] recommented hardened-sources

[Alex Efros]

Hi!

On Sat, May 15, 2010 at 12:37:58PM +0300, Constantine Kardaris wrote:
> add "anarchy" overlay

Hmm. So, what is recommended way to run reliable and secure server and/or
workstation today?

- use stable x86 kernel from main portage, which is outdated .28 without
  support from PaX/GrSec team?
- use development kernel from anarchy overlay, which is up-to-date (now,
  but doesn't guaranteed to be always up-to-date, I think), and which is ...
  hmm ... development/unstable?
- use latests stable x86 vanilla-sources and manually apply PaX/GrSec
  patches?
- use latests stable x86 gentoo-sources (which is expected to be better
  than vanilla) and manually apply PaX/GrSec patches (which isn't
  guaranteed to apply at all to gentoo-sources)?

-- 
			WBR, Alex.

[gentoo-hardened] Re: recommented hardened-sources

[7v5w7go9ub0o]

On 05/15/10 11:25, Alex Efros wrote:
[]
>
> Hmm. So, what is recommended way to run reliable and secure server
> and/or workstation today?
>
> - use stable x86 kernel from main portage, which is outdated .28
> without support from PaX/GrSec team? - use development kernel from
> anarchy overlay, which is up-to-date (now, but doesn't guaranteed to
> be always up-to-date, I think), and which is ... hmm ...
> development/unstable? - use latests stable x86 vanilla-sources and
> manually apply PaX/GrSec patches? - use latests stable x86
> gentoo-sources (which is expected to be better than vanilla) and
> manually apply PaX/GrSec patches (which isn't guaranteed to apply at
> all to gentoo-sources)?
>

That seems to sum it up.

And when I advise folks on how much I like gentoo hardened, and what
great work the  hardened team is doing, it can be a little awkward
referring them to:

  "Anarchy overlay"; "gentoo-hardened at freenode"; and of course the
"bible":
<http://forums.gentoo.org/viewtopic-t-705939.html >


What IMHO should be the single starting point is:
<http://www.gentoo.org/proj/en/hardened/>

(The "last revised" date on this starting point is missing; the pages to
which it refers seem to be all 3-5 years old.)

I'm guessing that the hardened team is working to bring their efforts up
to standard, before officially updating the official gentoo hardened
page with appropriate links. Wrong policy, IMHO. In the interim, 'twould
be nice

- the good work of the hardened herd should be moved into the Gentoo
documentation structure, and noted as "developmental, but deployed
widely and successfully".

- the anarchy overlay should either be brought into portage core, or at
least renamed (e.g. "hardened" overlay) and documented in the official
Gentoo hardened pages.

I fear that folks looking for a hardened OS are passing Gentoo by,
because of the present situation.

Re: [gentoo-hardened] Re: recommented hardened-sources

[Matthew Summers]

[-- Attachment #1: Type: text/plain, Size: 1093 bytes --]

On Sat, May 15, 2010 at 11:33 AM, 7v5w7go9ub0o <7v5w7go9ub0o@gmail.com>wrote:

> On 05/15/10 11:25, Alex Efros wrote:
> []
>

<snip>

Hello, I just wanted to drop a note here that there is a team of devs
actively working on support for the 2.6.32 & 2.6.33 hardened-sources
kernels. There are a number of issues that have needed resolution for quite
some time.

Many of the points raised regarding documentation are good ones. The main
reason for the current state of things is a dramatic deficiency in man
power. So, if you have some time and the requisite experience I would
encourage you to join us in freenode's #gentoo-hardened to lend a hand. We
could really use the extra help.

To summarize, we are actively working on this & we are on pace to have h-s
and toolchain up to par and tree-ready, if not in the tree, within the next
couple of weeks. Stay tuned, as we will notify uses via as many comm
channels as possible.

We apologize for any inconvenience this has caused, but bear with us, we
will have things in shape very soon.

Best Regards,
Matt Summers

quantumsummers@gentoo.org

[-- Attachment #2: Type: text/html, Size: 1484 bytes --]

[gentoo-hardened] Re: recommented hardened-sources

[7v5w7go9ub0o]

On 05/15/10 13:19, Matthew Summers wrote:
[]
>
> Hello, I just wanted to drop a note here that there is a team of devs
> actively working on support for the 2.6.32&  2.6.33 hardened-sources
> kernels. There are a number of issues that have needed resolution for
> quite some time.
>
> Many of the points raised regarding documentation are good ones. The
> main reason for the current state of things is a dramatic deficiency
> in man power. So, if you have some time and the requisite experience
> I would encourage you to join us in freenode's #gentoo-hardened to
> lend a hand. We could really use the extra help.

I'll be happy to try to update the "starting point" mentioned above.

Please contact me at Gmail or at silcnet. I'm a newbie, and will need a
recommendation on how to proceed; perhaps a WYSIWYG html editor and an
address to which to mail drafts, or some sort of online access; perhaps
an online editor access to a work copy of the page. (is this something a
Firefox plugin could do?)

- I would try to do this quickly. Once the page is 95-6% (it is presently
less), one of you can officially transfer it to the gentoo server - I
don't need any keys to the kingdom. I'll have some time tomorrow and Sunday.

(Going for 99% should occur after you guys get things up to spec
(lurking at gentoo-hardened at freenode indicates that is in a few weeks
:-) )).

- Obviously, I'd need someone to proofread what I propose as it nears
completion; best is to do it via chat (please.... something easy,
friendly, and spontaneous; e.g. gentoo-hardened at silcnet; I'm there now).

- If you can get someone more "ready to go" than I, please do so.