[gentoo-dev] Bleeding edge hardened-sources: move PaX markings from ELF to Extended Attributes

[gentoo-dev] Bleeding edge hardened-sources: move PaX markings from ELF to Extended Attributes

[Anthony G. Basile]

Hi everyone,

I've been doing some experimental work with PaX enabled kernels and I
wanted to share it with the community at large for feedback.

Motivation: There are two (soon three) ways of doing PaX markings so
that a PaX enabled kernel knows what restrictions to put on the running
process.  These are:

1) EI_PAX markings.  This puts the pax flags in the ELF header in bytes
14 and 15 of the e_ident[] field.  This was a "hijacked" area and is now
broken. [1]

2) PT_PAX markings.  This puts the flags in an ELF program header.  On
Gentoo systems, all binaries are compiled with a PT_PAX header ready to
go because of a patch against binutils [2].  The problem is precompiled
binaries which lack a PT_PAX header and cannot have one added without
breaking.  (eg. skype).

3) XT_PAX markings.  This is the new experimental way of doing the
markings using xattrs for PaX markings.  Currently, I'm using the name
space "user.pax" so as to allow users to mark their own binaries, but
this may change to "security.pax" depending on what direction upstream
(ie pipacs) wants to go.  The advantage here is that the ELF binary is
not mangled in any way since the xattrs live in the inodes not the
blocks.  The disadvantage is that xattrs is not supported on all
filesystems and in all our utilities we need for portage to work.  I'm
working to get xattrs supported where we need it.  This will also help
with supporting other features like ACL and CAPS.  To this end:

a) There is a patch against tar to support xattrs based on a Fedora's
patch.  [3]
b) Kernels 3.0 and above support xattrs in tmpfs, squashfs and other
filesystems.
c) Python 3.3 and above support os.getxattr and os.setxattr and zmedico
and Arfrever have patched portage to copy xattrs from ${D} to ${ROOT}.
d) There's probably more .... feedback welcome!


I've built two test systems, amd64 and x86, and so far so go. 
Prometheanfire tested too and help find some snags.  If anyone is
interested, I've got a HOWTO on converting any gentoo system to a *pure*
XT_PAX hardened system [4], ie one with *no* EI_PAX or PT_PAX.  This
will not be the final situation where we will have backwards compat with
PT_PAX but not EI_PAX.  However, for testing it will force any issues
with XT_PAX to the foreground.

Since many of you know more about the internals of Gentoo than I, I
would appreciate any suggestions regarding what I might be missing if we
eventually migrate in this direction.


References:

[1] https://bugs.gentoo.org/show_bug.cgi?id=387459

[2] As of this writing, PT_PAX support is provided by patch
63_all_binutils-2.21.1-pt-pax-flags-20110918.patch  which can be
obtained from the patch bundles found at
http://dev.gentoo.org/~vapier/dist/ among other places.

[3] https://bugs.gentoo.org/show_bug.cgi?id=382067

[4]
http://git.overlays.gentoo.org/gitweb/?p=proj/hardened-dev.git;a=blob;f=HOWTO.txt;h=9edc600f0d81d5e77c6cd7e961a05b56f51b51ec;hb=f4d0da5dcaf12e4b9a70c1d2528becf649b1de61

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 8040 5A4D 8709 21B1 1A88  33CE 979C AF40 D045 5535
GnuPG ID  : D0455535

[gentoo-dev] Re: Bleeding edge hardened-sources: move PaX markings from ELF to Extended Attributes

[Duncan]

Anthony G. Basile posted on Thu, 01 Dec 2011 11:08:37 -0500 as excerpted:

> I've been doing some experimental work with PaX enabled kernels

> The disadvantage is that xattrs is not supported on all filesystems

> b) Kernels 3.0 and above support xattrs in tmpfs, squashfs and other
> filesystems.

[I debated sending this offlist as my interest isn't entirely on-topic, 
but as I wrote it up, it became apparent that the question itself isn't 
as off-topic as I thought, so on-list it stayed.]


TL;DR:  reiserfs (v3), for both caps and XT_PAX ??


Longer:  Does reiserfs (v3) support xattrs and thus, presumably caps and 
XT_PAX?  Kernel reiserfs options suggest yes, but everything I've read 
elsewhere (including gentoo-dev caps project discussions) seems to 
indicate no.  Is the "no" simply outdated, since reiserfs xattrs support 
was added relatively late in the game, or is it still correct and I have 
the purpose of those kernel options all wrong, or ???

If both reiserfs and tmpfs (my $PORTAGE_TMPDIR) support xattrs, both caps 
and XT_PAX should be good to go, correct?


Ignorable background:  I've been running reiserfs since early in the 
kernel 2.4 era, before switching to gentoo, and have had quite a good 
experience on it since it got ordered journals by default (before that, 
let's not talk about), despite a period with bad RAM that was hard-
crashing the system frequently, no UPS, and apparently missing barriers 
until quite recently.  I've even been stable when others had problems 
with ext3 due to its writeback-by-default period.  I intend to eventually 
switch to btrfs but there's still more kernel churn there than I'm 
comfortable with, especially running live-git kernels as I normally do, 
and of course, it's still lacking a proper fsck (tho arguably reiserfs' 
fsck support has always been "interesting" as well, and that's not such a 
big deal given the backups I already keep based on pre-ordered era 
experience).

My interest is therefore in whether USE=caps does people on reiserfs any 
good or not, but the same concerns would appear to apply to XT_PAX on 
reiserfs, as well.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Bleeding edge hardened-sources: move PaX markings from ELF to Extended Attributes

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 337 bytes --]

On 05:16 Fri 02 Dec     , Duncan wrote:
> TL;DR: reiserfs (v3), for both caps and XT_PAX ??

A bit OT, but I find it incredibly ironic that perhaps the shortest 
email you've ever written contained a TL;DR segment.

-- 
Thanks,
Donnie

Donnie Berkholz
Council Member / Sr. Developer
Gentoo Linux
Blog: http://dberkholz.com

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Bleeding edge hardened-sources: move PaX markings from ELF to Extended Attributes

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1105 bytes --]

On Friday 02 December 2011 00:16:43 Duncan wrote:
> Longer:  Does reiserfs (v3) support xattrs and thus, presumably caps and
> XT_PAX?  Kernel reiserfs options suggest yes, but everything I've read
> elsewhere (including gentoo-dev caps project discussions) seems to
> indicate no.  Is the "no" simply outdated, since reiserfs xattrs support
> was added relatively late in the game, or is it still correct and I have
> the purpose of those kernel options all wrong, or ???

i found reiserfs useful in the ext2/ext3 days, but now i find it completely 
irrelevant with ext4

> If both reiserfs and tmpfs (my $PORTAGE_TMPDIR) support xattrs, both caps 
> and XT_PAX should be good to go, correct?

my understanding of libattr/libacl is that the userland interface is FS 
independent.  once the FS supports xattrs (if the kernel says it does, i'd 
believe that), then that should be all you need.

while i've never tested xattrs on reiserfs (as alluded earlier, i've dropped 
all my reiserfs usage in favor of ext4), but i know tmpfs works (once you've 
enabled it in the kernel).
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Bleeding edge hardened-sources: move PaX markings from ELF to Extended Attributes

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1551 bytes --]

On Thursday 01 December 2011 11:08:37 Anthony G. Basile wrote:
> 2) PT_PAX markings.  This puts the flags in an ELF program header.  On
> Gentoo systems, all binaries are compiled with a PT_PAX header ready to
> go because of a patch against binutils [2].  The problem is precompiled
> binaries which lack a PT_PAX header and cannot have one added without
> breaking.  (eg. skype).
> 
> 3) XT_PAX markings.  This is the new experimental way of doing the
> markings using xattrs for PaX markings.  Currently, I'm using the name
> space "user.pax" so as to allow users to mark their own binaries, but
> this may change to "security.pax" depending on what direction upstream
> (ie pipacs) wants to go.  The advantage here is that the ELF binary is
> not mangled in any way since the xattrs live in the inodes not the
> blocks.  The disadvantage is that xattrs is not supported on all
> filesystems and in all our utilities we need for portage to work.  I'm
> working to get xattrs supported where we need it.  This will also help
> with supporting other features like ACL and CAPS.  To this end:

i happily look forward to the time where we can deprecate PT_PAX support in 
binutils.  it is, by far, the largest thorn in my side when it comes to 
stabilization and false positive test failures in binutils.

> a) There is a patch against tar to support xattrs based on a Fedora's
> patch.  [3]

sorry, now that i know this is a bit more important than "i've been playing 
with this stuff", i'll try and get to it faster
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[gentoo-dev] Re: Bleeding edge hardened-sources: move PaX markings from ELF to Extended Attributes

[Duncan]

Donnie Berkholz posted on Wed, 07 Dec 2011 15:47:01 -0500 as excerpted:

> On 05:16 Fri 02 Dec     , Duncan wrote:
>> TL;DR: reiserfs (v3), for both caps and XT_PAX ??
> 
> A bit OT, but I find it incredibly ironic that perhaps the shortest
> email you've ever written contained a TL;DR segment.

=;^)

If I fit any of the qualities said to be associated with Asperger (which 
FWIW I don't claim to have), surely "excess verbosity" must be it.  So 
I'm working on it and the more I thought about tl;dr, the more it seemed 
to do exactly what I needed.  Thus I'm trying to make it a habit.

Thanks for noticing.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Bleeding edge hardened-sources: move PaX markings from ELF to Extended Attributes

[Anthony G. Basile]

On 12/07/2011 05:10 PM, Mike Frysinger wrote:
> On Friday 02 December 2011 00:16:43 Duncan wrote:
>> Longer:  Does reiserfs (v3) support xattrs and thus, presumably caps and
>> XT_PAX?  Kernel reiserfs options suggest yes, but everything I've read
>> elsewhere (including gentoo-dev caps project discussions) seems to
>> indicate no.  Is the "no" simply outdated, since reiserfs xattrs support
>> was added relatively late in the game, or is it still correct and I have
>> the purpose of those kernel options all wrong, or ???
> i found reiserfs useful in the ext2/ext3 days, but now i find it completely 
> irrelevant with ext4
>
>> If both reiserfs and tmpfs (my $PORTAGE_TMPDIR) support xattrs, both caps 
>> and XT_PAX should be good to go, correct?
> my understanding of libattr/libacl is that the userland interface is FS 
> independent.  once the FS supports xattrs (if the kernel says it does, i'd 
> believe that), then that should be all you need.
>
> while i've never tested xattrs on reiserfs (as alluded earlier, i've dropped 
> all my reiserfs usage in favor of ext4), but i know tmpfs works (once you've 
> enabled it in the kernel).
> -mike

I just tested with reiser3 and xattr works just fine.  Just make sure
its enabled in the kernel and when you mount the fs use option
user_xattr for the user. namespace.

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 8040 5A4D 8709 21B1 1A88  33CE 979C AF40 D045 5535
GnuPG ID  : D0455535

[gentoo-dev] Re: Bleeding edge hardened-sources: move PaX markings from ELF to Extended Attributes

[Duncan]

Mike Frysinger posted on Wed, 07 Dec 2011 17:10:01 -0500 as excerpted:

> while i've never tested xattrs on reiserfs (as alluded earlier, i've
> dropped all my reiserfs usage in favor of ext4), but i know tmpfs works
> (once you've enabled it in the kernel).

Thanks.

FWIW:

(1) Tail-packing.

(2) Good experience since data=ordered, even thru bad ram induced kernel 
lockups, plus a drive overheat recovery (dd-rescue) after an AC failure.

(3) The kernel folks don't screw with it as much as they do ext*.  
Reference the infamous ext3 data=writeback period during which reiserfs 
stayed safely data=ordered.  I know people on ext3 that were wondering 
why they were losing data during that period, until I explained the 
situation and they switched to explicit data=ordered.

btrfs ftw, tho, when I think it's ready.  Perhaps after the larger 
metadata blocks that LWN says is penciled in for 3.3, which should be 
early January merge-window and mid-March release, assuming 10-week cycles 
and given current 3.2-rc4.  But I'll omit listing my reasons as I did for 
reiserfs, since I guess anyone interested has already read up on btrfs.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[gentoo-dev] Re: Bleeding edge hardened-sources: move PaX markings from ELF to Extended Attributes

[Duncan]

Anthony G. Basile posted on Wed, 07 Dec 2011 20:03:22 -0500 as excerpted:

> I just tested with reiser3 and xattr works just fine.  Just make sure
> its enabled in the kernel and when you mount the fs use option
> user_xattr for the user. namespace.

Thanks.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Bleeding edge hardened-sources: move PaX markings from ELF to Extended Attributes

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 615 bytes --]

On Wednesday 07 December 2011 20:45:28 Duncan wrote:
> (1) Tail-packing.

ext4 will be doing something similar:
	http://lwn.net/Articles/469805/

although i don't find the overhead w/out tail packing to be a deal breaker

> (3) The kernel folks don't screw with it as much as they do ext*.
> Reference the infamous ext3 data=writeback period during which reiserfs
> stayed safely data=ordered.  I know people on ext3 that were wondering
> why they were losing data during that period, until I explained the
> situation and they switched to explicit data=ordered.

i have no sympathy for broken userspace code
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Re: Bleeding edge hardened-sources: move PaX markings from ELF to Extended Attributes

[Rich Freeman]

On Wed, Dec 7, 2011 at 8:58 PM, Mike Frysinger <vapier@gentoo.org> wrote:
> i have no sympathy for broken userspace code

I define broken userspace code as anything that uses fsync except for
transactional synchronization with external sources.

My system is a bit beefier now, but one of the biggest performance
issues before the upgrade was that mythtv did an fsync all the time.
What's the point in having RAM and buffer if some process is going to
tell the kernel that it is worth taking a huge impact on everything
else just to write one file a little faster.  The irony in the end was
that the result usually was dropped data in mythtv on another tuner
since you'd end up with two processes stepping all over each other and
the kernel probably could have easily flushed all that data if it were
allowed to have more than a few MB of data in the cache between disk
seeks.

Filesystems need to fail gracefully.  If I write data to disk and the
power fails then the file should either contain what it did before the
write, or the correct data after the write.  As a compromise since COW
is still new I'm fine with it even being partially modified, but
things like zero-length files or other odd behavior is just dumb.  And
the correct userspace implementation is to call write, and close when
you're done.  I don't want every process on my system doing syncs just
because some ext3 purist thinks that data is disposable by default, or
in the magical world where every system is on a UPS, hardware never
fails, and kernels never panic.  I might live dangerously for a 50%
performance gain, but when we're talking about shaving an extra 0.1%
off of some benchmark then lets go ahead and write the data out in the
right order.

If we do want to have an API that tells the kernel that data isn't
disposable or to safely overwrite a file in place then we should
invent a new system call for that, and not try to overload some other
behavior (like a combination of fsyncing and moves).  The problem with
the latter is that it often results in extra disk churn as the
filesystem does more than is strictly necessary, and since every
filesystem author has their own religious preference about the right
series of system calls to use programs end up only working correctly
on a handful of filesystems.

And of course the real solution is COW once that becomes stable.  And
the next step after that is real transaction support.  If only btrfs
were ready for production use...

Fortunately for the moment it seems Linus is reasonably sane and will
fix filesystems if the upstream maintainers are unwilling to do so.
:)

Ok, off the soap box.  My issue is that people seem to talk about
"correct userspace behavior" as if there were some kind of standard we
were referring to.  By all means get the crazy behavior added to POSIX
and then we can all go on and on about it (and then once COW comes
along scrach our heads as we do all kinds of extra unnecessary file
manipulation to do something that just works by default).  If there is
such a standard that has been endorsed by all the major unix
filesystems, please do point it out.

And sorry, Mike, not really directed at you in particular - I know
that many people feel the same as you do...

Rich

[gentoo-dev] Re: Bleeding edge hardened-sources: move PaX markings from ELF to Extended Attributes

[Duncan]

Mike Frysinger posted on Wed, 07 Dec 2011 20:58:15 -0500 as excerpted:

> On Wednesday 07 December 2011 20:45:28 Duncan wrote:
>> (1) Tail-packing.
> 
> ext4 will be doing something similar:	http://lwn.net/Articles/469805/

Thanks.  I was ~3 months behind on LWN for awhile but am down to 4 weeks 
(including the 8th's weekly) behind now, but that does mean I hadn't seen 
that article yet.  I just read it. =:^)

Looks like they're adding metadata checksumming too, tho not data, and 
not until 3.4-ish.

I like, but as I think I mentioned, I hope to finally be switching to 
btrfs with 3.3 or 3.4, after the bigger metadata blocks hit (an earlier 
LWN said 3.3-ish).  I'll be keeping the backup copies (both hot and cold) 
on reiserfs for awhile, tho.  I'm not insane enough to trust /everything/ 
to the still-no-fsck-only-recovery-tool btrfs just yet.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman