[gentoo-dev] Textrels in packages policy

[gentoo-dev] Textrels in packages policy

[Mark Loeser]

[-- Attachment #1: Type: text/plain, Size: 1016 bytes --]

Basically what I'm looking for here is an easy to understand explanation of
what textrels are, why they are bad, and why they should hold back marking a
package stable.  The only information I've been able to find states that they
could cause a performance hit, but this doesn't seem to warrant banning them
completely in my eyes.

Getting a clear cut policy on exactly what issues should hold a package back 
from being marked stable is what I'm looking for.  Issues like textrels, 
executable stacks, etc is what I'm looking for to be defined and explained why 
we are to always avoid them.  This should be added to existing documentation
policy so it is somewhere for new devs to know about, and existing devs to
have for a reference.

Thanks

-- 
Mark Loeser   -   Gentoo Developer (cpp gcc-porting toolchain x86)
email         -   halcy0n AT gentoo DOT org
                  mark AT halcy0n DOT com
web           -   http://dev.gentoo.org/~halcy0n/
                  http://www.halcy0n.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Textrels in packages policy

[Mike Frysinger]

On Tue, Dec 13, 2005 at 03:59:03PM -0500, Mark Loeser wrote:
> Basically what I'm looking for here is an easy to understand explanation of
> what textrels are, why they are bad, and why they should hold back marking a
> package stable.  The only information I've been able to find states that they
> could cause a performance hit, but this doesn't seem to warrant banning them
> completely in my eyes.

erm, this e-mail comes a bit early ... i was hoping to have more stuff
written before moving this info into gentoo-dev mailing list realm ...
-mike
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Textrels in packages policy

[Saleem A.]

[-- Attachment #1: Type: text/plain, Size: 2209 bytes --]

On Tue, 13 Dec 2005, Mark Loeser wrote:

> Basically what I'm looking for here is an easy to understand explanation of
> what textrels are, why they are bad, and why they should hold back marking a
> package stable.  The only information I've been able to find states that they
> could cause a performance hit, but this doesn't seem to warrant banning them
> completely in my eyes.

Given my limited knowledge on this, this is my understanding.

TEXTRELS are basically text relocations.  What this is, is relocation
within the text segment of the process image.  This brings up the
question of what a relocation is.  A relocation is simply the
replacement of some text with a memory location.  The big issue with
this is that the text segment is usually suppose to be read only for
security reasons.  But because the text segment needs a relocation, it
needs to be read-write since the relocation happens at runtime
dynamically.  The constant need to look up the address is what causes
the performance degredation.  The performance degredation however is of
no worry to us.  The issue is that since the text segment is now
read-write, the image of the process is no longer guaranteed to remain
the same as it can be overwritten (allowing code modifications at
runtime which can happen other ways as well).  Because of this, the
application is far more vurnerable to arbitrary code execution as if an
exploit manages to overwrite the text segment properly, it can execute
code that it wants.

I am not sure how correct this explanation is or it is even what you
were looking for.

> Getting a clear cut policy on exactly what issues should hold a package back 
> from being marked stable is what I'm looking for.  Issues like textrels, 
> executable stacks, etc is what I'm looking for to be defined and explained why 
> we are to always avoid them.  This should be added to existing documentation
> policy so it is somewhere for new devs to know about, and existing devs to
> have for a reference.

I agree, this would be very nice to have.  It would make stabilization
of packages a little bit easier.


Thanks.

Saleem Abdulrasool
compnerd (at) gentoo (dot) org

[-- Attachment #2: Type: application/pgp-signature, Size: 307 bytes --]

Re: [gentoo-dev] Textrels in packages policy

[Jory A. Pratt]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Mark Loeser wrote:
> Basically what I'm looking for here is an easy to understand explanation of
> what textrels are, why they are bad, and why they should hold back marking a
> package stable.  The only information I've been able to find states that they
> could cause a performance hit, but this doesn't seem to warrant banning them
> completely in my eyes.
> 
> Getting a clear cut policy on exactly what issues should hold a package back 
> from being marked stable is what I'm looking for.  Issues like textrels, 
> executable stacks, etc is what I'm looking for to be defined and explained why 
> we are to always avoid them.  This should be added to existing documentation
> policy so it is somewhere for new devs to know about, and existing devs to
> have for a reference.
> 
> Thanks
> 
Only problem I see with this is binary packages. We can not control
upstream binaries as everyone is aware of. So when does it become safe
to override stable packages that have texrel's and executable stacks?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFDn2BVGDfjNg8unQIRAupGAKCiTPJseSVrklDjWXqwEdeHFDxnRQCcD0xQ
mzjn2yXHiNSdBcnFkCTD+u0=
=RYEw
-----END PGP SIGNATURE-----
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Textrels in packages policy

[Mike Frysinger]

On Tue, Dec 13, 2005 at 10:30:59PM +0000, Saleem A. wrote:
> On Tue, 13 Dec 2005, Mark Loeser wrote:
> 
> > Basically what I'm looking for here is an easy to understand explanation of
> > what textrels are, why they are bad, and why they should hold back marking a
> > package stable.  The only information I've been able to find states that they
> > could cause a performance hit, but this doesn't seem to warrant banning them
> > completely in my eyes.
> 
> The big issue with
> this is that the text segment is usually suppose to be read only for
> security reasons.  But because the text segment needs a relocation, it
> needs to be read-write since the relocation happens at runtime
> dynamically.

this is correct, a very good reason to fix TEXTRELs.  another good
reason is that since the segment cannot be mapped readonly, the memory
cannot be shared across multiple processes ... each will need to have
its own copy, thus wasting what could be significant memory resources.

> The constant need to look up the address is what causes
> the performance degredation.  The performance degredation however is of
> no worry to us.

if by "constant" you mean "once everytime the code is loaded", then you
are correct ... the relocations need to be rewritten everytime the image
is loaded into memory by the dynamic loader (ldso), but after the first
fixup, that's it, nothing else to be done.

> > Getting a clear cut policy on exactly what issues should hold a package back 
> > from being marked stable is what I'm looking for.  Issues like textrels, 
> > executable stacks, etc is what I'm looking for to be defined and explained why 
> > we are to always avoid them.  This should be added to existing documentation
> > policy so it is somewhere for new devs to know about, and existing devs to
> > have for a reference.
> 
> I agree, this would be very nice to have.  It would make stabilization
> of packages a little bit easier.

working on it as i said ... i wish this e-mail could have been posted
once i had more easier things to read :p
-mike
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Textrels in packages policy

[Mike Frysinger]

On Tue, Dec 13, 2005 at 05:59:17PM -0600, Jory A. Pratt wrote:
> Mark Loeser wrote:
> > Basically what I'm looking for here is an easy to understand explanation of
> > what textrels are, why they are bad, and why they should hold back marking a
> > package stable.  The only information I've been able to find states that they
> > could cause a performance hit, but this doesn't seem to warrant banning them
> > completely in my eyes.
> > 
> > Getting a clear cut policy on exactly what issues should hold a package back 
> > from being marked stable is what I'm looking for.  Issues like textrels, 
> > executable stacks, etc is what I'm looking for to be defined and explained why 
> > we are to always avoid them.  This should be added to existing documentation
> > policy so it is somewhere for new devs to know about, and existing devs to
> > have for a reference.
>
> Only problem I see with this is binary packages. We can not control
> upstream binaries as everyone is aware of. So when does it become safe
> to override stable packages that have texrel's and executable stacks?

no idea what you mean by "override", but here's a crazy idea ... ask
upstream to fix the issues.  for example, we just reported executable
stacks with the ut2004 game and Ryan of epicgames was so kind as to
fix it up for us.  some upstream peeps dont even know about these sort
of things until you point them out.
-mike
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Textrels in packages policy

[Mark Loeser]

[-- Attachment #1: Type: text/plain, Size: 1515 bytes --]

Mike Frysinger <vapier@gentoo.org> said:
> this is correct, a very good reason to fix TEXTRELs.  another good
> reason is that since the segment cannot be mapped readonly, the memory
> cannot be shared across multiple processes ... each will need to have
> its own copy, thus wasting what could be significant memory resources.

Sounds like something nice to do, but not something that should be _required_
for a package to be deemed stable.

> > The constant need to look up the address is what causes
> > the performance degredation.  The performance degredation however is of
> > no worry to us.
> 
> if by "constant" you mean "once everytime the code is loaded", then you
> are correct ... the relocations need to be rewritten everytime the image
> is loaded into memory by the dynamic loader (ldso), but after the first
> fixup, that's it, nothing else to be done.

Again, nice to fix, but does not seem necessary.

> working on it as i said ... i wish this e-mail could have been posted
> once i had more easier things to read :p

You are working on a policy, or just docs to explain the issues?  From what
was listed above, I'm not sure why we should require that people fix these
issues just to have a package deemed stable.

-- 
Mark Loeser   -   Gentoo Developer (cpp gcc-porting toolchain x86)
email         -   halcy0n AT gentoo DOT org
                  mark AT halcy0n DOT com
web           -   http://dev.gentoo.org/~halcy0n/
                  http://www.halcy0n.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Textrels in packages policy

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 2013 bytes --]

On Wed, 14 Dec 2005 00:22:36 +0000 Mike Frysinger <vapier@gentoo.org>
wrote:
| > The big issue with
| > this is that the text segment is usually suppose to be read only for
| > security reasons.  But because the text segment needs a relocation,
| > it needs to be read-write since the relocation happens at runtime
| > dynamically.
| 
| this is correct, a very good reason to fix TEXTRELs.

This is only an issue if an application is already insecure. Thus,
TEXTRELs shouldn't be considered sufficient reason to avoid marking
something stable, any more than we avoid marking code that uses sprintf
stable.

|  another good reason is that since the segment cannot be mapped
| readonly, the memory cannot be shared across multiple processes ...
| each will need to have its own copy, thus wasting what could be
| significant memory resources.

Again, that's a big "could be". We don't avoid marking stable code
that, say, mallocs lots of space, then fills it with some calculated
numbers (for example, the first million prime numbers), even though a
better program would allow for that data to be shared.

So yes, TEXTRELs when used accidentally are rather sucky. On the other
hand, there are legitimate uses for them, and they aren't insecure, nor
are they necessarily any worse performance-wise than code that uses
other methods.

Banning TEXTRELs outright makes no more sense than banning code that
uses goto or sprintf -- if TEXTRELs are used accidentally and there's
an easy fix, take it, but don't let them stop you from providing the
most stable version of a package to your users.

Oh, and don't accept reasons like "but they don't work if we enable
$obscure_voodoo in the compiler" either. If $obscure_voodoo breaks on
legitimate TEXTRELs then $obscure_voodoo is broken, not the code using
TEXTRELs.

-- 
Ciaran McCreesh : Gentoo Developer (I can kill you with my brain)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Textrels in packages policy

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 821 bytes --]

On Tue, 13 Dec 2005 20:02:27 -0500 Mark Loeser <halcy0n@gentoo.org>
wrote:
| You are working on a policy, or just docs to explain the issues?
| From what was listed above, I'm not sure why we should require that
| people fix these issues just to have a package deemed stable.

Some people want no TEXTRELs to be a requirement because they're
superstitious, or because they read somewhere that TEXTRELs are
naughty and are repeating the conclusion without understanding the
underlying reasons. It's approximately on par with trying to ban all
code that uses 'goto' because it breaks certain security auditing
tools that rely upon call graphs.

-- 
Ciaran McCreesh : Gentoo Developer (I can kill you with my brain)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Textrels in packages policy

[Mike Frysinger]

On Tue, Dec 13, 2005 at 08:02:27PM -0500, Mark Loeser wrote:
> Mike Frysinger <vapier@gentoo.org> said:
> > working on it as i said ... i wish this e-mail could have been posted
> > once i had more easier things to read :p
> 
> You are working on a policy, or just docs to explain the issues?

documentation on PIC/TEXTRELs/etc...

the policy i consider a no-brainer, fix TEXTRELs
-mike
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Textrels in packages policy

[Mike Frysinger]

On Wed, Dec 14, 2005 at 01:07:53AM +0000, Ciaran McCreesh wrote:
> On Wed, 14 Dec 2005 00:22:36 +0000 Mike Frysinger <vapier@gentoo.org>
> |  another good reason is that since the segment cannot be mapped
> | readonly, the memory cannot be shared across multiple processes ...
> | each will need to have its own copy, thus wasting what could be
> | significant memory resources.
>
> Again, that's a big "could be".

it's more often an "is be" considering the fact we're talking about
shared library code here.

> We don't avoid marking stable code
> that, say, mallocs lots of space, then fills it with some calculated
> numbers (for example, the first million prime numbers), even though a
> better program would allow for that data to be shared.

no one said that broken code with TEXTRELs cannot be marked stable

they're something to be fixed down the road as time permits

> Oh, and don't accept reasons like "but they don't work if we enable
> $obscure_voodoo in the compiler" either. If $obscure_voodoo breaks on
> legitimate TEXTRELs then $obscure_voodoo is broken, not the code using
> TEXTRELs.

majority of the time, if a build process is generating poor code with
textrels, it wont work on most architectures.  x86 just tends to be
pretty lenient when it comes to poor code, so no one notices/cares.
-mike
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Textrels in packages policy

[Mark Loeser]

[-- Attachment #1: Type: text/plain, Size: 942 bytes --]

Mike Frysinger <vapier@gentoo.org> said:
> > We don't avoid marking stable code
> > that, say, mallocs lots of space, then fills it with some calculated
> > numbers (for example, the first million prime numbers), even though a
> > better program would allow for that data to be shared.
> 
> no one said that broken code with TEXTRELs cannot be marked stable
> 
> they're something to be fixed down the road as time permits

This I completely agree with, but was not what was being pushed as policy for
the x86 arch team.  If a package reintroduces textrels or such, I don't see
the problem with marking it stable personally.  The issues listed don't seem
to warrant that.

-- 
Mark Loeser   -   Gentoo Developer (cpp gcc-porting toolchain x86)
email         -   halcy0n AT gentoo DOT org
                  mark AT halcy0n DOT com
web           -   http://dev.gentoo.org/~halcy0n/
                  http://www.halcy0n.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Textrels in packages policy

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1361 bytes --]

On Wed, 14 Dec 2005 01:20:29 +0000 Mike Frysinger <vapier@gentoo.org>
wrote:
| the policy i consider a no-brainer, fix TEXTRELs

So... Say libfoo is a library that decodes files in the foo format. Say
also that libfoo-2.1 is currently marked stable, and does not contain
any TEXTRELs, but only supports foo files using the foo-1 format, which
is rapidly becoming obsolete. Then, if libfoo-2.2 comes along, and
introduces support for the rapidly becoming very widely used foo-2
format, should libfoo-2.2 be held back from being marked stable purely
because of a TEXTREL?

What about if, instead, libfoo-2.1 contains a security bug that is
fixed by libfoo-2.2?

What about if, instead, libfoo-2.1 has a nasty bug that will cause it
to lose data under certain circumstances?

See, any policy on TEXTRELs will have to be *very* carefully worded to
avoid having people think that they're something important... Otherwise
we'll end up with broken packages being left in stable simply because
newer versions contain TEXTRELs. Don't overhype the problem -- make it
clear that it's a "would be nice" issue, or else you'll encourage even
more superstition than we have already...

-- 
Ciaran McCreesh : Gentoo Developer (I can kill you with my brain)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Textrels in packages policy

[Mark Loeser]

[-- Attachment #1: Type: text/plain, Size: 778 bytes --]

Mike Frysinger <vapier@gentoo.org> said:
> On Tue, Dec 13, 2005 at 08:02:27PM -0500, Mark Loeser wrote:
> > You are working on a policy, or just docs to explain the issues?
> 
> documentation on PIC/TEXTRELs/etc...
> 
> the policy i consider a no-brainer, fix TEXTRELs

By policy, I mean things to add to the ebuild policy stating, "These
conditions must be met in order to consider a package stable:".  I'll assume
that is not what you are working on since nothing you have listed so far
seems to imply this.

-- 
Mark Loeser   -   Gentoo Developer (cpp gcc-porting toolchain x86)
email         -   halcy0n AT gentoo DOT org
                  mark AT halcy0n DOT com
web           -   http://dev.gentoo.org/~halcy0n/
                  http://www.halcy0n.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Textrels in packages policy

[Mike Frysinger]

On Wed, Dec 14, 2005 at 01:37:57AM +0000, Ciaran McCreesh wrote:
> On Wed, 14 Dec 2005 01:20:29 +0000 Mike Frysinger <vapier@gentoo.org>
> wrote:
> | the policy i consider a no-brainer, fix TEXTRELs
> 
> So... Say libfoo is 
> <snip>

blah blah blah i didnt read this e-mail, i imagine it's your normal
stuff though, so i'm not missing much

as i said, i'm pro-fixing TEXTRELs, not 
pro-preventing-packages-from-getting-into-stable-because-it-has-TEXTRELs
-mike
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Textrels in packages policy

[Jason Wever]

[-- Attachment #1: Type: text/plain, Size: 742 bytes --]

On Wed, 14 Dec 2005 00:25:57 +0000
Mike Frysinger <vapier@gentoo.org> wrote:

> no idea what you mean by "override", but here's a crazy idea ... ask
> upstream to fix the issues.  for example, we just reported executable
> stacks with the ut2004 game and Ryan of epicgames was so kind as to
> fix it up for us.  some upstream peeps dont even know about these sort
> of things until you point them out.

Not to redirect the thread, but can someone point me to stuff on
executable stacks (what they are and the background info on the
warnings in portage)?

If these are anywhere near critical, then I think a lot of us non-x86
arch monkeys have a lot of work on our hands.

Cheers,
-- 
Jason Wever
Gentoo/Sparc Team Co-Lead

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Textrels in packages policy

[Mark Loeser]

[-- Attachment #1: Type: text/plain, Size: 1029 bytes --]

Jason Wever <weeve@gentoo.org> said:
> Not to redirect the thread, but can someone point me to stuff on
> executable stacks (what they are and the background info on the
> warnings in portage)?

Not really redirecting the thread since this was another thing I wanted to
find out about :)  Basically, any of the things that "stricter" may complain
about, which are "critical", in the sense that they can not be marked stable
under any circumstances?  Also, what exactly do they mean? :)

> If these are anywhere near critical, then I think a lot of us non-x86
> arch monkeys have a lot of work on our hands.

Yea, we need to know what is absolutely critical so we can add this to the
ebuild policy if something should not go stable if the problem is present.

-- 
Mark Loeser   -   Gentoo Developer (cpp gcc-porting toolchain x86)
email         -   halcy0n AT gentoo DOT org
                  mark AT halcy0n DOT com
web           -   http://dev.gentoo.org/~halcy0n/
                  http://www.halcy0n.com

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Textrels in packages policy

[Mike Frysinger]

On Tue, Dec 13, 2005 at 07:59:02PM -0700, Jason Wever wrote:
> On Wed, 14 Dec 2005 00:25:57 +0000
> Mike Frysinger <vapier@gentoo.org> wrote:
> 
> > no idea what you mean by "override", but here's a crazy idea ... ask
> > upstream to fix the issues.  for example, we just reported executable
> > stacks with the ut2004 game and Ryan of epicgames was so kind as to
> > fix it up for us.  some upstream peeps dont even know about these sort
> > of things until you point them out.
> 
> Not to redirect the thread, but can someone point me to stuff on
> executable stacks (what they are and the background info on the
> warnings in portage)?

my gnu stack docs are actually complete:
http://hardened.gentoo.org/gnu-stack.xml

> If these are anywhere near critical, then I think a lot of us non-x86
> arch monkeys have a lot of work on our hands.

gnu-stack tends to be broken for everyone, not just x86 or !x86
-mike
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Textrels in packages policy

[Harald van Dijk]

[-- Attachment #1: Type: text/plain, Size: 451 bytes --]

On Wed, Dec 14, 2005 at 03:50:16AM +0000, Mike Frysinger wrote:
> my gnu stack docs are actually complete:
> http://hardened.gentoo.org/gnu-stack.xml

A question about that: you discourage fixing this with --noexecstack
because it's better to be able to submit a patch upstream. What's your
take on patches that modify configure scripts or similar files to check
for this flag, keeping it out of the ebuild? Is that good, acceptable,
or bad, and why?

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Textrels in packages policy

[Kevin F. Quinn]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, 13 Dec 2005 15:59:03 -0500
Mark Loeser <halcy0n@gentoo.org> wrote:

> Basically what I'm looking for here is an easy to understand
> explanation of what textrels are, why they are bad, and why they
> should hold back marking a package stable.  The only information I've
> been able to find states that they could cause a performance hit, but
> this doesn't seem to warrant banning them completely in my eyes.

The seriousness of the textrel issue is different for Hardened Gentoo
and normal Gentoo.  For Hardened Gentoo they cause real problems and
must to be fixed to avoid compromising the overall strategy.  For
non-hardened Gentoo it's not so serious.  I'll focus on non-hardened
Gentoo here.

Textrels (short for "text relocations") are function or data pointers
within the code of a shared library that need to be adjusted to take
account of the address at which a shared library is loaded.

Shared libraries can be loaded at varying addresses depending on how the
loader brings shared libraries into a process' memory image.  This can
be affected by factors including the size of the main executable, which
other shared libraries are loaded for the same process and in which
order.

When a shared library is loaded, any absolute address references within
the code have to be changed to reflect the actual address according to
the address at which the shared library is loaded, before that code is
executed.  Since the loader can't tell what will be executed and when,
it has to resolve all such references before the process can be
permitted to start execution (this is the performance overhead
incurred at load time).  It also means that the image of the shared
library is specific to that process - other processes can only use the
same image if they load the library at the same address which is
unlikely under normal circumstances.  Thus libraries that are truly
shared between applications, consume separate memory for each
application that is using them (which is therefore a resource overhead).
One workaround for this is to use prelink.  This examines your whole
system, and marks all shared libraries with hints for their
load addresses, which the loader follows if it can. The addresses are
chosen to minimise the amount of collisions, 

If a shared library has no text relocations, it is properly Position
Independent Code (PIC).  Instead of absolute addresses, the code always
uses relative addresses which work no matter at which address the code
is loaded.  Typically this is achieved via an indirection table (the
Global Offset Table) which contains the offsets for each symbol.  Thus
such code can be mapped without write permissions on the code, and you
only ever have one copy of the code in memory, no matter how many
applications are using that code.

The downside of position-independent code is that it does have a
performance impact as currently implemented (i.e. it generates slower
code) due to the indirection introduced.  This is made worse by the
bad use of shared libraries, and the fact that the visibility stuff
in the GNU tools is so recent (shockingly) and so libraries have far
too much stuff unnecessarily in their global offset tables.

Shared libraries (.so's) are used for various things:

1) To share code between multiple applications, the original intention
of shared libraries. Best example is libc which if it were to be
duplicated for each running application would have a significant impact
on memory usage. For such libraries, TEXTRELs are a problem; the
seriousness of the problem depends on how many applications use them.

2) To modularise an application; good examples here are X and
web-browser plugins.  Here, code is loaded if it is used, but not
otherwise.  Loading all the device drivers into X would be wasteful,
for example.  This is a secondary use of shared libraries that arises
from the way shared libraries are implemented, making them easy to use
for this purpose.  TEXTRELs here cause a penalty on load, but 

3) To break an application up into bits.  This is the worst reason to
create a shared library, as it gains nothing but loses much.  It means a
developer abstraction has leaked unnecessarily into the implementation.
These simply shouldn't be shared libraries, instead the code should
have been linked into the executable.


Textrels can be caused by:

1) asm code that isn't PIC

2) incorrect makefiles etc (if a build uses libtool properly, it'll
create shared libraries with no textrels)


Frequently these are easy to fix, and are simply oversights upstream.
There's no reason not to fix these.

Occasionally they're a pain to fix, especially non-PIC asm, and if
upstream are non-PIC on purpose (usually because they want to
extract every last cycle from the processor) we end up having to
maintain large patches which amongst other things is risky.


> Getting a clear cut policy on exactly what issues should hold a
> package back from being marked stable is what I'm looking for.
> Issues like textrels, executable stacks, etc is what I'm looking for
> to be defined and explained why we are to always avoid them.

Executable stack is a separate issue, and nothing to do with textrels.
For the moment it's not really relevant outside of Hardened Gentoo,
however it's quite possible that standard Linux may support some kind
of executable stack protection in the future once the venerable x86
architecture gives way to NX-enabled systems.

However, typically executable stack is caused by one or more of:

1) objects that aren't marked as needing or not needing executable
stack.  Typically this occurs with assembler code, but can also occur
with code built with compilers other than our gcc. Here, the linker
decides that the code might need executable stack, so cannot mark it as
not needing executable stack.  This is trivially fixed by adding the
appropriate markings to the asm code.

2) Nested functions.  These are a GCC extension to C, although the
underlying concept exists in other languages.  In GCC they are
implemented using a trick called a 'trampoline', which is a little
piece of code place on the stack at runtime.  Note that in C, it's
easy to pass around a pointer to a nested function (i.e. a pointer to
the trampoline) outside of scope; outside of scope a call through such
a pointer will have unpredictable behaviour - probably segfault.
Frequently, the nested functions do not really need to be nested and
the code can trivially be adjusted to avoid them.

3) Custom software that builds code dynamically in data local to a
function (i.e. data on the stack).  This is rare, unlikely to occur on
anything other than x86.  Dynamically-generated code should really be
created on the heap, with the appropriate mprotect() calls made to make
sure that malloc()'ed data is executable (by default it is not
executable in theory, however in practice on x86 it usually is because
x86 has a limited memory protection scheme).

Where the changes to eliminate executable stack are trivial, there's no
reason not to make them as they have no performance impact and
upstream maintainers are usually in favour, and in the long term it's
good to avoid executable stack where possible.

>  This
> should be added to existing documentation policy so it is somewhere
> for new devs to know about, and existing devs to have for a reference.

Agreed.

As far as normal Gentoo is concerned, I think policy should be to fix
textrels at least where it is simple to do so and upstream are happy to
have the issues fixed, and we should be most insistent for shared
libraries that are actually used as libraries shared between
applications. For profiles that must refuse textrels, unfixed packages
can be masked. Similarly with executable stack.

- -- 
Kevin F. Quinn
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFDn81l9G2S8dekcG0RAlz4AKDB8V2j9yJUaWmWRpn6vOXbHK34+wCeNLup
NgGh1BZemplmgQW3/2050Kw=
=XjZ8
-----END PGP SIGNATURE-----

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Textrels in packages policy

[Kevin F. Quinn]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, 14 Dec 2005 07:59:23 +0100
Harald van Dijk <truedfx@gentoo.org> wrote:

> On Wed, Dec 14, 2005 at 03:50:16AM +0000, Mike Frysinger wrote:
> > my gnu stack docs are actually complete:
> > http://hardened.gentoo.org/gnu-stack.xml
> 
> A question about that: you discourage fixing this with --noexecstack
> because it's better to be able to submit a patch upstream. What's your
> take on patches that modify configure scripts or similar files to
> check for this flag, keeping it out of the ebuild? Is that good,
> acceptable, or bad, and why?

Using '--noexecstack' overrides anything the compiler works out for
itself, so applying it indiscriminately is a bad idea.  For example, if
an application contains asm code with no markings, but also contains
code that creates trampolines, it should be marked for executable stack
even if the asm code is fixed.  Applying '--noexecstack' via LDFLAGS
would break such an application.

Regarding patches, it's usually much simpler to patch asm source code
compared to patching an application's make process.  Patching asm
source code just means appending a few lines depending on the type of
assembler used.

As far as ebuilds are concerned, if you add it to LDFLAGS you will need
to re-check the application every time you bump the ebuild, and it's
difficult to find new occurrences of nested functions for example if
you've applied '--noexecstack'.

- -- 
Kevin F. Quinn
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFDn88O9G2S8dekcG0RAsdDAJ9bhfqc44mtQgBsPu5OFjfNGG0GWQCg0eTA
vU+j9b8nxMtodf5MSXgkfsE=
=nVnR
-----END PGP SIGNATURE-----

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Textrels in packages policy

[Harald van Dijk]

[-- Attachment #1: Type: text/plain, Size: 1744 bytes --]

On Wed, Dec 14, 2005 at 08:51:42AM +0100, Kevin F. Quinn wrote:
> On Wed, 14 Dec 2005 07:59:23 +0100
> Harald van Dijk <truedfx@gentoo.org> wrote:
> 
> > On Wed, Dec 14, 2005 at 03:50:16AM +0000, Mike Frysinger wrote:
> > > my gnu stack docs are actually complete:
> > > http://hardened.gentoo.org/gnu-stack.xml
> > 
> > A question about that: you discourage fixing this with --noexecstack
> > because it's better to be able to submit a patch upstream. What's your
> > take on patches that modify configure scripts or similar files to
> > check for this flag, keeping it out of the ebuild? Is that good,
> > acceptable, or bad, and why?
> 
> Using '--noexecstack' overrides anything the compiler works out for
> itself, so applying it indiscriminately is a bad idea.  For example, if
> an application contains asm code with no markings, but also contains
> code that creates trampolines, it should be marked for executable stack
> even if the asm code is fixed.  Applying '--noexecstack' via LDFLAGS
> would break such an application.
> 
> Regarding patches, it's usually much simpler to patch asm source code
> compared to patching an application's make process.  Patching asm
> source code just means appending a few lines depending on the type of
> assembler used.
> 
> As far as ebuilds are concerned, if you add it to LDFLAGS you will need
> to re-check the application every time you bump the ebuild, and it's
> difficult to find new occurrences of nested functions for example if
> you've applied '--noexecstack'.

LDFLAGS? Assuming you meant ASFLAGS, this doesn't affect C files, and
would need rechecking of the assembly code on updates just as much as
patches which add .note.GNU-stack would, right?

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Textrels in packages policy

[Henrik Brix Andersen]

[-- Attachment #1: Type: text/plain, Size: 545 bytes --]

On Wed, Dec 14, 2005 at 08:44:33AM +0100, Kevin F. Quinn wrote:
> The seriousness of the textrel issue is different for Hardened Gentoo
> and normal Gentoo.  For Hardened Gentoo they cause real problems and
> must to be fixed to avoid compromising the overall strategy.  For
> non-hardened Gentoo it's not so serious.  I'll focus on non-hardened
> Gentoo here.
[snip]

Thank you for this splendid explaination of the issue.

Regards,
Brix
-- 
Henrik Brix Andersen <brix@gentoo.org>
Gentoo Metadistribution | Mobile computing herd

[-- Attachment #2: Type: application/pgp-signature, Size: 211 bytes --]

Re: [gentoo-dev] Textrels in packages policy

[Mike Frysinger]

On Wed, Dec 14, 2005 at 09:19:56AM +0100, Harald van D??k wrote:
> LDFLAGS? Assuming you meant ASFLAGS, this doesn't affect C files,

correct

> would need rechecking of the assembly code on updates just as much as
> patches which add .note.GNU-stack would, right?

no

you were supposed to send that patch upstream, i guess you didnt huh

bad dev
-mike
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Textrels in packages policy

[Harald van Dijk]

[-- Attachment #1: Type: text/plain, Size: 785 bytes --]

On Wed, Dec 14, 2005 at 01:43:28PM +0000, Mike Frysinger wrote:
> On Wed, Dec 14, 2005 at 09:19:56AM +0100, Harald van D??k wrote:
> > LDFLAGS? Assuming you meant ASFLAGS, this doesn't affect C files,
> 
> correct
> 
> > would need rechecking of the assembly code on updates just as much as
> > patches which add .note.GNU-stack would, right?
> 
> no
> 
> you were supposed to send that patch upstream, i guess you didnt huh

In my first message I explained the reason for asking about these kinds
of patches: because your reason against them -- that they can't be sent
upstream -- didn't apply. So I guess this situation would be that I did
send a patch upstream, but it simply wasn't accepted yet. This can
happen with any patch, even those of your preferred kind.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Textrels in packages policy

[Mike Frysinger]

On Wed, Dec 14, 2005 at 03:27:54PM +0100, Harald van D??k wrote:
> On Wed, Dec 14, 2005 at 01:43:28PM +0000, Mike Frysinger wrote:
> > On Wed, Dec 14, 2005 at 09:19:56AM +0100, Harald van D??k wrote:
> > > would need rechecking of the assembly code on updates just as much as
> > > patches which add .note.GNU-stack would, right?
> > 
> > no
> > 
> > you were supposed to send that patch upstream, i guess you didnt huh
> 
> In my first message I explained the reason for asking about these kinds
> of patches: because your reason against them -- that they can't be sent
> upstream -- didn't apply.

there's a few problems with trying to get configure to detect whether
the host assembler supports the --noexecstack option:
 - it's very easy to get the detection wrong and i'd bet money that
   anyone doing it for the first time would screw it up
 - it'll only fix the package if the host binutils are new enough ...
   this is becoming less of a problem every day, but it's still
   not uncommon
 - patching the code to use .note.GNU-stack would work regardless of
   the binutils version

> So I guess this situation would be that I did
> send a patch upstream, but it simply wasn't accepted yet. This can
> happen with any patch, even those of your preferred kind.

then you convince upstream that they're being stupid
-mike
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Textrels in packages policy

[Harald van Dijk]

[-- Attachment #1: Type: text/plain, Size: 601 bytes --]

On Wed, Dec 14, 2005 at 02:38:00PM +0000, Mike Frysinger wrote:
> there's a few problems with trying to get configure to detect whether
> the host assembler supports the --noexecstack option:
>  - it's very easy to get the detection wrong and i'd bet money that
>    anyone doing it for the first time would screw it up
>  - it'll only fix the package if the host binutils are new enough ...
>    this is becoming less of a problem every day, but it's still
>    not uncommon
>  - patching the code to use .note.GNU-stack would work regardless of
>    the binutils version

Okay, thanks for the info.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Textrels in packages policy

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 1824 bytes --]

On Wed, 2005-12-14 at 00:25 +0000, Mike Frysinger wrote:
> On Tue, Dec 13, 2005 at 05:59:17PM -0600, Jory A. Pratt wrote:
> > Mark Loeser wrote:
> > > Basically what I'm looking for here is an easy to understand explanation of
> > > what textrels are, why they are bad, and why they should hold back marking a
> > > package stable.  The only information I've been able to find states that they
> > > could cause a performance hit, but this doesn't seem to warrant banning them
> > > completely in my eyes.
> > > 
> > > Getting a clear cut policy on exactly what issues should hold a package back 
> > > from being marked stable is what I'm looking for.  Issues like textrels, 
> > > executable stacks, etc is what I'm looking for to be defined and explained why 
> > > we are to always avoid them.  This should be added to existing documentation
> > > policy so it is somewhere for new devs to know about, and existing devs to
> > > have for a reference.
> >
> > Only problem I see with this is binary packages. We can not control
> > upstream binaries as everyone is aware of. So when does it become safe
> > to override stable packages that have texrel's and executable stacks?
> 
> no idea what you mean by "override", but here's a crazy idea ... ask
> upstream to fix the issues.  for example, we just reported executable
> stacks with the ut2004 game and Ryan of epicgames was so kind as to
> fix it up for us.  some upstream peeps dont even know about these sort
> of things until you point them out.

Actually, it is not fixed yet.  Ryan is aware of the issue, but it was
not fixed in the latest version.  There was an error on my part that
made me think it was fixed.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead
x86 Architecture Team
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Textrels in packages policy

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 1866 bytes --]

On Wed, 2005-12-14 at 00:25 +0000, Mike Frysinger wrote:
> On Tue, Dec 13, 2005 at 05:59:17PM -0600, Jory A. Pratt wrote:
> > Mark Loeser wrote:
> > > Basically what I'm looking for here is an easy to understand explanation of
> > > what textrels are, why they are bad, and why they should hold back marking a
> > > package stable.  The only information I've been able to find states that they
> > > could cause a performance hit, but this doesn't seem to warrant banning them
> > > completely in my eyes.
> > > 
> > > Getting a clear cut policy on exactly what issues should hold a package back 
> > > from being marked stable is what I'm looking for.  Issues like textrels, 
> > > executable stacks, etc is what I'm looking for to be defined and explained why 
> > > we are to always avoid them.  This should be added to existing documentation
> > > policy so it is somewhere for new devs to know about, and existing devs to
> > > have for a reference.
> >
> > Only problem I see with this is binary packages. We can not control
> > upstream binaries as everyone is aware of. So when does it become safe
> > to override stable packages that have texrel's and executable stacks?
> 
> no idea what you mean by "override", but here's a crazy idea ... ask
> upstream to fix the issues.  for example, we just reported executable
> stacks with the ut2004 game and Ryan of epicgames was so kind as to
> fix it up for us.  some upstream peeps dont even know about these sort
> of things until you point them out.

Let me take that back again.  This is resolved on amd64, but not on x86.
There's 2 binaries.  This also explains why I didn't see the error and
closed the bug, I'm on amd64, whereas the bug reporter is on x86.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead
x86 Architecture Team
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Textrels in packages policy

[Mike Frysinger]

On Wed, Dec 14, 2005 at 10:27:19AM -0500, Chris Gianelloni wrote:
> On Wed, 2005-12-14 at 00:25 +0000, Mike Frysinger wrote:
> > On Tue, Dec 13, 2005 at 05:59:17PM -0600, Jory A. Pratt wrote:
> > > Only problem I see with this is binary packages. We can not control
> > > upstream binaries as everyone is aware of. So when does it become safe
> > > to override stable packages that have texrel's and executable stacks?
> > 
> > no idea what you mean by "override", but here's a crazy idea ... ask
> > upstream to fix the issues.  for example, we just reported executable
> > stacks with the ut2004 game and Ryan of epicgames was so kind as to
> > fix it up for us.  some upstream peeps dont even know about these sort
> > of things until you point them out.
> 
> Let me take that back again.  This is resolved on amd64, but not on x86.
> There's 2 binaries.  This also explains why I didn't see the error and
> closed the bug, I'm on amd64, whereas the bug reporter is on x86.

yeah, Ryan's one cool cat
-mike
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Textrels in packages policy

[Kevin F. Quinn]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, 14 Dec 2005 09:19:56 +0100
Harald van Dijk <truedfx@gentoo.org> wrote:

> On Wed, Dec 14, 2005 at 08:51:42AM +0100, Kevin F. Quinn wrote:
> > On Wed, 14 Dec 2005 07:59:23 +0100
> > Harald van Dijk <truedfx@gentoo.org> wrote:
> > 
> > > On Wed, Dec 14, 2005 at 03:50:16AM +0000, Mike Frysinger wrote:
> > > > my gnu stack docs are actually complete:
> > > > http://hardened.gentoo.org/gnu-stack.xml
> > > 
> > > A question about that: you discourage fixing this with
> > > --noexecstack because it's better to be able to submit a patch
> > > upstream. What's your take on patches that modify configure
> > > scripts or similar files to check for this flag, keeping it out
> > > of the ebuild? Is that good, acceptable, or bad, and why?
> > 
> > Using '--noexecstack' overrides anything the compiler works out for
> > itself, so applying it indiscriminately is a bad idea.  For
> > example, if an application contains asm code with no markings, but
> > also contains code that creates trampolines, it should be marked
> > for executable stack even if the asm code is fixed.  Applying
> > '--noexecstack' via LDFLAGS would break such an application.
> > 
> > Regarding patches, it's usually much simpler to patch asm source
> > code compared to patching an application's make process.  Patching
> > asm source code just means appending a few lines depending on the
> > type of assembler used.
> > 
> > As far as ebuilds are concerned, if you add it to LDFLAGS you will
> > need to re-check the application every time you bump the ebuild,
> > and it's difficult to find new occurrences of nested functions for
> > example if you've applied '--noexecstack'.
> 
> LDFLAGS? Assuming you meant ASFLAGS, this doesn't affect C files, and
> would need rechecking of the assembly code on updates just as much as
> patches which add .note.GNU-stack would, right?

You're right there.  I was thinking of '-Wl,-z,[no]execstack' which
can be used on LDFLAGS, but overrides the setting for the whole
application.

- -- 
Kevin F. Quinn
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFDoRfM9G2S8dekcG0RAoiRAKDcjEaXjLU4AmC+1NLM8zzOZ7DoDQCeJILV
oncYVeaOrMf77XZyRwWCBUA=
=ua9o
-----END PGP SIGNATURE-----

-- 
gentoo-dev@gentoo.org mailing list