[gentoo-dev] RFC: Should preserve-libs be enabled by default?

[gentoo-dev] RFC: Should preserve-libs be enabled by default?

[Marius Mauch]

[-- Attachment #1: Type: text/plain, Size: 2730 bytes --]

As portage-2.2 is about to be unmasked into ~arch soon (there is one
weird bug to solve before) it's time to ask for some input on one of
the important new features, FEATURES=preserve-libs.

(if you're already familiar with it you can skip this paragraph)
Simply said, when this feature is enabled portage keeps track of all
installed libraries and binaries linked against them, and if a package
upgrade would remove a library that's still in use portage will keep
the library around, owned by the new version and also registered in a
separate file. There is also an internal package set that can be used
to rebuild all packages linked against libraries preserved in this way,
and the user is notified after each emerge operation that he should do
that (the example is from an expat downgrade in case you wonder about
the versions): 
!!! existing preserved libs:
>>> package: dev-libs/expat-1.95.8
 *  - /usr/lib64/libexpat.so.1
 *  - /usr/lib64/libexpat.so.1.5.2
Use emerge @preserved-rebuild to rebuild packages using these libraries

The purpose of this is to keep the system operational after library
upgrades until all affected packages could be rebuilt and to simplify
the process, not to avoid the rebuilds.

Now the question is if this behavior should be enabled by default?

In the existing prereleases it has been enabled to get some real-world
testing, and it's been quite effective, though there are still a few
issues to be worked out (e.g. if libraries are moved between packages).
And no doubt a few more bugs will turn up over time.
Also it is not going to be a perfect solution against all runtime link
errors, but if enabled it should eliminate the need for revdep-rebuild
in most cases.
One concern raised by some people is that it might cause old libraries
with security issues to stay on the system for eternity even though
the package was upgraded, and eventually be preferred by new builds.
I can't rule this out completely but thinks it's very unlikely, as
preserved libraries are specially tracked and the user is notified
about their existance after every emerge operation (similar to glep42
news). And new builds shouldn't find them as the unversioned .so
symlinks ar going to point to the current versions.
So personally I'm not too worried about this concern becoming reality,
but I can understand if others are.

So, do you think it should be enabled by default?

Marius

PS: Obviously, if you haven't tested portage-2.2 yet, now would be a
good time.

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-dev] Re: RFC: Should preserve-libs be enabled by default?

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 452 bytes --]

On Thu, 29 May 2008 01:13:16 +0200
Marius Mauch <genone@gentoo.org> wrote:

> So, do you think it should be enabled by default?

Yes please. :)  I haven't had any problems in the couple of months
i've been using it.


-- 
fonts, gcc-porting,                               by design, by neglect
mips, treecleaner,                        for a fact or just for effect
wxwidgets @ gentoo     EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: Should preserve-libs be enabled by default?

[Donnie Berkholz]

On 01:13 Thu 29 May     , Marius Mauch wrote:
> One concern raised by some people is that it might cause old libraries
> with security issues to stay on the system for eternity even though
> the package was upgraded, and eventually be preferred by new builds.
> I can't rule this out completely but thinks it's very unlikely, as
> preserved libraries are specially tracked and the user is notified
> about their existance after every emerge operation (similar to glep42
> news).

Part of this should be addressable by keeping track of the version that 
installed them and checking it against the distributed GLSAs...

Thanks,
Donnie
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: Should preserve-libs be enabled by default?

[Rémi Cardona]

Marius Mauch a écrit :
> So, do you think it should be enabled by default?

Does portage have a way to report which libraries it is keeping around 
because of preserve-libs ? If there's an easy way to figure that out, 
then enabling it by default is a very sane and sound idea.

Cheers,

Rémi
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: Should preserve-libs be enabled by default?

[Fabian Groffen]

On 29-05-2008 08:54:48 +0200, Rémi Cardona wrote:
> Marius Mauch a écrit :
>> So, do you think it should be enabled by default?
>
> Does portage have a way to report which libraries it is keeping around  
> because of preserve-libs ? If there's an easy way to figure that out,  
> then enabling it by default is a very sane and sound idea.

It does so after every merge IIRC, and you can also find them in a file
somewhere.


-- 
Fabian Groffen
Gentoo on a different level
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: Should preserve-libs be enabled by default?

[Mike Auty]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Marius Mauch wrote:
| The purpose of this is to keep the system operational after library
| upgrades until all affected packages could be rebuilt and to simplify
| the process, not to avoid the rebuilds.

I couldn't find it mentioned in your email, but if portage is
effectively doing reference counts, what happens when its reference
count gets to 0?  Once no ebuilds rely on the old library is it removed
automatically, or do the "you need to rebuild these" message just go away?

Is there a way to have portage delete the libraries once it's sure
they're no longer necessary?  If so, is that done by rebuilding the
owning package itself, or by editing the owning pacakge's contents and
removing the old library?

Does @preserved-rebuild contain just the affected packages, or the
package containing the old library as well?  (i.e. Does an "emerge
@preserved-rebuild" ensure that the old library will no longer exist on
your system, or not?)

Basically, if I can safely replace "revdep-rebuild" with "emerge
@preserved-rebuild" then I'd be happy to keep it enabled.  If it's going
to leave cruft on the system (or then require manual rebuilds of
packages containing preserved libraries to clear out the cruft) then I'd
personally be inclined to turn it off and stick with revdep-rebuild...

Mike  5:)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)

iEYEARECAAYFAkg+aSAACgkQu7rWomwgFXoR2ACeJnf+J/pd/GEEh5Ds/Q80sjOR
vIkAoKEyLD2lTGfehoSoYLP6pH/R++2J
=0sv1
-----END PGP SIGNATURE-----
-- 
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] Re: RFC: Should preserve-libs be enabled by default?

[Diego 'Flameeyes' Pettenò]

[-- Attachment #1: Type: text/plain, Size: 1868 bytes --]

Marius Mauch <genone@gentoo.org> writes:

> Also it is not going to be a perfect solution against all runtime link
> errors, but if enabled it should eliminate the need for revdep-rebuild
> in most cases.

I'm afraid that it will turn, for complex libraries like libexpat and
users not using --as-needed, the message telling you the program cannot
be started with subtle crashes for symbol collision.

preserve-libs would be quite perfect if all libraries out there used
versioned symbol, but this is far from true (and some systems Gentoo
runs on don't even consider versioned symbol to begin with).

Example at hand? When the libexpat transition started, the choice of
keeling .so.0 around with .so.1 was discarded right away because:

- library libfoo links to libexpat;
- program bar links to libfoo;
- user is not using --as-needed, so bar has a NEEDED against both libfoo
  and libexpat;
- user rebuilds libfoo, but not bar; or bar and not libfoo, the result
  is the same;
- KABOOM! symbol collision and bar crashes.

As much as we want preserve-libs to be an all-curing magic, it's
not. When you need to replace a library you need to do so _for all its
users at once_, if you allow it to be gradually you're opening the
hellgate of symbol collision.

My solution would be to disallow _building_ anything that is or depends
directly or indirectly on a package on the set until it is removed, or
at the request of merging "mickeymouse", depending on "bar", re-emerging
libfoo first, and bar if the user is not using --as-needed (checking the
NEEDED lines).

With all due respect to everybody, the right course of action here has
to be selected by people who knows how the runtime linker works, symbol
collision and all the rest, as that's what's at stake here.

-- 
Diego "Flameeyes" Pettenò
http://blog.flameeyes.eu/

[-- Attachment #2: Type: application/pgp-signature, Size: 196 bytes --]

Re: [gentoo-dev] RFC: Should preserve-libs be enabled by default?

[Arfrever Frehtes Taifersar Arahesis]

[-- Attachment #1: Type: text/plain, Size: 468 bytes --]

2008-05-29 01:13:16 Marius Mauch napisał(a):
> One concern raised by some people is that it might cause old libraries
> with security issues to stay on the system for eternity even though
> the package was upgraded, and eventually be preferred by new builds.

This shouldn't be a problem, because unneeded preserved libraries are
automatically deleted.

> So, do you think it should be enabled by default?

Yes.

-- 
Arfrever Frehtes Taifersar Arahesis

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: Should preserve-libs be enabled by default?

[Arfrever Frehtes Taifersar Arahesis]

[-- Attachment #1: Type: text/plain, Size: 231 bytes --]

2008-05-29 08:54:48 Rémi Cardona napisał(a):
> Does portage have a way to report which libraries it is keeping around 
> because of preserve-libs ?

portageq list_preserved_libs /

-- 
Arfrever Frehtes Taifersar Arahesis

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: Should preserve-libs be enabled by default?

[Marius Mauch]

On Thu, 29 May 2008 09:28:16 +0100
Mike Auty <ikelos@gentoo.org> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Marius Mauch wrote:
> | The purpose of this is to keep the system operational after library
> | upgrades until all affected packages could be rebuilt and to
> | simplify the process, not to avoid the rebuilds.
> 
> I couldn't find it mentioned in your email, but if portage is
> effectively doing reference counts, what happens when its reference
> count gets to 0?  Once no ebuilds rely on the old library is it
> removed automatically, or do the "you need to rebuild these" message
> just go away?

They are removed automatically.

Marius
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: RFC: Should preserve-libs be enabled by default?

[Marius Mauch]

On Thu, 29 May 2008 11:02:55 +0200
flameeyes@gmail.com (Diego 'Flameeyes' Pettenò) wrote:

> Marius Mauch <genone@gentoo.org> writes:
> 
> As much as we want preserve-libs to be an all-curing magic, it's
> not. When you need to replace a library you need to do so _for all its
> users at once_, if you allow it to be gradually you're opening the
> hellgate of symbol collision.

That's what `emerge @preserved-rebuild` does, or do you mean something
different?

> My solution would be to disallow _building_ anything that is or
> depends directly or indirectly on a package on the set until it is
> removed, or at the request of merging "mickeymouse", depending on
> "bar", re-emerging libfoo first, and bar if the user is not using
> --as-needed (checking the NEEDED lines).

Well, with preserve-libs the situation is this (using your example):
- user upgrades expat, portage keeps libexpat.so.0 around
(some packages might now be linked against both versions if the session
included other packages as well)
- emerge tells the user to rebuild all affected packages (affected =
contains libexpat.so.0 in NEEDED, so includes both libfoo and bar) by
using `emerge @preserved-rebuild` (in the future this could also be done
automatically, but that won't be before 2.2 final)
- when all affected packages have been rebuilt (so their NEEDED entries
don't contain libexpat.so.0 anymore) libexpat.so.0 is automatically
removed

So, if I understand you correctly (probably not), you want portage to
prevent the user from building any packages depending on any affected
package before the last step is completed?

> With all due respect to everybody, the right course of action here has
> to be selected by people who knows how the runtime linker works,
> symbol collision and all the rest, as that's what's at stake here.

Whoever that is is welcome to voice his opinion here, that's the point
of this thread after all.

Marius
--
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: RFC: Should preserve-libs be enabled by default?

[Peter Volkov]

В Чтв, 29/05/2008 в 11:02 +0200, Diego 'Flameeyes' Pettenò пишет:
> I'm afraid that it will turn, for complex libraries like libexpat and
> users not using --as-needed, the message telling you the program cannot
> be started with subtle crashes for symbol collision.

Is there any reason why --as-needed is not enabled "by default"?

-- 
Peter.

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: RFC: Should preserve-libs be enabled by default?

[Mike Auty]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Peter Volkov wrote:
| Is there any reason why --as-needed is not enabled "by default"?

There's still about 18 open bugs on the tracker[1] for it.  You can see
how many problems it had been causing by the huge number of blocking bugs.

I've been using it for a pretty long time now (probably a couple weeks
after Diego first blogged about it) and don't have many problems at all
(now), but every once in a while a version bump or a new package will
just fail to compile properly and the problem leads back to as-needed.
I'm not sure whether ~arch users would be able to catch all the
as-needed bugs before they hit stable, so I couldn't say whether it
should be enabled by default or not.

I also don't think it would completely eliminate the problems that Diego
mentioned with preserve-libs (there could still be instances where bar
and foo both NEEDED thing.so, but bar had been compiled more recently
and needed thing.so.1 whilst foo needed thing.so.0, and starting bar
would break trying to load both)...

Mike  5:)

[1] http://bugs.gentoo.org/show_bug.cgi?id=129413
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)

iEYEARECAAYFAkg/qb4ACgkQu7rWomwgFXpl7wCdFhDuZbQOVy1b12dgXbZbSWtj
dIMAn3Z6FDx5HW1KD83JxdboNrQOOap1
=Nca2
-----END PGP SIGNATURE-----
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: RFC: Should preserve-libs be enabled by default?

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 553 bytes --]

On Fri, 30 May 2008 10:55:51 +0400
Peter Volkov <pva@gentoo.org> wrote:
> В Чтв, 29/05/2008 в 11:02 +0200, Diego 'Flameeyes' Pettenò пишет:
> > I'm afraid that it will turn, for complex libraries like libexpat
> > and users not using --as-needed, the message telling you the
> > program cannot be started with subtle crashes for symbol collision.
> 
> Is there any reason why --as-needed is not enabled "by default"?

--as-needed is fundamentally broken by design and causes legitimate code
to break.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-dev] Re: RFC: Should preserve-libs be enabled by default?

[Diego 'Flameeyes' Pettenò]

[-- Attachment #1: Type: text/plain, Size: 1945 bytes --]

Marius Mauch <genone@gentoo.org> writes:

> That's what `emerge @preserved-rebuild` does, or do you mean something
> different?

I meant something different, see the rst.
> Well, with preserve-libs the situation is this (using your example):
> - user upgrades expat, portage keeps libexpat.so.0 around
> (some packages might now be linked against both versions if the session
> included other packages as well)
> - emerge tells the user to rebuild all affected packages (affected =
> contains libexpat.so.0 in NEEDED, so includes both libfoo and bar) by
> using `emerge @preserved-rebuild` (in the future this could also be done
> automatically, but that won't be before 2.2 final)
> - when all affected packages have been rebuilt (so their NEEDED entries
> don't contain libexpat.so.0 anymore) libexpat.so.0 is automatically
> removed

Okay this works if the user follows the procedure and tries not to bend
the rules...

> So, if I understand you correctly (probably not), you want portage to
> prevent the user from building any packages depending on any affected
> package before the last step is completed?

Yes this is exactly what I meant. Whenever a dependency is in the
@preserved-rebuild set, it should not be linked against. It could still
be used, but as now we don't have an easy way to distinguish between the
two, I'd say it's better to check both DEPEND and RDEPEND and disallow
its usage as a dependency until it's removed from the set.

> Whoever that is is welcome to voice his opinion here, that's the point
> of this thread after all.

(It was mostly a disclaimer so that users don't feel like they get
ignored without just cause if they want behaviour X and instead we go
with behaviour Y... I'm sure a lot of people wouldn't like the option I
proposed above, but trust me there's a reason why I voiced that concern
:) ).

-- 
Diego "Flameeyes" Pettenò
http://blog.flameeyes.eu/

[-- Attachment #2: Type: application/pgp-signature, Size: 196 bytes --]

Re: [gentoo-dev] Re: RFC: Should preserve-libs be enabled by default?

[Santiago M. Mola]

On Fri, May 30, 2008 at 9:16 AM, Mike Auty <ikelos@gentoo.org> wrote:
>
> Peter Volkov wrote:
> | Is there any reason why --as-needed is not enabled "by default"?
>
> There's still about 18 open bugs on the tracker[1] for it.  You can see
> how many problems it had been causing by the huge number of blocking bugs.
>
> I've been using it for a pretty long time now (probably a couple weeks
> after Diego first blogged about it) and don't have many problems at all
> (now), but every once in a while a version bump or a new package will
> just fail to compile properly and the problem leads back to as-needed.
> I'm not sure whether ~arch users would be able to catch all the
> as-needed bugs before they hit stable, so I couldn't say whether it
> should be enabled by default or not.

That's not a problem at all. If we choose to support --as-needed by
default we'd get testing from maintainers when adding new ebuilds, and
from arch teams before ebuilds hit stable.

--as-needed breaking legitimate code is a problem, though. I wonder if
we have that kind of code in any application in the tree and if we
have some way to detect it.

Regards,
-- 
Santiago M. Mola
Jabber ID: cooldwind@gmail.com
-- 
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] Re: RFC: Should preserve-libs be enabled by default?

[Diego 'Flameeyes' Pettenò]

[-- Attachment #1: Type: text/plain, Size: 821 bytes --]

"Santiago M. Mola" <coldwind@gentoo.org> writes:

> --as-needed breaking legitimate code is a problem, though. I wonder if
> we have that kind of code in any application in the tree and if we
> have some way to detect it.

You could be looking for code not supposed to work under Solaris or
Windows, as the --as-needed behaviour behaves a lot like the standard
Sun linker and the forced behaviour of the PE executable format.

The only thing that can be broken by using --as-needed is code that
assumes the order in calling the .init sections of a set of shared
objects. Such an order is not only changed by --as-needed usage but by
any other change in the loading mechanism. It is strictly related to
glibc at this point as far as I can tell.

-- 
Diego "Flameeyes" Pettenò
http://blog.flameeyes.eu/

[-- Attachment #2: Type: application/pgp-signature, Size: 196 bytes --]

Re: [gentoo-dev] Re: RFC: Should preserve-libs be enabled by default?

[Rémi Cardona]

Santiago M. Mola a écrit :
> On Fri, May 30, 2008 at 9:16 AM, Mike Auty <ikelos@gentoo.org> wrote:
>> I've been using it for a pretty long time now (probably a couple weeks
>> after Diego first blogged about it) and don't have many problems at all
>> (now), but every once in a while a version bump or a new package will
>> just fail to compile properly and the problem leads back to as-needed.
>> I'm not sure whether ~arch users would be able to catch all the
>> as-needed bugs before they hit stable, so I couldn't say whether it
>> should be enabled by default or not.

Our experience in the gnome herd is that most --as-needed issues come up 
*very* quickly: either the build fails, or application plugins don't get 
loaded. As such, they're quite easy to catch. Fixing them can be trickier.

> --as-needed breaking legitimate code is a problem, though. I wonder if
> we have that kind of code in any application in the tree and if we
> have some way to detect it.

--as-needed breaks legitimate C++ code, I have yet to see it break plain 
C code (but I could be wrong). Ciaran posted an example code a while 
ago, either here or on his blog.

His example is valid C++ code (basically it was about implicitly loading 
.so plugins) but IMHO, it's just not a good engineering practice. Plugin 
loading should be explicit (readdir + dlopen). But that's my opinion. 
There could be a couple other odd cases but I don't remember them.

All I can say is that --as-needed *today* does much more good than harm.

Cheers

-- 
Rémi Cardona
LRI, INRIA
remi.cardona@lri.fr
remi@gentoo.org
-- 
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] Re: RFC: Should preserve-libs be enabled by default?

[Diego 'Flameeyes' Pettenò]

[-- Attachment #1: Type: text/plain, Size: 878 bytes --]

Rémi Cardona <remi@gentoo.org> writes:

> --as-needed breaks legitimate C++ code, I have yet to see it break
> plain C code (but I could be wrong).

This because C does not have constructors or static objects. I suppose I
can write a couple of C files that can present the problem, but really,
they would break as soon as I change the loading rules by moving to a
non-ELF system or a different dynamic loader.

It's interesting to note that Microsoft _did_ think of this problem when
designing the .NET framework, and they answer is ensuring just that the
static constructor will be called (right) before any static method or
attribute is referenced and (right) before any object of the given class
is istantiated. They don't get otherwise an absolute order in which
static constructors are called.

-- 
Diego "Flameeyes" Pettenò
http://blog.flameeyes.eu/

[-- Attachment #2: Type: application/pgp-signature, Size: 196 bytes --]

Re: [gentoo-dev] Re: RFC: Should preserve-libs be enabled by default?

[David Leverton]

On Friday 30 May 2008 13:22:15 Diego 'Flameeyes' Pettenò wrote:
> The only thing that can be broken by using --as-needed is code that
> assumes the order in calling the .init sections of a set of shared
> objects. Such an order is not only changed by --as-needed usage but by
> any other change in the loading mechanism. It is strictly related to
> glibc at this point as far as I can tell.

It's not just the order, it also breaks things that rely on the .init section 
being called at all to register themselves with the core application 
(with --as-needed, the .so doesn't get loaded in the first place, so it 
doesn't get a chance to run anything).

> It's interesting to note that Microsoft _did_ think of this problem when
> designing the .NET framework, and they answer is ensuring just that the
> static constructor will be called (right) before any static method or
> attribute is referenced and (right) before any object of the given class
> is istantiated. They don't get otherwise an absolute order in which
> static constructors are called.

That's /an/ answer, but it doesn't provide all the functionality that static 
objects in C++ have.
--
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] Re: RFC: Should preserve-libs be enabled by default?

[Diego 'Flameeyes' Pettenò]

[-- Attachment #1: Type: text/plain, Size: 1094 bytes --]

David Leverton <levertond@googlemail.com> writes:

> It's not just the order, it also breaks things that rely on the .init
> section being called at all to register themselves with the core
> application (with --as-needed, the .so doesn't get loaded in the first
> place, so it doesn't get a chance to run anything).

This really is backward, solution-wise: you expect the "core
application" to know enough of the plugins to link them together, but
not enough to call their init functions...

And still, it breaks not only with --as-needed but also with the Sun
linker, with the PE file format and other non-ELF file formats. So it's
really not a problem _limited_ to --as-needed.

Beside, you can _force_ a link by using --no-as-needed before a given
library, that's what it's supposed to happen if you don't want to be
helped by the linker.

To a very minimum this can be said to be a _clash_ between two
designs. Saying that --as-needed is _broken_ because it breaks this case
is _quite_ an exaggeration...

-- 
Diego "Flameeyes" Pettenò
http://blog.flameeyes.eu/

[-- Attachment #2: Type: application/pgp-signature, Size: 196 bytes --]

Re: [gentoo-dev] Re: RFC: Should preserve-libs be enabled by default?

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1565 bytes --]

On Fri, 30 May 2008 18:29:49 +0200
flameeyes@gmail.com (Diego 'Flameeyes' Pettenò) wrote:
> David Leverton <levertond@googlemail.com> writes:
> > It's not just the order, it also breaks things that rely on
> > the .init section being called at all to register themselves with
> > the core application (with --as-needed, the .so doesn't get loaded
> > in the first place, so it doesn't get a chance to run anything).
> 
> This really is backward, solution-wise: you expect the "core
> application" to know enough of the plugins to link them together, but
> not enough to call their init functions...

Actually, no. The core application doesn't have to be doing the
linking. The linking can be done by an intermediate library.

> And still, it breaks not only with --as-needed but also with the Sun
> linker, with the PE file format and other non-ELF file formats. So
> it's really not a problem _limited_ to --as-needed.

It works with any standard-compliant C++ setup.

> Beside, you can _force_ a link by using --no-as-needed before a given
> library, that's what it's supposed to happen if you don't want to be
> helped by the linker.

And next you'll be saying "and you can force a compile using
--no-broken-behaviour-that-goes-against-the-standard".

> To a very minimum this can be said to be a _clash_ between two
> designs. Saying that --as-needed is _broken_ because it breaks this
> case is _quite_ an exaggeration...

Saying it's broken because it goes against an international standard
isn't...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: RFC: Should preserve-libs be enabled by default?

[David Leverton]

On Friday 30 May 2008 17:29:49 Diego 'Flameeyes' Pettenò wrote:
> This really is backward, solution-wise: you expect the "core
> application" to know enough of the plugins to link them together, but
> not enough to call their init functions...

Why should it call their init functions, when a static object with a 
constructor can do the job just fine?
--
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: RFC: Should preserve-libs be enabled by default?

[Luca Barbato]

David Leverton wrote:
> On Friday 30 May 2008 17:29:49 Diego 'Flameeyes' Pettenò wrote:
>> This really is backward, solution-wise: you expect the "core
>> application" to know enough of the plugins to link them together, but
>> not enough to call their init functions...
> 
> Why should it call their init functions, when a static object with a 
> constructor can do the job just fine?

Talk to the upstream about this, probably getting a satisfying solution 
isn't that difficult.

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: RFC: Should preserve-libs be enabled by default?

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 343 bytes --]

On Fri, 30 May 2008 21:13:32 +0200
Luca Barbato <lu_zero@gentoo.org> wrote:
> Talk to the upstream about this, probably getting a satisfying
> solution isn't that difficult.

The solution is to use --as-needed in the same way that -ffast-math is
used: only with applications specifically designed to support it.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: RFC: Should preserve-libs be enabled by default?

[Luca Barbato]

Ciaran McCreesh wrote:
> --as-needed is fundamentally broken by design and causes legitimate code
> to break.

Basically C++ quasi-standard corner cases nobody uses, that happen to 
work on ELF only and that should be removed in the next revision of 0x ?

Implicit plugin loading isn't exactly a sane thing.

lu - less excuses to laziness please.

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: RFC: Should preserve-libs be enabled by default?

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 483 bytes --]

On Fri, 30 May 2008 21:29:26 +0200
Luca Barbato <lu_zero@gentoo.org> wrote:
> Ciaran McCreesh wrote:
> > --as-needed is fundamentally broken by design and causes legitimate
> > code to break.
> 
> Basically C++ quasi-standard corner cases nobody uses, that happen to 
> work on ELF only and that should be removed in the next revision of
> 0x ?

It's going to end up being used more, not less, in 0x, thanks to
constexpr and the new thread rules.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Re: RFC: Should preserve-libs be enabled by default?

[Mart Raudsepp]

On R, 2008-05-30 at 20:20 +0100, Ciaran McCreesh wrote:
> On Fri, 30 May 2008 21:13:32 +0200
> Luca Barbato <lu_zero@gentoo.org> wrote:
> > Talk to the upstream about this, probably getting a satisfying
> > solution isn't that difficult.
> 
> The solution is to use --as-needed in the same way that -ffast-math is
> used: only with applications specifically designed to support it.

You mean everything but paludis?

Doesn't your grand plan include supporting Prefix and Interix with PE
binaries and so on?

I know projects that need to work around static initialization not being
reliable - they only happen to have done that for other reasons (such as
Windows PE format, iirc) years before --as-needed was implemented for
binutils.
Standards is one thing - reality is something quite different.
The reality is that everything designed to work everywhere is just
mighty happy with --as-needed and lots of benefits to gain from it.

-- 
Mart Raudsepp
Gentoo Developer
Mail: leio@gentoo.org
Weblog: http://planet.gentoo.org/developers/leio

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: RFC: Should preserve-libs be enabled by default?

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1556 bytes --]

On Sat, 31 May 2008 00:31:22 +0300
Mart Raudsepp <leio@gentoo.org> wrote:
> On R, 2008-05-30 at 20:20 +0100, Ciaran McCreesh wrote:
> > On Fri, 30 May 2008 21:13:32 +0200
> > Luca Barbato <lu_zero@gentoo.org> wrote:
> > > Talk to the upstream about this, probably getting a satisfying
> > > solution isn't that difficult.
> > 
> > The solution is to use --as-needed in the same way that -ffast-math
> > is used: only with applications specifically designed to support it.
> 
> You mean everything but paludis?

Paludis is fine with as-needed. But hey, don't let reality get in the
way of your pathetic attempts at turning everything into Paludis
bashing.

> Doesn't your grand plan include supporting Prefix and Interix with PE
> binaries and so on?

I have no particular interest in supporting any platform that can't
ship a Standard-compliant C++ environment.

> I know projects that need to work around static initialization not
> being reliable - they only happen to have done that for other reasons
> (such as Windows PE format, iirc) years before --as-needed was
> implemented for binutils.
> Standards is one thing - reality is something quite different.
> The reality is that everything designed to work everywhere is just
> mighty happy with --as-needed and lots of benefits to gain from it.

And twenty years ago C++ had to work around linkers that only supported
eight character symbol names. Reality moves forward, except in
situations like these where people try to rice it backwards.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Mart Raudsepp]

On R, 2008-05-30 at 22:37 +0100, Ciaran McCreesh wrote:
> On Sat, 31 May 2008 00:31:22 +0300
> Mart Raudsepp <leio@gentoo.org> wrote:
> > On R, 2008-05-30 at 20:20 +0100, Ciaran McCreesh wrote:
> > > On Fri, 30 May 2008 21:13:32 +0200
> > > Luca Barbato <lu_zero@gentoo.org> wrote:
> > > > Talk to the upstream about this, probably getting a satisfying
> > > > solution isn't that difficult.
> > > 
> > > The solution is to use --as-needed in the same way that -ffast-math
> > > is used: only with applications specifically designed to support it.
> > 
> > You mean everything but paludis?
> 
> Paludis is fine with as-needed. But hey, don't let reality get in the
> way of your pathetic attempts at turning everything into Paludis
> bashing.

It happens to be the only package that I know of that couldn't be fixed
to work with --as-needed (fix for others being to actually state linking
with a library whose symbols are directly used). I have not heard of
anything else.

> > Doesn't your grand plan include supporting Prefix and Interix with PE
> > binaries and so on?
> 
> I have no particular interest in supporting any platform that can't
> ship a Standard-compliant C++ environment.

That doesn't mean Gentoo progress, in maintainability of a running
system through the ease of ABI breaks meaning magnitudes of less
recompilations, should be inhibited.

> > I know projects that need to work around static initialization not
> > being reliable - they only happen to have done that for other reasons
> > (such as Windows PE format, iirc) years before --as-needed was
> > implemented for binutils.
> > Standards is one thing - reality is something quite different.
> > The reality is that everything designed to work everywhere is just
> > mighty happy with --as-needed and lots of benefits to gain from it.
> 
> And twenty years ago C++ had to work around linkers that only supported
> eight character symbol names. Reality moves forward, except in
> situations like these where people try to rice it backwards.

Maybe you'd like to tell that to the authors of the platforms that don't
support this extreme corner case, but are amongst the platforms that we
do somewhat support in Gentoo?

The story that matters here is, that a C++ corner case that does not
work on 0.01% of packages with --as-needed and breaks on non-ELF
platforms, should not cause good things for our users to be shot down.

99.9% packages in the tree work just great with --as-needed with many
benefits, including ABI break pain reduction (and less importantly
memory savings from dirty library private memory pages), so given that
percentage the default should be what makes things better for users with
exceptions for those tiny percentage of packages that fall into the
corner case (that break on more exotic platforms anyway and arguably
should be fixed).

Portage developers - is there anything we should do to get --as-needed
to make.conf.example and other places, beyond fixing the known bugs on
the appropriate bug tracker?

-- 
Mart Raudsepp
Gentoo Developer
Mail: leio@gentoo.org
Weblog: http://planet.gentoo.org/developers/leio

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1554 bytes --]

On Sat, 31 May 2008 00:47:44 +0300
Mart Raudsepp <leio@gentoo.org> wrote:
> > Paludis is fine with as-needed. But hey, don't let reality get in
> > the way of your pathetic attempts at turning everything into Paludis
> > bashing.
> 
> It happens to be the only package that I know of that couldn't be
> fixed to work with --as-needed (fix for others being to actually
> state linking with a library whose symbols are directly used). I have
> not heard of anything else.

Except that Paludis is fine with --as-needed.

> That doesn't mean Gentoo progress, in maintainability of a running
> system through the ease of ABI breaks meaning magnitudes of less
> recompilations, should be inhibited.

as-needed isn't the right way to do this.

> > And twenty years ago C++ had to work around linkers that only
> > supported eight character symbol names. Reality moves forward,
> > except in situations like these where people try to rice it
> > backwards.
> 
> Maybe you'd like to tell that to the authors of the platforms that
> don't support this extreme corner case, but are amongst the platforms
> that we do somewhat support in Gentoo?

Not really. They'll fix it sooner or later. Probably sooner, once C++0x
starts being widely used.

> The story that matters here is, that a C++ corner case that does not
> work on 0.01% of packages with --as-needed and breaks on non-ELF
> platforms, should not cause good things for our users to be shot down.

You could say the same thing for -ffast-math...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Donnie Berkholz]

On 22:53 Fri 30 May     , Ciaran McCreesh wrote:
> On Sat, 31 May 2008 00:47:44 +0300
> Mart Raudsepp <leio@gentoo.org> wrote:
> > The story that matters here is, that a C++ corner case that does not
> > work on 0.01% of packages with --as-needed and breaks on non-ELF
> > platforms, should not cause good things for our users to be shot down.
> 
> You could say the same thing for -ffast-math...

When there's a feature that only breaks one package that we know of, 
wouldn't it make more sense to enable it globally and add an exception 
than to do it the other way around?

I see that a number of packages in the tree explicitly filter 
-ffast-math.

Thanks,
Donnie
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1435 bytes --]

On Fri, 30 May 2008 15:07:43 -0700
Donnie Berkholz <dberkholz@gentoo.org> wrote:
> On 22:53 Fri 30 May     , Ciaran McCreesh wrote:
> > On Sat, 31 May 2008 00:47:44 +0300
> > Mart Raudsepp <leio@gentoo.org> wrote:
> > > The story that matters here is, that a C++ corner case that does
> > > not work on 0.01% of packages with --as-needed and breaks on
> > > non-ELF platforms, should not cause good things for our users to
> > > be shot down.
> > 
> > You could say the same thing for -ffast-math...
> 
> When there's a feature that only breaks one package that we know of, 
> wouldn't it make more sense to enable it globally and add an
> exception than to do it the other way around?

Both -ffast-math and --as-needed make the compiler / linker violate
various standards in ways that can't be used safely unless a package
has been explicitly designed to work with it. For packages that have
been explicitly designed to work with either, upstream can add the
options to the build system themselves. For packages that haven't, it's
not Gentoo's place to try to guess whether upstream has designed their
software with ricer flags in mind, and whether if it works by fluke
now it'll still work in the next version.

> I see that a number of packages in the tree explicitly filter 
> -ffast-math.

That's mostly from the bad old days when users were encouraged to use
silly CFLAGS...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Luca Barbato]

Ciaran McCreesh wrote:
> On Sat, 31 May 2008 00:47:44 +0300
> Mart Raudsepp <leio@gentoo.org> wrote:
>>> Paludis is fine with as-needed. But hey, don't let reality get in
>>> the way of your pathetic attempts at turning everything into Paludis
>>> bashing.
>> It happens to be the only package that I know of that couldn't be
>> fixed to work with --as-needed (fix for others being to actually
>> state linking with a library whose symbols are directly used). I have
>> not heard of anything else.
> 
> Except that Paludis is fine with --as-needed.
> 

Ok, then everything in the tree is covered and we can move to having 
--as-needed as default.

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Luca Barbato]

Ciaran McCreesh wrote:
> On Fri, 30 May 2008 15:07:43 -0700
> Donnie Berkholz <dberkholz@gentoo.org> wrote:
>> On 22:53 Fri 30 May     , Ciaran McCreesh wrote:
>>> On Sat, 31 May 2008 00:47:44 +0300
>>> Mart Raudsepp <leio@gentoo.org> wrote:
>>>> The story that matters here is, that a C++ corner case that does
>>>> not work on 0.01% of packages with --as-needed and breaks on
>>>> non-ELF platforms, should not cause good things for our users to
>>>> be shot down.
>>> You could say the same thing for -ffast-math...
>> When there's a feature that only breaks one package that we know of, 
>> wouldn't it make more sense to enable it globally and add an
>> exception than to do it the other way around?
> 
> Both -ffast-math and --as-needed make the compiler / linker violate
> various standards in ways that can't be used safely unless a package
> has been explicitly designed to work with it.

I know exactly which standard -ffast-math violates (IEEE/ISO floating 
point spec) and how (the man page is quite complete about this), 
--as-needed doesn't have any warning about this, there isn't any 
standard that it violates since it's the default behavior at least for 2 
platform (one from those who wrote most of the ELF spec...).
Point the spec, and the paragraph violated.

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 564 bytes --]

On Sat, 31 May 2008 01:08:21 +0200
Luca Barbato <lu_zero@gentoo.org> wrote:
> > Except that Paludis is fine with --as-needed.
> 
> Ok, then everything in the tree is covered and we can move to having 
> --as-needed as default.

Is the next version of everything in the tree covered? Have you made
sure that software isn't merely working by fluke? Is Gentoo really that
desperate to turn everyone into a ricer?

I'd bet you could get a pretty long way by shoving -ffast-math into
CFLAGS by default before anyone would notice...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Luca Barbato]

Ciaran McCreesh wrote:
> I'd bet you could get a pretty long way by shoving -ffast-math into
> CFLAGS by default before anyone would notice...
> 

Non sequitur. We are talking about --as-needed, not -ffast-math.

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 550 bytes --]

On Sat, 31 May 2008 01:13:58 +0200
Luca Barbato <lu_zero@gentoo.org> wrote:
> I know exactly which standard -ffast-math violates (IEEE/ISO floating 
> point spec) and how (the man page is quite complete about this), 
> --as-needed doesn't have any warning about this, there isn't any 
> standard that it violates since it's the default behavior at least
> for 2 platform (one from those who wrote most of the ELF spec...).
> Point the spec, and the paragraph violated.

ISO/IEC 14882:1998 section 3.7.1 paragraph 2.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 698 bytes --]

On Sat, May 31, 2008 at 12:26:44AM +0100, Ciaran McCreesh wrote:
> On Sat, 31 May 2008 01:13:58 +0200
> Luca Barbato <lu_zero@gentoo.org> wrote:
> > I know exactly which standard -ffast-math violates (IEEE/ISO floating 
> > point spec) and how (the man page is quite complete about this), 
> > --as-needed doesn't have any warning about this, there isn't any 
> > standard that it violates since it's the default behavior at least
> > for 2 platform (one from those who wrote most of the ELF spec...).
> > Point the spec, and the paragraph violated.
> 
> ISO/IEC 14882:1998 section 3.7.1 paragraph 2.

Might want to confirm it's in 14882:2003, since the '98 was withdrawn...
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 349 bytes --]

On Fri, 30 May 2008 16:43:38 -0700
Brian Harring <ferringb@gmail.com> wrote:
> > ISO/IEC 14882:1998 section 3.7.1 paragraph 2.
> 
> Might want to confirm it's in 14882:2003, since the '98 was
> withdrawn... ~harring

It's in all the way up to the current 0x draft. It even has the same
section and paragraph number.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Luca Barbato]

Ciaran McCreesh wrote:
> On Sat, 31 May 2008 01:13:58 +0200
> Luca Barbato <lu_zero@gentoo.org> wrote:
>> I know exactly which standard -ffast-math violates (IEEE/ISO floating 
>> point spec) and how (the man page is quite complete about this), 
>> --as-needed doesn't have any warning about this, there isn't any 
>> standard that it violates since it's the default behavior at least
>> for 2 platform (one from those who wrote most of the ELF spec...).
>> Point the spec, and the paragraph violated.
> 
> ISO/IEC 14882:1998 section 3.7.1 paragraph 2.
> 

"If an object of static storage duration has initialization or a 
destructor with side effects, it shall not be eliminated even if
it appears to be unused, except that a class object or its copy
may be eliminated as specified in 12.8."

Unchanged in the 2003 revision.

Is that related to linking? I don't think so. Still, PE and ELF are 
older than the first C++ spec so, IFF your reading of this chapter is 
correct, C++ is broken by design.

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 972 bytes --]

On Sat, 31 May 2008 01:54:45 +0200
Luca Barbato <lu_zero@gentoo.org> wrote:
> > ISO/IEC 14882:1998 section 3.7.1 paragraph 2.
> 
> "If an object of static storage duration has initialization or a 
> destructor with side effects, it shall not be eliminated even if
> it appears to be unused, except that a class object or its copy
> may be eliminated as specified in 12.8."
> 
> Unchanged in the 2003 revision.
> 
> Is that related to linking? I don't think so.

Linking with as-needed is the stage in which the elimination occurs,
and as-needed is the cause of the elimination. So yes, it is related.

> Still, PE and ELF are older than the first C++ spec so, IFF your
> reading of this chapter is correct, C++ is broken by design.

Not at all. Read "The Design and Evolution of C++", and you shall see
that requiring changes to the linker where necessary for sensible
behaviour was considered acceptable, and with good reason.
-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Luca Barbato]

Ciaran McCreesh wrote:
> Linking with as-needed is the stage in which the elimination occurs,
> and as-needed is the cause of the elimination. So yes, it is related.

The linker just does bookkeeping, if there aren't symbols used, the 
library won't be in the list.

>> Still, PE and ELF are older than the first C++ spec so, IFF your
>> reading of this chapter is correct, C++ is broken by design.
> 
> Not at all. Read "The Design and Evolution of C++", and you shall see
> that requiring changes to the linker where necessary for sensible
> behaviour was considered acceptable, and with good reason.

As in "we have a square wheels, let's make routes for them"...

Anyway is the book a standard? Is it available as pdf so you can point 
me the exact paragraph?

lu - changing the world so non euclidean aberrations fit isn't sensible

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Marius Mauch]

On Sat, 31 May 2008 00:47:44 +0300
Mart Raudsepp <leio@gentoo.org> wrote:

> Portage developers - is there anything we should do to get --as-needed
> to make.conf.example and other places, beyond fixing the known bugs on
> the appropriate bug tracker?

make.conf.example is no big deal, that's just documentation, though
personally I'd prefer to remove the toolchain (CHOST, *FLAGS) stuff from
there so we don't have to maintain all the arch-specific diffs.

If you want to really enable it by default you'll have to persuade the
profile maintainers to add it into the appropriate make.defaults.

Marius
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1985 bytes --]

On Sat, 31 May 2008 02:17:15 +0200
Luca Barbato <lu_zero@gentoo.org> wrote:
> Ciaran McCreesh wrote:
> > Linking with as-needed is the stage in which the elimination occurs,
> > and as-needed is the cause of the elimination. So yes, it is
> > related.
> 
> The linker just does bookkeeping, if there aren't symbols used, the 
> library won't be in the list.

Which is where the design flaw is -- as-needed incorrectly assumes that
the only type of dependency between shared objects is a name
dependency. This isn't true with C++ static initialisers.

> >> Still, PE and ELF are older than the first C++ spec so, IFF your
> >> reading of this chapter is correct, C++ is broken by design.
> > 
> > Not at all. Read "The Design and Evolution of C++", and you shall
> > see that requiring changes to the linker where necessary for
> > sensible behaviour was considered acceptable, and with good reason.
> 
> As in "we have a square wheels, let's make routes for them"...

More like "getting the linker right is important enough to us that
we'll pester people to make their wheels at least octagonal rather than
the current square". Unfortunately, as-needed is moving back to square.

> Anyway is the book a standard? Is it available as pdf so you can
> point me the exact paragraph?

The book is an explanation of why the standard is the way it is. You
can find it at your local library.

Whilst we're on the subject... You'll note that as-needed overrides
explicit instructions from the programmer. When you say "link A to B",
you aren't say "link A to B unless you feel like not doing the link".
Unfortunately, the ricers shoving as-needed upon everyone aren't smart
enough to fix libtool, which is the real problem here, so they go for
the thing they think they understand instead, without thinking the
implications through -- as-needed, like fast-math, is for programs
explicitly designed for it, not for universal use.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Luca Barbato]

Ciaran McCreesh wrote:
> Which is where the design flaw is -- as-needed incorrectly assumes that
> the only type of dependency between shared objects is a name
> dependency. This isn't true with C++ static initialisers.

I don't see why should be different than abusing .init in any other 
language that let you do (ok, C, C++, asm mostly).

> Unfortunately, the ricers shoving as-needed upon everyone aren't smart

Asking people to not do stuff that is unportable (Solaris and PE based 
systems) seems sensible and not ricing.

> enough to fix libtool, which is the real problem here, so they go for
> the thing they think they understand instead, without thinking the
> implications through -- as-needed, like fast-math, is for programs
> explicitly designed for it, not for universal use.

Differently -ffast-math is setting up a slightly different behavior than 
the usual standard, --as-needed enforce what is the default standard in 
determined architectures, thus the exception and the universality are 
quite reverted.

We already started to think how to fix libtool, or at least make it less 
annoying, removing .la files when they are not necessary.

Similarly we started proposing upstream to use pkg-config if they aren't 
already.

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 2077 bytes --]

On Sat, 31 May 2008 03:03:42 +0200
Luca Barbato <lu_zero@gentoo.org> wrote:
> Ciaran McCreesh wrote:
> > Which is where the design flaw is -- as-needed incorrectly assumes
> > that the only type of dependency between shared objects is a name
> > dependency. This isn't true with C++ static initialisers.
> 
> I don't see why should be different than abusing .init in any other 
> language that let you do (ok, C, C++, asm mostly).

In C++ it's not abuse. It's using the language as specified and
designed.

> > Unfortunately, the ricers shoving as-needed upon everyone aren't
> > smart
> 
> Asking people to not do stuff that is unportable (Solaris and PE
> based systems) seems sensible and not ricing.

Not where "unportable" means "doesn't work on systems that fail to
correctly implement widely used international standards" it doesn't.

You might as well say "you shouldn't use lchown() because BSD 2 doesn't
support it".

> > enough to fix libtool, which is the real problem here, so they go
> > for the thing they think they understand instead, without thinking
> > the implications through -- as-needed, like fast-math, is for
> > programs explicitly designed for it, not for universal use.
> 
> Differently -ffast-math is setting up a slightly different behavior
> than the usual standard, --as-needed enforce what is the default
> standard in determined architectures, thus the exception and the
> universality are quite reverted.

Both are standard-violating options that are useful for applications
designed to work with them explicitly.

> We already started to think how to fix libtool, or at least make it
> less annoying, removing .la files when they are not necessary.

Again, that's silly ricing. Saving a few kBytes is irrelevant. Instead,
you should be focusing your efforts upon something that will really
make a difference, like getting something based upon this into upstream:

http://patches.ubuntu.com/by-release/extracted/debian/libt/libtool/1.9+20051221-1/link_all_deplibs.dpatch

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Josh Saddler]

[-- Attachment #1: Type: text/plain, Size: 764 bytes --]

Ciaran McCreesh wrote:
> On Fri, 30 May 2008 15:07:43 -0700
> Donnie Berkholz <dberkholz@gentoo.org> wrote:
>> I see that a number of packages in the tree explicitly filter 
>> -ffast-math.
> 
> That's mostly from the bad old days when users were encouraged to use
> silly CFLAGS...

1. _When_ was this?
2. _Who_ was encouraging this?

. . . because it sure as hell hasn't ever been in the official 
documentation.

Last year I wrote the optimization guide[1] in part to stem the flood of 
bad advice on the unofficial gentoo-wiki, and by uninformed users on the 
forums.

There hasn't been any official documentation telling users to do foolish 
things with their CFLAGS or LDFLAGS.

[1] http://www.gentoo.org/doc/en/gcc-optimization.xml


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 1280 bytes --]

While we can continually loop around w/ the "--as-needed is evil since 
c++ does this one odd thing occasionally" argument, why not hear from 
the folks using it, specifically finding out what breaks in their 
usage?

Ciaran: yes, just because the tree works now w/ --as-needed doesn't 
mean that future pkg versions will work.  Dumb argument however 
(has shades of 'the sky is falling') since *every* new version is 
untested and has the potential to break against our accepted build 
environments (or to break pre-existing pkgs).  That's a known issue, 
and dealt with (30 days stablization among other things).

So... folks have pointed out a benefit to using --as-needed.  The 
benefit itself doesn't seem particularly in dispute, analyze the 
fallout from it- if the best that is offered is "the spec says 
otherwise", screw the spec frankly- a .01% breakage w/ 99.99% pkgs 
getting a positive gain is a strong argument for doing exemptions 
where needed.

Basically, pull out the stats of the breakage.  There is *always* risk 
in changes (new gcc, new bash breaking paludis/portage, etc), someone 
kindly come back w/ stats backing their specific viewpoint.

Arguing over the spec at this point isn't going anywhere, so just 
drop it.

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 519 bytes --]

On Fri, 30 May 2008 18:43:56 -0700
Brian Harring <ferringb@gmail.com> wrote:
> So... folks have pointed out a benefit to using --as-needed.

But they haven't. They've pointed out a flaw in libtool that is sort of
worked around sometimes at the expense of breaking things by using
as-needed. The correct solution is to fix libtool, but that's evidently
beyond the abilities of people who're only interested in increasing
their epenis size by throwing more silly options in config files.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 969 bytes --]

On Sat, May 31, 2008 at 02:50:20AM +0100, Ciaran McCreesh wrote:
> On Fri, 30 May 2008 18:43:56 -0700
> Brian Harring <ferringb@gmail.com> wrote:
> > So... folks have pointed out a benefit to using --as-needed.
> 
> But they haven't. They've pointed out a flaw in libtool that is sort of
> worked around sometimes at the expense of breaking things by using
> as-needed.

> The correct solution is to fix libtool., but that's evidently
> beyond the abilities of people who're only interested in increasing
> their epenis size by throwing more silly options in config files.

Then go do it.  You don't like the solution folks favor because you 
view another as more "correct and proper", either put up, or shut up.  

Meanwhile, obviously w/ the exemption of you and your "epenis" 
insults, most other folk don't seem to mind this approach as at least 
an interim solution- so again I ask, where are the stats stating the 
sky shall fall?

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Nirbheek Chauhan]

On Sat, May 31, 2008 at 7:20 AM, Ciaran McCreesh
<ciaran.mccreesh@googlemail.com> wrote:
> On Fri, 30 May 2008 18:43:56 -0700
> Brian Harring <ferringb@gmail.com> wrote:
>> So... folks have pointed out a benefit to using --as-needed.
>
> But they haven't. They've pointed out a flaw in libtool that is sort of
> worked around sometimes at the expense of breaking things by using
> as-needed. The correct solution is to fix libtool, but that's evidently
> beyond the abilities of people who're only interested in increasing
> their epenis size by throwing more silly options in config files.

This is incredible. I have never seen a more logic-lacking and
flamebait reply to a post full of reason and logic.

1) You say the benefits haven't been pointed out, while several posts
have already done so. You seem to be the only one pretending to be
unaware of them.
2) The "expense of breaking things" is completely unqualified in your
post. Here's some context: "expense" is minimal since the problem is
easily fixable, and "breaking things" is the list of bugs on the
tracker bug -- 19 with most of them already having patches that just
need to be committed
3) You say fixing libtool is the correct solution but you don't say
why or explain how. You don't give any information at all, and due to
the non-existant evidence, I am going to take the statement with a
fist of salt.
4) epenis size? Really.

You grabbed one line in his entire post and used it to divert the
conversation's direction from constructive to time-wasting-argument.

Personally, I agree wholeheartedly with what Brian has said. There are
advantages (they've have already been stated earlier), and they far
outweigh the disadvantages (breakages are easily fixable afaict).


~Nirbheek Chauhan who awaits the reply by ciaranm quoting one sentence
from his post and flaming/trolling him
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 605 bytes --]

On Fri, 30 May 2008 19:01:24 -0700
Brian Harring <ferringb@gmail.com> wrote:
> > The correct solution is to fix libtool., but that's evidently
> > beyond the abilities of people who're only interested in increasing
> > their epenis size by throwing more silly options in config files.
> 
> Then go do it.  You don't like the solution folks favor because you 
> view another as more "correct and proper", either put up, or shut
> up.  

Ah, brilliant. The old "yes, we know we're wrong but we're going to do
it anyway so there!" approach. Nice design principle, that...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1528 bytes --]

On Sat, 31 May 2008 07:38:12 +0530
"Nirbheek Chauhan" <nirbheek.chauhan@gmail.com> wrote:
> 1) You say the benefits haven't been pointed out, while several posts
> have already done so. You seem to be the only one pretending to be
> unaware of them.

No no no. The benefits described would be obtained by fixing libtool.
What you get from as-needed is a half-arsed sometimes-working subset of
those benefits. as-needed is not the fix for the libtool problems.

> 2) The "expense of breaking things" is completely unqualified in your
> post. Here's some context: "expense" is minimal since the problem is
> easily fixable, and "breaking things" is the list of bugs on the
> tracker bug -- 19 with most of them already having patches that just
> need to be committed

And all of which are utterly pointless.

> 3) You say fixing libtool is the correct solution but you don't say
> why or explain how. You don't give any information at all, and due to
> the non-existant evidence, I am going to take the statement with a
> fist of salt.

I'm assuming everyone contributing to this thread knows exactly what the
libtool problems are... But from the looks of things, plenty of people
are quite happy to jump in and yell when they don't have the slightest
clue what the root problem is, what as-needed changes, what as-needed
breaks or how as-needed is unrelated to the problem. And unfortunately,
it looks like those people are the ones that're going to be making the
decisions.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Nirbheek Chauhan]

On Sat, May 31, 2008 at 7:44 AM, Ciaran McCreesh
<ciaran.mccreesh@googlemail.com> wrote:
> On Sat, 31 May 2008 07:38:12 +0530
> "Nirbheek Chauhan" <nirbheek.chauhan@gmail.com> wrote:
>> 1) You say the benefits haven't been pointed out, while several posts
>> have already done so. You seem to be the only one pretending to be
>> unaware of them.
>
> No no no. The benefits described would be obtained by fixing libtool.
> What you get from as-needed is a half-arsed sometimes-working subset of
> those benefits. as-needed is not the fix for the libtool problems.

Once again, you do not support your argument with anything but your
own word. Don't make me choke on the salt please :)

>
>> 2) The "expense of breaking things" is completely unqualified in your
>> post. Here's some context: "expense" is minimal since the problem is
>> easily fixable, and "breaking things" is the list of bugs on the
>> tracker bug -- 19 with most of them already having patches that just
>> need to be committed
>
> And all of which are utterly pointless.

Ah, that's assuming your arguments are right, and your arguments
aren't supported, so I'll take some more salt with this one.

>
>> 3) You say fixing libtool is the correct solution but you don't say
>> why or explain how. You don't give any information at all, and due to
>> the non-existant evidence, I am going to take the statement with a
>> fist of salt.
>
> I'm assuming everyone contributing to this thread knows exactly what the
> libtool problems are... But from the looks of things, plenty of people
> are quite happy to jump in and yell when they don't have the slightest
> clue what the root problem is, what as-needed changes, what as-needed
> breaks or how as-needed is unrelated to the problem.

Once again, rhetoric and insults without logic or reason. We all know
you know that you need facts to convince people, but you're not
providing any facts. One can only conclude that your purpose is not to
convince. I honestly am baffled what purpose you have in mind.

> And unfortunately,
> it looks like those people are the ones that're going to be making the
> decisions.

Excellent, then you are free to point and laugh when we trip and fall.
In the meantime, if you truly think everyone is making the wrong
decision, talk with some facts and/or statistics.

-- 
~Nirbheek Chauhan
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1941 bytes --]

On Sat, 31 May 2008 07:53:05 +0530
"Nirbheek Chauhan" <nirbheek.chauhan@gmail.com> wrote:
> > No no no. The benefits described would be obtained by fixing
> > libtool. What you get from as-needed is a half-arsed
> > sometimes-working subset of those benefits. as-needed is not the
> > fix for the libtool problems.
> 
> Once again, you do not support your argument with anything but your
> own word. Don't make me choke on the salt please :)

Uhm. You're suggesting that the underlying issue is not a libtool
problem? Or you're suggesting that as-needed fixes the libtool bug?
Which basic fact that everyone discussing this should already know are
you disputing?

> Once again, rhetoric and insults without logic or reason. We all know
> you know that you need facts to convince people, but you're not
> providing any facts. One can only conclude that your purpose is not to
> convince. I honestly am baffled what purpose you have in mind.

I expect people to do their homework and understand what we're
discussing. Do you expect me to start every post by explain what a
linker is?

> > And unfortunately, it looks like those people are the ones that're
> > going to be making the decisions.
> 
> Excellent, then you are free to point and laugh when we trip and fall.
> In the meantime, if you truly think everyone is making the wrong
> decision, talk with some facts and/or statistics.

Fact: the underlying issue is a libtool bug.

Fact: as-needed does not fix this bug. It attempts to work around it.

Fact: as-needed breaks standard-compliant code.

Fact: fixing the libtool bug would give all the benefits purportedly
given by using as-needed, without the drawbacks.

It's quite simple, and if there're any of the above that you didn't
already know then why are you wasting everyone else's time discussing
things in this thread without doing some basic research first?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Ravi Pinjala]

Ciaran McCreesh wrote:
> On Sat, 31 May 2008 07:38:12 +0530
> "Nirbheek Chauhan" <nirbheek.chauhan@gmail.com> wrote:
>> 1) You say the benefits haven't been pointed out, while several posts
>> have already done so. You seem to be the only one pretending to be
>> unaware of them.
> 
> No no no. The benefits described would be obtained by fixing libtool.
> What you get from as-needed is a half-arsed sometimes-working subset of
> those benefits. as-needed is not the fix for the libtool problems.
> 
>> 2) The "expense of breaking things" is completely unqualified in your
>> post. Here's some context: "expense" is minimal since the problem is
>> easily fixable, and "breaking things" is the list of bugs on the
>> tracker bug -- 19 with most of them already having patches that just
>> need to be committed
> 
> And all of which are utterly pointless.
> 
>> 3) You say fixing libtool is the correct solution but you don't say
>> why or explain how. You don't give any information at all, and due to
>> the non-existant evidence, I am going to take the statement with a
>> fist of salt.
> 
> I'm assuming everyone contributing to this thread knows exactly what the
> libtool problems are... But from the looks of things, plenty of people
> are quite happy to jump in and yell when they don't have the slightest
> clue what the root problem is, what as-needed changes, what as-needed
> breaks or how as-needed is unrelated to the problem. And unfortunately,
> it looks like those people are the ones that're going to be making the
> decisions.
> 

Could you explain, for the benefit of us spectators, what these libtool 
problems are, and what cleaner solution you have in mind? It'd make this 
whole discussion a lot more comprehensible.

--Ravi
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Nirbheek Chauhan]

On Sat, May 31, 2008 at 8:03 AM, Ciaran McCreesh
<ciaran.mccreesh@googlemail.com> wrote:
>> Once again, you do not support your argument with anything but your
>> own word. Don't make me choke on the salt please :)
>
> Uhm. You're suggesting that the underlying issue is not a libtool
> problem? Or you're suggesting that as-needed fixes the libtool bug?
> Which basic fact that everyone discussing this should already know are
> you disputing?

I'm disputing your claim that fixing libtool is the correct solution
*right now*, and that it's sanely doable in a reasonable time-frame.
The former has been questioned by pretty much everyone else in the
thread, and the latter is probably false since you don't just do it
yourself.

>
>> Once again, rhetoric and insults without logic or reason. We all know
>> you know that you need facts to convince people, but you're not
>> providing any facts. One can only conclude that your purpose is not to
>> convince. I honestly am baffled what purpose you have in mind.
>
> I expect people to do their homework and understand what we're
> discussing. Do you expect me to start every post by explain what a
> linker is?

EDOESNOTMAKESENSE
You said

"[...]plenty of people are quite happy to jump in and yell when they
don't have the slightest clue what the root problem is[...]"

I replied saying that the paragraph was full of "rhetoric and insults
without logic or reason". The latter part of the post is purely about
how you're doing the same even where you should be talking with facts.

Your reply seriously does not make sense to me.

>
>> > And unfortunately, it looks like those people are the ones that're
>> > going to be making the decisions.
>>
>> Excellent, then you are free to point and laugh when we trip and fall.
>> In the meantime, if you truly think everyone is making the wrong
>> decision, talk with some facts and/or statistics.
>
> Fact: the underlying issue is a libtool bug.

Fact: It can't be fixed easily and/or in a reasonable time-frame. Else
someone would've done it -- heck you could've fixed it.

>
> Fact: as-needed does not fix this bug. It attempts to work around it.

Fact: It works. Unlike your vapour-proposal to "fix libtool".

>
> Fact: as-needed breaks standard-compliant code.

Fact: Breakages are rare, code which causes it is discouraged anyway,
and is fixable in any case. We're not a standards organisation.

>
> Fact: fixing the libtool bug would give all the benefits purportedly
> given by using as-needed, without the drawbacks.

Fact: It hasn't been done forever, and won't be done anytime soon.

>
> It's quite simple, and if there're any of the above that you didn't
> already know then why are you wasting everyone else's time discussing
> things in this thread without doing some basic research first?

It's quite simple, and you already knew all of the above, so why are
you wasting everyone's time and energy discussing these things in this
thread?

Quite refreshing, seeing you type out your points in a clear manner
for clearer rebuttal. I am going to assume that the matter is settled
now.

-- 
~Nirbheek Chauhan
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1344 bytes --]

On Sat, 31 May 2008 08:28:27 +0530
"Nirbheek Chauhan" <nirbheek.chauhan@gmail.com> wrote:
> > Fact: the underlying issue is a libtool bug.
> 
> Fact: It can't be fixed easily and/or in a reasonable time-frame. Else
> someone would've done it -- heck you could've fixed it.

Untrue. The amount of effort that's been wasted messing around with
as-needed could easily have been directed to fixing the root cause
instead. Debian have already done most of the work.

> > Fact: as-needed does not fix this bug. It attempts to work around
> > it.
> 
> Fact: It works. Unlike your vapour-proposal to "fix libtool".

But it doesn't work. And fixing libtool isn't vapour. Read the Debian
patch.

> > Fact: as-needed breaks standard-compliant code.
> 
> Fact: Breakages are rare, code which causes it is discouraged anyway,
> and is fixable in any case. We're not a standards organisation.

You seriously think Gentoo has the manpower to go around making
unnecessary changes to upstream code? And there's nothing in the C++
standard discouraging static initialisation.

> > Fact: fixing the libtool bug would give all the benefits purportedly
> > given by using as-needed, without the drawbacks.
> 
> Fact: It hasn't been done forever, and won't be done anytime soon.

And the Debian patch is...?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1241 bytes --]

On Fri, 30 May 2008 21:50:49 -0500
Ravi Pinjala <ravi@p-static.net> wrote:
> Could you explain, for the benefit of us spectators, what these
> libtool problems are, and what cleaner solution you have in mind?
> It'd make this whole discussion a lot more comprehensible.

libtool links against dependencies-of-dependencies, rather than just
direct dependencies. This is correct behaviour on some platforms under
some situations (one example is static linking). But when linking
shared objects on modern Unixy platforms it's a pain in the ass since
you end up with a load of bogus shared object dependencies that break
things unnecessarily upon upgrades.

as-needed works around this by making the linker only link things from
the command line that resolve a name dependency (but name dependencies
aren't the only dependency type). This means that most of the extras
libtool adds in end up getting ignored, but it also means that things
the programmer has explicitly said to link in get ignored too.

The correct fix is to make libtool only link to dependencies of
dependencies when doing, for example, static linking. Debian has a
half-working patch for this that I posted earlier in the thread.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Nirbheek Chauhan]

On Sat, May 31, 2008 at 8:33 AM, Ciaran McCreesh
<ciaran.mccreesh@googlemail.com> wrote:
> On Sat, 31 May 2008 08:28:27 +0530
> "Nirbheek Chauhan" <nirbheek.chauhan@gmail.com> wrote:
>> Fact: It can't be fixed easily and/or in a reasonable time-frame. Else
>> someone would've done it -- heck you could've fixed it.
>
> Untrue. The amount of effort that's been wasted messing around with
> as-needed could easily have been directed to fixing the root cause
> instead. Debian have already done most of the work.

And the time you just wasted spewing rhetoric on this thread? Since
you have such a deep understanding of everything, couldn't you have
done the rest of the work, posted it here and instantly convinced
everyone?

>> Fact: It works. Unlike your vapour-proposal to "fix libtool".
>
> But it doesn't work. And fixing libtool isn't vapour. Read the Debian
> patch.

Fixing libtool isn't vapour, neither is the debian patch, but your
plan/proposal to do is non-existant. You're just saying "Do this
because I say you should, I don't know how, but you should. Oh, and if
you don't you're all idiots".

>> Fact: Breakages are rare, code which causes it is discouraged anyway,
>> and is fixable in any case. We're not a standards organisation.
>
> You seriously think Gentoo has the manpower to go around making
> unnecessary changes to upstream code? And there's nothing in the C++
> standard discouraging static initialisation.

The tracker bug (bug 129413) seems to say otherwise.

>> Fact: It hasn't been done forever, and won't be done anytime soon.
>
> And the Debian patch is...?

Useless unless it's complete. As I said above, fix it and convince us.
Show us how wrong we are. We'll be glad and grateful. Don't whine
about it and waste everyone's time and energy.


-- 
~Nirbheek Chauhan
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 1963 bytes --]

On Sat, May 31, 2008 at 08:45:09AM +0530, Nirbheek Chauhan wrote:
> On Sat, May 31, 2008 at 8:33 AM, Ciaran McCreesh
> <ciaran.mccreesh@googlemail.com> wrote:
> > On Sat, 31 May 2008 08:28:27 +0530
> > "Nirbheek Chauhan" <nirbheek.chauhan@gmail.com> wrote:
> >> Fact: It works. Unlike your vapour-proposal to "fix libtool".
> >
> > But it doesn't work. And fixing libtool isn't vapour. Read the Debian
> > patch.
> 
> Fixing libtool isn't vapour, neither is the debian patch, but your
> plan/proposal to do is non-existant. You're just saying "Do this
> because I say you should, I don't know how, but you should. Oh, and if
> you don't you're all idiots".

It's worth noting the debian patch was also rejected by upstream-
http://lists.gnu.org/archive/html/libtool/2004-05/msg00118.html

Full thread in question-
http://lists.gnu.org/archive/html/libtool/2004-05/msg00097.html

Basically, libtool upstream points at several rather hard issues to 
fixing it fully.  Worth noting that was also in '04- 4 years later 
(with said debian patch in use), issue still is unfixed upstream.

> >> Fact: Breakages are rare, code which causes it is discouraged anyway,
> >> and is fixable in any case. We're not a standards organisation.
> >
> > You seriously think Gentoo has the manpower to go around making
> > unnecessary changes to upstream code? And there's nothing in the C++
> > standard discouraging static initialisation.

Considering static initialization doesn't always play nice on other 
platforms (porting rekall 2.2 on osx comes to mind), I'd expect the 
sources in question to be addressing the issue themselves frankly 
(which is what occured with rekall).

Either way, basically it's coming down to if gentoo wants to follow 
the definition of 'academic' right, or 'pragmatic' right.  Exempting 
ciaran, vote seems to be pragmatic.

Cc'ing council, would like them to discuss this for upcoming meeting.
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Michal Kurgan]

On Sat, 31 May 2008 02:17:48 +0100
Ciaran McCreesh <ciaran.mccreesh@googlemail.com> wrote:

> On Sat, 31 May 2008 03:03:42 +0200
> Luca Barbato <lu_zero@gentoo.org> wrote:
> > Ciaran McCreesh wrote:
> > > Which is where the design flaw is -- as-needed incorrectly assumes
> > > that the only type of dependency between shared objects is a name
> > > dependency. This isn't true with C++ static initialisers.
> > 
> > I don't see why should be different than abusing .init in any other 
> > language that let you do (ok, C, C++, asm mostly).
> 
> In C++ it's not abuse. It's using the language as specified and
> designed.
> 

Would that be possible for you (or anyone else) to point or provide
example of code that breaks after use of as-needed?

-- 
Michal Kurgan
http://dev.gentoo.org/~moloh


-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Roy Marples]

On Saturday 31 May 2008 00:16:31 Ciaran McCreesh wrote:
> > Ok, then everything in the tree is covered and we can move to having
> > --as-needed as default.
>
> Is the next version of everything in the tree covered? Have you made
> sure that software isn't merely working by fluke? 

We interupt this thread with breaking news!
A troll claims that future software packages may break!
This reporter sure hopes that the developers in question actually tests 
packages before they commit.
I, for one, welcome our new --as-needed Overlords.

> Is Gentoo really that desperate to turn everyone into a ricer?

Everyones ride should be pimped.

Thanks

Roy
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Peter Volkov]

В Птн, 30/05/2008 в 20:28 -0700, Brian Harring пишет:
> Either way, basically it's coming down to if gentoo wants to follow 
> the definition of 'academic' right, or 'pragmatic' right. Exempting 
> ciaran, vote seems to be pragmatic.

Well, although I've asked about problems with having --as-needed by
default, I'd better go with academic. C++ is quite common language to
ignore its design problems and in the end it's not hard to define
LDFLAGS in make.conf.

-- 
Peter.

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Luca Barbato]

Ciaran McCreesh wrote:
> Fact: the underlying issue is a libtool bug.

Wrong, it isn't just that, --as-needed and libtool are unrelated.

> Fact: as-needed does not fix this bug. It attempts to work around it.

Wrong, --as-needed does exactly what is supposed to do, precise bookkeeping.

> Fact: as-needed breaks standard-compliant code.

Wrong, --as-needed breaks disputable code that happens to be 
standard-compliant by a specific read of the standard. The fact the 
specific code is something wrong from the security/style/maintainability 
point makes it a bonus.

> Fact: fixing the libtool bug would give all the benefits purportedly
> given by using as-needed, without the drawbacks.

Wrong, fixing libtool gives other benefits, so it's worth trying to fix 
it as well. The new autotools and proper usage of them makes life easier 
so it's worth improving on this side.

> It's quite simple,

Probably but is an empty sentence w/out supporting code.

> and if there're any of the above that you didn't
> already know then why are you wasting everyone else's time discussing
> things in this thread without doing some basic research first?

Basically most people is discussing with you since thinks, wrongly, that 
could be possible take something good from this discussion. The patch 
you pointed doesn't look complete nor acceptable to upstream as is, yet 
could help.

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[David Leverton]

On Saturday 31 May 2008 11:14:33 Luca Barbato wrote:
> Ciaran McCreesh wrote:
> > Fact: the underlying issue is a libtool bug.
>
> Wrong, it isn't just that, --as-needed and libtool are unrelated.

The issue that as-needed tries to solve is libraries being linked to binaries 
or other libraries that don't use said library directly.  While it's true 
that libtool isn't the only cause, it does produce by far the most.

> > Fact: as-needed does not fix this bug. It attempts to work around it.
>
> Wrong, --as-needed does exactly what is supposed to do, precise
> bookkeeping.

It does do what it's supposed to do, unfortunately "what it's supposed to do" 
isn't the right thing in all cases.  And it's not "precise", it simply uses a 
different criterion that's better in some cases and worse in others.

> > Fact: as-needed breaks standard-compliant code.
>
> Wrong, --as-needed breaks disputable code that happens to be
> standard-compliant by a specific read of the standard. The fact the
> specific code is something wrong from the security/style/maintainability
> point makes it a bonus.

No-one's given any reason why it's "disputable", worse "style" or 
less "maintainable", other than "it doesn't work with --as-needed", quite a 
circular argument.  As for "security"... please show evidence, or I'll have 
to assume that that's just desperate FUD.

> > Fact: fixing the libtool bug would give all the benefits purportedly
> > given by using as-needed, without the drawbacks.
>
> Wrong, fixing libtool gives other benefits, so it's worth trying to fix
> it as well. The new autotools and proper usage of them makes life easier
> so it's worth improving on this side.

I really don't see what you're trying to say there....
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Luca Barbato]

Peter Volkov wrote:
> В Птн, 30/05/2008 в 20:28 -0700, Brian Harring пишет:
>> Either way, basically it's coming down to if gentoo wants to follow 
>> the definition of 'academic' right, or 'pragmatic' right. Exempting 
>> ciaran, vote seems to be pragmatic.
> 
> Well, although I've asked about problems with having --as-needed by
> default, I'd better go with academic. C++ is quite common language to
> ignore its design problems and in the end it's not hard to define
> LDFLAGS in make.conf.
> 

To clarify:

- static initializers (as in __attribute__((constructor), so no, it 
isn't a C++ only feature) have nothing wrong with --as-needed.

- ugly code that refers to undefined symbols that are resolved to ones 
from the main binary and written in the constructor is broken already in 
  systems not allowing undefined refs.

- you don't have guarantees about the order in witch the .init sections 
are parsed and constructor function are called, they can be called in 
parallel and you have no means to have a predictable behavior, all you 
know is that everything will be called right before main() or as the 
first thing in dlopen().

- doing such stuff is uncommon since it isn't the simplest thing to do, 
doesn't work in every place, you have to be particular perverse and 
convoluted even to think about this.

- making such thing go away is good for security, maintainability and 
sanity.

lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Alec Warner]

On 5/31/08, Luca Barbato <lu_zero@gentoo.org> wrote:
> Peter Volkov wrote:
>
> > В Птн, 30/05/2008 в 20:28 -0700, Brian Harring пишет:
> >
> > > Either way, basically it's coming down to if gentoo wants to follow the
> definition of 'academic' right, or 'pragmatic' right. Exempting ciaran, vote
> seems to be pragmatic.
> > >
> >
> > Well, although I've asked about problems with having --as-needed by
> > default, I'd better go with academic. C++ is quite common language to
> > ignore its design problems and in the end it's not hard to define
> > LDFLAGS in make.conf.
> >
> >
>

Just to jump in quickly; this thread is about adding --as-needed to
the default CFLAGS.  To get this accomplished you need to:

A.  Convince the portage developers to put it in make.conf/make.defaults.
B.  Convince the profile maintainers to put it in their profile.
C.  Do A or B yourself and hope no one gets pissed.
D.  Convince the council at the next meeting that either A or B is a good idea.

Personally I think D is the best choice here.  Your point should not
to be to convince random people such at Ciaran.  I'm pretty sure his
mind is made up and I kind of agree on his position.  The point being
regardless of what the 'tehcnically correct' decision with regards to
the standard is, the Gentoo community needs to make its own choice as
to whether the risks of breaking code is worth the gains of making
--as-needed the default.

Presently community decisions regarding technical issues are made via
the council; so maybe you should start convincing them.

-Alec

>  To clarify:
>
>  - static initializers (as in __attribute__((constructor), so no, it isn't a
> C++ only feature) have nothing wrong with --as-needed.
>
>  - ugly code that refers to undefined symbols that are resolved to ones from
> the main binary and written in the constructor is broken already in  systems
> not allowing undefined refs.
>
>  - you don't have guarantees about the order in witch the .init sections are
> parsed and constructor function are called, they can be called in parallel
> and you have no means to have a predictable behavior, all you know is that
> everything will be called right before main() or as the first thing in
> dlopen().
>
>  - doing such stuff is uncommon since it isn't the simplest thing to do,
> doesn't work in every place, you have to be particular perverse and
> convoluted even to think about this.
>
>  - making such thing go away is good for security, maintainability and
> sanity.
>
>  lu
>
>  --
>
>  Luca Barbato
>  Gentoo Council Member
>  Gentoo/linux Gentoo/PPC
>  http://dev.gentoo.org/~lu_zero
>
>  --
>
> gentoo-dev@lists.gentoo.org mailing list
>
>

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Alexis Ballier]

[-- Attachment #1: Type: text/plain, Size: 435 bytes --]


> A.  Convince the portage developers to put it in
> make.conf/make.defaults.


By the way, I'm strongly opposed to this: it should be, at best, in the
profiles.
For instance, as long as bug #192403 isn't fixed, as-needed will cause
*a lot* of build failures on fbsd since gcc specs are broken and wont
append the -lc for shared libraries and -lpthread when -pthread is
used if I remember correctly.

Regards,

Alexis.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Marius Mauch]

On Sat, 31 May 2008 04:26:39 -0700
"Alec Warner" <antarus@gentoo.org> wrote:

> Just to jump in quickly; this thread is about adding --as-needed to
> the default CFLAGS.  To get this accomplished you need to:
> 
> A.  Convince the portage developers to put it in
> make.conf/make.defaults.

Wrong.

We don't control make.conf (the user does) or make.defaults (profile
maintainers do). 
We only control make.conf.example (which these days
only serves as documentation) and make.globals (which isn't the right
place for any arch-specific stuff, and CFLAGS are arch-specific by
definition).

Marius
-- 
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] Re: RFC: --as-needed to default LDFLAGS (Was: RFC: Should preserve-libs be enabled by default?)

[Duncan]

Ciaran McCreesh <ciaran.mccreesh@googlemail.com> posted
20080531040338.26a64f31@snowcone, excerpted below, on  Sat, 31 May 2008
04:03:38 +0100:

> The correct fix is to make libtool only link to dependencies of
> dependencies when doing, for example, static linking. Debian has a
> half-working patch for this that I posted earlier in the thread.

Thanks.  That explanation (mostly snipped for brevity) was a great "plain 
English" explanation for those of us trying to follow along but not 
making any claim to be great programmers or at understanding the depths 
of libtool.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] Re: RFC: --as-needed to default LDFLAGS

[Ulrich Mueller]

[Answering to some random message in this long thread.]

>>>>> On Sat, 31 May 2008, Brian Harring wrote:

> So... folks have pointed out a benefit to using --as-needed.
> The benefit itself doesn't seem particularly in dispute, analyze
> the fallout from it- if the best that is offered is "the spec says
> otherwise", screw the spec frankly- a .01% breakage w/ 99.99% pkgs
> getting a positive gain is a strong argument for doing exemptions
> where needed.

Speaking about statistics: Either I have missed it, or so far nobody
has presented any solid numbers showing what the benefit of
--as-needed in terms of memory usage or program startup time is.

Could someone please show this comparison for some common programs?

I've just done this for Emacs (22.2-r2), virtual set size directly
after startup is 25280 and 25276 kB, for Emacs built without and with
--as-needed, respectively (resident set size is 14412 and 14396 kB).
I don't see any difference in startup time.

But maybe Emacs is an uncommon application, or I am looking for the
wrong things? Could one of the experts please shed some light on this?

Ulrich
-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: RFC: --as-needed to default LDFLAGS

[Mike Auty]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ulrich Mueller wrote:
| But maybe Emacs is an uncommon application, or I am looking for the
| wrong things? Could one of the experts please shed some light on this?

I think you're looking for the wrong things.  I'm not an expert, but I
think --as-needed means that if there are 20 libraries on your system
that use libexpat.so.0 and 400 programs that use those 20 libraries,
when libexpat is updated to libexpat.so.1, you only need to rebuild the
20 libraries, not all 420 packages (as you would do otherwise).  I
believe that's the main reason for using as-needed...

Mike  5:)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)

iEYEARECAAYFAkhBp9oACgkQu7rWomwgFXr8JQCfYFDwcebduPVaY3yqUIEfVOxp
G80AoKV6SsAewxyyfv+fsiwbc6M1BHsc
=12e5
-----END PGP SIGNATURE-----
-- 
gentoo-dev@lists.gentoo.org mailing list

[gentoo-dev] Re: RFC: --as-needed to default LDFLAGS

[Duncan]

Mike Auty <ikelos@gentoo.org> posted 4841A7DA.7060605@gentoo.org,
excerpted below, on  Sat, 31 May 2008 20:32:42 +0100:

> I think you're looking for the wrong things.  I'm not an expert, but I
> think --as-needed means that if there are 20 libraries on your system
> that use libexpat.so.0 and 400 programs that use those 20 libraries,
> when libexpat is updated to libexpat.so.1, you only need to rebuild the
> 20 libraries, not all 420 packages (as you would do otherwise).  I
> believe that's the main reason for using as-needed...

That has certainly been my experience.  I've had way less rebuilds to 
worry about since I added that to my LDFLAGS and rebuilt the system.  
revdep-rebuild -p, which I run regularly after major world upgrades, 
returns far fewer packages to rebuild, now.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-dev@lists.gentoo.org mailing list

Re: [gentoo-dev] Re: RFC: --as-needed to default LDFLAGS

[Rémi Cardona]

Ulrich Mueller a écrit :
> Speaking about statistics: Either I have missed it, or so far nobody
> has presented any solid numbers showing what the benefit of
> --as-needed in terms of memory usage or program startup time is.

The reduction in startup time may not be noticeable.

The real win is when low level libs change ABI, like expat. On a 
standard Gnome system, without --as-needed, I had over 280 packages to 
rebuild (that was a very slow Duron 700Mhz, I ended up moving to 
--as-needed and did emerge -e world).

On my other box which had had --as-needed for a while (so some useless 
rebuild could have been further avoided), I only had around 45 packages.

And for the sake of the thread, had libtool been smarter, I'm sure that 
the final number could have gone down to 20 or so packages.

My opinion:
  - we need --as-needed because it's useful (maybe ld could echo the 
libs that's it's dropping and then we could have a QA warning?)
  - we *do* need to fix libtool too
  - we need to make sure upstream packages provide correct .pc files

Just a thought :)

Rémi
-- 
gentoo-dev@lists.gentoo.org mailing list