[gentoo-dev] Developers, please work on underlinking issues!

[gentoo-dev] Developers, please work on underlinking issues!

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1518 bytes --]

Hello, fellow developers.

TL;DR: switch to the darn gold linker, and start fixing your darn
packages!

After many years, the underlinking tracker [1] is still full of bugs.
This is a huge QA problem, and most of the developers don't test for
it.

Long story short, the thing usually looks like this: a developer
switches to the gold linker, starts hitting build failures
and reporting bugs. Most of the time, nobody gives a damn about those
bugs and the developer eventually gives up, wants to be able to upgrade
his system again and switches back to bfd.

So I'm asking shortly: please start caring. Switch to gold:

$ sudo binutils-config --linker ld.gold

Test your packages. Report bugs upstream. Get them fixed.

If someone already tested it and reported the bug, proxy it upstream.
Don't expect everyone else to do the dirty work on your packages.

This is not some problem with a fancy non-standard configuration.
ld.gold is just the messenger, and the message is: you are not linking
to the libraries you are using. This is serious, and your package only
works by accident. Of course, you don't have to pass my exact words
upstream.

That said, I'd really like for Gentoo to be able to finally switch to
ld.gold by default. However, that can't happen if every common system is
going to have 1-3 packages failing to build.

Thanks in advance.

[1]:https://bugs.gentoo.org/show_bug.cgi?id=underlinking

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Developers, please work on underlinking issues!

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 218 bytes --]

>>>>> On Wed, 17 Aug 2016, Michał Górny wrote:

> $ sudo binutils-config --linker ld.gold

"sudo", really? Are we Ubuntu now? :)

$ sudo binutils-config --linker ld.gold
bash: sudo: command not found

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Developers, please work on underlinking issues!

[Daniel Campbell]

[-- Attachment #1.1: Type: text/plain, Size: 2089 bytes --]

On 08/17/2016 01:37 PM, Michał Górny wrote:
> Hello, fellow developers.
> 
> TL;DR: switch to the darn gold linker, and start fixing your darn
> packages!
> 
> After many years, the underlinking tracker [1] is still full of bugs.
> This is a huge QA problem, and most of the developers don't test for
> it.
> 
> Long story short, the thing usually looks like this: a developer
> switches to the gold linker, starts hitting build failures
> and reporting bugs. Most of the time, nobody gives a damn about those
> bugs and the developer eventually gives up, wants to be able to upgrade
> his system again and switches back to bfd.
> 
> So I'm asking shortly: please start caring. Switch to gold:
> 
> $ sudo binutils-config --linker ld.gold
> 
> Test your packages. Report bugs upstream. Get them fixed.
> 
> If someone already tested it and reported the bug, proxy it upstream.
> Don't expect everyone else to do the dirty work on your packages.
> 
> This is not some problem with a fancy non-standard configuration.
> ld.gold is just the messenger, and the message is: you are not linking
> to the libraries you are using. This is serious, and your package only
> works by accident. Of course, you don't have to pass my exact words
> upstream.
> 
> That said, I'd really like for Gentoo to be able to finally switch to
> ld.gold by default. However, that can't happen if every common system is
> going to have 1-3 packages failing to build.
> 
> Thanks in advance.
> 
> [1]:https://bugs.gentoo.org/show_bug.cgi?id=underlinking
> 

Is it as simple as switching the linker and re-merging packages that one
maintains? Is gold supposed to be a big deal? Does it do the job of
linking better? I read the blog post and all but nobody's explaining
what gold does better than standard ld.

That said, if it's that simple and then just requires some DEPEND
updating, doesn't sound all that bad to me.
-- 
Daniel Campbell - Gentoo Developer
OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net
fpr: AE03 9064 AE00 053C 270C  1DE4 6F7A 9091 1EA0 55D6


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Developers, please work on underlinking issues!

[Alexis Ballier]

On Wed, 17 Aug 2016 22:39:37 -0700
Daniel Campbell <zlg@gentoo.org> wrote:

> Is it as simple as switching the linker and re-merging packages that
> one maintains? Is gold supposed to be a big deal? Does it do the job
> of linking better? I read the blog post and all but nobody's
> explaining what gold does better than standard ld.

I think gold is faster and doesnt eat 8+gb of ram when linking chromium
with debug symbols :)

> That said, if it's that simple and then just requires some DEPEND
> updating, doesn't sound all that bad to me.

Nah, it's adding proper -lfoo when foo is used; more like asneeded
fixes.

Re: [gentoo-dev] Developers, please work on underlinking issues!

[Rich Freeman]

On Thu, Aug 18, 2016 at 1:39 AM, Daniel Campbell <zlg@gentoo.org> wrote:
>
> Is it as simple as switching the linker and re-merging packages that one
> maintains? Is gold supposed to be a big deal? Does it do the job of
> linking better? I read the blog post and all but nobody's explaining
> what gold does better than standard ld.
>

There are a bunch of reasons why people prefer it for the long term.
However, one of the biggest drives to get devs to use it is that it
serves as a good canary for underlinking.

If you underlink with the bfd linker the package will build and run
fine typically, at least initially.  However, down the road as
libraries are updated you can run into problems.  Since you didn't
specify all your dependencies portage won't do rebuilds when they're
necessary.  My understanding is that if the needed libraries aren't
even linked then even preserve-rebuild will fail.  The package is
basically relying on unspecified behavior to work.  If you need a
symbol you're supposed to link to the providing library, not just
assume that it will happen to be around because some other library
you're linked to happens to pull it in.

The gold linker generates errors if you attempt to underlink, which
turns this into a build-time error, and not a
maybe-you-see-it-maybe-you-don't runtime issue for random users in six
months.

Correctly specifying dependencies and ensuring they're linked is of
benefit to all users, and it will prevent subtle problems for users of
the default bfd linker.

At least, that is my understanding of the issue.  It is entirely
possible I missed something; I don't profess to be an ELF expert.  :)

-- 
Rich

Re: [gentoo-dev] Developers, please work on underlinking issues!

[Andrew Savchenko]

[-- Attachment #1: Type: text/plain, Size: 490 bytes --]

Hi,

On Wed, 17 Aug 2016 22:37:42 +0200 Michał Górny wrote:
[...]
> That said, I'd really like for Gentoo to be able to finally switch to
> ld.gold by default.

It still has problems with kernel or kernel modules here and there.

Also from my experience gold demands more memory on linking large
binaries, thus it it unsuitable for low-memory setups.

While gold indeed has its benefits, it is still too immature to
become system default.

Best regards,
Andrew Savchenko

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Developers, please work on underlinking issues!

[Lars Wendler]

[-- Attachment #1: Type: text/plain, Size: 979 bytes --]

On Thu, 18 Aug 2016 13:43:42 +0300 Andrew Savchenko wrote:

>Hi,
>
>On Wed, 17 Aug 2016 22:37:42 +0200 Michał Górny wrote:
>[...]
>> That said, I'd really like for Gentoo to be able to finally switch to
>> ld.gold by default.  
>
>It still has problems with kernel or kernel modules here and there.
>
>Also from my experience gold demands more memory on linking large
>binaries, thus it it unsuitable for low-memory setups.
>
>While gold indeed has its benefits, it is still too immature to
>become system default.

And as long as I keep reading such statements I won't use ld.gold
anywhere on my (dev-)systems. A linker IMHO is a far too crucial
toolchain component to blindly play around with.

>Best regards,
>Andrew Savchenko

Cheers
Lars

-- 
Lars Wendler
Gentoo package maintainer
GPG: 21CC CF02 4586 0A07 ED93  9F68 498F E765 960E 9B39

Attention! New gpg key! See
https://www.gentoofan.org/blog/index.php?/archives/9-New-gpg-keys.html

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Developers, please work on underlinking issues!

[Rich Freeman]

On Thu, Aug 18, 2016 at 7:47 AM, Lars Wendler <polynomial-c@gentoo.org> wrote:
>
> And as long as I keep reading such statements I won't use ld.gold
> anywhere on my (dev-)systems. A linker IMHO is a far too crucial
> toolchain component to blindly play around with.
>

There really isn't any need to use it as a daily driver.

If you just check your packages occassionally to make sure they build
with gold it completely achieves the goal, and it will actually result
in fewer bugs using the non-gold linker as well.

-- 
Rich

Re: [gentoo-dev] Developers, please work on underlinking issues!

[Alexis Ballier]

On Thu, 18 Aug 2016 08:13:14 -0400
Rich Freeman <rich0@gentoo.org> wrote:

> If you just check your packages occassionally to make sure they build
> with gold it completely achieves the goal, and it will actually result
> in fewer bugs using the non-gold linker as well.


That's what a tinderbox is for. The only QA problem I see here is that
QA doesn't automate that kind of checks anymore since Diego left. Maybe
QA should ask Toralf to run a ld.gold tinderbox and avoid asking people
to randomly test random packages ?

Re: [gentoo-dev] Developers, please work on underlinking issues!

[Daniel Campbell]

[-- Attachment #1.1: Type: text/plain, Size: 1097 bytes --]

On 08/18/2016 06:21 AM, Alexis Ballier wrote:
> On Thu, 18 Aug 2016 08:13:14 -0400
> Rich Freeman <rich0@gentoo.org> wrote:
> 
>> If you just check your packages occassionally to make sure they build
>> with gold it completely achieves the goal, and it will actually result
>> in fewer bugs using the non-gold linker as well.
> 
> 
> That's what a tinderbox is for. The only QA problem I see here is that
> QA doesn't automate that kind of checks anymore since Diego left. Maybe
> QA should ask Toralf to run a ld.gold tinderbox and avoid asking people
> to randomly test random packages ?
> 
I dunno, if testing packages that one maintains is as simple as
reconfiguring a package, testing, and switching back then I don't think
it's unreasonable to ask us to test our own packages. We're supposed to
do that already, and for packages whose dependencies aren't 100% hashed
out, it can help us figure out what the real deps are.

-- 
Daniel Campbell - Gentoo Developer
OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net
fpr: AE03 9064 AE00 053C 270C  1DE4 6F7A 9091 1EA0 55D6


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] Developers, please work on underlinking issues!

[Alexis Ballier]

On Thu, 18 Aug 2016 14:20:41 -0700
Daniel Campbell <zlg@gentoo.org> wrote:

> On 08/18/2016 06:21 AM, Alexis Ballier wrote:
> > On Thu, 18 Aug 2016 08:13:14 -0400
> > Rich Freeman <rich0@gentoo.org> wrote:
> >   
> >> If you just check your packages occassionally to make sure they
> >> build with gold it completely achieves the goal, and it will
> >> actually result in fewer bugs using the non-gold linker as well.  
> > 
> > 
> > That's what a tinderbox is for. The only QA problem I see here is
> > that QA doesn't automate that kind of checks anymore since Diego
> > left. Maybe QA should ask Toralf to run a ld.gold tinderbox and
> > avoid asking people to randomly test random packages ?
> >   
> I dunno, if testing packages that one maintains is as simple as
> reconfiguring a package, testing, and switching back then I don't
> think it's unreasonable to ask us to test our own packages. We're
> supposed to do that already, and for packages whose dependencies
> aren't 100% hashed out, it can help us figure out what the real deps
> are.


test against... all linkers, all compilers, all libcs, all kernels, all
userlannds, all useflags, ... ? :)


by all means, please do it, but there are things machines are better
at, like ensuring all packages have been tested against gold linker and
every failure has been reported

Re: [gentoo-dev] Developers, please work on underlinking issues!

[Mart Raudsepp]

Ühel kenal päeval, N, 18.08.2016 kell 23:56, kirjutas Alexis Ballier:
> On Thu, 18 Aug 2016 14:20:41 -0700
> Daniel Campbell <zlg@gentoo.org> wrote:
> 
> > On 08/18/2016 06:21 AM, Alexis Ballier wrote:
> > > On Thu, 18 Aug 2016 08:13:14 -0400
> > > Rich Freeman <rich0@gentoo.org> wrote:
> > >   
> > > > If you just check your packages occassionally to make sure they
> > > > build with gold it completely achieves the goal, and it will
> > > > actually result in fewer bugs using the non-gold linker as
> > > > well.  
> > > 
> > > 
> > > That's what a tinderbox is for. The only QA problem I see here is
> > > that QA doesn't automate that kind of checks anymore since Diego
> > > left. Maybe QA should ask Toralf to run a ld.gold tinderbox and
> > > avoid asking people to randomly test random packages ?
> > >   
> > I dunno, if testing packages that one maintains is as simple as
> > reconfiguring a package, testing, and switching back then I don't
> > think it's unreasonable to ask us to test our own packages. We're
> > supposed to do that already, and for packages whose dependencies
> > aren't 100% hashed out, it can help us figure out what the real
> > deps
> > are.
> 
> 
> test against... all linkers, all compilers, all libcs, all kernels,
> all
> userlannds, all useflags, ... ? :)
> 
> 
> by all means, please do it, but there are things machines are better
> at, like ensuring all packages have been tested against gold linker
> and
> every failure has been reported

The tl;dr did say to switch to ld.gold, but the main point was to
actually fix the bugs reported against your packages about it by other
developers, users and any future tinderbox runs, instead of ignoring
them as something that isn't supposed to matter and very low priority.
I think that should be sufficient, and we don't need to all rush to
switching to it, as long as we take care of the bugs about it when
others notice and file a bug.
Though it gives some good benefits when you are able to use it, afaik,
so hey, why not use it when you can :)


Mart

Re: [gentoo-dev] Developers, please work on underlinking issues!

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 901 bytes --]

On Thu, 18 Aug 2016 15:21:16 +0200
Alexis Ballier <aballier@gentoo.org> wrote:

> On Thu, 18 Aug 2016 08:13:14 -0400
> Rich Freeman <rich0@gentoo.org> wrote:
> 
> > If you just check your packages occassionally to make sure they build
> > with gold it completely achieves the goal, and it will actually result
> > in fewer bugs using the non-gold linker as well.  
> 
> That's what a tinderbox is for. The only QA problem I see here is that
> QA doesn't automate that kind of checks anymore since Diego left. Maybe
> QA should ask Toralf to run a ld.gold tinderbox and avoid asking people
> to randomly test random packages ?

Yes, tinderboxing makes a lot of sense if the bugs are afterwards
ignored by package maintainers. Or in the best case, the maintainer
tells reporter (Toralf) to file the bug upstream.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Developers, please work on underlinking issues!

[Rich Freeman]

On Fri, Aug 19, 2016 at 12:58 AM, Michał Górny <mgorny@gentoo.org> wrote:
> On Thu, 18 Aug 2016 15:21:16 +0200
> Alexis Ballier <aballier@gentoo.org> wrote:
>
>> On Thu, 18 Aug 2016 08:13:14 -0400
>> Rich Freeman <rich0@gentoo.org> wrote:
>>
>> > If you just check your packages occassionally to make sure they build
>> > with gold it completely achieves the goal, and it will actually result
>> > in fewer bugs using the non-gold linker as well.
>>
>> That's what a tinderbox is for. The only QA problem I see here is that
>> QA doesn't automate that kind of checks anymore since Diego left. Maybe
>> QA should ask Toralf to run a ld.gold tinderbox and avoid asking people
>> to randomly test random packages ?
>
> Yes, tinderboxing makes a lot of sense if the bugs are afterwards
> ignored by package maintainers. Or in the best case, the maintainer
> tells reporter (Toralf) to file the bug upstream.
>

TBH, these are really two different problems.

1.  I think raising awareness of underlinking is good.

2.  I think encouraging developers to test their own packages with the
gold linker is good, because it helps accomplish #1, and increases
their awareness in general.

3.  I think that having a tinderbox systematically testing using the
gold linker is also good.

4.  I think that hitting devs with a cluebat when they ignore valid
bugs is good.

The flip side of this is that we're not necessarily better off if
maintainers just abandon packages because they have terrible build
systems.  At some point you need to work with them.  However, if
they're not willing to at least stick in a slot operator dependency
when asked to, then sure we should have a talk with them.  (A slot op
dep will of course help by triggering rebuilds, but it doesn't
actually directly fix the underlinking issue, which would require
fixing the build system.)

I think the big thing is acknowledging that packages that are missing
dependencies or which are underlinked are defective.  Sure, it would
be nice if somebody else came along and helped find our mistakes.
However, that in itself doesn't excuse us from having made them in the
first place.  And it certainly doesn't excuse giving people a hard
time when they politely point them out.

-- 
Rich

Re: [gentoo-dev] Developers, please work on underlinking issues!

[james]

On 08/19/2016 03:49 AM, Rich Freeman wrote:
> On Fri, Aug 19, 2016 at 12:58 AM, Michał Górny <mgorny@gentoo.org> wrote:
>> On Thu, 18 Aug 2016 15:21:16 +0200
>> Alexis Ballier <aballier@gentoo.org> wrote:
>>
>>> On Thu, 18 Aug 2016 08:13:14 -0400
>>> Rich Freeman <rich0@gentoo.org> wrote:
>>>
>>>> If you just check your packages occassionally to make sure they build
>>>> with gold it completely achieves the goal, and it will actually result
>>>> in fewer bugs using the non-gold linker as well.
>>>
>>> That's what a tinderbox is for. The only QA problem I see here is that
>>> QA doesn't automate that kind of checks anymore since Diego left. Maybe
>>> QA should ask Toralf to run a ld.gold tinderbox and avoid asking people
>>> to randomly test random packages ?
>>
>> Yes, tinderboxing makes a lot of sense if the bugs are afterwards
>> ignored by package maintainers. Or in the best case, the maintainer
>> tells reporter (Toralf) to file the bug upstream.
>>
>
> TBH, these are really two different problems.
>
> 1.  I think raising awareness of underlinking is good.
>
> 2.  I think encouraging developers to test their own packages with the
> gold linker is good, because it helps accomplish #1, and increases
> their awareness in general.
>
> 3.  I think that having a tinderbox systematically testing using the
> gold linker is also good.
>
> 4.  I think that hitting devs with a cluebat when they ignore valid
> bugs is good.
>
> The flip side of this is that we're not necessarily better off if
> maintainers just abandon packages because they have terrible build
> systems.  At some point you need to work with them.  However, if
> they're not willing to at least stick in a slot operator dependency
> when asked to, then sure we should have a talk with them.  (A slot op
> dep will of course help by triggering rebuilds, but it doesn't
> actually directly fix the underlinking issue, which would require
> fixing the build system.)
>
> I think the big thing is acknowledging that packages that are missing
> dependencies or which are underlinked are defective.  Sure, it would
> be nice if somebody else came along and helped find our mistakes.
> However, that in itself doesn't excuse us from having made them in the
> first place.  And it certainly doesn't excuse giving people a hard
> time when they politely point them out.

+1

Perhaps much of the mechanics and ordinary part of these aforementioned 
task, would make for fertile ground of skills diversification for the 
'proxy-maintainers' ? Understanding and routine usage of a full suite of 
tools available under gentoo, could easily be missed during the proxy 
period. In fact, putting the tinderbox out there as part of the proxy 
training and perhaps available to a portion of the wider gentoo-user 
base, might also be a fertile area for technical growth or the gentoo 
community?

Access to there and other dev tools might be a powerful incentive, if 
packaged up attactively, for the gentoo user community to participate 
more in the less risky parts of gentoo development workflows?


hth,
James

Re: [gentoo-dev] Developers, please work on underlinking issues!

[Alexis Ballier]

On Fri, 19 Aug 2016 06:58:44 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> On Thu, 18 Aug 2016 15:21:16 +0200
> Alexis Ballier <aballier@gentoo.org> wrote:
> 
> > On Thu, 18 Aug 2016 08:13:14 -0400
> > Rich Freeman <rich0@gentoo.org> wrote:
> >   
> > > If you just check your packages occassionally to make sure they
> > > build with gold it completely achieves the goal, and it will
> > > actually result in fewer bugs using the non-gold linker as
> > > well.    
> > 
> > That's what a tinderbox is for. The only QA problem I see here is
> > that QA doesn't automate that kind of checks anymore since Diego
> > left. Maybe QA should ask Toralf to run a ld.gold tinderbox and
> > avoid asking people to randomly test random packages ?  
> 
> Yes, tinderboxing makes a lot of sense if the bugs are afterwards
> ignored by package maintainers.

"ignored" is rather strong here; considering people join gentoo and
basically work for free, I'd rather say "not considered important
enough" which yields basically the same result but without accusing
people of refusing to fix a bug. With that in mind, everybody is free
to submit patches to bugzie, and if it is a QA goal, QA is free to
apply it after some time. Nothing is blocked here, expect when people
prefer to hit fellow developers "with a cluebat" rather than getting
things done.

> Or in the best case, the maintainer
> tells reporter (Toralf) to file the bug upstream.

It's not because Toralf reported it that he is the only one that has
the right to report it upstream. Obtaining fixes from those that know
the package best is good, isn't it ? I thought we learned from the past
and tried to avoid, e.g., fixing valgrind warnings in libssl by our
own :)

Alexis.