[gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014

[gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014

[Brian Dolbec]

On Fri, 2013-07-05 at 22:18 -0400, Rick "Zero_Chaos" Farina wrote:
> On 07/05/2013 09:41 PM, Ryan Hill wrote:
> > On Fri, 05 Jul 2013 15:47:08 -0400
> > "Rick \"Zero_Chaos\" Farina" <zerochaos@gentoo.org> wrote:
> > 
> >> -----BEGIN PGP SIGNED MESSAGE-----
> >> Hash: SHA1
> >>
> >>> Paweł was nice enough to write a patch for us to get toolchain.eclass up to
> >>> EAPI 5.  I believe it still needs some pieces like prefix support and I
> >>> haven't reviewed it in depth but it looks good so far (and much simpler
> >>> than I thought (oops)).  I'm planning on moving up an EAPI at a time,
> >>> bumping it whenever we could use new features or people start hucking fruit.
> >>>
> >>>
> >> I would be forever in your debt if toolchain were on eapi 5 and had
> >> proper subslot deps.
> > 
> > What use case are you encountering?  I hadn't planned on using subslots, but
> > I'm open to good reasons/bribery.
> > 
> > 
> 
> Livecd builds work like this:
> 
> stage3 tarball is unpacked, then toolchain (minimum package set) is
> built with ROOT=/tmp/stage1root and is linked to the original (seed
> stage) while being built.
> 
> When we then move onto stage 2, it uses just the packages built during
> stage1 (/tmp/stage1root becomes /).  This means, if seed stage has
> mpc.so.0.1 but portage has since included mpc.so.2 that the gcc in
> stage2 is linked against mpc.so.0.1 but only mpc.so.0.2 is installed.
> 
> To combat this kind of failure we are currently running "emerge --update
> --deep --newuse --complete-graph --rebuild-if-new-ver gcc" which could
> just be "emerge --update gcc" if eapi 5 subslots were used properly.
> 
> Please forgive me if any of these details aren't perfect, I'm using my
> best recollection of the most recent issue which happened when mpc was
> bumped a few months ago.
> 
> I am available on irc quite a bit if you would like to discuss, however,
> if you want to keep it on the ML we really need to move this to -dev.
> 
> Thanks,
> Zero

Continuing this on -dev mail list.


The other thing we needed to do was completely remove the use of or
building of binpkgs during the update_seed stage.  Portage has no
capability to check binpkg linking to ensure the binpkg was properly
usable.  EAPI 5 subslots also put that info into the DEPENDS so portage
then considers that information.  It would then reject a binpkg properly
and build from source due to the abi mismatch.  I might also add that
using broken binpkgs in catalyst (like the mpc update) cause delayed
breakage, making trouble shooting the problem more difficult.

With proper use of subslots for base system pkgs, binpkgs can be re-used
safely, saving both build and turn around time to get fully working
stages and livecd's etc. in catalyst released.   The same holds true for
user systems that build and try to reuse binkgs.

[gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 3311 bytes --]

On Fri, 05 Jul 2013 21:27:46 -0700
Brian Dolbec <dolsen@gentoo.org> wrote:

> On Fri, 2013-07-05 at 22:18 -0400, Rick "Zero_Chaos" Farina wrote:
> > On 07/05/2013 09:41 PM, Ryan Hill wrote:
> > > On Fri, 05 Jul 2013 15:47:08 -0400
> > > "Rick \"Zero_Chaos\" Farina" <zerochaos@gentoo.org> wrote:
> > > 
> > >> -----BEGIN PGP SIGNED MESSAGE-----
> > >> Hash: SHA1
> > >>
> > >>> Paweł was nice enough to write a patch for us to get toolchain.eclass
> > >>> up to EAPI 5.  I believe it still needs some pieces like prefix support
> > >>> and I haven't reviewed it in depth but it looks good so far (and much
> > >>> simpler than I thought (oops)).  I'm planning on moving up an EAPI at a
> > >>> time, bumping it whenever we could use new features or people start
> > >>> hucking fruit.
> > >>>
> > >>>
> > >> I would be forever in your debt if toolchain were on eapi 5 and had
> > >> proper subslot deps.
> > > 
> > > What use case are you encountering?  I hadn't planned on using subslots,
> > > but I'm open to good reasons/bribery.
> > > 
> > > 
> > 
> > Livecd builds work like this:
> > 
> > stage3 tarball is unpacked, then toolchain (minimum package set) is
> > built with ROOT=/tmp/stage1root and is linked to the original (seed
> > stage) while being built.
> > 
> > When we then move onto stage 2, it uses just the packages built during
> > stage1 (/tmp/stage1root becomes /).  This means, if seed stage has
> > mpc.so.0.1 but portage has since included mpc.so.2 that the gcc in
> > stage2 is linked against mpc.so.0.1 but only mpc.so.0.2 is installed.
> > 
> > To combat this kind of failure we are currently running "emerge --update
> > --deep --newuse --complete-graph --rebuild-if-new-ver gcc" which could
> > just be "emerge --update gcc" if eapi 5 subslots were used properly.
> > 
> > Please forgive me if any of these details aren't perfect, I'm using my
> > best recollection of the most recent issue which happened when mpc was
> > bumped a few months ago.
> > 
> > I am available on irc quite a bit if you would like to discuss, however,
> > if you want to keep it on the ML we really need to move this to -dev.
> > 
> > Thanks,
> > Zero
> 
> Continuing this on -dev mail list.
> 
> 
> The other thing we needed to do was completely remove the use of or
> building of binpkgs during the update_seed stage.  Portage has no
> capability to check binpkg linking to ensure the binpkg was properly
> usable.  EAPI 5 subslots also put that info into the DEPENDS so portage
> then considers that information.  It would then reject a binpkg properly
> and build from source due to the abi mismatch.  I might also add that
> using broken binpkgs in catalyst (like the mpc update) cause delayed
> breakage, making trouble shooting the problem more difficult.
> 
> With proper use of subslots for base system pkgs, binpkgs can be re-used
> safely, saving both build and turn around time to get fully working
> stages and livecd's etc. in catalyst released.   The same holds true for
> user systems that build and try to reuse binkgs.

Thanks, these are good reasons.


-- 
Ryan Hill                        psn: dirtyepic_sk
   gcc-porting/toolchain/wxwidgets @ gentoo.org

47C3 6D62 4864 0E49 8E9E  7F92 ED38 BD49 957A 8463

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014

[Matt Turner]

On Fri, Jul 5, 2013 at 7:18 PM, Rick "Zero_Chaos" Farina
<zerochaos@gentoo.org> wrote:
> When we then move onto stage 2, it uses just the packages built during
> stage1 (/tmp/stage1root becomes /).  This means, if seed stage has
> mpc.so.0.1 but portage has since included mpc.so.2 that the gcc in
> stage2 is linked against mpc.so.0.1 but only mpc.so.0.2 is installed.
>
> To combat this kind of failure we are currently running "emerge --update
> - --deep --newuse --complete-graph --rebuild-if-new-ver gcc" which could
> just be "emerge --update gcc" if eapi 5 subslots were used properly.

The best solution to this is, and has always been, just updating gcc's
deps during update_seed. Or am I misremembering something?

As far as I know, you don't need to waste time rebuilding the seed
stage's gcc, since gcc is rebuilt in stage2 and then everything is
built by it in stage3.

Re: [gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014

[Brian Dolbec]

On Sat, 2013-07-06 at 00:08 -0700, Matt Turner wrote:
> On Fri, Jul 5, 2013 at 7:18 PM, Rick "Zero_Chaos" Farina
> <zerochaos@gentoo.org> wrote:
> > When we then move onto stage 2, it uses just the packages built during
> > stage1 (/tmp/stage1root becomes /).  This means, if seed stage has
> > mpc.so.0.1 but portage has since included mpc.so.2 that the gcc in
> > stage2 is linked against mpc.so.0.1 but only mpc.so.0.2 is installed.
> >
> > To combat this kind of failure we are currently running "emerge --update
> > - --deep --newuse --complete-graph --rebuild-if-new-ver gcc" which could
> > just be "emerge --update gcc" if eapi 5 subslots were used properly.
> 
> The best solution to this is, and has always been, just updating gcc's
> deps during update_seed. Or am I misremembering something?
> 
> As far as I know, you don't need to waste time rebuilding the seed
> stage's gcc, since gcc is rebuilt in stage2 and then everything is
> built by it in stage3.
> 

Yes, it does need to be rebuilt if key deps are updated.  The gcc
produced in the stage1 is broken, so won't run to rebuild itself during
the stage2 run.
Also the "--rebuild-if-new-ver gcc" only rebuilds gcc if it's deps are
updated to new versions (likely breaking lib linkage), it is not
unconditional.  It is not as accurate as subslots, so may rebuild more
often than possibly needed.  But it is better than having unreliable
stage builds, not knowing when they may break.
-- 
Brian Dolbec <dolsen@gentoo.org>

Re: [gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/06/2013 03:08 AM, Matt Turner wrote:
> On Fri, Jul 5, 2013 at 7:18 PM, Rick "Zero_Chaos" Farina
> <zerochaos@gentoo.org> wrote:
>> When we then move onto stage 2, it uses just the packages built during
>> stage1 (/tmp/stage1root becomes /).  This means, if seed stage has
>> mpc.so.0.1 but portage has since included mpc.so.2 that the gcc in
>> stage2 is linked against mpc.so.0.1 but only mpc.so.0.2 is installed.
>>
>> To combat this kind of failure we are currently running "emerge --update
>> - --deep --newuse --complete-graph --rebuild-if-new-ver gcc" which could
>> just be "emerge --update gcc" if eapi 5 subslots were used properly.
> 
> The best solution to this is, and has always been, just updating gcc's
> deps during update_seed. Or am I misremembering something?

You are misremembering that we are using preserve_libs to save our butts
when mpc is updated and gcc is still linked to the old mpc.  I feel very
uncomfortable as the recommendation of preserve-libs is to remerge as
soon as possible not "build a whole system like this".  Is there an
actual failure here? Not that I've seen yet, but it's an awkward way to
build in my opinion.

- -Zero
> 
> As far as I know, you don't need to waste time rebuilding the seed
> stage's gcc, since gcc is rebuilt in stage2 and then everything is
> built by it in stage3.
> 
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJR2DeYAAoJEKXdFCfdEflKDwwP/1Pr2eB9GjSKncEabiB9WkEb
eziSaccKcsmjdXq9Svg0dfTM9m9rgroK0iM15IWLHhbAoU/5beVPv4bOYVYejYkP
NBMWp2+MoIE7VRhziFToj7tHxTnXsUg1l3dMFWECewWpMewo9lZw1eYTn/iaUGaI
tkfNi7iQV9PvfknUgtQZ8lfgSUjUz2CdtjCyjaoMpO+vls+gVvH74vGETIMMrHWr
CN7iMfx7F6FGpc1+FxZ0CJ1zKSifY/1R+dLABass8xaLRMPNqTIpm8b37xD2tHOA
f2pfzCIgkwLEo8moJrmkl21CqC2CjqZ0HPqd3dd/wSTg2x1ccslNHVOUf8+mZu1I
4zwJUwS7e2w8rxcq/UTu9x3J18D2doFjLg1mLUtWgmptn9Tydr/GYL+yYJei0yK+
MiADpdK+UI5frUo1B8bZ+Gs0N5IIh2pcGkjdupz4HXRAeD+2VN5G0HBpTZ8I4vNI
rK9wRQN1iyxb4sn0Wr0n+GwSlxyTao6yuUSJwT5nfD1k9gSGI9Zh7tERlD/D9ceN
Vfvv0No+ikh47TDjm3hSmz2fdbTT5vxKecnXT72EpYQwIZFoVMo7tnRVwxd4gBbM
Fx4LDUaSn3ommMBZF/jRt5zsn+cGMB/7qHkNo1DPIbKgXgExEQWRz7Lxhy8Hm4er
YYVhPl4Zhc0/BCBwBpK/
=zakW
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014

[Matt Turner]

On Sat, Jul 6, 2013 at 12:46 AM, Brian Dolbec <dolsen@gentoo.org> wrote:
> Yes, it does need to be rebuilt if key deps are updated.  The gcc
> produced in the stage1 is broken, so won't run to rebuild itself during
> the stage2 run.

But we keep the old libs around via preserve_libs, and once stage1 is
done we effectively throw the seed stage out.

Re: [gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014

[Matt Turner]

On Sat, Jul 6, 2013 at 8:28 AM, Rick "Zero_Chaos" Farina
<zerochaos@gentoo.org> wrote:
> You are misremembering that we are using preserve_libs to save our butts
> when mpc is updated and gcc is still linked to the old mpc.  I feel very
> uncomfortable as the recommendation of preserve-libs is to remerge as
> soon as possible not "build a whole system like this".  Is there an
> actual failure here? Not that I've seen yet, but it's an awkward way to
> build in my opinion.

Keeping the old libs seems perfectly fine, since it's in a seed stage
that we don't care about after stage1 is complete.

An unnecessary build of gcc may not matter much on a relatively fast
amd64, but it's going to be a pain on a bunch of slower architectures.
And on mips/multilib it'll be even worse since we get to build the
libraries for three ABIs.

Re: [gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014

[Brian Dolbec]

On Sat, 2013-07-06 at 09:36 -0700, Matt Turner wrote:
> On Sat, Jul 6, 2013 at 12:46 AM, Brian Dolbec <dolsen@gentoo.org> wrote:
> > Yes, it does need to be rebuilt if key deps are updated.  The gcc
> > produced in the stage1 is broken, so won't run to rebuild itself during
> > the stage2 run.
> 
> But we keep the old libs around via preserve_libs, and once stage1 is
> done we effectively throw the seed stage out.
> 

It could be that I missed that combination of changes that could occur
during my testing.  But I am quite sure that it failed for me.  You
could always do some re-runs of those tests using an older seed stage
built against libmpc.so.1.

Re: [gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014

[Anthony G. Basile]

On 07/06/2013 12:39 PM, Matt Turner wrote:
> On Sat, Jul 6, 2013 at 8:28 AM, Rick "Zero_Chaos" Farina
> <zerochaos@gentoo.org> wrote:
>> You are misremembering that we are using preserve_libs to save our butts
>> when mpc is updated and gcc is still linked to the old mpc.  I feel very
>> uncomfortable as the recommendation of preserve-libs is to remerge as
>> soon as possible not "build a whole system like this".  Is there an
>> actual failure here? Not that I've seen yet, but it's an awkward way to
>> build in my opinion.
>
> Keeping the old libs seems perfectly fine, since it's in a seed stage
> that we don't care about after stage1 is complete.
>
> An unnecessary build of gcc may not matter much on a relatively fast
> amd64, but it's going to be a pain on a bunch of slower architectures.
> And on mips/multilib it'll be even worse since we get to build the
> libraries for three ABIs.
>

Thanks Matt, I was going to make that point. ~3 days to recompile world 
on the lemote yeelong.

-- 
Anthony G. Basile, Ph. D.
Chair of Information Technology
D'Youville College
Buffalo, NY 14201
(716) 829-8197

Re: [gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014

[Robin H. Johnson]

On Fri, Jul 05, 2013 at 09:27:46PM -0700, Brian Dolbec wrote:
> The other thing we needed to do was completely remove the use of or
> building of binpkgs during the update_seed stage.  Portage has no
> capability to check binpkg linking to ensure the binpkg was properly
> usable. 
Can somebody actually please implement this, to run before the binpkg
merge phase?

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/10/2013 10:03 PM, Robin H. Johnson wrote:
> On Fri, Jul 05, 2013 at 09:27:46PM -0700, Brian Dolbec wrote:
>> The other thing we needed to do was completely remove the use of
>> or building of binpkgs during the update_seed stage.  Portage has
>> no capability to check binpkg linking to ensure the binpkg was
>> properly usable.
> Can somebody actually please implement this, to run before the
> binpkg merge phase?
> 
Please be more specific, this is currently implemented...

- -Zero
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJR3ihMAAoJEKXdFCfdEflKH4sP/1dswIQytHVOXtI6mqzntqjy
9rBl2tkBft2gdQp7Vru3d4s0HiicpeLImSyvIM8H2FVmbEVutMwOOUWp2tllOUNh
bb98BnrAoSMz4GEPtcS2d0vmZ3HdDj99mrnkOOWcqvHRPMn/J0KogAyrCVRa0+Pb
bNEo2dU5g4Jegs8gl4l1ilH0hIyb7uqurwjd5G8NayJIs0U5gIpTHZzPvRJo0B98
tnzEsoSTY0JekgSGC+8xgR4ijmR0AicF5nCciKUjFf7OjEDitK6sF6g3m92zM8hU
fjPLTW0vjR63KbtOJNgw992DpQoUfRj2jYn4ErjDB1jKz5+xvLy0+rpwZPYA4hgf
QNck4OnLRqK6rMy45Tzu58gfUsH1GIdmdH0JJ0x2rBLu89GHB4HSR+UlE5ugxmvj
uzs2ZJ8MPIT7mYnrqJ8XbiMzDmkvspkixbi6YpbiNCi76R08ljNXFjT0Ju5nI29u
W+6p/we4FMPLo47W5zSAS4Csa0reIayaIpJTbeyfHVe88DcS6jO+JAs416mod9ec
1SYogk+SFrmFC0PyW9uQeP0gE12U20bw0NCI0R5+PmXIs54jrowIJgjB8hPH8Cpz
df7+ZNGuDaOqQnJT5j1EwzfAPEGVHmHtIaMu3bRMv0EJZj3WMa1rABtFFX+65S1x
dz87xG1hys5ZZmz8IzOQ
=+Lzs
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014

[Matt Turner]

On Wed, Jul 10, 2013 at 8:36 PM, Rick "Zero_Chaos" Farina
<zerochaos@gentoo.org> wrote:
> On 07/10/2013 10:03 PM, Robin H. Johnson wrote:
>> On Fri, Jul 05, 2013 at 09:27:46PM -0700, Brian Dolbec wrote:
>>> The other thing we needed to do was completely remove the use of
>>> or building of binpkgs during the update_seed stage.  Portage has
>>> no capability to check binpkg linking to ensure the binpkg was
>>> properly usable.
>> Can somebody actually please implement this, to run before the
>> binpkg merge phase?
>>
> Please be more specific, this is currently implemented...

He's talking about portage, not catalyst.

Re: [gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014

[Robin H. Johnson]

On Wed, Jul 10, 2013 at 11:36:44PM -0400, Rick "Zero_Chaos" Farina wrote:
> On 07/10/2013 10:03 PM, Robin H. Johnson wrote:
> > On Fri, Jul 05, 2013 at 09:27:46PM -0700, Brian Dolbec wrote:
> >> The other thing we needed to do was completely remove the use of
> >> or building of binpkgs during the update_seed stage.  Portage has
> >> no capability to check binpkg linking to ensure the binpkg was
> >> properly usable.
> > Can somebody actually please implement this, to run before the
> > binpkg merge phase?
> Please be more specific, this is currently implemented...
I was partially responding to dolsen's comment that it wasn't capable,
as well as a discussion I had with somebody on the ChromeOS team a few
months ago.

In the ChromeOS case, if a library was removed off the system, but
Portage still thought it was there, and you went to install a binpkg
that had an ELF dependency on the removed library, you'd have broken
binaries on the system.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-dev] Re: toolchain update was Re: [gentoo-project] Re: Questions for candidates for Gentoo Council 2013/2014

[Zac Medico]

On 07/10/2013 08:43 PM, Robin H. Johnson wrote:
> On Wed, Jul 10, 2013 at 11:36:44PM -0400, Rick "Zero_Chaos" Farina wrote:
>> On 07/10/2013 10:03 PM, Robin H. Johnson wrote:
>>> On Fri, Jul 05, 2013 at 09:27:46PM -0700, Brian Dolbec wrote:
>>>> The other thing we needed to do was completely remove the use of
>>>> or building of binpkgs during the update_seed stage.  Portage has
>>>> no capability to check binpkg linking to ensure the binpkg was
>>>> properly usable.
>>> Can somebody actually please implement this, to run before the
>>> binpkg merge phase?
>> Please be more specific, this is currently implemented...
> I was partially responding to dolsen's comment that it wasn't capable,
> as well as a discussion I had with somebody on the ChromeOS team a few
> months ago.
> 
> In the ChromeOS case, if a library was removed off the system, but
> Portage still thought it was there, and you went to install a binpkg
> that had an ELF dependency on the removed library, you'd have broken
> binaries on the system.

Ideally, this would be handled entirely by EAPI 5 slot-operator
dependencies. However, it seems like we'll need some form of sub-slot
pass-through [1] and/or sub-slot dictionaries [2] before we'll be able
to achieve complete adoption of slot-operators.

Meanwhile, adding a preinst sanity check seems like a good idea. The
LinkageMap class that's used for preserve-libs already has all of the
info that we need about installed libraries, so it's just a matter of
checking the binary package's NEEDED.ELF.2 entries against that.

[1] https://bugs.gentoo.org/show_bug.cgi?id=449094
[2] https://bugs.gentoo.org/show_bug.cgi?id=462138
-- 
Thanks,
Zac