[gentoo-dev] [RFC] First (experimental) 17.1 profiles news item for review

[gentoo-dev] [RFC] First (experimental) 17.1 profiles news item for review

[Michał Górny]

Hello, everyone.

The first news item I'd like to submit for 17.1 profiles follows.
The item is aimed at ~amd64 users who'd like to test the new profiles.
When they become stable, a separate news item for all our users will be
published.

===

Title: Experimental amd64 17.1 profiles up for testing
Author: Michał Górny <mgorny@gentoo.org>
Posted: 2017-12-xx
Revision: 1
News-Item-Format: 2.0
Display-If-Keyword: ~amd64

A new set of 17.1 amd64 profiles has been added to the Gentoo
repository. Those profiles switch to a more standard 'no SYMLINK_LIB'
multilib layout, and require explicit migration as described below. They
are considered experimental at the moment, and have a fair risk
of breaking your system. We would therefore like to ask our users to
test them on their non-production ~amd64 systems.

In those profiles, the lib->lib64 compatibility symlink is removed.
The 'lib' directory becomes a separate directory, that is used
for cross-arch and native non-library packages (gcc, clang) and 32-bit
libraries on the multilib profile (for better compatibility with
prebuilt x86 packages).

Migration from both 13.0 and 17.0 profiles is supported. In case
of the former, please read the news item for 17.0 upgrade first
and enable gcc 6.4.0 or newer first as explained there.

The migration is performed using app-portage/unsymlink-lib tool.
The following steps can be used to upgrade your system:

1. Sync and upgrade your system to the newest package versions
   to reduce the risk of issues.

2. Install the tool, e.g. via 'emerge -1v app-portage/unsymlink-lib'

3. Run 'unsymlink-lib --analyze' and check the output for obvious
   mistakes. If you need to perform any changes to the system, remember
   to run 'unsymlink-lib --analyze' again afterwards.

[past this point do not call emerge or modify /usr manually]

4. This is a very good time to make a backup.

5. Run 'unsymlink-lib --migrate'. You can add '--pretend' first to see
   what is going to happen.

6. Reboot your system and see if it still boots. Check if important
   programs work. In particular, check if e.g. 'emerge --info' works
   (but do not install anything). If you hit any serious problems,
   you can use 'unsymlink-lib --rollback' to revert the changes
   and return to step 3.

7. Run 'unsymlink-lib --finish'. You can add '--pretend' first to see
   what is going to happen but note that you're going to see a very long
   list of files to remove.

8. Switch the profile, e.g.:

     eselect profile set --force default/linux/amd64/17.1/desktop

[at this point you can start using emerge again]

9. Rebuild sys-devel/gcc. If you are switching from 13.0 profiles,
   rebuild sys-devel/binutils and sys-libs/glibc afterwards.

10. If you are using a multilib profile, rebuild all 32-bit packages.
    This can be done using:

      emerge -1v /lib32 /usr/lib32

    Alternatively, if you are switching from one of the 13.0 profiles
    you can rebuild all packages as detailed in the 17.0 news item.

11. Once the last 32-bit package is rebuilt, your package manager
    should remove the orphaned /lib32 and /usr/lib32 symlinks. If that
    does not happen, remove them manually.

For known issues, please see bug #506276 [1]. If you have any problems
with the new profiles or the migration procedure, please report a bug
and make it block the tracker.

[1]:https://bugs.gentoo.org/506276

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] [RFC] First (experimental) 17.1 profiles news item for review

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 381 bytes --]

>>>>> On Mon, 18 Dec 2017, Michał Górny wrote:

> Display-If-Keyword: ~amd64

Only keyword names [1] allowed there.

> multilib layout, and require explicit migration as described below. They
> are considered experimental at the moment, and have a fair risk

Your line wrapping has room for improvement. :)

[1] https://projects.gentoo.org/pms/6/pms.html#x1-280003.1.7

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] [RFC] First (experimental) 17.1 profiles news item for review

[Michał Górny]

W dniu wto, 19.12.2017 o godzinie 09∶08 +0100, użytkownik Ulrich Mueller
napisał:
> > > > > > On Mon, 18 Dec 2017, Michał Górny wrote:
> > Display-If-Keyword: ~amd64
> 
> Only keyword names [1] allowed there.

Fixed already. Though I'm thinking that it'd probably be better to use
Display-If-Profile to avoid displaying it on new installs using 17.1
already.

> 
> > multilib layout, and require explicit migration as described below. They
> > are considered experimental at the moment, and have a fair risk
> 
> Your line wrapping has room for improvement. :)
> 
> [1] https://projects.gentoo.org/pms/6/pms.html#x1-280003.1.7

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] [RFC] First (experimental) 17.1 profiles news item for review

[Brian Evans]

[-- Attachment #1.1: Type: text/plain, Size: 670 bytes --]

On 12/18/2017 1:51 PM, Michał Górny wrote:
> Hello, everyone.
> 
> The first news item I'd like to submit for 17.1 profiles follows.
> The item is aimed at ~amd64 users who'd like to test the new profiles.
> When they become stable, a separate news item for all our users will be
> published.
> 

I'm not sure if it is valid to put in the news item or modify the tool,
but users of PostgreSQL will need to either 'mv /usr/lib/postgresql
/usr/lib64' or, probably better, re-execute 'eselect postgresql'. The
symlink will not be moved by the unsymlink-lib tool and build failures
are likely to occur if libraries are searched for in this method

Brian


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] [RFC] First (experimental) 17.1 profiles news item for review

[Michał Górny]

W dniu śro, 20.12.2017 o godzinie 08∶28 -0500, użytkownik Brian Evans
napisał:
> On 12/18/2017 1:51 PM, Michał Górny wrote:
> > Hello, everyone.
> > 
> > The first news item I'd like to submit for 17.1 profiles follows.
> > The item is aimed at ~amd64 users who'd like to test the new profiles.
> > When they become stable, a separate news item for all our users will be
> > published.
> > 
> 
> I'm not sure if it is valid to put in the news item or modify the tool,
> but users of PostgreSQL will need to either 'mv /usr/lib/postgresql
> /usr/lib64' or, probably better, re-execute 'eselect postgresql'. The
> symlink will not be moved by the unsymlink-lib tool and build failures
> are likely to occur if libraries are searched for in this method
> 

Not sure if you got my message on IRC but the proper fix for this is for
eselect package to own the symlink. Orphan files are always a major PITA
for everyone.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] [RFC] First (experimental) 17.1 profiles news item for review (v2)

[Michał Górny]

Here's an update using Display-If-Profile header instead. The item
applies to users of 13.0 and 17.0 profiles, except for x32 profiles that
have been using the correct layout already.

===
Title: Experimental amd64 17.1 profiles up for testing
Author: Michał Górny <mgorny@gentoo.org>
Posted: 2017-12-xx
Revision: 1
News-Item-Format: 2.0
Display-If-Profile: default/linux/amd64/13.0
Display-If-Profile: default/linux/amd64/13.0/selinux
Display-If-Profile: default/linux/amd64/13.0/desktop
Display-If-Profile: default/linux/amd64/13.0/desktop/gnome
Display-If-Profile: default/linux/amd64/13.0/desktop/gnome/systemd
Display-If-Profile: default/linux/amd64/13.0/desktop/plasma
Display-If-Profile: default/linux/amd64/13.0/desktop/plasma/systemd
Display-If-Profile: default/linux/amd64/13.0/developer
Display-If-Profile: default/linux/amd64/13.0/no-multilib
Display-If-Profile: default/linux/amd64/13.0/systemd
Display-If-Profile: default/linux/amd64/17.0
Display-If-Profile: default/linux/amd64/17.0/selinux
Display-If-Profile: default/linux/amd64/17.0/hardened
Display-If-Profile: default/linux/amd64/17.0/hardened/selinux
Display-If-Profile: default/linux/amd64/17.0/desktop
Display-If-Profile: default/linux/amd64/17.0/desktop/gnome
Display-If-Profile: default/linux/amd64/17.0/desktop/gnome/systemd
Display-If-Profile: default/linux/amd64/17.0/desktop/plasma
Display-If-Profile: default/linux/amd64/17.0/desktop/plasma/systemd
Display-If-Profile: default/linux/amd64/17.0/developer
Display-If-Profile: default/linux/amd64/17.0/no-multilib
Display-If-Profile: default/linux/amd64/17.0/no-multilib/hardened
Display-If-Profile: default/linux/amd64/17.0/no-multilib/hardened/selinux
Display-If-Profile: default/linux/amd64/17.0/systemd

A new set of 17.1 amd64 profiles has been added to the Gentoo
repository. Those profiles switch to a more standard 'no SYMLINK_LIB'
multilib layout, and require explicit migration as described below. They
are considered experimental at the moment, and have a fair risk
of breaking your system. We would therefore like to ask our users to
test them on their non-production ~amd64 systems.

In those profiles, the lib->lib64 compatibility symlink is removed.
The 'lib' directory becomes a separate directory, that is used
for cross-arch and native non-library packages (gcc, clang) and 32-bit
libraries on the multilib profile (for better compatibility with
prebuilt x86 packages).

Migration from both 13.0 and 17.0 profiles is supported. In case
of the former, please read the news item for 17.0 upgrade first
and enable gcc 6.4.0 or newer first as explained there.

The migration is performed using app-portage/unsymlink-lib tool.
The following steps can be used to upgrade your system:

1. Sync and upgrade your system to the newest package versions
   to reduce the risk of issues.

2. Install the tool, e.g. via 'emerge -1v app-portage/unsymlink-lib'

3. Run 'unsymlink-lib --analyze' and check the output for obvious
   mistakes. If you need to perform any changes to the system, remember
   to run 'unsymlink-lib --analyze' again afterwards.

[past this point do not call emerge or modify /usr manually]

4. This is a very good time to make a backup.

5. Run 'unsymlink-lib --migrate'. You can add '--pretend' first to see
   what is going to happen.

6. Reboot your system and see if it still boots. Check if important
   programs work. In particular, check if e.g. 'emerge --info' works
   (but do not install anything). If you hit any serious problems,
   you can use 'unsymlink-lib --rollback' to revert the changes
   and return to step 3.

7. Run 'unsymlink-lib --finish'. You can add '--pretend' first to see
   what is going to happen but note that you're going to see a very long
   list of files to remove.

8. Switch the profile, e.g.:

     eselect profile set --force default/linux/amd64/17.1/desktop

[at this point you can start using emerge again]

9. Rebuild sys-devel/gcc. If you are switching from 13.0 profiles,
   rebuild sys-devel/binutils and sys-libs/glibc afterwards.

10. If you are using a multilib profile, rebuild all 32-bit packages.
    This can be done using:

      emerge -1v /lib32 /usr/lib32

    Alternatively, if you are switching from one of the 13.0 profiles
    you can rebuild all packages as detailed in the 17.0 news item.

11. Once the last 32-bit package is rebuilt, your package manager
    should remove the orphaned /lib32 and /usr/lib32 symlinks. If that
    does not happen, remove them manually.

For known issues, please see bug #506276 [1]. If you have any problems
with the new profiles or the migration procedure, please report a bug
and make it block the tracker.

[1]:https://bugs.gentoo.org/506276

-- 
Best regards,
Michał Górny

[gentoo-dev] Re: [RFC] First (experimental) 17.1 profiles news item for review (v2)

[Duncan]

Michał Górny posted on Wed, 20 Dec 2017 14:40:27 +0100 as excerpted:

> A new set of 17.1 amd64 profiles has been added to the Gentoo
> repository. Those profiles switch to a more standard 'no SYMLINK_LIB'
> multilib layout,
> and require explicit migration as described below. They are considered
> experimental at the moment, and have a fair risk of breaking your
> system. We would therefore like to ask our users to test them on their
> non-production ~amd64 systems.
> 
> In those profiles, the lib->lib64 compatibility symlink is removed.
> The 'lib' directory becomes a separate directory, that is used for
> cross-arch and native non-library packages (gcc, clang) and 32-bit
> libraries on the multilib profile (for better compatibility with
> prebuilt x86 packages).


In all this I don't see an answer to one question:

Will this eventually be the only supported choice, or is the 
compatibility-symlinked version going to be supported going forward too?  
If it's to be only-supported, what's the timeline?


Here's why I'm asking:  I'm on nomultilib and already have usrmerge (tho 
reverse, with / being canonical and /usr -> .), and (s)bin merge, so I 
already have a single canonical /bin and a single canonical /lib64, with 
various symlinks making the other paths work as well.

So there's no reason or benefit to me splitting /lib and /lib64 again, as 
that would go against the concept of the usr and sbin merges I've already 
done, and the long-time lib merges that gentoo has had on amd64 since 
before I switched to gentoo in 2004.  I've found I quite /like/ having a 
single bin dir and a single lib dir for everything, and this would undo 
that, forcing me to mentally track separate lib locations once again.


So I'll probably keep my merged lib here, managing it much like I do my 
merged bin and root/usr, but it'd be nice to know whether that's going to 
remain an official layout or not, and if not, what the timeframe for 
removing it is.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: [RFC] First (experimental) 17.1 profiles news item for review (v2)

[Michał Górny]

W dniu czw, 21.12.2017 o godzinie 05∶29 +0000, użytkownik Duncan
napisał:
> Michał Górny posted on Wed, 20 Dec 2017 14:40:27 +0100 as excerpted:
> 
> > A new set of 17.1 amd64 profiles has been added to the Gentoo
> > repository. Those profiles switch to a more standard 'no SYMLINK_LIB'
> > multilib layout,
> > and require explicit migration as described below. They are considered
> > experimental at the moment, and have a fair risk of breaking your
> > system. We would therefore like to ask our users to test them on their
> > non-production ~amd64 systems.
> > 
> > In those profiles, the lib->lib64 compatibility symlink is removed.
> > The 'lib' directory becomes a separate directory, that is used for
> > cross-arch and native non-library packages (gcc, clang) and 32-bit
> > libraries on the multilib profile (for better compatibility with
> > prebuilt x86 packages).
> 
> 
> In all this I don't see an answer to one question:
> 
> Will this eventually be the only supported choice, or is the 
> compatibility-symlinked version going to be supported going forward too?  
> If it's to be only-supported, what's the timeline?

The former. We'll make a timeline when the profiles are tested
and stable.

> Here's why I'm asking:  I'm on nomultilib and already have usrmerge (tho 
> reverse, with / being canonical and /usr -> .), and (s)bin merge, so I 
> already have a single canonical /bin and a single canonical /lib64, with 
> various symlinks making the other paths work as well.
> 
> So there's no reason or benefit to me splitting /lib and /lib64 again, as 
> that would go against the concept of the usr and sbin merges I've already 
> done, and the long-time lib merges that gentoo has had on amd64 since 
> before I switched to gentoo in 2004.  I've found I quite /like/ having a 
> single bin dir and a single lib dir for everything, and this would undo 
> that, forcing me to mentally track separate lib locations once again.

Custom setups were never really supported. It may work, it may not.
If you report a bug, it may be fixed or someone may close it as INVALID
or UPSTREAM. In particular, you'll probably have to deal with upstreams
yourself.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] Re: [RFC] First (experimental) 17.1 profiles news item for review (v2)

[Matthew Thode]

[-- Attachment #1: Type: text/plain, Size: 655 bytes --]

On 17-12-21 08:34:31, Michał Górny wrote:
> W dniu czw, 21.12.2017 o godzinie 05∶29 +0000, użytkownik Duncan
> napisał:
> > Michał Górny posted on Wed, 20 Dec 2017 14:40:27 +0100 as excerpted:
> > 
> > In all this I don't see an answer to one question:
> > 
> > Will this eventually be the only supported choice, or is the 
> > compatibility-symlinked version going to be supported going forward too?  
> > If it's to be only-supported, what's the timeline?
> 
> The former. We'll make a timeline when the profiles are tested
> and stable.
> 

What group are the ones making this decision?

-- 
Matthew Thode (prometheanfire)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Re: [RFC] First (experimental) 17.1 profiles news item for review (v2)

[Mike Gilbert]

On Thu, Dec 21, 2017 at 2:44 AM, Matthew Thode
<prometheanfire@gentoo.org> wrote:
> On 17-12-21 08:34:31, Michał Górny wrote:
>> W dniu czw, 21.12.2017 o godzinie 05∶29 +0000, użytkownik Duncan
>> napisał:
>> > Michał Górny posted on Wed, 20 Dec 2017 14:40:27 +0100 as excerpted:
>> >
>> > In all this I don't see an answer to one question:
>> >
>> > Will this eventually be the only supported choice, or is the
>> > compatibility-symlinked version going to be supported going forward too?
>> > If it's to be only-supported, what's the timeline?
>>
>> The former. We'll make a timeline when the profiles are tested
>> and stable.
>>
>
> What group are the ones making this decision?

The decision was effectively made by vapier in 2014 (see bug 506276);
mgorny is the one doing most of the work in 2017. There's also a group
of us who have been following the bug and experimenting with our own
systems since then.

If you disagree with this plan for some reason, please start a new thread.

Re: [gentoo-dev] Re: [RFC] First (experimental) 17.1 profiles news item for review (v2)

[Matthew Thode]

[-- Attachment #1: Type: text/plain, Size: 1186 bytes --]

On 17-12-21 09:10:09, Mike Gilbert wrote:
> On Thu, Dec 21, 2017 at 2:44 AM, Matthew Thode
> <prometheanfire@gentoo.org> wrote:
> > On 17-12-21 08:34:31, Michał Górny wrote:
> >> W dniu czw, 21.12.2017 o godzinie 05∶29 +0000, użytkownik Duncan
> >> napisał:
> >> > Michał Górny posted on Wed, 20 Dec 2017 14:40:27 +0100 as excerpted:
> >> >
> >> > In all this I don't see an answer to one question:
> >> >
> >> > Will this eventually be the only supported choice, or is the
> >> > compatibility-symlinked version going to be supported going forward too?
> >> > If it's to be only-supported, what's the timeline?
> >>
> >> The former. We'll make a timeline when the profiles are tested
> >> and stable.
> >>
> >
> > What group are the ones making this decision?
> 
> The decision was effectively made by vapier in 2014 (see bug 506276);
> mgorny is the one doing most of the work in 2017. There's also a group
> of us who have been following the bug and experimenting with our own
> systems since then.
> 
> If you disagree with this plan for some reason, please start a new thread.
> 

Nope, just curious :D

-- 
Matthew Thode (prometheanfire)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[gentoo-dev] Re: [RFC] First (experimental) 17.1 profiles news item for review (v2)

[Duncan]

Mike Gilbert posted on Thu, 21 Dec 2017 09:10:09 -0500 as excerpted:

> On Thu, Dec 21, 2017 at 2:44 AM, Matthew Thode
> <prometheanfire@gentoo.org> wrote:
>> On 17-12-21 08:34:31, Michał Górny wrote:
>>> W dniu czw, 21.12.2017 o godzinie 05∶29 +0000, użytkownik Duncan
>>> napisał:
>>> > Michał Górny posted on Wed, 20 Dec 2017 14:40:27 +0100 as excerpted:
>>> >
>>> > In all this I don't see an answer to one question:
>>> >
>>> > Will this eventually be the only supported choice, or is the
>>> > compatibility-symlinked version going to be supported going forward
>>> > too?
>>> > If it's to be only-supported, what's the timeline?
>>>
>>> The former. We'll make a timeline when the profiles are tested and
>>> stable.
>>>
>>>
>> What group are the ones making this decision?
> 
> The decision was effectively made by vapier in 2014 (see bug 506276);
> mgorny is the one doing most of the work in 2017. There's also a group
> of us who have been following the bug and experimenting with our own
> systems since then.
> 
> If you disagree with this plan for some reason, please start a new
> thread.

Reading the bug (506276), it seems to me it's all about /supporting/ 
symlink=no, discussing migration scripts to make it possible to migrate 
existing installations, etc, not /mandating/ it.

I don't see anything suggesting that vapier, or anyone else for that 
matter, decided it was going to be the _only_ choice, just that it would 
eventually be _a_ choice.

Much like x32 didn't replace x86 or mulitlib, but became another choice 
for those who wanted it.

Thus the only thing suggesting it'll be mandatory remains the direct 
answer to my direct question above, asking about it.

Meanwhile, I don't really disagree at this point, certainly not if much 
like usrmerge and (s)bin merge a gentooer is free to decide on their own, 
regardless of official support for it, but I'm wondering whether that 
will continue to be the case, or if the test (discussed in the bug) to 
die if the symlink remains when people are on the new profiles will 
effectively enforce it against an admin's will, even if that's not the 
intent.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: [RFC] First (experimental) 17.1 profiles news item for review (v2)

[Mike Gilbert]

On Thu, Dec 21, 2017 at 11:28 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> Reading the bug (506276), it seems to me it's all about /supporting/
> symlink=no, discussing migration scripts to make it possible to migrate
> existing installations, etc, not /mandating/ it.
>
> I don't see anything suggesting that vapier, or anyone else for that
> matter, decided it was going to be the _only_ choice, just that it would
> eventually be _a_ choice.

On 2014-03-31, vapier changed the bug summary to "kill SYMLINK_LIB=yes
support". I'm not sure how else that could be interpreted.