public inbox for gentoo-dev@lists.gentoo.org
 help / color / mirror / Atom feed
* [gentoo-dev] New SYMLINK_LIB=no migration tool for review
@ 2017-08-02 15:58 Michał Górny
  2017-08-02 17:51 ` [gentoo-dev] " Martin Vaeth
                   ` (2 more replies)
  0 siblings, 3 replies; 19+ messages in thread
From: Michał Górny @ 2017-08-02 15:58 UTC (permalink / raw)
  To: gentoo-dev

[-- Attachment #1: Type: text/plain, Size: 1788 bytes --]

Hi, everyone.

I've finally gotten around to writing a new tool for migrating amd64
systems to SYMLINK_LIB=no layout [1]. I've put it in symlink-lib-
migration [2] repository along with a README. Please review it and give
it more testing.

The tool has a few advantages over the one attached to the bug. Most
notably:

1. It runs in three-stage semi-automatic mode. This gives the user
an explicit opportunity to verify the action plan for any obvious
mistakes, and to test a temporary 'lib.new' layout to confirm that
the migration won't break anything before it's final.

2. It supports a mid-migration rollback -- if 'lib.new' layout breaks
stuff, the user runs './migrate.py --rollback' and he's back home.

3. It works on top-directory level whenever possible. The stuff destined
for /usr/lib is moved correctly along with any user-created files. When
a top-directory is split between lib+lib64, all files except for those
explicitly destined for lib64 land in lib (arbitrary decision).

4. It does not reinvent the wheel poorly to copy files. Instead, it
calls 'cp -a --reflink=auto ...' to guarantee that everything will be
preserved correctly.

5. It does not reinvent the wheel to parse vdb. Instead, it uses
the Portage API to get installed file list. Portage is only required
during the initial analysis phase, and the actual migration/rollback can
be done without it (or with it being broken).

I've limited the scope of the tool to do the filesystem manipulation.
Afterwards, it tells user to update the profile or adjust make.conf,
and to rebuild all the packages.

What are your thoughts?


[1]:https://bugs.gentoo.org/show_bug.cgi?id=506276
[2]:https://github.com/mgorny/symlink-lib-migration

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 988 bytes --]

^ permalink raw reply	[flat|nested] 19+ messages in thread

* [gentoo-dev] Re: New SYMLINK_LIB=no migration tool for review
  2017-08-02 15:58 [gentoo-dev] New SYMLINK_LIB=no migration tool for review Michał Górny
@ 2017-08-02 17:51 ` Martin Vaeth
  2017-08-02 18:10   ` Mike Gilbert
  2017-08-02 18:10   ` Holger Hoffstätte
  2017-08-11 23:56 ` [gentoo-dev] " Gerogy Yakovlev
  2017-08-12 21:33 ` Michał Górny
  2 siblings, 2 replies; 19+ messages in thread
From: Martin Vaeth @ 2017-08-02 17:51 UTC (permalink / raw)
  To: gentoo-dev

Michał Górny <mgorny@gentoo.org> wrote:
>
> What are your thoughts?

If this already was discussed then sorry for the noise:

What is the rationale for merging lib32 with lib?
Wouldn't it be somewhat cleaner to have a completely
split structure

lib64
lib32
libx32 (possibly)
lib



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [gentoo-dev] Re: New SYMLINK_LIB=no migration tool for review
  2017-08-02 17:51 ` [gentoo-dev] " Martin Vaeth
@ 2017-08-02 18:10   ` Mike Gilbert
  2017-08-02 19:07     ` Martin Vaeth
  2017-08-02 18:10   ` Holger Hoffstätte
  1 sibling, 1 reply; 19+ messages in thread
From: Mike Gilbert @ 2017-08-02 18:10 UTC (permalink / raw)
  To: Gentoo Dev

On Wed, Aug 2, 2017 at 1:51 PM, Martin Vaeth <martin@mvath.de> wrote:
> Michał Górny <mgorny@gentoo.org> wrote:
>>
>> What are your thoughts?
>
> If this already was discussed then sorry for the noise:
>
> What is the rationale for merging lib32 with lib?
> Wouldn't it be somewhat cleaner to have a completely
> split structure
>
> lib64
> lib32
> libx32 (possibly)
> lib

Here are a couple of reasons:

1. Other distros (notably Red Hat and Fedora) put 32-bit libs in "lib".
2. The path to the 32-bit runtime linker (/lib/ld-linux.so.2) is
hard-coded in every x86 binary on your system.


^ permalink raw reply	[flat|nested] 19+ messages in thread

* [gentoo-dev] Re: New SYMLINK_LIB=no migration tool for review
  2017-08-02 17:51 ` [gentoo-dev] " Martin Vaeth
  2017-08-02 18:10   ` Mike Gilbert
@ 2017-08-02 18:10   ` Holger Hoffstätte
  1 sibling, 0 replies; 19+ messages in thread
From: Holger Hoffstätte @ 2017-08-02 18:10 UTC (permalink / raw)
  To: gentoo-dev

On Wed, 02 Aug 2017 17:51:43 +0000, Martin Vaeth wrote:

> Michał Górny <mgorny@gentoo.org> wrote:
>>
>> What are your thoughts?
> 
> If this already was discussed then sorry for the noise:
> 
> What is the rationale for merging lib32 with lib?
> Wouldn't it be somewhat cleaner to have a completely
> split structure
> 
> lib64
> lib32
> libx32 (possibly)
> lib

I wondered the same. If anything it would make more sense to merge
lib64 into lib, since it's the platform's default (assuming 64bit).
Merging lib32 into lib is going to cause nothing but problems.

-h



^ permalink raw reply	[flat|nested] 19+ messages in thread

* [gentoo-dev] Re: New SYMLINK_LIB=no migration tool for review
  2017-08-02 18:10   ` Mike Gilbert
@ 2017-08-02 19:07     ` Martin Vaeth
  2017-08-02 19:25       ` Mike Gilbert
  2017-08-02 19:44       ` Michał Górny
  0 siblings, 2 replies; 19+ messages in thread
From: Martin Vaeth @ 2017-08-02 19:07 UTC (permalink / raw)
  To: gentoo-dev

Mike Gilbert <floppym@gentoo.org> wrote:
> On Wed, Aug 2, 2017 at 1:51 PM, Martin Vaeth <martin@mvath.de> wrote:
>> If this already was discussed then sorry for the noise:
>>
>> What is the rationale for merging lib32 with lib?
>> Wouldn't it be somewhat cleaner to have a completely
>> split structure
>>
>> lib64
>> lib32
>> libx32 (possibly)
>> lib
>
> Here are a couple of reasons:
>
> 1. Other distros (notably Red Hat and Fedora) put 32-bit libs in "lib".

According to bug 506276, Debian has instead merged 64-bit to lib.
So it seems to me that there is no "mainstream" to follow.
Perhaps striving for the cleanest solution would be the best?

> 2. The path to the 32-bit runtime linker (/lib/ld-linux.so.2) is
> hard-coded in every x86 binary on your system.

I am afraid that these must stay exceptional in any case:
Also currently, gentoo (and if I understood correctly, also Debian
and Red Hat) has the possible ld-linux{,-x86-64,-x32}.so.2 symlinks
in _all_ /lib* directories; I suppose that this is not intended to
change. (I assume that a change might break some proprietary binaries
which might have hard-coded the "wrong" directory.)



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [gentoo-dev] Re: New SYMLINK_LIB=no migration tool for review
  2017-08-02 19:07     ` Martin Vaeth
@ 2017-08-02 19:25       ` Mike Gilbert
  2017-08-03  1:02         ` Walter Dnes
                           ` (2 more replies)
  2017-08-02 19:44       ` Michał Górny
  1 sibling, 3 replies; 19+ messages in thread
From: Mike Gilbert @ 2017-08-02 19:25 UTC (permalink / raw)
  To: Gentoo Dev

On Wed, Aug 2, 2017 at 3:07 PM, Martin Vaeth <martin@mvath.de> wrote:
> Mike Gilbert <floppym@gentoo.org> wrote:
>> On Wed, Aug 2, 2017 at 1:51 PM, Martin Vaeth <martin@mvath.de> wrote:
>>> If this already was discussed then sorry for the noise:
>>>
>>> What is the rationale for merging lib32 with lib?
>>> Wouldn't it be somewhat cleaner to have a completely
>>> split structure
>>>
>>> lib64
>>> lib32
>>> libx32 (possibly)
>>> lib
>>
>> Here are a couple of reasons:
>>
>> 1. Other distros (notably Red Hat and Fedora) put 32-bit libs in "lib".
>
> According to bug 506276, Debian has instead merged 64-bit to lib.
> So it seems to me that there is no "mainstream" to follow.
> Perhaps striving for the cleanest solution would be the best?

Debian puts 64-bit libs in /lib/(host), where (host) is something like
x86_64-linux-gnu. They don't get put in /lib directly. They call this
"multiarch".

Migrating Gentoo to a "multiarch" config is a larger project.


^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [gentoo-dev] Re: New SYMLINK_LIB=no migration tool for review
  2017-08-02 19:07     ` Martin Vaeth
  2017-08-02 19:25       ` Mike Gilbert
@ 2017-08-02 19:44       ` Michał Górny
  1 sibling, 0 replies; 19+ messages in thread
From: Michał Górny @ 2017-08-02 19:44 UTC (permalink / raw)
  To: gentoo-dev

[-- Attachment #1: Type: text/plain, Size: 1855 bytes --]

On śro, 2017-08-02 at 19:07 +0000, Martin Vaeth wrote:
> Mike Gilbert <floppym@gentoo.org> wrote:
> > On Wed, Aug 2, 2017 at 1:51 PM, Martin Vaeth <martin@mvath.de> wrote:
> > > If this already was discussed then sorry for the noise:
> > > 
> > > What is the rationale for merging lib32 with lib?
> > > Wouldn't it be somewhat cleaner to have a completely
> > > split structure
> > > 
> > > lib64
> > > lib32
> > > libx32 (possibly)
> > > lib
> > 
> > Here are a couple of reasons:
> > 
> > 1. Other distros (notably Red Hat and Fedora) put 32-bit libs in "lib".
> 
> According to bug 506276, Debian has instead merged 64-bit to lib.
> So it seems to me that there is no "mainstream" to follow.
> Perhaps striving for the cleanest solution would be the best?

'No mainstream' as you claim it doesn't mean it's fine to invent yet
another completely incompatible solution.

> > 2. The path to the 32-bit runtime linker (/lib/ld-linux.so.2) is
> > hard-coded in every x86 binary on your system.
> 
> I am afraid that these must stay exceptional in any case:
> Also currently, gentoo (and if I understood correctly, also Debian
> and Red Hat) has the possible ld-linux{,-x86-64,-x32}.so.2 symlinks
> in _all_ /lib* directories; I suppose that this is not intended to
> change. (I assume that a change might break some proprietary binaries
> which might have hard-coded the "wrong" directory.)

This is not true.

$ ls /lib*/ld-linux*
/lib64/ld-linux-x86-64.so.2  /lib/ld-linux.so.2  /libx32/ld-linux-x32.so.2

The only symlinks ever there was in /lib64 because of the Gentoo symlink
deviation.

The 32-bit proprietary binaries use exactly /lib which is the main
reason for the switch. Plus, the linker uses /lib path independently of
how multilib is done on the system.

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 988 bytes --]

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [gentoo-dev] Re: New SYMLINK_LIB=no migration tool for review
  2017-08-02 19:25       ` Mike Gilbert
@ 2017-08-03  1:02         ` Walter Dnes
  2017-08-03  2:09           ` Benda Xu
  2017-08-03  6:55           ` Michał Górny
  2017-08-03  7:23         ` Martin Vaeth
  2017-08-03  7:50         ` Martin Vaeth
  2 siblings, 2 replies; 19+ messages in thread
From: Walter Dnes @ 2017-08-03  1:02 UTC (permalink / raw)
  To: gentoo-dev

On Wed, Aug 02, 2017 at 03:25:01PM -0400, Mike Gilbert wrote
> On Wed, Aug 2, 2017 at 3:07 PM, Martin Vaeth <martin@mvath.de> wrote:
> > Mike Gilbert <floppym@gentoo.org> wrote:
> >> On Wed, Aug 2, 2017 at 1:51 PM, Martin Vaeth <martin@mvath.de> wrote:
> >>> If this already was discussed then sorry for the noise:
> >>>
> >>> What is the rationale for merging lib32 with lib?
> >>> Wouldn't it be somewhat cleaner to have a completely
> >>> split structure
> >>>
> >>> lib64
> >>> lib32
> >>> libx32 (possibly)
> >>> lib
> >>
> >> Here are a couple of reasons:
> >>
> >> 1. Other distros (notably Red Hat and Fedora) put 32-bit libs in "lib".
> >
> > According to bug 506276, Debian has instead merged 64-bit to lib.
> > So it seems to me that there is no "mainstream" to follow.
> > Perhaps striving for the cleanest solution would be the best?
> 
> Debian puts 64-bit libs in /lib/(host), where (host) is something like
> x86_64-linux-gnu. They don't get put in /lib directly. They call this
> "multiarch".
> 
> Migrating Gentoo to a "multiarch" config is a larger project.

  And what happens when 128-bit cpus debut? /lib128?

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications


^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [gentoo-dev] Re: New SYMLINK_LIB=no migration tool for review
  2017-08-03  1:02         ` Walter Dnes
@ 2017-08-03  2:09           ` Benda Xu
  2017-08-03  6:55           ` Michał Górny
  1 sibling, 0 replies; 19+ messages in thread
From: Benda Xu @ 2017-08-03  2:09 UTC (permalink / raw)
  To: gentoo-dev

"Walter Dnes" <waltdnes@waltdnes.org> writes:

>   And what happens when 128-bit cpus debut? /lib128?

In this case CHOST makes more sense.  From my understanding, the Exherbo
approach is the cleanest.

Benda


^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [gentoo-dev] Re: New SYMLINK_LIB=no migration tool for review
  2017-08-03  1:02         ` Walter Dnes
  2017-08-03  2:09           ` Benda Xu
@ 2017-08-03  6:55           ` Michał Górny
  1 sibling, 0 replies; 19+ messages in thread
From: Michał Górny @ 2017-08-03  6:55 UTC (permalink / raw)
  To: gentoo-dev

[-- Attachment #1: Type: text/plain, Size: 1406 bytes --]

On śro, 2017-08-02 at 21:02 -0400, Walter Dnes wrote:
> On Wed, Aug 02, 2017 at 03:25:01PM -0400, Mike Gilbert wrote
> > On Wed, Aug 2, 2017 at 3:07 PM, Martin Vaeth <martin@mvath.de> wrote:
> > > Mike Gilbert <floppym@gentoo.org> wrote:
> > > > On Wed, Aug 2, 2017 at 1:51 PM, Martin Vaeth <martin@mvath.de> wrote:
> > > > > If this already was discussed then sorry for the noise:
> > > > > 
> > > > > What is the rationale for merging lib32 with lib?
> > > > > Wouldn't it be somewhat cleaner to have a completely
> > > > > split structure
> > > > > 
> > > > > lib64
> > > > > lib32
> > > > > libx32 (possibly)
> > > > > lib
> > > > 
> > > > Here are a couple of reasons:
> > > > 
> > > > 1. Other distros (notably Red Hat and Fedora) put 32-bit libs in "lib".
> > > 
> > > According to bug 506276, Debian has instead merged 64-bit to lib.
> > > So it seems to me that there is no "mainstream" to follow.
> > > Perhaps striving for the cleanest solution would be the best?
> > 
> > Debian puts 64-bit libs in /lib/(host), where (host) is something like
> > x86_64-linux-gnu. They don't get put in /lib directly. They call this
> > "multiarch".
> > 
> > Migrating Gentoo to a "multiarch" config is a larger project.
> 
>   And what happens when 128-bit cpus debut? /lib128?

Then we use whatever path is used in the appropriate ABI.

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 988 bytes --]

^ permalink raw reply	[flat|nested] 19+ messages in thread

* [gentoo-dev] Re: New SYMLINK_LIB=no migration tool for review
  2017-08-02 19:25       ` Mike Gilbert
  2017-08-03  1:02         ` Walter Dnes
@ 2017-08-03  7:23         ` Martin Vaeth
  2017-08-03  7:50         ` Martin Vaeth
  2 siblings, 0 replies; 19+ messages in thread
From: Martin Vaeth @ 2017-08-03  7:23 UTC (permalink / raw)
  To: gentoo-dev

Michał Górny <mgorny@gentoo.org> wrote:
>
> 'No mainstream' as you claim it doesn't mean it's fine to invent yet
> another completely incompatible solution.

As I understand, the compatibility with Debian might be increased
(keeping /lib32), at the cost of slightly decreasing the compatibility
with Red Hat (concerning 32bit emulation).
So depending on the use case, it might as well be more compatible.

>> in _all_ /lib* directories
>
> This is not true.

You are right, I had made a stupid mistake. (I forgot that I still
have the symlink lib->lib64 so that of course I should not have
wondered why all ld-* existed in both libs...)

> the linker uses /lib path independently of how multilib is done
> on the system.

I am not sure whether I understand what you mean.
Do you mean that practically sys-libs/binutils or sys-libs/glibc
have hardcoded
/lib -> 32bit
/lib64 -> 64bit
(concerning ld-*) unless one patches it out?
In this case you are right that there is an upstream to follow.
In this case I wonder why Debian decided to patch out this
upstream decision to support /lib32.

> The 32-bit proprietary binaries use exactly /lib which is the main
> reason for the switch.

I thought the main reason for the switch is to avoid the
merging of /lib and /lib64 which in bug 506276 led to problems
for debugging.
For this reason I am afraid that now merging /lib and /lib32
might again lead to a similar type of problems.
I am aware that exactly the cited problem is now mitigated,
because this time the merging does not happen over a symlink.
But anyway, it seems unnatural when "curing" from a merge to
decide to merge something else.



^ permalink raw reply	[flat|nested] 19+ messages in thread

* [gentoo-dev] Re: New SYMLINK_LIB=no migration tool for review
  2017-08-02 19:25       ` Mike Gilbert
  2017-08-03  1:02         ` Walter Dnes
  2017-08-03  7:23         ` Martin Vaeth
@ 2017-08-03  7:50         ` Martin Vaeth
  2017-08-03  7:57           ` Michał Górny
  2 siblings, 1 reply; 19+ messages in thread
From: Martin Vaeth @ 2017-08-03  7:50 UTC (permalink / raw)
  To: gentoo-dev

Mike Gilbert <floppym@gentoo.org> wrote:
> Debian puts 64-bit libs in /lib/(host)

Yes, this is somewhat weird:
They have /lib/i386-linux-gnu/ and /lib/x86_64-linux-gnu/
but anyway they use /lib32 instead of e.g. /lib/i686-linux-gnu/
Their reasons for this are mysterious to me.

> Migrating Gentoo to a "multiarch" config is a larger project.

Migrating to the completely split setting /lib{,32,64,x32}
would be _less_ intrusive than to simultaneously omitting /lib32:
With a slight modification of Michał's tool (in the instructions
and keeping lib32 untouched) not even a re-emerge of anything
would be necessary (as is the case now).

This is one of the reasons why a completely split layout appears
safer to me than to combine it with yet another new merge we
have no experience with.



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [gentoo-dev] Re: New SYMLINK_LIB=no migration tool for review
  2017-08-03  7:50         ` Martin Vaeth
@ 2017-08-03  7:57           ` Michał Górny
  2017-08-03 14:30             ` Martin Vaeth
  0 siblings, 1 reply; 19+ messages in thread
From: Michał Górny @ 2017-08-03  7:57 UTC (permalink / raw)
  To: gentoo-dev

[-- Attachment #1: Type: text/plain, Size: 1371 bytes --]

On czw, 2017-08-03 at 07:50 +0000, Martin Vaeth wrote:
> Mike Gilbert <floppym@gentoo.org> wrote:
> > Debian puts 64-bit libs in /lib/(host)
> 
> Yes, this is somewhat weird:
> They have /lib/i386-linux-gnu/ and /lib/x86_64-linux-gnu/
> but anyway they use /lib32 instead of e.g. /lib/i686-linux-gnu/
> Their reasons for this are mysterious to me.
> 
> > Migrating Gentoo to a "multiarch" config is a larger project.
> 
> Migrating to the completely split setting /lib{,32,64,x32}
> would be _less_ intrusive than to simultaneously omitting /lib32:
> With a slight modification of Michał's tool (in the instructions
> and keeping lib32 untouched) not even a re-emerge of anything
> would be necessary (as is the case now).

I have been running such a layout for over a year. I've stopped when
base-system proactively blocked it in baselayout, and multiple people
in Gentoo rejected providing support for it.

> This is one of the reasons why a completely split layout appears
> safer to me than to combine it with yet another new merge we
> have no experience with.

Except that it breaks every 32-bit x86 pre-built executable.

Anyway, this is off-topic and a waste of everyone's time. Please review
the tool instead of waking up last-minute to redesign everything that's
been designed for 10+ years.

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 988 bytes --]

^ permalink raw reply	[flat|nested] 19+ messages in thread

* [gentoo-dev] Re: New SYMLINK_LIB=no migration tool for review
  2017-08-03  7:57           ` Michał Górny
@ 2017-08-03 14:30             ` Martin Vaeth
  0 siblings, 0 replies; 19+ messages in thread
From: Martin Vaeth @ 2017-08-03 14:30 UTC (permalink / raw)
  To: gentoo-dev

Michał Górny <mgorny@gentoo.org> wrote:
>
> I have been running such a layout for over a year. [...]

Thanks for clarifying that this already was discussed.
Obviously, I was not aware about this discussion, and perhaps
I was not the only one.

> instead of waking up last-minute to redesign [...]

Pointing me to the discussion would have been sufficient:
If this was already discussed, it makes no sense to open
that bottle again.
As mentioned in my first posting:
In this case I am sorry for the noise.



^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [gentoo-dev] New SYMLINK_LIB=no migration tool for review
  2017-08-02 15:58 [gentoo-dev] New SYMLINK_LIB=no migration tool for review Michał Górny
  2017-08-02 17:51 ` [gentoo-dev] " Martin Vaeth
@ 2017-08-11 23:56 ` Gerogy Yakovlev
  2017-08-12  7:12   ` Michał Górny
  2017-08-12 21:33 ` Michał Górny
  2 siblings, 1 reply; 19+ messages in thread
From: Gerogy Yakovlev @ 2017-08-11 23:56 UTC (permalink / raw)
  To: gentoo-dev

Hi,

I was able to test this one a bit.
The test subjects were:
 ~amd64/openrc/desktop system which I was going to wipe anyway.
 amd64/openrc latest snapshot, updated to ~amd64 with boostrtap.sh

for the latter symlink migration was done before bootstrapping,


So far I found 3 issues so far:

1) kernel modules. modprobe looks into /lib/modules/$(uname -r), but
after running the tool they end up in /lib64/modules.

2) app-eselect/eselect-mesa  has some hardcoded paths:
	MESA_CONF_DIR="${EROOT}/usr/share/mesa"
	MESA_DIR_64="${EROOT}/usr/lib/mesa"
	DRI_DIR_64="${EROOT}/usr/lib/dri"
	MESA_DIR_32="${EROOT}/usr/lib32/mesa"
	DRI_DIR_32="${EROOT}/usr/lib32/dri"
that leads to many funny breakages.

3) and minor one, sys-boot/refind will fails to build, because it looks
for 64bit libs in lib.


I'm going to play with the system next couple of weeks and If I find
anything else I'll let you know.


Regards,

Georgy Yakovlev.

On 08/02/2017 08:58 AM, Michał Górny wrote:
> Hi, everyone.
> 
> I've finally gotten around to writing a new tool for migrating amd64
> systems to SYMLINK_LIB=no layout [1]. I've put it in symlink-lib-
> migration [2] repository along with a README. Please review it and give
> it more testing.
> 
> The tool has a few advantages over the one attached to the bug. Most
> notably:
> 
> 1. It runs in three-stage semi-automatic mode. This gives the user
> an explicit opportunity to verify the action plan for any obvious
> mistakes, and to test a temporary 'lib.new' layout to confirm that
> the migration won't break anything before it's final.
> 
> 2. It supports a mid-migration rollback -- if 'lib.new' layout breaks
> stuff, the user runs './migrate.py --rollback' and he's back home.
> 
> 3. It works on top-directory level whenever possible. The stuff destined
> for /usr/lib is moved correctly along with any user-created files. When
> a top-directory is split between lib+lib64, all files except for those
> explicitly destined for lib64 land in lib (arbitrary decision).
> 
> 4. It does not reinvent the wheel poorly to copy files. Instead, it
> calls 'cp -a --reflink=auto ...' to guarantee that everything will be
> preserved correctly.
> 
> 5. It does not reinvent the wheel to parse vdb. Instead, it uses
> the Portage API to get installed file list. Portage is only required
> during the initial analysis phase, and the actual migration/rollback can
> be done without it (or with it being broken).
> 
> I've limited the scope of the tool to do the filesystem manipulation.
> Afterwards, it tells user to update the profile or adjust make.conf,
> and to rebuild all the packages.
> 
> What are your thoughts?
> 
> 
> [1]:https://bugs.gentoo.org/show_bug.cgi?id=506276
> [2]:https://github.com/mgorny/symlink-lib-migration
> 


^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [gentoo-dev] New SYMLINK_LIB=no migration tool for review
  2017-08-11 23:56 ` [gentoo-dev] " Gerogy Yakovlev
@ 2017-08-12  7:12   ` Michał Górny
  0 siblings, 0 replies; 19+ messages in thread
From: Michał Górny @ 2017-08-12  7:12 UTC (permalink / raw)
  To: gentoo-dev

[-- Attachment #1: Type: text/plain, Size: 1400 bytes --]

On pią, 2017-08-11 at 16:56 -0700, Gerogy Yakovlev wrote:
> Hi,
> 
> I was able to test this one a bit.
> The test subjects were:
>  ~amd64/openrc/desktop system which I was going to wipe anyway.
>  amd64/openrc latest snapshot, updated to ~amd64 with boostrtap.sh
> 
> for the latter symlink migration was done before bootstrapping,

Thanks a lot for testing.

> So far I found 3 issues so far:
> 
> 1) kernel modules. modprobe looks into /lib/modules/$(uname -r), but
> after running the tool they end up in /lib64/modules.

Hmm, my test chroot did not have Linux modules. I'm going to test that. 
I suspect it's because 'modules' is not owned by any package on your
system, so it doesn't touch it at all.

I need to rethink that. Maybe it'd be better to just move everything
unowned to /lib.

> 2) app-eselect/eselect-mesa  has some hardcoded paths:
> 	MESA_CONF_DIR="${EROOT}/usr/share/mesa"
> 	MESA_DIR_64="${EROOT}/usr/lib/mesa"
> 	DRI_DIR_64="${EROOT}/usr/lib/dri"
> 	MESA_DIR_32="${EROOT}/usr/lib32/mesa"
> 	DRI_DIR_32="${EROOT}/usr/lib32/dri"
> that leads to many funny breakages.

https://bugs.gentoo.org/486712

> 
> 3) and minor one, sys-boot/refind will fails to build, because it looks
> for 64bit libs in lib.

Could you file a bug with a build log for it? Leave a note that it's on
SYMLINK_LIB=no system.

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 988 bytes --]

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [gentoo-dev] New SYMLINK_LIB=no migration tool for review
  2017-08-02 15:58 [gentoo-dev] New SYMLINK_LIB=no migration tool for review Michał Górny
  2017-08-02 17:51 ` [gentoo-dev] " Martin Vaeth
  2017-08-11 23:56 ` [gentoo-dev] " Gerogy Yakovlev
@ 2017-08-12 21:33 ` Michał Górny
  2017-08-19 22:25   ` Georgy Yakovlev
  2 siblings, 1 reply; 19+ messages in thread
From: Michał Górny @ 2017-08-12 21:33 UTC (permalink / raw)
  To: gentoo-dev

[-- Attachment #1: Type: text/plain, Size: 874 bytes --]

On śro, 2017-08-02 at 17:58 +0200, Michał Górny wrote:
> Hi, everyone.
> 
> I've finally gotten around to writing a new tool for migrating amd64
> systems to SYMLINK_LIB=no layout [1]. I've put it in symlink-lib-
> migration [2] repository along with a README. Please review it and give
> it more testing.

I've pushed two important fixes now:

a. The tool now processes unowned files as well -- *.{a,la,so} are left
in lib64 (i.e. the symlinks created by db.eclass and alikes) while
everything else goes into lib.

b. I've fixed cleanup phase to also remove top-level files
and directories that were moved out of lib64. Also, I've fixed it not to
complain about trying to remove non-empty directories.


> [1]:https://bugs.gentoo.org/show_bug.cgi?id=506276
> [2]:https://github.com/mgorny/symlink-lib-migration

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 988 bytes --]

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [gentoo-dev] New SYMLINK_LIB=no migration tool for review
  2017-08-12 21:33 ` Michał Górny
@ 2017-08-19 22:25   ` Georgy Yakovlev
  2017-08-19 22:50     ` Michał Górny
  0 siblings, 1 reply; 19+ messages in thread
From: Georgy Yakovlev @ 2017-08-19 22:25 UTC (permalink / raw)
  To: gentoo-dev

I've found couple of issues, or maybe not.

systemd installs to /usr/lib/systemd (or /lib/systemd since 234)
unconditionally.
I'm not sure if it's special and should be allowed to do that, but it's
the only package on the system (except gcc/$CHOST dir) that has 64-bit
libraries and binaries in lib.

Here is the list of the bugs I've found and opened, maybe you can add
them as blockers for #506276

https://bugs.gentoo.org/show_bug.cgi?id=627744
https://bugs.gentoo.org/show_bug.cgi?id=627746
https://bugs.gentoo.org/show_bug.cgi?id=628338



On 08/12/2017 02:33 PM, Michał Górny wrote:
> On śro, 2017-08-02 at 17:58 +0200, Michał Górny wrote:
>> Hi, everyone.
>>
>> I've finally gotten around to writing a new tool for migrating amd64
>> systems to SYMLINK_LIB=no layout [1]. I've put it in symlink-lib-
>> migration [2] repository along with a README. Please review it and give
>> it more testing.
> 
> I've pushed two important fixes now:
> 
> a. The tool now processes unowned files as well -- *.{a,la,so} are left
> in lib64 (i.e. the symlinks created by db.eclass and alikes) while
> everything else goes into lib.
> 
> b. I've fixed cleanup phase to also remove top-level files
> and directories that were moved out of lib64. Also, I've fixed it not to
> complain about trying to remove non-empty directories.
> 
> 
>> [1]:https://bugs.gentoo.org/show_bug.cgi?id=506276
>> [2]:https://github.com/mgorny/symlink-lib-migration
> 


^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [gentoo-dev] New SYMLINK_LIB=no migration tool for review
  2017-08-19 22:25   ` Georgy Yakovlev
@ 2017-08-19 22:50     ` Michał Górny
  0 siblings, 0 replies; 19+ messages in thread
From: Michał Górny @ 2017-08-19 22:50 UTC (permalink / raw)
  To: gentoo-dev

W dniu sob, 19.08.2017 o godzinie 15∶25 -0700, użytkownik Georgy
Yakovlev napisał:
> I've found couple of issues, or maybe not.
> 
> systemd installs to /usr/lib/systemd (or /lib/systemd since 234)
> unconditionally.
> I'm not sure if it's special and should be allowed to do that, but it's
> the only package on the system (except gcc/$CHOST dir) that has 64-bit
> libraries and binaries in lib.

Yes, this is valid. /usr/lib/foo is equivalent to /usr/libexec/foo
on modern systems, and some packages prefer the former over the latter.

> Here is the list of the bugs I've found and opened, maybe you can add
> them as blockers for #506276
> 
> https://bugs.gentoo.org/show_bug.cgi?id=627744
> https://bugs.gentoo.org/show_bug.cgi?id=627746

Assigned and blocked, thanks.

> https://bugs.gentoo.org/show_bug.cgi?id=628338
> 
> 
> 
> On 08/12/2017 02:33 PM, Michał Górny wrote:
> > On śro, 2017-08-02 at 17:58 +0200, Michał Górny wrote:
> > > Hi, everyone.
> > > 
> > > I've finally gotten around to writing a new tool for migrating amd64
> > > systems to SYMLINK_LIB=no layout [1]. I've put it in symlink-lib-
> > > migration [2] repository along with a README. Please review it and give
> > > it more testing.
> > 
> > I've pushed two important fixes now:
> > 
> > a. The tool now processes unowned files as well -- *.{a,la,so} are left
> > in lib64 (i.e. the symlinks created by db.eclass and alikes) while
> > everything else goes into lib.
> > 
> > b. I've fixed cleanup phase to also remove top-level files
> > and directories that were moved out of lib64. Also, I've fixed it not to
> > complain about trying to remove non-empty directories.
> > 
> > 
> > > [1]:https://bugs.gentoo.org/show_bug.cgi?id=506276
> > > [2]:https://github.com/mgorny/symlink-lib-migration
> 
> 

-- 
Best regards,
Michał Górny



^ permalink raw reply	[flat|nested] 19+ messages in thread

end of thread, other threads:[~2017-08-19 22:50 UTC | newest]

Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-08-02 15:58 [gentoo-dev] New SYMLINK_LIB=no migration tool for review Michał Górny
2017-08-02 17:51 ` [gentoo-dev] " Martin Vaeth
2017-08-02 18:10   ` Mike Gilbert
2017-08-02 19:07     ` Martin Vaeth
2017-08-02 19:25       ` Mike Gilbert
2017-08-03  1:02         ` Walter Dnes
2017-08-03  2:09           ` Benda Xu
2017-08-03  6:55           ` Michał Górny
2017-08-03  7:23         ` Martin Vaeth
2017-08-03  7:50         ` Martin Vaeth
2017-08-03  7:57           ` Michał Górny
2017-08-03 14:30             ` Martin Vaeth
2017-08-02 19:44       ` Michał Górny
2017-08-02 18:10   ` Holger Hoffstätte
2017-08-11 23:56 ` [gentoo-dev] " Gerogy Yakovlev
2017-08-12  7:12   ` Michał Górny
2017-08-12 21:33 ` Michał Górny
2017-08-19 22:25   ` Georgy Yakovlev
2017-08-19 22:50     ` Michał Górny

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox