[gentoo-dev] Making a common sub-profile for no-multilib

[gentoo-dev] Making a common sub-profile for no-multilib

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hey everyone -- so right now (and it's been this way for a very long
time, it seems), amd64 no-multilib profiles will always report repoman
'dependency.badindev' warnings whenever a package either depends on an
emul-* package or an abi_x86_32 use dependency.

At the moment there are two profiles in particular that do this,
amd64/no-multilib and hardened/linux/uclibc/amd64 ..  It's possible or
likely there are others, too, on other arches perhaps.

In general, it's good that repoman notices this because those profiles
don't support having the 32bit deps installed.  However, if one of
those profiles ever moves from a dev profile to a stable one (and
blueness mentioned he would like to make uclibc/amd64 stable), then
those dependency.badindev warnings will suddenly turn into
dependency.bad errors.

The solution to this would seem to be to package.mask all of these
32-bit packages in the pure 64bit profiles.  However, having to do
this in multiple locations is annoying.

I would like to propose adding 'no-multilib/[arch]/package.mask'
sub-profile(s), to allow all of these masks to go in one place.

Populating package.mask should be fairly easy for amd64 at least,
since anything depending on an app-emulation/emul-* will need to be
masked.  However the merits of where the package.mask will go needs
discussion.  Perhaps, for instance, it's time to adjust the profile
tree hierarchy more aggressively instead of "piling on" with yet
another subdir.

Thoughts?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iF4EAREIAAYFAlOrAHwACgkQ2ugaI38ACPA1lgD/eTAWJz5gBHAL49HFP9StYlcj
aWcBKp7I8/5yP5KxXcgA/3jcn2v/yNt3nhbNWvhRWvmj1FIau6kWnjTXNyS1uVYh
=9p7z
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Making a common sub-profile for no-multilib

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2648 bytes --]

Dnia 2014-06-25, o godz. 13:01:48
Ian Stakenvicius <axs@gentoo.org> napisał(a):

> At the moment there are two profiles in particular that do this,
> amd64/no-multilib and hardened/linux/uclibc/amd64 ..  It's possible or
> likely there are others, too, on other arches perhaps.
> 
> In general, it's good that repoman notices this because those profiles
> don't support having the 32bit deps installed.  However, if one of
> those profiles ever moves from a dev profile to a stable one (and
> blueness mentioned he would like to make uclibc/amd64 stable), then
> those dependency.badindev warnings will suddenly turn into
> dependency.bad errors.
> 
> The solution to this would seem to be to package.mask all of these
> 32-bit packages in the pure 64bit profiles.  However, having to do
> this in multiple locations is annoying.
> 
> I would like to propose adding 'no-multilib/[arch]/package.mask'
> sub-profile(s), to allow all of these masks to go in one place.
> 
> Populating package.mask should be fairly easy for amd64 at least,
> since anything depending on an app-emulation/emul-* will need to be
> masked.  However the merits of where the package.mask will go needs
> discussion.  Perhaps, for instance, it's time to adjust the profile
> tree hierarchy more aggressively instead of "piling on" with yet
> another subdir.

Forgive me for using such a words, but the profile tree is a well
stacked pile of crap. We have a dozen random profiles for each arch
then a dozen other profiles forked off them 'because the generic
arch profiles have crap' and a lot of profiles that inherit others
in a complex and semi-random manner.

For example, abi_x86_64 and abi_x86_32 each need to be forced in 4
profiles. This proves that we have 4 distinct profiles for amd64 which
need to be kept in sync. If I change something, I need to perform
the change in 4 different profiles and make sure I didn't screw
something up.

Then there's the x32 profile that inherits amd64 profile and therefore
requires reverting some of the stuff done on amd64. To do a complete
common change to x86-family multilib (like adding IUSE_IMPLICIT) I have
to modify 9 profiles.

Now let's add to this circa 8 mips profiles, around 3 ppc profiles, 4
arm profiles and some more. Every arch organized in somewhat different
and totally non-documented manner.

Long story short, doing anything to Gentoo profiles is utter pain
and comes with random breakage guarantee. Therefore, I'm asking -- nuke
those damn profiles, and start over! The current situation is
completely unmaintainable.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Making a common sub-profile for no-multilib

[Chris Reffett]

On June 25, 2014 2:44:57 PM EDT, "Michał Górny" <mgorny@gentoo.org> wrote:
>Dnia 2014-06-25, o godz. 13:01:48
>Ian Stakenvicius <axs@gentoo.org> napisał(a):
>
>> At the moment there are two profiles in particular that do this,
>> amd64/no-multilib and hardened/linux/uclibc/amd64 ..  It's possible
>or
>> likely there are others, too, on other arches perhaps.
>> 
>> In general, it's good that repoman notices this because those
>profiles
>> don't support having the 32bit deps installed.  However, if one of
>> those profiles ever moves from a dev profile to a stable one (and
>> blueness mentioned he would like to make uclibc/amd64 stable), then
>> those dependency.badindev warnings will suddenly turn into
>> dependency.bad errors.
>> 
>> The solution to this would seem to be to package.mask all of these
>> 32-bit packages in the pure 64bit profiles.  However, having to do
>> this in multiple locations is annoying.
>> 
>> I would like to propose adding 'no-multilib/[arch]/package.mask'
>> sub-profile(s), to allow all of these masks to go in one place.
>> 
>> Populating package.mask should be fairly easy for amd64 at least,
>> since anything depending on an app-emulation/emul-* will need to be
>> masked.  However the merits of where the package.mask will go needs
>> discussion.  Perhaps, for instance, it's time to adjust the profile
>> tree hierarchy more aggressively instead of "piling on" with yet
>> another subdir.
>
>Forgive me for using such a words, but the profile tree is a well
>stacked pile of crap. We have a dozen random profiles for each arch
>then a dozen other profiles forked off them 'because the generic
>arch profiles have crap' and a lot of profiles that inherit others
>in a complex and semi-random manner.
>
>For example, abi_x86_64 and abi_x86_32 each need to be forced in 4
>profiles. This proves that we have 4 distinct profiles for amd64 which
>need to be kept in sync. If I change something, I need to perform
>the change in 4 different profiles and make sure I didn't screw
>something up.
>
>Then there's the x32 profile that inherits amd64 profile and therefore
>requires reverting some of the stuff done on amd64. To do a complete
>common change to x86-family multilib (like adding IUSE_IMPLICIT) I have
>to modify 9 profiles.
>
>Now let's add to this circa 8 mips profiles, around 3 ppc profiles, 4
>arm profiles and some more. Every arch organized in somewhat different
>and totally non-documented manner.
>
>Long story short, doing anything to Gentoo profiles is utter pain
>and comes with random breakage guarantee. Therefore, I'm asking -- nuke
>those damn profiles, and start over! The current situation is
>completely unmaintainable.

+1, I'm all for replacing it with something more usable. I personally would favor the mix-in approach, but just about anything would be better than the weird setup we have right now.

Chris

Re: [gentoo-dev] Making a common sub-profile for no-multilib

[Anthony G. Basile]

On 06/25/14 15:00, Chris Reffett wrote:
>
> On June 25, 2014 2:44:57 PM EDT, "Michał Górny" <mgorny@gentoo.org> wrote:
>> Dnia 2014-06-25, o godz. 13:01:48
>> Ian Stakenvicius <axs@gentoo.org> napisał(a):
>>
>>> At the moment there are two profiles in particular that do this,
>>> amd64/no-multilib and hardened/linux/uclibc/amd64 ..  It's possible
>> or
>>> likely there are others, too, on other arches perhaps.
>>>
>>> In general, it's good that repoman notices this because those
>> profiles
>>> don't support having the 32bit deps installed.  However, if one of
>>> those profiles ever moves from a dev profile to a stable one (and
>>> blueness mentioned he would like to make uclibc/amd64 stable), then
>>> those dependency.badindev warnings will suddenly turn into
>>> dependency.bad errors.
>>>
>>> The solution to this would seem to be to package.mask all of these
>>> 32-bit packages in the pure 64bit profiles.  However, having to do
>>> this in multiple locations is annoying.
>>>
>>> I would like to propose adding 'no-multilib/[arch]/package.mask'
>>> sub-profile(s), to allow all of these masks to go in one place.
>>>
>>> Populating package.mask should be fairly easy for amd64 at least,
>>> since anything depending on an app-emulation/emul-* will need to be
>>> masked.  However the merits of where the package.mask will go needs
>>> discussion.  Perhaps, for instance, it's time to adjust the profile
>>> tree hierarchy more aggressively instead of "piling on" with yet
>>> another subdir.
>> Forgive me for using such a words, but the profile tree is a well
>> stacked pile of crap. We have a dozen random profiles for each arch
>> then a dozen other profiles forked off them 'because the generic
>> arch profiles have crap' and a lot of profiles that inherit others
>> in a complex and semi-random manner.
>>
>> For example, abi_x86_64 and abi_x86_32 each need to be forced in 4
>> profiles. This proves that we have 4 distinct profiles for amd64 which
>> need to be kept in sync. If I change something, I need to perform
>> the change in 4 different profiles and make sure I didn't screw
>> something up.
>>
>> Then there's the x32 profile that inherits amd64 profile and therefore
>> requires reverting some of the stuff done on amd64. To do a complete
>> common change to x86-family multilib (like adding IUSE_IMPLICIT) I have
>> to modify 9 profiles.
>>
>> Now let's add to this circa 8 mips profiles, around 3 ppc profiles, 4
>> arm profiles and some more. Every arch organized in somewhat different
>> and totally non-documented manner.
>>
>> Long story short, doing anything to Gentoo profiles is utter pain
>> and comes with random breakage guarantee. Therefore, I'm asking -- nuke
>> those damn profiles, and start over! The current situation is
>> completely unmaintainable.
> +1, I'm all for replacing it with something more usable. I personally would favor the mix-in approach, but just about anything would be better than the weird setup we have right now.
>
> Chris
>
The profiles have caused us no end of suffering in hardened.  The 
hardened/linux/uclibc profiles are my attempt to avoid the "well stacked 
pile of crap" without creating more "crap", although that's debatable 
;)  The problem is the way stacking works.  You don't have control over 
the inheritance and so wind up turn things on, then reverting, then 
turning them on again on in the next layer etc.  We had luck 
disentangling selinux because it was orthogonal to the rest of hardened, 
but not so much luck when it came to the hardened/desktop subprofiles.

I've been trying to keep up with the multilib stuff for uclibc, so don't 
fret too much about those profiles, although any help is appreciated 
like repoman -d warnings.  I'd worry more about the rest of them.

However, in the long run, before we nuke them all and start over (and 
loose a lot of memory in the process) we may want to look into designs 
where we can control the inheritance better via portage. More syntax in 
the parent file may help here.

The issue has vexed me enough that I blogged about it: 
http://blogs.gentoo.org/blueness/2014/02/07/the-gentoo-profile-stacking-problem/

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] Making a common sub-profile for no-multilib

[Rich Freeman]

On Wed, Jun 25, 2014 at 2:44 PM, Michał Górny <mgorny@gentoo.org> wrote:
> Long story short, doing anything to Gentoo profiles is utter pain
> and comes with random breakage guarantee. Therefore, I'm asking -- nuke
> those damn profiles, and start over! The current situation is
> completely unmaintainable.

++

But, would it make sense to just go the Funtoo route with "mix-ins."
Basically instead of symlinking make.profile to a profile directory
they turn make.profile into an actual profile which just has a parent
that inherits all the real profiles, and eselect is designed to handle
it.

That would get rid of the 47 permutations in our current tree.  It
would also reduce the barrier to adding more options, since it means
adding one subdir that only affects those who use/maintain it, and not
23 combinations that drive everybody mad.

Rich

Re: Re: [gentoo-dev] Making a common sub-profile for no-multilib

[Andreas K. Huettel]

Am Mittwoch 25 Juni 2014, 15:11:40 schrieb Rich Freeman:
> On Wed, Jun 25, 2014 at 2:44 PM, Michał Górny <mgorny@gentoo.org> wrote:
> > Long story short, doing anything to Gentoo profiles is utter pain
> > and comes with random breakage guarantee. Therefore, I'm asking -- nuke
> > those damn profiles, and start over! The current situation is
> > completely unmaintainable.
> 
> ++
> 
> But, would it make sense to just go the Funtoo route with "mix-ins."
> Basically instead of symlinking make.profile to a profile directory
> they turn make.profile into an actual profile which just has a parent
> that inherits all the real profiles, and eselect is designed to handle
> it.
> 
> That would get rid of the 47 permutations in our current tree.  It
> would also reduce the barrier to adding more options, since it means
> adding one subdir that only affects those who use/maintain it, and not
> 23 combinations that drive everybody mad.

++

this is what we've been just discussing on the irc channel

-- 
Andreas K. Huettel
Gentoo Linux developer
kde, council

Re: Re: [gentoo-dev] Making a common sub-profile for no-multilib

[Rich Freeman]

On Wed, Jun 25, 2014 at 4:01 PM, Andreas K. Huettel
<dilfridge@gentoo.org> wrote:
> Am Mittwoch 25 Juni 2014, 15:11:40 schrieb Rich Freeman:
>> On Wed, Jun 25, 2014 at 2:44 PM, Michał Górny <mgorny@gentoo.org> wrote:
>> > Long story short, doing anything to Gentoo profiles is utter pain
>> > and comes with random breakage guarantee. Therefore, I'm asking -- nuke
>> > those damn profiles, and start over! The current situation is
>> > completely unmaintainable.
>>
>> ++
>>
>> But, would it make sense to just go the Funtoo route with "mix-ins."
> ++
>
> this is what we've been just discussing on the irc channel

So, not wanting this to die on the vine.

If we did the mix-in approach, would we just follow the example of Funtoo?

They use an arch profile, a stability profile (~arch vs arch), a
"flavor" profile (core, minimal, desktop), and then users can layer as
much other stuff on top of that as they want (gnome, kde, multimedia,
etc).

Do we want to do things the same way?

Some things to think about include multilib (just another arch?),
systemd, and usr-merge.  I'm not saying that we need to implement any
of that stuff completely - but when planning the profile layout we
should at least consider whether it will handle things like this in
the future.  Should some types of profiles be only additive?  Etc...

Rich

[gentoo-dev] Re: Making a common sub-profile for no-multilib

[Duncan]

Rich Freeman posted on Wed, 02 Jul 2014 09:30:50 -0400 as excerpted:

> Some things to think about include multilib (just another arch?),
> systemd,
> and usr-merge.  I'm not saying that we need to implement any of that
> stuff completely - but when planning the profile layout we should at
> least consider whether it will handle things like this in the future. 
> Should some types of profiles be only additive?  Etc...

FWIW, systemd with merge to / (/usr being a symlink to . and sbin to bin) 
here.  In particular, the merge "just works" with current portage.  I'm 
no-multilib.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[gentoo-dev] Re: Making a common sub-profile for no-multilib

[Jonathan Callen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 07/02/2014 02:14 PM, Duncan wrote:
> Rich Freeman posted on Wed, 02 Jul 2014 09:30:50 -0400 as
> excerpted:
> 
>> Some things to think about include multilib (just another
>> arch?), systemd, and usr-merge.  I'm not saying that we need to
>> implement any of that stuff completely - but when planning the
>> profile layout we should at least consider whether it will handle
>> things like this in the future. Should some types of profiles be
>> only additive?  Etc...
> 
> FWIW, systemd with merge to / (/usr being a symlink to . and sbin
> to bin) here.  In particular, the merge "just works" with current
> portage.  I'm no-multilib.
> 

That merge only works because you happened to get lucky, and have
portage merge coreutils in a particular manner (/usr/bin/uname
installed before /bin/uname).  If portage had installed /bin/uname
first, you would have ended up with a /bin/uname symlink pointing to
/bin/uname.  The same is also true of basename, chroot, cut, dir,
dirname, du, env, expr, head, mkfifo, mktemp, readlink, seq, sleep,
sort, tail, touch, tr, tty, vdir, wc, and yes.

I myself am currently running a system with the opposite merge (/bin,
/lib, /lib64, and /sbin are symlinks to /usr/bin, /usr/lib,
/usr/lib64, and /usr/sbin) although I do not yet have the sbin -> bin
merge yet.  I added a post_src_install and post_pkg_preinst hook in my
/etc/portage/bashrc to ensure that there are no conflicts before the
package is merged and a pre_pkg_setup hook to disarm gen_usr_ldscript
(so as not to create a conflict).

The only two packages I have installed that I had to modify to make
this work were coreutils (as noted above) and plymouth.

- -- 
Jonathan Callen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCgAGBQJTtJCTAAoJELHSF2kinlg41W0P/19TAJErj3y7eMFGJG2BMVIM
TtAj2wM+whB0Md1+TGypFS0ESO+hJAxNyyuIT215afRIqZJgYjZG5NZCnmygI71t
9das1k2JHn+PRO5KSZcw3Z3VcpspzADTwR+avaxiqRuslzyTvAYrsPj+7oQd6iDp
4XzDKFSEh2hNJHUQUQ3eT5NxlJNQu9uxKLc4aM+1/GImlSAVbbiKHUHWy3njSVWN
twFkQJ59IHod9aZgn0txydd9hhMr43Et6uDJywjGuIeVQncAO/eBvT9hRE5tB3aN
x/pd1Dcf0V5FTN/459kcoRwTBQ2k3quhGJBeSSGLUwNlTjdOWDbqU5HcgOf0d7v4
NXMIOr95ung3LgeqCyZU5S4XTyoh9w/nZRXuSTYL0sQL3IxffLRkVPcvis02cY6a
jZaSAwkVkPhyAGkV0QxMsfhEP9+2wdtB7JMc28kslO0kGpdV5Oa8ES+QqnwVXgi7
umB+5D4IjTDK6uPcM5K5p+8wKWcsFMvn5X/+3Fh49S6KDiWZRsjq45pHC7+5lC0p
S8EJb8XybieRvQsVGo8NfgucvRQFbeUn+BaUfztoNRvKG3NxsnxRJ+JycK9bi1pj
znu7YKmCZMmt8PurliQPKQ6vtPjFQK19gZZf72zTVBycc2hdaKnbowAV0rBgTETN
+8xryIRBNfISkUIjEip/
=Muzz
-----END PGP SIGNATURE-----

[gentoo-dev] Re: Making a common sub-profile for no-multilib

[Duncan]

Jonathan Callen posted on Wed, 02 Jul 2014 19:06:59 -0400 as excerpted:

> On 07/02/2014 02:14 PM, Duncan wrote:
>> Rich Freeman posted on Wed, 02 Jul 2014 09:30:50 -0400 as excerpted:
>> 
>>> Some things to think about include multilib (just another arch?),
>>> systemd, and usr-merge.
>> 
>> FWIW, systemd with merge to / (/usr being a symlink to . and sbin to
>> bin) here.  In particular, the merge "just works" with current portage.
>>  I'm no-multilib.
>> 
> That merge only works because you happened to get lucky, and have
> portage merge coreutils in a particular manner (/usr/bin/uname installed
> before /bin/uname).  If portage had installed /bin/uname first, you
> would have ended up with a /bin/uname symlink pointing to /bin/uname. 
> The same is also true of basename, chroot, cut, dir, dirname, du, env,
> expr, head, mkfifo, mktemp, readlink, seq, sleep, sort, tail, touch, tr,
> tty, vdir, wc, and yes.

No, it works because, as I specifically asked the portage folks about 
before I tried it, portage has had symlink merge intelligence for some 
time now.  Apparently well before I asked (in May, 2013, according to my 
portage-devel list archives), some portage dev considered the possibility 
of directory symlinks triggering overwrites of targets with the symlinks 
pointing to them, and ensured that portage's qmerge code "just did the 
right thing."

I did nothing with the answer for some time, but eventually decided to 
try it.  As I was a bit skeptical/careful at first, after doing the 
initial manual moves and thus finding which files existed in both target 
dirs, then deleting the individual file symlinks, moving the remaining 
files to the new location and making the old directory a symlink, I 
equery-belonged the symlinks and manually remerged (from binpkg, which 
still stores the symlinks in their usual location in the binpkg tarball) 
several of the packages one at a time, first a non-critical one, then I 
think coreutils itself, just to be sure, then 2-3 others together, and 
sure enough, it "just worked" the way it was supposed to work.

=:^)

> I myself am currently running a system with the opposite merge (/bin,
> /lib, /lib64, and /sbin are symlinks to /usr/bin, /usr/lib, /usr/lib64,
> and /usr/sbin) although I do not yet have the sbin -> bin merge yet.

FWIW I decided the single /usr -> . symlink was simpler, and besides, I 
liked the shorter direct paths. =:^)  And I did the /usr merge first, 
thinking I wanted to keep the bin/sbin distinction at first, until 
sometime later I decided to simplify my PATH var to remove /usr, and 
realized I had sbin in my normal user's PATH too.  Completing the merge 
would/did allow me to simplify PATH even further, and since I has sbin in 
my normal user's PATH, there really wasn't a reason to keep them separate 
at that point, and I was already comfortable with merged bins by then 
anyway, so it wasn't much of a stretch at all.

> I
> added a post_src_install and post_pkg_preinst hook in my
> /etc/portage/bashrc to ensure that there are no conflicts before the
> package is merged and a pre_pkg_setup hook to disarm gen_usr_ldscript
> (so as not to create a conflict).

That's all unnecessary, because as I said, in general portage "just 
works" with the merge now, since it's designed to work with pretty much 
/any/ strange site-specific symlinking local gentoo admins might throw at 
it, altho I expect the ldcache stuff is still doing a bunch of extra work 
by going over all the dirs that are actually the same thing, and I 
suspect revdep-rebuild (which I still run religiously after every @world 
update) is doing a bunch of extra scanning too.  But as I'm on SSD the 
extra filesystem IO isn't a big issue, meaning it's simply the CPU cycle 
cost, and so far simply spending the extra CPU cycles has been cheaper 
for me than actually researching what I might do about it, so...

> The only two packages I have installed that I had to modify to make this
> work were coreutils (as noted above) and plymouth.

I did find one package with a specific post-install quirk, that was there 
to work around a binary move from many years ago, that ended up 
specifically removing the binary after portage had done its thing and 
actually put the binary in place instead of the symlink, but that binary 
wasn't a system-critical binary (dircolors, FWIW, pkg=coreutils), and 
upon investigation and filing a bug suggesting to test that it was a 
symlink, not the actual binary removed, the maintainer (vapier) decided 
that the reason that particular post-install quirk was there was far 
enough back in history he could just remove it from the ebuild.  I didn't 
check whether he removed it from stable, but at least leading ~arch 
coreutils doesn't have that quirk any longer, and leaves dircolors alone.

FWIW, https://bugs.gentoo.org/show_bug.cgi?id=509596


The only other relatively minor irritation that remains is a portage bug, 
but as I said it's relatively minor as it doesn't affect functionality.

But what it /does/ mean is that for nearly EVERY package upgrade, I have 
an identical message complaining about /usr being a possibly stale 
symlink, with no easy/apparent way to quite the warning using 
UNINSTALL_IGNORE, as I can and have with other symlinks such as
/var/run -> run, etc.

When it stops being a relatively minor irritation and gets big enough to 
really annoy me (if simply mentioning it here doesn't get it fixed before 
I actually get to that point), I'll file a bug on it and I expect that 
annoyance will go away as well. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[gentoo-dev] Re: Making a common sub-profile for no-multilib

[Jonathan Callen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 07/03/2014 12:59 AM, Duncan wrote:
> Jonathan Callen posted on Wed, 02 Jul 2014 19:06:59 -0400 as
> excerpted:
> 
>> On 07/02/2014 02:14 PM, Duncan wrote:
>>> Rich Freeman posted on Wed, 02 Jul 2014 09:30:50 -0400 as
>>> excerpted:
>>> 
>>>> Some things to think about include multilib (just another
>>>> arch?), systemd, and usr-merge.
>>> 
>>> FWIW, systemd with merge to / (/usr being a symlink to . and
>>> sbin to bin) here.  In particular, the merge "just works" with
>>> current portage. I'm no-multilib.
>>> 
>> That merge only works because you happened to get lucky, and
>> have portage merge coreutils in a particular manner
>> (/usr/bin/uname installed before /bin/uname).  If portage had
>> installed /bin/uname first, you would have ended up with a
>> /bin/uname symlink pointing to /bin/uname. The same is also true
>> of basename, chroot, cut, dir, dirname, du, env, expr, head,
>> mkfifo, mktemp, readlink, seq, sleep, sort, tail, touch, tr, tty,
>> vdir, wc, and yes.
> 
> No, it works because, as I specifically asked the portage folks
> about before I tried it, portage has had symlink merge intelligence
> for some time now.  Apparently well before I asked (in May, 2013,
> according to my portage-devel list archives), some portage dev
> considered the possibility of directory symlinks triggering
> overwrites of targets with the symlinks pointing to them, and
> ensured that portage's qmerge code "just did the right thing."
> 
> I did nothing with the answer for some time, but eventually decided
> to try it.  As I was a bit skeptical/careful at first, after doing
> the initial manual moves and thus finding which files existed in
> both target dirs, then deleting the individual file symlinks,
> moving the remaining files to the new location and making the old
> directory a symlink, I equery-belonged the symlinks and manually
> remerged (from binpkg, which still stores the symlinks in their
> usual location in the binpkg tarball) several of the packages one
> at a time, first a non-critical one, then I think coreutils itself,
> just to be sure, then 2-3 others together, and sure enough, it
> "just worked" the way it was supposed to work.
> 
> =:^)
> 
>> I myself am currently running a system with the opposite merge
>> (/bin, /lib, /lib64, and /sbin are symlinks to /usr/bin,
>> /usr/lib, /usr/lib64, and /usr/sbin) although I do not yet have
>> the sbin -> bin merge yet.
> 
> FWIW I decided the single /usr -> . symlink was simpler, and
> besides, I liked the shorter direct paths. =:^)  And I did the /usr
> merge first, thinking I wanted to keep the bin/sbin distinction at
> first, until sometime later I decided to simplify my PATH var to
> remove /usr, and realized I had sbin in my normal user's PATH too.
> Completing the merge would/did allow me to simplify PATH even
> further, and since I has sbin in my normal user's PATH, there
> really wasn't a reason to keep them separate at that point, and I
> was already comfortable with merged bins by then anyway, so it
> wasn't much of a stretch at all.
> 
>> I added a post_src_install and post_pkg_preinst hook in my 
>> /etc/portage/bashrc to ensure that there are no conflicts before
>> the package is merged and a pre_pkg_setup hook to disarm
>> gen_usr_ldscript (so as not to create a conflict).
> 
> That's all unnecessary, because as I said, in general portage "just
>  works" with the merge now, since it's designed to work with pretty
> much /any/ strange site-specific symlinking local gentoo admins
> might throw at it, altho I expect the ldcache stuff is still doing
> a bunch of extra work by going over all the dirs that are actually
> the same thing, and I suspect revdep-rebuild (which I still run
> religiously after every @world update) is doing a bunch of extra
> scanning too.  But as I'm on SSD the extra filesystem IO isn't a
> big issue, meaning it's simply the CPU cycle cost, and so far
> simply spending the extra CPU cycles has been cheaper for me than
> actually researching what I might do about it, so...
> 

That is good to know.  I did, however test when a package installs two
(different) regular files into paths that end up symlinked, and found
that portage does break in that case (as the only sensible option at
that point is to fail, as something will be lost in either case).

- -- 
Jonathan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCgAGBQJTtQDKAAoJELHSF2kinlg4eIIP/0Jmia/151IPAAt3oG7OalVr
1SyUSdAchM/cCb6m8uf3BDPIxMQa704Zh/TKd3wAxz64BJGmJtDmEBlAq+04nXnw
UFB+dJVJsphN/X/jS7iln1tVqn8099mntYWG9mjv4nQLXNf6U0H1i9iBQtz+uvi5
9tzO6Ghoen6VvyV7ykI/r2uByI4Y4+anVgXKT2s99akbIWFR9qQuHOHWlcxZxurd
QA17Wfnxene7FfpGtsanORvpiKaQamUo75zXlR7l5FydiANH5QKt1Qw5RZIC/k27
PVc+oV2A+7HJVWHrMqEBwwGyn8LokzzX644TpiL17rTHkGa9jYYnsfwO/mIksEh2
O7HzbHyGjD6yUVbY/G3bxUfjpEi0C02DASSlOrHnDsZf4U5joYf9D//jTFQhCkp8
TtPcU7h4HIlyniberknM/WMqQ8B4lnjlCGNcVoaZtqabQ6YtIV+9eu32Gzd0LP29
TN/MQFm98pRGTwAAoyoq8QzOUUpoEBZP05htsW5UvgSe5CAzfi2Tm2TEsF/ulbMs
IoZBdovmv1mpTnNESJ6rTos9QesAbErV0FQn6NC5WcvtWpTrpiRFf3QeqOCcgpZY
ogH241MJaaxnu3vUdKnCD5zd4UBJOjWaViLKIK29v3pWVAgbMeHhC0inNW6gIZIu
Nn8aKkjoyjyEhyJT9/tj
=3Nrz
-----END PGP SIGNATURE-----

[gentoo-dev] Re: Making a common sub-profile for no-multilib

[Duncan]

Jonathan Callen posted on Thu, 03 Jul 2014 03:05:46 -0400 as excerpted:

> I did, however test when a package installs two (different) regular
> files into paths that end up symlinked, and found that portage does
> break in that case (as the only sensible option at that point is to
> fail, as something will be lost in either case).

Indeed, and good point.

I guess at that point it's basically a pkg-collision, except that it's in 
a single package instead of two different packages.  Too bad a package 
can't block itself and thus handle it the way different packages blocking 
each other handle that case! =;^)

That's why symlinking both lib64 and lib32 to lib isn't a particularly 
good idea! =;^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Making a common sub-profile for no-multilib

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1068 bytes --]

Dnia 2014-07-03, o godz. 09:05:47
Duncan <1i5t5.duncan@cox.net> napisał(a):

> Jonathan Callen posted on Thu, 03 Jul 2014 03:05:46 -0400 as excerpted:
> 
> > I did, however test when a package installs two (different) regular
> > files into paths that end up symlinked, and found that portage does
> > break in that case (as the only sensible option at that point is to
> > fail, as something will be lost in either case).
> 
> Indeed, and good point.
> 
> I guess at that point it's basically a pkg-collision, except that it's in 
> a single package instead of two different packages.  Too bad a package 
> can't block itself and thus handle it the way different packages blocking 
> each other handle that case! =;^)
> 
> That's why symlinking both lib64 and lib32 to lib isn't a particularly 
> good idea! =;^)

Symlinking anything to lib wasn't ever a particularly good idea. We
will spend a lot of effort fixing that (see the SYMLINK_LIB=no bug [1]).

[1]:https://bugs.gentoo.org/show_bug.cgi?id=506276

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Making a common sub-profile for no-multilib

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 3383 bytes --]

Dnia 2014-06-25, o godz. 13:01:48
Ian Stakenvicius <axs@gentoo.org> napisał(a):

> I would like to propose adding 'no-multilib/[arch]/package.mask'
> sub-profile(s), to allow all of these masks to go in one place.
> 
> Populating package.mask should be fairly easy for amd64 at least,
> since anything depending on an app-emulation/emul-* will need to be
> masked.  However the merits of where the package.mask will go needs
> discussion.  Perhaps, for instance, it's time to adjust the profile
> tree hierarchy more aggressively instead of "piling on" with yet
> another subdir.
> 
> Thoughts?

I would go for a bit different way of handling arch profiles. This is
an early idea that probably can't work but could make things a little
bit easier to mangle.

The arch/ tree starts with 'generic' subdirectories matching main
arches -- like arm, mips, x86, sparc, powerpc, s390 (but not amd64).

Each of arch trees contains an 'abis' subtree that contains mix-ins
describing particular ABIs supported. For example, arch/x86/abis/x86,
arch/x86/abis/amd64, arch/mips/abis/o32.

Each of those mix-ins describes that basic stuff for ABI (like LIBDIR,
standard CHOST), unmasks relevant ABI flags and p.masks them on
packages that don't support a particular ABI.

Now, each of those mix-ins may come with a sub-profile called 'default'
that sets use.force & make.defaults for making the ABI the default ABI.
I'm currently not sure if this will be really helpful since the small
amount of work we may also put directly into 'real' profiles (described
below).

Moreover, each of those mix-ins may come with a 'no-multilib'
sub-profile. This one -- aside from setting all the standard variables
-- package.masks all the packages that were package.use.mask in parent
profile. This way, we can keep all the masks almost in a single place.

Then, we have multilib profiles like arch/x86/multilib/amd64,
arch/x86/multilib/x32. Those inherit from all the relevant ABI profiles
-- e.g. amd64 would inherit arch/x86/abis/x86 &
arch/x86/abis/amd64[/default], and set the remaining variables for
multilib.


While I feel it's a bit complex, I think that's somehow a sane way to
express what we have now without moving back and forth. Few extra key
points:

- minimal no of profiles in each 'parent' -- supposedly abis/ would
  have no parents, and possibly final profiles would have 'arch/base'
  as parent,

- as already mentioned somewhere else, i'd rather see 'arch/base'
  instead of plain 'base' -- the idea is to put everything that's
  easier reverted than copied through all arch/ profiles, and have it
  inherited only by the final arch profiles.


For example, the expanded inherits for arch/x86/multilib/amd64 would
go like:

1. arch/base -- that disables a lot of uncommon stuff,

2. arch/x86/base [optionally] -- setting some generic defaults,

3. arch/x86/abis/x86 -- setting support for 'x86' ABI,

4. arch/x86/abis/x86/lib32 [optionally] -- overriding LIBDIR_x86 for
compatibility with current SYMLINK_LIB screwup,

5. arch/x86/abis/amd64 -- setting support for 'amd64' ABI,

6. arch/x86/abis/amd64/default [optionally] -- setting 'amd64'
as default ABI,

7. arch/x86/multilib/amd64 -- finishing multilib setup.

The key point here being that no profile is run twice.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Making a common sub-profile for no-multilib

[Peter Stuge]

[-- Attachment #1: Type: text/plain, Size: 1221 bytes --]

Michał Górny wrote:
> The arch/ tree starts with 'generic' subdirectories matching main
> arches -- like arm, mips, x86, sparc, powerpc, s390 (but not amd64).

I like this idea, but..


> Each of arch trees contains an 'abis' subtree that contains mix-ins

..please please call this 'abi' instead.


> For example, the expanded inherits for arch/x86/multilib/amd64 would
> go like:
> 
> 1. arch/base -- that disables a lot of uncommon stuff,
> 2. arch/x86/base [optionally] -- setting some generic defaults,

Fine so far.

> 3. arch/x86/abis/x86 -- setting support for 'x86' ABI,
> 4. arch/x86/abis/x86/lib32 [optionally] -- overriding LIBDIR_x86 for
> compatibility with current SYMLINK_LIB screwup,

This can't work; abi/x86 can't be both a file and a subdir.
Maybe call them abi/x86 and abi/x86_SYMLINK_LIB_compatibility ?
(Or x86_lib32, although that is a lot less descriptive.)


> 5. arch/x86/abis/amd64 -- setting support for 'amd64' ABI,
> 6. arch/x86/abis/amd64/default [optionally] -- setting 'amd64'
> as default ABI,

Same here with abi/amd64 being both file and subdir.


> 7. arch/x86/multilib/amd64 -- finishing multilib setup.

I think it looks good.


//Peter

[-- Attachment #2: Type: application/pgp-signature, Size: 190 bytes --]

Re: [gentoo-dev] Making a common sub-profile for no-multilib

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1110 bytes --]

Dnia 2014-07-03, o godz. 14:07:28
Peter Stuge <peter@stuge.se> napisał(a):

> Michał Górny wrote:
> > The arch/ tree starts with 'generic' subdirectories matching main
> > arches -- like arm, mips, x86, sparc, powerpc, s390 (but not amd64).
> 
> I like this idea, but..
> 
> 
> > Each of arch trees contains an 'abis' subtree that contains mix-ins
> 
> ..please please call this 'abi' instead.
> 
> 
> > For example, the expanded inherits for arch/x86/multilib/amd64 would
> > go like:
> > 
> > 1. arch/base -- that disables a lot of uncommon stuff,
> > 2. arch/x86/base [optionally] -- setting some generic defaults,
> 
> Fine so far.
> 
> > 3. arch/x86/abis/x86 -- setting support for 'x86' ABI,
> > 4. arch/x86/abis/x86/lib32 [optionally] -- overriding LIBDIR_x86 for
> > compatibility with current SYMLINK_LIB screwup,
> 
> This can't work; abi/x86 can't be both a file and a subdir.
> Maybe call them abi/x86 and abi/x86_SYMLINK_LIB_compatibility ?
> (Or x86_lib32, although that is a lot less descriptive.)

Profiles are directories...

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Making a common sub-profile for no-multilib

[Peter Stuge]

[-- Attachment #1: Type: text/plain, Size: 192 bytes --]

Michał Górny wrote:
> > abi/x86 can't be both a file and a subdir.
> 
> Profiles are directories...

Doh, yes of course. :\

abi rather than abis is still important. :)


//Peter

[-- Attachment #2: Type: application/pgp-signature, Size: 190 bytes --]

Re: [gentoo-dev] Making a common sub-profile for no-multilib

[Rich Freeman]

On Thu, Jul 3, 2014 at 8:07 AM, Peter Stuge <peter@stuge.se> wrote:
> This can't work; abi/x86 can't be both a file and a subdir.

Profiles ARE subdirs.  Most of our existing profiles are stacked in
this way.  This would become less-so if we went with the mix-in
approach, but it wouldn't be entirely flat in this proposal.

Rich