[gentoo-dev] RFC: multilib and the compatibility to singlelib

[gentoo-dev] RFC: multilib and the compatibility to singlelib

[Michael Haubenwallner]

Hi devs,

while there is the appreciated multiple ABI portage support going on,
a thought on the intentions of the multilib profiles.

Some background:
I do have to support building an older, but still maintained large
application software, that simply does not work when built as 64bit.
As it does not have the need itself for >32bit pointers, and all
the 64bit target platforms do support 32bit binaries too, the plan
to fix the code for 64bits has been cancelled.

I'm using gcc to build the application on all the target unices,
as well as autotools for build management.

As I'm building the toolchain myself too, I configure it with the
32bit host triplet on each platform, usually disabling multilib.

This simply works for ppc-aix, hppa-hpux, ia64-hpux, sparc-solaris,
x86-solaris, even if the underlying OS/kernel is 64bit.
It isn't even necessary to explicitly use --{build,host} at all,
as config.guess tells the 32bit triplet on these platforms.
Additionally, even their default compiler output is 32bit.

But Linux "is not Unix":
Configuring both binutils and gcc needs to be done with:
  $ CC="gcc -m32" /path/to/configure --{build,host}=i686-pc-linux-gnu ...
This works on 64bit RHEL as well as on 64bit SLES (both have 32bit in /lib,
64bit in /lib64, no /lib32), but breaks on amd64+multilib Gentoo Linux.

Problem is that, as x86-linux isn't a multilib target, both ld and gcc
search for 32bit libs&objects in /usr/lib:/lib, as they don't know about
/usr/lib32:/lib32. The error when bootstrapping gcc is:
  /usr/lib/crti.o: file not recognized: File format not recognized

While it is possible to patch binutils[1] and gcc[2] to search in
/usr/lib32:/lib32 even with this otherways non-multilib target,
it doesn't feel like the "right thing" to enable multilib for
just one multilib option.

Isn't the intention of multilib to have a new (64bit) system
be compatible with the corresponding old (32bit) system?

Please comment, thank you!
/haubi/

[1] http://overlays.gentoo.org/proj/alt/changeset/51324
[2] http://overlays.gentoo.org/proj/alt/changeset/51325

[gentoo-dev] Re: RFC: multilib and the compatibility to singlelib

[Nikos Chantziaras]

On 10/20/2009 04:06 PM, Michael Haubenwallner wrote:
>
> Isn't the intention of multilib to have a new (64bit) system
> be compatible with the corresponding old (32bit) system?

I'm not sure I understand the whole procedure you use to build this app. 
  Why not simply use -m32 when building it?  Why build a whole toolchain?

Anyway, as for the question of multilib's intention, yes, it's to have a 
64-bit and 32-bit userland at the same time.  But it doesn't work fully 
on Gentoo, which is why the effort for real multilib support has started 
in the first place.

Re: [gentoo-dev] RFC: multilib and the compatibility to singlelib

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 797 bytes --]

Michael Haubenwallner schrieb:
> Isn't the intention of multilib to have a new (64bit) system
> be compatible with the corresponding old (32bit) system?
> 
> Please comment, thank you!
> /haubi/

If you have a 64bit system, the default should be 64bit, both for libs and for binaries. The
additional multilib support allows to install and use additional 32bit libs and binaries. Since they
are not the system dafault, they shouldnt be in some default place like lib, but instead a
different, but clearly labeled dir like lib32. So the Gentoo way looks like the right way to me.

And better support for this part is my current multilib-portage project, which allows compilation
and installation of additional 32bit libs and binaries.

-- 
Thomas Sachau

Gentoo Linux Developer


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

[gentoo-dev] Re: RFC: multilib and the compatibility to singlelib

[Duncan]

Thomas Sachau posted on Tue, 20 Oct 2009 17:29:25 +0200 as excerpted:

> Michael Haubenwallner schrieb:
>> Isn't the intention of multilib to have a new (64bit) system be
>> compatible with the corresponding old (32bit) system?
>> 
>> Please comment, thank you!
>> /haubi/
> 
> If you have a 64bit system, the default should be 64bit, both for libs
> and for binaries. The additional multilib support allows to install and
> use additional 32bit libs and binaries. Since they are not the system
> dafault, they shouldnt be in some default place like lib, but instead a
> different, but clearly labeled dir like lib32. So the Gentoo way looks
> like the right way to me.

Except... it isn't, at least not if Gentoo is at all concerned about 
standards, especially when they'll make things far easier for it.

What you just described was the logic applied with, for example, ia64, 
which is true 64-bit (only) native.  However (as I understand it), the 
Linux FHS and LSB used somewhat different logic for x86_64.

See, x86_64 is hardware native dual-bitness, so both 32-bit and 64-bit 
can be said to be true native hardware bitness (this is NOT the case with 
ia64).  Apparently due to that and to the vast number of legacy 32-bit 
apps around, many binary-only apps doing obscure and unpredictable things 
that could well break if assumptions about lib proved invalid, they 
decided to keep the 32-bit lib location just as it was, believing it 
easier to create the new lib64 for 64-bit, then to worry about whatever 
obscure and exotic stuff various binary-only apps might be doing. Since 
this was just one more change to add to the list of changes already being 
made to port apps to amd64, it was considered easier than the other way, 
and thus became the standard.

The problem was that Gentoo's early amd64 implementation predated this 
standardization, and we had chosen the other way.  While we've defaulted 
to lib64 for 64-bit libs for years, it has never been considered anything 
but experimental to break the lib --> lib64 link.  AFAIK stable 
baselayout still doesn't get its libdir usage consistent, putting files 
in one but actually calling them using the other path, and boot breaks in 
various frustrating ways if lib and lib64 are not the same directory.  
Openrc gets it better now, but I'm not sure it's all fixed either -- it 
certainly wasn't last time I tried breaking the link.

So before Gentoo can switch to the FHS/LSB 32-bit lib on amd64 multilib, 
it must first fix those last inconsistent usages, and make it possible to 
break the lib --> lib64 link.  Then in theory at least, after awhile with 
no bugs, or at least no big ones related to this issue, it might be 
considered safe to move 32-bit back to lib, where the LSB/FHS says it 
should be.

> And better support for this part is my current multilib-portage project,
> which allows compilation and installation of additional 32bit libs and
> binaries.

While I'm a no-multilib user here, I'm glad someone's finally getting 
that close enough to mainline merge to talk about it!  There's a lot of 
folks for whom it will make life far easier. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: RFC: multilib and the compatibility to singlelib

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 3824 bytes --]

On Tuesday 20 October 2009 12:25:15 Duncan wrote:
> Thomas Sachau posted on Tue, 20 Oct 2009 17:29:25 +0200 as excerpted:
> > Michael Haubenwallner schrieb:
> >> Isn't the intention of multilib to have a new (64bit) system be
> >> compatible with the corresponding old (32bit) system?
> >>
> >> Please comment, thank you!
> >> /haubi/
> >
> > If you have a 64bit system, the default should be 64bit, both for libs
> > and for binaries. The additional multilib support allows to install and
> > use additional 32bit libs and binaries. Since they are not the system
> > dafault, they shouldnt be in some default place like lib, but instead a
> > different, but clearly labeled dir like lib32. So the Gentoo way looks
> > like the right way to me.
> 
> Except... it isn't, at least not if Gentoo is at all concerned about
> standards, especially when they'll make things far easier for it.
> 
> What you just described was the logic applied with, for example, ia64,
> which is true 64-bit (only) native.  However (as I understand it), the
> Linux FHS and LSB used somewhat different logic for x86_64.

if you read FHS you'll see that both implementations are allowed.  Gentoo isnt 
violating anything here.  wrt LSB, who knows.  there are a ton of things we 
dont follow with LSB.

> See, x86_64 is hardware native dual-bitness, so both 32-bit and 64-bit
> can be said to be true native hardware bitness (this is NOT the case with
> ia64).

you can word "true native hardware bitness" (i dont even know what that means) 
however you like.  fact is, first ia64 gen had hardware support for x86.  no 
software levels needed.  we decided to not support ia64 multilib because (1) 
the hardware sucked so bad and (2) no one actually wanted it and (3) none of 
the mainline packages were/are doing it and (4) newer ia64 gens dropped 
support for it.

> Apparently due to that and to the vast number of legacy 32-bit
> apps around, many binary-only apps doing obscure and unpredictable things
> that could well break if assumptions about lib proved invalid, they
> decided to keep the 32-bit lib location just as it was, believing it
> easier to create the new lib64 for 64-bit, then to worry about whatever
> obscure and exotic stuff various binary-only apps might be doing. Since
> this was just one more change to add to the list of changes already being
> made to port apps to amd64, it was considered easier than the other way,
> and thus became the standard.

the only binary encoded 32bit /lib/ path is the ldso which is why we symlinked 
that too

> The problem was that Gentoo's early amd64 implementation predated this
> standardization, and we had chosen the other way.  While we've defaulted
> to lib64 for 64-bit libs for years, it has never been considered anything
> but experimental to break the lib --> lib64 link.  AFAIK stable
> baselayout still doesn't get its libdir usage consistent, putting files
> in one but actually calling them using the other path, and boot breaks in
> various frustrating ways if lib and lib64 are not the same directory.
> Openrc gets it better now, but I'm not sure it's all fixed either -- it
> certainly wasn't last time I tried breaking the link.

your "AFAIK" isnt useful.  there are no open bugs about either version and 
people assume that it's doing the right thing.

> So before Gentoo can switch to the FHS/LSB 32-bit lib on amd64 multilib,
> it must first fix those last inconsistent usages, and make it possible to
> break the lib --> lib64 link.  Then in theory at least, after awhile with
> no bugs, or at least no big ones related to this issue, it might be
> considered safe to move 32-bit back to lib, where the LSB/FHS says it
> should be.

we've already switched to the FHS implementation
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] RFC: multilib and the compatibility to singlelib

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 2135 bytes --]

On Tuesday 20 October 2009 09:06:29 Michael Haubenwallner wrote:
> As I'm building the toolchain myself too, I configure it with the
> 32bit host triplet on each platform, usually disabling multilib.

this doesnt make any sense to me

> This simply works for ppc-aix, hppa-hpux, ia64-hpux, sparc-solaris,
> x86-solaris, even if the underlying OS/kernel is 64bit.
> It isn't even necessary to explicitly use --{build,host} at all,
> as config.guess tells the 32bit triplet on these platforms.
> Additionally, even their default compiler output is 32bit.
> 
> But Linux "is not Unix":

you're right, so i'm not terribly concerned with compatibility with non-Linux 
systems.  comparing the native Gentoo/Linux multilib setup to another Linux 
multilib setup is the only useful comparison.

> Configuring both binutils and gcc needs to be done with:
>   $ CC="gcc -m32" /path/to/configure --{build,host}=i686-pc-linux-gnu ...
> This works on 64bit RHEL as well as on 64bit SLES (both have 32bit in /lib,
> 64bit in /lib64, no /lib32), but breaks on amd64+multilib Gentoo Linux.
> 
> Problem is that, as x86-linux isn't a multilib target, both ld and gcc
> search for 32bit libs&objects in /usr/lib:/lib, as they don't know about
> /usr/lib32:/lib32. The error when bootstrapping gcc is:
>   /usr/lib/crti.o: file not recognized: File format not recognized

i dont get it.  why does the i686-pc-linux-gnu toolchain target matter on an 
amd64 multilib system ?  the native x86_64-pc-linux-gnu toolchain should 
already do the right thing when given -m32.

> While it is possible to patch binutils[1] and gcc[2] to search in
> /usr/lib32:/lib32 even with this otherways non-multilib target,
> it doesn't feel like the "right thing" to enable multilib for
> just one multilib option.
> 
> Isn't the intention of multilib to have a new (64bit) system
> be compatible with the corresponding old (32bit) system?

your description of "compatible" is pretty vague.  ignoring /lib -> /lib64 
symlink (which shouldnt matter to any binaries), i'm not aware of any 
differences off the top of my head.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[gentoo-dev] Re: RFC: multilib and the compatibility to singlelib

[Jonathan Callen]

[-- Attachment #1.1: Type: text/plain, Size: 1696 bytes --]

Mike Frysinger wrote:
> if you read FHS you'll see that both implementations are allowed.  Gentoo isnt 
> violating anything here.  wrt LSB, who knows.  there are a ton of things we 
> dont follow with LSB.

Actually, at first, FHS says that any /lib<qual> would be allowed, but
it then goes into specifics in Chapter 6, Operating System Specific
Annex. Under the heading "/lib64 and /lib32 : 64/32-bit libraries
(architecture dependent)", it says, "The 64-bit architectures PPC64,
s390x, sparc64 and AMD64 must place 64-bit libraries in /lib64, and
32-bit (or 31-bit on s390) libraries in /lib."

>> The problem was that Gentoo's early amd64 implementation predated this
>> standardization, and we had chosen the other way.  While we've defaulted
>> to lib64 for 64-bit libs for years, it has never been considered anything
>> but experimental to break the lib --> lib64 link.  AFAIK stable
>> baselayout still doesn't get its libdir usage consistent, putting files
>> in one but actually calling them using the other path, and boot breaks in
>> various frustrating ways if lib and lib64 are not the same directory.
>> Openrc gets it better now, but I'm not sure it's all fixed either -- it
>> certainly wasn't last time I tried breaking the link.
> 
> your "AFAIK" isnt useful.  there are no open bugs about either version and 
> people assume that it's doing the right thing.

Personally, I do have a ~amd64 Gentoo chroot with LIBDIR_x86="lib".
There is only one place that I've found that it is still broken, namely
one line in toolchain.eclass (patch attached). I've been meaning to file
a bug for quite a while now, but never got around to it.

-- 
Jonathan

[-- Attachment #1.2: toolchain.eclass.patch --]
[-- Type: text/plain, Size: 395 bytes --]

--- /usr/portage/eclass/toolchain.eclass
+++ toolchain.eclass
@@ -1956,8 +1956,7 @@ gcc_movelibs() {
 			${LIBPATH}/${OS_MULTIDIR} \
 			${LIBPATH}/../${MULTIDIR} \
 			${PREFIX}/lib/${OS_MULTIDIR} \
-			${PREFIX}/${CTARGET}/lib/${OS_MULTIDIR} \
-			${PREFIX}/lib/${MULTIDIR}
+			${PREFIX}/${CTARGET}/lib/${OS_MULTIDIR}
 		do
 			removedirs="${removedirs} ${FROMDIR}"
 			FROMDIR=${D}${FROMDIR}

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 261 bytes --]

Re: [gentoo-dev] Re: RFC: multilib and the compatibility to singlelib

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1470 bytes --]

On Tuesday 20 October 2009 16:47:50 Jonathan Callen wrote:
> Mike Frysinger wrote:
> >> The problem was that Gentoo's early amd64 implementation predated this
> >> standardization, and we had chosen the other way.  While we've defaulted
> >> to lib64 for 64-bit libs for years, it has never been considered
> >> anything but experimental to break the lib --> lib64 link.  AFAIK stable
> >> baselayout still doesn't get its libdir usage consistent, putting files
> >> in one but actually calling them using the other path, and boot breaks
> >> in various frustrating ways if lib and lib64 are not the same directory.
> >> Openrc gets it better now, but I'm not sure it's all fixed either -- it
> >> certainly wasn't last time I tried breaking the link.
> >
> > your "AFAIK" isnt useful.  there are no open bugs about either version
> > and people assume that it's doing the right thing.
> 
> Personally, I do have a ~amd64 Gentoo chroot with LIBDIR_x86="lib".
> There is only one place that I've found that it is still broken, namely
> one line in toolchain.eclass (patch attached). I've been meaning to file
> a bug for quite a while now, but never got around to it.

off the top of my head, that doesnt really look like a correct fix.  please 
open a bug with info on what you're actually doing.

plus, your LIBDIR_x86 isnt really respected.  there are places where 
lib32/lib64 are hardcoded when modifying gcc/binutils (patch or sed).
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] RFC: multilib and the compatibility to singlelib

[Michael Haubenwallner]

Mike Frysinger wrote:
> On Tuesday 20 October 2009 09:06:29 Michael Haubenwallner wrote:
>> As I'm building the toolchain myself too, I configure it with the
>> 32bit host triplet on each platform, usually disabling multilib.
> 
> this doesnt make any sense to me

What exactly doesn't make sense to you:
* building the toolchain in general:
   The application is quality assured to one specific gcc version, and
   I cannot expect that exactly this one version is available on the
   target machine, especially when some specific patches are required,
   or there is no gcc available at all, or the installation is just broken.
   And sometimes I do have multiple application releases on the same
   machine, requiring different gcc versions...
* using the 32bit host triplet:
   see below
* or disabling multilib:
   whenever possible I'd like to avoid messing with multilib.

>> But Linux "is not Unix":
> 
> you're right, so i'm not terribly concerned with compatibility with non-Linux 
> systems.  comparing the native Gentoo/Linux multilib setup to another Linux 
> multilib setup is the only useful comparison.

Agreed. Although it is uncommon that Linux causes more headaches than Unix,
especially when it is about GNU packages.

>> Configuring both binutils and gcc needs to be done with:
>>   $ CC="gcc -m32" /path/to/configure --{build,host}=i686-pc-linux-gnu ...
>> This works on 64bit RHEL as well as on 64bit SLES (both have 32bit in /lib,
>> 64bit in /lib64, no /lib32), but breaks on amd64+multilib Gentoo Linux.

> i dont get it.  why does the i686-pc-linux-gnu toolchain target matter on an 
> amd64 multilib system ?  the native x86_64-pc-linux-gnu toolchain should 
> already do the right thing when given -m32.

It is more an administrative thing: On Unix as well as x86-linux I simply
get 32bits from gcc. But for x86_64-linux I'm in need of an exception to
build/use some x86_64-gcc and wrap it with -m32, because I don't want to
force the application-maintainers to add this exception to add CFLAGS=-m32,
which can be interpreted as "require some change to keep it unchanged".
And it is ways easier to use --{build,host} than to create wrappers.

>> Isn't the intention of multilib to have a new (64bit) system
>> be compatible with the corresponding old (32bit) system?
> 
> your description of "compatible" is pretty vague.  ignoring /lib -> /lib64 
> symlink (which shouldnt matter to any binaries), i'm not aware of any 
> differences off the top of my head.

Well, "compatible" here means to me that when I do
$ configure --{build,host}=i686-pc-linux-gnu
on x86-linux, I'd like to expect this working on x86_64-linux too, as the
"_64" can be seen as an "extension"[1] to x86 I just do not want to use.

It turns out that it is the "/lib resolving to 64bit" thing only that
causes me headaches here, which actually is distro-specific.

[1] http://en.wikipedia.org/wiki/X86-64

/haubi/

Re: [gentoo-dev] RFC: multilib and the compatibility to singlelib

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1869 bytes --]

On Wednesday 21 October 2009 07:34:18 Michael Haubenwallner wrote:
> Mike Frysinger wrote:
> > On Tuesday 20 October 2009 09:06:29 Michael Haubenwallner wrote:
> >> As I'm building the toolchain myself too, I configure it with the
> >> 32bit host triplet on each platform, usually disabling multilib.
> >
> > this doesnt make any sense to me
> 
> What exactly doesn't make sense to you:

it doesnt make sense to build your own toolchain when the default native one 
Gentoo provides includes all multilib support already.

but i guess when you're commercially developing a binary-only package, people 
tend to not have such freedoms as the binary-only mentality infects all 
layers.

> >> Isn't the intention of multilib to have a new (64bit) system
> >> be compatible with the corresponding old (32bit) system?
> >
> > your description of "compatible" is pretty vague.  ignoring /lib ->
> > /lib64 symlink (which shouldnt matter to any binaries), i'm not aware of
> > any differences off the top of my head.
> 
> Well, "compatible" here means to me that when I do
> $ configure --{build,host}=i686-pc-linux-gnu

assuming you simply forgot the forcing of -m32 here, or you have a fully named 
i686-pc-linux-gnu-... toolchain

> on x86-linux, I'd like to expect this working on x86_64-linux too, as the
> "_64" can be seen as an "extension"[1] to x86 I just do not want to use.
> 
> It turns out that it is the "/lib resolving to 64bit" thing only that
> causes me headaches here, which actually is distro-specific.

i'm not against changing things to fall in line with what other distros have 
settled on (guess that's the risk you take when you're one of the first 
distros to do multilib), i just want this kind of decision to be fully 
informed / thought out before making it.  it's not something i'd label 
"trivial".
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] RFC: multilib and the compatibility to singlelib

[Michael Haubenwallner]

Mike Frysinger wrote:
> On Wednesday 21 October 2009 07:34:18 Michael Haubenwallner wrote:
>> Mike Frysinger wrote:
>>> On Tuesday 20 October 2009 09:06:29 Michael Haubenwallner wrote:
>>>> As I'm building the toolchain myself too, I configure it with the
>>>> 32bit host triplet on each platform, usually disabling multilib.
>>> this doesnt make any sense to me
>> What exactly doesn't make sense to you:
> 
> it doesnt make sense to build your own toolchain when the default native one 
> Gentoo provides includes all multilib support already.
> 
> but i guess when you're commercially developing a binary-only package, people 
> tend to not have such freedoms as the binary-only mentality infects all 
> layers.

Even if it's commercially, it isn't binary-only here. But it is bound
to a specific set of (likely older) toolchain versions usually not
available on the target system.
I just don't want to make an exception for Gentoo Linux hosts when it
does work on both RedHat and SuSE Linux as well as *nix.

>>>> Isn't the intention of multilib to have a new (64bit) system
>>>> be compatible with the corresponding old (32bit) system?
>>> your description of "compatible" is pretty vague.  ignoring /lib ->
>>> /lib64 symlink (which shouldnt matter to any binaries), i'm not aware of
>>> any differences off the top of my head.
>> Well, "compatible" here means to me that when I do
>> $ configure --{build,host}=i686-pc-linux-gnu
> 
> assuming you simply forgot the forcing of -m32 here, or you have a fully named 
> i686-pc-linux-gnu-... toolchain

I do (like to) have a fully qualified i686-pc-linux-gnu-* toolchain.
Adding -m32 would require to create the i686-pc-linux-gnu-gcc wrapper,
resulting in some kind of a fully qualified i686 toolchain again.

>> It turns out that it is the "/lib resolving to 64bit" thing only that
>> causes me headaches here, which actually is distro-specific.
> 
> i'm not against changing things to fall in line with what other distros have 
> settled on (guess that's the risk you take when you're one of the first 
> distros to do multilib), i just want this kind of decision to be fully 
> informed / thought out before making it.  it's not something i'd label 
> "trivial".

Fully agreed. But as I don't have time to carry on this symlink change,
I'm going to live with the patches for now (in Prefix).
OTOH, Debian uses /lib->/lib64 symlink too IIRC...

Thank you!
/haubi/