[gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 1875 bytes --]

Let me introduce a nice project, which was started by some users:

Since the emul-linux-x86-* packages for 32bit libs for amd64 users are neither easy to maintain nor
up-to-date, some users started to implement an eclass, which allows to build requested libs with
additional 32bit support. Later i joined them and helped them improving it a bit, but it was and
still is mainly their project, they do the main work keeping this overlay up-to-date.

Also this overlay is a nice idea to drop emul-linux-x86-* packages, it either requires continual
work or modification of many ebuilds in main tree to support this in long term. To avoid this, i
took the original multilib portage patch from kanaka, adjusted it to the current portage code and
added the ideas and code from the eclass version. The result is now a portage, which is able to
build any ebuild with additional 32bit lib support.

The current main regression are ebuilds and eclasses, which do not support this (e.g. perl modules
and mysql).

If anyone is interested:

-for the eclass version, which is mainly maintained by users and is mainly intended to only replace
the emul-linux-x86-* package: just add it via "layman -a multilib" (it should be pretty stable and
mostly working).

-for the portage version: It is also in the multilib overlay, but in a different branch called
portage-multilib. To use this, you should read the instructions at [1]
(doc/portage-multilib-instructions). This one should also mainly work, but there is probably a good
amount of packages in the main tree, which may refuse to work with it.

Bugreports: preferred way is #gentoo-multilib-overlay at irc.freenode.org, but we also have an
alias, where you can contact us: multilib@g.o

[1]: http://github.com/sjnewbury/multilib-overlay/tree/portage-multilib
-- 
Thomas Sachau

Gentoo Linux Developer


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

[gentoo-dev] Re: Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[Nikos Chantziaras]

On 08/16/2009 03:37 PM, Thomas Sachau wrote:
> The result is now a portage, which is able to
> build any ebuild with additional 32bit lib support.

This is great news!  It's one feature many of us were missing badly in 
Gentoo.

Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[Paul de Vrieze]

On Sunday 16 August 2009, Thomas Sachau wrote:
> Let me introduce a nice project, which was started by some users:
>
> Since the emul-linux-x86-* packages for 32bit libs for amd64 users are
> neither easy to maintain nor up-to-date, some users started to implement an
> eclass, which allows to build requested libs with additional 32bit support.
> Later i joined them and helped them improving it a bit, but it was and
> still is mainly their project, they do the main work keeping this overlay
> up-to-date.
>
> Also this overlay is a nice idea to drop emul-linux-x86-* packages, it
> either requires continual work or modification of many ebuilds in main tree
> to support this in long term. To avoid this, i took the original multilib
> portage patch from kanaka, adjusted it to the current portage code and
> added the ideas and code from the eclass version. The result is now a
> portage, which is able to build any ebuild with additional 32bit lib
> support.
>
> The current main regression are ebuilds and eclasses, which do not support
> this (e.g. perl modules and mysql).
>
> If anyone is interested:
>
> -for the eclass version, which is mainly maintained by users and is mainly
> intended to only replace the emul-linux-x86-* package: just add it via
> "layman -a multilib" (it should be pretty stable and mostly working).
>
> -for the portage version: It is also in the multilib overlay, but in a
> different branch called portage-multilib. To use this, you should read the
> instructions at [1] (doc/portage-multilib-instructions). This one should
> also mainly work, but there is probably a good amount of packages in the
> main tree, which may refuse to work with it.
>
> Bugreports: preferred way is #gentoo-multilib-overlay at irc.freenode.org,
> but we also have an alias, where you can contact us: multilib@g.o
>
> [1]: http://github.com/sjnewbury/multilib-overlay/tree/portage-multilib

Good work,

Unfortunately my 64 bit system is currently non-functional, but when it is 
working again (when I replace parts) I'll try the portage stuff out.

Paul

-- 
Paul de Vrieze
Email: pauldv@gentoo.org

Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1013 bytes --]

On Sunday 16 August 2009 08:37:37 Thomas Sachau wrote:
> -for the portage version: It is also in the multilib overlay, but in a
>  different branch called portage-multilib. To use this, you should read the
>  instructions at [1] (doc/portage-multilib-instructions). This one should
>  also mainly work, but there is probably a good amount of packages in the
>  main tree, which may refuse to work with it.

the abi-wrapper doesnt look terribly appealing.  why dont we use broken/custom 
-config handling as incentive to convert packages to .pc files.  pkg-config 
handles ABI/cross-compile splitting cleanly and transparently.

bash-4 is stable, so we could start depending on it.

you dont save/restore CPPFLAGS

is there documentation that covers the proposed changes/design to portage ?  
i'm not looking for high level (it runs src_compile twice).  i dont recall 
seeing details posted to the portage or gentoo mailing lists ...  it's hard to 
review `git diff portage-svn..master`.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 2518 bytes --]

Mike Frysinger schrieb:
> On Sunday 16 August 2009 08:37:37 Thomas Sachau wrote:
>> -for the portage version: It is also in the multilib overlay, but in a
>>  different branch called portage-multilib. To use this, you should read the
>>  instructions at [1] (doc/portage-multilib-instructions). This one should
>>  also mainly work, but there is probably a good amount of packages in the
>>  main tree, which may refuse to work with it.
> 
> the abi-wrapper doesnt look terribly appealing.  why dont we use broken/custom 
> -config handling as incentive to convert packages to .pc files.  pkg-config 
> handles ABI/cross-compile splitting cleanly and transparently.

I am totally unfamiliar with pkg-config, so that would take some time or a helping hand.

> bash-4 is stable, so we could start depending on it.

It still has 3 unstable KEYWORDS including mips.

> you dont save/restore CPPFLAGS

Are there any initial values it should get?

> is there documentation that covers the proposed changes/design to portage ?  
> i'm not looking for high level (it runs src_compile twice).  i dont recall 
> seeing details posted to the portage or gentoo mailing lists ...  it's hard to 
> review `git diff portage-svn..master`.
> -mike

There is currently no documentation for the actual code version.

In short, there are those changes:

1. ability to run phases multiple times in ebuild.sh:

change from
...
dyn_unpack()
    src_unpack

dyn_compile()
    src_compile

dyn_install()
    src_install
...

to basicly

...
dyn_unpack()
    for each ABI
        call set_abi
            create ABI workdir
            set ABI envronment variables
        call src_unpack

dyn_compile()
    for each ABI
        call set_abi
            use workdir for current ABI
            set ABI environment variables
        call src_compile

dyn_install()
    for each ABI
        call set_abi
            use workdir for current ABI
            set ABI environment variables
        call src_install
        call _finalize_abi_install
            create gentoo-multilib headers (if needed)
...

2. Adding the internal lib32 useflag and usedeps with some workarounds

3. Added bin/auto-multilib.sh, which contains most functions, merged together from the older version
kanaka presented ( => http://dev.gentoo.org/~kanaka/auto-multilib/) and the multilib-native.eclass
from master branch of multilib overlay.

-- 
Thomas Sachau

Gentoo Linux Developer


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[Nirbheek Chauhan]

On Tue, Oct 13, 2009 at 1:19 AM, Thomas Sachau <tommy@gentoo.org> wrote:
>> the abi-wrapper doesnt look terribly appealing.  why dont we use broken/custom
>> -config handling as incentive to convert packages to .pc files.  pkg-config
>> handles ABI/cross-compile splitting cleanly and transparently.
>
> I am totally unfamiliar with pkg-config, so that would take some time or a helping hand.
>

You can take a look at http://is.gd/4fSfN for an overview (google
cache of original since fdo downtime wiped home directories). I
completely agree with Mike that pkg-config makes all this much simpler
and consistent.

>> bash-4 is stable, so we could start depending on it.
>
> It still has 3 unstable KEYWORDS including mips.
>

~mips ~sparc-fbsd ~x86-fbsd

^^ None of these arches do stable keywords...


-- 
~Nirbheek Chauhan

GNOME+Mozilla Team, Gentoo

Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 5574 bytes --]

On Monday 12 October 2009 15:49:48 Thomas Sachau wrote:
> Mike Frysinger schrieb:
> > On Sunday 16 August 2009 08:37:37 Thomas Sachau wrote:
> >> -for the portage version: It is also in the multilib overlay, but in a
> >>  different branch called portage-multilib. To use this, you should read
> >> the instructions at [1] (doc/portage-multilib-instructions). This one
> >> should also mainly work, but there is probably a good amount of packages
> >> in the main tree, which may refuse to work with it.
> >
> > the abi-wrapper doesnt look terribly appealing.  why dont we use
> > broken/custom -config handling as incentive to convert packages to .pc
> > files.  pkg-config handles ABI/cross-compile splitting cleanly and
> > transparently.
> 
> I am totally unfamiliar with pkg-config, so that would take some time or a
>  helping hand.

going by the pkg-config wrapper we use for cross-compiling, the multilib code 
should unset:
	PKG_CONFIG_ALLOW_SYSTEM_CFLAGS
	PKG_CONFIG_ALLOW_SYSTEM_LIBS

then it should export export PKG_CONFIG_LIBDIR to the ABI-specific pkgconfig 
dir.  so it should be something akin to:
	export PKG_CONFIG_LIBDIR=/usr/$(get_libdir)/pkgconfig

any package that adds things to PKG_CONFIG_PATH (grep /etc/env.d/) will be a 
problem here, but it seems only qt-3 currently falls into this category.  the 
easiest thing will probably be to just disallow this behavior completely.

then pkg-config should automatically locate the right abi-specific .pc files 
and filter the related abi-specific flags.

> > bash-4 is stable, so we could start depending on it.
> 
> It still has 3 unstable KEYWORDS including mips.

mips doesnt count -- it's moving all to ~arch, and you've counted the bsd 
arches which are never stable (currently).

> > you dont save/restore CPPFLAGS
> 
> Are there any initial values it should get?

the current profiles dont set any, but it should be treated as a precious 
toolchain variable same as any other compiler flag

another quick look at _setup_abi_env() looks like it needs work:
 - LD should not default to `ld`
 - the -L paths to system dirs in LDFLAGS should not be there -- the toolchain 
can handle these just fine
 - missing CPPFLAGS handling
 - see previous comments re-pkg-config
 - you dont declare pyver local
 - the abi if check is uncommon syntax.  "! [[ a == b ]]" -> "[[ a != b ]]"

> > is there documentation that covers the proposed changes/design to portage
> > ? i'm not looking for high level (it runs src_compile twice).  i dont
> > recall seeing details posted to the portage or gentoo mailing lists ... 
> > it's hard to review `git diff portage-svn..master`.
> 
> There is currently no documentation for the actual code version.

what about documentation for ebuild writers ?  only thing i see is how to 
install the new version of portage.

> In short, there are those changes:
> 
> 1. ability to run phases multiple times in ebuild.sh:
> 
> change from
> ...
> dyn_unpack()
>     src_unpack
> 
> dyn_compile()
>     src_compile
> 
> dyn_install()
>     src_install
> ...
> 
> to basicly
> 
> ...
> dyn_unpack()
>     for each ABI
>         call set_abi
>             create ABI workdir
>             set ABI envronment variables
>         call src_unpack
> 
> dyn_compile()
>     for each ABI
>         call set_abi
>             use workdir for current ABI
>             set ABI environment variables
>         call src_compile
> 
> dyn_install()
>     for each ABI
>         call set_abi
>             use workdir for current ABI
>             set ABI environment variables
>         call src_install
>         call _finalize_abi_install
>             create gentoo-multilib headers (if needed)
> ...

so basically what we already do in like glibc and sandbox.  at the base level, 
should be fine since it's been working so far with these few ebuilds.

how do you control whether the multilib headers are needed ?  it'll probably 
make sense in general, but there are def some packages where this will only 
get in the way (the toolchain ones).

how do you differentiate between packages where multi ABIs make no sense ?  
for example, most packages that dont install any libraries but just binaries.  
let's use the simple package app-text/tree.

a lot of this multilib code should probably continue to live in the tree as 
it's a pretty big base of code to formalize that could do with fixes over 
time.  i.e. we figure out that certain paths / define protections dont work so 
well, so changing them to something else would require PMS changing !?  there 
has been talk before about pushing a lot of basic stuff to the tree so things 
dont have to be encoded in the PMS.

how are packages handled that can only be used via 1 ABI ?  any of the 
packages listed in the amd64 no-multilib package.mask for example.  while 
these are mostly binary-only, this isnt a binary-specific issue.  there are a 
number of packages which only work on x86/ppc but could easily work in a 
multilib amd64/ppc64 as a 32bit binary (source code sucks, is heavily asm, 
something else).

> 2. Adding the internal lib32 useflag and usedeps with some workarounds

what exactly does this "lib32" do ?  naming USE flags according to specific 
ABI implementations is a bad idea.  you have to forget special casing anything 
to "lib32 vs lib64".  amd64, while the most common, is hardly extensible.  we 
must handle multiple ABIs which easily might have the same bitsize.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 936 bytes --]

On Mon, Oct 12, 2009 at 04:50:23PM -0400, Mike Frysinger wrote:
> what exactly does this "lib32" do ?  naming USE flags according to specific 
> ABI implementations is a bad idea.  you have to forget special casing anything 
> to "lib32 vs lib64".  amd64, while the most common, is hardly extensible.  we 
> must handle multiple ABIs which easily might have the same bitsize.
The canonical example for this does still remain MIPS I believe.

The most common ABIs in MIPS are:
o32, n32 - Both in Gentoo releases
n64 - Was experimentally done in Gentoo (default-linux/mips/2007.1-dev/generic-be/n64)
o64, eabi, meabi, nubi - Not sure if they were were ever released in any way.

Crossdev DOES support the full swath of these last I checked it.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 330 bytes --]

Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 3317 bytes --]

Mike Frysinger schrieb:

> 
> another quick look at _setup_abi_env() looks like it needs work:
>  - LD should not default to `ld`

Whats your suggestion?

>  - the -L paths to system dirs in LDFLAGS should not be there -- the toolchain 
> can handle these just fine

Last time i tried without, some packages failed to compile, will test it again to check, if its
still needed

>  - missing CPPFLAGS handling

Added

>  - see previous comments re-pkg-config

Will have a look the next days

>  - you dont declare pyver local

fixed

>  - the abi if check is uncommon syntax.  "! [[ a == b ]]" -> "[[ a != b ]]"

fixed

> how do you control whether the multilib headers are needed ?  it'll probably 
> make sense in general, but there are def some packages where this will only 
> get in the way (the toolchain ones).

What do you mean here? If the diff between ABIs makes sense to install seperate versions?

> how do you differentiate between packages where multi ABIs make no sense ?  
> for example, most packages that dont install any libraries but just binaries.  
> let's use the simple package app-text/tree.

I dont differentiate. Currently i build for every ABI, if lib32 useflag is set and multilib useflag
is not set. The idea behind it is to allow the installation of additional 32bit binaries, if requested.

> a lot of this multilib code should probably continue to live in the tree as 
> it's a pretty big base of code to formalize that could do with fixes over 
> time.  i.e. we figure out that certain paths / define protections dont work so 
> well, so changing them to something else would require PMS changing !?  there 
> has been talk before about pushing a lot of basic stuff to the tree so things 
> dont have to be encoded in the PMS.

How do you want to do this? Require package managers to inherit some file/eclass?

> how are packages handled that can only be used via 1 ABI ?  any of the 
> packages listed in the amd64 no-multilib package.mask for example.  while 
> these are mostly binary-only, this isnt a binary-specific issue.  there are a 
> number of packages which only work on x86/ppc but could easily work in a 
> multilib amd64/ppc64 as a 32bit binary (source code sucks, is heavily asm, 
> something else).

The binary-only ones are easy. Since they dont interact with the env, they can be installed as
usual. If they depend on a new enough package manager with multilib support, they could also depend
on the useflag for the additional 32bit libs they need.

> 
>> 2. Adding the internal lib32 useflag and usedeps with some workarounds
> 
> what exactly does this "lib32" do ?  naming USE flags according to specific 
> ABI implementations is a bad idea.  you have to forget special casing anything 
> to "lib32 vs lib64".  amd64, while the most common, is hardly extensible.  we 
> must handle multiple ABIs which easily might have the same bitsize.

"lib32" is added to IUSE, you can enable as as every other useflag. Internally, portage does add
[lib32?] usedeps to all dependencies. So if you enable the lib32 useflag, portage will require this
useflag for all dependencies too. I dont mind renaming it, if there is some other sane naming for it.

-- 
Thomas Sachau

Gentoo Linux Developer


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 1215 bytes --]

Robin H. Johnson schrieb:
> On Mon, Oct 12, 2009 at 04:50:23PM -0400, Mike Frysinger wrote:
>> what exactly does this "lib32" do ?  naming USE flags according to specific 
>> ABI implementations is a bad idea.  you have to forget special casing anything 
>> to "lib32 vs lib64".  amd64, while the most common, is hardly extensible.  we 
>> must handle multiple ABIs which easily might have the same bitsize.
> The canonical example for this does still remain MIPS I believe.
> 
> The most common ABIs in MIPS are:
> o32, n32 - Both in Gentoo releases
> n64 - Was experimentally done in Gentoo (default-linux/mips/2007.1-dev/generic-be/n64)
> o64, eabi, meabi, nubi - Not sure if they were were ever released in any way.
> 
> Crossdev DOES support the full swath of these last I checked it.
> 


There is a difference between creating a toolchain and supporting all packages for that arch and
every possible ABI you can crosscompile.

Currently i only support amd64 since thats the only ARCH i know and have access to. If i get enough
details to implement other ARCHes and some way to test it there, i might try it for those other
ARCHes too.

-- 
Thomas Sachau

Gentoo Linux Developer


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1473 bytes --]

On Sunday 18 October 2009 14:49:09 Thomas Sachau wrote:
> Robin H. Johnson schrieb:
> > On Mon, Oct 12, 2009 at 04:50:23PM -0400, Mike Frysinger wrote:
> >> what exactly does this "lib32" do ?  naming USE flags according to
> >> specific ABI implementations is a bad idea.  you have to forget special
> >> casing anything to "lib32 vs lib64".  amd64, while the most common, is
> >> hardly extensible.  we must handle multiple ABIs which easily might have
> >> the same bitsize.
> >
> > The canonical example for this does still remain MIPS I believe.
> >
> > The most common ABIs in MIPS are:
> > o32, n32 - Both in Gentoo releases
> > n64 - Was experimentally done in Gentoo
> > (default-linux/mips/2007.1-dev/generic-be/n64) o64, eabi, meabi, nubi -
> > Not sure if they were were ever released in any way.
> >
> > Crossdev DOES support the full swath of these last I checked it.
> 
> There is a difference between creating a toolchain and supporting all
>  packages for that arch and every possible ABI you can crosscompile.
> 
> Currently i only support amd64 since thats the only ARCH i know and have
>  access to. If i get enough details to implement other ARCHes and some way
>  to test it there, i might try it for those other ARCHes too.

we're not asking you to implement support for these ABIs, just to keep in mind 
that any implementation that does not scale and/or hardcodes to a single set 
of ABIs isnt a proper solution
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 2478 bytes --]

On Sun, Oct 18, 2009 at 10:26:37PM -0400, Mike Frysinger wrote:
> On Sunday 18 October 2009 14:49:09 Thomas Sachau wrote:
> > Robin H. Johnson schrieb:
> > > On Mon, Oct 12, 2009 at 04:50:23PM -0400, Mike Frysinger wrote:
> > >> what exactly does this "lib32" do ?  naming USE flags according to
> > >> specific ABI implementations is a bad idea.  you have to forget special
> > >> casing anything to "lib32 vs lib64".  amd64, while the most common, is
> > >> hardly extensible.  we must handle multiple ABIs which easily might have
> > >> the same bitsize.
> > >
> > > The canonical example for this does still remain MIPS I believe.
> > >
> > > The most common ABIs in MIPS are:
> > > o32, n32 - Both in Gentoo releases
> > > n64 - Was experimentally done in Gentoo
> > > (default-linux/mips/2007.1-dev/generic-be/n64) o64, eabi, meabi, nubi -
> > > Not sure if they were were ever released in any way.
> > >
> > > Crossdev DOES support the full swath of these last I checked it.
> > 
> > There is a difference between creating a toolchain and supporting all
> >  packages for that arch and every possible ABI you can crosscompile.
> > 
> > Currently i only support amd64 since thats the only ARCH i know and have
> >  access to. If i get enough details to implement other ARCHes and some way
> >  to test it there, i might try it for those other ARCHes too.
> we're not asking you to implement support for these ABIs, just to keep in mind 
> that any implementation that does not scale and/or hardcodes to a single set 
> of ABIs isnt a proper solution
My main objection thusfar is hardcoding the names as lib64/lib32.

Is there any specific reason you're using the content of the
LIBDIR_${ABI} variable? Maybe using the $ABI string as the USE flag
would be better, in some form of USE_EXPAND manner (cannot use them raw
as I believe we still have some code in the form of 'use $ARCH', which
might not behave sanely if say both amd64 and x86 were set.

Egs:
multilib AMD64:
USE="abi_x86 abi_amd64"

Multilib PPC64:
USE="abi_ppc abi_ppc64"

Multilib MIPS (SGI hardware)
USE="abi_n32 abi_n64"

Possible valid multilib sets on MIPS are:
[n32,n64] - at least one Gentoo system like this has been built.
[o32,o64]
[eabi,meabi] - docs fuzzy
[nubi]

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 330 bytes --]

Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 5604 bytes --]

On Sunday 18 October 2009 14:46:07 Thomas Sachau wrote:
> Mike Frysinger schrieb:
> > another quick look at _setup_abi_env() looks like it needs work:
> >  - LD should not default to `ld`
> 
> Whats your suggestion?

the majority of the time, the compiler driver (i.e. `gcc`) should be used for 
linking.  very few packages should invoke the linker directly.  that is why 
currently the toolchain-func.eclass has tc-getLD return `ld` -- a few packages 
need it, but not most.  if we're going to be forcing the setting of the LD env 
var all the time, then it needs to default to ${CC}.  packages that need funky 
behavior should still work as they will be calling $(tc-getLD) anyways.

> >  - the -L paths to system dirs in LDFLAGS should not be there -- the
> > toolchain can handle these just fine
> 
> Last time i tried without, some packages failed to compile, will test it
>  again to check, if its still needed

if things are failing, then we should look at gcc/binutils to make sure they 
use the right default search paths when given -m32/-m64

> > how do you control whether the multilib headers are needed ?  it'll
> > probably make sense in general, but there are def some packages where
> > this will only get in the way (the toolchain ones).
> 
> What do you mean here? If the diff between ABIs makes sense to install
>  seperate versions?

some packages (like glibc and linux-headers) already handle the multilib 
situation.  forcing the unnecessary Gentoo layer into the stack can easily 
break things.

i imagine there are many other packages out there that use the same exact 
header regardless of the ABI in use -- let the compiler figure out padding of 
structs and sizes of types.

> > how do you differentiate between packages where multi ABIs make no sense
> > ? for example, most packages that dont install any libraries but just
> > binaries. let's use the simple package app-text/tree.
> 
> I dont differentiate. Currently i build for every ABI, if lib32 useflag is
>  set and multilib useflag is not set. The idea behind it is to allow the
>  installation of additional 32bit binaries, if requested.

that's is an immense waste of time.  if we ran all the source phases for a 
single ABI in one go, it should be easy to dynamically detect when a multilib 
multipass is necessary (by looking at library paths in $D).  and for the odd 
package out, create a hook of some sort (EMULTIABI=true) to force behavior.

i dont think there is any inherit reliance on running the multilib pass on 
each src step every time (other than that was easiest to implement) ?

> > a lot of this multilib code should probably continue to live in the tree
> > as it's a pretty big base of code to formalize that could do with fixes
> > over time.  i.e. we figure out that certain paths / define protections
> > dont work so well, so changing them to something else would require PMS
> > changing !?  there has been talk before about pushing a lot of basic
> > stuff to the tree so things dont have to be encoded in the PMS.
> 
> How do you want to do this? Require package managers to inherit some
>  file/eclass?

considering this requires changes to the PM already, i dont see why not.  and 
it wouldnt really be an inherit in the current sense of the word.  more like a 
simple source.

> > how are packages handled that can only be used via 1 ABI ?  any of the
> > packages listed in the amd64 no-multilib package.mask for example.  while
> > these are mostly binary-only, this isnt a binary-specific issue.  there
> > are a number of packages which only work on x86/ppc but could easily work
> > in a multilib amd64/ppc64 as a 32bit binary (source code sucks, is
> > heavily asm, something else).
> 
> The binary-only ones are easy. Since they dont interact with the env, they
>  can be installed as usual. If they depend on a new enough package manager
>  with multilib support, they could also depend on the useflag for the
>  additional 32bit libs they need.

if it's a binary package, we know the ABI of it ahead of time.  so if the 
package declared the binary ABI that it uses, then portage could handle the 
rest (making sure the deps are resolved against the ABI that it requires).  we 
dont want to rewrite every binary ebuild to include an explicit [foo] ABI flag 
on each of its deps.

> >> 2. Adding the internal lib32 useflag and usedeps with some workarounds
> >
> > what exactly does this "lib32" do ?  naming USE flags according to
> > specific ABI implementations is a bad idea.  you have to forget special
> > casing anything to "lib32 vs lib64".  amd64, while the most common, is
> > hardly extensible.  we must handle multiple ABIs which easily might have
> > the same bitsize.
> 
> "lib32" is added to IUSE, you can enable as as every other useflag.
>  Internally, portage does add [lib32?] usedeps to all dependencies. So if
>  you enable the lib32 useflag, portage will require this useflag for all
>  dependencies too. I dont mind renaming it, if there is some other sane
>  naming for it.

i think every package will need tagging for each ABI, not just relative to the 
default one.  so the profile on an amd64 multilib would declare that it wants 
both x86 and x86_64 binaries by default.  in the normal case, the PM can then 
automatically resolve all dependencies as requiring all ABIs.  if a binary 
package is emerged, the ebuild itself declares the ABIs it provides, and then 
the PM only resolves the dependencies for the ABIs the package provides.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 2810 bytes --]

Mike Frysinger schrieb:
> the majority of the time, the compiler driver (i.e. `gcc`) should be used for 
> linking.  very few packages should invoke the linker directly.  that is why 
> currently the toolchain-func.eclass has tc-getLD return `ld` -- a few packages 
> need it, but not most.  if we're going to be forcing the setting of the LD env 
> var all the time, then it needs to default to ${CC}.  packages that need funky 
> behavior should still work as they will be calling $(tc-getLD) anyways.
> 
>>>  - the -L paths to system dirs in LDFLAGS should not be there -- the
>>> toolchain can handle these just fine
>> Last time i tried without, some packages failed to compile, will test it
>>  again to check, if its still needed
> 
> if things are failing, then we should look at gcc/binutils to make sure they 
> use the right default search paths when given -m32/-m64

This is an example from configure failure with ABI=x86 for cvs-1.12.12-r6

last lines from configure:

checking for ssh... ssh
checking for vim... /bin/nano
checking for temporary directory... /tmp
checking security/pam_appl.h usability... yes
checking security/pam_appl.h presence... yes
checking for security/pam_appl.h... yes
checking for pam_start in -lpam... no
configure: error: Could not find PAM libraries but the headers exist.
      Give the --disable-pam option to compile without PAM support (or fix
      your broken configuration)

!!! Please attach the following file when seeking support:
!!! /var/tmp/portage/dev-util/cvs-1.12.12-r6/work/cvs-1.12.12/config.log
 * ERROR: dev-util/cvs-1.12.12-r6 failed:
 *   econf failed

relevant lines from config.log:

configure:38697: checking for pam_start in -lpam
configure:38727: x86_64-pc-linux-gnu-gcc -o conftest -march=nocona -O2 -pipe -m32
-L/emul/linux/x86/lib -L/emul/linux/x86/usr/lib -march=nocona -O2 -pipe  -Wl,--as-needed conftest.c
-lpam  -lnsl  -lz >&5
/usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/../../../../lib32/libpam.a(pam_dynamic.o): In function
`_pam_dlerror':
(.text+0x1f): undefined reference to `dlerror'
/usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/../../../../lib32/libpam.a(pam_dynamic.o): In function
`_pam_dlclose':
(.text+0x5f): undefined reference to `dlclose'
/usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/../../../../lib32/libpam.a(pam_dynamic.o): In function
`_pam_dlsym':
(.text+0xa6): undefined reference to `dlsym'
/usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/../../../../lib32/libpam.a(pam_dynamic.o): In function
`_pam_dlopen':
(.text+0xf2): undefined reference to `dlopen'
collect2: ld returned 1 exit status
configure:38733: $? = 1

If you need some more lines or complete build.log/config.log, feel free to tell me and i will send
them directly.

-- 
Thomas Sachau

Gentoo Linux Developer


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 2608 bytes --]

Robin H. Johnson schrieb:
> On Sun, Oct 18, 2009 at 10:26:37PM -0400, Mike Frysinger wrote:
>> On Sunday 18 October 2009 14:49:09 Thomas Sachau wrote:
>>> Robin H. Johnson schrieb:
>>>> On Mon, Oct 12, 2009 at 04:50:23PM -0400, Mike Frysinger wrote:
>>>>> what exactly does this "lib32" do ?  naming USE flags according to
>>>>> specific ABI implementations is a bad idea.  you have to forget special
>>>>> casing anything to "lib32 vs lib64".  amd64, while the most common, is
>>>>> hardly extensible.  we must handle multiple ABIs which easily might have
>>>>> the same bitsize.
>>>> The canonical example for this does still remain MIPS I believe.
>>>>
>>>> The most common ABIs in MIPS are:
>>>> o32, n32 - Both in Gentoo releases
>>>> n64 - Was experimentally done in Gentoo
>>>> (default-linux/mips/2007.1-dev/generic-be/n64) o64, eabi, meabi, nubi -
>>>> Not sure if they were were ever released in any way.
>>>>
>>>> Crossdev DOES support the full swath of these last I checked it.
>>> There is a difference between creating a toolchain and supporting all
>>>  packages for that arch and every possible ABI you can crosscompile.
>>>
>>> Currently i only support amd64 since thats the only ARCH i know and have
>>>  access to. If i get enough details to implement other ARCHes and some way
>>>  to test it there, i might try it for those other ARCHes too.
>> we're not asking you to implement support for these ABIs, just to keep in mind 
>> that any implementation that does not scale and/or hardcodes to a single set 
>> of ABIs isnt a proper solution
> My main objection thusfar is hardcoding the names as lib64/lib32.
> 
> Is there any specific reason you're using the content of the
> LIBDIR_${ABI} variable? Maybe using the $ABI string as the USE flag
> would be better, in some form of USE_EXPAND manner (cannot use them raw
> as I believe we still have some code in the form of 'use $ARCH', which
> might not behave sanely if say both amd64 and x86 were set.
> 
> Egs:
> multilib AMD64:
> USE="abi_x86 abi_amd64"
> 
> Multilib PPC64:
> USE="abi_ppc abi_ppc64"
> 
> Multilib MIPS (SGI hardware)
> USE="abi_n32 abi_n64"
> 
> Possible valid multilib sets on MIPS are:
> [n32,n64] - at least one Gentoo system like this has been built.
> [o32,o64]
> [eabi,meabi] - docs fuzzy
> [nubi]
> 

I dont mind using some other string here. So those USE flags would be fine for me. They could be
USE_EXPANDed to e.g. for multilib amd64 ABI="x86 amd64"

Any other comment or suggestions on this part?

-- 
Thomas Sachau

Gentoo Linux Developer


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 2797 bytes --]

On Monday 19 October 2009 17:02:50 Thomas Sachau wrote:
> Robin H. Johnson schrieb:
> > On Sun, Oct 18, 2009 at 10:26:37PM -0400, Mike Frysinger wrote:
> >> On Sunday 18 October 2009 14:49:09 Thomas Sachau wrote:
> >>> Robin H. Johnson schrieb:
> >>>> On Mon, Oct 12, 2009 at 04:50:23PM -0400, Mike Frysinger wrote:
> >>>>> what exactly does this "lib32" do ?  naming USE flags according to
> >>>>> specific ABI implementations is a bad idea.  you have to forget
> >>>>> special casing anything to "lib32 vs lib64".  amd64, while the most
> >>>>> common, is hardly extensible.  we must handle multiple ABIs which
> >>>>> easily might have the same bitsize.
> >>>>
> >>>> The canonical example for this does still remain MIPS I believe.
> >>>>
> >>>> The most common ABIs in MIPS are:
> >>>> o32, n32 - Both in Gentoo releases
> >>>> n64 - Was experimentally done in Gentoo
> >>>> (default-linux/mips/2007.1-dev/generic-be/n64) o64, eabi, meabi, nubi
> >>>> - Not sure if they were were ever released in any way.
> >>>>
> >>>> Crossdev DOES support the full swath of these last I checked it.
> >>>
> >>> There is a difference between creating a toolchain and supporting all
> >>>  packages for that arch and every possible ABI you can crosscompile.
> >>>
> >>> Currently i only support amd64 since thats the only ARCH i know and
> >>> have access to. If i get enough details to implement other ARCHes and
> >>> some way to test it there, i might try it for those other ARCHes too.
> >>
> >> we're not asking you to implement support for these ABIs, just to keep
> >> in mind that any implementation that does not scale and/or hardcodes to
> >> a single set of ABIs isnt a proper solution
> >
> > My main objection thusfar is hardcoding the names as lib64/lib32.
> >
> > Is there any specific reason you're using the content of the
> > LIBDIR_${ABI} variable? Maybe using the $ABI string as the USE flag
> > would be better, in some form of USE_EXPAND manner (cannot use them raw
> > as I believe we still have some code in the form of 'use $ARCH', which
> > might not behave sanely if say both amd64 and x86 were set.
> >
> > Egs:
> > multilib AMD64:
> > USE="abi_x86 abi_amd64"
> >
> > Multilib PPC64:
> > USE="abi_ppc abi_ppc64"
> >
> > Multilib MIPS (SGI hardware)
> > USE="abi_n32 abi_n64"
> >
> > Possible valid multilib sets on MIPS are:
> > [n32,n64] - at least one Gentoo system like this has been built.
> > [o32,o64]
> > [eabi,meabi] - docs fuzzy
> > [nubi]
> 
> I dont mind using some other string here. So those USE flags would be fine
>  for me. They could be USE_EXPANDed to e.g. for multilib amd64 ABI="x86
>  amd64"
> 
> Any other comment or suggestions on this part?

what Robin has proposed makes sense to me
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[Robin H. Johnson]

On Mon, Oct 19, 2009 at 11:02:50PM +0200, Thomas Sachau wrote:
> > Is there any specific reason you're using the content of the
> > LIBDIR_${ABI} variable? Maybe using the $ABI string as the USE flag
> > would be better, in some form of USE_EXPAND manner (cannot use them raw
> > as I believe we still have some code in the form of 'use $ARCH', which
> > might not behave sanely if say both amd64 and x86 were set.
> > 
> > Egs:
> > multilib AMD64:
> > USE="abi_x86 abi_amd64"
> > 
> > Multilib PPC64:
> > USE="abi_ppc abi_ppc64"
> > 
> > Multilib MIPS (SGI hardware)
> > USE="abi_n32 abi_n64"
> > 
> > Possible valid multilib sets on MIPS are:
> > [n32,n64] - at least one Gentoo system like this has been built.
> > [o32,o64]
> > [eabi,meabi] - docs fuzzy
> > [nubi]
> I dont mind using some other string here. So those USE flags would be fine for me. They could be
> USE_EXPANDed to e.g. for multilib amd64 ABI="x86 amd64"
I just one thought overnight on it, are we going to have a namespace
conflict with the variable?

make.defaults already uses ABI= to contain a single value, not be
multi-valued.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1285 bytes --]

On Monday 19 October 2009 18:53:10 Robin H. Johnson wrote:
> On Mon, Oct 19, 2009 at 11:02:50PM +0200, Thomas Sachau wrote:
> > > Is there any specific reason you're using the content of the
> > > LIBDIR_${ABI} variable? Maybe using the $ABI string as the USE flag
> > > would be better, in some form of USE_EXPAND manner (cannot use them raw
> > > as I believe we still have some code in the form of 'use $ARCH', which
> > > might not behave sanely if say both amd64 and x86 were set.
> > >
> > > Egs:
> > > multilib AMD64:
> > > USE="abi_x86 abi_amd64"
> > >
> > > Multilib PPC64:
> > > USE="abi_ppc abi_ppc64"
> > >
> > > Multilib MIPS (SGI hardware)
> > > USE="abi_n32 abi_n64"
> > >
> > > Possible valid multilib sets on MIPS are:
> > > [n32,n64] - at least one Gentoo system like this has been built.
> > > [o32,o64]
> > > [eabi,meabi] - docs fuzzy
> > > [nubi]
> >
> > I dont mind using some other string here. So those USE flags would be
> > fine for me. They could be USE_EXPANDed to e.g. for multilib amd64
> > ABI="x86 amd64"
> 
> I just one thought overnight on it, are we going to have a namespace
> conflict with the variable?
> 
> make.defaults already uses ABI= to contain a single value, not be
> multi-valued.

ABIS or EABIS
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 3058 bytes --]

On Monday 19 October 2009 16:59:55 Thomas Sachau wrote:
> Mike Frysinger schrieb:
> > the majority of the time, the compiler driver (i.e. `gcc`) should be used
> > for linking.  very few packages should invoke the linker directly.  that
> > is why currently the toolchain-func.eclass has tc-getLD return `ld` -- a
> > few packages need it, but not most.  if we're going to be forcing the
> > setting of the LD env var all the time, then it needs to default to
> > ${CC}.  packages that need funky behavior should still work as they will
> > be calling $(tc-getLD) anyways.
> >
> >>>  - the -L paths to system dirs in LDFLAGS should not be there -- the
> >>> toolchain can handle these just fine
> >>
> >> Last time i tried without, some packages failed to compile, will test it
> >>  again to check, if its still needed
> >
> > if things are failing, then we should look at gcc/binutils to make sure
> > they use the right default search paths when given -m32/-m64
> 
> This is an example from configure failure with ABI=x86 for cvs-1.12.12-r6
> 
> last lines from configure:
> 
> checking for ssh... ssh
> checking for vim... /bin/nano
> checking for temporary directory... /tmp
> checking security/pam_appl.h usability... yes
> checking security/pam_appl.h presence... yes
> checking for security/pam_appl.h... yes
> checking for pam_start in -lpam... no
> configure: error: Could not find PAM libraries but the headers exist.
>       Give the --disable-pam option to compile without PAM support (or fix
>       your broken configuration)
> 
> !!! Please attach the following file when seeking support:
> !!! /var/tmp/portage/dev-util/cvs-1.12.12-r6/work/cvs-1.12.12/config.log
>  * ERROR: dev-util/cvs-1.12.12-r6 failed:
>  *   econf failed
> 
> relevant lines from config.log:
> 
> configure:38697: checking for pam_start in -lpam
> configure:38727: x86_64-pc-linux-gnu-gcc -o conftest -march=nocona -O2
>  -pipe -m32 -L/emul/linux/x86/lib -L/emul/linux/x86/usr/lib -march=nocona
>  -O2 -pipe  -Wl,--as-needed conftest.c -lpam  -lnsl  -lz >&5
> /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/../../../../lib32/libpam.a(pam_dynam
> ic.o): In function `_pam_dlerror':
> (.text+0x1f): undefined reference to `dlerror'
> /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/../../../../lib32/libpam.a(pam_dynam
> ic.o): In function `_pam_dlclose':
> (.text+0x5f): undefined reference to `dlclose'
> /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/../../../../lib32/libpam.a(pam_dynam
> ic.o): In function `_pam_dlsym':
> (.text+0xa6): undefined reference to `dlsym'
> /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/../../../../lib32/libpam.a(pam_dynam
> ic.o): In function `_pam_dlopen':
> (.text+0xf2): undefined reference to `dlopen'
> collect2: ld returned 1 exit status
> configure:38733: $? = 1
> 
> If you need some more lines or complete build.log/config.log, feel free to
>  tell me and i will send them directly.

please open a bug about this for the toolchain guys.  i dont know when i'll 
get to researching this.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 3256 bytes --]

Mike Frysinger schrieb:
> On Monday 19 October 2009 16:59:55 Thomas Sachau wrote:
>> Mike Frysinger schrieb:
>>> the majority of the time, the compiler driver (i.e. `gcc`) should be used
>>> for linking.  very few packages should invoke the linker directly.  that
>>> is why currently the toolchain-func.eclass has tc-getLD return `ld` -- a
>>> few packages need it, but not most.  if we're going to be forcing the
>>> setting of the LD env var all the time, then it needs to default to
>>> ${CC}.  packages that need funky behavior should still work as they will
>>> be calling $(tc-getLD) anyways.
>>>
>>>>>  - the -L paths to system dirs in LDFLAGS should not be there -- the
>>>>> toolchain can handle these just fine
>>>> Last time i tried without, some packages failed to compile, will test it
>>>>  again to check, if its still needed
>>> if things are failing, then we should look at gcc/binutils to make sure
>>> they use the right default search paths when given -m32/-m64
>> This is an example from configure failure with ABI=x86 for cvs-1.12.12-r6
>>
>> last lines from configure:
>>
>> checking for ssh... ssh
>> checking for vim... /bin/nano
>> checking for temporary directory... /tmp
>> checking security/pam_appl.h usability... yes
>> checking security/pam_appl.h presence... yes
>> checking for security/pam_appl.h... yes
>> checking for pam_start in -lpam... no
>> configure: error: Could not find PAM libraries but the headers exist.
>>       Give the --disable-pam option to compile without PAM support (or fix
>>       your broken configuration)
>>
>> !!! Please attach the following file when seeking support:
>> !!! /var/tmp/portage/dev-util/cvs-1.12.12-r6/work/cvs-1.12.12/config.log
>>  * ERROR: dev-util/cvs-1.12.12-r6 failed:
>>  *   econf failed
>>
>> relevant lines from config.log:
>>
>> configure:38697: checking for pam_start in -lpam
>> configure:38727: x86_64-pc-linux-gnu-gcc -o conftest -march=nocona -O2
>>  -pipe -m32 -L/emul/linux/x86/lib -L/emul/linux/x86/usr/lib -march=nocona
>>  -O2 -pipe  -Wl,--as-needed conftest.c -lpam  -lnsl  -lz >&5
>> /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/../../../../lib32/libpam.a(pam_dynam
>> ic.o): In function `_pam_dlerror':
>> (.text+0x1f): undefined reference to `dlerror'
>> /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/../../../../lib32/libpam.a(pam_dynam
>> ic.o): In function `_pam_dlclose':
>> (.text+0x5f): undefined reference to `dlclose'
>> /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/../../../../lib32/libpam.a(pam_dynam
>> ic.o): In function `_pam_dlsym':
>> (.text+0xa6): undefined reference to `dlsym'
>> /usr/lib/gcc/x86_64-pc-linux-gnu/4.4.2/../../../../lib32/libpam.a(pam_dynam
>> ic.o): In function `_pam_dlopen':
>> (.text+0xf2): undefined reference to `dlopen'
>> collect2: ld returned 1 exit status
>> configure:38733: $? = 1
>>
>> If you need some more lines or complete build.log/config.log, feel free to
>>  tell me and i will send them directly.
> 
> please open a bug about this for the toolchain guys.  i dont know when i'll 
> get to researching this.
> -mike

Seems like it was some temporary issue, cannot reproduce it now, so ignore it for now.

-- 
Thomas Sachau

Gentoo Linux Developer


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 7344 bytes --]

Mike Frysinger schrieb:
> On Sunday 18 October 2009 14:46:07 Thomas Sachau wrote:
>> Mike Frysinger schrieb:
>>> another quick look at _setup_abi_env() looks like it needs work:
>>>  - LD should not default to `ld`
>> Whats your suggestion?
> 
> the majority of the time, the compiler driver (i.e. `gcc`) should be used for 
> linking.  very few packages should invoke the linker directly.  that is why 
> currently the toolchain-func.eclass has tc-getLD return `ld` -- a few packages 
> need it, but not most.  if we're going to be forcing the setting of the LD env 
> var all the time, then it needs to default to ${CC}.  packages that need funky 
> behavior should still work as they will be calling $(tc-getLD) anyways.

Dropped $LD for now, seems to work fine for me.

> 
>>>  - the -L paths to system dirs in LDFLAGS should not be there -- the
>>> toolchain can handle these just fine
>> Last time i tried without, some packages failed to compile, will test it
>>  again to check, if its still needed
> 
> if things are failing, then we should look at gcc/binutils to make sure they 
> use the right default search paths when given -m32/-m64

I currently dont export any explicit LDFLAGS, but instead only save the value at the end of each
phase, seems to work fine for me.

> 
>>> how do you control whether the multilib headers are needed ?  it'll
>>> probably make sense in general, but there are def some packages where
>>> this will only get in the way (the toolchain ones).
>> What do you mean here? If the diff between ABIs makes sense to install
>>  seperate versions?
> 
> some packages (like glibc and linux-headers) already handle the multilib 
> situation.  forcing the unnecessary Gentoo layer into the stack can easily 
> break things.

For glibc, it is avoided since it has the "multilib" useflag. If this is enabled, it indicates for
me that the package does handle everything for itself, so multilib-portage does handle it as if it
would be a normal package without multi-ABI request.
Since linux-headers do also some special multilib handling, could you also add a "multilib" useflag
for them? Currently i have an exception for them in my code to prevent problems for other packages.

I assume that ever package not having a multilib useflag does not any multilib-specific action. To
check, if the header files differ per ABI, i save them for both ABIs, then diff them and create
ABI-specific header files with a wrapper for all header files, that differ. The rest is just
installed as usual.

>>> how do you differentiate between packages where multi ABIs make no sense
>>> ? for example, most packages that dont install any libraries but just
>>> binaries. let's use the simple package app-text/tree.
>> I dont differentiate. Currently i build for every ABI, if lib32 useflag is
>>  set and multilib useflag is not set. The idea behind it is to allow the
>>  installation of additional 32bit binaries, if requested.
> 
> that's is an immense waste of time.  if we ran all the source phases for a 
> single ABI in one go, it should be easy to dynamically detect when a multilib 
> multipass is necessary (by looking at library paths in $D).  and for the odd 
> package out, create a hook of some sort (EMULTIABI=true) to force behavior.
> 
> i dont think there is any inherit reliance on running the multilib pass on 
> each src step every time (other than that was easiest to implement) ?

For packages with header files, i need to run all phases for both ABIs to check, if the header files
are ABI specific.
So it would require a check for installed libs and installed header files. And its more work since
it requires both changes to the ebuild and emerge command.

> 
>>> a lot of this multilib code should probably continue to live in the tree
>>> as it's a pretty big base of code to formalize that could do with fixes
>>> over time.  i.e. we figure out that certain paths / define protections
>>> dont work so well, so changing them to something else would require PMS
>>> changing !?  there has been talk before about pushing a lot of basic
>>> stuff to the tree so things dont have to be encoded in the PMS.
>> How do you want to do this? Require package managers to inherit some
>>  file/eclass?
> 
> considering this requires changes to the PM already, i dont see why not.  and 
> it wouldnt really be an inherit in the current sense of the word.  more like a 
> simple source.

I am ok with this suggestion.

> 
>>> how are packages handled that can only be used via 1 ABI ?  any of the
>>> packages listed in the amd64 no-multilib package.mask for example.  while
>>> these are mostly binary-only, this isnt a binary-specific issue.  there
>>> are a number of packages which only work on x86/ppc but could easily work
>>> in a multilib amd64/ppc64 as a 32bit binary (source code sucks, is
>>> heavily asm, something else).
>> The binary-only ones are easy. Since they dont interact with the env, they
>>  can be installed as usual. If they depend on a new enough package manager
>>  with multilib support, they could also depend on the useflag for the
>>  additional 32bit libs they need.
> 
> if it's a binary package, we know the ABI of it ahead of time.  so if the 
> package declared the binary ABI that it uses, then portage could handle the 
> rest (making sure the deps are resolved against the ABI that it requires).  we 
> dont want to rewrite every binary ebuild to include an explicit [foo] ABI flag 
> on each of its deps.

This would require additional vars for multilib handling, which would require inclusion in EAPI-3 or
in a future EAPI, if the current process with EAPIs is continued.

With the current implementation, i try to stay EAPI-independent, so the changes can be implemented
without having to wait for aproval of another EAPI.

> 
>>>> 2. Adding the internal lib32 useflag and usedeps with some workarounds
>>> what exactly does this "lib32" do ?  naming USE flags according to
>>> specific ABI implementations is a bad idea.  you have to forget special
>>> casing anything to "lib32 vs lib64".  amd64, while the most common, is
>>> hardly extensible.  we must handle multiple ABIs which easily might have
>>> the same bitsize.
>> "lib32" is added to IUSE, you can enable as as every other useflag.
>>  Internally, portage does add [lib32?] usedeps to all dependencies. So if
>>  you enable the lib32 useflag, portage will require this useflag for all
>>  dependencies too. I dont mind renaming it, if there is some other sane
>>  naming for it.
> 
> i think every package will need tagging for each ABI, not just relative to the 
> default one.  so the profile on an amd64 multilib would declare that it wants 
> both x86 and x86_64 binaries by default.  in the normal case, the PM can then 
> automatically resolve all dependencies as requiring all ABIs.  if a binary 
> package is emerged, the ebuild itself declares the ABIs it provides, and then 
> the PM only resolves the dependencies for the ABIs the package provides.

In general, this looks ok. As said above, i just want to prevent the multilib-implementation from
being delayed until another EAPI is created, aproved and implemented.

-- 
Thomas Sachau

Gentoo Linux Developer


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 4962 bytes --]

On Thursday 22 October 2009 11:26:58 Thomas Sachau wrote:
> Mike Frysinger schrieb:
> > On Sunday 18 October 2009 14:46:07 Thomas Sachau wrote:
> >> Mike Frysinger schrieb:
> >>> how do you control whether the multilib headers are needed ?  it'll
> >>> probably make sense in general, but there are def some packages where
> >>> this will only get in the way (the toolchain ones).
> >>
> >> What do you mean here? If the diff between ABIs makes sense to install
> >>  seperate versions?
> >
> > some packages (like glibc and linux-headers) already handle the multilib
> > situation.  forcing the unnecessary Gentoo layer into the stack can
> > easily break things.
> 
> For glibc, it is avoided since it has the "multilib" useflag. If this is
>  enabled, it indicates for me that the package does handle everything for
>  itself, so multilib-portage does handle it as if it would be a normal
>  package without multi-ABI request.
> Since linux-headers do also some special multilib handling, could you also
>  add a "multilib" useflag for them? Currently i have an exception for them
>  in my code to prevent problems for other packages.

the linux-headers package doesnt have any multilib handling in them.  the 
linux kernel itself takes care of installing proper headers for all ABIs 
(since they differ very little).  it isnt possible to enable/disable this 
behavior afaik, so USE=multilib in the package makes no sense.

perhaps cram all of these into a hidden USE expanded variable "EABI" ... but 
this is kind of a bad poor hack in face of creating a new dedicated variable 
to declare/control multilib settings.

> I assume that ever package not having a multilib useflag does not any
>  multilib-specific action. To check, if the header files differ per ABI, i
>  save them for both ABIs, then diff them and create ABI-specific header
>  files with a wrapper for all header files, that differ. The rest is just
>  installed as usual.

that should obviously work in general

> >>> how do you differentiate between packages where multi ABIs make no
> >>> sense ? for example, most packages that dont install any libraries but
> >>> just binaries. let's use the simple package app-text/tree.
> >>
> >> I dont differentiate. Currently i build for every ABI, if lib32 useflag
> >> is set and multilib useflag is not set. The idea behind it is to allow
> >> the installation of additional 32bit binaries, if requested.
> >
> > that's is an immense waste of time.  if we ran all the source phases for
> > a single ABI in one go, it should be easy to dynamically detect when a
> > multilib multipass is necessary (by looking at library paths in $D).  and
> > for the odd package out, create a hook of some sort (EMULTIABI=true) to
> > force behavior.
> >
> > i dont think there is any inherit reliance on running the multilib pass
> > on each src step every time (other than that was easiest to implement) ?
> 
> For packages with header files, i need to run all phases for both ABIs to
>  check, if the header files are ABI specific.
> So it would require a check for installed libs and installed header files.
>  And its more work since it requires both changes to the ebuild and emerge
>  command.

my point is to skip multilib phases for a package that only installs 
executables.  i dont think you need any changes to ebuilds to support this.  
if you run all src steps and then check for include files/libs in the $D dir, 
you know at that point whether you need to re-run the src steps for all ABIs.

> > if it's a binary package, we know the ABI of it ahead of time.  so if the
> > package declared the binary ABI that it uses, then portage could handle
> > the rest (making sure the deps are resolved against the ABI that it
> > requires).  we dont want to rewrite every binary ebuild to include an
> > explicit [foo] ABI flag on each of its deps.
> 
> This would require additional vars for multilib handling, which would
>  require inclusion in EAPI-3 or in a future EAPI, if the current process
>  with EAPIs is continued.
> 
> With the current implementation, i try to stay EAPI-independent, so the
>  changes can be implemented without having to wait for aproval of another
>  EAPI.

this is an edge case that the rest of the implementation doesnt really need to 
rely on.  in other words, we spec/prototype the right solution for this part 
for future EAPIs.

now that i think about it more, i dont think explicit USE deps on these 
packages is really EAPI compliant either.  for example, you cant change 
quake4-bin's depends from like "x11-libs/libXext" to "x11-libs/libXext[x86]" 
because that package doesnt have x86 in IUSE, nor does it make sense to add 
it.

an EAPI change would be required anyways to handle funky source packages like 
wine where normally it's a 32bit binary, but it can be built as a 64bit binary 
via USE flags ...
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Support for multiple ABIs for amd64 (64bit,32bit) in multilib overlay

[Denis Dupeyron]

(I'm resending this email as the original apparently did not make it
to the list, although Thomas probably received it)

Hi Thomas,

I'm replying to the original thread below to allow those who have
missed it to have the full context.

On Sun, Aug 16, 2009 at 5:37 AM, Thomas Sachau <tommy@gentoo.org> wrote:
> Let me introduce a nice project, which was started by some users:
>
> Since the emul-linux-x86-* packages for 32bit libs for amd64 users are neither easy to maintain nor
> up-to-date, some users started to implement an eclass, which allows to build requested libs with
> additional 32bit support. Later i joined them and helped them improving it a bit, but it was and
> still is mainly their project, they do the main work keeping this overlay up-to-date.
>
> Also this overlay is a nice idea to drop emul-linux-x86-* packages, it either requires continual
> work or modification of many ebuilds in main tree to support this in long term. To avoid this, i
> took the original multilib portage patch from kanaka, adjusted it to the current portage code and
> added the ideas and code from the eclass version. The result is now a portage, which is able to
> build any ebuild with additional 32bit lib support.
>
> The current main regression are ebuilds and eclasses, which do not support this (e.g. perl modules
> and mysql).
>
> If anyone is interested:
>
> -for the eclass version, which is mainly maintained by users and is mainly intended to only replace
> the emul-linux-x86-* package: just add it via "layman -a multilib" (it should be pretty stable and
> mostly working).
>
> -for the portage version: It is also in the multilib overlay, but in a different branch called
> portage-multilib. To use this, you should read the instructions at [1]
> (doc/portage-multilib-instructions). This one should also mainly work, but there is probably a good
> amount of packages in the main tree, which may refuse to work with it.
>
> Bugreports: preferred way is #gentoo-multilib-overlay at irc.freenode.org, but we also have an
> alias, where you can contact us: multilib@g.o
>
> [1]: http://github.com/sjnewbury/multilib-overlay/tree/portage-multilib
> --
> Thomas Sachau
>
> Gentoo Linux Developer

I have already explained that I think it's a wonderful project and I
definitely would like to see it in the tree sooner rather than later.
There was a discussion on the council alias where everybody who
participated seemed to like it too.

However the consensus was that the project wasn't mature enough (I
will let the other, more technical, council members comment on that).
There are still open questions on this here thread, there is a request
for low-level documentation and a high-level doc is also required
(make it a request from me if you want), a preliminary PMS patch is
needed, possibly a devmanual patch too, etc... I'm not saying we're
asking you to do all this alone because this isn't how a collaborative
project like Gentoo should work. We have resources and they are at
everybody's disposition. We (I) will help you coordinate that effort
if you need/want it.

I have noted somewhere that you are concerned about having to do an
EAPI bump and were trying to work around that. I understand your
concerns and would tend to agree with you since in the past these
things were not addressed smoothly and timely enough. This council
showed however that we were ready to change plans and create EAPI
bumps when needed [1]. If multilib is ready before or at the same time
as prefix we can add multilib to EAPI3. If not, well, we will bump as
needed by multilib or any worthy project/enhancement anyway. There is
no point carving (the former) EAPI3 into stone and having it block
everything else due to its implementation taking longer than
anticipated.

Also, there is no good reason for doing things the wrong way. If an
EAPI bump is needed for multilib then let's just do it. I will
personally see to putting this EAPI bump on the agenda when multilib
is ready. And I'm convinced that at that time my fellow council
members will simply vote "yes".

As you have noticed on the portage irc channel I discussed the
maintenance of your branch with Zac. He has agreed to help you with
that, and I understand that's your main concern at the moment. It
appears that the portage repo is in the process of being converted to
git [2] and this should make it a lot easier. I suggest you talk to
Zac directly about this. Still on this subject, I will put the
question of whether we should add this new multilib to the portage 2.2
branch or something more experimental on the agenda for the next
meeting. I will also add multilib as a topic for the open floor
discussion.

Feel free to contact me in private if you have any question or need
help with the above requests. I will do my best to assist you.

Denis.

[1] http://www.gentoo.org/proj/en/council/meeting-logs/20091207-summary.txt
[2] https://bugs.gentoo.org/show_bug.cgi?id=196025#c34 and further