[gentoo-dev] [RFC] multilib-build.eclass and restricting unsupported ABIs

[gentoo-dev] [RFC] multilib-build.eclass and restricting unsupported ABIs

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2649 bytes --]

Hello,

With the introduction of support for x32 ABI it has become necessary to
enhance the multilib-build eclass with some kind of support for
specifying the supported/unsupported ABIs.

In this particular context, tetromino has noted that many packages
don't support the x32 ABI. From the ones currently using the eclass,
the one is app-emulation/wine.

I would like to enhance the eclass with the ability to specify
supported and unsupported ABIs. For that reason, I'd like to gather
your opinion on what would be the best solution. Preferably, I'd see
one that could work both for the eclass and multilib-portage so that we
wouldn't need to duplicate the same information.


1) opt-in or opt-out?

So far, the multilib-capable packages did get support for all multilib
ABIs on given architecture enabled (assuming that the package is
keyworded for the arch).

As a next step from that, I think an opt-out solution be the simplest
and most consistent one. In this particular context:

  MULTILIB_RESTRICT_ABIS=( ... )

which would be an optional variable disabling support for problematic
ABIs in the packages which need it.


An alternative solution would be an opt-in like in python-r1:

  MULTILIB_COMPAT=( ... )

but so far, that would mean that all current packages will have to be
updated to list the currently supported ABIs. And it all sucks a bit
due to the gray zone between amd64/x86 keyword and ABIs.


And no, optional MULTILIB_COMPAT is a no-go. It's a weird breed of
opt-in and opt-out which is just awful.



2) USE flag names or ABI names?

Next thing to decide would be: whether the restrict should specify USE
flag names (like the eclass solution) or ABI names (like
portage-multilib and profiles).

The advantage of USE flags is that they are guaranteed to be unique
and clear. As in, two arches won't ever have the same USE flag for ABI
with the same name.

  MULTILIB_RESTRICT_ABIS=( abi_x86_x32 )

The advantage of ABI names is that multilib-portage is aware of them.
So, it's mostly about supporting a poor choice done without consulting
other developers.

  MULTILIB_RESTRICT_ABIS=( x32 )

The problem with that is that a new arch can define an ABI with exactly
the same name (since all ABI variables are profile-local). In that
case, the restriction will unexpectedly apply to that arch.


By the way, maybe we should move the flag -> ABI mapping from
the eclass to some global location in profiles? That would make it
possible to use the global flags from multilib-portage as well.

What are your thoughts?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] [RFC] multilib-build.eclass and restricting unsupported ABIs

[Alexis Ballier]

On Sun, 3 Mar 2013 00:02:30 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> Hello,
> 
> With the introduction of support for x32 ABI it has become necessary
> to enhance the multilib-build eclass with some kind of support for
> specifying the supported/unsupported ABIs.
> 
> In this particular context, tetromino has noted that many packages
> don't support the x32 ABI. From the ones currently using the eclass,
> the one is app-emulation/wine.
> 
> I would like to enhance the eclass with the ability to specify
> supported and unsupported ABIs. For that reason, I'd like to gather
> your opinion on what would be the best solution. Preferably, I'd see
> one that could work both for the eclass and multilib-portage so that
> we wouldn't need to duplicate the same information.
> 
> 
> 1) opt-in or opt-out?
> 
> So far, the multilib-capable packages did get support for all multilib
> ABIs on given architecture enabled (assuming that the package is
> keyworded for the arch).
> 
> As a next step from that, I think an opt-out solution be the simplest
> and most consistent one. In this particular context:
> 
>   MULTILIB_RESTRICT_ABIS=( ... )
> 
> which would be an optional variable disabling support for problematic
> ABIs in the packages which need it.
> 
> 
> An alternative solution would be an opt-in like in python-r1:
> 
>   MULTILIB_COMPAT=( ... )
> 
> but so far, that would mean that all current packages will have to be
> updated to list the currently supported ABIs. And it all sucks a bit
> due to the gray zone between amd64/x86 keyword and ABIs.
> 
> 
> And no, optional MULTILIB_COMPAT is a no-go. It's a weird breed of
> opt-in and opt-out which is just awful.


I'd go for opt-out (MULTILIB_RESTRICT_ABIS); Ideally we'd want all
packages to support all abis, so what we should aim at is building
for every abi. Also, opt-in has the big disadvantage that introducing a
new abi requires a lot of tree-wide changes, which we tried to avoid
since the beginning.


> 2) USE flag names or ABI names?
> 
> Next thing to decide would be: whether the restrict should specify USE
> flag names (like the eclass solution) or ABI names (like
> portage-multilib and profiles).
> 
> The advantage of USE flags is that they are guaranteed to be unique
> and clear. As in, two arches won't ever have the same USE flag for ABI
> with the same name.
> 
>   MULTILIB_RESTRICT_ABIS=( abi_x86_x32 )
> 
> The advantage of ABI names is that multilib-portage is aware of them.
> So, it's mostly about supporting a poor choice done without consulting
> other developers.
> 
>   MULTILIB_RESTRICT_ABIS=( x32 )
> 
> The problem with that is that a new arch can define an ABI with
> exactly the same name (since all ABI variables are profile-local). In
> that case, the restriction will unexpectedly apply to that arch.
> 
> 
> By the way, maybe we should move the flag -> ABI mapping from
> the eclass to some global location in profiles? That would make it
> possible to use the global flags from multilib-portage as well.
> 
> What are your thoughts?

I'd prefer the useflag names for the sake of unicity, but I'm not sure
I understand why and how multilib-portage needs it.
What will multilib-portage uses it for ? If that's to gather and use
its  information to restrict some ABIs, then I assume you will have
something like 'if multilib-portage then dont do anything multilib' in
the eclass; well, you can very well export a variable translating the
useflag names to abi names that multilib-portage can use too. I'm not
sure you need the mapping on the profiles.

A.

Re: [gentoo-dev] [RFC] multilib-build.eclass and restricting unsupported ABIs

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1233 bytes --]

On Sun, 3 Mar 2013 12:41:07 +0100
Alexis Ballier <aballier@gentoo.org> wrote:

> I'd prefer the useflag names for the sake of unicity, but I'm not sure
> I understand why and how multilib-portage needs it.

Well, I mostly thought that if we were to introduce that information,
multilib-portage could use it as well rather than expecting
the information to be duplicated for the sake of it.

Not that I see a good reason for multilib-portage to work-around our
multilib but they like it.

> What will multilib-portage uses it for ? If that's to gather and use
> its  information to restrict some ABIs, then I assume you will have
> something like 'if multilib-portage then dont do anything multilib' in
> the eclass; well, you can very well export a variable translating the
> useflag names to abi names that multilib-portage can use too. I'm not
> sure you need the mapping on the profiles.

Well, right now multilib-portage is using the 'fallback' mechanism
in the eclass (the same as used on non-multilib arches). They mask all
the multilib flags and build the package one ABI at a time. The foreach
loop does not find any enabled flag and uses ${ABI:-${DEFAULT_ABI}}.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

[gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 3197 bytes --]

Michał Górny schrieb:
> Hello,
> 
> With the introduction of support for x32 ABI it has become necessary to
> enhance the multilib-build eclass with some kind of support for
> specifying the supported/unsupported ABIs.
> 
> In this particular context, tetromino has noted that many packages
> don't support the x32 ABI. From the ones currently using the eclass,
> the one is app-emulation/wine.
> 
> I would like to enhance the eclass with the ability to specify
> supported and unsupported ABIs. For that reason, I'd like to gather
> your opinion on what would be the best solution. Preferably, I'd see
> one that could work both for the eclass and multilib-portage so that we
> wouldn't need to duplicate the same information.
> 
> 
> 1) opt-in or opt-out?
> 
> So far, the multilib-capable packages did get support for all multilib
> ABIs on given architecture enabled (assuming that the package is
> keyworded for the arch).
> 
> As a next step from that, I think an opt-out solution be the simplest
> and most consistent one. In this particular context:
> 
>   MULTILIB_RESTRICT_ABIS=( ... )
> 
> which would be an optional variable disabling support for problematic
> ABIs in the packages which need it.

+1 for this one, better disable support for some packages with reported
issues then having to update all packages, when an ABI is added.

> 
> 
> An alternative solution would be an opt-in like in python-r1:
> 
>   MULTILIB_COMPAT=( ... )
> 
> but so far, that would mean that all current packages will have to be
> updated to list the currently supported ABIs. And it all sucks a bit
> due to the gray zone between amd64/x86 keyword and ABIs.
> 
> 
> And no, optional MULTILIB_COMPAT is a no-go. It's a weird breed of
> opt-in and opt-out which is just awful.
> 
> 
> 
> 2) USE flag names or ABI names?
> 
> Next thing to decide would be: whether the restrict should specify USE
> flag names (like the eclass solution) or ABI names (like
> portage-multilib and profiles).
> 
> The advantage of USE flags is that they are guaranteed to be unique
> and clear. As in, two arches won't ever have the same USE flag for ABI
> with the same name.
> 
>   MULTILIB_RESTRICT_ABIS=( abi_x86_x32 )
> 
> The advantage of ABI names is that multilib-portage is aware of them.
> So, it's mostly about supporting a poor choice done without consulting
> other developers.
> 
>   MULTILIB_RESTRICT_ABIS=( x32 )
> 
> The problem with that is that a new arch can define an ABI with exactly
> the same name (since all ABI variables are profile-local). In that
> case, the restriction will unexpectedly apply to that arch.
> 
> 
> By the way, maybe we should move the flag -> ABI mapping from
> the eclass to some global location in profiles? That would make it
> possible to use the global flags from multilib-portage as well.
> 
> What are your thoughts?
> 

Once the eclass has per-ABI header and binaries support, i would see
multilib-portage as fallback option for packages/arches, which dont yet
have multilib support via eclass. So i am ok with the USE flag names.

-- 

Thomas Sachau
Gentoo Linux Developer


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 379 bytes --]

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Alexis Ballier]

On Sun, 03 Mar 2013 14:02:58 +0100
Thomas Sachau <tommy@gentoo.org> wrote:
> 
> Once the eclass has per-ABI header

I think this is needed.

> and binaries support,

but here, could you enlighten me on its use cases ? I can't imagine
why having multi binaries support would be useful.

Alexis.

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 583 bytes --]

Alexis Ballier schrieb:
> On Sun, 03 Mar 2013 14:02:58 +0100
> Thomas Sachau <tommy@gentoo.org> wrote:
>>
>> Once the eclass has per-ABI header
> 
> I think this is needed.
> 
>> and binaries support,
> 
> but here, could you enlighten me on its use cases ? I can't imagine
> why having multi binaries support would be useful.
> 
> Alexis.
> 


At least some binaries do have abi-specific output, which is used by
other applications. As a good example of this, have a look at qmake and
qmake based build systems.

-- 

Thomas Sachau
Gentoo Linux Developer


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 379 bytes --]

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Chí-Thanh Christopher Nguyễn]

Alexis Ballier schrieb:
>> and binaries support,
> but here, could you enlighten me on its use cases ? I can't imagine
> why having multi binaries support would be useful.

One example is glxinfo from the mesa-progs package, it will display the
information for the ABI it is compiled with. So if you want to know
about the state of GLX/OpenGL acceleration for x86 applications, you
will need x86 glxinfo.


Best regards,
Chí-Thanh Christopher Nguyễn

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Alexis Ballier]

On Sun, 03 Mar 2013 16:47:43 +0100
Thomas Sachau <tommy@gentoo.org> wrote:

> Alexis Ballier schrieb:
> > On Sun, 03 Mar 2013 14:02:58 +0100
> > Thomas Sachau <tommy@gentoo.org> wrote:
> >>
> >> Once the eclass has per-ABI header
> > 
> > I think this is needed.
> > 
> >> and binaries support,
> > 
> > but here, could you enlighten me on its use cases ? I can't imagine
> > why having multi binaries support would be useful.
> > 
> > Alexis.
> > 
> 
> 
> At least some binaries do have abi-specific output, which is used by
> other applications. As a good example of this, have a look at qmake
> and qmake based build systems.

hmm, qmake doesnt seem to be the perfect example: how do you handle
this?

- install qmake-${abi}
- ln -s qmake-${DEFAULT_ABI} qmake
- modify eqmake4 to call the right qmake when doing multilib?

this sounds hackish for a behavior that doesn't make much sense to me
(its an upstream problem here); the glxinfo example seems perfectly
valid though.

Alexis.

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 1679 bytes --]

Alexis Ballier schrieb:
> On Sun, 03 Mar 2013 16:47:43 +0100
> Thomas Sachau <tommy@gentoo.org> wrote:
> 
>> Alexis Ballier schrieb:
>>> On Sun, 03 Mar 2013 14:02:58 +0100
>>> Thomas Sachau <tommy@gentoo.org> wrote:
>>>>
>>>> Once the eclass has per-ABI header
>>>
>>> I think this is needed.
>>>
>>>> and binaries support,
>>>
>>> but here, could you enlighten me on its use cases ? I can't imagine
>>> why having multi binaries support would be useful.
>>>
>>> Alexis.
>>>
>>
>>
>> At least some binaries do have abi-specific output, which is used by
>> other applications. As a good example of this, have a look at qmake
>> and qmake based build systems.
> 
> hmm, qmake doesnt seem to be the perfect example: how do you handle
> this?
> 
> - install qmake-${abi}

ok

> - ln -s qmake-${DEFAULT_ABI} qmake

Just the same as with headers:

You dont symlink the headers for the default ABI, but instead a wrapper
is placed, which does then call/include the real target, so in this
case, qmake is then a symlink to the abiwrapper, which does execute the
real abi-specific binary, depending on the current ABI.
We can of course place this abiwrapper in every place, where it is
needed instead of the symlink, but having one central and package
provided wrapper instead is easier to maintain and update.

> - modify eqmake4 to call the right qmake when doing multilib?

not needed at all (with multilib-portage), since when any package calls
qmake to get any abi-specific details, the abiwrapper executes the
binary, that matches the ABI and you get the right details for your ABI.

-- 

Thomas Sachau
Gentoo Linux Developer


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 379 bytes --]

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Alexis Ballier]

On Sun, 03 Mar 2013 17:27:50 +0100
Thomas Sachau <tommy@gentoo.org> wrote:

> Alexis Ballier schrieb:
> > On Sun, 03 Mar 2013 16:47:43 +0100
> > Thomas Sachau <tommy@gentoo.org> wrote:
> > 
> >> Alexis Ballier schrieb:
> >>> On Sun, 03 Mar 2013 14:02:58 +0100
> >>> Thomas Sachau <tommy@gentoo.org> wrote:
> >>>>
> >>>> Once the eclass has per-ABI header
> >>>
> >>> I think this is needed.
> >>>
> >>>> and binaries support,
> >>>
> >>> but here, could you enlighten me on its use cases ? I can't
> >>> imagine why having multi binaries support would be useful.
> >>>
> >>> Alexis.
> >>>
> >>
> >>
> >> At least some binaries do have abi-specific output, which is used
> >> by other applications. As a good example of this, have a look at
> >> qmake and qmake based build systems.
> > 
> > hmm, qmake doesnt seem to be the perfect example: how do you handle
> > this?
> > 
> > - install qmake-${abi}
> 
> ok
> 
> > - ln -s qmake-${DEFAULT_ABI} qmake
> 
> Just the same as with headers:
> 
> You dont symlink the headers for the default ABI, but instead a
> wrapper is placed, which does then call/include the real target, so
> in this case, qmake is then a symlink to the abiwrapper, which does
> execute the real abi-specific binary, depending on the current ABI.
> We can of course place this abiwrapper in every place, where it is
> needed instead of the symlink, but having one central and package
> provided wrapper instead is easier to maintain and update.
> 
> > - modify eqmake4 to call the right qmake when doing multilib?
> 
> not needed at all (with multilib-portage), since when any package
> calls qmake to get any abi-specific details, the abiwrapper executes
> the binary, that matches the ABI and you get the right details for
> your ABI.
> 

Indeed, nice idea. The wrapper can just call argv[0]-${ABI} or argv[0]
if ABI is unset or argv[0]-${ABI} does not exist.
Do you install this abiwrapper with multilib-portage? What would you
think about splitting it and adding such a package to the tree?

Alexis.

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2181 bytes --]

On Sun, 03 Mar 2013 17:27:50 +0100
Thomas Sachau <tommy@gentoo.org> wrote:

> Alexis Ballier schrieb:
> > On Sun, 03 Mar 2013 16:47:43 +0100
> > Thomas Sachau <tommy@gentoo.org> wrote:
> > 
> >> Alexis Ballier schrieb:
> >>> On Sun, 03 Mar 2013 14:02:58 +0100
> >>> Thomas Sachau <tommy@gentoo.org> wrote:
> >>>>
> >>>> Once the eclass has per-ABI header
> >>>
> >>> I think this is needed.
> >>>
> >>>> and binaries support,
> >>>
> >>> but here, could you enlighten me on its use cases ? I can't imagine
> >>> why having multi binaries support would be useful.
> >>>
> >>> Alexis.
> >>>
> >>
> >>
> >> At least some binaries do have abi-specific output, which is used by
> >> other applications. As a good example of this, have a look at qmake
> >> and qmake based build systems.
> > 
> > hmm, qmake doesnt seem to be the perfect example: how do you handle
> > this?
> > 
> > - install qmake-${abi}
> 
> ok
> 
> > - ln -s qmake-${DEFAULT_ABI} qmake
> 
> Just the same as with headers:
> 
> You dont symlink the headers for the default ABI, but instead a wrapper
> is placed, which does then call/include the real target, so in this
> case, qmake is then a symlink to the abiwrapper, which does execute the
> real abi-specific binary, depending on the current ABI.
> We can of course place this abiwrapper in every place, where it is
> needed instead of the symlink, but having one central and package
> provided wrapper instead is easier to maintain and update.
> 
> > - modify eqmake4 to call the right qmake when doing multilib?
> 
> not needed at all (with multilib-portage), since when any package calls
> qmake to get any abi-specific details, the abiwrapper executes the
> binary, that matches the ABI and you get the right details for your ABI.

What do we need that wrapper for? What does the wrapper do? Does it
just rely on custom 'ABI' variable? Or maybe should it try to detect
whether it was called by a 64- or 32-bit app? What for?

It's just a needless complexity, a big tool to handle a few corner
cases. Alexis just pointed out a perfectly good way of handling it.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Alexis Ballier]

On Sun, 3 Mar 2013 17:58:26 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> On Sun, 03 Mar 2013 17:27:50 +0100
> Thomas Sachau <tommy@gentoo.org> wrote:
> 
> > Alexis Ballier schrieb:
> > > On Sun, 03 Mar 2013 16:47:43 +0100
> > > Thomas Sachau <tommy@gentoo.org> wrote:
> > > 
> > >> Alexis Ballier schrieb:
> > >>> On Sun, 03 Mar 2013 14:02:58 +0100
> > >>> Thomas Sachau <tommy@gentoo.org> wrote:
> > >>>>
> > >>>> Once the eclass has per-ABI header
> > >>>
> > >>> I think this is needed.
> > >>>
> > >>>> and binaries support,
> > >>>
> > >>> but here, could you enlighten me on its use cases ? I can't
> > >>> imagine why having multi binaries support would be useful.
> > >>>
> > >>> Alexis.
> > >>>
> > >>
> > >>
> > >> At least some binaries do have abi-specific output, which is
> > >> used by other applications. As a good example of this, have a
> > >> look at qmake and qmake based build systems.
> > > 
> > > hmm, qmake doesnt seem to be the perfect example: how do you
> > > handle this?
> > > 
> > > - install qmake-${abi}
> > 
> > ok
> > 
> > > - ln -s qmake-${DEFAULT_ABI} qmake
> > 
> > Just the same as with headers:
> > 
> > You dont symlink the headers for the default ABI, but instead a
> > wrapper is placed, which does then call/include the real target, so
> > in this case, qmake is then a symlink to the abiwrapper, which does
> > execute the real abi-specific binary, depending on the current ABI.
> > We can of course place this abiwrapper in every place, where it is
> > needed instead of the symlink, but having one central and package
> > provided wrapper instead is easier to maintain and update.
> > 
> > > - modify eqmake4 to call the right qmake when doing multilib?
> > 
> > not needed at all (with multilib-portage), since when any package
> > calls qmake to get any abi-specific details, the abiwrapper
> > executes the binary, that matches the ABI and you get the right
> > details for your ABI.
> 
> What do we need that wrapper for? What does the wrapper do? Does it
> just rely on custom 'ABI' variable?

yes -- it must perform some checks though.

> Or maybe should it try to detect
> whether it was called by a 64- or 32-bit app?

this wont work: think about a build system, your shell/make will likely
be your default abi's but may call abi-specific tools depending on what
you build _for_ not what you build _with_

> What for?

in order to be transparent from the ebuild perspective.

> It's just a needless complexity, a big tool to handle a few corner
> cases. Alexis just pointed out a perfectly good way of handling it.

well, I don't like the way of handling it I pointed out to be honest :p

Alexis.

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 2554 bytes --]

Alexis Ballier schrieb:
> On Sun, 03 Mar 2013 17:27:50 +0100
> Thomas Sachau <tommy@gentoo.org> wrote:
> 
>> Alexis Ballier schrieb:
>>> On Sun, 03 Mar 2013 16:47:43 +0100
>>> Thomas Sachau <tommy@gentoo.org> wrote:
>>>
>>>> Alexis Ballier schrieb:
>>>>> On Sun, 03 Mar 2013 14:02:58 +0100
>>>>> Thomas Sachau <tommy@gentoo.org> wrote:
>>>>>>
>>>>>> Once the eclass has per-ABI header
>>>>>
>>>>> I think this is needed.
>>>>>
>>>>>> and binaries support,
>>>>>
>>>>> but here, could you enlighten me on its use cases ? I can't
>>>>> imagine why having multi binaries support would be useful.
>>>>>
>>>>> Alexis.
>>>>>
>>>>
>>>>
>>>> At least some binaries do have abi-specific output, which is used
>>>> by other applications. As a good example of this, have a look at
>>>> qmake and qmake based build systems.
>>>
>>> hmm, qmake doesnt seem to be the perfect example: how do you handle
>>> this?
>>>
>>> - install qmake-${abi}
>>
>> ok
>>
>>> - ln -s qmake-${DEFAULT_ABI} qmake
>>
>> Just the same as with headers:
>>
>> You dont symlink the headers for the default ABI, but instead a
>> wrapper is placed, which does then call/include the real target, so
>> in this case, qmake is then a symlink to the abiwrapper, which does
>> execute the real abi-specific binary, depending on the current ABI.
>> We can of course place this abiwrapper in every place, where it is
>> needed instead of the symlink, but having one central and package
>> provided wrapper instead is easier to maintain and update.
>>
>>> - modify eqmake4 to call the right qmake when doing multilib?
>>
>> not needed at all (with multilib-portage), since when any package
>> calls qmake to get any abi-specific details, the abiwrapper executes
>> the binary, that matches the ABI and you get the right details for
>> your ABI.
>>
> 
> Indeed, nice idea. The wrapper can just call argv[0]-${ABI} or argv[0]
> if ABI is unset or argv[0]-${ABI} does not exist.
> Do you install this abiwrapper with multilib-portage? What would you
> think about splitting it and adding such a package to the tree?
> 
> Alexis.
> 

The wrapper is already in a seperate package [1], the currently active
and tested version (1.0) is in bash, also there is a (currently masked)
version 2.0 written by binki in C doing the same natively. So you even
have a choice in what to use. :-)

[1]:
http://git.overlays.gentoo.org/gitweb/?p=proj/multilib-portage.git;a=tree;f=sys-apps/abi-wrapper

-- 

Thomas Sachau
Gentoo Linux Developer


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 379 bytes --]

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1778 bytes --]

On Sun, 3 Mar 2013 18:18:12 +0100
Alexis Ballier <aballier@gentoo.org> wrote:

> On Sun, 3 Mar 2013 17:58:26 +0100
> Michał Górny <mgorny@gentoo.org> wrote:
> 
> > What do we need that wrapper for? What does the wrapper do? Does it
> > just rely on custom 'ABI' variable?
> 
> yes -- it must perform some checks though.

What kind of checks?

> > Or maybe should it try to detect
> > whether it was called by a 64- or 32-bit app?
> 
> this wont work: think about a build system, your shell/make will likely
> be your default abi's but may call abi-specific tools depending on what
> you build _for_ not what you build _with_

That's one side of it. The other is that if it worked, it may be
something really unexpected. Do you expect things to work differently
when called by a 32-bit program?

> > What for?
> 
> in order to be transparent from the ebuild perspective.

That depends on what kind of transparency do we want. I prefer being
explicit here rather than assuming something you can't know for sure.

I think we're starting to miss the point of multilib. Multilib was
intended as a cheap way of getting things working. I believe that we
should still consider it so, and keep it in cages rather than believing
that the world is more fun with tigers jumping around.

That said, I wouldn't say that making random executables in system work
differently on ${ABI} is a way to go. That leaves the tricky imprint
of multilib visible to users who shouldn't care about it. If they do,
they're looking for multilib-portage.

The whole 'switching' part of multilib should be kept part of our build
system -- eclasses, ebuilds or just some specificities like libdir or
pkg-config path switching.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Alexis Ballier]

On Sun, 03 Mar 2013 22:39:42 +0100
Thomas Sachau <tommy@gentoo.org> wrote:
> The wrapper is already in a seperate package [1], the currently active
> and tested version (1.0) is in bash, also there is a (currently
> masked) version 2.0 written by binki in C doing the same natively. So
> you even have a choice in what to use. :-)

Good and not so good: We don't really have a choice, for portability we
need a C version otherwise shebangs are broken with older linux kernels
and non-linux platforms. ( Do you remember the python-wrapper mess? :) )
It'd be nice to have a well tested C version.

Alexis.

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Alexis Ballier]

On Sun, 3 Mar 2013 23:25:03 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> On Sun, 3 Mar 2013 18:18:12 +0100
> Alexis Ballier <aballier@gentoo.org> wrote:
> 
> > On Sun, 3 Mar 2013 17:58:26 +0100
> > Michał Górny <mgorny@gentoo.org> wrote:
> > 
> > > What do we need that wrapper for? What does the wrapper do? Does
> > > it just rely on custom 'ABI' variable?
> > 
> > yes -- it must perform some checks though.
> 
> What kind of checks?

you are called with ABI=sth argv[0] = your name
if argv[0] = abiwrapper -> print some information and exit
getenv ABI -> if nothing then set ABI=$DEFAULT_ABI (hardcoded at
buildtime of the wrapper)
execvp(argv[0]_$ABI, argv)
if execvp returns: print a warning, execvp argv[0]_$DEFAULT_ABI


python-wrapper.c is very likely to have such a logic already.

btw, IMHO ABI is a too common name for such a variable, I'd better name
it something like _GENTOO_MULTILIB_ABI so that collisions are much less
likely.

> > > Or maybe should it try to detect
> > > whether it was called by a 64- or 32-bit app?
> > 
> > this wont work: think about a build system, your shell/make will
> > likely be your default abi's but may call abi-specific tools
> > depending on what you build _for_ not what you build _with_
> 
> That's one side of it. The other is that if it worked, it may be
> something really unexpected. Do you expect things to work differently
> when called by a 32-bit program?

That's why I asked for examples :)
qmake may do it, I don't think its sane, but that's life for now.
having glxinfo for each abi is useful from a user perspective (it does
not need the wrapper a priori though)


> > > What for?
> > 
> > in order to be transparent from the ebuild perspective.
> 
> That depends on what kind of transparency do we want. I prefer being
> explicit here rather than assuming something you can't know for sure.

See it something like python-wrapper. EPYTHON=python3.2 python -> runs
python3.2 :)

> I think we're starting to miss the point of multilib. Multilib was
> intended as a cheap way of getting things working. I believe that we
> should still consider it so, and keep it in cages rather than
> believing that the world is more fun with tigers jumping around.
> 
> That said, I wouldn't say that making random executables in system
> work differently on ${ABI} is a way to go. That leaves the tricky
> imprint of multilib visible to users who shouldn't care about it. If
> they do, they're looking for multilib-portage.

To some extent that's what happened to python too :) As a python
maintainer, you could share your thoughts on the topic. python slotting
was intended to make switching between python versions easy but has
been needing wrappers for the python binary.

> The whole 'switching' part of multilib should be kept part of our
> build system -- eclasses, ebuilds or just some specificities like
> libdir or pkg-config path switching.

Maybe, but that would involve perfectly working setups being "broken".
It's like packages not respecting CC being broken for cross-compiling,
those not respecting CFLAGS being broken for multilib, etc. packages
calling directly binaries having ABI specific output will be broken for
multilib too (and I don't know of anyone checking for this while the
other two have been long standing issues we tried to fix). We can fix
this, but the fact is that we need multi-binary support for users, then
the only choice to make is if we want to provide a wrapper so that we
do not need to fix build systems or if we want to fix them. The latter
is likely preferred but I do not know what kind of work it will involve.
It'd help if tommy could provide a list of binaries he needed to wrap
through the abiwrapper.

Alexis.

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 4824 bytes --]

Alexis Ballier schrieb:
> On Sun, 3 Mar 2013 23:25:03 +0100
> Michał Górny <mgorny@gentoo.org> wrote:
> 
>> On Sun, 3 Mar 2013 18:18:12 +0100
>> Alexis Ballier <aballier@gentoo.org> wrote:
>>
>>> On Sun, 3 Mar 2013 17:58:26 +0100
>>> Michał Górny <mgorny@gentoo.org> wrote:
>>>
>>>> What do we need that wrapper for? What does the wrapper do? Does
>>>> it just rely on custom 'ABI' variable?
>>>
>>> yes -- it must perform some checks though.
>>
>> What kind of checks?
> 
> you are called with ABI=sth argv[0] = your name
> if argv[0] = abiwrapper -> print some information and exit
> getenv ABI -> if nothing then set ABI=$DEFAULT_ABI (hardcoded at
> buildtime of the wrapper)
> execvp(argv[0]_$ABI, argv)
> if execvp returns: print a warning, execvp argv[0]_$DEFAULT_ABI
> 
> 
> python-wrapper.c is very likely to have such a logic already.
> 
> btw, IMHO ABI is a too common name for such a variable, I'd better name
> it something like _GENTOO_MULTILIB_ABI so that collisions are much less
> likely.
> 
>>>> Or maybe should it try to detect
>>>> whether it was called by a 64- or 32-bit app?
>>>
>>> this wont work: think about a build system, your shell/make will
>>> likely be your default abi's but may call abi-specific tools
>>> depending on what you build _for_ not what you build _with_
>>
>> That's one side of it. The other is that if it worked, it may be
>> something really unexpected. Do you expect things to work differently
>> when called by a 32-bit program?
> 
> That's why I asked for examples :)
> qmake may do it, I don't think its sane, but that's life for now.
> having glxinfo for each abi is useful from a user perspective (it does
> not need the wrapper a priori though)
> 
> 
>>>> What for?
>>>
>>> in order to be transparent from the ebuild perspective.
>>
>> That depends on what kind of transparency do we want. I prefer being
>> explicit here rather than assuming something you can't know for sure.
> 
> See it something like python-wrapper. EPYTHON=python3.2 python -> runs
> python3.2 :)
> 
>> I think we're starting to miss the point of multilib. Multilib was
>> intended as a cheap way of getting things working. I believe that we
>> should still consider it so, and keep it in cages rather than
>> believing that the world is more fun with tigers jumping around.
>>
>> That said, I wouldn't say that making random executables in system
>> work differently on ${ABI} is a way to go. That leaves the tricky
>> imprint of multilib visible to users who shouldn't care about it. If
>> they do, they're looking for multilib-portage.
> 
> To some extent that's what happened to python too :) As a python
> maintainer, you could share your thoughts on the topic. python slotting
> was intended to make switching between python versions easy but has
> been needing wrappers for the python binary.
> 
>> The whole 'switching' part of multilib should be kept part of our
>> build system -- eclasses, ebuilds or just some specificities like
>> libdir or pkg-config path switching.
> 
> Maybe, but that would involve perfectly working setups being "broken".
> It's like packages not respecting CC being broken for cross-compiling,
> those not respecting CFLAGS being broken for multilib, etc. packages
> calling directly binaries having ABI specific output will be broken for
> multilib too (and I don't know of anyone checking for this while the
> other two have been long standing issues we tried to fix). We can fix
> this, but the fact is that we need multi-binary support for users, then
> the only choice to make is if we want to provide a wrapper so that we
> do not need to fix build systems or if we want to fix them. The latter
> is likely preferred but I do not know what kind of work it will involve.
> It'd help if tommy could provide a list of binaries he needed to wrap
> through the abiwrapper.
> 
> Alexis.
> 

The actual list of packages with wrapped binaries is in the profiles dir
of the multilib-portage overlay (expect this to be a minimal list, there
are probably more out there we did not yet catch):

dev-db/mysql abiwrapper
dev-lang/perl abiwrapper
dev-lang/python abiwrapper
dev-lang/ruby abiwrapper
dev-libs/gobject-introspection abiwrapper
dev-libs/libIDL abiwrapper
dev-scheme/guile abiwrapper
net-libs/courier-authlib abiwrapper
dev-qt/qtcore abiwrapper
dev-qt/qtgui abiwrapper
media-libs/fontconfig abiwrapper
www-servers/apache abiwrapper
x11-libs/pango abiwrapper
x11-libs/gtk+ abiwrapper

Keep in mind, that multilib-portage does always and unconditionally wrap
*-config binaries. If you dont want to add that logic to the eclass, you
need to add all packages providing such files to the list.

-- 

Thomas Sachau
Gentoo Linux Developer


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 379 bytes --]

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 5721 bytes --]

On Mon, 4 Mar 2013 11:02:40 +0100
Alexis Ballier <aballier@gentoo.org> wrote:

> On Sun, 3 Mar 2013 23:25:03 +0100
> Michał Górny <mgorny@gentoo.org> wrote:
> 
> > On Sun, 3 Mar 2013 18:18:12 +0100
> > Alexis Ballier <aballier@gentoo.org> wrote:
> > 
> > > On Sun, 3 Mar 2013 17:58:26 +0100
> > > Michał Górny <mgorny@gentoo.org> wrote:
> > > 
> > > > What do we need that wrapper for? What does the wrapper do? Does
> > > > it just rely on custom 'ABI' variable?
> > > 
> > > yes -- it must perform some checks though.
> > 
> > What kind of checks?
> 
> you are called with ABI=sth argv[0] = your name

I'm afraid that's the first potential point of failure. Relying
on argv[0] is a poor idea and means that any application calling exec()
with changed argv[0] on a wrapped binary will fail terribly.

> if argv[0] = abiwrapper -> print some information and exit
> getenv ABI -> if nothing then set ABI=$DEFAULT_ABI (hardcoded at
> buildtime of the wrapper)
> execvp(argv[0]_$ABI, argv)
> if execvp returns: print a warning, execvp argv[0]_$DEFAULT_ABI
> 
> 
> python-wrapper.c is very likely to have such a logic already.
> 
> btw, IMHO ABI is a too common name for such a variable, I'd better name
> it something like _GENTOO_MULTILIB_ABI so that collisions are much less
> likely.

I'm afraid you are actually starting to go the other way.

Global wrapper means that it is potentially useful to our users.
However I don't really see people who want to compile 32-bit
executables think of setting some custom variable like ABI.

Moreover, if we replace that with a specific, 'private' environment
variable like you suggested, we're actually discouraging people from
using it. So, we're force-installing wrappers which make things fragile
and at the same time provide no benefit for regular users.

> > > > Or maybe should it try to detect
> > > > whether it was called by a 64- or 32-bit app?
> > > 
> > > this wont work: think about a build system, your shell/make will
> > > likely be your default abi's but may call abi-specific tools
> > > depending on what you build _for_ not what you build _with_
> > 
> > That's one side of it. The other is that if it worked, it may be
> > something really unexpected. Do you expect things to work differently
> > when called by a 32-bit program?
> 
> That's why I asked for examples :)
> qmake may do it, I don't think its sane, but that's life for now.
> having glxinfo for each abi is useful from a user perspective (it does
> not need the wrapper a priori though)

Yep, I intended to just have the additional variant of glxinfo directly
callable, with a name chosen specifically by the X11 team. Wrapper
would be more confusing than beneficial here IMO.

> > > > What for?
> > > 
> > > in order to be transparent from the ebuild perspective.
> > 
> > That depends on what kind of transparency do we want. I prefer being
> > explicit here rather than assuming something you can't know for sure.
> 
> See it something like python-wrapper. EPYTHON=python3.2 python -> runs
> python3.2 :)

Err, python-wrapper respects quite complex logic involving EPYTHON,
and eselect-python. We don't really want people to have eselect-abi,
do we?

If we were to implement abi-wrapper, it will be much simpler than any
logic needed for Python.

> > I think we're starting to miss the point of multilib. Multilib was
> > intended as a cheap way of getting things working. I believe that we
> > should still consider it so, and keep it in cages rather than
> > believing that the world is more fun with tigers jumping around.
> > 
> > That said, I wouldn't say that making random executables in system
> > work differently on ${ABI} is a way to go. That leaves the tricky
> > imprint of multilib visible to users who shouldn't care about it. If
> > they do, they're looking for multilib-portage.
> 
> To some extent that's what happened to python too :) As a python
> maintainer, you could share your thoughts on the topic. python slotting
> was intended to make switching between python versions easy but has
> been needing wrappers for the python binary.

I'm doing just that. Any kind of wrapping is an increasing mess. I'm
still trying to find out good solutions for Python wrapping but there's
no such thing. It's always about choosing one evil over the other.

> > The whole 'switching' part of multilib should be kept part of our
> > build system -- eclasses, ebuilds or just some specificities like
> > libdir or pkg-config path switching.
> 
> Maybe, but that would involve perfectly working setups being "broken".
> It's like packages not respecting CC being broken for cross-compiling,
> those not respecting CFLAGS being broken for multilib, etc.

Just to make it clear, multilib.eclass overrides CC for multilib. It
does not rely on CFLAGS. I don't know the rationale, I can guess only.

> packages
> calling directly binaries having ABI specific output will be broken for
> multilib too (and I don't know of anyone checking for this while the
> other two have been long standing issues we tried to fix). We can fix
> this, but the fact is that we need multi-binary support for users, then
> the only choice to make is if we want to provide a wrapper so that we
> do not need to fix build systems or if we want to fix them. The latter
> is likely preferred but I do not know what kind of work it will involve.
> It'd help if tommy could provide a list of binaries he needed to wrap
> through the abiwrapper.

We don't want it for users in an automagic and fragile way. And ebuilds
we can fix while migrating.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 5508 bytes --]

Michał Górny schrieb:
> On Mon, 4 Mar 2013 11:02:40 +0100
> Alexis Ballier <aballier@gentoo.org> wrote:
> 
>> On Sun, 3 Mar 2013 23:25:03 +0100
>> Michał Górny <mgorny@gentoo.org> wrote:
>>
>>> On Sun, 3 Mar 2013 18:18:12 +0100
>>> Alexis Ballier <aballier@gentoo.org> wrote:
>>>
>>>> On Sun, 3 Mar 2013 17:58:26 +0100
>>>> Michał Górny <mgorny@gentoo.org> wrote:
>>>>
>>>>> What do we need that wrapper for? What does the wrapper do? Does
>>>>> it just rely on custom 'ABI' variable?
>>>>
>>>> yes -- it must perform some checks though.
>>>
>>> What kind of checks?
>>
>> you are called with ABI=sth argv[0] = your name
> 
> I'm afraid that's the first potential point of failure. Relying
> on argv[0] is a poor idea and means that any application calling exec()
> with changed argv[0] on a wrapped binary will fail terribly.

Nobody said, that one cannot create situations, where such a wrapper
does fail, the point is more an easy and general solution for wrapping
binaries without implementing different solutions for the same issue in
every ebuild.

If you have a better, yet still easy and general solution not requiring
every ebuild to create something on its own, please write it instead of
just complaining how bad the wrapper solution actually is.

> 
>> if argv[0] = abiwrapper -> print some information and exit
>> getenv ABI -> if nothing then set ABI=$DEFAULT_ABI (hardcoded at
>> buildtime of the wrapper)
>> execvp(argv[0]_$ABI, argv)
>> if execvp returns: print a warning, execvp argv[0]_$DEFAULT_ABI
>>
>>
>> python-wrapper.c is very likely to have such a logic already.
>>
>> btw, IMHO ABI is a too common name for such a variable, I'd better name
>> it something like _GENTOO_MULTILIB_ABI so that collisions are much less
>> likely.
> 
> I'm afraid you are actually starting to go the other way.
> 
> Global wrapper means that it is potentially useful to our users.
> However I don't really see people who want to compile 32-bit
> executables think of setting some custom variable like ABI.

Unless you implied, that you want users to compile from hand instead of
using an ebuild, this makes no sense to me.

This ABI variable is of course set when setting up the environment, so
it is around in every phase and any call to an abi-wrapped binary
directly gets the right one for the current ABI.

>>>>> Or maybe should it try to detect
>>>>> whether it was called by a 64- or 32-bit app?
>>>>
>>>> this wont work: think about a build system, your shell/make will
>>>> likely be your default abi's but may call abi-specific tools
>>>> depending on what you build _for_ not what you build _with_
>>>
>>> That's one side of it. The other is that if it worked, it may be
>>> something really unexpected. Do you expect things to work differently
>>> when called by a 32-bit program?
>>
>> That's why I asked for examples :)
>> qmake may do it, I don't think its sane, but that's life for now.
>> having glxinfo for each abi is useful from a user perspective (it does
>> not need the wrapper a priori though)
> 
> Yep, I intended to just have the additional variant of glxinfo directly
> callable, with a name chosen specifically by the X11 team. Wrapper
> would be more confusing than beneficial here IMO.

Ah, you actually want each ebuild to have its own custom hack instead of
one global solution usable everywhere?

> 
>>>>> What for?
>>>>
>>>> in order to be transparent from the ebuild perspective.
>>>
>>> That depends on what kind of transparency do we want. I prefer being
>>> explicit here rather than assuming something you can't know for sure.
>>
>> See it something like python-wrapper. EPYTHON=python3.2 python -> runs
>> python3.2 :)
> 
> Err, python-wrapper respects quite complex logic involving EPYTHON,
> and eselect-python. We don't really want people to have eselect-abi,
> do we?
> 
> If we were to implement abi-wrapper, it will be much simpler than any
> logic needed for Python.

Exactly: Just the environment variable, no eselect module, simple, easy
to understand and working.

> 
>>> I think we're starting to miss the point of multilib. Multilib was
>>> intended as a cheap way of getting things working. I believe that we
>>> should still consider it so, and keep it in cages rather than
>>> believing that the world is more fun with tigers jumping around.
>>>
>>> That said, I wouldn't say that making random executables in system
>>> work differently on ${ABI} is a way to go. That leaves the tricky
>>> imprint of multilib visible to users who shouldn't care about it. If
>>> they do, they're looking for multilib-portage.
>>
>> To some extent that's what happened to python too :) As a python
>> maintainer, you could share your thoughts on the topic. python slotting
>> was intended to make switching between python versions easy but has
>> been needing wrappers for the python binary.
> 
> I'm doing just that. Any kind of wrapping is an increasing mess. I'm
> still trying to find out good solutions for Python wrapping but there's
> no such thing. It's always about choosing one evil over the other.

So you are wrapping python, have not yet found anything better and still
dont want to wrap abi-specific binaries, while you dont have a better
solution at hand? Saying no to everything is easy, providing something
better if you dont like a suggestion is the challenge.

-- 

Thomas Sachau
Gentoo Linux Developer


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 379 bytes --]

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2545 bytes --]

On Mon, 04 Mar 2013 23:21:36 +0100
Thomas Sachau <tommy@gentoo.org> wrote:

> Michał Górny schrieb:
> > On Mon, 4 Mar 2013 11:02:40 +0100
> > Alexis Ballier <aballier@gentoo.org> wrote:
> > 
> >> you are called with ABI=sth argv[0] = your name
> > 
> > I'm afraid that's the first potential point of failure. Relying
> > on argv[0] is a poor idea and means that any application calling exec()
> > with changed argv[0] on a wrapped binary will fail terribly.
> 
> Nobody said, that one cannot create situations, where such a wrapper
> does fail, the point is more an easy and general solution for wrapping
> binaries without implementing different solutions for the same issue in
> every ebuild.

There's no such thing as 'easy and general solution'. You always
sacrifice something.

And in this case, you're creating a point of failure which is
completely custom to Gentoo and actually quite hard to track. Just to
support a specific package manager feature specific to Gentoo.

> If you have a better, yet still easy and general solution not requiring
> every ebuild to create something on its own, please write it instead of
> just complaining how bad the wrapper solution actually is.

The solution is called eclasses.

> > Yep, I intended to just have the additional variant of glxinfo directly
> > callable, with a name chosen specifically by the X11 team. Wrapper
> > would be more confusing than beneficial here IMO.
> 
> Ah, you actually want each ebuild to have its own custom hack instead of
> one global solution usable everywhere?

Yes.

> >> To some extent that's what happened to python too :) As a python
> >> maintainer, you could share your thoughts on the topic. python slotting
> >> was intended to make switching between python versions easy but has
> >> been needing wrappers for the python binary.
> > 
> > I'm doing just that. Any kind of wrapping is an increasing mess. I'm
> > still trying to find out good solutions for Python wrapping but there's
> > no such thing. It's always about choosing one evil over the other.
> 
> So you are wrapping python, have not yet found anything better and still
> dont want to wrap abi-specific binaries, while you dont have a better
> solution at hand? Saying no to everything is easy, providing something
> better if you dont like a suggestion is the challenge.

Yes, it is easy and mess-free. Using a cheap hack is mess-full, and is
just asking for trouble which will eventually rise.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Alexis Ballier]

On Mon, 4 Mar 2013 21:49:48 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> On Mon, 4 Mar 2013 11:02:40 +0100
> Alexis Ballier <aballier@gentoo.org> wrote:
> 
> > On Sun, 3 Mar 2013 23:25:03 +0100
> > Michał Górny <mgorny@gentoo.org> wrote:
> > 
> > > On Sun, 3 Mar 2013 18:18:12 +0100
> > > Alexis Ballier <aballier@gentoo.org> wrote:
> > > 
> > > > On Sun, 3 Mar 2013 17:58:26 +0100
> > > > Michał Górny <mgorny@gentoo.org> wrote:
> > > > 
> > > > > What do we need that wrapper for? What does the wrapper do?
> > > > > Does it just rely on custom 'ABI' variable?
> > > > 
> > > > yes -- it must perform some checks though.
> > > 
> > > What kind of checks?
> > 
> > you are called with ABI=sth argv[0] = your name
> 
> I'm afraid that's the first potential point of failure. Relying
> on argv[0] is a poor idea and means that any application calling
> exec() with changed argv[0] on a wrapped binary will fail terribly.

Go tell that to those that wrote busybox, tex or git: They all use
that argv trick :) I'm sure there are plenty of other examples and
don't consider that a poor idea. Note that the wrapper does not lie on
argv: it just makes the binary think it has its original name (by
passing argv as is) while it has been moved to allow parallel
installation of different flavors of it.

> > if argv[0] = abiwrapper -> print some information and exit
> > getenv ABI -> if nothing then set ABI=$DEFAULT_ABI (hardcoded at
> > buildtime of the wrapper)
> > execvp(argv[0]_$ABI, argv)
> > if execvp returns: print a warning, execvp argv[0]_$DEFAULT_ABI
> > 
> > 
> > python-wrapper.c is very likely to have such a logic already.
> > 
> > btw, IMHO ABI is a too common name for such a variable, I'd better
> > name it something like _GENTOO_MULTILIB_ABI so that collisions are
> > much less likely.
> 
> I'm afraid you are actually starting to go the other way.
> 
> Global wrapper means that it is potentially useful to our users.
> However I don't really see people who want to compile 32-bit
> executables think of setting some custom variable like ABI.

They will not: Users will call the name-suffixed version directly. If
they call the non-suffixed version then the above logic will run the
DEFAULT_ABI version. There's no variable to set.
The variable is more so that ebuilds/eclasses can set the variable and
be done. Exactly like EPYTHON I think.


[...]
> > That's why I asked for examples :)
> > qmake may do it, I don't think its sane, but that's life for now.
> > having glxinfo for each abi is useful from a user perspective (it
> > does not need the wrapper a priori though)
> 
> Yep, I intended to just have the additional variant of glxinfo
> directly callable, with a name chosen specifically by the X11 team.
> Wrapper would be more confusing than beneficial here IMO.

Having a wrapper or not, for the end user that does not know the
internal variable name, the end-result will be exactly the same :)

[...]
> > See it something like python-wrapper. EPYTHON=python3.2 python ->
> > runs python3.2 :)
> 
> Err, python-wrapper respects quite complex logic involving EPYTHON,
> and eselect-python. We don't really want people to have eselect-abi,
> do we?

No we don't. The wrapper has nothing to do with that.

[...]
> > To some extent that's what happened to python too :) As a python
> > maintainer, you could share your thoughts on the topic. python
> > slotting was intended to make switching between python versions
> > easy but has been needing wrappers for the python binary.
> 
> I'm doing just that. Any kind of wrapping is an increasing mess. I'm
> still trying to find out good solutions for Python wrapping but
> there's no such thing. It's always about choosing one evil over the
> other.

I don't understand here. What is python-wrapper good for if that's not
to be able to chose which python to run within ebuilds? If that's just
for eselecting the active version, a symlink is likely to be enough.
python-wrapper is an elegant idea for not having to invent a new way
and change everything else: Within an ebuild, setting EPYTHON lets you
be certain that everything will be run by the version you want. You do
not have to mess with packages calling 'python' directly.

[...]
> > packages
> > calling directly binaries having ABI specific output will be broken
> > for multilib too (and I don't know of anyone checking for this
> > while the other two have been long standing issues we tried to
> > fix). We can fix this, but the fact is that we need multi-binary
> > support for users, then the only choice to make is if we want to
> > provide a wrapper so that we do not need to fix build systems or if
> > we want to fix them. The latter is likely preferred but I do not
> > know what kind of work it will involve. It'd help if tommy could
> > provide a list of binaries he needed to wrap through the abiwrapper.
> 
> We don't want it for users in an automagic and fragile way. And
> ebuilds we can fix while migrating.

I think you misunderstood the idea of the wrapper: It seems more robust
than anything else.

Consider the following scenario: 
The binary foo transforms a .foo file into a .h file. .foo files define
types foo32 and foo64. The foo utility, while processing its .foo
files, #define them to, respectively, long and long long on x86
and int and long on amd64. Its output depends on the ABI, so we install
foo and foo_x86 for an amd64/x86 multilib system.

Now I have a package that uses .foo files: it installs several
libraries and binaries and seems to accept FOO='name of the foo utility'
variables. Good, so we set FOO=foo_${ABI} or FOO=foo depending on what
we are building for. All seems good. But one of all its libraries
has an hardcoded foo call. Your package will build fine but x86
binaries using this library will except the foo_x86 types of abi,
however this library was built with foo_amd64 types: the foo64
types will actually be 32bits for this sole library. Calling a
function with a foo64 argument from this library will corrupt the stack
and you will get weird crashes, usually very hard to debug.


This is an extreme and stupid example, but without any wrapper and with
ABI-specific output used within ebuilds and build systems, you have to
be *very* careful that the correct tool is *always* called.
The wrapper is only a way to ensure that after all :)


Alexis.

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Alexis Ballier]

On Mon, 04 Mar 2013 21:17:50 +0100
Thomas Sachau <tommy@gentoo.org> wrote:
> The actual list of packages with wrapped binaries is in the profiles
> dir of the multilib-portage overlay (expect this to be a minimal
> list, there are probably more out there we did not yet catch):

Thanks.

> dev-db/mysql abiwrapper
> dev-lang/perl abiwrapper
> dev-lang/python abiwrapper
> dev-lang/ruby abiwrapper
> dev-libs/gobject-introspection abiwrapper
> dev-libs/libIDL abiwrapper
> dev-scheme/guile abiwrapper
> net-libs/courier-authlib abiwrapper
> dev-qt/qtcore abiwrapper
> dev-qt/qtgui abiwrapper
> media-libs/fontconfig abiwrapper
> www-servers/apache abiwrapper
> x11-libs/pango abiwrapper
> x11-libs/gtk+ abiwrapper


Since you are doing this per-package and not per-binary: Do you happen
to remember/have a list of the binaries that have abi-specific output?

Also, for the qmake example, do you have an example of difference
between qmake32 and qmake64?

Alexis.

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 1685 bytes --]

Alexis Ballier schrieb:
> On Mon, 04 Mar 2013 21:17:50 +0100
> Thomas Sachau <tommy@gentoo.org> wrote:
>> dev-db/mysql abiwrapper
>> dev-lang/perl abiwrapper
>> dev-lang/python abiwrapper
>> dev-lang/ruby abiwrapper
>> dev-libs/gobject-introspection abiwrapper
>> dev-libs/libIDL abiwrapper
>> dev-scheme/guile abiwrapper
>> net-libs/courier-authlib abiwrapper
>> dev-qt/qtcore abiwrapper
>> dev-qt/qtgui abiwrapper
>> media-libs/fontconfig abiwrapper
>> www-servers/apache abiwrapper
>> x11-libs/pango abiwrapper
>> x11-libs/gtk+ abiwrapper
> 
> 
> Since you are doing this per-package and not per-binary: Do you happen
> to remember/have a list of the binaries that have abi-specific output?
> 
> Also, for the qmake example, do you have an example of difference
> between qmake32 and qmake64?
> 
> Alexis.
> 

I dont have a list of binaries, i either noticed myself some
abi-specific behaviour or got user reports for abi-specific behaviour.
As an example i remember, dev-libs/libIDL has a config binary not
matching the usual *-config scheme (libIDL-config-2), so instead of
adding a random list of patterns, i simply added that package to the
list of packages with wrapped binaries. The same applies to mysql, which
has a mysql_config binary.

I am not sure about the target of your qmake question, so as a general
answer:

qmake is something like configure for qmake based build systems. If you
want to see the difference between qmake (32bit) and qmake (64bit), run
a 64bit qmake on a qmake based package and do the same with a 32bit
qmake and check the difference between the 2 runs.

-- 

Thomas Sachau
Gentoo Linux Developer


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 379 bytes --]

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Davide Pesavento]

On Thu, Mar 7, 2013 at 10:59 AM, Thomas Sachau <tommy@gentoo.org> wrote:
> Alexis Ballier schrieb:
>> On Mon, 04 Mar 2013 21:17:50 +0100
>> Thomas Sachau <tommy@gentoo.org> wrote:
>>> dev-db/mysql abiwrapper
>>> dev-lang/perl abiwrapper
>>> dev-lang/python abiwrapper
>>> dev-lang/ruby abiwrapper
>>> dev-libs/gobject-introspection abiwrapper
>>> dev-libs/libIDL abiwrapper
>>> dev-scheme/guile abiwrapper
>>> net-libs/courier-authlib abiwrapper
>>> dev-qt/qtcore abiwrapper
>>> dev-qt/qtgui abiwrapper
>>> media-libs/fontconfig abiwrapper
>>> www-servers/apache abiwrapper
>>> x11-libs/pango abiwrapper
>>> x11-libs/gtk+ abiwrapper
>>
>>
>> Since you are doing this per-package and not per-binary: Do you happen
>> to remember/have a list of the binaries that have abi-specific output?
>>
>> Also, for the qmake example, do you have an example of difference
>> between qmake32 and qmake64?
>>
>> Alexis.
>>
>
> I dont have a list of binaries, i either noticed myself some
> abi-specific behaviour or got user reports for abi-specific behaviour.
> As an example i remember, dev-libs/libIDL has a config binary not
> matching the usual *-config scheme (libIDL-config-2), so instead of
> adding a random list of patterns, i simply added that package to the
> list of packages with wrapped binaries. The same applies to mysql, which
> has a mysql_config binary.
>
> I am not sure about the target of your qmake question, so as a general
> answer:
>
> qmake is something like configure for qmake based build systems. If you
> want to see the difference between qmake (32bit) and qmake (64bit), run
> a 64bit qmake on a qmake based package and do the same with a 32bit
> qmake and check the difference between the 2 runs.
>

Isn't this solvable by passing the right mkspec to a 64-bit qmake?

Thanks,
Pesa

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 607 bytes --]

On Thu, 07 Mar 2013 19:59:35 +0100
Thomas Sachau <tommy@gentoo.org> wrote:

> I am not sure about the target of your qmake question, so as a general
> answer:
> 
> qmake is something like configure for qmake based build systems. If you
> want to see the difference between qmake (32bit) and qmake (64bit), run
> a 64bit qmake on a qmake based package and do the same with a 32bit
> qmake and check the difference between the 2 runs.

Did you try to work with our Qt maintainers to find a solution? Or did
you just wrap it the Fedora way of solving things?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Alexis Ballier]

On Thu, 07 Mar 2013 19:59:35 +0100
Thomas Sachau <tommy@gentoo.org> wrote:
> 
> I dont have a list of binaries, i either noticed myself some
> abi-specific behaviour or got user reports for abi-specific behaviour.
> As an example i remember, dev-libs/libIDL has a config binary not
> matching the usual *-config scheme (libIDL-config-2), so instead of
> adding a random list of patterns, i simply added that package to the
> list of packages with wrapped binaries. The same applies to mysql,
> which has a mysql_config binary.

Ok, so those make perfect sense for being wrapped and should be done
per-binary not per-package: We do not really want to wrap all mysql
binaries just for mysql_config.
Ebuilds should declare the binaries they want to be wrapped, so we can
grep the tree for getting a list of what to fight against if we want a
cleaner multilib system :)

> I am not sure about the target of your qmake question, so as a general
> answer:
> 
> qmake is something like configure for qmake based build systems. If
> you want to see the difference between qmake (32bit) and qmake
> (64bit), run a 64bit qmake on a qmake based package and do the same
> with a 32bit qmake and check the difference between the 2 runs.

Well, I'm asking this because I don't have access to a 32bit qmake
here so a diff of the files generated by the two will be useful :P And I
believe it makes sense to study in details this case to understand
whether we want to wrap it or not. As Davide said, it is likely that
overriding the correct variable may make qmake output not abi-specific.
The difference between g++-64 and g++-32 qmake mkspec is only a -m32 vs
-m64 cflag which I think is overridden elsewhere in our ebuilds so this
should not matter much.

Alexis.

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Alexis Ballier]

On Fri, 8 Mar 2013 05:47:23 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> On Thu, 07 Mar 2013 19:59:35 +0100
> Thomas Sachau <tommy@gentoo.org> wrote:
> 
> > I am not sure about the target of your qmake question, so as a
> > general answer:
> > 
> > qmake is something like configure for qmake based build systems. If
> > you want to see the difference between qmake (32bit) and qmake
> > (64bit), run a 64bit qmake on a qmake based package and do the same
> > with a 32bit qmake and check the difference between the 2 runs.
> 
> Did you try to work with our Qt maintainers to find a solution? Or did
> you just wrap it the Fedora way of solving things?

You cannot blame him for that since the main idea behind
multilib-portage is to be transparent for ebuild developers while
portage will do all the multilib magic behind our back :) This approach
has its pros and cons.

A.

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 2498 bytes --]

Davide Pesavento schrieb:
> On Thu, Mar 7, 2013 at 10:59 AM, Thomas Sachau <tommy@gentoo.org> wrote:
>> Alexis Ballier schrieb:
>>> On Mon, 04 Mar 2013 21:17:50 +0100
>>> Thomas Sachau <tommy@gentoo.org> wrote:
>>>> dev-db/mysql abiwrapper
>>>> dev-lang/perl abiwrapper
>>>> dev-lang/python abiwrapper
>>>> dev-lang/ruby abiwrapper
>>>> dev-libs/gobject-introspection abiwrapper
>>>> dev-libs/libIDL abiwrapper
>>>> dev-scheme/guile abiwrapper
>>>> net-libs/courier-authlib abiwrapper
>>>> dev-qt/qtcore abiwrapper
>>>> dev-qt/qtgui abiwrapper
>>>> media-libs/fontconfig abiwrapper
>>>> www-servers/apache abiwrapper
>>>> x11-libs/pango abiwrapper
>>>> x11-libs/gtk+ abiwrapper
>>>
>>>
>>> Since you are doing this per-package and not per-binary: Do you happen
>>> to remember/have a list of the binaries that have abi-specific output?
>>>
>>> Also, for the qmake example, do you have an example of difference
>>> between qmake32 and qmake64?
>>>
>>> Alexis.
>>>
>>
>> I dont have a list of binaries, i either noticed myself some
>> abi-specific behaviour or got user reports for abi-specific behaviour.
>> As an example i remember, dev-libs/libIDL has a config binary not
>> matching the usual *-config scheme (libIDL-config-2), so instead of
>> adding a random list of patterns, i simply added that package to the
>> list of packages with wrapped binaries. The same applies to mysql, which
>> has a mysql_config binary.
>>
>> I am not sure about the target of your qmake question, so as a general
>> answer:
>>
>> qmake is something like configure for qmake based build systems. If you
>> want to see the difference between qmake (32bit) and qmake (64bit), run
>> a 64bit qmake on a qmake based package and do the same with a 32bit
>> qmake and check the difference between the 2 runs.
>>
> 
> Isn't this solvable by passing the right mkspec to a 64-bit qmake?
> 
> Thanks,
> Pesa
> 
> 

Bug 304971 - dev-qt/qtcore stores machine-specific information in
/usr/share/qt4/mkspecs

in short: since the install location for the mkspec files is not
abi-specific, you only have either 32bit or 64bit ones around, but not both.

So if you get upstream to place those files into an abi-specific dir,
get qmake to use the mkspec files from the right dir for the currently
active ABI and got all older versions of qmake removed from user
systems, then we might not need to anymore wrap qmake.

-- 

Thomas Sachau
Gentoo Linux Developer


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 379 bytes --]

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 2540 bytes --]

Alexis Ballier schrieb:
> On Thu, 07 Mar 2013 19:59:35 +0100
> Thomas Sachau <tommy@gentoo.org> wrote:
>>
>> I dont have a list of binaries, i either noticed myself some
>> abi-specific behaviour or got user reports for abi-specific behaviour.
>> As an example i remember, dev-libs/libIDL has a config binary not
>> matching the usual *-config scheme (libIDL-config-2), so instead of
>> adding a random list of patterns, i simply added that package to the
>> list of packages with wrapped binaries. The same applies to mysql,
>> which has a mysql_config binary.
> 
> Ok, so those make perfect sense for being wrapped and should be done
> per-binary not per-package: We do not really want to wrap all mysql
> binaries just for mysql_config.
> Ebuilds should declare the binaries they want to be wrapped, so we can
> grep the tree for getting a list of what to fight against if we want a
> cleaner multilib system :)

This is possible, but will make the solution more complex or limited
(either the solution does not allow users to opt-in for wrapping all
binaries of a package or you need an additional option for users to
opt-in beside the variable, which includes the needed binaries).

> 
>> I am not sure about the target of your qmake question, so as a general
>> answer:
>>
>> qmake is something like configure for qmake based build systems. If
>> you want to see the difference between qmake (32bit) and qmake
>> (64bit), run a 64bit qmake on a qmake based package and do the same
>> with a 32bit qmake and check the difference between the 2 runs.
> 
> Well, I'm asking this because I don't have access to a 32bit qmake
> here so a diff of the files generated by the two will be useful :P And I
> believe it makes sense to study in details this case to understand
> whether we want to wrap it or not. As Davide said, it is likely that
> overriding the correct variable may make qmake output not abi-specific.
> The difference between g++-64 and g++-32 qmake mkspec is only a -m32 vs
> -m64 cflag which I think is overridden elsewhere in our ebuilds so this
> should not matter much.

If you want an example diff, can you point me to a small qmake based
package?
For the difference: It might work with mkspec files installed per-abi,
but i did not have a close look at it, so cannot say for sure. Maybe
someone from qt herd can give you a clear answer on the abi-specific
content inside qmake, which is used to generate the makefiles.


-- 

Thomas Sachau
Gentoo Linux Developer


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 379 bytes --]

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 3912 bytes --]

On Thu, 7 Mar 2013 17:25:23 +0100
Alexis Ballier <aballier@gentoo.org> wrote:

> On Mon, 4 Mar 2013 21:49:48 +0100
> Michał Górny <mgorny@gentoo.org> wrote:
> 
> > I'm afraid that's the first potential point of failure. Relying
> > on argv[0] is a poor idea and means that any application calling
> > exec() with changed argv[0] on a wrapped binary will fail terribly.
> 
> Go tell that to those that wrote busybox, tex or git: They all use
> that argv trick :) I'm sure there are plenty of other examples and
> don't consider that a poor idea. Note that the wrapper does not lie on
> argv: it just makes the binary think it has its original name (by
> passing argv as is) while it has been moved to allow parallel
> installation of different flavors of it.

We're talking about two different tricks.

Busybox checks argv to support symlinking for a standard tool. With
invalid argv[0], it still runs as the standard busybox binary. You can
use it that way, the 'safe way'.

Unless I'm missing something, git checks argv to support symlinking
into a few dozen tools which is optional. I doubt most of our users
rely on that rather than using the 'git foo' syntax.

A wrapper needs valid argv[0] to find the actual executable. Since
argv[0] is unlikely to contain a path, that's either reinventing some
logic or execvp() which is a bit fragile. Plus symlink resolution loop.
Hard links do not work at all.

And that's just theory. I may be missing some of the dangers.

> > I'm afraid you are actually starting to go the other way.
> > 
> > Global wrapper means that it is potentially useful to our users.
> > However I don't really see people who want to compile 32-bit
> > executables think of setting some custom variable like ABI.
> 
> They will not: Users will call the name-suffixed version directly. If
> they call the non-suffixed version then the above logic will run the
> DEFAULT_ABI version. There's no variable to set.

So there's no benefit from the wrapper to the users. We're talking
about a solution which is purely disadvantageous to them.

> The variable is more so that ebuilds/eclasses can set the variable and
> be done. Exactly like EPYTHON I think.

Python is almost a completely different issue. Most importantly, we
support multiple Python versions, and have to maintain all of them.
That support is extended to all packages in the tree, and is achieved
with no major losses.

We're talking about having multilib for a few chosen packages, and yet
introducing a tree-wide solution to hack-around even less of them.
A hack-around which has visible disadvantages.

> [...]
> > > That's why I asked for examples :)
> > > qmake may do it, I don't think its sane, but that's life for now.
> > > having glxinfo for each abi is useful from a user perspective (it
> > > does not need the wrapper a priori though)
> > 
> > Yep, I intended to just have the additional variant of glxinfo
> > directly callable, with a name chosen specifically by the X11 team.
> > Wrapper would be more confusing than beneficial here IMO.
> 
> Having a wrapper or not, for the end user that does not know the
> internal variable name, the end-result will be exactly the same :)

Unless he symlinked the tool. Then it simply won't work for him anymore.

> This is an extreme and stupid example, but without any wrapper and with
> ABI-specific output used within ebuilds and build systems, you have to
> be *very* careful that the correct tool is *always* called.
> The wrapper is only a way to ensure that after all :)

The main point is to enforce ebuild developers to be careful while
handling multilib. This is the main difference from Tommy's solution --
I don't want people to switch the 'multilib' flag on ebuilds
thoughtlessly and expect everything to work without changing the ebuild.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

[gentoo-dev] Re: Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Steven J. Long]

On Mon, Mar 04, 2013 at 11:21:36PM +0100, Thomas Sachau wrote:
> Michał Górny schrieb:
> > On Mon, 4 Mar 2013 11:02:40 +0100
> > Alexis Ballier <aballier@gentoo.org> wrote:
> >> you are called with ABI=sth argv[0] = your name
> > 
> > I'm afraid that's the first potential point of failure. Relying
> > on argv[0] is a poor idea and means that any application calling exec()
> > with changed argv[0] on a wrapped binary will fail terribly.
> 
> Nobody said, that one cannot create situations, where such a wrapper
> does fail, the point is more an easy and general solution for wrapping
> binaries without implementing different solutions for the same issue in
> every ebuild.
> 
> If you have a better, yet still easy and general solution not requiring
> every ebuild to create something on its own, please write it instead of
> just complaining how bad the wrapper solution actually is.

Why not just custom-compile the wrapper so the path to what it is wrapping
is hard-coded? After all, you've moved the original binary out of PATH so
that the wrapper gets called instead, and you're hardcoding the default ABI
at build-time.

That way it doesn't matter what argv[0] is, it'll always exec either of the
binaries it was installed for, and the user can symlink to it as normal.
(testing for 'abiwrapper' can still be done.)

Admittedly it adds a little complexity, but not much, and it's automated so
once the kinks are worked out it should be fire and forget. Personally, I'd
go for explicitly stating which binaries to wrap, but either way it's hardly
going to add a significant amount of compilation time to a mysql build.

> >> if argv[0] = abiwrapper -> print some information and exit
> >> getenv ABI -> if nothing then set ABI=$DEFAULT_ABI (hardcoded at
> >> buildtime of the wrapper)
> >> execvp(argv[0]_$ABI, argv)
> >> if execvp returns: print a warning, execvp argv[0]_$DEFAULT_ABI
> >>
> >> python-wrapper.c is very likely to have such a logic already.
> >>
> >> btw, IMHO ABI is a too common name for such a variable, I'd better name
> >> it something like _GENTOO_MULTILIB_ABI so that collisions are much less
> >> likely.
> > 
> > I'm afraid you are actually starting to go the other way.
> >
> > Global wrapper means that it is potentially useful to our users.
> > However I don't really see people who want to compile 32-bit
> > executables think of setting some custom variable like ABI.
> 
> Unless you implied, that you want users to compile from hand instead of
> using an ebuild, this makes no sense to me.

I think there's confusion over ebuild and runtime here. But you did mention
getenv("ABI"); in the wrapper, so it's only natural, and I'm not sure what
you mean either ;)

I agree with Thomas that a longer name makes sense in ebuild space (ABI is
far too likely to crop up in a build-system), and with Michal that using
ABI=x86 foo .. is a bit odd. But how else are you going to switch ABI at
runtime if not with an environment variable? --abi=x86 would be nice, but that
impinges on the called binary, and is nowhere near as convenient as a simple
export.

GENTOO_ABI=foo gets my vote so far, fwtw.

-- 
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Alexis Ballier]

On Fri, 8 Mar 2013 17:30:10 +0100
Michał Górny <mgorny@gentoo.org> wrote:

> On Thu, 7 Mar 2013 17:25:23 +0100
> Alexis Ballier <aballier@gentoo.org> wrote:
> 
> > On Mon, 4 Mar 2013 21:49:48 +0100
> > Michał Górny <mgorny@gentoo.org> wrote:
> > 
> > > I'm afraid that's the first potential point of failure. Relying
> > > on argv[0] is a poor idea and means that any application calling
> > > exec() with changed argv[0] on a wrapped binary will fail
> > > terribly.
> > 
> > Go tell that to those that wrote busybox, tex or git: They all use
> > that argv trick :) I'm sure there are plenty of other examples and
> > don't consider that a poor idea. Note that the wrapper does not lie
> > on argv: it just makes the binary think it has its original name (by
> > passing argv as is) while it has been moved to allow parallel
> > installation of different flavors of it.
> 
> We're talking about two different tricks.
> 
> Busybox checks argv to support symlinking for a standard tool. With
> invalid argv[0], it still runs as the standard busybox binary. You can
> use it that way, the 'safe way'.
> 
> Unless I'm missing something, git checks argv to support symlinking
> into a few dozen tools which is optional. I doubt most of our users
> rely on that rather than using the 'git foo' syntax.
> 
> A wrapper needs valid argv[0] to find the actual executable. Since
> argv[0] is unlikely to contain a path, that's either reinventing some
> logic or execvp() which is a bit fragile. Plus symlink resolution
> loop. Hard links do not work at all.
> 
> And that's just theory. I may be missing some of the dangers.

abiwrapper with invalid argv will print a standard help message too :)
I don't see why you consider execvp fragile: Don't tell me you run all
your binaries with full path.

Of course you can break it, but the question is more: Is it likely to
break in a real life situation?

If you consider the argv trick fragile, which is understandable, the
same wrapper idea can be used differently: If you want to wrap a
binary foo, move it to foo_${abi} and build a wrapper with hardcoded
'foo' instead of argv[0] that you install as foo. You can even hardcode
its path and use execv instead of execvp. That would invalidate all
your above objections I think :)

> > > I'm afraid you are actually starting to go the other way.
> > > 
> > > Global wrapper means that it is potentially useful to our users.
> > > However I don't really see people who want to compile 32-bit
> > > executables think of setting some custom variable like ABI.
> > 
> > They will not: Users will call the name-suffixed version directly.
> > If they call the non-suffixed version then the above logic will run
> > the DEFAULT_ABI version. There's no variable to set.
> 
> So there's no benefit from the wrapper to the users. We're talking
> about a solution which is purely disadvantageous to them.

The wrapper is not for users at all...

You can very well have two different ways of installing
multilib-binaries also: one with an array mapping (abi,binary) to
(renamed_binary), without wrapping at all, which is what you seem to
want for glxinfo. Another one with the full wrapping mechanism to make
multilib builds of dependent packages easier/doable.

> > The variable is more so that ebuilds/eclasses can set the variable
> > and be done. Exactly like EPYTHON I think.
> 
> Python is almost a completely different issue. Most importantly, we
> support multiple Python versions, and have to maintain all of them.
> That support is extended to all packages in the tree, and is achieved
> with no major losses.
> 
> We're talking about having multilib for a few chosen packages, and yet
> introducing a tree-wide solution to hack-around even less of them.
> A hack-around which has visible disadvantages.

That's why we are currently discussing how to do it properly :P
Let alone qmake, how do you suggest dealing with *-config scripts?
Some packages are just not written with multilib in mind, or have
to maintain historical compatibility which was not, wrapping the
problematic binaries is one solution to this. I've not seen any other
solution.

> > [...]
> > > > That's why I asked for examples :)
> > > > qmake may do it, I don't think its sane, but that's life for
> > > > now. having glxinfo for each abi is useful from a user
> > > > perspective (it does not need the wrapper a priori though)
> > > 
> > > Yep, I intended to just have the additional variant of glxinfo
> > > directly callable, with a name chosen specifically by the X11
> > > team. Wrapper would be more confusing than beneficial here IMO.
> > 
> > Having a wrapper or not, for the end user that does not know the
> > internal variable name, the end-result will be exactly the same :)
> 
> Unless he symlinked the tool. Then it simply won't work for him
> anymore.

Well, this is a non-issue with the 'one wrapper with hardcoded names
per binary' approach above.

> > This is an extreme and stupid example, but without any wrapper and
> > with ABI-specific output used within ebuilds and build systems, you
> > have to be *very* careful that the correct tool is *always* called.
> > The wrapper is only a way to ensure that after all :)
> 
> The main point is to enforce ebuild developers to be careful while
> handling multilib. This is the main difference from Tommy's solution
> -- I don't want people to switch the 'multilib' flag on ebuilds
> thoughtlessly and expect everything to work without changing the
> ebuild.

Yes, and we want to do it properly. So far the only solution I've seen
for *-config scripts is a wrapper. The qmake case has to be analyzed,
maybe its doable in a better way without wrapper.
Unless someone comes with a sane different solution for *-config
scripts, I still think the wrapper is the only solution to these.

Alexis.

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Alexis Ballier]

On Fri, 08 Mar 2013 15:44:16 +0100
Thomas Sachau <tommy@gentoo.org> wrote:

> Alexis Ballier schrieb:
> > On Thu, 07 Mar 2013 19:59:35 +0100
> > Thomas Sachau <tommy@gentoo.org> wrote:
> >>
> >> I dont have a list of binaries, i either noticed myself some
> >> abi-specific behaviour or got user reports for abi-specific
> >> behaviour. As an example i remember, dev-libs/libIDL has a config
> >> binary not matching the usual *-config scheme (libIDL-config-2),
> >> so instead of adding a random list of patterns, i simply added
> >> that package to the list of packages with wrapped binaries. The
> >> same applies to mysql, which has a mysql_config binary.
> > 
> > Ok, so those make perfect sense for being wrapped and should be done
> > per-binary not per-package: We do not really want to wrap all mysql
> > binaries just for mysql_config.
> > Ebuilds should declare the binaries they want to be wrapped, so we
> > can grep the tree for getting a list of what to fight against if we
> > want a cleaner multilib system :)
> 
> This is possible, but will make the solution more complex or limited
> (either the solution does not allow users to opt-in for wrapping all
> binaries of a package or you need an additional option for users to
> opt-in beside the variable, which includes the needed binaries).

Why do you want _users_ to do that? Ebuilds should do it. We are
talking about multi_lib_ not multi_arch_ after all :)
If you want to wrap all binaries, it may be allowable to have a regexp
there so that '*/bin/*' would wrap them all, but I don't think it is a
good idea to do that.

> >> I am not sure about the target of your qmake question, so as a
> >> general answer:
> >>
> >> qmake is something like configure for qmake based build systems. If
> >> you want to see the difference between qmake (32bit) and qmake
> >> (64bit), run a 64bit qmake on a qmake based package and do the same
> >> with a 32bit qmake and check the difference between the 2 runs.
> > 
> > Well, I'm asking this because I don't have access to a 32bit qmake
> > here so a diff of the files generated by the two will be useful :P
> > And I believe it makes sense to study in details this case to
> > understand whether we want to wrap it or not. As Davide said, it is
> > likely that overriding the correct variable may make qmake output
> > not abi-specific. The difference between g++-64 and g++-32 qmake
> > mkspec is only a -m32 vs -m64 cflag which I think is overridden
> > elsewhere in our ebuilds so this should not matter much.
> 
> If you want an example diff, can you point me to a small qmake based
> package?

media-video/smplayer or media-sound/qjackctl but this one also uses
autoconf before


Alexis.

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 3786 bytes --]

On Sat, 9 Mar 2013 11:10:11 +0100
Alexis Ballier <aballier@gentoo.org> wrote:

> On Fri, 8 Mar 2013 17:30:10 +0100
> Michał Górny <mgorny@gentoo.org> wrote:
> 
> > We're talking about two different tricks.
> > 
> > Busybox checks argv to support symlinking for a standard tool. With
> > invalid argv[0], it still runs as the standard busybox binary. You can
> > use it that way, the 'safe way'.
> > 
> > Unless I'm missing something, git checks argv to support symlinking
> > into a few dozen tools which is optional. I doubt most of our users
> > rely on that rather than using the 'git foo' syntax.
> > 
> > A wrapper needs valid argv[0] to find the actual executable. Since
> > argv[0] is unlikely to contain a path, that's either reinventing some
> > logic or execvp() which is a bit fragile. Plus symlink resolution
> > loop. Hard links do not work at all.
> > 
> > And that's just theory. I may be missing some of the dangers.
> 
> abiwrapper with invalid argv will print a standard help message too :)

Help message won't help users.

  We're very sorry, that won't work. Don't bother reporting it to
  upstream, it's Gentoo-specific hack, they'll just WONTFIX them. Feel
  free to report it here, we may be able to clean the mess up someday.

We're basically talking about opt-out. We introduce a trick which fails
in corner cases by default, hoping those cases won't happen. People
start to rely on it. Then the corner case happens and we need to
opt-out, and we need to fix everything that started to rely
on the common case working.

> I don't see why you consider execvp fragile: Don't tell me you run all
> your binaries with full path.
> 
> Of course you can break it, but the question is more: Is it likely to
> break in a real life situation?

I think that was one of the things that were pointed out in the initial
python-exec implementation. It's not broken that much as a bit
inconsistent by design. As in, wrapper /usr/bin/foo-config doesn't
guarantee that /usr/bin/foo-config-${ARCH} will be called (maybe
something in /usr/local/bin or somewhere completely different?).

> If you consider the argv trick fragile, which is understandable, the
> same wrapper idea can be used differently: If you want to wrap a
> binary foo, move it to foo_${abi} and build a wrapper with hardcoded
> 'foo' instead of argv[0] that you install as foo. You can even hardcode
> its path and use execv instead of execvp. That would invalidate all
> your above objections I think :)

Well, yes. In the worst case we could go that way, although it's a bit
of overkill, don't you think? While we're talking about a few programs
which will actually need it, and even less which will require
modification in more than one place.

> Let alone qmake, how do you suggest dealing with *-config scripts?
> Some packages are just not written with multilib in mind, or have
> to maintain historical compatibility which was not, wrapping the
> problematic binaries is one solution to this. I've not seen any other
> solution.

Choose optimal solution per-package. In case of freetype, the solution
is simple enough: make multilib-aware packages use pkg-config.
fontconfig does exactly that, and in the earlier version it was enough
to export something like FREETYPE_CONFIG='pkg-config freetype2'
(or ac_cv...).

In LLVM, for example, we were considering just calling the proper
config by hand. It's a single multilib package which will need it now,
so it's not that problematic.

If you have any specific -config scripts in mind, please point me
to them. By the way, from what I recall Tommy said that llvm is
basically the only -config script being actually compiled executable.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Alexis Ballier]

On Sun, 10 Mar 2013 14:42:43 +0100
Michał Górny <mgorny@gentoo.org> wrote:
> > If you consider the argv trick fragile, which is understandable, the
> > same wrapper idea can be used differently: If you want to wrap a
> > binary foo, move it to foo_${abi} and build a wrapper with hardcoded
> > 'foo' instead of argv[0] that you install as foo. You can even
> > hardcode its path and use execv instead of execvp. That would
> > invalidate all your above objections I think :)
> 
> Well, yes. In the worst case we could go that way, although it's a bit
> of overkill, don't you think? While we're talking about a few programs
> which will actually need it, and even less which will require
> modification in more than one place.

You are starting to convince me a bin wrapper may not be needed. qt
will have to be converted to multilib at some point and we'll be able
to discuss if we want this or not at that point. We have a generic
solution (the bin wrapper) but should likely prefer the longer and more
difficult road of making packages more multilib-aware. Let's see the
outcome when we'll be at it.

> > Let alone qmake, how do you suggest dealing with *-config scripts?
> > Some packages are just not written with multilib in mind, or have
> > to maintain historical compatibility which was not, wrapping the
> > problematic binaries is one solution to this. I've not seen any
> > other solution.
> 
> Choose optimal solution per-package. In case of freetype, the solution
> is simple enough: make multilib-aware packages use pkg-config.
> fontconfig does exactly that, and in the earlier version it was enough
> to export something like FREETYPE_CONFIG='pkg-config freetype2'
> (or ac_cv...).

In this case there's also cmake, and all the other build system types,
maybe even custom ones. pkg-config should be prefered over *-config
scripts in 99% of the cases I'd say; however, setting FREETYPE_CONFIG
isn't less of a hack than the bin wrapper if patches for using
pkg-config to detect freetype are not merged upstream.


Alexis.

Re: [gentoo-dev] Re: [RFC] multilib-build.eclass and restricting unsupported ABIs

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1745 bytes --]

On Fri, 15 Mar 2013 11:32:31 +0100
Alexis Ballier <aballier@gentoo.org> wrote:

> On Sun, 10 Mar 2013 14:42:43 +0100
> Michał Górny <mgorny@gentoo.org> wrote:
> > > Let alone qmake, how do you suggest dealing with *-config scripts?
> > > Some packages are just not written with multilib in mind, or have
> > > to maintain historical compatibility which was not, wrapping the
> > > problematic binaries is one solution to this. I've not seen any
> > > other solution.
> > 
> > Choose optimal solution per-package. In case of freetype, the solution
> > is simple enough: make multilib-aware packages use pkg-config.
> > fontconfig does exactly that, and in the earlier version it was enough
> > to export something like FREETYPE_CONFIG='pkg-config freetype2'
> > (or ac_cv...).
> 
> In this case there's also cmake, and all the other build system types,
> maybe even custom ones. pkg-config should be prefered over *-config
> scripts in 99% of the cases I'd say; however, setting FREETYPE_CONFIG
> isn't less of a hack than the bin wrapper if patches for using
> pkg-config to detect freetype are not merged upstream.

Yes, the freetype case has already been solved through removing
the trouble-making part of header (it wasn't actually ever used) which
made the headers compatible.

In the longer run, we need more tools for that. During the weekend I'll
attempt to write an acceptably good header wrapping function.

We may also want a simpler ignore/exclude option but not sure how to do
it properly. Certainly I wouldn't want the function to dumbly ignore
the whole file, esp. considering that its contents (and thus
ABI-specific data) can change over time (versions).

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 966 bytes --]