[gentoo-dev] spec draft for cross-compile support in future EAPI (EAPI-5)

[gentoo-dev] spec draft for cross-compile support in future EAPI (EAPI-5)

[Thomas Sachau]

[-- Attachment #1.1: Type: text/plain, Size: 203 bytes --]

Since i am not that sure about my ability to write formal specs, i am
presenting my first draft for further review and suggestions for
improvement.


-- 

Thomas Sachau
Gentoo Linux Developer


[-- Attachment #1.2: multilib-portage-spec-draft --]
[-- Type: text/plain, Size: 5177 bytes --]

For amd64 users, there is sometimes the issue, that they need 32bit libs for certain packages (e.g.
wine or many binary-only packages). Currently, they only get them prepackaged in binary form with
the emul-linux-x86-* packages. But those packages have a few issues (list does not have to be complete):

-they only contain a limited set of 32bit packages
-they are precompiled, so the user cannot define his own flags
-they have to be manually maintained and updated


So the idea was to add the ability to compile 32bit packages with support from the package manager,
so there is no need for additional packages to maintain. After this was originally implemented, it
was further extended to allow cross-compilation for other targets, not only limited to 32bit packages.


The basic way, how this should work:

1. Check for the current profile being multilib, this means, that the MULTILIB_ABIS var exists and has more then 1 (space seperated) value. Additionally, the DEFAULT_ABI var has to be defined and its content should be part of the MULTILIB_ABIS var. Only continue with the following steps, if this is true
2.1 Take the entries from MULTILIB_ABIS var and add a USE flag for each of them in the form of multilib_abi_$value
2.2 Add "abiwrapper" as a USE flag
3. Check, if the package has USE=multilib enabled. Only continue with the following steps, if this is false.
4. Add a use dependency for each USE flag added in step 2 for all dependencies except those defined in a space seperated list of the NO_AUTO_FLAG var. The dependency should then look like category/package[multilib_abi_$value?]
5. Find the first enabled USE flag from the list of step 2, start with multilib_abi_$DEFAULT_ABI during the search. If none is enabled, use multilib_abi_$DEFAULT_ABI. In the following, $ABI will reference the $value part of the selected USE flag.
6. Before the pkg_setup phase is executed, setup the environment as following:
	-export CHOST with $CHOST_$DEFAULT_ABI
	-export $CC with $CHOST-gcc
	-export $CXX with $CHOST-g++
	-export $FC with $CHOST-gfortran
	-export $CHOST with $CHOST_$ABI
	-export $CBUILD with $CHOST_$ABI
	-export $CDEFINE with $CDEFINE_$ABI
	-export $CCASFLAGS with ${CCASFLAGS:-${CFLAGS}} and append $CFLAGS_$ABI
	-export $CFLAGS with $CFLAGS and append $CFLAGS_$ABI
	-export $CPPFLAGS with $CPPFLAGS and append $CPPFLAGS_$ABI
	-export $CXXFLAGS with $CXXFLAGS and append $CFLAGS_$ABI
	-export $FCFLAGS with $FCFLAGS and append $CFLAGS_$ABI
	-export $FFLAGS with $FFLAGS and append $CFLAGS_$ABI
	-export $ASFLAGS with $ASFLAGS and append $ASFLAGS_$ABI
	-export $LDFLAGS with $LDFLAGS and append $CFLAGS_$ABI
	-export $PKG_CONFIG_PATH with /usr/$LIBDIR_$ABI/pkgconfig

7. After src_install is finished:
7.1 Execute the following or equivalent code (prep_ml_binaries is a function from multilib-portage from [1]):
prep_ml_binaries $(find "${D}"usr/bin "${D}"usr/sbin "${D}"bin "${D}"sbin -type f -name '*-config' 2>/dev/null)
7.2 If USE flag abiwrapper is enabled, execute:
local noabi=()
for i in ${MULTILIB_ABIS}; do
	noabi+=( ! -name '*-'${i} )
done
if use abiwrapper ; then
	for i in $(find "${D}"usr/bin/ "${D}"usr/sbin "${D}"bin "${D}"sbin -type f ${noabi[@]} 2>/dev/null); do
		prep_ml_binaries "${i}"
	done
fi
7.3. After src_install, do the following checks:
	-is $Dusr/include not empty
	-is $usr/$LIBDIR_$ABI not empty
	-is abiwrapper USE flag enabled and any of /bin /usr/bin /sbin /usr/sbin not empty
If any of those checks is true, save the content of $D to $D.$ABI. Check if another USE flag from step 2 is enabled, if no, then continue with step 8, otherwise reset everything to the state before pkg_setup. Find the next enabled USE flag from step 5 and continue with that on step 6.
8. If there have been at least 2 USE flags from step 2 enabled, check if the header files between the different target $ABIs differ. Move the files, that dont differ, back to $D. If files differ, they are moved to $Dusr/include/gentoo-multilib/$ABI and the following or equivalent code is executed in $D ($ALL_ABIS is the content of $MULTILIB_ABIS, for which the USE flags from step 2 have been enabled) (_create_abi_includes is a function from multilib-portage from [1]):
set --
for diffabi in ${ALL_ABIS}; do
	local define_var=CDEFINE_${diffabi}
	set -- "$@" "${!define_var}:${dir}/gentoo-multilib/${diffabi}"
done
_create_abi_includes "$@"
Now move the rest of the content of the temporary dirs back into $D with the content of $D.$DEFAULT_ABI being the last.
9. pkg_preinst, pkg_postinst, pkg_prerm and pkg_postrm are executed within a loop of the following form
for each enabled USE flag from step 2 do
	setup environment as described in step 6
	execute phase
done

[1]: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=blob;f=bin/auto-multilib.sh;hb=refs/heads/multilib

Optional: The code from step 6, 7.1, 7.2 and 8 (for managing the headers) could be moved into an eclass. Then the PM executes this or an equivalent internal function with newer versions of that eclass overwrite the internal functions. That way, those parts could be updated without the need to update change 
the PMS or to wait for another EAPI for an update/change.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 380 bytes --]

[gentoo-dev] Re: spec draft for cross-compile support in future EAPI (EAPI-5)

[Duncan]

Thomas Sachau posted on Sat, 16 Jun 2012 12:31:40 +0200 as excerpted:

> Since i am not that sure about my ability to write formal specs, i am
> presenting my first draft for further review and suggestions for
> improvement.

Just a format suggestion.  Call it nitpicky if you want, and yes, my 
client isn't perfect, but I'm sure people with a bit of experience 
writing such specs will tell you I'm not alone...

Several of your points ended up as very long single lines.  My client can 
wrap, but that wraps the points as well (so for example 2.1 starts in the 
middle of a line).  So I was left with the choice to either massively 
horizontally scroll, or of trying to figure out where one point ended and 
another began, since wrapping it... /wrapped/ it, so points appeared in 
the middle of a line.

Please:

* If you use long lines, leave a vertical space (blank line) between 
points so when a client wraps them, they wrap as individual paragraphs.

* Alternatively, wrap at something sensible. (The traditional wrap for 
posting is 72 chars or so, 80 minus a few to allow a few levels of 
quoting without rewrap.  I wouldn't complain at 90, but if you're going 
to bother, you might as well go the standard route and avoid further 
issues.)

Long lines as paragraphs would probably be easier especially early in the 
process when you're modifying a lot, but you still risk (even more) 
limited clients having issues with it.  YMMV.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: spec draft for cross-compile support in future EAPI (EAPI-5)

[Thomas Sachau]

[-- Attachment #1.1: Type: text/plain, Size: 1748 bytes --]

Duncan schrieb:
> Thomas Sachau posted on Sat, 16 Jun 2012 12:31:40 +0200 as excerpted:
> 
>> Since i am not that sure about my ability to write formal specs, i am
>> presenting my first draft for further review and suggestions for
>> improvement.
> 
> Just a format suggestion.  Call it nitpicky if you want, and yes, my 
> client isn't perfect, but I'm sure people with a bit of experience 
> writing such specs will tell you I'm not alone...
> 
> Several of your points ended up as very long single lines.  My client can 
> wrap, but that wraps the points as well (so for example 2.1 starts in the 
> middle of a line).  So I was left with the choice to either massively 
> horizontally scroll, or of trying to figure out where one point ended and 
> another began, since wrapping it... /wrapped/ it, so points appeared in 
> the middle of a line.
> 
> Please:
> 
> * If you use long lines, leave a vertical space (blank line) between 
> points so when a client wraps them, they wrap as individual paragraphs.
> 
> * Alternatively, wrap at something sensible. (The traditional wrap for 
> posting is 72 chars or so, 80 minus a few to allow a few levels of 
> quoting without rewrap.  I wouldn't complain at 90, but if you're going 
> to bother, you might as well go the standard route and avoid further 
> issues.)
> 
> Long lines as paragraphs would probably be easier especially early in the 
> process when you're modifying a lot, but you still risk (even more) 
> limited clients having issues with it.  YMMV.
> 

I suggest you look for a better client to handle the line wrapping
better. ;-) In the meantime, the same file attached with wrapped lines.

-- 

Thomas Sachau
Gentoo Linux Developer



[-- Attachment #1.2: multilib-portage-spec-draft --]
[-- Type: text/plain, Size: 5179 bytes --]

For amd64 users, there is sometimes the issue, that they need 32bit libs for
certain packages (e.g. wine or many binary-only packages). Currently,
they only get them prepackaged in binary form with the emul-linux-x86-*
packages. But those packages have a few issues (list does not have to be
complete):

-they only contain a limited set of 32bit packages
-they are precompiled, so the user cannot define his own flags
-they have to be manually maintained and updated


So the idea was to add the ability to compile 32bit packages with support from
the package manager, so there is no need for additional packages to
maintain. After this was originally implemented, it was further extended
to allow cross-compilation for other targets, not only limited to 32bit
packages.


The basic way, how this should work:

1. Check for the current profile being multilib, this means, that the
MULTILIB_ABIS var exists and has more then 1 (space seperated) value.
Additionally, the DEFAULT_ABI var has to be defined and its content should
be part of the MULTILIB_ABIS var. Only continue with the following steps,
if this is true
2.1 Take the entries from MULTILIB_ABIS var and add a USE flag for each of them
in the form of multilib_abi_$value
2.2 Add "abiwrapper" as a USE flag
3. Check, if the package has USE=multilib enabled. Only continue with the
following steps, if this is false.
4. Add a use dependency for each USE flag added in step 2 for all dependencies
except those defined in a space seperated list of the NO_AUTO_FLAG var. The
dependency should then look like category/package[multilib_abi_$value?]
5. Find the first enabled USE flag from the list of step 2, start with
multilib_abi_$DEFAULT_ABI during the search. If none is enabled, use
multilib_abi_$DEFAULT_ABI. In the following, $ABI will reference the $value
part of the selected USE flag.
6. Before the pkg_setup phase is executed, setup the environment as following:
	-export CHOST with $CHOST_$DEFAULT_ABI
	-export $CC with $CHOST-gcc
	-export $CXX with $CHOST-g++
	-export $FC with $CHOST-gfortran
	-export $CHOST with $CHOST_$ABI
	-export $CBUILD with $CHOST_$ABI
	-export $CDEFINE with $CDEFINE_$ABI
	-export $CCASFLAGS with ${CCASFLAGS:-${CFLAGS}} and append $CFLAGS_$ABI
	-export $CFLAGS with $CFLAGS and append $CFLAGS_$ABI
	-export $CPPFLAGS with $CPPFLAGS and append $CPPFLAGS_$ABI
	-export $CXXFLAGS with $CXXFLAGS and append $CFLAGS_$ABI
	-export $FCFLAGS with $FCFLAGS and append $CFLAGS_$ABI
	-export $FFLAGS with $FFLAGS and append $CFLAGS_$ABI
	-export $ASFLAGS with $ASFLAGS and append $ASFLAGS_$ABI
	-export $LDFLAGS with $LDFLAGS and append $CFLAGS_$ABI
	-export $PKG_CONFIG_PATH with /usr/$LIBDIR_$ABI/pkgconfig

7. After src_install is finished:
7.1 Execute the following or equivalent code (prep_ml_binaries is a function
from multilib-portage from [1]):
prep_ml_binaries $(find "${D}"usr/bin "${D}"usr/sbin "${D}"bin "${D}"sbin -type f -name '*-config' 2>/dev/null)
7.2 If USE flag abiwrapper is enabled, execute:
local noabi=()
for i in ${MULTILIB_ABIS}; do
	noabi+=( ! -name '*-'${i} )
done
if use abiwrapper ; then
	for i in $(find "${D}"usr/bin/ "${D}"usr/sbin "${D}"bin "${D}"sbin -type f ${noabi[@]} 2>/dev/null); do
		prep_ml_binaries "${i}"
	done
fi
7.3. After src_install, do the following checks:
	-is $Dusr/include not empty
	-is $usr/$LIBDIR_$ABI not empty
	-is abiwrapper USE flag enabled and any of /bin /usr/bin /sbin
	 /usr/sbin not empty
If any of those checks is true, save the content of $D to $D.$ABI. Check if
another USE flag from step 2 is enabled, if no, then continue with step 8,
otherwise reset everything to the state before pkg_setup. Find the next enabled
USE flag from step 5 and continue with that on step 6.
8. If there have been at least 2 USE flags from step 2 enabled, check if the
header files between the different target $ABIs differ. Move the files, that
dont differ, back to $D. If files differ, they are moved to 
$Dusr/include/gentoo-multilib/$ABI and the following or equivalent code is
executed in $D ($ALL_ABIS is the content of $MULTILIB_ABIS, for which the USE
flags from step 2 have been enabled) (_create_abi_includes is a function from
multilib-portage from [1]):
set --
for diffabi in ${ALL_ABIS}; do
	local define_var=CDEFINE_${diffabi}
	set -- "$@" "${!define_var}:${dir}/gentoo-multilib/${diffabi}"
done
_create_abi_includes "$@"
Now move the rest of the content of the temporary dirs back into $D with the
content of $D.$DEFAULT_ABI being the last.
9. pkg_preinst, pkg_postinst, pkg_prerm and pkg_postrm are executed within a
loop of the following form
for each enabled USE flag from step 2 do
	setup environment as described in step 6
	execute phase
done

[1]: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=blob;f=bin/auto-multilib.sh;hb=refs/heads/multilib

Optional: The code from step 6, 7.1, 7.2 and 8 (for managing the headers) could
be moved into an eclass. Then the PM executes this or an equivalent internal
function with newer versions of that eclass overwrite the internal functions.
That way, those parts could be updated without the need to update change
the PMS or to wait for another EAPI for an update/change.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 380 bytes --]

[gentoo-dev] Re: spec draft for cross-compile support in future EAPI (EAPI-5)

[Duncan]

Thomas Sachau posted on Sun, 17 Jun 2012 14:02:26 +0200 as excerpted:

> I suggest you look for a better client to handle the line wrapping
> better. ;-) In the meantime, the same file attached with wrapped lines.

Thanks.  (And I'm actually involved upstream tho not as a coder, so 
maybe...)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] spec draft for cross-compile support in future EAPI (EAPI-5)

[Luca Barbato]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/16/2012 12:31 PM, Thomas Sachau wrote:
> Since i am not that sure about my ability to write formal specs, i am
> presenting my first draft for further review and suggestions for
> improvement.

Currently I'm experimenting with evil hack with qemu-static (hopefully
that would make my student life happier later).

What would be nice on a full multilib/multiabi system is to have

/lib/ld-linux-foo-arch.so.2 for each active architecture

/usr/$CTUPLE/ as destdir for each of them

PATH and ld.so.conf pointing to them accordingly.

and possibly split RDEPEND/DEPEND to have HDEPEND to list build
dependencies that need to be run on host.

lu

- -- 

Luca Barbato
Gentoo/linux
http://dev.gentoo.org/~lu_zero

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/gvVYACgkQ6Ex4woTpDjQlaQCg1W/PZxOo/2EDktU0XR6/48nf
9RcAoOg40vFoSIX7gNpD82qdym54MB4m
=hEyq
-----END PGP SIGNATURE-----

Re: [gentoo-dev] spec draft for cross-compile support in future EAPI (EAPI-5)

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 1139 bytes --]

Luca Barbato schrieb:
> On 06/16/2012 12:31 PM, Thomas Sachau wrote:
>> Since i am not that sure about my ability to write formal specs, i am
>> presenting my first draft for further review and suggestions for
>> improvement.
> 
> Currently I'm experimenting with evil hack with qemu-static (hopefully
> that would make my student life happier later).
> 
> What would be nice on a full multilib/multiabi system is to have
> 
> /lib/ld-linux-foo-arch.so.2 for each active architecture

This should be under control by the libc ebuild/package, not controlled
by the package manager.

> 
> /usr/$CTUPLE/ as destdir for each of them

if you mean $CTUPLE as an arch, so that every arch should have all files
for each package, then something like that does not exist. It may be
possible with an extended multilib-portage, but since i did not think
about it nor tested it, i cant say for sure.

> and possibly split RDEPEND/DEPEND to have HDEPEND to list build
> dependencies that need to be run on host.

What should the difference between DEPEND and HDEPEND be?

-- 

Thomas Sachau
Gentoo Linux Developer




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 380 bytes --]

Re: [gentoo-dev] Re: spec draft for cross-compile support in future EAPI (EAPI-5)

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 1974 bytes --]

Thomas Sachau schrieb:
> Duncan schrieb:
>> Thomas Sachau posted on Sat, 16 Jun 2012 12:31:40 +0200 as excerpted:
>>
>>> Since i am not that sure about my ability to write formal specs, i am
>>> presenting my first draft for further review and suggestions for
>>> improvement.
>>
>> Just a format suggestion.  Call it nitpicky if you want, and yes, my 
>> client isn't perfect, but I'm sure people with a bit of experience 
>> writing such specs will tell you I'm not alone...
>>
>> Several of your points ended up as very long single lines.  My client can 
>> wrap, but that wraps the points as well (so for example 2.1 starts in the 
>> middle of a line).  So I was left with the choice to either massively 
>> horizontally scroll, or of trying to figure out where one point ended and 
>> another began, since wrapping it... /wrapped/ it, so points appeared in 
>> the middle of a line.
>>
>> Please:
>>
>> * If you use long lines, leave a vertical space (blank line) between 
>> points so when a client wraps them, they wrap as individual paragraphs.
>>
>> * Alternatively, wrap at something sensible. (The traditional wrap for 
>> posting is 72 chars or so, 80 minus a few to allow a few levels of 
>> quoting without rewrap.  I wouldn't complain at 90, but if you're going 
>> to bother, you might as well go the standard route and avoid further 
>> issues.)
>>
>> Long lines as paragraphs would probably be easier especially early in the 
>> process when you're modifying a lot, but you still risk (even more) 
>> limited clients having issues with it.  YMMV.
>>
> 
> I suggest you look for a better client to handle the line wrapping
> better. ;-) In the meantime, the same file attached with wrapped lines.
> 

Since there is again no response at all, it seems like everyone is ok
with this, so i will propose to add this to the next council agenda for
EAPI-5 addition.

-- 

Thomas Sachau
Gentoo Linux Developer




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 380 bytes --]

Re: [gentoo-dev] Re: spec draft for cross-compile support in future EAPI (EAPI-5)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 292 bytes --]

On Tue, 19 Jun 2012 20:16:39 +0200
Thomas Sachau <tommy@gentoo.org> wrote:
> Since there is again no response at all, it seems like everyone is ok
> with this, so i will propose to add this to the next council agenda
> for EAPI-5 addition.

Got a diff for PMS?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: spec draft for cross-compile support in future EAPI (EAPI-5)

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 837 bytes --]

Ciaran McCreesh schrieb:
> On Tue, 19 Jun 2012 20:16:39 +0200
> Thomas Sachau <tommy@gentoo.org> wrote:
>> Since there is again no response at all, it seems like everyone is ok
>> with this, so i will propose to add this to the next council agenda
>> for EAPI-5 addition.
> 
> Got a diff for PMS?
> 

Last time you only requested enough details for implementation, you did
not require a PMS diff, so i wrote more details for the implementation.

If you, Brian (for pkgcore) and zmedico accept this for EAPI-5, i might
look into creating a diff against PMS but until then, i dont want to
waste my time, especially since noone commented on the implementation
details or the technical details and any change would require even more
work to rework/adjust the PMS diff.

-- 

Thomas Sachau
Gentoo Linux Developer




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 380 bytes --]

Re: [gentoo-dev] Re: spec draft for cross-compile support in future EAPI (EAPI-5)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1548 bytes --]

On Tue, 19 Jun 2012 20:54:07 +0200
Thomas Sachau <tommy@gentoo.org> wrote:
> Ciaran McCreesh schrieb:
> > On Tue, 19 Jun 2012 20:16:39 +0200
> > Thomas Sachau <tommy@gentoo.org> wrote:
> >> Since there is again no response at all, it seems like everyone is
> >> ok with this, so i will propose to add this to the next council
> >> agenda for EAPI-5 addition.
> > 
> > Got a diff for PMS?
> 
> Last time you only requested enough details for implementation, you
> did not require a PMS diff, so i wrote more details for the
> implementation.
> 
> If you, Brian (for pkgcore) and zmedico accept this for EAPI-5, i
> might look into creating a diff against PMS but until then, i dont
> want to waste my time, especially since noone commented on the
> implementation details or the technical details and any change would
> require even more work to rework/adjust the PMS diff.

Here's the thing: I doubt the lack of feedback is down to everyone
being happy with the proposal. I strongly suspect it's because people
look at it and go "huh?". You've provided exactly the kind of
information that no-one cares about (e.g. long lists of variable names,
which will probably just come from a var in profiles), and none of what
really matters.

I think you'll get a better response if you write this up in GLEP form
(i.e. motivation, examples etc) to describe to ebuild developers what's
in it for them, and as a diff against PMS so that package mangler
authors can work out exactly what you're saying.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: spec draft for cross-compile support in future EAPI (EAPI-5)

[Brian Harring]

On Tue, Jun 19, 2012 at 08:54:07PM +0200, Thomas Sachau wrote:
> Ciaran McCreesh schrieb:
> > On Tue, 19 Jun 2012 20:16:39 +0200
> > Thomas Sachau <tommy@gentoo.org> wrote:
> >> Since there is again no response at all, it seems like everyone is ok
> >> with this, so i will propose to add this to the next council agenda
> >> for EAPI-5 addition.
> > 
> > Got a diff for PMS?
> > 
> 
> Last time you only requested enough details for implementation, you did
> not require a PMS diff, so i wrote more details for the implementation.
> 
> If you, Brian (for pkgcore) and zmedico accept this for EAPI-5, i might
> look into creating a diff against PMS but until then, i dont want to
> waste my time, especially since noone commented on the implementation
> details or the technical details and any change would require even more
> work to rework/adjust the PMS diff.

You need a glep here frankly; per the norm, if you want things to move 
faster, then put in time- aka, generate a patch against PMS, write a 
patch for portage, etc, you get the idea.  The bit re: a PMS patch is 
mostly that in looking at your proposal... well, I personally don't 
want to write that patch (nor do I suspect ulm/ciaran do either).

One thing to note; this has been posted for all of 2-3 days; that's 
not exactly much time for 1) people to comment, 2) people to frankly 
comprehend the quite dense description you wrote.

Please write a glep covering details of the implementation, 
background, preferablly why this route over others.  Bluntly... clue 
everyone else in rather than hoping they'll just sign off on a fairly 
opaque list of things. :)  It'll be useful for dev education also- 
which is a bit of a requirement for stuff of this sort considering 
it's not going to be a magic deploy/shit works everywhere situation I 
suspect.

Would also be useful getting commentary from crossdev folk considering 
your solution is intended to be (best I can tell) full cross 
compilation support, and they've been leading that front for many, 
many years.

Cheers-
~brian

Re: [gentoo-dev] Re: spec draft for cross-compile support in future EAPI (EAPI-5)

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 1918 bytes --]

Brian Harring schrieb:
> On Tue, Jun 19, 2012 at 08:54:07PM +0200, Thomas Sachau wrote:
>> Ciaran McCreesh schrieb:
>>> On Tue, 19 Jun 2012 20:16:39 +0200
>>> Thomas Sachau <tommy@gentoo.org> wrote:
>>>> Since there is again no response at all, it seems like everyone is ok
>>>> with this, so i will propose to add this to the next council agenda
>>>> for EAPI-5 addition.
>>>
>>> Got a diff for PMS?
>>>
>>
>> Last time you only requested enough details for implementation, you did
>> not require a PMS diff, so i wrote more details for the implementation.
>>
>> If you, Brian (for pkgcore) and zmedico accept this for EAPI-5, i might
>> look into creating a diff against PMS but until then, i dont want to
>> waste my time, especially since noone commented on the implementation
>> details or the technical details and any change would require even more
>> work to rework/adjust the PMS diff.
> 
> You need a glep here frankly; per the norm, if you want things to move 
> faster, then put in time- aka, generate a patch against PMS, write a 
> patch for portage, etc, you get the idea.

Since multilib-portage code is just a git branch for the portage code, i
effectively have a patch for portage. ;-)


Before i start investing more time, a question to the PM developers:

Do you prefer having everything hardcoded in PMS or can you accept
outsourcing bigger code pieces into some sort of eclass (i am thinking
about some external code base, which can be duplicated by the package
manager with internal code, but has to be used, if the external eclass
has a newer version/revision then the duplicated internal code)?

I am especially thinking about the setup of the environment and the code
details for the wrappers for binaries and headers, hardcoding those
details into PMS makes it hard to change/fix issues later on.

-- 

Thomas Sachau
Gentoo Linux Developer




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 380 bytes --]

Re: [gentoo-dev] Re: spec draft for cross-compile support in future EAPI (EAPI-5)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 886 bytes --]

On Tue, 19 Jun 2012 23:07:01 +0200
Thomas Sachau <tommy@gentoo.org> wrote:
> Do you prefer having everything hardcoded in PMS or can you accept
> outsourcing bigger code pieces into some sort of eclass (i am thinking
> about some external code base, which can be duplicated by the package
> manager with internal code, but has to be used, if the external eclass
> has a newer version/revision then the duplicated internal code)?

The package manager mustn't require any particular eclass to be
present, and there mustn't be duplication between eclasses and the
package manager.

> I am especially thinking about the setup of the environment and the
> code details for the wrappers for binaries and headers, hardcoding
> those details into PMS makes it hard to change/fix issues later on.

Sounds like you haven't really got a clean design then.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] spec draft for cross-compile support in future EAPI (EAPI-5)

[Luca Barbato]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/19/2012 08:14 PM, Thomas Sachau wrote:

>> and possibly split RDEPEND/DEPEND to have HDEPEND to list build
>> dependencies that need to be run on host.
> 
> What should the difference between DEPEND and HDEPEND be?

Not library but program that have to run. Think about generators.

lu




- -- 

Luca Barbato
Gentoo/linux
http://dev.gentoo.org/~lu_zero

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/hfm8ACgkQ6Ex4woTpDjRrtgCfXm2/b3FlZldoKfbVoNA8DKOf
Sx4AoIAy1WEHulrBY3LsDxIyv6JUMjPV
=MCO8
-----END PGP SIGNATURE-----

GLEP draf for cross-compile support in multilib profiles (was: Re: [gentoo-dev] Re: spec draft for cross-compile support in future EAPI (EAPI-5))

[Thomas Sachau]

[-- Attachment #1.1: Type: text/plain, Size: 2243 bytes --]

Brian Harring schrieb:
> 
> You need a glep here frankly; per the norm, if you want things to move 
> faster, then put in time- aka, generate a patch against PMS, write a 
> patch for portage, etc, you get the idea.  The bit re: a PMS patch is 
> mostly that in looking at your proposal... well, I personally don't 
> want to write that patch (nor do I suspect ulm/ciaran do either).

As written earlier, a patch for portage exists already (just the diff
between multilib and master branch of portage).
Now, in addition, a GLEP draft for cross-compile support in
multilib-portage.

> 
> One thing to note; this has been posted for all of 2-3 days; that's 
> not exactly much time for 1) people to comment, 2) people to frankly 
> comprehend the quite dense description you wrote.

This may be true, but as you may have seen afterwards, beside the
responses to my mail about planned council asking, there have not been
any further responses, so the relatively short time since my mail doesnt
seem like the reason for missing responses.

> 
> Please write a glep covering details of the implementation, 
> background, preferablly why this route over others.  Bluntly... clue 
> everyone else in rather than hoping they'll just sign off on a fairly 
> opaque list of things. :)  It'll be useful for dev education also- 
> which is a bit of a requirement for stuff of this sort considering 
> it's not going to be a magic deploy/shit works everywhere situation I 
> suspect.

See attached GLEP draft.

> 
> Would also be useful getting commentary from crossdev folk considering 
> your solution is intended to be (best I can tell) full cross 
> compilation support, and they've been leading that front for many, 
> many years.

There is an important difference between my cross-compile support for
multilib profiles and crossdev: My suggestion works with the default
toolchain, which itself is able to cross-compile for other targets (like
the amd64/multilib gcc is able to create 32bit libs), while crossdev is
using a different toolchain for the targets.

Of course, if someone from crossdev maintainers wants to comment, he is
free to do so.


-- 

Thomas Sachau
Gentoo Linux Developer



[-- Attachment #1.2: glep-XXXX.txt --]
[-- Type: text/plain, Size: 5358 bytes --]

GLEP: XXX
Title: Crosscompile support for multilib profiles
Version: $Revision: 1.4 $
Last-Modified: $Date: 2003/07/19 12:09:20 $
Author:	Thomas Sachau <tommy@gentoo.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 24 Jun 2012
Post-History: 2-Jun-2003


Abstract
========

Currently, multilib profiles like amd64 do support other targets, but there
is no simple way to build packages for those seperate targets.  A workaround
for this issue are emul packages, which contain precompiled 32bit libs.  
This GLEP extends the package manager to allow the user to build packages
from source for targets supported by the profile and toolchain.  


Motivation
==========

There is no way to build packages for other then the default target also
the profile and toolchain do support them.  For amd64, there are precompiled
emul packages, which only support a predefined subset of packages, are not
compiled with the respect for the user USE or comiler flags and tend to
become outdated due to the amount of packages they include.  


Specification
=============

This GLEP will extend the functionality of package managers with a future
EAPI.  With this EAPI, users will be able to select additional or different
targets per package by adjusting the additional USE flags.  Ebuilds
themselves dont have to be modified to support this.  If a dependency does
support the future EAPI, depending packages may require a specific target
to be enabled via use dependencies.  

Exmaple:

profile: amd64/multilib
package dev-libs/a uses the future EAPI
packge app-misc/b requires 32bit libs from dev-libs/a
It can now define this in the dependency section as following:
DEPEND="dev-libs/a[multilib_abi_x86]"


Rationale
=========

The proposal does add USE flags for each supported target ABI.  This allows
the user to select per package, if he wants to compile it for the default
target, additionally for another target or just for another target.  
An alternative would have been a global variable defining this support for
all packages at once. This was not done, because it would force the user
to either build all (even unneeded packages) or nothing with additional or
different target support without the ability to choose on a per package level.


The proposel does add a USE flag called "abiwrapper" to control the
behaviour for binaries.  If this flag is enabled and there is a none-default
target enabled, the binaries for each target are preserved and installed
with their target as suffix.  A symlink is created for the binary name to a
wrapper, which does execute the real binary based on a set environment
variable.  Following alternative implementations have not been selected:

* no wrapper: This would mean, that the installed binaries change, depending
  on the selected targets. Since some binaries have target specific output,
  this would reduce the possible targets for depending packages.
* a variable to control the packages with wrapped binaries: This has the
  same effect as a USE flag, but requires the user to look into an additional
  location to see, if a package gets wrapped binaries or not.


If a package has dependencies, then those dependencies are required to have
at least the same targets enabled as the package. This makes sure, that
the needed libs for that target do exist.  The alternative to go without
those dependencies was not choosen, because this would result in compile
failures due to missing libs for the specified target and in the need for
the user to manually find the needed libs and to install them with the needed
target.


An enabled USE flag called "multilib" does restrict any cross-compilation
and compiles and installs the package for the default target.  The added
USE flags still exist, but are ignored.  This allows packages to do their
own cross-compilation without conflicting with this feature.  The
alternative to not have such a flag would make it much harder for certain
packages (e.g. gcc, glibc or linux-headers) to be transferred to this
future EAPI.


Backwards Compatibility
=======================

To allow packages, which already compile and/or install files for different
targets, to use this future EAPI, the multilib USE flag is used as a special
flag to indicate, that this package already does the needed steps, so that
the package manager wont attempt to additionally do this.


Further backward compatibility issues are not known.


Reference Implementation
========================

A reference implementation is done in a portage branch. The code can be
found at [#1].
An overlay containing an ebuild and the needed profile details can be
found via layman.  It is called "multilib-portage".  The docs dir contains
basic setup instructions.  For help with this overlay, ask in [#2].
A stage 4 qemu image can be found on the Gentoo distfiles mirrors [#3].

References
==========

.. [#1] http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=shortlog;h=refs/heads/multilib

.. [#2] IRC network: Freenode, IRC channel: #gentoo-multilib-overlay

.. [#3] http://distfiles.gentoo.org/experimental/amd64/qemu/ contains
    multilib-amd64-qemu-2012-01-04.tar.lzma with DIGESTS file and
    signature. Those file are also on any other distfiles mirror
    including the experimental dir.

Copyright
=========

This document has been placed in the public domain.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 380 bytes --]

Re: GLEP draf for cross-compile support in multilib profiles (was: Re: [gentoo-dev] Re: spec draft for cross-compile support in future EAPI (EAPI-5))

[Matt Turner]

On Fri, Jun 29, 2012 at 10:30 AM, Thomas Sachau <tommy@gentoo.org> wrote:
>

I'm interested in this because I'm regularly annoyed with the emul-
packages and also because multilib is pretty important for mips.

> If a package has dependencies, then those dependencies are required to have
> at least the same targets enabled as the package

That seems like the obvious (but perhaps naive) choice. What about
depending on packages that don't install libraries, like x11-proto/
packages or generators like dev-util/indent?

Maybe I just don't understand. Would these packages even have ABI flags?

It's clear to me that libraries would be installed in different lib*
directories dependent on their ABI, but how would typical executables
be handled?

For mips, we'd like to have gcc built as an n64 binary, which would
require its run-time dependencies (mpfr, mpc, gmp, etc.) to be
available as n64 as well, but the rest of the system to be n32
binaries. Does this fit with your understanding (and proposed
solution) of the problem?

Thanks,
Matt

[gentoo-dev] Re: GLEP draf for cross-compile support in multilib profiles

[Luca Barbato]

On 06/29/2012 04:30 PM, Thomas Sachau wrote:

It is interesting, still you need a way to define HOST dependencies
(stuff you need that has to be built on the host since you run it, e.g.
xcb python code generator), something to play properly with ld and
cross-vs-host paths for compilers (that part is _really_ annoying for
qemu-chroots)

lu

-- 

Luca Barbato
Gentoo/linux
http://dev.gentoo.org/~lu_zero

[gentoo-dev] Re: GLEP draf for cross-compile support in multilib profiles

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 2360 bytes --]

Matt Turner schrieb:
> On Fri, Jun 29, 2012 at 10:30 AM, Thomas Sachau <tommy@gentoo.org> wrote:
>>
> 
> I'm interested in this because I'm regularly annoyed with the emul-
> packages and also because multilib is pretty important for mips.
> 
>> If a package has dependencies, then those dependencies are required to have
>> at least the same targets enabled as the package
> 
> That seems like the obvious (but perhaps naive) choice. What about
> depending on packages that don't install libraries, like x11-proto/
> packages or generators like dev-util/indent?
> 
> Maybe I just don't understand. Would these packages even have ABI flags?

All packages do get the ABI flags (with the needed EAPI or via enabled
portage feature, which is currently in the multilib branch).

If a package does not install anything ABI-specific (no headers, no libs
and no binaries), then there is no overhead, since it will just get
compiled/installed for one ABI, even if multiple ABI flags are enabled.

> 
> It's clear to me that libraries would be installed in different lib*
> directories dependent on their ABI, but how would typical executables
> be handled?

By default, only the binaries for the first enabled ABI are preserved
(which usually is the default ABI). If you enable the abiwrapper USE
flag, the binaries for all target ABIs are preserved, installed in the
form of $binary-$ABI and $binary will be a symlink to a wrapper, which
calls the right binary for the currently selected ABI.

> 
> For mips, we'd like to have gcc built as an n64 binary, which would
> require its run-time dependencies (mpfr, mpc, gmp, etc.) to be
> available as n64 as well, but the rest of the system to be n32
> binaries. Does this fit with your understanding (and proposed
> solution) of the problem?

I guess, you require additional n64 libs for gcc dependencies like mpfr,
mpc and others, while your default ABI will be n32.

This should work fine with my proposal, you just have the (already
default enabled) n32 ABI flag enabled, which results in everything being
built just for n32. For the packages, where you require additional n64
libs, you just enable the n64 ABI flag in addition. And if you need n64
binaries too, you enable the abiwrapper USE flag for them.


-- 

Thomas Sachau
Gentoo Linux Developer






[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 380 bytes --]

Re: [gentoo-dev] Re: GLEP draf for cross-compile support in multilib profiles

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 1061 bytes --]

Luca Barbato schrieb:
> On 06/29/2012 04:30 PM, Thomas Sachau wrote:
> 
> It is interesting, still you need a way to define HOST dependencies
> (stuff you need that has to be built on the host since you run it, e.g.
> xcb python code generator), something to play properly with ld and
> cross-vs-host paths for compilers (that part is _really_ annoying for
> qemu-chroots)
> 
> lu
> 

I guess, you are mixing cross-compile support in multilib profiles and
cross-compile support with cross-toolchains, multilib-portage is for the
first one, while crossdev is for the second one.

My suggestion does not support e.g. compiling for ppc with an amd64
profile, on amd64 it only can support x86 and x32. Since all of these
binaries can run with an amd64 kernel and you build for at least one
target, you always have a binary around, no need for an extra HOST
dependency.
I dont know, what exactly you mean with "play properly with ld" and
"cross-vs-host paths", so cannot respond to those.

-- 

Thomas Sachau
Gentoo Linux Developer




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 380 bytes --]

Re: [gentoo-dev] Re: GLEP draf for cross-compile support in multilib profiles

[Matt Turner]

On Sun, Jul 1, 2012 at 7:29 AM, Thomas Sachau <tommy@gentoo.org> wrote:
> Matt Turner schrieb:
>> On Fri, Jun 29, 2012 at 10:30 AM, Thomas Sachau <tommy@gentoo.org> wrote:
>>>
>>
>> I'm interested in this because I'm regularly annoyed with the emul-
>> packages and also because multilib is pretty important for mips.
>>
>>> If a package has dependencies, then those dependencies are required to have
>>> at least the same targets enabled as the package
>>
>> That seems like the obvious (but perhaps naive) choice. What about
>> depending on packages that don't install libraries, like x11-proto/
>> packages or generators like dev-util/indent?
>>
>> Maybe I just don't understand. Would these packages even have ABI flags?
>
> All packages do get the ABI flags (with the needed EAPI or via enabled
> portage feature, which is currently in the multilib branch).
>
> If a package does not install anything ABI-specific (no headers, no libs
> and no binaries), then there is no overhead, since it will just get
> compiled/installed for one ABI, even if multiple ABI flags are enabled.

I suppose that's just for ease of implementation? Not having to
special-case packages that don't install binaries.

That seem /okay/ to me, but I'm not writing the implementation so I
only have so much weight. I would be interested to hear what Ciaran,
Brian, and Zac think though.

>> It's clear to me that libraries would be installed in different lib*
>> directories dependent on their ABI, but how would typical executables
>> be handled?
>
> By default, only the binaries for the first enabled ABI are preserved
> (which usually is the default ABI). If you enable the abiwrapper USE
> flag, the binaries for all target ABIs are preserved, installed in the
> form of $binary-$ABI and $binary will be a symlink to a wrapper, which
> calls the right binary for the currently selected ABI.

Ahh, I see. That sounds reasonable.

>> For mips, we'd like to have gcc built as an n64 binary, which would
>> require its run-time dependencies (mpfr, mpc, gmp, etc.) to be
>> available as n64 as well, but the rest of the system to be n32
>> binaries. Does this fit with your understanding (and proposed
>> solution) of the problem?
>
> I guess, you require additional n64 libs for gcc dependencies like mpfr,
> mpc and others, while your default ABI will be n32.
>
> This should work fine with my proposal, you just have the (already
> default enabled) n32 ABI flag enabled, which results in everything being
> built just for n32. For the packages, where you require additional n64
> libs, you just enable the n64 ABI flag in addition. And if you need n64
> binaries too, you enable the abiwrapper USE flag for them.

One follow-on question. Consider having a package dev-libs/libpkg
already installed for ABI="n32 -n64" and wanting to emerge another
package that depends on it with ABI="-n32 n64". Will dev-libs/libpkg
have to be rebuilt twice (once for the existing n32 ABI and again for
the newly needed n64), or can we optimize this case by recognizing
that building for multiple ABIs means entirely separate builds?

Thanks,
Matt

Re: [gentoo-dev] Re: GLEP draf for cross-compile support in multilib profiles

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 3160 bytes --]

Matt Turner schrieb:
> On Sun, Jul 1, 2012 at 7:29 AM, Thomas Sachau <tommy@gentoo.org> wrote:
>> Matt Turner schrieb:
>>> On Fri, Jun 29, 2012 at 10:30 AM, Thomas Sachau <tommy@gentoo.org> wrote:
>>>>
>>>
>>> I'm interested in this because I'm regularly annoyed with the emul-
>>> packages and also because multilib is pretty important for mips.
>>>
>>>> If a package has dependencies, then those dependencies are required to have
>>>> at least the same targets enabled as the package
>>>
>>> That seems like the obvious (but perhaps naive) choice. What about
>>> depending on packages that don't install libraries, like x11-proto/
>>> packages or generators like dev-util/indent?
>>>
>>> Maybe I just don't understand. Would these packages even have ABI flags?
>>
>> All packages do get the ABI flags (with the needed EAPI or via enabled
>> portage feature, which is currently in the multilib branch).
>>
>> If a package does not install anything ABI-specific (no headers, no libs
>> and no binaries), then there is no overhead, since it will just get
>> compiled/installed for one ABI, even if multiple ABI flags are enabled.
> 
> I suppose that's just for ease of implementation? Not having to
> special-case packages that don't install binaries.

I dont follow. Did you think about only having additional ABI flags for
certain cases?

>>> For mips, we'd like to have gcc built as an n64 binary, which would
>>> require its run-time dependencies (mpfr, mpc, gmp, etc.) to be
>>> available as n64 as well, but the rest of the system to be n32
>>> binaries. Does this fit with your understanding (and proposed
>>> solution) of the problem?
>>
>> I guess, you require additional n64 libs for gcc dependencies like mpfr,
>> mpc and others, while your default ABI will be n32.
>>
>> This should work fine with my proposal, you just have the (already
>> default enabled) n32 ABI flag enabled, which results in everything being
>> built just for n32. For the packages, where you require additional n64
>> libs, you just enable the n64 ABI flag in addition. And if you need n64
>> binaries too, you enable the abiwrapper USE flag for them.
> 
> One follow-on question. Consider having a package dev-libs/libpkg
> already installed for ABI="n32 -n64" and wanting to emerge another
> package that depends on it with ABI="-n32 n64". Will dev-libs/libpkg
> have to be rebuilt twice (once for the existing n32 ABI and again for
> the newly needed n64), or can we optimize this case by recognizing
> that building for multiple ABIs means entirely separate builds?

Those ABI flags behave the same as other USE flags: When you change
them, you have to recompile the package including all the content, that
has been compiled previously.
If you want the compile for each ABI seperate, then you would have to
handle them more like SLOTS, but with a new syntax, with a collision
handler for the common content and how to handle that in the user UI. I
fear, that this would be way more complicated to implement in a sane
way, so i dont plan to go that route.


-- 

Thomas Sachau
Gentoo Linux Developer




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 380 bytes --]

Re: [gentoo-dev] Re: GLEP draf for cross-compile support in multilib profiles

[Zac Medico]

On 07/01/2012 04:29 AM, Thomas Sachau wrote:
> Matt Turner schrieb:
>> On Fri, Jun 29, 2012 at 10:30 AM, Thomas Sachau <tommy@gentoo.org> wrote:
>>>
>>
>> I'm interested in this because I'm regularly annoyed with the emul-
>> packages and also because multilib is pretty important for mips.
>>
>>> If a package has dependencies, then those dependencies are required to have
>>> at least the same targets enabled as the package
>>
>> That seems like the obvious (but perhaps naive) choice. What about
>> depending on packages that don't install libraries, like x11-proto/
>> packages or generators like dev-util/indent?
>>
>> Maybe I just don't understand. Would these packages even have ABI flags?
> 
> All packages do get the ABI flags (with the needed EAPI or via enabled
> portage feature, which is currently in the multilib branch).
> 
> If a package does not install anything ABI-specific (no headers, no libs
> and no binaries), then there is no overhead, since it will just get
> compiled/installed for one ABI, even if multiple ABI flags are enabled.

For a package like this that does not install anything ABI-specific,
does the package manager still execute phases for each enabled ABI, or
is there some way for the ebuild to indicate whether or not its phases
need to be executed for each enabled ABI?
-- 
Thanks,
Zac

Re: [gentoo-dev] Re: GLEP draf for cross-compile support in multilib profiles

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 1881 bytes --]

Zac Medico schrieb:
> On 07/01/2012 04:29 AM, Thomas Sachau wrote:
>> Matt Turner schrieb:
>>> On Fri, Jun 29, 2012 at 10:30 AM, Thomas Sachau <tommy@gentoo.org> wrote:
>>>>
>>>
>>> I'm interested in this because I'm regularly annoyed with the emul-
>>> packages and also because multilib is pretty important for mips.
>>>
>>>> If a package has dependencies, then those dependencies are required to have
>>>> at least the same targets enabled as the package
>>>
>>> That seems like the obvious (but perhaps naive) choice. What about
>>> depending on packages that don't install libraries, like x11-proto/
>>> packages or generators like dev-util/indent?
>>>
>>> Maybe I just don't understand. Would these packages even have ABI flags?
>>
>> All packages do get the ABI flags (with the needed EAPI or via enabled
>> portage feature, which is currently in the multilib branch).
>>
>> If a package does not install anything ABI-specific (no headers, no libs
>> and no binaries), then there is no overhead, since it will just get
>> compiled/installed for one ABI, even if multiple ABI flags are enabled.
> 
> For a package like this that does not install anything ABI-specific,
> does the package manager still execute phases for each enabled ABI, or
> is there some way for the ebuild to indicate whether or not its phases
> need to be executed for each enabled ABI?
> 


This is dynamicly checked at runtime, no need to modify the ebuilds and
also no needless compilation, when there is no ABI-specific content.

A more detailed answer at package manager level:
After the src_install phase for the first requested ABI has been
finished, the content of $DESTDIR is checked. If there is no ABI
specific content, the other enabled ABIs are skipped and the following
steps are done as usual.

-- 

Thomas Sachau
Gentoo Linux Developer




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 380 bytes --]

Re: [gentoo-dev] Re: GLEP draf for cross-compile support in multilib profiles

[Matt Turner]

On Sun, Jul 1, 2012 at 4:52 PM, Thomas Sachau <tommy@gentoo.org> wrote:
> Matt Turner schrieb:
>> I suppose that's just for ease of implementation? Not having to
>> special-case packages that don't install binaries.
>
> I dont follow. Did you think about only having additional ABI flags for
> certain cases?

Right. It seems to me that not having the ABI flags for packages like
x11-proto/* would make more sense, but may be more work to implement.

> Those ABI flags behave the same as other USE flags: When you change
> them, you have to recompile the package including all the content, that
> has been compiled previously.
> If you want the compile for each ABI seperate, then you would have to
> handle them more like SLOTS, but with a new syntax, with a collision
> handler for the common content and how to handle that in the user UI. I
> fear, that this would be way more complicated to implement in a sane
> way, so i dont plan to go that route.

Ouch. I already think a mechanism for telling the package manager "you
don't need to rebuild this entire package to turn on /this/ USE flag"
is needed. For ABIs, the situation seems much more clear-cut. In fact,
it seems rather important.

Re: [gentoo-dev] Re: GLEP draf for cross-compile support in multilib profiles

[Zac Medico]

On 07/01/2012 02:39 PM, Matt Turner wrote:
> On Sun, Jul 1, 2012 at 4:52 PM, Thomas Sachau <tommy@gentoo.org> wrote:
>> Matt Turner schrieb:
>>> I suppose that's just for ease of implementation? Not having to
>>> special-case packages that don't install binaries.
>>
>> I dont follow. Did you think about only having additional ABI flags for
>> certain cases?
> 
> Right. It seems to me that not having the ABI flags for packages like
> x11-proto/* would make more sense, but may be more work to implement.

It seems like this could easily be controlled by setting a PROPERTIES or
RESTRICT value in the ebuild. Also, it might be useful to give the
ebuild similar control over the abiwrapper flag via PROPERTIES or RESTRICT.
-- 
Thanks,
Zac

Re: [gentoo-dev] Re: GLEP draf for cross-compile support in multilib profiles

[Zac Medico]

On 07/01/2012 02:34 PM, Thomas Sachau wrote:
> Zac Medico schrieb:
>> On 07/01/2012 04:29 AM, Thomas Sachau wrote:
>>> Matt Turner schrieb:
>>>> On Fri, Jun 29, 2012 at 10:30 AM, Thomas Sachau <tommy@gentoo.org> wrote:
>>>>>
>>>>
>>>> I'm interested in this because I'm regularly annoyed with the emul-
>>>> packages and also because multilib is pretty important for mips.
>>>>
>>>>> If a package has dependencies, then those dependencies are required to have
>>>>> at least the same targets enabled as the package
>>>>
>>>> That seems like the obvious (but perhaps naive) choice. What about
>>>> depending on packages that don't install libraries, like x11-proto/
>>>> packages or generators like dev-util/indent?
>>>>
>>>> Maybe I just don't understand. Would these packages even have ABI flags?
>>>
>>> All packages do get the ABI flags (with the needed EAPI or via enabled
>>> portage feature, which is currently in the multilib branch).
>>>
>>> If a package does not install anything ABI-specific (no headers, no libs
>>> and no binaries), then there is no overhead, since it will just get
>>> compiled/installed for one ABI, even if multiple ABI flags are enabled.
>>
>> For a package like this that does not install anything ABI-specific,
>> does the package manager still execute phases for each enabled ABI, or
>> is there some way for the ebuild to indicate whether or not its phases
>> need to be executed for each enabled ABI?
>>
> 
> 
> This is dynamicly checked at runtime, no need to modify the ebuilds and
> also no needless compilation, when there is no ABI-specific content.
> 
> A more detailed answer at package manager level:
> After the src_install phase for the first requested ABI has been
> finished, the content of $DESTDIR is checked. If there is no ABI
> specific content, the other enabled ABIs are skipped and the following
> steps are done as usual.

I think it would be helpful to include a short explanation of these
kinds of details in the GLEP, in order to help people wrap their heads
around the whole thing (possibly avoiding the "Huh?" kind of reaction
that you got from Brian).
-- 
Thanks,
Zac

Re: [gentoo-dev] Re: GLEP draf for cross-compile support in multilib profiles

[Zac Medico]

On 07/01/2012 02:52 PM, Zac Medico wrote:
> On 07/01/2012 02:39 PM, Matt Turner wrote:
>> On Sun, Jul 1, 2012 at 4:52 PM, Thomas Sachau <tommy@gentoo.org> wrote:
>>> Matt Turner schrieb:
>>>> I suppose that's just for ease of implementation? Not having to
>>>> special-case packages that don't install binaries.
>>>
>>> I dont follow. Did you think about only having additional ABI flags for
>>> certain cases?
>>
>> Right. It seems to me that not having the ABI flags for packages like
>> x11-proto/* would make more sense, but may be more work to implement.
> 
> It seems like this could easily be controlled by setting a PROPERTIES or
> RESTRICT value in the ebuild. Also, it might be useful to give the
> ebuild similar control over the abiwrapper flag via PROPERTIES or RESTRICT.

On irc, I asked Tommy about the abiwrapper thing, and he suggested to
set IUSE="+abiwrapper" for packages containing programs like qmake which
are known to produce ABI specific output.
-- 
Thanks,
Zac

Re: [gentoo-dev] Re: GLEP draf for cross-compile support in multilib profiles

[Zac Medico]

On 07/01/2012 02:34 PM, Thomas Sachau wrote:
> Zac Medico schrieb:
>> On 07/01/2012 04:29 AM, Thomas Sachau wrote:
>>> Matt Turner schrieb:
>>>> On Fri, Jun 29, 2012 at 10:30 AM, Thomas Sachau <tommy@gentoo.org> wrote:
>>>>>
>>>>
>>>> I'm interested in this because I'm regularly annoyed with the emul-
>>>> packages and also because multilib is pretty important for mips.
>>>>
>>>>> If a package has dependencies, then those dependencies are required to have
>>>>> at least the same targets enabled as the package
>>>>
>>>> That seems like the obvious (but perhaps naive) choice. What about
>>>> depending on packages that don't install libraries, like x11-proto/
>>>> packages or generators like dev-util/indent?
>>>>
>>>> Maybe I just don't understand. Would these packages even have ABI flags?
>>>
>>> All packages do get the ABI flags (with the needed EAPI or via enabled
>>> portage feature, which is currently in the multilib branch).
>>>
>>> If a package does not install anything ABI-specific (no headers, no libs
>>> and no binaries), then there is no overhead, since it will just get
>>> compiled/installed for one ABI, even if multiple ABI flags are enabled.
>>
>> For a package like this that does not install anything ABI-specific,
>> does the package manager still execute phases for each enabled ABI, or
>> is there some way for the ebuild to indicate whether or not its phases
>> need to be executed for each enabled ABI?
>>
> 
> 
> This is dynamicly checked at runtime, no need to modify the ebuilds and
> also no needless compilation, when there is no ABI-specific content.
> 
> A more detailed answer at package manager level:
> After the src_install phase for the first requested ABI has been
> finished, the content of $DESTDIR is checked. If there is no ABI
> specific content, the other enabled ABIs are skipped and the following
> steps are done as usual.

In case anyone want some more detail, here's a follow-up question from irc:

<zmedico> Tommy[D]: does any ELF executable qualify as "abi specific"
and trigger builds for all ABIs?
<Tommy[D]> zmedico: if it goes into any of /bin /usr/bin /sbin /usr/sbin
and you enable the abiwrapper USE flag, yes, otherwise you just request
a binary for the default ABI, so no need to rebuild everything for no need
-- 
Thanks,
Zac

[gentoo-dev] Re: GLEP draf for cross-compile support in multilib profiles

[Jonathan Callen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 07/01/2012 05:39 PM, Matt Turner wrote:
> On Sun, Jul 1, 2012 at 4:52 PM, Thomas Sachau <tommy@gentoo.org>
> wrote:
>> Matt Turner schrieb:
>>> I suppose that's just for ease of implementation? Not having
>>> to special-case packages that don't install binaries.
>> 
>> I dont follow. Did you think about only having additional ABI
>> flags for certain cases?
> 
> Right. It seems to me that not having the ABI flags for packages
> like x11-proto/* would make more sense, but may be more work to
> implement.
> 

For x11-proto/* you currently do need to build the package for each
individual ABI because they install a file in
/usr/$(get_libdir)/pkgconfig.  This should probably be changed
upstream to install to /usr/share/pkgconfig, as there is nothing
ABI-specific in there.

- -- 
Jonathan Callen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCgAGBQJP8O8VAAoJELHSF2kinlg4uToP/1swJ2Dm0W1R5z15L41kCO8F
jKtc1Q+3xN/gfDuHzAR1rJpzNLGhOzX0tE+9TKePp4TLcoDKSIeOi9wR+MwncULE
aOqIDPQE5n06z06FEuKqkyWUSJZjJ3sObWiH0+hhWssIorks+LEVYZ8EeLNCw0rb
4zmobbckGeMcbZvcvtISw/N7K3QuUoEDM5rj+059ViqShLxzZegEnO8WU/+h1qi7
E6G5PwKK2MhMGHdWT/Gt+yMT6bSQbNpHu78YHElQFuYSagCcBe75JMfjTKup5iu+
15hM7ZBbiU6yyx45CV6SSXzdqdpkTBkWY7lfrMzXqa3ZtIqgDhmtdxZiNyJ0sHDe
dOtYhJC2FGIGkgv3ZarCEqlybOHrgFxYfBSo0dyFzbr9bWebJgb+mg8CWVbNb4j4
3pe3jVOy5r0CNQ6/ZKRN2ZhzMBhjB0sIsoVmnAOFyqKMNCXHTBHNngQ3jMdxkrJD
fhBVMkEnRr2Gmzmsgi1DZZMvRI9W/6SJo03zwYrik0VBBqyRubRRvEp7mNJO1kV/
eW+8tfStDyOHXuQherGJfCdZiHdP6puj4ZmySURhFHAyeRTEQ1Gxq+vPDIGed1Jv
23odxq+l4MW9lktPgUIrTeku4zUG//FWva74JPBUld1VyicINSCU7plFkLwF5z0o
XEzrHuDZRO+k9Vdi5y1P
=RrNC
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Re: GLEP draf for cross-compile support in multilib profiles

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2658 bytes --]

On Sun, 01 Jul 2012 15:30:44 -0700
Zac Medico <zmedico@gentoo.org> wrote:

> On 07/01/2012 02:34 PM, Thomas Sachau wrote:
> > Zac Medico schrieb:
> >> On 07/01/2012 04:29 AM, Thomas Sachau wrote:
> >>> Matt Turner schrieb:
> >>>> On Fri, Jun 29, 2012 at 10:30 AM, Thomas Sachau
> >>>> <tommy@gentoo.org> wrote:
> >>>>>
> >>>>
> >>>> I'm interested in this because I'm regularly annoyed with the
> >>>> emul- packages and also because multilib is pretty important for
> >>>> mips.
> >>>>
> >>>>> If a package has dependencies, then those dependencies are
> >>>>> required to have at least the same targets enabled as the
> >>>>> package
> >>>>
> >>>> That seems like the obvious (but perhaps naive) choice. What
> >>>> about depending on packages that don't install libraries, like
> >>>> x11-proto/ packages or generators like dev-util/indent?
> >>>>
> >>>> Maybe I just don't understand. Would these packages even have
> >>>> ABI flags?
> >>>
> >>> All packages do get the ABI flags (with the needed EAPI or via
> >>> enabled portage feature, which is currently in the multilib
> >>> branch).
> >>>
> >>> If a package does not install anything ABI-specific (no headers,
> >>> no libs and no binaries), then there is no overhead, since it
> >>> will just get compiled/installed for one ABI, even if multiple
> >>> ABI flags are enabled.
> >>
> >> For a package like this that does not install anything
> >> ABI-specific, does the package manager still execute phases for
> >> each enabled ABI, or is there some way for the ebuild to indicate
> >> whether or not its phases need to be executed for each enabled ABI?
> >>
> > 
> > 
> > This is dynamicly checked at runtime, no need to modify the ebuilds
> > and also no needless compilation, when there is no ABI-specific
> > content.
> > 
> > A more detailed answer at package manager level:
> > After the src_install phase for the first requested ABI has been
> > finished, the content of $DESTDIR is checked. If there is no ABI
> > specific content, the other enabled ABIs are skipped and the
> > following steps are done as usual.
> 
> In case anyone want some more detail, here's a follow-up question
> from irc:
> 
> <zmedico> Tommy[D]: does any ELF executable qualify as "abi specific"
> and trigger builds for all ABIs?
> <Tommy[D]> zmedico: if it goes into any
> of /bin /usr/bin /sbin /usr/sbin and you enable the abiwrapper USE
> flag, yes, otherwise you just request a binary for the default ABI,
> so no need to rebuild everything for no need

How about executables which go into /lib or /libXX?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Re: GLEP draf for cross-compile support in multilib profiles

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 2075 bytes --]

Matt Turner schrieb:
> On Sun, Jul 1, 2012 at 4:52 PM, Thomas Sachau <tommy@gentoo.org> wrote:
>> Matt Turner schrieb:
>>> I suppose that's just for ease of implementation? Not having to
>>> special-case packages that don't install binaries.
>>
>> I dont follow. Did you think about only having additional ABI flags for
>> certain cases?
> 
> Right. It seems to me that not having the ABI flags for packages like
> x11-proto/* would make more sense, but may be more work to implement.

The problem here is the following: How do you know before the
src_install phase, that a package has no ABI-specific content? And even,
if you know that, how would you define the dependency part? This would
probably become pretty complex, since it also has to support corner cases.

> 
>> Those ABI flags behave the same as other USE flags: When you change
>> them, you have to recompile the package including all the content, that
>> has been compiled previously.
>> If you want the compile for each ABI seperate, then you would have to
>> handle them more like SLOTS, but with a new syntax, with a collision
>> handler for the common content and how to handle that in the user UI. I
>> fear, that this would be way more complicated to implement in a sane
>> way, so i dont plan to go that route.
> 
> Ouch. I already think a mechanism for telling the package manager "you
> don't need to rebuild this entire package to turn on /this/ USE flag"
> is needed. For ABIs, the situation seems much more clear-cut. In fact,
> it seems rather important.

This would require a good amount of changes. Since my work with
multilib-portage started in 2009 and is still not in the PMS, also i
tried to keep it simple, i would not even try to guess an ETA for both
of your, in my eyes more complex, ideas. ;-)

I of course wont prevent you from preparing them and getting them in,
they may be nice, especially removing the need to recompile already
compiled code. Just dont underestimate the work with it.


-- 

Thomas Sachau
Gentoo Linux Developer




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 380 bytes --]

Re: [gentoo-dev] Re: GLEP draf for cross-compile support in multilib profiles

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 321 bytes --]

On Mon, 02 Jul 2012 19:06:43 +0200
Thomas Sachau <tommy@gentoo.org> wrote:
> The problem here is the following: How do you know before the
> src_install phase, that a package has no ABI-specific content?

You make every package that has ABI specific content explicitly say so,
as metadata.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: GLEP draf for cross-compile support in multilib profiles

[Luca Barbato]

On 07/01/2012 01:41 PM, Thomas Sachau wrote:
> I guess, you are mixing cross-compile support in multilib profiles and
> cross-compile support with cross-toolchains, multilib-portage is for the
> first one, while crossdev is for the second one.
> 
> My suggestion does not support e.g. compiling for ppc with an amd64
> profile, on amd64 it only can support x86 and x32. Since all of these
> binaries can run with an amd64 kernel and you build for at least one
> target, you always have a binary around, no need for an extra HOST
> dependency.

You can run an arm binary on amd64 (through binfmt+qemu-user static)

> I dont know, what exactly you mean with "play properly with ld" and
> "cross-vs-host paths", so cannot respond to those.

multilib works because the runtime linker picked is the right one for
each ABI, thanks to qemu makes no difference if that ABI is native or not.

cross vs host paths is an annoying problem due the slightly different
behaviour between native and cross compiler toolchains, it tends to
ignore environment variables and other small differences making dropping
an native cross compiler in a qemu chroot, QUITE a creative activity.

lu

-- 

Luca Barbato
Gentoo/linux
http://dev.gentoo.org/~lu_zero