[gentoo-dev] RFC: method of checking for cross compilation from ebuild functions

[gentoo-dev] RFC: method of checking for cross compilation from ebuild functions

[Ambroz Bizjak]

I'm working on some EAPI extensions with the goal of making Portage
more powerful for cross-compilation. See
https://bugs.gentoo.org/show_bug.cgi?id=317337
Currently, it's come down to the following:

- A new dependency type HDEPEND for packages which must be installed
in / at build time. With HDEPEND present in the new EAPI (probably
4-hdepend), DEPEND will always mean ROOT (target) dependencies, and
will no longer be affected by --root-deps option.

- Another dependency type CROSS_HDEPEND, like HDEPEND, but in effect
only when cross compiling. This in particular is required for packages
which need themselves on the host when cross compiling, and adding
this to HDEPEND would make a package not buildable on native due to a
circular dependency.

- A --crosscompile option to tell emerge when we are cross compiling,
and this triggers CROSS_HDEPEND (except for installs to /). Basically
we don't want emerge doing any black magic guessing like comparing
CHOST.

But the one thing missing currently is a way to check from within an
ebuild function whether we are cross compiling, in a way that would
correspond directly to --crosscompile (again, except for /). There is
currently "tc-is-cross-compiler" from toolchain-funcs.eclass, but this
may not work right when not at build time (pkg_{pre,post}{inst,rm}),
and it also may not correspond to presence of --crosscompile.

The question now is, how should this method for checking
--crosscompile be implemented? In particular, we have two options:

- Environment variable. If so, how to call it? Possible names are
CROSSCOMPILE, GENTOO_CROSSCOMPILE, PORTAGE_CROSSCOMPILE,
ECROSSCOMPILE... For more generic names (CROSSCOMPILE) it needs to be
taken into account that they may inadvertently affect packages.
However environment vars have the benefit that it's easy to pass them
through programs and scripts.

- Internal function, similar to "use". Probably "is_crosscompile".
This may look nicer and reduces the risk of collisions.

This RFC is only about checking for presence of --crosscompile. Please
do not discuss HDEPEND in general here, or anything that would require
major effort to get done.

Re: [gentoo-dev] RFC: method of checking for cross compilation from ebuild functions

[Zac Medico]

On 09/20/2012 10:34 AM, Ambroz Bizjak wrote:
> The question now is, how should this method for checking
> --crosscompile be implemented? In particular, we have two options:
> 
> - Environment variable. If so, how to call it? Possible names are
> CROSSCOMPILE, GENTOO_CROSSCOMPILE, PORTAGE_CROSSCOMPILE,
> ECROSSCOMPILE... For more generic names (CROSSCOMPILE) it needs to be
> taken into account that they may inadvertently affect packages.
> However environment vars have the benefit that it's easy to pass them
> through programs and scripts.
> 
> - Internal function, similar to "use". Probably "is_crosscompile".
> This may look nicer and reduces the risk of collisions.

Since it's just a boolean flag, we could have a special "crosscompile"
USE flag for this, so that the use() function could be used like we
currently use it for the "test" USE flag. The flag would be forced on or
off based on your configuration, similar to the "test" flag [1], so
there wouldn't be any danger of the flag being accidentally enabled or
disabled. The flag could be bound to FEATURES=crosscompile, or some
other package manager configuration variable. Note that if we add a
--crosscompile option to emerge, then we'll also have to add it to the
ebuid(1) command, so maybe it's better to forgo the commandline option
and just toggle it with a configuration variable like
FEATURES=crosscompile. Also, it's conceivable that you could drop the
CROSS_HDEPEND variable, in favor of HDEPEND="crosscompile? ( foo )"
syntax (somewhat in alignment with Brian Harring's DEPENDENCIES proposal).

[1] https://bugs.gentoo.org/show_bug.cgi?id=373209
-- 
Thanks,
Zac

Re: [gentoo-dev] RFC: method of checking for cross compilation from ebuild functions

[Luca Barbato]

On 09/21/2012 06:06 PM, Zac Medico wrote:
> On 09/20/2012 10:34 AM, Ambroz Bizjak wrote:
>> The question now is, how should this method for checking
>> --crosscompile be implemented? In particular, we have two options:
>>
>> - Environment variable. If so, how to call it? Possible names are
>> CROSSCOMPILE, GENTOO_CROSSCOMPILE, PORTAGE_CROSSCOMPILE,
>> ECROSSCOMPILE... For more generic names (CROSSCOMPILE) it needs to be
>> taken into account that they may inadvertently affect packages.
>> However environment vars have the benefit that it's easy to pass them
>> through programs and scripts.
>>
>> - Internal function, similar to "use". Probably "is_crosscompile".
>> This may look nicer and reduces the risk of collisions.
> 
> Since it's just a boolean flag, we could have a special "crosscompile"
> USE flag for this, so that the use() function could be used like we
> currently use it for the "test" USE flag. The flag would be forced on or
> off based on your configuration, similar to the "test" flag [1], so
> there wouldn't be any danger of the flag being accidentally enabled or
> disabled. The flag could be bound to FEATURES=crosscompile, or some
> other package manager configuration variable. Note that if we add a
> --crosscompile option to emerge, then we'll also have to add it to the
> ebuid(1) command, so maybe it's better to forgo the commandline option
> and just toggle it with a configuration variable like
> FEATURES=crosscompile. Also, it's conceivable that you could drop the
> CROSS_HDEPEND variable, in favor of HDEPEND="crosscompile? ( foo )"
> syntax (somewhat in alignment with Brian Harring's DEPENDENCIES proposal).
> 
> [1] https://bugs.gentoo.org/show_bug.cgi?id=373209
> 


I like the idea.

lu

Re: [gentoo-dev] RFC: method of checking for cross compilation from ebuild functions

[Ambroz Bizjak]

Yes, I think this is a good idea, it would allow the dependencies to
be expressed nicely as conditions.

But I'm not sure how this would be a USE flag. It should behave like
one during the build, but it would be best if it was not written into
the VDB as such, at least in a way that would be considered by
--newuse. It don't want "emerge -unD" on the booted system want to
reinstall all packages because the current ones were cross-compiled.
Does the test flag already behave nicely like that? In that case, all
is good, and I can try to implement this.

On a slightly different subject: I've been porting some packages to
HDEPEND and I've seen problems with packages that want to use the
programs they build during the build (or in postinst). Of couse this
works for native builds, and it can be fixed to work for cross-compile
builds (build native version or HDEPEND on host package).

But what do we do with the strange case where ROOT!=/ but
--crosscompile/FEATURES=crosscompile is not in affect? Can we expect
that we will be able to run the programs that were built? What if they
link to libraries only available in ROOT?

So, I think it would make sense for a lot of packages to treat ROOT!=/
equivalently to cross-compilation, i.e. require host tool to be
present. But with what has currently been proposed there is no
conditional dependency on ROOT!=/, so a package cannot demand that a
tool be present on the host. Then, it may be a good idea to add a
conditional dependency on ROOT!=/.

In fact, I think that --crosscompile or FEATURES=crosscompile could
actually be abolished and only this condition would be available. It's
true that some packages would only use the host dependency if there's
actual cross-compilation going on, but nothing will break. This would
ease configuration and reduce the number of cases to be tested.

On Fri, Sep 21, 2012 at 6:06 PM, Zac Medico <zmedico@gentoo.org> wrote:
> On 09/20/2012 10:34 AM, Ambroz Bizjak wrote:
>> The question now is, how should this method for checking
>> --crosscompile be implemented? In particular, we have two options:
>>
>> - Environment variable. If so, how to call it? Possible names are
>> CROSSCOMPILE, GENTOO_CROSSCOMPILE, PORTAGE_CROSSCOMPILE,
>> ECROSSCOMPILE... For more generic names (CROSSCOMPILE) it needs to be
>> taken into account that they may inadvertently affect packages.
>> However environment vars have the benefit that it's easy to pass them
>> through programs and scripts.
>>
>> - Internal function, similar to "use". Probably "is_crosscompile".
>> This may look nicer and reduces the risk of collisions.
>
> Since it's just a boolean flag, we could have a special "crosscompile"
> USE flag for this, so that the use() function could be used like we
> currently use it for the "test" USE flag. The flag would be forced on or
> off based on your configuration, similar to the "test" flag [1], so
> there wouldn't be any danger of the flag being accidentally enabled or
> disabled. The flag could be bound to FEATURES=crosscompile, or some
> other package manager configuration variable. Note that if we add a
> --crosscompile option to emerge, then we'll also have to add it to the
> ebuid(1) command, so maybe it's better to forgo the commandline option
> and just toggle it with a configuration variable like
> FEATURES=crosscompile. Also, it's conceivable that you could drop the
> CROSS_HDEPEND variable, in favor of HDEPEND="crosscompile? ( foo )"
> syntax (somewhat in alignment with Brian Harring's DEPENDENCIES proposal).
>
> [1] https://bugs.gentoo.org/show_bug.cgi?id=373209
> --
> Thanks,
> Zac

Re: [gentoo-dev] RFC: method of checking for cross compilation from ebuild functions

[Zac Medico]

On 09/22/2012 09:08 AM, Ambroz Bizjak wrote:
> Yes, I think this is a good idea, it would allow the dependencies to
> be expressed nicely as conditions.
> 
> But I'm not sure how this would be a USE flag. It should behave like
> one during the build, but it would be best if it was not written into
> the VDB as such, at least in a way that would be considered by
> --newuse. It don't want "emerge -unD" on the booted system want to
> reinstall all packages because the current ones were cross-compiled.
> Does the test flag already behave nicely like that? In that case, all
> is good, and I can try to implement this.

Simply add your special flag to the _feature_flags variable in
config.py, and it will be exempt from --newuse. See this commit:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=6b19f71b39b6af43307abf20654511bace041217

> On a slightly different subject: I've been porting some packages to
> HDEPEND and I've seen problems with packages that want to use the
> programs they build during the build (or in postinst). Of couse this
> works for native builds, and it can be fixed to work for cross-compile
> builds (build native version or HDEPEND on host package).
> 
> But what do we do with the strange case where ROOT!=/ but
> --crosscompile/FEATURES=crosscompile is not in affect? Can we expect
> that we will be able to run the programs that were built? What if they
> link to libraries only available in ROOT?
> 
> So, I think it would make sense for a lot of packages to treat ROOT!=/
> equivalently to cross-compilation, i.e. require host tool to be
> present. But with what has currently been proposed there is no
> conditional dependency on ROOT!=/, so a package cannot demand that a
> tool be present on the host. Then, it may be a good idea to add a
> conditional dependency on ROOT!=/.

If I understand correctly, that would be like a CROSS_TDEPEND? If we
translate that to a conditional, it would become DEPEND="crosscompile? (
foo )", since our plan was to make DEPEND apply to ROOT!=/ and HDEPEND
apply to ROOT=/, right?

> In fact, I think that --crosscompile or FEATURES=crosscompile could
> actually be abolished and only this condition would be available. It's
> true that some packages would only use the host dependency if there's
> actual cross-compilation going on, but nothing will break. This would
> ease configuration and reduce the number of cases to be tested.

Yeah, the split between HDEPEND and DEPEND might be enough so that you
don't need these conditionals. If you're not really sure that the
conditionals are needed, then maybe it's better to eliminate them for now.
-- 
Thanks,
Zac

[gentoo-dev] RFC: method of checking for cross compilation from ebuild functions

[Ambroz Bizjak]

Zac, I think you misunderstood me here. Crosscompile-only HDEPEND is
definitely necessary, I've seen many packages need this. But what I'm
suggesting is that we also, and maybe only, need "ROOT != /" - only
HDEPEND dependencies. This means that the dependency would only be
required if the package is being built for a ROOT that is not /. The
idea is to eliminate the strange case that is ROOT!=/ but FEATURES has
no crosscompile. If the package requires tools that it would build
itself if ROOT was /, it will use the host's version of the tool
whenever ROOT!=/  It wouldn't have to worry about whether the tools it
builds link to libraries in ROOT.

So my proposal is basically, instead of:
HDEPEND="crosscompile? ( ~${CATEGORY}/${P} )    (yes, that seems to be
the most common case)

there would also, and maybe only, be:
HDEPEND="root_not_slash? ( ~${CATEGORY}/${P} )"

root_not_slash (or another name) would essentially be a superset of
crosscompile, since crosscompile implies ROOT!=/.

P.S. sorry Zac I sent you this twice, damn GMail :)

On Sat, Sep 22, 2012 at 6:31 PM, Zac Medico <zmedico@gentoo.org> wrote:
> On 09/22/2012 09:08 AM, Ambroz Bizjak wrote:
>> Yes, I think this is a good idea, it would allow the dependencies to
>> be expressed nicely as conditions.
>>
>> But I'm not sure how this would be a USE flag. It should behave like
>> one during the build, but it would be best if it was not written into
>> the VDB as such, at least in a way that would be considered by
>> --newuse. It don't want "emerge -unD" on the booted system want to
>> reinstall all packages because the current ones were cross-compiled.
>> Does the test flag already behave nicely like that? In that case, all
>> is good, and I can try to implement this.
>
> Simply add your special flag to the _feature_flags variable in
> config.py, and it will be exempt from --newuse. See this commit:
>
> http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=6b19f71b39b6af43307abf20654511bace041217
>
>> On a slightly different subject: I've been porting some packages to
>> HDEPEND and I've seen problems with packages that want to use the
>> programs they build during the build (or in postinst). Of couse this
>> works for native builds, and it can be fixed to work for cross-compile
>> builds (build native version or HDEPEND on host package).
>>
>> But what do we do with the strange case where ROOT!=/ but
>> --crosscompile/FEATURES=crosscompile is not in affect? Can we expect
>> that we will be able to run the programs that were built? What if they
>> link to libraries only available in ROOT?
>>
>> So, I think it would make sense for a lot of packages to treat ROOT!=/
>> equivalently to cross-compilation, i.e. require host tool to be
>> present. But with what has currently been proposed there is no
>> conditional dependency on ROOT!=/, so a package cannot demand that a
>> tool be present on the host. Then, it may be a good idea to add a
>> conditional dependency on ROOT!=/.
>
> If I understand correctly, that would be like a CROSS_TDEPEND? If we
> translate that to a conditional, it would become DEPEND="crosscompile? (
> foo )", since our plan was to make DEPEND apply to ROOT!=/ and HDEPEND
> apply to ROOT=/, right?
>
>> In fact, I think that --crosscompile or FEATURES=crosscompile could
>> actually be abolished and only this condition would be available. It's
>> true that some packages would only use the host dependency if there's
>> actual cross-compilation going on, but nothing will break. This would
>> ease configuration and reduce the number of cases to be tested.
>
> Yeah, the split between HDEPEND and DEPEND might be enough so that you
> don't need these conditionals. If you're not really sure that the
> conditionals are needed, then maybe it's better to eliminate them for now.
> --
> Thanks,
> Zac

Re: [gentoo-dev] RFC: method of checking for cross compilation from ebuild functions

[Zac Medico]

On 09/22/2012 09:48 AM, Ambroz Bizjak wrote:
> Zac, I think you misunderstood me here. Crosscompile-only HDEPEND is
> definitely necessary, I've seen many packages need this. But what I'm
> suggesting is that we also, and maybe only, need "ROOT != /" - only
> HDEPEND dependencies. This means that the dependency would only be
> required if the package is being built for a ROOT that is not /. The
> idea is to eliminate the strange case that is ROOT!=/ but FEATURES has
> no crosscompile. If the package requires tools that it would build
> itself if ROOT was /, it will use the host's version of the tool
> whenever ROOT!=/  It wouldn't have to worry about whether the tools it
> builds link to libraries in ROOT.
> 
> So my proposal is basically, instead of:
> HDEPEND="crosscompile? ( ~${CATEGORY}/${P} )    (yes, that seems to be
> the most common case)
> 
> there would also, and maybe only, be:
> HDEPEND="root_not_slash? ( ~${CATEGORY}/${P} )"
> 
> root_not_slash (or another name) would essentially be a superset of
> crosscompile, since crosscompile implies ROOT!=/.

Maybe call it "sysroot" instead of root_not_slash, since gcc and other
tools seem to refer to it as sysroot.
-- 
Thanks,
Zac

Re: [gentoo-dev] RFC: method of checking for cross compilation from ebuild functions

[Ambroz Bizjak]

Yes, sysroot is much better, thanks :)

So, does anyone have any objections to just having a sysroot condition
and no --crosscompile or FEATURES=crosscompile?

Essentially, there's still tc-is-cross-compiler, if you want the real
cross-compile semantic, and sysroot conditional dependencies cover any
host programs you may need within a tc-is-cross-compiler block, even
though it may occasionally install stuff that is not used. But I think
it's worth the reduction of complexity.  (yes, tc-is-cross-compiler
doesn't work at install time, but from my experience you really want
the sysroot semantic there, which is available via use sysroot)

If all is well, I'll proceed to implement this.

On Sat, Sep 22, 2012 at 7:01 PM, Zac Medico <zmedico@gentoo.org> wrote:
> On 09/22/2012 09:48 AM, Ambroz Bizjak wrote:
>> Zac, I think you misunderstood me here. Crosscompile-only HDEPEND is
>> definitely necessary, I've seen many packages need this. But what I'm
>> suggesting is that we also, and maybe only, need "ROOT != /" - only
>> HDEPEND dependencies. This means that the dependency would only be
>> required if the package is being built for a ROOT that is not /. The
>> idea is to eliminate the strange case that is ROOT!=/ but FEATURES has
>> no crosscompile. If the package requires tools that it would build
>> itself if ROOT was /, it will use the host's version of the tool
>> whenever ROOT!=/  It wouldn't have to worry about whether the tools it
>> builds link to libraries in ROOT.
>>
>> So my proposal is basically, instead of:
>> HDEPEND="crosscompile? ( ~${CATEGORY}/${P} )    (yes, that seems to be
>> the most common case)
>>
>> there would also, and maybe only, be:
>> HDEPEND="root_not_slash? ( ~${CATEGORY}/${P} )"
>>
>> root_not_slash (or another name) would essentially be a superset of
>> crosscompile, since crosscompile implies ROOT!=/.
>
> Maybe call it "sysroot" instead of root_not_slash, since gcc and other
> tools seem to refer to it as sysroot.
> --
> Thanks,
> Zac

Re: [gentoo-dev] RFC: method of checking for cross compilation from ebuild functions

[Zac Medico]

On 09/22/2012 10:14 AM, Ambroz Bizjak wrote:
> Yes, sysroot is much better, thanks :)
> 
> So, does anyone have any objections to just having a sysroot condition
> and no --crosscompile or FEATURES=crosscompile?
> 
> Essentially, there's still tc-is-cross-compiler, if you want the real
> cross-compile semantic, and sysroot conditional dependencies cover any
> host programs you may need within a tc-is-cross-compiler block, even
> though it may occasionally install stuff that is not used. But I think
> it's worth the reduction of complexity.  (yes, tc-is-cross-compiler
> doesn't work at install time, but from my experience you really want
> the sysroot semantic there, which is available via use sysroot)
> 
> If all is well, I'll proceed to implement this.

Sounds good to me. Concerning IUSE, there are a few of different ways we
could handle this:

1) ebuilds that utilize this flag will simply add it to IUSE
2) We set USE_IMPLICIT="sysroot" in profiles/base/make.defaults (applies
to EAPI 5 and later)
3) We define our HDEPEND EAPI such that "sysroot" is automatically
considered to be an implicit member of IUSE

You have to pick one of these, since otherwise USE conditionals
involving this flag will be considered as invalid. Maybe option #1 is
best, since that will make it easy to identify ebuilds that are known to
support sysroot (as long as you add "sysroot" to that _feature_flags
variable that I've mentioned before, it will be exempt from emerge
--newuse).
-- 
Thanks,
Zac