[gentoo-dev] Support for multiple ABIs (for Python modules etc.)

[gentoo-dev] Support for multiple ABIs (for Python modules etc.)

[Arfrever Frehtes Taifersar Arahesis]

[-- Attachment #1: Type: text/plain, Size: 5318 bytes --]

I would like to present the plan of support for multiple ABIs. It should be sufficient for
Python modules and might be also appropriate for some other ABI types (e.g. for Ruby modules).

1. Portage will support repository-specific, profile-independent ${REPOSITORY}/profiles/settings
configuration files (filename can be changed). They will set some variables (similarly to
make.defaults files). These variables shouldn't be overrideable by profiles, eclasses, ebuilds
or by user's configuration files (e.g. /etc/portage/*). For future compatibility these files
could also support lists/indexed arrays and dictionaries/associative arrays/maps.

2. ${REPOSITORY}/profiles/settings will set ABI_TYPES and ${ABI_TYPE^^}_ABI_SUPPORTED_VALUES variables.
ABI_TYPES defines supported ABI types. ${ABI_TYPE^^}_ABI_SUPPORTED_VALUES defines supported ABIs
for given ABI type.

  Example (${REPOSITORY}/profiles/settings):
    ABI_TYPES="python ruby ..."
    PYTHON_ABI_SUPPORTED_VALUES="2.4 2.5 2.6 3.0 3.1"
    RUBY_ABI_SUPPORTED_VALUES="1.8 1.9"

3. make.defaults files will support ${ABI_TYPE^^}_ABI_FORCED_VALUES/${ABI_TYPE^^}_ABI_MASKED_VALUES
variables to force/mask specific ABIs in given profiles.

  Example (make.defaults):
    PYTHON_ABI_MASKED_VALUES="2.4 2.5"
    PYTHON_ABI_FORCED_VALUES="3.1"

4. For convenience of developers ABI dependencies will be able to be specified in explicit and
slightly implicit way. Ebuilds/eclasses, which support multiple ABIs, will set USED_ABI_TYPES.
Ebuilds/eclasses will be able to set RESTRICT_${ABI_TYPE^^}_ABIS variables which will support
'*' wildcard. USED_ABI_TYPES and RESTRICT_${ABI_TYPE^^}_ABIS variables should be cumulative
(across eclasses and ebuild).

  Example (ebuild):
    USED_ABI_TYPES="python"
    # This package doesn't work yet with Python 2.6 and 3.*
    RESTRICT_PYTHON_ABIS="2.6 3*"

4.1. Implicitly specified ABI dependencies. During calculation of dependencies of given package,
Portage will verify if all dependencies, which use given ABI type, have been built with enabled
support for these ABIs, which are enabled for given package.

  Example (ebuild):
    # dev-python/bsddb3
    USED_ABI_TYPES="python"
    RDEPEND=">=sys-libs/db-4.6"
    DEPEND="${RDEPEND}
        dev-python/setuptools
        doc? ( dev-python/sphinx )"

    Assuming that dev-python/setuptools and dev-python/sphinx set USED_ABI_TYPES="python",
    Portage will verify that, when user runs `USE_PYTHON="2.5 2.6" emerge dev-python/bsddb3`,
    dev-python/setuptools and dev-python/sphinx (when USE="doc" is enabled) are built with
    enabled support for at least "2.5" and "2.6" Python ABIs. If they are already installed
    without support for e.g. "2.5" Python ABI, Portage will print appropriate error message
    (similarly to unsatisfied USE dependencies) and exit.

4.2. Explicitly specified ABI dependencies. {,P,R}DEPEND variables will support specifying
ABI dependencies in explicit way. {,P,R}DEPEND variables will also support ABI conditionals.
I suggest using {ABI_type[comma-delimited values]} syntax, but it can be changed.

  Example (ebuild):
    DEPEND="category/package1{python[2.5,2.6],ruby[-1.8]}"
    RDEPEND="{python[2.6]}? ( category/package2 )

  Example (python.eclass):
    DEPEND="
        {python[2.4]}? ( dev-lang/python:2.4 )
        {python[2.5]}? ( dev-lang/python:2.5 )
        {python[2.6]}? ( dev-lang/python:2.6 )
        {python[3.0]}? ( dev-lang/python:3.0 )
        {python[3.1]}? ( dev-lang/python:3.1 )
    "
    RDEPEND="${RDEPEND}"

5. 18th line of metadata cache files will contain the list of ABI types and values supported
(i.e. not restricted by RESTRICT_${ABI_TYPE^^}_ABIS variables) by given package.

  Example (18th line of metadata cache file):
    python[2.5,2.6,3.0,3.1] ruby[1.8,1.9]

Every change to ${REPOSITORY}/profiles/settings should probably invalidate whole metadata
cache of given repository.

6. Portage will support USE_${ABI_TYPE^^} variables in /etc/make.conf to specify which ABIs
should be enabled.

  Example (/etc/make.conf):
    USE_PYTHON="2.5 2.6 3.1"
    USE_RUBY="1.8 1.9"

7. Portage will set ${ABI_TYPE^^}_ABIS variables in ebuild environment. ${ABI_TYPE^^}_ABIS
should contain ABIs forced by ${ABI_TYPE^^}_ABI_FORCED_VALUES and these ABIs enabled by user
in /etc/make.conf by USE_${ABI_TYPE^^} which aren't masked by ${ABI_TYPE^^}_ABI_MASKED_VALUES.

  Example (ebuild):
    src_compile() {
        for abi in ${PYTHON_ABIS}; do
            emake PYTHON=$(get_PYTHON ${abi})
        done
    }

Ebuilds/eclasses should implicitly enable support for one ("default") ABI, when user hasn't
set appropriate USE_${ABI_TYPE^^} variable. For example Python modules would use ABI of
/usr/bin/python.

8. Portage will store the list of supported ABIs in /var/db/pkg/${CATEGORY}/${PF}/SUPPORTED_ABIS
and the list of enabled ABIs in /var/db/pkg/${CATEGORY}/${PF}/ABIS.

  Example (/var/db/pkg/${CATEGORY}/${PF}/SUPPORTED_ABIS):
    python[2.4,2.5,2.6,3.0,3.1] ruby[1.8,1.9]

  Example (/var/db/pkg/${CATEGORY}/${PF}/ABIS):
    python[2.6,3.1] ruby[1.9]

Names/syntax of specific variables etc. can be changed. Please write constructive comments.

-- 
Arfrever Frehtes Taifersar Arahesis

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Support for multiple ABIs (for Python modules etc.)

[Arfrever Frehtes Taifersar Arahesis]

[-- Attachment #1: Type: text/plain, Size: 6287 bytes --]

2009-07-25 12:28:44 Arfrever Frehtes Taifersar Arahesis napisał(a):
> I would like to present the plan of support for multiple ABIs. It should be sufficient for
> Python modules and might be also appropriate for some other ABI types (e.g. for Ruby modules).
> 
> 1. Portage will support repository-specific, profile-independent ${REPOSITORY}/profiles/settings
> configuration files (filename can be changed). They will set some variables (similarly to
> make.defaults files). These variables shouldn't be overrideable by profiles, eclasses, ebuilds
> or by user's configuration files (e.g. /etc/portage/*). For future compatibility these files
> could also support lists/indexed arrays and dictionaries/associative arrays/maps.
> 
> 2. ${REPOSITORY}/profiles/settings will set ABI_TYPES and ${ABI_TYPE^^}_ABI_SUPPORTED_VALUES variables.
> ABI_TYPES defines supported ABI types. ${ABI_TYPE^^}_ABI_SUPPORTED_VALUES defines supported ABIs
> for given ABI type.
> 
>   Example (${REPOSITORY}/profiles/settings):
>     ABI_TYPES="python ruby ..."
>     PYTHON_ABI_SUPPORTED_VALUES="2.4 2.5 2.6 3.0 3.1"
>     RUBY_ABI_SUPPORTED_VALUES="1.8 1.9"
> 
> 3. make.defaults files will support ${ABI_TYPE^^}_ABI_FORCED_VALUES/${ABI_TYPE^^}_ABI_MASKED_VALUES
> variables to force/mask specific ABIs in given profiles.
> 
>   Example (make.defaults):
>     PYTHON_ABI_MASKED_VALUES="2.4 2.5"
>     PYTHON_ABI_FORCED_VALUES="3.1"
> 
> 4. For convenience of developers ABI dependencies will be able to be specified in explicit and
> slightly implicit way. Ebuilds/eclasses, which support multiple ABIs, will set USED_ABI_TYPES.
> Ebuilds/eclasses will be able to set RESTRICT_${ABI_TYPE^^}_ABIS variables which will support
> '*' wildcard. USED_ABI_TYPES and RESTRICT_${ABI_TYPE^^}_ABIS variables should be cumulative
> (across eclasses and ebuild).
> 
>   Example (ebuild):
>     USED_ABI_TYPES="python"
>     # This package doesn't work yet with Python 2.6 and 3.*
>     RESTRICT_PYTHON_ABIS="2.6 3*"
> 
> 4.1. Implicitly specified ABI dependencies. During calculation of dependencies of given package,
> Portage will verify if all dependencies, which use given ABI type, have been built with enabled
> support for these ABIs, which are enabled for given package.
> 
>   Example (ebuild):
>     # dev-python/bsddb3
>     USED_ABI_TYPES="python"
>     RDEPEND=">=sys-libs/db-4.6"
>     DEPEND="${RDEPEND}
>         dev-python/setuptools
>         doc? ( dev-python/sphinx )"
> 
>     Assuming that dev-python/setuptools and dev-python/sphinx set USED_ABI_TYPES="python",
>     Portage will verify that, when user runs `USE_PYTHON="2.5 2.6" emerge dev-python/bsddb3`,
>     dev-python/setuptools and dev-python/sphinx (when USE="doc" is enabled) are built with
>     enabled support for at least "2.5" and "2.6" Python ABIs. If they are already installed
>     without support for e.g. "2.5" Python ABI, Portage will print appropriate error message
>     (similarly to unsatisfied USE dependencies) and exit.
> 
> 4.2. Explicitly specified ABI dependencies. {,P,R}DEPEND variables will support specifying
> ABI dependencies in explicit way. {,P,R}DEPEND variables will also support ABI conditionals.
> I suggest using {ABI_type[comma-delimited values]} syntax, but it can be changed.
> 
>   Example (ebuild):
>     DEPEND="category/package1{python[2.5,2.6],ruby[-1.8]}"
>     RDEPEND="{python[2.6]}? ( category/package2 )
> 
>   Example (python.eclass):
>     DEPEND="
>         {python[2.4]}? ( dev-lang/python:2.4 )
>         {python[2.5]}? ( dev-lang/python:2.5 )
>         {python[2.6]}? ( dev-lang/python:2.6 )
>         {python[3.0]}? ( dev-lang/python:3.0 )
>         {python[3.1]}? ( dev-lang/python:3.1 )
>     "
>     RDEPEND="${RDEPEND}"
> 
> 5. 18th line of metadata cache files will contain the list of ABI types and values supported
> (i.e. not restricted by RESTRICT_${ABI_TYPE^^}_ABIS variables) by given package.
> 
>   Example (18th line of metadata cache file):
>     python[2.5,2.6,3.0,3.1] ruby[1.8,1.9]
> 
> Every change to ${REPOSITORY}/profiles/settings should probably invalidate whole metadata
> cache of given repository.
> 
> 6. Portage will support USE_${ABI_TYPE^^} variables in /etc/make.conf to specify which ABIs
> should be enabled.

Portage will also allow to set USE_${ABI_TYPE^^} variables per-package etc.

> 
>   Example (/etc/make.conf):
>     USE_PYTHON="2.5 2.6 3.1"
>     USE_RUBY="1.8 1.9"
> 
> 7. Portage will set ${ABI_TYPE^^}_ABIS variables in ebuild environment. ${ABI_TYPE^^}_ABIS
> should contain ABIs forced by ${ABI_TYPE^^}_ABI_FORCED_VALUES and these ABIs enabled by user
> in /etc/make.conf by USE_${ABI_TYPE^^} which aren't masked by ${ABI_TYPE^^}_ABI_MASKED_VALUES.

${ABI_TYPE^^}_ABIS variables won't contain ABIs restricted by RESTRICT_${ABI_TYPE^^}_ABIS
variables. Eclasses/ebuilds will be able to change the values of ${ABI_TYPE^^}_ABIS variables
in ebuild phases. For example python.eclass will support ignoring of build failures for Python
ABIs specified by user in FAILURE_TOLERANT_PYTHON_ABIS variable. Python ABIs, for which
building has failed, would be removed from PYTHON_ABIS variable.

> 
>   Example (ebuild):
>     src_compile() {
>         for abi in ${PYTHON_ABIS}; do
>             emake PYTHON=$(get_PYTHON ${abi})
>         done
>     }
> 
> Ebuilds/eclasses should implicitly enable support for one ("default") ABI, when user hasn't
> set appropriate USE_${ABI_TYPE^^} variable. For example Python modules would use ABI of
> /usr/bin/python.
> 
> 8. Portage will store the list of supported ABIs in /var/db/pkg/${CATEGORY}/${PF}/SUPPORTED_ABIS
> and the list of enabled ABIs in /var/db/pkg/${CATEGORY}/${PF}/ABIS.

The list of enabled ABIs in /var/db/pkg/${CATEGORY}/${PF}/ABIS will depend on final values of
${ABI_TYPE^^}_ABIS variables after installation.

> 
>   Example (/var/db/pkg/${CATEGORY}/${PF}/SUPPORTED_ABIS):
>     python[2.4,2.5,2.6,3.0,3.1] ruby[1.8,1.9]
> 
>   Example (/var/db/pkg/${CATEGORY}/${PF}/ABIS):
>     python[2.6,3.1] ruby[1.9]
> 
> Names/syntax of specific variables etc. can be changed. Please write constructive comments.

-- 
Arfrever Frehtes Taifersar Arahesis

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Support for multiple ABIs (for Python modules etc.)

[Marijn Schouten (hkBst)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Arfrever Frehtes Taifersar Arahesis wrote:
> I would like to present the plan of support for multiple ABIs. It should be sufficient for
> Python modules and might be also appropriate for some other ABI types (e.g. for Ruby modules).

In the context of which problem are you brainstorming?

Marijn

- --
If you cannot read my mind, then listen to what I say.

Marijn Schouten (hkBst), Gentoo Lisp project, Gentoo ML
<http://www.gentoo.org/proj/en/lisp/>, #gentoo-{lisp,ml} on FreeNode
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkpsTq0ACgkQp/VmCx0OL2xqtACglJ7tpOXL4DAeSvo+sLpSc5ir
NsEAoLmyH9EbXydfk4dpCJiHxLqr6WFS
=lmdT
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Support for multiple ABIs (for Python modules etc.)

[Arfrever Frehtes Taifersar Arahesis]

[-- Attachment #1: Type: text/plain, Size: 1019 bytes --]

2009-07-26 14:40:13 Marijn Schouten (hkBst) napisał(a):
> Arfrever Frehtes Taifersar Arahesis wrote:
> > I would like to present the plan of support for multiple ABIs. It should be sufficient for
> > Python modules and might be also appropriate for some other ABI types (e.g. for Ruby modules).
> 
> In the context of which problem are you brainstorming?

This proposition is intended to solve multiple problems, but Gentoo Python Project especially
would like to have it available during transitions to new Python versions (e.g. Python 3.*).
Python 3.1 will be added to the tree in the next week. Over 10 Python modules should work with
Python 3, but the majority of packages doesn't work yet. We want to provide possibility of
installing Python modules into site-packages directories of multiple Python versions (e.g. 2.6
and 3.1). In currently existing EAPIs we *will* support it, but without checking Python ABI
dependencies during dependency calculation.

-- 
Arfrever Frehtes Taifersar Arahesis

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Support for multiple ABIs (for Python modules etc.)

[Paul de Vrieze]

On Sun, Jul 26, 2009 at 5:21 PM, Arfrever Frehtes Taifersar
Arahesis<Arfrever@gentoo.org> wrote:
> 2009-07-26 14:40:13 Marijn Schouten (hkBst) napisał(a):
>> Arfrever Frehtes Taifersar Arahesis wrote:
>> > I would like to present the plan of support for multiple ABIs. It should be sufficient for
>> > Python modules and might be also appropriate for some other ABI types (e.g. for Ruby modules).
>>
>> In the context of which problem are you brainstorming?
>
> This proposition is intended to solve multiple problems, but Gentoo Python Project especially
> would like to have it available during transitions to new Python versions (e.g. Python 3.*).
> Python 3.1 will be added to the tree in the next week. Over 10 Python modules should work with
> Python 3, but the majority of packages doesn't work yet. We want to provide possibility of
> installing Python modules into site-packages directories of multiple Python versions (e.g. 2.6
> and 3.1). In currently existing EAPIs we *will* support it, but without checking Python ABI
> dependencies during dependency calculation.
>

I don't think this is easy to do, but I think the solution to this
problem should be the same as the (as yet not existing) solution to
the multi-ABI problem as in (x86_64 vs. ix86). The biggest issue is to
handle multiple instances of the same package and how to handle
overlapping (ABI independent) files.

Paul

-- 
Paul de Vrieze

Re: [gentoo-dev] Support for multiple ABIs (for Python modules etc.)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1805 bytes --]

On Sat, 25 Jul 2009 12:28:44 +0200
Arfrever Frehtes Taifersar Arahesis <Arfrever@gentoo.org> wrote:
> I would like to present the plan of support for multiple ABIs. It
> should be sufficient for Python modules and might be also appropriate
> for some other ABI types (e.g. for Ruby modules).

How do you plan to handle the intersection of ABIs? What if you have to
build or depend upon a Python module for both 32 and 64 bit ABIs, and
for both 2.5 and 2.6? What if you have a package that provides both
Ruby and Python code, where the two ABIs are independent rather than a
product?

> 4.1. Implicitly specified ABI dependencies. During calculation of
> dependencies of given package, Portage will verify if all
> dependencies, which use given ABI type, have been built with enabled
> support for these ABIs, which are enabled for given package.

How do you say "I need only a single ABI for this, even though it looks
like I need every ABI I'm built with"? For example, if your Python
module, being built for 2.5 and 2.6, runs (but does not use in a
library sense) a Python program that comes as part of a Python package
that is buildable with multiple ABIs?

> 4.2. Explicitly specified ABI dependencies. {,P,R}DEPEND variables
> will support specifying ABI dependencies in explicit way.
> {,P,R}DEPEND variables will also support ABI conditionals. I suggest
> using {ABI_type[comma-delimited values]} syntax, but it can be
> changed.

I think we're trying to avoid introducing new special characters if we
can get away with using existing ones. You can overload [] and existing
conditionals if you're careful.

Having said that, you can probably do everything you described slightly
less elegantly just using things that're already in EAPI 3.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-dev] Support for multiple ABIs (for Python modules etc.)

[Arfrever Frehtes Taifersar Arahesis]

[-- Attachment #1: Type: text/plain, Size: 2346 bytes --]

2009-08-01 20:10:49 Ciaran McCreesh napisał(a):
> On Sat, 25 Jul 2009 12:28:44 +0200
> Arfrever Frehtes Taifersar Arahesis <Arfrever@gentoo.org> wrote:
> > I would like to present the plan of support for multiple ABIs. It
> > should be sufficient for Python modules and might be also appropriate
> > for some other ABI types (e.g. for Ruby modules).
> 
> How do you plan to handle the intersection of ABIs? What if you have to
> build or depend upon a Python module for both 32 and 64 bit ABIs, and
> for both 2.5 and 2.6? What if you have a package that provides both
> Ruby and Python code, where the two ABIs are independent rather than a
> product?

The proposition already handles these cases. Please describe in more details what you don't
understand.

> > 4.1. Implicitly specified ABI dependencies. During calculation of
> > dependencies of given package, Portage will verify if all
> > dependencies, which use given ABI type, have been built with enabled
> > support for these ABIs, which are enabled for given package.
> 
> How do you say "I need only a single ABI for this, even though it looks
> like I need every ABI I'm built with"? For example, if your Python
> module, being built for 2.5 and 2.6, runs (but does not use in a
> library sense) a Python program that comes as part of a Python package
> that is buildable with multiple ABIs?

In such case a Python package, which is a dependency of another Python package, shouldn't
be marked as supporting multiple Python ABIs. But anyway I suggest the following syntax:
  category/package_name{python[*]}

> > 4.2. Explicitly specified ABI dependencies. {,P,R}DEPEND variables
> > will support specifying ABI dependencies in explicit way.
> > {,P,R}DEPEND variables will also support ABI conditionals. I suggest
> > using {ABI_type[comma-delimited values]} syntax, but it can be
> > changed.
> 
> I think we're trying to avoid introducing new special characters if we
> can get away with using existing ones. You can overload [] and existing
> conditionals if you're careful.

The syntax suggested by me looks better for this purpose.

> Having said that, you can probably do everything you described slightly
> less elegantly just using things that're already in EAPI 3.

Not everything.

-- 
Arfrever Frehtes Taifersar Arahesis

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]