[gentoo-dev] [RFC] python-r1 + distutils-r1, v2 (+ comparison)

[gentoo-dev] [RFC] python-r1 + distutils-r1, v2 (+ comparison)

[Michał Górny]

[-- Attachment #1.1: Type: text/plain, Size: 7282 bytes --]

Hello,

This is the second and hopefully last version of python-r1 +
distutils-r1 eclasses before committing. I would also like to shortly
point out the goals and the differences between various python eclasses
in Gentoo.

Both are attached. For those who prefer hosted form:
https://bitbucket.org/mgorny/gx86-working-tree/src/5a2d39e69d6d/gx86/eclass/python-r1.eclass
https://bitbucket.org/mgorny/gx86-working-tree/src/5a2d39e69d6d/gx86/eclass/distutils-r1.eclass

Changes from the previous version:
- support for DOCS and HTML_DOCS (alike EAPI 4);
- minimal support for passing arguments to setup.py (through
  distutils-r1_python_compile & distutils-r1_python_install);
- EPYTHON values adjusted to match python.eclass;
- scripts are -${EPYTHON} suffixed -- i.e. foo-python2.7,
  foo-pypy-c1.8;
- a wrapper script[1] is used to choose the correct script instead of
  constant symlink. Thus, EPYTHON & eselect-python are respected,
- PYTHON_COMPAT can be an array only.

What still needs considering:
- a nicer way of passing setup.py arguments (mysetuppyargs=()?,
  esetuppy wrapper?);
- Prefix support.

[1]:https://bitbucket.org/mgorny/python-exec


Now, for the differences part.

Goals
-----

python.eclass aims to support every single Python package out there,
including rare corner cases supported in-eclass. It supports both
packages supporting one and multiple Python implementations. distutils
support is provided by second eclass, and both provide phase functions
(python.eclass providing default ph.f. wrappers).

python-distutils-ng aims to support majority of Python packages using
distutils. With a bit of hackery it can support non-distutils packages,
but the use is limited to the common cases. In an uncommon case, it's
not flexible enough. It supports Python packages supporting multiple
implementations only.

python-r1 aims mostly to provide tools to support majority of Python
packages. It tries to be simple & flexible, thus allowing handling
various corner cases without special branches of code in the eclass.
It doesn't export any phase functions, nor set dependencies by default
(just provides a variable with them).

Right now, it supports packages supporting multiple implementations
only; if necessary, the support will be extended -- either through
additional code if that wouldn't add too much complexity, or through
additional eclass.

It is accompanied by distutils-r1 which provides a simple overlay for
the very common case of distutils packages. This eclass exports phase
functions and sets dependencies implicitly. It also handles renaming
the distutils-installed scripts and linking the python-exec wrapper.


Python targets
--------------

python.eclass uses implicit Python target specifications. It provides
no ebuild-transparent way of enabling/disabling them.

python-distutils-ng and python-r1 use USE flags to explicitly list
enabled Python implementations. Both eclasses use the same values for
PYTHON_TARGETS USE_EXPAND.


Installed scripts
-----------------

python-distutils-ng rework the installed scripts creating copies for
Python implementations and changing the shebang. The created copies
don't differ any other way. The old script name is then symlinked to
the version for default implementation.

python.eclass & python-r1 instead let distutils rework the scripts
and just rename them before merging. A wrapper is used to choose
the correct version, respecting EPYTHON & eselect-python.

python.eclass installs the files to separate installation images
and merges them. python-r1 installs them to the main image directly,
renaming the installed scripts between successive implementations.

python.eclass creates a wrapper script for each package. The script is
written in python. python-r1 instead installs the compiled wrapper
once (in an ebuild), and symlinks it for the packages. The wrapper is
written in much simpler C.

The implementation suffixes for all three eclasses are different:
- python.eclass uses -2.6, -2.7 for Python, -2.7-pypy-1.8,
  -2.7-pypy-1.9 for pypy and -2.5-jython for jython (enjoy!);
- p-d-ng uses Python target names (-python2_6, -python2_7, -pypy1_8),
- p-r1 uses $EPYTHON values (-python2.6, -python2.7, -pypy-c1.8 (sic!)).


Dependency on Python
--------------------

python.eclass has a few variables necessary to set dependencies
on Python implementation, including internal sub-syntax (and an even
more complete generator sub-syntax in progress overlay). I suspect that
most of the possible variants can be achieved using it, just please
don't make me try learning it all.

Additionally, packages supporting multiple Python implementations are
required to specify their supported Python versions twice -- with
PYTHON_DEPEND and RESTRICT_PYTHON_ABIS.

USE flag dependencies are specified using three variables, and I'd like
to avoid getting anywhere deeper.

p-d-ng has a very simple dependency setup. It has a variable for
listing supported Python versions and a variable to make Python
dependencies conditional under the 'python' flag. Lately, it has been
added a method of listing requested USE flags.

p-d-ng does not provide any support for more complex dependency
specifications, nor a way to disable adding default Python dependencies.

python-r1 does not append any dependencies by default, and instead
exports them in a variable. The variable can be used to easily express
simple cases, and avoids polluting the ebuild in more complex cases. It
also supports providing USE dependencies for the implementation.

distutils-r1 adds the Python dependencies to RDEPEND & DEPEND;
additionally, it adds a dependency on python-exec.


Dependencies on other packages
------------------------------

As python.eclass (as of gx86 state) does not express enabled
implementations explicitly, it is not possible to request those
implementations from other packages. The dependencies on other Python
packages are thus simple.

p-d-ng does not provide any simpler way of writing dependencies
on other Python packages using the eclass. Thus, the necessary USE
dependencies have to be written by hand.

python-r1 does provide a simple PYTHON_USEDEP variable which can be
used in dependency atoms to request the same Python implementations
being enabled.


Phase functions for distutils
-----------------------------

distutils.eclass allows specifying additional arguments to setup.py.
src_install() installs default documentation files (*unconditionally*)
and ${DOCS}, it also does installation image magic.

p-d-ng has mixed unconditional and user-overridable stuff. Copying
sources is done unconditionally; redoing scripts can be disabled using
a separate variable. setup.py invocations are overridable.

d-r1 splits each phase into 'global' and 'per-implementation' stages
which are both overridable. The default stages apply PATCHES (and user
patches), copy sources, run setup.py, install DOCS & HTML_DOCS and link
python-exec wrappers.



Thank you for reading up to this. I'd appreciate any comments, ideas,
and especially any API suggestions while the eclass still ain't live.

-- 
Best regards,
Michał Górny

[-- Attachment #1.2: python-r1.eclass --]
[-- Type: text/plain, Size: 6234 bytes --]

# Copyright 1999-2012 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

# @ECLASS: python-r1
# @MAINTAINER:
# Python herd <python@gentoo.org>
# @AUTHOR:
# Author: Michał Górny <mgorny@gentoo.org>
# Based on work of: Krzysztof Pawlik <nelchael@gentoo.org>
# @BLURB: A common, simple eclass for Python packages.
# @DESCRIPTION:
# A common eclass providing helper functions to build and install
# packages supporting being installed for multiple Python
# implementations.
#
# This eclass sets correct IUSE and REQUIRED_USE. It exports PYTHON_DEPS
# and PYTHON_USEDEP so you can create correct dependencies for your
# package easily. It also provides methods to easily run a command for
# each enabled Python implementation and duplicate the sources for them.

case "${EAPI}" in
	0|1|2|3)
		die "Unsupported EAPI=${EAPI} (too old) for ${ECLASS}"
		;;
	4)
		# EAPI=4 needed for REQUIRED_USE
		;;
	*)
		die "Unsupported EAPI=${EAPI} (unknown) for ${ECLASS}"
		;;
esac

# @ECLASS-VARIABLE: _PYTHON_ALL_IMPLS
# @INTERNAL
# @DESCRIPTION:
# All supported Python implementations, most preferred last.
_PYTHON_ALL_IMPLS=(
	jython2_5
	pypy1_8 pypy1_9
	python3_1 python3_2
	python2_5 python2_6 python2_7
)

# @ECLASS-VARIABLE: PYTHON_COMPAT
# @DESCRIPTION:
# This variable contains a list of Python implementations the package
# supports. It must be set before the `inherit' call.  The default is to
# enable all implementations. It has to be an array.
: ${PYTHON_COMPAT:=${_PYTHON_ALL_IMPLS[@]}}

# @ECLASS-VARIABLE: PYTHON_REQ_USE
# @DEFAULT_UNSET
# @DESCRIPTION:
# The list of USEflags required to be enabled on the chosen Python
# implementations, formed as a USE-dependency string. It should be valid
# for all implementations in PYTHON_COMPAT, so it may be necessary to
# use USE defaults.
#
# Example:
# @CODE
# PYTHON_REQ_USE="gdbm,ncurses(-)?"
# @CODE
#
# Will cause the Python dependencies to look like:
# @CODE
# python_targets_pythonX_Y? (
#   dev-lang/python:X_Y[gdbm,ncurses(-)?] )
# @CODE

# @ECLASS-VARIABLE: PYTHON_DEPS
# @DESCRIPTION:
# This is an eclass-generated Python dependency string for all
# implementations listed in PYTHON_COMPAT. It should be used
# in RDEPEND and/or DEPEND like:
#
# @CODE
# RDEPEND="${PYTHON_DEPS}
#   dev-foo/mydep"
# DEPEND="${RDEPEND}"
# @CODE

# @ECLASS-VARIABLE: PYTHON_USEDEP
# @DESCRIPTION:
# This is an eclass-generated USE-dependency string which can be used to
# depend on another Python package being built for the same Python
# implementations. It should be used like:
#
# @CODE
# RDEPEND="dev-python/foo[${PYTHON_USEDEP}]"
# @CODE

_python_set_globals() {
	local flags=( "${PYTHON_COMPAT[@]/#/python_targets_}" )
	local optflags=${flags[@]/%/?}

	IUSE=${flags[*]}
	REQUIRED_USE="|| ( ${flags[*]} )"
	PYTHON_USEDEP=${optflags// /,}

	PYTHON_DEPS=
	local i
	for i in ${PYTHON_COMPAT[@]}; do
		local d
		case ${i} in
			python*)
				d='dev-lang/python';;
			jython*)
				d='dev-java/jython';;
			pypy*)
				d='dev-python/pypy';;
			*)
				die "Invalid implementation: ${i}"
		esac

		local v=${i##*[a-z]}
		local usestr
		[[ ${PYTHON_REQ_USE} ]] && usestr="[${PYTHON_REQ_USE}]"
		PYTHON_DEPS+=" python_targets_${i}? (
			${d}:${v/_/.}${usestr} )"
	done
}
_python_set_globals

# @FUNCTION: python_get_PYTHON
# @USAGE: <impl>
# @DESCRIPTION:
# Get the Python executable name for the given implementation. Please
# note that this this function returns the 'basename' only.
# If using it for a hashbang, please use #!/usr/bin/env.
python_get_PYTHON() {
	debug-print-function ${FUNCNAME} "${@}"

	local impl=${1/_/.}
	local ret

	case "${impl}" in
		python*|jython*)
			ret=${impl}
			;;
		pypy*)
			ret=pypy-c${impl#pypy}
			;;
		*)
			die "Invalid argument to python_get_PYTHON: ${1}"
			;;
	esac

	debug-print "${FUNCNAME}: ${impl} -> ${ret}"
	echo "${ret}"
}

# @FUNCTION: python_copy_sources
# @DESCRIPTION:
# Create a single copy of the package sources (${S}) for each enabled
# Python implementation.
python_copy_sources() {
	debug-print-function ${FUNCNAME} "${@}"

	local impl
	local bdir=${BUILD_DIR:-${S}}

	debug-print "${FUNCNAME}: bdir = ${bdir}"
	einfo "Will copy sources from ${S}"
	# the order is irrelevant here
	for impl in ${PYTHON_COMPAT[@]}; do
		if use python_targets_${impl}
		then
			local BUILD_DIR=${bdir%%/}-${impl}

			einfo "${impl}: copying to ${BUILD_DIR}"
			debug-print "${FUNCNAME}: [${impl}] cp ${S} => ${BUILD_DIR}"
			cp -pr "${S}" "${BUILD_DIR}" || die
		fi
	done
}

# @FUNCTION: python_foreach_impl
# @USAGE: <command> [<args>...]
# @DESCRIPTION:
# Run the given command for each of the enabled Python implementations.
# If additional parameters are passed, they will be passed through
# to the command. If the command fails, python_foreach_impl dies.
# If necessary, use ':' to force a successful return.
#
# Before the command is run, EPYTHON is set to the name of the current
# Python implementation, PYTHON is set to the correct Python executable
# name and exported, and BUILD_DIR is set to a 'default' build directory
# for given implementation (e.g. ${BUILD_DIR:-${S}}-python2_7).
#
# The command is run inside the build directory. If it doesn't exist
# yet, it is created (as an empty directory!). If your build system does
# not support out-of-source builds, you will likely want to use
# python_copy_sources first.
python_foreach_impl() {
	debug-print-function ${FUNCNAME} "${@}"

	local impl
	local bdir=${BUILD_DIR:-${S}}

	debug-print "${FUNCNAME}: bdir = ${bdir}"
	for impl in ${_PYTHON_ALL_IMPLS[@]}; do
		if has ${impl} ${PYTHON_COMPAT[@]} && use python_targets_${impl}
		then
			local PYTHON=$(python_get_PYTHON "${impl}")
			local EPYTHON=${PYTHON}
			local BUILD_DIR=${bdir%%/}-${impl}
			export PYTHON

			debug-print "${FUNCNAME}: [${impl}] build_dir = ${BUILD_DIR}"

			mkdir -p "${BUILD_DIR}" || die
			pushd "${BUILD_DIR}" &>/dev/null || die
			einfo "${EPYTHON}: running ${@}"
			"${@}" || die "${EPYTHON}: ${1} failed"
			popd &>/dev/null || die
		fi
	done
}

[-- Attachment #1.3: distutils-r1.eclass --]
[-- Type: text/plain, Size: 8731 bytes --]

# Copyright 1999-2012 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

# @ECLASS: distutils-r1
# @MAINTAINER:
# Python herd <python@gentoo.org>
# @AUTHOR:
# Author: Michał Górny <mgorny@gentoo.org>
# Based on the work of: Krzysztof Pawlik <nelchael@gentoo.org>
# @BLURB: A simple eclass to build Python packages using distutils.
# @DESCRIPTION:
# A simple eclass providing functions to build Python packages using
# the distutils build system. It exports phase functions for all
# the src_* phases. Each of the phases runs two pseudo-phases:
# python_..._all() (e.g. python_prepare_all()) once in ${S}, then
# python_...() (e.g. python_prepare()) for each implementation
# (see: python_foreach_impl() in python-r1).
#
# In distutils-r1_src_prepare(), the 'all' function is run before
# per-implementation ones (because it creates the implementations),
# per-implementation functions are run in a random order.
#
# In remaining phase functions, the per-implementation functions are run
# before the 'all' one, and they are ordered from the least to the most
# preferred implementation (so that 'better' files overwrite 'worse'
# ones).
#
# If the ebuild doesn't specify a particular pseudo-phase function,
# the default one will be used (distutils-r1_...). Defaults are provided
# for all per-implementation pseudo-phases, python_prepare_all()
# and python_install_all(); whenever writing your own pseudo-phase
# functions, you should consider calling the defaults (and especially
# distutils-r1_python_prepare_all).
#
# Please note that distutils-r1 sets RDEPEND and DEPEND for you.

case "${EAPI}" in
	0|1|2|3)
		die "Unsupported EAPI=${EAPI} (too old) for ${ECLASS}"
		;;
	4)
		;;
	*)
		die "Unsupported EAPI=${EAPI} (unknown) for ${ECLASS}"
		;;
esac

inherit eutils python-r1

EXPORT_FUNCTIONS src_prepare src_configure src_compile src_test src_install

RDEPEND="${PYTHON_DEPS}
	dev-python/python-exec"
DEPEND=${PYTHON_DEPS}

# @ECLASS-VARIABLE: DOCS
# @DEFAULT_UNSET
# @DESCRIPTION:
# Array containing documents installed using dodoc.
#
# If unset, the default filename list (from PMS) will be used.
#
# Example:
# @CODE
# DOCS=( NEWS README )
# @CODE

# @ECLASS-VARIABLE: HTML_DOCS
# @DEFAULT_UNSET
# @DESCRIPTION:
# Array containing documents installed using dohtml.
#
# Example:
# @CODE
# HTML_DOCS=( doc/html/ )
# @CODE

# @FUNCTION: distutils-r1_python_prepare_all
# @DESCRIPTION:
# The default python_prepare_all(). It applies the patches from PATCHES
# array, then user patches and finally calls python_copy_sources to
# create copies of resulting sources for each Python implementation.
distutils-r1_python_prepare_all() {
	debug-print-function ${FUNCNAME} "${@}"

	[[ ${PATCHES} ]] && epatch "${PATCHES[@]}"

	epatch_user

	# create source copies for each implementation
	python_copy_sources
}

# @FUNCTION: distutils-r1_python_prepare
# @DESCRIPTION:
# The default python_prepare(). Currently it is a no-op
# but in the future it may apply implementation-specific quirks
# to the build system.
distutils-r1_python_prepare() {
	debug-print-function ${FUNCNAME} "${@}"

	:
}

# @FUNCTION: distutils-r1_python_configure
# @DESCRIPTION:
# The default python_configure(). Currently a no-op.
distutils-r1_python_configure() {
	debug-print-function ${FUNCNAME} "${@}"

	:
}

# @FUNCTION: distutils-r1_python_compile
# @USAGE: [additional-args...]
# @DESCRIPTION:
# The default python_compile(). Runs 'setup.py build' using the correct
# Python implementation. Any parameters passed to this function will be
# passed to setup.py.
distutils-r1_python_compile() {
	debug-print-function ${FUNCNAME} "${@}"

	cd "${BUILD_DIR}" || die
	set -- "${PYTHON}" setup.py build "${@}"
	echo "${@}"
	"${@}" || die
}

# @FUNCTION: distutils-r1_python_test
# @DESCRIPTION:
# The default python_test(). Currently a no-op.
distutils-r1_python_test() {
	debug-print-function ${FUNCNAME} "${@}"

	:
}

# @FUNCTION: distutils-r1_rename_scripts
# @DESCRIPTION:
# Renames installed Python scripts to be implementation-suffixed.
# ${PYTHON} has to be set to the expected Python executable (which
# hashbang will be grepped for), and ${EPYTHON} to the implementation
# name (for new name).
distutils-r1_rename_scripts() {
	debug-print-function ${FUNCNAME} "${@}"

	local f
	# XXX: change this if we ever allow directories in bin/sbin
	for f in "${D}"/{bin,sbin,usr/bin,usr/sbin}/*; do
		if [[ -x ${f} ]]; then
			debug-print "${FUNCNAME}: found executable at ${f#${D}/}"

			if [[ "$(head -n 1 "${f}")" == '#!'*${PYTHON}* ]]
			then
				debug-print "${FUNCNAME}: matching shebang: $(head -n 1 "${f}")"

				local newf=${f}-${EPYTHON}
				debug-print "${FUNCNAME}: renamed to ${newf#${D}/}"
				mv "${f}" "${newf}" || die
			fi
		fi
	done
}

# @FUNCTION: distutils-r1_python_install
# @USAGE: [additional-args...]
# @DESCRIPTION:
# The default python_install(). Runs 'setup.py install' using
# the correct Python implementation, and appending the optimization
# flags. Then calls distutils-r1_rename_scripts. Any parameters passed
# to this function will be passed to setup.py.
distutils-r1_python_install() {
	debug-print-function ${FUNCNAME} "${@}"

	local flags

	case "${EPYTHON}" in
		jython*)
			flags='--compile';;
		*)
			flags='--compile -O2';;
	esac
	debug-print "${FUNCNAME}: [${EPYTHON}] flags: ${flags}"

	unset PYTHONDONTWRITEBYTECODE

	cd "${BUILD_DIR}" || die
	set -- "${PYTHON}" setup.py install ${flags} --root="${D}" "${@}"
	echo "${@}"
	"${@}" || die

	distutils-r1_rename_scripts
}

# @FUNCTION: distutils-r1_python_install_all
# @DESCRIPTION:
# The default python_install_all(). It symlinks wrappers
# for the implementation-suffixed executables and installs
# documentation.
distutils-r1_python_install_all() {
	debug-print-function ${FUNCNAME} "${@}"

	if declare -p DOCS &>/dev/null; then
		dodoc "${DOCS[@]}" || die "dodoc failed"
	else
		local f
		# same list as in PMS
		for f in README* ChangeLog AUTHORS NEWS TODO CHANGES \
				THANKS BUGS FAQ CREDITS CHANGELOG; do
			if [[ -s ${f} ]]; then
				dodoc "${f}" || die "(default) dodoc ${f} failed"
			fi
		done
	fi

	if declare -p HTML_DOCS &>/dev/null; then
		dohtml -r "${HTML_DOCS[@]}" || die "dohtml failed"
	fi

	# note: keep in sync with ...rename_scripts()
	# also, we assume that each script is installed for all impls
	local any_impl=$(python_get_PYTHON ${PYTHON_COMPAT[0]})
	for f in "${D}"/{bin,sbin,usr/bin,usr/sbin}/*-${any_impl}; do
		if [[ -x ${f} ]]; then
			debug-print "${FUNCNAME}: found executable at ${f#${D}/}"

			local wrapf=${f%-${any_impl}}
			debug-print "${FUNCNAME}: will link wrapper to ${wrapf#${D}/}"
			local wrapfrom
			case "${f#${D}/}" in
				usr/bin*)
					wrapfrom=
					;;
				usr/sbin*)
					wrapfrom=../bin/
					;;
				*)
					wrapfrom=../usr/bin/
					;;
			esac
			debug-print "${FUNCNAME}: (from ${wrapfrom}python-exec)"

			ln -s ${wrapfrom}python-exec "${wrapf}" || die
		fi
	done
}

distutils-r1_src_prepare() {
	debug-print-function ${FUNCNAME} "${@}"

	# common preparations
	if declare -f python_prepare_all >/dev/null; then
		python_prepare_all
	else
		distutils-r1_python_prepare_all
	fi

	if declare -f python_prepare >/dev/null; then
		python_foreach_impl python_prepare
	else
		distutils-r1_python_prepare
	fi
}

distutils-r1_src_configure() {
	if declare -f python_configure >/dev/null; then
		python_foreach_impl python_configure
	else
		distutils-r1_python_configure
	fi

	if declare -f python_configure_all >/dev/null; then
		python_configure_all
	fi
}

distutils-r1_src_compile() {
	debug-print-function ${FUNCNAME} "${@}"

	if declare -f python_compile >/dev/null; then
		python_foreach_impl python_compile
	else
		python_foreach_impl distutils-r1_python_compile
	fi

	if declare -f python_compile_all >/dev/null; then
		python_compile_all
	fi
}

distutils-r1_src_test() {
	debug-print-function ${FUNCNAME} "${@}"

	if declare -f python_test >/dev/null; then
		python_foreach_impl python_test
	else
		distutils-r1_python_test
	fi

	if declare -f python_test_all >/dev/null; then
		python_test_all
	fi
}

distutils-r1_src_install() {
	debug-print-function ${FUNCNAME} "${@}"

	if declare -f python_install >/dev/null; then
		python_foreach_impl python_install
	else
		python_foreach_impl distutils-r1_python_install
	fi

	if declare -f python_install_all >/dev/null; then
		python_install_all
	else
		distutils-r1_python_install_all
	fi
}

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] [RFC] python-r1 + distutils-r1, v2 (+ comparison)

[Reinis Danne]

On Mon, Oct 08, 2012 at 05:16:05PM +0200, Michał Górny wrote:
> Hello,
> 
> This is the second and hopefully last version of python-r1 +
> distutils-r1 eclasses before committing. I would also like to shortly
> point out the goals and the differences between various python eclasses
> in Gentoo.
> 
> Both are attached. For those who prefer hosted form:
> https://bitbucket.org/mgorny/gx86-working-tree/src/5a2d39e69d6d/gx86/eclass/python-r1.eclass
> https://bitbucket.org/mgorny/gx86-working-tree/src/5a2d39e69d6d/gx86/eclass/distutils-r1.eclass
> 
> Changes from the previous version:
> - support for DOCS and HTML_DOCS (alike EAPI 4);
> - minimal support for passing arguments to setup.py (through
>   distutils-r1_python_compile & distutils-r1_python_install);
> - EPYTHON values adjusted to match python.eclass;
> - scripts are -${EPYTHON} suffixed -- i.e. foo-python2.7,
>   foo-pypy-c1.8;
> - a wrapper script[1] is used to choose the correct script instead of
>   constant symlink. Thus, EPYTHON & eselect-python are respected,
> - PYTHON_COMPAT can be an array only.
> 
> What still needs considering:
> - a nicer way of passing setup.py arguments (mysetuppyargs=()?,
>   esetuppy wrapper?);
> - Prefix support.
> 
> [1]:https://bitbucket.org/mgorny/python-exec
> 
> 
> Now, for the differences part.
> 
> Goals
> -----
> 
> python.eclass aims to support every single Python package out there,
> including rare corner cases supported in-eclass. It supports both
> packages supporting one and multiple Python implementations. distutils
> support is provided by second eclass, and both provide phase functions
> (python.eclass providing default ph.f. wrappers).
> 
> python-distutils-ng aims to support majority of Python packages using
> distutils. With a bit of hackery it can support non-distutils packages,
> but the use is limited to the common cases. In an uncommon case, it's
> not flexible enough. It supports Python packages supporting multiple
> implementations only.
> 
> python-r1 aims mostly to provide tools to support majority of Python
> packages. It tries to be simple & flexible, thus allowing handling
> various corner cases without special branches of code in the eclass.
> It doesn't export any phase functions, nor set dependencies by default
> (just provides a variable with them).
> 
> Right now, it supports packages supporting multiple implementations
> only; if necessary, the support will be extended -- either through
> additional code if that wouldn't add too much complexity, or through
> additional eclass.
> 
> It is accompanied by distutils-r1 which provides a simple overlay for
> the very common case of distutils packages. This eclass exports phase
> functions and sets dependencies implicitly. It also handles renaming
> the distutils-installed scripts and linking the python-exec wrapper.
> 
> 
> Python targets
> --------------
> 
> python.eclass uses implicit Python target specifications. It provides
> no ebuild-transparent way of enabling/disabling them.
> 
> python-distutils-ng and python-r1 use USE flags to explicitly list
> enabled Python implementations. Both eclasses use the same values for
> PYTHON_TARGETS USE_EXPAND.
> 
> 
> Installed scripts
> -----------------
> 
> python-distutils-ng rework the installed scripts creating copies for
> Python implementations and changing the shebang. The created copies
> don't differ any other way. The old script name is then symlinked to
> the version for default implementation.
> 
> python.eclass & python-r1 instead let distutils rework the scripts
> and just rename them before merging. A wrapper is used to choose
> the correct version, respecting EPYTHON & eselect-python.
> 
> python.eclass installs the files to separate installation images
> and merges them. python-r1 installs them to the main image directly,
> renaming the installed scripts between successive implementations.
> 
> python.eclass creates a wrapper script for each package. The script is
> written in python. python-r1 instead installs the compiled wrapper
> once (in an ebuild), and symlinks it for the packages. The wrapper is
> written in much simpler C.
> 
> The implementation suffixes for all three eclasses are different:
> - python.eclass uses -2.6, -2.7 for Python, -2.7-pypy-1.8,
>   -2.7-pypy-1.9 for pypy and -2.5-jython for jython (enjoy!);
> - p-d-ng uses Python target names (-python2_6, -python2_7, -pypy1_8),
> - p-r1 uses $EPYTHON values (-python2.6, -python2.7, -pypy-c1.8 (sic!)).
> 
> 
> Dependency on Python
> --------------------
> 
> python.eclass has a few variables necessary to set dependencies
> on Python implementation, including internal sub-syntax (and an even
> more complete generator sub-syntax in progress overlay). I suspect that
> most of the possible variants can be achieved using it, just please
> don't make me try learning it all.
> 
> Additionally, packages supporting multiple Python implementations are
> required to specify their supported Python versions twice -- with
> PYTHON_DEPEND and RESTRICT_PYTHON_ABIS.
> 
> USE flag dependencies are specified using three variables, and I'd like
> to avoid getting anywhere deeper.
> 
> p-d-ng has a very simple dependency setup. It has a variable for
> listing supported Python versions and a variable to make Python
> dependencies conditional under the 'python' flag. Lately, it has been
> added a method of listing requested USE flags.
> 
> p-d-ng does not provide any support for more complex dependency
> specifications, nor a way to disable adding default Python dependencies.
> 
> python-r1 does not append any dependencies by default, and instead
> exports them in a variable. The variable can be used to easily express
> simple cases, and avoids polluting the ebuild in more complex cases. It
> also supports providing USE dependencies for the implementation.
> 
> distutils-r1 adds the Python dependencies to RDEPEND & DEPEND;
> additionally, it adds a dependency on python-exec.
> 
> 
> Dependencies on other packages
> ------------------------------
> 
> As python.eclass (as of gx86 state) does not express enabled
> implementations explicitly, it is not possible to request those
> implementations from other packages. The dependencies on other Python
> packages are thus simple.
> 
> p-d-ng does not provide any simpler way of writing dependencies
> on other Python packages using the eclass. Thus, the necessary USE
> dependencies have to be written by hand.
> 
> python-r1 does provide a simple PYTHON_USEDEP variable which can be
> used in dependency atoms to request the same Python implementations
> being enabled.
> 
> 
> Phase functions for distutils
> -----------------------------
> 
> distutils.eclass allows specifying additional arguments to setup.py.
> src_install() installs default documentation files (*unconditionally*)
> and ${DOCS}, it also does installation image magic.
> 
> p-d-ng has mixed unconditional and user-overridable stuff. Copying
> sources is done unconditionally; redoing scripts can be disabled using
> a separate variable. setup.py invocations are overridable.
> 
> d-r1 splits each phase into 'global' and 'per-implementation' stages
> which are both overridable. The default stages apply PATCHES (and user
> patches), copy sources, run setup.py, install DOCS & HTML_DOCS and link
> python-exec wrappers.
> 
> 
> 
> Thank you for reading up to this. I'd appreciate any comments, ideas,
> and especially any API suggestions while the eclass still ain't live.

Hi!

Do you have any examples of ebuild conversion for non-distutils
using ebuild? I was reading trough these threads but could find
only examples for distutils-r1.

I have a package which requires for each supported python
version to know python executable, include dir and library to
build python bindings (uses cmake) and location for
site-packages so I can manually install them.

Of these there is function only for executable
(python_get_PYTHON). I hope there is a reasonable way to provide
the rest of them without the need for me to resort on guessing
the locations.


Reinis

Re: [gentoo-dev] [RFC] python-r1 + distutils-r1, v2 (+ comparison)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 9761 bytes --]

On Fri, 12 Oct 2012 13:35:44 +0300
Reinis Danne <rei4dan@gmail.com> wrote:

> On Mon, Oct 08, 2012 at 05:16:05PM +0200, Michał Górny wrote:
> > Hello,
> > 
> > This is the second and hopefully last version of python-r1 +
> > distutils-r1 eclasses before committing. I would also like to shortly
> > point out the goals and the differences between various python eclasses
> > in Gentoo.
> > 
> > Both are attached. For those who prefer hosted form:
> > https://bitbucket.org/mgorny/gx86-working-tree/src/5a2d39e69d6d/gx86/eclass/python-r1.eclass
> > https://bitbucket.org/mgorny/gx86-working-tree/src/5a2d39e69d6d/gx86/eclass/distutils-r1.eclass
> > 
> > Changes from the previous version:
> > - support for DOCS and HTML_DOCS (alike EAPI 4);
> > - minimal support for passing arguments to setup.py (through
> >   distutils-r1_python_compile & distutils-r1_python_install);
> > - EPYTHON values adjusted to match python.eclass;
> > - scripts are -${EPYTHON} suffixed -- i.e. foo-python2.7,
> >   foo-pypy-c1.8;
> > - a wrapper script[1] is used to choose the correct script instead of
> >   constant symlink. Thus, EPYTHON & eselect-python are respected,
> > - PYTHON_COMPAT can be an array only.
> > 
> > What still needs considering:
> > - a nicer way of passing setup.py arguments (mysetuppyargs=()?,
> >   esetuppy wrapper?);
> > - Prefix support.
> > 
> > [1]:https://bitbucket.org/mgorny/python-exec
> > 
> > 
> > Now, for the differences part.
> > 
> > Goals
> > -----
> > 
> > python.eclass aims to support every single Python package out there,
> > including rare corner cases supported in-eclass. It supports both
> > packages supporting one and multiple Python implementations. distutils
> > support is provided by second eclass, and both provide phase functions
> > (python.eclass providing default ph.f. wrappers).
> > 
> > python-distutils-ng aims to support majority of Python packages using
> > distutils. With a bit of hackery it can support non-distutils packages,
> > but the use is limited to the common cases. In an uncommon case, it's
> > not flexible enough. It supports Python packages supporting multiple
> > implementations only.
> > 
> > python-r1 aims mostly to provide tools to support majority of Python
> > packages. It tries to be simple & flexible, thus allowing handling
> > various corner cases without special branches of code in the eclass.
> > It doesn't export any phase functions, nor set dependencies by default
> > (just provides a variable with them).
> > 
> > Right now, it supports packages supporting multiple implementations
> > only; if necessary, the support will be extended -- either through
> > additional code if that wouldn't add too much complexity, or through
> > additional eclass.
> > 
> > It is accompanied by distutils-r1 which provides a simple overlay for
> > the very common case of distutils packages. This eclass exports phase
> > functions and sets dependencies implicitly. It also handles renaming
> > the distutils-installed scripts and linking the python-exec wrapper.
> > 
> > 
> > Python targets
> > --------------
> > 
> > python.eclass uses implicit Python target specifications. It provides
> > no ebuild-transparent way of enabling/disabling them.
> > 
> > python-distutils-ng and python-r1 use USE flags to explicitly list
> > enabled Python implementations. Both eclasses use the same values for
> > PYTHON_TARGETS USE_EXPAND.
> > 
> > 
> > Installed scripts
> > -----------------
> > 
> > python-distutils-ng rework the installed scripts creating copies for
> > Python implementations and changing the shebang. The created copies
> > don't differ any other way. The old script name is then symlinked to
> > the version for default implementation.
> > 
> > python.eclass & python-r1 instead let distutils rework the scripts
> > and just rename them before merging. A wrapper is used to choose
> > the correct version, respecting EPYTHON & eselect-python.
> > 
> > python.eclass installs the files to separate installation images
> > and merges them. python-r1 installs them to the main image directly,
> > renaming the installed scripts between successive implementations.
> > 
> > python.eclass creates a wrapper script for each package. The script is
> > written in python. python-r1 instead installs the compiled wrapper
> > once (in an ebuild), and symlinks it for the packages. The wrapper is
> > written in much simpler C.
> > 
> > The implementation suffixes for all three eclasses are different:
> > - python.eclass uses -2.6, -2.7 for Python, -2.7-pypy-1.8,
> >   -2.7-pypy-1.9 for pypy and -2.5-jython for jython (enjoy!);
> > - p-d-ng uses Python target names (-python2_6, -python2_7, -pypy1_8),
> > - p-r1 uses $EPYTHON values (-python2.6, -python2.7, -pypy-c1.8 (sic!)).
> > 
> > 
> > Dependency on Python
> > --------------------
> > 
> > python.eclass has a few variables necessary to set dependencies
> > on Python implementation, including internal sub-syntax (and an even
> > more complete generator sub-syntax in progress overlay). I suspect that
> > most of the possible variants can be achieved using it, just please
> > don't make me try learning it all.
> > 
> > Additionally, packages supporting multiple Python implementations are
> > required to specify their supported Python versions twice -- with
> > PYTHON_DEPEND and RESTRICT_PYTHON_ABIS.
> > 
> > USE flag dependencies are specified using three variables, and I'd like
> > to avoid getting anywhere deeper.
> > 
> > p-d-ng has a very simple dependency setup. It has a variable for
> > listing supported Python versions and a variable to make Python
> > dependencies conditional under the 'python' flag. Lately, it has been
> > added a method of listing requested USE flags.
> > 
> > p-d-ng does not provide any support for more complex dependency
> > specifications, nor a way to disable adding default Python dependencies.
> > 
> > python-r1 does not append any dependencies by default, and instead
> > exports them in a variable. The variable can be used to easily express
> > simple cases, and avoids polluting the ebuild in more complex cases. It
> > also supports providing USE dependencies for the implementation.
> > 
> > distutils-r1 adds the Python dependencies to RDEPEND & DEPEND;
> > additionally, it adds a dependency on python-exec.
> > 
> > 
> > Dependencies on other packages
> > ------------------------------
> > 
> > As python.eclass (as of gx86 state) does not express enabled
> > implementations explicitly, it is not possible to request those
> > implementations from other packages. The dependencies on other Python
> > packages are thus simple.
> > 
> > p-d-ng does not provide any simpler way of writing dependencies
> > on other Python packages using the eclass. Thus, the necessary USE
> > dependencies have to be written by hand.
> > 
> > python-r1 does provide a simple PYTHON_USEDEP variable which can be
> > used in dependency atoms to request the same Python implementations
> > being enabled.
> > 
> > 
> > Phase functions for distutils
> > -----------------------------
> > 
> > distutils.eclass allows specifying additional arguments to setup.py.
> > src_install() installs default documentation files (*unconditionally*)
> > and ${DOCS}, it also does installation image magic.
> > 
> > p-d-ng has mixed unconditional and user-overridable stuff. Copying
> > sources is done unconditionally; redoing scripts can be disabled using
> > a separate variable. setup.py invocations are overridable.
> > 
> > d-r1 splits each phase into 'global' and 'per-implementation' stages
> > which are both overridable. The default stages apply PATCHES (and user
> > patches), copy sources, run setup.py, install DOCS & HTML_DOCS and link
> > python-exec wrappers.
> > 
> > 
> > 
> > Thank you for reading up to this. I'd appreciate any comments, ideas,
> > and especially any API suggestions while the eclass still ain't live.
> 
> Hi!
> 
> Do you have any examples of ebuild conversion for non-distutils
> using ebuild? I was reading trough these threads but could find
> only examples for distutils-r1.

Well, yes, in the earlier thread there was one non-distutils package
requested as an example. It's autotools but the rule is similar. Here
it is:
http://git.overlays.gentoo.org/gitweb/?p=dev/mgorny.git;a=blob;f=dev-python/pygobject/pygobject-3.2.2.ebuild;h=289eacee0a0f52744a16e80d7e91d7bc5987b6a2;hb=7baa032fc4f3bf158d76c187920057c7da57f41d

Basically, you have two options:
1. Use just python-r1, get your own phase functions and use
python_foreach_impl(), If you're using out-of-source builds (which is
quite likely with cmake), it will be probably beneficial to make
cmake-utils.eclass support BUILD_DIR (without prefix).
2. Use distutils-r1 and override python_compile(), python_install()
(and probably python_prepare_all()). Well, assuming it will all play
together nice.

> I have a package which requires for each supported python
> version to know python executable, include dir and library to
> build python bindings (uses cmake) and location for
> site-packages so I can manually install them.
> 
> Of these there is function only for executable
> (python_get_PYTHON). I hope there is a reasonable way to provide
> the rest of them without the need for me to resort on guessing
> the locations.

It should be pretty straightforward to add the remaining getters
to the eclass. But if you don't mind, I'd rather get on to that after
the initial eclass is committed since it's a specific case rather than
a common one.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]