From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 8C2F0138997 for ; Mon, 11 Feb 2013 10:36:22 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 8DC41E0495; Mon, 11 Feb 2013 10:36:21 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id F31ED21C097 for ; Mon, 11 Feb 2013 00:20:57 +0000 (UTC) Received: from pomiocik.lan (unknown [213.146.33.74]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: mgorny) by smtp.gentoo.org (Postfix) with ESMTPSA id 8F5EA33DF13 for ; Mon, 11 Feb 2013 00:20:56 +0000 (UTC) Date: Mon, 11 Feb 2013 01:20:56 +0100 From: =?UTF-8?B?TWljaGHFgiBHw7Nybnk=?= To: gentoo-dev-announce@lists.gentoo.org Subject: [gentoo-dev-announce] New Python eclasses -- a summary and reminder Message-ID: <20130211012056.3df19cc5@pomiocik.lan> Reply-To: gentoo-dev@lists.gentoo.org Organization: Gentoo X-Mailer: Claws Mail 3.9.0 (GTK+ 2.24.14; x86_64-pc-linux-gnu) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo development announcement list X-BeenThere: gentoo-dev-announce@lists.gentoo.org Mime-Version: 1.0 Content-Type: multipart/signed; micalg=PGP-SHA256; boundary="Sig_/X9dBd064ups42V44httpVmg"; protocol="application/pgp-signature" X-Archives-Salt: 843b0b72-f838-4361-95b3-b72340937000 X-Archives-Hash: 0e12be06f2542e7b1c3e182907b42b06 --Sig_/X9dBd064ups42V44httpVmg Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hello, fellow developers. It seems that most of the new Python eclass development has been kept in gentoo-python mailing list and most developers hasn't been properly introduced to them and lack necessary information to use them. For that reason, I'd like to quickly summarize what has changed and how Python ebuilds shall be written nowadays. First of all, I would like to apologize for the official eclass guides [1] being a bit out-of-date. I was short on time lately and I tried to use the remaining bits of it to work on ebuilds and eclasses. For the most recent documentation, please use eclass manpages [2]. Secondly, I'd like to make it clear that the old python.eclass is 'almost' deprecated. We're in process of converting the in-tree packages to use the new eclasses but that's a lot of work [3]. When committing new packages, you are strongly encouraged to use the new eclasses. If you'd like to convert existing package, feel free to -- but please make sure to do so properly, since the new eclasses aren't supposed to be 1:1-compatible with python.eclass. The new eclasses (except for python-any-r1) require EAPI=3D5. That's mostly due to USE_EXPAND USE-deps being useless in PMS variant of EAPI<5. That should not be a problem for most of the packages since EAPI=3D5 portage is stable already and we're expecting users to migrate to an EAPI=3D5 profile. Now, a few quick details you may want to know about the new eclasses and which may confuse you since it's different to what python.eclass used to do. It's a bit late but I hope it helps you. [1]:http://www.gentoo.org/proj/en/Python/#doc_chap5 [2]:http://devmanual.gentoo.org/eclass-reference/index.html [3]:http://dev.gentoo.org/~mgorny/python-r1/conversion.xhtml The eclasses ------------ There are 5 eclasses to choose from. You always inherit only one of the eclasses, the 'parent' ones will be pulled in for you. 1) distutils-r1.eclass -- for *all* packages which use the distutils build system. Replacement for distutils.eclass. 2) python-r1.eclass -- for packages which are being installed for multiple Python implementations at the same time. Replacement for python.eclass with SUPPORT_PYTHON_ABIS=3D1. 3) python-single-r1.eclass -- for packages which can be built for one Python implementation at a time only. Replacement for python.eclass without SUPPORT_PYTHON_ABIS. 4) python-any-r1.eclass -- for packages which need only simple, build-time dependency on Python. Doesn't export USE flags and only uses '|| ( python:X.Y python:X.Z )' kind of deps. Usually replaces non-eclass deps on Python. 5) python-utils-r1.eclass -- helper functions. Inheriting it directly sounds a bit fragile. So, if you package uses distutils (calls setup.py), you always inherit distutils-r1 only. Otherwise, you inherit 2-4 of your choice. Variable and function substitutions ----------------------------------- PYTHON_DEPEND, RESTRICT_PYTHON_ABIS -> PYTHON_COMPAT (an array!) SUPPORT_PYTHON_ABIS -> depends on eclass PYTHON_USE_WITH* -> hand-written USE-dep in PYTHON_REQ_USE DISTUTILS_SRC_TEST -> manual IUSE, DEPEND, python_test() PYTHON_CFLAGS -> has to be done by hand PYTHON_MODNAME -> removed, not necessary anymore DISTUTILS_USE_SEPARATE_SOURCE_DIRECTORIES -> DISTUTILS_IN_SOURCE_BUILD DISTUTILS_GLOBAL_OPTIONS -> mydistutilsargs (an array) DOCS -> an array now, replaces defaults (instead of appending) added HTML_DOCS, PATCHES $(PYTHON) -> ${PYTHON} (path) or ${EPYTHON} (basename) ${PYTHON_ABI} -> use ${EPYTHON} instead [outside of distutils-r1, for distutils-r1 see below] python_execute_function -> python_foreach_impl python_execute_function -f -> python_export_best; ... python_mod_{optimize,cleanup} -> removed $(python_get_sitedir) and friends -> now contain ${EPREFIX} as well insinto $(python_get_...) -> python_domodule, python_doinclude, ... python_convert_shebangs: - on already-installed script -> python_replicate_script, - on to-be-installed scripts -> python_foreach_impl python_doscript ..., - in python-single-r1 -> python_fix_shebang "${D}"... Former two create impl-variants and wrapper. The latter converts only. I think that's all common cases. PYTHON_COMPAT ------------- Eclasses 1-4 use PYTHON_COMPAT to denote supported Python implementations. You have to list all the implementations there. There's no 'but my package supports even Python 4!' Please test packages before adding something there. Grep bugzie for bugs, run test suites, *check dependencies*. If in doubt, add only python2.7 and 3.2 (if both work). We'll add other versions if somebody needs them. distutils-r1 ------------ Most importantly, please don't migrate packages 1:1 from the old eclass. The new eclass design is based on python-distutils-ng. Feel free to look at some of dev-python/ packages to get a feeling how to solve some of the common issues. The packages which can provide solutions for most of the common problems (by having those problems) include: - dev-python/matplotlib -- py2-only deps in py2+3 package (please always open bugs for those), weird config command, - dev-python/numpy -- a lot of horrible, random stuff, - dev-python/pygments -- nosetests and 2to3 conversion, - dev-python/unittest2 -- separate sources for py2 & py3, tests having problems with parallel builds. A few quick notes: 1) src_prepare() to src_install() are replaced each with two sub-phases: - python_prepare_all() to python_install_all() -- which are run once for the whole build process. There you put the common stuff like applying patches, building and installing docs. python_prepare_all() and python_install_all() (only the two) have default functions so remember to call them (as distutils-r1_python_prepare_all). - python_prepare() to python_install() -- which are run repeatedly for each Python implementation being built. There you put the specific stuff like building and installing the sources, running the tests. python_compile() and python_install() have defaults, and those defaults shall be usually called (distutils-r1_python_compile). Use of python_prepare() should be avoided as that implies in-source build. You declare src_*() phases only in special cases (like disabling parallel build for only one phase, see dev-python/nose). Moving 1:1 distutils.eclass python_execute_function snippets is simply wrong. As in, WRONG: src_test() { testing() { footest } python_foreach_impl testing } CORRECT: python_test() { footest || die "Tests fail with ${EPYTHON}" } 2) out-of-source builds + parallel builds usually mean fun but trouble. When working on an ebuild, it's best to set PYTHON_TARGETS=3Dpython2_7 temporarily and get it to work like this. Then you try it with all targets enabled and built in parallel. If you can't fix the package to work properly with that kind of build, try DISTUTILS_IN_SOURCE_BUILD=3D1 first. That usually solves most of the issues, with the cost of keeping multiple source copies. As a last resort, use DISTUTILS_NO_PARALLEL_BUILD=3D1. You may have to use it to avoid excessive use of memory or disk space. If you have to use it for any other reason, please report a bug upstream. If possible, please try to set it only during specific src_*() phases as it helps a lot in making builds faster. 3) non-standard distutils ebuilds The distutils-r1 eclass was designed to handle most common case of distutils use gracefully -- the Python packages. Therefore, it exports phase functions and adds python to DEP and RDEP by default. If you have a package with Python bindings where distutils is only flag-conditional, you want to set DISTUTILS_OPTIONAL=3D1. Then add ${PYTHON_DEPS} to your DEP+RDEP yourself (as with python-r1) and call the phases in 'use python &&'. Or just split the package, it avoids unnecessary rebuilds and makes linking easier/more correct. If you have a distutils package which can be installed for one Python implementation only (like python-single-r1 or distutils.eclass without SUPPORT_PYTHON_ABIS), set DISTUTILS_SINGLE_IMPL=3D1. This will switch the code to use python-single-r1 instead of python-r1. Both of the above variables have to be set before inheriting distutils-r1. python-r1 and python-single-r1 ------------------------------ This is closer to 1:1 from python.eclass. A few notes though: 1) use python-r1 *only* if your package and all of its Python deps support being built for multiple Python implementations. Otherwise, use python-single-r1. As in, WRONG: PYTHON_COMPAT=3D( python2_7 ) # hahaha, tricked him! inherit python-r1 CORRECT: PYTHON_COMPAT=3D( python{2_6,2_7} ) inherit python-single-r1 2) no Python deps are added by default. Use: RDEPEND=3D"${PYTHON_DEPS}" or: RDEPEND=3D"python? ( ${PYTHON_DEPS} )" 3) Python package deps need USE-deps to enforce same implementations being enabled. That is: DEPEND=3D"dev-python/setuptools[${PYTHON_USEDEP}]" If necessary to hack py3 package with py2 deps (please keep a bug open in that case!): RDEPEND=3D"gtk? ( dev-python/pygtk[$(python_gen_usedep python2*)] )" REQUIRED_USE=3D"gtk? ( || ( $(python_gen_useflags python2*) ) )" 4) Please try to use out-of-source builds and avoid copying sources unnecessarily. python_foreach_impl() exports BUILD_DIR to help you with that. autotools-utils and cmake-utils respect that. inherit autotools-utils src_configure() { local myeconfargs=3D( --with-foo #... ) # it builds the package in implementation-specific dir! python_foreach_impl autotools-utils_src_configure } src_compile() { python_foreach_impl autotools-utils_src_compile } If you really have to use an in-source build, python_copy_sources() and run_in_build_dir() functions are at your service: src_prepare() { python_copy_sources } src_configure() { python_foreach_impl run_in_build_dir econf ... } 5) python-r1 doesn't export pkg_setup(). The implementation data gets set in python_foreach_impl or by python_export_best. python-single-r1 exports pkg_setup(). It finds out which implementation was selected and exports all data for it. 6) The generic idea of 'converting shebangs' is discouraged. Packages supporting multiple implementations are supposed to create impl-variants of *all* the scripts. This is only way to enforce that PYTHON_COMPAT is respected properly. In distutils-r1 this is done automatically. In python-r1 packages, you can either: a) install scripts to temporary locations and use python_doscript() to install them. Note that this is done for a single implementation only (so you put it in python_foreach_impl). It installs the script as 'foo-${EPYTHON}' and symlinks the wrapper to 'foo'. If your build system tries to write to 'foo' then, this may end up really bad. b) use python_replicate_script() on an installed script. It will copy the script into variants and then symlink the wrapper. All in one run, don't use python_foreach_impl here. In python-single-r1, you can mangle shebangs to match the selected Python implementation. Use python_fix_shebang for that. Please note that the shebang conversion function is very strict in order to avoid mistakenly breaking something. If it fails with your shebang, please let us know. python-any-r1 ------------- This eclass is something special. Unlike previous eclasses, it does not set any USE flags, so no rebuilds are forced when user changes enabled Python implementations. It can be used to handle build-time deps on Python. Alike with other eclasses, you need to set PYTHON_COMPAT and use PYTHON_DEPS to get proper dependencies. # Build system needs python2.6+ PYTHON_COMPAT=3D( python{2_6,2_7} ) inherit python-any-r1 DEPEND=3D"${PYTHON_DEPS}" It adds '|| ( python:{2_7,2_6} )'. pkg_setup() finds out best installed one and uses it. --=20 Best regards, Micha=C5=82 G=C3=B3rny --Sig_/X9dBd064ups42V44httpVmg Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (GNU/Linux) iJwEAQEIAAYFAlEYOWgACgkQfXuS5UK5QB2vXgP9HuKzTseKtmxbrq90R7it5VQU VK5/4L/YfGPptip4yKtPr8fyHptPb2nVn8Y+QZ7k4agK65YcZmM+JT2QNSnNze/x OQHs65MIzUyIuRcJCj9bu4nNM+YsdxhQdx2LvX4/UGgR4Pn1+xctdUfJ5KXt3iEC 1X3Isb50crcCjKWiUJg= =FoPU -----END PGP SIGNATURE----- --Sig_/X9dBd064ups42V44httpVmg--