* [gentoo-portage-dev] [PATCHES] setup.py install for Portage
@ 2014-08-21 20:19 Michał Górny
2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 1/6] self-update: Copy only relevant packages from PORTAGE_PYM_PATH Michał Górny
` (9 more replies)
0 siblings, 10 replies; 23+ messages in thread
From: Michał Górny @ 2014-08-21 20:19 UTC (permalink / raw
To: gentoo-portage-dev
Hello, everyone.
Here is a rebase of my recent work on bringing setup.py install for
Portage.
About the patches:
(1) teaches the self-update function to deal with PORTAGE_PYM_PATH that
contains more packages than Portage itself. In particular, it makes it
copy packages of Portage only rather than everything found in 'pym' :).
(2) makes PORTAGE_PYM_PATH contain wherever Portage is loaded from
rather than 'pym' subdirectory of PORTAGE_BASE_PATH. This makes the code
work well when 'pym' directory is removed.
(3) makes sure that all important test files are suffixed with .py so
that they are installed by setup.py without extra hackery.
(4) fixes all the remaining issues with tests, allowing them to be run
on the installed copy of Portage.
(5) replaces the Makefiles with setup.py. It's distutils with a few
extensions -- mostly making it possible to install scripts to multiple
locations, install data files recursively and override all the paths.
(6) updates .travis.yml to run tests using setup.py, and also try
installing and re-running tests after install.
I will follow this thread with updated ebuild.
About the new install layout:
1. /usr/lib/portage/pym no longer exists. Python files are installed
directly to site-packages for each implementation (by default).
2. /usr/lib/portage/bin no longer contains copies of the scripts that
are installed to /usr/bin and /usr/sbin.
3. All Python scripts have proper shebangs set by distutils. To fit this
with how distutils usually works and how the eclass expects it to be
done, my ebuild had done a few more changes:
3a. all programs from /usr/sbin are moved to /usr/bin, and all programs
in /usr/bin are wrapped.
3b. The Portage internal tools are moved from /usr/lib/portage/bin
to /usr/lib/portage/${EPYTHON} so that no Python mismatch is possible.
3c. /usr/lib/portage/bin became a symlink for external tool
compatibility. It uses the 'last' Python implementation as chosen
by distutils-r1.
I have tested these changes for a while and I think Portage itself can
handle them well. However, some of the external tools overrelying on
Portage itself can be broken, eix in particular.
In particular, tools that want to play with Portage internals need to
export proper PORTAGE_BIN_PATH and PORTAGE_PYM_PATH before spawning
them. The correct values can be obtained using portageq:
$ portageq envvar PORTAGE_BIN_PATH
/usr/lib/portage/python3.4
$ portageq envvar PORTAGE_PYM_PATH
/usr/lib64/python3.4/site-packages
Please look through the patches and test at will :). Thanks.
--
Michał Górny
^ permalink raw reply [flat|nested] 23+ messages in thread* [gentoo-portage-dev] [PATCH 1/6] self-update: Copy only relevant packages from PORTAGE_PYM_PATH 2014-08-21 20:19 [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny @ 2014-08-21 20:19 ` Michał Górny 2014-08-22 21:47 ` Brian Dolbec 2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 2/6] portage.const: Make PORTAGE_PYM_PATH point to the real package tree Michał Górny ` (8 subsequent siblings) 9 siblings, 1 reply; 23+ messages in thread From: Michał Górny @ 2014-08-21 20:19 UTC (permalink / raw To: gentoo-portage-dev; +Cc: Michał Górny Update the self-update code to copy only packages relevant to Portage, to avoid copying other packages when Portage is installed alongside them. --- pym/portage/const.py | 3 +++ pym/portage/package/ebuild/doebuild.py | 8 ++++++-- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/pym/portage/const.py b/pym/portage/const.py index aab6e8a..4b01ff9 100644 --- a/pym/portage/const.py +++ b/pym/portage/const.py @@ -270,6 +270,9 @@ SUPPORTED_BINPKG_FORMATS = ("tar", "rpm") # Time formats used in various places like metadata.chk. TIMESTAMP_FORMAT = "%a, %d %b %Y %H:%M:%S +0000" # to be used with time.gmtime() +# Top-level names of Python packages installed by Portage. +PORTAGE_PYM_PACKAGES = ("_emerge", "portage", "repoman") + # =========================================================================== # END OF CONSTANTS -- END OF CONSTANTS -- END OF CONSTANTS -- END OF CONSTANT # =========================================================================== diff --git a/pym/portage/package/ebuild/doebuild.py b/pym/portage/package/ebuild/doebuild.py index 01707ae..d3e3f5a 100644 --- a/pym/portage/package/ebuild/doebuild.py +++ b/pym/portage/package/ebuild/doebuild.py @@ -45,7 +45,7 @@ from portage import auxdbkeys, bsd_chflags, \ unmerge, _encodings, _os_merge, \ _shell_quote, _unicode_decode, _unicode_encode from portage.const import EBUILD_SH_ENV_FILE, EBUILD_SH_ENV_DIR, \ - EBUILD_SH_BINARY, INVALID_ENV_FILE, MISC_SH_BINARY + EBUILD_SH_BINARY, INVALID_ENV_FILE, MISC_SH_BINARY, PORTAGE_PYM_PACKAGES from portage.data import portage_gid, portage_uid, secpass, \ uid, userpriv_groups from portage.dbapi.porttree import _parse_uri_map @@ -2327,7 +2327,11 @@ def _prepare_self_update(settings): orig_pym_path = portage._pym_path portage._pym_path = os.path.join(base_path_tmp, "pym") - shutil.copytree(orig_pym_path, portage._pym_path, symlinks=True) + os.mkdir(portage._pym_path) + for pmod in PORTAGE_PYM_PACKAGES: + shutil.copytree(os.path.join(orig_pym_path, pmod), + os.path.join(portage._pym_path, pmod), + symlinks=True) for dir_path in (base_path_tmp, portage._bin_path, portage._pym_path): os.chmod(dir_path, 0o755) -- 2.0.4 ^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [gentoo-portage-dev] [PATCH 1/6] self-update: Copy only relevant packages from PORTAGE_PYM_PATH 2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 1/6] self-update: Copy only relevant packages from PORTAGE_PYM_PATH Michał Górny @ 2014-08-22 21:47 ` Brian Dolbec 0 siblings, 0 replies; 23+ messages in thread From: Brian Dolbec @ 2014-08-22 21:47 UTC (permalink / raw To: gentoo-portage-dev On Thu, 21 Aug 2014 22:19:39 +0200 Michał Górny <mgorny@gentoo.org> wrote: > Update the self-update code to copy only packages relevant to Portage, > to avoid copying other packages when Portage is installed alongside > them. > --- > pym/portage/const.py | 3 +++ > pym/portage/package/ebuild/doebuild.py | 8 ++++++-- > 2 files changed, 9 insertions(+), 2 deletions(-) > Seems fine to me... -- Brian Dolbec <dolsen> ^ permalink raw reply [flat|nested] 23+ messages in thread
* [gentoo-portage-dev] [PATCH 2/6] portage.const: Make PORTAGE_PYM_PATH point to the real package tree 2014-08-21 20:19 [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny 2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 1/6] self-update: Copy only relevant packages from PORTAGE_PYM_PATH Michał Górny @ 2014-08-21 20:19 ` Michał Górny 2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 3/6] tests: Append .py to files that need to be installed Michał Górny ` (7 subsequent siblings) 9 siblings, 0 replies; 23+ messages in thread From: Michał Górny @ 2014-08-21 20:19 UTC (permalink / raw To: gentoo-portage-dev; +Cc: Michał Górny Change PORTAGE_PYM_PATH to use real package tree path rather than one based on PORTAGE_BASE_PATH. --- pym/portage/const.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pym/portage/const.py b/pym/portage/const.py index 4b01ff9..f518b47 100644 --- a/pym/portage/const.py +++ b/pym/portage/const.py @@ -63,7 +63,7 @@ GLOBAL_CONFIG_PATH = "/usr/share/portage/config" PORTAGE_BASE_PATH = os.path.join(os.sep, os.sep.join(os.path.realpath( __file__.rstrip("co")).split(os.sep)[:-3])) PORTAGE_BIN_PATH = PORTAGE_BASE_PATH + "/bin" -PORTAGE_PYM_PATH = PORTAGE_BASE_PATH + "/pym" +PORTAGE_PYM_PATH = os.path.realpath(os.path.join(__file__, '../..')) LOCALE_DATA_PATH = PORTAGE_BASE_PATH + "/locale" # FIXME: not used EBUILD_SH_BINARY = PORTAGE_BIN_PATH + "/ebuild.sh" MISC_SH_BINARY = PORTAGE_BIN_PATH + "/misc-functions.sh" -- 2.0.4 ^ permalink raw reply related [flat|nested] 23+ messages in thread
* [gentoo-portage-dev] [PATCH 3/6] tests: Append .py to files that need to be installed 2014-08-21 20:19 [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny 2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 1/6] self-update: Copy only relevant packages from PORTAGE_PYM_PATH Michał Górny 2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 2/6] portage.const: Make PORTAGE_PYM_PATH point to the real package tree Michał Górny @ 2014-08-21 20:19 ` Michał Górny 2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 4/6] tests: Fix running on installed copy of Portage Michał Górny ` (6 subsequent siblings) 9 siblings, 0 replies; 23+ messages in thread From: Michał Górny @ 2014-08-21 20:19 UTC (permalink / raw To: gentoo-portage-dev; +Cc: Michał Górny Append .py to the runTests script and __test__ files so that they could be easily picked up by setup.py. --- .travis.yml | 2 +- Makefile | 2 +- pym/portage/tests/__init__.py | 4 +- pym/portage/tests/bin/__test__ | 0 pym/portage/tests/bin/__test__.py | 0 pym/portage/tests/dbapi/__test__ | 0 pym/portage/tests/dbapi/__test__.py | 0 pym/portage/tests/dep/__test__ | 0 pym/portage/tests/dep/__test__.py | 0 pym/portage/tests/ebuild/__test__ | 0 pym/portage/tests/ebuild/__test__.py | 0 pym/portage/tests/emerge/__test__ | 0 pym/portage/tests/emerge/__test__.py | 0 pym/portage/tests/env/__test__ | 0 pym/portage/tests/env/__test__.py | 0 pym/portage/tests/env/config/__test__ | 0 pym/portage/tests/env/config/__test__.py | 0 pym/portage/tests/glsa/__test__ | 0 pym/portage/tests/glsa/__test__.py | 0 pym/portage/tests/lafilefixer/__test__ | 0 pym/portage/tests/lafilefixer/__test__.py | 0 pym/portage/tests/lazyimport/__test__ | 0 pym/portage/tests/lazyimport/__test__.py | 0 pym/portage/tests/lint/__test__ | 0 pym/portage/tests/lint/__test__.py | 0 pym/portage/tests/locks/__test__ | 0 pym/portage/tests/locks/__test__.py | 0 pym/portage/tests/news/__test__ | 0 pym/portage/tests/news/__test__.py | 0 pym/portage/tests/process/__test__ | 0 pym/portage/tests/process/__test__.py | 0 pym/portage/tests/repoman/__test__ | 0 pym/portage/tests/repoman/__test__.py | 0 pym/portage/tests/resolver/__test__ | 0 pym/portage/tests/resolver/__test__.py | 0 pym/portage/tests/runTests | 61 ------------------------------- pym/portage/tests/runTests.py | 61 +++++++++++++++++++++++++++++++ pym/portage/tests/sets/base/__test__ | 0 pym/portage/tests/sets/base/__test__.py | 0 pym/portage/tests/sets/files/__test__ | 0 pym/portage/tests/sets/files/__test__.py | 0 pym/portage/tests/sets/shell/__test__ | 0 pym/portage/tests/sets/shell/__test__.py | 0 pym/portage/tests/unicode/__test__ | 0 pym/portage/tests/unicode/__test__.py | 0 pym/portage/tests/update/__test__ | 0 pym/portage/tests/update/__test__.py | 0 pym/portage/tests/util/__test__ | 0 pym/portage/tests/util/__test__.py | 0 pym/portage/tests/versions/__test__ | 0 pym/portage/tests/versions/__test__.py | 0 pym/portage/tests/xpak/__test__ | 0 pym/portage/tests/xpak/__test__.py | 0 runtests.sh | 2 +- 54 files changed, 66 insertions(+), 66 deletions(-) delete mode 100644 pym/portage/tests/bin/__test__ create mode 100644 pym/portage/tests/bin/__test__.py delete mode 100644 pym/portage/tests/dbapi/__test__ create mode 100644 pym/portage/tests/dbapi/__test__.py delete mode 100644 pym/portage/tests/dep/__test__ create mode 100644 pym/portage/tests/dep/__test__.py delete mode 100644 pym/portage/tests/ebuild/__test__ create mode 100644 pym/portage/tests/ebuild/__test__.py delete mode 100644 pym/portage/tests/emerge/__test__ create mode 100644 pym/portage/tests/emerge/__test__.py delete mode 100644 pym/portage/tests/env/__test__ create mode 100644 pym/portage/tests/env/__test__.py delete mode 100644 pym/portage/tests/env/config/__test__ create mode 100644 pym/portage/tests/env/config/__test__.py delete mode 100644 pym/portage/tests/glsa/__test__ create mode 100644 pym/portage/tests/glsa/__test__.py delete mode 100644 pym/portage/tests/lafilefixer/__test__ create mode 100644 pym/portage/tests/lafilefixer/__test__.py delete mode 100644 pym/portage/tests/lazyimport/__test__ create mode 100644 pym/portage/tests/lazyimport/__test__.py delete mode 100644 pym/portage/tests/lint/__test__ create mode 100644 pym/portage/tests/lint/__test__.py delete mode 100644 pym/portage/tests/locks/__test__ create mode 100644 pym/portage/tests/locks/__test__.py delete mode 100644 pym/portage/tests/news/__test__ create mode 100644 pym/portage/tests/news/__test__.py delete mode 100644 pym/portage/tests/process/__test__ create mode 100644 pym/portage/tests/process/__test__.py delete mode 100644 pym/portage/tests/repoman/__test__ create mode 100644 pym/portage/tests/repoman/__test__.py delete mode 100644 pym/portage/tests/resolver/__test__ create mode 100644 pym/portage/tests/resolver/__test__.py delete mode 100755 pym/portage/tests/runTests create mode 100755 pym/portage/tests/runTests.py delete mode 100644 pym/portage/tests/sets/base/__test__ create mode 100644 pym/portage/tests/sets/base/__test__.py delete mode 100644 pym/portage/tests/sets/files/__test__ create mode 100644 pym/portage/tests/sets/files/__test__.py delete mode 100644 pym/portage/tests/sets/shell/__test__ create mode 100644 pym/portage/tests/sets/shell/__test__.py delete mode 100644 pym/portage/tests/unicode/__test__ create mode 100644 pym/portage/tests/unicode/__test__.py delete mode 100644 pym/portage/tests/update/__test__ create mode 100644 pym/portage/tests/update/__test__.py delete mode 100644 pym/portage/tests/util/__test__ create mode 100644 pym/portage/tests/util/__test__.py delete mode 100644 pym/portage/tests/versions/__test__ create mode 100644 pym/portage/tests/versions/__test__.py delete mode 100644 pym/portage/tests/xpak/__test__ create mode 100644 pym/portage/tests/xpak/__test__.py diff --git a/.travis.yml b/.travis.yml index 6c8d873..52ca444 100644 --- a/.travis.yml +++ b/.travis.yml @@ -7,4 +7,4 @@ python: - 3.4 - pypy script: - - python -b -Wd pym/portage/tests/runTests + - python -b -Wd pym/portage/tests/runTests.py diff --git a/Makefile b/Makefile index 92ea195..9eb6e66 100644 --- a/Makefile +++ b/Makefile @@ -66,7 +66,7 @@ epydoc: test: set -e; \ - "$(srcdir)/pym/portage/tests/runTests"; \ + "$(srcdir)/pym/portage/tests/runTests.py"; \ install: set -e; \ diff --git a/pym/portage/tests/__init__.py b/pym/portage/tests/__init__.py index 84e732a..697b800 100644 --- a/pym/portage/tests/__init__.py +++ b/pym/portage/tests/__init__.py @@ -83,11 +83,11 @@ def getTestFromCommandLine(args, base_path): return result def getTestDirs(base_path): - TEST_FILE = b'__test__' + TEST_FILE = b'__test__.py' testDirs = [] # the os.walk help mentions relative paths as being quirky - # I was tired of adding dirs to the list, so now we add __test__ + # I was tired of adding dirs to the list, so now we add __test__.py # to each dir we want tested. for root, dirs, files in os.walk(base_path): try: diff --git a/pym/portage/tests/bin/__test__ b/pym/portage/tests/bin/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/bin/__test__.py b/pym/portage/tests/bin/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/pym/portage/tests/dbapi/__test__ b/pym/portage/tests/dbapi/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/dbapi/__test__.py b/pym/portage/tests/dbapi/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/pym/portage/tests/dep/__test__ b/pym/portage/tests/dep/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/dep/__test__.py b/pym/portage/tests/dep/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/pym/portage/tests/ebuild/__test__ b/pym/portage/tests/ebuild/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/ebuild/__test__.py b/pym/portage/tests/ebuild/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/pym/portage/tests/emerge/__test__ b/pym/portage/tests/emerge/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/emerge/__test__.py b/pym/portage/tests/emerge/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/pym/portage/tests/env/__test__ b/pym/portage/tests/env/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/env/__test__.py b/pym/portage/tests/env/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/pym/portage/tests/env/config/__test__ b/pym/portage/tests/env/config/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/env/config/__test__.py b/pym/portage/tests/env/config/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/pym/portage/tests/glsa/__test__ b/pym/portage/tests/glsa/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/glsa/__test__.py b/pym/portage/tests/glsa/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/pym/portage/tests/lafilefixer/__test__ b/pym/portage/tests/lafilefixer/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/lafilefixer/__test__.py b/pym/portage/tests/lafilefixer/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/pym/portage/tests/lazyimport/__test__ b/pym/portage/tests/lazyimport/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/lazyimport/__test__.py b/pym/portage/tests/lazyimport/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/pym/portage/tests/lint/__test__ b/pym/portage/tests/lint/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/lint/__test__.py b/pym/portage/tests/lint/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/pym/portage/tests/locks/__test__ b/pym/portage/tests/locks/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/locks/__test__.py b/pym/portage/tests/locks/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/pym/portage/tests/news/__test__ b/pym/portage/tests/news/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/news/__test__.py b/pym/portage/tests/news/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/pym/portage/tests/process/__test__ b/pym/portage/tests/process/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/process/__test__.py b/pym/portage/tests/process/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/pym/portage/tests/repoman/__test__ b/pym/portage/tests/repoman/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/repoman/__test__.py b/pym/portage/tests/repoman/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/pym/portage/tests/resolver/__test__ b/pym/portage/tests/resolver/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/resolver/__test__.py b/pym/portage/tests/resolver/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/pym/portage/tests/runTests b/pym/portage/tests/runTests deleted file mode 100755 index 9c45276..0000000 --- a/pym/portage/tests/runTests +++ /dev/null @@ -1,61 +0,0 @@ -#!/usr/bin/python -bWd -# runTests.py -- Portage Unit Test Functionality -# Copyright 2006-2014 Gentoo Foundation -# Distributed under the terms of the GNU General Public License v2 - -import os, sys -import os.path as osp -import grp -import platform -import pwd -import signal - -def debug_signal(signum, frame): - import pdb - pdb.set_trace() - -if platform.python_implementation() == 'Jython': - debug_signum = signal.SIGUSR2 # bug #424259 -else: - debug_signum = signal.SIGUSR1 - -signal.signal(debug_signum, debug_signal) - -# Pretend that the current user's uid/gid are the 'portage' uid/gid, -# so things go smoothly regardless of the current user and global -# user/group configuration. -os.environ["PORTAGE_USERNAME"] = pwd.getpwuid(os.getuid()).pw_name -os.environ["PORTAGE_GRPNAME"] = grp.getgrgid(os.getgid()).gr_name - -# Insert our parent dir so we can do shiny import "tests" -# This line courtesy of Marienz and Pkgcore ;) -sys.path.insert(0, osp.dirname(osp.dirname(osp.dirname(osp.realpath(__file__))))) - -import portage -portage._internal_caller = True - -# Ensure that we don't instantiate portage.settings, so that tests should -# work the same regardless of global configuration file state/existence. -portage._disable_legacy_globals() - -if os.environ.get('NOCOLOR') in ('yes', 'true'): - portage.output.nocolor() - -import portage.tests as tests -from portage.const import PORTAGE_BIN_PATH -path = os.environ.get("PATH", "").split(":") -path = [x for x in path if x] - -insert_bin_path = True -try: - insert_bin_path = not path or \ - not os.path.samefile(path[0], PORTAGE_BIN_PATH) -except OSError: - pass - -if insert_bin_path: - path.insert(0, PORTAGE_BIN_PATH) - os.environ["PATH"] = ":".join(path) - -if __name__ == "__main__": - sys.exit(tests.main()) diff --git a/pym/portage/tests/runTests.py b/pym/portage/tests/runTests.py new file mode 100755 index 0000000..9c45276 --- /dev/null +++ b/pym/portage/tests/runTests.py @@ -0,0 +1,61 @@ +#!/usr/bin/python -bWd +# runTests.py -- Portage Unit Test Functionality +# Copyright 2006-2014 Gentoo Foundation +# Distributed under the terms of the GNU General Public License v2 + +import os, sys +import os.path as osp +import grp +import platform +import pwd +import signal + +def debug_signal(signum, frame): + import pdb + pdb.set_trace() + +if platform.python_implementation() == 'Jython': + debug_signum = signal.SIGUSR2 # bug #424259 +else: + debug_signum = signal.SIGUSR1 + +signal.signal(debug_signum, debug_signal) + +# Pretend that the current user's uid/gid are the 'portage' uid/gid, +# so things go smoothly regardless of the current user and global +# user/group configuration. +os.environ["PORTAGE_USERNAME"] = pwd.getpwuid(os.getuid()).pw_name +os.environ["PORTAGE_GRPNAME"] = grp.getgrgid(os.getgid()).gr_name + +# Insert our parent dir so we can do shiny import "tests" +# This line courtesy of Marienz and Pkgcore ;) +sys.path.insert(0, osp.dirname(osp.dirname(osp.dirname(osp.realpath(__file__))))) + +import portage +portage._internal_caller = True + +# Ensure that we don't instantiate portage.settings, so that tests should +# work the same regardless of global configuration file state/existence. +portage._disable_legacy_globals() + +if os.environ.get('NOCOLOR') in ('yes', 'true'): + portage.output.nocolor() + +import portage.tests as tests +from portage.const import PORTAGE_BIN_PATH +path = os.environ.get("PATH", "").split(":") +path = [x for x in path if x] + +insert_bin_path = True +try: + insert_bin_path = not path or \ + not os.path.samefile(path[0], PORTAGE_BIN_PATH) +except OSError: + pass + +if insert_bin_path: + path.insert(0, PORTAGE_BIN_PATH) + os.environ["PATH"] = ":".join(path) + +if __name__ == "__main__": + sys.exit(tests.main()) diff --git a/pym/portage/tests/sets/base/__test__ b/pym/portage/tests/sets/base/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/sets/base/__test__.py b/pym/portage/tests/sets/base/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/pym/portage/tests/sets/files/__test__ b/pym/portage/tests/sets/files/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/sets/files/__test__.py b/pym/portage/tests/sets/files/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/pym/portage/tests/sets/shell/__test__ b/pym/portage/tests/sets/shell/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/sets/shell/__test__.py b/pym/portage/tests/sets/shell/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/pym/portage/tests/unicode/__test__ b/pym/portage/tests/unicode/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/unicode/__test__.py b/pym/portage/tests/unicode/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/pym/portage/tests/update/__test__ b/pym/portage/tests/update/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/update/__test__.py b/pym/portage/tests/update/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/pym/portage/tests/util/__test__ b/pym/portage/tests/util/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/util/__test__.py b/pym/portage/tests/util/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/pym/portage/tests/versions/__test__ b/pym/portage/tests/versions/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/versions/__test__.py b/pym/portage/tests/versions/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/pym/portage/tests/xpak/__test__ b/pym/portage/tests/xpak/__test__ deleted file mode 100644 index e69de29..0000000 diff --git a/pym/portage/tests/xpak/__test__.py b/pym/portage/tests/xpak/__test__.py new file mode 100644 index 0000000..e69de29 diff --git a/runtests.sh b/runtests.sh index 7999220..599727b 100755 --- a/runtests.sh +++ b/runtests.sh @@ -67,7 +67,7 @@ for version in ${PYTHON_VERSIONS}; do fi if [[ -x "${executable}" ]]; then echo -e "${GOOD}Testing with Python ${version}...${NORMAL}" - "${executable}" -b -Wd pym/portage/tests/runTests "$@" + "${executable}" -b -Wd pym/portage/tests/runTests.py "$@" status=$? status_array[${#status_array[@]}]=${status} found_versions[${#found_versions[@]}]=${version} -- 2.0.4 ^ permalink raw reply related [flat|nested] 23+ messages in thread
* [gentoo-portage-dev] [PATCH 4/6] tests: Fix running on installed copy of Portage 2014-08-21 20:19 [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny ` (2 preceding siblings ...) 2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 3/6] tests: Append .py to files that need to be installed Michał Górny @ 2014-08-21 20:19 ` Michał Górny 2014-08-22 22:19 ` Brian Dolbec 2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 5/6] Install Python modules using setup.py Michał Górny ` (5 subsequent siblings) 9 siblings, 1 reply; 23+ messages in thread From: Michał Górny @ 2014-08-21 20:19 UTC (permalink / raw To: gentoo-portage-dev; +Cc: Michał Górny --- pym/portage/tests/__init__.py | 20 ++++++++++++++++++ pym/portage/tests/dbapi/test_portdb_cache.py | 5 ++--- pym/portage/tests/emerge/test_emerge_slot_abi.py | 7 +++---- pym/portage/tests/emerge/test_simple.py | 26 ++++++++++++------------ pym/portage/tests/lint/test_compile_modules.py | 10 +++++---- pym/portage/tests/lint/test_import_modules.py | 8 ++++++-- pym/portage/tests/repoman/test_simple.py | 8 +++++--- pym/portage/tests/resolver/ResolverPlayground.py | 8 +++++--- pym/portage/tests/util/test_getconfig.py | 5 ++--- 9 files changed, 62 insertions(+), 35 deletions(-) diff --git a/pym/portage/tests/__init__.py b/pym/portage/tests/__init__.py index 697b800..708dd59 100644 --- a/pym/portage/tests/__init__.py +++ b/pym/portage/tests/__init__.py @@ -25,8 +25,24 @@ import portage from portage import os from portage import _encodings from portage import _unicode_decode +from portage.const import (EPREFIX, GLOBAL_CONFIG_PATH, PORTAGE_BASE_PATH, + PORTAGE_BIN_PATH) from portage.util._argparse import ArgumentParser + +if portage._not_installed: + cnf_path = os.path.join(PORTAGE_BASE_PATH, 'cnf') + cnf_etc_path = cnf_path + cnf_bindir = PORTAGE_BIN_PATH + cnf_sbindir = cnf_bindir +else: + cnf_path = os.path.join(EPREFIX or '/', GLOBAL_CONFIG_PATH) + cnf_etc_path = os.path.join(EPREFIX or '/', 'etc') + cnf_eprefix = EPREFIX + cnf_bindir = os.path.join(EPREFIX or '/', 'usr', 'bin') + cnf_sbindir = os.path.join(EPREFIX or '/', 'usr', 'sbin') + + def main(): suite = unittest.TestSuite() basedir = os.path.dirname(os.path.realpath(__file__)) @@ -178,6 +194,10 @@ class TestCase(unittest.TestCase): unittest.TestCase.__init__(self, *pargs, **kwargs) self.todo = False self.portage_skip = None + self.cnf_path = cnf_path + self.cnf_etc_path = cnf_etc_path + self.bindir = cnf_bindir + self.sbindir = cnf_sbindir def defaultTestResult(self): return TextTestResult() diff --git a/pym/portage/tests/dbapi/test_portdb_cache.py b/pym/portage/tests/dbapi/test_portdb_cache.py index 94af96e..25e65f6 100644 --- a/pym/portage/tests/dbapi/test_portdb_cache.py +++ b/pym/portage/tests/dbapi/test_portdb_cache.py @@ -8,8 +8,7 @@ import textwrap import portage from portage import os from portage import _unicode_decode -from portage.const import (BASH_BINARY, PORTAGE_BIN_PATH, - PORTAGE_PYM_PATH, USER_CONFIG_PATH) +from portage.const import (BASH_BINARY, PORTAGE_PYM_PATH, USER_CONFIG_PATH) from portage.tests import TestCase from portage.tests.resolver.ResolverPlayground import ResolverPlayground from portage.util import ensure_dirs @@ -38,7 +37,7 @@ class PortdbCacheTestCase(TestCase): portage_python = portage._python_interpreter egencache_cmd = (portage_python, "-b", "-Wd", - os.path.join(PORTAGE_BIN_PATH, "egencache"), + os.path.join(self.bindir, "egencache"), "--repo", "test_repo", "--repositories-configuration", settings.repositories.config_string()) python_cmd = (portage_python, "-b", "-Wd", "-c") diff --git a/pym/portage/tests/emerge/test_emerge_slot_abi.py b/pym/portage/tests/emerge/test_emerge_slot_abi.py index fd7ec0e..d1f2d92 100644 --- a/pym/portage/tests/emerge/test_emerge_slot_abi.py +++ b/pym/portage/tests/emerge/test_emerge_slot_abi.py @@ -7,8 +7,7 @@ import sys import portage from portage import os from portage import _unicode_decode -from portage.const import (BASH_BINARY, PORTAGE_BIN_PATH, - PORTAGE_PYM_PATH, USER_CONFIG_PATH) +from portage.const import (BASH_BINARY, PORTAGE_PYM_PATH, USER_CONFIG_PATH) from portage.process import find_binary from portage.tests import TestCase from portage.tests.resolver.ResolverPlayground import ResolverPlayground @@ -70,9 +69,9 @@ class SlotAbiEmergeTestCase(TestCase): portage_python = portage._python_interpreter ebuild_cmd = (portage_python, "-b", "-Wd", - os.path.join(PORTAGE_BIN_PATH, "ebuild")) + os.path.join(self.bindir, "ebuild")) emerge_cmd = (portage_python, "-b", "-Wd", - os.path.join(PORTAGE_BIN_PATH, "emerge")) + os.path.join(self.bindir, "emerge")) test_ebuild = portdb.findname("dev-libs/dbus-glib-0.98") self.assertFalse(test_ebuild is None) diff --git a/pym/portage/tests/emerge/test_simple.py b/pym/portage/tests/emerge/test_simple.py index bf0af8b..9c1b1bf 100644 --- a/pym/portage/tests/emerge/test_simple.py +++ b/pym/portage/tests/emerge/test_simple.py @@ -8,7 +8,7 @@ import portage from portage import os from portage import _unicode_decode from portage.const import (BASH_BINARY, PORTAGE_BASE_PATH, - PORTAGE_BIN_PATH, PORTAGE_PYM_PATH, USER_CONFIG_PATH) + PORTAGE_PYM_PATH, USER_CONFIG_PATH) from portage.process import find_binary from portage.tests import TestCase from portage.tests.resolver.ResolverPlayground import ResolverPlayground @@ -175,29 +175,29 @@ pkg_preinst() { portage_python = portage._python_interpreter dispatch_conf_cmd = (portage_python, "-b", "-Wd", - os.path.join(PORTAGE_BIN_PATH, "dispatch-conf")) + os.path.join(self.sbindir, "dispatch-conf")) ebuild_cmd = (portage_python, "-b", "-Wd", - os.path.join(PORTAGE_BIN_PATH, "ebuild")) + os.path.join(self.bindir, "ebuild")) egencache_cmd = (portage_python, "-b", "-Wd", - os.path.join(PORTAGE_BIN_PATH, "egencache"), + os.path.join(self.bindir, "egencache"), "--repo", "test_repo", "--repositories-configuration", settings.repositories.config_string()) emerge_cmd = (portage_python, "-b", "-Wd", - os.path.join(PORTAGE_BIN_PATH, "emerge")) + os.path.join(self.bindir, "emerge")) emaint_cmd = (portage_python, "-b", "-Wd", - os.path.join(PORTAGE_BIN_PATH, "emaint")) + os.path.join(self.sbindir, "emaint")) env_update_cmd = (portage_python, "-b", "-Wd", - os.path.join(PORTAGE_BIN_PATH, "env-update")) + os.path.join(self.sbindir, "env-update")) etc_update_cmd = (BASH_BINARY, - os.path.join(PORTAGE_BIN_PATH, "etc-update")) + os.path.join(self.sbindir, "etc-update")) fixpackages_cmd = (portage_python, "-b", "-Wd", - os.path.join(PORTAGE_BIN_PATH, "fixpackages")) + os.path.join(self.sbindir, "fixpackages")) portageq_cmd = (portage_python, "-b", "-Wd", - os.path.join(PORTAGE_BIN_PATH, "portageq")) + os.path.join(self.bindir, "portageq")) quickpkg_cmd = (portage_python, "-b", "-Wd", - os.path.join(PORTAGE_BIN_PATH, "quickpkg")) + os.path.join(self.bindir, "quickpkg")) regenworld_cmd = (portage_python, "-b", "-Wd", - os.path.join(PORTAGE_BIN_PATH, "regenworld")) + os.path.join(self.sbindir, "regenworld")) rm_binary = find_binary("rm") self.assertEqual(rm_binary is None, False, @@ -368,7 +368,7 @@ pkg_preinst() { for x in true_symlinks: os.symlink(true_binary, os.path.join(fake_bin, x)) for x in etc_symlinks: - os.symlink(os.path.join(PORTAGE_BASE_PATH, "cnf", x), + os.symlink(os.path.join(self.cnf_etc_path, x), os.path.join(eprefix, "etc", x)) with open(os.path.join(var_cache_edb, "counter"), 'wb') as f: f.write(b"100") diff --git a/pym/portage/tests/lint/test_compile_modules.py b/pym/portage/tests/lint/test_compile_modules.py index ce7e3fb..4826cad 100644 --- a/pym/portage/tests/lint/test_compile_modules.py +++ b/pym/portage/tests/lint/test_compile_modules.py @@ -5,7 +5,7 @@ import errno import itertools import stat -from portage.const import PORTAGE_BIN_PATH, PORTAGE_PYM_PATH +from portage.const import PORTAGE_BIN_PATH, PORTAGE_PYM_PATH, PORTAGE_PYM_PACKAGES from portage.tests import TestCase from portage import os from portage import _encodings @@ -14,9 +14,11 @@ from portage import _unicode_decode, _unicode_encode class CompileModulesTestCase(TestCase): def testCompileModules(self): - for parent, _dirs, files in itertools.chain( - os.walk(PORTAGE_BIN_PATH), - os.walk(PORTAGE_PYM_PATH)): + iters = [os.walk(os.path.join(PORTAGE_PYM_PATH, x)) + for x in PORTAGE_PYM_PACKAGES] + iters.append(os.walk(PORTAGE_BIN_PATH)) + + for parent, _dirs, files in itertools.chain(*iters): parent = _unicode_decode(parent, encoding=_encodings['fs'], errors='strict') for x in files: diff --git a/pym/portage/tests/lint/test_import_modules.py b/pym/portage/tests/lint/test_import_modules.py index 34261f4..fcdcb3b 100644 --- a/pym/portage/tests/lint/test_import_modules.py +++ b/pym/portage/tests/lint/test_import_modules.py @@ -1,7 +1,9 @@ # Copyright 2011-2012 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 -from portage.const import PORTAGE_PYM_PATH +from itertools import chain + +from portage.const import PORTAGE_PYM_PATH, PORTAGE_PYM_PACKAGES from portage.tests import TestCase from portage import os from portage import _encodings @@ -13,7 +15,9 @@ class ImportModulesTestCase(TestCase): expected_failures = frozenset(( )) - for mod in self._iter_modules(PORTAGE_PYM_PATH): + iters = (self._iter_modules(os.path.join(PORTAGE_PYM_PATH, x)) + for x in PORTAGE_PYM_PACKAGES) + for mod in chain(*iters): try: __import__(mod) except ImportError as e: diff --git a/pym/portage/tests/repoman/test_simple.py b/pym/portage/tests/repoman/test_simple.py index 69eb36d..5dbb767 100644 --- a/pym/portage/tests/repoman/test_simple.py +++ b/pym/portage/tests/repoman/test_simple.py @@ -9,7 +9,7 @@ import portage from portage import os from portage import shutil from portage import _unicode_decode -from portage.const import PORTAGE_BASE_PATH, PORTAGE_BIN_PATH, PORTAGE_PYM_PATH +from portage.const import PORTAGE_BASE_PATH, PORTAGE_PYM_PATH from portage.process import find_binary from portage.tests import TestCase from portage.tests.resolver.ResolverPlayground import ResolverPlayground @@ -171,7 +171,7 @@ class SimpleRepomanTestCase(TestCase): license_dir = os.path.join(test_repo_location, "licenses") repoman_cmd = (portage._python_interpreter, "-b", "-Wd", - os.path.join(PORTAGE_BIN_PATH, "repoman")) + os.path.join(self.bindir, "repoman")) git_binary = find_binary("git") git_cmd = (git_binary,) @@ -274,7 +274,9 @@ class SimpleRepomanTestCase(TestCase): os.symlink(test_repo_location, test_repo_symlink) # repoman checks metadata.dtd for recent CTIME, so copy the file in # order to ensure that the CTIME is current - shutil.copyfile(metadata_dtd, os.path.join(distdir, "metadata.dtd")) + # NOTE: if we don't have the file around, let repoman try to fetch it. + if os.path.exists(metadata_dtd): + shutil.copyfile(metadata_dtd, os.path.join(distdir, "metadata.dtd")) if debug: # The subprocess inherits both stdout and stderr, for diff --git a/pym/portage/tests/resolver/ResolverPlayground.py b/pym/portage/tests/resolver/ResolverPlayground.py index 077e271..675e23f 100644 --- a/pym/portage/tests/resolver/ResolverPlayground.py +++ b/pym/portage/tests/resolver/ResolverPlayground.py @@ -15,6 +15,7 @@ from portage.package.ebuild.config import config from portage.package.ebuild.digestgen import digestgen from portage._sets import load_default_config from portage._sets.base import InternalPackageSet +from portage.tests import cnf_path from portage.util import ensure_dirs, normalize_path from portage.versions import catsplit @@ -65,6 +66,7 @@ class ResolverPlayground(object): If a metadata key is missing, it gets a default value. profile: settings defined by the profile. """ + self.debug = debug self.eprefix = normalize_path(tempfile.mkdtemp()) portage.const.EPREFIX = self.eprefix.rstrip(os.sep) @@ -414,7 +416,7 @@ class ResolverPlayground(object): make_globals_path = os.path.join(self.eroot, GLOBAL_CONFIG_PATH.lstrip(os.sep), "make.globals") ensure_dirs(os.path.dirname(make_globals_path)) - os.symlink(os.path.join(PORTAGE_BASE_PATH, "cnf", "make.globals"), + os.symlink(os.path.join(cnf_path, "make.globals"), make_globals_path) #Create /usr/share/portage/config/sets/portage.conf @@ -425,8 +427,8 @@ class ResolverPlayground(object): except os.error: pass - provided_sets_portage_conf = \ - os.path.join(PORTAGE_BASE_PATH, "cnf/sets/portage.conf") + provided_sets_portage_conf = ( + os.path.join(cnf_path, "sets", "portage.conf")) os.symlink(provided_sets_portage_conf, os.path.join(default_sets_conf_dir, "portage.conf")) set_config_dir = os.path.join(user_config_dir, "sets") diff --git a/pym/portage/tests/util/test_getconfig.py b/pym/portage/tests/util/test_getconfig.py index e5fd60f..16f415c 100644 --- a/pym/portage/tests/util/test_getconfig.py +++ b/pym/portage/tests/util/test_getconfig.py @@ -3,6 +3,7 @@ import tempfile +import portage from portage import os from portage import shutil from portage import _unicode_encode @@ -26,9 +27,7 @@ class GetConfigTestCase(TestCase): } def testGetConfig(self): - - make_globals_file = os.path.join(PORTAGE_BASE_PATH, - 'cnf', 'make.globals') + make_globals_file = os.path.join(self.cnf_path, "make.globals") d = getconfig(make_globals_file) for k, v in self._cases.items(): self.assertEqual(d[k], v) -- 2.0.4 ^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [gentoo-portage-dev] [PATCH 4/6] tests: Fix running on installed copy of Portage 2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 4/6] tests: Fix running on installed copy of Portage Michał Górny @ 2014-08-22 22:19 ` Brian Dolbec 2014-08-22 22:52 ` Michał Górny 0 siblings, 1 reply; 23+ messages in thread From: Brian Dolbec @ 2014-08-22 22:19 UTC (permalink / raw To: gentoo-portage-dev On Thu, 21 Aug 2014 22:19:42 +0200 Michał Górny <mgorny@gentoo.org> wrote: > --- > pym/portage/tests/__init__.py | 20 > ++++++++++++++++++ pym/portage/tests/dbapi/test_portdb_cache.py > | 5 ++--- pym/portage/tests/emerge/test_emerge_slot_abi.py | 7 > +++---- pym/portage/tests/emerge/test_simple.py | 26 > ++++++++++++------------ > pym/portage/tests/lint/test_compile_modules.py | 10 +++++---- > pym/portage/tests/lint/test_import_modules.py | 8 ++++++-- > pym/portage/tests/repoman/test_simple.py | 8 +++++--- > pym/portage/tests/resolver/ResolverPlayground.py | 8 +++++--- > pym/portage/tests/util/test_getconfig.py | 5 ++--- 9 files > changed, 62 insertions(+), 35 deletions(-) > diff --git a/pym/portage/tests/util/test_getconfig.py > b/pym/portage/tests/util/test_getconfig.py index e5fd60f..16f415c > 100644 --- a/pym/portage/tests/util/test_getconfig.py > +++ b/pym/portage/tests/util/test_getconfig.py > @@ -3,6 +3,7 @@ > > import tempfile > > +import portage > from portage import os > from portage import shutil > from portage import _unicode_encode > @@ -26,9 +27,7 @@ class GetConfigTestCase(TestCase): > } > > def testGetConfig(self): > - > - make_globals_file = os.path.join(PORTAGE_BASE_PATH, > - 'cnf', 'make.globals') > + make_globals_file = os.path.join(self.cnf_path, > "make.globals") d = getconfig(make_globals_file) > for k, v in self._cases.items(): > self.assertEqual(d[k], v) Why was "import portage" added but not used anywhere in additional code? -- Brian Dolbec <dolsen> ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [gentoo-portage-dev] [PATCH 4/6] tests: Fix running on installed copy of Portage 2014-08-22 22:19 ` Brian Dolbec @ 2014-08-22 22:52 ` Michał Górny 0 siblings, 0 replies; 23+ messages in thread From: Michał Górny @ 2014-08-22 22:52 UTC (permalink / raw To: Brian Dolbec; +Cc: gentoo-portage-dev [-- Attachment #1: Type: text/plain, Size: 1884 bytes --] Dnia 2014-08-22, o godz. 15:19:12 Brian Dolbec <dolsen@gentoo.org> napisał(a): > On Thu, 21 Aug 2014 22:19:42 +0200 > Michał Górny <mgorny@gentoo.org> wrote: > > > --- > > pym/portage/tests/__init__.py | 20 > > ++++++++++++++++++ pym/portage/tests/dbapi/test_portdb_cache.py > > | 5 ++--- pym/portage/tests/emerge/test_emerge_slot_abi.py | 7 > > +++---- pym/portage/tests/emerge/test_simple.py | 26 > > ++++++++++++------------ > > pym/portage/tests/lint/test_compile_modules.py | 10 +++++---- > > pym/portage/tests/lint/test_import_modules.py | 8 ++++++-- > > pym/portage/tests/repoman/test_simple.py | 8 +++++--- > > pym/portage/tests/resolver/ResolverPlayground.py | 8 +++++--- > > pym/portage/tests/util/test_getconfig.py | 5 ++--- 9 files > > changed, 62 insertions(+), 35 deletions(-) > > > > diff --git a/pym/portage/tests/util/test_getconfig.py > > b/pym/portage/tests/util/test_getconfig.py index e5fd60f..16f415c > > 100644 --- a/pym/portage/tests/util/test_getconfig.py > > +++ b/pym/portage/tests/util/test_getconfig.py > > @@ -3,6 +3,7 @@ > > > > import tempfile > > > > +import portage > > from portage import os > > from portage import shutil > > from portage import _unicode_encode > > @@ -26,9 +27,7 @@ class GetConfigTestCase(TestCase): > > } > > > > def testGetConfig(self): > > - > > - make_globals_file = os.path.join(PORTAGE_BASE_PATH, > > - 'cnf', 'make.globals') > > + make_globals_file = os.path.join(self.cnf_path, > > "make.globals") d = getconfig(make_globals_file) > > for k, v in self._cases.items(): > > self.assertEqual(d[k], v) > > Why was "import portage" added but not used anywhere in additional code? Good catch. It was used in the previous version of the patch. Fixed now. -- Best regards, Michał Górny [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 949 bytes --] ^ permalink raw reply [flat|nested] 23+ messages in thread
* [gentoo-portage-dev] [PATCH 5/6] Install Python modules using setup.py 2014-08-21 20:19 [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny ` (3 preceding siblings ...) 2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 4/6] tests: Fix running on installed copy of Portage Michał Górny @ 2014-08-21 20:19 ` Michał Górny 2014-08-22 22:30 ` Brian Dolbec 2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 6/6] travis: Use setup.py for running tests Michał Górny ` (4 subsequent siblings) 9 siblings, 1 reply; 23+ messages in thread From: Michał Górny @ 2014-08-21 20:19 UTC (permalink / raw To: gentoo-portage-dev; +Cc: Michał Górny --- .gitignore | 1 + Makefile | 215 -------------------- doc/Makefile | 11 - pym/portage/const.py | 4 +- setup.py | 557 +++++++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 560 insertions(+), 228 deletions(-) delete mode 100644 Makefile delete mode 100644 doc/Makefile create mode 100755 setup.py diff --git a/.gitignore b/.gitignore index 074bb86..c2dd534 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,5 @@ *.py[co] __pycache__/ *.class +/build /tags diff --git a/Makefile b/Makefile deleted file mode 100644 index 9eb6e66..0000000 --- a/Makefile +++ /dev/null @@ -1,215 +0,0 @@ -SHELL = /bin/sh -PN ?= portage -PF ?= portage -HOMEPAGE ?= http://www.gentoo.org/proj/en/portage/index.xml -PWD ?= $(shell pwd) -S ?= $(PWD) -WORKDIR ?= $(PWD) -DESTDIR = $(PWD)/image/ -srcdir = $(S) -prefix = /usr -sysconfdir = /etc -exec_prefix = $(prefix) -bindir = $(exec_prefix)/bin -sbindir = $(exec_prefix)/sbin -libdir = $(exec_prefix)/lib -datarootdir = $(prefix)/share -datadir = $(datarootdir) -mandir = $(datarootdir)/man -docdir = $(datarootdir)/doc/$(PF) -htmldir = $(docdir)/html -portage_datadir = $(datarootdir)/$(PN) -portage_confdir = $(portage_datadir)/config -portage_setsdir = $(portage_confdir)/sets -portage_base = $(libdir)/$(PN) -EPYDOC_OPTS = -qqqqq --no-frames --show-imports -INSMODE = 0644 -EXEMODE = 0755 -DIRMODE = 0755 -SYSCONFDIR_FILES = etc-update.conf dispatch-conf.conf -PORTAGE_CONFDIR_FILES = make.conf.example make.globals repos.conf -LOGROTATE_FILES = elog-save-summary -BINDIR_FILES = ebuild egencache emerge emerge-webrsync \ - emirrordist portageq quickpkg repoman -SBINDIR_FILES = archive-conf dispatch-conf emaint \ - env-update etc-update fixpackages regenworld -DOCS = ChangeLog NEWS RELEASE-NOTES -LINGUAS ?= $(shell cd "$(srcdir)/man" && find -mindepth 1 -type d) - -ifdef PYTHONPATH - PYTHONPATH := $(srcdir)/pym:$(PYTHONPATH) -else - PYTHONPATH := $(srcdir)/pym -endif - -all: docbook epydoc - -docbook: - set -e; \ - touch "$(srcdir)/doc/fragment/date"; \ - $(MAKE) -C "$(srcdir)/doc" xhtml xhtml-nochunks - -epydoc: - set -e; \ - env PYTHONPATH="$(PYTHONPATH)" epydoc \ - -o "$(WORKDIR)/epydoc" \ - --name $(PN) \ - --url "$(HOMEPAGE)" \ - $(EPYDOC_OPTS) \ - $$(cd "$(srcdir)" && find pym -name '*.py' | sed \ - -e s:/__init__.py$$:: \ - -e s:\.py$$:: \ - -e s:^pym/:: \ - -e s:/:.:g \ - | sort); \ - rm -f "$(WORKDIR)/epydoc/api-objects.txt"; \ - -test: - set -e; \ - "$(srcdir)/pym/portage/tests/runTests.py"; \ - -install: - set -e; \ - cd "$(srcdir)/cnf"; \ - install -d -m$(DIRMODE) "$(DESTDIR)$(sysconfdir)"; \ - install -m$(INSMODE) $(SYSCONFDIR_FILES) "$(DESTDIR)$(sysconfdir)"; \ - \ - install -d -m$(DIRMODE) "$(DESTDIR)$(portage_confdir)"; \ - cd "$(srcdir)/cnf"; \ - install -m$(INSMODE) $(PORTAGE_CONFDIR_FILES) \ - "$(DESTDIR)$(portage_confdir)"; \ - install -d -m$(DIRMODE) "$(DESTDIR)$(portage_setsdir)"; \ - cd "$(S)/cnf/sets"; \ - install -m$(INSMODE) *.conf "$(DESTDIR)$(portage_setsdir)"; \ - \ - install -d -m$(DIRMODE) "$(DESTDIR)$(sysconfdir)/logrotate.d"; \ - cd "$(srcdir)/cnf/logrotate.d"; \ - install -m$(INSMODE) $(LOGROTATE_FILES) \ - "$(DESTDIR)$(sysconfdir)/logrotate.d"; \ - \ - for x in $$(cd "$(srcdir)" && find bin -type d) ; do \ - cd "$(srcdir)/$$x"; \ - install -d -m$(DIRMODE) "$(DESTDIR)$(portage_base)/$$x"; \ - files=$$(find . -mindepth 1 -maxdepth 1 -type f ! -type l); \ - if [ -n "$$files" ] ; then \ - install -m$(EXEMODE) $$files \ - "$(DESTDIR)$(portage_base)/$$x"; \ - fi; \ - symlinks=$$(find . -mindepth 1 -maxdepth 1 -type l); \ - if [ -n "$$symlinks" ] ; then \ - cp -P $$symlinks "$(DESTDIR)$(portage_base)/$$x"; \ - fi; \ - done; \ - \ - for x in $$(cd "$(srcdir)" && find pym/* -type d \ - ! -path "pym/portage/tests*") ; do \ - cd "$(srcdir)/$$x"; \ - files=$$(echo *.py); \ - if [ -z "$$files" ] || [ "$$files" = "*.py" ]; then \ - # __pycache__ directories contain no py files \ - continue; \ - fi; \ - install -d -m$(DIRMODE) "$(DESTDIR)$(portage_base)/$$x"; \ - install -m$(INSMODE) $$files "$(DESTDIR)$(portage_base)/$$x"; \ - done; \ - \ - install -d -m$(DIRMODE) "$(DESTDIR)$(bindir)"; \ - relative_path=".."; \ - x=$(bindir) ; \ - y="$(portage_base)"; \ - if [ "$${x#$(prefix)}" != "$$x" ] && \ - [ "$${y#$(prefix)}" != "$$y" ]; then \ - x=$${x#$(prefix)}; \ - y=$${y#$(prefix)}; \ - fi; \ - x=$${x%/*}; \ - while [ -n "$$x" ] ; do \ - relative_path=$${relative_path}/..; \ - x=$${x%/*}; \ - done; \ - relative_path=$$relative_path$$y; \ - for x in $(BINDIR_FILES) ; do \ - ln -sf "$$relative_path/bin/$$x" \ - "$(DESTDIR)$(bindir)/$$x"; \ - done; \ - \ - install -d -m$(DIRMODE) "$(DESTDIR)$(sbindir)"; \ - relative_path=".."; \ - x=$(sbindir) ; \ - y="$(portage_base)"; \ - if [ "$${x#$(prefix)}" != "$$x" ] && \ - [ "$${y#$(prefix)}" != "$$y" ]; then \ - x=$${x#$(prefix)}; \ - y=$${y#$(prefix)}; \ - fi; \ - x=$${x%/*}; \ - while [ -n "$$x" ] ; do \ - relative_path=$${relative_path}/..; \ - x=$${x%/*}; \ - done; \ - relative_path=$$relative_path$$y; \ - for x in $(SBINDIR_FILES) ; do \ - ln -sf "$$relative_path/bin/$$x" \ - "$(DESTDIR)$(sbindir)/$$x"; \ - done; \ - \ - ln -sf "$$relative_path/bin/env-update" \ - "$(DESTDIR)$(sbindir)/update-env"; \ - ln -sf "$$relative_path/bin/etc-update" \ - "$(DESTDIR)$(sbindir)/update-etc"; \ - \ - # We install some minimal tests for use as a preinst sanity check. \ - # These tests must be able to run without a full source tree and \ - # without relying on a previous portage instance being installed. \ - install -d -m$(DIRMODE) \ - "$(DESTDIR)$(portage_base)/pym/portage/tests"; \ - install -m$(EXEMODE) "$(srcdir)/pym/portage/tests/runTests" \ - "$(DESTDIR)$(portage_base)/pym/portage/tests"; \ - cd "$(srcdir)/pym/portage/tests"; \ - install -m$(INSMODE) *.py \ - "$(DESTDIR)$(portage_base)/pym/portage/tests"; \ - install -d -m$(DIRMODE) \ - "$(DESTDIR)$(portage_base)/pym/portage/tests/lint"; \ - cd "$(srcdir)/pym/portage/tests/lint"; \ - install -m$(INSMODE) *.py __test__ \ - "$(DESTDIR)$(portage_base)/pym/portage/tests/lint"; \ - \ - install -d -m$(DIRMODE) "$(DESTDIR)$(docdir)"; \ - cd "$(srcdir)"; \ - install -m $(INSMODE) $(DOCS) "$(DESTDIR)$(docdir)"; \ - \ - for x in "" $(LINGUAS); do \ - for y in 1 5 ; do \ - if [ -d "$(srcdir)/man/$$x" ]; then \ - cd "$(srcdir)/man/$$x"; \ - files=$$(echo *.$$y); \ - if [ -z "$$files" ] || [ "$$files" = "*.$$y" ]; then \ - continue; \ - fi; \ - install -d -m$(DIRMODE) "$(DESTDIR)$(mandir)/$$x/man$$y"; \ - install -m$(INSMODE) *.$$y "$(DESTDIR)$(mandir)/$$x/man$$y"; \ - fi; \ - done; \ - done; \ - \ - if [ -f "$(srcdir)/doc/portage.html" ] ; then \ - install -d -m$(DIRMODE) "$(DESTDIR)$(htmldir)"; \ - cd "$(srcdir)/doc"; \ - install -m$(INSMODE) *.html "$(DESTDIR)$(htmldir)"; \ - fi; \ - \ - if [ -d "$(WORKDIR)/epydoc" ] ; then \ - install -d -m$(DIRMODE) "$(DESTDIR)$(htmldir)"; \ - cp -pPR "$(WORKDIR)/epydoc" \ - "$(DESTDIR)$(htmldir)/api"; \ - cd "$(DESTDIR)$(htmldir)/api"; \ - find . -type d | xargs chmod $(DIRMODE); \ - find . -type f | xargs chmod $(INSMODE); \ - fi; \ - -clean: - set -e; \ - $(MAKE) -C "$(srcdir)/doc" clean; \ - rm -rf "$(WORKDIR)/epydoc"; \ - -.PHONY: all clean docbook epydoc install test diff --git a/doc/Makefile b/doc/Makefile deleted file mode 100644 index 261a0b4..0000000 --- a/doc/Makefile +++ /dev/null @@ -1,11 +0,0 @@ -all: xhtml xhtml-nochunks - -XMLTO_FLAGS = -m custom.xsl -man pdf txt xhtml xhtml-nochunks: - xmlto $@ $(XMLTO_FLAGS) portage.docbook - -clean distclean: - rm -f *.1 *.html portage.txt - -.PHONY: all clean distclean \ - man pdf txt xhtml xhtml-nochunks diff --git a/pym/portage/const.py b/pym/portage/const.py index f518b47..acb90f9 100644 --- a/pym/portage/const.py +++ b/pym/portage/const.py @@ -60,8 +60,8 @@ GLOBAL_CONFIG_PATH = "/usr/share/portage/config" # these variables are not used with target_root or config_root # NOTE: Use realpath(__file__) so that python module symlinks in site-packages # are followed back to the real location of the whole portage installation. -PORTAGE_BASE_PATH = os.path.join(os.sep, os.sep.join(os.path.realpath( - __file__.rstrip("co")).split(os.sep)[:-3])) +# NOTE: Please keep PORTAGE_BASE_PATH in one line to help substitutions. +PORTAGE_BASE_PATH = os.path.join(os.sep, os.sep.join(os.path.realpath(__file__.rstrip("co")).split(os.sep)[:-3])) PORTAGE_BIN_PATH = PORTAGE_BASE_PATH + "/bin" PORTAGE_PYM_PATH = os.path.realpath(os.path.join(__file__, '../..')) LOCALE_DATA_PATH = PORTAGE_BASE_PATH + "/locale" # FIXME: not used diff --git a/setup.py b/setup.py new file mode 100755 index 0000000..bd6e506 --- /dev/null +++ b/setup.py @@ -0,0 +1,557 @@ +#!/usr/bin/env python +# vim:fileencoding=utf-8 +# (c) 2010 Michał Górny <mgorny@gentoo.org> +# Released under the terms of the 2-clause BSD license. + +from distutils.core import setup, Command +from distutils.command.build_scripts import build_scripts +from distutils.command.clean import clean +from distutils.command.install import install +from distutils.command.install_data import install_data +from distutils.command.install_lib import install_lib +from distutils.command.install_scripts import install_scripts +from distutils.dir_util import remove_tree +from distutils.util import change_root, subst_vars + +import codecs, collections, glob, os, os.path, re, subprocess, sys + +# TODO: +# - smarter rebuilds of docs w/ 'install_docbook' and 'install_epydoc'. + +package_name = 'portage' +package_version = '2.2.12' +package_homepage = 'https://wiki.gentoo.org/wiki/Project:Portage' + +x_scripts = { + 'bin': [ + 'bin/ebuild', 'bin/egencache', 'bin/emerge', 'bin/emerge-webrsync', + 'bin/emirrordist', 'bin/portageq', 'bin/quickpkg', 'bin/repoman' + ], + 'sbin': [ + 'bin/archive-conf', 'bin/dispatch-conf', 'bin/emaint', 'bin/env-update', + 'bin/etc-update', 'bin/fixpackages', 'bin/regenworld' + ], +} + + +class docbook(Command): + """ Build docs using docbook. """ + + user_options = [ + ('doc-formats=', None, 'Documentation formats to build (all xmlto formats for docbook are allowed, comma-separated'), + ] + + def initialize_options(self): + self.doc_formats = 'xhtml,xhtml-nochunks' + + def finalize_options(self): + self.doc_formats = self.doc_formats.replace(',', ' ').split() + + def run(self): + with open('doc/fragment/date', 'w'): + pass + + for f in self.doc_formats: + print('Building docs in %s format...' % f) + subprocess.check_call(['xmlto', '-o', 'doc', + '-m', 'doc/custom.xsl', f, 'doc/portage.docbook']) + + +class epydoc(Command): + """ Build API docs using epydoc. """ + + user_options = [ + ] + + def initialize_options(self): + self.build_lib = None + + def finalize_options(self): + self.set_undefined_options('build_py', ('build_lib', 'build_lib')) + + def run(self): + self.run_command('build_py') + + print('Building API documentation...') + + process_env = os.environ.copy() + pythonpath = self.build_lib + try: + pythonpath += ':' + process_env['PYTHONPATH'] + except KeyError: + pass + process_env['PYTHONPATH'] = pythonpath + + subprocess.check_call(['epydoc', '-o', 'epydoc', + '--name', package_name, + '--url', package_homepage, + '-qq', '--no-frames', '--show-imports', + '--exclude', 'portage.tests', + '_emerge', 'portage', 'repoman'], + env = process_env) + os.remove('epydoc/api-objects.txt') + + +class install_docbook(install_data): + """ install_data for docbook docs """ + + user_options = install_data.user_options + + def initialize_options(self): + install_data.initialize_options(self) + self.htmldir = None + + def finalize_options(self): + self.set_undefined_options('install', ('htmldir', 'htmldir')) + install_data.finalize_options(self) + + def run(self): + if not os.path.exists('doc/portage.html'): + self.run_command('docbook') + self.data_files = [ + (self.htmldir, glob.glob('doc/*.html')), + ] + install_data.run(self) + + +class install_epydoc(install_data): + """ install_data for epydoc docs """ + + user_options = install_data.user_options + + def initialize_options(self): + install_data.initialize_options(self) + self.htmldir = None + + def finalize_options(self): + self.set_undefined_options('install', ('htmldir', 'htmldir')) + install_data.finalize_options(self) + + def run(self): + if not os.path.exists('epydoc/index.html'): + self.run_command('epydoc') + self.data_files = [ + (os.path.join(self.htmldir, 'api'), glob.glob('epydoc/*')), + ] + install_data.run(self) + + +class x_build_scripts_custom(build_scripts): + def finalize_options(self): + build_scripts.finalize_options(self) + if 'dir_name' in dir(self): + self.build_dir = os.path.join(self.build_dir, self.dir_name) + if self.dir_name in x_scripts: + self.scripts = x_scripts[self.dir_name] + else: + self.scripts = set(self.scripts) + for other_files in x_scripts.values(): + self.scripts.difference_update(other_files) + + def run(self): + # group scripts by subdirectory + split_scripts = collections.defaultdict(list) + for f in self.scripts: + dir_name = os.path.dirname(f[len('bin/'):]) + split_scripts[dir_name].append(f) + + base_dir = self.build_dir + base_scripts = self.scripts + for d, files in split_scripts.items(): + self.build_dir = os.path.join(base_dir, d) + self.scripts = files + self.copy_scripts() + + # restore previous values + self.build_dir = base_dir + self.scripts = base_scripts + + +class x_build_scripts_bin(x_build_scripts_custom): + dir_name = 'bin' + + +class x_build_scripts_sbin(x_build_scripts_custom): + dir_name = 'sbin' + + +class x_build_scripts_portagebin(x_build_scripts_custom): + dir_name = 'portage' + + +class x_build_scripts(build_scripts): + def initialize_option(self): + pass + + def finalize_options(self): + pass + + def run(self): + self.run_command('build_scripts_bin') + self.run_command('build_scripts_portagebin') + self.run_command('build_scripts_sbin') + + +class x_clean(clean): + """ clean extended for doc & post-test cleaning """ + + def clean_docs(self): + def get_doc_outfiles(): + for dirpath, dirnames, filenames in os.walk('doc'): + for f in filenames: + if f.endswith('.docbook') or f == 'custom.xsl': + pass + else: + yield os.path.join(dirpath, f) + + # do not recurse + break + + + for f in get_doc_outfiles(): + print('removing %s' % repr(f)) + os.remove(f) + + if os.path.isdir('epydoc'): + remove_tree('epydoc') + + def clean_tests(self): + # do not remove incorrect dirs accidentally + top_dir = os.path.normpath(os.path.join(self.build_lib, '..')) + cprefix = os.path.commonprefix((self.build_base, top_dir)) + if cprefix != self.build_base: + return + + bin_dir = os.path.join(top_dir, 'bin') + if os.path.exists(bin_dir): + remove_tree(bin_dir) + + conf_dir = os.path.join(top_dir, 'cnf') + if os.path.islink(conf_dir): + print('removing %s symlink' % repr(conf_dir)) + os.unlink(conf_dir) + + pni_file = os.path.join(top_dir, '.portage_not_installed') + if os.path.exists(pni_file): + print('removing %s' % repr(pni_file)) + os.unlink(pni_file) + + def run(self): + if self.all: + self.clean_tests() + self.clean_docs() + + clean.run(self) + + +class x_install(install): + """ install command with extra Portage paths """ + + user_options = install.user_options + [ + # note: $prefix and $exec_prefix are reserved for Python install + ('system-prefix=', None, "Prefix for architecture-independent data"), + ('system-exec-prefix=', None, "Prefix for architecture-specific data"), + + ('bindir=', None, "Install directory for main executables"), + ('datarootdir=', None, "Data install root directory"), + ('docdir=', None, "Documentation install directory"), + ('htmldir=', None, "HTML documentation install directory"), + ('mandir=', None, "Manpage root install directory"), + ('portage-base=', 'b', "Portage install base"), + ('portage-bindir=', None, "Install directory for Portage internal-use executables"), + ('portage-datadir=', None, 'Install directory for data files'), + ('sbindir=', None, "Install directory for superuser-intended executables"), + ('sysconfdir=', None, 'System configuration path'), + ] + + # note: the order is important for proper substitution + paths = [ + ('system_prefix', '/usr'), + ('system_exec_prefix', '$system_prefix'), + + ('bindir', '$system_exec_prefix/bin'), + ('sbindir', '$system_exec_prefix/sbin'), + ('sysconfdir', '/etc'), + + ('datarootdir', '$system_prefix/share'), + ('docdir', '$datarootdir/doc/%s-%s' % (package_name, package_version)), + ('htmldir', '$docdir/html'), + ('mandir', '$datarootdir/man'), + + ('portage_base', '$system_exec_prefix/lib/portage'), + ('portage_bindir', '$portage_base/bin'), + ('portage_datadir', '$datarootdir/portage'), + + # not customized at the moment + ('logrotatedir', '$sysconfdir/logrotate'), + ('portage_confdir', '$portage_datadir/config'), + ('portage_setsdir', '$portage_confdir/sets'), + ] + + def initialize_options(self): + install.initialize_options(self) + + for key, default in self.paths: + setattr(self, key, default) + self.subst_paths = {} + + def finalize_options(self): + install.finalize_options(self) + + # substitute variables + new_paths = {} + for key, default in self.paths: + new_paths[key] = subst_vars(getattr(self, key), new_paths) + setattr(self, key, new_paths[key]) + self.subst_paths = new_paths + + +class x_install_data(install_data): + """ install_data with customized path support """ + + user_options = install_data.user_options + + def initialize_options(self): + install_data.initialize_options(self) + self.paths = None + + def finalize_options(self): + install_data.finalize_options(self) + self.set_undefined_options('install', + ('subst_paths', 'paths')) + + # substitute variables in data_files + for f in self.data_files: + f[0] = subst_vars(f[0], self.paths) + + +class x_install_lib(install_lib): + """ install_lib command with Portage path substitution """ + + user_options = install_lib.user_options + + def initialize_options(self): + install_lib.initialize_options(self) + self.portage_base = None + self.portage_bindir = None + self.portage_confdir = None + + def finalize_options(self): + install_lib.finalize_options(self) + self.set_undefined_options('install', + ('portage_base', 'portage_base'), + ('portage_bindir', 'portage_bindir'), + ('portage_confdir', 'portage_confdir')) + + def install(self): + ret = install_lib.install(self) + + base_re = re.compile(r'(^PORTAGE_BASE_PATH.*=) .*$', re.MULTILINE) + portage_bin_re = re.compile(r'(^PORTAGE_BIN_PATH.*=) .*$', re.MULTILINE) + global_config_re = re.compile(r'(^GLOBAL_CONFIG_PATH.*=) .*$', re.MULTILINE) + + constfile = os.path.join(self.install_dir, 'portage', 'const.py') + print('Rewriting %s' % constfile) + with codecs.open(constfile, 'r', 'utf-8') as f: + data = f.read() + data = base_re.sub('\\1 %s' % repr(self.portage_base), data) + data = portage_bin_re.sub('\\1 %s' % repr(self.portage_bindir), data) + data = global_config_re.sub('\\1 %s' % repr(self.portage_confdir), data) + with codecs.open(constfile, 'w', 'utf-8') as f: + f.write(data) + + return ret + + +class x_install_scripts_custom(install_scripts): + def initialize_options(self): + install_scripts.initialize_options(self) + self.root = None + + def finalize_options(self): + self.set_undefined_options('install', + ('root', 'root'), + (self.var_name, 'install_dir')) + install_scripts.finalize_options(self) + self.build_dir = os.path.join(self.build_dir, self.dir_name) + + # prepend root + if self.root is not None: + self.install_dir = change_root(self.root, self.install_dir) + + +class x_install_scripts_bin(x_install_scripts_custom): + dir_name = 'bin' + var_name = 'bindir' + + +class x_install_scripts_sbin(x_install_scripts_custom): + dir_name = 'sbin' + var_name = 'sbindir' + + +class x_install_scripts_portagebin(x_install_scripts_custom): + dir_name = 'portage' + var_name = 'portage_bindir' + + +class x_install_scripts(install_scripts): + def initialize_option(self): + pass + + def finalize_options(self): + pass + + def run(self): + self.run_command('install_scripts_bin') + self.run_command('install_scripts_portagebin') + self.run_command('install_scripts_sbin') + + +class build_tests(x_build_scripts_custom): + """ Prepare build dir for running tests. """ + + def initialize_options(self): + x_build_scripts_custom.initialize_options(self) + self.build_base = None + self.build_lib = None + + def finalize_options(self): + x_build_scripts_custom.finalize_options(self) + self.set_undefined_options('build', + ('build_base', 'build_base'), + ('build_lib', 'build_lib')) + + # since we will be writing to $build_lib/.., it is important + # that we do not leave $build_base + self.top_dir = os.path.normpath(os.path.join(self.build_lib, '..')) + cprefix = os.path.commonprefix((self.build_base, self.top_dir)) + if cprefix != self.build_base: + raise SystemError('build_lib must be a subdirectory of build_base') + + self.build_dir = os.path.join(self.top_dir, 'bin') + + def run(self): + self.run_command('build_py') + + # install all scripts $build_lib/../bin + # (we can't do a symlink since we want shebangs corrected) + x_build_scripts_custom.run(self) + + # symlink 'cnf' directory + conf_dir = os.path.join(self.top_dir, 'cnf') + if os.path.exists(conf_dir): + if not os.path.islink(conf_dir): + raise SystemError('%s exists and is not a symlink (collision)' + % repr(conf_dir)) + os.unlink(conf_dir) + conf_src = os.path.relpath('cnf', self.top_dir) + print('Symlinking %s -> %s' % (conf_dir, conf_src)) + os.symlink(conf_src, conf_dir) + + # create $build_lib/../.portage_not_installed + # to enable proper paths in tests + with open(os.path.join(self.top_dir, '.portage_not_installed'), 'w') as f: + pass + + +class test(Command): + """ run tests """ + + user_options = [] + + def initialize_options(self): + self.build_lib = None + + def finalize_options(self): + self.set_undefined_options('build', + ('build_lib', 'build_lib')) + + def run(self): + self.run_command('build_tests') + subprocess.check_call([ + sys.executable, '-bWd', + os.path.join(self.build_lib, 'portage/tests/runTests.py') + ]) + + +def find_packages(): + for dirpath, dirnames, filenames in os.walk('pym'): + if '__init__.py' in filenames: + yield os.path.relpath(dirpath, 'pym') + + +def find_scripts(): + for dirpath, dirnames, filenames in os.walk('bin'): + for f in filenames: + yield os.path.join(dirpath, f) + + +def get_manpages(): + linguas = os.environ.get('LINGUAS') + if linguas is not None: + linguas = linguas.split() + + for dirpath, dirnames, filenames in os.walk('man'): + groups = collections.defaultdict(list) + for f in filenames: + fn, suffix = f.rsplit('.', 1) + groups[suffix].append(os.path.join(dirpath, f)) + + topdir = dirpath[len('man/'):] + if not topdir or linguas is None or topdir in linguas: + for g, mans in groups.items(): + yield [os.path.join('$mandir', topdir, 'man%s' % g), mans] + +setup( + name = package_name, + version = package_version, + author = 'Gentoo Portage Development Team', + author_email = 'dev-portage@gentoo.org', + url = package_homepage, + + package_dir = {'': 'pym'}, + packages = list(find_packages()), + # something to cheat build & install commands + scripts = list(find_scripts()), + + data_files = list(get_manpages()) + [ + ['$sysconfdir', ['cnf/etc-update.conf', 'cnf/dispatch-conf.conf']], + ['$logrotatedir', ['cnf/logrotate.d/elog-save-summary']], + ['$portage_confdir', [ + 'cnf/make.conf.example', 'cnf/make.globals', 'cnf/repos.conf']], + ['$portage_setsdir', ['cnf/sets/portage.conf']], + ['$docdir', ['ChangeLog', 'NEWS', 'RELEASE-NOTES']], + ], + + cmdclass = { + 'build_scripts': x_build_scripts, + 'build_scripts_bin': x_build_scripts_bin, + 'build_scripts_portagebin': x_build_scripts_portagebin, + 'build_scripts_sbin': x_build_scripts_sbin, + 'build_tests': build_tests, + 'clean': x_clean, + 'docbook': docbook, + 'epydoc': epydoc, + 'install': x_install, + 'install_data': x_install_data, + 'install_docbook': install_docbook, + 'install_epydoc': install_epydoc, + 'install_lib': x_install_lib, + 'install_scripts': x_install_scripts, + 'install_scripts_bin': x_install_scripts_bin, + 'install_scripts_portagebin': x_install_scripts_portagebin, + 'install_scripts_sbin': x_install_scripts_sbin, + 'test': test, + }, + + classifiers = [ + 'Development Status :: 5 - Production/Stable', + 'Environment :: Console', + 'Intended Audience :: System Administrators', + 'License :: OSI Approved :: GNU General Public License v2 (GPLv2)', + 'Operating System :: POSIX', + 'Programming Language :: Python', + 'Topic :: System :: Installation/Setup' + ] +) -- 2.0.4 ^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [gentoo-portage-dev] [PATCH 5/6] Install Python modules using setup.py 2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 5/6] Install Python modules using setup.py Michał Górny @ 2014-08-22 22:30 ` Brian Dolbec 2014-08-22 22:55 ` Michał Górny 2014-08-23 20:30 ` [gentoo-portage-dev] [PATCH v2] Install Portage " Michał Górny 0 siblings, 2 replies; 23+ messages in thread From: Brian Dolbec @ 2014-08-22 22:30 UTC (permalink / raw To: gentoo-portage-dev On Thu, 21 Aug 2014 22:19:43 +0200 Michał Górny <mgorny@gentoo.org> wrote: > --- > .gitignore | 1 + > Makefile | 215 -------------------- > doc/Makefile | 11 - > pym/portage/const.py | 4 +- > setup.py | 557 > +++++++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, > 560 insertions(+), 228 deletions(-) delete mode 100644 Makefile > delete mode 100644 doc/Makefile > create mode 100755 setup.py > diff --git a/setup.py b/setup.py new file mode 100755 > index 0000000..bd6e506 --- /dev/null > +++ b/setup.py > @@ -0,0 +1,557 @@ > +#!/usr/bin/env python > +# vim:fileencoding=utf-8 > +# (c) 2010 Michał Górny <mgorny@gentoo.org> > +# Released under the terms of the 2-clause BSD license. > + > +from distutils.core import setup, Command > +from distutils.command.build_scripts import build_scripts > +from distutils.command.clean import clean > +from distutils.command.install import install > +from distutils.command.install_data import install_data > +from distutils.command.install_lib import install_lib > +from distutils.command.install_scripts import install_scripts > +from distutils.dir_util import remove_tree > +from distutils.util import change_root, subst_vars > + > +import codecs, collections, glob, os, os.path, re, subprocess, sys > + > +# TODO: > +# - smarter rebuilds of docs w/ 'install_docbook' and > 'install_epydoc'. + > +package_name = 'portage' > +package_version = '2.2.12' Why hard code the version? can you not import it? or did you add this to the sed list in mkrelease.sh? -- Brian Dolbec <dolsen> ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [gentoo-portage-dev] [PATCH 5/6] Install Python modules using setup.py 2014-08-22 22:30 ` Brian Dolbec @ 2014-08-22 22:55 ` Michał Górny 2014-08-23 20:30 ` [gentoo-portage-dev] [PATCH v2] Install Portage " Michał Górny 1 sibling, 0 replies; 23+ messages in thread From: Michał Górny @ 2014-08-22 22:55 UTC (permalink / raw To: Brian Dolbec; +Cc: gentoo-portage-dev [-- Attachment #1: Type: text/plain, Size: 2022 bytes --] Dnia 2014-08-22, o godz. 15:30:15 Brian Dolbec <dolsen@gentoo.org> napisał(a): > On Thu, 21 Aug 2014 22:19:43 +0200 > Michał Górny <mgorny@gentoo.org> wrote: > > > --- > > .gitignore | 1 + > > Makefile | 215 -------------------- > > doc/Makefile | 11 - > > pym/portage/const.py | 4 +- > > setup.py | 557 > > +++++++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, > > 560 insertions(+), 228 deletions(-) delete mode 100644 Makefile > > delete mode 100644 doc/Makefile > > create mode 100755 setup.py > > > > > > diff --git a/setup.py b/setup.py new file mode 100755 > > index 0000000..bd6e506 --- /dev/null > > +++ b/setup.py > > @@ -0,0 +1,557 @@ > > +#!/usr/bin/env python > > +# vim:fileencoding=utf-8 > > +# (c) 2010 Michał Górny <mgorny@gentoo.org> > > +# Released under the terms of the 2-clause BSD license. > > + > > +from distutils.core import setup, Command > > +from distutils.command.build_scripts import build_scripts > > +from distutils.command.clean import clean > > +from distutils.command.install import install > > +from distutils.command.install_data import install_data > > +from distutils.command.install_lib import install_lib > > +from distutils.command.install_scripts import install_scripts > > +from distutils.dir_util import remove_tree > > +from distutils.util import change_root, subst_vars > > + > > +import codecs, collections, glob, os, os.path, re, subprocess, sys > > + > > +# TODO: > > +# - smarter rebuilds of docs w/ 'install_docbook' and > > 'install_epydoc'. + > > +package_name = 'portage' > > +package_version = '2.2.12' > > Why hard code the version? can you not import it? or did you add this > to the sed list in mkrelease.sh? Left it in my todo and forgot about it. Which also makes me think that we may want to try to get 'setup.py sdist' working as well, and rethink how much we need mkrelease.sh afterwards. -- Best regards, Michał Górny [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 949 bytes --] ^ permalink raw reply [flat|nested] 23+ messages in thread
* [gentoo-portage-dev] [PATCH v2] Install Portage using setup.py 2014-08-22 22:30 ` Brian Dolbec 2014-08-22 22:55 ` Michał Górny @ 2014-08-23 20:30 ` Michał Górny 2014-08-23 21:56 ` Brian Dolbec ` (2 more replies) 1 sibling, 3 replies; 23+ messages in thread From: Michał Górny @ 2014-08-23 20:30 UTC (permalink / raw To: gentoo-portage-dev; +Cc: Michał Górny --- .gitignore | 1 + MANIFEST.in | 19 ++ Makefile | 215 ------------------- doc/Makefile | 11 - mkrelease.sh | 141 ------------- pym/portage/const.py | 4 +- setup.py | 578 +++++++++++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 600 insertions(+), 369 deletions(-) create mode 100644 MANIFEST.in delete mode 100644 Makefile delete mode 100644 doc/Makefile delete mode 100755 mkrelease.sh create mode 100755 setup.py diff --git a/.gitignore b/.gitignore index 074bb86..c2dd534 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,5 @@ *.py[co] __pycache__/ *.class +/build /tags diff --git a/MANIFEST.in b/MANIFEST.in new file mode 100644 index 0000000..a005ab9 --- /dev/null +++ b/MANIFEST.in @@ -0,0 +1,19 @@ +# docs +include DEVELOPING +include LICENSE +include TEST-NOTES + +# docbook sources +include doc/custom.xsl +include doc/fragment/* +recursive-include doc *.docbook + +# extra conf files used in ebuild +include cnf/make.conf.example.* + +# extra files for tests +include .portage_not_installed +include cnf/metadata.dtd + +# extra scripts +include misc/* diff --git a/Makefile b/Makefile deleted file mode 100644 index 9eb6e66..0000000 --- a/Makefile +++ /dev/null @@ -1,215 +0,0 @@ -SHELL = /bin/sh -PN ?= portage -PF ?= portage -HOMEPAGE ?= http://www.gentoo.org/proj/en/portage/index.xml -PWD ?= $(shell pwd) -S ?= $(PWD) -WORKDIR ?= $(PWD) -DESTDIR = $(PWD)/image/ -srcdir = $(S) -prefix = /usr -sysconfdir = /etc -exec_prefix = $(prefix) -bindir = $(exec_prefix)/bin -sbindir = $(exec_prefix)/sbin -libdir = $(exec_prefix)/lib -datarootdir = $(prefix)/share -datadir = $(datarootdir) -mandir = $(datarootdir)/man -docdir = $(datarootdir)/doc/$(PF) -htmldir = $(docdir)/html -portage_datadir = $(datarootdir)/$(PN) -portage_confdir = $(portage_datadir)/config -portage_setsdir = $(portage_confdir)/sets -portage_base = $(libdir)/$(PN) -EPYDOC_OPTS = -qqqqq --no-frames --show-imports -INSMODE = 0644 -EXEMODE = 0755 -DIRMODE = 0755 -SYSCONFDIR_FILES = etc-update.conf dispatch-conf.conf -PORTAGE_CONFDIR_FILES = make.conf.example make.globals repos.conf -LOGROTATE_FILES = elog-save-summary -BINDIR_FILES = ebuild egencache emerge emerge-webrsync \ - emirrordist portageq quickpkg repoman -SBINDIR_FILES = archive-conf dispatch-conf emaint \ - env-update etc-update fixpackages regenworld -DOCS = ChangeLog NEWS RELEASE-NOTES -LINGUAS ?= $(shell cd "$(srcdir)/man" && find -mindepth 1 -type d) - -ifdef PYTHONPATH - PYTHONPATH := $(srcdir)/pym:$(PYTHONPATH) -else - PYTHONPATH := $(srcdir)/pym -endif - -all: docbook epydoc - -docbook: - set -e; \ - touch "$(srcdir)/doc/fragment/date"; \ - $(MAKE) -C "$(srcdir)/doc" xhtml xhtml-nochunks - -epydoc: - set -e; \ - env PYTHONPATH="$(PYTHONPATH)" epydoc \ - -o "$(WORKDIR)/epydoc" \ - --name $(PN) \ - --url "$(HOMEPAGE)" \ - $(EPYDOC_OPTS) \ - $$(cd "$(srcdir)" && find pym -name '*.py' | sed \ - -e s:/__init__.py$$:: \ - -e s:\.py$$:: \ - -e s:^pym/:: \ - -e s:/:.:g \ - | sort); \ - rm -f "$(WORKDIR)/epydoc/api-objects.txt"; \ - -test: - set -e; \ - "$(srcdir)/pym/portage/tests/runTests.py"; \ - -install: - set -e; \ - cd "$(srcdir)/cnf"; \ - install -d -m$(DIRMODE) "$(DESTDIR)$(sysconfdir)"; \ - install -m$(INSMODE) $(SYSCONFDIR_FILES) "$(DESTDIR)$(sysconfdir)"; \ - \ - install -d -m$(DIRMODE) "$(DESTDIR)$(portage_confdir)"; \ - cd "$(srcdir)/cnf"; \ - install -m$(INSMODE) $(PORTAGE_CONFDIR_FILES) \ - "$(DESTDIR)$(portage_confdir)"; \ - install -d -m$(DIRMODE) "$(DESTDIR)$(portage_setsdir)"; \ - cd "$(S)/cnf/sets"; \ - install -m$(INSMODE) *.conf "$(DESTDIR)$(portage_setsdir)"; \ - \ - install -d -m$(DIRMODE) "$(DESTDIR)$(sysconfdir)/logrotate.d"; \ - cd "$(srcdir)/cnf/logrotate.d"; \ - install -m$(INSMODE) $(LOGROTATE_FILES) \ - "$(DESTDIR)$(sysconfdir)/logrotate.d"; \ - \ - for x in $$(cd "$(srcdir)" && find bin -type d) ; do \ - cd "$(srcdir)/$$x"; \ - install -d -m$(DIRMODE) "$(DESTDIR)$(portage_base)/$$x"; \ - files=$$(find . -mindepth 1 -maxdepth 1 -type f ! -type l); \ - if [ -n "$$files" ] ; then \ - install -m$(EXEMODE) $$files \ - "$(DESTDIR)$(portage_base)/$$x"; \ - fi; \ - symlinks=$$(find . -mindepth 1 -maxdepth 1 -type l); \ - if [ -n "$$symlinks" ] ; then \ - cp -P $$symlinks "$(DESTDIR)$(portage_base)/$$x"; \ - fi; \ - done; \ - \ - for x in $$(cd "$(srcdir)" && find pym/* -type d \ - ! -path "pym/portage/tests*") ; do \ - cd "$(srcdir)/$$x"; \ - files=$$(echo *.py); \ - if [ -z "$$files" ] || [ "$$files" = "*.py" ]; then \ - # __pycache__ directories contain no py files \ - continue; \ - fi; \ - install -d -m$(DIRMODE) "$(DESTDIR)$(portage_base)/$$x"; \ - install -m$(INSMODE) $$files "$(DESTDIR)$(portage_base)/$$x"; \ - done; \ - \ - install -d -m$(DIRMODE) "$(DESTDIR)$(bindir)"; \ - relative_path=".."; \ - x=$(bindir) ; \ - y="$(portage_base)"; \ - if [ "$${x#$(prefix)}" != "$$x" ] && \ - [ "$${y#$(prefix)}" != "$$y" ]; then \ - x=$${x#$(prefix)}; \ - y=$${y#$(prefix)}; \ - fi; \ - x=$${x%/*}; \ - while [ -n "$$x" ] ; do \ - relative_path=$${relative_path}/..; \ - x=$${x%/*}; \ - done; \ - relative_path=$$relative_path$$y; \ - for x in $(BINDIR_FILES) ; do \ - ln -sf "$$relative_path/bin/$$x" \ - "$(DESTDIR)$(bindir)/$$x"; \ - done; \ - \ - install -d -m$(DIRMODE) "$(DESTDIR)$(sbindir)"; \ - relative_path=".."; \ - x=$(sbindir) ; \ - y="$(portage_base)"; \ - if [ "$${x#$(prefix)}" != "$$x" ] && \ - [ "$${y#$(prefix)}" != "$$y" ]; then \ - x=$${x#$(prefix)}; \ - y=$${y#$(prefix)}; \ - fi; \ - x=$${x%/*}; \ - while [ -n "$$x" ] ; do \ - relative_path=$${relative_path}/..; \ - x=$${x%/*}; \ - done; \ - relative_path=$$relative_path$$y; \ - for x in $(SBINDIR_FILES) ; do \ - ln -sf "$$relative_path/bin/$$x" \ - "$(DESTDIR)$(sbindir)/$$x"; \ - done; \ - \ - ln -sf "$$relative_path/bin/env-update" \ - "$(DESTDIR)$(sbindir)/update-env"; \ - ln -sf "$$relative_path/bin/etc-update" \ - "$(DESTDIR)$(sbindir)/update-etc"; \ - \ - # We install some minimal tests for use as a preinst sanity check. \ - # These tests must be able to run without a full source tree and \ - # without relying on a previous portage instance being installed. \ - install -d -m$(DIRMODE) \ - "$(DESTDIR)$(portage_base)/pym/portage/tests"; \ - install -m$(EXEMODE) "$(srcdir)/pym/portage/tests/runTests" \ - "$(DESTDIR)$(portage_base)/pym/portage/tests"; \ - cd "$(srcdir)/pym/portage/tests"; \ - install -m$(INSMODE) *.py \ - "$(DESTDIR)$(portage_base)/pym/portage/tests"; \ - install -d -m$(DIRMODE) \ - "$(DESTDIR)$(portage_base)/pym/portage/tests/lint"; \ - cd "$(srcdir)/pym/portage/tests/lint"; \ - install -m$(INSMODE) *.py __test__ \ - "$(DESTDIR)$(portage_base)/pym/portage/tests/lint"; \ - \ - install -d -m$(DIRMODE) "$(DESTDIR)$(docdir)"; \ - cd "$(srcdir)"; \ - install -m $(INSMODE) $(DOCS) "$(DESTDIR)$(docdir)"; \ - \ - for x in "" $(LINGUAS); do \ - for y in 1 5 ; do \ - if [ -d "$(srcdir)/man/$$x" ]; then \ - cd "$(srcdir)/man/$$x"; \ - files=$$(echo *.$$y); \ - if [ -z "$$files" ] || [ "$$files" = "*.$$y" ]; then \ - continue; \ - fi; \ - install -d -m$(DIRMODE) "$(DESTDIR)$(mandir)/$$x/man$$y"; \ - install -m$(INSMODE) *.$$y "$(DESTDIR)$(mandir)/$$x/man$$y"; \ - fi; \ - done; \ - done; \ - \ - if [ -f "$(srcdir)/doc/portage.html" ] ; then \ - install -d -m$(DIRMODE) "$(DESTDIR)$(htmldir)"; \ - cd "$(srcdir)/doc"; \ - install -m$(INSMODE) *.html "$(DESTDIR)$(htmldir)"; \ - fi; \ - \ - if [ -d "$(WORKDIR)/epydoc" ] ; then \ - install -d -m$(DIRMODE) "$(DESTDIR)$(htmldir)"; \ - cp -pPR "$(WORKDIR)/epydoc" \ - "$(DESTDIR)$(htmldir)/api"; \ - cd "$(DESTDIR)$(htmldir)/api"; \ - find . -type d | xargs chmod $(DIRMODE); \ - find . -type f | xargs chmod $(INSMODE); \ - fi; \ - -clean: - set -e; \ - $(MAKE) -C "$(srcdir)/doc" clean; \ - rm -rf "$(WORKDIR)/epydoc"; \ - -.PHONY: all clean docbook epydoc install test diff --git a/doc/Makefile b/doc/Makefile deleted file mode 100644 index 261a0b4..0000000 --- a/doc/Makefile +++ /dev/null @@ -1,11 +0,0 @@ -all: xhtml xhtml-nochunks - -XMLTO_FLAGS = -m custom.xsl -man pdf txt xhtml xhtml-nochunks: - xmlto $@ $(XMLTO_FLAGS) portage.docbook - -clean distclean: - rm -f *.1 *.html portage.txt - -.PHONY: all clean distclean \ - man pdf txt xhtml xhtml-nochunks diff --git a/mkrelease.sh b/mkrelease.sh deleted file mode 100755 index f9f7564..0000000 --- a/mkrelease.sh +++ /dev/null @@ -1,141 +0,0 @@ -#!/bin/bash -# Copyright 2008-2014 Gentoo Foundation -# Distributed under the terms of the GNU General Public License v2 - -RELEASE_BUILDDIR=${RELEASE_BUILDDIR:-/var/tmp/portage-release} -SOURCE_DIR=${RELEASE_BUILDDIR}/checkout -BRANCH=${BRANCH:-master} -USE_TAG=false -CHANGELOG_REVISION= -UPLOAD_LOCATION= -RUNTESTS=false -USER= - -usage() { - echo "Usage: ${0##*/} [--changelog-rev <tree-ish>] [-t|--tag] [-u|--upload <location>] [--user <username>] [--runtests] <version>" - exit ${1:-0} -} - -die() { - printf 'error: %s\n' "$*" - usage 1 -} - -ARGS=$(getopt -o htu: --long help,changelog-rev:,runtests,tag,upload:,user: \ - -n "${0##*/}" -- "$@") -[ $? != 0 ] && die "initialization error" - -eval set -- "${ARGS}" - -while true; do - case $1 in - --changelog-rev) - CHANGELOG_REVISION=$2 - shift 2 - ;; - -t|--tag) - USE_TAG=true - shift - ;; - -u|--upload) - UPLOAD_LOCATION=$2 - shift 2 - ;; - --user) - USER=$2"@" - shift 2 - ;; - -h|--help) - usage - ;; - --runtests) - RUNTESTS=true - shift - ;; - --) - shift - break - ;; - *) - die "unknown option: $1" - ;; - esac -done - -[ $# != 1 ] && die "Need version argument" -[[ -n ${1/[0-9]*} ]] && die "Invalid version argument" - -VERSION=$1 -RELEASE=portage-${VERSION} -RELEASE_DIR=${RELEASE_BUILDDIR}/${RELEASE} -RELEASE_TARBALL="${RELEASE_BUILDDIR}/${RELEASE}.tar.bz2" -TREE_ISH=${BRANCH} -if [[ ${USE_TAG} == "true" ]] ; then - TREE_ISH="v${VERSION}" -fi - -echo ">>> Cleaning working directories ${RELEASE_DIR} ${SOURCE_DIR}" -rm -rf "${RELEASE_DIR}" "${SOURCE_DIR}" || die "directory cleanup failed" -mkdir -p "${RELEASE_DIR}" || die "directory creation failed" -mkdir -p "${SOURCE_DIR}" || die "mkdir failed" - -echo ">>> Starting GIT archive" -git archive --format=tar ${TREE_ISH} | \ - tar -xf - -C "${SOURCE_DIR}" || die "git archive failed" - -echo ">>> Building release tree" -cp -a "${SOURCE_DIR}/"{bin,cnf,doc,man,misc,pym} "${RELEASE_DIR}/" || die "directory copy failed" -cp "${SOURCE_DIR}/"{.portage_not_installed,DEVELOPING,LICENSE,Makefile,NEWS,README,RELEASE-NOTES,TEST-NOTES} \ - "${RELEASE_DIR}/" || die "file copy failed" - -if [[ ${RUNTESTS} == "true" ]] ; then - pushd "${SOURCE_DIR}" >/dev/null - ./runtests.sh --python-versions=supported || die "tests failed" - popd >/dev/null -fi - -rm -rf "${SOURCE_DIR}" || die "directory cleanup failed" - -echo ">>> Setting portage.VERSION" -sed -e "s/^VERSION = .*/VERSION = \"${VERSION}\"/" \ - -i "${RELEASE_DIR}/pym/portage/__init__.py" || \ - die "Failed to patch portage.VERSION" - -echo ">>> Creating Changelog" -git_log_opts="" -if [[ -n ${CHANGELOG_REVISION} ]] ; then - git_log_opts+=" ${CHANGELOG_REVISION}^..${TREE_ISH}" -else - git_log_opts+=" ${TREE_ISH}" -fi -skip_next=false -git log ${git_log_opts} | fmt -w 80 -p " " | while read -r ; do - if [[ ${skip_next} == "true" ]] ; then - skip_next=false - elif [[ ${REPLY} == " svn path="* ]] ; then - skip_next=true - else - echo "${REPLY}" - fi -done > "${RELEASE_DIR}/ChangeLog" || die "ChangeLog creation failed" - -cd "${RELEASE_BUILDDIR}" - -echo ">>> Creating release tarball ${RELEASE_TARBALL}" -tar --owner portage --group portage -cjf "${RELEASE_TARBALL}" "${RELEASE}" || \ - die "tarball creation failed" - -DISTDIR=$(portageq distdir) -if [[ -n ${DISTDIR} && -d ${DISTDIR} && -w ${DISTDIR} ]] ; then - echo ">>> Copying release tarball into ${DISTDIR}" - cp "${RELEASE_TARBALL}" "${DISTDIR}"/ || echo "!!! tarball copy failed" -fi - -if [[ -n ${UPLOAD_LOCATION} ]] ; then - echo ">>> Uploading ${RELEASE_TARBALL} to ${USER}dev.gentoo.org:${UPLOAD_LOCATION}" - scp "${RELEASE_TARBALL}" "${USER}dev.gentoo.org:${UPLOAD_LOCATION}" || die "upload failed" -else - du -h "${RELEASE_TARBALL}" -fi - -exit 0 diff --git a/pym/portage/const.py b/pym/portage/const.py index f518b47..acb90f9 100644 --- a/pym/portage/const.py +++ b/pym/portage/const.py @@ -60,8 +60,8 @@ GLOBAL_CONFIG_PATH = "/usr/share/portage/config" # these variables are not used with target_root or config_root # NOTE: Use realpath(__file__) so that python module symlinks in site-packages # are followed back to the real location of the whole portage installation. -PORTAGE_BASE_PATH = os.path.join(os.sep, os.sep.join(os.path.realpath( - __file__.rstrip("co")).split(os.sep)[:-3])) +# NOTE: Please keep PORTAGE_BASE_PATH in one line to help substitutions. +PORTAGE_BASE_PATH = os.path.join(os.sep, os.sep.join(os.path.realpath(__file__.rstrip("co")).split(os.sep)[:-3])) PORTAGE_BIN_PATH = PORTAGE_BASE_PATH + "/bin" PORTAGE_PYM_PATH = os.path.realpath(os.path.join(__file__, '../..')) LOCALE_DATA_PATH = PORTAGE_BASE_PATH + "/locale" # FIXME: not used diff --git a/setup.py b/setup.py new file mode 100755 index 0000000..7ffe7f0 --- /dev/null +++ b/setup.py @@ -0,0 +1,578 @@ +#!/usr/bin/env python +# vim:fileencoding=utf-8 +# (c) 2010 Michał Górny <mgorny@gentoo.org> +# Released under the terms of the 2-clause BSD license. + +from distutils.core import setup, Command +from distutils.command.build_scripts import build_scripts +from distutils.command.clean import clean +from distutils.command.install import install +from distutils.command.install_data import install_data +from distutils.command.install_lib import install_lib +from distutils.command.install_scripts import install_scripts +from distutils.command.sdist import sdist +from distutils.dir_util import remove_tree +from distutils.util import change_root, subst_vars + +import codecs, collections, glob, os, os.path, re, subprocess, sys + +# TODO: +# - smarter rebuilds of docs w/ 'install_docbook' and 'install_epydoc'. + +x_scripts = { + 'bin': [ + 'bin/ebuild', 'bin/egencache', 'bin/emerge', 'bin/emerge-webrsync', + 'bin/emirrordist', 'bin/portageq', 'bin/quickpkg', 'bin/repoman' + ], + 'sbin': [ + 'bin/archive-conf', 'bin/dispatch-conf', 'bin/emaint', 'bin/env-update', + 'bin/etc-update', 'bin/fixpackages', 'bin/regenworld' + ], +} + + +class docbook(Command): + """ Build docs using docbook. """ + + user_options = [ + ('doc-formats=', None, 'Documentation formats to build (all xmlto formats for docbook are allowed, comma-separated'), + ] + + def initialize_options(self): + self.doc_formats = 'xhtml,xhtml-nochunks' + + def finalize_options(self): + self.doc_formats = self.doc_formats.replace(',', ' ').split() + + def run(self): + with open('doc/fragment/date', 'w'): + pass + + for f in self.doc_formats: + print('Building docs in %s format...' % f) + subprocess.check_call(['xmlto', '-o', 'doc', + '-m', 'doc/custom.xsl', f, 'doc/portage.docbook']) + + +class epydoc(Command): + """ Build API docs using epydoc. """ + + user_options = [ + ] + + def initialize_options(self): + self.build_lib = None + + def finalize_options(self): + self.set_undefined_options('build_py', ('build_lib', 'build_lib')) + + def run(self): + self.run_command('build_py') + + print('Building API documentation...') + + process_env = os.environ.copy() + pythonpath = self.build_lib + try: + pythonpath += ':' + process_env['PYTHONPATH'] + except KeyError: + pass + process_env['PYTHONPATH'] = pythonpath + + subprocess.check_call(['epydoc', '-o', 'epydoc', + '--name', self.distribution.get_name(), + '--url', self.distribution.get_url(), + '-qq', '--no-frames', '--show-imports', + '--exclude', 'portage.tests', + '_emerge', 'portage', 'repoman'], + env = process_env) + os.remove('epydoc/api-objects.txt') + + +class install_docbook(install_data): + """ install_data for docbook docs """ + + user_options = install_data.user_options + + def initialize_options(self): + install_data.initialize_options(self) + self.htmldir = None + + def finalize_options(self): + self.set_undefined_options('install', ('htmldir', 'htmldir')) + install_data.finalize_options(self) + + def run(self): + if not os.path.exists('doc/portage.html'): + self.run_command('docbook') + self.data_files = [ + (self.htmldir, glob.glob('doc/*.html')), + ] + install_data.run(self) + + +class install_epydoc(install_data): + """ install_data for epydoc docs """ + + user_options = install_data.user_options + + def initialize_options(self): + install_data.initialize_options(self) + self.htmldir = None + + def finalize_options(self): + self.set_undefined_options('install', ('htmldir', 'htmldir')) + install_data.finalize_options(self) + + def run(self): + if not os.path.exists('epydoc/index.html'): + self.run_command('epydoc') + self.data_files = [ + (os.path.join(self.htmldir, 'api'), glob.glob('epydoc/*')), + ] + install_data.run(self) + + +class x_build_scripts_custom(build_scripts): + def finalize_options(self): + build_scripts.finalize_options(self) + if 'dir_name' in dir(self): + self.build_dir = os.path.join(self.build_dir, self.dir_name) + if self.dir_name in x_scripts: + self.scripts = x_scripts[self.dir_name] + else: + self.scripts = set(self.scripts) + for other_files in x_scripts.values(): + self.scripts.difference_update(other_files) + + def run(self): + # group scripts by subdirectory + split_scripts = collections.defaultdict(list) + for f in self.scripts: + dir_name = os.path.dirname(f[len('bin/'):]) + split_scripts[dir_name].append(f) + + base_dir = self.build_dir + base_scripts = self.scripts + for d, files in split_scripts.items(): + self.build_dir = os.path.join(base_dir, d) + self.scripts = files + self.copy_scripts() + + # restore previous values + self.build_dir = base_dir + self.scripts = base_scripts + + +class x_build_scripts_bin(x_build_scripts_custom): + dir_name = 'bin' + + +class x_build_scripts_sbin(x_build_scripts_custom): + dir_name = 'sbin' + + +class x_build_scripts_portagebin(x_build_scripts_custom): + dir_name = 'portage' + + +class x_build_scripts(build_scripts): + def initialize_option(self): + build_scripts.initialize_options(self) + + def finalize_options(self): + build_scripts.finalize_options(self) + + def run(self): + self.run_command('build_scripts_bin') + self.run_command('build_scripts_portagebin') + self.run_command('build_scripts_sbin') + + +class x_clean(clean): + """ clean extended for doc & post-test cleaning """ + + def clean_docs(self): + def get_doc_outfiles(): + for dirpath, dirnames, filenames in os.walk('doc'): + for f in filenames: + if f.endswith('.docbook') or f == 'custom.xsl': + pass + else: + yield os.path.join(dirpath, f) + + # do not recurse + break + + + for f in get_doc_outfiles(): + print('removing %s' % repr(f)) + os.remove(f) + + if os.path.isdir('epydoc'): + remove_tree('epydoc') + + def clean_tests(self): + # do not remove incorrect dirs accidentally + top_dir = os.path.normpath(os.path.join(self.build_lib, '..')) + cprefix = os.path.commonprefix((self.build_base, top_dir)) + if cprefix != self.build_base: + return + + bin_dir = os.path.join(top_dir, 'bin') + if os.path.exists(bin_dir): + remove_tree(bin_dir) + + conf_dir = os.path.join(top_dir, 'cnf') + if os.path.islink(conf_dir): + print('removing %s symlink' % repr(conf_dir)) + os.unlink(conf_dir) + + pni_file = os.path.join(top_dir, '.portage_not_installed') + if os.path.exists(pni_file): + print('removing %s' % repr(pni_file)) + os.unlink(pni_file) + + def run(self): + if self.all: + self.clean_tests() + self.clean_docs() + + clean.run(self) + + +class x_install(install): + """ install command with extra Portage paths """ + + user_options = install.user_options + [ + # note: $prefix and $exec_prefix are reserved for Python install + ('system-prefix=', None, "Prefix for architecture-independent data"), + ('system-exec-prefix=', None, "Prefix for architecture-specific data"), + + ('bindir=', None, "Install directory for main executables"), + ('datarootdir=', None, "Data install root directory"), + ('docdir=', None, "Documentation install directory"), + ('htmldir=', None, "HTML documentation install directory"), + ('mandir=', None, "Manpage root install directory"), + ('portage-base=', 'b', "Portage install base"), + ('portage-bindir=', None, "Install directory for Portage internal-use executables"), + ('portage-datadir=', None, 'Install directory for data files'), + ('sbindir=', None, "Install directory for superuser-intended executables"), + ('sysconfdir=', None, 'System configuration path'), + ] + + # note: the order is important for proper substitution + paths = [ + ('system_prefix', '/usr'), + ('system_exec_prefix', '$system_prefix'), + + ('bindir', '$system_exec_prefix/bin'), + ('sbindir', '$system_exec_prefix/sbin'), + ('sysconfdir', '/etc'), + + ('datarootdir', '$system_prefix/share'), + ('docdir', '$datarootdir/doc/$package-$version'), + ('htmldir', '$docdir/html'), + ('mandir', '$datarootdir/man'), + + ('portage_base', '$system_exec_prefix/lib/portage'), + ('portage_bindir', '$portage_base/bin'), + ('portage_datadir', '$datarootdir/portage'), + + # not customized at the moment + ('logrotatedir', '$sysconfdir/logrotate'), + ('portage_confdir', '$portage_datadir/config'), + ('portage_setsdir', '$portage_confdir/sets'), + ] + + def initialize_options(self): + install.initialize_options(self) + + for key, default in self.paths: + setattr(self, key, default) + self.subst_paths = {} + + def finalize_options(self): + install.finalize_options(self) + + # substitute variables + new_paths = { + 'package': self.distribution.get_name(), + 'version': self.distribution.get_version(), + } + for key, default in self.paths: + new_paths[key] = subst_vars(getattr(self, key), new_paths) + setattr(self, key, new_paths[key]) + self.subst_paths = new_paths + + +class x_install_data(install_data): + """ install_data with customized path support """ + + user_options = install_data.user_options + + def initialize_options(self): + install_data.initialize_options(self) + self.paths = None + + def finalize_options(self): + install_data.finalize_options(self) + self.set_undefined_options('install', + ('subst_paths', 'paths')) + + # substitute variables in data_files + for f in self.data_files: + f[0] = subst_vars(f[0], self.paths) + + +class x_install_lib(install_lib): + """ install_lib command with Portage path substitution """ + + user_options = install_lib.user_options + + def initialize_options(self): + install_lib.initialize_options(self) + self.portage_base = None + self.portage_bindir = None + self.portage_confdir = None + + def finalize_options(self): + install_lib.finalize_options(self) + self.set_undefined_options('install', + ('portage_base', 'portage_base'), + ('portage_bindir', 'portage_bindir'), + ('portage_confdir', 'portage_confdir')) + + def install(self): + ret = install_lib.install(self) + + def rewrite_file(path, val_dict): + path = os.path.join(self.install_dir, path) + print('Rewriting %s' % path) + with codecs.open(path, 'r', 'utf-8') as f: + data = f.read() + + for varname, val in val_dict.items(): + regexp = r'^(%s\s*=).*$' % varname + repl = r'\1 %s' % repr(val) + + data = re.sub(regexp, repl, data, 0, re.MULTILINE) + + with codecs.open(path, 'w', 'utf-8') as f: + f.write(data) + + rewrite_file('portage/__init__.py', { + 'VERSION': self.distribution.get_version(), + }) + rewrite_file('portage/const.py', { + 'PORTAGE_BASE_PATH': self.portage_base, + 'PORTAGE_BIN_PATH': self.portage_bindir, + 'PORTAGE_CONFIG_PATH': self.portage_confdir, + }) + + return ret + + +class x_install_scripts_custom(install_scripts): + def initialize_options(self): + install_scripts.initialize_options(self) + self.root = None + + def finalize_options(self): + self.set_undefined_options('install', + ('root', 'root'), + (self.var_name, 'install_dir')) + install_scripts.finalize_options(self) + self.build_dir = os.path.join(self.build_dir, self.dir_name) + + # prepend root + if self.root is not None: + self.install_dir = change_root(self.root, self.install_dir) + + +class x_install_scripts_bin(x_install_scripts_custom): + dir_name = 'bin' + var_name = 'bindir' + + +class x_install_scripts_sbin(x_install_scripts_custom): + dir_name = 'sbin' + var_name = 'sbindir' + + +class x_install_scripts_portagebin(x_install_scripts_custom): + dir_name = 'portage' + var_name = 'portage_bindir' + + +class x_install_scripts(install_scripts): + def initialize_option(self): + pass + + def finalize_options(self): + pass + + def run(self): + self.run_command('install_scripts_bin') + self.run_command('install_scripts_portagebin') + self.run_command('install_scripts_sbin') + + +class x_sdist(sdist): + """ sdist defaulting to .tar.bz2 format """ + + def finalize_options(self): + if self.formats is None: + self.formats = ['bztar'] + + sdist.finalize_options(self) + + +class build_tests(x_build_scripts_custom): + """ Prepare build dir for running tests. """ + + def initialize_options(self): + x_build_scripts_custom.initialize_options(self) + self.build_base = None + self.build_lib = None + + def finalize_options(self): + x_build_scripts_custom.finalize_options(self) + self.set_undefined_options('build', + ('build_base', 'build_base'), + ('build_lib', 'build_lib')) + + # since we will be writing to $build_lib/.., it is important + # that we do not leave $build_base + self.top_dir = os.path.normpath(os.path.join(self.build_lib, '..')) + cprefix = os.path.commonprefix((self.build_base, self.top_dir)) + if cprefix != self.build_base: + raise SystemError('build_lib must be a subdirectory of build_base') + + self.build_dir = os.path.join(self.top_dir, 'bin') + + def run(self): + self.run_command('build_py') + + # install all scripts $build_lib/../bin + # (we can't do a symlink since we want shebangs corrected) + x_build_scripts_custom.run(self) + + # symlink 'cnf' directory + conf_dir = os.path.join(self.top_dir, 'cnf') + if os.path.exists(conf_dir): + if not os.path.islink(conf_dir): + raise SystemError('%s exists and is not a symlink (collision)' + % repr(conf_dir)) + os.unlink(conf_dir) + conf_src = os.path.relpath('cnf', self.top_dir) + print('Symlinking %s -> %s' % (conf_dir, conf_src)) + os.symlink(conf_src, conf_dir) + + # create $build_lib/../.portage_not_installed + # to enable proper paths in tests + with open(os.path.join(self.top_dir, '.portage_not_installed'), 'w') as f: + pass + + +class test(Command): + """ run tests """ + + user_options = [] + + def initialize_options(self): + self.build_lib = None + + def finalize_options(self): + self.set_undefined_options('build', + ('build_lib', 'build_lib')) + + def run(self): + self.run_command('build_tests') + subprocess.check_call([ + sys.executable, '-bWd', + os.path.join(self.build_lib, 'portage/tests/runTests.py') + ]) + + +def find_packages(): + for dirpath, dirnames, filenames in os.walk('pym'): + if '__init__.py' in filenames: + yield os.path.relpath(dirpath, 'pym') + + +def find_scripts(): + for dirpath, dirnames, filenames in os.walk('bin'): + for f in filenames: + yield os.path.join(dirpath, f) + + +def get_manpages(): + linguas = os.environ.get('LINGUAS') + if linguas is not None: + linguas = linguas.split() + + for dirpath, dirnames, filenames in os.walk('man'): + groups = collections.defaultdict(list) + for f in filenames: + fn, suffix = f.rsplit('.', 1) + groups[suffix].append(os.path.join(dirpath, f)) + + topdir = dirpath[len('man/'):] + if not topdir or linguas is None or topdir in linguas: + for g, mans in groups.items(): + yield [os.path.join('$mandir', topdir, 'man%s' % g), mans] + +setup( + name = 'portage', + version = '2.2.12', + url = 'https://wiki.gentoo.org/wiki/Project:Portage', + author = 'Gentoo Portage Development Team', + author_email = 'dev-portage@gentoo.org', + + package_dir = {'': 'pym'}, + packages = list(find_packages()), + # something to cheat build & install commands + scripts = list(find_scripts()), + + data_files = list(get_manpages()) + [ + ['$sysconfdir', ['cnf/etc-update.conf', 'cnf/dispatch-conf.conf']], + ['$logrotatedir', ['cnf/logrotate.d/elog-save-summary']], + ['$portage_confdir', [ + 'cnf/make.conf.example', 'cnf/make.globals', 'cnf/repos.conf']], + ['$portage_setsdir', ['cnf/sets/portage.conf']], + ['$docdir', ['NEWS', 'RELEASE-NOTES']], + ], + + cmdclass = { + 'build_scripts': x_build_scripts, + 'build_scripts_bin': x_build_scripts_bin, + 'build_scripts_portagebin': x_build_scripts_portagebin, + 'build_scripts_sbin': x_build_scripts_sbin, + 'build_tests': build_tests, + 'clean': x_clean, + 'docbook': docbook, + 'epydoc': epydoc, + 'install': x_install, + 'install_data': x_install_data, + 'install_docbook': install_docbook, + 'install_epydoc': install_epydoc, + 'install_lib': x_install_lib, + 'install_scripts': x_install_scripts, + 'install_scripts_bin': x_install_scripts_bin, + 'install_scripts_portagebin': x_install_scripts_portagebin, + 'install_scripts_sbin': x_install_scripts_sbin, + 'sdist': x_sdist, + 'test': test, + }, + + classifiers = [ + 'Development Status :: 5 - Production/Stable', + 'Environment :: Console', + 'Intended Audience :: System Administrators', + 'License :: OSI Approved :: GNU General Public License v2 (GPLv2)', + 'Operating System :: POSIX', + 'Programming Language :: Python', + 'Topic :: System :: Installation/Setup' + ] +) -- 2.0.4 ^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [gentoo-portage-dev] [PATCH v2] Install Portage using setup.py 2014-08-23 20:30 ` [gentoo-portage-dev] [PATCH v2] Install Portage " Michał Górny @ 2014-08-23 21:56 ` Brian Dolbec 2014-08-29 17:08 ` Brian Dolbec 2014-09-05 6:58 ` [gentoo-portage-dev] [PATCH v3] " Michał Górny 2 siblings, 0 replies; 23+ messages in thread From: Brian Dolbec @ 2014-08-23 21:56 UTC (permalink / raw To: gentoo-portage-dev On Sat, 23 Aug 2014 22:30:56 +0200 Michał Górny <mgorny@gentoo.org> wrote: What about adding to the commit message that v2 also includes "setup.py sdist" capability? > diff --git a/mkrelease.sh b/mkrelease.sh > deleted file mode 100755 > index f9f7564..0000000 > --- a/mkrelease.sh > +++ /dev/null > @@ -1,141 +0,0 @@ > -#!/bin/bash > -# Copyright 2008-2014 Gentoo Foundation > -# Distributed under the terms of the GNU General Public License v2 > - > -RELEASE_BUILDDIR=${RELEASE_BUILDDIR:-/var/tmp/portage-release} > -SOURCE_DIR=${RELEASE_BUILDDIR}/checkout > -BRANCH=${BRANCH:-master} > -USE_TAG=false > -CHANGELOG_REVISION= > -UPLOAD_LOCATION= > -RUNTESTS=false > -USER= > - > -usage() { > - echo "Usage: ${0##*/} [--changelog-rev <tree-ish>] > [-t|--tag] [-u|--upload <location>] [--user <username>] [--runtests] > <version>" > - exit ${1:-0} > -} > - > -die() { > - printf 'error: %s\n' "$*" > - usage 1 > -} > - > -ARGS=$(getopt -o htu: --long > help,changelog-rev:,runtests,tag,upload:,user: \ > - -n "${0##*/}" -- "$@") > -[ $? != 0 ] && die "initialization error" > - > -eval set -- "${ARGS}" > - > -while true; do > - case $1 in > - --changelog-rev) > - CHANGELOG_REVISION=$2 > - shift 2 > - ;; > - -t|--tag) > - USE_TAG=true > - shift > - ;; > - -u|--upload) > - UPLOAD_LOCATION=$2 > - shift 2 > - ;; > - --user) > - USER=$2"@" > - shift 2 > - ;; > - -h|--help) > - usage > - ;; > - --runtests) > - RUNTESTS=true > - shift > - ;; > - --) > - shift > - break > - ;; > - *) > - die "unknown option: $1" > - ;; > - esac > -done > - > -[ $# != 1 ] && die "Need version argument" > -[[ -n ${1/[0-9]*} ]] && die "Invalid version argument" > - > -VERSION=$1 > -RELEASE=portage-${VERSION} > -RELEASE_DIR=${RELEASE_BUILDDIR}/${RELEASE} > -RELEASE_TARBALL="${RELEASE_BUILDDIR}/${RELEASE}.tar.bz2" > -TREE_ISH=${BRANCH} > -if [[ ${USE_TAG} == "true" ]] ; then > - TREE_ISH="v${VERSION}" > -fi > - > -echo ">>> Cleaning working directories ${RELEASE_DIR} ${SOURCE_DIR}" > -rm -rf "${RELEASE_DIR}" "${SOURCE_DIR}" || die "directory cleanup > failed" -mkdir -p "${RELEASE_DIR}" || die "directory creation failed" > -mkdir -p "${SOURCE_DIR}" || die "mkdir failed" > - > -echo ">>> Starting GIT archive" > -git archive --format=tar ${TREE_ISH} | \ > - tar -xf - -C "${SOURCE_DIR}" || die "git archive failed" > - > -echo ">>> Building release tree" > -cp -a "${SOURCE_DIR}/"{bin,cnf,doc,man,misc,pym} "${RELEASE_DIR}/" > || die "directory copy failed" -cp > "${SOURCE_DIR}/"{.portage_not_installed,DEVELOPING,LICENSE,Makefile,NEWS,README,RELEASE-NOTES,TEST-NOTES} > \ > - "${RELEASE_DIR}/" || die "file copy failed" > - > -if [[ ${RUNTESTS} == "true" ]] ; then > - pushd "${SOURCE_DIR}" >/dev/null > - ./runtests.sh --python-versions=supported || die "tests > failed" > - popd >/dev/null > -fi > - > -rm -rf "${SOURCE_DIR}" || die "directory cleanup failed" > - > -echo ">>> Setting portage.VERSION" > -sed -e "s/^VERSION = .*/VERSION = \"${VERSION}\"/" \ > - -i "${RELEASE_DIR}/pym/portage/__init__.py" || \ > - die "Failed to patch portage.VERSION" > - > -echo ">>> Creating Changelog" > -git_log_opts="" > -if [[ -n ${CHANGELOG_REVISION} ]] ; then > - git_log_opts+=" ${CHANGELOG_REVISION}^..${TREE_ISH}" > -else > - git_log_opts+=" ${TREE_ISH}" > -fi > -skip_next=false > -git log ${git_log_opts} | fmt -w 80 -p " " | while read -r ; do > - if [[ ${skip_next} == "true" ]] ; then > - skip_next=false > - elif [[ ${REPLY} == " svn path="* ]] ; then > - skip_next=true > - else > - echo "${REPLY}" > - fi > -done > "${RELEASE_DIR}/ChangeLog" || die "ChangeLog creation failed" > - > -cd "${RELEASE_BUILDDIR}" > - > -echo ">>> Creating release tarball ${RELEASE_TARBALL}" > -tar --owner portage --group portage -cjf "${RELEASE_TARBALL}" > "${RELEASE}" || \ > - die "tarball creation failed" > - > -DISTDIR=$(portageq distdir) > -if [[ -n ${DISTDIR} && -d ${DISTDIR} && -w ${DISTDIR} ]] ; then > - echo ">>> Copying release tarball into ${DISTDIR}" > - cp "${RELEASE_TARBALL}" "${DISTDIR}"/ || echo "!!! tarball > copy failed" -fi > - > -if [[ -n ${UPLOAD_LOCATION} ]] ; then > - echo ">>> Uploading ${RELEASE_TARBALL} to > ${USER}dev.gentoo.org:${UPLOAD_LOCATION}" > - scp "${RELEASE_TARBALL}" > "${USER}dev.gentoo.org:${UPLOAD_LOCATION}" || die "upload failed" > -else > - du -h "${RELEASE_TARBALL}" > -fi > - > -exit 0 > Why are you removing everything from mkrelease.sh? It does do some other useful things other than just make the release tarball. We have not decided to drop the Changelog as a team yet, only a few words on irc. Instead mkrelease could be modified to work with setup.py for the tarball creation, but it still creates the Changelog, copies the tarball to distfiles, uploads the file... I did not see any of that functionality added to setup.py below. > FIXME: not used diff --git a/setup.py b/setup.py new file mode 100755 > index 0000000..7ffe7f0 --- /dev/null > +++ b/setup.py > @@ -0,0 +1,578 @@ > +#!/usr/bin/env python > +# vim:fileencoding=utf-8 > +# (c) 2010 Michał Górny <mgorny@gentoo.org> > +# Released under the terms of the 2-clause BSD license. > + > +from distutils.core import setup, Command > +from distutils.command.build_scripts import build_scripts > +from distutils.command.clean import clean > +from distutils.command.install import install > +from distutils.command.install_data import install_data > +from distutils.command.install_lib import install_lib > +from distutils.command.install_scripts import install_scripts > +from distutils.command.sdist import sdist > +from distutils.dir_util import remove_tree > +from distutils.util import change_root, subst_vars > + > +import codecs, collections, glob, os, os.path, re, subprocess, sys > + > +# TODO: > +# - smarter rebuilds of docs w/ 'install_docbook' and > 'install_epydoc'. + > +x_scripts = { > + 'bin': [ > + 'bin/ebuild', 'bin/egencache', 'bin/emerge', > 'bin/emerge-webrsync', > + 'bin/emirrordist', 'bin/portageq', 'bin/quickpkg', > 'bin/repoman' > + ], > + 'sbin': [ > + 'bin/archive-conf', 'bin/dispatch-conf', > 'bin/emaint', 'bin/env-update', > + 'bin/etc-update', 'bin/fixpackages', 'bin/regenworld' > + ], > +} > + > + > +class docbook(Command): > + """ Build docs using docbook. """ > + > + user_options = [ > + ('doc-formats=', None, 'Documentation formats to > build (all xmlto formats for docbook are allowed, comma-separated'), > + ] > + > + def initialize_options(self): > + self.doc_formats = 'xhtml,xhtml-nochunks' > + > + def finalize_options(self): > + self.doc_formats = self.doc_formats.replace(',', ' > ').split() + > + def run(self): > + with open('doc/fragment/date', 'w'): > + pass > + > + for f in self.doc_formats: > + print('Building docs in %s format...' % f) > + subprocess.check_call(['xmlto', '-o', 'doc', > + '-m', 'doc/custom.xsl', f, > 'doc/portage.docbook']) + > + > +class epydoc(Command): > + """ Build API docs using epydoc. """ > + > + user_options = [ > + ] > + > + def initialize_options(self): > + self.build_lib = None > + > + def finalize_options(self): > + self.set_undefined_options('build_py', ('build_lib', > 'build_lib')) + > + def run(self): > + self.run_command('build_py') > + > + print('Building API documentation...') > + > + process_env = os.environ.copy() > + pythonpath = self.build_lib > + try: > + pythonpath += ':' + process_env['PYTHONPATH'] > + except KeyError: > + pass > + process_env['PYTHONPATH'] = pythonpath > + > + subprocess.check_call(['epydoc', '-o', 'epydoc', > + '--name', self.distribution.get_name(), > + '--url', self.distribution.get_url(), > + '-qq', '--no-frames', '--show-imports', > + '--exclude', 'portage.tests', > + '_emerge', 'portage', 'repoman'], > + env = process_env) > + os.remove('epydoc/api-objects.txt') > + > + > +class install_docbook(install_data): > + """ install_data for docbook docs """ > + > + user_options = install_data.user_options > + > + def initialize_options(self): > + install_data.initialize_options(self) > + self.htmldir = None > + > + def finalize_options(self): > + self.set_undefined_options('install', ('htmldir', > 'htmldir')) > + install_data.finalize_options(self) > + > + def run(self): > + if not os.path.exists('doc/portage.html'): > + self.run_command('docbook') > + self.data_files = [ > + (self.htmldir, glob.glob('doc/*.html')), > + ] > + install_data.run(self) > + > + > +class install_epydoc(install_data): > + """ install_data for epydoc docs """ > + > + user_options = install_data.user_options > + > + def initialize_options(self): > + install_data.initialize_options(self) > + self.htmldir = None > + > + def finalize_options(self): > + self.set_undefined_options('install', ('htmldir', > 'htmldir')) > + install_data.finalize_options(self) > + > + def run(self): > + if not os.path.exists('epydoc/index.html'): > + self.run_command('epydoc') > + self.data_files = [ > + (os.path.join(self.htmldir, 'api'), > glob.glob('epydoc/*')), > + ] > + install_data.run(self) > + > + > +class x_build_scripts_custom(build_scripts): > + def finalize_options(self): > + build_scripts.finalize_options(self) > + if 'dir_name' in dir(self): > + self.build_dir = > os.path.join(self.build_dir, self.dir_name) > + if self.dir_name in x_scripts: > + self.scripts = > x_scripts[self.dir_name] > + else: > + self.scripts = set(self.scripts) > + for other_files in > x_scripts.values(): > + > self.scripts.difference_update(other_files) + > + def run(self): > + # group scripts by subdirectory > + split_scripts = collections.defaultdict(list) > + for f in self.scripts: > + dir_name = os.path.dirname(f[len('bin/'):]) > + split_scripts[dir_name].append(f) > + > + base_dir = self.build_dir > + base_scripts = self.scripts > + for d, files in split_scripts.items(): > + self.build_dir = os.path.join(base_dir, d) > + self.scripts = files > + self.copy_scripts() > + > + # restore previous values > + self.build_dir = base_dir > + self.scripts = base_scripts > + > + > +class x_build_scripts_bin(x_build_scripts_custom): > + dir_name = 'bin' > + > + > +class x_build_scripts_sbin(x_build_scripts_custom): > + dir_name = 'sbin' > + > + > +class x_build_scripts_portagebin(x_build_scripts_custom): > + dir_name = 'portage' > + > + > +class x_build_scripts(build_scripts): > + def initialize_option(self): > + build_scripts.initialize_options(self) > + > + def finalize_options(self): > + build_scripts.finalize_options(self) > + > + def run(self): > + self.run_command('build_scripts_bin') > + self.run_command('build_scripts_portagebin') > + self.run_command('build_scripts_sbin') > + > + > +class x_clean(clean): > + """ clean extended for doc & post-test cleaning """ > + > + def clean_docs(self): > + def get_doc_outfiles(): > + for dirpath, dirnames, filenames in > os.walk('doc'): > + for f in filenames: > + if f.endswith('.docbook') or > f == 'custom.xsl': > + pass > + else: > + yield > os.path.join(dirpath, f) + > + # do not recurse > + break > + > + > + for f in get_doc_outfiles(): > + print('removing %s' % repr(f)) > + os.remove(f) > + > + if os.path.isdir('epydoc'): > + remove_tree('epydoc') > + > + def clean_tests(self): > + # do not remove incorrect dirs accidentally > + top_dir = > os.path.normpath(os.path.join(self.build_lib, '..')) > + cprefix = os.path.commonprefix((self.build_base, > top_dir)) > + if cprefix != self.build_base: > + return > + > + bin_dir = os.path.join(top_dir, 'bin') > + if os.path.exists(bin_dir): > + remove_tree(bin_dir) > + > + conf_dir = os.path.join(top_dir, 'cnf') > + if os.path.islink(conf_dir): > + print('removing %s symlink' % repr(conf_dir)) > + os.unlink(conf_dir) > + > + pni_file = os.path.join(top_dir, > '.portage_not_installed') > + if os.path.exists(pni_file): > + print('removing %s' % repr(pni_file)) > + os.unlink(pni_file) > + > + def run(self): > + if self.all: > + self.clean_tests() > + self.clean_docs() > + > + clean.run(self) > + > + > +class x_install(install): > + """ install command with extra Portage paths """ > + > + user_options = install.user_options + [ > + # note: $prefix and $exec_prefix are reserved for > Python install > + ('system-prefix=', None, "Prefix for > architecture-independent data"), > + ('system-exec-prefix=', None, "Prefix for > architecture-specific data"), + > + ('bindir=', None, "Install directory for main > executables"), > + ('datarootdir=', None, "Data install root > directory"), > + ('docdir=', None, "Documentation install directory"), > + ('htmldir=', None, "HTML documentation install > directory"), > + ('mandir=', None, "Manpage root install directory"), > + ('portage-base=', 'b', "Portage install base"), > + ('portage-bindir=', None, "Install directory for > Portage internal-use executables"), > + ('portage-datadir=', None, 'Install directory for > data files'), > + ('sbindir=', None, "Install directory for > superuser-intended executables"), > + ('sysconfdir=', None, 'System configuration path'), > + ] > + > + # note: the order is important for proper substitution > + paths = [ > + ('system_prefix', '/usr'), > + ('system_exec_prefix', '$system_prefix'), > + > + ('bindir', '$system_exec_prefix/bin'), > + ('sbindir', '$system_exec_prefix/sbin'), > + ('sysconfdir', '/etc'), > + > + ('datarootdir', '$system_prefix/share'), > + ('docdir', '$datarootdir/doc/$package-$version'), > + ('htmldir', '$docdir/html'), > + ('mandir', '$datarootdir/man'), > + > + ('portage_base', '$system_exec_prefix/lib/portage'), > + ('portage_bindir', '$portage_base/bin'), > + ('portage_datadir', '$datarootdir/portage'), > + > + # not customized at the moment > + ('logrotatedir', '$sysconfdir/logrotate'), > + ('portage_confdir', '$portage_datadir/config'), > + ('portage_setsdir', '$portage_confdir/sets'), > + ] > + > + def initialize_options(self): > + install.initialize_options(self) > + > + for key, default in self.paths: > + setattr(self, key, default) > + self.subst_paths = {} > + > + def finalize_options(self): > + install.finalize_options(self) > + > + # substitute variables > + new_paths = { > + 'package': self.distribution.get_name(), > + 'version': self.distribution.get_version(), > + } > + for key, default in self.paths: > + new_paths[key] = subst_vars(getattr(self, > key), new_paths) > + setattr(self, key, new_paths[key]) > + self.subst_paths = new_paths > + > + > +class x_install_data(install_data): > + """ install_data with customized path support """ > + > + user_options = install_data.user_options > + > + def initialize_options(self): > + install_data.initialize_options(self) > + self.paths = None > + > + def finalize_options(self): > + install_data.finalize_options(self) > + self.set_undefined_options('install', > + ('subst_paths', 'paths')) > + > + # substitute variables in data_files > + for f in self.data_files: > + f[0] = subst_vars(f[0], self.paths) > + > + > +class x_install_lib(install_lib): > + """ install_lib command with Portage path substitution """ > + > + user_options = install_lib.user_options > + > + def initialize_options(self): > + install_lib.initialize_options(self) > + self.portage_base = None > + self.portage_bindir = None > + self.portage_confdir = None > + > + def finalize_options(self): > + install_lib.finalize_options(self) > + self.set_undefined_options('install', > + ('portage_base', 'portage_base'), > + ('portage_bindir', 'portage_bindir'), > + ('portage_confdir', 'portage_confdir')) > + > + def install(self): > + ret = install_lib.install(self) > + > + def rewrite_file(path, val_dict): > + path = os.path.join(self.install_dir, path) > + print('Rewriting %s' % path) > + with codecs.open(path, 'r', 'utf-8') as f: > + data = f.read() > + > + for varname, val in val_dict.items(): > + regexp = r'^(%s\s*=).*$' % varname > + repl = r'\1 %s' % repr(val) > + > + data = re.sub(regexp, repl, data, 0, > re.MULTILINE) + > + with codecs.open(path, 'w', 'utf-8') as f: > + f.write(data) > + > + rewrite_file('portage/__init__.py', { > + 'VERSION': self.distribution.get_version(), > + }) > + rewrite_file('portage/const.py', { > + 'PORTAGE_BASE_PATH': self.portage_base, > + 'PORTAGE_BIN_PATH': self.portage_bindir, > + 'PORTAGE_CONFIG_PATH': self.portage_confdir, > + }) > + > + return ret > + > + > +class x_install_scripts_custom(install_scripts): > + def initialize_options(self): > + install_scripts.initialize_options(self) > + self.root = None > + > + def finalize_options(self): > + self.set_undefined_options('install', > + ('root', 'root'), > + (self.var_name, 'install_dir')) > + install_scripts.finalize_options(self) > + self.build_dir = os.path.join(self.build_dir, > self.dir_name) + > + # prepend root > + if self.root is not None: > + self.install_dir = change_root(self.root, > self.install_dir) + > + > +class x_install_scripts_bin(x_install_scripts_custom): > + dir_name = 'bin' > + var_name = 'bindir' > + > + > +class x_install_scripts_sbin(x_install_scripts_custom): > + dir_name = 'sbin' > + var_name = 'sbindir' > + > + > +class x_install_scripts_portagebin(x_install_scripts_custom): > + dir_name = 'portage' > + var_name = 'portage_bindir' > + > + > +class x_install_scripts(install_scripts): > + def initialize_option(self): > + pass > + > + def finalize_options(self): > + pass > + > + def run(self): > + self.run_command('install_scripts_bin') > + self.run_command('install_scripts_portagebin') > + self.run_command('install_scripts_sbin') > + > + > +class x_sdist(sdist): > + """ sdist defaulting to .tar.bz2 format """ > + > + def finalize_options(self): > + if self.formats is None: > + self.formats = ['bztar'] > + > + sdist.finalize_options(self) > + > + > +class build_tests(x_build_scripts_custom): > + """ Prepare build dir for running tests. """ > + > + def initialize_options(self): > + x_build_scripts_custom.initialize_options(self) > + self.build_base = None > + self.build_lib = None > + > + def finalize_options(self): > + x_build_scripts_custom.finalize_options(self) > + self.set_undefined_options('build', > + ('build_base', 'build_base'), > + ('build_lib', 'build_lib')) > + > + # since we will be writing to $build_lib/.., it is > important > + # that we do not leave $build_base > + self.top_dir = > os.path.normpath(os.path.join(self.build_lib, '..')) > + cprefix = os.path.commonprefix((self.build_base, > self.top_dir)) > + if cprefix != self.build_base: > + raise SystemError('build_lib must be a > subdirectory of build_base') + > + self.build_dir = os.path.join(self.top_dir, 'bin') > + > + def run(self): > + self.run_command('build_py') > + > + # install all scripts $build_lib/../bin > + # (we can't do a symlink since we want shebangs > corrected) > + x_build_scripts_custom.run(self) > + > + # symlink 'cnf' directory > + conf_dir = os.path.join(self.top_dir, 'cnf') > + if os.path.exists(conf_dir): > + if not os.path.islink(conf_dir): > + raise SystemError('%s exists and is > not a symlink (collision)' > + % repr(conf_dir)) > + os.unlink(conf_dir) > + conf_src = os.path.relpath('cnf', self.top_dir) > + print('Symlinking %s -> %s' % (conf_dir, conf_src)) > + os.symlink(conf_src, conf_dir) > + > + # create $build_lib/../.portage_not_installed > + # to enable proper paths in tests > + with open(os.path.join(self.top_dir, > '.portage_not_installed'), 'w') as f: > + pass > + > + > +class test(Command): > + """ run tests """ > + > + user_options = [] > + > + def initialize_options(self): > + self.build_lib = None > + > + def finalize_options(self): > + self.set_undefined_options('build', > + ('build_lib', 'build_lib')) > + > + def run(self): > + self.run_command('build_tests') > + subprocess.check_call([ > + sys.executable, '-bWd', > + os.path.join(self.build_lib, > 'portage/tests/runTests.py') > + ]) > + > + > +def find_packages(): > + for dirpath, dirnames, filenames in os.walk('pym'): > + if '__init__.py' in filenames: > + yield os.path.relpath(dirpath, 'pym') > + > + > +def find_scripts(): > + for dirpath, dirnames, filenames in os.walk('bin'): > + for f in filenames: > + yield os.path.join(dirpath, f) > + > + > +def get_manpages(): > + linguas = os.environ.get('LINGUAS') > + if linguas is not None: > + linguas = linguas.split() > + > + for dirpath, dirnames, filenames in os.walk('man'): > + groups = collections.defaultdict(list) > + for f in filenames: > + fn, suffix = f.rsplit('.', 1) > + groups[suffix].append(os.path.join(dirpath, > f)) + > + topdir = dirpath[len('man/'):] > + if not topdir or linguas is None or topdir in > linguas: > + for g, mans in groups.items(): > + yield [os.path.join('$mandir', > topdir, 'man%s' % g), mans] + > +setup( > + name = 'portage', > + version = '2.2.12', > + url = 'https://wiki.gentoo.org/wiki/Project:Portage', > + author = 'Gentoo Portage Development Team', > + author_email = 'dev-portage@gentoo.org', > + > + package_dir = {'': 'pym'}, > + packages = list(find_packages()), > + # something to cheat build & install commands > + scripts = list(find_scripts()), > + > + data_files = list(get_manpages()) + [ > + ['$sysconfdir', ['cnf/etc-update.conf', > 'cnf/dispatch-conf.conf']], > + ['$logrotatedir', > ['cnf/logrotate.d/elog-save-summary']], > + ['$portage_confdir', [ > + 'cnf/make.conf.example', 'cnf/make.globals', > 'cnf/repos.conf']], > + ['$portage_setsdir', ['cnf/sets/portage.conf']], > + ['$docdir', ['NEWS', 'RELEASE-NOTES']], > + ], > + > + cmdclass = { > + 'build_scripts': x_build_scripts, > + 'build_scripts_bin': x_build_scripts_bin, > + 'build_scripts_portagebin': > x_build_scripts_portagebin, > + 'build_scripts_sbin': x_build_scripts_sbin, > + 'build_tests': build_tests, > + 'clean': x_clean, > + 'docbook': docbook, > + 'epydoc': epydoc, > + 'install': x_install, > + 'install_data': x_install_data, > + 'install_docbook': install_docbook, > + 'install_epydoc': install_epydoc, > + 'install_lib': x_install_lib, > + 'install_scripts': x_install_scripts, > + 'install_scripts_bin': x_install_scripts_bin, > + 'install_scripts_portagebin': > x_install_scripts_portagebin, > + 'install_scripts_sbin': x_install_scripts_sbin, > + 'sdist': x_sdist, > + 'test': test, > + }, > + > + classifiers = [ > + 'Development Status :: 5 - Production/Stable', > + 'Environment :: Console', > + 'Intended Audience :: System Administrators', > + 'License :: OSI Approved :: GNU General Public > License v2 (GPLv2)', > + 'Operating System :: POSIX', > + 'Programming Language :: Python', > + 'Topic :: System :: Installation/Setup' > + ] > +) -- Brian Dolbec <dolsen> ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [gentoo-portage-dev] [PATCH v2] Install Portage using setup.py 2014-08-23 20:30 ` [gentoo-portage-dev] [PATCH v2] Install Portage " Michał Górny 2014-08-23 21:56 ` Brian Dolbec @ 2014-08-29 17:08 ` Brian Dolbec 2014-08-29 23:30 ` Brian Dolbec 2014-09-05 6:58 ` [gentoo-portage-dev] [PATCH v3] " Michał Górny 2 siblings, 1 reply; 23+ messages in thread From: Brian Dolbec @ 2014-08-29 17:08 UTC (permalink / raw To: gentoo-portage-dev [-- Attachment #1: Type: text/plain, Size: 5472 bytes --] On Sat, 23 Aug 2014 22:30:56 +0200 Michał Górny <mgorny@gentoo.org> wrote: > --- > .gitignore | 1 + > MANIFEST.in | 19 ++ > Makefile | 215 ------------------- > doc/Makefile | 11 - > mkrelease.sh | 141 ------------- > pym/portage/const.py | 4 +- > setup.py | 578 > +++++++++++++++++++++++++++++++++++++++++++++++++++ 7 files changed, > 600 insertions(+), 369 deletions(-) create mode 100644 MANIFEST.in > delete mode 100644 Makefile > delete mode 100644 doc/Makefile > delete mode 100755 mkrelease.sh > create mode 100755 setup.py > > > FIXME: not used diff --git a/setup.py b/setup.py new file mode 100755 > index 0000000..7ffe7f0 --- /dev/null > +++ b/setup.py ... > + > +class docbook(Command): > + """ Build docs using docbook. """ > + > + user_options = [ > + ('doc-formats=', None, 'Documentation formats to > build (all xmlto formats for docbook are allowed, comma-separated'), > + ] > + > + def initialize_options(self): > + self.doc_formats = 'xhtml,xhtml-nochunks' > + > + def finalize_options(self): > + self.doc_formats = self.doc_formats.replace(',', ' > ').split() + > + def run(self): > + with open('doc/fragment/date', 'w'): > + pass > + > + for f in self.doc_formats: > + print('Building docs in %s format...' % f) > + subprocess.check_call(['xmlto', '-o', 'doc', > + '-m', 'doc/custom.xsl', f, > 'doc/portage.docbook']) + > + > +class epydoc(Command): > + """ Build API docs using epydoc. """ > + > + user_options = [ > + ] > + > + def initialize_options(self): > + self.build_lib = None > + > + def finalize_options(self): > + self.set_undefined_options('build_py', ('build_lib', > 'build_lib')) + > + def run(self): > + self.run_command('build_py') > + > + print('Building API documentation...') > + > + process_env = os.environ.copy() > + pythonpath = self.build_lib > + try: > + pythonpath += ':' + process_env['PYTHONPATH'] > + except KeyError: > + pass > + process_env['PYTHONPATH'] = pythonpath > + > + subprocess.check_call(['epydoc', '-o', 'epydoc', > + '--name', self.distribution.get_name(), > + '--url', self.distribution.get_url(), > + '-qq', '--no-frames', '--show-imports', > + '--exclude', 'portage.tests', > + '_emerge', 'portage', 'repoman'], > + env = process_env) > + os.remove('epydoc/api-objects.txt') > + > + > +class install_docbook(install_data): > + """ install_data for docbook docs """ > + > + user_options = install_data.user_options > + > + def initialize_options(self): > + install_data.initialize_options(self) > + self.htmldir = None > + > + def finalize_options(self): > + self.set_undefined_options('install', ('htmldir', > 'htmldir')) > + install_data.finalize_options(self) > + > + def run(self): > + if not os.path.exists('doc/portage.html'): > + self.run_command('docbook') > + self.data_files = [ > + (self.htmldir, glob.glob('doc/*.html')), > + ] > + install_data.run(self) > + > + > +class install_epydoc(install_data): > + """ install_data for epydoc docs """ > + > + user_options = install_data.user_options > + > + def initialize_options(self): > + install_data.initialize_options(self) > + self.htmldir = None > + > + def finalize_options(self): > + self.set_undefined_options('install', ('htmldir', > 'htmldir')) > + install_data.finalize_options(self) > + > + def run(self): > + if not os.path.exists('epydoc/index.html'): > + self.run_command('epydoc') > + self.data_files = [ > + (os.path.join(self.htmldir, 'api'), > glob.glob('epydoc/*')), > + ] > + install_data.run(self) > + > + > Both doc and epydoc use flags, install is broken and fails with permission denied error to create the /usr/share/.../portage-9999 directory. ... > +setup( > + name = 'portage', > + version = '2.2.12', You've changed it to 2.2.13_pre1 in git. But it produces incorrect portage-2.2.13_pre1.egginfo version for an otherwise tagged/versioned 9999 install. So set this to 9999 in git and reset to the correct value for a release only it looks like. > + url = 'https://wiki.gentoo.org/wiki/Project:Portage', > + author = 'Gentoo Portage Development Team', > + author_email = 'dev-portage@gentoo.org', > + ... Also, setup.py sdist does not produce an installable tarball. It keeps failing with unicode decode errors on various files on the different target pythons. eg: copying and adjusting bin/fixpackages -> /var/tmp/portage/sys-apps/portage-2.2.13_pre2/work/portage-2.2.13_pre2-python2_7/scripts/sbin/ Traceback (most recent call last): File "/usr/lib64/python3.2/tokenize.py", line 298, in find_cookie line_string = line.decode('utf-8') UnicodeDecodeError: 'utf-8' codec can't decode byte 0xee in position 0: invalid continuation byte ------------------------------- I have changed the version to 9999 and rebased your new-install branch on currrent master, and pushed it to my github account. https://github.com/dol-sen/portage I also am attaching the ebuilds I edited slightly I have uploaded the 2.2.13_pre1 and _pre2 tarballs to my dev space. _pre2 differs from _pre1 in that _pre2 is with the rebased git sources, while _pre1 is Michal's original branch code. you can rename the _pre2.ebuild to _pre1.ebuild to test at your pleasure. -- Brian Dolbec <dolsen> [-- Attachment #2: portage-9999.ebuild --] [-- Type: application/octet-stream, Size: 7201 bytes --] # Copyright 1999-2014 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 # $Header: /var/cvsroot/gentoo-x86/sys-apps/portage/portage-9999.ebuild,v 1.99 2014/08/05 14:19:02 dolsen Exp $ EAPI=5 PYTHON_COMPAT=( pypy python3_2 python3_3 python3_4 python2_7 ) PYTHON_REQ_USE='ssl(+),bzip2(+)' inherit distutils-r1 git-r3 multilib DESCRIPTION="Portage is the package management and distribution system for Gentoo" HOMEPAGE="http://www.gentoo.org/proj/en/portage/index.xml" LICENSE="GPL-2" KEYWORDS="" SLOT="0" IUSE="build doc epydoc +ipc linguas_ru selinux xattr" DEPEND="${python_dep} >=sys-apps/sed-4.0.5 sys-devel/patch doc? ( app-text/xmlto ~app-text/docbook-xml-dtd-4.4 ) epydoc? ( >=dev-python/epydoc-2.0[$(python_gen_usedep 'python2*')] )" # Require sandbox-2.2 for bug #288863. # For xattr, we can spawn getfattr and setfattr from sys-apps/attr, but that's # quite slow, so it's not considered in the dependencies as an alternative to # to python-3.3 / pyxattr. Also, xattr support is only tested with Linux, so # for now, don't pull in xattr deps for other kernels. # For whirlpool hash, require python[ssl] or python-mhash (bug #425046). # For compgen, require bash[readline] (bug #445576). RDEPEND="${python_dep} !build? ( >=sys-apps/sed-4.0.5 || ( >=app-shells/bash-4.2_p37[readline] ( <app-shells/bash-4.2_p37 >=app-shells/bash-3.2_p17 ) ) >=app-admin/eselect-1.2 ) elibc_FreeBSD? ( sys-freebsd/freebsd-bin ) elibc_glibc? ( >=sys-apps/sandbox-2.2 ) elibc_uclibc? ( >=sys-apps/sandbox-2.2 ) >=app-misc/pax-utils-0.1.17 selinux? ( >=sys-libs/libselinux-2.0.94[python,${PYTHON_USEDEP}] ) xattr? ( kernel_linux? ( >=sys-apps/install-xattr-0.3 $(python_gen_cond_dep 'dev-python/pyxattr[${PYTHON_USEDEP}]' \ python{2_7,3_2} pypy) ) ) !<app-shells/bash-3.2_p17 !<app-admin/logrotate-3.8.0" PDEPEND=" !build? ( >=net-misc/rsync-2.6.4 userland_GNU? ( >=sys-apps/coreutils-6.4 ) )" # coreutils-6.4 rdep is for date format in emerge-webrsync #164532 # NOTE: FEATURES=installsources requires debugedit and rsync REQUIRED_USE="epydoc? ( $(python_gen_useflags 'python2*') )" SRC_ARCHIVES="http://dev.gentoo.org/~dolsen/releases/portage" prefix_src_archives() { local x y for x in ${@}; do for y in ${SRC_ARCHIVES}; do echo ${y}/${x} done done } EGIT_REPO_URI="https://github.com/dol-sen/portage.git" EGIT_BRANCH=new-install EGIT_MIN_CLONE_TYPE=single python_prepare_all() { distutils-r1_python_prepare_all einfo "Producing ChangeLog from Git history..." git log ebcf8975b37a8aae9735eb491a9b4cb63549bd5d^.. \ > "${S}"/ChangeLog || die local _version=$(git describe --tags | sed -e 's|-\([0-9]\+\)-.\+$|_p\1|') _version=${_version:1} einfo "Setting portage.VERSION to ${_version} ..." sed -e "s/^VERSION =.*/VERSION = '${_version}'/" -i pym/portage/__init__.py || \ die "Failed to patch portage.VERSION" sed -e "1s/VERSION/${_version}/" -i doc/fragment/version || \ die "Failed to patch VERSION in doc/fragment/version" sed -e "1s/VERSION/${_version}/" -i $(find man -type f) || \ die "Failed to patch VERSION in man page headers" if ! use ipc ; then einfo "Disabling ipc..." sed -e "s:_enable_ipc_daemon = True:_enable_ipc_daemon = False:" \ -i pym/_emerge/AbstractEbuildProcess.py || \ die "failed to patch AbstractEbuildProcess.py" fi if use xattr && use kernel_linux ; then einfo "Adding FEATURES=xattr to make.globals ..." echo -e '\nFEATURES="${FEATURES} xattr"' >> cnf/make.globals \ || die "failed to append to make.globals" fi if [[ -n ${EPREFIX} ]] ; then einfo "Setting portage.const.EPREFIX ..." sed -e "s|^\(SANDBOX_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/bin/sandbox\"\)|\\1${EPREFIX}\\2|" \ -e "s|^\(FAKEROOT_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/bin/fakeroot\"\)|\\1${EPREFIX}\\2|" \ -e "s|^\(BASH_BINARY[[:space:]]*=[[:space:]]*\"\)\(/bin/bash\"\)|\\1${EPREFIX}\\2|" \ -e "s|^\(MOVE_BINARY[[:space:]]*=[[:space:]]*\"\)\(/bin/mv\"\)|\\1${EPREFIX}\\2|" \ -e "s|^\(PRELINK_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/sbin/prelink\"\)|\\1${EPREFIX}\\2|" \ -e "s|^\(EPREFIX[[:space:]]*=[[:space:]]*\"\).*|\\1${EPREFIX}\"|" \ -i pym/portage/const.py || \ die "Failed to patch portage.const.EPREFIX" einfo "Prefixing shebangs ..." while read -r -d $'\0' ; do local shebang=$(head -n1 "$REPLY") if [[ ${shebang} == "#!"* && ! ${shebang} == "#!${EPREFIX}/"* ]] ; then sed -i -e "1s:.*:#!${EPREFIX}${shebang:2}:" "$REPLY" || \ die "sed failed" fi done < <(find . -type f -print0) einfo "Adjusting make.globals ..." sed -e "s|\(/usr/portage\)|${EPREFIX}\\1|" \ -e "s|^\(PORTAGE_TMPDIR=\"\)\(/var/tmp\"\)|\\1${EPREFIX}\\2|" \ -i cnf/make.globals || die "sed failed" einfo "Adjusting repos.conf ..." sed -e "s|^\(main-repo = \).*|\\1gentoo_prefix|" \ -e "s|^\\[gentoo\\]|[gentoo_prefix]|" \ -e "s|^\(location = \)\(/usr/portage\)|\\1${EPREFIX}\\2|" \ -e "s|^\(sync-uri = \).*|\\1rsync://prefix.gentooexperimental.org/gentoo-portage-prefix|" \ -i cnf/repos.conf || die "sed failed" einfo "Adding FEATURES=force-prefix to make.globals ..." echo -e '\nFEATURES="${FEATURES} force-prefix"' >> cnf/make.globals \ || die "failed to append to make.globals" fi cd "${S}/cnf" || die if [ -f "make.conf.example.${ARCH}".diff ]; then patch make.conf.example "make.conf.example.${ARCH}".diff || \ die "Failed to patch make.conf.example" else eerror "" eerror "Portage does not have an arch-specific configuration for this arch." eerror "Please notify the arch maintainer about this issue. Using generic." eerror "" fi } python_compile_all() { local targets=() use doc && targets+=( docbook ) use epydoc && targets+=( epydoc ) if [[ ${targets[@]} ]]; then esetup.py "${targets[@]}" fi } python_test() { esetup.py test } python_install() { distutils-r1_python_install \ --system-prefix="${EPREFIX}/usr" \ --bindir="$(python_get_scriptdir)" \ --docdir="${EPREFIX}/usr/share/doc/${PF}" \ --htmldir="${EPREFIX}/usr/share/doc/${PF}/html" \ --portage-bindir="${EPREFIX}/usr/lib/portage/${EPYTHON}" \ --sbindir="$(python_get_scriptdir)" \ --sysconfdir="${EPREFIX}/etc" \ "${@}" } python_install_all() { distutils-r1_python_install_all local targets=() use doc && targets+=( install_docbook ) use epydoc && targets+=( install_epydoc ) # install docs if [[ ${targets[@]} ]]; then esetup.py "${targets[@]}" fi } pkg_preinst() { if [[ $ROOT == / ]] ; then # Run some minimal tests as a sanity check. local test_runner=$(find "${ED}" -name runTests) if [[ -n $test_runner && -x $test_runner ]] ; then einfo "Running preinst sanity tests..." "$test_runner" || die "preinst sanity tests failed" fi fi # elog dir must exist to avoid logrotate error for bug #415911. # This code runs in preinst in order to bypass the mapping of # portage:portage to root:root which happens after src_install. keepdir /var/log/portage/elog # This is allowed to fail if the user/group are invalid for prefix users. if chown portage:portage "${ED}"var/log/portage{,/elog} 2>/dev/null ; then chmod g+s,ug+rwx "${ED}"var/log/portage{,/elog} fi } [-- Attachment #3: portage-2.2.13_pre2.ebuild --] [-- Type: application/octet-stream, Size: 7118 bytes --] # Copyright 1999-2014 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 # $Header: /var/cvsroot/gentoo-x86/sys-apps/portage/portage-9999.ebuild,v 1.99 2014/08/05 14:19:02 dolsen Exp $ EAPI=5 PYTHON_COMPAT=( pypy python3_2 python3_3 python3_4 python2_7 ) PYTHON_REQ_USE='ssl(+),bzip2(+)' inherit distutils-r1 multilib DESCRIPTION="Portage is the package management and distribution system for Gentoo" HOMEPAGE="http://www.gentoo.org/proj/en/portage/index.xml" LICENSE="GPL-2" KEYWORDS="~alpha ~amd64 ~arm ~arm64 ~hppa ~ia64 ~m68k ~mips ~ppc ~ppc64 ~s390 ~sh ~sparc ~x86 ~amd64-fbsd ~sparc-fbsd ~x86-fbsd" SLOT="0" IUSE="build doc epydoc +ipc linguas_ru selinux xattr" DEPEND="${python_dep} >=sys-apps/sed-4.0.5 sys-devel/patch doc? ( app-text/xmlto ~app-text/docbook-xml-dtd-4.4 ) epydoc? ( >=dev-python/epydoc-2.0[$(python_gen_usedep 'python2*')] )" # Require sandbox-2.2 for bug #288863. # For xattr, we can spawn getfattr and setfattr from sys-apps/attr, but that's # quite slow, so it's not considered in the dependencies as an alternative to # to python-3.3 / pyxattr. Also, xattr support is only tested with Linux, so # for now, don't pull in xattr deps for other kernels. # For whirlpool hash, require python[ssl] or python-mhash (bug #425046). # For compgen, require bash[readline] (bug #445576). RDEPEND="${python_dep} !build? ( >=sys-apps/sed-4.0.5 || ( >=app-shells/bash-4.2_p37[readline] ( <app-shells/bash-4.2_p37 >=app-shells/bash-3.2_p17 ) ) >=app-admin/eselect-1.2 ) elibc_FreeBSD? ( sys-freebsd/freebsd-bin ) elibc_glibc? ( >=sys-apps/sandbox-2.2 ) elibc_uclibc? ( >=sys-apps/sandbox-2.2 ) >=app-misc/pax-utils-0.1.17 selinux? ( >=sys-libs/libselinux-2.0.94[python,${PYTHON_USEDEP}] ) xattr? ( kernel_linux? ( >=sys-apps/install-xattr-0.3 $(python_gen_cond_dep 'dev-python/pyxattr[${PYTHON_USEDEP}]' \ python{2_7,3_2} pypy) ) ) !<app-shells/bash-3.2_p17 !<app-admin/logrotate-3.8.0" PDEPEND=" !build? ( >=net-misc/rsync-2.6.4 userland_GNU? ( >=sys-apps/coreutils-6.4 ) )" # coreutils-6.4 rdep is for date format in emerge-webrsync #164532 # NOTE: FEATURES=installsources requires debugedit and rsync REQUIRED_USE="epydoc? ( $(python_gen_useflags 'python2*') )" SRC_ARCHIVES="http://dev.gentoo.org/~dolsen/releases/portage" prefix_src_archives() { local x y for x in ${@}; do for y in ${SRC_ARCHIVES}; do echo ${y}/${x} done done } TARBALL_PV=${PV} SRC_URI="mirror://gentoo/${PN}-${TARBALL_PV}.tar.bz2 $(prefix_src_archives ${PN}-${TARBALL_PV}.tar.bz2)" python_prepare_all() { distutils-r1_python_prepare_all local _version=${PV} einfo "Setting portage.VERSION to ${_version} ..." sed -e "s/^VERSION =.*/VERSION = '${_version}'/" -i pym/portage/__init__.py || \ die "Failed to patch portage.VERSION" sed -e "1s/VERSION/${_version}/" -i doc/fragment/version || \ die "Failed to patch VERSION in doc/fragment/version" sed -e "1s/VERSION/${_version}/" -i $(find man -type f) || \ die "Failed to patch VERSION in man page headers" if ! use ipc ; then einfo "Disabling ipc..." sed -e "s:_enable_ipc_daemon = True:_enable_ipc_daemon = False:" \ -i pym/_emerge/AbstractEbuildProcess.py || \ die "failed to patch AbstractEbuildProcess.py" fi if use xattr && use kernel_linux ; then einfo "Adding FEATURES=xattr to make.globals ..." echo -e '\nFEATURES="${FEATURES} xattr"' >> cnf/make.globals \ || die "failed to append to make.globals" fi if [[ -n ${EPREFIX} ]] ; then einfo "Setting portage.const.EPREFIX ..." sed -e "s|^\(SANDBOX_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/bin/sandbox\"\)|\\1${EPREFIX}\\2|" \ -e "s|^\(FAKEROOT_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/bin/fakeroot\"\)|\\1${EPREFIX}\\2|" \ -e "s|^\(BASH_BINARY[[:space:]]*=[[:space:]]*\"\)\(/bin/bash\"\)|\\1${EPREFIX}\\2|" \ -e "s|^\(MOVE_BINARY[[:space:]]*=[[:space:]]*\"\)\(/bin/mv\"\)|\\1${EPREFIX}\\2|" \ -e "s|^\(PRELINK_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/sbin/prelink\"\)|\\1${EPREFIX}\\2|" \ -e "s|^\(EPREFIX[[:space:]]*=[[:space:]]*\"\).*|\\1${EPREFIX}\"|" \ -i pym/portage/const.py || \ die "Failed to patch portage.const.EPREFIX" einfo "Prefixing shebangs ..." while read -r -d $'\0' ; do local shebang=$(head -n1 "$REPLY") if [[ ${shebang} == "#!"* && ! ${shebang} == "#!${EPREFIX}/"* ]] ; then sed -i -e "1s:.*:#!${EPREFIX}${shebang:2}:" "$REPLY" || \ die "sed failed" fi done < <(find . -type f -print0) einfo "Adjusting make.globals ..." sed -e "s|\(/usr/portage\)|${EPREFIX}\\1|" \ -e "s|^\(PORTAGE_TMPDIR=\"\)\(/var/tmp\"\)|\\1${EPREFIX}\\2|" \ -i cnf/make.globals || die "sed failed" einfo "Adjusting repos.conf ..." sed -e "s|^\(main-repo = \).*|\\1gentoo_prefix|" \ -e "s|^\\[gentoo\\]|[gentoo_prefix]|" \ -e "s|^\(location = \)\(/usr/portage\)|\\1${EPREFIX}\\2|" \ -e "s|^\(sync-uri = \).*|\\1rsync://prefix.gentooexperimental.org/gentoo-portage-prefix|" \ -i cnf/repos.conf || die "sed failed" einfo "Adding FEATURES=force-prefix to make.globals ..." echo -e '\nFEATURES="${FEATURES} force-prefix"' >> cnf/make.globals \ || die "failed to append to make.globals" fi cd "${S}/cnf" || die if [ -f "make.conf.example.${ARCH}".diff ]; then patch make.conf.example "make.conf.example.${ARCH}".diff || \ die "Failed to patch make.conf.example" else eerror "" eerror "Portage does not have an arch-specific configuration for this arch." eerror "Please notify the arch maintainer about this issue. Using generic." eerror "" fi } python_compile_all() { local targets=() use doc && targets+=( docbook ) use epydoc && targets+=( epydoc ) if [[ ${targets[@]} ]]; then esetup.py "${targets[@]}" fi } python_test() { esetup.py test } python_install() { distutils-r1_python_install \ --system-prefix="${EPREFIX}/usr" \ --bindir="$(python_get_scriptdir)" \ --docdir="${EPREFIX}/usr/share/doc/${PF}" \ --htmldir="${EPREFIX}/usr/share/doc/${PF}/html" \ --portage-bindir="${EPREFIX}/usr/lib/portage/${EPYTHON}" \ --sbindir="$(python_get_scriptdir)" \ --sysconfdir="${EPREFIX}/etc" \ "${@}" } python_install_all() { distutils-r1_python_install_all local targets=() use doc && targets+=( install_docbook ) use epydoc && targets+=( install_epydoc ) # install docs if [[ ${targets[@]} ]]; then esetup.py "${targets[@]}" fi } pkg_preinst() { if [[ $ROOT == / ]] ; then # Run some minimal tests as a sanity check. local test_runner=$(find "${ED}" -name runTests) if [[ -n $test_runner && -x $test_runner ]] ; then einfo "Running preinst sanity tests..." "$test_runner" || die "preinst sanity tests failed" fi fi # elog dir must exist to avoid logrotate error for bug #415911. # This code runs in preinst in order to bypass the mapping of # portage:portage to root:root which happens after src_install. keepdir /var/log/portage/elog # This is allowed to fail if the user/group are invalid for prefix users. if chown portage:portage "${ED}"var/log/portage{,/elog} 2>/dev/null ; then chmod g+s,ug+rwx "${ED}"var/log/portage{,/elog} fi } ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [gentoo-portage-dev] [PATCH v2] Install Portage using setup.py 2014-08-29 17:08 ` Brian Dolbec @ 2014-08-29 23:30 ` Brian Dolbec 0 siblings, 0 replies; 23+ messages in thread From: Brian Dolbec @ 2014-08-29 23:30 UTC (permalink / raw To: gentoo-portage-dev [-- Attachment #1: Type: text/plain, Size: 1463 bytes --] On Fri, 29 Aug 2014 10:08:04 -0700 Brian Dolbec <dolsen@gentoo.org> wrote: > > > I have changed the version to 9999 and rebased your new-install branch > on currrent master, and pushed it to my github account. > > https://github.com/dol-sen/portage > > I also am attaching the ebuilds I edited slightly > > I have uploaded the 2.2.13_pre1 and _pre2 tarballs to my dev space. > _pre2 differs from _pre1 in that _pre2 is with the rebased git > sources, while _pre1 is Michal's original branch code. > > you can rename the _pre2.ebuild to _pre1.ebuild to test at your > pleasure. > OK, all previous problems I reported with setup.py were due to my checkout having some stale files or something sdist didn't like. Even though an updated mkrelease.sh worked on the same files. The doc, epydoc failure was due to my tree being a little too old, it was missing an update to the distutils-r1.eclass mgorny did. As for the version updating. I have removed it from the ebuild I have. There were too many places that updated the version, the ebuild code was missing the setup.py sed for it to be consistent. Also the ebuild should not have to. I think that was from a time when portage releases were just patches to previous tarballs. Thanks to Michal for helping me figure out the errors he was unable to reproduce. I've uploaded a 2.2.13_pre4.tar.bz2 tarball to my releases/portage dir. I've also attached the latest _pre4 ebuild. -- Brian Dolbec <dolsen> [-- Attachment #2: portage-2.2.13_pre4.ebuild --] [-- Type: application/octet-stream, Size: 6687 bytes --] # Copyright 1999-2014 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 # $Header: /var/cvsroot/gentoo-x86/sys-apps/portage/portage-9999.ebuild,v 1.99 2014/08/05 14:19:02 dolsen Exp $ EAPI=5 PYTHON_COMPAT=( pypy python3_2 python3_3 python3_4 python2_7 ) PYTHON_REQ_USE='ssl(+),bzip2(+)' inherit distutils-r1 multilib DESCRIPTION="Portage is the package management and distribution system for Gentoo" HOMEPAGE="http://www.gentoo.org/proj/en/portage/index.xml" LICENSE="GPL-2" KEYWORDS="~alpha ~amd64 ~arm ~arm64 ~hppa ~ia64 ~m68k ~mips ~ppc ~ppc64 ~s390 ~sh ~sparc ~x86 ~amd64-fbsd ~sparc-fbsd ~x86-fbsd" SLOT="0" IUSE="build doc epydoc +ipc linguas_ru selinux xattr" DEPEND="${python_dep} >=sys-apps/sed-4.0.5 sys-devel/patch doc? ( app-text/xmlto ~app-text/docbook-xml-dtd-4.4 ) epydoc? ( >=dev-python/epydoc-2.0[$(python_gen_usedep 'python2*')] )" # Require sandbox-2.2 for bug #288863. # For xattr, we can spawn getfattr and setfattr from sys-apps/attr, but that's # quite slow, so it's not considered in the dependencies as an alternative to # to python-3.3 / pyxattr. Also, xattr support is only tested with Linux, so # for now, don't pull in xattr deps for other kernels. # For whirlpool hash, require python[ssl] or python-mhash (bug #425046). # For compgen, require bash[readline] (bug #445576). RDEPEND="${python_dep} !build? ( >=sys-apps/sed-4.0.5 || ( >=app-shells/bash-4.2_p37[readline] ( <app-shells/bash-4.2_p37 >=app-shells/bash-3.2_p17 ) ) >=app-admin/eselect-1.2 ) elibc_FreeBSD? ( sys-freebsd/freebsd-bin ) elibc_glibc? ( >=sys-apps/sandbox-2.2 ) elibc_uclibc? ( >=sys-apps/sandbox-2.2 ) >=app-misc/pax-utils-0.1.17 selinux? ( >=sys-libs/libselinux-2.0.94[python,${PYTHON_USEDEP}] ) xattr? ( kernel_linux? ( >=sys-apps/install-xattr-0.3 $(python_gen_cond_dep 'dev-python/pyxattr[${PYTHON_USEDEP}]' \ python{2_7,3_2} pypy) ) ) !<app-shells/bash-3.2_p17 !<app-admin/logrotate-3.8.0" PDEPEND=" !build? ( >=net-misc/rsync-2.6.4 userland_GNU? ( >=sys-apps/coreutils-6.4 ) )" # coreutils-6.4 rdep is for date format in emerge-webrsync #164532 # NOTE: FEATURES=installsources requires debugedit and rsync REQUIRED_USE="epydoc? ( $(python_gen_useflags 'python2*') )" SRC_ARCHIVES="http://dev.gentoo.org/~dolsen/releases/portage" prefix_src_archives() { local x y for x in ${@}; do for y in ${SRC_ARCHIVES}; do echo ${y}/${x} done done } TARBALL_PV=${PV} SRC_URI="mirror://gentoo/${PN}-${TARBALL_PV}.tar.bz2 $(prefix_src_archives ${PN}-${TARBALL_PV}.tar.bz2)" python_prepare_all() { distutils-r1_python_prepare_all if ! use ipc ; then einfo "Disabling ipc..." sed -e "s:_enable_ipc_daemon = True:_enable_ipc_daemon = False:" \ -i pym/_emerge/AbstractEbuildProcess.py || \ die "failed to patch AbstractEbuildProcess.py" fi if use xattr && use kernel_linux ; then einfo "Adding FEATURES=xattr to make.globals ..." echo -e '\nFEATURES="${FEATURES} xattr"' >> cnf/make.globals \ || die "failed to append to make.globals" fi if [[ -n ${EPREFIX} ]] ; then einfo "Setting portage.const.EPREFIX ..." sed -e "s|^\(SANDBOX_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/bin/sandbox\"\)|\\1${EPREFIX}\\2|" \ -e "s|^\(FAKEROOT_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/bin/fakeroot\"\)|\\1${EPREFIX}\\2|" \ -e "s|^\(BASH_BINARY[[:space:]]*=[[:space:]]*\"\)\(/bin/bash\"\)|\\1${EPREFIX}\\2|" \ -e "s|^\(MOVE_BINARY[[:space:]]*=[[:space:]]*\"\)\(/bin/mv\"\)|\\1${EPREFIX}\\2|" \ -e "s|^\(PRELINK_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/sbin/prelink\"\)|\\1${EPREFIX}\\2|" \ -e "s|^\(EPREFIX[[:space:]]*=[[:space:]]*\"\).*|\\1${EPREFIX}\"|" \ -i pym/portage/const.py || \ die "Failed to patch portage.const.EPREFIX" einfo "Prefixing shebangs ..." while read -r -d $'\0' ; do local shebang=$(head -n1 "$REPLY") if [[ ${shebang} == "#!"* && ! ${shebang} == "#!${EPREFIX}/"* ]] ; then sed -i -e "1s:.*:#!${EPREFIX}${shebang:2}:" "$REPLY" || \ die "sed failed" fi done < <(find . -type f -print0) einfo "Adjusting make.globals ..." sed -e "s|\(/usr/portage\)|${EPREFIX}\\1|" \ -e "s|^\(PORTAGE_TMPDIR=\"\)\(/var/tmp\"\)|\\1${EPREFIX}\\2|" \ -i cnf/make.globals || die "sed failed" einfo "Adjusting repos.conf ..." sed -e "s|^\(main-repo = \).*|\\1gentoo_prefix|" \ -e "s|^\\[gentoo\\]|[gentoo_prefix]|" \ -e "s|^\(location = \)\(/usr/portage\)|\\1${EPREFIX}\\2|" \ -e "s|^\(sync-uri = \).*|\\1rsync://prefix.gentooexperimental.org/gentoo-portage-prefix|" \ -i cnf/repos.conf || die "sed failed" einfo "Adding FEATURES=force-prefix to make.globals ..." echo -e '\nFEATURES="${FEATURES} force-prefix"' >> cnf/make.globals \ || die "failed to append to make.globals" fi cd "${S}/cnf" || die if [ -f "make.conf.example.${ARCH}".diff ]; then patch make.conf.example "make.conf.example.${ARCH}".diff || \ die "Failed to patch make.conf.example" else eerror "" eerror "Portage does not have an arch-specific configuration for this arch." eerror "Please notify the arch maintainer about this issue. Using generic." eerror "" fi } python_compile_all() { local targets=() use doc && targets+=( docbook ) use epydoc && targets+=( epydoc ) if [[ ${targets[@]} ]]; then esetup.py "${targets[@]}" fi } python_test() { esetup.py test } python_install() { distutils-r1_python_install \ --system-prefix="${EPREFIX}/usr" \ --bindir="$(python_get_scriptdir)" \ --docdir="${EPREFIX}/usr/share/doc/${PF}" \ --htmldir="${EPREFIX}/usr/share/doc/${PF}/html" \ --portage-bindir="${EPREFIX}/usr/lib/portage/${EPYTHON}" \ --sbindir="$(python_get_scriptdir)" \ --sysconfdir="${EPREFIX}/etc" \ "${@}" } python_install_all() { distutils-r1_python_install_all local targets=() use doc && targets+=( install_docbook ) use epydoc && targets+=( install_epydoc ) # install docs if [[ ${targets[@]} ]]; then esetup.py "${targets[@]}" fi } pkg_preinst() { if [[ $ROOT == / ]] ; then # Run some minimal tests as a sanity check. local test_runner=$(find "${ED}" -name runTests) if [[ -n $test_runner && -x $test_runner ]] ; then einfo "Running preinst sanity tests..." "$test_runner" || die "preinst sanity tests failed" fi fi # elog dir must exist to avoid logrotate error for bug #415911. # This code runs in preinst in order to bypass the mapping of # portage:portage to root:root which happens after src_install. keepdir /var/log/portage/elog # This is allowed to fail if the user/group are invalid for prefix users. if chown portage:portage "${ED}"var/log/portage{,/elog} 2>/dev/null ; then chmod g+s,ug+rwx "${ED}"var/log/portage{,/elog} fi } ^ permalink raw reply [flat|nested] 23+ messages in thread
* [gentoo-portage-dev] [PATCH v3] Install Portage using setup.py 2014-08-23 20:30 ` [gentoo-portage-dev] [PATCH v2] Install Portage " Michał Górny 2014-08-23 21:56 ` Brian Dolbec 2014-08-29 17:08 ` Brian Dolbec @ 2014-09-05 6:58 ` Michał Górny 2 siblings, 0 replies; 23+ messages in thread From: Michał Górny @ 2014-09-05 6:58 UTC (permalink / raw To: gentoo-portage-dev; +Cc: Michał Górny Changes in v2: - 'sdist' support Changes in v3: - version substituted properly in docs & mans - cleaner handling of install_data --- .gitignore | 1 + MANIFEST.in | 18 ++ Makefile | 215 ----------------- doc/Makefile | 11 - doc/fragment/date | 0 doc/fragment/version | 1 - mkrelease.sh | 141 ----------- pym/portage/const.py | 4 +- setup.py | 657 +++++++++++++++++++++++++++++++++++++++++++++++++++ 9 files changed, 678 insertions(+), 370 deletions(-) create mode 100644 MANIFEST.in delete mode 100644 Makefile delete mode 100644 doc/Makefile delete mode 100644 doc/fragment/date delete mode 100644 doc/fragment/version delete mode 100755 mkrelease.sh create mode 100755 setup.py diff --git a/.gitignore b/.gitignore index 074bb86..c2dd534 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,5 @@ *.py[co] __pycache__/ *.class +/build /tags diff --git a/MANIFEST.in b/MANIFEST.in new file mode 100644 index 0000000..d65c874 --- /dev/null +++ b/MANIFEST.in @@ -0,0 +1,18 @@ +# docs +include DEVELOPING +include LICENSE +include TEST-NOTES + +# docbook sources +include doc/custom.xsl +recursive-include doc *.docbook + +# extra conf files used in ebuild +include cnf/make.conf.example.* + +# extra files for tests +include .portage_not_installed +include cnf/metadata.dtd + +# extra scripts +include misc/* diff --git a/Makefile b/Makefile deleted file mode 100644 index 9eb6e66..0000000 --- a/Makefile +++ /dev/null @@ -1,215 +0,0 @@ -SHELL = /bin/sh -PN ?= portage -PF ?= portage -HOMEPAGE ?= http://www.gentoo.org/proj/en/portage/index.xml -PWD ?= $(shell pwd) -S ?= $(PWD) -WORKDIR ?= $(PWD) -DESTDIR = $(PWD)/image/ -srcdir = $(S) -prefix = /usr -sysconfdir = /etc -exec_prefix = $(prefix) -bindir = $(exec_prefix)/bin -sbindir = $(exec_prefix)/sbin -libdir = $(exec_prefix)/lib -datarootdir = $(prefix)/share -datadir = $(datarootdir) -mandir = $(datarootdir)/man -docdir = $(datarootdir)/doc/$(PF) -htmldir = $(docdir)/html -portage_datadir = $(datarootdir)/$(PN) -portage_confdir = $(portage_datadir)/config -portage_setsdir = $(portage_confdir)/sets -portage_base = $(libdir)/$(PN) -EPYDOC_OPTS = -qqqqq --no-frames --show-imports -INSMODE = 0644 -EXEMODE = 0755 -DIRMODE = 0755 -SYSCONFDIR_FILES = etc-update.conf dispatch-conf.conf -PORTAGE_CONFDIR_FILES = make.conf.example make.globals repos.conf -LOGROTATE_FILES = elog-save-summary -BINDIR_FILES = ebuild egencache emerge emerge-webrsync \ - emirrordist portageq quickpkg repoman -SBINDIR_FILES = archive-conf dispatch-conf emaint \ - env-update etc-update fixpackages regenworld -DOCS = ChangeLog NEWS RELEASE-NOTES -LINGUAS ?= $(shell cd "$(srcdir)/man" && find -mindepth 1 -type d) - -ifdef PYTHONPATH - PYTHONPATH := $(srcdir)/pym:$(PYTHONPATH) -else - PYTHONPATH := $(srcdir)/pym -endif - -all: docbook epydoc - -docbook: - set -e; \ - touch "$(srcdir)/doc/fragment/date"; \ - $(MAKE) -C "$(srcdir)/doc" xhtml xhtml-nochunks - -epydoc: - set -e; \ - env PYTHONPATH="$(PYTHONPATH)" epydoc \ - -o "$(WORKDIR)/epydoc" \ - --name $(PN) \ - --url "$(HOMEPAGE)" \ - $(EPYDOC_OPTS) \ - $$(cd "$(srcdir)" && find pym -name '*.py' | sed \ - -e s:/__init__.py$$:: \ - -e s:\.py$$:: \ - -e s:^pym/:: \ - -e s:/:.:g \ - | sort); \ - rm -f "$(WORKDIR)/epydoc/api-objects.txt"; \ - -test: - set -e; \ - "$(srcdir)/pym/portage/tests/runTests.py"; \ - -install: - set -e; \ - cd "$(srcdir)/cnf"; \ - install -d -m$(DIRMODE) "$(DESTDIR)$(sysconfdir)"; \ - install -m$(INSMODE) $(SYSCONFDIR_FILES) "$(DESTDIR)$(sysconfdir)"; \ - \ - install -d -m$(DIRMODE) "$(DESTDIR)$(portage_confdir)"; \ - cd "$(srcdir)/cnf"; \ - install -m$(INSMODE) $(PORTAGE_CONFDIR_FILES) \ - "$(DESTDIR)$(portage_confdir)"; \ - install -d -m$(DIRMODE) "$(DESTDIR)$(portage_setsdir)"; \ - cd "$(S)/cnf/sets"; \ - install -m$(INSMODE) *.conf "$(DESTDIR)$(portage_setsdir)"; \ - \ - install -d -m$(DIRMODE) "$(DESTDIR)$(sysconfdir)/logrotate.d"; \ - cd "$(srcdir)/cnf/logrotate.d"; \ - install -m$(INSMODE) $(LOGROTATE_FILES) \ - "$(DESTDIR)$(sysconfdir)/logrotate.d"; \ - \ - for x in $$(cd "$(srcdir)" && find bin -type d) ; do \ - cd "$(srcdir)/$$x"; \ - install -d -m$(DIRMODE) "$(DESTDIR)$(portage_base)/$$x"; \ - files=$$(find . -mindepth 1 -maxdepth 1 -type f ! -type l); \ - if [ -n "$$files" ] ; then \ - install -m$(EXEMODE) $$files \ - "$(DESTDIR)$(portage_base)/$$x"; \ - fi; \ - symlinks=$$(find . -mindepth 1 -maxdepth 1 -type l); \ - if [ -n "$$symlinks" ] ; then \ - cp -P $$symlinks "$(DESTDIR)$(portage_base)/$$x"; \ - fi; \ - done; \ - \ - for x in $$(cd "$(srcdir)" && find pym/* -type d \ - ! -path "pym/portage/tests*") ; do \ - cd "$(srcdir)/$$x"; \ - files=$$(echo *.py); \ - if [ -z "$$files" ] || [ "$$files" = "*.py" ]; then \ - # __pycache__ directories contain no py files \ - continue; \ - fi; \ - install -d -m$(DIRMODE) "$(DESTDIR)$(portage_base)/$$x"; \ - install -m$(INSMODE) $$files "$(DESTDIR)$(portage_base)/$$x"; \ - done; \ - \ - install -d -m$(DIRMODE) "$(DESTDIR)$(bindir)"; \ - relative_path=".."; \ - x=$(bindir) ; \ - y="$(portage_base)"; \ - if [ "$${x#$(prefix)}" != "$$x" ] && \ - [ "$${y#$(prefix)}" != "$$y" ]; then \ - x=$${x#$(prefix)}; \ - y=$${y#$(prefix)}; \ - fi; \ - x=$${x%/*}; \ - while [ -n "$$x" ] ; do \ - relative_path=$${relative_path}/..; \ - x=$${x%/*}; \ - done; \ - relative_path=$$relative_path$$y; \ - for x in $(BINDIR_FILES) ; do \ - ln -sf "$$relative_path/bin/$$x" \ - "$(DESTDIR)$(bindir)/$$x"; \ - done; \ - \ - install -d -m$(DIRMODE) "$(DESTDIR)$(sbindir)"; \ - relative_path=".."; \ - x=$(sbindir) ; \ - y="$(portage_base)"; \ - if [ "$${x#$(prefix)}" != "$$x" ] && \ - [ "$${y#$(prefix)}" != "$$y" ]; then \ - x=$${x#$(prefix)}; \ - y=$${y#$(prefix)}; \ - fi; \ - x=$${x%/*}; \ - while [ -n "$$x" ] ; do \ - relative_path=$${relative_path}/..; \ - x=$${x%/*}; \ - done; \ - relative_path=$$relative_path$$y; \ - for x in $(SBINDIR_FILES) ; do \ - ln -sf "$$relative_path/bin/$$x" \ - "$(DESTDIR)$(sbindir)/$$x"; \ - done; \ - \ - ln -sf "$$relative_path/bin/env-update" \ - "$(DESTDIR)$(sbindir)/update-env"; \ - ln -sf "$$relative_path/bin/etc-update" \ - "$(DESTDIR)$(sbindir)/update-etc"; \ - \ - # We install some minimal tests for use as a preinst sanity check. \ - # These tests must be able to run without a full source tree and \ - # without relying on a previous portage instance being installed. \ - install -d -m$(DIRMODE) \ - "$(DESTDIR)$(portage_base)/pym/portage/tests"; \ - install -m$(EXEMODE) "$(srcdir)/pym/portage/tests/runTests" \ - "$(DESTDIR)$(portage_base)/pym/portage/tests"; \ - cd "$(srcdir)/pym/portage/tests"; \ - install -m$(INSMODE) *.py \ - "$(DESTDIR)$(portage_base)/pym/portage/tests"; \ - install -d -m$(DIRMODE) \ - "$(DESTDIR)$(portage_base)/pym/portage/tests/lint"; \ - cd "$(srcdir)/pym/portage/tests/lint"; \ - install -m$(INSMODE) *.py __test__ \ - "$(DESTDIR)$(portage_base)/pym/portage/tests/lint"; \ - \ - install -d -m$(DIRMODE) "$(DESTDIR)$(docdir)"; \ - cd "$(srcdir)"; \ - install -m $(INSMODE) $(DOCS) "$(DESTDIR)$(docdir)"; \ - \ - for x in "" $(LINGUAS); do \ - for y in 1 5 ; do \ - if [ -d "$(srcdir)/man/$$x" ]; then \ - cd "$(srcdir)/man/$$x"; \ - files=$$(echo *.$$y); \ - if [ -z "$$files" ] || [ "$$files" = "*.$$y" ]; then \ - continue; \ - fi; \ - install -d -m$(DIRMODE) "$(DESTDIR)$(mandir)/$$x/man$$y"; \ - install -m$(INSMODE) *.$$y "$(DESTDIR)$(mandir)/$$x/man$$y"; \ - fi; \ - done; \ - done; \ - \ - if [ -f "$(srcdir)/doc/portage.html" ] ; then \ - install -d -m$(DIRMODE) "$(DESTDIR)$(htmldir)"; \ - cd "$(srcdir)/doc"; \ - install -m$(INSMODE) *.html "$(DESTDIR)$(htmldir)"; \ - fi; \ - \ - if [ -d "$(WORKDIR)/epydoc" ] ; then \ - install -d -m$(DIRMODE) "$(DESTDIR)$(htmldir)"; \ - cp -pPR "$(WORKDIR)/epydoc" \ - "$(DESTDIR)$(htmldir)/api"; \ - cd "$(DESTDIR)$(htmldir)/api"; \ - find . -type d | xargs chmod $(DIRMODE); \ - find . -type f | xargs chmod $(INSMODE); \ - fi; \ - -clean: - set -e; \ - $(MAKE) -C "$(srcdir)/doc" clean; \ - rm -rf "$(WORKDIR)/epydoc"; \ - -.PHONY: all clean docbook epydoc install test diff --git a/doc/Makefile b/doc/Makefile deleted file mode 100644 index 261a0b4..0000000 --- a/doc/Makefile +++ /dev/null @@ -1,11 +0,0 @@ -all: xhtml xhtml-nochunks - -XMLTO_FLAGS = -m custom.xsl -man pdf txt xhtml xhtml-nochunks: - xmlto $@ $(XMLTO_FLAGS) portage.docbook - -clean distclean: - rm -f *.1 *.html portage.txt - -.PHONY: all clean distclean \ - man pdf txt xhtml xhtml-nochunks diff --git a/doc/fragment/date b/doc/fragment/date deleted file mode 100644 index e69de29..0000000 diff --git a/doc/fragment/version b/doc/fragment/version deleted file mode 100644 index f85674c..0000000 --- a/doc/fragment/version +++ /dev/null @@ -1 +0,0 @@ -<releaseinfo>VERSION</releaseinfo> diff --git a/mkrelease.sh b/mkrelease.sh deleted file mode 100755 index f9f7564..0000000 --- a/mkrelease.sh +++ /dev/null @@ -1,141 +0,0 @@ -#!/bin/bash -# Copyright 2008-2014 Gentoo Foundation -# Distributed under the terms of the GNU General Public License v2 - -RELEASE_BUILDDIR=${RELEASE_BUILDDIR:-/var/tmp/portage-release} -SOURCE_DIR=${RELEASE_BUILDDIR}/checkout -BRANCH=${BRANCH:-master} -USE_TAG=false -CHANGELOG_REVISION= -UPLOAD_LOCATION= -RUNTESTS=false -USER= - -usage() { - echo "Usage: ${0##*/} [--changelog-rev <tree-ish>] [-t|--tag] [-u|--upload <location>] [--user <username>] [--runtests] <version>" - exit ${1:-0} -} - -die() { - printf 'error: %s\n' "$*" - usage 1 -} - -ARGS=$(getopt -o htu: --long help,changelog-rev:,runtests,tag,upload:,user: \ - -n "${0##*/}" -- "$@") -[ $? != 0 ] && die "initialization error" - -eval set -- "${ARGS}" - -while true; do - case $1 in - --changelog-rev) - CHANGELOG_REVISION=$2 - shift 2 - ;; - -t|--tag) - USE_TAG=true - shift - ;; - -u|--upload) - UPLOAD_LOCATION=$2 - shift 2 - ;; - --user) - USER=$2"@" - shift 2 - ;; - -h|--help) - usage - ;; - --runtests) - RUNTESTS=true - shift - ;; - --) - shift - break - ;; - *) - die "unknown option: $1" - ;; - esac -done - -[ $# != 1 ] && die "Need version argument" -[[ -n ${1/[0-9]*} ]] && die "Invalid version argument" - -VERSION=$1 -RELEASE=portage-${VERSION} -RELEASE_DIR=${RELEASE_BUILDDIR}/${RELEASE} -RELEASE_TARBALL="${RELEASE_BUILDDIR}/${RELEASE}.tar.bz2" -TREE_ISH=${BRANCH} -if [[ ${USE_TAG} == "true" ]] ; then - TREE_ISH="v${VERSION}" -fi - -echo ">>> Cleaning working directories ${RELEASE_DIR} ${SOURCE_DIR}" -rm -rf "${RELEASE_DIR}" "${SOURCE_DIR}" || die "directory cleanup failed" -mkdir -p "${RELEASE_DIR}" || die "directory creation failed" -mkdir -p "${SOURCE_DIR}" || die "mkdir failed" - -echo ">>> Starting GIT archive" -git archive --format=tar ${TREE_ISH} | \ - tar -xf - -C "${SOURCE_DIR}" || die "git archive failed" - -echo ">>> Building release tree" -cp -a "${SOURCE_DIR}/"{bin,cnf,doc,man,misc,pym} "${RELEASE_DIR}/" || die "directory copy failed" -cp "${SOURCE_DIR}/"{.portage_not_installed,DEVELOPING,LICENSE,Makefile,NEWS,README,RELEASE-NOTES,TEST-NOTES} \ - "${RELEASE_DIR}/" || die "file copy failed" - -if [[ ${RUNTESTS} == "true" ]] ; then - pushd "${SOURCE_DIR}" >/dev/null - ./runtests.sh --python-versions=supported || die "tests failed" - popd >/dev/null -fi - -rm -rf "${SOURCE_DIR}" || die "directory cleanup failed" - -echo ">>> Setting portage.VERSION" -sed -e "s/^VERSION = .*/VERSION = \"${VERSION}\"/" \ - -i "${RELEASE_DIR}/pym/portage/__init__.py" || \ - die "Failed to patch portage.VERSION" - -echo ">>> Creating Changelog" -git_log_opts="" -if [[ -n ${CHANGELOG_REVISION} ]] ; then - git_log_opts+=" ${CHANGELOG_REVISION}^..${TREE_ISH}" -else - git_log_opts+=" ${TREE_ISH}" -fi -skip_next=false -git log ${git_log_opts} | fmt -w 80 -p " " | while read -r ; do - if [[ ${skip_next} == "true" ]] ; then - skip_next=false - elif [[ ${REPLY} == " svn path="* ]] ; then - skip_next=true - else - echo "${REPLY}" - fi -done > "${RELEASE_DIR}/ChangeLog" || die "ChangeLog creation failed" - -cd "${RELEASE_BUILDDIR}" - -echo ">>> Creating release tarball ${RELEASE_TARBALL}" -tar --owner portage --group portage -cjf "${RELEASE_TARBALL}" "${RELEASE}" || \ - die "tarball creation failed" - -DISTDIR=$(portageq distdir) -if [[ -n ${DISTDIR} && -d ${DISTDIR} && -w ${DISTDIR} ]] ; then - echo ">>> Copying release tarball into ${DISTDIR}" - cp "${RELEASE_TARBALL}" "${DISTDIR}"/ || echo "!!! tarball copy failed" -fi - -if [[ -n ${UPLOAD_LOCATION} ]] ; then - echo ">>> Uploading ${RELEASE_TARBALL} to ${USER}dev.gentoo.org:${UPLOAD_LOCATION}" - scp "${RELEASE_TARBALL}" "${USER}dev.gentoo.org:${UPLOAD_LOCATION}" || die "upload failed" -else - du -h "${RELEASE_TARBALL}" -fi - -exit 0 diff --git a/pym/portage/const.py b/pym/portage/const.py index f518b47..acb90f9 100644 --- a/pym/portage/const.py +++ b/pym/portage/const.py @@ -60,8 +60,8 @@ GLOBAL_CONFIG_PATH = "/usr/share/portage/config" # these variables are not used with target_root or config_root # NOTE: Use realpath(__file__) so that python module symlinks in site-packages # are followed back to the real location of the whole portage installation. -PORTAGE_BASE_PATH = os.path.join(os.sep, os.sep.join(os.path.realpath( - __file__.rstrip("co")).split(os.sep)[:-3])) +# NOTE: Please keep PORTAGE_BASE_PATH in one line to help substitutions. +PORTAGE_BASE_PATH = os.path.join(os.sep, os.sep.join(os.path.realpath(__file__.rstrip("co")).split(os.sep)[:-3])) PORTAGE_BIN_PATH = PORTAGE_BASE_PATH + "/bin" PORTAGE_PYM_PATH = os.path.realpath(os.path.join(__file__, '../..')) LOCALE_DATA_PATH = PORTAGE_BASE_PATH + "/locale" # FIXME: not used diff --git a/setup.py b/setup.py new file mode 100755 index 0000000..e5c3631 --- /dev/null +++ b/setup.py @@ -0,0 +1,657 @@ +#!/usr/bin/env python +# Copyright 1998-2014 Gentoo Foundation +# Distributed under the terms of the GNU General Public License v2 + +from distutils.core import setup, Command +from distutils.command.build import build +from distutils.command.build_scripts import build_scripts +from distutils.command.clean import clean +from distutils.command.install import install +from distutils.command.install_data import install_data +from distutils.command.install_lib import install_lib +from distutils.command.install_scripts import install_scripts +from distutils.command.sdist import sdist +from distutils.dep_util import newer +from distutils.dir_util import mkpath, remove_tree +from distutils.util import change_root, subst_vars + +import codecs, collections, errno, glob, os, os.path, re, subprocess, sys + +# TODO: +# - smarter rebuilds of docs w/ 'install_docbook' and 'install_epydoc'. + +x_scripts = { + 'bin': [ + 'bin/ebuild', 'bin/egencache', 'bin/emerge', 'bin/emerge-webrsync', + 'bin/emirrordist', 'bin/portageq', 'bin/quickpkg', 'bin/repoman' + ], + 'sbin': [ + 'bin/archive-conf', 'bin/dispatch-conf', 'bin/emaint', 'bin/env-update', + 'bin/etc-update', 'bin/fixpackages', 'bin/regenworld' + ], +} + + +class x_build(build): + """ Build command with extra build_man call. """ + + def run(self): + build.run(self) + self.run_command('build_man') + + +class build_man(Command): + """ Perform substitutions in manpages. """ + + user_options = [ + ] + + def initialize_options(self): + self.build_base = None + + def finalize_options(self): + self.set_undefined_options('build', + ('build_base', 'build_base')) + + def run(self): + for d, files in self.distribution.data_files: + if not d.startswith('$mandir/'): + continue + + for source in files: + target = os.path.join(self.build_base, source) + mkpath(os.path.dirname(target)) + + if not newer(source, target) and not newer(__file__, target): + continue + + print('copying and updating %s -> %s' % ( + source, target)) + + with codecs.open(source, 'r', 'utf8') as f: + data = f.readlines() + data[0] = data[0].replace('VERSION', + self.distribution.get_version()) + with codecs.open(target, 'w', 'utf8') as f: + f.writelines(data) + + +class docbook(Command): + """ Build docs using docbook. """ + + user_options = [ + ('doc-formats=', None, 'Documentation formats to build (all xmlto formats for docbook are allowed, comma-separated'), + ] + + def initialize_options(self): + self.doc_formats = 'xhtml,xhtml-nochunks' + + def finalize_options(self): + self.doc_formats = self.doc_formats.replace(',', ' ').split() + + def run(self): + if not os.path.isdir('doc/fragment'): + mkpath('doc/fragment') + + with open('doc/fragment/date', 'w'): + pass + with open('doc/fragment/version', 'w') as f: + f.write('<releaseinfo>%s</releaseinfo>' % self.distribution.get_version()) + + for f in self.doc_formats: + print('Building docs in %s format...' % f) + subprocess.check_call(['xmlto', '-o', 'doc', + '-m', 'doc/custom.xsl', f, 'doc/portage.docbook']) + + +class epydoc(Command): + """ Build API docs using epydoc. """ + + user_options = [ + ] + + def initialize_options(self): + self.build_lib = None + + def finalize_options(self): + self.set_undefined_options('build_py', ('build_lib', 'build_lib')) + + def run(self): + self.run_command('build_py') + + print('Building API documentation...') + + process_env = os.environ.copy() + pythonpath = self.build_lib + try: + pythonpath += ':' + process_env['PYTHONPATH'] + except KeyError: + pass + process_env['PYTHONPATH'] = pythonpath + + subprocess.check_call(['epydoc', '-o', 'epydoc', + '--name', self.distribution.get_name(), + '--url', self.distribution.get_url(), + '-qq', '--no-frames', '--show-imports', + '--exclude', 'portage.tests', + '_emerge', 'portage', 'repoman'], + env = process_env) + os.remove('epydoc/api-objects.txt') + + +class install_docbook(install_data): + """ install_data for docbook docs """ + + user_options = install_data.user_options + + def initialize_options(self): + install_data.initialize_options(self) + self.htmldir = None + + def finalize_options(self): + self.set_undefined_options('install', ('htmldir', 'htmldir')) + install_data.finalize_options(self) + + def run(self): + if not os.path.exists('doc/portage.html'): + self.run_command('docbook') + self.data_files = [ + (self.htmldir, glob.glob('doc/*.html')), + ] + install_data.run(self) + + +class install_epydoc(install_data): + """ install_data for epydoc docs """ + + user_options = install_data.user_options + + def initialize_options(self): + install_data.initialize_options(self) + self.htmldir = None + + def finalize_options(self): + self.set_undefined_options('install', ('htmldir', 'htmldir')) + install_data.finalize_options(self) + + def run(self): + if not os.path.exists('epydoc/index.html'): + self.run_command('epydoc') + self.data_files = [ + (os.path.join(self.htmldir, 'api'), glob.glob('epydoc/*')), + ] + install_data.run(self) + + +class x_build_scripts_custom(build_scripts): + def finalize_options(self): + build_scripts.finalize_options(self) + if 'dir_name' in dir(self): + self.build_dir = os.path.join(self.build_dir, self.dir_name) + if self.dir_name in x_scripts: + self.scripts = x_scripts[self.dir_name] + else: + self.scripts = set(self.scripts) + for other_files in x_scripts.values(): + self.scripts.difference_update(other_files) + + def run(self): + # group scripts by subdirectory + split_scripts = collections.defaultdict(list) + for f in self.scripts: + dir_name = os.path.dirname(f[len('bin/'):]) + split_scripts[dir_name].append(f) + + base_dir = self.build_dir + base_scripts = self.scripts + for d, files in split_scripts.items(): + self.build_dir = os.path.join(base_dir, d) + self.scripts = files + self.copy_scripts() + + # restore previous values + self.build_dir = base_dir + self.scripts = base_scripts + + +class x_build_scripts_bin(x_build_scripts_custom): + dir_name = 'bin' + + +class x_build_scripts_sbin(x_build_scripts_custom): + dir_name = 'sbin' + + +class x_build_scripts_portagebin(x_build_scripts_custom): + dir_name = 'portage' + + +class x_build_scripts(build_scripts): + def initialize_option(self): + build_scripts.initialize_options(self) + + def finalize_options(self): + build_scripts.finalize_options(self) + + def run(self): + self.run_command('build_scripts_bin') + self.run_command('build_scripts_portagebin') + self.run_command('build_scripts_sbin') + + +class x_clean(clean): + """ clean extended for doc & post-test cleaning """ + + def clean_docs(self): + def get_doc_outfiles(): + for dirpath, dirnames, filenames in os.walk('doc'): + for f in filenames: + if f.endswith('.docbook') or f == 'custom.xsl': + pass + else: + yield os.path.join(dirpath, f) + + # do not recurse + break + + + for f in get_doc_outfiles(): + print('removing %s' % repr(f)) + os.remove(f) + + if os.path.isdir('doc/fragment'): + remove_tree('doc/fragment') + + if os.path.isdir('epydoc'): + remove_tree('epydoc') + + def clean_tests(self): + # do not remove incorrect dirs accidentally + top_dir = os.path.normpath(os.path.join(self.build_lib, '..')) + cprefix = os.path.commonprefix((self.build_base, top_dir)) + if cprefix != self.build_base: + return + + bin_dir = os.path.join(top_dir, 'bin') + if os.path.exists(bin_dir): + remove_tree(bin_dir) + + conf_dir = os.path.join(top_dir, 'cnf') + if os.path.islink(conf_dir): + print('removing %s symlink' % repr(conf_dir)) + os.unlink(conf_dir) + + pni_file = os.path.join(top_dir, '.portage_not_installed') + if os.path.exists(pni_file): + print('removing %s' % repr(pni_file)) + os.unlink(pni_file) + + def clean_man(self): + man_dir = os.path.join(self.build_base, 'man') + if os.path.exists(man_dir): + remove_tree(man_dir) + + def run(self): + if self.all: + self.clean_tests() + self.clean_docs() + self.clean_man() + + clean.run(self) + + +class x_install(install): + """ install command with extra Portage paths """ + + user_options = install.user_options + [ + # note: $prefix and $exec_prefix are reserved for Python install + ('system-prefix=', None, "Prefix for architecture-independent data"), + ('system-exec-prefix=', None, "Prefix for architecture-specific data"), + + ('bindir=', None, "Install directory for main executables"), + ('datarootdir=', None, "Data install root directory"), + ('docdir=', None, "Documentation install directory"), + ('htmldir=', None, "HTML documentation install directory"), + ('mandir=', None, "Manpage root install directory"), + ('portage-base=', 'b', "Portage install base"), + ('portage-bindir=', None, "Install directory for Portage internal-use executables"), + ('portage-datadir=', None, 'Install directory for data files'), + ('sbindir=', None, "Install directory for superuser-intended executables"), + ('sysconfdir=', None, 'System configuration path'), + ] + + # note: the order is important for proper substitution + paths = [ + ('system_prefix', '/usr'), + ('system_exec_prefix', '$system_prefix'), + + ('bindir', '$system_exec_prefix/bin'), + ('sbindir', '$system_exec_prefix/sbin'), + ('sysconfdir', '/etc'), + + ('datarootdir', '$system_prefix/share'), + ('docdir', '$datarootdir/doc/$package-$version'), + ('htmldir', '$docdir/html'), + ('mandir', '$datarootdir/man'), + + ('portage_base', '$system_exec_prefix/lib/portage'), + ('portage_bindir', '$portage_base/bin'), + ('portage_datadir', '$datarootdir/portage'), + + # not customized at the moment + ('logrotatedir', '$sysconfdir/logrotate'), + ('portage_confdir', '$portage_datadir/config'), + ('portage_setsdir', '$portage_confdir/sets'), + ] + + def initialize_options(self): + install.initialize_options(self) + + for key, default in self.paths: + setattr(self, key, default) + self.subst_paths = {} + + def finalize_options(self): + install.finalize_options(self) + + # substitute variables + new_paths = { + 'package': self.distribution.get_name(), + 'version': self.distribution.get_version(), + } + for key, default in self.paths: + new_paths[key] = subst_vars(getattr(self, key), new_paths) + setattr(self, key, new_paths[key]) + self.subst_paths = new_paths + + +class x_install_data(install_data): + """ install_data with customized path support """ + + user_options = install_data.user_options + + def initialize_options(self): + install_data.initialize_options(self) + self.build_base = None + self.paths = None + + def finalize_options(self): + install_data.finalize_options(self) + self.set_undefined_options('build', + ('build_base', 'build_base')) + self.set_undefined_options('install', + ('subst_paths', 'paths')) + + def run(self): + self.run_command('build_man') + + def process_data_files(df): + for d, files in df: + # substitute man sources + if d.startswith('$mandir/'): + files = [os.path.join(self.build_base, v) for v in files] + + # substitute variables in path + d = subst_vars(d, self.paths) + yield (d, files) + + old_data_files = self.data_files + self.data_files = process_data_files(self.data_files) + + install_data.run(self) + self.data_files = old_data_files + + +class x_install_lib(install_lib): + """ install_lib command with Portage path substitution """ + + user_options = install_lib.user_options + + def initialize_options(self): + install_lib.initialize_options(self) + self.portage_base = None + self.portage_bindir = None + self.portage_confdir = None + + def finalize_options(self): + install_lib.finalize_options(self) + self.set_undefined_options('install', + ('portage_base', 'portage_base'), + ('portage_bindir', 'portage_bindir'), + ('portage_confdir', 'portage_confdir')) + + def install(self): + ret = install_lib.install(self) + + def rewrite_file(path, val_dict): + path = os.path.join(self.install_dir, path) + print('Rewriting %s' % path) + with codecs.open(path, 'r', 'utf-8') as f: + data = f.read() + + for varname, val in val_dict.items(): + regexp = r'(?m)^(%s\s*=).*$' % varname + repl = r'\1 %s' % repr(val) + + data = re.sub(regexp, repl, data) + + with codecs.open(path, 'w', 'utf-8') as f: + f.write(data) + + rewrite_file('portage/__init__.py', { + 'VERSION': self.distribution.get_version(), + }) + rewrite_file('portage/const.py', { + 'PORTAGE_BASE_PATH': self.portage_base, + 'PORTAGE_BIN_PATH': self.portage_bindir, + 'PORTAGE_CONFIG_PATH': self.portage_confdir, + }) + + return ret + + +class x_install_scripts_custom(install_scripts): + def initialize_options(self): + install_scripts.initialize_options(self) + self.root = None + + def finalize_options(self): + self.set_undefined_options('install', + ('root', 'root'), + (self.var_name, 'install_dir')) + install_scripts.finalize_options(self) + self.build_dir = os.path.join(self.build_dir, self.dir_name) + + # prepend root + if self.root is not None: + self.install_dir = change_root(self.root, self.install_dir) + + +class x_install_scripts_bin(x_install_scripts_custom): + dir_name = 'bin' + var_name = 'bindir' + + +class x_install_scripts_sbin(x_install_scripts_custom): + dir_name = 'sbin' + var_name = 'sbindir' + + +class x_install_scripts_portagebin(x_install_scripts_custom): + dir_name = 'portage' + var_name = 'portage_bindir' + + +class x_install_scripts(install_scripts): + def initialize_option(self): + pass + + def finalize_options(self): + pass + + def run(self): + self.run_command('install_scripts_bin') + self.run_command('install_scripts_portagebin') + self.run_command('install_scripts_sbin') + + +class x_sdist(sdist): + """ sdist defaulting to .tar.bz2 format """ + + def finalize_options(self): + if self.formats is None: + self.formats = ['bztar'] + + sdist.finalize_options(self) + + +class build_tests(x_build_scripts_custom): + """ Prepare build dir for running tests. """ + + def initialize_options(self): + x_build_scripts_custom.initialize_options(self) + self.build_base = None + self.build_lib = None + + def finalize_options(self): + x_build_scripts_custom.finalize_options(self) + self.set_undefined_options('build', + ('build_base', 'build_base'), + ('build_lib', 'build_lib')) + + # since we will be writing to $build_lib/.., it is important + # that we do not leave $build_base + self.top_dir = os.path.normpath(os.path.join(self.build_lib, '..')) + cprefix = os.path.commonprefix((self.build_base, self.top_dir)) + if cprefix != self.build_base: + raise SystemError('build_lib must be a subdirectory of build_base') + + self.build_dir = os.path.join(self.top_dir, 'bin') + + def run(self): + self.run_command('build_py') + + # install all scripts $build_lib/../bin + # (we can't do a symlink since we want shebangs corrected) + x_build_scripts_custom.run(self) + + # symlink 'cnf' directory + conf_dir = os.path.join(self.top_dir, 'cnf') + if os.path.exists(conf_dir): + if not os.path.islink(conf_dir): + raise SystemError('%s exists and is not a symlink (collision)' + % repr(conf_dir)) + os.unlink(conf_dir) + conf_src = os.path.relpath('cnf', self.top_dir) + print('Symlinking %s -> %s' % (conf_dir, conf_src)) + os.symlink(conf_src, conf_dir) + + # create $build_lib/../.portage_not_installed + # to enable proper paths in tests + with open(os.path.join(self.top_dir, '.portage_not_installed'), 'w') as f: + pass + + +class test(Command): + """ run tests """ + + user_options = [] + + def initialize_options(self): + self.build_lib = None + + def finalize_options(self): + self.set_undefined_options('build', + ('build_lib', 'build_lib')) + + def run(self): + self.run_command('build_tests') + subprocess.check_call([ + sys.executable, '-bWd', + os.path.join(self.build_lib, 'portage/tests/runTests.py') + ]) + + +def find_packages(): + for dirpath, dirnames, filenames in os.walk('pym'): + if '__init__.py' in filenames: + yield os.path.relpath(dirpath, 'pym') + + +def find_scripts(): + for dirpath, dirnames, filenames in os.walk('bin'): + for f in filenames: + yield os.path.join(dirpath, f) + + +def get_manpages(): + linguas = os.environ.get('LINGUAS') + if linguas is not None: + linguas = linguas.split() + + for dirpath, dirnames, filenames in os.walk('man'): + groups = collections.defaultdict(list) + for f in filenames: + fn, suffix = f.rsplit('.', 1) + groups[suffix].append(os.path.join(dirpath, f)) + + topdir = dirpath[len('man/'):] + if not topdir or linguas is None or topdir in linguas: + for g, mans in groups.items(): + yield [os.path.join('$mandir', topdir, 'man%s' % g), mans] + +setup( + name = 'portage', + version = '2.2.12', + url = 'https://wiki.gentoo.org/wiki/Project:Portage', + author = 'Gentoo Portage Development Team', + author_email = 'dev-portage@gentoo.org', + + package_dir = {'': 'pym'}, + packages = list(find_packages()), + # something to cheat build & install commands + scripts = list(find_scripts()), + + data_files = list(get_manpages()) + [ + ['$sysconfdir', ['cnf/etc-update.conf', 'cnf/dispatch-conf.conf']], + ['$logrotatedir', ['cnf/logrotate.d/elog-save-summary']], + ['$portage_confdir', [ + 'cnf/make.conf.example', 'cnf/make.globals', 'cnf/repos.conf']], + ['$portage_setsdir', ['cnf/sets/portage.conf']], + ['$docdir', ['NEWS', 'RELEASE-NOTES']], + ], + + cmdclass = { + 'build': x_build, + 'build_man': build_man, + 'build_scripts': x_build_scripts, + 'build_scripts_bin': x_build_scripts_bin, + 'build_scripts_portagebin': x_build_scripts_portagebin, + 'build_scripts_sbin': x_build_scripts_sbin, + 'build_tests': build_tests, + 'clean': x_clean, + 'docbook': docbook, + 'epydoc': epydoc, + 'install': x_install, + 'install_data': x_install_data, + 'install_docbook': install_docbook, + 'install_epydoc': install_epydoc, + 'install_lib': x_install_lib, + 'install_scripts': x_install_scripts, + 'install_scripts_bin': x_install_scripts_bin, + 'install_scripts_portagebin': x_install_scripts_portagebin, + 'install_scripts_sbin': x_install_scripts_sbin, + 'sdist': x_sdist, + 'test': test, + }, + + classifiers = [ + 'Development Status :: 5 - Production/Stable', + 'Environment :: Console', + 'Intended Audience :: System Administrators', + 'License :: OSI Approved :: GNU General Public License v2 (GPLv2)', + 'Operating System :: POSIX', + 'Programming Language :: Python', + 'Topic :: System :: Installation/Setup' + ] +) -- 2.1.0 ^ permalink raw reply related [flat|nested] 23+ messages in thread
* [gentoo-portage-dev] [PATCH 6/6] travis: Use setup.py for running tests 2014-08-21 20:19 [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny ` (4 preceding siblings ...) 2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 5/6] Install Python modules using setup.py Michał Górny @ 2014-08-21 20:19 ` Michał Górny 2014-08-21 20:22 ` [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny ` (3 subsequent siblings) 9 siblings, 0 replies; 23+ messages in thread From: Michał Górny @ 2014-08-21 20:19 UTC (permalink / raw To: gentoo-portage-dev; +Cc: Michał Górny --- .travis.yml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 52ca444..563f9ca 100644 --- a/.travis.yml +++ b/.travis.yml @@ -7,4 +7,8 @@ python: - 3.4 - pypy script: - - python -b -Wd pym/portage/tests/runTests.py + - ./setup.py test + - touch ChangeLog # seriously stupid + - ./setup.py install --root=/tmp/install-root + - sudo rsync -a /tmp/install-root/. / + - python -b -Wd -m portage.tests.runTests -- 2.0.4 ^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [gentoo-portage-dev] [PATCHES] setup.py install for Portage 2014-08-21 20:19 [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny ` (5 preceding siblings ...) 2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 6/6] travis: Use setup.py for running tests Michał Górny @ 2014-08-21 20:22 ` Michał Górny 2014-08-21 20:25 ` Michał Górny 2014-08-22 22:32 ` Brian Dolbec ` (2 subsequent siblings) 9 siblings, 1 reply; 23+ messages in thread From: Michał Górny @ 2014-08-21 20:22 UTC (permalink / raw To: gentoo-portage-dev [-- Attachment #1.1: Type: text/plain, Size: 893 bytes --] Dnia 2014-08-21, o godz. 22:19:38 Michał Górny <mgorny@gentoo.org> napisał(a): > I will follow this thread with updated ebuild. And here's the ebuild. You can also find it in ::mgorny. One minor note: after applying the distutils-r1 patch I submitted to gentoo-dev@ today, the following diff can be applied to improve the ebuild: diff --git a/sys-apps/portage/portage-9999.ebuild b/sys-apps/portage/portage-9999.ebuild index 3fee385..f11c62a 100644 --- a/sys-apps/portage/portage-9999.ebuild +++ b/sys-apps/portage/portage-9999.ebuild @@ -187,7 +187,7 @@ python_install_all() { use epydoc && targets+=( install_epydoc ) # install docs - python_install "${targets[@]}" + esetup.py "${targets[@]}" # symlink compatibility /usr/lib/portage/bin dosym "${EPYTHON}" /usr/lib/portage/bin -- Best regards, Michał Górny [-- Attachment #1.2: portage-9999.ebuild --] [-- Type: text/plain, Size: 7530 bytes --] # Copyright 1999-2014 Gentoo Foundation # Distributed under the terms of the GNU General Public License v2 # $Header: /var/cvsroot/gentoo-x86/sys-apps/portage/portage-9999.ebuild,v 1.99 2014/08/05 14:19:02 dolsen Exp $ EAPI=5 PYTHON_COMPAT=( pypy python3_2 python3_3 python3_4 python2_7 ) PYTHON_REQ_USE='ssl(+),bzip2(+)' inherit distutils-r1 git-r3 multilib DESCRIPTION="Portage is the package management and distribution system for Gentoo" HOMEPAGE="http://www.gentoo.org/proj/en/portage/index.xml" LICENSE="GPL-2" KEYWORDS="" SLOT="0" IUSE="build doc epydoc +ipc linguas_ru selinux xattr" DEPEND="${python_dep} >=sys-apps/sed-4.0.5 sys-devel/patch doc? ( app-text/xmlto ~app-text/docbook-xml-dtd-4.4 ) epydoc? ( >=dev-python/epydoc-2.0[$(python_gen_usedep 'python2*')] )" # Require sandbox-2.2 for bug #288863. # For xattr, we can spawn getfattr and setfattr from sys-apps/attr, but that's # quite slow, so it's not considered in the dependencies as an alternative to # to python-3.3 / pyxattr. Also, xattr support is only tested with Linux, so # for now, don't pull in xattr deps for other kernels. # For whirlpool hash, require python[ssl] or python-mhash (bug #425046). # For compgen, require bash[readline] (bug #445576). RDEPEND="${python_dep} !build? ( >=sys-apps/sed-4.0.5 || ( >=app-shells/bash-4.2_p37[readline] ( <app-shells/bash-4.2_p37 >=app-shells/bash-3.2_p17 ) ) >=app-admin/eselect-1.2 ) elibc_FreeBSD? ( sys-freebsd/freebsd-bin ) elibc_glibc? ( >=sys-apps/sandbox-2.2 ) elibc_uclibc? ( >=sys-apps/sandbox-2.2 ) >=app-misc/pax-utils-0.1.17 selinux? ( >=sys-libs/libselinux-2.0.94[python,${PYTHON_USEDEP}] ) xattr? ( kernel_linux? ( >=sys-apps/install-xattr-0.3 $(python_gen_cond_dep 'dev-python/pyxattr[${PYTHON_USEDEP}]' \ python{2_7,3_2} pypy) ) ) !<app-shells/bash-3.2_p17 !<app-admin/logrotate-3.8.0" PDEPEND=" !build? ( >=net-misc/rsync-2.6.4 userland_GNU? ( >=sys-apps/coreutils-6.4 ) )" # coreutils-6.4 rdep is for date format in emerge-webrsync #164532 # NOTE: FEATURES=installsources requires debugedit and rsync REQUIRED_USE="epydoc? ( $(python_gen_useflags 'python2*') )" SRC_ARCHIVES="http://dev.gentoo.org/~dolsen/releases/portage" prefix_src_archives() { local x y for x in ${@}; do for y in ${SRC_ARCHIVES}; do echo ${y}/${x} done done } EGIT_REPO_URI="https://github.com/mgorny/portage.git" EGIT_BRANCH=new-install EGIT_MIN_CLONE_TYPE=single python_prepare_all() { distutils-r1_python_prepare_all einfo "Producing ChangeLog from Git history..." git log ebcf8975b37a8aae9735eb491a9b4cb63549bd5d^.. \ > "${S}"/ChangeLog || die local _version=$(git describe --tags | sed -e 's|-\([0-9]\+\)-.\+$|_p\1|') _version=${_version:1} einfo "Setting portage.VERSION to ${_version} ..." sed -e "s/^VERSION =.*/VERSION = '${_version}'/" -i pym/portage/__init__.py || \ die "Failed to patch portage.VERSION" sed -e "1s/VERSION/${_version}/" -i doc/fragment/version || \ die "Failed to patch VERSION in doc/fragment/version" sed -e "1s/VERSION/${_version}/" -i $(find man -type f) || \ die "Failed to patch VERSION in man page headers" if ! use ipc ; then einfo "Disabling ipc..." sed -e "s:_enable_ipc_daemon = True:_enable_ipc_daemon = False:" \ -i pym/_emerge/AbstractEbuildProcess.py || \ die "failed to patch AbstractEbuildProcess.py" fi if use xattr && use kernel_linux ; then einfo "Adding FEATURES=xattr to make.globals ..." echo -e '\nFEATURES="${FEATURES} xattr"' >> cnf/make.globals \ || die "failed to append to make.globals" fi if [[ -n ${EPREFIX} ]] ; then einfo "Setting portage.const.EPREFIX ..." sed -e "s|^\(SANDBOX_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/bin/sandbox\"\)|\\1${EPREFIX}\\2|" \ -e "s|^\(FAKEROOT_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/bin/fakeroot\"\)|\\1${EPREFIX}\\2|" \ -e "s|^\(BASH_BINARY[[:space:]]*=[[:space:]]*\"\)\(/bin/bash\"\)|\\1${EPREFIX}\\2|" \ -e "s|^\(MOVE_BINARY[[:space:]]*=[[:space:]]*\"\)\(/bin/mv\"\)|\\1${EPREFIX}\\2|" \ -e "s|^\(PRELINK_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/sbin/prelink\"\)|\\1${EPREFIX}\\2|" \ -e "s|^\(EPREFIX[[:space:]]*=[[:space:]]*\"\).*|\\1${EPREFIX}\"|" \ -i pym/portage/const.py || \ die "Failed to patch portage.const.EPREFIX" einfo "Prefixing shebangs ..." while read -r -d $'\0' ; do local shebang=$(head -n1 "$REPLY") if [[ ${shebang} == "#!"* && ! ${shebang} == "#!${EPREFIX}/"* ]] ; then sed -i -e "1s:.*:#!${EPREFIX}${shebang:2}:" "$REPLY" || \ die "sed failed" fi done < <(find . -type f -print0) einfo "Adjusting make.globals ..." sed -e "s|\(/usr/portage\)|${EPREFIX}\\1|" \ -e "s|^\(PORTAGE_TMPDIR=\"\)\(/var/tmp\"\)|\\1${EPREFIX}\\2|" \ -i cnf/make.globals || die "sed failed" einfo "Adjusting repos.conf ..." sed -e "s|^\(main-repo = \).*|\\1gentoo_prefix|" \ -e "s|^\\[gentoo\\]|[gentoo_prefix]|" \ -e "s|^\(location = \)\(/usr/portage\)|\\1${EPREFIX}\\2|" \ -e "s|^\(sync-uri = \).*|\\1rsync://prefix.gentooexperimental.org/gentoo-portage-prefix|" \ -i cnf/repos.conf || die "sed failed" einfo "Adding FEATURES=force-prefix to make.globals ..." echo -e '\nFEATURES="${FEATURES} force-prefix"' >> cnf/make.globals \ || die "failed to append to make.globals" fi cd "${S}/cnf" || die if [ -f "make.conf.example.${ARCH}".diff ]; then patch make.conf.example "make.conf.example.${ARCH}".diff || \ die "Failed to patch make.conf.example" else eerror "" eerror "Portage does not have an arch-specific configuration for this arch." eerror "Please notify the arch maintainer about this issue. Using generic." eerror "" fi } python_compile_all() { local targets=() use doc && targets+=( docbook ) use epydoc && targets+=( epydoc ) if [[ ${targets[@]} ]]; then esetup.py "${targets[@]}" fi } python_test() { esetup.py test } python_install() { distutils-r1_python_install \ --system-prefix="${EPREFIX}/usr" \ --bindir="$(python_get_scriptdir)" \ --docdir="${EPREFIX}/usr/share/doc/${PF}" \ --htmldir="${EPREFIX}/usr/share/doc/${PF}/html" \ --portage-bindir="${EPREFIX}/usr/lib/portage/${EPYTHON}" \ --sbindir="$(python_get_scriptdir)" \ --sysconfdir="${EPREFIX}/etc" \ "${@}" } python_install_all() { distutils-r1_python_install_all local targets=() use doc && targets+=( install_docbook ) use epydoc && targets+=( install_epydoc ) # install docs esetup.py "${targets[@]}" # symlink compatibility /usr/lib/portage/bin dosym "${EPYTHON}" /usr/lib/portage/bin # used as safe location with no files keepdir /var/empty } pkg_preinst() { if [[ $ROOT == / ]] ; then # Run some minimal tests as a sanity check. local test_runner=$(find "${ED}" -name runTests) if [[ -n $test_runner && -x $test_runner ]] ; then einfo "Running preinst sanity tests..." "$test_runner" || die "preinst sanity tests failed" fi fi # elog dir must exist to avoid logrotate error for bug #415911. # This code runs in preinst in order to bypass the mapping of # portage:portage to root:root which happens after src_install. keepdir /var/log/portage/elog # This is allowed to fail if the user/group are invalid for prefix users. if chown portage:portage "${ED}"var/log/portage{,/elog} 2>/dev/null ; then chmod g+s,ug+rwx "${ED}"var/log/portage{,/elog} fi } [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 949 bytes --] ^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [gentoo-portage-dev] [PATCHES] setup.py install for Portage 2014-08-21 20:22 ` [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny @ 2014-08-21 20:25 ` Michał Górny 0 siblings, 0 replies; 23+ messages in thread From: Michał Górny @ 2014-08-21 20:25 UTC (permalink / raw To: gentoo-portage-dev [-- Attachment #1: Type: text/plain, Size: 637 bytes --] Dnia 2014-08-21, o godz. 22:22:30 Michał Górny <mgorny@gentoo.org> napisał(a): > # symlink compatibility /usr/lib/portage/bin > dosym "${EPYTHON}" /usr/lib/portage/bin I'm also wondering about removing this. Since the scripts won't work without setting PORTAGE_PYM_PATH correctly, people can obtain PORTAGE_BIN_PATH as well. Then, there's no need for compatibility symlink :). > # used as safe location with no files > keepdir /var/empty This one's not needed/used anymore. We decided to revert the change using it, and instead expect people to export proper PORTAGE_PYM_PATH. -- Best regards, Michał Górny [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 949 bytes --] ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [gentoo-portage-dev] [PATCHES] setup.py install for Portage 2014-08-21 20:19 [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny ` (6 preceding siblings ...) 2014-08-21 20:22 ` [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny @ 2014-08-22 22:32 ` Brian Dolbec 2014-08-24 19:42 ` Brian Dolbec 2014-09-12 7:17 ` Michał Górny 9 siblings, 0 replies; 23+ messages in thread From: Brian Dolbec @ 2014-08-22 22:32 UTC (permalink / raw To: gentoo-portage-dev On Thu, 21 Aug 2014 22:19:38 +0200 Michał Górny <mgorny@gentoo.org> wrote: > Hello, everyone. > > Here is a rebase of my recent work on bringing setup.py install for > Portage. > > > About the patches: > > (1) teaches the self-update function to deal with PORTAGE_PYM_PATH > that contains more packages than Portage itself. In particular, it > makes it copy packages of Portage only rather than everything found > in 'pym' :). > > (2) makes PORTAGE_PYM_PATH contain wherever Portage is loaded from > rather than 'pym' subdirectory of PORTAGE_BASE_PATH. This makes the > code work well when 'pym' directory is removed. > > (3) makes sure that all important test files are suffixed with .py so > that they are installed by setup.py without extra hackery. > > (4) fixes all the remaining issues with tests, allowing them to be run > on the installed copy of Portage. > > (5) replaces the Makefiles with setup.py. It's distutils with a few > extensions -- mostly making it possible to install scripts to multiple > locations, install data files recursively and override all the paths. > > (6) updates .travis.yml to run tests using setup.py, and also try > installing and re-running tests after install. > > I will follow this thread with updated ebuild. > > > About the new install layout: > > 1. /usr/lib/portage/pym no longer exists. Python files are installed > directly to site-packages for each implementation (by default). > > 2. /usr/lib/portage/bin no longer contains copies of the scripts that > are installed to /usr/bin and /usr/sbin. > > 3. All Python scripts have proper shebangs set by distutils. To fit > this with how distutils usually works and how the eclass expects it > to be done, my ebuild had done a few more changes: > > 3a. all programs from /usr/sbin are moved to /usr/bin, and all > programs in /usr/bin are wrapped. > > 3b. The Portage internal tools are moved from /usr/lib/portage/bin > to /usr/lib/portage/${EPYTHON} so that no Python mismatch is possible. > > 3c. /usr/lib/portage/bin became a symlink for external tool > compatibility. It uses the 'last' Python implementation as chosen > by distutils-r1. > > > I have tested these changes for a while and I think Portage itself can > handle them well. However, some of the external tools overrelying on > Portage itself can be broken, eix in particular. > > In particular, tools that want to play with Portage internals need to > export proper PORTAGE_BIN_PATH and PORTAGE_PYM_PATH before spawning > them. The correct values can be obtained using portageq: > > $ portageq envvar PORTAGE_BIN_PATH > /usr/lib/portage/python3.4 > $ portageq envvar PORTAGE_PYM_PATH > /usr/lib64/python3.4/site-packages > > > Please look through the patches and test at will :). Thanks. > > -- > Michał Górny > > I have not tested these yet, but they mostly look good aside for the couple questions I had (see individual replies). I will begin some testing. -- Brian Dolbec <dolsen> ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [gentoo-portage-dev] [PATCHES] setup.py install for Portage 2014-08-21 20:19 [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny ` (7 preceding siblings ...) 2014-08-22 22:32 ` Brian Dolbec @ 2014-08-24 19:42 ` Brian Dolbec 2014-08-24 21:19 ` Michał Górny 2014-09-12 7:17 ` Michał Górny 9 siblings, 1 reply; 23+ messages in thread From: Brian Dolbec @ 2014-08-24 19:42 UTC (permalink / raw To: gentoo-portage-dev On Thu, 21 Aug 2014 22:19:38 +0200 Michał Górny <mgorny@gentoo.org> wrote: > In particular, tools that want to play with Portage internals need to > export proper PORTAGE_BIN_PATH and PORTAGE_PYM_PATH before spawning > them. The correct values can be obtained using portageq: > > $ portageq envvar PORTAGE_BIN_PATH > /usr/lib/portage/python3.4 > $ portageq envvar PORTAGE_PYM_PATH > /usr/lib64/python3.4/site-packages > > > Please look through the patches and test at will :). Thanks. > > -- > Michał Górny > > There is breakage in catalyst runs. [08:26] <iamben> has anyone tried mgorny's new portage ebuild? after portage is emerged for stage1, it's missing /usr/lib/portage/bin/ altogether, for me I did a search thru the catalyst code. It is hard-coded in only 1 place: lines 35, 37 in stage1-controller.sh code snipit: clean) # Clean out man, info and doc files rm -rf usr/share/{man,doc,info}/* # Zap all .pyc and .pyo files find . -iname "*.py[co]" -exec rm -f {} \; # Cleanup all .a files except libgcc.a, *_nonshared.a and # /usr/lib/portage/bin/*.a find . -type f -iname "*.a" | grep -v 'libgcc.a' | \ grep -v 'nonshared.a' | grep -v '/usr/lib/portage/bin/' | \ grep -v 'libgcc_eh.a' | xargs rm -f ;; *) Michał can you supply me a patch to fix this in catalyst and I will apply it. I haven't started testing myself yet. -- Brian Dolbec <dolsen> ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [gentoo-portage-dev] [PATCHES] setup.py install for Portage 2014-08-24 19:42 ` Brian Dolbec @ 2014-08-24 21:19 ` Michał Górny 0 siblings, 0 replies; 23+ messages in thread From: Michał Górny @ 2014-08-24 21:19 UTC (permalink / raw To: Brian Dolbec; +Cc: gentoo-portage-dev [-- Attachment #1: Type: text/plain, Size: 1348 bytes --] Dnia 2014-08-24, o godz. 12:42:02 Brian Dolbec <dolsen@gentoo.org> napisał(a): > There is breakage in catalyst runs. > > [08:26] <iamben> has anyone tried mgorny's new portage ebuild? after > portage is emerged for stage1, it's missing /usr/lib/portage/bin/ > altogether, for me Well, that directory is no longer used, so it's fine missing as long as executables are where they're supposed to be ;). > I did a search thru the catalyst code. It is hard-coded in only 1 > place: > > lines 35, 37 in stage1-controller.sh > > code snipit: > > clean) > # Clean out man, info and doc files > rm -rf usr/share/{man,doc,info}/* > # Zap all .pyc and .pyo files > find . -iname "*.py[co]" -exec rm -f {} \; > # Cleanup all .a files except libgcc.a, *_nonshared.a and > # /usr/lib/portage/bin/*.a > find . -type f -iname "*.a" | grep -v 'libgcc.a' | \ > grep -v 'nonshared.a' | grep -v '/usr/lib/portage/bin/' | \ > grep -v 'libgcc_eh.a' | xargs rm -f > ;; > > *) > > Michał can you supply me a patch to fix this in catalyst and I will apply it. It should be enough to change that path to '/usr/lib/portage/'. Though it would generally be better to pass additional tests to find rather than playing with grep... -- Best regards, Michał Górny [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 949 bytes --] ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [gentoo-portage-dev] [PATCHES] setup.py install for Portage 2014-08-21 20:19 [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny ` (8 preceding siblings ...) 2014-08-24 19:42 ` Brian Dolbec @ 2014-09-12 7:17 ` Michał Górny 9 siblings, 0 replies; 23+ messages in thread From: Michał Górny @ 2014-09-12 7:17 UTC (permalink / raw To: gentoo-portage-dev [-- Attachment #1: Type: text/plain, Size: 261 bytes --] Dnia 2014-08-21, o godz. 22:19:38 Michał Górny <mgorny@gentoo.org> napisał(a): > Here is a rebase of my recent work on bringing setup.py install for > Portage. This one's committed in the ultimate newest version :). -- Best regards, Michał Górny [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 949 bytes --] ^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2014-09-12 7:18 UTC | newest] Thread overview: 23+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-08-21 20:19 [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny 2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 1/6] self-update: Copy only relevant packages from PORTAGE_PYM_PATH Michał Górny 2014-08-22 21:47 ` Brian Dolbec 2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 2/6] portage.const: Make PORTAGE_PYM_PATH point to the real package tree Michał Górny 2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 3/6] tests: Append .py to files that need to be installed Michał Górny 2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 4/6] tests: Fix running on installed copy of Portage Michał Górny 2014-08-22 22:19 ` Brian Dolbec 2014-08-22 22:52 ` Michał Górny 2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 5/6] Install Python modules using setup.py Michał Górny 2014-08-22 22:30 ` Brian Dolbec 2014-08-22 22:55 ` Michał Górny 2014-08-23 20:30 ` [gentoo-portage-dev] [PATCH v2] Install Portage " Michał Górny 2014-08-23 21:56 ` Brian Dolbec 2014-08-29 17:08 ` Brian Dolbec 2014-08-29 23:30 ` Brian Dolbec 2014-09-05 6:58 ` [gentoo-portage-dev] [PATCH v3] " Michał Górny 2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 6/6] travis: Use setup.py for running tests Michał Górny 2014-08-21 20:22 ` [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny 2014-08-21 20:25 ` Michał Górny 2014-08-22 22:32 ` Brian Dolbec 2014-08-24 19:42 ` Brian Dolbec 2014-08-24 21:19 ` Michał Górny 2014-09-12 7:17 ` Michał Górny
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox