[gentoo-portage-dev] [PATCHES] setup.py install for Portage

public inbox for gentoo-portage-dev@lists.gentoo.org
 help / color / mirror / Atom feed

* [gentoo-portage-dev] [PATCHES] setup.py install for Portage
@ 2014-08-21 20:19 Michał Górny
  2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 1/6] self-update: Copy only relevant packages from PORTAGE_PYM_PATH Michał Górny
                   ` (9 more replies)
  0 siblings, 10 replies; 23+ messages in thread
From: Michał Górny @ 2014-08-21 20:19 UTC (permalink / raw
  To: gentoo-portage-dev

Hello, everyone.

Here is a rebase of my recent work on bringing setup.py install for
Portage.

About the patches:

(1) teaches the self-update function to deal with PORTAGE_PYM_PATH that
contains more packages than Portage itself. In particular, it makes it
copy packages of Portage only rather than everything found in 'pym' :).

(2) makes PORTAGE_PYM_PATH contain wherever Portage is loaded from
rather than 'pym' subdirectory of PORTAGE_BASE_PATH. This makes the code
work well when 'pym' directory is removed.

(3) makes sure that all important test files are suffixed with .py so
that they are installed by setup.py without extra hackery.

(4) fixes all the remaining issues with tests, allowing them to be run
on the installed copy of Portage.

(5) replaces the Makefiles with setup.py. It's distutils with a few
extensions -- mostly making it possible to install scripts to multiple
locations, install data files recursively and override all the paths.

(6) updates .travis.yml to run tests using setup.py, and also try
installing and re-running tests after install.

I will follow this thread with updated ebuild.

About the new install layout:

1. /usr/lib/portage/pym no longer exists. Python files are installed
directly to site-packages for each implementation (by default).

2. /usr/lib/portage/bin no longer contains copies of the scripts that
are installed to /usr/bin and /usr/sbin.

3. All Python scripts have proper shebangs set by distutils. To fit this
with how distutils usually works and how the eclass expects it to be
done, my ebuild had done a few more changes:

3a. all programs from /usr/sbin are moved to /usr/bin, and all programs
in /usr/bin are wrapped.

3b. The Portage internal tools are moved from /usr/lib/portage/bin
to /usr/lib/portage/${EPYTHON} so that no Python mismatch is possible.

3c. /usr/lib/portage/bin became a symlink for external tool
compatibility. It uses the 'last' Python implementation as chosen
by distutils-r1.

I have tested these changes for a while and I think Portage itself can
handle them well. However, some of the external tools overrelying on
Portage itself can be broken, eix in particular.

In particular, tools that want to play with Portage internals need to
export proper PORTAGE_BIN_PATH and PORTAGE_PYM_PATH before spawning
them. The correct values can be obtained using portageq:

  $ portageq envvar PORTAGE_BIN_PATH
  /usr/lib/portage/python3.4
  $ portageq envvar PORTAGE_PYM_PATH
  /usr/lib64/python3.4/site-packages

Please look through the patches and test at will :). Thanks.

--
Michał Górny

^ permalink raw reply	[flat|nested] 23+ messages in thread

* [gentoo-portage-dev] [PATCH 1/6] self-update: Copy only relevant packages from PORTAGE_PYM_PATH
  2014-08-21 20:19 [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny
@ 2014-08-21 20:19 ` Michał Górny
  2014-08-22 21:47   ` Brian Dolbec
  2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 2/6] portage.const: Make PORTAGE_PYM_PATH point to the real package tree Michał Górny
                   ` (8 subsequent siblings)
  9 siblings, 1 reply; 23+ messages in thread
From: Michał Górny @ 2014-08-21 20:19 UTC (permalink / raw
  To: gentoo-portage-dev; +Cc: Michał Górny

Update the self-update code to copy only packages relevant to Portage,
to avoid copying other packages when Portage is installed alongside
them.
---
 pym/portage/const.py                   | 3 +++
 pym/portage/package/ebuild/doebuild.py | 8 ++++++--
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/pym/portage/const.py b/pym/portage/const.py
index aab6e8a..4b01ff9 100644
--- a/pym/portage/const.py
+++ b/pym/portage/const.py
@@ -270,6 +270,9 @@ SUPPORTED_BINPKG_FORMATS = ("tar", "rpm")
 # Time formats used in various places like metadata.chk.
 TIMESTAMP_FORMAT = "%a, %d %b %Y %H:%M:%S +0000"	# to be used with time.gmtime()
 
+# Top-level names of Python packages installed by Portage.
+PORTAGE_PYM_PACKAGES = ("_emerge", "portage", "repoman")
+
 # ===========================================================================
 # END OF CONSTANTS -- END OF CONSTANTS -- END OF CONSTANTS -- END OF CONSTANT
 # ===========================================================================
diff --git a/pym/portage/package/ebuild/doebuild.py b/pym/portage/package/ebuild/doebuild.py
index 01707ae..d3e3f5a 100644
--- a/pym/portage/package/ebuild/doebuild.py
+++ b/pym/portage/package/ebuild/doebuild.py
@@ -45,7 +45,7 @@ from portage import auxdbkeys, bsd_chflags, \
 	unmerge, _encodings, _os_merge, \
 	_shell_quote, _unicode_decode, _unicode_encode
 from portage.const import EBUILD_SH_ENV_FILE, EBUILD_SH_ENV_DIR, \
-	EBUILD_SH_BINARY, INVALID_ENV_FILE, MISC_SH_BINARY
+	EBUILD_SH_BINARY, INVALID_ENV_FILE, MISC_SH_BINARY, PORTAGE_PYM_PACKAGES
 from portage.data import portage_gid, portage_uid, secpass, \
 	uid, userpriv_groups
 from portage.dbapi.porttree import _parse_uri_map
@@ -2327,7 +2327,11 @@ def _prepare_self_update(settings):
 
 	orig_pym_path = portage._pym_path
 	portage._pym_path = os.path.join(base_path_tmp, "pym")
-	shutil.copytree(orig_pym_path, portage._pym_path, symlinks=True)
+	os.mkdir(portage._pym_path)
+	for pmod in PORTAGE_PYM_PACKAGES:
+		shutil.copytree(os.path.join(orig_pym_path, pmod),
+			os.path.join(portage._pym_path, pmod),
+			symlinks=True)
 
 	for dir_path in (base_path_tmp, portage._bin_path, portage._pym_path):
 		os.chmod(dir_path, 0o755)
-- 
2.0.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [gentoo-portage-dev] [PATCH 2/6] portage.const: Make PORTAGE_PYM_PATH point to the real package tree
  2014-08-21 20:19 [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny
  2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 1/6] self-update: Copy only relevant packages from PORTAGE_PYM_PATH Michał Górny
@ 2014-08-21 20:19 ` Michał Górny
  2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 3/6] tests: Append .py to files that need to be installed Michał Górny
                   ` (7 subsequent siblings)
  9 siblings, 0 replies; 23+ messages in thread
From: Michał Górny @ 2014-08-21 20:19 UTC (permalink / raw
  To: gentoo-portage-dev; +Cc: Michał Górny

Change PORTAGE_PYM_PATH to use real package tree path rather than one
based on PORTAGE_BASE_PATH.
---
 pym/portage/const.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/pym/portage/const.py b/pym/portage/const.py
index 4b01ff9..f518b47 100644
--- a/pym/portage/const.py
+++ b/pym/portage/const.py
@@ -63,7 +63,7 @@ GLOBAL_CONFIG_PATH       = "/usr/share/portage/config"
 PORTAGE_BASE_PATH        = os.path.join(os.sep, os.sep.join(os.path.realpath(
                                __file__.rstrip("co")).split(os.sep)[:-3]))
 PORTAGE_BIN_PATH         = PORTAGE_BASE_PATH + "/bin"
-PORTAGE_PYM_PATH         = PORTAGE_BASE_PATH + "/pym"
+PORTAGE_PYM_PATH         = os.path.realpath(os.path.join(__file__, '../..'))
 LOCALE_DATA_PATH         = PORTAGE_BASE_PATH + "/locale"  # FIXME: not used
 EBUILD_SH_BINARY         = PORTAGE_BIN_PATH + "/ebuild.sh"
 MISC_SH_BINARY           = PORTAGE_BIN_PATH + "/misc-functions.sh"
-- 
2.0.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [gentoo-portage-dev] [PATCH 3/6] tests: Append .py to files that need to be installed
  2014-08-21 20:19 [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny
  2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 1/6] self-update: Copy only relevant packages from PORTAGE_PYM_PATH Michał Górny
  2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 2/6] portage.const: Make PORTAGE_PYM_PATH point to the real package tree Michał Górny
@ 2014-08-21 20:19 ` Michał Górny
  2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 4/6] tests: Fix running on installed copy of Portage Michał Górny
                   ` (6 subsequent siblings)
  9 siblings, 0 replies; 23+ messages in thread
From: Michał Górny @ 2014-08-21 20:19 UTC (permalink / raw
  To: gentoo-portage-dev; +Cc: Michał Górny

Append .py to the runTests script and __test__ files so that they could
be easily picked up by setup.py.
---
 .travis.yml                               |  2 +-
 Makefile                                  |  2 +-
 pym/portage/tests/__init__.py             |  4 +-
 pym/portage/tests/bin/__test__            |  0
 pym/portage/tests/bin/__test__.py         |  0
 pym/portage/tests/dbapi/__test__          |  0
 pym/portage/tests/dbapi/__test__.py       |  0
 pym/portage/tests/dep/__test__            |  0
 pym/portage/tests/dep/__test__.py         |  0
 pym/portage/tests/ebuild/__test__         |  0
 pym/portage/tests/ebuild/__test__.py      |  0
 pym/portage/tests/emerge/__test__         |  0
 pym/portage/tests/emerge/__test__.py      |  0
 pym/portage/tests/env/__test__            |  0
 pym/portage/tests/env/__test__.py         |  0
 pym/portage/tests/env/config/__test__     |  0
 pym/portage/tests/env/config/__test__.py  |  0
 pym/portage/tests/glsa/__test__           |  0
 pym/portage/tests/glsa/__test__.py        |  0
 pym/portage/tests/lafilefixer/__test__    |  0
 pym/portage/tests/lafilefixer/__test__.py |  0
 pym/portage/tests/lazyimport/__test__     |  0
 pym/portage/tests/lazyimport/__test__.py  |  0
 pym/portage/tests/lint/__test__           |  0
 pym/portage/tests/lint/__test__.py        |  0
 pym/portage/tests/locks/__test__          |  0
 pym/portage/tests/locks/__test__.py       |  0
 pym/portage/tests/news/__test__           |  0
 pym/portage/tests/news/__test__.py        |  0
 pym/portage/tests/process/__test__        |  0
 pym/portage/tests/process/__test__.py     |  0
 pym/portage/tests/repoman/__test__        |  0
 pym/portage/tests/repoman/__test__.py     |  0
 pym/portage/tests/resolver/__test__       |  0
 pym/portage/tests/resolver/__test__.py    |  0
 pym/portage/tests/runTests                | 61 -------------------------------
 pym/portage/tests/runTests.py             | 61 +++++++++++++++++++++++++++++++
 pym/portage/tests/sets/base/__test__      |  0
 pym/portage/tests/sets/base/__test__.py   |  0
 pym/portage/tests/sets/files/__test__     |  0
 pym/portage/tests/sets/files/__test__.py  |  0
 pym/portage/tests/sets/shell/__test__     |  0
 pym/portage/tests/sets/shell/__test__.py  |  0
 pym/portage/tests/unicode/__test__        |  0
 pym/portage/tests/unicode/__test__.py     |  0
 pym/portage/tests/update/__test__         |  0
 pym/portage/tests/update/__test__.py      |  0
 pym/portage/tests/util/__test__           |  0
 pym/portage/tests/util/__test__.py        |  0
 pym/portage/tests/versions/__test__       |  0
 pym/portage/tests/versions/__test__.py    |  0
 pym/portage/tests/xpak/__test__           |  0
 pym/portage/tests/xpak/__test__.py        |  0
 runtests.sh                               |  2 +-
 54 files changed, 66 insertions(+), 66 deletions(-)
 delete mode 100644 pym/portage/tests/bin/__test__
 create mode 100644 pym/portage/tests/bin/__test__.py
 delete mode 100644 pym/portage/tests/dbapi/__test__
 create mode 100644 pym/portage/tests/dbapi/__test__.py
 delete mode 100644 pym/portage/tests/dep/__test__
 create mode 100644 pym/portage/tests/dep/__test__.py
 delete mode 100644 pym/portage/tests/ebuild/__test__
 create mode 100644 pym/portage/tests/ebuild/__test__.py
 delete mode 100644 pym/portage/tests/emerge/__test__
 create mode 100644 pym/portage/tests/emerge/__test__.py
 delete mode 100644 pym/portage/tests/env/__test__
 create mode 100644 pym/portage/tests/env/__test__.py
 delete mode 100644 pym/portage/tests/env/config/__test__
 create mode 100644 pym/portage/tests/env/config/__test__.py
 delete mode 100644 pym/portage/tests/glsa/__test__
 create mode 100644 pym/portage/tests/glsa/__test__.py
 delete mode 100644 pym/portage/tests/lafilefixer/__test__
 create mode 100644 pym/portage/tests/lafilefixer/__test__.py
 delete mode 100644 pym/portage/tests/lazyimport/__test__
 create mode 100644 pym/portage/tests/lazyimport/__test__.py
 delete mode 100644 pym/portage/tests/lint/__test__
 create mode 100644 pym/portage/tests/lint/__test__.py
 delete mode 100644 pym/portage/tests/locks/__test__
 create mode 100644 pym/portage/tests/locks/__test__.py
 delete mode 100644 pym/portage/tests/news/__test__
 create mode 100644 pym/portage/tests/news/__test__.py
 delete mode 100644 pym/portage/tests/process/__test__
 create mode 100644 pym/portage/tests/process/__test__.py
 delete mode 100644 pym/portage/tests/repoman/__test__
 create mode 100644 pym/portage/tests/repoman/__test__.py
 delete mode 100644 pym/portage/tests/resolver/__test__
 create mode 100644 pym/portage/tests/resolver/__test__.py
 delete mode 100755 pym/portage/tests/runTests
 create mode 100755 pym/portage/tests/runTests.py
 delete mode 100644 pym/portage/tests/sets/base/__test__
 create mode 100644 pym/portage/tests/sets/base/__test__.py
 delete mode 100644 pym/portage/tests/sets/files/__test__
 create mode 100644 pym/portage/tests/sets/files/__test__.py
 delete mode 100644 pym/portage/tests/sets/shell/__test__
 create mode 100644 pym/portage/tests/sets/shell/__test__.py
 delete mode 100644 pym/portage/tests/unicode/__test__
 create mode 100644 pym/portage/tests/unicode/__test__.py
 delete mode 100644 pym/portage/tests/update/__test__
 create mode 100644 pym/portage/tests/update/__test__.py
 delete mode 100644 pym/portage/tests/util/__test__
 create mode 100644 pym/portage/tests/util/__test__.py
 delete mode 100644 pym/portage/tests/versions/__test__
 create mode 100644 pym/portage/tests/versions/__test__.py
 delete mode 100644 pym/portage/tests/xpak/__test__
 create mode 100644 pym/portage/tests/xpak/__test__.py

diff --git a/.travis.yml b/.travis.yml
index 6c8d873..52ca444 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -7,4 +7,4 @@ python:
     - 3.4
     - pypy
 script:
-    - python -b -Wd pym/portage/tests/runTests
+    - python -b -Wd pym/portage/tests/runTests.py
diff --git a/Makefile b/Makefile
index 92ea195..9eb6e66 100644
--- a/Makefile
+++ b/Makefile
@@ -66,7 +66,7 @@ epydoc:
 
 test:
 	set -e; \
-	"$(srcdir)/pym/portage/tests/runTests"; \
+	"$(srcdir)/pym/portage/tests/runTests.py"; \
 
 install:
 	set -e; \
diff --git a/pym/portage/tests/__init__.py b/pym/portage/tests/__init__.py
index 84e732a..697b800 100644
--- a/pym/portage/tests/__init__.py
+++ b/pym/portage/tests/__init__.py
@@ -83,11 +83,11 @@ def getTestFromCommandLine(args, base_path):
 	return result
 
 def getTestDirs(base_path):
-	TEST_FILE = b'__test__'
+	TEST_FILE = b'__test__.py'
 	testDirs = []
 
 	# the os.walk help mentions relative paths as being quirky
-	# I was tired of adding dirs to the list, so now we add __test__
+	# I was tired of adding dirs to the list, so now we add __test__.py
 	# to each dir we want tested.
 	for root, dirs, files in os.walk(base_path):
 		try:
diff --git a/pym/portage/tests/bin/__test__ b/pym/portage/tests/bin/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/bin/__test__.py b/pym/portage/tests/bin/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/pym/portage/tests/dbapi/__test__ b/pym/portage/tests/dbapi/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/dbapi/__test__.py b/pym/portage/tests/dbapi/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/pym/portage/tests/dep/__test__ b/pym/portage/tests/dep/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/dep/__test__.py b/pym/portage/tests/dep/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/pym/portage/tests/ebuild/__test__ b/pym/portage/tests/ebuild/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/ebuild/__test__.py b/pym/portage/tests/ebuild/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/pym/portage/tests/emerge/__test__ b/pym/portage/tests/emerge/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/emerge/__test__.py b/pym/portage/tests/emerge/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/pym/portage/tests/env/__test__ b/pym/portage/tests/env/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/env/__test__.py b/pym/portage/tests/env/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/pym/portage/tests/env/config/__test__ b/pym/portage/tests/env/config/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/env/config/__test__.py b/pym/portage/tests/env/config/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/pym/portage/tests/glsa/__test__ b/pym/portage/tests/glsa/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/glsa/__test__.py b/pym/portage/tests/glsa/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/pym/portage/tests/lafilefixer/__test__ b/pym/portage/tests/lafilefixer/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/lafilefixer/__test__.py b/pym/portage/tests/lafilefixer/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/pym/portage/tests/lazyimport/__test__ b/pym/portage/tests/lazyimport/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/lazyimport/__test__.py b/pym/portage/tests/lazyimport/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/pym/portage/tests/lint/__test__ b/pym/portage/tests/lint/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/lint/__test__.py b/pym/portage/tests/lint/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/pym/portage/tests/locks/__test__ b/pym/portage/tests/locks/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/locks/__test__.py b/pym/portage/tests/locks/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/pym/portage/tests/news/__test__ b/pym/portage/tests/news/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/news/__test__.py b/pym/portage/tests/news/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/pym/portage/tests/process/__test__ b/pym/portage/tests/process/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/process/__test__.py b/pym/portage/tests/process/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/pym/portage/tests/repoman/__test__ b/pym/portage/tests/repoman/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/repoman/__test__.py b/pym/portage/tests/repoman/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/pym/portage/tests/resolver/__test__ b/pym/portage/tests/resolver/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/resolver/__test__.py b/pym/portage/tests/resolver/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/pym/portage/tests/runTests b/pym/portage/tests/runTests
deleted file mode 100755
index 9c45276..0000000
--- a/pym/portage/tests/runTests
+++ /dev/null
@@ -1,61 +0,0 @@
-#!/usr/bin/python -bWd
-# runTests.py -- Portage Unit Test Functionality
-# Copyright 2006-2014 Gentoo Foundation
-# Distributed under the terms of the GNU General Public License v2
-
-import os, sys
-import os.path as osp
-import grp
-import platform
-import pwd
-import signal
-
-def debug_signal(signum, frame):
-	import pdb
-	pdb.set_trace()
-
-if platform.python_implementation() == 'Jython':
-	debug_signum = signal.SIGUSR2 # bug #424259
-else:
-	debug_signum = signal.SIGUSR1
-
-signal.signal(debug_signum, debug_signal)
-
-# Pretend that the current user's uid/gid are the 'portage' uid/gid,
-# so things go smoothly regardless of the current user and global
-# user/group configuration.
-os.environ["PORTAGE_USERNAME"] = pwd.getpwuid(os.getuid()).pw_name
-os.environ["PORTAGE_GRPNAME"] = grp.getgrgid(os.getgid()).gr_name
-
-# Insert our parent dir so we can do shiny import "tests"
-# This line courtesy of Marienz and Pkgcore ;)
-sys.path.insert(0, osp.dirname(osp.dirname(osp.dirname(osp.realpath(__file__)))))
-
-import portage
-portage._internal_caller = True
-
-# Ensure that we don't instantiate portage.settings, so that tests should
-# work the same regardless of global configuration file state/existence.
-portage._disable_legacy_globals()
-
-if os.environ.get('NOCOLOR') in ('yes', 'true'):
-	portage.output.nocolor()
-
-import portage.tests as tests
-from portage.const import PORTAGE_BIN_PATH
-path = os.environ.get("PATH", "").split(":")
-path = [x for x in path if x]
-
-insert_bin_path = True
-try:
-	insert_bin_path = not path or \
-		not os.path.samefile(path[0], PORTAGE_BIN_PATH)
-except OSError:
-	pass
-
-if insert_bin_path:
-	path.insert(0, PORTAGE_BIN_PATH)
-	os.environ["PATH"] = ":".join(path)
-
-if __name__ == "__main__":
-	sys.exit(tests.main())
diff --git a/pym/portage/tests/runTests.py b/pym/portage/tests/runTests.py
new file mode 100755
index 0000000..9c45276
--- /dev/null
+++ b/pym/portage/tests/runTests.py
@@ -0,0 +1,61 @@
+#!/usr/bin/python -bWd
+# runTests.py -- Portage Unit Test Functionality
+# Copyright 2006-2014 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+import os, sys
+import os.path as osp
+import grp
+import platform
+import pwd
+import signal
+
+def debug_signal(signum, frame):
+	import pdb
+	pdb.set_trace()
+
+if platform.python_implementation() == 'Jython':
+	debug_signum = signal.SIGUSR2 # bug #424259
+else:
+	debug_signum = signal.SIGUSR1
+
+signal.signal(debug_signum, debug_signal)
+
+# Pretend that the current user's uid/gid are the 'portage' uid/gid,
+# so things go smoothly regardless of the current user and global
+# user/group configuration.
+os.environ["PORTAGE_USERNAME"] = pwd.getpwuid(os.getuid()).pw_name
+os.environ["PORTAGE_GRPNAME"] = grp.getgrgid(os.getgid()).gr_name
+
+# Insert our parent dir so we can do shiny import "tests"
+# This line courtesy of Marienz and Pkgcore ;)
+sys.path.insert(0, osp.dirname(osp.dirname(osp.dirname(osp.realpath(__file__)))))
+
+import portage
+portage._internal_caller = True
+
+# Ensure that we don't instantiate portage.settings, so that tests should
+# work the same regardless of global configuration file state/existence.
+portage._disable_legacy_globals()
+
+if os.environ.get('NOCOLOR') in ('yes', 'true'):
+	portage.output.nocolor()
+
+import portage.tests as tests
+from portage.const import PORTAGE_BIN_PATH
+path = os.environ.get("PATH", "").split(":")
+path = [x for x in path if x]
+
+insert_bin_path = True
+try:
+	insert_bin_path = not path or \
+		not os.path.samefile(path[0], PORTAGE_BIN_PATH)
+except OSError:
+	pass
+
+if insert_bin_path:
+	path.insert(0, PORTAGE_BIN_PATH)
+	os.environ["PATH"] = ":".join(path)
+
+if __name__ == "__main__":
+	sys.exit(tests.main())
diff --git a/pym/portage/tests/sets/base/__test__ b/pym/portage/tests/sets/base/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/sets/base/__test__.py b/pym/portage/tests/sets/base/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/pym/portage/tests/sets/files/__test__ b/pym/portage/tests/sets/files/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/sets/files/__test__.py b/pym/portage/tests/sets/files/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/pym/portage/tests/sets/shell/__test__ b/pym/portage/tests/sets/shell/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/sets/shell/__test__.py b/pym/portage/tests/sets/shell/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/pym/portage/tests/unicode/__test__ b/pym/portage/tests/unicode/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/unicode/__test__.py b/pym/portage/tests/unicode/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/pym/portage/tests/update/__test__ b/pym/portage/tests/update/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/update/__test__.py b/pym/portage/tests/update/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/pym/portage/tests/util/__test__ b/pym/portage/tests/util/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/util/__test__.py b/pym/portage/tests/util/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/pym/portage/tests/versions/__test__ b/pym/portage/tests/versions/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/versions/__test__.py b/pym/portage/tests/versions/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/pym/portage/tests/xpak/__test__ b/pym/portage/tests/xpak/__test__
deleted file mode 100644
index e69de29..0000000
diff --git a/pym/portage/tests/xpak/__test__.py b/pym/portage/tests/xpak/__test__.py
new file mode 100644
index 0000000..e69de29
diff --git a/runtests.sh b/runtests.sh
index 7999220..599727b 100755
--- a/runtests.sh
+++ b/runtests.sh
@@ -67,7 +67,7 @@ for version in ${PYTHON_VERSIONS}; do
 	fi
 	if [[ -x "${executable}" ]]; then
 		echo -e "${GOOD}Testing with Python ${version}...${NORMAL}"
-		"${executable}" -b -Wd pym/portage/tests/runTests "$@"
+		"${executable}" -b -Wd pym/portage/tests/runTests.py "$@"
 		status=$?
 		status_array[${#status_array[@]}]=${status}
 		found_versions[${#found_versions[@]}]=${version}
-- 
2.0.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [gentoo-portage-dev] [PATCH 4/6] tests: Fix running on installed copy of Portage
  2014-08-21 20:19 [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny
                   ` (2 preceding siblings ...)
  2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 3/6] tests: Append .py to files that need to be installed Michał Górny
@ 2014-08-21 20:19 ` Michał Górny
  2014-08-22 22:19   ` Brian Dolbec
  2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 5/6] Install Python modules using setup.py Michał Górny
                   ` (5 subsequent siblings)
  9 siblings, 1 reply; 23+ messages in thread
From: Michał Górny @ 2014-08-21 20:19 UTC (permalink / raw
  To: gentoo-portage-dev; +Cc: Michał Górny

---
 pym/portage/tests/__init__.py                    | 20 ++++++++++++++++++
 pym/portage/tests/dbapi/test_portdb_cache.py     |  5 ++---
 pym/portage/tests/emerge/test_emerge_slot_abi.py |  7 +++----
 pym/portage/tests/emerge/test_simple.py          | 26 ++++++++++++------------
 pym/portage/tests/lint/test_compile_modules.py   | 10 +++++----
 pym/portage/tests/lint/test_import_modules.py    |  8 ++++++--
 pym/portage/tests/repoman/test_simple.py         |  8 +++++---
 pym/portage/tests/resolver/ResolverPlayground.py |  8 +++++---
 pym/portage/tests/util/test_getconfig.py         |  5 ++---
 9 files changed, 62 insertions(+), 35 deletions(-)

diff --git a/pym/portage/tests/__init__.py b/pym/portage/tests/__init__.py
index 697b800..708dd59 100644
--- a/pym/portage/tests/__init__.py
+++ b/pym/portage/tests/__init__.py
@@ -25,8 +25,24 @@ import portage
 from portage import os
 from portage import _encodings
 from portage import _unicode_decode
+from portage.const import (EPREFIX, GLOBAL_CONFIG_PATH, PORTAGE_BASE_PATH,
+	PORTAGE_BIN_PATH)
 from portage.util._argparse import ArgumentParser
 
+
+if portage._not_installed:
+	cnf_path = os.path.join(PORTAGE_BASE_PATH, 'cnf')
+	cnf_etc_path = cnf_path
+	cnf_bindir = PORTAGE_BIN_PATH
+	cnf_sbindir = cnf_bindir
+else:
+	cnf_path = os.path.join(EPREFIX or '/', GLOBAL_CONFIG_PATH)
+	cnf_etc_path = os.path.join(EPREFIX or '/', 'etc')
+	cnf_eprefix = EPREFIX
+	cnf_bindir = os.path.join(EPREFIX or '/', 'usr', 'bin')
+	cnf_sbindir = os.path.join(EPREFIX or '/', 'usr', 'sbin')
+
+
 def main():
 	suite = unittest.TestSuite()
 	basedir = os.path.dirname(os.path.realpath(__file__))
@@ -178,6 +194,10 @@ class TestCase(unittest.TestCase):
 		unittest.TestCase.__init__(self, *pargs, **kwargs)
 		self.todo = False
 		self.portage_skip = None
+		self.cnf_path = cnf_path
+		self.cnf_etc_path = cnf_etc_path
+		self.bindir = cnf_bindir
+		self.sbindir = cnf_sbindir
 
 	def defaultTestResult(self):
 		return TextTestResult()
diff --git a/pym/portage/tests/dbapi/test_portdb_cache.py b/pym/portage/tests/dbapi/test_portdb_cache.py
index 94af96e..25e65f6 100644
--- a/pym/portage/tests/dbapi/test_portdb_cache.py
+++ b/pym/portage/tests/dbapi/test_portdb_cache.py
@@ -8,8 +8,7 @@ import textwrap
 import portage
 from portage import os
 from portage import _unicode_decode
-from portage.const import (BASH_BINARY, PORTAGE_BIN_PATH,
-	PORTAGE_PYM_PATH, USER_CONFIG_PATH)
+from portage.const import (BASH_BINARY, PORTAGE_PYM_PATH, USER_CONFIG_PATH)
 from portage.tests import TestCase
 from portage.tests.resolver.ResolverPlayground import ResolverPlayground
 from portage.util import ensure_dirs
@@ -38,7 +37,7 @@ class PortdbCacheTestCase(TestCase):
 
 		portage_python = portage._python_interpreter
 		egencache_cmd = (portage_python, "-b", "-Wd",
-			os.path.join(PORTAGE_BIN_PATH, "egencache"),
+			os.path.join(self.bindir, "egencache"),
 			"--repo", "test_repo",
 			"--repositories-configuration", settings.repositories.config_string())
 		python_cmd = (portage_python, "-b", "-Wd", "-c")
diff --git a/pym/portage/tests/emerge/test_emerge_slot_abi.py b/pym/portage/tests/emerge/test_emerge_slot_abi.py
index fd7ec0e..d1f2d92 100644
--- a/pym/portage/tests/emerge/test_emerge_slot_abi.py
+++ b/pym/portage/tests/emerge/test_emerge_slot_abi.py
@@ -7,8 +7,7 @@ import sys
 import portage
 from portage import os
 from portage import _unicode_decode
-from portage.const import (BASH_BINARY, PORTAGE_BIN_PATH,
-	PORTAGE_PYM_PATH, USER_CONFIG_PATH)
+from portage.const import (BASH_BINARY, PORTAGE_PYM_PATH, USER_CONFIG_PATH)
 from portage.process import find_binary
 from portage.tests import TestCase
 from portage.tests.resolver.ResolverPlayground import ResolverPlayground
@@ -70,9 +69,9 @@ class SlotAbiEmergeTestCase(TestCase):
 
 		portage_python = portage._python_interpreter
 		ebuild_cmd = (portage_python, "-b", "-Wd",
-			os.path.join(PORTAGE_BIN_PATH, "ebuild"))
+			os.path.join(self.bindir, "ebuild"))
 		emerge_cmd = (portage_python, "-b", "-Wd",
-			os.path.join(PORTAGE_BIN_PATH, "emerge"))
+			os.path.join(self.bindir, "emerge"))
 
 		test_ebuild = portdb.findname("dev-libs/dbus-glib-0.98")
 		self.assertFalse(test_ebuild is None)
diff --git a/pym/portage/tests/emerge/test_simple.py b/pym/portage/tests/emerge/test_simple.py
index bf0af8b..9c1b1bf 100644
--- a/pym/portage/tests/emerge/test_simple.py
+++ b/pym/portage/tests/emerge/test_simple.py
@@ -8,7 +8,7 @@ import portage
 from portage import os
 from portage import _unicode_decode
 from portage.const import (BASH_BINARY, PORTAGE_BASE_PATH,
-	PORTAGE_BIN_PATH, PORTAGE_PYM_PATH, USER_CONFIG_PATH)
+	PORTAGE_PYM_PATH, USER_CONFIG_PATH)
 from portage.process import find_binary
 from portage.tests import TestCase
 from portage.tests.resolver.ResolverPlayground import ResolverPlayground
@@ -175,29 +175,29 @@ pkg_preinst() {
 
 		portage_python = portage._python_interpreter
 		dispatch_conf_cmd = (portage_python, "-b", "-Wd",
-			os.path.join(PORTAGE_BIN_PATH, "dispatch-conf"))
+			os.path.join(self.sbindir, "dispatch-conf"))
 		ebuild_cmd = (portage_python, "-b", "-Wd",
-			os.path.join(PORTAGE_BIN_PATH, "ebuild"))
+			os.path.join(self.bindir, "ebuild"))
 		egencache_cmd = (portage_python, "-b", "-Wd",
-			os.path.join(PORTAGE_BIN_PATH, "egencache"),
+			os.path.join(self.bindir, "egencache"),
 			"--repo", "test_repo",
 			"--repositories-configuration", settings.repositories.config_string())
 		emerge_cmd = (portage_python, "-b", "-Wd",
-			os.path.join(PORTAGE_BIN_PATH, "emerge"))
+			os.path.join(self.bindir, "emerge"))
 		emaint_cmd = (portage_python, "-b", "-Wd",
-			os.path.join(PORTAGE_BIN_PATH, "emaint"))
+			os.path.join(self.sbindir, "emaint"))
 		env_update_cmd = (portage_python, "-b", "-Wd",
-			os.path.join(PORTAGE_BIN_PATH, "env-update"))
+			os.path.join(self.sbindir, "env-update"))
 		etc_update_cmd = (BASH_BINARY,
-			os.path.join(PORTAGE_BIN_PATH, "etc-update"))
+			os.path.join(self.sbindir, "etc-update"))
 		fixpackages_cmd = (portage_python, "-b", "-Wd",
-			os.path.join(PORTAGE_BIN_PATH, "fixpackages"))
+			os.path.join(self.sbindir, "fixpackages"))
 		portageq_cmd = (portage_python, "-b", "-Wd",
-			os.path.join(PORTAGE_BIN_PATH, "portageq"))
+			os.path.join(self.bindir, "portageq"))
 		quickpkg_cmd = (portage_python, "-b", "-Wd",
-			os.path.join(PORTAGE_BIN_PATH, "quickpkg"))
+			os.path.join(self.bindir, "quickpkg"))
 		regenworld_cmd = (portage_python, "-b", "-Wd",
-			os.path.join(PORTAGE_BIN_PATH, "regenworld"))
+			os.path.join(self.sbindir, "regenworld"))
 
 		rm_binary = find_binary("rm")
 		self.assertEqual(rm_binary is None, False,
@@ -368,7 +368,7 @@ pkg_preinst() {
 			for x in true_symlinks:
 				os.symlink(true_binary, os.path.join(fake_bin, x))
 			for x in etc_symlinks:
-				os.symlink(os.path.join(PORTAGE_BASE_PATH, "cnf", x),
+				os.symlink(os.path.join(self.cnf_etc_path, x),
 					os.path.join(eprefix, "etc", x))
 			with open(os.path.join(var_cache_edb, "counter"), 'wb') as f:
 				f.write(b"100")
diff --git a/pym/portage/tests/lint/test_compile_modules.py b/pym/portage/tests/lint/test_compile_modules.py
index ce7e3fb..4826cad 100644
--- a/pym/portage/tests/lint/test_compile_modules.py
+++ b/pym/portage/tests/lint/test_compile_modules.py
@@ -5,7 +5,7 @@ import errno
 import itertools
 import stat
 
-from portage.const import PORTAGE_BIN_PATH, PORTAGE_PYM_PATH
+from portage.const import PORTAGE_BIN_PATH, PORTAGE_PYM_PATH, PORTAGE_PYM_PACKAGES
 from portage.tests import TestCase
 from portage import os
 from portage import _encodings
@@ -14,9 +14,11 @@ from portage import _unicode_decode, _unicode_encode
 class CompileModulesTestCase(TestCase):
 
 	def testCompileModules(self):
-		for parent, _dirs, files in itertools.chain(
-			os.walk(PORTAGE_BIN_PATH),
-			os.walk(PORTAGE_PYM_PATH)):
+		iters = [os.walk(os.path.join(PORTAGE_PYM_PATH, x))
+			for x in PORTAGE_PYM_PACKAGES]
+		iters.append(os.walk(PORTAGE_BIN_PATH))
+
+		for parent, _dirs, files in itertools.chain(*iters):
 			parent = _unicode_decode(parent,
 				encoding=_encodings['fs'], errors='strict')
 			for x in files:
diff --git a/pym/portage/tests/lint/test_import_modules.py b/pym/portage/tests/lint/test_import_modules.py
index 34261f4..fcdcb3b 100644
--- a/pym/portage/tests/lint/test_import_modules.py
+++ b/pym/portage/tests/lint/test_import_modules.py
@@ -1,7 +1,9 @@
 # Copyright 2011-2012 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
-from portage.const import PORTAGE_PYM_PATH
+from itertools import chain
+
+from portage.const import PORTAGE_PYM_PATH, PORTAGE_PYM_PACKAGES
 from portage.tests import TestCase
 from portage import os
 from portage import _encodings
@@ -13,7 +15,9 @@ class ImportModulesTestCase(TestCase):
 		expected_failures = frozenset((
 		))
 
-		for mod in self._iter_modules(PORTAGE_PYM_PATH):
+		iters = (self._iter_modules(os.path.join(PORTAGE_PYM_PATH, x))
+			for x in PORTAGE_PYM_PACKAGES)
+		for mod in chain(*iters):
 			try:
 				__import__(mod)
 			except ImportError as e:
diff --git a/pym/portage/tests/repoman/test_simple.py b/pym/portage/tests/repoman/test_simple.py
index 69eb36d..5dbb767 100644
--- a/pym/portage/tests/repoman/test_simple.py
+++ b/pym/portage/tests/repoman/test_simple.py
@@ -9,7 +9,7 @@ import portage
 from portage import os
 from portage import shutil
 from portage import _unicode_decode
-from portage.const import PORTAGE_BASE_PATH, PORTAGE_BIN_PATH, PORTAGE_PYM_PATH
+from portage.const import PORTAGE_BASE_PATH, PORTAGE_PYM_PATH
 from portage.process import find_binary
 from portage.tests import TestCase
 from portage.tests.resolver.ResolverPlayground import ResolverPlayground
@@ -171,7 +171,7 @@ class SimpleRepomanTestCase(TestCase):
 		license_dir = os.path.join(test_repo_location, "licenses")
 
 		repoman_cmd = (portage._python_interpreter, "-b", "-Wd",
-			os.path.join(PORTAGE_BIN_PATH, "repoman"))
+			os.path.join(self.bindir, "repoman"))
 
 		git_binary = find_binary("git")
 		git_cmd = (git_binary,)
@@ -274,7 +274,9 @@ class SimpleRepomanTestCase(TestCase):
 			os.symlink(test_repo_location, test_repo_symlink)
 			# repoman checks metadata.dtd for recent CTIME, so copy the file in
 			# order to ensure that the CTIME is current
-			shutil.copyfile(metadata_dtd, os.path.join(distdir, "metadata.dtd"))
+			# NOTE: if we don't have the file around, let repoman try to fetch it.
+			if os.path.exists(metadata_dtd):
+				shutil.copyfile(metadata_dtd, os.path.join(distdir, "metadata.dtd"))
 
 			if debug:
 				# The subprocess inherits both stdout and stderr, for
diff --git a/pym/portage/tests/resolver/ResolverPlayground.py b/pym/portage/tests/resolver/ResolverPlayground.py
index 077e271..675e23f 100644
--- a/pym/portage/tests/resolver/ResolverPlayground.py
+++ b/pym/portage/tests/resolver/ResolverPlayground.py
@@ -15,6 +15,7 @@ from portage.package.ebuild.config import config
 from portage.package.ebuild.digestgen import digestgen
 from portage._sets import load_default_config
 from portage._sets.base import InternalPackageSet
+from portage.tests import cnf_path
 from portage.util import ensure_dirs, normalize_path
 from portage.versions import catsplit
 
@@ -65,6 +66,7 @@ class ResolverPlayground(object):
 			If a metadata key is missing, it gets a default value.
 		profile: settings defined by the profile.
 		"""
+
 		self.debug = debug
 		self.eprefix = normalize_path(tempfile.mkdtemp())
 		portage.const.EPREFIX = self.eprefix.rstrip(os.sep)
@@ -414,7 +416,7 @@ class ResolverPlayground(object):
 		make_globals_path = os.path.join(self.eroot,
 			GLOBAL_CONFIG_PATH.lstrip(os.sep), "make.globals")
 		ensure_dirs(os.path.dirname(make_globals_path))
-		os.symlink(os.path.join(PORTAGE_BASE_PATH, "cnf", "make.globals"),
+		os.symlink(os.path.join(cnf_path, "make.globals"),
 			make_globals_path)
 
 		#Create /usr/share/portage/config/sets/portage.conf
@@ -425,8 +427,8 @@ class ResolverPlayground(object):
 		except os.error:
 			pass
 
-		provided_sets_portage_conf = \
-			os.path.join(PORTAGE_BASE_PATH, "cnf/sets/portage.conf")
+		provided_sets_portage_conf = (
+			os.path.join(cnf_path, "sets", "portage.conf"))
 		os.symlink(provided_sets_portage_conf, os.path.join(default_sets_conf_dir, "portage.conf"))
 
 		set_config_dir = os.path.join(user_config_dir, "sets")
diff --git a/pym/portage/tests/util/test_getconfig.py b/pym/portage/tests/util/test_getconfig.py
index e5fd60f..16f415c 100644
--- a/pym/portage/tests/util/test_getconfig.py
+++ b/pym/portage/tests/util/test_getconfig.py
@@ -3,6 +3,7 @@
 
 import tempfile
 
+import portage
 from portage import os
 from portage import shutil
 from portage import _unicode_encode
@@ -26,9 +27,7 @@ class GetConfigTestCase(TestCase):
 	}
 
 	def testGetConfig(self):
-
-		make_globals_file = os.path.join(PORTAGE_BASE_PATH,
-			'cnf', 'make.globals')
+		make_globals_file = os.path.join(self.cnf_path, "make.globals")
 		d = getconfig(make_globals_file)
 		for k, v in self._cases.items():
 			self.assertEqual(d[k], v)
-- 
2.0.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [gentoo-portage-dev] [PATCH 5/6] Install Python modules using setup.py
  2014-08-21 20:19 [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny
                   ` (3 preceding siblings ...)
  2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 4/6] tests: Fix running on installed copy of Portage Michał Górny
@ 2014-08-21 20:19 ` Michał Górny
  2014-08-22 22:30   ` Brian Dolbec
  2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 6/6] travis: Use setup.py for running tests Michał Górny
                   ` (4 subsequent siblings)
  9 siblings, 1 reply; 23+ messages in thread
From: Michał Górny @ 2014-08-21 20:19 UTC (permalink / raw
  To: gentoo-portage-dev; +Cc: Michał Górny

---
 .gitignore           |   1 +
 Makefile             | 215 --------------------
 doc/Makefile         |  11 -
 pym/portage/const.py |   4 +-
 setup.py             | 557 +++++++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 560 insertions(+), 228 deletions(-)
 delete mode 100644 Makefile
 delete mode 100644 doc/Makefile
 create mode 100755 setup.py

diff --git a/.gitignore b/.gitignore
index 074bb86..c2dd534 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,4 +1,5 @@
 *.py[co]
 __pycache__/
 *.class
+/build
 /tags
diff --git a/Makefile b/Makefile
deleted file mode 100644
index 9eb6e66..0000000
--- a/Makefile
+++ /dev/null
@@ -1,215 +0,0 @@
-SHELL = /bin/sh
-PN ?= portage
-PF ?= portage
-HOMEPAGE ?= http://www.gentoo.org/proj/en/portage/index.xml
-PWD ?= $(shell pwd)
-S ?= $(PWD)
-WORKDIR ?= $(PWD)
-DESTDIR = $(PWD)/image/
-srcdir = $(S)
-prefix = /usr
-sysconfdir = /etc
-exec_prefix = $(prefix)
-bindir = $(exec_prefix)/bin
-sbindir = $(exec_prefix)/sbin
-libdir = $(exec_prefix)/lib
-datarootdir = $(prefix)/share
-datadir = $(datarootdir)
-mandir = $(datarootdir)/man
-docdir = $(datarootdir)/doc/$(PF)
-htmldir = $(docdir)/html
-portage_datadir = $(datarootdir)/$(PN)
-portage_confdir = $(portage_datadir)/config
-portage_setsdir = $(portage_confdir)/sets
-portage_base = $(libdir)/$(PN)
-EPYDOC_OPTS = -qqqqq --no-frames --show-imports
-INSMODE = 0644
-EXEMODE = 0755
-DIRMODE = 0755
-SYSCONFDIR_FILES = etc-update.conf dispatch-conf.conf
-PORTAGE_CONFDIR_FILES = make.conf.example make.globals repos.conf
-LOGROTATE_FILES = elog-save-summary
-BINDIR_FILES = ebuild egencache emerge emerge-webrsync \
-	emirrordist portageq quickpkg repoman
-SBINDIR_FILES = archive-conf dispatch-conf emaint \
-	env-update etc-update fixpackages regenworld
-DOCS = ChangeLog NEWS RELEASE-NOTES
-LINGUAS ?= $(shell cd "$(srcdir)/man" && find -mindepth 1 -type d)
-
-ifdef PYTHONPATH
-	PYTHONPATH := $(srcdir)/pym:$(PYTHONPATH)
-else
-	PYTHONPATH := $(srcdir)/pym
-endif
-
-all: docbook epydoc
-
-docbook:
-	set -e; \
-	touch "$(srcdir)/doc/fragment/date"; \
-	$(MAKE) -C "$(srcdir)/doc" xhtml xhtml-nochunks
-
-epydoc:
-	set -e; \
-	env PYTHONPATH="$(PYTHONPATH)" epydoc \
-		-o "$(WORKDIR)/epydoc" \
-		--name $(PN) \
-		--url "$(HOMEPAGE)" \
-		$(EPYDOC_OPTS) \
-		$$(cd "$(srcdir)" && find pym -name '*.py' | sed \
-		-e s:/__init__.py$$:: \
-		-e s:\.py$$:: \
-		-e s:^pym/:: \
-		-e s:/:.:g \
-		| sort); \
-	rm -f "$(WORKDIR)/epydoc/api-objects.txt"; \
-
-test:
-	set -e; \
-	"$(srcdir)/pym/portage/tests/runTests.py"; \
-
-install:
-	set -e; \
-	cd "$(srcdir)/cnf"; \
-	install -d -m$(DIRMODE) "$(DESTDIR)$(sysconfdir)"; \
-	install -m$(INSMODE) $(SYSCONFDIR_FILES) "$(DESTDIR)$(sysconfdir)"; \
-	\
-	install -d -m$(DIRMODE) "$(DESTDIR)$(portage_confdir)"; \
-	cd "$(srcdir)/cnf"; \
-	install -m$(INSMODE) $(PORTAGE_CONFDIR_FILES) \
-		"$(DESTDIR)$(portage_confdir)"; \
-	install -d -m$(DIRMODE) "$(DESTDIR)$(portage_setsdir)"; \
-	cd "$(S)/cnf/sets"; \
-	install -m$(INSMODE) *.conf "$(DESTDIR)$(portage_setsdir)"; \
-	\
-	install -d -m$(DIRMODE) "$(DESTDIR)$(sysconfdir)/logrotate.d"; \
-	cd "$(srcdir)/cnf/logrotate.d"; \
-	install -m$(INSMODE) $(LOGROTATE_FILES) \
-		"$(DESTDIR)$(sysconfdir)/logrotate.d"; \
-	\
-	for x in $$(cd "$(srcdir)" && find bin -type d) ; do \
-		cd "$(srcdir)/$$x"; \
-		install -d -m$(DIRMODE) "$(DESTDIR)$(portage_base)/$$x"; \
-		files=$$(find . -mindepth 1 -maxdepth 1 -type f ! -type l); \
-		if [ -n "$$files" ] ; then \
-			install -m$(EXEMODE) $$files \
-				"$(DESTDIR)$(portage_base)/$$x"; \
-		fi; \
-		symlinks=$$(find . -mindepth 1 -maxdepth 1 -type l); \
-		if [ -n "$$symlinks" ] ; then \
-			cp -P $$symlinks "$(DESTDIR)$(portage_base)/$$x"; \
-		fi; \
-	done; \
-	\
-	for x in $$(cd "$(srcdir)" && find pym/* -type d \
-		! -path "pym/portage/tests*") ; do \
-		cd "$(srcdir)/$$x"; \
-		files=$$(echo *.py); \
-		if [ -z "$$files" ] || [ "$$files" = "*.py" ]; then \
-			# __pycache__ directories contain no py files \
-			continue; \
-		fi; \
-		install -d -m$(DIRMODE) "$(DESTDIR)$(portage_base)/$$x"; \
-		install -m$(INSMODE) $$files "$(DESTDIR)$(portage_base)/$$x"; \
-	done; \
-	\
-	install -d -m$(DIRMODE) "$(DESTDIR)$(bindir)"; \
-	relative_path=".."; \
-	x=$(bindir) ; \
-	y="$(portage_base)"; \
-	if [ "$${x#$(prefix)}" != "$$x" ] && \
-		[ "$${y#$(prefix)}" != "$$y" ]; then \
-		x=$${x#$(prefix)}; \
-		y=$${y#$(prefix)}; \
-	fi; \
-	x=$${x%/*}; \
-	while [ -n "$$x" ] ; do \
-		relative_path=$${relative_path}/..; \
-		x=$${x%/*}; \
-	done; \
-	relative_path=$$relative_path$$y; \
-	for x in $(BINDIR_FILES) ; do \
-		ln -sf "$$relative_path/bin/$$x" \
-			"$(DESTDIR)$(bindir)/$$x"; \
-	done; \
-	\
-	install -d -m$(DIRMODE) "$(DESTDIR)$(sbindir)"; \
-	relative_path=".."; \
-	x=$(sbindir) ; \
-	y="$(portage_base)"; \
-	if [ "$${x#$(prefix)}" != "$$x" ] && \
-		[ "$${y#$(prefix)}" != "$$y" ]; then \
-		x=$${x#$(prefix)}; \
-		y=$${y#$(prefix)}; \
-	fi; \
-	x=$${x%/*}; \
-	while [ -n "$$x" ] ; do \
-		relative_path=$${relative_path}/..; \
-		x=$${x%/*}; \
-	done; \
-	relative_path=$$relative_path$$y; \
-	for x in $(SBINDIR_FILES) ; do \
-		ln -sf "$$relative_path/bin/$$x" \
-			"$(DESTDIR)$(sbindir)/$$x"; \
-	done; \
-	\
-	ln -sf "$$relative_path/bin/env-update" \
-		"$(DESTDIR)$(sbindir)/update-env"; \
-	ln -sf "$$relative_path/bin/etc-update" \
-		"$(DESTDIR)$(sbindir)/update-etc"; \
-	\
-	# We install some minimal tests for use as a preinst sanity check. \
-	# These tests must be able to run without a full source tree and \
-	# without relying on a previous portage instance being installed. \
-	install -d -m$(DIRMODE) \
-		"$(DESTDIR)$(portage_base)/pym/portage/tests"; \
-	install -m$(EXEMODE) "$(srcdir)/pym/portage/tests/runTests" \
-		"$(DESTDIR)$(portage_base)/pym/portage/tests"; \
-	cd "$(srcdir)/pym/portage/tests"; \
-	install -m$(INSMODE) *.py \
-		"$(DESTDIR)$(portage_base)/pym/portage/tests"; \
-	install -d -m$(DIRMODE) \
-		"$(DESTDIR)$(portage_base)/pym/portage/tests/lint"; \
-	cd "$(srcdir)/pym/portage/tests/lint"; \
-	install -m$(INSMODE) *.py __test__ \
-		"$(DESTDIR)$(portage_base)/pym/portage/tests/lint"; \
-	\
-	install -d -m$(DIRMODE) "$(DESTDIR)$(docdir)"; \
-	cd "$(srcdir)"; \
-	install -m $(INSMODE) $(DOCS) "$(DESTDIR)$(docdir)"; \
-	\
-	for x in "" $(LINGUAS); do \
-		for y in 1 5 ; do \
-			if [ -d "$(srcdir)/man/$$x" ]; then \
-				cd "$(srcdir)/man/$$x"; \
-				files=$$(echo *.$$y); \
-				if [ -z "$$files" ] || [ "$$files" = "*.$$y" ]; then \
-					continue; \
-				fi; \
-				install -d -m$(DIRMODE) "$(DESTDIR)$(mandir)/$$x/man$$y"; \
-				install -m$(INSMODE) *.$$y "$(DESTDIR)$(mandir)/$$x/man$$y"; \
-			fi; \
-		done; \
-	done; \
-	\
-	if [ -f "$(srcdir)/doc/portage.html" ] ; then \
-		install -d -m$(DIRMODE) "$(DESTDIR)$(htmldir)"; \
-		cd "$(srcdir)/doc"; \
-		install -m$(INSMODE) *.html "$(DESTDIR)$(htmldir)"; \
-	fi; \
-	\
-	if [ -d "$(WORKDIR)/epydoc" ] ; then \
-		install -d -m$(DIRMODE) "$(DESTDIR)$(htmldir)"; \
-		cp -pPR "$(WORKDIR)/epydoc" \
-			"$(DESTDIR)$(htmldir)/api"; \
-		cd "$(DESTDIR)$(htmldir)/api"; \
-		find . -type d | xargs chmod $(DIRMODE); \
-		find . -type f | xargs chmod $(INSMODE); \
-	fi; \
-
-clean:
-	set -e; \
-	$(MAKE) -C "$(srcdir)/doc" clean; \
-	rm -rf "$(WORKDIR)/epydoc"; \
-
-.PHONY: all clean docbook epydoc install test
diff --git a/doc/Makefile b/doc/Makefile
deleted file mode 100644
index 261a0b4..0000000
--- a/doc/Makefile
+++ /dev/null
@@ -1,11 +0,0 @@
-all: xhtml xhtml-nochunks
-
-XMLTO_FLAGS = -m custom.xsl
-man pdf txt xhtml xhtml-nochunks:
-	xmlto $@ $(XMLTO_FLAGS) portage.docbook
-
-clean distclean:
-	rm -f *.1 *.html portage.txt
-
-.PHONY: all clean distclean \
-        man pdf txt xhtml xhtml-nochunks
diff --git a/pym/portage/const.py b/pym/portage/const.py
index f518b47..acb90f9 100644
--- a/pym/portage/const.py
+++ b/pym/portage/const.py
@@ -60,8 +60,8 @@ GLOBAL_CONFIG_PATH       = "/usr/share/portage/config"
 # these variables are not used with target_root or config_root
 # NOTE: Use realpath(__file__) so that python module symlinks in site-packages
 # are followed back to the real location of the whole portage installation.
-PORTAGE_BASE_PATH        = os.path.join(os.sep, os.sep.join(os.path.realpath(
-                               __file__.rstrip("co")).split(os.sep)[:-3]))
+# NOTE: Please keep PORTAGE_BASE_PATH in one line to help substitutions.
+PORTAGE_BASE_PATH        = os.path.join(os.sep, os.sep.join(os.path.realpath(__file__.rstrip("co")).split(os.sep)[:-3]))
 PORTAGE_BIN_PATH         = PORTAGE_BASE_PATH + "/bin"
 PORTAGE_PYM_PATH         = os.path.realpath(os.path.join(__file__, '../..'))
 LOCALE_DATA_PATH         = PORTAGE_BASE_PATH + "/locale"  # FIXME: not used
diff --git a/setup.py b/setup.py
new file mode 100755
index 0000000..bd6e506
--- /dev/null
+++ b/setup.py
@@ -0,0 +1,557 @@
+#!/usr/bin/env python
+#	vim:fileencoding=utf-8
+# (c) 2010 Michał Górny <mgorny@gentoo.org>
+# Released under the terms of the 2-clause BSD license.
+
+from distutils.core import setup, Command
+from distutils.command.build_scripts import build_scripts
+from distutils.command.clean import clean
+from distutils.command.install import install
+from distutils.command.install_data import install_data
+from distutils.command.install_lib import install_lib
+from distutils.command.install_scripts import install_scripts
+from distutils.dir_util import remove_tree
+from distutils.util import change_root, subst_vars
+
+import codecs, collections, glob, os, os.path, re, subprocess, sys
+
+# TODO:
+# - smarter rebuilds of docs w/ 'install_docbook' and 'install_epydoc'.
+
+package_name = 'portage'
+package_version = '2.2.12'
+package_homepage = 'https://wiki.gentoo.org/wiki/Project:Portage'
+
+x_scripts = {
+	'bin': [
+		'bin/ebuild', 'bin/egencache', 'bin/emerge', 'bin/emerge-webrsync',
+		'bin/emirrordist', 'bin/portageq', 'bin/quickpkg', 'bin/repoman'
+	],
+	'sbin': [
+		'bin/archive-conf', 'bin/dispatch-conf', 'bin/emaint', 'bin/env-update',
+		'bin/etc-update', 'bin/fixpackages', 'bin/regenworld'
+	],
+}
+
+
+class docbook(Command):
+	""" Build docs using docbook. """
+
+	user_options = [
+		('doc-formats=', None, 'Documentation formats to build (all xmlto formats for docbook are allowed, comma-separated'),
+	]
+
+	def initialize_options(self):
+		self.doc_formats = 'xhtml,xhtml-nochunks'
+
+	def finalize_options(self):
+		self.doc_formats = self.doc_formats.replace(',', ' ').split()
+
+	def run(self):
+		with open('doc/fragment/date', 'w'):
+			pass
+
+		for f in self.doc_formats:
+			print('Building docs in %s format...' % f)
+			subprocess.check_call(['xmlto', '-o', 'doc',
+				'-m', 'doc/custom.xsl', f, 'doc/portage.docbook'])
+
+
+class epydoc(Command):
+	""" Build API docs using epydoc. """
+
+	user_options = [
+	]
+
+	def initialize_options(self):
+		self.build_lib = None
+
+	def finalize_options(self):
+		self.set_undefined_options('build_py', ('build_lib', 'build_lib'))
+
+	def run(self):
+		self.run_command('build_py')
+
+		print('Building API documentation...')
+
+		process_env = os.environ.copy()
+		pythonpath = self.build_lib
+		try:
+			pythonpath += ':' + process_env['PYTHONPATH']
+		except KeyError:
+			pass
+		process_env['PYTHONPATH'] = pythonpath
+
+		subprocess.check_call(['epydoc', '-o', 'epydoc',
+			'--name', package_name,
+			'--url', package_homepage,
+			'-qq', '--no-frames', '--show-imports',
+			'--exclude', 'portage.tests',
+			'_emerge', 'portage', 'repoman'],
+			env = process_env)
+		os.remove('epydoc/api-objects.txt')
+
+
+class install_docbook(install_data):
+	""" install_data for docbook docs """
+
+	user_options = install_data.user_options
+
+	def initialize_options(self):
+		install_data.initialize_options(self)
+		self.htmldir = None
+
+	def finalize_options(self):
+		self.set_undefined_options('install', ('htmldir', 'htmldir'))
+		install_data.finalize_options(self)
+
+	def run(self):
+		if not os.path.exists('doc/portage.html'):
+			self.run_command('docbook')
+		self.data_files = [
+			(self.htmldir, glob.glob('doc/*.html')),
+		]
+		install_data.run(self)
+
+
+class install_epydoc(install_data):
+	""" install_data for epydoc docs """
+
+	user_options = install_data.user_options
+
+	def initialize_options(self):
+		install_data.initialize_options(self)
+		self.htmldir = None
+
+	def finalize_options(self):
+		self.set_undefined_options('install', ('htmldir', 'htmldir'))
+		install_data.finalize_options(self)
+
+	def run(self):
+		if not os.path.exists('epydoc/index.html'):
+			self.run_command('epydoc')
+		self.data_files = [
+			(os.path.join(self.htmldir, 'api'), glob.glob('epydoc/*')),
+		]
+		install_data.run(self)
+
+
+class x_build_scripts_custom(build_scripts):
+	def finalize_options(self):
+		build_scripts.finalize_options(self)
+		if 'dir_name' in dir(self):
+			self.build_dir = os.path.join(self.build_dir, self.dir_name)
+			if self.dir_name in x_scripts:
+				self.scripts = x_scripts[self.dir_name]
+			else:
+				self.scripts = set(self.scripts)
+				for other_files in x_scripts.values():
+					self.scripts.difference_update(other_files)
+
+	def run(self):
+		# group scripts by subdirectory
+		split_scripts = collections.defaultdict(list)
+		for f in self.scripts:
+			dir_name = os.path.dirname(f[len('bin/'):])
+			split_scripts[dir_name].append(f)
+
+		base_dir = self.build_dir
+		base_scripts = self.scripts
+		for d, files in split_scripts.items():
+			self.build_dir = os.path.join(base_dir, d)
+			self.scripts = files
+			self.copy_scripts()
+
+		# restore previous values
+		self.build_dir = base_dir
+		self.scripts = base_scripts
+
+
+class x_build_scripts_bin(x_build_scripts_custom):
+	dir_name = 'bin'
+
+
+class x_build_scripts_sbin(x_build_scripts_custom):
+	dir_name = 'sbin'
+
+
+class x_build_scripts_portagebin(x_build_scripts_custom):
+	dir_name = 'portage'
+
+
+class x_build_scripts(build_scripts):
+	def initialize_option(self):
+		pass
+
+	def finalize_options(self):
+		pass
+
+	def run(self):
+		self.run_command('build_scripts_bin')
+		self.run_command('build_scripts_portagebin')
+		self.run_command('build_scripts_sbin')
+
+
+class x_clean(clean):
+	""" clean extended for doc & post-test cleaning """
+
+	def clean_docs(self):
+		def get_doc_outfiles():
+			for dirpath, dirnames, filenames in os.walk('doc'):
+				for f in filenames:
+					if f.endswith('.docbook') or f == 'custom.xsl':
+						pass
+					else:
+						yield os.path.join(dirpath, f)
+
+				# do not recurse
+				break
+
+
+		for f in get_doc_outfiles():
+			print('removing %s' % repr(f))
+			os.remove(f)
+
+		if os.path.isdir('epydoc'):
+			remove_tree('epydoc')
+
+	def clean_tests(self):
+		# do not remove incorrect dirs accidentally
+		top_dir = os.path.normpath(os.path.join(self.build_lib, '..'))
+		cprefix = os.path.commonprefix((self.build_base, top_dir))
+		if cprefix != self.build_base:
+			return
+
+		bin_dir = os.path.join(top_dir, 'bin')
+		if os.path.exists(bin_dir):
+			remove_tree(bin_dir)
+
+		conf_dir = os.path.join(top_dir, 'cnf')
+		if os.path.islink(conf_dir):
+			print('removing %s symlink' % repr(conf_dir))
+			os.unlink(conf_dir)
+
+		pni_file = os.path.join(top_dir, '.portage_not_installed')
+		if os.path.exists(pni_file):
+			print('removing %s' % repr(pni_file))
+			os.unlink(pni_file)
+
+	def run(self):
+		if self.all:
+			self.clean_tests()
+			self.clean_docs()
+
+		clean.run(self)
+
+
+class x_install(install):
+	""" install command with extra Portage paths """
+
+	user_options = install.user_options + [
+		# note: $prefix and $exec_prefix are reserved for Python install
+		('system-prefix=', None, "Prefix for architecture-independent data"),
+		('system-exec-prefix=', None, "Prefix for architecture-specific data"),
+
+		('bindir=', None, "Install directory for main executables"),
+		('datarootdir=', None, "Data install root directory"),
+		('docdir=', None, "Documentation install directory"),
+		('htmldir=', None, "HTML documentation install directory"),
+		('mandir=', None, "Manpage root install directory"),
+		('portage-base=', 'b', "Portage install base"),
+		('portage-bindir=', None, "Install directory for Portage internal-use executables"),
+		('portage-datadir=', None, 'Install directory for data files'),
+		('sbindir=', None, "Install directory for superuser-intended executables"),
+		('sysconfdir=', None, 'System configuration path'),
+	]
+
+	# note: the order is important for proper substitution
+	paths = [
+		('system_prefix', '/usr'),
+		('system_exec_prefix', '$system_prefix'),
+
+		('bindir', '$system_exec_prefix/bin'),
+		('sbindir', '$system_exec_prefix/sbin'),
+		('sysconfdir', '/etc'),
+
+		('datarootdir', '$system_prefix/share'),
+		('docdir', '$datarootdir/doc/%s-%s' % (package_name, package_version)),
+		('htmldir', '$docdir/html'),
+		('mandir', '$datarootdir/man'),
+
+		('portage_base', '$system_exec_prefix/lib/portage'),
+		('portage_bindir', '$portage_base/bin'),
+		('portage_datadir', '$datarootdir/portage'),
+
+		# not customized at the moment
+		('logrotatedir', '$sysconfdir/logrotate'),
+		('portage_confdir', '$portage_datadir/config'),
+		('portage_setsdir', '$portage_confdir/sets'),
+	]
+
+	def initialize_options(self):
+		install.initialize_options(self)
+
+		for key, default in self.paths:
+			setattr(self, key, default)
+		self.subst_paths = {}
+
+	def finalize_options(self):
+		install.finalize_options(self)
+
+		# substitute variables
+		new_paths = {}
+		for key, default in self.paths:
+			new_paths[key] = subst_vars(getattr(self, key), new_paths)
+			setattr(self, key, new_paths[key])
+		self.subst_paths = new_paths
+
+
+class x_install_data(install_data):
+	""" install_data with customized path support """
+
+	user_options = install_data.user_options
+
+	def initialize_options(self):
+		install_data.initialize_options(self)
+		self.paths = None
+
+	def finalize_options(self):
+		install_data.finalize_options(self)
+		self.set_undefined_options('install',
+			('subst_paths', 'paths'))
+
+		# substitute variables in data_files
+		for f in self.data_files:
+			f[0] = subst_vars(f[0], self.paths)
+
+
+class x_install_lib(install_lib):
+	""" install_lib command with Portage path substitution """
+
+	user_options = install_lib.user_options
+
+	def initialize_options(self):
+		install_lib.initialize_options(self)
+		self.portage_base = None
+		self.portage_bindir = None
+		self.portage_confdir = None
+
+	def finalize_options(self):
+		install_lib.finalize_options(self)
+		self.set_undefined_options('install',
+			('portage_base', 'portage_base'),
+			('portage_bindir', 'portage_bindir'),
+			('portage_confdir', 'portage_confdir'))
+
+	def install(self):
+		ret = install_lib.install(self)
+
+		base_re = re.compile(r'(^PORTAGE_BASE_PATH.*=) .*$', re.MULTILINE)
+		portage_bin_re = re.compile(r'(^PORTAGE_BIN_PATH.*=) .*$', re.MULTILINE)
+		global_config_re = re.compile(r'(^GLOBAL_CONFIG_PATH.*=) .*$', re.MULTILINE)
+
+		constfile = os.path.join(self.install_dir, 'portage', 'const.py')
+		print('Rewriting %s' % constfile)
+		with codecs.open(constfile, 'r', 'utf-8') as f:
+			data = f.read()
+		data = base_re.sub('\\1 %s' % repr(self.portage_base), data)
+		data = portage_bin_re.sub('\\1 %s' % repr(self.portage_bindir), data)
+		data = global_config_re.sub('\\1 %s' % repr(self.portage_confdir), data)
+		with codecs.open(constfile, 'w', 'utf-8') as f:
+			f.write(data)
+
+		return ret
+
+
+class x_install_scripts_custom(install_scripts):
+	def initialize_options(self):
+		install_scripts.initialize_options(self)
+		self.root = None
+
+	def finalize_options(self):
+		self.set_undefined_options('install',
+			('root', 'root'),
+			(self.var_name, 'install_dir'))
+		install_scripts.finalize_options(self)
+		self.build_dir = os.path.join(self.build_dir, self.dir_name)
+
+		# prepend root
+		if self.root is not None:
+			self.install_dir = change_root(self.root, self.install_dir)
+
+
+class x_install_scripts_bin(x_install_scripts_custom):
+	dir_name = 'bin'
+	var_name = 'bindir'
+
+
+class x_install_scripts_sbin(x_install_scripts_custom):
+	dir_name = 'sbin'
+	var_name = 'sbindir'
+
+
+class x_install_scripts_portagebin(x_install_scripts_custom):
+	dir_name = 'portage'
+	var_name = 'portage_bindir'
+
+
+class x_install_scripts(install_scripts):
+	def initialize_option(self):
+		pass
+
+	def finalize_options(self):
+		pass
+
+	def run(self):
+		self.run_command('install_scripts_bin')
+		self.run_command('install_scripts_portagebin')
+		self.run_command('install_scripts_sbin')
+
+
+class build_tests(x_build_scripts_custom):
+	""" Prepare build dir for running tests. """
+
+	def initialize_options(self):
+		x_build_scripts_custom.initialize_options(self)
+		self.build_base = None
+		self.build_lib = None
+
+	def finalize_options(self):
+		x_build_scripts_custom.finalize_options(self)
+		self.set_undefined_options('build',
+			('build_base', 'build_base'),
+			('build_lib', 'build_lib'))
+
+		# since we will be writing to $build_lib/.., it is important
+		# that we do not leave $build_base
+		self.top_dir = os.path.normpath(os.path.join(self.build_lib, '..'))
+		cprefix = os.path.commonprefix((self.build_base, self.top_dir))
+		if cprefix != self.build_base:
+			raise SystemError('build_lib must be a subdirectory of build_base')
+
+		self.build_dir = os.path.join(self.top_dir, 'bin')
+
+	def run(self):
+		self.run_command('build_py')
+
+		# install all scripts $build_lib/../bin
+		# (we can't do a symlink since we want shebangs corrected)
+		x_build_scripts_custom.run(self)
+
+		# symlink 'cnf' directory
+		conf_dir = os.path.join(self.top_dir, 'cnf')
+		if os.path.exists(conf_dir):
+			if not os.path.islink(conf_dir):
+				raise SystemError('%s exists and is not a symlink (collision)'
+					% repr(conf_dir))
+			os.unlink(conf_dir)
+		conf_src = os.path.relpath('cnf', self.top_dir)
+		print('Symlinking %s -> %s' % (conf_dir, conf_src))
+		os.symlink(conf_src, conf_dir)
+
+		# create $build_lib/../.portage_not_installed
+		# to enable proper paths in tests
+		with open(os.path.join(self.top_dir, '.portage_not_installed'), 'w') as f:
+			pass
+
+
+class test(Command):
+	""" run tests """
+
+	user_options = []
+
+	def initialize_options(self):
+		self.build_lib = None
+
+	def finalize_options(self):
+		self.set_undefined_options('build',
+			('build_lib', 'build_lib'))
+
+	def run(self):
+		self.run_command('build_tests')
+		subprocess.check_call([
+			sys.executable, '-bWd',
+			os.path.join(self.build_lib, 'portage/tests/runTests.py')
+		])
+
+
+def find_packages():
+	for dirpath, dirnames, filenames in os.walk('pym'):
+		if '__init__.py' in filenames:
+			yield os.path.relpath(dirpath, 'pym')
+
+
+def find_scripts():
+	for dirpath, dirnames, filenames in os.walk('bin'):
+		for f in filenames:
+			yield os.path.join(dirpath, f)
+
+
+def get_manpages():
+	linguas = os.environ.get('LINGUAS')
+	if linguas is not None:
+		linguas = linguas.split()
+
+	for dirpath, dirnames, filenames in os.walk('man'):
+		groups = collections.defaultdict(list)
+		for f in filenames:
+			fn, suffix = f.rsplit('.', 1)
+			groups[suffix].append(os.path.join(dirpath, f))
+
+		topdir = dirpath[len('man/'):]
+		if not topdir or linguas is None or topdir in linguas:
+			for g, mans in groups.items():
+				yield [os.path.join('$mandir', topdir, 'man%s' % g), mans]
+
+setup(
+		name = package_name,
+		version = package_version,
+		author = 'Gentoo Portage Development Team',
+		author_email = 'dev-portage@gentoo.org',
+		url = package_homepage,
+
+		package_dir = {'': 'pym'},
+		packages = list(find_packages()),
+		# something to cheat build & install commands
+		scripts = list(find_scripts()),
+
+		data_files = list(get_manpages()) + [
+			['$sysconfdir', ['cnf/etc-update.conf', 'cnf/dispatch-conf.conf']],
+			['$logrotatedir', ['cnf/logrotate.d/elog-save-summary']],
+			['$portage_confdir', [
+				'cnf/make.conf.example', 'cnf/make.globals', 'cnf/repos.conf']],
+			['$portage_setsdir', ['cnf/sets/portage.conf']],
+			['$docdir', ['ChangeLog', 'NEWS', 'RELEASE-NOTES']],
+		],
+
+		cmdclass = {
+			'build_scripts': x_build_scripts,
+			'build_scripts_bin': x_build_scripts_bin,
+			'build_scripts_portagebin': x_build_scripts_portagebin,
+			'build_scripts_sbin': x_build_scripts_sbin,
+			'build_tests': build_tests,
+			'clean': x_clean,
+			'docbook': docbook,
+			'epydoc': epydoc,
+			'install': x_install,
+			'install_data': x_install_data,
+			'install_docbook': install_docbook,
+			'install_epydoc': install_epydoc,
+			'install_lib': x_install_lib,
+			'install_scripts': x_install_scripts,
+			'install_scripts_bin': x_install_scripts_bin,
+			'install_scripts_portagebin': x_install_scripts_portagebin,
+			'install_scripts_sbin': x_install_scripts_sbin,
+			'test': test,
+		},
+
+		classifiers = [
+			'Development Status :: 5 - Production/Stable',
+			'Environment :: Console',
+			'Intended Audience :: System Administrators',
+			'License :: OSI Approved :: GNU General Public License v2 (GPLv2)',
+			'Operating System :: POSIX',
+			'Programming Language :: Python',
+			'Topic :: System :: Installation/Setup'
+		]
+)
-- 
2.0.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread

* [gentoo-portage-dev] [PATCH 6/6] travis: Use setup.py for running tests
  2014-08-21 20:19 [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny
                   ` (4 preceding siblings ...)
  2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 5/6] Install Python modules using setup.py Michał Górny
@ 2014-08-21 20:19 ` Michał Górny
  2014-08-21 20:22 ` [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny
                   ` (3 subsequent siblings)
  9 siblings, 0 replies; 23+ messages in thread
From: Michał Górny @ 2014-08-21 20:19 UTC (permalink / raw
  To: gentoo-portage-dev; +Cc: Michał Górny

---
 .travis.yml | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index 52ca444..563f9ca 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -7,4 +7,8 @@ python:
     - 3.4
     - pypy
 script:
-    - python -b -Wd pym/portage/tests/runTests.py
+    - ./setup.py test
+    - touch ChangeLog # seriously stupid
+    - ./setup.py install --root=/tmp/install-root
+    - sudo rsync -a /tmp/install-root/. /
+    - python -b -Wd -m portage.tests.runTests
-- 
2.0.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread

* Re: [gentoo-portage-dev] [PATCHES] setup.py install for Portage
  2014-08-21 20:19 [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny
                   ` (5 preceding siblings ...)
  2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 6/6] travis: Use setup.py for running tests Michał Górny
@ 2014-08-21 20:22 ` Michał Górny
  2014-08-21 20:25   ` Michał Górny
  2014-08-22 22:32 ` Brian Dolbec
                   ` (2 subsequent siblings)
  9 siblings, 1 reply; 23+ messages in thread
From: Michał Górny @ 2014-08-21 20:22 UTC (permalink / raw
  To: gentoo-portage-dev


[-- Attachment #1.1: Type: text/plain, Size: 893 bytes --]

Dnia 2014-08-21, o godz. 22:19:38
Michał Górny <mgorny@gentoo.org> napisał(a):

> I will follow this thread with updated ebuild.

And here's the ebuild. You can also find it in ::mgorny.

One minor note: after applying the distutils-r1 patch I submitted to
gentoo-dev@ today, the following diff can be applied to improve
the ebuild:

diff --git a/sys-apps/portage/portage-9999.ebuild b/sys-apps/portage/portage-9999.ebuild
index 3fee385..f11c62a 100644
--- a/sys-apps/portage/portage-9999.ebuild
+++ b/sys-apps/portage/portage-9999.ebuild
@@ -187,7 +187,7 @@ python_install_all() {
        use epydoc && targets+=( install_epydoc )
 
        # install docs
-       python_install "${targets[@]}"
+       esetup.py "${targets[@]}"
 
        # symlink compatibility /usr/lib/portage/bin
        dosym "${EPYTHON}" /usr/lib/portage/bin

-- 
Best regards,
Michał Górny

[-- Attachment #1.2: portage-9999.ebuild --]
[-- Type: text/plain, Size: 7530 bytes --]

# Copyright 1999-2014 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo-x86/sys-apps/portage/portage-9999.ebuild,v 1.99 2014/08/05 14:19:02 dolsen Exp $

EAPI=5

PYTHON_COMPAT=(
	pypy
	python3_2 python3_3 python3_4
	python2_7
)
PYTHON_REQ_USE='ssl(+),bzip2(+)'

inherit distutils-r1 git-r3 multilib

DESCRIPTION="Portage is the package management and distribution system for Gentoo"
HOMEPAGE="http://www.gentoo.org/proj/en/portage/index.xml"

LICENSE="GPL-2"
KEYWORDS=""
SLOT="0"
IUSE="build doc epydoc +ipc linguas_ru selinux xattr"

DEPEND="${python_dep}
	>=sys-apps/sed-4.0.5 sys-devel/patch
	doc? ( app-text/xmlto ~app-text/docbook-xml-dtd-4.4 )
	epydoc? ( >=dev-python/epydoc-2.0[$(python_gen_usedep 'python2*')] )"
# Require sandbox-2.2 for bug #288863.
# For xattr, we can spawn getfattr and setfattr from sys-apps/attr, but that's
# quite slow, so it's not considered in the dependencies as an alternative to
# to python-3.3 / pyxattr. Also, xattr support is only tested with Linux, so
# for now, don't pull in xattr deps for other kernels.
# For whirlpool hash, require python[ssl] or python-mhash (bug #425046).
# For compgen, require bash[readline] (bug #445576).
RDEPEND="${python_dep}
	!build? ( >=sys-apps/sed-4.0.5
		|| ( >=app-shells/bash-4.2_p37[readline] ( <app-shells/bash-4.2_p37 >=app-shells/bash-3.2_p17 ) )
		>=app-admin/eselect-1.2
	)
	elibc_FreeBSD? ( sys-freebsd/freebsd-bin )
	elibc_glibc? ( >=sys-apps/sandbox-2.2 )
	elibc_uclibc? ( >=sys-apps/sandbox-2.2 )
	>=app-misc/pax-utils-0.1.17
	selinux? ( >=sys-libs/libselinux-2.0.94[python,${PYTHON_USEDEP}] )
	xattr? ( kernel_linux? (
		>=sys-apps/install-xattr-0.3
		$(python_gen_cond_dep 'dev-python/pyxattr[${PYTHON_USEDEP}]' \
			python{2_7,3_2} pypy)
	) )
	!<app-shells/bash-3.2_p17
	!<app-admin/logrotate-3.8.0"
PDEPEND="
	!build? (
		>=net-misc/rsync-2.6.4
		userland_GNU? ( >=sys-apps/coreutils-6.4 )
	)"
# coreutils-6.4 rdep is for date format in emerge-webrsync #164532
# NOTE: FEATURES=installsources requires debugedit and rsync

REQUIRED_USE="epydoc? ( $(python_gen_useflags 'python2*') )"

SRC_ARCHIVES="http://dev.gentoo.org/~dolsen/releases/portage"

prefix_src_archives() {
	local x y
	for x in ${@}; do
		for y in ${SRC_ARCHIVES}; do
			echo ${y}/${x}
		done
	done
}

EGIT_REPO_URI="https://github.com/mgorny/portage.git"
EGIT_BRANCH=new-install
EGIT_MIN_CLONE_TYPE=single

python_prepare_all() {
	distutils-r1_python_prepare_all

	einfo "Producing ChangeLog from Git history..."
	git log ebcf8975b37a8aae9735eb491a9b4cb63549bd5d^.. \
		> "${S}"/ChangeLog || die

	local _version=$(git describe --tags | sed -e 's|-\([0-9]\+\)-.\+$|_p\1|')
	_version=${_version:1}
	einfo "Setting portage.VERSION to ${_version} ..."
	sed -e "s/^VERSION =.*/VERSION = '${_version}'/" -i pym/portage/__init__.py || \
		die "Failed to patch portage.VERSION"
	sed -e "1s/VERSION/${_version}/" -i doc/fragment/version || \
		die "Failed to patch VERSION in doc/fragment/version"
	sed -e "1s/VERSION/${_version}/" -i $(find man -type f) || \
		die "Failed to patch VERSION in man page headers"

	if ! use ipc ; then
		einfo "Disabling ipc..."
		sed -e "s:_enable_ipc_daemon = True:_enable_ipc_daemon = False:" \
			-i pym/_emerge/AbstractEbuildProcess.py || \
			die "failed to patch AbstractEbuildProcess.py"
	fi

	if use xattr && use kernel_linux ; then
		einfo "Adding FEATURES=xattr to make.globals ..."
		echo -e '\nFEATURES="${FEATURES} xattr"' >> cnf/make.globals \
			|| die "failed to append to make.globals"
	fi

	if [[ -n ${EPREFIX} ]] ; then
		einfo "Setting portage.const.EPREFIX ..."
		sed -e "s|^\(SANDBOX_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/bin/sandbox\"\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(FAKEROOT_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/bin/fakeroot\"\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(BASH_BINARY[[:space:]]*=[[:space:]]*\"\)\(/bin/bash\"\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(MOVE_BINARY[[:space:]]*=[[:space:]]*\"\)\(/bin/mv\"\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(PRELINK_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/sbin/prelink\"\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(EPREFIX[[:space:]]*=[[:space:]]*\"\).*|\\1${EPREFIX}\"|" \
			-i pym/portage/const.py || \
			die "Failed to patch portage.const.EPREFIX"

		einfo "Prefixing shebangs ..."
		while read -r -d $'\0' ; do
			local shebang=$(head -n1 "$REPLY")
			if [[ ${shebang} == "#!"* && ! ${shebang} == "#!${EPREFIX}/"* ]] ; then
				sed -i -e "1s:.*:#!${EPREFIX}${shebang:2}:" "$REPLY" || \
					die "sed failed"
			fi
		done < <(find . -type f -print0)

		einfo "Adjusting make.globals ..."
		sed -e "s|\(/usr/portage\)|${EPREFIX}\\1|" \
			-e "s|^\(PORTAGE_TMPDIR=\"\)\(/var/tmp\"\)|\\1${EPREFIX}\\2|" \
			-i cnf/make.globals || die "sed failed"

		einfo "Adjusting repos.conf ..."
		sed -e "s|^\(main-repo = \).*|\\1gentoo_prefix|" \
			-e "s|^\\[gentoo\\]|[gentoo_prefix]|" \
			-e "s|^\(location = \)\(/usr/portage\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(sync-uri = \).*|\\1rsync://prefix.gentooexperimental.org/gentoo-portage-prefix|" \
			-i cnf/repos.conf || die "sed failed"

		einfo "Adding FEATURES=force-prefix to make.globals ..."
		echo -e '\nFEATURES="${FEATURES} force-prefix"' >> cnf/make.globals \
			|| die "failed to append to make.globals"
	fi

	cd "${S}/cnf" || die
	if [ -f "make.conf.example.${ARCH}".diff ]; then
		patch make.conf.example "make.conf.example.${ARCH}".diff || \
			die "Failed to patch make.conf.example"
	else
		eerror ""
		eerror "Portage does not have an arch-specific configuration for this arch."
		eerror "Please notify the arch maintainer about this issue. Using generic."
		eerror ""
	fi
}

python_compile_all() {
	local targets=()
	use doc && targets+=( docbook )
	use epydoc && targets+=( epydoc )

	if [[ ${targets[@]} ]]; then
		esetup.py "${targets[@]}"
	fi
}

python_test() {
	esetup.py test
}

python_install() {
	distutils-r1_python_install \
		--system-prefix="${EPREFIX}/usr" \
		--bindir="$(python_get_scriptdir)" \
		--docdir="${EPREFIX}/usr/share/doc/${PF}" \
		--htmldir="${EPREFIX}/usr/share/doc/${PF}/html" \
		--portage-bindir="${EPREFIX}/usr/lib/portage/${EPYTHON}" \
		--sbindir="$(python_get_scriptdir)" \
		--sysconfdir="${EPREFIX}/etc" \
		"${@}"
}

python_install_all() {
	distutils-r1_python_install_all

	local targets=()
	use doc && targets+=( install_docbook )
	use epydoc && targets+=( install_epydoc )

	# install docs
	esetup.py "${targets[@]}"

	# symlink compatibility /usr/lib/portage/bin
	dosym "${EPYTHON}" /usr/lib/portage/bin

	# used as safe location with no files
	keepdir /var/empty
}

pkg_preinst() {
	if [[ $ROOT == / ]] ; then
		# Run some minimal tests as a sanity check.
		local test_runner=$(find "${ED}" -name runTests)
		if [[ -n $test_runner && -x $test_runner ]] ; then
			einfo "Running preinst sanity tests..."
			"$test_runner" || die "preinst sanity tests failed"
		fi
	fi

	# elog dir must exist to avoid logrotate error for bug #415911.
	# This code runs in preinst in order to bypass the mapping of
	# portage:portage to root:root which happens after src_install.
	keepdir /var/log/portage/elog
	# This is allowed to fail if the user/group are invalid for prefix users.
	if chown portage:portage "${ED}"var/log/portage{,/elog} 2>/dev/null ; then
		chmod g+s,ug+rwx "${ED}"var/log/portage{,/elog}
	fi
}

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

^ permalink raw reply related	[flat|nested] 23+ messages in thread

* Re: [gentoo-portage-dev] [PATCHES] setup.py install for Portage
  2014-08-21 20:22 ` [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny
@ 2014-08-21 20:25   ` Michał Górny
  0 siblings, 0 replies; 23+ messages in thread
From: Michał Górny @ 2014-08-21 20:25 UTC (permalink / raw
  To: gentoo-portage-dev

[-- Attachment #1: Type: text/plain, Size: 637 bytes --]

Dnia 2014-08-21, o godz. 22:22:30
Michał Górny <mgorny@gentoo.org> napisał(a):

> 	# symlink compatibility /usr/lib/portage/bin
> 	dosym "${EPYTHON}" /usr/lib/portage/bin

I'm also wondering about removing this. Since the scripts won't work
without setting PORTAGE_PYM_PATH correctly, people can obtain
PORTAGE_BIN_PATH as well. Then, there's no need for compatibility
symlink :).

> 	# used as safe location with no files
> 	keepdir /var/empty

This one's not needed/used anymore. We decided to revert the change
using it, and instead expect people to export proper PORTAGE_PYM_PATH.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [gentoo-portage-dev] [PATCH 1/6] self-update: Copy only relevant packages from PORTAGE_PYM_PATH
  2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 1/6] self-update: Copy only relevant packages from PORTAGE_PYM_PATH Michał Górny
@ 2014-08-22 21:47   ` Brian Dolbec
  0 siblings, 0 replies; 23+ messages in thread
From: Brian Dolbec @ 2014-08-22 21:47 UTC (permalink / raw
  To: gentoo-portage-dev

On Thu, 21 Aug 2014 22:19:39 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> Update the self-update code to copy only packages relevant to Portage,
> to avoid copying other packages when Portage is installed alongside
> them.
> ---
>  pym/portage/const.py                   | 3 +++
>  pym/portage/package/ebuild/doebuild.py | 8 ++++++--
>  2 files changed, 9 insertions(+), 2 deletions(-)
> 

Seems fine to me...


-- 
Brian Dolbec <dolsen>



^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [gentoo-portage-dev] [PATCH 4/6] tests: Fix running on installed copy of Portage
  2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 4/6] tests: Fix running on installed copy of Portage Michał Górny
@ 2014-08-22 22:19   ` Brian Dolbec
  2014-08-22 22:52     ` Michał Górny
  0 siblings, 1 reply; 23+ messages in thread
From: Brian Dolbec @ 2014-08-22 22:19 UTC (permalink / raw
  To: gentoo-portage-dev

On Thu, 21 Aug 2014 22:19:42 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> ---
>  pym/portage/tests/__init__.py                    | 20
> ++++++++++++++++++ pym/portage/tests/dbapi/test_portdb_cache.py
> |  5 ++--- pym/portage/tests/emerge/test_emerge_slot_abi.py |  7
> +++---- pym/portage/tests/emerge/test_simple.py          | 26
> ++++++++++++------------
> pym/portage/tests/lint/test_compile_modules.py   | 10 +++++----
> pym/portage/tests/lint/test_import_modules.py    |  8 ++++++--
> pym/portage/tests/repoman/test_simple.py         |  8 +++++---
> pym/portage/tests/resolver/ResolverPlayground.py |  8 +++++---
> pym/portage/tests/util/test_getconfig.py         |  5 ++--- 9 files
> changed, 62 insertions(+), 35 deletions(-)
> 
 
diff --git a/pym/portage/tests/util/test_getconfig.py
> b/pym/portage/tests/util/test_getconfig.py index e5fd60f..16f415c
> 100644 --- a/pym/portage/tests/util/test_getconfig.py
> +++ b/pym/portage/tests/util/test_getconfig.py
> @@ -3,6 +3,7 @@
>  
>  import tempfile
>  
> +import portage
>  from portage import os
>  from portage import shutil
>  from portage import _unicode_encode
> @@ -26,9 +27,7 @@ class GetConfigTestCase(TestCase):
>  	}
>  
>  	def testGetConfig(self):
> -
> -		make_globals_file = os.path.join(PORTAGE_BASE_PATH,
> -			'cnf', 'make.globals')
> +		make_globals_file = os.path.join(self.cnf_path,
> "make.globals") d = getconfig(make_globals_file)
>  		for k, v in self._cases.items():
>  			self.assertEqual(d[k], v)

Why was "import portage" added but not used anywhere in additional code?


-- 
Brian Dolbec <dolsen>



^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [gentoo-portage-dev] [PATCH 5/6] Install Python modules using setup.py
  2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 5/6] Install Python modules using setup.py Michał Górny
@ 2014-08-22 22:30   ` Brian Dolbec
  2014-08-22 22:55     ` Michał Górny
  2014-08-23 20:30     ` [gentoo-portage-dev] [PATCH v2] Install Portage " Michał Górny
  0 siblings, 2 replies; 23+ messages in thread
From: Brian Dolbec @ 2014-08-22 22:30 UTC (permalink / raw
  To: gentoo-portage-dev

On Thu, 21 Aug 2014 22:19:43 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> ---
>  .gitignore           |   1 +
>  Makefile             | 215 --------------------
>  doc/Makefile         |  11 -
>  pym/portage/const.py |   4 +-
>  setup.py             | 557
> +++++++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed,
> 560 insertions(+), 228 deletions(-) delete mode 100644 Makefile
>  delete mode 100644 doc/Makefile
>  create mode 100755 setup.py




> diff --git a/setup.py b/setup.py new file mode 100755
> index 0000000..bd6e506 --- /dev/null
> +++ b/setup.py
> @@ -0,0 +1,557 @@
> +#!/usr/bin/env python
> +#	vim:fileencoding=utf-8
> +# (c) 2010 Michał Górny <mgorny@gentoo.org>
> +# Released under the terms of the 2-clause BSD license.
> +
> +from distutils.core import setup, Command
> +from distutils.command.build_scripts import build_scripts
> +from distutils.command.clean import clean
> +from distutils.command.install import install
> +from distutils.command.install_data import install_data
> +from distutils.command.install_lib import install_lib
> +from distutils.command.install_scripts import install_scripts
> +from distutils.dir_util import remove_tree
> +from distutils.util import change_root, subst_vars
> +
> +import codecs, collections, glob, os, os.path, re, subprocess, sys
> +
> +# TODO:
> +# - smarter rebuilds of docs w/ 'install_docbook' and
> 'install_epydoc'. +
> +package_name = 'portage'
> +package_version = '2.2.12'

Why hard code the version?  can you not import it?  or did you add this
to the sed list in mkrelease.sh?

-- 
Brian Dolbec <dolsen>



^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [gentoo-portage-dev] [PATCHES] setup.py install for Portage
  2014-08-21 20:19 [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny
                   ` (6 preceding siblings ...)
  2014-08-21 20:22 ` [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny
@ 2014-08-22 22:32 ` Brian Dolbec
  2014-08-24 19:42 ` Brian Dolbec
  2014-09-12  7:17 ` Michał Górny
  9 siblings, 0 replies; 23+ messages in thread
From: Brian Dolbec @ 2014-08-22 22:32 UTC (permalink / raw
  To: gentoo-portage-dev

On Thu, 21 Aug 2014 22:19:38 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> Hello, everyone.
> 
> Here is a rebase of my recent work on bringing setup.py install for
> Portage.
> 
> 
> About the patches:
> 
> (1) teaches the self-update function to deal with PORTAGE_PYM_PATH
> that contains more packages than Portage itself. In particular, it
> makes it copy packages of Portage only rather than everything found
> in 'pym' :).
> 
> (2) makes PORTAGE_PYM_PATH contain wherever Portage is loaded from
> rather than 'pym' subdirectory of PORTAGE_BASE_PATH. This makes the
> code work well when 'pym' directory is removed.
> 
> (3) makes sure that all important test files are suffixed with .py so
> that they are installed by setup.py without extra hackery.
> 
> (4) fixes all the remaining issues with tests, allowing them to be run
> on the installed copy of Portage.
> 
> (5) replaces the Makefiles with setup.py. It's distutils with a few
> extensions -- mostly making it possible to install scripts to multiple
> locations, install data files recursively and override all the paths.
> 
> (6) updates .travis.yml to run tests using setup.py, and also try
> installing and re-running tests after install.
> 
> I will follow this thread with updated ebuild.
> 
> 
> About the new install layout:
> 
> 1. /usr/lib/portage/pym no longer exists. Python files are installed
> directly to site-packages for each implementation (by default).
> 
> 2. /usr/lib/portage/bin no longer contains copies of the scripts that
> are installed to /usr/bin and /usr/sbin.
> 
> 3. All Python scripts have proper shebangs set by distutils. To fit
> this with how distutils usually works and how the eclass expects it
> to be done, my ebuild had done a few more changes:
> 
> 3a. all programs from /usr/sbin are moved to /usr/bin, and all
> programs in /usr/bin are wrapped.
> 
> 3b. The Portage internal tools are moved from /usr/lib/portage/bin
> to /usr/lib/portage/${EPYTHON} so that no Python mismatch is possible.
> 
> 3c. /usr/lib/portage/bin became a symlink for external tool
> compatibility. It uses the 'last' Python implementation as chosen
> by distutils-r1.
> 
> 
> I have tested these changes for a while and I think Portage itself can
> handle them well. However, some of the external tools overrelying on
> Portage itself can be broken, eix in particular.
> 
> In particular, tools that want to play with Portage internals need to
> export proper PORTAGE_BIN_PATH and PORTAGE_PYM_PATH before spawning
> them. The correct values can be obtained using portageq:
> 
>   $ portageq envvar PORTAGE_BIN_PATH
>   /usr/lib/portage/python3.4
>   $ portageq envvar PORTAGE_PYM_PATH
>   /usr/lib64/python3.4/site-packages
> 
> 
> Please look through the patches and test at will :). Thanks.
> 
> --
> Michał Górny
> 
> 

I have not tested these yet, but they mostly look good aside for the
couple questions I had (see individual replies).

I will begin some testing.

-- 
Brian Dolbec <dolsen>



^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [gentoo-portage-dev] [PATCH 4/6] tests: Fix running on installed copy of Portage
  2014-08-22 22:19   ` Brian Dolbec
@ 2014-08-22 22:52     ` Michał Górny
  0 siblings, 0 replies; 23+ messages in thread
From: Michał Górny @ 2014-08-22 22:52 UTC (permalink / raw
  To: Brian Dolbec; +Cc: gentoo-portage-dev

[-- Attachment #1: Type: text/plain, Size: 1884 bytes --]

Dnia 2014-08-22, o godz. 15:19:12
Brian Dolbec <dolsen@gentoo.org> napisał(a):

> On Thu, 21 Aug 2014 22:19:42 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
> 
> > ---
> >  pym/portage/tests/__init__.py                    | 20
> > ++++++++++++++++++ pym/portage/tests/dbapi/test_portdb_cache.py
> > |  5 ++--- pym/portage/tests/emerge/test_emerge_slot_abi.py |  7
> > +++---- pym/portage/tests/emerge/test_simple.py          | 26
> > ++++++++++++------------
> > pym/portage/tests/lint/test_compile_modules.py   | 10 +++++----
> > pym/portage/tests/lint/test_import_modules.py    |  8 ++++++--
> > pym/portage/tests/repoman/test_simple.py         |  8 +++++---
> > pym/portage/tests/resolver/ResolverPlayground.py |  8 +++++---
> > pym/portage/tests/util/test_getconfig.py         |  5 ++--- 9 files
> > changed, 62 insertions(+), 35 deletions(-)
> > 
>  
> diff --git a/pym/portage/tests/util/test_getconfig.py
> > b/pym/portage/tests/util/test_getconfig.py index e5fd60f..16f415c
> > 100644 --- a/pym/portage/tests/util/test_getconfig.py
> > +++ b/pym/portage/tests/util/test_getconfig.py
> > @@ -3,6 +3,7 @@
> >  
> >  import tempfile
> >  
> > +import portage
> >  from portage import os
> >  from portage import shutil
> >  from portage import _unicode_encode
> > @@ -26,9 +27,7 @@ class GetConfigTestCase(TestCase):
> >  	}
> >  
> >  	def testGetConfig(self):
> > -
> > -		make_globals_file = os.path.join(PORTAGE_BASE_PATH,
> > -			'cnf', 'make.globals')
> > +		make_globals_file = os.path.join(self.cnf_path,
> > "make.globals") d = getconfig(make_globals_file)
> >  		for k, v in self._cases.items():
> >  			self.assertEqual(d[k], v)
> 
> Why was "import portage" added but not used anywhere in additional code?

Good catch. It was used in the previous version of the patch. Fixed now.


-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [gentoo-portage-dev] [PATCH 5/6] Install Python modules using setup.py
  2014-08-22 22:30   ` Brian Dolbec
@ 2014-08-22 22:55     ` Michał Górny
  2014-08-23 20:30     ` [gentoo-portage-dev] [PATCH v2] Install Portage " Michał Górny
  1 sibling, 0 replies; 23+ messages in thread
From: Michał Górny @ 2014-08-22 22:55 UTC (permalink / raw
  To: Brian Dolbec; +Cc: gentoo-portage-dev

[-- Attachment #1: Type: text/plain, Size: 2022 bytes --]

Dnia 2014-08-22, o godz. 15:30:15
Brian Dolbec <dolsen@gentoo.org> napisał(a):

> On Thu, 21 Aug 2014 22:19:43 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
> 
> > ---
> >  .gitignore           |   1 +
> >  Makefile             | 215 --------------------
> >  doc/Makefile         |  11 -
> >  pym/portage/const.py |   4 +-
> >  setup.py             | 557
> > +++++++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed,
> > 560 insertions(+), 228 deletions(-) delete mode 100644 Makefile
> >  delete mode 100644 doc/Makefile
> >  create mode 100755 setup.py
> 
> 
> 
> 
> > diff --git a/setup.py b/setup.py new file mode 100755
> > index 0000000..bd6e506 --- /dev/null
> > +++ b/setup.py
> > @@ -0,0 +1,557 @@
> > +#!/usr/bin/env python
> > +#	vim:fileencoding=utf-8
> > +# (c) 2010 Michał Górny <mgorny@gentoo.org>
> > +# Released under the terms of the 2-clause BSD license.
> > +
> > +from distutils.core import setup, Command
> > +from distutils.command.build_scripts import build_scripts
> > +from distutils.command.clean import clean
> > +from distutils.command.install import install
> > +from distutils.command.install_data import install_data
> > +from distutils.command.install_lib import install_lib
> > +from distutils.command.install_scripts import install_scripts
> > +from distutils.dir_util import remove_tree
> > +from distutils.util import change_root, subst_vars
> > +
> > +import codecs, collections, glob, os, os.path, re, subprocess, sys
> > +
> > +# TODO:
> > +# - smarter rebuilds of docs w/ 'install_docbook' and
> > 'install_epydoc'. +
> > +package_name = 'portage'
> > +package_version = '2.2.12'
> 
> Why hard code the version?  can you not import it?  or did you add this
> to the sed list in mkrelease.sh?

Left it in my todo and forgot about it. Which also makes me think that
we may want to try to get 'setup.py sdist' working as well, and rethink
how much we need mkrelease.sh afterwards.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

^ permalink raw reply	[flat|nested] 23+ messages in thread

* [gentoo-portage-dev] [PATCH v2] Install Portage using setup.py
  2014-08-22 22:30   ` Brian Dolbec
  2014-08-22 22:55     ` Michał Górny
@ 2014-08-23 20:30     ` Michał Górny
  2014-08-23 21:56       ` Brian Dolbec
                         ` (2 more replies)
  1 sibling, 3 replies; 23+ messages in thread
From: Michał Górny @ 2014-08-23 20:30 UTC (permalink / raw
  To: gentoo-portage-dev; +Cc: Michał Górny

---
 .gitignore           |   1 +
 MANIFEST.in          |  19 ++
 Makefile             | 215 -------------------
 doc/Makefile         |  11 -
 mkrelease.sh         | 141 -------------
 pym/portage/const.py |   4 +-
 setup.py             | 578 +++++++++++++++++++++++++++++++++++++++++++++++++++
 7 files changed, 600 insertions(+), 369 deletions(-)
 create mode 100644 MANIFEST.in
 delete mode 100644 Makefile
 delete mode 100644 doc/Makefile
 delete mode 100755 mkrelease.sh
 create mode 100755 setup.py

diff --git a/.gitignore b/.gitignore
index 074bb86..c2dd534 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,4 +1,5 @@
 *.py[co]
 __pycache__/
 *.class
+/build
 /tags
diff --git a/MANIFEST.in b/MANIFEST.in
new file mode 100644
index 0000000..a005ab9
--- /dev/null
+++ b/MANIFEST.in
@@ -0,0 +1,19 @@
+# docs
+include DEVELOPING
+include LICENSE
+include TEST-NOTES
+
+# docbook sources
+include doc/custom.xsl
+include doc/fragment/*
+recursive-include doc *.docbook
+
+# extra conf files used in ebuild
+include cnf/make.conf.example.*
+
+# extra files for tests
+include .portage_not_installed
+include cnf/metadata.dtd
+
+# extra scripts
+include misc/*
diff --git a/Makefile b/Makefile
deleted file mode 100644
index 9eb6e66..0000000
--- a/Makefile
+++ /dev/null
@@ -1,215 +0,0 @@
-SHELL = /bin/sh
-PN ?= portage
-PF ?= portage
-HOMEPAGE ?= http://www.gentoo.org/proj/en/portage/index.xml
-PWD ?= $(shell pwd)
-S ?= $(PWD)
-WORKDIR ?= $(PWD)
-DESTDIR = $(PWD)/image/
-srcdir = $(S)
-prefix = /usr
-sysconfdir = /etc
-exec_prefix = $(prefix)
-bindir = $(exec_prefix)/bin
-sbindir = $(exec_prefix)/sbin
-libdir = $(exec_prefix)/lib
-datarootdir = $(prefix)/share
-datadir = $(datarootdir)
-mandir = $(datarootdir)/man
-docdir = $(datarootdir)/doc/$(PF)
-htmldir = $(docdir)/html
-portage_datadir = $(datarootdir)/$(PN)
-portage_confdir = $(portage_datadir)/config
-portage_setsdir = $(portage_confdir)/sets
-portage_base = $(libdir)/$(PN)
-EPYDOC_OPTS = -qqqqq --no-frames --show-imports
-INSMODE = 0644
-EXEMODE = 0755
-DIRMODE = 0755
-SYSCONFDIR_FILES = etc-update.conf dispatch-conf.conf
-PORTAGE_CONFDIR_FILES = make.conf.example make.globals repos.conf
-LOGROTATE_FILES = elog-save-summary
-BINDIR_FILES = ebuild egencache emerge emerge-webrsync \
-	emirrordist portageq quickpkg repoman
-SBINDIR_FILES = archive-conf dispatch-conf emaint \
-	env-update etc-update fixpackages regenworld
-DOCS = ChangeLog NEWS RELEASE-NOTES
-LINGUAS ?= $(shell cd "$(srcdir)/man" && find -mindepth 1 -type d)
-
-ifdef PYTHONPATH
-	PYTHONPATH := $(srcdir)/pym:$(PYTHONPATH)
-else
-	PYTHONPATH := $(srcdir)/pym
-endif
-
-all: docbook epydoc
-
-docbook:
-	set -e; \
-	touch "$(srcdir)/doc/fragment/date"; \
-	$(MAKE) -C "$(srcdir)/doc" xhtml xhtml-nochunks
-
-epydoc:
-	set -e; \
-	env PYTHONPATH="$(PYTHONPATH)" epydoc \
-		-o "$(WORKDIR)/epydoc" \
-		--name $(PN) \
-		--url "$(HOMEPAGE)" \
-		$(EPYDOC_OPTS) \
-		$$(cd "$(srcdir)" && find pym -name '*.py' | sed \
-		-e s:/__init__.py$$:: \
-		-e s:\.py$$:: \
-		-e s:^pym/:: \
-		-e s:/:.:g \
-		| sort); \
-	rm -f "$(WORKDIR)/epydoc/api-objects.txt"; \
-
-test:
-	set -e; \
-	"$(srcdir)/pym/portage/tests/runTests.py"; \
-
-install:
-	set -e; \
-	cd "$(srcdir)/cnf"; \
-	install -d -m$(DIRMODE) "$(DESTDIR)$(sysconfdir)"; \
-	install -m$(INSMODE) $(SYSCONFDIR_FILES) "$(DESTDIR)$(sysconfdir)"; \
-	\
-	install -d -m$(DIRMODE) "$(DESTDIR)$(portage_confdir)"; \
-	cd "$(srcdir)/cnf"; \
-	install -m$(INSMODE) $(PORTAGE_CONFDIR_FILES) \
-		"$(DESTDIR)$(portage_confdir)"; \
-	install -d -m$(DIRMODE) "$(DESTDIR)$(portage_setsdir)"; \
-	cd "$(S)/cnf/sets"; \
-	install -m$(INSMODE) *.conf "$(DESTDIR)$(portage_setsdir)"; \
-	\
-	install -d -m$(DIRMODE) "$(DESTDIR)$(sysconfdir)/logrotate.d"; \
-	cd "$(srcdir)/cnf/logrotate.d"; \
-	install -m$(INSMODE) $(LOGROTATE_FILES) \
-		"$(DESTDIR)$(sysconfdir)/logrotate.d"; \
-	\
-	for x in $$(cd "$(srcdir)" && find bin -type d) ; do \
-		cd "$(srcdir)/$$x"; \
-		install -d -m$(DIRMODE) "$(DESTDIR)$(portage_base)/$$x"; \
-		files=$$(find . -mindepth 1 -maxdepth 1 -type f ! -type l); \
-		if [ -n "$$files" ] ; then \
-			install -m$(EXEMODE) $$files \
-				"$(DESTDIR)$(portage_base)/$$x"; \
-		fi; \
-		symlinks=$$(find . -mindepth 1 -maxdepth 1 -type l); \
-		if [ -n "$$symlinks" ] ; then \
-			cp -P $$symlinks "$(DESTDIR)$(portage_base)/$$x"; \
-		fi; \
-	done; \
-	\
-	for x in $$(cd "$(srcdir)" && find pym/* -type d \
-		! -path "pym/portage/tests*") ; do \
-		cd "$(srcdir)/$$x"; \
-		files=$$(echo *.py); \
-		if [ -z "$$files" ] || [ "$$files" = "*.py" ]; then \
-			# __pycache__ directories contain no py files \
-			continue; \
-		fi; \
-		install -d -m$(DIRMODE) "$(DESTDIR)$(portage_base)/$$x"; \
-		install -m$(INSMODE) $$files "$(DESTDIR)$(portage_base)/$$x"; \
-	done; \
-	\
-	install -d -m$(DIRMODE) "$(DESTDIR)$(bindir)"; \
-	relative_path=".."; \
-	x=$(bindir) ; \
-	y="$(portage_base)"; \
-	if [ "$${x#$(prefix)}" != "$$x" ] && \
-		[ "$${y#$(prefix)}" != "$$y" ]; then \
-		x=$${x#$(prefix)}; \
-		y=$${y#$(prefix)}; \
-	fi; \
-	x=$${x%/*}; \
-	while [ -n "$$x" ] ; do \
-		relative_path=$${relative_path}/..; \
-		x=$${x%/*}; \
-	done; \
-	relative_path=$$relative_path$$y; \
-	for x in $(BINDIR_FILES) ; do \
-		ln -sf "$$relative_path/bin/$$x" \
-			"$(DESTDIR)$(bindir)/$$x"; \
-	done; \
-	\
-	install -d -m$(DIRMODE) "$(DESTDIR)$(sbindir)"; \
-	relative_path=".."; \
-	x=$(sbindir) ; \
-	y="$(portage_base)"; \
-	if [ "$${x#$(prefix)}" != "$$x" ] && \
-		[ "$${y#$(prefix)}" != "$$y" ]; then \
-		x=$${x#$(prefix)}; \
-		y=$${y#$(prefix)}; \
-	fi; \
-	x=$${x%/*}; \
-	while [ -n "$$x" ] ; do \
-		relative_path=$${relative_path}/..; \
-		x=$${x%/*}; \
-	done; \
-	relative_path=$$relative_path$$y; \
-	for x in $(SBINDIR_FILES) ; do \
-		ln -sf "$$relative_path/bin/$$x" \
-			"$(DESTDIR)$(sbindir)/$$x"; \
-	done; \
-	\
-	ln -sf "$$relative_path/bin/env-update" \
-		"$(DESTDIR)$(sbindir)/update-env"; \
-	ln -sf "$$relative_path/bin/etc-update" \
-		"$(DESTDIR)$(sbindir)/update-etc"; \
-	\
-	# We install some minimal tests for use as a preinst sanity check. \
-	# These tests must be able to run without a full source tree and \
-	# without relying on a previous portage instance being installed. \
-	install -d -m$(DIRMODE) \
-		"$(DESTDIR)$(portage_base)/pym/portage/tests"; \
-	install -m$(EXEMODE) "$(srcdir)/pym/portage/tests/runTests" \
-		"$(DESTDIR)$(portage_base)/pym/portage/tests"; \
-	cd "$(srcdir)/pym/portage/tests"; \
-	install -m$(INSMODE) *.py \
-		"$(DESTDIR)$(portage_base)/pym/portage/tests"; \
-	install -d -m$(DIRMODE) \
-		"$(DESTDIR)$(portage_base)/pym/portage/tests/lint"; \
-	cd "$(srcdir)/pym/portage/tests/lint"; \
-	install -m$(INSMODE) *.py __test__ \
-		"$(DESTDIR)$(portage_base)/pym/portage/tests/lint"; \
-	\
-	install -d -m$(DIRMODE) "$(DESTDIR)$(docdir)"; \
-	cd "$(srcdir)"; \
-	install -m $(INSMODE) $(DOCS) "$(DESTDIR)$(docdir)"; \
-	\
-	for x in "" $(LINGUAS); do \
-		for y in 1 5 ; do \
-			if [ -d "$(srcdir)/man/$$x" ]; then \
-				cd "$(srcdir)/man/$$x"; \
-				files=$$(echo *.$$y); \
-				if [ -z "$$files" ] || [ "$$files" = "*.$$y" ]; then \
-					continue; \
-				fi; \
-				install -d -m$(DIRMODE) "$(DESTDIR)$(mandir)/$$x/man$$y"; \
-				install -m$(INSMODE) *.$$y "$(DESTDIR)$(mandir)/$$x/man$$y"; \
-			fi; \
-		done; \
-	done; \
-	\
-	if [ -f "$(srcdir)/doc/portage.html" ] ; then \
-		install -d -m$(DIRMODE) "$(DESTDIR)$(htmldir)"; \
-		cd "$(srcdir)/doc"; \
-		install -m$(INSMODE) *.html "$(DESTDIR)$(htmldir)"; \
-	fi; \
-	\
-	if [ -d "$(WORKDIR)/epydoc" ] ; then \
-		install -d -m$(DIRMODE) "$(DESTDIR)$(htmldir)"; \
-		cp -pPR "$(WORKDIR)/epydoc" \
-			"$(DESTDIR)$(htmldir)/api"; \
-		cd "$(DESTDIR)$(htmldir)/api"; \
-		find . -type d | xargs chmod $(DIRMODE); \
-		find . -type f | xargs chmod $(INSMODE); \
-	fi; \
-
-clean:
-	set -e; \
-	$(MAKE) -C "$(srcdir)/doc" clean; \
-	rm -rf "$(WORKDIR)/epydoc"; \
-
-.PHONY: all clean docbook epydoc install test
diff --git a/doc/Makefile b/doc/Makefile
deleted file mode 100644
index 261a0b4..0000000
--- a/doc/Makefile
+++ /dev/null
@@ -1,11 +0,0 @@
-all: xhtml xhtml-nochunks
-
-XMLTO_FLAGS = -m custom.xsl
-man pdf txt xhtml xhtml-nochunks:
-	xmlto $@ $(XMLTO_FLAGS) portage.docbook
-
-clean distclean:
-	rm -f *.1 *.html portage.txt
-
-.PHONY: all clean distclean \
-        man pdf txt xhtml xhtml-nochunks
diff --git a/mkrelease.sh b/mkrelease.sh
deleted file mode 100755
index f9f7564..0000000
--- a/mkrelease.sh
+++ /dev/null
@@ -1,141 +0,0 @@
-#!/bin/bash
-# Copyright 2008-2014 Gentoo Foundation
-# Distributed under the terms of the GNU General Public License v2
-
-RELEASE_BUILDDIR=${RELEASE_BUILDDIR:-/var/tmp/portage-release}
-SOURCE_DIR=${RELEASE_BUILDDIR}/checkout
-BRANCH=${BRANCH:-master}
-USE_TAG=false
-CHANGELOG_REVISION=
-UPLOAD_LOCATION=
-RUNTESTS=false
-USER=
-
-usage() {
-	echo "Usage: ${0##*/} [--changelog-rev <tree-ish>] [-t|--tag] [-u|--upload <location>]  [--user <username>] [--runtests] <version>"
-	exit ${1:-0}
-}
-
-die() {
-	printf 'error: %s\n' "$*"
-	usage 1
-}
-
-ARGS=$(getopt -o htu: --long help,changelog-rev:,runtests,tag,upload:,user: \
-	-n "${0##*/}" -- "$@")
-[ $? != 0 ] && die "initialization error"
-
-eval set -- "${ARGS}"
-
-while true; do
-	case $1 in
-		--changelog-rev)
-			CHANGELOG_REVISION=$2
-			shift 2
-			;;
-		-t|--tag)
-			USE_TAG=true
-			shift
-			;;
-		-u|--upload)
-			UPLOAD_LOCATION=$2
-			shift 2
-			;;
-		--user)
-			USER=$2"@"
-			shift 2
-			;;
-		-h|--help)
-			usage
-			;;
-		--runtests)
-			RUNTESTS=true
-			shift
-			;;
-		--)
-			shift
-			break
-			;;
-		*)
-			die "unknown option: $1"
-			;;
-	esac
-done
-
-[ $# != 1 ] && die "Need version argument"
-[[ -n ${1/[0-9]*} ]] && die "Invalid version argument"
-
-VERSION=$1
-RELEASE=portage-${VERSION}
-RELEASE_DIR=${RELEASE_BUILDDIR}/${RELEASE}
-RELEASE_TARBALL="${RELEASE_BUILDDIR}/${RELEASE}.tar.bz2"
-TREE_ISH=${BRANCH}
-if [[ ${USE_TAG} == "true" ]] ; then
-	TREE_ISH="v${VERSION}"
-fi
-
-echo ">>> Cleaning working directories ${RELEASE_DIR} ${SOURCE_DIR}"
-rm -rf "${RELEASE_DIR}" "${SOURCE_DIR}" || die "directory cleanup failed"
-mkdir -p "${RELEASE_DIR}" || die "directory creation failed"
-mkdir -p "${SOURCE_DIR}" || die "mkdir failed"
-
-echo ">>> Starting GIT archive"
-git archive --format=tar ${TREE_ISH} | \
-	tar -xf - -C "${SOURCE_DIR}" || die "git archive failed"
-
-echo ">>> Building release tree"
-cp -a "${SOURCE_DIR}/"{bin,cnf,doc,man,misc,pym} "${RELEASE_DIR}/" || die "directory copy failed"
-cp "${SOURCE_DIR}/"{.portage_not_installed,DEVELOPING,LICENSE,Makefile,NEWS,README,RELEASE-NOTES,TEST-NOTES} \
-	"${RELEASE_DIR}/" || die "file copy failed"
-
-if [[ ${RUNTESTS} == "true" ]] ; then
-	pushd "${SOURCE_DIR}" >/dev/null
-	./runtests.sh --python-versions=supported || die "tests failed"
-	popd >/dev/null
-fi
-
-rm -rf "${SOURCE_DIR}" || die "directory cleanup failed"
-
-echo ">>> Setting portage.VERSION"
-sed -e "s/^VERSION = .*/VERSION = \"${VERSION}\"/" \
-	-i "${RELEASE_DIR}/pym/portage/__init__.py" || \
-	die "Failed to patch portage.VERSION"
-
-echo ">>> Creating Changelog"
-git_log_opts=""
-if [[ -n ${CHANGELOG_REVISION} ]] ; then
-	git_log_opts+=" ${CHANGELOG_REVISION}^..${TREE_ISH}"
-else
-	git_log_opts+=" ${TREE_ISH}"
-fi
-skip_next=false
-git log ${git_log_opts} | fmt -w 80 -p "    " | while read -r ; do
-	if [[ ${skip_next} == "true" ]] ; then
-		skip_next=false
-	elif [[ ${REPLY} == "    svn path="* ]] ; then
-		skip_next=true
-	else
-		echo "${REPLY}"
-	fi
-done > "${RELEASE_DIR}/ChangeLog" || die "ChangeLog creation failed"
-
-cd "${RELEASE_BUILDDIR}"
-
-echo ">>> Creating release tarball ${RELEASE_TARBALL}"
-tar --owner portage --group portage -cjf "${RELEASE_TARBALL}" "${RELEASE}" || \
-	die "tarball creation failed"
-
-DISTDIR=$(portageq distdir)
-if [[ -n ${DISTDIR} && -d ${DISTDIR} && -w ${DISTDIR} ]] ; then
-	echo ">>> Copying release tarball into ${DISTDIR}"
-	cp "${RELEASE_TARBALL}" "${DISTDIR}"/ || echo "!!! tarball copy failed"
-fi
-
-if [[ -n ${UPLOAD_LOCATION} ]] ; then
-	echo ">>> Uploading ${RELEASE_TARBALL} to ${USER}dev.gentoo.org:${UPLOAD_LOCATION}"
-	scp "${RELEASE_TARBALL}" "${USER}dev.gentoo.org:${UPLOAD_LOCATION}" || die "upload failed"
-else
-	du -h "${RELEASE_TARBALL}"
-fi
-
-exit 0
diff --git a/pym/portage/const.py b/pym/portage/const.py
index f518b47..acb90f9 100644
--- a/pym/portage/const.py
+++ b/pym/portage/const.py
@@ -60,8 +60,8 @@ GLOBAL_CONFIG_PATH       = "/usr/share/portage/config"
 # these variables are not used with target_root or config_root
 # NOTE: Use realpath(__file__) so that python module symlinks in site-packages
 # are followed back to the real location of the whole portage installation.
-PORTAGE_BASE_PATH        = os.path.join(os.sep, os.sep.join(os.path.realpath(
-                               __file__.rstrip("co")).split(os.sep)[:-3]))
+# NOTE: Please keep PORTAGE_BASE_PATH in one line to help substitutions.
+PORTAGE_BASE_PATH        = os.path.join(os.sep, os.sep.join(os.path.realpath(__file__.rstrip("co")).split(os.sep)[:-3]))
 PORTAGE_BIN_PATH         = PORTAGE_BASE_PATH + "/bin"
 PORTAGE_PYM_PATH         = os.path.realpath(os.path.join(__file__, '../..'))
 LOCALE_DATA_PATH         = PORTAGE_BASE_PATH + "/locale"  # FIXME: not used
diff --git a/setup.py b/setup.py
new file mode 100755
index 0000000..7ffe7f0
--- /dev/null
+++ b/setup.py
@@ -0,0 +1,578 @@
+#!/usr/bin/env python
+#	vim:fileencoding=utf-8
+# (c) 2010 Michał Górny <mgorny@gentoo.org>
+# Released under the terms of the 2-clause BSD license.
+
+from distutils.core import setup, Command
+from distutils.command.build_scripts import build_scripts
+from distutils.command.clean import clean
+from distutils.command.install import install
+from distutils.command.install_data import install_data
+from distutils.command.install_lib import install_lib
+from distutils.command.install_scripts import install_scripts
+from distutils.command.sdist import sdist
+from distutils.dir_util import remove_tree
+from distutils.util import change_root, subst_vars
+
+import codecs, collections, glob, os, os.path, re, subprocess, sys
+
+# TODO:
+# - smarter rebuilds of docs w/ 'install_docbook' and 'install_epydoc'.
+
+x_scripts = {
+	'bin': [
+		'bin/ebuild', 'bin/egencache', 'bin/emerge', 'bin/emerge-webrsync',
+		'bin/emirrordist', 'bin/portageq', 'bin/quickpkg', 'bin/repoman'
+	],
+	'sbin': [
+		'bin/archive-conf', 'bin/dispatch-conf', 'bin/emaint', 'bin/env-update',
+		'bin/etc-update', 'bin/fixpackages', 'bin/regenworld'
+	],
+}
+
+
+class docbook(Command):
+	""" Build docs using docbook. """
+
+	user_options = [
+		('doc-formats=', None, 'Documentation formats to build (all xmlto formats for docbook are allowed, comma-separated'),
+	]
+
+	def initialize_options(self):
+		self.doc_formats = 'xhtml,xhtml-nochunks'
+
+	def finalize_options(self):
+		self.doc_formats = self.doc_formats.replace(',', ' ').split()
+
+	def run(self):
+		with open('doc/fragment/date', 'w'):
+			pass
+
+		for f in self.doc_formats:
+			print('Building docs in %s format...' % f)
+			subprocess.check_call(['xmlto', '-o', 'doc',
+				'-m', 'doc/custom.xsl', f, 'doc/portage.docbook'])
+
+
+class epydoc(Command):
+	""" Build API docs using epydoc. """
+
+	user_options = [
+	]
+
+	def initialize_options(self):
+		self.build_lib = None
+
+	def finalize_options(self):
+		self.set_undefined_options('build_py', ('build_lib', 'build_lib'))
+
+	def run(self):
+		self.run_command('build_py')
+
+		print('Building API documentation...')
+
+		process_env = os.environ.copy()
+		pythonpath = self.build_lib
+		try:
+			pythonpath += ':' + process_env['PYTHONPATH']
+		except KeyError:
+			pass
+		process_env['PYTHONPATH'] = pythonpath
+
+		subprocess.check_call(['epydoc', '-o', 'epydoc',
+			'--name', self.distribution.get_name(),
+			'--url', self.distribution.get_url(),
+			'-qq', '--no-frames', '--show-imports',
+			'--exclude', 'portage.tests',
+			'_emerge', 'portage', 'repoman'],
+			env = process_env)
+		os.remove('epydoc/api-objects.txt')
+
+
+class install_docbook(install_data):
+	""" install_data for docbook docs """
+
+	user_options = install_data.user_options
+
+	def initialize_options(self):
+		install_data.initialize_options(self)
+		self.htmldir = None
+
+	def finalize_options(self):
+		self.set_undefined_options('install', ('htmldir', 'htmldir'))
+		install_data.finalize_options(self)
+
+	def run(self):
+		if not os.path.exists('doc/portage.html'):
+			self.run_command('docbook')
+		self.data_files = [
+			(self.htmldir, glob.glob('doc/*.html')),
+		]
+		install_data.run(self)
+
+
+class install_epydoc(install_data):
+	""" install_data for epydoc docs """
+
+	user_options = install_data.user_options
+
+	def initialize_options(self):
+		install_data.initialize_options(self)
+		self.htmldir = None
+
+	def finalize_options(self):
+		self.set_undefined_options('install', ('htmldir', 'htmldir'))
+		install_data.finalize_options(self)
+
+	def run(self):
+		if not os.path.exists('epydoc/index.html'):
+			self.run_command('epydoc')
+		self.data_files = [
+			(os.path.join(self.htmldir, 'api'), glob.glob('epydoc/*')),
+		]
+		install_data.run(self)
+
+
+class x_build_scripts_custom(build_scripts):
+	def finalize_options(self):
+		build_scripts.finalize_options(self)
+		if 'dir_name' in dir(self):
+			self.build_dir = os.path.join(self.build_dir, self.dir_name)
+			if self.dir_name in x_scripts:
+				self.scripts = x_scripts[self.dir_name]
+			else:
+				self.scripts = set(self.scripts)
+				for other_files in x_scripts.values():
+					self.scripts.difference_update(other_files)
+
+	def run(self):
+		# group scripts by subdirectory
+		split_scripts = collections.defaultdict(list)
+		for f in self.scripts:
+			dir_name = os.path.dirname(f[len('bin/'):])
+			split_scripts[dir_name].append(f)
+
+		base_dir = self.build_dir
+		base_scripts = self.scripts
+		for d, files in split_scripts.items():
+			self.build_dir = os.path.join(base_dir, d)
+			self.scripts = files
+			self.copy_scripts()
+
+		# restore previous values
+		self.build_dir = base_dir
+		self.scripts = base_scripts
+
+
+class x_build_scripts_bin(x_build_scripts_custom):
+	dir_name = 'bin'
+
+
+class x_build_scripts_sbin(x_build_scripts_custom):
+	dir_name = 'sbin'
+
+
+class x_build_scripts_portagebin(x_build_scripts_custom):
+	dir_name = 'portage'
+
+
+class x_build_scripts(build_scripts):
+	def initialize_option(self):
+		build_scripts.initialize_options(self)
+
+	def finalize_options(self):
+		build_scripts.finalize_options(self)
+
+	def run(self):
+		self.run_command('build_scripts_bin')
+		self.run_command('build_scripts_portagebin')
+		self.run_command('build_scripts_sbin')
+
+
+class x_clean(clean):
+	""" clean extended for doc & post-test cleaning """
+
+	def clean_docs(self):
+		def get_doc_outfiles():
+			for dirpath, dirnames, filenames in os.walk('doc'):
+				for f in filenames:
+					if f.endswith('.docbook') or f == 'custom.xsl':
+						pass
+					else:
+						yield os.path.join(dirpath, f)
+
+				# do not recurse
+				break
+
+
+		for f in get_doc_outfiles():
+			print('removing %s' % repr(f))
+			os.remove(f)
+
+		if os.path.isdir('epydoc'):
+			remove_tree('epydoc')
+
+	def clean_tests(self):
+		# do not remove incorrect dirs accidentally
+		top_dir = os.path.normpath(os.path.join(self.build_lib, '..'))
+		cprefix = os.path.commonprefix((self.build_base, top_dir))
+		if cprefix != self.build_base:
+			return
+
+		bin_dir = os.path.join(top_dir, 'bin')
+		if os.path.exists(bin_dir):
+			remove_tree(bin_dir)
+
+		conf_dir = os.path.join(top_dir, 'cnf')
+		if os.path.islink(conf_dir):
+			print('removing %s symlink' % repr(conf_dir))
+			os.unlink(conf_dir)
+
+		pni_file = os.path.join(top_dir, '.portage_not_installed')
+		if os.path.exists(pni_file):
+			print('removing %s' % repr(pni_file))
+			os.unlink(pni_file)
+
+	def run(self):
+		if self.all:
+			self.clean_tests()
+			self.clean_docs()
+
+		clean.run(self)
+
+
+class x_install(install):
+	""" install command with extra Portage paths """
+
+	user_options = install.user_options + [
+		# note: $prefix and $exec_prefix are reserved for Python install
+		('system-prefix=', None, "Prefix for architecture-independent data"),
+		('system-exec-prefix=', None, "Prefix for architecture-specific data"),
+
+		('bindir=', None, "Install directory for main executables"),
+		('datarootdir=', None, "Data install root directory"),
+		('docdir=', None, "Documentation install directory"),
+		('htmldir=', None, "HTML documentation install directory"),
+		('mandir=', None, "Manpage root install directory"),
+		('portage-base=', 'b', "Portage install base"),
+		('portage-bindir=', None, "Install directory for Portage internal-use executables"),
+		('portage-datadir=', None, 'Install directory for data files'),
+		('sbindir=', None, "Install directory for superuser-intended executables"),
+		('sysconfdir=', None, 'System configuration path'),
+	]
+
+	# note: the order is important for proper substitution
+	paths = [
+		('system_prefix', '/usr'),
+		('system_exec_prefix', '$system_prefix'),
+
+		('bindir', '$system_exec_prefix/bin'),
+		('sbindir', '$system_exec_prefix/sbin'),
+		('sysconfdir', '/etc'),
+
+		('datarootdir', '$system_prefix/share'),
+		('docdir', '$datarootdir/doc/$package-$version'),
+		('htmldir', '$docdir/html'),
+		('mandir', '$datarootdir/man'),
+
+		('portage_base', '$system_exec_prefix/lib/portage'),
+		('portage_bindir', '$portage_base/bin'),
+		('portage_datadir', '$datarootdir/portage'),
+
+		# not customized at the moment
+		('logrotatedir', '$sysconfdir/logrotate'),
+		('portage_confdir', '$portage_datadir/config'),
+		('portage_setsdir', '$portage_confdir/sets'),
+	]
+
+	def initialize_options(self):
+		install.initialize_options(self)
+
+		for key, default in self.paths:
+			setattr(self, key, default)
+		self.subst_paths = {}
+
+	def finalize_options(self):
+		install.finalize_options(self)
+
+		# substitute variables
+		new_paths = {
+			'package': self.distribution.get_name(),
+			'version': self.distribution.get_version(),
+		}
+		for key, default in self.paths:
+			new_paths[key] = subst_vars(getattr(self, key), new_paths)
+			setattr(self, key, new_paths[key])
+		self.subst_paths = new_paths
+
+
+class x_install_data(install_data):
+	""" install_data with customized path support """
+
+	user_options = install_data.user_options
+
+	def initialize_options(self):
+		install_data.initialize_options(self)
+		self.paths = None
+
+	def finalize_options(self):
+		install_data.finalize_options(self)
+		self.set_undefined_options('install',
+			('subst_paths', 'paths'))
+
+		# substitute variables in data_files
+		for f in self.data_files:
+			f[0] = subst_vars(f[0], self.paths)
+
+
+class x_install_lib(install_lib):
+	""" install_lib command with Portage path substitution """
+
+	user_options = install_lib.user_options
+
+	def initialize_options(self):
+		install_lib.initialize_options(self)
+		self.portage_base = None
+		self.portage_bindir = None
+		self.portage_confdir = None
+
+	def finalize_options(self):
+		install_lib.finalize_options(self)
+		self.set_undefined_options('install',
+			('portage_base', 'portage_base'),
+			('portage_bindir', 'portage_bindir'),
+			('portage_confdir', 'portage_confdir'))
+
+	def install(self):
+		ret = install_lib.install(self)
+
+		def rewrite_file(path, val_dict):
+			path = os.path.join(self.install_dir, path)
+			print('Rewriting %s' % path)
+			with codecs.open(path, 'r', 'utf-8') as f:
+				data = f.read()
+
+			for varname, val in val_dict.items():
+				regexp = r'^(%s\s*=).*$' % varname
+				repl = r'\1 %s' % repr(val)
+
+				data = re.sub(regexp, repl, data, 0, re.MULTILINE)
+
+			with codecs.open(path, 'w', 'utf-8') as f:
+				f.write(data)
+
+		rewrite_file('portage/__init__.py', {
+			'VERSION': self.distribution.get_version(),
+		})
+		rewrite_file('portage/const.py', {
+			'PORTAGE_BASE_PATH': self.portage_base,
+			'PORTAGE_BIN_PATH': self.portage_bindir,
+			'PORTAGE_CONFIG_PATH': self.portage_confdir,
+		})
+
+		return ret
+
+
+class x_install_scripts_custom(install_scripts):
+	def initialize_options(self):
+		install_scripts.initialize_options(self)
+		self.root = None
+
+	def finalize_options(self):
+		self.set_undefined_options('install',
+			('root', 'root'),
+			(self.var_name, 'install_dir'))
+		install_scripts.finalize_options(self)
+		self.build_dir = os.path.join(self.build_dir, self.dir_name)
+
+		# prepend root
+		if self.root is not None:
+			self.install_dir = change_root(self.root, self.install_dir)
+
+
+class x_install_scripts_bin(x_install_scripts_custom):
+	dir_name = 'bin'
+	var_name = 'bindir'
+
+
+class x_install_scripts_sbin(x_install_scripts_custom):
+	dir_name = 'sbin'
+	var_name = 'sbindir'
+
+
+class x_install_scripts_portagebin(x_install_scripts_custom):
+	dir_name = 'portage'
+	var_name = 'portage_bindir'
+
+
+class x_install_scripts(install_scripts):
+	def initialize_option(self):
+		pass
+
+	def finalize_options(self):
+		pass
+
+	def run(self):
+		self.run_command('install_scripts_bin')
+		self.run_command('install_scripts_portagebin')
+		self.run_command('install_scripts_sbin')
+
+
+class x_sdist(sdist):
+	""" sdist defaulting to .tar.bz2 format """
+
+	def finalize_options(self):
+		if self.formats is None:
+			self.formats = ['bztar']
+
+		sdist.finalize_options(self)
+
+
+class build_tests(x_build_scripts_custom):
+	""" Prepare build dir for running tests. """
+
+	def initialize_options(self):
+		x_build_scripts_custom.initialize_options(self)
+		self.build_base = None
+		self.build_lib = None
+
+	def finalize_options(self):
+		x_build_scripts_custom.finalize_options(self)
+		self.set_undefined_options('build',
+			('build_base', 'build_base'),
+			('build_lib', 'build_lib'))
+
+		# since we will be writing to $build_lib/.., it is important
+		# that we do not leave $build_base
+		self.top_dir = os.path.normpath(os.path.join(self.build_lib, '..'))
+		cprefix = os.path.commonprefix((self.build_base, self.top_dir))
+		if cprefix != self.build_base:
+			raise SystemError('build_lib must be a subdirectory of build_base')
+
+		self.build_dir = os.path.join(self.top_dir, 'bin')
+
+	def run(self):
+		self.run_command('build_py')
+
+		# install all scripts $build_lib/../bin
+		# (we can't do a symlink since we want shebangs corrected)
+		x_build_scripts_custom.run(self)
+
+		# symlink 'cnf' directory
+		conf_dir = os.path.join(self.top_dir, 'cnf')
+		if os.path.exists(conf_dir):
+			if not os.path.islink(conf_dir):
+				raise SystemError('%s exists and is not a symlink (collision)'
+					% repr(conf_dir))
+			os.unlink(conf_dir)
+		conf_src = os.path.relpath('cnf', self.top_dir)
+		print('Symlinking %s -> %s' % (conf_dir, conf_src))
+		os.symlink(conf_src, conf_dir)
+
+		# create $build_lib/../.portage_not_installed
+		# to enable proper paths in tests
+		with open(os.path.join(self.top_dir, '.portage_not_installed'), 'w') as f:
+			pass
+
+
+class test(Command):
+	""" run tests """
+
+	user_options = []
+
+	def initialize_options(self):
+		self.build_lib = None
+
+	def finalize_options(self):
+		self.set_undefined_options('build',
+			('build_lib', 'build_lib'))
+
+	def run(self):
+		self.run_command('build_tests')
+		subprocess.check_call([
+			sys.executable, '-bWd',
+			os.path.join(self.build_lib, 'portage/tests/runTests.py')
+		])
+
+
+def find_packages():
+	for dirpath, dirnames, filenames in os.walk('pym'):
+		if '__init__.py' in filenames:
+			yield os.path.relpath(dirpath, 'pym')
+
+
+def find_scripts():
+	for dirpath, dirnames, filenames in os.walk('bin'):
+		for f in filenames:
+			yield os.path.join(dirpath, f)
+
+
+def get_manpages():
+	linguas = os.environ.get('LINGUAS')
+	if linguas is not None:
+		linguas = linguas.split()
+
+	for dirpath, dirnames, filenames in os.walk('man'):
+		groups = collections.defaultdict(list)
+		for f in filenames:
+			fn, suffix = f.rsplit('.', 1)
+			groups[suffix].append(os.path.join(dirpath, f))
+
+		topdir = dirpath[len('man/'):]
+		if not topdir or linguas is None or topdir in linguas:
+			for g, mans in groups.items():
+				yield [os.path.join('$mandir', topdir, 'man%s' % g), mans]
+
+setup(
+	name = 'portage',
+	version = '2.2.12',
+	url = 'https://wiki.gentoo.org/wiki/Project:Portage',
+	author = 'Gentoo Portage Development Team',
+	author_email = 'dev-portage@gentoo.org',
+
+	package_dir = {'': 'pym'},
+	packages = list(find_packages()),
+	# something to cheat build & install commands
+	scripts = list(find_scripts()),
+
+	data_files = list(get_manpages()) + [
+		['$sysconfdir', ['cnf/etc-update.conf', 'cnf/dispatch-conf.conf']],
+		['$logrotatedir', ['cnf/logrotate.d/elog-save-summary']],
+		['$portage_confdir', [
+			'cnf/make.conf.example', 'cnf/make.globals', 'cnf/repos.conf']],
+		['$portage_setsdir', ['cnf/sets/portage.conf']],
+		['$docdir', ['NEWS', 'RELEASE-NOTES']],
+	],
+
+	cmdclass = {
+		'build_scripts': x_build_scripts,
+		'build_scripts_bin': x_build_scripts_bin,
+		'build_scripts_portagebin': x_build_scripts_portagebin,
+		'build_scripts_sbin': x_build_scripts_sbin,
+		'build_tests': build_tests,
+		'clean': x_clean,
+		'docbook': docbook,
+		'epydoc': epydoc,
+		'install': x_install,
+		'install_data': x_install_data,
+		'install_docbook': install_docbook,
+		'install_epydoc': install_epydoc,
+		'install_lib': x_install_lib,
+		'install_scripts': x_install_scripts,
+		'install_scripts_bin': x_install_scripts_bin,
+		'install_scripts_portagebin': x_install_scripts_portagebin,
+		'install_scripts_sbin': x_install_scripts_sbin,
+		'sdist': x_sdist,
+		'test': test,
+	},
+
+	classifiers = [
+		'Development Status :: 5 - Production/Stable',
+		'Environment :: Console',
+		'Intended Audience :: System Administrators',
+		'License :: OSI Approved :: GNU General Public License v2 (GPLv2)',
+		'Operating System :: POSIX',
+		'Programming Language :: Python',
+		'Topic :: System :: Installation/Setup'
+	]
+)
-- 
2.0.4



^ permalink raw reply related	[flat|nested] 23+ messages in thread

* Re: [gentoo-portage-dev] [PATCH v2] Install Portage using setup.py
  2014-08-23 20:30     ` [gentoo-portage-dev] [PATCH v2] Install Portage " Michał Górny
@ 2014-08-23 21:56       ` Brian Dolbec
  2014-08-29 17:08       ` Brian Dolbec
  2014-09-05  6:58       ` [gentoo-portage-dev] [PATCH v3] " Michał Górny
  2 siblings, 0 replies; 23+ messages in thread
From: Brian Dolbec @ 2014-08-23 21:56 UTC (permalink / raw
  To: gentoo-portage-dev

On Sat, 23 Aug 2014 22:30:56 +0200
Michał Górny <mgorny@gentoo.org> wrote:


What about adding to the commit message that v2 also includes "setup.py
sdist" capability?



> diff --git a/mkrelease.sh b/mkrelease.sh
> deleted file mode 100755
> index f9f7564..0000000
> --- a/mkrelease.sh
> +++ /dev/null
> @@ -1,141 +0,0 @@
> -#!/bin/bash
> -# Copyright 2008-2014 Gentoo Foundation
> -# Distributed under the terms of the GNU General Public License v2
> -
> -RELEASE_BUILDDIR=${RELEASE_BUILDDIR:-/var/tmp/portage-release}
> -SOURCE_DIR=${RELEASE_BUILDDIR}/checkout
> -BRANCH=${BRANCH:-master}
> -USE_TAG=false
> -CHANGELOG_REVISION=
> -UPLOAD_LOCATION=
> -RUNTESTS=false
> -USER=
> -
> -usage() {
> -	echo "Usage: ${0##*/} [--changelog-rev <tree-ish>]
> [-t|--tag] [-u|--upload <location>]  [--user <username>] [--runtests]
> <version>"
> -	exit ${1:-0}
> -}
> -
> -die() {
> -	printf 'error: %s\n' "$*"
> -	usage 1
> -}
> -
> -ARGS=$(getopt -o htu: --long
> help,changelog-rev:,runtests,tag,upload:,user: \
> -	-n "${0##*/}" -- "$@")
> -[ $? != 0 ] && die "initialization error"
> -
> -eval set -- "${ARGS}"
> -
> -while true; do
> -	case $1 in
> -		--changelog-rev)
> -			CHANGELOG_REVISION=$2
> -			shift 2
> -			;;
> -		-t|--tag)
> -			USE_TAG=true
> -			shift
> -			;;
> -		-u|--upload)
> -			UPLOAD_LOCATION=$2
> -			shift 2
> -			;;
> -		--user)
> -			USER=$2"@"
> -			shift 2
> -			;;
> -		-h|--help)
> -			usage
> -			;;
> -		--runtests)
> -			RUNTESTS=true
> -			shift
> -			;;
> -		--)
> -			shift
> -			break
> -			;;
> -		*)
> -			die "unknown option: $1"
> -			;;
> -	esac
> -done
> -
> -[ $# != 1 ] && die "Need version argument"
> -[[ -n ${1/[0-9]*} ]] && die "Invalid version argument"
> -
> -VERSION=$1
> -RELEASE=portage-${VERSION}
> -RELEASE_DIR=${RELEASE_BUILDDIR}/${RELEASE}
> -RELEASE_TARBALL="${RELEASE_BUILDDIR}/${RELEASE}.tar.bz2"
> -TREE_ISH=${BRANCH}
> -if [[ ${USE_TAG} == "true" ]] ; then
> -	TREE_ISH="v${VERSION}"
> -fi
> -
> -echo ">>> Cleaning working directories ${RELEASE_DIR} ${SOURCE_DIR}"
> -rm -rf "${RELEASE_DIR}" "${SOURCE_DIR}" || die "directory cleanup
> failed" -mkdir -p "${RELEASE_DIR}" || die "directory creation failed"
> -mkdir -p "${SOURCE_DIR}" || die "mkdir failed"
> -
> -echo ">>> Starting GIT archive"
> -git archive --format=tar ${TREE_ISH} | \
> -	tar -xf - -C "${SOURCE_DIR}" || die "git archive failed"
> -
> -echo ">>> Building release tree"
> -cp -a "${SOURCE_DIR}/"{bin,cnf,doc,man,misc,pym} "${RELEASE_DIR}/"
> || die "directory copy failed" -cp
> "${SOURCE_DIR}/"{.portage_not_installed,DEVELOPING,LICENSE,Makefile,NEWS,README,RELEASE-NOTES,TEST-NOTES}
> \
> -	"${RELEASE_DIR}/" || die "file copy failed"
> -
> -if [[ ${RUNTESTS} == "true" ]] ; then
> -	pushd "${SOURCE_DIR}" >/dev/null
> -	./runtests.sh --python-versions=supported || die "tests
> failed"
> -	popd >/dev/null
> -fi
> -
> -rm -rf "${SOURCE_DIR}" || die "directory cleanup failed"
> -
> -echo ">>> Setting portage.VERSION"
> -sed -e "s/^VERSION = .*/VERSION = \"${VERSION}\"/" \
> -	-i "${RELEASE_DIR}/pym/portage/__init__.py" || \
> -	die "Failed to patch portage.VERSION"
> -
> -echo ">>> Creating Changelog"
> -git_log_opts=""
> -if [[ -n ${CHANGELOG_REVISION} ]] ; then
> -	git_log_opts+=" ${CHANGELOG_REVISION}^..${TREE_ISH}"
> -else
> -	git_log_opts+=" ${TREE_ISH}"
> -fi
> -skip_next=false
> -git log ${git_log_opts} | fmt -w 80 -p "    " | while read -r ; do
> -	if [[ ${skip_next} == "true" ]] ; then
> -		skip_next=false
> -	elif [[ ${REPLY} == "    svn path="* ]] ; then
> -		skip_next=true
> -	else
> -		echo "${REPLY}"
> -	fi
> -done > "${RELEASE_DIR}/ChangeLog" || die "ChangeLog creation failed"
> -
> -cd "${RELEASE_BUILDDIR}"
> -
> -echo ">>> Creating release tarball ${RELEASE_TARBALL}"
> -tar --owner portage --group portage -cjf "${RELEASE_TARBALL}"
> "${RELEASE}" || \
> -	die "tarball creation failed"
> -
> -DISTDIR=$(portageq distdir)
> -if [[ -n ${DISTDIR} && -d ${DISTDIR} && -w ${DISTDIR} ]] ; then
> -	echo ">>> Copying release tarball into ${DISTDIR}"
> -	cp "${RELEASE_TARBALL}" "${DISTDIR}"/ || echo "!!! tarball
> copy failed" -fi
> -
> -if [[ -n ${UPLOAD_LOCATION} ]] ; then
> -	echo ">>> Uploading ${RELEASE_TARBALL} to
> ${USER}dev.gentoo.org:${UPLOAD_LOCATION}"
> -	scp "${RELEASE_TARBALL}"
> "${USER}dev.gentoo.org:${UPLOAD_LOCATION}" || die "upload failed"
> -else
> -	du -h "${RELEASE_TARBALL}"
> -fi
> -
> -exit 0
>

Why are you removing everything from mkrelease.sh?

It does do some other useful things other than just make the release
tarball.
We have not decided to drop the Changelog as a team yet, only a few
words on irc.  Instead mkrelease could be modified to work with
setup.py for the tarball creation, but it still creates the Changelog,
copies the tarball to distfiles, uploads the file...

I did not see any of that functionality added to setup.py below.


> FIXME: not used diff --git a/setup.py b/setup.py new file mode 100755
> index 0000000..7ffe7f0 --- /dev/null
> +++ b/setup.py
> @@ -0,0 +1,578 @@
> +#!/usr/bin/env python
> +#	vim:fileencoding=utf-8
> +# (c) 2010 Michał Górny <mgorny@gentoo.org>
> +# Released under the terms of the 2-clause BSD license.
> +
> +from distutils.core import setup, Command
> +from distutils.command.build_scripts import build_scripts
> +from distutils.command.clean import clean
> +from distutils.command.install import install
> +from distutils.command.install_data import install_data
> +from distutils.command.install_lib import install_lib
> +from distutils.command.install_scripts import install_scripts
> +from distutils.command.sdist import sdist
> +from distutils.dir_util import remove_tree
> +from distutils.util import change_root, subst_vars
> +
> +import codecs, collections, glob, os, os.path, re, subprocess, sys
> +
> +# TODO:
> +# - smarter rebuilds of docs w/ 'install_docbook' and
> 'install_epydoc'. +
> +x_scripts = {
> +	'bin': [
> +		'bin/ebuild', 'bin/egencache', 'bin/emerge',
> 'bin/emerge-webrsync',
> +		'bin/emirrordist', 'bin/portageq', 'bin/quickpkg',
> 'bin/repoman'
> +	],
> +	'sbin': [
> +		'bin/archive-conf', 'bin/dispatch-conf',
> 'bin/emaint', 'bin/env-update',
> +		'bin/etc-update', 'bin/fixpackages', 'bin/regenworld'
> +	],
> +}
> +
> +
> +class docbook(Command):
> +	""" Build docs using docbook. """
> +
> +	user_options = [
> +		('doc-formats=', None, 'Documentation formats to
> build (all xmlto formats for docbook are allowed, comma-separated'),
> +	]
> +
> +	def initialize_options(self):
> +		self.doc_formats = 'xhtml,xhtml-nochunks'
> +
> +	def finalize_options(self):
> +		self.doc_formats = self.doc_formats.replace(',', '
> ').split() +
> +	def run(self):
> +		with open('doc/fragment/date', 'w'):
> +			pass
> +
> +		for f in self.doc_formats:
> +			print('Building docs in %s format...' % f)
> +			subprocess.check_call(['xmlto', '-o', 'doc',
> +				'-m', 'doc/custom.xsl', f,
> 'doc/portage.docbook']) +
> +
> +class epydoc(Command):
> +	""" Build API docs using epydoc. """
> +
> +	user_options = [
> +	]
> +
> +	def initialize_options(self):
> +		self.build_lib = None
> +
> +	def finalize_options(self):
> +		self.set_undefined_options('build_py', ('build_lib',
> 'build_lib')) +
> +	def run(self):
> +		self.run_command('build_py')
> +
> +		print('Building API documentation...')
> +
> +		process_env = os.environ.copy()
> +		pythonpath = self.build_lib
> +		try:
> +			pythonpath += ':' + process_env['PYTHONPATH']
> +		except KeyError:
> +			pass
> +		process_env['PYTHONPATH'] = pythonpath
> +
> +		subprocess.check_call(['epydoc', '-o', 'epydoc',
> +			'--name', self.distribution.get_name(),
> +			'--url', self.distribution.get_url(),
> +			'-qq', '--no-frames', '--show-imports',
> +			'--exclude', 'portage.tests',
> +			'_emerge', 'portage', 'repoman'],
> +			env = process_env)
> +		os.remove('epydoc/api-objects.txt')
> +
> +
> +class install_docbook(install_data):
> +	""" install_data for docbook docs """
> +
> +	user_options = install_data.user_options
> +
> +	def initialize_options(self):
> +		install_data.initialize_options(self)
> +		self.htmldir = None
> +
> +	def finalize_options(self):
> +		self.set_undefined_options('install', ('htmldir',
> 'htmldir'))
> +		install_data.finalize_options(self)
> +
> +	def run(self):
> +		if not os.path.exists('doc/portage.html'):
> +			self.run_command('docbook')
> +		self.data_files = [
> +			(self.htmldir, glob.glob('doc/*.html')),
> +		]
> +		install_data.run(self)
> +
> +
> +class install_epydoc(install_data):
> +	""" install_data for epydoc docs """
> +
> +	user_options = install_data.user_options
> +
> +	def initialize_options(self):
> +		install_data.initialize_options(self)
> +		self.htmldir = None
> +
> +	def finalize_options(self):
> +		self.set_undefined_options('install', ('htmldir',
> 'htmldir'))
> +		install_data.finalize_options(self)
> +
> +	def run(self):
> +		if not os.path.exists('epydoc/index.html'):
> +			self.run_command('epydoc')
> +		self.data_files = [
> +			(os.path.join(self.htmldir, 'api'),
> glob.glob('epydoc/*')),
> +		]
> +		install_data.run(self)
> +
> +
> +class x_build_scripts_custom(build_scripts):
> +	def finalize_options(self):
> +		build_scripts.finalize_options(self)
> +		if 'dir_name' in dir(self):
> +			self.build_dir =
> os.path.join(self.build_dir, self.dir_name)
> +			if self.dir_name in x_scripts:
> +				self.scripts =
> x_scripts[self.dir_name]
> +			else:
> +				self.scripts = set(self.scripts)
> +				for other_files in
> x_scripts.values():
> +
> self.scripts.difference_update(other_files) +
> +	def run(self):
> +		# group scripts by subdirectory
> +		split_scripts = collections.defaultdict(list)
> +		for f in self.scripts:
> +			dir_name = os.path.dirname(f[len('bin/'):])
> +			split_scripts[dir_name].append(f)
> +
> +		base_dir = self.build_dir
> +		base_scripts = self.scripts
> +		for d, files in split_scripts.items():
> +			self.build_dir = os.path.join(base_dir, d)
> +			self.scripts = files
> +			self.copy_scripts()
> +
> +		# restore previous values
> +		self.build_dir = base_dir
> +		self.scripts = base_scripts
> +
> +
> +class x_build_scripts_bin(x_build_scripts_custom):
> +	dir_name = 'bin'
> +
> +
> +class x_build_scripts_sbin(x_build_scripts_custom):
> +	dir_name = 'sbin'
> +
> +
> +class x_build_scripts_portagebin(x_build_scripts_custom):
> +	dir_name = 'portage'
> +
> +
> +class x_build_scripts(build_scripts):
> +	def initialize_option(self):
> +		build_scripts.initialize_options(self)
> +
> +	def finalize_options(self):
> +		build_scripts.finalize_options(self)
> +
> +	def run(self):
> +		self.run_command('build_scripts_bin')
> +		self.run_command('build_scripts_portagebin')
> +		self.run_command('build_scripts_sbin')
> +
> +
> +class x_clean(clean):
> +	""" clean extended for doc & post-test cleaning """
> +
> +	def clean_docs(self):
> +		def get_doc_outfiles():
> +			for dirpath, dirnames, filenames in
> os.walk('doc'):
> +				for f in filenames:
> +					if f.endswith('.docbook') or
> f == 'custom.xsl':
> +						pass
> +					else:
> +						yield
> os.path.join(dirpath, f) +
> +				# do not recurse
> +				break
> +
> +
> +		for f in get_doc_outfiles():
> +			print('removing %s' % repr(f))
> +			os.remove(f)
> +
> +		if os.path.isdir('epydoc'):
> +			remove_tree('epydoc')
> +
> +	def clean_tests(self):
> +		# do not remove incorrect dirs accidentally
> +		top_dir =
> os.path.normpath(os.path.join(self.build_lib, '..'))
> +		cprefix = os.path.commonprefix((self.build_base,
> top_dir))
> +		if cprefix != self.build_base:
> +			return
> +
> +		bin_dir = os.path.join(top_dir, 'bin')
> +		if os.path.exists(bin_dir):
> +			remove_tree(bin_dir)
> +
> +		conf_dir = os.path.join(top_dir, 'cnf')
> +		if os.path.islink(conf_dir):
> +			print('removing %s symlink' % repr(conf_dir))
> +			os.unlink(conf_dir)
> +
> +		pni_file = os.path.join(top_dir,
> '.portage_not_installed')
> +		if os.path.exists(pni_file):
> +			print('removing %s' % repr(pni_file))
> +			os.unlink(pni_file)
> +
> +	def run(self):
> +		if self.all:
> +			self.clean_tests()
> +			self.clean_docs()
> +
> +		clean.run(self)
> +
> +
> +class x_install(install):
> +	""" install command with extra Portage paths """
> +
> +	user_options = install.user_options + [
> +		# note: $prefix and $exec_prefix are reserved for
> Python install
> +		('system-prefix=', None, "Prefix for
> architecture-independent data"),
> +		('system-exec-prefix=', None, "Prefix for
> architecture-specific data"), +
> +		('bindir=', None, "Install directory for main
> executables"),
> +		('datarootdir=', None, "Data install root
> directory"),
> +		('docdir=', None, "Documentation install directory"),
> +		('htmldir=', None, "HTML documentation install
> directory"),
> +		('mandir=', None, "Manpage root install directory"),
> +		('portage-base=', 'b', "Portage install base"),
> +		('portage-bindir=', None, "Install directory for
> Portage internal-use executables"),
> +		('portage-datadir=', None, 'Install directory for
> data files'),
> +		('sbindir=', None, "Install directory for
> superuser-intended executables"),
> +		('sysconfdir=', None, 'System configuration path'),
> +	]
> +
> +	# note: the order is important for proper substitution
> +	paths = [
> +		('system_prefix', '/usr'),
> +		('system_exec_prefix', '$system_prefix'),
> +
> +		('bindir', '$system_exec_prefix/bin'),
> +		('sbindir', '$system_exec_prefix/sbin'),
> +		('sysconfdir', '/etc'),
> +
> +		('datarootdir', '$system_prefix/share'),
> +		('docdir', '$datarootdir/doc/$package-$version'),
> +		('htmldir', '$docdir/html'),
> +		('mandir', '$datarootdir/man'),
> +
> +		('portage_base', '$system_exec_prefix/lib/portage'),
> +		('portage_bindir', '$portage_base/bin'),
> +		('portage_datadir', '$datarootdir/portage'),
> +
> +		# not customized at the moment
> +		('logrotatedir', '$sysconfdir/logrotate'),
> +		('portage_confdir', '$portage_datadir/config'),
> +		('portage_setsdir', '$portage_confdir/sets'),
> +	]
> +
> +	def initialize_options(self):
> +		install.initialize_options(self)
> +
> +		for key, default in self.paths:
> +			setattr(self, key, default)
> +		self.subst_paths = {}
> +
> +	def finalize_options(self):
> +		install.finalize_options(self)
> +
> +		# substitute variables
> +		new_paths = {
> +			'package': self.distribution.get_name(),
> +			'version': self.distribution.get_version(),
> +		}
> +		for key, default in self.paths:
> +			new_paths[key] = subst_vars(getattr(self,
> key), new_paths)
> +			setattr(self, key, new_paths[key])
> +		self.subst_paths = new_paths
> +
> +
> +class x_install_data(install_data):
> +	""" install_data with customized path support """
> +
> +	user_options = install_data.user_options
> +
> +	def initialize_options(self):
> +		install_data.initialize_options(self)
> +		self.paths = None
> +
> +	def finalize_options(self):
> +		install_data.finalize_options(self)
> +		self.set_undefined_options('install',
> +			('subst_paths', 'paths'))
> +
> +		# substitute variables in data_files
> +		for f in self.data_files:
> +			f[0] = subst_vars(f[0], self.paths)
> +
> +
> +class x_install_lib(install_lib):
> +	""" install_lib command with Portage path substitution """
> +
> +	user_options = install_lib.user_options
> +
> +	def initialize_options(self):
> +		install_lib.initialize_options(self)
> +		self.portage_base = None
> +		self.portage_bindir = None
> +		self.portage_confdir = None
> +
> +	def finalize_options(self):
> +		install_lib.finalize_options(self)
> +		self.set_undefined_options('install',
> +			('portage_base', 'portage_base'),
> +			('portage_bindir', 'portage_bindir'),
> +			('portage_confdir', 'portage_confdir'))
> +
> +	def install(self):
> +		ret = install_lib.install(self)
> +
> +		def rewrite_file(path, val_dict):
> +			path = os.path.join(self.install_dir, path)
> +			print('Rewriting %s' % path)
> +			with codecs.open(path, 'r', 'utf-8') as f:
> +				data = f.read()
> +
> +			for varname, val in val_dict.items():
> +				regexp = r'^(%s\s*=).*$' % varname
> +				repl = r'\1 %s' % repr(val)
> +
> +				data = re.sub(regexp, repl, data, 0,
> re.MULTILINE) +
> +			with codecs.open(path, 'w', 'utf-8') as f:
> +				f.write(data)
> +
> +		rewrite_file('portage/__init__.py', {
> +			'VERSION': self.distribution.get_version(),
> +		})
> +		rewrite_file('portage/const.py', {
> +			'PORTAGE_BASE_PATH': self.portage_base,
> +			'PORTAGE_BIN_PATH': self.portage_bindir,
> +			'PORTAGE_CONFIG_PATH': self.portage_confdir,
> +		})
> +
> +		return ret
> +
> +
> +class x_install_scripts_custom(install_scripts):
> +	def initialize_options(self):
> +		install_scripts.initialize_options(self)
> +		self.root = None
> +
> +	def finalize_options(self):
> +		self.set_undefined_options('install',
> +			('root', 'root'),
> +			(self.var_name, 'install_dir'))
> +		install_scripts.finalize_options(self)
> +		self.build_dir = os.path.join(self.build_dir,
> self.dir_name) +
> +		# prepend root
> +		if self.root is not None:
> +			self.install_dir = change_root(self.root,
> self.install_dir) +
> +
> +class x_install_scripts_bin(x_install_scripts_custom):
> +	dir_name = 'bin'
> +	var_name = 'bindir'
> +
> +
> +class x_install_scripts_sbin(x_install_scripts_custom):
> +	dir_name = 'sbin'
> +	var_name = 'sbindir'
> +
> +
> +class x_install_scripts_portagebin(x_install_scripts_custom):
> +	dir_name = 'portage'
> +	var_name = 'portage_bindir'
> +
> +
> +class x_install_scripts(install_scripts):
> +	def initialize_option(self):
> +		pass
> +
> +	def finalize_options(self):
> +		pass
> +
> +	def run(self):
> +		self.run_command('install_scripts_bin')
> +		self.run_command('install_scripts_portagebin')
> +		self.run_command('install_scripts_sbin')
> +
> +
> +class x_sdist(sdist):
> +	""" sdist defaulting to .tar.bz2 format """
> +
> +	def finalize_options(self):
> +		if self.formats is None:
> +			self.formats = ['bztar']
> +
> +		sdist.finalize_options(self)
> +
> +
> +class build_tests(x_build_scripts_custom):
> +	""" Prepare build dir for running tests. """
> +
> +	def initialize_options(self):
> +		x_build_scripts_custom.initialize_options(self)
> +		self.build_base = None
> +		self.build_lib = None
> +
> +	def finalize_options(self):
> +		x_build_scripts_custom.finalize_options(self)
> +		self.set_undefined_options('build',
> +			('build_base', 'build_base'),
> +			('build_lib', 'build_lib'))
> +
> +		# since we will be writing to $build_lib/.., it is
> important
> +		# that we do not leave $build_base
> +		self.top_dir =
> os.path.normpath(os.path.join(self.build_lib, '..'))
> +		cprefix = os.path.commonprefix((self.build_base,
> self.top_dir))
> +		if cprefix != self.build_base:
> +			raise SystemError('build_lib must be a
> subdirectory of build_base') +
> +		self.build_dir = os.path.join(self.top_dir, 'bin')
> +
> +	def run(self):
> +		self.run_command('build_py')
> +
> +		# install all scripts $build_lib/../bin
> +		# (we can't do a symlink since we want shebangs
> corrected)
> +		x_build_scripts_custom.run(self)
> +
> +		# symlink 'cnf' directory
> +		conf_dir = os.path.join(self.top_dir, 'cnf')
> +		if os.path.exists(conf_dir):
> +			if not os.path.islink(conf_dir):
> +				raise SystemError('%s exists and is
> not a symlink (collision)'
> +					% repr(conf_dir))
> +			os.unlink(conf_dir)
> +		conf_src = os.path.relpath('cnf', self.top_dir)
> +		print('Symlinking %s -> %s' % (conf_dir, conf_src))
> +		os.symlink(conf_src, conf_dir)
> +
> +		# create $build_lib/../.portage_not_installed
> +		# to enable proper paths in tests
> +		with open(os.path.join(self.top_dir,
> '.portage_not_installed'), 'w') as f:
> +			pass
> +
> +
> +class test(Command):
> +	""" run tests """
> +
> +	user_options = []
> +
> +	def initialize_options(self):
> +		self.build_lib = None
> +
> +	def finalize_options(self):
> +		self.set_undefined_options('build',
> +			('build_lib', 'build_lib'))
> +
> +	def run(self):
> +		self.run_command('build_tests')
> +		subprocess.check_call([
> +			sys.executable, '-bWd',
> +			os.path.join(self.build_lib,
> 'portage/tests/runTests.py')
> +		])
> +
> +
> +def find_packages():
> +	for dirpath, dirnames, filenames in os.walk('pym'):
> +		if '__init__.py' in filenames:
> +			yield os.path.relpath(dirpath, 'pym')
> +
> +
> +def find_scripts():
> +	for dirpath, dirnames, filenames in os.walk('bin'):
> +		for f in filenames:
> +			yield os.path.join(dirpath, f)
> +
> +
> +def get_manpages():
> +	linguas = os.environ.get('LINGUAS')
> +	if linguas is not None:
> +		linguas = linguas.split()
> +
> +	for dirpath, dirnames, filenames in os.walk('man'):
> +		groups = collections.defaultdict(list)
> +		for f in filenames:
> +			fn, suffix = f.rsplit('.', 1)
> +			groups[suffix].append(os.path.join(dirpath,
> f)) +
> +		topdir = dirpath[len('man/'):]
> +		if not topdir or linguas is None or topdir in
> linguas:
> +			for g, mans in groups.items():
> +				yield [os.path.join('$mandir',
> topdir, 'man%s' % g), mans] +
> +setup(
> +	name = 'portage',
> +	version = '2.2.12',
> +	url = 'https://wiki.gentoo.org/wiki/Project:Portage',
> +	author = 'Gentoo Portage Development Team',
> +	author_email = 'dev-portage@gentoo.org',
> +
> +	package_dir = {'': 'pym'},
> +	packages = list(find_packages()),
> +	# something to cheat build & install commands
> +	scripts = list(find_scripts()),
> +
> +	data_files = list(get_manpages()) + [
> +		['$sysconfdir', ['cnf/etc-update.conf',
> 'cnf/dispatch-conf.conf']],
> +		['$logrotatedir',
> ['cnf/logrotate.d/elog-save-summary']],
> +		['$portage_confdir', [
> +			'cnf/make.conf.example', 'cnf/make.globals',
> 'cnf/repos.conf']],
> +		['$portage_setsdir', ['cnf/sets/portage.conf']],
> +		['$docdir', ['NEWS', 'RELEASE-NOTES']],
> +	],
> +
> +	cmdclass = {
> +		'build_scripts': x_build_scripts,
> +		'build_scripts_bin': x_build_scripts_bin,
> +		'build_scripts_portagebin':
> x_build_scripts_portagebin,
> +		'build_scripts_sbin': x_build_scripts_sbin,
> +		'build_tests': build_tests,
> +		'clean': x_clean,
> +		'docbook': docbook,
> +		'epydoc': epydoc,
> +		'install': x_install,
> +		'install_data': x_install_data,
> +		'install_docbook': install_docbook,
> +		'install_epydoc': install_epydoc,
> +		'install_lib': x_install_lib,
> +		'install_scripts': x_install_scripts,
> +		'install_scripts_bin': x_install_scripts_bin,
> +		'install_scripts_portagebin':
> x_install_scripts_portagebin,
> +		'install_scripts_sbin': x_install_scripts_sbin,
> +		'sdist': x_sdist,
> +		'test': test,
> +	},
> +
> +	classifiers = [
> +		'Development Status :: 5 - Production/Stable',
> +		'Environment :: Console',
> +		'Intended Audience :: System Administrators',
> +		'License :: OSI Approved :: GNU General Public
> License v2 (GPLv2)',
> +		'Operating System :: POSIX',
> +		'Programming Language :: Python',
> +		'Topic :: System :: Installation/Setup'
> +	]
> +)



-- 
Brian Dolbec <dolsen>



^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [gentoo-portage-dev] [PATCHES] setup.py install for Portage
  2014-08-21 20:19 [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny
                   ` (7 preceding siblings ...)
  2014-08-22 22:32 ` Brian Dolbec
@ 2014-08-24 19:42 ` Brian Dolbec
  2014-08-24 21:19   ` Michał Górny
  2014-09-12  7:17 ` Michał Górny
  9 siblings, 1 reply; 23+ messages in thread
From: Brian Dolbec @ 2014-08-24 19:42 UTC (permalink / raw
  To: gentoo-portage-dev

On Thu, 21 Aug 2014 22:19:38 +0200
Michał Górny <mgorny@gentoo.org> wrote:


> In particular, tools that want to play with Portage internals need to
> export proper PORTAGE_BIN_PATH and PORTAGE_PYM_PATH before spawning
> them. The correct values can be obtained using portageq:
> 
>   $ portageq envvar PORTAGE_BIN_PATH
>   /usr/lib/portage/python3.4
>   $ portageq envvar PORTAGE_PYM_PATH
>   /usr/lib64/python3.4/site-packages
> 
> 
> Please look through the patches and test at will :). Thanks.
> 
> --
> Michał Górny
> 
> 

There is breakage in catalyst runs.  

[08:26] <iamben> has anyone tried mgorny's new portage ebuild? after
portage is emerged for stage1, it's missing /usr/lib/portage/bin/
altogether, for me

I did a search thru the catalyst code.  It is hard-coded in only 1
place:

lines 35, 37 in stage1-controller.sh

code snipit:

	clean)
		# Clean out man, info and doc files
		rm -rf usr/share/{man,doc,info}/*
		# Zap all .pyc and .pyo files
		find . -iname "*.py[co]" -exec rm -f {} \;
		# Cleanup all .a files except libgcc.a, *_nonshared.a and 
                # /usr/lib/portage/bin/*.a
		find . -type f -iname "*.a" | grep -v 'libgcc.a' | \
			grep -v 'nonshared.a' | grep -v	'/usr/lib/portage/bin/' | \ 
                        grep -v 'libgcc_eh.a' | xargs rm -f
	;;

	*)

Michał can you supply me a patch to fix this in catalyst and I will apply it.

I haven't started testing myself yet.
-- 
Brian Dolbec <dolsen>



^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [gentoo-portage-dev] [PATCHES] setup.py install for Portage
  2014-08-24 19:42 ` Brian Dolbec
@ 2014-08-24 21:19   ` Michał Górny
  0 siblings, 0 replies; 23+ messages in thread
From: Michał Górny @ 2014-08-24 21:19 UTC (permalink / raw
  To: Brian Dolbec; +Cc: gentoo-portage-dev

[-- Attachment #1: Type: text/plain, Size: 1348 bytes --]

Dnia 2014-08-24, o godz. 12:42:02
Brian Dolbec <dolsen@gentoo.org> napisał(a):

> There is breakage in catalyst runs.  
> 
> [08:26] <iamben> has anyone tried mgorny's new portage ebuild? after
> portage is emerged for stage1, it's missing /usr/lib/portage/bin/
> altogether, for me

Well, that directory is no longer used, so it's fine missing as long
as executables are where they're supposed to be ;).

> I did a search thru the catalyst code.  It is hard-coded in only 1
> place:
> 
> lines 35, 37 in stage1-controller.sh
> 
> code snipit:
> 
> 	clean)
> 		# Clean out man, info and doc files
> 		rm -rf usr/share/{man,doc,info}/*
> 		# Zap all .pyc and .pyo files
> 		find . -iname "*.py[co]" -exec rm -f {} \;
> 		# Cleanup all .a files except libgcc.a, *_nonshared.a and 
>                 # /usr/lib/portage/bin/*.a
> 		find . -type f -iname "*.a" | grep -v 'libgcc.a' | \
> 			grep -v 'nonshared.a' | grep -v	'/usr/lib/portage/bin/' | \ 
>                         grep -v 'libgcc_eh.a' | xargs rm -f
> 	;;
> 
> 	*)
> 
> Michał can you supply me a patch to fix this in catalyst and I will apply it.

It should be enough to change that path to '/usr/lib/portage/'. Though
it would generally be better to pass additional tests to find rather
than playing with grep...

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [gentoo-portage-dev] [PATCH v2] Install Portage using setup.py
  2014-08-23 20:30     ` [gentoo-portage-dev] [PATCH v2] Install Portage " Michał Górny
  2014-08-23 21:56       ` Brian Dolbec
@ 2014-08-29 17:08       ` Brian Dolbec
  2014-08-29 23:30         ` Brian Dolbec
  2014-09-05  6:58       ` [gentoo-portage-dev] [PATCH v3] " Michał Górny
  2 siblings, 1 reply; 23+ messages in thread
From: Brian Dolbec @ 2014-08-29 17:08 UTC (permalink / raw
  To: gentoo-portage-dev

[-- Attachment #1: Type: text/plain, Size: 5472 bytes --]

On Sat, 23 Aug 2014 22:30:56 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> ---
>  .gitignore           |   1 +
>  MANIFEST.in          |  19 ++
>  Makefile             | 215 -------------------
>  doc/Makefile         |  11 -
>  mkrelease.sh         | 141 -------------
>  pym/portage/const.py |   4 +-
>  setup.py             | 578
> +++++++++++++++++++++++++++++++++++++++++++++++++++ 7 files changed,
> 600 insertions(+), 369 deletions(-) create mode 100644 MANIFEST.in
>  delete mode 100644 Makefile
>  delete mode 100644 doc/Makefile
>  delete mode 100755 mkrelease.sh
>  create mode 100755 setup.py
> 
>
> FIXME: not used diff --git a/setup.py b/setup.py new file mode 100755
> index 0000000..7ffe7f0 --- /dev/null
> +++ b/setup.py

...

> +
> +class docbook(Command):
> +	""" Build docs using docbook. """
> +
> +	user_options = [
> +		('doc-formats=', None, 'Documentation formats to
> build (all xmlto formats for docbook are allowed, comma-separated'),
> +	]
> +
> +	def initialize_options(self):
> +		self.doc_formats = 'xhtml,xhtml-nochunks'
> +
> +	def finalize_options(self):
> +		self.doc_formats = self.doc_formats.replace(',', '
> ').split() +
> +	def run(self):
> +		with open('doc/fragment/date', 'w'):
> +			pass
> +
> +		for f in self.doc_formats:
> +			print('Building docs in %s format...' % f)
> +			subprocess.check_call(['xmlto', '-o', 'doc',
> +				'-m', 'doc/custom.xsl', f,
> 'doc/portage.docbook']) +
> +
> +class epydoc(Command):
> +	""" Build API docs using epydoc. """
> +
> +	user_options = [
> +	]
> +
> +	def initialize_options(self):
> +		self.build_lib = None
> +
> +	def finalize_options(self):
> +		self.set_undefined_options('build_py', ('build_lib',
> 'build_lib')) +
> +	def run(self):
> +		self.run_command('build_py')
> +
> +		print('Building API documentation...')
> +
> +		process_env = os.environ.copy()
> +		pythonpath = self.build_lib
> +		try:
> +			pythonpath += ':' + process_env['PYTHONPATH']
> +		except KeyError:
> +			pass
> +		process_env['PYTHONPATH'] = pythonpath
> +
> +		subprocess.check_call(['epydoc', '-o', 'epydoc',
> +			'--name', self.distribution.get_name(),
> +			'--url', self.distribution.get_url(),
> +			'-qq', '--no-frames', '--show-imports',
> +			'--exclude', 'portage.tests',
> +			'_emerge', 'portage', 'repoman'],
> +			env = process_env)
> +		os.remove('epydoc/api-objects.txt')
> +
> +
> +class install_docbook(install_data):
> +	""" install_data for docbook docs """
> +
> +	user_options = install_data.user_options
> +
> +	def initialize_options(self):
> +		install_data.initialize_options(self)
> +		self.htmldir = None
> +
> +	def finalize_options(self):
> +		self.set_undefined_options('install', ('htmldir',
> 'htmldir'))
> +		install_data.finalize_options(self)
> +
> +	def run(self):
> +		if not os.path.exists('doc/portage.html'):
> +			self.run_command('docbook')
> +		self.data_files = [
> +			(self.htmldir, glob.glob('doc/*.html')),
> +		]
> +		install_data.run(self)
> +
> +
> +class install_epydoc(install_data):
> +	""" install_data for epydoc docs """
> +
> +	user_options = install_data.user_options
> +
> +	def initialize_options(self):
> +		install_data.initialize_options(self)
> +		self.htmldir = None
> +
> +	def finalize_options(self):
> +		self.set_undefined_options('install', ('htmldir',
> 'htmldir'))
> +		install_data.finalize_options(self)
> +
> +	def run(self):
> +		if not os.path.exists('epydoc/index.html'):
> +			self.run_command('epydoc')
> +		self.data_files = [
> +			(os.path.join(self.htmldir, 'api'),
> glob.glob('epydoc/*')),
> +		]
> +		install_data.run(self)
> +
> +
> 

Both doc and epydoc use flags, install is broken and fails with
permission denied error to create the /usr/share/.../portage-9999
directory.

...

> +setup(
> +	name = 'portage',
> +	version = '2.2.12',

You've changed it to 2.2.13_pre1 in git.  But it produces incorrect
portage-2.2.13_pre1.egginfo version for an otherwise tagged/versioned
9999 install.

So set this to 9999 in git and reset to the correct value for a release
only it looks like.

> +	url = 'https://wiki.gentoo.org/wiki/Project:Portage',
> +	author = 'Gentoo Portage Development Team',
> +	author_email = 'dev-portage@gentoo.org',
> +
...


Also, setup.py sdist does not produce an installable tarball.  It keeps
failing with unicode decode errors on various files on the different
target pythons.

eg:
copying and adjusting bin/fixpackages
-> /var/tmp/portage/sys-apps/portage-2.2.13_pre2/work/portage-2.2.13_pre2-python2_7/scripts/sbin/
Traceback (most recent call last): File
"/usr/lib64/python3.2/tokenize.py", line 298, in find_cookie
line_string = line.decode('utf-8') UnicodeDecodeError: 'utf-8' codec
can't decode byte 0xee in position 0: invalid continuation byte

-------------------------------

I have changed the version to 9999 and rebased your new-install branch
on currrent master, and pushed it to my github account.

https://github.com/dol-sen/portage

I also am attaching the ebuilds I edited slightly

I have uploaded the 2.2.13_pre1 and _pre2 tarballs to my dev space.
_pre2 differs from _pre1 in that _pre2 is with the rebased git sources,
while _pre1 is Michal's original branch code.

you can rename the _pre2.ebuild to _pre1.ebuild to test at your
pleasure.

-- 
Brian Dolbec <dolsen>


[-- Attachment #2: portage-9999.ebuild --]
[-- Type: application/octet-stream, Size: 7201 bytes --]

# Copyright 1999-2014 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo-x86/sys-apps/portage/portage-9999.ebuild,v 1.99 2014/08/05 14:19:02 dolsen Exp $

EAPI=5

PYTHON_COMPAT=(
	pypy
	python3_2 python3_3 python3_4
	python2_7
)
PYTHON_REQ_USE='ssl(+),bzip2(+)'

inherit distutils-r1 git-r3 multilib

DESCRIPTION="Portage is the package management and distribution system for Gentoo"
HOMEPAGE="http://www.gentoo.org/proj/en/portage/index.xml"

LICENSE="GPL-2"
KEYWORDS=""
SLOT="0"
IUSE="build doc epydoc +ipc linguas_ru selinux xattr"

DEPEND="${python_dep}
	>=sys-apps/sed-4.0.5 sys-devel/patch
	doc? ( app-text/xmlto ~app-text/docbook-xml-dtd-4.4 )
	epydoc? ( >=dev-python/epydoc-2.0[$(python_gen_usedep 'python2*')] )"
# Require sandbox-2.2 for bug #288863.
# For xattr, we can spawn getfattr and setfattr from sys-apps/attr, but that's
# quite slow, so it's not considered in the dependencies as an alternative to
# to python-3.3 / pyxattr. Also, xattr support is only tested with Linux, so
# for now, don't pull in xattr deps for other kernels.
# For whirlpool hash, require python[ssl] or python-mhash (bug #425046).
# For compgen, require bash[readline] (bug #445576).
RDEPEND="${python_dep}
	!build? ( >=sys-apps/sed-4.0.5
		|| ( >=app-shells/bash-4.2_p37[readline] ( <app-shells/bash-4.2_p37 >=app-shells/bash-3.2_p17 ) )
		>=app-admin/eselect-1.2
	)
	elibc_FreeBSD? ( sys-freebsd/freebsd-bin )
	elibc_glibc? ( >=sys-apps/sandbox-2.2 )
	elibc_uclibc? ( >=sys-apps/sandbox-2.2 )
	>=app-misc/pax-utils-0.1.17
	selinux? ( >=sys-libs/libselinux-2.0.94[python,${PYTHON_USEDEP}] )
	xattr? ( kernel_linux? (
		>=sys-apps/install-xattr-0.3
		$(python_gen_cond_dep 'dev-python/pyxattr[${PYTHON_USEDEP}]' \
			python{2_7,3_2} pypy)
	) )
	!<app-shells/bash-3.2_p17
	!<app-admin/logrotate-3.8.0"
PDEPEND="
	!build? (
		>=net-misc/rsync-2.6.4
		userland_GNU? ( >=sys-apps/coreutils-6.4 )
	)"
# coreutils-6.4 rdep is for date format in emerge-webrsync #164532
# NOTE: FEATURES=installsources requires debugedit and rsync

REQUIRED_USE="epydoc? ( $(python_gen_useflags 'python2*') )"

SRC_ARCHIVES="http://dev.gentoo.org/~dolsen/releases/portage"

prefix_src_archives() {
	local x y
	for x in ${@}; do
		for y in ${SRC_ARCHIVES}; do
			echo ${y}/${x}
		done
	done
}

EGIT_REPO_URI="https://github.com/dol-sen/portage.git"
EGIT_BRANCH=new-install
EGIT_MIN_CLONE_TYPE=single

python_prepare_all() {
	distutils-r1_python_prepare_all

	einfo "Producing ChangeLog from Git history..."
	git log ebcf8975b37a8aae9735eb491a9b4cb63549bd5d^.. \
		> "${S}"/ChangeLog || die

	local _version=$(git describe --tags | sed -e 's|-\([0-9]\+\)-.\+$|_p\1|')
	_version=${_version:1}
	einfo "Setting portage.VERSION to ${_version} ..."
	sed -e "s/^VERSION =.*/VERSION = '${_version}'/" -i pym/portage/__init__.py || \
		die "Failed to patch portage.VERSION"
	sed -e "1s/VERSION/${_version}/" -i doc/fragment/version || \
		die "Failed to patch VERSION in doc/fragment/version"
	sed -e "1s/VERSION/${_version}/" -i $(find man -type f) || \
		die "Failed to patch VERSION in man page headers"

	if ! use ipc ; then
		einfo "Disabling ipc..."
		sed -e "s:_enable_ipc_daemon = True:_enable_ipc_daemon = False:" \
			-i pym/_emerge/AbstractEbuildProcess.py || \
			die "failed to patch AbstractEbuildProcess.py"
	fi

	if use xattr && use kernel_linux ; then
		einfo "Adding FEATURES=xattr to make.globals ..."
		echo -e '\nFEATURES="${FEATURES} xattr"' >> cnf/make.globals \
			|| die "failed to append to make.globals"
	fi

	if [[ -n ${EPREFIX} ]] ; then
		einfo "Setting portage.const.EPREFIX ..."
		sed -e "s|^\(SANDBOX_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/bin/sandbox\"\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(FAKEROOT_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/bin/fakeroot\"\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(BASH_BINARY[[:space:]]*=[[:space:]]*\"\)\(/bin/bash\"\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(MOVE_BINARY[[:space:]]*=[[:space:]]*\"\)\(/bin/mv\"\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(PRELINK_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/sbin/prelink\"\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(EPREFIX[[:space:]]*=[[:space:]]*\"\).*|\\1${EPREFIX}\"|" \
			-i pym/portage/const.py || \
			die "Failed to patch portage.const.EPREFIX"

		einfo "Prefixing shebangs ..."
		while read -r -d $'\0' ; do
			local shebang=$(head -n1 "$REPLY")
			if [[ ${shebang} == "#!"* && ! ${shebang} == "#!${EPREFIX}/"* ]] ; then
				sed -i -e "1s:.*:#!${EPREFIX}${shebang:2}:" "$REPLY" || \
					die "sed failed"
			fi
		done < <(find . -type f -print0)

		einfo "Adjusting make.globals ..."
		sed -e "s|\(/usr/portage\)|${EPREFIX}\\1|" \
			-e "s|^\(PORTAGE_TMPDIR=\"\)\(/var/tmp\"\)|\\1${EPREFIX}\\2|" \
			-i cnf/make.globals || die "sed failed"

		einfo "Adjusting repos.conf ..."
		sed -e "s|^\(main-repo = \).*|\\1gentoo_prefix|" \
			-e "s|^\\[gentoo\\]|[gentoo_prefix]|" \
			-e "s|^\(location = \)\(/usr/portage\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(sync-uri = \).*|\\1rsync://prefix.gentooexperimental.org/gentoo-portage-prefix|" \
			-i cnf/repos.conf || die "sed failed"

		einfo "Adding FEATURES=force-prefix to make.globals ..."
		echo -e '\nFEATURES="${FEATURES} force-prefix"' >> cnf/make.globals \
			|| die "failed to append to make.globals"
	fi

	cd "${S}/cnf" || die
	if [ -f "make.conf.example.${ARCH}".diff ]; then
		patch make.conf.example "make.conf.example.${ARCH}".diff || \
			die "Failed to patch make.conf.example"
	else
		eerror ""
		eerror "Portage does not have an arch-specific configuration for this arch."
		eerror "Please notify the arch maintainer about this issue. Using generic."
		eerror ""
	fi
}

python_compile_all() {
	local targets=()
	use doc && targets+=( docbook )
	use epydoc && targets+=( epydoc )

	if [[ ${targets[@]} ]]; then
		esetup.py "${targets[@]}"
	fi
}

python_test() {
	esetup.py test
}

python_install() {
	distutils-r1_python_install \
		--system-prefix="${EPREFIX}/usr" \
		--bindir="$(python_get_scriptdir)" \
		--docdir="${EPREFIX}/usr/share/doc/${PF}" \
		--htmldir="${EPREFIX}/usr/share/doc/${PF}/html" \
		--portage-bindir="${EPREFIX}/usr/lib/portage/${EPYTHON}" \
		--sbindir="$(python_get_scriptdir)" \
		--sysconfdir="${EPREFIX}/etc" \
		"${@}"
}

python_install_all() {
	distutils-r1_python_install_all

	local targets=()
	use doc && targets+=( install_docbook )
	use epydoc && targets+=( install_epydoc )

	# install docs
	if [[ ${targets[@]} ]]; then
		esetup.py "${targets[@]}"
	fi
}

pkg_preinst() {
	if [[ $ROOT == / ]] ; then
		# Run some minimal tests as a sanity check.
		local test_runner=$(find "${ED}" -name runTests)
		if [[ -n $test_runner && -x $test_runner ]] ; then
			einfo "Running preinst sanity tests..."
			"$test_runner" || die "preinst sanity tests failed"
		fi
	fi

	# elog dir must exist to avoid logrotate error for bug #415911.
	# This code runs in preinst in order to bypass the mapping of
	# portage:portage to root:root which happens after src_install.
	keepdir /var/log/portage/elog
	# This is allowed to fail if the user/group are invalid for prefix users.
	if chown portage:portage "${ED}"var/log/portage{,/elog} 2>/dev/null ; then
		chmod g+s,ug+rwx "${ED}"var/log/portage{,/elog}
	fi
}

[-- Attachment #3: portage-2.2.13_pre2.ebuild --]
[-- Type: application/octet-stream, Size: 7118 bytes --]

# Copyright 1999-2014 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo-x86/sys-apps/portage/portage-9999.ebuild,v 1.99 2014/08/05 14:19:02 dolsen Exp $

EAPI=5

PYTHON_COMPAT=(
	pypy
	python3_2 python3_3 python3_4
	python2_7
)
PYTHON_REQ_USE='ssl(+),bzip2(+)'

inherit distutils-r1 multilib

DESCRIPTION="Portage is the package management and distribution system for Gentoo"
HOMEPAGE="http://www.gentoo.org/proj/en/portage/index.xml"

LICENSE="GPL-2"
KEYWORDS="~alpha ~amd64 ~arm ~arm64 ~hppa ~ia64 ~m68k ~mips ~ppc ~ppc64 ~s390 ~sh ~sparc ~x86 ~amd64-fbsd ~sparc-fbsd ~x86-fbsd"
SLOT="0"
IUSE="build doc epydoc +ipc linguas_ru selinux xattr"

DEPEND="${python_dep}
	>=sys-apps/sed-4.0.5 sys-devel/patch
	doc? ( app-text/xmlto ~app-text/docbook-xml-dtd-4.4 )
	epydoc? ( >=dev-python/epydoc-2.0[$(python_gen_usedep 'python2*')] )"
# Require sandbox-2.2 for bug #288863.
# For xattr, we can spawn getfattr and setfattr from sys-apps/attr, but that's
# quite slow, so it's not considered in the dependencies as an alternative to
# to python-3.3 / pyxattr. Also, xattr support is only tested with Linux, so
# for now, don't pull in xattr deps for other kernels.
# For whirlpool hash, require python[ssl] or python-mhash (bug #425046).
# For compgen, require bash[readline] (bug #445576).
RDEPEND="${python_dep}
	!build? ( >=sys-apps/sed-4.0.5
		|| ( >=app-shells/bash-4.2_p37[readline] ( <app-shells/bash-4.2_p37 >=app-shells/bash-3.2_p17 ) )
		>=app-admin/eselect-1.2
	)
	elibc_FreeBSD? ( sys-freebsd/freebsd-bin )
	elibc_glibc? ( >=sys-apps/sandbox-2.2 )
	elibc_uclibc? ( >=sys-apps/sandbox-2.2 )
	>=app-misc/pax-utils-0.1.17
	selinux? ( >=sys-libs/libselinux-2.0.94[python,${PYTHON_USEDEP}] )
	xattr? ( kernel_linux? (
		>=sys-apps/install-xattr-0.3
		$(python_gen_cond_dep 'dev-python/pyxattr[${PYTHON_USEDEP}]' \
			python{2_7,3_2} pypy)
	) )
	!<app-shells/bash-3.2_p17
	!<app-admin/logrotate-3.8.0"
PDEPEND="
	!build? (
		>=net-misc/rsync-2.6.4
		userland_GNU? ( >=sys-apps/coreutils-6.4 )
	)"
# coreutils-6.4 rdep is for date format in emerge-webrsync #164532
# NOTE: FEATURES=installsources requires debugedit and rsync

REQUIRED_USE="epydoc? ( $(python_gen_useflags 'python2*') )"

SRC_ARCHIVES="http://dev.gentoo.org/~dolsen/releases/portage"

prefix_src_archives() {
	local x y
	for x in ${@}; do
		for y in ${SRC_ARCHIVES}; do
			echo ${y}/${x}
		done
	done
}

TARBALL_PV=${PV}
SRC_URI="mirror://gentoo/${PN}-${TARBALL_PV}.tar.bz2
	$(prefix_src_archives ${PN}-${TARBALL_PV}.tar.bz2)"

python_prepare_all() {
	distutils-r1_python_prepare_all

	local _version=${PV}
		einfo "Setting portage.VERSION to ${_version} ..."
	sed -e "s/^VERSION =.*/VERSION = '${_version}'/" -i pym/portage/__init__.py || \
		die "Failed to patch portage.VERSION"
	sed -e "1s/VERSION/${_version}/" -i doc/fragment/version || \
		die "Failed to patch VERSION in doc/fragment/version"
	sed -e "1s/VERSION/${_version}/" -i $(find man -type f) || \
		die "Failed to patch VERSION in man page headers"

	if ! use ipc ; then
		einfo "Disabling ipc..."
		sed -e "s:_enable_ipc_daemon = True:_enable_ipc_daemon = False:" \
			-i pym/_emerge/AbstractEbuildProcess.py || \
			die "failed to patch AbstractEbuildProcess.py"
	fi

	if use xattr && use kernel_linux ; then
		einfo "Adding FEATURES=xattr to make.globals ..."
		echo -e '\nFEATURES="${FEATURES} xattr"' >> cnf/make.globals \
			|| die "failed to append to make.globals"
	fi

	if [[ -n ${EPREFIX} ]] ; then
		einfo "Setting portage.const.EPREFIX ..."
		sed -e "s|^\(SANDBOX_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/bin/sandbox\"\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(FAKEROOT_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/bin/fakeroot\"\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(BASH_BINARY[[:space:]]*=[[:space:]]*\"\)\(/bin/bash\"\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(MOVE_BINARY[[:space:]]*=[[:space:]]*\"\)\(/bin/mv\"\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(PRELINK_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/sbin/prelink\"\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(EPREFIX[[:space:]]*=[[:space:]]*\"\).*|\\1${EPREFIX}\"|" \
			-i pym/portage/const.py || \
			die "Failed to patch portage.const.EPREFIX"

		einfo "Prefixing shebangs ..."
		while read -r -d $'\0' ; do
			local shebang=$(head -n1 "$REPLY")
			if [[ ${shebang} == "#!"* && ! ${shebang} == "#!${EPREFIX}/"* ]] ; then
				sed -i -e "1s:.*:#!${EPREFIX}${shebang:2}:" "$REPLY" || \
					die "sed failed"
			fi
		done < <(find . -type f -print0)

		einfo "Adjusting make.globals ..."
		sed -e "s|\(/usr/portage\)|${EPREFIX}\\1|" \
			-e "s|^\(PORTAGE_TMPDIR=\"\)\(/var/tmp\"\)|\\1${EPREFIX}\\2|" \
			-i cnf/make.globals || die "sed failed"

		einfo "Adjusting repos.conf ..."
		sed -e "s|^\(main-repo = \).*|\\1gentoo_prefix|" \
			-e "s|^\\[gentoo\\]|[gentoo_prefix]|" \
			-e "s|^\(location = \)\(/usr/portage\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(sync-uri = \).*|\\1rsync://prefix.gentooexperimental.org/gentoo-portage-prefix|" \
			-i cnf/repos.conf || die "sed failed"

		einfo "Adding FEATURES=force-prefix to make.globals ..."
		echo -e '\nFEATURES="${FEATURES} force-prefix"' >> cnf/make.globals \
			|| die "failed to append to make.globals"
	fi

	cd "${S}/cnf" || die
	if [ -f "make.conf.example.${ARCH}".diff ]; then
		patch make.conf.example "make.conf.example.${ARCH}".diff || \
			die "Failed to patch make.conf.example"
	else
		eerror ""
		eerror "Portage does not have an arch-specific configuration for this arch."
		eerror "Please notify the arch maintainer about this issue. Using generic."
		eerror ""
	fi
}

python_compile_all() {
	local targets=()
	use doc && targets+=( docbook )
	use epydoc && targets+=( epydoc )

	if [[ ${targets[@]} ]]; then
		esetup.py "${targets[@]}"
	fi
}

python_test() {
	esetup.py test
}

python_install() {
	distutils-r1_python_install \
		--system-prefix="${EPREFIX}/usr" \
		--bindir="$(python_get_scriptdir)" \
		--docdir="${EPREFIX}/usr/share/doc/${PF}" \
		--htmldir="${EPREFIX}/usr/share/doc/${PF}/html" \
		--portage-bindir="${EPREFIX}/usr/lib/portage/${EPYTHON}" \
		--sbindir="$(python_get_scriptdir)" \
		--sysconfdir="${EPREFIX}/etc" \
		"${@}"
}

python_install_all() {
	distutils-r1_python_install_all

	local targets=()
	use doc && targets+=( install_docbook )
	use epydoc && targets+=( install_epydoc )

	# install docs
	if [[ ${targets[@]} ]]; then
		esetup.py "${targets[@]}"
	fi
}

pkg_preinst() {
	if [[ $ROOT == / ]] ; then
		# Run some minimal tests as a sanity check.
		local test_runner=$(find "${ED}" -name runTests)
		if [[ -n $test_runner && -x $test_runner ]] ; then
			einfo "Running preinst sanity tests..."
			"$test_runner" || die "preinst sanity tests failed"
		fi
	fi

	# elog dir must exist to avoid logrotate error for bug #415911.
	# This code runs in preinst in order to bypass the mapping of
	# portage:portage to root:root which happens after src_install.
	keepdir /var/log/portage/elog
	# This is allowed to fail if the user/group are invalid for prefix users.
	if chown portage:portage "${ED}"var/log/portage{,/elog} 2>/dev/null ; then
		chmod g+s,ug+rwx "${ED}"var/log/portage{,/elog}
	fi
}

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [gentoo-portage-dev] [PATCH v2] Install Portage using setup.py
  2014-08-29 17:08       ` Brian Dolbec
@ 2014-08-29 23:30         ` Brian Dolbec
  0 siblings, 0 replies; 23+ messages in thread
From: Brian Dolbec @ 2014-08-29 23:30 UTC (permalink / raw
  To: gentoo-portage-dev

[-- Attachment #1: Type: text/plain, Size: 1463 bytes --]

On Fri, 29 Aug 2014 10:08:04 -0700
Brian Dolbec <dolsen@gentoo.org> wrote:

>
> 
> I have changed the version to 9999 and rebased your new-install branch
> on currrent master, and pushed it to my github account.
> 
> https://github.com/dol-sen/portage
> 
> I also am attaching the ebuilds I edited slightly
> 
> I have uploaded the 2.2.13_pre1 and _pre2 tarballs to my dev space.
> _pre2 differs from _pre1 in that _pre2 is with the rebased git
> sources, while _pre1 is Michal's original branch code.
> 
> you can rename the _pre2.ebuild to _pre1.ebuild to test at your
> pleasure.
> 


OK, all previous problems I reported with setup.py were due to my
checkout having some stale files or something sdist didn't like. Even
though an updated mkrelease.sh worked on the same files. The doc,
epydoc failure was due to my tree being a little too old, it was
missing an update to the distutils-r1.eclass mgorny did.

As for the version updating. I have removed it from the ebuild I have.
There were too many places that updated the version, the ebuild code
was missing the setup.py sed for it to be consistent.  Also the ebuild
should not have to.  I think that was from a time when portage releases
were just patches to previous tarballs.

Thanks to Michal for helping me figure out the errors he was unable to
reproduce.

I've uploaded a 2.2.13_pre4.tar.bz2 tarball to my releases/portage dir.
I've also attached the latest _pre4 ebuild.
-- 
Brian Dolbec <dolsen>


[-- Attachment #2: portage-2.2.13_pre4.ebuild --]
[-- Type: application/octet-stream, Size: 6687 bytes --]

# Copyright 1999-2014 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo-x86/sys-apps/portage/portage-9999.ebuild,v 1.99 2014/08/05 14:19:02 dolsen Exp $

EAPI=5

PYTHON_COMPAT=(
	pypy
	python3_2 python3_3 python3_4
	python2_7
)
PYTHON_REQ_USE='ssl(+),bzip2(+)'

inherit distutils-r1 multilib

DESCRIPTION="Portage is the package management and distribution system for Gentoo"
HOMEPAGE="http://www.gentoo.org/proj/en/portage/index.xml"

LICENSE="GPL-2"
KEYWORDS="~alpha ~amd64 ~arm ~arm64 ~hppa ~ia64 ~m68k ~mips ~ppc ~ppc64 ~s390 ~sh ~sparc ~x86 ~amd64-fbsd ~sparc-fbsd ~x86-fbsd"
SLOT="0"
IUSE="build doc epydoc +ipc linguas_ru selinux xattr"

DEPEND="${python_dep}
	>=sys-apps/sed-4.0.5 sys-devel/patch
	doc? ( app-text/xmlto ~app-text/docbook-xml-dtd-4.4 )
	epydoc? ( >=dev-python/epydoc-2.0[$(python_gen_usedep 'python2*')] )"
# Require sandbox-2.2 for bug #288863.
# For xattr, we can spawn getfattr and setfattr from sys-apps/attr, but that's
# quite slow, so it's not considered in the dependencies as an alternative to
# to python-3.3 / pyxattr. Also, xattr support is only tested with Linux, so
# for now, don't pull in xattr deps for other kernels.
# For whirlpool hash, require python[ssl] or python-mhash (bug #425046).
# For compgen, require bash[readline] (bug #445576).
RDEPEND="${python_dep}
	!build? ( >=sys-apps/sed-4.0.5
		|| ( >=app-shells/bash-4.2_p37[readline] ( <app-shells/bash-4.2_p37 >=app-shells/bash-3.2_p17 ) )
		>=app-admin/eselect-1.2
	)
	elibc_FreeBSD? ( sys-freebsd/freebsd-bin )
	elibc_glibc? ( >=sys-apps/sandbox-2.2 )
	elibc_uclibc? ( >=sys-apps/sandbox-2.2 )
	>=app-misc/pax-utils-0.1.17
	selinux? ( >=sys-libs/libselinux-2.0.94[python,${PYTHON_USEDEP}] )
	xattr? ( kernel_linux? (
		>=sys-apps/install-xattr-0.3
		$(python_gen_cond_dep 'dev-python/pyxattr[${PYTHON_USEDEP}]' \
			python{2_7,3_2} pypy)
	) )
	!<app-shells/bash-3.2_p17
	!<app-admin/logrotate-3.8.0"
PDEPEND="
	!build? (
		>=net-misc/rsync-2.6.4
		userland_GNU? ( >=sys-apps/coreutils-6.4 )
	)"
# coreutils-6.4 rdep is for date format in emerge-webrsync #164532
# NOTE: FEATURES=installsources requires debugedit and rsync

REQUIRED_USE="epydoc? ( $(python_gen_useflags 'python2*') )"

SRC_ARCHIVES="http://dev.gentoo.org/~dolsen/releases/portage"

prefix_src_archives() {
	local x y
	for x in ${@}; do
		for y in ${SRC_ARCHIVES}; do
			echo ${y}/${x}
		done
	done
}

TARBALL_PV=${PV}
SRC_URI="mirror://gentoo/${PN}-${TARBALL_PV}.tar.bz2
	$(prefix_src_archives ${PN}-${TARBALL_PV}.tar.bz2)"

python_prepare_all() {
	distutils-r1_python_prepare_all

	if ! use ipc ; then
		einfo "Disabling ipc..."
		sed -e "s:_enable_ipc_daemon = True:_enable_ipc_daemon = False:" \
			-i pym/_emerge/AbstractEbuildProcess.py || \
			die "failed to patch AbstractEbuildProcess.py"
	fi

	if use xattr && use kernel_linux ; then
		einfo "Adding FEATURES=xattr to make.globals ..."
		echo -e '\nFEATURES="${FEATURES} xattr"' >> cnf/make.globals \
			|| die "failed to append to make.globals"
	fi

	if [[ -n ${EPREFIX} ]] ; then
		einfo "Setting portage.const.EPREFIX ..."
		sed -e "s|^\(SANDBOX_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/bin/sandbox\"\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(FAKEROOT_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/bin/fakeroot\"\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(BASH_BINARY[[:space:]]*=[[:space:]]*\"\)\(/bin/bash\"\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(MOVE_BINARY[[:space:]]*=[[:space:]]*\"\)\(/bin/mv\"\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(PRELINK_BINARY[[:space:]]*=[[:space:]]*\"\)\(/usr/sbin/prelink\"\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(EPREFIX[[:space:]]*=[[:space:]]*\"\).*|\\1${EPREFIX}\"|" \
			-i pym/portage/const.py || \
			die "Failed to patch portage.const.EPREFIX"

		einfo "Prefixing shebangs ..."
		while read -r -d $'\0' ; do
			local shebang=$(head -n1 "$REPLY")
			if [[ ${shebang} == "#!"* && ! ${shebang} == "#!${EPREFIX}/"* ]] ; then
				sed -i -e "1s:.*:#!${EPREFIX}${shebang:2}:" "$REPLY" || \
					die "sed failed"
			fi
		done < <(find . -type f -print0)

		einfo "Adjusting make.globals ..."
		sed -e "s|\(/usr/portage\)|${EPREFIX}\\1|" \
			-e "s|^\(PORTAGE_TMPDIR=\"\)\(/var/tmp\"\)|\\1${EPREFIX}\\2|" \
			-i cnf/make.globals || die "sed failed"

		einfo "Adjusting repos.conf ..."
		sed -e "s|^\(main-repo = \).*|\\1gentoo_prefix|" \
			-e "s|^\\[gentoo\\]|[gentoo_prefix]|" \
			-e "s|^\(location = \)\(/usr/portage\)|\\1${EPREFIX}\\2|" \
			-e "s|^\(sync-uri = \).*|\\1rsync://prefix.gentooexperimental.org/gentoo-portage-prefix|" \
			-i cnf/repos.conf || die "sed failed"

		einfo "Adding FEATURES=force-prefix to make.globals ..."
		echo -e '\nFEATURES="${FEATURES} force-prefix"' >> cnf/make.globals \
			|| die "failed to append to make.globals"
	fi

	cd "${S}/cnf" || die
	if [ -f "make.conf.example.${ARCH}".diff ]; then
		patch make.conf.example "make.conf.example.${ARCH}".diff || \
			die "Failed to patch make.conf.example"
	else
		eerror ""
		eerror "Portage does not have an arch-specific configuration for this arch."
		eerror "Please notify the arch maintainer about this issue. Using generic."
		eerror ""
	fi
}

python_compile_all() {
	local targets=()
	use doc && targets+=( docbook )
	use epydoc && targets+=( epydoc )

	if [[ ${targets[@]} ]]; then
		esetup.py "${targets[@]}"
	fi
}

python_test() {
	esetup.py test
}

python_install() {
	distutils-r1_python_install \
		--system-prefix="${EPREFIX}/usr" \
		--bindir="$(python_get_scriptdir)" \
		--docdir="${EPREFIX}/usr/share/doc/${PF}" \
		--htmldir="${EPREFIX}/usr/share/doc/${PF}/html" \
		--portage-bindir="${EPREFIX}/usr/lib/portage/${EPYTHON}" \
		--sbindir="$(python_get_scriptdir)" \
		--sysconfdir="${EPREFIX}/etc" \
		"${@}"
}

python_install_all() {
	distutils-r1_python_install_all

	local targets=()
	use doc && targets+=( install_docbook )
	use epydoc && targets+=( install_epydoc )

	# install docs
	if [[ ${targets[@]} ]]; then
		esetup.py "${targets[@]}"
	fi
}

pkg_preinst() {
	if [[ $ROOT == / ]] ; then
		# Run some minimal tests as a sanity check.
		local test_runner=$(find "${ED}" -name runTests)
		if [[ -n $test_runner && -x $test_runner ]] ; then
			einfo "Running preinst sanity tests..."
			"$test_runner" || die "preinst sanity tests failed"
		fi
	fi

	# elog dir must exist to avoid logrotate error for bug #415911.
	# This code runs in preinst in order to bypass the mapping of
	# portage:portage to root:root which happens after src_install.
	keepdir /var/log/portage/elog
	# This is allowed to fail if the user/group are invalid for prefix users.
	if chown portage:portage "${ED}"var/log/portage{,/elog} 2>/dev/null ; then
		chmod g+s,ug+rwx "${ED}"var/log/portage{,/elog}
	fi
}

^ permalink raw reply	[flat|nested] 23+ messages in thread

* [gentoo-portage-dev] [PATCH v3] Install Portage using setup.py
  2014-08-23 20:30     ` [gentoo-portage-dev] [PATCH v2] Install Portage " Michał Górny
  2014-08-23 21:56       ` Brian Dolbec
  2014-08-29 17:08       ` Brian Dolbec
@ 2014-09-05  6:58       ` Michał Górny
  2 siblings, 0 replies; 23+ messages in thread
From: Michał Górny @ 2014-09-05  6:58 UTC (permalink / raw
  To: gentoo-portage-dev; +Cc: Michał Górny

Changes in v2:
- 'sdist' support

Changes in v3:
- version substituted properly in docs & mans
- cleaner handling of install_data
---
 .gitignore           |   1 +
 MANIFEST.in          |  18 ++
 Makefile             | 215 -----------------
 doc/Makefile         |  11 -
 doc/fragment/date    |   0
 doc/fragment/version |   1 -
 mkrelease.sh         | 141 -----------
 pym/portage/const.py |   4 +-
 setup.py             | 657 +++++++++++++++++++++++++++++++++++++++++++++++++++
 9 files changed, 678 insertions(+), 370 deletions(-)
 create mode 100644 MANIFEST.in
 delete mode 100644 Makefile
 delete mode 100644 doc/Makefile
 delete mode 100644 doc/fragment/date
 delete mode 100644 doc/fragment/version
 delete mode 100755 mkrelease.sh
 create mode 100755 setup.py

diff --git a/.gitignore b/.gitignore
index 074bb86..c2dd534 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,4 +1,5 @@
 *.py[co]
 __pycache__/
 *.class
+/build
 /tags
diff --git a/MANIFEST.in b/MANIFEST.in
new file mode 100644
index 0000000..d65c874
--- /dev/null
+++ b/MANIFEST.in
@@ -0,0 +1,18 @@
+# docs
+include DEVELOPING
+include LICENSE
+include TEST-NOTES
+
+# docbook sources
+include doc/custom.xsl
+recursive-include doc *.docbook
+
+# extra conf files used in ebuild
+include cnf/make.conf.example.*
+
+# extra files for tests
+include .portage_not_installed
+include cnf/metadata.dtd
+
+# extra scripts
+include misc/*
diff --git a/Makefile b/Makefile
deleted file mode 100644
index 9eb6e66..0000000
--- a/Makefile
+++ /dev/null
@@ -1,215 +0,0 @@
-SHELL = /bin/sh
-PN ?= portage
-PF ?= portage
-HOMEPAGE ?= http://www.gentoo.org/proj/en/portage/index.xml
-PWD ?= $(shell pwd)
-S ?= $(PWD)
-WORKDIR ?= $(PWD)
-DESTDIR = $(PWD)/image/
-srcdir = $(S)
-prefix = /usr
-sysconfdir = /etc
-exec_prefix = $(prefix)
-bindir = $(exec_prefix)/bin
-sbindir = $(exec_prefix)/sbin
-libdir = $(exec_prefix)/lib
-datarootdir = $(prefix)/share
-datadir = $(datarootdir)
-mandir = $(datarootdir)/man
-docdir = $(datarootdir)/doc/$(PF)
-htmldir = $(docdir)/html
-portage_datadir = $(datarootdir)/$(PN)
-portage_confdir = $(portage_datadir)/config
-portage_setsdir = $(portage_confdir)/sets
-portage_base = $(libdir)/$(PN)
-EPYDOC_OPTS = -qqqqq --no-frames --show-imports
-INSMODE = 0644
-EXEMODE = 0755
-DIRMODE = 0755
-SYSCONFDIR_FILES = etc-update.conf dispatch-conf.conf
-PORTAGE_CONFDIR_FILES = make.conf.example make.globals repos.conf
-LOGROTATE_FILES = elog-save-summary
-BINDIR_FILES = ebuild egencache emerge emerge-webrsync \
-	emirrordist portageq quickpkg repoman
-SBINDIR_FILES = archive-conf dispatch-conf emaint \
-	env-update etc-update fixpackages regenworld
-DOCS = ChangeLog NEWS RELEASE-NOTES
-LINGUAS ?= $(shell cd "$(srcdir)/man" && find -mindepth 1 -type d)
-
-ifdef PYTHONPATH
-	PYTHONPATH := $(srcdir)/pym:$(PYTHONPATH)
-else
-	PYTHONPATH := $(srcdir)/pym
-endif
-
-all: docbook epydoc
-
-docbook:
-	set -e; \
-	touch "$(srcdir)/doc/fragment/date"; \
-	$(MAKE) -C "$(srcdir)/doc" xhtml xhtml-nochunks
-
-epydoc:
-	set -e; \
-	env PYTHONPATH="$(PYTHONPATH)" epydoc \
-		-o "$(WORKDIR)/epydoc" \
-		--name $(PN) \
-		--url "$(HOMEPAGE)" \
-		$(EPYDOC_OPTS) \
-		$$(cd "$(srcdir)" && find pym -name '*.py' | sed \
-		-e s:/__init__.py$$:: \
-		-e s:\.py$$:: \
-		-e s:^pym/:: \
-		-e s:/:.:g \
-		| sort); \
-	rm -f "$(WORKDIR)/epydoc/api-objects.txt"; \
-
-test:
-	set -e; \
-	"$(srcdir)/pym/portage/tests/runTests.py"; \
-
-install:
-	set -e; \
-	cd "$(srcdir)/cnf"; \
-	install -d -m$(DIRMODE) "$(DESTDIR)$(sysconfdir)"; \
-	install -m$(INSMODE) $(SYSCONFDIR_FILES) "$(DESTDIR)$(sysconfdir)"; \
-	\
-	install -d -m$(DIRMODE) "$(DESTDIR)$(portage_confdir)"; \
-	cd "$(srcdir)/cnf"; \
-	install -m$(INSMODE) $(PORTAGE_CONFDIR_FILES) \
-		"$(DESTDIR)$(portage_confdir)"; \
-	install -d -m$(DIRMODE) "$(DESTDIR)$(portage_setsdir)"; \
-	cd "$(S)/cnf/sets"; \
-	install -m$(INSMODE) *.conf "$(DESTDIR)$(portage_setsdir)"; \
-	\
-	install -d -m$(DIRMODE) "$(DESTDIR)$(sysconfdir)/logrotate.d"; \
-	cd "$(srcdir)/cnf/logrotate.d"; \
-	install -m$(INSMODE) $(LOGROTATE_FILES) \
-		"$(DESTDIR)$(sysconfdir)/logrotate.d"; \
-	\
-	for x in $$(cd "$(srcdir)" && find bin -type d) ; do \
-		cd "$(srcdir)/$$x"; \
-		install -d -m$(DIRMODE) "$(DESTDIR)$(portage_base)/$$x"; \
-		files=$$(find . -mindepth 1 -maxdepth 1 -type f ! -type l); \
-		if [ -n "$$files" ] ; then \
-			install -m$(EXEMODE) $$files \
-				"$(DESTDIR)$(portage_base)/$$x"; \
-		fi; \
-		symlinks=$$(find . -mindepth 1 -maxdepth 1 -type l); \
-		if [ -n "$$symlinks" ] ; then \
-			cp -P $$symlinks "$(DESTDIR)$(portage_base)/$$x"; \
-		fi; \
-	done; \
-	\
-	for x in $$(cd "$(srcdir)" && find pym/* -type d \
-		! -path "pym/portage/tests*") ; do \
-		cd "$(srcdir)/$$x"; \
-		files=$$(echo *.py); \
-		if [ -z "$$files" ] || [ "$$files" = "*.py" ]; then \
-			# __pycache__ directories contain no py files \
-			continue; \
-		fi; \
-		install -d -m$(DIRMODE) "$(DESTDIR)$(portage_base)/$$x"; \
-		install -m$(INSMODE) $$files "$(DESTDIR)$(portage_base)/$$x"; \
-	done; \
-	\
-	install -d -m$(DIRMODE) "$(DESTDIR)$(bindir)"; \
-	relative_path=".."; \
-	x=$(bindir) ; \
-	y="$(portage_base)"; \
-	if [ "$${x#$(prefix)}" != "$$x" ] && \
-		[ "$${y#$(prefix)}" != "$$y" ]; then \
-		x=$${x#$(prefix)}; \
-		y=$${y#$(prefix)}; \
-	fi; \
-	x=$${x%/*}; \
-	while [ -n "$$x" ] ; do \
-		relative_path=$${relative_path}/..; \
-		x=$${x%/*}; \
-	done; \
-	relative_path=$$relative_path$$y; \
-	for x in $(BINDIR_FILES) ; do \
-		ln -sf "$$relative_path/bin/$$x" \
-			"$(DESTDIR)$(bindir)/$$x"; \
-	done; \
-	\
-	install -d -m$(DIRMODE) "$(DESTDIR)$(sbindir)"; \
-	relative_path=".."; \
-	x=$(sbindir) ; \
-	y="$(portage_base)"; \
-	if [ "$${x#$(prefix)}" != "$$x" ] && \
-		[ "$${y#$(prefix)}" != "$$y" ]; then \
-		x=$${x#$(prefix)}; \
-		y=$${y#$(prefix)}; \
-	fi; \
-	x=$${x%/*}; \
-	while [ -n "$$x" ] ; do \
-		relative_path=$${relative_path}/..; \
-		x=$${x%/*}; \
-	done; \
-	relative_path=$$relative_path$$y; \
-	for x in $(SBINDIR_FILES) ; do \
-		ln -sf "$$relative_path/bin/$$x" \
-			"$(DESTDIR)$(sbindir)/$$x"; \
-	done; \
-	\
-	ln -sf "$$relative_path/bin/env-update" \
-		"$(DESTDIR)$(sbindir)/update-env"; \
-	ln -sf "$$relative_path/bin/etc-update" \
-		"$(DESTDIR)$(sbindir)/update-etc"; \
-	\
-	# We install some minimal tests for use as a preinst sanity check. \
-	# These tests must be able to run without a full source tree and \
-	# without relying on a previous portage instance being installed. \
-	install -d -m$(DIRMODE) \
-		"$(DESTDIR)$(portage_base)/pym/portage/tests"; \
-	install -m$(EXEMODE) "$(srcdir)/pym/portage/tests/runTests" \
-		"$(DESTDIR)$(portage_base)/pym/portage/tests"; \
-	cd "$(srcdir)/pym/portage/tests"; \
-	install -m$(INSMODE) *.py \
-		"$(DESTDIR)$(portage_base)/pym/portage/tests"; \
-	install -d -m$(DIRMODE) \
-		"$(DESTDIR)$(portage_base)/pym/portage/tests/lint"; \
-	cd "$(srcdir)/pym/portage/tests/lint"; \
-	install -m$(INSMODE) *.py __test__ \
-		"$(DESTDIR)$(portage_base)/pym/portage/tests/lint"; \
-	\
-	install -d -m$(DIRMODE) "$(DESTDIR)$(docdir)"; \
-	cd "$(srcdir)"; \
-	install -m $(INSMODE) $(DOCS) "$(DESTDIR)$(docdir)"; \
-	\
-	for x in "" $(LINGUAS); do \
-		for y in 1 5 ; do \
-			if [ -d "$(srcdir)/man/$$x" ]; then \
-				cd "$(srcdir)/man/$$x"; \
-				files=$$(echo *.$$y); \
-				if [ -z "$$files" ] || [ "$$files" = "*.$$y" ]; then \
-					continue; \
-				fi; \
-				install -d -m$(DIRMODE) "$(DESTDIR)$(mandir)/$$x/man$$y"; \
-				install -m$(INSMODE) *.$$y "$(DESTDIR)$(mandir)/$$x/man$$y"; \
-			fi; \
-		done; \
-	done; \
-	\
-	if [ -f "$(srcdir)/doc/portage.html" ] ; then \
-		install -d -m$(DIRMODE) "$(DESTDIR)$(htmldir)"; \
-		cd "$(srcdir)/doc"; \
-		install -m$(INSMODE) *.html "$(DESTDIR)$(htmldir)"; \
-	fi; \
-	\
-	if [ -d "$(WORKDIR)/epydoc" ] ; then \
-		install -d -m$(DIRMODE) "$(DESTDIR)$(htmldir)"; \
-		cp -pPR "$(WORKDIR)/epydoc" \
-			"$(DESTDIR)$(htmldir)/api"; \
-		cd "$(DESTDIR)$(htmldir)/api"; \
-		find . -type d | xargs chmod $(DIRMODE); \
-		find . -type f | xargs chmod $(INSMODE); \
-	fi; \
-
-clean:
-	set -e; \
-	$(MAKE) -C "$(srcdir)/doc" clean; \
-	rm -rf "$(WORKDIR)/epydoc"; \
-
-.PHONY: all clean docbook epydoc install test
diff --git a/doc/Makefile b/doc/Makefile
deleted file mode 100644
index 261a0b4..0000000
--- a/doc/Makefile
+++ /dev/null
@@ -1,11 +0,0 @@
-all: xhtml xhtml-nochunks
-
-XMLTO_FLAGS = -m custom.xsl
-man pdf txt xhtml xhtml-nochunks:
-	xmlto $@ $(XMLTO_FLAGS) portage.docbook
-
-clean distclean:
-	rm -f *.1 *.html portage.txt
-
-.PHONY: all clean distclean \
-        man pdf txt xhtml xhtml-nochunks
diff --git a/doc/fragment/date b/doc/fragment/date
deleted file mode 100644
index e69de29..0000000
diff --git a/doc/fragment/version b/doc/fragment/version
deleted file mode 100644
index f85674c..0000000
--- a/doc/fragment/version
+++ /dev/null
@@ -1 +0,0 @@
-<releaseinfo>VERSION</releaseinfo>
diff --git a/mkrelease.sh b/mkrelease.sh
deleted file mode 100755
index f9f7564..0000000
--- a/mkrelease.sh
+++ /dev/null
@@ -1,141 +0,0 @@
-#!/bin/bash
-# Copyright 2008-2014 Gentoo Foundation
-# Distributed under the terms of the GNU General Public License v2
-
-RELEASE_BUILDDIR=${RELEASE_BUILDDIR:-/var/tmp/portage-release}
-SOURCE_DIR=${RELEASE_BUILDDIR}/checkout
-BRANCH=${BRANCH:-master}
-USE_TAG=false
-CHANGELOG_REVISION=
-UPLOAD_LOCATION=
-RUNTESTS=false
-USER=
-
-usage() {
-	echo "Usage: ${0##*/} [--changelog-rev <tree-ish>] [-t|--tag] [-u|--upload <location>]  [--user <username>] [--runtests] <version>"
-	exit ${1:-0}
-}
-
-die() {
-	printf 'error: %s\n' "$*"
-	usage 1
-}
-
-ARGS=$(getopt -o htu: --long help,changelog-rev:,runtests,tag,upload:,user: \
-	-n "${0##*/}" -- "$@")
-[ $? != 0 ] && die "initialization error"
-
-eval set -- "${ARGS}"
-
-while true; do
-	case $1 in
-		--changelog-rev)
-			CHANGELOG_REVISION=$2
-			shift 2
-			;;
-		-t|--tag)
-			USE_TAG=true
-			shift
-			;;
-		-u|--upload)
-			UPLOAD_LOCATION=$2
-			shift 2
-			;;
-		--user)
-			USER=$2"@"
-			shift 2
-			;;
-		-h|--help)
-			usage
-			;;
-		--runtests)
-			RUNTESTS=true
-			shift
-			;;
-		--)
-			shift
-			break
-			;;
-		*)
-			die "unknown option: $1"
-			;;
-	esac
-done
-
-[ $# != 1 ] && die "Need version argument"
-[[ -n ${1/[0-9]*} ]] && die "Invalid version argument"
-
-VERSION=$1
-RELEASE=portage-${VERSION}
-RELEASE_DIR=${RELEASE_BUILDDIR}/${RELEASE}
-RELEASE_TARBALL="${RELEASE_BUILDDIR}/${RELEASE}.tar.bz2"
-TREE_ISH=${BRANCH}
-if [[ ${USE_TAG} == "true" ]] ; then
-	TREE_ISH="v${VERSION}"
-fi
-
-echo ">>> Cleaning working directories ${RELEASE_DIR} ${SOURCE_DIR}"
-rm -rf "${RELEASE_DIR}" "${SOURCE_DIR}" || die "directory cleanup failed"
-mkdir -p "${RELEASE_DIR}" || die "directory creation failed"
-mkdir -p "${SOURCE_DIR}" || die "mkdir failed"
-
-echo ">>> Starting GIT archive"
-git archive --format=tar ${TREE_ISH} | \
-	tar -xf - -C "${SOURCE_DIR}" || die "git archive failed"
-
-echo ">>> Building release tree"
-cp -a "${SOURCE_DIR}/"{bin,cnf,doc,man,misc,pym} "${RELEASE_DIR}/" || die "directory copy failed"
-cp "${SOURCE_DIR}/"{.portage_not_installed,DEVELOPING,LICENSE,Makefile,NEWS,README,RELEASE-NOTES,TEST-NOTES} \
-	"${RELEASE_DIR}/" || die "file copy failed"
-
-if [[ ${RUNTESTS} == "true" ]] ; then
-	pushd "${SOURCE_DIR}" >/dev/null
-	./runtests.sh --python-versions=supported || die "tests failed"
-	popd >/dev/null
-fi
-
-rm -rf "${SOURCE_DIR}" || die "directory cleanup failed"
-
-echo ">>> Setting portage.VERSION"
-sed -e "s/^VERSION = .*/VERSION = \"${VERSION}\"/" \
-	-i "${RELEASE_DIR}/pym/portage/__init__.py" || \
-	die "Failed to patch portage.VERSION"
-
-echo ">>> Creating Changelog"
-git_log_opts=""
-if [[ -n ${CHANGELOG_REVISION} ]] ; then
-	git_log_opts+=" ${CHANGELOG_REVISION}^..${TREE_ISH}"
-else
-	git_log_opts+=" ${TREE_ISH}"
-fi
-skip_next=false
-git log ${git_log_opts} | fmt -w 80 -p "    " | while read -r ; do
-	if [[ ${skip_next} == "true" ]] ; then
-		skip_next=false
-	elif [[ ${REPLY} == "    svn path="* ]] ; then
-		skip_next=true
-	else
-		echo "${REPLY}"
-	fi
-done > "${RELEASE_DIR}/ChangeLog" || die "ChangeLog creation failed"
-
-cd "${RELEASE_BUILDDIR}"
-
-echo ">>> Creating release tarball ${RELEASE_TARBALL}"
-tar --owner portage --group portage -cjf "${RELEASE_TARBALL}" "${RELEASE}" || \
-	die "tarball creation failed"
-
-DISTDIR=$(portageq distdir)
-if [[ -n ${DISTDIR} && -d ${DISTDIR} && -w ${DISTDIR} ]] ; then
-	echo ">>> Copying release tarball into ${DISTDIR}"
-	cp "${RELEASE_TARBALL}" "${DISTDIR}"/ || echo "!!! tarball copy failed"
-fi
-
-if [[ -n ${UPLOAD_LOCATION} ]] ; then
-	echo ">>> Uploading ${RELEASE_TARBALL} to ${USER}dev.gentoo.org:${UPLOAD_LOCATION}"
-	scp "${RELEASE_TARBALL}" "${USER}dev.gentoo.org:${UPLOAD_LOCATION}" || die "upload failed"
-else
-	du -h "${RELEASE_TARBALL}"
-fi
-
-exit 0
diff --git a/pym/portage/const.py b/pym/portage/const.py
index f518b47..acb90f9 100644
--- a/pym/portage/const.py
+++ b/pym/portage/const.py
@@ -60,8 +60,8 @@ GLOBAL_CONFIG_PATH       = "/usr/share/portage/config"
 # these variables are not used with target_root or config_root
 # NOTE: Use realpath(__file__) so that python module symlinks in site-packages
 # are followed back to the real location of the whole portage installation.
-PORTAGE_BASE_PATH        = os.path.join(os.sep, os.sep.join(os.path.realpath(
-                               __file__.rstrip("co")).split(os.sep)[:-3]))
+# NOTE: Please keep PORTAGE_BASE_PATH in one line to help substitutions.
+PORTAGE_BASE_PATH        = os.path.join(os.sep, os.sep.join(os.path.realpath(__file__.rstrip("co")).split(os.sep)[:-3]))
 PORTAGE_BIN_PATH         = PORTAGE_BASE_PATH + "/bin"
 PORTAGE_PYM_PATH         = os.path.realpath(os.path.join(__file__, '../..'))
 LOCALE_DATA_PATH         = PORTAGE_BASE_PATH + "/locale"  # FIXME: not used
diff --git a/setup.py b/setup.py
new file mode 100755
index 0000000..e5c3631
--- /dev/null
+++ b/setup.py
@@ -0,0 +1,657 @@
+#!/usr/bin/env python
+# Copyright 1998-2014 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+from distutils.core import setup, Command
+from distutils.command.build import build
+from distutils.command.build_scripts import build_scripts
+from distutils.command.clean import clean
+from distutils.command.install import install
+from distutils.command.install_data import install_data
+from distutils.command.install_lib import install_lib
+from distutils.command.install_scripts import install_scripts
+from distutils.command.sdist import sdist
+from distutils.dep_util import newer
+from distutils.dir_util import mkpath, remove_tree
+from distutils.util import change_root, subst_vars
+
+import codecs, collections, errno, glob, os, os.path, re, subprocess, sys
+
+# TODO:
+# - smarter rebuilds of docs w/ 'install_docbook' and 'install_epydoc'.
+
+x_scripts = {
+	'bin': [
+		'bin/ebuild', 'bin/egencache', 'bin/emerge', 'bin/emerge-webrsync',
+		'bin/emirrordist', 'bin/portageq', 'bin/quickpkg', 'bin/repoman'
+	],
+	'sbin': [
+		'bin/archive-conf', 'bin/dispatch-conf', 'bin/emaint', 'bin/env-update',
+		'bin/etc-update', 'bin/fixpackages', 'bin/regenworld'
+	],
+}
+
+
+class x_build(build):
+	""" Build command with extra build_man call. """
+
+	def run(self):
+		build.run(self)
+		self.run_command('build_man')
+
+
+class build_man(Command):
+	""" Perform substitutions in manpages. """
+
+	user_options = [
+	]
+
+	def initialize_options(self):
+		self.build_base = None
+
+	def finalize_options(self):
+		self.set_undefined_options('build',
+			('build_base', 'build_base'))
+
+	def run(self):
+		for d, files in self.distribution.data_files:
+			if not d.startswith('$mandir/'):
+				continue
+
+			for source in files:
+				target = os.path.join(self.build_base, source)
+				mkpath(os.path.dirname(target))
+
+				if not newer(source, target) and not newer(__file__, target):
+					continue
+
+				print('copying and updating %s -> %s' % (
+					source, target))
+
+				with codecs.open(source, 'r', 'utf8') as f:
+					data = f.readlines()
+				data[0] = data[0].replace('VERSION',
+						self.distribution.get_version())
+				with codecs.open(target, 'w', 'utf8') as f:
+					f.writelines(data)
+
+
+class docbook(Command):
+	""" Build docs using docbook. """
+
+	user_options = [
+		('doc-formats=', None, 'Documentation formats to build (all xmlto formats for docbook are allowed, comma-separated'),
+	]
+
+	def initialize_options(self):
+		self.doc_formats = 'xhtml,xhtml-nochunks'
+
+	def finalize_options(self):
+		self.doc_formats = self.doc_formats.replace(',', ' ').split()
+
+	def run(self):
+		if not os.path.isdir('doc/fragment'):
+			mkpath('doc/fragment')
+
+		with open('doc/fragment/date', 'w'):
+			pass
+		with open('doc/fragment/version', 'w') as f:
+			f.write('<releaseinfo>%s</releaseinfo>' % self.distribution.get_version())
+
+		for f in self.doc_formats:
+			print('Building docs in %s format...' % f)
+			subprocess.check_call(['xmlto', '-o', 'doc',
+				'-m', 'doc/custom.xsl', f, 'doc/portage.docbook'])
+
+
+class epydoc(Command):
+	""" Build API docs using epydoc. """
+
+	user_options = [
+	]
+
+	def initialize_options(self):
+		self.build_lib = None
+
+	def finalize_options(self):
+		self.set_undefined_options('build_py', ('build_lib', 'build_lib'))
+
+	def run(self):
+		self.run_command('build_py')
+
+		print('Building API documentation...')
+
+		process_env = os.environ.copy()
+		pythonpath = self.build_lib
+		try:
+			pythonpath += ':' + process_env['PYTHONPATH']
+		except KeyError:
+			pass
+		process_env['PYTHONPATH'] = pythonpath
+
+		subprocess.check_call(['epydoc', '-o', 'epydoc',
+			'--name', self.distribution.get_name(),
+			'--url', self.distribution.get_url(),
+			'-qq', '--no-frames', '--show-imports',
+			'--exclude', 'portage.tests',
+			'_emerge', 'portage', 'repoman'],
+			env = process_env)
+		os.remove('epydoc/api-objects.txt')
+
+
+class install_docbook(install_data):
+	""" install_data for docbook docs """
+
+	user_options = install_data.user_options
+
+	def initialize_options(self):
+		install_data.initialize_options(self)
+		self.htmldir = None
+
+	def finalize_options(self):
+		self.set_undefined_options('install', ('htmldir', 'htmldir'))
+		install_data.finalize_options(self)
+
+	def run(self):
+		if not os.path.exists('doc/portage.html'):
+			self.run_command('docbook')
+		self.data_files = [
+			(self.htmldir, glob.glob('doc/*.html')),
+		]
+		install_data.run(self)
+
+
+class install_epydoc(install_data):
+	""" install_data for epydoc docs """
+
+	user_options = install_data.user_options
+
+	def initialize_options(self):
+		install_data.initialize_options(self)
+		self.htmldir = None
+
+	def finalize_options(self):
+		self.set_undefined_options('install', ('htmldir', 'htmldir'))
+		install_data.finalize_options(self)
+
+	def run(self):
+		if not os.path.exists('epydoc/index.html'):
+			self.run_command('epydoc')
+		self.data_files = [
+			(os.path.join(self.htmldir, 'api'), glob.glob('epydoc/*')),
+		]
+		install_data.run(self)
+
+
+class x_build_scripts_custom(build_scripts):
+	def finalize_options(self):
+		build_scripts.finalize_options(self)
+		if 'dir_name' in dir(self):
+			self.build_dir = os.path.join(self.build_dir, self.dir_name)
+			if self.dir_name in x_scripts:
+				self.scripts = x_scripts[self.dir_name]
+			else:
+				self.scripts = set(self.scripts)
+				for other_files in x_scripts.values():
+					self.scripts.difference_update(other_files)
+
+	def run(self):
+		# group scripts by subdirectory
+		split_scripts = collections.defaultdict(list)
+		for f in self.scripts:
+			dir_name = os.path.dirname(f[len('bin/'):])
+			split_scripts[dir_name].append(f)
+
+		base_dir = self.build_dir
+		base_scripts = self.scripts
+		for d, files in split_scripts.items():
+			self.build_dir = os.path.join(base_dir, d)
+			self.scripts = files
+			self.copy_scripts()
+
+		# restore previous values
+		self.build_dir = base_dir
+		self.scripts = base_scripts
+
+
+class x_build_scripts_bin(x_build_scripts_custom):
+	dir_name = 'bin'
+
+
+class x_build_scripts_sbin(x_build_scripts_custom):
+	dir_name = 'sbin'
+
+
+class x_build_scripts_portagebin(x_build_scripts_custom):
+	dir_name = 'portage'
+
+
+class x_build_scripts(build_scripts):
+	def initialize_option(self):
+		build_scripts.initialize_options(self)
+
+	def finalize_options(self):
+		build_scripts.finalize_options(self)
+
+	def run(self):
+		self.run_command('build_scripts_bin')
+		self.run_command('build_scripts_portagebin')
+		self.run_command('build_scripts_sbin')
+
+
+class x_clean(clean):
+	""" clean extended for doc & post-test cleaning """
+
+	def clean_docs(self):
+		def get_doc_outfiles():
+			for dirpath, dirnames, filenames in os.walk('doc'):
+				for f in filenames:
+					if f.endswith('.docbook') or f == 'custom.xsl':
+						pass
+					else:
+						yield os.path.join(dirpath, f)
+
+				# do not recurse
+				break
+
+
+		for f in get_doc_outfiles():
+			print('removing %s' % repr(f))
+			os.remove(f)
+
+		if os.path.isdir('doc/fragment'):
+			remove_tree('doc/fragment')
+
+		if os.path.isdir('epydoc'):
+			remove_tree('epydoc')
+
+	def clean_tests(self):
+		# do not remove incorrect dirs accidentally
+		top_dir = os.path.normpath(os.path.join(self.build_lib, '..'))
+		cprefix = os.path.commonprefix((self.build_base, top_dir))
+		if cprefix != self.build_base:
+			return
+
+		bin_dir = os.path.join(top_dir, 'bin')
+		if os.path.exists(bin_dir):
+			remove_tree(bin_dir)
+
+		conf_dir = os.path.join(top_dir, 'cnf')
+		if os.path.islink(conf_dir):
+			print('removing %s symlink' % repr(conf_dir))
+			os.unlink(conf_dir)
+
+		pni_file = os.path.join(top_dir, '.portage_not_installed')
+		if os.path.exists(pni_file):
+			print('removing %s' % repr(pni_file))
+			os.unlink(pni_file)
+
+	def clean_man(self):
+		man_dir = os.path.join(self.build_base, 'man')
+		if os.path.exists(man_dir):
+			remove_tree(man_dir)
+
+	def run(self):
+		if self.all:
+			self.clean_tests()
+			self.clean_docs()
+			self.clean_man()
+
+		clean.run(self)
+
+
+class x_install(install):
+	""" install command with extra Portage paths """
+
+	user_options = install.user_options + [
+		# note: $prefix and $exec_prefix are reserved for Python install
+		('system-prefix=', None, "Prefix for architecture-independent data"),
+		('system-exec-prefix=', None, "Prefix for architecture-specific data"),
+
+		('bindir=', None, "Install directory for main executables"),
+		('datarootdir=', None, "Data install root directory"),
+		('docdir=', None, "Documentation install directory"),
+		('htmldir=', None, "HTML documentation install directory"),
+		('mandir=', None, "Manpage root install directory"),
+		('portage-base=', 'b', "Portage install base"),
+		('portage-bindir=', None, "Install directory for Portage internal-use executables"),
+		('portage-datadir=', None, 'Install directory for data files'),
+		('sbindir=', None, "Install directory for superuser-intended executables"),
+		('sysconfdir=', None, 'System configuration path'),
+	]
+
+	# note: the order is important for proper substitution
+	paths = [
+		('system_prefix', '/usr'),
+		('system_exec_prefix', '$system_prefix'),
+
+		('bindir', '$system_exec_prefix/bin'),
+		('sbindir', '$system_exec_prefix/sbin'),
+		('sysconfdir', '/etc'),
+
+		('datarootdir', '$system_prefix/share'),
+		('docdir', '$datarootdir/doc/$package-$version'),
+		('htmldir', '$docdir/html'),
+		('mandir', '$datarootdir/man'),
+
+		('portage_base', '$system_exec_prefix/lib/portage'),
+		('portage_bindir', '$portage_base/bin'),
+		('portage_datadir', '$datarootdir/portage'),
+
+		# not customized at the moment
+		('logrotatedir', '$sysconfdir/logrotate'),
+		('portage_confdir', '$portage_datadir/config'),
+		('portage_setsdir', '$portage_confdir/sets'),
+	]
+
+	def initialize_options(self):
+		install.initialize_options(self)
+
+		for key, default in self.paths:
+			setattr(self, key, default)
+		self.subst_paths = {}
+
+	def finalize_options(self):
+		install.finalize_options(self)
+
+		# substitute variables
+		new_paths = {
+			'package': self.distribution.get_name(),
+			'version': self.distribution.get_version(),
+		}
+		for key, default in self.paths:
+			new_paths[key] = subst_vars(getattr(self, key), new_paths)
+			setattr(self, key, new_paths[key])
+		self.subst_paths = new_paths
+
+
+class x_install_data(install_data):
+	""" install_data with customized path support """
+
+	user_options = install_data.user_options
+
+	def initialize_options(self):
+		install_data.initialize_options(self)
+		self.build_base = None
+		self.paths = None
+
+	def finalize_options(self):
+		install_data.finalize_options(self)
+		self.set_undefined_options('build',
+			('build_base', 'build_base'))
+		self.set_undefined_options('install',
+			('subst_paths', 'paths'))
+
+	def run(self):
+		self.run_command('build_man')
+
+		def process_data_files(df):
+			for d, files in df:
+				# substitute man sources
+				if d.startswith('$mandir/'):
+					files = [os.path.join(self.build_base, v) for v in files]
+
+				# substitute variables in path
+				d = subst_vars(d, self.paths)
+				yield (d, files)
+
+		old_data_files = self.data_files
+		self.data_files = process_data_files(self.data_files)
+
+		install_data.run(self)
+		self.data_files = old_data_files
+
+
+class x_install_lib(install_lib):
+	""" install_lib command with Portage path substitution """
+
+	user_options = install_lib.user_options
+
+	def initialize_options(self):
+		install_lib.initialize_options(self)
+		self.portage_base = None
+		self.portage_bindir = None
+		self.portage_confdir = None
+
+	def finalize_options(self):
+		install_lib.finalize_options(self)
+		self.set_undefined_options('install',
+			('portage_base', 'portage_base'),
+			('portage_bindir', 'portage_bindir'),
+			('portage_confdir', 'portage_confdir'))
+
+	def install(self):
+		ret = install_lib.install(self)
+
+		def rewrite_file(path, val_dict):
+			path = os.path.join(self.install_dir, path)
+			print('Rewriting %s' % path)
+			with codecs.open(path, 'r', 'utf-8') as f:
+				data = f.read()
+
+			for varname, val in val_dict.items():
+				regexp = r'(?m)^(%s\s*=).*$' % varname
+				repl = r'\1 %s' % repr(val)
+
+				data = re.sub(regexp, repl, data)
+
+			with codecs.open(path, 'w', 'utf-8') as f:
+				f.write(data)
+
+		rewrite_file('portage/__init__.py', {
+			'VERSION': self.distribution.get_version(),
+		})
+		rewrite_file('portage/const.py', {
+			'PORTAGE_BASE_PATH': self.portage_base,
+			'PORTAGE_BIN_PATH': self.portage_bindir,
+			'PORTAGE_CONFIG_PATH': self.portage_confdir,
+		})
+
+		return ret
+
+
+class x_install_scripts_custom(install_scripts):
+	def initialize_options(self):
+		install_scripts.initialize_options(self)
+		self.root = None
+
+	def finalize_options(self):
+		self.set_undefined_options('install',
+			('root', 'root'),
+			(self.var_name, 'install_dir'))
+		install_scripts.finalize_options(self)
+		self.build_dir = os.path.join(self.build_dir, self.dir_name)
+
+		# prepend root
+		if self.root is not None:
+			self.install_dir = change_root(self.root, self.install_dir)
+
+
+class x_install_scripts_bin(x_install_scripts_custom):
+	dir_name = 'bin'
+	var_name = 'bindir'
+
+
+class x_install_scripts_sbin(x_install_scripts_custom):
+	dir_name = 'sbin'
+	var_name = 'sbindir'
+
+
+class x_install_scripts_portagebin(x_install_scripts_custom):
+	dir_name = 'portage'
+	var_name = 'portage_bindir'
+
+
+class x_install_scripts(install_scripts):
+	def initialize_option(self):
+		pass
+
+	def finalize_options(self):
+		pass
+
+	def run(self):
+		self.run_command('install_scripts_bin')
+		self.run_command('install_scripts_portagebin')
+		self.run_command('install_scripts_sbin')
+
+
+class x_sdist(sdist):
+	""" sdist defaulting to .tar.bz2 format """
+
+	def finalize_options(self):
+		if self.formats is None:
+			self.formats = ['bztar']
+
+		sdist.finalize_options(self)
+
+
+class build_tests(x_build_scripts_custom):
+	""" Prepare build dir for running tests. """
+
+	def initialize_options(self):
+		x_build_scripts_custom.initialize_options(self)
+		self.build_base = None
+		self.build_lib = None
+
+	def finalize_options(self):
+		x_build_scripts_custom.finalize_options(self)
+		self.set_undefined_options('build',
+			('build_base', 'build_base'),
+			('build_lib', 'build_lib'))
+
+		# since we will be writing to $build_lib/.., it is important
+		# that we do not leave $build_base
+		self.top_dir = os.path.normpath(os.path.join(self.build_lib, '..'))
+		cprefix = os.path.commonprefix((self.build_base, self.top_dir))
+		if cprefix != self.build_base:
+			raise SystemError('build_lib must be a subdirectory of build_base')
+
+		self.build_dir = os.path.join(self.top_dir, 'bin')
+
+	def run(self):
+		self.run_command('build_py')
+
+		# install all scripts $build_lib/../bin
+		# (we can't do a symlink since we want shebangs corrected)
+		x_build_scripts_custom.run(self)
+
+		# symlink 'cnf' directory
+		conf_dir = os.path.join(self.top_dir, 'cnf')
+		if os.path.exists(conf_dir):
+			if not os.path.islink(conf_dir):
+				raise SystemError('%s exists and is not a symlink (collision)'
+					% repr(conf_dir))
+			os.unlink(conf_dir)
+		conf_src = os.path.relpath('cnf', self.top_dir)
+		print('Symlinking %s -> %s' % (conf_dir, conf_src))
+		os.symlink(conf_src, conf_dir)
+
+		# create $build_lib/../.portage_not_installed
+		# to enable proper paths in tests
+		with open(os.path.join(self.top_dir, '.portage_not_installed'), 'w') as f:
+			pass
+
+
+class test(Command):
+	""" run tests """
+
+	user_options = []
+
+	def initialize_options(self):
+		self.build_lib = None
+
+	def finalize_options(self):
+		self.set_undefined_options('build',
+			('build_lib', 'build_lib'))
+
+	def run(self):
+		self.run_command('build_tests')
+		subprocess.check_call([
+			sys.executable, '-bWd',
+			os.path.join(self.build_lib, 'portage/tests/runTests.py')
+		])
+
+
+def find_packages():
+	for dirpath, dirnames, filenames in os.walk('pym'):
+		if '__init__.py' in filenames:
+			yield os.path.relpath(dirpath, 'pym')
+
+
+def find_scripts():
+	for dirpath, dirnames, filenames in os.walk('bin'):
+		for f in filenames:
+			yield os.path.join(dirpath, f)
+
+
+def get_manpages():
+	linguas = os.environ.get('LINGUAS')
+	if linguas is not None:
+		linguas = linguas.split()
+
+	for dirpath, dirnames, filenames in os.walk('man'):
+		groups = collections.defaultdict(list)
+		for f in filenames:
+			fn, suffix = f.rsplit('.', 1)
+			groups[suffix].append(os.path.join(dirpath, f))
+
+		topdir = dirpath[len('man/'):]
+		if not topdir or linguas is None or topdir in linguas:
+			for g, mans in groups.items():
+				yield [os.path.join('$mandir', topdir, 'man%s' % g), mans]
+
+setup(
+	name = 'portage',
+	version = '2.2.12',
+	url = 'https://wiki.gentoo.org/wiki/Project:Portage',
+	author = 'Gentoo Portage Development Team',
+	author_email = 'dev-portage@gentoo.org',
+
+	package_dir = {'': 'pym'},
+	packages = list(find_packages()),
+	# something to cheat build & install commands
+	scripts = list(find_scripts()),
+
+	data_files = list(get_manpages()) + [
+		['$sysconfdir', ['cnf/etc-update.conf', 'cnf/dispatch-conf.conf']],
+		['$logrotatedir', ['cnf/logrotate.d/elog-save-summary']],
+		['$portage_confdir', [
+			'cnf/make.conf.example', 'cnf/make.globals', 'cnf/repos.conf']],
+		['$portage_setsdir', ['cnf/sets/portage.conf']],
+		['$docdir', ['NEWS', 'RELEASE-NOTES']],
+	],
+
+	cmdclass = {
+		'build': x_build,
+		'build_man': build_man,
+		'build_scripts': x_build_scripts,
+		'build_scripts_bin': x_build_scripts_bin,
+		'build_scripts_portagebin': x_build_scripts_portagebin,
+		'build_scripts_sbin': x_build_scripts_sbin,
+		'build_tests': build_tests,
+		'clean': x_clean,
+		'docbook': docbook,
+		'epydoc': epydoc,
+		'install': x_install,
+		'install_data': x_install_data,
+		'install_docbook': install_docbook,
+		'install_epydoc': install_epydoc,
+		'install_lib': x_install_lib,
+		'install_scripts': x_install_scripts,
+		'install_scripts_bin': x_install_scripts_bin,
+		'install_scripts_portagebin': x_install_scripts_portagebin,
+		'install_scripts_sbin': x_install_scripts_sbin,
+		'sdist': x_sdist,
+		'test': test,
+	},
+
+	classifiers = [
+		'Development Status :: 5 - Production/Stable',
+		'Environment :: Console',
+		'Intended Audience :: System Administrators',
+		'License :: OSI Approved :: GNU General Public License v2 (GPLv2)',
+		'Operating System :: POSIX',
+		'Programming Language :: Python',
+		'Topic :: System :: Installation/Setup'
+	]
+)
-- 
2.1.0



^ permalink raw reply related	[flat|nested] 23+ messages in thread

* Re: [gentoo-portage-dev] [PATCHES] setup.py install for Portage
  2014-08-21 20:19 [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny
                   ` (8 preceding siblings ...)
  2014-08-24 19:42 ` Brian Dolbec
@ 2014-09-12  7:17 ` Michał Górny
  9 siblings, 0 replies; 23+ messages in thread
From: Michał Górny @ 2014-09-12  7:17 UTC (permalink / raw
  To: gentoo-portage-dev

[-- Attachment #1: Type: text/plain, Size: 261 bytes --]

Dnia 2014-08-21, o godz. 22:19:38
Michał Górny <mgorny@gentoo.org> napisał(a):

> Here is a rebase of my recent work on bringing setup.py install for
> Portage.

This one's committed in the ultimate newest version :).

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

^ permalink raw reply	[flat|nested] 23+ messages in thread

end of thread, other threads:[~2014-09-12  7:18 UTC | newest]

Thread overview: 23+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-08-21 20:19 [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny
2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 1/6] self-update: Copy only relevant packages from PORTAGE_PYM_PATH Michał Górny
2014-08-22 21:47   ` Brian Dolbec
2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 2/6] portage.const: Make PORTAGE_PYM_PATH point to the real package tree Michał Górny
2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 3/6] tests: Append .py to files that need to be installed Michał Górny
2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 4/6] tests: Fix running on installed copy of Portage Michał Górny
2014-08-22 22:19   ` Brian Dolbec
2014-08-22 22:52     ` Michał Górny
2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 5/6] Install Python modules using setup.py Michał Górny
2014-08-22 22:30   ` Brian Dolbec
2014-08-22 22:55     ` Michał Górny
2014-08-23 20:30     ` [gentoo-portage-dev] [PATCH v2] Install Portage " Michał Górny
2014-08-23 21:56       ` Brian Dolbec
2014-08-29 17:08       ` Brian Dolbec
2014-08-29 23:30         ` Brian Dolbec
2014-09-05  6:58       ` [gentoo-portage-dev] [PATCH v3] " Michał Górny
2014-08-21 20:19 ` [gentoo-portage-dev] [PATCH 6/6] travis: Use setup.py for running tests Michał Górny
2014-08-21 20:22 ` [gentoo-portage-dev] [PATCHES] setup.py install for Portage Michał Górny
2014-08-21 20:25   ` Michał Górny
2014-08-22 22:32 ` Brian Dolbec
2014-08-24 19:42 ` Brian Dolbec
2014-08-24 21:19   ` Michał Górny
2014-09-12  7:17 ` Michał Górny

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox