[gentoo-portage-dev] prefix portage chaining

[gentoo-portage-dev] prefix portage chaining

[Markus Duft]

Hey guys :)

Just wanted to stop by and get some opinions on a patch i wrote for the
prefix branch.

i'll try and explain what i want in the first place: i'm porting things
to native windows. since windows isn't too cooperative, i'm unable to
merge most things (and with other things, i simply don't want to), and
thus i need to take those things from somewhere else (more or less the
complete @system). I _am_ able to build all those things for interix
(which is the host system in the windows case). So what i want is a
setup of two prefix instances with a certain relation to each other: the
native windows prefix should be able to recognize installed packages
from the other instance, and resolve dependencies by eventually using
the other .../var/db/...

This could be (and is) quite usefull for all other platforms too. For
exmaple i could use prefix chaining on a linux box. I could create a
prefix containing a base system, and then for testing of
i-don't-know-whatever, i could create another small prefix inheriting
all installed packages from the other one. this way new prefixes can
stay very slim, but still the "parent" prefix is not altered on merges.

one issue not handled by the current patch is, that prefixes can have
different CHOST/ARCH/... (which is the case with x86-interix and
x86-winnt for example).

another thing is, that i plan to add some output in the merge list,
telling the user, which packages have been readonly-resolved from
another portage instance. right now the dependency is treated as if it
didn't exist.

all together, i'm pretty sure i did the one or the other forbidden thing
in my patch, but that's why i'm asking the guys-who-know :) it was hard
enough to read the portage source and get where i am, so i'm happy with
the result ;)

Waiting for comments, suggestions, etc.

Thanks in advance,
Cheers, Markus

Re: [gentoo-portage-dev] prefix portage chaining

[Markus Duft]

[-- Attachment #1: Type: text/plain, Size: 2219 bytes --]

On Fri, 2009-03-20 at 11:35 +0100, Markus Duft wrote:
> Hey guys :)
> 
> Just wanted to stop by and get some opinions on a patch i wrote for the
> prefix branch.

argh... managed to fail to attach the patch _again_ :) at least i
managed to add the output stuff i mentioned before to the patch in the
meantime. so this patch should be pretty much complete.

Thanks and Cheers, Markus

> 
> i'll try and explain what i want in the first place: i'm porting things
> to native windows. since windows isn't too cooperative, i'm unable to
> merge most things (and with other things, i simply don't want to), and
> thus i need to take those things from somewhere else (more or less the
> complete @system). I _am_ able to build all those things for interix
> (which is the host system in the windows case). So what i want is a
> setup of two prefix instances with a certain relation to each other: the
> native windows prefix should be able to recognize installed packages
> from the other instance, and resolve dependencies by eventually using
> the other .../var/db/...
> 
> This could be (and is) quite usefull for all other platforms too. For
> exmaple i could use prefix chaining on a linux box. I could create a
> prefix containing a base system, and then for testing of
> i-don't-know-whatever, i could create another small prefix inheriting
> all installed packages from the other one. this way new prefixes can
> stay very slim, but still the "parent" prefix is not altered on merges.
> 
> one issue not handled by the current patch is, that prefixes can have
> different CHOST/ARCH/... (which is the case with x86-interix and
> x86-winnt for example).
> 
> another thing is, that i plan to add some output in the merge list,
> telling the user, which packages have been readonly-resolved from
> another portage instance. right now the dependency is treated as if it
> didn't exist.
> 
> all together, i'm pretty sure i did the one or the other forbidden thing
> in my patch, but that's why i'm asking the guys-who-know :) it was hard
> enough to read the portage source and get where i am, so i'm happy with
> the result ;)
> 
> Waiting for comments, suggestions, etc.
> 
> Thanks in advance,
> Cheers, Markus
> 
> 

[-- Attachment #2: portage-chaining-2.patch --]
[-- Type: text/x-patch, Size: 11393 bytes --]

diff -ru portage.orig/pym/_emerge/__init__.py portage/pym/_emerge/__init__.py
--- portage.orig/pym/_emerge/__init__.py	2009-03-18 10:13:34.000000000 +0100
+++ portage/pym/_emerge/__init__.py	2009-03-20 12:08:49.000000000 +0100
@@ -49,7 +49,7 @@
 import portage.xpak, commands, errno, re, socket, time
 from portage.output import blue, bold, colorize, darkblue, darkgreen, darkred, green, \
 	nc_len, red, teal, turquoise, xtermTitle, \
-	xtermTitleReset, yellow
+	xtermTitleReset, yellow, purple
 from portage.output import create_color_func
 good = create_color_func("GOOD")
 bad = create_color_func("BAD")
@@ -69,6 +69,7 @@
 from portage.util import cmp_sort_key, writemsg, writemsg_level
 from portage.sets import load_default_config, SETPREFIX
 from portage.sets.base import InternalPackageSet
+from portage.dbapi.vartree import vartree
 
 from itertools import chain, izip
 
@@ -4692,6 +4693,7 @@
 		self._unsatisfied_deps_for_display = []
 		self._unsatisfied_blockers_for_display = None
 		self._circular_deps_for_display = None
+		self._injected_readonly_deps_for_display = []
 		self._dep_stack = []
 		self._unsatisfied_deps = []
 		self._initially_unsatisfied_deps = []
@@ -5270,18 +5272,35 @@
 		if removal_action and self.myopts.get("--with-bdeps", "y") == "n":
 			edepend["DEPEND"] = ""
 
+		# MDUFT: create additional vartrees for every readonly root here
+		# (maybe FakeVartree's?). Then below search those trees and set
+		# mypriority.satisfied to True.
+		# the ro_vartrees instances are created below as they are needed to
+		# avoid reading vartrees of portage instances which aren't required
+		# while resolving this dependencies.
+		ro_trees = {}
+		ro_vartrees = {}
+		
+		for type in ("DEPEND","RDEPEND", "PDEPEND"):
+			ro_trees[type] = []
+			
+			for ro_root, ro_dep_types in self.settings.readonly_roots.items():
+				if type in ro_dep_types:
+					ro_trees[type].append(ro_root)
+
 		deps = (
-			("/", edepend["DEPEND"],
+			("/", "DEPEND",
 				self._priority(buildtime=(not bdeps_optional),
 				optional=bdeps_optional)),
-			(myroot, edepend["RDEPEND"], self._priority(runtime=True)),
-			(myroot, edepend["PDEPEND"], self._priority(runtime_post=True))
+			(myroot, "RDEPEND", self._priority(runtime=True)),
+			(myroot, "PDEPEND", self._priority(runtime_post=True))
 		)
 
 		debug = "--debug" in self.myopts
 		strict = mytype != "installed"
 		try:
-			for dep_root, dep_string, dep_priority in deps:
+			for dep_root, dep_type, dep_priority in deps:
+				dep_string = edepend[dep_type]
 				if not dep_string:
 					continue
 				if debug:
@@ -5309,6 +5328,32 @@
 						if not atom.blocker and vardb.match(atom):
 							mypriority.satisfied = True
 
+						# MDUFT: How erver we do it - if we find "atom" beeing installed
+						# in a valid readonly root for the current dependency type, then
+						# _DONT_ call the below, but rather return 1 immediately.
+						for ro_root in ro_trees[dep_type]:
+							if not ro_vartrees.has_key(ro_root):
+								# target_root=ro_root ok? or should it be the real target_root?
+								_tmp_settings = portage.config(config_root=ro_root, target_root=ro_root,
+									config_incrementals=portage.const.INCREMENTALS)
+								#_tmp_trees = portage.util.LazyItemsDict()
+								#_tmp_trees.addLazySingleton("vartree", vartree, ro_root,
+								#	categories=_tmp_settings.categories, settings=_tmp_settings)
+								
+								#setconfig = load_default_config(_tmp_trees["vartree"].settings, _tmp_trees)
+								#ro_vartrees[ro_root] = FakeVartree(RootConfig(_tmp_trees["vartree"].settings,
+								#	_tmp_trees, setconfig), acquire_lock=0)
+								
+								ro_vartrees[ro_root] = vartree(root=ro_root, categories=_tmp_settings.categories, settings=_tmp_settings, kill_eprefix=True)
+									
+							ro_matches = ro_vartrees[ro_root].dbapi.match(atom)
+							
+							if ro_matches:
+								#TODO: print notice, just the same as the rest of the
+								# self.*_for_display stuff.
+								self._injected_readonly_deps_for_display.append({"ro_root" : ro_root, "atom" : atom, "matches": ro_matches, "type": dep_type})
+								return 1
+
 						if not self._add_dep(Dependency(atom=atom,
 							blocker=atom.blocker, depth=depth, parent=pkg,
 							priority=mypriority, root=dep_root),
@@ -8457,6 +8502,19 @@
 		for x in blockers:
 			print x
 
+		# print _injected_readonly_deps_for_display.
+		if len(self._injected_readonly_deps_for_display) > 0:
+			out.write("\n%s\n" % (darkgreen("Packages resolved from readonly installations:")))
+
+		for x in self._injected_readonly_deps_for_display:
+			tmp_type = x["type"]
+			while len(tmp_type) < 9:
+				tmp_type += " "
+			out.write("  [ %s ] %s %s %s (matched: %s)" % (teal(tmp_type), green(str(x["atom"])), yellow("from"), blue(x["ro_root"]), turquoise(str(x["matches"]))))
+
+		if len(self._injected_readonly_deps_for_display) > 0:
+			out.write("\n")
+
 		if verbosity == 3:
 			print
 			print counters
@@ -11825,6 +11883,7 @@
 		for x in candidate_catpkgs:
 			# cycle through all our candidate deps and determine
 			# what will and will not get unmerged
+
 			try:
 				mymatch = vartree.dbapi.match(x)
 			except portage.exception.AmbiguousPackageName, errpkgs:
diff -ru portage.orig/pym/portage/__init__.py portage/pym/portage/__init__.py
--- portage.orig/pym/portage/__init__.py	2009-03-18 10:13:35.000000000 +0100
+++ portage/pym/portage/__init__.py	2009-03-20 09:05:56.000000000 +0100
@@ -1236,6 +1236,7 @@
 
 			self._accept_license = copy.deepcopy(clone._accept_license)
 			self._plicensedict = copy.deepcopy(clone._plicensedict)
+			self.readonly_roots = copy.deepcopy(clone.readonly_roots)
 		else:
 
 			def check_var_directory(varname, var):
@@ -1802,6 +1803,50 @@
 			# but needs to be available using portageq
 			self["EPREFIX"] = EPREFIX
 
+			# expand READONLY_EROOT to a list of all readonly portage instances
+			# all the way down to the last one. beware that ATM a deeper instance
+			# in the chain can provide more than the toplevel! this means that
+			# if you only inherit DEPENDS from one instance, that instance may
+			# inherit RDEPENDs from another one, making the top-level instance
+			# inherit RDEPENDs from there too - even if the intermediate prefix
+			# does not do this.
+			self.readonly_roots = {}
+			my_ro_current_instance = config_root
+
+			while True:
+				my_ro_current_make_conf_file = os.path.join(my_ro_current_instance,MAKE_CONF_FILE.lstrip(os.path.sep))
+
+				if os.path.exists(my_ro_current_make_conf_file):
+					my_ro_cfg = getconfig(my_ro_current_make_conf_file)
+					
+					if my_ro_cfg.has_key("READONLY_EROOT"):
+
+						if not my_ro_cfg["READONLY_EROOT"].find(":"):
+							raise portage.exception.InvalidReadonlyRoot("ERROR: malformed READONLY_EROOT in %s" % (my_ro_current_make_conf_file))
+
+						(my_ro_cfg_root,my_ro_cfg_root_deps) = my_ro_cfg["READONLY_EROOT"].rsplit(":",1)
+
+						if not os.path.exists(my_ro_cfg_root):
+							raise portage.exception.InvalidReadonlyRoot("ERROR: malformed READONLY_EROOT in %s: path does not exist!" % (my_ro_current_instance))
+
+						if self.readonly_roots.has_key(my_ro_cfg_root):
+							raise portage.exception.InvalidReadonlyRoot("ERROR: circular READONLY_EROOT's in %s. %s already checked for %s" % (my_ro_current_make_conf_file, my_ro_cfg_root, self.readonly_roots[my_ro_cfg_root]))
+
+						if my_ro_cfg_root == config_root:
+							raise portage.exception.InvalidReadonlyRoot("ERROR: cannot add this instance as READONLY_EROOT in %s." % (my_ro_current_make_conf_file))
+
+						self.readonly_roots[my_ro_cfg_root] = my_ro_cfg_root_deps.split(",")
+						my_ro_current_instance = my_ro_cfg_root
+
+						continue
+
+				break
+
+			if len(self.readonly_roots) > 0:
+				writemsg("Found readonly portage instance to resolve dependencies:\n",noiselevel=0)
+				for k,v in self.readonly_roots.items():
+					writemsg("%20s %-s\n" % (v,k), noiselevel=0)
+
 			self._init_dirs()
 
 		if mycpv:
diff -ru portage.orig/pym/portage/dbapi/vartree.py portage/pym/portage/dbapi/vartree.py
--- portage.orig/pym/portage/dbapi/vartree.py	2009-03-18 10:13:35.000000000 +0100
+++ portage/pym/portage/dbapi/vartree.py	2009-03-20 11:09:49.000000000 +0100
@@ -1270,9 +1270,20 @@
 		self._counter_path = os.path.join(root,
 			CACHE_PATH.lstrip(os.path.sep), "counter")
 
+		plibreg_path = os.path.join(self.root, EPREFIX_LSTRIP, PRIVATE_PATH, "preserved_libs_registry")
+
+		if vartree:
+			self._kill_eprefix = vartree._kill_eprefix
+		else:
+			self._kill_eprefix = False
+
+		if self._kill_eprefix:
+			self._aux_cache_filename = os.path.join(self.root, self._aux_cache_filename.lstrip(EPREFIX.rstrip('/')))
+			self._counter_path = os.path.join(self.root, self._counter_path.lstrip(EPREFIX.rstrip('/')))
+			plibreg_path = os.path.join(self.root, plibreg_path.lstrip(EPREFIX.rstrip('/')))
+
 		try:
-			self.plib_registry = PreservedLibsRegistry(
-				os.path.join(self.root, EPREFIX_LSTRIP, PRIVATE_PATH, "preserved_libs_registry"))
+			self.plib_registry = PreservedLibsRegistry(plibreg_path)
 		except PermissionDenied:
 			# apparently this user isn't allowed to access PRIVATE_PATH
 			self.plib_registry = None
@@ -1285,6 +1296,10 @@
 
 	def getpath(self, mykey, filename=None):
 		rValue = os.path.join(self.root, VDB_PATH, mykey)
+
+		if self._kill_eprefix:
+			rValue = os.path.join(self.root, rValue.lstrip(EPREFIX.rstrip('/')))
+
 		if filename != None:
 			rValue = os.path.join(rValue, filename)
 		return rValue
@@ -1433,6 +1448,9 @@
 		returnme = []
 		basepath = os.path.join(self.root, VDB_PATH) + os.path.sep
 
+		if self._kill_eprefix:
+			basepath = os.path.join(self.root, basepath.lstrip(EPREFIX.rstrip('/')))
+
 		if use_cache:
 			from portage import listdir
 		else:
@@ -1522,7 +1540,12 @@
 			return list(self._iter_match(mydep,
 				self.cp_list(mydep.cp, use_cache=use_cache)))
 		try:
-			curmtime = os.stat(self.root+VDB_PATH+"/"+mycat).st_mtime
+			_tmp_path = self.root+VDB_PATH+"/"+mycat
+			
+			if self._kill_eprefix:
+				_tmp_path = os.path.join(self.root, _tmp_path.lstrip(EPREFIX.rstrip('/')))
+
+			curmtime = os.stat(_tmp_path).st_mtime
 		except (IOError, OSError):
 			curmtime=0
 
@@ -2063,7 +2086,7 @@
 class vartree(object):
 	"this tree will scan a var/db/pkg database located at root (passed to init)"
 	def __init__(self, root="/", virtual=None, clone=None, categories=None,
-		settings=None):
+		settings=None, kill_eprefix=False):
 		if clone:
 			writemsg("vartree.__init__(): deprecated " + \
 				"use of clone parameter\n", noiselevel=-1)
@@ -2072,6 +2095,7 @@
 			self.populated = 1
 			from portage import config
 			self.settings = config(clone=clone.settings)
+			self._kill_eprefix = clone._kill_eprefix
 		else:
 			self.root = root[:]
 			if settings is None:
@@ -2079,6 +2103,7 @@
 			self.settings = settings # for key_expand calls
 			if categories is None:
 				categories = settings.categories
+			self._kill_eprefix=kill_eprefix
 			self.dbapi = vardbapi(self.root, categories=categories,
 				settings=settings, vartree=self)
 			self.populated = 1
@@ -2110,6 +2135,10 @@
 			raise
 		except Exception, e:
 			mydir = os.path.join(self.root, VDB_PATH, mycpv)
+
+			if self._kill_eprefix:
+				mydir = os.path.join(self.root, mydir.lstrip(EPREFIX.rstrip('/')))
+
 			writemsg("\nParse Error reading PROVIDE and USE in '%s'\n" % mydir,
 				noiselevel=-1)
 			if mylines:

Re: [gentoo-portage-dev] prefix portage chaining

[Markus Duft]

On Fri, 2009-03-20 at 12:11 +0100, Markus Duft wrote:
> On Fri, 2009-03-20 at 11:35 +0100, Markus Duft wrote:
> > Hey guys :)
> > 
> > Just wanted to stop by and get some opinions on a patch i wrote for the
> > prefix branch.
> 
[snip]

Hey there. Seemingly nobody has any comments on this..? :) here's a
working version of the patch. i realized that the last one worked only
"by accident" :) maybe now, some comments (or questions)?

Thanks, Cheers, Markus

[snip]
> > 
> > Waiting for comments, suggestions, etc.
> > 
> > Thanks in advance,
> > Cheers, Markus
> > 
> > 

Re: [gentoo-portage-dev] prefix portage chaining

[Markus Duft]

[-- Attachment #1: Type: text/plain, Size: 790 bytes --]

On Mon, 2009-03-23 at 08:27 +0100, Markus Duft wrote:
> On Fri, 2009-03-20 at 12:11 +0100, Markus Duft wrote:
> > On Fri, 2009-03-20 at 11:35 +0100, Markus Duft wrote:
> > > Hey guys :)
> > > 
> > > Just wanted to stop by and get some opinions on a patch i wrote for the
> > > prefix branch.
> > 
> [snip]
> 
> Hey there. Seemingly nobody has any comments on this..? :) here's a
> working version of the patch. i realized that the last one worked only
> "by accident" :) maybe now, some comments (or questions)?

i'm eager to find out how many times i can forget to attach the patch :)

sorry for the noise

Cheers, Markus

> 
> Thanks, Cheers, Markus
> 
> [snip]
> > > 
> > > Waiting for comments, suggestions, etc.
> > > 
> > > Thanks in advance,
> > > Cheers, Markus
> > > 
> > > 
> 
> 

[-- Attachment #2: portage-chaining-6.patch --]
[-- Type: text/x-patch, Size: 11497 bytes --]

diff -ru portage.orig/pym/_emerge/__init__.py portage/pym/_emerge/__init__.py
--- portage.orig/pym/_emerge/__init__.py	2009-03-18 10:13:34.000000000 +0100
+++ portage/pym/_emerge/__init__.py	2009-03-23 08:46:59.000000000 +0100
@@ -49,7 +49,7 @@
 import portage.xpak, commands, errno, re, socket, time
 from portage.output import blue, bold, colorize, darkblue, darkgreen, darkred, green, \
 	nc_len, red, teal, turquoise, xtermTitle, \
-	xtermTitleReset, yellow
+	xtermTitleReset, yellow, purple
 from portage.output import create_color_func
 good = create_color_func("GOOD")
 bad = create_color_func("BAD")
@@ -69,6 +69,7 @@
 from portage.util import cmp_sort_key, writemsg, writemsg_level
 from portage.sets import load_default_config, SETPREFIX
 from portage.sets.base import InternalPackageSet
+from portage.dbapi.vartree import vartree
 
 from itertools import chain, izip
 
@@ -4692,6 +4693,7 @@
 		self._unsatisfied_deps_for_display = []
 		self._unsatisfied_blockers_for_display = None
 		self._circular_deps_for_display = None
+		self._injected_readonly_deps_for_display = []
 		self._dep_stack = []
 		self._unsatisfied_deps = []
 		self._initially_unsatisfied_deps = []
@@ -5270,18 +5272,35 @@
 		if removal_action and self.myopts.get("--with-bdeps", "y") == "n":
 			edepend["DEPEND"] = ""
 
+		# MDUFT: create additional vartrees for every readonly root here
+		# (maybe FakeVartree's?). Then below search those trees and set
+		# mypriority.satisfied to True.
+		# the ro_vartrees instances are created below as they are needed to
+		# avoid reading vartrees of portage instances which aren't required
+		# while resolving this dependencies.
+		ro_trees = {}
+		ro_vartrees = {}
+		
+		for type in ("DEPEND","RDEPEND", "PDEPEND"):
+			ro_trees[type] = []
+			
+			for ro_root, ro_dep_types in self.settings.readonly_roots.items():
+				if type in ro_dep_types:
+					ro_trees[type].append(ro_root)
+
 		deps = (
-			("/", edepend["DEPEND"],
+			("/", "DEPEND",
 				self._priority(buildtime=(not bdeps_optional),
 				optional=bdeps_optional)),
-			(myroot, edepend["RDEPEND"], self._priority(runtime=True)),
-			(myroot, edepend["PDEPEND"], self._priority(runtime_post=True))
+			(myroot, "RDEPEND", self._priority(runtime=True)),
+			(myroot, "PDEPEND", self._priority(runtime_post=True))
 		)
 
 		debug = "--debug" in self.myopts
 		strict = mytype != "installed"
 		try:
-			for dep_root, dep_string, dep_priority in deps:
+			for dep_root, dep_type, dep_priority in deps:
+				dep_string = edepend[dep_type]
 				if not dep_string:
 					continue
 				if debug:
@@ -5309,6 +5328,43 @@
 						if not atom.blocker and vardb.match(atom):
 							mypriority.satisfied = True
 
+						# MDUFT: How erver we do it - if we find "atom" beeing installed
+						# in a valid readonly root for the current dependency type, then
+						# _DONT_ call the below, but rather return 1 immediately.
+						ro_matched = False
+						for ro_root in ro_trees[dep_type]:
+							if not ro_vartrees.has_key(ro_root):
+								# target_root=ro_root ok? or should it be the real target_root?
+								_tmp_settings = portage.config(config_root=ro_root, target_root=ro_root,
+									config_incrementals=portage.const.INCREMENTALS)
+								#_tmp_trees = portage.util.LazyItemsDict()
+								#_tmp_trees.addLazySingleton("vartree", vartree, ro_root,
+								#	categories=_tmp_settings.categories, settings=_tmp_settings)
+								
+								#setconfig = load_default_config(_tmp_trees["vartree"].settings, _tmp_trees)
+								#ro_vartrees[ro_root] = FakeVartree(RootConfig(_tmp_trees["vartree"].settings,
+								#	_tmp_trees, setconfig), acquire_lock=0)
+								
+								ro_vartrees[ro_root] = vartree(root=ro_root, categories=_tmp_settings.categories, 
+									settings=_tmp_settings, kill_eprefix=True)
+									
+							ro_matches = ro_vartrees[ro_root].dbapi.match(atom)
+							
+							if ro_matches:
+								if dep_type is "RDEPEND":
+									# we need to assure binary compatability, so it needs to be
+									# the same CHOST! But how? for now i cannot do anything...
+									pass
+
+								# injected dep for display. those get shown with the merge list.
+								self._injected_readonly_deps_for_display.append({"ro_root" : ro_root, "atom" : atom, 
+									"matches": ro_matches, "type": dep_type, "parent": pkg})
+								ro_matched = True
+								break
+								
+						if ro_matched:
+							continue
+
 						if not self._add_dep(Dependency(atom=atom,
 							blocker=atom.blocker, depth=depth, parent=pkg,
 							priority=mypriority, root=dep_root),
@@ -8457,6 +8513,21 @@
 		for x in blockers:
 			print x
 
+		# print _injected_readonly_deps_for_display.
+		if len(self._injected_readonly_deps_for_display) > 0:
+			out.write("\n%s\n\n" % (darkgreen("Packages resolved from readonly installations:")))
+
+		for x in self._injected_readonly_deps_for_display:
+			tmp_type = x["type"].replace("END","")
+			while len(tmp_type) < 4:
+				tmp_type += " "
+			out.write("[%s %s] %s %s %s (%s by %s)\n" % (teal("external"), 
+				green(tmp_type), green(str(x["matches"][0])), yellow("from"), 
+				blue(x["ro_root"]), turquoise(str(x["atom"])), green(x["parent"].cpv)))
+
+		if len(self._injected_readonly_deps_for_display) > 0:
+			out.write("\n")
+
 		if verbosity == 3:
 			print
 			print counters
@@ -11825,6 +11896,7 @@
 		for x in candidate_catpkgs:
 			# cycle through all our candidate deps and determine
 			# what will and will not get unmerged
+
 			try:
 				mymatch = vartree.dbapi.match(x)
 			except portage.exception.AmbiguousPackageName, errpkgs:
diff -ru portage.orig/pym/portage/__init__.py portage/pym/portage/__init__.py
--- portage.orig/pym/portage/__init__.py	2009-03-18 10:13:35.000000000 +0100
+++ portage/pym/portage/__init__.py	2009-03-20 12:12:34.000000000 +0100
@@ -1236,6 +1236,7 @@
 
 			self._accept_license = copy.deepcopy(clone._accept_license)
 			self._plicensedict = copy.deepcopy(clone._plicensedict)
+			self.readonly_roots = copy.deepcopy(clone.readonly_roots)
 		else:
 
 			def check_var_directory(varname, var):
@@ -1802,6 +1803,45 @@
 			# but needs to be available using portageq
 			self["EPREFIX"] = EPREFIX
 
+			# expand READONLY_EROOT to a list of all readonly portage instances
+			# all the way down to the last one. beware that ATM a deeper instance
+			# in the chain can provide more than the toplevel! this means that
+			# if you only inherit DEPENDS from one instance, that instance may
+			# inherit RDEPENDs from another one, making the top-level instance
+			# inherit RDEPENDs from there too - even if the intermediate prefix
+			# does not do this.
+			self.readonly_roots = {}
+			my_ro_current_instance = config_root
+
+			while True:
+				my_ro_current_make_conf_file = os.path.join(my_ro_current_instance,MAKE_CONF_FILE.lstrip(os.path.sep))
+
+				if os.path.exists(my_ro_current_make_conf_file):
+					my_ro_cfg = getconfig(my_ro_current_make_conf_file)
+					
+					if my_ro_cfg.has_key("READONLY_EROOT"):
+
+						if not my_ro_cfg["READONLY_EROOT"].find(":"):
+							raise portage.exception.InvalidReadonlyRoot("ERROR: malformed READONLY_EROOT in %s" % (my_ro_current_make_conf_file))
+
+						(my_ro_cfg_root,my_ro_cfg_root_deps) = my_ro_cfg["READONLY_EROOT"].rsplit(":",1)
+
+						if not os.path.exists(my_ro_cfg_root):
+							raise portage.exception.InvalidReadonlyRoot("ERROR: malformed READONLY_EROOT in %s: path does not exist!" % (my_ro_current_instance))
+
+						if self.readonly_roots.has_key(my_ro_cfg_root):
+							raise portage.exception.InvalidReadonlyRoot("ERROR: circular READONLY_EROOT's in %s. %s already checked for %s" % (my_ro_current_make_conf_file, my_ro_cfg_root, self.readonly_roots[my_ro_cfg_root]))
+
+						if my_ro_cfg_root == config_root:
+							raise portage.exception.InvalidReadonlyRoot("ERROR: cannot add this instance as READONLY_EROOT in %s." % (my_ro_current_make_conf_file))
+
+						self.readonly_roots[my_ro_cfg_root] = my_ro_cfg_root_deps.split(",")
+						my_ro_current_instance = my_ro_cfg_root
+
+						continue
+
+				break
+
 			self._init_dirs()
 
 		if mycpv:
diff -ru portage.orig/pym/portage/dbapi/vartree.py portage/pym/portage/dbapi/vartree.py
--- portage.orig/pym/portage/dbapi/vartree.py	2009-03-18 10:13:35.000000000 +0100
+++ portage/pym/portage/dbapi/vartree.py	2009-03-23 08:23:23.000000000 +0100
@@ -1270,9 +1270,20 @@
 		self._counter_path = os.path.join(root,
 			CACHE_PATH.lstrip(os.path.sep), "counter")
 
+		plibreg_path = os.path.join(self.root, EPREFIX_LSTRIP, PRIVATE_PATH, "preserved_libs_registry")
+
+		if vartree:
+			self._kill_eprefix = vartree._kill_eprefix
+		else:
+			self._kill_eprefix = False
+
+		if self._kill_eprefix:
+			self._aux_cache_filename = os.path.join(self.root, self._aux_cache_filename.replace(EPREFIX, ""))
+			self._counter_path = os.path.join(self.root, self._counter_path.replace(EPREFIX, ""))
+			plibreg_path = os.path.join(self.root, plibreg_path.replace(EPREFIX, ""))
+
 		try:
-			self.plib_registry = PreservedLibsRegistry(
-				os.path.join(self.root, EPREFIX_LSTRIP, PRIVATE_PATH, "preserved_libs_registry"))
+			self.plib_registry = PreservedLibsRegistry(plibreg_path)
 		except PermissionDenied:
 			# apparently this user isn't allowed to access PRIVATE_PATH
 			self.plib_registry = None
@@ -1285,6 +1296,10 @@
 
 	def getpath(self, mykey, filename=None):
 		rValue = os.path.join(self.root, VDB_PATH, mykey)
+
+		if self._kill_eprefix:
+			rValue = os.path.join(self.root, rValue.replace(EPREFIX, ""))
+
 		if filename != None:
 			rValue = os.path.join(rValue, filename)
 		return rValue
@@ -1433,6 +1448,9 @@
 		returnme = []
 		basepath = os.path.join(self.root, VDB_PATH) + os.path.sep
 
+		if self._kill_eprefix:
+			basepath = os.path.join(self.root, basepath.replace(EPREFIX, ""))
+
 		if use_cache:
 			from portage import listdir
 		else:
@@ -1522,7 +1540,12 @@
 			return list(self._iter_match(mydep,
 				self.cp_list(mydep.cp, use_cache=use_cache)))
 		try:
-			curmtime = os.stat(self.root+VDB_PATH+"/"+mycat).st_mtime
+			_tmp_path = self.root+VDB_PATH+"/"+mycat
+			
+			if self._kill_eprefix:
+				_tmp_path = os.path.join(self.root, _tmp_path.replace(EPREFIX, ""))
+
+			curmtime = os.stat(_tmp_path).st_mtime
 		except (IOError, OSError):
 			curmtime=0
 
@@ -2063,7 +2086,7 @@
 class vartree(object):
 	"this tree will scan a var/db/pkg database located at root (passed to init)"
 	def __init__(self, root="/", virtual=None, clone=None, categories=None,
-		settings=None):
+		settings=None, kill_eprefix=False):
 		if clone:
 			writemsg("vartree.__init__(): deprecated " + \
 				"use of clone parameter\n", noiselevel=-1)
@@ -2072,6 +2095,7 @@
 			self.populated = 1
 			from portage import config
 			self.settings = config(clone=clone.settings)
+			self._kill_eprefix = clone._kill_eprefix
 		else:
 			self.root = root[:]
 			if settings is None:
@@ -2079,6 +2103,7 @@
 			self.settings = settings # for key_expand calls
 			if categories is None:
 				categories = settings.categories
+			self._kill_eprefix=kill_eprefix
 			self.dbapi = vardbapi(self.root, categories=categories,
 				settings=settings, vartree=self)
 			self.populated = 1
@@ -2110,6 +2135,10 @@
 			raise
 		except Exception, e:
 			mydir = os.path.join(self.root, VDB_PATH, mycpv)
+
+			if self._kill_eprefix:
+				mydir = os.path.join(self.root, mydir.replace(EPREFIX, ""))
+
 			writemsg("\nParse Error reading PROVIDE and USE in '%s'\n" % mydir,
 				noiselevel=-1)
 			if mylines:

Re: [gentoo-portage-dev] prefix portage chaining

[Fabian Groffen]

On 20-03-2009 11:35:09 +0100, Markus Duft wrote:
> i'll try and explain what i want in the first place: i'm porting things
> to native windows. since windows isn't too cooperative, i'm unable to
> merge most things (and with other things, i simply don't want to), and
> thus i need to take those things from somewhere else (more or less the
> complete @system). I _am_ able to build all those things for interix
> (which is the host system in the windows case). So what i want is a
> setup of two prefix instances with a certain relation to each other: the
> native windows prefix should be able to recognize installed packages
> from the other instance, and resolve dependencies by eventually using
> the other .../var/db/...

Since Windows and Interix seem to be two different things to me, can you
explain why in this case Portage can resolve the dependencies from the
Interix system to use for the Windows system?  What dependencies are we
talking about?  Build tools?  Libraries?  Runtime dependencies?

> This could be (and is) quite usefull for all other platforms too. For
> exmaple i could use prefix chaining on a linux box. I could create a
> prefix containing a base system, and then for testing of
> i-don't-know-whatever, i could create another small prefix inheriting
> all installed packages from the other one. this way new prefixes can
> stay very slim, but still the "parent" prefix is not altered on merges.

That potentially is useful, in the case where you want to upgrade a
critical system package and test it out or something.  Can't think of an
example other than Portage itself for the moment, though.  (And that one
can be hairy, for instance if the vdb format changes NEEDED ->
NEEDED.ELF.2)

> one issue not handled by the current patch is, that prefixes can have
> different CHOST/ARCH/... (which is the case with x86-interix and
> x86-winnt for example).

Then how does it work for you?


-- 
Fabian Groffen
Gentoo on a different level

Re: [gentoo-portage-dev] prefix portage chaining

[Markus Duft]

On Wed, 2009-03-25 at 12:00 +0100, Fabian Groffen wrote:
> On 20-03-2009 11:35:09 +0100, Markus Duft wrote:
> > i'll try and explain what i want in the first place: i'm porting things
> > to native windows. since windows isn't too cooperative, i'm unable to
> > merge most things (and with other things, i simply don't want to), and
> > thus i need to take those things from somewhere else (more or less the
> > complete @system). I _am_ able to build all those things for interix
> > (which is the host system in the windows case). So what i want is a
> > setup of two prefix instances with a certain relation to each other: the
> > native windows prefix should be able to recognize installed packages
> > from the other instance, and resolve dependencies by eventually using
> > the other .../var/db/...
> 
> Since Windows and Interix seem to be two different things to me, can you
> explain why in this case Portage can resolve the dependencies from the
> Interix system to use for the Windows system?  What dependencies are we
> talking about?  Build tools?  Libraries?  Runtime dependencies?

i'm using it to resolve DEPEND atoms only. of course my notion of valid
DEPENDs and RDEPENDs is influenced by this, and i'm alergic against
RDEPEND=$DEPEND and such :) since it doesn't work in this setup. however
right now most things i need don't make too much problems.

> 
> > This could be (and is) quite usefull for all other platforms too. For
> > exmaple i could use prefix chaining on a linux box. I could create a
> > prefix containing a base system, and then for testing of
> > i-don't-know-whatever, i could create another small prefix inheriting
> > all installed packages from the other one. this way new prefixes can
> > stay very slim, but still the "parent" prefix is not altered on merges.
> 
> That potentially is useful, in the case where you want to upgrade a
> critical system package and test it out or something.  Can't think of an
> example other than Portage itself for the moment, though.  (And that one
> can be hairy, for instance if the vdb format changes NEEDED ->
> NEEDED.ELF.2)

++ :)

> 
> > one issue not handled by the current patch is, that prefixes can have
> > different CHOST/ARCH/... (which is the case with x86-interix and
> > x86-winnt for example).
> 
> Then how does it work for you?

as i said, i'm using DEPENDs only from the other prefix with the
different CHOST/ARCH. this works, since on interix i can execute windows
binaries, and vice versa (limited).

the patch i proposed here and on gentoo-alt@ (btw. i have a fixed
version lying around since a few minutes ...) allows the user to set
which atoms should be resolve-able from the chain. for exmaple for
windows i'm doing this in /my/winnt/prefix/etc/make.conf:

READONLY_EROOT=/my/interix/prefix:DEPEND

on linux, if both prefixes are x86-linux for example i could to
in /my/prefix/two/etc/make.conf:

READONLY_EROOT=/my/prefix/one:DEPEND,RDEPEND

(btw. this forces PDEPENDs to merge in /my/prefix/two. i guess most of
the time this makes sense, since with a PDEPEND from a lib, i want the
PDEPEND package to link against this lib... you know what i mean? of
course PDEPEND can still be added to the above list...)

(btw2. the name READONLY_EROOT is discussed on gentoo-alt@ already, so i
won't comment on it beeing cool/uncool here... ;) )

Cheers, Markus

> 
> 

Re: [gentoo-portage-dev] prefix portage chaining

[Ned Ludd]

On Wed, 2009-03-25 at 14:14 +0100, Markus Duft wrote:
> On Wed, 2009-03-25 at 12:00 +0100, Fabian Groffen wrote:
> > On 20-03-2009 11:35:09 +0100, Markus Duft wrote:
> > > i'll try and explain what i want in the first place: i'm porting things
> > > to native windows. since windows isn't too cooperative, i'm unable to
> > > merge most things (and with other things, i simply don't want to), and
> > > thus i need to take those things from somewhere else (more or less the
> > > complete @system). I _am_ able to build all those things for interix
> > > (which is the host system in the windows case). So what i want is a
> > > setup of two prefix instances with a certain relation to each other: the
> > > native windows prefix should be able to recognize installed packages
> > > from the other instance, and resolve dependencies by eventually using
> > > the other .../var/db/...
> > 
> > Since Windows and Interix seem to be two different things to me, can you
> > explain why in this case Portage can resolve the dependencies from the
> > Interix system to use for the Windows system?  What dependencies are we
> > talking about?  Build tools?  Libraries?  Runtime dependencies?
> 
> i'm using it to resolve DEPEND atoms only. of course my notion of valid
> DEPENDs and RDEPENDs is influenced by this, and i'm alergic against
> RDEPEND=$DEPEND and such :) since it doesn't work in this setup. however
> right now most things i need don't make too much problems.
> 
> > 
> > > This could be (and is) quite usefull for all other platforms too. For
> > > exmaple i could use prefix chaining on a linux box. I could create a
> > > prefix containing a base system, and then for testing of
> > > i-don't-know-whatever, i could create another small prefix inheriting
> > > all installed packages from the other one. this way new prefixes can
> > > stay very slim, but still the "parent" prefix is not altered on merges.
> > 
> > That potentially is useful, in the case where you want to upgrade a
> > critical system package and test it out or something.  Can't think of an
> > example other than Portage itself for the moment, though.  (And that one
> > can be hairy, for instance if the vdb format changes NEEDED ->
> > NEEDED.ELF.2)
> 
> ++ :)
> 
> > 
> > > one issue not handled by the current patch is, that prefixes can have
> > > different CHOST/ARCH/... (which is the case with x86-interix and
> > > x86-winnt for example).
> > 
> > Then how does it work for you?
> 
> as i said, i'm using DEPENDs only from the other prefix with the
> different CHOST/ARCH. this works, since on interix i can execute windows
> binaries, and vice versa (limited).
> 
> the patch i proposed here and on gentoo-alt@ (btw. i have a fixed
> version lying around since a few minutes ...) allows the user to set
> which atoms should be resolve-able from the chain. for exmaple for
> windows i'm doing this in /my/winnt/prefix/etc/make.conf:
> 
> READONLY_EROOT=/my/interix/prefix:DEPEND
> 
> on linux, if both prefixes are x86-linux for example i could to
> in /my/prefix/two/etc/make.conf:
> 
> READONLY_EROOT=/my/prefix/one:DEPEND,RDEPEND
> 
> (btw. this forces PDEPENDs to merge in /my/prefix/two. i guess most of
> the time this makes sense, since with a PDEPEND from a lib, i want the
> PDEPEND package to link against this lib... you know what i mean? of
> course PDEPEND can still be added to the above list...)
> 
> (btw2. the name READONLY_EROOT is discussed on gentoo-alt@ already, so i
> won't comment on it beeing cool/uncool here... ;) )
> 
> Cheers, Markus
> 
> > 
> > 
> 
> \


While much of what you are talking about here mainly applies to prefix,
it looks to me from glancing over the code that you might of solved a
long standing problem in the embedded world with cross compiling via
portage. 222895  If that is the case, then I owe you a beer. one about
the size of a keg.


-- 
Ned Ludd <solar@gentoo.org>
Gentoo Linux

Re: [gentoo-portage-dev] prefix portage chaining

[Markus Duft]

[-- Attachment #1: Type: text/plain, Size: 1347 bytes --]

On Wed, 2009-03-25 at 11:44 -0700, Ned Ludd wrote:
[snip]
> 
> 
> While much of what you are talking about here mainly applies to prefix,
> it looks to me from glancing over the code that you might of solved a
> long standing problem in the embedded world with cross compiling via
> portage. 222895  If that is the case, then I owe you a beer. one about
> the size of a keg.
> 

lol, thx for the beer ;)

hmm... looking over that patch again, the only EPREFIX dependent thing
is, that i'm removing EPREFIX from the vartree class again :) so this
should pretty much plain apply to main too, and simply work. you may
want to rename READONLY_EPREFIX to READONLY_ROOT, but thats it :)

the other stuff besides portage modification (baselayout patchery, etc.
is prefix specific again, so all you'd need is the portage changes.

if you will try it, please let me know if it worked :) with the attached
patch "sed -i -e 's,READONLY_EPREFIX,READONLY_ROOT,g'", and applying to
an installed /usr/lib/portage should enable you to do it.
(backup /usr/lib/portage - i trust my work, but... we never know for
sure :))

then add to make.conf: READONLY_ROOT=/my/other/root:DEPEND

i hope this is what you where looking for...! and i hope it doesn't
somehow clash with the existing cross compile logic in portage regarding
where to merge to...

Cheers, Markus

> 

[-- Attachment #2: portage-chaining-11.patch --]
[-- Type: text/x-patch, Size: 16980 bytes --]

diff -ru portage.orig/bin/ebuild.sh portage/bin/ebuild.sh
--- portage.orig/bin/ebuild.sh	2009-03-24 16:44:15.000000000 +0100
+++ portage/bin/ebuild.sh	2009-03-25 07:32:07.000000000 +0100
@@ -69,7 +69,38 @@
 # Unset some variables that break things.
 unset GZIP BZIP BZIP2 CDPATH GREP_OPTIONS GREP_COLOR GLOBIGNORE
 
-export PATH="${DEFAULT_PATH}:$PORTAGE_BIN_PATH/ebuild-helpers:${ROOTPATH}"
+if [[ -n "${PORTAGE_READONLY_EROOTS}" ]]; then
+	new_PATH="$PORTAGE_BIN_PATH/ebuild-helpers:${ROOTPATH}"
+
+	save_IFS=$IFS
+	IFS=':'
+
+	for eroot in ${PORTAGE_READONLY_EROOTS}:${EPREFIX}; do
+		IFS=$save_IFS
+		[[ -f ${eroot}/usr/share/portage/config/make.globals ]] || continue
+		defpath="$(. ${eroot}/etc/make.globals && echo $DEFAULT_PATH)"
+		okpath=
+		save_IFS2=$IFS
+		IFS=':'
+		for p in $defpath; do
+			IFS=$save_IFS2
+			[[ ":${new_PATH}:" == *":$p:"* ]] && continue
+			if [[ -z "${okpath}" ]]; then
+				okpath="${p}"
+			else
+				okpath="${okpath}:${p}"
+			fi
+		done
+		IFS=$save_IFS2
+
+		new_PATH="${okpath}:${new_PATH}"
+	done
+	IFS=$save_IFS
+
+	export PATH=$new_PATH
+else
+	export PATH="${DEFAULT_PATH}:$PORTAGE_BIN_PATH/ebuild-helpers:${ROOTPATH}"
+fi
 [ ! -z "$PREROOTPATH" ] && export PATH="${PREROOTPATH%%:}:$PATH"
 
 source "${PORTAGE_BIN_PATH}/isolated-functions.sh"  &>/dev/null
diff -ru portage.orig/pym/_emerge/__init__.py portage/pym/_emerge/__init__.py
--- portage.orig/pym/_emerge/__init__.py	2009-03-24 16:43:02.000000000 +0100
+++ portage/pym/_emerge/__init__.py	2009-03-24 16:42:51.000000000 +0100
@@ -5269,17 +5269,18 @@
 			edepend["DEPEND"] = ""
 
 		deps = (
-			("/", edepend["DEPEND"],
+			("/", "DEPEND",
 				self._priority(buildtime=(not bdeps_optional),
 				optional=bdeps_optional)),
-			(myroot, edepend["RDEPEND"], self._priority(runtime=True)),
-			(myroot, edepend["PDEPEND"], self._priority(runtime_post=True))
+			(myroot, "RDEPEND", self._priority(runtime=True)),
+			(myroot, "PDEPEND", self._priority(runtime_post=True))
 		)
 
 		debug = "--debug" in self.myopts
 		strict = mytype != "installed"
 		try:
-			for dep_root, dep_string, dep_priority in deps:
+			for dep_root, dep_type, dep_priority in deps:
+				dep_string = edepend[dep_type]
 				if not dep_string:
 					continue
 				if debug:
@@ -5289,9 +5290,11 @@
 					print "Priority:", dep_priority
 				vardb = self.roots[dep_root].trees["vartree"].dbapi
 				try:
+					# MDUFT: selected_atoms will not contain anything
+					# that can be resolved from a readonly root!
 					selected_atoms = self._select_atoms(dep_root,
 						dep_string, myuse=myuse, parent=pkg, strict=strict,
-						priority=dep_priority)
+						priority=dep_priority, dep_type=dep_type)
 				except portage.exception.InvalidDependString, e:
 					show_invalid_depstring_notice(jbigkey, dep_string, str(e))
 					return 0
@@ -5916,12 +5919,18 @@
 		return self._select_atoms_highest_available(*pargs, **kwargs)
 
 	def _select_atoms_highest_available(self, root, depstring,
-		myuse=None, parent=None, strict=True, trees=None, priority=None):
+		myuse=None, parent=None, strict=True, trees=None, priority=None, dep_type=None):
 		"""This will raise InvalidDependString if necessary. If trees is
 		None then self._filtered_trees is used."""
 		pkgsettings = self.pkgsettings[root]
 		if trees is None:
 			trees = self._filtered_trees
+
+		# this one is needed to guarantee good readonly root
+		# resolution display in the merge list. required since
+		# parent (below) can be None
+		trees[root]["disp_parent"] = parent
+
 		if not getattr(priority, "buildtime", False):
 			# The parent should only be passed to dep_check() for buildtime
 			# dependencies since that's the only case when it's appropriate
@@ -5938,7 +5947,7 @@
 					portage.dep._dep_check_strict = False
 				mycheck = portage.dep_check(depstring, None,
 					pkgsettings, myuse=myuse,
-					myroot=root, trees=trees)
+					myroot=root, trees=trees, dep_type=dep_type)
 			finally:
 				if parent is not None:
 					trees[root].pop("parent")
@@ -5946,6 +5955,8 @@
 			if not mycheck[0]:
 				raise portage.exception.InvalidDependString(mycheck[1])
 			selected_atoms = mycheck[1]
+
+		trees[root].pop("disp_parent")
 		return selected_atoms
 
 	def _show_unsatisfied_dep(self, root, atom, myparent=None, arg=None):
@@ -8455,6 +8466,37 @@
 		for x in blockers:
 			print x
 
+		# print readonly selected packages
+		if len(portage.ro_selected) > 0:
+			out.write("\n%s\n\n" % (darkgreen("Packages resolved from readonly installations:")))
+
+		ro_mismatch_warning = False
+		ro_dupcheck = []
+		for x in portage.ro_selected:
+			tmp_type = x["type"].replace("END","")
+			while len(tmp_type) < 4:
+				tmp_type += " "
+			if str(x["atom"]) not in ro_dupcheck:
+				out.write("[%s %s] %s %s %s (%s by %s)" % (teal("readonly"), 
+					green(tmp_type), green(str(x["matches"][0])), yellow("from"), 
+					blue(x["ro_root"]), turquoise(str(x["atom"])), green(x["parent"].cpv)))
+
+				ro_dupcheck.append(str(x["atom"]))
+
+				if x["host_mismatch"]:
+					ro_mismatch_warning = True
+					out.write(" %s\n" % (red("**")))
+				else:
+					out.write("\n")
+
+		if ro_mismatch_warning:
+			out.write("\n%s:" % (red("**")))
+			out.write(yellow(" WARNING: packages marked with ** have been resolved as a\n"))
+			out.write(yellow("    runtime dependency, but the CHOST variable for the parent\n"))
+			out.write(yellow("    and dependency package don't match. This could cause link\n"))
+			out.write(yellow("    errors. It is recommended to use RDEPEND READONLY_EROOT's\n"))
+			out.write(yellow("    only with matching CHOST portage instances.\n"))
+
 		if verbosity == 3:
 			print
 			print counters
@@ -11823,6 +11865,7 @@
 		for x in candidate_catpkgs:
 			# cycle through all our candidate deps and determine
 			# what will and will not get unmerged
+
 			try:
 				mymatch = vartree.dbapi.match(x)
 			except portage.exception.AmbiguousPackageName, errpkgs:
diff -ru portage.orig/pym/portage/__init__.py portage/pym/portage/__init__.py
--- portage.orig/pym/portage/__init__.py	2009-03-24 16:43:49.000000000 +0100
+++ portage/pym/portage/__init__.py	2009-03-25 13:40:59.000000000 +0100
@@ -1240,6 +1240,7 @@
 
 			self._accept_license = copy.deepcopy(clone._accept_license)
 			self._plicensedict = copy.deepcopy(clone._plicensedict)
+			self.readonly_roots = copy.deepcopy(clone.readonly_roots)
 		else:
 
 			def check_var_directory(varname, var):
@@ -1807,6 +1808,50 @@
 			# but needs to be available using portageq
 			self["EPREFIX"] = EPREFIX
 
+			# expand READONLY_EROOT to a list of all readonly portage instances
+			# all the way down to the last one. beware that ATM a deeper instance
+			# in the chain can provide more than the toplevel! this means that
+			# if you only inherit DEPENDS from one instance, that instance may
+			# inherit RDEPENDs from another one, making the top-level instance
+			# inherit RDEPENDs from there too - even if the intermediate prefix
+			# does not do this.
+			self.readonly_roots = {}
+			my_ro_current_instance = config_root
+			my_ro_widest_depset = set(['DEPEND', 'RDEPEND', 'PDEPEND'])
+
+			while True:
+				my_ro_current_make_conf_file = os.path.join(my_ro_current_instance,MAKE_CONF_FILE.lstrip(os.path.sep))
+
+				if os.path.exists(my_ro_current_make_conf_file):
+					my_ro_cfg = getconfig(my_ro_current_make_conf_file, tolerant=1)
+					
+					if my_ro_cfg.has_key("READONLY_EROOT"):
+						if not my_ro_cfg["READONLY_EROOT"].find(":"):
+							raise portage.exception.InvalidReadonlyERoot("ERROR: malformed READONLY_EROOT in %s" % (my_ro_current_make_conf_file))
+
+						(my_ro_cfg_root,my_ro_cfg_root_deps) = my_ro_cfg["READONLY_EROOT"].rsplit(":",1)
+
+						if not os.path.exists(my_ro_cfg_root):
+							raise portage.exception.InvalidReadonlyERoot("ERROR: malformed READONLY_EROOT in %s: path does not exist!" % (my_ro_current_instance))
+
+						if self.readonly_roots.has_key(my_ro_cfg_root):
+							raise portage.exception.InvalidReadonlyERoot("ERROR: circular READONLY_EROOT's in %s. %s already checked for %s" % (my_ro_current_make_conf_file, my_ro_cfg_root, self.readonly_roots[my_ro_cfg_root]))
+
+						if my_ro_cfg_root == config_root:
+							raise portage.exception.InvalidReadonlyERoot("ERROR: cannot add this instance as READONLY_EROOT in %s." % (my_ro_current_make_conf_file))
+
+						# intersect the widest depset with the current one to strip down
+						# the allowed dependency resolution to not be wider than the
+						# next higher one. this way we can prevent for a given prefix
+						# to resolve RDEPENDs from a prefix with a different CHOST that
+						# is a few levels deeper in the chain.
+						my_ro_widest_depset = set(my_ro_cfg_root_deps.split(",")) & my_ro_widest_depset
+						self.readonly_roots[my_ro_cfg_root] = my_ro_widest_depset
+						my_ro_current_instance = my_ro_cfg_root
+						continue
+
+				break
+
 			self._init_dirs()
 
 		if mycpv:
@@ -3056,6 +3101,10 @@
 		if rootpath:
 			mydict["PORTAGE_ROOTPATH"] = rootpath
 
+		# populate with PORTAGE_READONLY_EROOTS
+		if self.readonly_roots and len(self.readonly_roots) > 0:
+			mydict["PORTAGE_READONLY_EROOTS"] = ':'.join(self.readonly_roots)
+
 		return mydict
 
 	def thirdpartymirrors(self):
@@ -6740,6 +6789,73 @@
 				return 0
 		return 1
 
+ro_trees={}
+ro_vartrees={}
+ro_selected=[]
+
+def dep_match_readonly_roots(settings, atom, dep_type, parent=None):
+	if len(ro_trees) < len(settings.readonly_roots):
+		# MDUFT: create additional vartrees for every readonly root here.
+		# the ro_vartrees instances are created below as they are needed to
+		# avoid reading vartrees of portage instances which aren't required
+		# while resolving this dependencies.
+		for type in ("DEPEND","RDEPEND", "PDEPEND"):
+			ro_trees[type] = []
+			
+			for ro_root, ro_dep_types in settings.readonly_roots.items():
+				if type in ro_dep_types:
+					ro_trees[type].append(ro_root)
+
+	if len(ro_trees) == 0:
+		return []
+	
+	matches = []
+
+	for ro_root in ro_trees[dep_type]:
+		if not ro_vartrees.has_key(ro_root):
+			# target_root=ro_root ok? or should it be the real target_root?
+			_tmp_settings = config(config_root=ro_root, target_root=ro_root,
+				config_incrementals=portage.const.INCREMENTALS)
+			
+			ro_vartrees[ro_root] = vartree(root=ro_root, 
+				categories=_tmp_settings.categories, 
+				settings=_tmp_settings, kill_eprefix=True)
+				
+		ro_matches = ro_vartrees[ro_root].dbapi.match(atom)
+
+		if ro_matches:
+			ro_host_mismatch = False
+			if dep_type is "RDEPEND":
+				# we need to assure binary compatability, so it needs to be
+				# the same CHOST! But how? for now i cannot do anything...
+				if parent and parent.metadata["CHOST"] != ro_vartrees[ro_root].settings.get("CHOST", ""):
+					# provocate a big fat warning in the list of external packages.
+					ro_host_mismatch = True
+				pass
+
+			matches.append({ "ro_root": ro_root, "atom": atom, "matches": ro_matches, 
+				"type": dep_type, "parent": parent, "host_mismatch": ro_host_mismatch })
+
+	return matches
+
+def dep_wordreduce_readonly(reduced, unreduced, settings, dep_type, parent):
+	for mypos, token in enumerate(unreduced):
+		# recurse if it's a list.
+		if isinstance(reduced[mypos], list):
+			reduced[mypos] = dep_wordreduce_readonly(reduced[mypos], 
+				unreduced[mypos], settings, dep_type, parent)
+		# do nothing if it's satisfied already.
+		elif not reduced[mypos]:
+			ro_matches = dep_match_readonly_roots(settings, unreduced[mypos], dep_type, parent)
+
+			if ro_matches:
+				# TODO: select a match if there are more than one?
+				# for now, the first match is taken...
+				ro_selected.append(ro_matches[0])
+				reduced[mypos] = True
+
+	return reduced
+
 def dep_zapdeps(unreduced, reduced, myroot, use_binaries=0, trees=None):
 	"""Takes an unreduced and reduced deplist and removes satisfied dependencies.
 	Returned deplist contains steps that must be taken to satisfy dependencies."""
@@ -6775,7 +6891,7 @@
 	preferred_any_slot = []
 	possible_upgrades = []
 	other = []
-
+	
 	# Alias the trees we'll be checking availability against
 	parent   = trees[myroot].get("parent")
 	graph_db = trees[myroot].get("graph_db")
@@ -6958,7 +7074,7 @@
 		return portage.dep.Atom("=" + prefix + expanded + postfix)
 
 def dep_check(depstring, mydbapi, mysettings, use="yes", mode=None, myuse=None,
-	use_cache=1, use_binaries=0, myroot="/", trees=None):
+	use_cache=1, use_binaries=0, myroot="/", trees=None, dep_type=None):
 	"""Takes a depend string and parses the condition."""
 	edebug = mysettings.get("PORTAGE_DEBUG", None) == "1"
 	#check_config_instance(mysettings)
@@ -7036,6 +7152,14 @@
 	writemsg("mysplit:  %s\n" % (mysplit), 1)
 	writemsg("mysplit2: %s\n" % (mysplit2), 1)
 
+	if dep_type is not None:
+		mysplit2=dep_wordreduce_readonly(unreduced=mysplit[:], 
+			reduced=mysplit2, settings=mysettings, 
+			dep_type=dep_type, parent=trees[myroot].get("disp_parent"))
+
+		writemsg("\n", 1)
+		writemsg("mysplit2 after readonly reduce: %s\n" % (mysplit2), 1)
+
 	try:
 		myzaps = dep_zapdeps(mysplit, mysplit2, myroot,
 			use_binaries=use_binaries, trees=trees)
diff -ru portage.orig/pym/portage/dbapi/vartree.py portage/pym/portage/dbapi/vartree.py
--- portage.orig/pym/portage/dbapi/vartree.py	2009-03-24 16:43:02.000000000 +0100
+++ portage/pym/portage/dbapi/vartree.py	2009-03-24 16:42:51.000000000 +0100
@@ -1276,9 +1276,20 @@
 		self._counter_path = os.path.join(root,
 			CACHE_PATH.lstrip(os.path.sep), "counter")
 
+		plibreg_path = os.path.join(self.root, EPREFIX_LSTRIP, PRIVATE_PATH, "preserved_libs_registry")
+
+		if vartree:
+			self._kill_eprefix = vartree._kill_eprefix
+		else:
+			self._kill_eprefix = False
+
+		if self._kill_eprefix:
+			self._aux_cache_filename = os.path.join(self.root, self._aux_cache_filename.replace(EPREFIX, ""))
+			self._counter_path = os.path.join(self.root, self._counter_path.replace(EPREFIX, ""))
+			plibreg_path = os.path.join(self.root, plibreg_path.replace(EPREFIX, ""))
+
 		try:
-			self.plib_registry = PreservedLibsRegistry(
-				os.path.join(self.root, EPREFIX_LSTRIP, PRIVATE_PATH, "preserved_libs_registry"))
+			self.plib_registry = PreservedLibsRegistry(plibreg_path)
 		except PermissionDenied:
 			# apparently this user isn't allowed to access PRIVATE_PATH
 			self.plib_registry = None
@@ -1291,6 +1302,10 @@
 
 	def getpath(self, mykey, filename=None):
 		rValue = os.path.join(self.root, VDB_PATH, mykey)
+
+		if self._kill_eprefix:
+			rValue = os.path.join(self.root, rValue.replace(EPREFIX, ""))
+
 		if filename != None:
 			rValue = os.path.join(rValue, filename)
 		return rValue
@@ -1439,6 +1454,9 @@
 		returnme = []
 		basepath = os.path.join(self.root, VDB_PATH) + os.path.sep
 
+		if self._kill_eprefix:
+			basepath = os.path.join(self.root, basepath.replace(EPREFIX, ""))
+
 		if use_cache:
 			from portage import listdir
 		else:
@@ -1528,7 +1546,12 @@
 			return list(self._iter_match(mydep,
 				self.cp_list(mydep.cp, use_cache=use_cache)))
 		try:
-			curmtime = os.stat(self.root+VDB_PATH+"/"+mycat).st_mtime
+			_tmp_path = self.root+VDB_PATH+"/"+mycat
+			
+			if self._kill_eprefix:
+				_tmp_path = os.path.join(self.root, _tmp_path.replace(EPREFIX, ""))
+
+			curmtime = os.stat(_tmp_path).st_mtime
 		except (IOError, OSError):
 			curmtime=0
 
@@ -2073,7 +2096,7 @@
 class vartree(object):
 	"this tree will scan a var/db/pkg database located at root (passed to init)"
 	def __init__(self, root="/", virtual=None, clone=None, categories=None,
-		settings=None):
+		settings=None, kill_eprefix=False):
 		if clone:
 			writemsg("vartree.__init__(): deprecated " + \
 				"use of clone parameter\n", noiselevel=-1)
@@ -2082,6 +2105,7 @@
 			self.populated = 1
 			from portage import config
 			self.settings = config(clone=clone.settings)
+			self._kill_eprefix = clone._kill_eprefix
 		else:
 			self.root = root[:]
 			if settings is None:
@@ -2089,6 +2113,7 @@
 			self.settings = settings # for key_expand calls
 			if categories is None:
 				categories = settings.categories
+			self._kill_eprefix=kill_eprefix
 			self.dbapi = vardbapi(self.root, categories=categories,
 				settings=settings, vartree=self)
 			self.populated = 1
@@ -2120,6 +2145,10 @@
 			raise
 		except Exception, e:
 			mydir = os.path.join(self.root, VDB_PATH, mycpv)
+
+			if self._kill_eprefix:
+				mydir = os.path.join(self.root, mydir.replace(EPREFIX, ""))
+
 			writemsg("\nParse Error reading PROVIDE and USE in '%s'\n" % mydir,
 				noiselevel=-1)
 			if mylines:
diff -ru portage.orig/pym/portage/exception.py portage/pym/portage/exception.py
--- portage.orig/pym/portage/exception.py	2009-03-24 16:43:02.000000000 +0100
+++ portage/pym/portage/exception.py	2009-03-24 16:42:52.000000000 +0100
@@ -118,3 +118,6 @@
 class UntrustedSignature(SignatureException):
 	"""Signature was not certified to the desired security level"""
 
+class InvalidReadonlyERoot(PortageException):
+	"""Readonly EROOT definition string in make.conf invalid."""
+

Re: [gentoo-portage-dev] prefix portage chaining

[Michael Haubenwallner]

On Thu, 2009-03-26 at 08:26 +0100, Markus Duft wrote:
> On Wed, 2009-03-25 at 11:44 -0700, Ned Ludd wrote:
> > While much of what you are talking about here mainly applies to prefix,
> > it looks to me from glancing over the code that you might of solved a
> > long standing problem in the embedded world with cross compiling via
> > portage. 222895  If that is the case, then I owe you a beer. one about
> > the size of a keg.

> hmm... looking over that patch again, the only EPREFIX dependent thing
> is, that i'm removing EPREFIX from the vartree class again :) so this
> should pretty much plain apply to main too, and simply work. you may
> want to rename READONLY_EPREFIX to READONLY_ROOT, but thats it :)

> then add to make.conf: READONLY_ROOT=/my/other/root:DEPEND

IMO, for non-prefix this is useful for [[ $ROOT != '/' ]] only, with
either CHOST (root=$ROOT) being equal to CBUILD (root='/') or not.
And the only two senseful values for READONLY_ROOT in make.conf are:
1. "/:DEPEND"
   To skip merging cmdline utilities (=DEPEND) to host-root (=$ROOT), by
   resolving them from build-root (='/').
2. <unset>
   To merge all DEPENDs to host-root (=$ROOT), although they are still
   used from build-root (='/') during build.

/haubi/
-- 
Michael Haubenwallner
Gentoo on a different level

Re: [gentoo-portage-dev] prefix portage chaining

[Ned Ludd]

On Thu, 2009-03-26 at 08:26 +0100, Markus Duft wrote:
> On Wed, 2009-03-25 at 11:44 -0700, Ned Ludd wrote:
> [snip]
> > 
> > 
> > While much of what you are talking about here mainly applies to prefix,
> > it looks to me from glancing over the code that you might of solved a
> > long standing problem in the embedded world with cross compiling via
> > portage. 222895  If that is the case, then I owe you a beer. one about
> > the size of a keg.
> > 
> 
> lol, thx for the beer ;)
> 
> hmm... looking over that patch again, the only EPREFIX dependent thing
> is, that i'm removing EPREFIX from the vartree class again :) so this
> should pretty much plain apply to main too, and simply work. you may
> want to rename READONLY_EPREFIX to READONLY_ROOT, but thats it :)
> 
> the other stuff besides portage modification (baselayout patchery, etc.
> is prefix specific again, so all you'd need is the portage changes.
> 
> if you will try it, please let me know if it worked :) with the attached
> patch "sed -i -e 's,READONLY_EPREFIX,READONLY_ROOT,g'", and applying to
> an installed /usr/lib/portage should enable you to do it.
> (backup /usr/lib/portage - i trust my work, but... we never know for
> sure :))
> 
> then add to make.conf: READONLY_ROOT=/my/other/root:DEPEND
> 
> i hope this is what you where looking for...! and i hope it doesn't
> somehow clash with the existing cross compile logic in portage regarding
> where to merge to...
> 
> Cheers, Markus


patch failed a few hunks for me on ~arch vanilla-portage. I did point
out the patch to one of the gentoo embedded/openmoko guys. Think they
will be the most eager to test this.

In our case if the code did work out whatever you call READONLY_ROOT we
would probably need to expand to allow for more that one ROOT if it has
to be defined in the parent /etc/make.conf


-- 
Ned Ludd <solar@gentoo.org>
Gentoo Linux

Re: [gentoo-portage-dev] prefix portage chaining

[Markus Duft]

On Thu, 2009-03-26 at 10:08 -0700, Ned Ludd wrote:
> On Thu, 2009-03-26 at 08:26 +0100, Markus Duft wrote:
> > On Wed, 2009-03-25 at 11:44 -0700, Ned Ludd wrote:
> > [snip]
> > > 
> > > 
> > > While much of what you are talking about here mainly applies to prefix,
> > > it looks to me from glancing over the code that you might of solved a
> > > long standing problem in the embedded world with cross compiling via
> > > portage. 222895  If that is the case, then I owe you a beer. one about
> > > the size of a keg.
> > > 
> > 
> > lol, thx for the beer ;)
> > 
> > hmm... looking over that patch again, the only EPREFIX dependent thing
> > is, that i'm removing EPREFIX from the vartree class again :) so this
> > should pretty much plain apply to main too, and simply work. you may
> > want to rename READONLY_EPREFIX to READONLY_ROOT, but thats it :)
> > 
> > the other stuff besides portage modification (baselayout patchery, etc.
> > is prefix specific again, so all you'd need is the portage changes.
> > 
> > if you will try it, please let me know if it worked :) with the attached
> > patch "sed -i -e 's,READONLY_EPREFIX,READONLY_ROOT,g'", and applying to
> > an installed /usr/lib/portage should enable you to do it.
> > (backup /usr/lib/portage - i trust my work, but... we never know for
> > sure :))
> > 
> > then add to make.conf: READONLY_ROOT=/my/other/root:DEPEND
> > 
> > i hope this is what you where looking for...! and i hope it doesn't
> > somehow clash with the existing cross compile logic in portage regarding
> > where to merge to...
> > 
> > Cheers, Markus
> 
> 
> patch failed a few hunks for me on ~arch vanilla-portage. I did point
> out the patch to one of the gentoo embedded/openmoko guys. Think they
> will be the most eager to test this.
> 
> In our case if the code did work out whatever you call READONLY_ROOT we
> would probably need to expand to allow for more that one ROOT if it has
> to be defined in the parent /etc/make.conf

not the parent one, but the "childs" one. so you'd say
READONLY_ROOT="/:DEPEND" in your $ROOT. so when merging to $ROOT, '/'
will be searched for DEPENDs.

er... you can remove all hunks for vartree.py and remove the argument
"kill_eprefix" from all calls to vartree (should be one only i guess).
sorry - forgot about that.

Cheers, Markus

> 
>