[gentoo-portage-dev] [PATCH] Split @selected into @selected-packages and @selected-sets

[gentoo-portage-dev] [PATCH] Split @selected into @selected-packages and @selected-sets

[Michał Górny]

Split the big @selected set into separate @selected-packages for plain
packages and @selected-sets for subsets. The @selected set is still
provided as compatibility wrapper for both, though the new sets are
preferred for more flexibility.
---
 bin/regenworld                |   4 +-
 cnf/sets/portage.conf         |   7 +-
 doc/config/sets.docbook       |  18 +++--
 man/emerge.1                  |  16 +++--
 pym/portage/_sets/__init__.py |   8 +++
 pym/portage/_sets/files.py    | 149 ++++++++++++++++++++++++++++--------------
 6 files changed, 137 insertions(+), 65 deletions(-)

diff --git a/bin/regenworld b/bin/regenworld
index e07424d..eef4cfd 100755
--- a/bin/regenworld
+++ b/bin/regenworld
@@ -11,7 +11,7 @@ if osp.isfile(osp.join(osp.dirname(osp.dirname(osp.realpath(__file__))), ".porta
 import portage
 portage._internal_caller = True
 from portage import os
-from portage._sets.files import StaticFileSet, WorldSelectedSet
+from portage._sets.files import StaticFileSet, WorldSelectedPackagesSet
 
 import re
 import tempfile
@@ -105,7 +105,7 @@ for mykey in biglist:
 if not worldlist:
 	pass
 else:
-	existing_set = WorldSelectedSet(eroot)
+	existing_set = WorldSelectedPackagesSet(eroot)
 	existing_set.load()
 
 	if not existing_set:
diff --git a/cnf/sets/portage.conf b/cnf/sets/portage.conf
index fd2c387..8ffcedc 100644
--- a/cnf/sets/portage.conf
+++ b/cnf/sets/portage.conf
@@ -9,10 +9,15 @@
 class = portage.sets.base.DummyPackageSet
 packages = @selected @system
 
-# Not much that could be changed for world, so better leave it alone
 [selected]
 class = portage.sets.files.WorldSelectedSet
 
+[selected-packages]
+class = portage.sets.files.WorldSelectedPackagesSet
+
+[selected-sets]
+class = portage.sets.files.WorldSelectedSetsSet
+
 # Same as for world, though later portage versions might use a different class
 [system]
 class = portage.sets.profiles.PackagesSystemSet
diff --git a/doc/config/sets.docbook b/doc/config/sets.docbook
index 8f74412..cf8dcf3 100644
--- a/doc/config/sets.docbook
+++ b/doc/config/sets.docbook
@@ -71,9 +71,9 @@
 			class = portage.sets.base.DummyPackageSet
 			packages = @selected @system
 
-			# The selected set
-			[selected]
-			class = portage.sets.files.WorldSelectedSet
+			# The selected-packages set
+			[selected-packages]
+			class = portage.sets.files.WorldSelectedPackagesSet
 
 			# The classic system set
 			[system]
@@ -247,8 +247,16 @@
 			</sect3>
 		</sect2>
 		
-		<sect2 id='config-set-classes-WorldSelectedSet'>
-		<title>portage.sets.files.WorldSelectedSet</title>
+		<sect2 id='config-set-classes-WorldSelectedPackagesSet'>
+		<title>portage.sets.files.WorldSelectedPackagesSet</title>
+		<para>
+		A minor variation of <classname>StaticFileSet</classname>, mainly for implementation 
+		reasons. It should never be used in user configurations as it's already configured
+		by default, doesn't support any options and will eventually be removed in a future version.
+		</para>
+
+		<sect2 id='config-set-classes-WorldSelectedSetsSet'>
+		<title>portage.sets.files.WorldSelectedSetsSet</title>
 		<para>
 		A minor variation of <classname>StaticFileSet</classname>, mainly for implementation 
 		reasons. It should never be used in user configurations as it's already configured
diff --git a/man/emerge.1 b/man/emerge.1
index a206e8e..91697b4 100644
--- a/man/emerge.1
+++ b/man/emerge.1
@@ -61,13 +61,15 @@ would like to query the owners of one or more files or directories.
 .TP
 .BR set
 A \fIset\fR is a convenient shorthand for a large group of
-packages. Three sets are currently always available: \fBselected\fR,
-\fBsystem\fR and \fBworld\fR. \fBselected\fR contains the user-selected
-"world" packages that are listed in \fB/var/lib/portage/world\fR,
-and nested sets that may be listed
-in \fB/var/lib/portage/world_sets\fR. \fBsystem\fR refers to a set of
-packages deemed necessary for your system to run properly. \fBworld\fR
-encompasses both the \fBselected\fR and \fBsystem\fR sets. [See
+packages. Five sets are currently always available: \fBselected-packages\fR,
+\fBselected-sets\fR, \fBselected\fR, \fBsystem\fR and \fBworld\fR.
+\fBselected-packages\fR contains the user-selected "world" packages that
+are listed in \fB/var/lib/portage/world\fR, while \fBselected-sets\fR
+contains the nested sets that may be listed in \fB/var/lib/portage/world_sets\fR.
+\fBsystem\fR refers to a set of packages deemed necessary for your system
+to run properly. \fBselected\fR encompasses both the \fBselected-packages\fR
+and \fBselected-sets\fR sets, while \fBworld\fR encompasses the \fBselected\fR
+and \fBsystem\fR sets. [See
 \fBFILES\fR below for more information.] Other sets can exist depending
 on the current configuration. The default set configuration is located
 in the \fB/usr/share/portage/config/sets\fR directory.
diff --git a/pym/portage/_sets/__init__.py b/pym/portage/_sets/__init__.py
index 75d1df7..a652227 100644
--- a/pym/portage/_sets/__init__.py
+++ b/pym/portage/_sets/__init__.py
@@ -121,6 +121,14 @@ class SetConfig(object):
 		parser.add_section("selected")
 		parser.set("selected", "class", "portage.sets.files.WorldSelectedSet")
 
+		parser.remove_section("selected-packages")
+		parser.add_section("selected-packages")
+		parser.set("selected-packages", "class", "portage.sets.files.WorldSelectedPackagesSet")
+
+		parser.remove_section("selected-sets")
+		parser.add_section("selected-sets")
+		parser.set("selected-sets", "class", "portage.sets.files.WorldSelectedSetsSet")
+
 		parser.remove_section("system")
 		parser.add_section("system")
 		parser.set("system", "class", "portage.sets.profiles.PackagesSystemSet")
diff --git a/pym/portage/_sets/files.py b/pym/portage/_sets/files.py
index 2fb64de..e76fa83 100644
--- a/pym/portage/_sets/files.py
+++ b/pym/portage/_sets/files.py
@@ -20,7 +20,8 @@ from portage.env.loaders import ItemFileLoader, KeyListFileLoader
 from portage.env.validators import ValidAtomValidator
 from portage import cpv_getkey
 
-__all__ = ["StaticFileSet", "ConfigFileSet", "WorldSelectedSet"]
+__all__ = ["StaticFileSet", "ConfigFileSet", "WorldSelectedSet",
+		"WorldSelectedPackagesSet", "WorldSelectedSetsSet"]
 
 class StaticFileSet(EditablePackageSet):
 	_operations = ["merge", "unmerge"]
@@ -205,41 +206,59 @@ class ConfigFileSet(PackageSet):
 	multiBuilder = classmethod(multiBuilder)
 
 class WorldSelectedSet(EditablePackageSet):
-	description = "Set of packages that were directly installed by the user"
-	
+	description = "Set of packages and subsets that were directly installed by the user"
+
 	def __init__(self, eroot):
 		super(WorldSelectedSet, self).__init__(allow_repo=True)
-		# most attributes exist twice as atoms and non-atoms are stored in 
-		# separate files
+		self._pkgset = WorldSelectedPackagesSet(eroot)
+		self._setset = WorldSelectedSetsSet(eroot)
+
+	def write(self):
+		self._pkgset._atoms = self._atoms
+		self._pkgset.write()
+		self._setset._nonatoms = self._nonatoms
+		self._setset.write()
+
+	def load(self):
+		self._pkgset.load()
+		self._setset.load()
+		self._setAtoms(self._pkgset._atoms | self._pkgset._nonatoms)
+
+	def lock(self):
+		self._pkgset.lock()
+		self._setset.lock()
+
+	def unlock(self):
+		self._pkgset.unlock()
+		self._setset.unlock()
+
+	def cleanPackage(self, vardb, cpv):
+		self._pkgset.cleanPackage(vardb, cpv)
+
+	def singleBuilder(self, options, settings, trees):
+		return WorldSelectedSet(settings["EROOT"])
+	singleBuilder = classmethod(singleBuilder)
+
+class WorldSelectedPackagesSet(EditablePackageSet):
+	description = "Set of packages that were directly installed by the user"
+
+	def __init__(self, eroot):
+		super(WorldSelectedPackagesSet, self).__init__(allow_repo=True)
 		self._lock = None
 		self._filename = os.path.join(eroot, WORLD_FILE)
 		self.loader = ItemFileLoader(self._filename, self._validate)
 		self._mtime = None
-		
-		self._filename2 = os.path.join(eroot, WORLD_SETS_FILE)
-		self.loader2 = ItemFileLoader(self._filename2, self._validate2)
-		self._mtime2 = None
-		
+
 	def _validate(self, atom):
 		return ValidAtomValidator(atom, allow_repo=True)
 
-	def _validate2(self, setname):
-		return setname.startswith(SETPREFIX)
-
 	def write(self):
 		write_atomic(self._filename,
 			"".join(sorted("%s\n" % x for x in self._atoms)))
 
-		write_atomic(self._filename2,
-			"".join(sorted("%s\n" % x for x in self._nonatoms)))
-
 	def load(self):
 		atoms = []
-		nonatoms = []
 		atoms_changed = False
-		# load atoms and non-atoms from different files so the worldfile is 
-		# backwards-compatible with older versions and other PMs, even though 
-		# it's supposed to be private state data :/
 		try:
 			mtime = os.stat(self._filename).st_mtime
 		except (OSError, IOError):
@@ -261,36 +280,8 @@ class WorldSelectedSet(EditablePackageSet):
 		else:
 			atoms.extend(self._atoms)
 
-		changed2, nonatoms = self._load2()
-		atoms_changed |= changed2
-
 		if atoms_changed:
-			self._setAtoms(atoms+nonatoms)
-
-	def _load2(self):
-		changed = False
-		try:
-			mtime = os.stat(self._filename2).st_mtime
-		except (OSError, IOError):
-			mtime = None
-		if (not self._loaded or self._mtime2 != mtime):
-			try:
-				data, errors = self.loader2.load()
-				for fname in errors:
-					for e in errors[fname]:
-						self.errors.append(fname+": "+e)
-			except EnvironmentError as e:
-				if e.errno != errno.ENOENT:
-					raise
-				del e
-				data = {}
-			nonatoms = list(data)
-			self._mtime2 = mtime
-			changed = True
-		else:
-			nonatoms = list(self._nonatoms)
-
-		return changed, nonatoms
+			self._setAtoms(atoms)
 
 	def _ensure_dirs(self):
 		ensure_dirs(os.path.dirname(self._filename), gid=portage_gid, mode=0o2750, mask=0o2)
@@ -338,5 +329,63 @@ class WorldSelectedSet(EditablePackageSet):
 		self.replace(newworldlist)
 
 	def singleBuilder(self, options, settings, trees):
-		return WorldSelectedSet(settings["EROOT"])
+		return WorldSelectedPackagesSet(settings["EROOT"])
+	singleBuilder = classmethod(singleBuilder)
+
+class WorldSelectedSetsSet(EditablePackageSet):
+	description = "Set of sets that were directly installed by the user"
+
+	def __init__(self, eroot):
+		super(WorldSelectedSetsSet, self).__init__(allow_repo=True)
+		self._lock = None
+		self._filename = os.path.join(eroot, WORLD_SETS_FILE)
+		self.loader = ItemFileLoader(self._filename, self._validate)
+		self._mtime = None
+
+	def _validate(self, setname):
+		return setname.startswith(SETPREFIX)
+
+	def write(self):
+		write_atomic(self._filename,
+			"".join(sorted("%s\n" % x for x in self._nonatoms)))
+
+	def load(self):
+		atoms_changed = False
+		try:
+			mtime = os.stat(self._filename).st_mtime
+		except (OSError, IOError):
+			mtime = None
+		if (not self._loaded or self._mtime != mtime):
+			try:
+				data, errors = self.loader.load()
+				for fname in errors:
+					for e in errors[fname]:
+						self.errors.append(fname+": "+e)
+			except EnvironmentError as e:
+				if e.errno != errno.ENOENT:
+					raise
+				del e
+				data = {}
+			nonatoms = list(data)
+			self._mtime = mtime
+			atoms_changed = True
+		else:
+			nonatoms = list(self._nonatoms)
+
+		if atoms_changed:
+			self._setAtoms(nonatoms)
+
+	def lock(self):
+		if self._lock is not None:
+			raise AssertionError("already locked")
+		self._lock = lockfile(self._filename, wantnewlockfile=1)
+
+	def unlock(self):
+		if self._lock is None:
+			raise AssertionError("not locked")
+		unlockfile(self._lock)
+		self._lock = None
+
+	def singleBuilder(self, options, settings, trees):
+		return WorldSelectedSetsSet(settings["EROOT"])
 	singleBuilder = classmethod(singleBuilder)
-- 
2.1.3

Re: [gentoo-portage-dev] [PATCH] Split @selected into @selected-packages and @selected-sets

[Zac Medico]

On 11/30/2014 03:04 PM, Michał Górny wrote:
> Split the big @selected set into separate @selected-packages for plain
> packages and @selected-sets for subsets. The @selected set is still
> provided as compatibility wrapper for both, though the new sets are
> preferred for more flexibility.

LGTM.

Later, we might replace WorldSelectedSet with this:

	[selected]
	class = portage.sets.base.DummyPackageSet
	packages = @selected-packages @selected-sets

-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] Split @selected into @selected-packages and @selected-sets

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 724 bytes --]

Dnia 2014-11-30, o godz. 17:52:29
Zac Medico <zmedico@gentoo.org> napisał(a):

> On 11/30/2014 03:04 PM, Michał Górny wrote:
> > Split the big @selected set into separate @selected-packages for plain
> > packages and @selected-sets for subsets. The @selected set is still
> > provided as compatibility wrapper for both, though the new sets are
> > preferred for more flexibility.
> 
> LGTM.
> 
> Later, we might replace WorldSelectedSet with this:
> 
> 	[selected]
> 	class = portage.sets.base.DummyPackageSet
> 	packages = @selected-packages @selected-sets
> 

I've tried but the current code relies on using 'selected' a lot,
and DummyPackageSet isn't mutable.

-- 
Best regards,
Michał Górny

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]