[gentoo-portage-dev] [PATCH 0/3] INSTALL_MASK refurbishing resubmit

[gentoo-portage-dev] [PATCH 0/3] INSTALL_MASK refurbishing resubmit

[Michał Górny]

Hi,

Here are three of four INSTALL_MASK updates I've sent long time ago
which were not really reviewed. The fourth patch added support
for repo-defined install-mask.conf and I'll do that separately.

Those patches focus on smaller changes. What they change, in order:

1. Removes explicit file removal code for FEATURES=no*. Instead, those
   values are converted into additional INSTALL_MASK entries
   and handled directly via INSTALL_MASK processing.

2. Rework INSTALL_MASK to filter files while installing instead of
   pre-stripping them. In other words, before: INSTALL_MASK removes
   files from ${D} before merge. After: ${D} contains all the files,
   Portage just skip INSTALL_MASK-ed stuff, verbosely indicating that.

3. Adds support for exclusions in INSTALL_MASK. In other words, you
   can do stuff like:

     INSTALL_MASK="/usr/share/locale -/usr/share/locale/en_US"

I have been using this via user patches since the last submission.
Guessing by 'git log', this means almost 2 years now.

--
Best regards,
Michał Górny

Michał Górny (3):
  portage.package.ebuild.config: Move FEATURES=no* handling there
  portage.dbapi.vartree: Move INSTALL_MASK handling into merging
  portage.dbapi.vartree: Support exclusions in INSTALL_MASK

 bin/misc-functions.sh                |  30 ----------
 pym/portage/dbapi/vartree.py         | 104 ++++++++++++++++++++++-------------
 pym/portage/package/ebuild/config.py |  11 ++++
 3 files changed, 77 insertions(+), 68 deletions(-)

-- 
2.16.2

[gentoo-portage-dev] [PATCH 1/3] portage.package.ebuild.config: Move FEATURES=no* handling there

[Michał Górny]

Move the code responsible for adding additional paths to INSTALL_MASK
into portage.package.ebuild.config.
---
 bin/misc-functions.sh                | 13 -------------
 pym/portage/package/ebuild/config.py | 10 ++++++++++
 2 files changed, 10 insertions(+), 13 deletions(-)

diff --git a/bin/misc-functions.sh b/bin/misc-functions.sh
index 7643af7b5..6a5c2ea05 100755
--- a/bin/misc-functions.sh
+++ b/bin/misc-functions.sh
@@ -375,20 +375,7 @@ preinst_mask() {
 	# in there in case any tools were built with -pg in CFLAGS.
 	cd "${T}"
 
-	# remove man pages, info pages, docs if requested
-	local f
-	for f in man info doc; do
-		if has no${f} $FEATURES; then
-			INSTALL_MASK="${INSTALL_MASK} /usr/share/${f}"
-		fi
-	done
-
 	install_mask "${ED}" "${INSTALL_MASK}"
-
-	# remove share dir if unnessesary
-	if has nodoc $FEATURES || has noman $FEATURES || has noinfo $FEATURES; then
-		rmdir "${ED%/}/usr/share" &> /dev/null
-	fi
 }
 
 preinst_sfperms() {
diff --git a/pym/portage/package/ebuild/config.py b/pym/portage/package/ebuild/config.py
index 3dc64a067..3f575fcaf 100644
--- a/pym/portage/package/ebuild/config.py
+++ b/pym/portage/package/ebuild/config.py
@@ -2465,6 +2465,16 @@ class config(object):
 		myflags.difference_update(self.usemask)
 		self.configlist[-1]["USE"]= " ".join(sorted(myflags))
 
+		# Prepare the final value of INSTALL_MASK
+		install_mask = self.get("INSTALL_MASK", '').split()
+		if 'nodoc' in self.features:
+			install_mask.append("/usr/share/doc")
+		if 'noinfo' in self.features:
+			install_mask.append("/usr/share/info")
+		if 'noman' in self.features:
+			install_mask.append("/usr/share/man")
+		self["INSTALL_MASK"] = ' '.join(install_mask)
+
 		if self.mycpv is None:
 			# Generate global USE_EXPAND variables settings that are
 			# consistent with USE, for display by emerge --info. For
-- 
2.16.2

[gentoo-portage-dev] [PATCH 2/3] portage.dbapi.vartree: Move INSTALL_MASK handling into merging

[Michał Górny]

Introduce a new logic for INSTALL_MASK handling in merging code,
replacing the old code that removed matching files and directories
from imagedir in bash. The new code actually ignores matching files
on-the-fly while testing for file collisions and merging files.
The files are still written to CONTENTS, and output using "###" zing
to indicate being masked, yet are not actually merged to the filesystem.
---
 bin/misc-functions.sh                |  17 ------
 pym/portage/dbapi/vartree.py         | 102 ++++++++++++++++++++++-------------
 pym/portage/package/ebuild/config.py |   3 +-
 3 files changed, 66 insertions(+), 56 deletions(-)

diff --git a/bin/misc-functions.sh b/bin/misc-functions.sh
index 6a5c2ea05..59391816b 100755
--- a/bin/misc-functions.sh
+++ b/bin/misc-functions.sh
@@ -361,23 +361,6 @@ install_mask() {
 	set -${shopts}
 }
 
-preinst_mask() {
-	if [ -z "${D}" ]; then
-		 eerror "${FUNCNAME}: D is unset"
-		 return 1
-	fi
-
-	if ! ___eapi_has_prefix_variables; then
-		local ED=${D}
-	fi
-
-	# Make sure $PWD is not ${D} so that we don't leave gmon.out files
-	# in there in case any tools were built with -pg in CFLAGS.
-	cd "${T}"
-
-	install_mask "${ED}" "${INSTALL_MASK}"
-}
-
 preinst_sfperms() {
 	if [ -z "${D}" ]; then
 		 eerror "${FUNCNAME}: D is unset"
diff --git a/pym/portage/dbapi/vartree.py b/pym/portage/dbapi/vartree.py
index 8b1b77f7d..21904edca 100644
--- a/pym/portage/dbapi/vartree.py
+++ b/pym/portage/dbapi/vartree.py
@@ -2491,7 +2491,7 @@ class dblink(object):
 								(statobj.st_dev, statobj.st_ino),
 								[]).append(relative_path)
 
-					if is_owned:
+					if is_owned and not self._is_install_masked(relative_path[1:]):
 						show_unmerge("---", unmerge_desc["replaced"], file_type, obj)
 						continue
 					elif relative_path in cfgfiledict:
@@ -3689,6 +3689,24 @@ class dblink(object):
 	def _emerge_log(self, msg):
 		emergelog(False, msg)
 
+	def _is_install_masked(self, relative_path):
+		ret = False
+		for pattern in self.settings.install_mask:
+			# absolute path pattern
+			if pattern.startswith('/'):
+				# match either exact path or one of parent dirs
+				# the latter is done via matching pattern/*
+				if (fnmatch.fnmatch(relative_path, pattern[1:])
+						or fnmatch.fnmatch(relative_path, pattern[1:] + '/*')):
+					ret = True
+					break
+			# filename
+			else:
+				if fnmatch.fnmatch(os.path.basename(relative_path), pattern):
+					ret = True
+					break
+		return ret
+
 	def treewalk(self, srcroot, destroot, inforoot, myebuild, cleanup=0,
 		mydbapi=None, prev_mtimes=None, counter=None):
 		"""
@@ -3848,16 +3866,6 @@ class dblink(object):
 					max_dblnk = dblnk
 			self._installed_instance = max_dblnk
 
-		# Apply INSTALL_MASK before collision-protect, since it may
-		# be useful to avoid collisions in some scenarios.
-		# We cannot detect if this is needed or not here as INSTALL_MASK can be
-		# modified by bashrc files.
-		phase = MiscFunctionsProcess(background=False,
-			commands=["preinst_mask"], phase="preinst",
-			scheduler=self._scheduler, settings=self.settings)
-		phase.start()
-		phase.wait()
-
 		# We check for unicode encoding issues after src_install. However,
 		# the check must be repeated here for binary packages (it's
 		# inexpensive since we call os.walk() here anyway).
@@ -3929,6 +3937,10 @@ class dblink(object):
 
 					relative_path = fpath[srcroot_len:]
 
+					# filter on INSTALL_MASK
+					if self._is_install_masked(relative_path):
+						continue
+
 					if line_ending_re.search(relative_path) is not None:
 						paths_with_newlines.append(relative_path)
 
@@ -4658,6 +4670,7 @@ class dblink(object):
 		while mergelist:
 
 			relative_path = mergelist.pop()
+			instmasked = self._is_install_masked(relative_path)
 			mysrc = join(srcroot, relative_path)
 			mydest = join(destroot, relative_path)
 			# myrealdest is mydest without the $ROOT prefix (makes a difference if ROOT!="/")
@@ -4744,7 +4757,7 @@ class dblink(object):
 				destmd5 = None
 
 			moveme = True
-			if protected:
+			if protected and not instmasked:
 				mydest, protected, moveme = self._protect(cfgfiledict,
 					protect_if_modified, mymd5, myto, mydest,
 					myrealdest, mydmode, destmd5, mydest_link)
@@ -4772,7 +4785,7 @@ class dblink(object):
 				# we can simply test for existence of this file to see if the target has been merged yet
 				myrealto = normalize_path(os.path.join(destroot, myabsto))
 				if mydmode is not None and stat.S_ISDIR(mydmode):
-					if not protected:
+					if not protected and not instmasked:
 						# we can't merge a symlink over a directory
 						newdest = self._new_backup_path(mydest)
 						msg = []
@@ -4792,26 +4805,32 @@ class dblink(object):
 					# it later.
 					secondhand.append(mysrc[len(srcroot):])
 					continue
-				# unlinking no longer necessary; "movefile" will overwrite symlinks atomically and correctly
-				if moveme:
-					zing = ">>>"
-					mymtime = movefile(mysrc, mydest, newmtime=thismtime,
-						sstat=mystat, mysettings=self.settings,
-						encoding=_encodings['merge'])
 
-				try:
-					self._merged_path(mydest, os.lstat(mydest))
-				except OSError:
-					pass
+				if instmasked:
+					zing = "###"
+					# pass mymtime through from initial stat
+				else:
+					# unlinking no longer necessary; "movefile" will overwrite symlinks atomically and correctly
+					if moveme:
+						zing = ">>>"
+						mymtime = movefile(mysrc, mydest, newmtime=thismtime,
+							sstat=mystat, mysettings=self.settings,
+							encoding=_encodings['merge'])
+
+					try:
+						self._merged_path(mydest, os.lstat(mydest))
+					except OSError:
+						pass
 
 				if mymtime != None:
-					# Use lexists, since if the target happens to be a broken
-					# symlink then that should trigger an independent warning.
-					if not (os.path.lexists(myrealto) or
-						os.path.lexists(join(srcroot, myabsto))):
-						self._eqawarn('preinst',
-							[_("QA Notice: Symbolic link /%s points to /%s which does not exist.")
-							% (relative_path, myabsto)])
+					if not instmasked:
+						# Use lexists, since if the target happens to be a broken
+						# symlink then that should trigger an independent warning.
+						if not (os.path.lexists(myrealto) or
+							os.path.lexists(join(srcroot, myabsto))):
+							self._eqawarn('preinst',
+								[_("QA Notice: Symbolic link /%s points to /%s which does not exist.")
+								% (relative_path, myabsto)])
 
 					showMessage("%s %s -> %s\n" % (zing, mydest, myto))
 					if sys.hexversion >= 0x3030000:
@@ -4826,7 +4845,9 @@ class dblink(object):
 					return 1
 			elif stat.S_ISDIR(mymode):
 				# we are merging a directory
-				if mydmode != None:
+				if instmasked:
+					showMessage("### %s/\n" % mydest)
+				elif mydmode != None:
 					# destination exists
 
 					if bsd_chflags:
@@ -4913,10 +4934,11 @@ class dblink(object):
 					os.chown(mydest, mystat[4], mystat[5])
 					showMessage(">>> %s/\n" % mydest)
 
-				try:
-					self._merged_path(mydest, os.lstat(mydest))
-				except OSError:
-					pass
+				if not instmasked:
+					try:
+						self._merged_path(mydest, os.lstat(mydest))
+					except OSError:
+						pass
 
 				outfile.write("dir "+myrealdest+"\n")
 				# recurse and merge this directory
@@ -4925,7 +4947,7 @@ class dblink(object):
 
 			elif stat.S_ISREG(mymode):
 				# we are merging a regular file
-				if not protected and \
+				if not protected and not instmasked and \
 					mydmode is not None and stat.S_ISDIR(mydmode):
 						# install of destination is blocked by an existing directory with the same name
 						newdest = self._new_backup_path(mydest)
@@ -4939,9 +4961,11 @@ class dblink(object):
 						self._eerror("preinst", msg)
 						mydest = newdest
 
+				if instmasked:
+					zing = "###"
 				# whether config protection or not, we merge the new file the
 				# same way.  Unless moveme=0 (blocking directory)
-				if moveme:
+				elif moveme:
 					# Create hardlinks only for source files that already exist
 					# as hardlinks (having identical st_dev and st_ino).
 					hardlink_key = (mystat.st_dev, mystat.st_ino)
@@ -4974,7 +4998,9 @@ class dblink(object):
 			else:
 				# we are merging a fifo or device node
 				zing = "!!!"
-				if mydmode is None:
+				if instmasked:
+					zing = "###"
+				elif mydmode is None:
 					# destination doesn't exist
 					if movefile(mysrc, mydest, newmtime=thismtime,
 						sstat=mystat, mysettings=self.settings,
diff --git a/pym/portage/package/ebuild/config.py b/pym/portage/package/ebuild/config.py
index 3f575fcaf..d04baacf9 100644
--- a/pym/portage/package/ebuild/config.py
+++ b/pym/portage/package/ebuild/config.py
@@ -271,6 +271,7 @@ class config(object):
 			self.mycpv = clone.mycpv
 			self._setcpv_args_hash = clone._setcpv_args_hash
 			self._soname_provided = clone._soname_provided
+			self.install_mask = clone.install_mask
 
 			# immutable attributes (internal policy ensures lack of mutation)
 			self._locations_manager = clone._locations_manager
@@ -2473,7 +2474,7 @@ class config(object):
 			install_mask.append("/usr/share/info")
 		if 'noman' in self.features:
 			install_mask.append("/usr/share/man")
-		self["INSTALL_MASK"] = ' '.join(install_mask)
+		self.install_mask = tuple(install_mask)
 
 		if self.mycpv is None:
 			# Generate global USE_EXPAND variables settings that are
-- 
2.16.2

[gentoo-portage-dev] [PATCH 3/3] portage.dbapi.vartree: Support exclusions in INSTALL_MASK

[Michał Górny]

Allow INSTALL_MASK patterns to start with '-' to indicate that
a specific match is to be excluded from being masked. In this case,
the last matching pattern determines whether the file is actually
filtered out or kept.
---
 pym/portage/dbapi/vartree.py | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/pym/portage/dbapi/vartree.py b/pym/portage/dbapi/vartree.py
index 21904edca..16c246b11 100644
--- a/pym/portage/dbapi/vartree.py
+++ b/pym/portage/dbapi/vartree.py
@@ -3692,19 +3692,21 @@ class dblink(object):
 	def _is_install_masked(self, relative_path):
 		ret = False
 		for pattern in self.settings.install_mask:
+			# if pattern starts with -, possibly exclude this path
+			pat_res = not pattern.startswith('-')
+			if not pat_res:
+				pattern = pattern[1:]
 			# absolute path pattern
 			if pattern.startswith('/'):
 				# match either exact path or one of parent dirs
 				# the latter is done via matching pattern/*
 				if (fnmatch.fnmatch(relative_path, pattern[1:])
 						or fnmatch.fnmatch(relative_path, pattern[1:] + '/*')):
-					ret = True
-					break
+					ret = pat_res
 			# filename
 			else:
 				if fnmatch.fnmatch(os.path.basename(relative_path), pattern):
-					ret = True
-					break
+					ret = pat_res
 		return ret
 
 	def treewalk(self, srcroot, destroot, inforoot, myebuild, cleanup=0,
-- 
2.16.2

Re: [gentoo-portage-dev] [PATCH 3/3] portage.dbapi.vartree: Support exclusions in INSTALL_MASK

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 2323 bytes --]

On Thu, Mar 15, 2018 at 3:22 PM, Michał Górny <mgorny@gentoo.org> wrote:

> Allow INSTALL_MASK patterns to start with '-' to indicate that
> a specific match is to be excluded from being masked. In this case,
> the last matching pattern determines whether the file is actually
> filtered out or kept.
> ---
>  pym/portage/dbapi/vartree.py | 10 ++++++----
>  1 file changed, 6 insertions(+), 4 deletions(-)
>
> diff --git a/pym/portage/dbapi/vartree.py b/pym/portage/dbapi/vartree.py
> index 21904edca..16c246b11 100644
> --- a/pym/portage/dbapi/vartree.py
> +++ b/pym/portage/dbapi/vartree.py
> @@ -3692,19 +3692,21 @@ class dblink(object):
>         def _is_install_masked(self, relative_path):
>                 ret = False
>                 for pattern in self.settings.install_mask:
>
+                       # if pattern starts with -, possibly exclude this
> path
> +                       pat_res = not pattern.startswith('-')
> +                       if not pat_res:
> +                               pattern = pattern[1:]
>

Maybe consider:

pattern = pattern[1:] if pattern.startswith('-') else pattern

I'm not super keen on this pattern in python, but it seems doable here.


>                         # absolute path pattern
>                         if pattern.startswith('/'):
>                                 # match either exact path or one of parent
> dirs
>                                 # the latter is done via matching pattern/*
>                                 if (fnmatch.fnmatch(relative_path,
> pattern[1:])
>                                                 or
> fnmatch.fnmatch(relative_path, pattern[1:] + '/*')):
> -                                       ret = True
> -                                       break
> +                                       ret = pat_res
>                         # filename
>                         else:
>                                 if fnmatch.fnmatch(os.path.basename(relative_path),
> pattern):
> -                                       ret = True
> -                                       break
> +                                       ret = pat_res
>                 return ret
>
>         def treewalk(self, srcroot, destroot, inforoot, myebuild,
> cleanup=0,
> --
> 2.16.2
>
>
>

[-- Attachment #2: Type: text/html, Size: 4660 bytes --]

Re: [gentoo-portage-dev] [PATCH 3/3] portage.dbapi.vartree: Support exclusions in INSTALL_MASK

[Michał Górny]

W dniu czw, 15.03.2018 o godzinie 17∶02 -0400, użytkownik Alec Warner
napisał:
> On Thu, Mar 15, 2018 at 3:22 PM, Michał Górny <mgorny@gentoo.org> wrote:
> 
> > Allow INSTALL_MASK patterns to start with '-' to indicate that
> > a specific match is to be excluded from being masked. In this case,
> > the last matching pattern determines whether the file is actually
> > filtered out or kept.
> > ---
> >  pym/portage/dbapi/vartree.py | 10 ++++++----
> >  1 file changed, 6 insertions(+), 4 deletions(-)
> > 
> > diff --git a/pym/portage/dbapi/vartree.py b/pym/portage/dbapi/vartree.py
> > index 21904edca..16c246b11 100644
> > --- a/pym/portage/dbapi/vartree.py
> > +++ b/pym/portage/dbapi/vartree.py
> > @@ -3692,19 +3692,21 @@ class dblink(object):
> >         def _is_install_masked(self, relative_path):
> >                 ret = False
> >                 for pattern in self.settings.install_mask:
> > 
> 
> +                       # if pattern starts with -, possibly exclude this
> > path
> > +                       pat_res = not pattern.startswith('-')
> > +                       if not pat_res:
> > +                               pattern = pattern[1:]
> > 
> 
> Maybe consider:
> 
> pattern = pattern[1:] if pattern.startswith('-') else pattern
> 
> I'm not super keen on this pattern in python, but it seems doable here.

I still need pat_res to know whether it's '+' or '-'.

> 
> 
> >                         # absolute path pattern
> >                         if pattern.startswith('/'):
> >                                 # match either exact path or one of parent
> > dirs
> >                                 # the latter is done via matching pattern/*
> >                                 if (fnmatch.fnmatch(relative_path,
> > pattern[1:])
> >                                                 or
> > fnmatch.fnmatch(relative_path, pattern[1:] + '/*')):
> > -                                       ret = True
> > -                                       break
> > +                                       ret = pat_res
> >                         # filename
> >                         else:
> >                                 if fnmatch.fnmatch(os.path.basename(relative_path),
> > pattern):
> > -                                       ret = True
> > -                                       break
> > +                                       ret = pat_res
> >                 return ret
> > 
> >         def treewalk(self, srcroot, destroot, inforoot, myebuild,
> > cleanup=0,
> > --
> > 2.16.2
> > 
> > 
> > 

-- 
Best regards,
Michał Górny

Re: [gentoo-portage-dev] [PATCH 3/3] portage.dbapi.vartree: Support exclusions in INSTALL_MASK

[Joakim Tjernlund]

On Thu, 2018-03-15 at 17:02 -0400, Alec Warner wrote:
> 
> 
> On Thu, Mar 15, 2018 at 3:22 PM, Michał Górny <mgorny@gentoo.org> wrote:
> > Allow INSTALL_MASK patterns to start with '-' to indicate that
> > a specific match is to be excluded from being masked. In this case,
> > the last matching pattern determines whether the file is actually
> > filtered out or kept.
> > ---

Yes, please ! I just needed this feature 2 hours ago.
I need the same for PKG_INSTALL_MASK 

I hope this allows me to do:
INSTALL_MASK="/usr/share/i18n/locales/* -/usr/share/i18n/locales/en_GB -/usr/share/i18n/locales/sv_SE"
to all in /usr/share/i18n/locales/* but /usr/share/i18n/locales/en_GB and /usr/share/i18n/locales/sv_SE
removed in usr/share/i18n/locales/ ?

 Jocke

Re: [gentoo-portage-dev] [PATCH 0/3] INSTALL_MASK refurbishing resubmit

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 2349 bytes --]

On 03/15/2018 12:22 PM, Michał Górny wrote:
> Hi,
> 
> Here are three of four INSTALL_MASK updates I've sent long time ago
> which were not really reviewed. The fourth patch added support
> for repo-defined install-mask.conf and I'll do that separately.
> 
> Those patches focus on smaller changes. What they change, in order:
> 
> 1. Removes explicit file removal code for FEATURES=no*. Instead, those
>    values are converted into additional INSTALL_MASK entries
>    and handled directly via INSTALL_MASK processing.
> 
> 2. Rework INSTALL_MASK to filter files while installing instead of
>    pre-stripping them. In other words, before: INSTALL_MASK removes
>    files from ${D} before merge. After: ${D} contains all the files,
>    Portage just skip INSTALL_MASK-ed stuff, verbosely indicating that.
> 
> 3. Adds support for exclusions in INSTALL_MASK. In other words, you
>    can do stuff like:
> 
>      INSTALL_MASK="/usr/share/locale -/usr/share/locale/en_US"
> 
> I have been using this via user patches since the last submission.
> Guessing by 'git log', this means almost 2 years now.
> 
> --
> Best regards,
> Michał Górny
> 
> Michał Górny (3):
>   portage.package.ebuild.config: Move FEATURES=no* handling there
>   portage.dbapi.vartree: Move INSTALL_MASK handling into merging
>   portage.dbapi.vartree: Support exclusions in INSTALL_MASK
> 
>  bin/misc-functions.sh                |  30 ----------
>  pym/portage/dbapi/vartree.py         | 104 ++++++++++++++++++++++-------------
>  pym/portage/package/ebuild/config.py |  11 ++++
>  3 files changed, 77 insertions(+), 68 deletions(-)

I like this patch set but here are some important things that I want it
to do differently:

1) For the unmerge code, it needs to read the appropriate
/var/db/pkg/*/*/{PKG,}INSTALL_MASK file in order to account for the
{PKG,}INSTALL_MASK settings that existed when the package was built
(PKG_INSTALL_MASK) and merged (INSTALL_MASK). A binary package should
use the value of INSTALL_MASK that existed at build time.

2) In order to support bashrc {PKG,}INSTALL_MASK settings, we need to
write the values from the environment to
${PORTAGE_BUILDDIR}/build-info/{PKG,}INSTALL_MASK and read them from
there (we do this for many other variables including QA_PREBUILT).
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 224 bytes --]

Re: [gentoo-portage-dev] [PATCH 3/3] portage.dbapi.vartree: Support exclusions in INSTALL_MASK

[Michał Górny]

W dniu czw, 15.03.2018 o godzinie 21∶44 +0000, użytkownik Joakim
Tjernlund napisał:
> On Thu, 2018-03-15 at 17:02 -0400, Alec Warner wrote:
> > 
> > 
> > On Thu, Mar 15, 2018 at 3:22 PM, Michał Górny <mgorny@gentoo.org> wrote:
> > > Allow INSTALL_MASK patterns to start with '-' to indicate that
> > > a specific match is to be excluded from being masked. In this case,
> > > the last matching pattern determines whether the file is actually
> > > filtered out or kept.
> > > ---
> 
> Yes, please ! I just needed this feature 2 hours ago.
> I need the same for PKG_INSTALL_MASK 
> 
> I hope this allows me to do:
> INSTALL_MASK="/usr/share/i18n/locales/* -/usr/share/i18n/locales/en_GB -/usr/share/i18n/locales/sv_SE"
> to all in /usr/share/i18n/locales/* but /usr/share/i18n/locales/en_GB and /usr/share/i18n/locales/sv_SE
> removed in usr/share/i18n/locales/ ?
> 

Yes, that is the intended use case.

-- 
Best regards,
Michał Górny

Re: [gentoo-portage-dev] [PATCH 3/3] portage.dbapi.vartree: Support exclusions in INSTALL_MASK

[Joakim Tjernlund]

On Fri, 2018-03-16 at 08:50 +0100, Michał Górny wrote:
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> 
> 
> W dniu czw, 15.03.2018 o godzinie 21∶44 +0000, użytkownik Joakim
> Tjernlund napisał:
> > On Thu, 2018-03-15 at 17:02 -0400, Alec Warner wrote:
> > > 
> > > 
> > > On Thu, Mar 15, 2018 at 3:22 PM, Michał Górny <mgorny@gentoo.org> wrote:
> > > > Allow INSTALL_MASK patterns to start with '-' to indicate that
> > > > a specific match is to be excluded from being masked. In this case,
> > > > the last matching pattern determines whether the file is actually
> > > > filtered out or kept.
> > > > ---
> > 
> > Yes, please ! I just needed this feature 2 hours ago.
> > I need the same for PKG_INSTALL_MASK
> > 
> > I hope this allows me to do:
> > INSTALL_MASK="/usr/share/i18n/locales/* -/usr/share/i18n/locales/en_GB -/usr/share/i18n/locales/sv_SE"
> > to all in /usr/share/i18n/locales/* but /usr/share/i18n/locales/en_GB and /usr/share/i18n/locales/sv_SE
> > removed in usr/share/i18n/locales/ ?
> > 
> 
> Yes, that is the intended use case.

Thanks, while on the subject I had a similar idea for stripping:

Today one have to build all files in glibc with debug syms just to get debug syms for
perf/valgrind in ld.so so it would be great if one could specify that just some files
should have debug syms.

 Jocke

Re: [gentoo-portage-dev] [PATCH 0/3] INSTALL_MASK refurbishing resubmit

[Joakim Tjernlund]

On Thu, 2018-03-15 at 20:22 +0100, Michał Górny wrote:
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> 
> 
> Hi,
> 
> Here are three of four INSTALL_MASK updates I've sent long time ago
> which were not really reviewed. The fourth patch added support
> for repo-defined install-mask.conf and I'll do that separately.
> 
> Those patches focus on smaller changes. What they change, in order:
> 
> 1. Removes explicit file removal code for FEATURES=no*. Instead, those
>    values are converted into additional INSTALL_MASK entries
>    and handled directly via INSTALL_MASK processing.
> 
> 2. Rework INSTALL_MASK to filter files while installing instead of
>    pre-stripping them. In other words, before: INSTALL_MASK removes
>    files from ${D} before merge. After: ${D} contains all the files,
>    Portage just skip INSTALL_MASK-ed stuff, verbosely indicating that.

Will this also remove corresponding split debug files?
There would be little/no point in keeping debug syms if the binary has been
MASKed 

 Jocke

Re: [gentoo-portage-dev] [PATCH 0/3] INSTALL_MASK refurbishing resubmit

[Michał Górny]

W dniu pią, 16.03.2018 o godzinie 08∶11 +0000, użytkownik Joakim
Tjernlund napisał:
> On Thu, 2018-03-15 at 20:22 +0100, Michał Górny wrote:
> > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > 
> > 
> > Hi,
> > 
> > Here are three of four INSTALL_MASK updates I've sent long time ago
> > which were not really reviewed. The fourth patch added support
> > for repo-defined install-mask.conf and I'll do that separately.
> > 
> > Those patches focus on smaller changes. What they change, in order:
> > 
> > 1. Removes explicit file removal code for FEATURES=no*. Instead, those
> >    values are converted into additional INSTALL_MASK entries
> >    and handled directly via INSTALL_MASK processing.
> > 
> > 2. Rework INSTALL_MASK to filter files while installing instead of
> >    pre-stripping them. In other words, before: INSTALL_MASK removes
> >    files from ${D} before merge. After: ${D} contains all the files,
> >    Portage just skip INSTALL_MASK-ed stuff, verbosely indicating that.
> 
> Will this also remove corresponding split debug files?
> There would be little/no point in keeping debug syms if the binary has been
> MASKed 
> 

Nope. Add both paths to INSTALL_MASK. Expecting it to do implicit magic
is a very bad idea.

-- 
Best regards,
Michał Górny

Re: [gentoo-portage-dev] [PATCH 0/3] INSTALL_MASK refurbishing resubmit

[Joakim Tjernlund]

On Thu, 2018-03-15 at 22:10 -0700, Zac Medico wrote:
> On 03/15/2018 12:22 PM, Michał Górny wrote:
> > Hi,
> > 
> > Here are three of four INSTALL_MASK updates I've sent long time ago
> > which were not really reviewed. The fourth patch added support
> > for repo-defined install-mask.conf and I'll do that separately.
> > 
> > Those patches focus on smaller changes. What they change, in order:
> > 
> > 1. Removes explicit file removal code for FEATURES=no*. Instead, those
> >    values are converted into additional INSTALL_MASK entries
> >    and handled directly via INSTALL_MASK processing.
> > 
> > 2. Rework INSTALL_MASK to filter files while installing instead of
> >    pre-stripping them. In other words, before: INSTALL_MASK removes
> >    files from ${D} before merge. After: ${D} contains all the files,
> >    Portage just skip INSTALL_MASK-ed stuff, verbosely indicating that.
> > 
> > 3. Adds support for exclusions in INSTALL_MASK. In other words, you
> >    can do stuff like:
> > 
> >      INSTALL_MASK="/usr/share/locale -/usr/share/locale/en_US"
> > 
> > I have been using this via user patches since the last submission.
> > Guessing by 'git log', this means almost 2 years now.
> > 
> > --
> > Best regards,
> > Michał Górny
> > 
> > Michał Górny (3):
> >   portage.package.ebuild.config: Move FEATURES=no* handling there
> >   portage.dbapi.vartree: Move INSTALL_MASK handling into merging
> >   portage.dbapi.vartree: Support exclusions in INSTALL_MASK
> > 
> >  bin/misc-functions.sh                |  30 ----------
> >  pym/portage/dbapi/vartree.py         | 104 ++++++++++++++++++++++-------------
> >  pym/portage/package/ebuild/config.py |  11 ++++
> >  3 files changed, 77 insertions(+), 68 deletions(-)
> 
> I like this patch set but here are some important things that I want it
> to do differently:
> 
> 1) For the unmerge code, it needs to read the appropriate
> /var/db/pkg/*/*/{PKG,}INSTALL_MASK file in order to account for the
> {PKG,}INSTALL_MASK settings that existed when the package was built
> (PKG_INSTALL_MASK) and merged (INSTALL_MASK). A binary package should
> use the value of INSTALL_MASK that existed at build time.

Why does unmerge code need to know PKG_INSTALL_MASK? The files are not
installed so nothing to filter ?
similarly for merge of binary pkgs I guess, PKG_INSTALL_MASKed files are not
in the binary packed so no need to something special in this case ?

> 
> 2) In order to support bashrc {PKG,}INSTALL_MASK settings, we need to
> write the values from the environment to
> ${PORTAGE_BUILDDIR}/build-info/{PKG,}INSTALL_MASK and read them from
> there (we do this for many other variables including QA_PREBUILT).

Re: [gentoo-portage-dev] [PATCH 0/3] INSTALL_MASK refurbishing resubmit

[Michał Górny]

W dniu czw, 15.03.2018 o godzinie 22∶10 -0700, użytkownik Zac Medico
napisał:
> On 03/15/2018 12:22 PM, Michał Górny wrote:
> > Hi,
> > 
> > Here are three of four INSTALL_MASK updates I've sent long time ago
> > which were not really reviewed. The fourth patch added support
> > for repo-defined install-mask.conf and I'll do that separately.
> > 
> > Those patches focus on smaller changes. What they change, in order:
> > 
> > 1. Removes explicit file removal code for FEATURES=no*. Instead, those
> >    values are converted into additional INSTALL_MASK entries
> >    and handled directly via INSTALL_MASK processing.
> > 
> > 2. Rework INSTALL_MASK to filter files while installing instead of
> >    pre-stripping them. In other words, before: INSTALL_MASK removes
> >    files from ${D} before merge. After: ${D} contains all the files,
> >    Portage just skip INSTALL_MASK-ed stuff, verbosely indicating that.
> > 
> > 3. Adds support for exclusions in INSTALL_MASK. In other words, you
> >    can do stuff like:
> > 
> >      INSTALL_MASK="/usr/share/locale -/usr/share/locale/en_US"
> > 
> > I have been using this via user patches since the last submission.
> > Guessing by 'git log', this means almost 2 years now.
> > 
> > --
> > Best regards,
> > Michał Górny
> > 
> > Michał Górny (3):
> >   portage.package.ebuild.config: Move FEATURES=no* handling there
> >   portage.dbapi.vartree: Move INSTALL_MASK handling into merging
> >   portage.dbapi.vartree: Support exclusions in INSTALL_MASK
> > 
> >  bin/misc-functions.sh                |  30 ----------
> >  pym/portage/dbapi/vartree.py         | 104 ++++++++++++++++++++++-------------
> >  pym/portage/package/ebuild/config.py |  11 ++++
> >  3 files changed, 77 insertions(+), 68 deletions(-)
> 
> I like this patch set but here are some important things that I want it
> to do differently:
> 
> 1) For the unmerge code, it needs to read the appropriate
> /var/db/pkg/*/*/{PKG,}INSTALL_MASK file in order to account for the
> {PKG,}INSTALL_MASK settings that existed when the package was built
> (PKG_INSTALL_MASK) and merged (INSTALL_MASK). A binary package should
> use the value of INSTALL_MASK that existed at build time.

> 2) In order to support bashrc {PKG,}INSTALL_MASK settings, we need to
> write the values from the environment to
> ${PORTAGE_BUILDDIR}/build-info/{PKG,}INSTALL_MASK and read them from
> there (we do this for many other variables including QA_PREBUILT).

I presume bin/phase-functions.sh __dyn_install is where I'm supposed to
write them. Could you suggest where is the best place to read them back?
Should the merge code do that explicitly while handling INSTALL_MASK, or
should some of the config classes do that?

-- 
Best regards,
Michał Górny

Re: [gentoo-portage-dev] [PATCH 0/3] INSTALL_MASK refurbishing resubmit

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 3485 bytes --]

On 03/16/2018 03:08 AM, Michał Górny wrote:
> W dniu czw, 15.03.2018 o godzinie 22∶10 -0700, użytkownik Zac Medico
> napisał:
>> On 03/15/2018 12:22 PM, Michał Górny wrote:
>>> Hi,
>>>
>>> Here are three of four INSTALL_MASK updates I've sent long time ago
>>> which were not really reviewed. The fourth patch added support
>>> for repo-defined install-mask.conf and I'll do that separately.
>>>
>>> Those patches focus on smaller changes. What they change, in order:
>>>
>>> 1. Removes explicit file removal code for FEATURES=no*. Instead, those
>>>    values are converted into additional INSTALL_MASK entries
>>>    and handled directly via INSTALL_MASK processing.
>>>
>>> 2. Rework INSTALL_MASK to filter files while installing instead of
>>>    pre-stripping them. In other words, before: INSTALL_MASK removes
>>>    files from ${D} before merge. After: ${D} contains all the files,
>>>    Portage just skip INSTALL_MASK-ed stuff, verbosely indicating that.
>>>
>>> 3. Adds support for exclusions in INSTALL_MASK. In other words, you
>>>    can do stuff like:
>>>
>>>      INSTALL_MASK="/usr/share/locale -/usr/share/locale/en_US"
>>>
>>> I have been using this via user patches since the last submission.
>>> Guessing by 'git log', this means almost 2 years now.
>>>
>>> --
>>> Best regards,
>>> Michał Górny
>>>
>>> Michał Górny (3):
>>>   portage.package.ebuild.config: Move FEATURES=no* handling there
>>>   portage.dbapi.vartree: Move INSTALL_MASK handling into merging
>>>   portage.dbapi.vartree: Support exclusions in INSTALL_MASK
>>>
>>>  bin/misc-functions.sh                |  30 ----------
>>>  pym/portage/dbapi/vartree.py         | 104 ++++++++++++++++++++++-------------
>>>  pym/portage/package/ebuild/config.py |  11 ++++
>>>  3 files changed, 77 insertions(+), 68 deletions(-)
>>
>> I like this patch set but here are some important things that I want it
>> to do differently:
>>
>> 1) For the unmerge code, it needs to read the appropriate
>> /var/db/pkg/*/*/{PKG,}INSTALL_MASK file in order to account for the
>> {PKG,}INSTALL_MASK settings that existed when the package was built
>> (PKG_INSTALL_MASK) and merged (INSTALL_MASK). A binary package should
>> use the value of INSTALL_MASK that existed at build time.
> 
>> 2) In order to support bashrc {PKG,}INSTALL_MASK settings, we need to
>> write the values from the environment to
>> ${PORTAGE_BUILDDIR}/build-info/{PKG,}INSTALL_MASK and read them from
>> there (we do this for many other variables including QA_PREBUILT).
> 
> I presume bin/phase-functions.sh __dyn_install is where I'm supposed to
> write them. Could you suggest where is the best place to read them back?

We can read them back just when they are needed.

PKG_INSTALL_MASK should be handled in the EbuildPhase class when
self.phase is "package". In order to preserve behavior, EbuildPhase will
have to create a temporary copy of ${D} and apply PKG_INSTALL_MASK to
it, for __dyn_package to use.

INSTALL_MASK should be handled in the dblink treewalk method like it is now.


> Should the merge code do that explicitly while handling INSTALL_MASK, or
> should some of the config classes do that?

The config class only needs to be involved if we want to expose some API
related to {PKG,}INSTALL_MASK there, but the config class is bloated
enough as it is so it's better to expose a helper class like the
ConfigProtect class.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 224 bytes --]

Re: [gentoo-portage-dev] [PATCH 0/3] INSTALL_MASK refurbishing resubmit

[Michał Górny]

W dniu pią, 16.03.2018 o godzinie 10∶07 -0700, użytkownik Zac Medico
napisał:
> On 03/16/2018 03:08 AM, Michał Górny wrote:
> > W dniu czw, 15.03.2018 o godzinie 22∶10 -0700, użytkownik Zac Medico
> > napisał:
> > > On 03/15/2018 12:22 PM, Michał Górny wrote:
> > > > Hi,
> > > > 
> > > > Here are three of four INSTALL_MASK updates I've sent long time ago
> > > > which were not really reviewed. The fourth patch added support
> > > > for repo-defined install-mask.conf and I'll do that separately.
> > > > 
> > > > Those patches focus on smaller changes. What they change, in order:
> > > > 
> > > > 1. Removes explicit file removal code for FEATURES=no*. Instead, those
> > > >    values are converted into additional INSTALL_MASK entries
> > > >    and handled directly via INSTALL_MASK processing.
> > > > 
> > > > 2. Rework INSTALL_MASK to filter files while installing instead of
> > > >    pre-stripping them. In other words, before: INSTALL_MASK removes
> > > >    files from ${D} before merge. After: ${D} contains all the files,
> > > >    Portage just skip INSTALL_MASK-ed stuff, verbosely indicating that.
> > > > 
> > > > 3. Adds support for exclusions in INSTALL_MASK. In other words, you
> > > >    can do stuff like:
> > > > 
> > > >      INSTALL_MASK="/usr/share/locale -/usr/share/locale/en_US"
> > > > 
> > > > I have been using this via user patches since the last submission.
> > > > Guessing by 'git log', this means almost 2 years now.
> > > > 
> > > > --
> > > > Best regards,
> > > > Michał Górny
> > > > 
> > > > Michał Górny (3):
> > > >   portage.package.ebuild.config: Move FEATURES=no* handling there
> > > >   portage.dbapi.vartree: Move INSTALL_MASK handling into merging
> > > >   portage.dbapi.vartree: Support exclusions in INSTALL_MASK
> > > > 
> > > >  bin/misc-functions.sh                |  30 ----------
> > > >  pym/portage/dbapi/vartree.py         | 104 ++++++++++++++++++++++-------------
> > > >  pym/portage/package/ebuild/config.py |  11 ++++
> > > >  3 files changed, 77 insertions(+), 68 deletions(-)
> > > 
> > > I like this patch set but here are some important things that I want it
> > > to do differently:
> > > 
> > > 1) For the unmerge code, it needs to read the appropriate
> > > /var/db/pkg/*/*/{PKG,}INSTALL_MASK file in order to account for the
> > > {PKG,}INSTALL_MASK settings that existed when the package was built
> > > (PKG_INSTALL_MASK) and merged (INSTALL_MASK). A binary package should
> > > use the value of INSTALL_MASK that existed at build time.
> > > 2) In order to support bashrc {PKG,}INSTALL_MASK settings, we need to
> > > write the values from the environment to
> > > ${PORTAGE_BUILDDIR}/build-info/{PKG,}INSTALL_MASK and read them from
> > > there (we do this for many other variables including QA_PREBUILT).
> > 
> > I presume bin/phase-functions.sh __dyn_install is where I'm supposed to
> > write them. Could you suggest where is the best place to read them back?
> 
> We can read them back just when they are needed.
> 
> PKG_INSTALL_MASK should be handled in the EbuildPhase class when
> self.phase is "package". In order to preserve behavior, EbuildPhase will
> have to create a temporary copy of ${D} and apply PKG_INSTALL_MASK to
> it, for __dyn_package to use.

But do I need to change anything for PKG_INSTALL_MASK? My original patch
did not touch that, so it can just continue happening as it is now.

> INSTALL_MASK should be handled in the dblink treewalk method like it is now.

But we also need to read it for unmerge, correct?

> > Should the merge code do that explicitly while handling INSTALL_MASK, or
> > should some of the config classes do that?
> 
> The config class only needs to be involved if we want to expose some API
> related to {PKG,}INSTALL_MASK there, but the config class is bloated
> enough as it is so it's better to expose a helper class like the
> ConfigProtect class.

-- 
Best regards,
Michał Górny

Re: [gentoo-portage-dev] [PATCH 0/3] INSTALL_MASK refurbishing resubmit

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 4166 bytes --]

On 03/16/2018 02:13 PM, Michał Górny wrote:
> W dniu pią, 16.03.2018 o godzinie 10∶07 -0700, użytkownik Zac Medico
> napisał:
>> On 03/16/2018 03:08 AM, Michał Górny wrote:
>>> W dniu czw, 15.03.2018 o godzinie 22∶10 -0700, użytkownik Zac Medico
>>> napisał:
>>>> On 03/15/2018 12:22 PM, Michał Górny wrote:
>>>>> Hi,
>>>>>
>>>>> Here are three of four INSTALL_MASK updates I've sent long time ago
>>>>> which were not really reviewed. The fourth patch added support
>>>>> for repo-defined install-mask.conf and I'll do that separately.
>>>>>
>>>>> Those patches focus on smaller changes. What they change, in order:
>>>>>
>>>>> 1. Removes explicit file removal code for FEATURES=no*. Instead, those
>>>>>    values are converted into additional INSTALL_MASK entries
>>>>>    and handled directly via INSTALL_MASK processing.
>>>>>
>>>>> 2. Rework INSTALL_MASK to filter files while installing instead of
>>>>>    pre-stripping them. In other words, before: INSTALL_MASK removes
>>>>>    files from ${D} before merge. After: ${D} contains all the files,
>>>>>    Portage just skip INSTALL_MASK-ed stuff, verbosely indicating that.
>>>>>
>>>>> 3. Adds support for exclusions in INSTALL_MASK. In other words, you
>>>>>    can do stuff like:
>>>>>
>>>>>      INSTALL_MASK="/usr/share/locale -/usr/share/locale/en_US"
>>>>>
>>>>> I have been using this via user patches since the last submission.
>>>>> Guessing by 'git log', this means almost 2 years now.
>>>>>
>>>>> --
>>>>> Best regards,
>>>>> Michał Górny
>>>>>
>>>>> Michał Górny (3):
>>>>>   portage.package.ebuild.config: Move FEATURES=no* handling there
>>>>>   portage.dbapi.vartree: Move INSTALL_MASK handling into merging
>>>>>   portage.dbapi.vartree: Support exclusions in INSTALL_MASK
>>>>>
>>>>>  bin/misc-functions.sh                |  30 ----------
>>>>>  pym/portage/dbapi/vartree.py         | 104 ++++++++++++++++++++++-------------
>>>>>  pym/portage/package/ebuild/config.py |  11 ++++
>>>>>  3 files changed, 77 insertions(+), 68 deletions(-)
>>>>
>>>> I like this patch set but here are some important things that I want it
>>>> to do differently:
>>>>
>>>> 1) For the unmerge code, it needs to read the appropriate
>>>> /var/db/pkg/*/*/{PKG,}INSTALL_MASK file in order to account for the
>>>> {PKG,}INSTALL_MASK settings that existed when the package was built
>>>> (PKG_INSTALL_MASK) and merged (INSTALL_MASK). A binary package should
>>>> use the value of INSTALL_MASK that existed at build time.
>>>> 2) In order to support bashrc {PKG,}INSTALL_MASK settings, we need to
>>>> write the values from the environment to
>>>> ${PORTAGE_BUILDDIR}/build-info/{PKG,}INSTALL_MASK and read them from
>>>> there (we do this for many other variables including QA_PREBUILT).
>>>
>>> I presume bin/phase-functions.sh __dyn_install is where I'm supposed to
>>> write them. Could you suggest where is the best place to read them back?
>>
>> We can read them back just when they are needed.
>>
>> PKG_INSTALL_MASK should be handled in the EbuildPhase class when
>> self.phase is "package". In order to preserve behavior, EbuildPhase will
>> have to create a temporary copy of ${D} and apply PKG_INSTALL_MASK to
>> it, for __dyn_package to use.
> 
> But do I need to change anything for PKG_INSTALL_MASK? My original patch
> did not touch that, so it can just continue happening as it is now.
> 
>> INSTALL_MASK should be handled in the dblink treewalk method like it is now.
> 
> But we also need to read it for unmerge, correct?

Oh right. We should load it in the dblink _match_contents or getcontents
method, since we need it for operation of the _match_contents method
which is called by isowner.

>>> Should the merge code do that explicitly while handling INSTALL_MASK, or
>>> should some of the config classes do that?
>>
>> The config class only needs to be involved if we want to expose some API
>> related to {PKG,}INSTALL_MASK there, but the config class is bloated
>> enough as it is so it's better to expose a helper class like the
>> ConfigProtect class.
> 


-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 224 bytes --]

Re: [gentoo-portage-dev] [PATCH 0/3] INSTALL_MASK refurbishing resubmit

[Michał Górny]

W dniu czw, 15.03.2018 o godzinie 22∶10 -0700, użytkownik Zac Medico
napisał:
>  A binary package should
> use the value of INSTALL_MASK that existed at build time.
> 

Wait a minute! This doesn't make any sense. The whole point of having
separate PKG_INSTALL_MASK and INSTALL_MASK is to be able to strip stuff
from more complete binary packages, not to force original restrictions
forever.

-- 
Best regards,
Michał Górny

Re: [gentoo-portage-dev] [PATCH 0/3] INSTALL_MASK refurbishing resubmit

[Joakim Tjernlund]

On Fri, 2018-03-16 at 09:13 +0100, Michał Górny wrote:
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> 
> 
> W dniu pią, 16.03.2018 o godzinie 08∶11 +0000, użytkownik Joakim
> Tjernlund napisał:
> > On Thu, 2018-03-15 at 20:22 +0100, Michał Górny wrote:
> > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > 
> > > 
> > > Hi,
> > > 
> > > Here are three of four INSTALL_MASK updates I've sent long time ago
> > > which were not really reviewed. The fourth patch added support
> > > for repo-defined install-mask.conf and I'll do that separately.
> > > 
> > > Those patches focus on smaller changes. What they change, in order:
> > > 
> > > 1. Removes explicit file removal code for FEATURES=no*. Instead, those
> > >    values are converted into additional INSTALL_MASK entries
> > >    and handled directly via INSTALL_MASK processing.
> > > 
> > > 2. Rework INSTALL_MASK to filter files while installing instead of
> > >    pre-stripping them. In other words, before: INSTALL_MASK removes
> > >    files from ${D} before merge. After: ${D} contains all the files,
> > >    Portage just skip INSTALL_MASK-ed stuff, verbosely indicating that.
> > 
> > Will this also remove corresponding split debug files?
> > There would be little/no point in keeping debug syms if the binary has been
> > MASKed
> > 
> 
> Nope. Add both paths to INSTALL_MASK. Expecting it to do implicit magic
> is a very bad idea.

Maybe but it also makes senses to get rid of them. To me it is only a matter
of applying PKG_INSTALL_MASK before applying strip debug, does that make sense ?

 Jocke

Re: [gentoo-portage-dev] [PATCH 0/3] INSTALL_MASK refurbishing resubmit

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 953 bytes --]

On 03/18/2018 02:03 AM, Michał Górny wrote:
> W dniu czw, 15.03.2018 o godzinie 22∶10 -0700, użytkownik Zac Medico
> napisał:
>>  A binary package should
>> use the value of INSTALL_MASK that existed at build time.
>>
> 
> Wait a minute! This doesn't make any sense. The whole point of having
> separate PKG_INSTALL_MASK and INSTALL_MASK is to be able to strip stuff
> from more complete binary packages, not to force original restrictions
> forever.

Okay, we should apply latest INSTALL_MASK settings when installing a
binary package? That seems reasonable.

I want to respect settings embedded in the binary package whenever it
could be useful, since my intention if for binhost clients to be able to
treat the binhost as a single source of truth, so that binary packages
can be installed without dependency on source ebuild
repositories/profiles, as discussed here:

https://bugs.gentoo.org/644990
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 224 bytes --]

Re: [gentoo-portage-dev] [PATCH 0/3] INSTALL_MASK refurbishing resubmit

[Joakim Tjernlund]

On Sun, 2018-03-18 at 10:03 +0100, Michał Górny wrote:
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> 
> 
> W dniu czw, 15.03.2018 o godzinie 22∶10 -0700, użytkownik Zac Medico
> napisał:
> >  A binary package should
> > use the value of INSTALL_MASK that existed at build time.
> > 
> 
> Wait a minute! This doesn't make any sense. The whole point of having
> separate PKG_INSTALL_MASK and INSTALL_MASK is to be able to strip stuff
> from more complete binary packages, not to force original restrictions
> forever.

These discussions also mentions PKG_INSTALL_MASK while the actual patches
only mention INSTALL_MASK. I am getting somewhat confused, does
the patches support PKG_INSTALL_MASK too or do you only intend to support
this new exclusion syntax in INSTALL_MASK?

    Jocke

Re: [gentoo-portage-dev] [PATCH 0/3] INSTALL_MASK refurbishing resubmit

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 2177 bytes --]

On 03/15/2018 12:22 PM, Michał Górny wrote:
> Hi,
> 
> Here are three of four INSTALL_MASK updates I've sent long time ago
> which were not really reviewed. The fourth patch added support
> for repo-defined install-mask.conf and I'll do that separately.
> 
> Those patches focus on smaller changes. What they change, in order:
> 
> 1. Removes explicit file removal code for FEATURES=no*. Instead, those
>    values are converted into additional INSTALL_MASK entries
>    and handled directly via INSTALL_MASK processing.
> 
> 2. Rework INSTALL_MASK to filter files while installing instead of
>    pre-stripping them. In other words, before: INSTALL_MASK removes
>    files from ${D} before merge. After: ${D} contains all the files,
>    Portage just skip INSTALL_MASK-ed stuff, verbosely indicating that.
> 
> 3. Adds support for exclusions in INSTALL_MASK. In other words, you
>    can do stuff like:
> 
>      INSTALL_MASK="/usr/share/locale -/usr/share/locale/en_US"
> 
> I have been using this via user patches since the last submission.
> Guessing by 'git log', this means almost 2 years now.
> 
> --
> Best regards,
> Michał Górny
> 
> Michał Górny (3):
>   portage.package.ebuild.config: Move FEATURES=no* handling there
>   portage.dbapi.vartree: Move INSTALL_MASK handling into merging
>   portage.dbapi.vartree: Support exclusions in INSTALL_MASK
> 
>  bin/misc-functions.sh                |  30 ----------
>  pym/portage/dbapi/vartree.py         | 104 ++++++++++++++++++++++-------------
>  pym/portage/package/ebuild/config.py |  11 ++++
>  3 files changed, 77 insertions(+), 68 deletions(-)
> 

As mentioned in #gentoo-portage today, the rationale for including the
INSTALL_MASKed files in CONTENTS is to that we can detect collisions
that would have occurred had people not been using INSTALL_MASK.

Since people can use INSTALL_MASK to intentionally prevent collisions,
in cases where COLLISION_IGNORE is not appropriate (this is common
practice at my workplace), we'll need a new FEATURES setting to trigger
the new behavior where INSTALL_MASKed files still trigger file collisions.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 224 bytes --]

Re: [gentoo-portage-dev] [PATCH 0/3] INSTALL_MASK refurbishing resubmit

[Joakim Tjernlund]

On Mon, 2018-03-19 at 15:59 -0700, Zac Medico wrote:
> On 03/15/2018 12:22 PM, Michał Górny wrote:
> > Hi,
> > 
> > Here are three of four INSTALL_MASK updates I've sent long time ago
> > which were not really reviewed. The fourth patch added support
> > for repo-defined install-mask.conf and I'll do that separately.
> > 
> > Those patches focus on smaller changes. What they change, in order:
> > 
> > 1. Removes explicit file removal code for FEATURES=no*. Instead, those
> >    values are converted into additional INSTALL_MASK entries
> >    and handled directly via INSTALL_MASK processing.
> > 
> > 2. Rework INSTALL_MASK to filter files while installing instead of
> >    pre-stripping them. In other words, before: INSTALL_MASK removes
> >    files from ${D} before merge. After: ${D} contains all the files,
> >    Portage just skip INSTALL_MASK-ed stuff, verbosely indicating that.
> > 
> > 3. Adds support for exclusions in INSTALL_MASK. In other words, you
> >    can do stuff like:
> > 
> >      INSTALL_MASK="/usr/share/locale -/usr/share/locale/en_US"
> > 
> > I have been using this via user patches since the last submission.
> > Guessing by 'git log', this means almost 2 years now.
> > 
> > --
> > Best regards,
> > Michał Górny
> > 
> > Michał Górny (3):
> >   portage.package.ebuild.config: Move FEATURES=no* handling there
> >   portage.dbapi.vartree: Move INSTALL_MASK handling into merging
> >   portage.dbapi.vartree: Support exclusions in INSTALL_MASK
> > 
> >  bin/misc-functions.sh                |  30 ----------
> >  pym/portage/dbapi/vartree.py         | 104 ++++++++++++++++++++++-------------
> >  pym/portage/package/ebuild/config.py |  11 ++++
> >  3 files changed, 77 insertions(+), 68 deletions(-)
> > 
> 
> As mentioned in #gentoo-portage today, the rationale for including the
> INSTALL_MASKed files in CONTENTS is to that we can detect collisions
> that would have occurred had people not been using INSTALL_MASK.
> 
> Since people can use INSTALL_MASK to intentionally prevent collisions,
> in cases where COLLISION_IGNORE is not appropriate (this is common
> practice at my workplace), we'll need a new FEATURES setting to trigger
> the new behavior where INSTALL_MASKed files still trigger file collisions.

Are we going to see this in Portage soon? And PKG_INSTALL_MASK too ?

Re: [gentoo-portage-dev] [PATCH 0/3] INSTALL_MASK refurbishing resubmit

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 2548 bytes --]

On 03/22/2018 05:52 PM, Joakim Tjernlund wrote:
> On Mon, 2018-03-19 at 15:59 -0700, Zac Medico wrote:
>> On 03/15/2018 12:22 PM, Michał Górny wrote:
>>> Hi,
>>>
>>> Here are three of four INSTALL_MASK updates I've sent long time ago
>>> which were not really reviewed. The fourth patch added support
>>> for repo-defined install-mask.conf and I'll do that separately.
>>>
>>> Those patches focus on smaller changes. What they change, in order:
>>>
>>> 1. Removes explicit file removal code for FEATURES=no*. Instead, those
>>>    values are converted into additional INSTALL_MASK entries
>>>    and handled directly via INSTALL_MASK processing.
>>>
>>> 2. Rework INSTALL_MASK to filter files while installing instead of
>>>    pre-stripping them. In other words, before: INSTALL_MASK removes
>>>    files from ${D} before merge. After: ${D} contains all the files,
>>>    Portage just skip INSTALL_MASK-ed stuff, verbosely indicating that.
>>>
>>> 3. Adds support for exclusions in INSTALL_MASK. In other words, you
>>>    can do stuff like:
>>>
>>>      INSTALL_MASK="/usr/share/locale -/usr/share/locale/en_US"
>>>
>>> I have been using this via user patches since the last submission.
>>> Guessing by 'git log', this means almost 2 years now.
>>>
>>> --
>>> Best regards,
>>> Michał Górny
>>>
>>> Michał Górny (3):
>>>   portage.package.ebuild.config: Move FEATURES=no* handling there
>>>   portage.dbapi.vartree: Move INSTALL_MASK handling into merging
>>>   portage.dbapi.vartree: Support exclusions in INSTALL_MASK
>>>
>>>  bin/misc-functions.sh                |  30 ----------
>>>  pym/portage/dbapi/vartree.py         | 104 ++++++++++++++++++++++-------------
>>>  pym/portage/package/ebuild/config.py |  11 ++++
>>>  3 files changed, 77 insertions(+), 68 deletions(-)
>>>
>>
>> As mentioned in #gentoo-portage today, the rationale for including the
>> INSTALL_MASKed files in CONTENTS is to that we can detect collisions
>> that would have occurred had people not been using INSTALL_MASK.
>>
>> Since people can use INSTALL_MASK to intentionally prevent collisions,
>> in cases where COLLISION_IGNORE is not appropriate (this is common
>> practice at my workplace), we'll need a new FEATURES setting to trigger
>> the new behavior where INSTALL_MASKed files still trigger file collisions.
> 
> Are we going to see this in Portage soon? And PKG_INSTALL_MASK too ?

Yes, I'll clean up the patches an resubmit them. Bug filed:
https://bugs.gentoo.org/651214
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 224 bytes --]

Re: [gentoo-portage-dev] [PATCH 0/3] INSTALL_MASK refurbishing resubmit

[Michał Górny]

W dniu pią, 23.03.2018 o godzinie 00∶52 +0000, użytkownik Joakim
Tjernlund napisał:
> On Mon, 2018-03-19 at 15:59 -0700, Zac Medico wrote:
> > On 03/15/2018 12:22 PM, Michał Górny wrote:
> > > Hi,
> > > 
> > > Here are three of four INSTALL_MASK updates I've sent long time ago
> > > which were not really reviewed. The fourth patch added support
> > > for repo-defined install-mask.conf and I'll do that separately.
> > > 
> > > Those patches focus on smaller changes. What they change, in order:
> > > 
> > > 1. Removes explicit file removal code for FEATURES=no*. Instead, those
> > >    values are converted into additional INSTALL_MASK entries
> > >    and handled directly via INSTALL_MASK processing.
> > > 
> > > 2. Rework INSTALL_MASK to filter files while installing instead of
> > >    pre-stripping them. In other words, before: INSTALL_MASK removes
> > >    files from ${D} before merge. After: ${D} contains all the files,
> > >    Portage just skip INSTALL_MASK-ed stuff, verbosely indicating that.
> > > 
> > > 3. Adds support for exclusions in INSTALL_MASK. In other words, you
> > >    can do stuff like:
> > > 
> > >      INSTALL_MASK="/usr/share/locale -/usr/share/locale/en_US"
> > > 
> > > I have been using this via user patches since the last submission.
> > > Guessing by 'git log', this means almost 2 years now.
> > > 
> > > --
> > > Best regards,
> > > Michał Górny
> > > 
> > > Michał Górny (3):
> > >   portage.package.ebuild.config: Move FEATURES=no* handling there
> > >   portage.dbapi.vartree: Move INSTALL_MASK handling into merging
> > >   portage.dbapi.vartree: Support exclusions in INSTALL_MASK
> > > 
> > >  bin/misc-functions.sh                |  30 ----------
> > >  pym/portage/dbapi/vartree.py         | 104 ++++++++++++++++++++++-------------
> > >  pym/portage/package/ebuild/config.py |  11 ++++
> > >  3 files changed, 77 insertions(+), 68 deletions(-)
> > > 
> > 
> > As mentioned in #gentoo-portage today, the rationale for including the
> > INSTALL_MASKed files in CONTENTS is to that we can detect collisions
> > that would have occurred had people not been using INSTALL_MASK.
> > 
> > Since people can use INSTALL_MASK to intentionally prevent collisions,
> > in cases where COLLISION_IGNORE is not appropriate (this is common
> > practice at my workplace), we'll need a new FEATURES setting to trigger
> > the new behavior where INSTALL_MASKed files still trigger file collisions.
> 
> Are we going to see this in Portage soon? And PKG_INSTALL_MASK too ?

It's in sys-apps/portage-mgorny. Whatever's going to land in sys-
apps/portage, it's probably going to be half-broken to satisfy
somebody's colleague's corner case of misusing INSTALL_MASK.

-- 
Best regards,
Michał Górny

Re: [gentoo-portage-dev] [PATCH 0/3] INSTALL_MASK refurbishing resubmit

[Joakim Tjernlund]

On Fri, 2018-03-23 at 09:33 +0100, Michał Górny wrote:
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> 
> 
> W dniu pią, 23.03.2018 o godzinie 00∶52 +0000, użytkownik Joakim
> Tjernlund napisał:
> > On Mon, 2018-03-19 at 15:59 -0700, Zac Medico wrote:
> > > On 03/15/2018 12:22 PM, Michał Górny wrote:
> > > > Hi,
> > > > 
> > > > Here are three of four INSTALL_MASK updates I've sent long time ago
> > > > which were not really reviewed. The fourth patch added support
> > > > for repo-defined install-mask.conf and I'll do that separately.
> > > > 
> > > > Those patches focus on smaller changes. What they change, in order:
> > > > 
> > > > 1. Removes explicit file removal code for FEATURES=no*. Instead, those
> > > >    values are converted into additional INSTALL_MASK entries
> > > >    and handled directly via INSTALL_MASK processing.
> > > > 
> > > > 2. Rework INSTALL_MASK to filter files while installing instead of
> > > >    pre-stripping them. In other words, before: INSTALL_MASK removes
> > > >    files from ${D} before merge. After: ${D} contains all the files,
> > > >    Portage just skip INSTALL_MASK-ed stuff, verbosely indicating that.
> > > > 
> > > > 3. Adds support for exclusions in INSTALL_MASK. In other words, you
> > > >    can do stuff like:
> > > > 
> > > >      INSTALL_MASK="/usr/share/locale -/usr/share/locale/en_US"
> > > > 
> > > > I have been using this via user patches since the last submission.
> > > > Guessing by 'git log', this means almost 2 years now.
> > > > 
> > > > --
> > > > Best regards,
> > > > Michał Górny
> > > > 
> > > > Michał Górny (3):
> > > >   portage.package.ebuild.config: Move FEATURES=no* handling there
> > > >   portage.dbapi.vartree: Move INSTALL_MASK handling into merging
> > > >   portage.dbapi.vartree: Support exclusions in INSTALL_MASK
> > > > 
> > > >  bin/misc-functions.sh                |  30 ----------
> > > >  pym/portage/dbapi/vartree.py         | 104 ++++++++++++++++++++++-------------
> > > >  pym/portage/package/ebuild/config.py |  11 ++++
> > > >  3 files changed, 77 insertions(+), 68 deletions(-)
> > > > 
> > > 
> > > As mentioned in #gentoo-portage today, the rationale for including the
> > > INSTALL_MASKed files in CONTENTS is to that we can detect collisions
> > > that would have occurred had people not been using INSTALL_MASK.
> > > 
> > > Since people can use INSTALL_MASK to intentionally prevent collisions,
> > > in cases where COLLISION_IGNORE is not appropriate (this is common
> > > practice at my workplace), we'll need a new FEATURES setting to trigger
> > > the new behavior where INSTALL_MASKed files still trigger file collisions.
> > 
> > Are we going to see this in Portage soon? And PKG_INSTALL_MASK too ?
> 
> It's in sys-apps/portage-mgorny. Whatever's going to land in sys-
> apps/portage, it's probably going to be half-broken to satisfy
> somebody's colleague's corner case of misusing INSTALL_MASK.

I looked and saw the INSTALL_MASK part but no mention of PKG_INSTALL_MASK(which is what I need)