[gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Bertrand Simonnet]

[-- Attachment #1.1: Type: text/plain, Size: 1152 bytes --]

Hi guys,

I have been working on a patch to make package.env (and env/ files)
stackable with the profiles.
The main goal would be to have a mechanism to be able to define a
per-package, per-profile environment in a simple way.

We can currently do this with profile.bashrc but we have to add some logic
there to decide which package are relevant which already exist in ebuild.sh
and pym/*.

I attached my current patch which contain two main modifications:
- ebuild.sh walk the profile and source all scripts with path:
${profile_path}/env/${category}/${PN}
- config.py imports package.env from all profiles and then from
/etc/portage/package.env

Note:
* I gated the env/$CATEGORY/$PN scripts with a new feature: per-profile-env.
   I couldn't do this with package.env as the scripts are parsed before we
initialize self.features.
* I am not particularly attach to both package.env and env/$CATEGORY/$PN.
It might make more sense to have only env/$CATEGORY/$PN, especially
considering the problem with package.env.

Do you have any suggestions on this patch ?

Would you be willing to accept this patch once the few problems are sorted
out ?

Thanks!

[-- Attachment #1.2: Type: text/html, Size: 2513 bytes --]

[-- Attachment #2: 0001-per-profile-package-env.patch --]
[-- Type: application/octet-stream, Size: 8587 bytes --]

From 550d25fc93d087186438bbbf1cc7aacbfd6bbc8f Mon Sep 17 00:00:00 2001
From: Bertrand SIMONNET <bsimonnet@chromium.org>
Date: Tue, 29 Jul 2014 15:14:40 -0700
Subject: [PATCH] Make package.env and env profile specific

Allows profiles to override environment variables throught package.env and env.
package.env becomes a per-profile configuration like package.use and
package.mask.

The environment bash file invoked by package.env must me placed in the same
profile.

Change-Id: I16bcd75790213d2204f83d4aa6e8b910f8829b6e
---
 bin/ebuild.sh                        | 78 ++++++++++++++++++++++--------------
 pym/portage/const.py                 |  1 +
 pym/portage/package/ebuild/config.py | 55 ++++++++++++++-----------
 3 files changed, 80 insertions(+), 54 deletions(-)

diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index be044e0..e188df7 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -67,11 +67,11 @@ unset BASH_ENV
 # earlier portage versions do not detect a version change in this case
 # (9999 to 9999) and therefore they try execute an incompatible version of
 # ebuild.sh during the upgrade.
-export PORTAGE_BZIP2_COMMAND=${PORTAGE_BZIP2_COMMAND:-bzip2} 
+export PORTAGE_BZIP2_COMMAND=${PORTAGE_BZIP2_COMMAND:-bzip2}
 
 # These two functions wrap sourcing and calling respectively.  At present they
 # perform a qa check to make sure eclasses and ebuilds and profiles don't mess
-# with shell opts (shopts).  Ebuilds/eclasses changing shopts should reset them 
+# with shell opts (shopts).  Ebuilds/eclasses changing shopts should reset them
 # when they are done.
 
 __qa_source() {
@@ -275,7 +275,7 @@ inherit() {
 		set +f
 
 		__qa_source "$location" || die "died sourcing $location in inherit()"
-		
+
 		#turn off glob expansion
 		set -f
 
@@ -290,7 +290,7 @@ inherit() {
 
 		[ "${B_IUSE+set}"     = set ] && IUSE="${B_IUSE}"
 		[ "${B_IUSE+set}"     = set ] || unset IUSE
-		
+
 		[ "${B_REQUIRED_USE+set}"     = set ] && REQUIRED_USE="${B_REQUIRED_USE}"
 		[ "${B_REQUIRED_USE+set}"     = set ] || unset REQUIRED_USE
 
@@ -372,42 +372,60 @@ __source_all_bashrcs() {
 		restore_IFS
 		for x in "${path_array[@]}" ; do
 			[ -f "$x/profile.bashrc" ] && __qa_source "$x/profile.bashrc"
+			[[ " ${FEATURES} " == *" per-profile-env "* ]] && \
+				__source_env_files "$x/env"
 		done
 	fi
 
-	if [ -r "${PORTAGE_BASHRC}" ] ; then
-		if [ "$PORTAGE_DEBUG" != "1" ] || [ "${-/x/}" != "$-" ]; then
-			source "${PORTAGE_BASHRC}"
-		else
-			set -x
-			source "${PORTAGE_BASHRC}"
-			set +x
-		fi
-	fi
+	# The user's bashrc is the ONLY non-portage bit of code
+	# that can change shopts without a QA violation.
+	__try_source "${PORTAGE_BASHRC}" "no_qa"
 
 	if [[ $EBUILD_PHASE != depend ]] ; then
-		# The user's bashrc is the ONLY non-portage bit of code that can
-		# change shopts without a QA violation.
-		for x in "${PM_EBUILD_HOOK_DIR}"/${CATEGORY}/{${PN},${PN}:${SLOT%/*},${P},${PF}}; do
-			if [ -r "${x}" ]; then
-				# If $- contains x, then tracing has already been enabled
-				# elsewhere for some reason. We preserve it's state so as
-				# not to interfere.
-				if [ "$PORTAGE_DEBUG" != "1" ] || [ "${-/x/}" != "$-" ]; then
-					source "${x}"
-				else
-					set -x
-					source "${x}"
-					set +x
-				fi
-			fi
-		done
+		__source_env_files "${PM_EBUILD_HOOK_DIR}" "no_qa"
 	fi
 
 	[ ! -z "${OCC}" ] && export CC="${OCC}"
 	[ ! -z "${OCXX}" ] && export CXX="${OCXX}"
 }
 
+# @FUNCTION: __source_env_files
+# @DESCRIPTION:
+# Source the files relevant to the current package from the given path.
+# If $2 is not 'no_qa', all the scripts are sourced with __qa_source.
+__source_env_files() {
+	for x in "${1}"/${CATEGORY}/{${PN},${PN}:${SLOT%/*},${P},${PF}}; do
+		__try_source "${x}" "${2}"
+	done
+}
+
+# @FUNCTION: __try_source
+# @DESCRIPTION:
+# If the path given as argument exists, source the file while preserving
+# $-.
+# If $2 is not 'no_qa', the file is sourced with __qa_source.
+__try_source() {
+	if [[ -r "$1" ]]; then
+		# If $- contains x, then tracing has already been enabled
+		# elsewhere for some reason. We preserve it's state so as
+		# not to interfere.
+		if [[ "$PORTAGE_DEBUG" != "1" ]] || [[ "${-/x/}" != "$-" ]]; then
+			if [[ "${2}" == "no_qa" ]]; then
+				source "${x}"
+			else
+				__qa_source "${x}"
+			fi
+		else
+			set -x
+			if [[ "${2}" == "no_qa" ]]; then
+				source "${x}"
+			else
+				__qa_source "${x}"
+			fi
+			set +x
+		fi
+	fi
+}
 # === === === === === === === === === === === === === === === === === ===
 # === === === === === functions end, main part begins === === === === ===
 # === === === === === === === === === === === === === === === === === ===
@@ -572,7 +590,7 @@ if ! has "$EBUILD_PHASE" clean cleanrm ; then
 		PDEPEND+="${PDEPEND:+ }${E_PDEPEND}"
 		HDEPEND+="${HDEPEND:+ }${E_HDEPEND}"
 		REQUIRED_USE+="${REQUIRED_USE:+ }${E_REQUIRED_USE}"
-		
+
 		unset ECLASS E_IUSE E_REQUIRED_USE E_DEPEND E_RDEPEND E_PDEPEND E_HDEPEND \
 			__INHERITED_QA_CACHE
 
diff --git a/pym/portage/const.py b/pym/portage/const.py
index aab6e8a..3043656 100644
--- a/pym/portage/const.py
+++ b/pym/portage/const.py
@@ -166,6 +166,7 @@ SUPPORTED_FEATURES       = frozenset([
 	"notitles",
 	"parallel-fetch",
 	"parallel-install",
+	"per-profile-env",
 	"prelink-checksums",
 	"preserve-libs",
 	"protect-owned",
diff --git a/pym/portage/package/ebuild/config.py b/pym/portage/package/ebuild/config.py
index f639e14..6c695fe 100644
--- a/pym/portage/package/ebuild/config.py
+++ b/pym/portage/package/ebuild/config.py
@@ -719,28 +719,38 @@ class config(object):
 					self._paccept_restrict.setdefault(k.cp, {})[k] = v
 
 				#package.env
-				penvdict = grabdict_package(os.path.join(
-					abs_user_config, "package.env"), recursive=1, allow_wildcard=True, \
-					allow_repo=True, verify_eapi=False)
-				v = penvdict.pop("*/*", None)
-				if v is not None:
-					global_wildcard_conf = {}
-					self._grab_pkg_env(v, global_wildcard_conf)
-					incrementals = self.incrementals
-					conf_configdict = self.configdict["conf"]
-					for k, v in global_wildcard_conf.items():
-						if k in incrementals:
-							if k in conf_configdict:
-								conf_configdict[k] = \
-									conf_configdict[k] + " " + v
+				envlocations = [v.location for v in profiles_complex]
+				envlocations.append(os.path.join(self['PORTAGE_CONFIGROOT'], USER_CONFIG_PATH))
+				penvdicts = [(grabdict_package(os.path.join(x, "package.env"),
+								recursive=1, allow_wildcard=True,
+								allow_repo=True, verify_eapi=False), x) \
+				 for x in envlocations]
+
+				for (penvdict, location) in penvdicts:
+					v = penvdict.pop("*/*", None)
+					if v is not None:
+						global_wildcard_conf = {}
+						v = [os.path.join(location, 'env', envname) \
+							for envname in v]
+						self._grab_pkg_env(v, global_wildcard_conf)
+						incrementals = self.incrementals
+						conf_configdict = self.configdict["conf"]
+						for k, v in global_wildcard_conf.items():
+							if k in incrementals:
+								if k in conf_configdict:
+									conf_configdict[k] = \
+										conf_configdict[k] + " " + v
+								else:
+									conf_configdict[k] = v
 							else:
 								conf_configdict[k] = v
-						else:
-							conf_configdict[k] = v
-						expand_map[k] = v
+								expand_map[k] = v
 
-				for k, v in penvdict.items():
-					self._penvdict.setdefault(k.cp, {})[k] = v
+					for k, v in penvdict.items():
+						penvfiles = [os.path.join(location, 'env', envname) \
+							for envname in v]
+						self._penvdict.setdefault(k.cp, {})\
+							.setdefault(k, []).extend(penvfiles)
 
 			#getting categories from an external file now
 			self.categories = [grabfile(os.path.join(x, "categories")) \
@@ -1692,18 +1702,15 @@ class config(object):
 		# setcpv triggers lazy instantiation of things like _use_manager.
 		_eapi_cache.clear()
 
-	def _grab_pkg_env(self, penv, container, protected_keys=None):
+	def _grab_pkg_env(self, penvfiles, container, protected_keys=None):
 		if protected_keys is None:
 			protected_keys = ()
-		abs_user_config = os.path.join(
-			self['PORTAGE_CONFIGROOT'], USER_CONFIG_PATH)
 		non_user_variables = self._non_user_variables
 		# Make a copy since we don't want per-package settings
 		# to pollute the global expand_map.
 		expand_map = self._expand_map.copy()
 		incrementals = self.incrementals
-		for envname in penv:
-			penvfile = os.path.join(abs_user_config, "env", envname)
+		for penvfile in penvfiles:
 			penvconfig = getconfig(penvfile, tolerant=self._tolerant,
 				allow_sourcing=True, expand=expand_map)
 			if penvconfig is None:
-- 
2.1.0.rc2.206.gedb03e5

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Zac Medico]

On 09/11/2014 04:18 AM, Bertrand Simonnet wrote:
> Hi guys,
> 
> I have been working on a patch to make package.env (and env/ files)
> stackable with the profiles.
> The main goal would be to have a mechanism to be able to define a
> per-package, per-profile environment in a simple way.
> 
> We can currently do this with profile.bashrc but we have to add some
> logic there to decide which package are relevant which already exist in
> ebuild.sh and pym/*.
> 
> I attached my current patch which contain two main modifications:
> - ebuild.sh walk the profile and source all scripts with path:
> ${profile_path}/env/${category}/${PN}
> - config.py imports package.env from all profiles and then from
> /etc/portage/package.env

It seems like duplication of work, to load the per-package env in
_grab_pkg_env on the python side, and to source it again on the bash
side. Why load the environments in both places? Note that the python
getconfig function only supports variable settings (no executable code),
which is very limited to what can be done when sourcing the files in bash.

> Note:
> * I gated the env/$CATEGORY/$PN scripts with a new feature: per-profile-env.
>    I couldn't do this with package.env as the scripts are parsed before
> we initialize self.features.
> * I am not particularly attach to both package.env and
> env/$CATEGORY/$PN. It might make more sense to have only
> env/$CATEGORY/$PN, especially considering the problem with package.env.
> 
> Do you have any suggestions on this patch ?

1) If you want both the python-side env loading and the bash-side env
loading, I would make those into separate features, since sourcing the
files in bash supports full bash syntax while the python-side getconfig
function does not.

2) When adding new global functions in ebuild.sh, please unset them
inside bin/save-ebuild-env.sh, when internal functions are stripped from
the environment.

3) You might want to look into using the repository's
metadata/layout.conf profile-formats setting, as an alternative to using
FEATURES to control profile-related behavior.

> Would you be willing to accept this patch once the few problems are
> sorted out ?

I can't speak for everyone, but I'm pretty sure we can work something out.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Bertrand Simonnet]

[-- Attachment #1.1: Type: text/plain, Size: 3020 bytes --]

Second patch containing only the ebuild.sh change.
Answers to suggestions inline.

On Thu, Sep 11, 2014 at 12:49 PM, Zac Medico <zmedico@gentoo.org> wrote:

> On 09/11/2014 04:18 AM, Bertrand Simonnet wrote:
> > Hi guys,
> >
> > I have been working on a patch to make package.env (and env/ files)
> > stackable with the profiles.
> > The main goal would be to have a mechanism to be able to define a
> > per-package, per-profile environment in a simple way.
> >
> > We can currently do this with profile.bashrc but we have to add some
> > logic there to decide which package are relevant which already exist in
> > ebuild.sh and pym/*.
> >
> > I attached my current patch which contain two main modifications:
> > - ebuild.sh walk the profile and source all scripts with path:
> > ${profile_path}/env/${category}/${PN}
> > - config.py imports package.env from all profiles and then from
> > /etc/portage/package.env
>
> It seems like duplication of work, to load the per-package env in
> _grab_pkg_env on the python side, and to source it again on the bash
> side. Why load the environments in both places? Note that the python
> getconfig function only supports variable settings (no executable code),
> which is very limited to what can be done when sourcing the files in bash.
>
> Right.
The python part was mostly useful from a reuse point of view (write few
scripts
and enable them for some packages only. As the python mechanism is less
powerful
than I thought, I removed it.


> > Note:
> > * I gated the env/$CATEGORY/$PN scripts with a new feature:
> per-profile-env.
> >    I couldn't do this with package.env as the scripts are parsed before
> > we initialize self.features.
> > * I am not particularly attach to both package.env and
> > env/$CATEGORY/$PN. It might make more sense to have only
> > env/$CATEGORY/$PN, especially considering the problem with package.env.
> >
> > Do you have any suggestions on this patch ?
>
> 1) If you want both the python-side env loading and the bash-side env
> loading, I would make those into separate features, since sourcing the
> files in bash supports full bash syntax while the python-side getconfig
> function does not.
>
Fixed: removed the python part completly

2) When adding new global functions in ebuild.sh, please unset them
> inside bin/save-ebuild-env.sh, when internal functions are stripped from
> the environment.
>
Done.


> 3) You might want to look into using the repository's
> metadata/layout.conf profile-formats setting, as an alternative to using
> FEATURES to control profile-related behavior.

I replaced the FEATURES gate by a profile-formats option.
I added some logic to find layout.conf for a given profile path. (find the
first
parent dir named profiles then check ../metadata/layout.conf)
Is there a corner case that I missed?

> Would you be willing to accept this patch once the few problems are
> > sorted out ?
>
> I can't speak for everyone, but I'm pretty sure we can work something out.
> --
> Thanks,
> Zac
>
>
Thanks,
Bertrand

[-- Attachment #1.2: Type: text/html, Size: 4469 bytes --]

[-- Attachment #2: 0001-Make-env-bash-scripts-profile-specific.patch --]
[-- Type: text/x-patch, Size: 6771 bytes --]

From 8a7e493d343bd986fa14e3c7148e76012e6f98d6 Mon Sep 17 00:00:00 2001
From: Bertrand SIMONNET <bsimonnet@chromium.org>
Date: Fri, 12 Sep 2014 05:07:20 -0700
Subject: [PATCH] Make env/ bash scripts profile specific

This generalize the /etc/portage/env mechanism to be stackable with profiles.
ebuild.sh will walk the profiles and sources scripts in env/ following the same
matching rules as for /etc/portage/env.

Change-Id: I16bcd75790213d2204f83d4aa6e8b910f8829b6e
---
 bin/ebuild.sh                    | 78 ++++++++++++++++++++++++----------------
 bin/isolated-functions.sh        | 23 ++++++++++++
 bin/save-ebuild-env.sh           |  1 +
 pym/portage/repository/config.py |  2 +-
 4 files changed, 73 insertions(+), 31 deletions(-)

diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index be044e0..87e44f4 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -67,11 +67,11 @@ unset BASH_ENV
 # earlier portage versions do not detect a version change in this case
 # (9999 to 9999) and therefore they try execute an incompatible version of
 # ebuild.sh during the upgrade.
-export PORTAGE_BZIP2_COMMAND=${PORTAGE_BZIP2_COMMAND:-bzip2} 
+export PORTAGE_BZIP2_COMMAND=${PORTAGE_BZIP2_COMMAND:-bzip2}
 
 # These two functions wrap sourcing and calling respectively.  At present they
 # perform a qa check to make sure eclasses and ebuilds and profiles don't mess
-# with shell opts (shopts).  Ebuilds/eclasses changing shopts should reset them 
+# with shell opts (shopts).  Ebuilds/eclasses changing shopts should reset them
 # when they are done.
 
 __qa_source() {
@@ -275,7 +275,7 @@ inherit() {
 		set +f
 
 		__qa_source "$location" || die "died sourcing $location in inherit()"
-		
+
 		#turn off glob expansion
 		set -f
 
@@ -290,7 +290,7 @@ inherit() {
 
 		[ "${B_IUSE+set}"     = set ] && IUSE="${B_IUSE}"
 		[ "${B_IUSE+set}"     = set ] || unset IUSE
-		
+
 		[ "${B_REQUIRED_USE+set}"     = set ] && REQUIRED_USE="${B_REQUIRED_USE}"
 		[ "${B_REQUIRED_USE+set}"     = set ] || unset REQUIRED_USE
 
@@ -372,42 +372,60 @@ __source_all_bashrcs() {
 		restore_IFS
 		for x in "${path_array[@]}" ; do
 			[ -f "$x/profile.bashrc" ] && __qa_source "$x/profile.bashrc"
+			__profile_env_enabled "${x}" && \
+				__source_env_files "$x/env"
 		done
 	fi
 
-	if [ -r "${PORTAGE_BASHRC}" ] ; then
-		if [ "$PORTAGE_DEBUG" != "1" ] || [ "${-/x/}" != "$-" ]; then
-			source "${PORTAGE_BASHRC}"
-		else
-			set -x
-			source "${PORTAGE_BASHRC}"
-			set +x
-		fi
-	fi
+	# The user's bashrc is the ONLY non-portage bit of code
+	# that can change shopts without a QA violation.
+	__try_source "${PORTAGE_BASHRC}" "no_qa"
 
 	if [[ $EBUILD_PHASE != depend ]] ; then
-		# The user's bashrc is the ONLY non-portage bit of code that can
-		# change shopts without a QA violation.
-		for x in "${PM_EBUILD_HOOK_DIR}"/${CATEGORY}/{${PN},${PN}:${SLOT%/*},${P},${PF}}; do
-			if [ -r "${x}" ]; then
-				# If $- contains x, then tracing has already been enabled
-				# elsewhere for some reason. We preserve it's state so as
-				# not to interfere.
-				if [ "$PORTAGE_DEBUG" != "1" ] || [ "${-/x/}" != "$-" ]; then
-					source "${x}"
-				else
-					set -x
-					source "${x}"
-					set +x
-				fi
-			fi
-		done
+		__source_env_files "${PM_EBUILD_HOOK_DIR}" "no_qa"
 	fi
 
 	[ ! -z "${OCC}" ] && export CC="${OCC}"
 	[ ! -z "${OCXX}" ] && export CXX="${OCXX}"
 }
 
+# @FUNCTION: __source_env_files
+# @DESCRIPTION:
+# Source the files relevant to the current package from the given path.
+# If $2 is not 'no_qa', all the scripts are sourced with __qa_source.
+__source_env_files() {
+	for x in "${1}"/${CATEGORY}/{${PN},${PN}:${SLOT%/*},${P},${PF}}; do
+		__try_source "${x}" "${2}"
+	done
+}
+
+# @FUNCTION: __try_source
+# @DESCRIPTION:
+# If the path given as argument exists, source the file while preserving
+# $-.
+# If $2 is not 'no_qa', the file is sourced with __qa_source.
+__try_source() {
+	if [[ -r "$1" ]]; then
+		# If $- contains x, then tracing has already been enabled
+		# elsewhere for some reason. We preserve it's state so as
+		# not to interfere.
+		if [[ "$PORTAGE_DEBUG" != "1" ]] || [[ "${-/x/}" != "$-" ]]; then
+			if [[ "${2}" == "no_qa" ]]; then
+				source "${x}"
+			else
+				__qa_source "${x}"
+			fi
+		else
+			set -x
+			if [[ "${2}" == "no_qa" ]]; then
+				source "${x}"
+			else
+				__qa_source "${x}"
+			fi
+			set +x
+		fi
+	fi
+}
 # === === === === === === === === === === === === === === === === === ===
 # === === === === === functions end, main part begins === === === === ===
 # === === === === === === === === === === === === === === === === === ===
@@ -572,7 +590,7 @@ if ! has "$EBUILD_PHASE" clean cleanrm ; then
 		PDEPEND+="${PDEPEND:+ }${E_PDEPEND}"
 		HDEPEND+="${HDEPEND:+ }${E_HDEPEND}"
 		REQUIRED_USE+="${REQUIRED_USE:+ }${E_REQUIRED_USE}"
-		
+
 		unset ECLASS E_IUSE E_REQUIRED_USE E_DEPEND E_RDEPEND E_PDEPEND E_HDEPEND \
 			__INHERITED_QA_CACHE
 
diff --git a/bin/isolated-functions.sh b/bin/isolated-functions.sh
index a22af57..17a4c61 100644
--- a/bin/isolated-functions.sh
+++ b/bin/isolated-functions.sh
@@ -482,4 +482,27 @@ __repo_attr() {
 	return ${exit_status}
 }
 
+# @FUNCTION: __profile_env_enabled
+# @DESCRIPTION:
+# Return "yes" if the layout.conf file of a given profile $1 has profile-env
+# enabled.
+__profile_env_enabled() {
+	local current_dir="$1"
+	while ! [[ "$(basename "${current_dir}")" == "profiles" ]]; do
+		if [[ "${current_dir}" == "/" ]]; then
+			return 1
+		fi
+		current_dir="$(dirname "${current_dir}")"
+	done
+	local layout="$(dirname "${current_dir}")/metadata/layout.conf"
+	if [[ -f "${layout}" ]]; then
+		while read line; do
+			if [[ "${line}" =~ ^profile-format.*profile-env ]]; then
+				return 0
+			fi
+		done < "${layout}"
+	fi
+	return 1
+}
+
 true
diff --git a/bin/save-ebuild-env.sh b/bin/save-ebuild-env.sh
index 98cff83..8e0cac5 100644
--- a/bin/save-ebuild-env.sh
+++ b/bin/save-ebuild-env.sh
@@ -75,6 +75,7 @@ __save_ebuild_env() {
 		__ebuild_main __ebuild_phase __ebuild_phase_with_hooks \
 		__ebuild_arg_to_phase __ebuild_phase_funcs default \
 		__unpack_tar __unset_colors \
+		__profile_env_enabled __source_env_files __try_source \
 		${QA_INTERCEPTORS}
 
 	___eapi_has_usex && unset -f usex
diff --git a/pym/portage/repository/config.py b/pym/portage/repository/config.py
index 5e0d055..ef8054e 100644
--- a/pym/portage/repository/config.py
+++ b/pym/portage/repository/config.py
@@ -40,7 +40,7 @@ if sys.hexversion >= 0x3000000:
 _invalid_path_char_re = re.compile(r'[^a-zA-Z0-9._\-+:/]')
 
 _valid_profile_formats = frozenset(
-	['pms', 'portage-1', 'portage-2'])
+	['pms', 'portage-1', 'portage-2', 'profile-env'])
 
 _portage1_profiles_allow_directories = frozenset(
 	["portage-1-compat", "portage-1", 'portage-2'])
-- 
2.1.0.rc2.206.gedb03e5

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Zac Medico]

On 09/15/2014 11:42 AM, Bertrand Simonnet wrote:
> I replaced the FEATURES gate by a profile-formats option.
> I added some logic to find layout.conf for a given profile path. (find
> the first
> parent dir named profiles then check ../metadata/layout.conf)
> Is there a corner case that I missed?

I all looks good to me, except that I don't like the way that
__profile_env_enabled parses the layout.conf for each directory, since
it's terribly inefficient. It would be much better to parse the
layout.conf data on the python side, and pass it to bash as an
environment variable. For example, the PORTAGE_ECLASS_LOCATIONS variable
is generated in python such that eval can be used to convert it to an
array. We can add a similar array variable called
PORTAGE_PROFILE_ATTRIBUTES, and each element of the array will
correspond to the profile path at the same index in the path_array variable.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Bertrand Simonnet]

[-- Attachment #1.1: Type: text/plain, Size: 1171 bytes --]

I moved the profile attributes detection logic into the python side as
suggested (much cleaner).

The updated patch is attached.

Thanks,
Bertrand

On Mon, Sep 15, 2014 at 1:40 PM, Zac Medico <zmedico@gentoo.org> wrote:

> On 09/15/2014 11:42 AM, Bertrand Simonnet wrote:
> > I replaced the FEATURES gate by a profile-formats option.
> > I added some logic to find layout.conf for a given profile path. (find
> > the first
> > parent dir named profiles then check ../metadata/layout.conf)
> > Is there a corner case that I missed?
>
> I all looks good to me, except that I don't like the way that
> __profile_env_enabled parses the layout.conf for each directory, since
> it's terribly inefficient. It would be much better to parse the
> layout.conf data on the python side, and pass it to bash as an
> environment variable. For example, the PORTAGE_ECLASS_LOCATIONS variable
> is generated in python such that eval can be used to convert it to an
> array. We can add a similar array variable called
> PORTAGE_PROFILE_ATTRIBUTES, and each element of the array will
> correspond to the profile path at the same index in the path_array
> variable.
> --
> Thanks,
> Zac
>
>

[-- Attachment #1.2: Type: text/html, Size: 1686 bytes --]

[-- Attachment #2: 0001-Make-env-bash-scripts-profile-specific.patch --]
[-- Type: text/x-patch, Size: 9401 bytes --]

From 921290494664b356e7bb82f5a5a86b40287a49d2 Mon Sep 17 00:00:00 2001
From: Bertrand SIMONNET <bsimonnet@chromium.org>
Date: Fri, 12 Sep 2014 05:07:20 -0700
Subject: [PATCH] Make env/ bash scripts profile specific

This generalize the /etc/portage/env mechanism to be stackable with profiles.
ebuild.sh will walk the profiles and sources scripts in env/ following the same
matching rules as for /etc/portage/env.

Change-Id: I16bcd75790213d2204f83d4aa6e8b910f8829b6e
---
 bin/ebuild.sh                                      | 86 ++++++++++++++--------
 bin/save-ebuild-env.sh                             |  1 +
 .../package/ebuild/_config/special_env_vars.py     |  2 +-
 pym/portage/package/ebuild/config.py               | 20 +++++
 pym/portage/package/ebuild/doebuild.py             |  1 +
 pym/portage/repository/config.py                   |  2 +-
 6 files changed, 78 insertions(+), 34 deletions(-)

diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index be044e0..26b9190 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -67,11 +67,11 @@ unset BASH_ENV
 # earlier portage versions do not detect a version change in this case
 # (9999 to 9999) and therefore they try execute an incompatible version of
 # ebuild.sh during the upgrade.
-export PORTAGE_BZIP2_COMMAND=${PORTAGE_BZIP2_COMMAND:-bzip2} 
+export PORTAGE_BZIP2_COMMAND=${PORTAGE_BZIP2_COMMAND:-bzip2}
 
 # These two functions wrap sourcing and calling respectively.  At present they
 # perform a qa check to make sure eclasses and ebuilds and profiles don't mess
-# with shell opts (shopts).  Ebuilds/eclasses changing shopts should reset them 
+# with shell opts (shopts).  Ebuilds/eclasses changing shopts should reset them
 # when they are done.
 
 __qa_source() {
@@ -275,7 +275,7 @@ inherit() {
 		set +f
 
 		__qa_source "$location" || die "died sourcing $location in inherit()"
-		
+
 		#turn off glob expansion
 		set -f
 
@@ -290,7 +290,7 @@ inherit() {
 
 		[ "${B_IUSE+set}"     = set ] && IUSE="${B_IUSE}"
 		[ "${B_IUSE+set}"     = set ] || unset IUSE
-		
+
 		[ "${B_REQUIRED_USE+set}"     = set ] && REQUIRED_USE="${B_REQUIRED_USE}"
 		[ "${B_REQUIRED_USE+set}"     = set ] || unset REQUIRED_USE
 
@@ -369,45 +369,67 @@ __source_all_bashrcs() {
 		save_IFS
 		IFS=$'\n'
 		local path_array=($PROFILE_PATHS)
+		local profile_attributes=($PORTAGE_PROFILE_ATTRIBUTES)
+		local profile_path
 		restore_IFS
-		for x in "${path_array[@]}" ; do
-			[ -f "$x/profile.bashrc" ] && __qa_source "$x/profile.bashrc"
+		for i in "${!path_array[@]}" ; do
+			profile_path="${path_array[$i]}"
+			[ -f "${profile_path}/profile.bashrc" ] && \
+				__qa_source "${profile_path}/profile.bashrc"
+			[[ "${profile_attributes[$i]}" == *profile-env* ]] && \
+				__source_env_files "${profile_path}/env"
 		done
 	fi
 
-	if [ -r "${PORTAGE_BASHRC}" ] ; then
-		if [ "$PORTAGE_DEBUG" != "1" ] || [ "${-/x/}" != "$-" ]; then
-			source "${PORTAGE_BASHRC}"
-		else
-			set -x
-			source "${PORTAGE_BASHRC}"
-			set +x
-		fi
-	fi
+	# The user's bashrc is the ONLY non-portage bit of code
+	# that can change shopts without a QA violation.
+	__try_source "${PORTAGE_BASHRC}" "no_qa"
 
 	if [[ $EBUILD_PHASE != depend ]] ; then
-		# The user's bashrc is the ONLY non-portage bit of code that can
-		# change shopts without a QA violation.
-		for x in "${PM_EBUILD_HOOK_DIR}"/${CATEGORY}/{${PN},${PN}:${SLOT%/*},${P},${PF}}; do
-			if [ -r "${x}" ]; then
-				# If $- contains x, then tracing has already been enabled
-				# elsewhere for some reason. We preserve it's state so as
-				# not to interfere.
-				if [ "$PORTAGE_DEBUG" != "1" ] || [ "${-/x/}" != "$-" ]; then
-					source "${x}"
-				else
-					set -x
-					source "${x}"
-					set +x
-				fi
-			fi
-		done
+		__source_env_files "${PM_EBUILD_HOOK_DIR}" "no_qa"
 	fi
 
 	[ ! -z "${OCC}" ] && export CC="${OCC}"
 	[ ! -z "${OCXX}" ] && export CXX="${OCXX}"
 }
 
+# @FUNCTION: __source_env_files
+# @DESCRIPTION:
+# Source the files relevant to the current package from the given path.
+# If $2 is not 'no_qa', all the scripts are sourced with __qa_source.
+__source_env_files() {
+	for x in "${1}"/${CATEGORY}/{${PN},${PN}:${SLOT%/*},${P},${PF}}; do
+		__try_source "${x}" "${2}"
+	done
+}
+
+# @FUNCTION: __try_source
+# @DESCRIPTION:
+# If the path given as argument exists, source the file while preserving
+# $-.
+# If $2 is not 'no_qa', the file is sourced with __qa_source.
+__try_source() {
+	if [[ -r "$1" ]]; then
+		# If $- contains x, then tracing has already been enabled
+		# elsewhere for some reason. We preserve it's state so as
+		# not to interfere.
+		if [[ "$PORTAGE_DEBUG" != "1" ]] || [[ "${-/x/}" != "$-" ]]; then
+			if [[ "${2}" == "no_qa" ]]; then
+				source "${x}"
+			else
+				__qa_source "${x}"
+			fi
+		else
+			set -x
+			if [[ "${2}" == "no_qa" ]]; then
+				source "${x}"
+			else
+				__qa_source "${x}"
+			fi
+			set +x
+		fi
+	fi
+}
 # === === === === === === === === === === === === === === === === === ===
 # === === === === === functions end, main part begins === === === === ===
 # === === === === === === === === === === === === === === === === === ===
@@ -572,7 +594,7 @@ if ! has "$EBUILD_PHASE" clean cleanrm ; then
 		PDEPEND+="${PDEPEND:+ }${E_PDEPEND}"
 		HDEPEND+="${HDEPEND:+ }${E_HDEPEND}"
 		REQUIRED_USE+="${REQUIRED_USE:+ }${E_REQUIRED_USE}"
-		
+
 		unset ECLASS E_IUSE E_REQUIRED_USE E_DEPEND E_RDEPEND E_PDEPEND E_HDEPEND \
 			__INHERITED_QA_CACHE
 
diff --git a/bin/save-ebuild-env.sh b/bin/save-ebuild-env.sh
index 98cff83..f114c48 100644
--- a/bin/save-ebuild-env.sh
+++ b/bin/save-ebuild-env.sh
@@ -75,6 +75,7 @@ __save_ebuild_env() {
 		__ebuild_main __ebuild_phase __ebuild_phase_with_hooks \
 		__ebuild_arg_to_phase __ebuild_phase_funcs default \
 		__unpack_tar __unset_colors \
+		__source_env_files __try_source \
 		${QA_INTERCEPTORS}
 
 	___eapi_has_usex && unset -f usex
diff --git a/pym/portage/package/ebuild/_config/special_env_vars.py b/pym/portage/package/ebuild/_config/special_env_vars.py
index 74fedd6..b929042 100644
--- a/pym/portage/package/ebuild/_config/special_env_vars.py
+++ b/pym/portage/package/ebuild/_config/special_env_vars.py
@@ -68,7 +68,7 @@ environ_whitelist += [
 	"PORTAGE_INST_GID", "PORTAGE_INST_UID",
 	"PORTAGE_IPC_DAEMON", "PORTAGE_IUSE", "PORTAGE_ECLASS_LOCATIONS",
 	"PORTAGE_LOG_FILE", "PORTAGE_OVERRIDE_EPREFIX", "PORTAGE_PIPE_FD",
-	"PORTAGE_PYM_PATH", "PORTAGE_PYTHON",
+	"PORTAGE_PROFILE_ATTRIBUTES", "PORTAGE_PYM_PATH", "PORTAGE_PYTHON",
 	"PORTAGE_PYTHONPATH", "PORTAGE_QUIET",
 	"PORTAGE_REPO_NAME", "PORTAGE_REPOSITORIES", "PORTAGE_RESTRICT",
 	"PORTAGE_SIGPIPE_STATUS",
diff --git a/pym/portage/package/ebuild/config.py b/pym/portage/package/ebuild/config.py
index f639e14..38b1df3 100644
--- a/pym/portage/package/ebuild/config.py
+++ b/pym/portage/package/ebuild/config.py
@@ -118,6 +118,24 @@ def _lazy_iuse_regex(iuse_implicit):
 	regex = regex.replace("\\.\\*", ".*")
 	return regex
 
+def _get_profile_attributes(profile_path):
+	current_path = profile_path
+	while os.path.basename(current_path) != "profiles":
+		if current_path == os.path.dirname(current_path):
+			return ""
+		current_path = os.path.dirname(current_path)
+	layout_path = os.path.join(current_path,
+			 "..",
+			 "metadata",
+			 "layout.conf")
+	if not os.path.isfile(layout_path):
+		return ""
+	with open(layout_path) as f:
+		for line in f:
+			if line.startswith("profile-formats"):
+				return line.split("=")[1].strip()
+	return ""
+
 class _iuse_implicit_match_cache(object):
 
 	def __init__(self, settings):
@@ -247,6 +265,7 @@ class config(object):
 			self.module_priority = clone.module_priority
 			self.profile_path = clone.profile_path
 			self.profiles = clone.profiles
+			self.profile_formats = clone.profile_formats
 			self.packages = clone.packages
 			self.repositories = clone.repositories
 			self.unpack_dependencies = clone.unpack_dependencies
@@ -554,6 +573,7 @@ class config(object):
 
 			profiles_complex = locations_manager.profiles_complex
 			self.profiles = locations_manager.profiles
+			self.profile_formats = [_get_profile_attributes(x) for x in self.profiles]
 			self.profile_path = locations_manager.profile_path
 			self.user_profile_dir = locations_manager.user_profile_dir
 
diff --git a/pym/portage/package/ebuild/doebuild.py b/pym/portage/package/ebuild/doebuild.py
index 01707ae..ba4e887 100644
--- a/pym/portage/package/ebuild/doebuild.py
+++ b/pym/portage/package/ebuild/doebuild.py
@@ -336,6 +336,7 @@ def doebuild_environment(myebuild, mydo, myroot=None, settings=None,
 	mysettings["SANDBOX_LOG"] = mycpv.replace("/", "_-_")
 
 	mysettings["PROFILE_PATHS"] = "\n".join(mysettings.profiles)
+	mysettings["PORTAGE_PROFILE_ATTRIBUTES"] = "\n".join(mysettings.profile_formats)
 	mysettings["P"]  = mysplit[0]+"-"+mysplit[1]
 	mysettings["PN"] = mysplit[0]
 	mysettings["PV"] = mysplit[1]
diff --git a/pym/portage/repository/config.py b/pym/portage/repository/config.py
index 5e0d055..ef8054e 100644
--- a/pym/portage/repository/config.py
+++ b/pym/portage/repository/config.py
@@ -40,7 +40,7 @@ if sys.hexversion >= 0x3000000:
 _invalid_path_char_re = re.compile(r'[^a-zA-Z0-9._\-+:/]')
 
 _valid_profile_formats = frozenset(
-	['pms', 'portage-1', 'portage-2'])
+	['pms', 'portage-1', 'portage-2', 'profile-env'])
 
 _portage1_profiles_allow_directories = frozenset(
 	["portage-1-compat", "portage-1", 'portage-2'])
-- 
2.1.0.rc2.206.gedb03e5

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Zac Medico]

On 09/16/2014 11:17 AM, Bertrand Simonnet wrote:
> I moved the profile attributes detection logic into the python side as
> suggested (much cleaner).

Thanks, that's better. I've got a couple more issues though:

1) Like global functions, global variables should also be unset in
__save_ebuild_env. So, we should unset PORTAGE_PROFILE_ATTRIBUTES there.

2) Instead of having _get_profile_attributes read the layout.conf files
directly, it would be nicer if we could integrated it with the
LocationsManager layout.conf parsing. For example, see the
intersecting_repos code inside the _addProfile method.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Bertrand Simonnet]

[-- Attachment #1.1: Type: text/plain, Size: 757 bytes --]

On Tue, Sep 16, 2014 at 12:17 PM, Zac Medico <zmedico@gentoo.org> wrote:

> On 09/16/2014 11:17 AM, Bertrand Simonnet wrote:
> > I moved the profile attributes detection logic into the python side as
> > suggested (much cleaner).
>
> Thanks, that's better. I've got a couple more issues though:
>
> 1) Like global functions, global variables should also be unset in
> __save_ebuild_env. So, we should unset PORTAGE_PROFILE_ATTRIBUTES there.
>
Done

2) Instead of having _get_profile_attributes read the layout.conf files
> directly, it would be nicer if we could integrated it with the
> LocationsManager layout.conf parsing. For example, see the
> intersecting_repos code inside the _addProfile method.
>
This is much better. :)


> --
> Thanks,
> Zac
>
>

[-- Attachment #1.2: Type: text/html, Size: 1473 bytes --]

[-- Attachment #2: 0001-Make-env-bash-scripts-profile-specific.patch --]
[-- Type: text/x-patch, Size: 11193 bytes --]

From 4bf1ee98bb97136e3f6f2182e205aed7ca597640 Mon Sep 17 00:00:00 2001
From: Bertrand SIMONNET <bsimonnet@chromium.org>
Date: Fri, 12 Sep 2014 05:07:20 -0700
Subject: [PATCH] Make env/ bash scripts profile specific

This generalize the /etc/portage/env mechanism to be stackable with profiles.
ebuild.sh will walk the profiles and sources scripts in env/ following the same
matching rules as for /etc/portage/env.

Change-Id: I16bcd75790213d2204f83d4aa6e8b910f8829b6e
---
 bin/ebuild.sh                                      | 86 ++++++++++++++--------
 bin/save-ebuild-env.sh                             |  3 +-
 .../package/ebuild/_config/LocationsManager.py     |  6 ++
 .../package/ebuild/_config/special_env_vars.py     |  2 +-
 pym/portage/package/ebuild/config.py               |  2 +
 pym/portage/package/ebuild/doebuild.py             |  2 +
 pym/portage/repository/config.py                   |  2 +-
 7 files changed, 68 insertions(+), 35 deletions(-)

diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index be044e0..26b9190 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -67,11 +67,11 @@ unset BASH_ENV
 # earlier portage versions do not detect a version change in this case
 # (9999 to 9999) and therefore they try execute an incompatible version of
 # ebuild.sh during the upgrade.
-export PORTAGE_BZIP2_COMMAND=${PORTAGE_BZIP2_COMMAND:-bzip2} 
+export PORTAGE_BZIP2_COMMAND=${PORTAGE_BZIP2_COMMAND:-bzip2}
 
 # These two functions wrap sourcing and calling respectively.  At present they
 # perform a qa check to make sure eclasses and ebuilds and profiles don't mess
-# with shell opts (shopts).  Ebuilds/eclasses changing shopts should reset them 
+# with shell opts (shopts).  Ebuilds/eclasses changing shopts should reset them
 # when they are done.
 
 __qa_source() {
@@ -275,7 +275,7 @@ inherit() {
 		set +f
 
 		__qa_source "$location" || die "died sourcing $location in inherit()"
-		
+
 		#turn off glob expansion
 		set -f
 
@@ -290,7 +290,7 @@ inherit() {
 
 		[ "${B_IUSE+set}"     = set ] && IUSE="${B_IUSE}"
 		[ "${B_IUSE+set}"     = set ] || unset IUSE
-		
+
 		[ "${B_REQUIRED_USE+set}"     = set ] && REQUIRED_USE="${B_REQUIRED_USE}"
 		[ "${B_REQUIRED_USE+set}"     = set ] || unset REQUIRED_USE
 
@@ -369,45 +369,67 @@ __source_all_bashrcs() {
 		save_IFS
 		IFS=$'\n'
 		local path_array=($PROFILE_PATHS)
+		local profile_attributes=($PORTAGE_PROFILE_ATTRIBUTES)
+		local profile_path
 		restore_IFS
-		for x in "${path_array[@]}" ; do
-			[ -f "$x/profile.bashrc" ] && __qa_source "$x/profile.bashrc"
+		for i in "${!path_array[@]}" ; do
+			profile_path="${path_array[$i]}"
+			[ -f "${profile_path}/profile.bashrc" ] && \
+				__qa_source "${profile_path}/profile.bashrc"
+			[[ "${profile_attributes[$i]}" == *profile-env* ]] && \
+				__source_env_files "${profile_path}/env"
 		done
 	fi
 
-	if [ -r "${PORTAGE_BASHRC}" ] ; then
-		if [ "$PORTAGE_DEBUG" != "1" ] || [ "${-/x/}" != "$-" ]; then
-			source "${PORTAGE_BASHRC}"
-		else
-			set -x
-			source "${PORTAGE_BASHRC}"
-			set +x
-		fi
-	fi
+	# The user's bashrc is the ONLY non-portage bit of code
+	# that can change shopts without a QA violation.
+	__try_source "${PORTAGE_BASHRC}" "no_qa"
 
 	if [[ $EBUILD_PHASE != depend ]] ; then
-		# The user's bashrc is the ONLY non-portage bit of code that can
-		# change shopts without a QA violation.
-		for x in "${PM_EBUILD_HOOK_DIR}"/${CATEGORY}/{${PN},${PN}:${SLOT%/*},${P},${PF}}; do
-			if [ -r "${x}" ]; then
-				# If $- contains x, then tracing has already been enabled
-				# elsewhere for some reason. We preserve it's state so as
-				# not to interfere.
-				if [ "$PORTAGE_DEBUG" != "1" ] || [ "${-/x/}" != "$-" ]; then
-					source "${x}"
-				else
-					set -x
-					source "${x}"
-					set +x
-				fi
-			fi
-		done
+		__source_env_files "${PM_EBUILD_HOOK_DIR}" "no_qa"
 	fi
 
 	[ ! -z "${OCC}" ] && export CC="${OCC}"
 	[ ! -z "${OCXX}" ] && export CXX="${OCXX}"
 }
 
+# @FUNCTION: __source_env_files
+# @DESCRIPTION:
+# Source the files relevant to the current package from the given path.
+# If $2 is not 'no_qa', all the scripts are sourced with __qa_source.
+__source_env_files() {
+	for x in "${1}"/${CATEGORY}/{${PN},${PN}:${SLOT%/*},${P},${PF}}; do
+		__try_source "${x}" "${2}"
+	done
+}
+
+# @FUNCTION: __try_source
+# @DESCRIPTION:
+# If the path given as argument exists, source the file while preserving
+# $-.
+# If $2 is not 'no_qa', the file is sourced with __qa_source.
+__try_source() {
+	if [[ -r "$1" ]]; then
+		# If $- contains x, then tracing has already been enabled
+		# elsewhere for some reason. We preserve it's state so as
+		# not to interfere.
+		if [[ "$PORTAGE_DEBUG" != "1" ]] || [[ "${-/x/}" != "$-" ]]; then
+			if [[ "${2}" == "no_qa" ]]; then
+				source "${x}"
+			else
+				__qa_source "${x}"
+			fi
+		else
+			set -x
+			if [[ "${2}" == "no_qa" ]]; then
+				source "${x}"
+			else
+				__qa_source "${x}"
+			fi
+			set +x
+		fi
+	fi
+}
 # === === === === === === === === === === === === === === === === === ===
 # === === === === === functions end, main part begins === === === === ===
 # === === === === === === === === === === === === === === === === === ===
@@ -572,7 +594,7 @@ if ! has "$EBUILD_PHASE" clean cleanrm ; then
 		PDEPEND+="${PDEPEND:+ }${E_PDEPEND}"
 		HDEPEND+="${HDEPEND:+ }${E_HDEPEND}"
 		REQUIRED_USE+="${REQUIRED_USE:+ }${E_REQUIRED_USE}"
-		
+
 		unset ECLASS E_IUSE E_REQUIRED_USE E_DEPEND E_RDEPEND E_PDEPEND E_HDEPEND \
 			__INHERITED_QA_CACHE
 
diff --git a/bin/save-ebuild-env.sh b/bin/save-ebuild-env.sh
index 98cff83..326410b 100644
--- a/bin/save-ebuild-env.sh
+++ b/bin/save-ebuild-env.sh
@@ -75,6 +75,7 @@ __save_ebuild_env() {
 		__ebuild_main __ebuild_phase __ebuild_phase_with_hooks \
 		__ebuild_arg_to_phase __ebuild_phase_funcs default \
 		__unpack_tar __unset_colors \
+		__source_env_files __try_source \
 		${QA_INTERCEPTORS}
 
 	___eapi_has_usex && unset -f usex
@@ -101,7 +102,7 @@ __save_ebuild_env() {
 		PORTAGE_DOHTML_UNWARNED_SKIPPED_EXTENSIONS \
 		PORTAGE_DOHTML_UNWARNED_SKIPPED_FILES \
 		PORTAGE_DOHTML_WARN_ON_SKIPPED_FILES \
-		PORTAGE_NONFATAL PORTAGE_QUIET \
+		PORTAGE_NONFATAL PORTAGE_PROFILE_ATTRIBUTES PORTAGE_QUIET \
 		PORTAGE_SANDBOX_DENY PORTAGE_SANDBOX_PREDICT \
 		PORTAGE_SANDBOX_READ PORTAGE_SANDBOX_WRITE PREROOTPATH \
 		QA_INTERCEPTORS \
diff --git a/pym/portage/package/ebuild/_config/LocationsManager.py b/pym/portage/package/ebuild/_config/LocationsManager.py
index 4427f1d..345679d 100644
--- a/pym/portage/package/ebuild/_config/LocationsManager.py
+++ b/pym/portage/package/ebuild/_config/LocationsManager.py
@@ -114,6 +114,7 @@ class LocationsManager(object):
 		# The symlink might not exist or might not be a symlink.
 		self.profiles = []
 		self.profiles_complex = []
+		self.profile_formats = []
 		if self.profile_path:
 			try:
 				self._addProfile(os.path.realpath(self.profile_path),
@@ -124,6 +125,7 @@ class LocationsManager(object):
 					writemsg("!!! ParseError: %s\n" % str(e), noiselevel=-1)
 				self.profiles = []
 				self.profiles_complex = []
+				self.profile_formats = []
 
 		if self._user_config and self.profiles:
 			custom_prof = os.path.join(
@@ -137,6 +139,7 @@ class LocationsManager(object):
 
 		self.profiles = tuple(self.profiles)
 		self.profiles_complex = tuple(self.profiles_complex)
+		self.profile_formats = tuple(self.profile_formats)
 
 	def _check_var_directory(self, varname, var):
 		if not isdir_raise_eaccess(var):
@@ -151,6 +154,7 @@ class LocationsManager(object):
 		allow_parent_colon = True
 		repo_loc = None
 		compat_mode = False
+		current_formats = ()
 
 		eapi_file = os.path.join(currentPath, "eapi")
 		eapi = "0"
@@ -183,6 +187,7 @@ class LocationsManager(object):
 				layout_data['profile-formats'] == ('portage-1-compat',)
 			allow_parent_colon = any(x in _allow_parent_colon
 				for x in layout_data['profile-formats'])
+			current_formats = tuple(layout_data['profile-formats'])
 
 		if compat_mode:
 			offenders = _PORTAGE1_DIRECTORIES.intersection(os.listdir(currentPath))
@@ -234,6 +239,7 @@ class LocationsManager(object):
 		self.profiles.append(currentPath)
 		self.profiles_complex.append(
 			_profile_node(currentPath, allow_directories, False))
+		self.profile_formats.append(current_formats)
 
 	def _expand_parent_colon(self, parentsFile, parentPath,
 		repo_loc, repositories):
diff --git a/pym/portage/package/ebuild/_config/special_env_vars.py b/pym/portage/package/ebuild/_config/special_env_vars.py
index 74fedd6..b929042 100644
--- a/pym/portage/package/ebuild/_config/special_env_vars.py
+++ b/pym/portage/package/ebuild/_config/special_env_vars.py
@@ -68,7 +68,7 @@ environ_whitelist += [
 	"PORTAGE_INST_GID", "PORTAGE_INST_UID",
 	"PORTAGE_IPC_DAEMON", "PORTAGE_IUSE", "PORTAGE_ECLASS_LOCATIONS",
 	"PORTAGE_LOG_FILE", "PORTAGE_OVERRIDE_EPREFIX", "PORTAGE_PIPE_FD",
-	"PORTAGE_PYM_PATH", "PORTAGE_PYTHON",
+	"PORTAGE_PROFILE_ATTRIBUTES", "PORTAGE_PYM_PATH", "PORTAGE_PYTHON",
 	"PORTAGE_PYTHONPATH", "PORTAGE_QUIET",
 	"PORTAGE_REPO_NAME", "PORTAGE_REPOSITORIES", "PORTAGE_RESTRICT",
 	"PORTAGE_SIGPIPE_STATUS",
diff --git a/pym/portage/package/ebuild/config.py b/pym/portage/package/ebuild/config.py
index f639e14..37e71f9 100644
--- a/pym/portage/package/ebuild/config.py
+++ b/pym/portage/package/ebuild/config.py
@@ -247,6 +247,7 @@ class config(object):
 			self.module_priority = clone.module_priority
 			self.profile_path = clone.profile_path
 			self.profiles = clone.profiles
+			self.profile_formats = clone.profile_formats
 			self.packages = clone.packages
 			self.repositories = clone.repositories
 			self.unpack_dependencies = clone.unpack_dependencies
@@ -554,6 +555,7 @@ class config(object):
 
 			profiles_complex = locations_manager.profiles_complex
 			self.profiles = locations_manager.profiles
+			self.profile_formats = locations_manager.profile_formats
 			self.profile_path = locations_manager.profile_path
 			self.user_profile_dir = locations_manager.user_profile_dir
 
diff --git a/pym/portage/package/ebuild/doebuild.py b/pym/portage/package/ebuild/doebuild.py
index 01707ae..0b8c3fa 100644
--- a/pym/portage/package/ebuild/doebuild.py
+++ b/pym/portage/package/ebuild/doebuild.py
@@ -336,6 +336,8 @@ def doebuild_environment(myebuild, mydo, myroot=None, settings=None,
 	mysettings["SANDBOX_LOG"] = mycpv.replace("/", "_-_")
 
 	mysettings["PROFILE_PATHS"] = "\n".join(mysettings.profiles)
+	mysettings["PORTAGE_PROFILE_ATTRIBUTES"] = \
+		"\n".join([" ".join(formats) for formats in mysettings.profile_formats])
 	mysettings["P"]  = mysplit[0]+"-"+mysplit[1]
 	mysettings["PN"] = mysplit[0]
 	mysettings["PV"] = mysplit[1]
diff --git a/pym/portage/repository/config.py b/pym/portage/repository/config.py
index 5e0d055..ef8054e 100644
--- a/pym/portage/repository/config.py
+++ b/pym/portage/repository/config.py
@@ -40,7 +40,7 @@ if sys.hexversion >= 0x3000000:
 _invalid_path_char_re = re.compile(r'[^a-zA-Z0-9._\-+:/]')
 
 _valid_profile_formats = frozenset(
-	['pms', 'portage-1', 'portage-2'])
+	['pms', 'portage-1', 'portage-2', 'profile-env'])
 
 _portage1_profiles_allow_directories = frozenset(
 	["portage-1-compat", "portage-1", 'portage-2'])
-- 
2.1.0.rc2.206.gedb03e5

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Zac Medico]

On 09/16/2014 02:37 PM, Bertrand Simonnet wrote:
> 
> 
> On Tue, Sep 16, 2014 at 12:17 PM, Zac Medico <zmedico@gentoo.org
> <mailto:zmedico@gentoo.org>> wrote:
> 
>     On 09/16/2014 11:17 AM, Bertrand Simonnet wrote:
>     > I moved the profile attributes detection logic into the python side as
>     > suggested (much cleaner).
> 
>     Thanks, that's better. I've got a couple more issues though:
> 
>     1) Like global functions, global variables should also be unset in
>     __save_ebuild_env. So, we should unset PORTAGE_PROFILE_ATTRIBUTES there.
> 
> Done 
> 
>     2) Instead of having _get_profile_attributes read the layout.conf files
>     directly, it would be nicer if we could integrated it with the
>     LocationsManager layout.conf parsing. For example, see the
>     intersecting_repos code inside the _addProfile method.
> 
> This is much better. :)

I all looks very well done to me now. I'd encourage others on the list
to review it now, in case there's anything that I missed.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Alexander Berntsen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 17/09/14 00:13, Zac Medico wrote:
> I all looks very well done to me now. I'd encourage others on the
> list to review it now, in case there's anything that I missed.
At a glance, it looks OK to me. You can commit it if you vouch for it,
Zac.
- -- 
Alexander
bernalex@gentoo.org
https://secure.plaimi.net/~alexander
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlQZNMIACgkQRtClrXBQc7XA3QEArOd0SFCPLA720DY2RD3hAfj3
GFytv08BkuoVGjyz/o4A/0iMdN7SeI66ctPNEWK4lNMxxUqCezT5o24winp13//b
=/wy+
-----END PGP SIGNATURE-----

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Zac Medico]

On 09/17/2014 12:14 AM, Alexander Berntsen wrote:
> On 17/09/14 00:13, Zac Medico wrote:
>> I all looks very well done to me now. I'd encourage others on the
>> list to review it now, in case there's anything that I missed.
> At a glance, it looks OK to me. You can commit it if you vouch for it,
> Zac.

Great, thanks for your review.

I have just realized that we forgot to document the new layout.conf
profile-formats profile-env setting in man/portage.5.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Bertrand Simonnet]

[-- Attachment #1: Type: text/plain, Size: 737 bytes --]

Thanks Alexander and Zac for your review!

I'll add documentation for profile-env.
Do you prefer having it in the same patch or in a separate one ?

Bertrand

On Wed, Sep 17, 2014 at 11:02 AM, Zac Medico <zmedico@gentoo.org> wrote:

> On 09/17/2014 12:14 AM, Alexander Berntsen wrote:
> > On 17/09/14 00:13, Zac Medico wrote:
> >> I all looks very well done to me now. I'd encourage others on the
> >> list to review it now, in case there's anything that I missed.
> > At a glance, it looks OK to me. You can commit it if you vouch for it,
> > Zac.
>
> Great, thanks for your review.
>
> I have just realized that we forgot to document the new layout.conf
> profile-formats profile-env setting in man/portage.5.
> --
> Thanks,
> Zac
>
>

[-- Attachment #2: Type: text/html, Size: 1228 bytes --]

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Zac Medico]

On 09/17/2014 11:12 AM, Bertrand Simonnet wrote:
> Thanks Alexander and Zac for your review!
> 
> I'll add documentation for profile-env.
> Do you prefer having it in the same patch or in a separate one ?

I'm not too particular, but all in one patch sounds good to me.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 7075 bytes --]

Thanks for the patch. I have a few comments though.

Dnia 2014-09-16, o godz. 14:37:04
Bertrand Simonnet <bsimonnet@google.com> napisał(a):

> From 4bf1ee98bb97136e3f6f2182e205aed7ca597640 Mon Sep 17 00:00:00 2001
> From: Bertrand SIMONNET <bsimonnet@chromium.org>
> Date: Fri, 12 Sep 2014 05:07:20 -0700
> Subject: [PATCH] Make env/ bash scripts profile specific
> 
> This generalize the /etc/portage/env mechanism to be stackable with profiles.
> ebuild.sh will walk the profiles and sources scripts in env/ following the same
> matching rules as for /etc/portage/env.
> 
> Change-Id: I16bcd75790213d2204f83d4aa6e8b910f8829b6e

  ^^^^^^^^^ What is this? I don't know this tag.

> ---
>  bin/ebuild.sh                                      | 86 ++++++++++++++--------
>  bin/save-ebuild-env.sh                             |  3 +-
>  .../package/ebuild/_config/LocationsManager.py     |  6 ++
>  .../package/ebuild/_config/special_env_vars.py     |  2 +-
>  pym/portage/package/ebuild/config.py               |  2 +
>  pym/portage/package/ebuild/doebuild.py             |  2 +
>  pym/portage/repository/config.py                   |  2 +-
>  7 files changed, 68 insertions(+), 35 deletions(-)
> 
> diff --git a/bin/ebuild.sh b/bin/ebuild.sh
> index be044e0..26b9190 100755
> --- a/bin/ebuild.sh
> +++ b/bin/ebuild.sh
> @@ -67,11 +67,11 @@ unset BASH_ENV
>  # earlier portage versions do not detect a version change in this case
>  # (9999 to 9999) and therefore they try execute an incompatible version of
>  # ebuild.sh during the upgrade.
> -export PORTAGE_BZIP2_COMMAND=${PORTAGE_BZIP2_COMMAND:-bzip2} 
> +export PORTAGE_BZIP2_COMMAND=${PORTAGE_BZIP2_COMMAND:-bzip2}

While whitespace fixes are generally a good idea, please don't mix them
with feature patches. It makes reviewing harder, for a start.

[...]

> @@ -369,45 +369,67 @@ __source_all_bashrcs() {
>  		save_IFS
>  		IFS=$'\n'
>  		local path_array=($PROFILE_PATHS)
> +		local profile_attributes=($PORTAGE_PROFILE_ATTRIBUTES)
> +		local profile_path
>  		restore_IFS
> -		for x in "${path_array[@]}" ; do
> -			[ -f "$x/profile.bashrc" ] && __qa_source "$x/profile.bashrc"
> +		for i in "${!path_array[@]}" ; do
> +			profile_path="${path_array[$i]}"
> +			[ -f "${profile_path}/profile.bashrc" ] && \
> +				__qa_source "${profile_path}/profile.bashrc"
> +			[[ "${profile_attributes[$i]}" == *profile-env* ]] && \

I think you should use 'has' helper function here. The pattern match
would also succeed with 'subprofile-envofoobar' and we don't want
that :).

> +				__source_env_files "${profile_path}/env"

Could you split the patch into smaller, logical changes? If that's too
much of a hassle, then nevermind but I would prefer to have it like:

1) refactor the code to use __source_env_files and so on without
having anything directly related to new features yet,

2) add support for querying profile attributes,

3) add new profile-env support.

This way, we could distinguish better between what you refactored
and what you added.

>  		done
>  	fi
>  
> -	if [ -r "${PORTAGE_BASHRC}" ] ; then
> -		if [ "$PORTAGE_DEBUG" != "1" ] || [ "${-/x/}" != "$-" ]; then
> -			source "${PORTAGE_BASHRC}"
> -		else
> -			set -x
> -			source "${PORTAGE_BASHRC}"
> -			set +x
> -		fi
> -	fi
> +	# The user's bashrc is the ONLY non-portage bit of code
> +	# that can change shopts without a QA violation.
> +	__try_source "${PORTAGE_BASHRC}" "no_qa"
                                         ^^^^^^^

This looks like a very confusing usage for a function. It would be
really preferable to use syntax similar to standard utilities, e.g.:

  __try_source --no-qa "${PORTAGE_BASHRC}"

>  	if [[ $EBUILD_PHASE != depend ]] ; then
> -		# The user's bashrc is the ONLY non-portage bit of code that can
> -		# change shopts without a QA violation.
> -		for x in "${PM_EBUILD_HOOK_DIR}"/${CATEGORY}/{${PN},${PN}:${SLOT%/*},${P},${PF}}; do
> -			if [ -r "${x}" ]; then
> -				# If $- contains x, then tracing has already been enabled
> -				# elsewhere for some reason. We preserve it's state so as
> -				# not to interfere.
> -				if [ "$PORTAGE_DEBUG" != "1" ] || [ "${-/x/}" != "$-" ]; then
> -					source "${x}"
> -				else
> -					set -x
> -					source "${x}"
> -					set +x
> -				fi
> -			fi
> -		done
> +		__source_env_files "${PM_EBUILD_HOOK_DIR}" "no_qa"
>  	fi
>  
>  	[ ! -z "${OCC}" ] && export CC="${OCC}"
>  	[ ! -z "${OCXX}" ] && export CXX="${OCXX}"
>  }
>  
> +# @FUNCTION: __source_env_files

Please describe the parameters as well (@USAGE).

> +# @DESCRIPTION:
> +# Source the files relevant to the current package from the given path.
> +# If $2 is not 'no_qa', all the scripts are sourced with __qa_source.
> +__source_env_files() {
> +	for x in "${1}"/${CATEGORY}/{${PN},${PN}:${SLOT%/*},${P},${PF}}; do
> +		__try_source "${x}" "${2}"
> +	done
> +}

Now I understand what the patch does. I don't think we want to actually
expand use of this verbose-style env files. I'd rather go for
package.env only since it is more in line with other package.* files.

> +
> +# @FUNCTION: __try_source
> +# @DESCRIPTION:
> +# If the path given as argument exists, source the file while preserving
> +# $-.
> +# If $2 is not 'no_qa', the file is sourced with __qa_source.
> +__try_source() {
> +	if [[ -r "$1" ]]; then
> +		# If $- contains x, then tracing has already been enabled
> +		# elsewhere for some reason. We preserve it's state so as
> +		# not to interfere.
> +		if [[ "$PORTAGE_DEBUG" != "1" ]] || [[ "${-/x/}" != "$-" ]]; then
> +			if [[ "${2}" == "no_qa" ]]; then
> +				source "${x}"
> +			else
> +				__qa_source "${x}"
> +			fi
> +		else
> +			set -x
> +			if [[ "${2}" == "no_qa" ]]; then
> +				source "${x}"
> +			else
> +				__qa_source "${x}"
> +			fi
> +			set +x
> +		fi
> +	fi
> +}

I'm pretty sure you could get this smaller if you put the long
conditional in a variable and use it twice just to control 'set -x' /
'set +x'. In pseudocode:

  [[ ${debug_on} ]] && set -x
  if [[ ... ]]; then
    source
  else
    qa_source
  fi
  [[ ${debug_on} ]] && set +x

> diff --git a/pym/portage/repository/config.py b/pym/portage/repository/config.py
> index 5e0d055..ef8054e 100644
> --- a/pym/portage/repository/config.py
> +++ b/pym/portage/repository/config.py
> @@ -40,7 +40,7 @@ if sys.hexversion >= 0x3000000:
>  _invalid_path_char_re = re.compile(r'[^a-zA-Z0-9._\-+:/]')
>  
>  _valid_profile_formats = frozenset(
> -	['pms', 'portage-1', 'portage-2'])
> +	['pms', 'portage-1', 'portage-2', 'profile-env'])

I'm not sure if dedicated profile format is the best way to go. If I
understand your patch correctly, this means that 'profile-env' has only
features of PMS profile but not of 'portage-*' formats. This could mean
that some people would have to choose between features of one or
the other. Since both are compatible, i suggest 'portage-3' instead.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Zac Medico]

On 09/17/2014 02:28 PM, Michał Górny wrote:
>> diff --git a/pym/portage/repository/config.py b/pym/portage/repository/config.py
>> index 5e0d055..ef8054e 100644
>> --- a/pym/portage/repository/config.py
>> +++ b/pym/portage/repository/config.py
>> @@ -40,7 +40,7 @@ if sys.hexversion >= 0x3000000:
>>  _invalid_path_char_re = re.compile(r'[^a-zA-Z0-9._\-+:/]')
>>  
>>  _valid_profile_formats = frozenset(
>> -	['pms', 'portage-1', 'portage-2'])
>> +	['pms', 'portage-1', 'portage-2', 'profile-env'])
> 
> I'm not sure if dedicated profile format is the best way to go. If I
> understand your patch correctly, this means that 'profile-env' has only
> features of PMS profile but not of 'portage-*' formats. This could mean
> that some people would have to choose between features of one or
> the other. Since both are compatible, i suggest 'portage-3' instead.

The profile-formats field is designed to allow mixing of formats. So,
it's legal to mix profile-env with any of the other formats.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Bertrand Simonnet]

[-- Attachment #1: Type: text/plain, Size: 1615 bytes --]

Michal, not opposed to splitting the patch into three parts.

I'd rather use the env/ mechanism instead of the package.env one as it is
more flexible.
It also feels better as ebuild.sh will walk the profiles to source the
bashrc script so a bashrc from a
"low priority" profile may override a package.env definition from a high
priority profile.

I'll remove the with spaces and the Change-Id (gerrit specific tag I
believe)

Bertrand

On Wed, Sep 17, 2014 at 2:43 PM, Zac Medico <zmedico@gentoo.org> wrote:

> On 09/17/2014 02:28 PM, Michał Górny wrote:
> >> diff --git a/pym/portage/repository/config.py
> b/pym/portage/repository/config.py
> >> index 5e0d055..ef8054e 100644
> >> --- a/pym/portage/repository/config.py
> >> +++ b/pym/portage/repository/config.py
> >> @@ -40,7 +40,7 @@ if sys.hexversion >= 0x3000000:
> >>  _invalid_path_char_re = re.compile(r'[^a-zA-Z0-9._\-+:/]')
> >>
> >>  _valid_profile_formats = frozenset(
> >> -    ['pms', 'portage-1', 'portage-2'])
> >> +    ['pms', 'portage-1', 'portage-2', 'profile-env'])
> >
> > I'm not sure if dedicated profile format is the best way to go. If I
> > understand your patch correctly, this means that 'profile-env' has only
> > features of PMS profile but not of 'portage-*' formats. This could mean
> > that some people would have to choose between features of one or
> > the other. Since both are compatible, i suggest 'portage-3' instead.
>
> The profile-formats field is designed to allow mixing of formats. So,
> it's legal to mix profile-env with any of the other formats.
> --
> Thanks,
> Zac
>
>

[-- Attachment #2: Type: text/html, Size: 2288 bytes --]

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Bertrand Simonnet]

[-- Attachment #1.1: Type: text/plain, Size: 2163 bytes --]

Please find the patch split into three smaller patch:
* refactoring: contains the Michal's suggestions. I also removed the
whitespace changes.
* profile attributes export: same as in the previous patch
* profile-env patch: uses "has" to check if profile-env is set. I also
included the documentation in portage.5.

The Change-Id git hook was disabled for this round of patches ;)

Bertrand


On Wed, Sep 17, 2014 at 2:57 PM, Bertrand Simonnet <bsimonnet@google.com>
wrote:

> Michal, not opposed to splitting the patch into three parts.
>
> I'd rather use the env/ mechanism instead of the package.env one as it is
> more flexible.
> It also feels better as ebuild.sh will walk the profiles to source the
> bashrc script so a bashrc from a
> "low priority" profile may override a package.env definition from a high
> priority profile.
>
> I'll remove the with spaces and the Change-Id (gerrit specific tag I
> believe)
>
> Bertrand
>
>
> On Wed, Sep 17, 2014 at 2:43 PM, Zac Medico <zmedico@gentoo.org> wrote:
>
>> On 09/17/2014 02:28 PM, Michał Górny wrote:
>> >> diff --git a/pym/portage/repository/config.py
>> b/pym/portage/repository/config.py
>> >> index 5e0d055..ef8054e 100644
>> >> --- a/pym/portage/repository/config.py
>> >> +++ b/pym/portage/repository/config.py
>> >> @@ -40,7 +40,7 @@ if sys.hexversion >= 0x3000000:
>> >>  _invalid_path_char_re = re.compile(r'[^a-zA-Z0-9._\-+:/]')
>> >>
>> >>  _valid_profile_formats = frozenset(
>> >> -    ['pms', 'portage-1', 'portage-2'])
>> >> +    ['pms', 'portage-1', 'portage-2', 'profile-env'])
>> >
>> > I'm not sure if dedicated profile format is the best way to go. If I
>> > understand your patch correctly, this means that 'profile-env' has only
>> > features of PMS profile but not of 'portage-*' formats. This could mean
>> > that some people would have to choose between features of one or
>> > the other. Since both are compatible, i suggest 'portage-3' instead.
>>
>> The profile-formats field is designed to allow mixing of formats. So,
>> it's legal to mix profile-env with any of the other formats.
>> --
>> Thanks,
>> Zac
>>
>>
>

[-- Attachment #1.2: Type: text/html, Size: 3321 bytes --]

[-- Attachment #2: 0001-Refactor-bashrc-scripts-sourcing.patch --]
[-- Type: text/x-patch, Size: 3686 bytes --]

From 3fb3e4923425af46c084e1f600c16c344ed5a26c Mon Sep 17 00:00:00 2001
From: Bertrand SIMONNET <bsimonnet@chromium.org>
Date: Wed, 17 Sep 2014 15:10:13 -0700
Subject: [PATCH 1/3] Refactor bashrc scripts sourcing

Creates two new helper functions __try_source and __source_env_files to simplify
__source_all_bashrcs.
---
 bin/ebuild.sh          | 75 +++++++++++++++++++++++++++++++++-----------------
 bin/save-ebuild-env.sh |  1 +
 2 files changed, 51 insertions(+), 25 deletions(-)

diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index be044e0..bca5b59 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -375,39 +375,64 @@ __source_all_bashrcs() {
 		done
 	fi
 
-	if [ -r "${PORTAGE_BASHRC}" ] ; then
-		if [ "$PORTAGE_DEBUG" != "1" ] || [ "${-/x/}" != "$-" ]; then
-			source "${PORTAGE_BASHRC}"
-		else
-			set -x
-			source "${PORTAGE_BASHRC}"
-			set +x
-		fi
-	fi
+	# The user's bashrc is the ONLY non-portage bit of code
+	# that can change shopts without a QA violation.
+	__try_source --no-qa "${PORTAGE_BASHRC}"
 
 	if [[ $EBUILD_PHASE != depend ]] ; then
-		# The user's bashrc is the ONLY non-portage bit of code that can
-		# change shopts without a QA violation.
-		for x in "${PM_EBUILD_HOOK_DIR}"/${CATEGORY}/{${PN},${PN}:${SLOT%/*},${P},${PF}}; do
-			if [ -r "${x}" ]; then
-				# If $- contains x, then tracing has already been enabled
-				# elsewhere for some reason. We preserve it's state so as
-				# not to interfere.
-				if [ "$PORTAGE_DEBUG" != "1" ] || [ "${-/x/}" != "$-" ]; then
-					source "${x}"
-				else
-					set -x
-					source "${x}"
-					set +x
-				fi
-			fi
-		done
+		__source_env_files --no-qa "${PM_EBUILD_HOOK_DIR}"
 	fi
 
 	[ ! -z "${OCC}" ] && export CC="${OCC}"
 	[ ! -z "${OCXX}" ] && export CXX="${OCXX}"
 }
 
+# @FUNCTION: __source_env_files
+# @USAGE: [--no-qa] <ENV_DIRECTORY>
+# @DESCRIPTION:
+# Source the files relevant to the current package from the given path.
+# If --no-qa is specified, use source instead of __qa_source to source the
+# files.
+__source_env_files() {
+	local argument=()
+	if [[ $1 == --no-qa ]]; then
+		argument=( --no-qa )
+	shift
+	fi
+	for x in "${1}"/${CATEGORY}/{${PN},${PN}:${SLOT%/*},${P},${PF}}; do
+		__try_source "${argument[@]}" "${x}"
+	done
+}
+
+# @FUNCTION: __try_source
+# @USAGE: [--no-qa] <FILE>
+# @DESCRIPTION:
+# If the path given as argument exists, source the file while preserving
+# $-.
+# If --no-qa is specified, source the file with source instead of __qa_source.
+__try_source() {
+	local qa=true
+	if [[ $1 == --no-qa ]]; then
+		qa=false
+		shift
+	fi
+	if [[ -r "$1" ]]; then
+		local debug_on=false
+		if [[ "$PORTAGE_DEBUG" == "1" ]] && [[ "${-/x/}" == "$-" ]]; then
+			debug_on=true
+		fi
+		[[ ${debug_on} ]] && set -x
+		# If $- contains x, then tracing has already been enabled
+		# elsewhere for some reason. We preserve it's state so as
+		# not to interfere.
+		if [[ ${qa} ]]; then
+			source "${1}"
+		else
+			__qa_source "${1}"
+		fi
+		[[ ${debug_on} ]] && set +x
+	fi
+}
 # === === === === === === === === === === === === === === === === === ===
 # === === === === === functions end, main part begins === === === === ===
 # === === === === === === === === === === === === === === === === === ===
diff --git a/bin/save-ebuild-env.sh b/bin/save-ebuild-env.sh
index 98cff83..f114c48 100644
--- a/bin/save-ebuild-env.sh
+++ b/bin/save-ebuild-env.sh
@@ -75,6 +75,7 @@ __save_ebuild_env() {
 		__ebuild_main __ebuild_phase __ebuild_phase_with_hooks \
 		__ebuild_arg_to_phase __ebuild_phase_funcs default \
 		__unpack_tar __unset_colors \
+		__source_env_files __try_source \
 		${QA_INTERCEPTORS}
 
 	___eapi_has_usex && unset -f usex
-- 
2.1.0.rc2.206.gedb03e5


[-- Attachment #3: 0002-Export-profile-attributes-for-ebuild.sh.patch --]
[-- Type: text/x-patch, Size: 5564 bytes --]

From 6b0693574dc2cd71f3089f01eea59ce61142dd77 Mon Sep 17 00:00:00 2001
From: Bertrand SIMONNET <bsimonnet@chromium.org>
Date: Wed, 17 Sep 2014 15:12:50 -0700
Subject: [PATCH 2/3] Export profile attributes for ebuild.sh

Expose the profile-formats for all profiles listed in PROFILE_PATHS.
---
 bin/save-ebuild-env.sh                                 | 2 +-
 pym/portage/package/ebuild/_config/LocationsManager.py | 6 ++++++
 pym/portage/package/ebuild/_config/special_env_vars.py | 2 +-
 pym/portage/package/ebuild/config.py                   | 2 ++
 pym/portage/package/ebuild/doebuild.py                 | 2 ++
 5 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/bin/save-ebuild-env.sh b/bin/save-ebuild-env.sh
index f114c48..326410b 100644
--- a/bin/save-ebuild-env.sh
+++ b/bin/save-ebuild-env.sh
@@ -102,7 +102,7 @@ __save_ebuild_env() {
 		PORTAGE_DOHTML_UNWARNED_SKIPPED_EXTENSIONS \
 		PORTAGE_DOHTML_UNWARNED_SKIPPED_FILES \
 		PORTAGE_DOHTML_WARN_ON_SKIPPED_FILES \
-		PORTAGE_NONFATAL PORTAGE_QUIET \
+		PORTAGE_NONFATAL PORTAGE_PROFILE_ATTRIBUTES PORTAGE_QUIET \
 		PORTAGE_SANDBOX_DENY PORTAGE_SANDBOX_PREDICT \
 		PORTAGE_SANDBOX_READ PORTAGE_SANDBOX_WRITE PREROOTPATH \
 		QA_INTERCEPTORS \
diff --git a/pym/portage/package/ebuild/_config/LocationsManager.py b/pym/portage/package/ebuild/_config/LocationsManager.py
index 4427f1d..345679d 100644
--- a/pym/portage/package/ebuild/_config/LocationsManager.py
+++ b/pym/portage/package/ebuild/_config/LocationsManager.py
@@ -114,6 +114,7 @@ class LocationsManager(object):
 		# The symlink might not exist or might not be a symlink.
 		self.profiles = []
 		self.profiles_complex = []
+		self.profile_formats = []
 		if self.profile_path:
 			try:
 				self._addProfile(os.path.realpath(self.profile_path),
@@ -124,6 +125,7 @@ class LocationsManager(object):
 					writemsg("!!! ParseError: %s\n" % str(e), noiselevel=-1)
 				self.profiles = []
 				self.profiles_complex = []
+				self.profile_formats = []
 
 		if self._user_config and self.profiles:
 			custom_prof = os.path.join(
@@ -137,6 +139,7 @@ class LocationsManager(object):
 
 		self.profiles = tuple(self.profiles)
 		self.profiles_complex = tuple(self.profiles_complex)
+		self.profile_formats = tuple(self.profile_formats)
 
 	def _check_var_directory(self, varname, var):
 		if not isdir_raise_eaccess(var):
@@ -151,6 +154,7 @@ class LocationsManager(object):
 		allow_parent_colon = True
 		repo_loc = None
 		compat_mode = False
+		current_formats = ()
 
 		eapi_file = os.path.join(currentPath, "eapi")
 		eapi = "0"
@@ -183,6 +187,7 @@ class LocationsManager(object):
 				layout_data['profile-formats'] == ('portage-1-compat',)
 			allow_parent_colon = any(x in _allow_parent_colon
 				for x in layout_data['profile-formats'])
+			current_formats = tuple(layout_data['profile-formats'])
 
 		if compat_mode:
 			offenders = _PORTAGE1_DIRECTORIES.intersection(os.listdir(currentPath))
@@ -234,6 +239,7 @@ class LocationsManager(object):
 		self.profiles.append(currentPath)
 		self.profiles_complex.append(
 			_profile_node(currentPath, allow_directories, False))
+		self.profile_formats.append(current_formats)
 
 	def _expand_parent_colon(self, parentsFile, parentPath,
 		repo_loc, repositories):
diff --git a/pym/portage/package/ebuild/_config/special_env_vars.py b/pym/portage/package/ebuild/_config/special_env_vars.py
index 74fedd6..b929042 100644
--- a/pym/portage/package/ebuild/_config/special_env_vars.py
+++ b/pym/portage/package/ebuild/_config/special_env_vars.py
@@ -68,7 +68,7 @@ environ_whitelist += [
 	"PORTAGE_INST_GID", "PORTAGE_INST_UID",
 	"PORTAGE_IPC_DAEMON", "PORTAGE_IUSE", "PORTAGE_ECLASS_LOCATIONS",
 	"PORTAGE_LOG_FILE", "PORTAGE_OVERRIDE_EPREFIX", "PORTAGE_PIPE_FD",
-	"PORTAGE_PYM_PATH", "PORTAGE_PYTHON",
+	"PORTAGE_PROFILE_ATTRIBUTES", "PORTAGE_PYM_PATH", "PORTAGE_PYTHON",
 	"PORTAGE_PYTHONPATH", "PORTAGE_QUIET",
 	"PORTAGE_REPO_NAME", "PORTAGE_REPOSITORIES", "PORTAGE_RESTRICT",
 	"PORTAGE_SIGPIPE_STATUS",
diff --git a/pym/portage/package/ebuild/config.py b/pym/portage/package/ebuild/config.py
index f639e14..37e71f9 100644
--- a/pym/portage/package/ebuild/config.py
+++ b/pym/portage/package/ebuild/config.py
@@ -247,6 +247,7 @@ class config(object):
 			self.module_priority = clone.module_priority
 			self.profile_path = clone.profile_path
 			self.profiles = clone.profiles
+			self.profile_formats = clone.profile_formats
 			self.packages = clone.packages
 			self.repositories = clone.repositories
 			self.unpack_dependencies = clone.unpack_dependencies
@@ -554,6 +555,7 @@ class config(object):
 
 			profiles_complex = locations_manager.profiles_complex
 			self.profiles = locations_manager.profiles
+			self.profile_formats = locations_manager.profile_formats
 			self.profile_path = locations_manager.profile_path
 			self.user_profile_dir = locations_manager.user_profile_dir
 
diff --git a/pym/portage/package/ebuild/doebuild.py b/pym/portage/package/ebuild/doebuild.py
index 01707ae..0b8c3fa 100644
--- a/pym/portage/package/ebuild/doebuild.py
+++ b/pym/portage/package/ebuild/doebuild.py
@@ -336,6 +336,8 @@ def doebuild_environment(myebuild, mydo, myroot=None, settings=None,
 	mysettings["SANDBOX_LOG"] = mycpv.replace("/", "_-_")
 
 	mysettings["PROFILE_PATHS"] = "\n".join(mysettings.profiles)
+	mysettings["PORTAGE_PROFILE_ATTRIBUTES"] = \
+		"\n".join([" ".join(formats) for formats in mysettings.profile_formats])
 	mysettings["P"]  = mysplit[0]+"-"+mysplit[1]
 	mysettings["PN"] = mysplit[0]
 	mysettings["PV"] = mysplit[1]
-- 
2.1.0.rc2.206.gedb03e5


[-- Attachment #4: 0003-Make-env-bash-scripts-profile-specific.patch --]
[-- Type: text/x-patch, Size: 3425 bytes --]

From def62f7e1785c65b33623b1a4547738e05b901ae Mon Sep 17 00:00:00 2001
From: Bertrand SIMONNET <bsimonnet@chromium.org>
Date: Wed, 17 Sep 2014 15:13:31 -0700
Subject: [PATCH 3/3] Make env/ bash scripts profile specific

This generalize the /etc/portage/env mechanism to be stackable with profiles.
ebuild.sh will walk the profiles and sources scripts in env/ following the
same matching rules as for /etc/portage/env.
---
 bin/ebuild.sh                    | 14 ++++++++++----
 man/portage.5                    |  5 ++++-
 pym/portage/repository/config.py |  2 +-
 3 files changed, 15 insertions(+), 6 deletions(-)

diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index bca5b59..7d5e15a 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -369,9 +369,15 @@ __source_all_bashrcs() {
 		save_IFS
 		IFS=$'\n'
 		local path_array=($PROFILE_PATHS)
+		local profile_attributes=($PORTAGE_PROFILE_ATTRIBUTES)
+		local profile_path
 		restore_IFS
-		for x in "${path_array[@]}" ; do
-			[ -f "$x/profile.bashrc" ] && __qa_source "$x/profile.bashrc"
+		for i in "${!path_array[@]}" ; do
+			profile_path="${path_array[$i]}"
+			[ -f "${profile_path}/profile.bashrc" ] && \
+				__qa_source "${profile_path}/profile.bashrc"
+			has "profile-env" ${profile_attributes[$i]} && \
+				__source_env_files "${profile_path}/env"
 		done
 	fi
 
@@ -421,7 +427,7 @@ __try_source() {
 		if [[ "$PORTAGE_DEBUG" == "1" ]] && [[ "${-/x/}" == "$-" ]]; then
 			debug_on=true
 		fi
-		[[ ${debug_on} ]] && set -x
+		${debug_on} && set -x
 		# If $- contains x, then tracing has already been enabled
 		# elsewhere for some reason. We preserve it's state so as
 		# not to interfere.
@@ -430,7 +436,7 @@ __try_source() {
 		else
 			__qa_source "${1}"
 		fi
-		[[ ${debug_on} ]] && set +x
+		${debug_on} && set +x
 	fi
 }
 # === === === === === === === === === === === === === === === === === ===
diff --git a/man/portage.5 b/man/portage.5
index e399f0f..0c1cce9 100644
--- a/man/portage.5
+++ b/man/portage.5
@@ -1047,11 +1047,14 @@ The default setting for repoman's --echangelog option.
 The cache formats supported in the metadata tree.  There is the old "pms" format
 and the newer/faster "md5-dict" format.  Default is to detect dirs.
 .TP
-.BR profile\-formats " = [pms|portage-1|portage-2]"
+.BR profile\-formats " = [pms|portage-1|portage-2|profile-env]"
 Control functionality available to profiles in this repo such as which files
 may be dirs, or the syntax available in parent files.  Use "portage-2" if you're
 unsure.  The default is "portage-1-compat" mode which is meant to be compatible
 with old profiles, but is not allowed to be opted into directly.
+"profile-env" allows profiles to modify the environment for each any package.
+The environment modifications are defined in an "env" directory in each profile,
+following the same conventions as for /etc/portage/env.
 .RE
 .RE
 
diff --git a/pym/portage/repository/config.py b/pym/portage/repository/config.py
index 5e0d055..ef8054e 100644
--- a/pym/portage/repository/config.py
+++ b/pym/portage/repository/config.py
@@ -40,7 +40,7 @@ if sys.hexversion >= 0x3000000:
 _invalid_path_char_re = re.compile(r'[^a-zA-Z0-9._\-+:/]')
 
 _valid_profile_formats = frozenset(
-	['pms', 'portage-1', 'portage-2'])
+	['pms', 'portage-1', 'portage-2', 'profile-env'])
 
 _portage1_profiles_allow_directories = frozenset(
 	["portage-1-compat", "portage-1", 'portage-2'])
-- 
2.1.0.rc2.206.gedb03e5

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Alexander Berntsen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

For future reference, please use git's send-email option with
- --in-reply-to, to send the patches inline. The way they are sent now,
(I assume) most of us have to download your patches and open them in a
text viewer to review it. If the patches were inline, I could just
look at them in the email client.

Thanks for splitting them up, and thanks for your work on this.
- -- 
Alexander
bernalex@gentoo.org
https://secure.plaimi.net/~alexander
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlQajFsACgkQRtClrXBQc7We4gD/UztD3pvIRr2DDP5l/qZBFPFq
zwJ+A2ijpKkBDGAF9kwBAJ8fii7loZ+AHw+InoHQat7s2BOFh6fkvz/uNx2J4iDb
=IGeT
-----END PGP SIGNATURE-----

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 602 bytes --]

Dnia 2014-09-17, o godz. 14:57:10
Bertrand Simonnet <bsimonnet@google.com> napisał(a):

> I'd rather use the env/ mechanism instead of the package.env one as it is
> more flexible.

It depends on what you aim to do. As portage(5) points out, both have
their advantages:

- package.env is parsed early, and so allows you override more
  variables, like FEATURES,

- env/ is used as bashrc extension.

The other difference is that package.env supports any atom syntax that
the particular EAPI supports, while env/ has hardcoded list of
possibilities.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Bertrand Simonnet]

[-- Attachment #1: Type: text/plain, Size: 1068 bytes --]

For my purpose, I think bash scripting would be more useful. I thought
about package.env
in the beginning (see first few messages on this thread) as it would help
us reuse code but
variable setting only will limit what we can do.

If you want package.env, we can implement it too and have both mechanisms
available.

Thanks,
Bertrand

On Thu, Sep 18, 2014 at 1:02 AM, Michał Górny <mgorny@gentoo.org> wrote:

> Dnia 2014-09-17, o godz. 14:57:10
> Bertrand Simonnet <bsimonnet@google.com> napisał(a):
>
> > I'd rather use the env/ mechanism instead of the package.env one as it is
> > more flexible.
>
> It depends on what you aim to do. As portage(5) points out, both have
> their advantages:
>
> - package.env is parsed early, and so allows you override more
>   variables, like FEATURES,
>
> - env/ is used as bashrc extension.
>
> The other difference is that package.env supports any atom syntax that
> the particular EAPI supports, while env/ has hardcoded list of
> possibilities.
>
> --
> Best regards,
> Michał Górny
>

[-- Attachment #2: Type: text/html, Size: 1590 bytes --]

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Bertrand Simonnet]

[-- Attachment #1: Type: text/plain, Size: 1500 bytes --]

Zac, Michal,

Would you be willing to merge this ?
If env/ instead of package.env is a deal breaker, I can change that.
A bashrc like mechanism is more practical for us but package.env will do
the trick too
and we really want to have this in mainline portage.

Thanks,
Bertrand

On Thu, Sep 18, 2014 at 10:54 AM, Bertrand Simonnet <bsimonnet@google.com>
wrote:

> For my purpose, I think bash scripting would be more useful. I thought
> about package.env
> in the beginning (see first few messages on this thread) as it would help
> us reuse code but
> variable setting only will limit what we can do.
>
> If you want package.env, we can implement it too and have both mechanisms
> available.
>
> Thanks,
> Bertrand
>
> On Thu, Sep 18, 2014 at 1:02 AM, Michał Górny <mgorny@gentoo.org> wrote:
>
>> Dnia 2014-09-17, o godz. 14:57:10
>> Bertrand Simonnet <bsimonnet@google.com> napisał(a):
>>
>> > I'd rather use the env/ mechanism instead of the package.env one as it
>> is
>> > more flexible.
>>
>> It depends on what you aim to do. As portage(5) points out, both have
>> their advantages:
>>
>> - package.env is parsed early, and so allows you override more
>>   variables, like FEATURES,
>>
>> - env/ is used as bashrc extension.
>>
>> The other difference is that package.env supports any atom syntax that
>> the particular EAPI supports, while env/ has hardcoded list of
>> possibilities.
>>
>> --
>> Best regards,
>> Michał Górny
>>
>
>

[-- Attachment #2: Type: text/html, Size: 2368 bytes --]

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Zac Medico]

On 09/22/2014 09:16 AM, Bertrand Simonnet wrote:
> Zac, Michal,
> 
> Would you be willing to merge this ?
> If env/ instead of package.env is a deal breaker, I can change that.
> A bashrc like mechanism is more practical for us but package.env will do
> the trick too
> and we really want to have this in mainline portage.
> 
> Thanks,
> Bertrand

In order to solve Michał's concern about the "hardcoded list of atoms",
we might choose this slightly different approach:

In python, parse package.env atom/file associations from the profile,
and pass the associated file names to bash as array, so that those files
can be sourced by bash.

Thoughts?
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Zac Medico]

On 09/22/2014 11:16 AM, Zac Medico wrote:
> On 09/22/2014 09:16 AM, Bertrand Simonnet wrote:
>> Zac, Michal,
>>
>> Would you be willing to merge this ?
>> If env/ instead of package.env is a deal breaker, I can change that.
>> A bashrc like mechanism is more practical for us but package.env will do
>> the trick too
>> and we really want to have this in mainline portage.
>>
>> Thanks,
>> Bertrand
> 
> In order to solve Michał's concern about the "hardcoded list of atoms",
> we might choose this slightly different approach:
> 
> In python, parse package.env atom/file associations from the profile,
> and pass the associated file names to bash as array, so that those files
> can be sourced by bash.
> 
> Thoughts?
> 

Also, note that with this hybrid bashrc like approach, you still won't
be able to adjust FEATURES. However, I'm not so sure that per-package
FEATURES adjustment is really that useful at the profile level.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Bertrand Simonnet]

[-- Attachment #1: Type: text/plain, Size: 1586 bytes --]

Did you mean env/ files ? Having a different behaviour for
/etc/portage/package.env and $profile/package.env would be confusing.

Parsing env/ atom/files association in python would be a good idea. We
should get the benefits of both I believe.
The list of files to be sourced by bash could contain all the files
relevant to the package in one profile then profile.bashrc, for each
profile so that
we don't even need to walk all the profiles in ebuild.sh.

Bertrand

On Mon, Sep 22, 2014 at 11:24 AM, Zac Medico <zmedico@gentoo.org> wrote:

> On 09/22/2014 11:16 AM, Zac Medico wrote:
> > On 09/22/2014 09:16 AM, Bertrand Simonnet wrote:
> >> Zac, Michal,
> >>
> >> Would you be willing to merge this ?
> >> If env/ instead of package.env is a deal breaker, I can change that.
> >> A bashrc like mechanism is more practical for us but package.env will do
> >> the trick too
> >> and we really want to have this in mainline portage.
> >>
> >> Thanks,
> >> Bertrand
> >
> > In order to solve Michał's concern about the "hardcoded list of atoms",
> > we might choose this slightly different approach:
> >
> > In python, parse package.env atom/file associations from the profile,
> > and pass the associated file names to bash as array, so that those files
> > can be sourced by bash.
> >
> > Thoughts?
> >
>
> Also, note that with this hybrid bashrc like approach, you still won't
> be able to adjust FEATURES. However, I'm not so sure that per-package
> FEATURES adjustment is really that useful at the profile level.
> --
> Thanks,
> Zac
>
>

[-- Attachment #2: Type: text/html, Size: 2216 bytes --]

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Zac Medico]

On 09/22/2014 11:43 AM, Bertrand Simonnet wrote:
> Did you mean env/ files ?

Yes.

> Having a different behaviour for
> /etc/portage/package.env and $profile/package.env would be confusing.

We could call package.bashenv (or something like that), in order to make
the difference from package.env clear.

> Parsing env/ atom/files association in python would be a good idea. We
> should get the benefits of both I believe.
> The list of files to be sourced by bash could contain all the files
> relevant to the package in one profile then profile.bashrc, for each
> profile so that
> we don't even need to walk all the profiles in ebuild.sh.

Right, that's what I was thinking.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Bertrand Simonnet]

[-- Attachment #1: Type: text/plain, Size: 894 bytes --]

Sounds good :)

Michal, would that work for you ? If so, I'll start on it tomorrow.

On Mon, Sep 22, 2014 at 5:48 PM, Zac Medico <zmedico@gentoo.org> wrote:

> On 09/22/2014 11:43 AM, Bertrand Simonnet wrote:
> > Did you mean env/ files ?
>
> Yes.
>
> > Having a different behaviour for
> > /etc/portage/package.env and $profile/package.env would be confusing.
>
> We could call package.bashenv (or something like that), in order to make
> the difference from package.env clear.
>
> > Parsing env/ atom/files association in python would be a good idea. We
> > should get the benefits of both I believe.
> > The list of files to be sourced by bash could contain all the files
> > relevant to the package in one profile then profile.bashrc, for each
> > profile so that
> > we don't even need to walk all the profiles in ebuild.sh.
>
> Right, that's what I was thinking.
> --
> Thanks,
> Zac
>
>

[-- Attachment #2: Type: text/html, Size: 1446 bytes --]

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Zac Medico]

On 09/22/2014 05:57 PM, Bertrand Simonnet wrote:
> Sounds good :)
> 
> Michal, would that work for you ? If so, I'll start on it tomorrow.

I don't know why he hasn't responded yet. I've just pinged him in the
#gentoo-portage irc channel.

Anyway, I think the proposed package.bashenv approach is great. I think
we should also support this in /etc/portage as a replacement for the
existing hardcoded atoms method.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 506 bytes --]

Dnia 2014-09-22, o godz. 17:57:26
Bertrand Simonnet <bsimonnet@google.com> napisał(a):

> Sounds good :)
> 
> Michal, would that work for you ? If so, I'll start on it tomorrow.

I have mixed feeling about having yet another package.* file but
the idea sounds good. However, I'd prefer naming it
'package.bashrc' (since it pretty much points to bashrcs, doesn't it?)
and using a new subdirectory like 'bashrc/' instead of re-using 'env/'
to avoid confusion.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Zac Medico]

On 09/26/2014 09:22 AM, Michał Górny wrote:
> Dnia 2014-09-22, o godz. 17:57:26
> Bertrand Simonnet <bsimonnet@google.com> napisał(a):
> 
>> Sounds good :)
>>
>> Michal, would that work for you ? If so, I'll start on it tomorrow.
> 
> I have mixed feeling about having yet another package.* file but
> the idea sounds good. However, I'd prefer naming it
> 'package.bashrc' (since it pretty much points to bashrcs, doesn't it?)
> and using a new subdirectory like 'bashrc/' instead of re-using 'env/'
> to avoid confusion.

That sounds pretty reasonable to me. I really like the idea of using a
new directory name. When we added package.env support, I wasn't very
comfortable with re-using the /etc/portage/env directory which was
already being used for bashrc, but we did it anyway because others
thought it would be fine.
-- 
Thanks,
Zac

[gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Bertrand SIMONNET]

Hi all,

Please find the patches implementing package.bashrc, the bashrc mechanism
parsing the atom names in python.

I split the change into three smaller changes:
* Refactoring ebuild.sh  (same as the first refactoring patch)
* Include profile-formats in profile_complex
* package.bashrc change

I also included the documentation for package.bashrc in the latest patch.
Let me know what you think :)

Thanks,
Bertrand

[gentoo-portage-dev] [PATCH 1/3] Refactor bashrc scripts sourcing

[Bertrand SIMONNET]

Creates two new helper functions __try_source and __source_env_files to simplify
__source_all_bashrcs.
---
 bin/ebuild.sh          | 75 +++++++++++++++++++++++++++++++++-----------------
 bin/save-ebuild-env.sh |  1 +
 2 files changed, 51 insertions(+), 25 deletions(-)

diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index be044e0..14cc321 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -375,39 +375,64 @@ __source_all_bashrcs() {
 		done
 	fi
 
-	if [ -r "${PORTAGE_BASHRC}" ] ; then
-		if [ "$PORTAGE_DEBUG" != "1" ] || [ "${-/x/}" != "$-" ]; then
-			source "${PORTAGE_BASHRC}"
-		else
-			set -x
-			source "${PORTAGE_BASHRC}"
-			set +x
-		fi
-	fi
+	# The user's bashrc is the ONLY non-portage bit of code
+	# that can change shopts without a QA violation.
+	__try_source --no-qa "${PORTAGE_BASHRC}"
 
 	if [[ $EBUILD_PHASE != depend ]] ; then
-		# The user's bashrc is the ONLY non-portage bit of code that can
-		# change shopts without a QA violation.
-		for x in "${PM_EBUILD_HOOK_DIR}"/${CATEGORY}/{${PN},${PN}:${SLOT%/*},${P},${PF}}; do
-			if [ -r "${x}" ]; then
-				# If $- contains x, then tracing has already been enabled
-				# elsewhere for some reason. We preserve it's state so as
-				# not to interfere.
-				if [ "$PORTAGE_DEBUG" != "1" ] || [ "${-/x/}" != "$-" ]; then
-					source "${x}"
-				else
-					set -x
-					source "${x}"
-					set +x
-				fi
-			fi
-		done
+		__source_env_files --no-qa "${PM_EBUILD_HOOK_DIR}"
 	fi
 
 	[ ! -z "${OCC}" ] && export CC="${OCC}"
 	[ ! -z "${OCXX}" ] && export CXX="${OCXX}"
 }
 
+# @FUNCTION: __source_env_files
+# @USAGE: [--no-qa] <ENV_DIRECTORY>
+# @DESCRIPTION:
+# Source the files relevant to the current package from the given path.
+# If --no-qa is specified, use source instead of __qa_source to source the
+# files.
+__source_env_files() {
+	local argument=()
+	if [[ $1 == --no-qa ]]; then
+		argument=( --no-qa )
+	shift
+	fi
+	for x in "${1}"/${CATEGORY}/{${PN},${PN}:${SLOT%/*},${P},${PF}}; do
+		__try_source "${argument[@]}" "${x}"
+	done
+}
+
+# @FUNCTION: __try_source
+# @USAGE: [--no-qa] <FILE>
+# @DESCRIPTION:
+# If the path given as argument exists, source the file while preserving
+# $-.
+# If --no-qa is specified, source the file with source instead of __qa_source.
+__try_source() {
+	local qa=true
+	if [[ $1 == --no-qa ]]; then
+		qa=false
+		shift
+	fi
+	if [[ -r "$1" ]]; then
+		local debug_on=false
+		if [[ "$PORTAGE_DEBUG" == "1" ]] && [[ "${-/x/}" == "$-" ]]; then
+			debug_on=true
+		fi
+		$debug_on && set -x
+		# If $- contains x, then tracing has already been enabled
+		# elsewhere for some reason. We preserve it's state so as
+		# not to interfere.
+		if [[ ${qa} ]]; then
+			source "${1}"
+		else
+			__qa_source "${1}"
+		fi
+		$debug_on && set +x
+	fi
+}
 # === === === === === === === === === === === === === === === === === ===
 # === === === === === functions end, main part begins === === === === ===
 # === === === === === === === === === === === === === === === === === ===
diff --git a/bin/save-ebuild-env.sh b/bin/save-ebuild-env.sh
index 98cff83..f114c48 100644
--- a/bin/save-ebuild-env.sh
+++ b/bin/save-ebuild-env.sh
@@ -75,6 +75,7 @@ __save_ebuild_env() {
 		__ebuild_main __ebuild_phase __ebuild_phase_with_hooks \
 		__ebuild_arg_to_phase __ebuild_phase_funcs default \
 		__unpack_tar __unset_colors \
+		__source_env_files __try_source \
 		${QA_INTERCEPTORS}
 
 	___eapi_has_usex && unset -f usex
-- 
2.1.0.rc2.206.gedb03e5

[gentoo-portage-dev] [PATCH 2/3] Add profile-formats to profile_complex

[Bertrand SIMONNET]

---
 pym/portage/package/ebuild/_config/LocationsManager.py | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/pym/portage/package/ebuild/_config/LocationsManager.py b/pym/portage/package/ebuild/_config/LocationsManager.py
index 4427f1d..8bf321c 100644
--- a/pym/portage/package/ebuild/_config/LocationsManager.py
+++ b/pym/portage/package/ebuild/_config/LocationsManager.py
@@ -31,7 +31,7 @@ _PORTAGE1_DIRECTORIES = frozenset([
 	'use.mask', 'use.force'])
 
 _profile_node = collections.namedtuple('_profile_node',
-	'location portage1_directories user_config')
+	'location portage1_directories user_config profile_formats')
 
 _allow_parent_colon = frozenset(
 	["portage-2"])
@@ -132,7 +132,7 @@ class LocationsManager(object):
 				self.user_profile_dir = custom_prof
 				self.profiles.append(custom_prof)
 				self.profiles_complex.append(
-					_profile_node(custom_prof, True, True))
+					_profile_node(custom_prof, True, True, ()))
 			del custom_prof
 
 		self.profiles = tuple(self.profiles)
@@ -151,6 +151,7 @@ class LocationsManager(object):
 		allow_parent_colon = True
 		repo_loc = None
 		compat_mode = False
+		current_formats = ()
 
 		eapi_file = os.path.join(currentPath, "eapi")
 		eapi = "0"
@@ -183,6 +184,8 @@ class LocationsManager(object):
 				layout_data['profile-formats'] == ('portage-1-compat',)
 			allow_parent_colon = any(x in _allow_parent_colon
 				for x in layout_data['profile-formats'])
+			current_formats = tuple(layout_data['profile-formats'])
+
 
 		if compat_mode:
 			offenders = _PORTAGE1_DIRECTORIES.intersection(os.listdir(currentPath))
@@ -233,7 +236,8 @@ class LocationsManager(object):
 
 		self.profiles.append(currentPath)
 		self.profiles_complex.append(
-			_profile_node(currentPath, allow_directories, False))
+			_profile_node(currentPath, allow_directories, False,
+				current_formats))
 
 	def _expand_parent_colon(self, parentsFile, parentPath,
 		repo_loc, repositories):
-- 
2.1.0.rc2.206.gedb03e5

[gentoo-portage-dev] [PATCH 3/3] package.bashrc: per profile, per-package bashrc mechanism

[Bertrand SIMONNET]

Profiles can define per-package bashrc files to be sourced before emerging.
Each line in package.bashrc must be an atom name then a list of space-delimited
bashrc files (stored in $profile/bashrc/).
---
 bin/ebuild.sh                                      |  6 ++--
 bin/phase-functions.sh                             |  2 +-
 bin/save-ebuild-env.sh                             |  2 +-
 man/portage.5                                      | 30 +++++++++++++++++-
 .../package/ebuild/_config/special_env_vars.py     |  4 +--
 pym/portage/package/ebuild/config.py               | 36 ++++++++++++++++++++++
 pym/portage/package/ebuild/doebuild.py             |  4 ++-
 pym/portage/repository/config.py                   |  2 +-
 8 files changed, 76 insertions(+), 10 deletions(-)

diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index 14cc321..50909e1 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -368,10 +368,10 @@ __source_all_bashrcs() {
 		# source the existing profile.bashrcs.
 		save_IFS
 		IFS=$'\n'
-		local path_array=($PROFILE_PATHS)
+		local bashenv_files=($PORTAGE_BASHRC_FILES)
 		restore_IFS
-		for x in "${path_array[@]}" ; do
-			[ -f "$x/profile.bashrc" ] && __qa_source "$x/profile.bashrc"
+		for x in "${bashenv_files[@]}" ; do
+			__try_source "${x}"
 		done
 	fi
 
diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index f39a024..5dff3bb 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -31,7 +31,7 @@ PORTAGE_READONLY_VARS="D EBUILD EBUILD_PHASE EBUILD_PHASE_FUNC \
 	PORTAGE_TMPDIR PORTAGE_UPDATE_ENV PORTAGE_USERNAME \
 	PORTAGE_VERBOSE PORTAGE_WORKDIR_MODE PORTAGE_XATTR_EXCLUDE \
 	PORTDIR \
-	PROFILE_PATHS REPLACING_VERSIONS REPLACED_BY_VERSION T WORKDIR \
+	REPLACING_VERSIONS REPLACED_BY_VERSION T WORKDIR \
 	__PORTAGE_HELPER __PORTAGE_TEST_HARDLINK_LOCKS"
 
 PORTAGE_SAVED_READONLY_VARS="A CATEGORY P PF PN PR PV PVR"
diff --git a/bin/save-ebuild-env.sh b/bin/save-ebuild-env.sh
index f114c48..1a684b9 100644
--- a/bin/save-ebuild-env.sh
+++ b/bin/save-ebuild-env.sh
@@ -97,7 +97,7 @@ __save_ebuild_env() {
 		GOOD HILITE HOME \
 		LAST_E_CMD LAST_E_LEN LD_PRELOAD MISC_FUNCTIONS_ARGS MOPREFIX \
 		NOCOLOR NORMAL PKGDIR PKGUSE PKG_LOGDIR PKG_TMPDIR \
-		PORTAGE_BASHRCS_SOURCED PORTAGE_COMPRESS \
+		PORTAGE_BASHRC_FILES PORTAGE_BASHRCS_SOURCED PORTAGE_COMPRESS \
 		PORTAGE_COMPRESS_EXCLUDE_SUFFIXES \
 		PORTAGE_DOHTML_UNWARNED_SKIPPED_EXTENSIONS \
 		PORTAGE_DOHTML_UNWARNED_SKIPPED_FILES \
diff --git a/man/portage.5 b/man/portage.5
index e399f0f..309e259 100644
--- a/man/portage.5
+++ b/man/portage.5
@@ -24,6 +24,7 @@ make.defaults
 packages
 packages.build
 package.accept_keywords
+package.bashrc
 package.keywords
 package.mask
 package.provided
@@ -358,6 +359,31 @@ a '\-'.
 A list of packages (one per line) that make up a stage1 tarball.  Really only
 useful for stage builders.
 .TP
+.BR package.bashrc
+Per-package bashrc mechanism.  Contains a list of bashrc files to be sourced
+before emerging a given atom.  The bashrc files must be stored in bashrc/, in
+the profile directory.
+
+.I Note:
+.nf
+\- The bashrc files will be sourced after profile.bashrc for the same profile.
+\- profile-formats in metadata/layout.conf must contain profile-bashrcs for this
+to be enabled.
+.fi
+
+.I Format:
+.nf
+\- comments begin with # (no inline comments).
+\- one atom per line with space-delimited list of bashrc files.
+.fi
+
+.I Example:
+.nf
+# By setting INSTALL_MASK in bashrc/nostandardconf.conf, we can avoid installing
+# the standard configuration and enable another package to install it.
+net-misc/dhcp nostardardconf.conf
+.fi
+.TP
 .BR package.provided
 A list of packages (one per line) that portage should assume have been
 provided.  Useful for porting to non-Linux systems. Basically, it's a
@@ -1047,11 +1073,13 @@ The default setting for repoman's --echangelog option.
 The cache formats supported in the metadata tree.  There is the old "pms" format
 and the newer/faster "md5-dict" format.  Default is to detect dirs.
 .TP
-.BR profile\-formats " = [pms|portage-1|portage-2]"
+.BR profile\-formats " = [pms|portage-1|portage-2|profile-bashrcs]"
 Control functionality available to profiles in this repo such as which files
 may be dirs, or the syntax available in parent files.  Use "portage-2" if you're
 unsure.  The default is "portage-1-compat" mode which is meant to be compatible
 with old profiles, but is not allowed to be opted into directly.
+Setting profile-bashrcs will enable the per-profile bashrc mechanism
+\fBpackage.bashrc\fR.
 .RE
 .RE
 
diff --git a/pym/portage/package/ebuild/_config/special_env_vars.py b/pym/portage/package/ebuild/_config/special_env_vars.py
index 74fedd6..387f4ae 100644
--- a/pym/portage/package/ebuild/_config/special_env_vars.py
+++ b/pym/portage/package/ebuild/_config/special_env_vars.py
@@ -49,7 +49,7 @@ environ_whitelist += [
 	"FEATURES", "FILESDIR", "HOME", "MERGE_TYPE", "NOCOLOR", "PATH",
 	"PKGDIR",
 	"PKGUSE", "PKG_LOGDIR", "PKG_TMPDIR",
-	"PORTAGE_ACTUAL_DISTDIR", "PORTAGE_ARCHLIST",
+	"PORTAGE_ACTUAL_DISTDIR", "PORTAGE_ARCHLIST", "PORTAGE_BASHRC_FILES",
 	"PORTAGE_BASHRC", "PM_EBUILD_HOOK_DIR",
 	"PORTAGE_BINPKG_FILE", "PORTAGE_BINPKG_TAR_OPTS",
 	"PORTAGE_BINPKG_TMPFILE",
@@ -74,7 +74,7 @@ environ_whitelist += [
 	"PORTAGE_SIGPIPE_STATUS",
 	"PORTAGE_TMPDIR", "PORTAGE_UPDATE_ENV", "PORTAGE_USERNAME",
 	"PORTAGE_VERBOSE", "PORTAGE_WORKDIR_MODE", "PORTAGE_XATTR_EXCLUDE",
-	"PORTDIR", "PORTDIR_OVERLAY", "PREROOTPATH", "PROFILE_PATHS",
+	"PORTDIR", "PORTDIR_OVERLAY", "PREROOTPATH",
 	"REPLACING_VERSIONS", "REPLACED_BY_VERSION",
 	"ROOT", "ROOTPATH", "T", "TMP", "TMPDIR",
 	"USE_EXPAND", "USE_ORDER", "WORKDIR",
diff --git a/pym/portage/package/ebuild/config.py b/pym/portage/package/ebuild/config.py
index f639e14..183627f 100644
--- a/pym/portage/package/ebuild/config.py
+++ b/pym/portage/package/ebuild/config.py
@@ -316,6 +316,7 @@ class config(object):
 			self._accept_restrict = copy.deepcopy(clone._accept_restrict)
 			self._paccept_restrict = copy.deepcopy(clone._paccept_restrict)
 			self._penvdict = copy.deepcopy(clone._penvdict)
+			self._pbashrcdict = copy.deepcopy(clone._pbashrcdict)
 			self._expand_map = copy.deepcopy(clone._expand_map)
 
 		else:
@@ -661,6 +662,7 @@ class config(object):
 			self._ppropertiesdict = portage.dep.ExtendedAtomDict(dict)
 			self._paccept_restrict = portage.dep.ExtendedAtomDict(dict)
 			self._penvdict = portage.dep.ExtendedAtomDict(dict)
+			self._pbashrcdict = {}
 
 			self._repo_make_defaults = {}
 			for repo in self.repositories.repos_with_profiles():
@@ -742,6 +744,25 @@ class config(object):
 				for k, v in penvdict.items():
 					self._penvdict.setdefault(k.cp, {})[k] = v
 
+				# package.bashrc
+				for profile in profiles_complex:
+					if not 'profile-bashrcs' in profile.profile_formats:
+						continue
+					self._pbashrcdict[profile] = \
+						portage.dep.ExtendedAtomDict(dict)
+					bashrc = grabdict_package(os.path.join(profile.location,
+						"package.bashrc"), recursive=1, allow_wildcard=True,
+								allow_repo=True, verify_eapi=False)
+					if not bashrc:
+						continue
+
+					for k, v in bashrc.items():
+						envfiles = [os.path.join(profile.location,
+							"bashrc",
+							envname) for envname in v]
+						self._pbashrcdict[profile].setdefault(k.cp, {})\
+							.setdefault(k, []).extend(envfiles)
+
 			#getting categories from an external file now
 			self.categories = [grabfile(os.path.join(x, "categories")) \
 				for x in locations_manager.profile_and_user_locations]
@@ -1501,6 +1522,21 @@ class config(object):
 				for x in penv_matches:
 					self._penv.extend(x)
 
+		bashrc_files = []
+
+		for profile in self._locations_manager.profiles_complex:
+			bashrc_files.append(os.path.join(profile.location,
+				'profile.bashrc'))
+			if profile in self._pbashrcdict:
+				cpdict = self._pbashrcdict[profile].get(cp)
+				if cpdict:
+					bashrc_matches = \
+						ordered_by_atom_specificity(cpdict, cpv_slot)
+					for x in bashrc_matches:
+						bashrc_files.extend(x)
+
+		self.configdict["BASHRC_FILES"] = bashrc_files
+
 		protected_pkg_keys = set(pkg_configdict)
 		protected_pkg_keys.discard('USE')
 
diff --git a/pym/portage/package/ebuild/doebuild.py b/pym/portage/package/ebuild/doebuild.py
index 01707ae..5224775 100644
--- a/pym/portage/package/ebuild/doebuild.py
+++ b/pym/portage/package/ebuild/doebuild.py
@@ -335,7 +335,9 @@ def doebuild_environment(myebuild, mydo, myroot=None, settings=None,
 	mysettings["ECLASSDIR"]   = mysettings["PORTDIR"]+"/eclass"
 	mysettings["SANDBOX_LOG"] = mycpv.replace("/", "_-_")
 
-	mysettings["PROFILE_PATHS"] = "\n".join(mysettings.profiles)
+	mysettings["PORTAGE_BASHRC_FILES"] = \
+		"\n".join(mysettings.configdict["BASHRC_FILES"])
+
 	mysettings["P"]  = mysplit[0]+"-"+mysplit[1]
 	mysettings["PN"] = mysplit[0]
 	mysettings["PV"] = mysplit[1]
diff --git a/pym/portage/repository/config.py b/pym/portage/repository/config.py
index 5e0d055..bef643d 100644
--- a/pym/portage/repository/config.py
+++ b/pym/portage/repository/config.py
@@ -40,7 +40,7 @@ if sys.hexversion >= 0x3000000:
 _invalid_path_char_re = re.compile(r'[^a-zA-Z0-9._\-+:/]')
 
 _valid_profile_formats = frozenset(
-	['pms', 'portage-1', 'portage-2'])
+	['pms', 'portage-1', 'portage-2', 'profile-bashrcs'])
 
 _portage1_profiles_allow_directories = frozenset(
 	["portage-1-compat", "portage-1", 'portage-2'])
-- 
2.1.0.rc2.206.gedb03e5

[gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Bertrand SIMONNET]

Hi all,

Here is the implementation of package.bashrc, split into three patches:
* refactoring of ebuild.sh (same as the refactor earlier in this thread)
* adding profile-formats to profile_complex (so that it can be used in
  ebuild/config.py)
* package.bashrc change (including documentation)

Let me know what you think :)

Thanks,
Bertrand

[gentoo-portage-dev] [PATCH 1/3] Refactor bashrc scripts sourcing

[Bertrand SIMONNET]

Creates two new helper functions __try_source and __source_env_files to simplify
__source_all_bashrcs.
---
 bin/ebuild.sh          | 75 +++++++++++++++++++++++++++++++++-----------------
 bin/save-ebuild-env.sh |  1 +
 2 files changed, 51 insertions(+), 25 deletions(-)

diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index be044e0..14cc321 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -375,39 +375,64 @@ __source_all_bashrcs() {
 		done
 	fi
 
-	if [ -r "${PORTAGE_BASHRC}" ] ; then
-		if [ "$PORTAGE_DEBUG" != "1" ] || [ "${-/x/}" != "$-" ]; then
-			source "${PORTAGE_BASHRC}"
-		else
-			set -x
-			source "${PORTAGE_BASHRC}"
-			set +x
-		fi
-	fi
+	# The user's bashrc is the ONLY non-portage bit of code
+	# that can change shopts without a QA violation.
+	__try_source --no-qa "${PORTAGE_BASHRC}"
 
 	if [[ $EBUILD_PHASE != depend ]] ; then
-		# The user's bashrc is the ONLY non-portage bit of code that can
-		# change shopts without a QA violation.
-		for x in "${PM_EBUILD_HOOK_DIR}"/${CATEGORY}/{${PN},${PN}:${SLOT%/*},${P},${PF}}; do
-			if [ -r "${x}" ]; then
-				# If $- contains x, then tracing has already been enabled
-				# elsewhere for some reason. We preserve it's state so as
-				# not to interfere.
-				if [ "$PORTAGE_DEBUG" != "1" ] || [ "${-/x/}" != "$-" ]; then
-					source "${x}"
-				else
-					set -x
-					source "${x}"
-					set +x
-				fi
-			fi
-		done
+		__source_env_files --no-qa "${PM_EBUILD_HOOK_DIR}"
 	fi
 
 	[ ! -z "${OCC}" ] && export CC="${OCC}"
 	[ ! -z "${OCXX}" ] && export CXX="${OCXX}"
 }
 
+# @FUNCTION: __source_env_files
+# @USAGE: [--no-qa] <ENV_DIRECTORY>
+# @DESCRIPTION:
+# Source the files relevant to the current package from the given path.
+# If --no-qa is specified, use source instead of __qa_source to source the
+# files.
+__source_env_files() {
+	local argument=()
+	if [[ $1 == --no-qa ]]; then
+		argument=( --no-qa )
+	shift
+	fi
+	for x in "${1}"/${CATEGORY}/{${PN},${PN}:${SLOT%/*},${P},${PF}}; do
+		__try_source "${argument[@]}" "${x}"
+	done
+}
+
+# @FUNCTION: __try_source
+# @USAGE: [--no-qa] <FILE>
+# @DESCRIPTION:
+# If the path given as argument exists, source the file while preserving
+# $-.
+# If --no-qa is specified, source the file with source instead of __qa_source.
+__try_source() {
+	local qa=true
+	if [[ $1 == --no-qa ]]; then
+		qa=false
+		shift
+	fi
+	if [[ -r "$1" ]]; then
+		local debug_on=false
+		if [[ "$PORTAGE_DEBUG" == "1" ]] && [[ "${-/x/}" == "$-" ]]; then
+			debug_on=true
+		fi
+		$debug_on && set -x
+		# If $- contains x, then tracing has already been enabled
+		# elsewhere for some reason. We preserve it's state so as
+		# not to interfere.
+		if [[ ${qa} ]]; then
+			source "${1}"
+		else
+			__qa_source "${1}"
+		fi
+		$debug_on && set +x
+	fi
+}
 # === === === === === === === === === === === === === === === === === ===
 # === === === === === functions end, main part begins === === === === ===
 # === === === === === === === === === === === === === === === === === ===
diff --git a/bin/save-ebuild-env.sh b/bin/save-ebuild-env.sh
index 98cff83..f114c48 100644
--- a/bin/save-ebuild-env.sh
+++ b/bin/save-ebuild-env.sh
@@ -75,6 +75,7 @@ __save_ebuild_env() {
 		__ebuild_main __ebuild_phase __ebuild_phase_with_hooks \
 		__ebuild_arg_to_phase __ebuild_phase_funcs default \
 		__unpack_tar __unset_colors \
+		__source_env_files __try_source \
 		${QA_INTERCEPTORS}
 
 	___eapi_has_usex && unset -f usex
-- 
2.1.0.rc2.206.gedb03e5

[gentoo-portage-dev] [PATCH 2/3] Add profile-formats to profile_complex

[Bertrand SIMONNET]

---
 pym/portage/package/ebuild/_config/LocationsManager.py | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/pym/portage/package/ebuild/_config/LocationsManager.py b/pym/portage/package/ebuild/_config/LocationsManager.py
index 4427f1d..8bf321c 100644
--- a/pym/portage/package/ebuild/_config/LocationsManager.py
+++ b/pym/portage/package/ebuild/_config/LocationsManager.py
@@ -31,7 +31,7 @@ _PORTAGE1_DIRECTORIES = frozenset([
 	'use.mask', 'use.force'])
 
 _profile_node = collections.namedtuple('_profile_node',
-	'location portage1_directories user_config')
+	'location portage1_directories user_config profile_formats')
 
 _allow_parent_colon = frozenset(
 	["portage-2"])
@@ -132,7 +132,7 @@ class LocationsManager(object):
 				self.user_profile_dir = custom_prof
 				self.profiles.append(custom_prof)
 				self.profiles_complex.append(
-					_profile_node(custom_prof, True, True))
+					_profile_node(custom_prof, True, True, ()))
 			del custom_prof
 
 		self.profiles = tuple(self.profiles)
@@ -151,6 +151,7 @@ class LocationsManager(object):
 		allow_parent_colon = True
 		repo_loc = None
 		compat_mode = False
+		current_formats = ()
 
 		eapi_file = os.path.join(currentPath, "eapi")
 		eapi = "0"
@@ -183,6 +184,8 @@ class LocationsManager(object):
 				layout_data['profile-formats'] == ('portage-1-compat',)
 			allow_parent_colon = any(x in _allow_parent_colon
 				for x in layout_data['profile-formats'])
+			current_formats = tuple(layout_data['profile-formats'])
+
 
 		if compat_mode:
 			offenders = _PORTAGE1_DIRECTORIES.intersection(os.listdir(currentPath))
@@ -233,7 +236,8 @@ class LocationsManager(object):
 
 		self.profiles.append(currentPath)
 		self.profiles_complex.append(
-			_profile_node(currentPath, allow_directories, False))
+			_profile_node(currentPath, allow_directories, False,
+				current_formats))
 
 	def _expand_parent_colon(self, parentsFile, parentPath,
 		repo_loc, repositories):
-- 
2.1.0.rc2.206.gedb03e5

[gentoo-portage-dev] [PATCH 3/3] package.bashrc: per profile, per-package bashrc mechanism

[Bertrand SIMONNET]

Profiles can define per-package bashrc files to be sourced before emerging.
Each line in package.bashrc must be an atom name then a list of space-delimited
bashrc files (stored in $profile/bashrc/).
---
 bin/ebuild.sh                                      |  6 ++--
 bin/phase-functions.sh                             |  2 +-
 bin/save-ebuild-env.sh                             |  2 +-
 man/portage.5                                      | 30 +++++++++++++++++-
 .../package/ebuild/_config/special_env_vars.py     |  4 +--
 pym/portage/package/ebuild/config.py               | 36 ++++++++++++++++++++++
 pym/portage/package/ebuild/doebuild.py             |  4 ++-
 pym/portage/repository/config.py                   |  2 +-
 8 files changed, 76 insertions(+), 10 deletions(-)

diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index 14cc321..50909e1 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -368,10 +368,10 @@ __source_all_bashrcs() {
 		# source the existing profile.bashrcs.
 		save_IFS
 		IFS=$'\n'
-		local path_array=($PROFILE_PATHS)
+		local bashenv_files=($PORTAGE_BASHRC_FILES)
 		restore_IFS
-		for x in "${path_array[@]}" ; do
-			[ -f "$x/profile.bashrc" ] && __qa_source "$x/profile.bashrc"
+		for x in "${bashenv_files[@]}" ; do
+			__try_source "${x}"
 		done
 	fi
 
diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index f39a024..5dff3bb 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -31,7 +31,7 @@ PORTAGE_READONLY_VARS="D EBUILD EBUILD_PHASE EBUILD_PHASE_FUNC \
 	PORTAGE_TMPDIR PORTAGE_UPDATE_ENV PORTAGE_USERNAME \
 	PORTAGE_VERBOSE PORTAGE_WORKDIR_MODE PORTAGE_XATTR_EXCLUDE \
 	PORTDIR \
-	PROFILE_PATHS REPLACING_VERSIONS REPLACED_BY_VERSION T WORKDIR \
+	REPLACING_VERSIONS REPLACED_BY_VERSION T WORKDIR \
 	__PORTAGE_HELPER __PORTAGE_TEST_HARDLINK_LOCKS"
 
 PORTAGE_SAVED_READONLY_VARS="A CATEGORY P PF PN PR PV PVR"
diff --git a/bin/save-ebuild-env.sh b/bin/save-ebuild-env.sh
index f114c48..1a684b9 100644
--- a/bin/save-ebuild-env.sh
+++ b/bin/save-ebuild-env.sh
@@ -97,7 +97,7 @@ __save_ebuild_env() {
 		GOOD HILITE HOME \
 		LAST_E_CMD LAST_E_LEN LD_PRELOAD MISC_FUNCTIONS_ARGS MOPREFIX \
 		NOCOLOR NORMAL PKGDIR PKGUSE PKG_LOGDIR PKG_TMPDIR \
-		PORTAGE_BASHRCS_SOURCED PORTAGE_COMPRESS \
+		PORTAGE_BASHRC_FILES PORTAGE_BASHRCS_SOURCED PORTAGE_COMPRESS \
 		PORTAGE_COMPRESS_EXCLUDE_SUFFIXES \
 		PORTAGE_DOHTML_UNWARNED_SKIPPED_EXTENSIONS \
 		PORTAGE_DOHTML_UNWARNED_SKIPPED_FILES \
diff --git a/man/portage.5 b/man/portage.5
index e399f0f..309e259 100644
--- a/man/portage.5
+++ b/man/portage.5
@@ -24,6 +24,7 @@ make.defaults
 packages
 packages.build
 package.accept_keywords
+package.bashrc
 package.keywords
 package.mask
 package.provided
@@ -358,6 +359,31 @@ a '\-'.
 A list of packages (one per line) that make up a stage1 tarball.  Really only
 useful for stage builders.
 .TP
+.BR package.bashrc
+Per-package bashrc mechanism.  Contains a list of bashrc files to be sourced
+before emerging a given atom.  The bashrc files must be stored in bashrc/, in
+the profile directory.
+
+.I Note:
+.nf
+\- The bashrc files will be sourced after profile.bashrc for the same profile.
+\- profile-formats in metadata/layout.conf must contain profile-bashrcs for this
+to be enabled.
+.fi
+
+.I Format:
+.nf
+\- comments begin with # (no inline comments).
+\- one atom per line with space-delimited list of bashrc files.
+.fi
+
+.I Example:
+.nf
+# By setting INSTALL_MASK in bashrc/nostandardconf.conf, we can avoid installing
+# the standard configuration and enable another package to install it.
+net-misc/dhcp nostardardconf.conf
+.fi
+.TP
 .BR package.provided
 A list of packages (one per line) that portage should assume have been
 provided.  Useful for porting to non-Linux systems. Basically, it's a
@@ -1047,11 +1073,13 @@ The default setting for repoman's --echangelog option.
 The cache formats supported in the metadata tree.  There is the old "pms" format
 and the newer/faster "md5-dict" format.  Default is to detect dirs.
 .TP
-.BR profile\-formats " = [pms|portage-1|portage-2]"
+.BR profile\-formats " = [pms|portage-1|portage-2|profile-bashrcs]"
 Control functionality available to profiles in this repo such as which files
 may be dirs, or the syntax available in parent files.  Use "portage-2" if you're
 unsure.  The default is "portage-1-compat" mode which is meant to be compatible
 with old profiles, but is not allowed to be opted into directly.
+Setting profile-bashrcs will enable the per-profile bashrc mechanism
+\fBpackage.bashrc\fR.
 .RE
 .RE
 
diff --git a/pym/portage/package/ebuild/_config/special_env_vars.py b/pym/portage/package/ebuild/_config/special_env_vars.py
index 74fedd6..387f4ae 100644
--- a/pym/portage/package/ebuild/_config/special_env_vars.py
+++ b/pym/portage/package/ebuild/_config/special_env_vars.py
@@ -49,7 +49,7 @@ environ_whitelist += [
 	"FEATURES", "FILESDIR", "HOME", "MERGE_TYPE", "NOCOLOR", "PATH",
 	"PKGDIR",
 	"PKGUSE", "PKG_LOGDIR", "PKG_TMPDIR",
-	"PORTAGE_ACTUAL_DISTDIR", "PORTAGE_ARCHLIST",
+	"PORTAGE_ACTUAL_DISTDIR", "PORTAGE_ARCHLIST", "PORTAGE_BASHRC_FILES",
 	"PORTAGE_BASHRC", "PM_EBUILD_HOOK_DIR",
 	"PORTAGE_BINPKG_FILE", "PORTAGE_BINPKG_TAR_OPTS",
 	"PORTAGE_BINPKG_TMPFILE",
@@ -74,7 +74,7 @@ environ_whitelist += [
 	"PORTAGE_SIGPIPE_STATUS",
 	"PORTAGE_TMPDIR", "PORTAGE_UPDATE_ENV", "PORTAGE_USERNAME",
 	"PORTAGE_VERBOSE", "PORTAGE_WORKDIR_MODE", "PORTAGE_XATTR_EXCLUDE",
-	"PORTDIR", "PORTDIR_OVERLAY", "PREROOTPATH", "PROFILE_PATHS",
+	"PORTDIR", "PORTDIR_OVERLAY", "PREROOTPATH",
 	"REPLACING_VERSIONS", "REPLACED_BY_VERSION",
 	"ROOT", "ROOTPATH", "T", "TMP", "TMPDIR",
 	"USE_EXPAND", "USE_ORDER", "WORKDIR",
diff --git a/pym/portage/package/ebuild/config.py b/pym/portage/package/ebuild/config.py
index f639e14..183627f 100644
--- a/pym/portage/package/ebuild/config.py
+++ b/pym/portage/package/ebuild/config.py
@@ -316,6 +316,7 @@ class config(object):
 			self._accept_restrict = copy.deepcopy(clone._accept_restrict)
 			self._paccept_restrict = copy.deepcopy(clone._paccept_restrict)
 			self._penvdict = copy.deepcopy(clone._penvdict)
+			self._pbashrcdict = copy.deepcopy(clone._pbashrcdict)
 			self._expand_map = copy.deepcopy(clone._expand_map)
 
 		else:
@@ -661,6 +662,7 @@ class config(object):
 			self._ppropertiesdict = portage.dep.ExtendedAtomDict(dict)
 			self._paccept_restrict = portage.dep.ExtendedAtomDict(dict)
 			self._penvdict = portage.dep.ExtendedAtomDict(dict)
+			self._pbashrcdict = {}
 
 			self._repo_make_defaults = {}
 			for repo in self.repositories.repos_with_profiles():
@@ -742,6 +744,25 @@ class config(object):
 				for k, v in penvdict.items():
 					self._penvdict.setdefault(k.cp, {})[k] = v
 
+				# package.bashrc
+				for profile in profiles_complex:
+					if not 'profile-bashrcs' in profile.profile_formats:
+						continue
+					self._pbashrcdict[profile] = \
+						portage.dep.ExtendedAtomDict(dict)
+					bashrc = grabdict_package(os.path.join(profile.location,
+						"package.bashrc"), recursive=1, allow_wildcard=True,
+								allow_repo=True, verify_eapi=False)
+					if not bashrc:
+						continue
+
+					for k, v in bashrc.items():
+						envfiles = [os.path.join(profile.location,
+							"bashrc",
+							envname) for envname in v]
+						self._pbashrcdict[profile].setdefault(k.cp, {})\
+							.setdefault(k, []).extend(envfiles)
+
 			#getting categories from an external file now
 			self.categories = [grabfile(os.path.join(x, "categories")) \
 				for x in locations_manager.profile_and_user_locations]
@@ -1501,6 +1522,21 @@ class config(object):
 				for x in penv_matches:
 					self._penv.extend(x)
 
+		bashrc_files = []
+
+		for profile in self._locations_manager.profiles_complex:
+			bashrc_files.append(os.path.join(profile.location,
+				'profile.bashrc'))
+			if profile in self._pbashrcdict:
+				cpdict = self._pbashrcdict[profile].get(cp)
+				if cpdict:
+					bashrc_matches = \
+						ordered_by_atom_specificity(cpdict, cpv_slot)
+					for x in bashrc_matches:
+						bashrc_files.extend(x)
+
+		self.configdict["BASHRC_FILES"] = bashrc_files
+
 		protected_pkg_keys = set(pkg_configdict)
 		protected_pkg_keys.discard('USE')
 
diff --git a/pym/portage/package/ebuild/doebuild.py b/pym/portage/package/ebuild/doebuild.py
index 01707ae..5224775 100644
--- a/pym/portage/package/ebuild/doebuild.py
+++ b/pym/portage/package/ebuild/doebuild.py
@@ -335,7 +335,9 @@ def doebuild_environment(myebuild, mydo, myroot=None, settings=None,
 	mysettings["ECLASSDIR"]   = mysettings["PORTDIR"]+"/eclass"
 	mysettings["SANDBOX_LOG"] = mycpv.replace("/", "_-_")
 
-	mysettings["PROFILE_PATHS"] = "\n".join(mysettings.profiles)
+	mysettings["PORTAGE_BASHRC_FILES"] = \
+		"\n".join(mysettings.configdict["BASHRC_FILES"])
+
 	mysettings["P"]  = mysplit[0]+"-"+mysplit[1]
 	mysettings["PN"] = mysplit[0]
 	mysettings["PV"] = mysplit[1]
diff --git a/pym/portage/repository/config.py b/pym/portage/repository/config.py
index 5e0d055..bef643d 100644
--- a/pym/portage/repository/config.py
+++ b/pym/portage/repository/config.py
@@ -40,7 +40,7 @@ if sys.hexversion >= 0x3000000:
 _invalid_path_char_re = re.compile(r'[^a-zA-Z0-9._\-+:/]')
 
 _valid_profile_formats = frozenset(
-	['pms', 'portage-1', 'portage-2'])
+	['pms', 'portage-1', 'portage-2', 'profile-bashrcs'])
 
 _portage1_profiles_allow_directories = frozenset(
 	["portage-1-compat", "portage-1", 'portage-2'])
-- 
2.1.0.rc2.206.gedb03e5

Re: [gentoo-portage-dev] [PATCH 3/3] package.bashrc: per profile, per-package bashrc mechanism

[Zac Medico]

On 09/29/2014 05:12 PM, Bertrand SIMONNET wrote:
> +.I Example:
> +.nf
> +# By setting INSTALL_MASK in bashrc/nostandardconf.conf, we can avoid installing
> +# the standard configuration and enable another package to install it.
> +net-misc/dhcp nostardardconf.conf

In order to ensure that INSTALL_MASK set via bashrc will always works as
intended, you'll have to modify the INSTALL_MASK logic inside the
dblink.treewalk method:

if self.settings.get("INSTALL_MASK") or \
	"nodoc" in self.settings.features or \
	"noinfo" in self.settings.features or \
	"noman" in self.settings.features:
	# Apply INSTALL_MASK before collision-protect, since it may
	# be useful to avoid collisions in some scenarios.
	phase = MiscFunctionsProcess(background=False,
		commands=["preinst_mask"], phase="preinst",
		scheduler=self._scheduler, settings=self.settings)
	phase.start()
	phase.wait()

We could simply run the preinst_mask function unconditionally here,
since we can't tell whether or not the bashrc contains an INSTALL_MASK
setting.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] per package environment: generalize the mechanism to be profile specific

[Zac Medico]

On 09/29/2014 05:12 PM, Bertrand SIMONNET wrote:
> Hi all,
> 
> Here is the implementation of package.bashrc, split into three patches:
> * refactoring of ebuild.sh (same as the refactor earlier in this thread)
> * adding profile-formats to profile_complex (so that it can be used in
>   ebuild/config.py)
> * package.bashrc change (including documentation)
> 
> Let me know what you think :)
> 
> Thanks,
> Bertrand

It all looks very good to me, aside from the issue about INSTALL_MASK.
-- 
Thanks,
Zac

[gentoo-portage-dev] [PATCH] package.bashrc: per profile, per-package bashrc mechanism

[Bertrand SIMONNET]

Profiles can define per-package bashrc files to be sourced before emerging.
Each line in package.bashrc must be an atom name then a list of space-delimited
bashrc files (stored in $profile/bashrc/).
---
 bin/ebuild.sh                                      |  6 ++--
 bin/phase-functions.sh                             |  2 +-
 bin/save-ebuild-env.sh                             |  2 +-
 man/portage.5                                      | 30 +++++++++++++++++-
 pym/portage/dbapi/vartree.py                       | 20 ++++++------
 .../package/ebuild/_config/special_env_vars.py     |  4 +--
 pym/portage/package/ebuild/config.py               | 36 ++++++++++++++++++++++
 pym/portage/package/ebuild/doebuild.py             |  4 ++-
 pym/portage/repository/config.py                   |  2 +-
 9 files changed, 85 insertions(+), 21 deletions(-)

diff --git a/bin/ebuild.sh b/bin/ebuild.sh
index 14cc321..50909e1 100755
--- a/bin/ebuild.sh
+++ b/bin/ebuild.sh
@@ -368,10 +368,10 @@ __source_all_bashrcs() {
 		# source the existing profile.bashrcs.
 		save_IFS
 		IFS=$'\n'
-		local path_array=($PROFILE_PATHS)
+		local bashenv_files=($PORTAGE_BASHRC_FILES)
 		restore_IFS
-		for x in "${path_array[@]}" ; do
-			[ -f "$x/profile.bashrc" ] && __qa_source "$x/profile.bashrc"
+		for x in "${bashenv_files[@]}" ; do
+			__try_source "${x}"
 		done
 	fi
 
diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
index f39a024..5dff3bb 100644
--- a/bin/phase-functions.sh
+++ b/bin/phase-functions.sh
@@ -31,7 +31,7 @@ PORTAGE_READONLY_VARS="D EBUILD EBUILD_PHASE EBUILD_PHASE_FUNC \
 	PORTAGE_TMPDIR PORTAGE_UPDATE_ENV PORTAGE_USERNAME \
 	PORTAGE_VERBOSE PORTAGE_WORKDIR_MODE PORTAGE_XATTR_EXCLUDE \
 	PORTDIR \
-	PROFILE_PATHS REPLACING_VERSIONS REPLACED_BY_VERSION T WORKDIR \
+	REPLACING_VERSIONS REPLACED_BY_VERSION T WORKDIR \
 	__PORTAGE_HELPER __PORTAGE_TEST_HARDLINK_LOCKS"
 
 PORTAGE_SAVED_READONLY_VARS="A CATEGORY P PF PN PR PV PVR"
diff --git a/bin/save-ebuild-env.sh b/bin/save-ebuild-env.sh
index f114c48..1a684b9 100644
--- a/bin/save-ebuild-env.sh
+++ b/bin/save-ebuild-env.sh
@@ -97,7 +97,7 @@ __save_ebuild_env() {
 		GOOD HILITE HOME \
 		LAST_E_CMD LAST_E_LEN LD_PRELOAD MISC_FUNCTIONS_ARGS MOPREFIX \
 		NOCOLOR NORMAL PKGDIR PKGUSE PKG_LOGDIR PKG_TMPDIR \
-		PORTAGE_BASHRCS_SOURCED PORTAGE_COMPRESS \
+		PORTAGE_BASHRC_FILES PORTAGE_BASHRCS_SOURCED PORTAGE_COMPRESS \
 		PORTAGE_COMPRESS_EXCLUDE_SUFFIXES \
 		PORTAGE_DOHTML_UNWARNED_SKIPPED_EXTENSIONS \
 		PORTAGE_DOHTML_UNWARNED_SKIPPED_FILES \
diff --git a/man/portage.5 b/man/portage.5
index e399f0f..309e259 100644
--- a/man/portage.5
+++ b/man/portage.5
@@ -24,6 +24,7 @@ make.defaults
 packages
 packages.build
 package.accept_keywords
+package.bashrc
 package.keywords
 package.mask
 package.provided
@@ -358,6 +359,31 @@ a '\-'.
 A list of packages (one per line) that make up a stage1 tarball.  Really only
 useful for stage builders.
 .TP
+.BR package.bashrc
+Per-package bashrc mechanism.  Contains a list of bashrc files to be sourced
+before emerging a given atom.  The bashrc files must be stored in bashrc/, in
+the profile directory.
+
+.I Note:
+.nf
+\- The bashrc files will be sourced after profile.bashrc for the same profile.
+\- profile-formats in metadata/layout.conf must contain profile-bashrcs for this
+to be enabled.
+.fi
+
+.I Format:
+.nf
+\- comments begin with # (no inline comments).
+\- one atom per line with space-delimited list of bashrc files.
+.fi
+
+.I Example:
+.nf
+# By setting INSTALL_MASK in bashrc/nostandardconf.conf, we can avoid installing
+# the standard configuration and enable another package to install it.
+net-misc/dhcp nostardardconf.conf
+.fi
+.TP
 .BR package.provided
 A list of packages (one per line) that portage should assume have been
 provided.  Useful for porting to non-Linux systems. Basically, it's a
@@ -1047,11 +1073,13 @@ The default setting for repoman's --echangelog option.
 The cache formats supported in the metadata tree.  There is the old "pms" format
 and the newer/faster "md5-dict" format.  Default is to detect dirs.
 .TP
-.BR profile\-formats " = [pms|portage-1|portage-2]"
+.BR profile\-formats " = [pms|portage-1|portage-2|profile-bashrcs]"
 Control functionality available to profiles in this repo such as which files
 may be dirs, or the syntax available in parent files.  Use "portage-2" if you're
 unsure.  The default is "portage-1-compat" mode which is meant to be compatible
 with old profiles, but is not allowed to be opted into directly.
+Setting profile-bashrcs will enable the per-profile bashrc mechanism
+\fBpackage.bashrc\fR.
 .RE
 .RE
 
diff --git a/pym/portage/dbapi/vartree.py b/pym/portage/dbapi/vartree.py
index 5b947dd..72c15db 100644
--- a/pym/portage/dbapi/vartree.py
+++ b/pym/portage/dbapi/vartree.py
@@ -3663,17 +3663,15 @@ class dblink(object):
 					max_dblnk = dblnk
 			self._installed_instance = max_dblnk
 
-		if self.settings.get("INSTALL_MASK") or \
-			"nodoc" in self.settings.features or \
-			"noinfo" in self.settings.features or \
-			"noman" in self.settings.features:
-			# Apply INSTALL_MASK before collision-protect, since it may
-			# be useful to avoid collisions in some scenarios.
-			phase = MiscFunctionsProcess(background=False,
-				commands=["preinst_mask"], phase="preinst",
-				scheduler=self._scheduler, settings=self.settings)
-			phase.start()
-			phase.wait()
+		# Apply INSTALL_MASK before collision-protect, since it may
+		# be useful to avoid collisions in some scenarios.
+		# We cannot detect if this is needed or not here as INSTALL_MASK can be
+		# modified by bashrc files.
+		phase = MiscFunctionsProcess(background=False,
+			commands=["preinst_mask"], phase="preinst",
+			scheduler=self._scheduler, settings=self.settings)
+		phase.start()
+		phase.wait()
 
 		# We check for unicode encoding issues after src_install. However,
 		# the check must be repeated here for binary packages (it's
diff --git a/pym/portage/package/ebuild/_config/special_env_vars.py b/pym/portage/package/ebuild/_config/special_env_vars.py
index 74fedd6..387f4ae 100644
--- a/pym/portage/package/ebuild/_config/special_env_vars.py
+++ b/pym/portage/package/ebuild/_config/special_env_vars.py
@@ -49,7 +49,7 @@ environ_whitelist += [
 	"FEATURES", "FILESDIR", "HOME", "MERGE_TYPE", "NOCOLOR", "PATH",
 	"PKGDIR",
 	"PKGUSE", "PKG_LOGDIR", "PKG_TMPDIR",
-	"PORTAGE_ACTUAL_DISTDIR", "PORTAGE_ARCHLIST",
+	"PORTAGE_ACTUAL_DISTDIR", "PORTAGE_ARCHLIST", "PORTAGE_BASHRC_FILES",
 	"PORTAGE_BASHRC", "PM_EBUILD_HOOK_DIR",
 	"PORTAGE_BINPKG_FILE", "PORTAGE_BINPKG_TAR_OPTS",
 	"PORTAGE_BINPKG_TMPFILE",
@@ -74,7 +74,7 @@ environ_whitelist += [
 	"PORTAGE_SIGPIPE_STATUS",
 	"PORTAGE_TMPDIR", "PORTAGE_UPDATE_ENV", "PORTAGE_USERNAME",
 	"PORTAGE_VERBOSE", "PORTAGE_WORKDIR_MODE", "PORTAGE_XATTR_EXCLUDE",
-	"PORTDIR", "PORTDIR_OVERLAY", "PREROOTPATH", "PROFILE_PATHS",
+	"PORTDIR", "PORTDIR_OVERLAY", "PREROOTPATH",
 	"REPLACING_VERSIONS", "REPLACED_BY_VERSION",
 	"ROOT", "ROOTPATH", "T", "TMP", "TMPDIR",
 	"USE_EXPAND", "USE_ORDER", "WORKDIR",
diff --git a/pym/portage/package/ebuild/config.py b/pym/portage/package/ebuild/config.py
index f639e14..183627f 100644
--- a/pym/portage/package/ebuild/config.py
+++ b/pym/portage/package/ebuild/config.py
@@ -316,6 +316,7 @@ class config(object):
 			self._accept_restrict = copy.deepcopy(clone._accept_restrict)
 			self._paccept_restrict = copy.deepcopy(clone._paccept_restrict)
 			self._penvdict = copy.deepcopy(clone._penvdict)
+			self._pbashrcdict = copy.deepcopy(clone._pbashrcdict)
 			self._expand_map = copy.deepcopy(clone._expand_map)
 
 		else:
@@ -661,6 +662,7 @@ class config(object):
 			self._ppropertiesdict = portage.dep.ExtendedAtomDict(dict)
 			self._paccept_restrict = portage.dep.ExtendedAtomDict(dict)
 			self._penvdict = portage.dep.ExtendedAtomDict(dict)
+			self._pbashrcdict = {}
 
 			self._repo_make_defaults = {}
 			for repo in self.repositories.repos_with_profiles():
@@ -742,6 +744,25 @@ class config(object):
 				for k, v in penvdict.items():
 					self._penvdict.setdefault(k.cp, {})[k] = v
 
+				# package.bashrc
+				for profile in profiles_complex:
+					if not 'profile-bashrcs' in profile.profile_formats:
+						continue
+					self._pbashrcdict[profile] = \
+						portage.dep.ExtendedAtomDict(dict)
+					bashrc = grabdict_package(os.path.join(profile.location,
+						"package.bashrc"), recursive=1, allow_wildcard=True,
+								allow_repo=True, verify_eapi=False)
+					if not bashrc:
+						continue
+
+					for k, v in bashrc.items():
+						envfiles = [os.path.join(profile.location,
+							"bashrc",
+							envname) for envname in v]
+						self._pbashrcdict[profile].setdefault(k.cp, {})\
+							.setdefault(k, []).extend(envfiles)
+
 			#getting categories from an external file now
 			self.categories = [grabfile(os.path.join(x, "categories")) \
 				for x in locations_manager.profile_and_user_locations]
@@ -1501,6 +1522,21 @@ class config(object):
 				for x in penv_matches:
 					self._penv.extend(x)
 
+		bashrc_files = []
+
+		for profile in self._locations_manager.profiles_complex:
+			bashrc_files.append(os.path.join(profile.location,
+				'profile.bashrc'))
+			if profile in self._pbashrcdict:
+				cpdict = self._pbashrcdict[profile].get(cp)
+				if cpdict:
+					bashrc_matches = \
+						ordered_by_atom_specificity(cpdict, cpv_slot)
+					for x in bashrc_matches:
+						bashrc_files.extend(x)
+
+		self.configdict["BASHRC_FILES"] = bashrc_files
+
 		protected_pkg_keys = set(pkg_configdict)
 		protected_pkg_keys.discard('USE')
 
diff --git a/pym/portage/package/ebuild/doebuild.py b/pym/portage/package/ebuild/doebuild.py
index 01707ae..5224775 100644
--- a/pym/portage/package/ebuild/doebuild.py
+++ b/pym/portage/package/ebuild/doebuild.py
@@ -335,7 +335,9 @@ def doebuild_environment(myebuild, mydo, myroot=None, settings=None,
 	mysettings["ECLASSDIR"]   = mysettings["PORTDIR"]+"/eclass"
 	mysettings["SANDBOX_LOG"] = mycpv.replace("/", "_-_")
 
-	mysettings["PROFILE_PATHS"] = "\n".join(mysettings.profiles)
+	mysettings["PORTAGE_BASHRC_FILES"] = \
+		"\n".join(mysettings.configdict["BASHRC_FILES"])
+
 	mysettings["P"]  = mysplit[0]+"-"+mysplit[1]
 	mysettings["PN"] = mysplit[0]
 	mysettings["PV"] = mysplit[1]
diff --git a/pym/portage/repository/config.py b/pym/portage/repository/config.py
index 5e0d055..bef643d 100644
--- a/pym/portage/repository/config.py
+++ b/pym/portage/repository/config.py
@@ -40,7 +40,7 @@ if sys.hexversion >= 0x3000000:
 _invalid_path_char_re = re.compile(r'[^a-zA-Z0-9._\-+:/]')
 
 _valid_profile_formats = frozenset(
-	['pms', 'portage-1', 'portage-2'])
+	['pms', 'portage-1', 'portage-2', 'profile-bashrcs'])
 
 _portage1_profiles_allow_directories = frozenset(
 	["portage-1-compat", "portage-1", 'portage-2'])
-- 
2.1.0.rc2.206.gedb03e5

[gentoo-portage-dev] Re: [PATCH 3/3] package.bashrc: per profile, per-package bashrc mechanism

[Bertrand Simonnet]

[-- Attachment #1: Type: text/plain, Size: 11927 bytes --]

Please discard the last serie of patches. git send-email was not configured
correctly and some commands  I typed yesterday are being send now.
Sorry about that.

Bertrand

On Mon, Sep 29, 2014 at 3:31 PM, Bertrand SIMONNET <bsimonnet@chromium.org>
wrote:

> Profiles can define per-package bashrc files to be sourced before emerging.
> Each line in package.bashrc must be an atom name then a list of
> space-delimited
> bashrc files (stored in $profile/bashrc/).
> ---
>  bin/ebuild.sh                                      |  6 ++--
>  bin/phase-functions.sh                             |  2 +-
>  bin/save-ebuild-env.sh                             |  2 +-
>  man/portage.5                                      | 30 +++++++++++++++++-
>  .../package/ebuild/_config/special_env_vars.py     |  4 +--
>  pym/portage/package/ebuild/config.py               | 36
> ++++++++++++++++++++++
>  pym/portage/package/ebuild/doebuild.py             |  4 ++-
>  pym/portage/repository/config.py                   |  2 +-
>  8 files changed, 76 insertions(+), 10 deletions(-)
>
> diff --git a/bin/ebuild.sh b/bin/ebuild.sh
> index 14cc321..50909e1 100755
> --- a/bin/ebuild.sh
> +++ b/bin/ebuild.sh
> @@ -368,10 +368,10 @@ __source_all_bashrcs() {
>                 # source the existing profile.bashrcs.
>                 save_IFS
>                 IFS=$'\n'
> -               local path_array=($PROFILE_PATHS)
> +               local bashenv_files=($PORTAGE_BASHRC_FILES)
>                 restore_IFS
> -               for x in "${path_array[@]}" ; do
> -                       [ -f "$x/profile.bashrc" ] && __qa_source
> "$x/profile.bashrc"
> +               for x in "${bashenv_files[@]}" ; do
> +                       __try_source "${x}"
>                 done
>         fi
>
> diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
> index f39a024..5dff3bb 100644
> --- a/bin/phase-functions.sh
> +++ b/bin/phase-functions.sh
> @@ -31,7 +31,7 @@ PORTAGE_READONLY_VARS="D EBUILD EBUILD_PHASE
> EBUILD_PHASE_FUNC \
>         PORTAGE_TMPDIR PORTAGE_UPDATE_ENV PORTAGE_USERNAME \
>         PORTAGE_VERBOSE PORTAGE_WORKDIR_MODE PORTAGE_XATTR_EXCLUDE \
>         PORTDIR \
> -       PROFILE_PATHS REPLACING_VERSIONS REPLACED_BY_VERSION T WORKDIR \
> +       REPLACING_VERSIONS REPLACED_BY_VERSION T WORKDIR \
>         __PORTAGE_HELPER __PORTAGE_TEST_HARDLINK_LOCKS"
>
>  PORTAGE_SAVED_READONLY_VARS="A CATEGORY P PF PN PR PV PVR"
> diff --git a/bin/save-ebuild-env.sh b/bin/save-ebuild-env.sh
> index f114c48..1a684b9 100644
> --- a/bin/save-ebuild-env.sh
> +++ b/bin/save-ebuild-env.sh
> @@ -97,7 +97,7 @@ __save_ebuild_env() {
>                 GOOD HILITE HOME \
>                 LAST_E_CMD LAST_E_LEN LD_PRELOAD MISC_FUNCTIONS_ARGS
> MOPREFIX \
>                 NOCOLOR NORMAL PKGDIR PKGUSE PKG_LOGDIR PKG_TMPDIR \
> -               PORTAGE_BASHRCS_SOURCED PORTAGE_COMPRESS \
> +               PORTAGE_BASHRC_FILES PORTAGE_BASHRCS_SOURCED
> PORTAGE_COMPRESS \
>                 PORTAGE_COMPRESS_EXCLUDE_SUFFIXES \
>                 PORTAGE_DOHTML_UNWARNED_SKIPPED_EXTENSIONS \
>                 PORTAGE_DOHTML_UNWARNED_SKIPPED_FILES \
> diff --git a/man/portage.5 b/man/portage.5
> index e399f0f..309e259 100644
> --- a/man/portage.5
> +++ b/man/portage.5
> @@ -24,6 +24,7 @@ make.defaults
>  packages
>  packages.build
>  package.accept_keywords
> +package.bashrc
>  package.keywords
>  package.mask
>  package.provided
> @@ -358,6 +359,31 @@ a '\-'.
>  A list of packages (one per line) that make up a stage1 tarball.  Really
> only
>  useful for stage builders.
>  .TP
> +.BR package.bashrc
> +Per-package bashrc mechanism.  Contains a list of bashrc files to be
> sourced
> +before emerging a given atom.  The bashrc files must be stored in
> bashrc/, in
> +the profile directory.
> +
> +.I Note:
> +.nf
> +\- The bashrc files will be sourced after profile.bashrc for the same
> profile.
> +\- profile-formats in metadata/layout.conf must contain profile-bashrcs
> for this
> +to be enabled.
> +.fi
> +
> +.I Format:
> +.nf
> +\- comments begin with # (no inline comments).
> +\- one atom per line with space-delimited list of bashrc files.
> +.fi
> +
> +.I Example:
> +.nf
> +# By setting INSTALL_MASK in bashrc/nostandardconf.conf, we can avoid
> installing
> +# the standard configuration and enable another package to install it.
> +net-misc/dhcp nostardardconf.conf
> +.fi
> +.TP
>  .BR package.provided
>  A list of packages (one per line) that portage should assume have been
>  provided.  Useful for porting to non-Linux systems. Basically, it's a
> @@ -1047,11 +1073,13 @@ The default setting for repoman's --echangelog
> option.
>  The cache formats supported in the metadata tree.  There is the old "pms"
> format
>  and the newer/faster "md5-dict" format.  Default is to detect dirs.
>  .TP
> -.BR profile\-formats " = [pms|portage-1|portage-2]"
> +.BR profile\-formats " = [pms|portage-1|portage-2|profile-bashrcs]"
>  Control functionality available to profiles in this repo such as which
> files
>  may be dirs, or the syntax available in parent files.  Use "portage-2" if
> you're
>  unsure.  The default is "portage-1-compat" mode which is meant to be
> compatible
>  with old profiles, but is not allowed to be opted into directly.
> +Setting profile-bashrcs will enable the per-profile bashrc mechanism
> +\fBpackage.bashrc\fR.
>  .RE
>  .RE
>
> diff --git a/pym/portage/package/ebuild/_config/special_env_vars.py
> b/pym/portage/package/ebuild/_config/special_env_vars.py
> index 74fedd6..387f4ae 100644
> --- a/pym/portage/package/ebuild/_config/special_env_vars.py
> +++ b/pym/portage/package/ebuild/_config/special_env_vars.py
> @@ -49,7 +49,7 @@ environ_whitelist += [
>         "FEATURES", "FILESDIR", "HOME", "MERGE_TYPE", "NOCOLOR", "PATH",
>         "PKGDIR",
>         "PKGUSE", "PKG_LOGDIR", "PKG_TMPDIR",
> -       "PORTAGE_ACTUAL_DISTDIR", "PORTAGE_ARCHLIST",
> +       "PORTAGE_ACTUAL_DISTDIR", "PORTAGE_ARCHLIST",
> "PORTAGE_BASHRC_FILES",
>         "PORTAGE_BASHRC", "PM_EBUILD_HOOK_DIR",
>         "PORTAGE_BINPKG_FILE", "PORTAGE_BINPKG_TAR_OPTS",
>         "PORTAGE_BINPKG_TMPFILE",
> @@ -74,7 +74,7 @@ environ_whitelist += [
>         "PORTAGE_SIGPIPE_STATUS",
>         "PORTAGE_TMPDIR", "PORTAGE_UPDATE_ENV", "PORTAGE_USERNAME",
>         "PORTAGE_VERBOSE", "PORTAGE_WORKDIR_MODE", "PORTAGE_XATTR_EXCLUDE",
> -       "PORTDIR", "PORTDIR_OVERLAY", "PREROOTPATH", "PROFILE_PATHS",
> +       "PORTDIR", "PORTDIR_OVERLAY", "PREROOTPATH",
>         "REPLACING_VERSIONS", "REPLACED_BY_VERSION",
>         "ROOT", "ROOTPATH", "T", "TMP", "TMPDIR",
>         "USE_EXPAND", "USE_ORDER", "WORKDIR",
> diff --git a/pym/portage/package/ebuild/config.py
> b/pym/portage/package/ebuild/config.py
> index f639e14..183627f 100644
> --- a/pym/portage/package/ebuild/config.py
> +++ b/pym/portage/package/ebuild/config.py
> @@ -316,6 +316,7 @@ class config(object):
>                         self._accept_restrict =
> copy.deepcopy(clone._accept_restrict)
>                         self._paccept_restrict =
> copy.deepcopy(clone._paccept_restrict)
>                         self._penvdict = copy.deepcopy(clone._penvdict)
> +                       self._pbashrcdict =
> copy.deepcopy(clone._pbashrcdict)
>                         self._expand_map = copy.deepcopy(clone._expand_map)
>
>                 else:
> @@ -661,6 +662,7 @@ class config(object):
>                         self._ppropertiesdict =
> portage.dep.ExtendedAtomDict(dict)
>                         self._paccept_restrict =
> portage.dep.ExtendedAtomDict(dict)
>                         self._penvdict = portage.dep.ExtendedAtomDict(dict)
> +                       self._pbashrcdict = {}
>
>                         self._repo_make_defaults = {}
>                         for repo in
> self.repositories.repos_with_profiles():
> @@ -742,6 +744,25 @@ class config(object):
>                                 for k, v in penvdict.items():
>                                         self._penvdict.setdefault(k.cp,
> {})[k] = v
>
> +                               # package.bashrc
> +                               for profile in profiles_complex:
> +                                       if not 'profile-bashrcs' in
> profile.profile_formats:
> +                                               continue
> +                                       self._pbashrcdict[profile] = \
> +
>  portage.dep.ExtendedAtomDict(dict)
> +                                       bashrc =
> grabdict_package(os.path.join(profile.location,
> +                                               "package.bashrc"),
> recursive=1, allow_wildcard=True,
> +
>  allow_repo=True, verify_eapi=False)
> +                                       if not bashrc:
> +                                               continue
> +
> +                                       for k, v in bashrc.items():
> +                                               envfiles =
> [os.path.join(profile.location,
> +                                                       "bashrc",
> +                                                       envname) for
> envname in v]
> +
>  self._pbashrcdict[profile].setdefault(k.cp, {})\
> +                                                       .setdefault(k,
> []).extend(envfiles)
> +
>                         #getting categories from an external file now
>                         self.categories = [grabfile(os.path.join(x,
> "categories")) \
>                                 for x in
> locations_manager.profile_and_user_locations]
> @@ -1501,6 +1522,21 @@ class config(object):
>                                 for x in penv_matches:
>                                         self._penv.extend(x)
>
> +               bashrc_files = []
> +
> +               for profile in self._locations_manager.profiles_complex:
> +                       bashrc_files.append(os.path.join(profile.location,
> +                               'profile.bashrc'))
> +                       if profile in self._pbashrcdict:
> +                               cpdict = self._pbashrcdict[profile].get(cp)
> +                               if cpdict:
> +                                       bashrc_matches = \
> +
>  ordered_by_atom_specificity(cpdict, cpv_slot)
> +                                       for x in bashrc_matches:
> +                                               bashrc_files.extend(x)
> +
> +               self.configdict["BASHRC_FILES"] = bashrc_files
> +
>                 protected_pkg_keys = set(pkg_configdict)
>                 protected_pkg_keys.discard('USE')
>
> diff --git a/pym/portage/package/ebuild/doebuild.py
> b/pym/portage/package/ebuild/doebuild.py
> index 01707ae..5224775 100644
> --- a/pym/portage/package/ebuild/doebuild.py
> +++ b/pym/portage/package/ebuild/doebuild.py
> @@ -335,7 +335,9 @@ def doebuild_environment(myebuild, mydo, myroot=None,
> settings=None,
>         mysettings["ECLASSDIR"]   = mysettings["PORTDIR"]+"/eclass"
>         mysettings["SANDBOX_LOG"] = mycpv.replace("/", "_-_")
>
> -       mysettings["PROFILE_PATHS"] = "\n".join(mysettings.profiles)
> +       mysettings["PORTAGE_BASHRC_FILES"] = \
> +               "\n".join(mysettings.configdict["BASHRC_FILES"])
> +
>         mysettings["P"]  = mysplit[0]+"-"+mysplit[1]
>         mysettings["PN"] = mysplit[0]
>         mysettings["PV"] = mysplit[1]
> diff --git a/pym/portage/repository/config.py
> b/pym/portage/repository/config.py
> index 5e0d055..bef643d 100644
> --- a/pym/portage/repository/config.py
> +++ b/pym/portage/repository/config.py
> @@ -40,7 +40,7 @@ if sys.hexversion >= 0x3000000:
>  _invalid_path_char_re = re.compile(r'[^a-zA-Z0-9._\-+:/]')
>
>  _valid_profile_formats = frozenset(
> -       ['pms', 'portage-1', 'portage-2'])
> +       ['pms', 'portage-1', 'portage-2', 'profile-bashrcs'])
>
>  _portage1_profiles_allow_directories = frozenset(
>         ["portage-1-compat", "portage-1", 'portage-2'])
> --
> 2.1.0.rc2.206.gedb03e5
>
>

[-- Attachment #2: Type: text/html, Size: 15442 bytes --]

[gentoo-portage-dev] Re: [PATCH 3/3] package.bashrc: per profile, per-package bashrc mechanism

[Bertrand Simonnet]

[-- Attachment #1: Type: text/plain, Size: 12407 bytes --]

Michal, Zac, did you have time to look at this serie of patches ?

Bertrand

On Tue, Sep 30, 2014 at 3:16 PM, Bertrand Simonnet <bsimonnet@chromium.org>
wrote:

> Please discard the last serie of patches. git send-email was not
> configured correctly and some commands  I typed yesterday are being send
> now.
> Sorry about that.
>
> Bertrand
>
> On Mon, Sep 29, 2014 at 3:31 PM, Bertrand SIMONNET <bsimonnet@chromium.org
> > wrote:
>
>> Profiles can define per-package bashrc files to be sourced before
>> emerging.
>> Each line in package.bashrc must be an atom name then a list of
>> space-delimited
>> bashrc files (stored in $profile/bashrc/).
>> ---
>>  bin/ebuild.sh                                      |  6 ++--
>>  bin/phase-functions.sh                             |  2 +-
>>  bin/save-ebuild-env.sh                             |  2 +-
>>  man/portage.5                                      | 30
>> +++++++++++++++++-
>>  .../package/ebuild/_config/special_env_vars.py     |  4 +--
>>  pym/portage/package/ebuild/config.py               | 36
>> ++++++++++++++++++++++
>>  pym/portage/package/ebuild/doebuild.py             |  4 ++-
>>  pym/portage/repository/config.py                   |  2 +-
>>  8 files changed, 76 insertions(+), 10 deletions(-)
>>
>> diff --git a/bin/ebuild.sh b/bin/ebuild.sh
>> index 14cc321..50909e1 100755
>> --- a/bin/ebuild.sh
>> +++ b/bin/ebuild.sh
>> @@ -368,10 +368,10 @@ __source_all_bashrcs() {
>>                 # source the existing profile.bashrcs.
>>                 save_IFS
>>                 IFS=$'\n'
>> -               local path_array=($PROFILE_PATHS)
>> +               local bashenv_files=($PORTAGE_BASHRC_FILES)
>>                 restore_IFS
>> -               for x in "${path_array[@]}" ; do
>> -                       [ -f "$x/profile.bashrc" ] && __qa_source
>> "$x/profile.bashrc"
>> +               for x in "${bashenv_files[@]}" ; do
>> +                       __try_source "${x}"
>>                 done
>>         fi
>>
>> diff --git a/bin/phase-functions.sh b/bin/phase-functions.sh
>> index f39a024..5dff3bb 100644
>> --- a/bin/phase-functions.sh
>> +++ b/bin/phase-functions.sh
>> @@ -31,7 +31,7 @@ PORTAGE_READONLY_VARS="D EBUILD EBUILD_PHASE
>> EBUILD_PHASE_FUNC \
>>         PORTAGE_TMPDIR PORTAGE_UPDATE_ENV PORTAGE_USERNAME \
>>         PORTAGE_VERBOSE PORTAGE_WORKDIR_MODE PORTAGE_XATTR_EXCLUDE \
>>         PORTDIR \
>> -       PROFILE_PATHS REPLACING_VERSIONS REPLACED_BY_VERSION T WORKDIR \
>> +       REPLACING_VERSIONS REPLACED_BY_VERSION T WORKDIR \
>>         __PORTAGE_HELPER __PORTAGE_TEST_HARDLINK_LOCKS"
>>
>>  PORTAGE_SAVED_READONLY_VARS="A CATEGORY P PF PN PR PV PVR"
>> diff --git a/bin/save-ebuild-env.sh b/bin/save-ebuild-env.sh
>> index f114c48..1a684b9 100644
>> --- a/bin/save-ebuild-env.sh
>> +++ b/bin/save-ebuild-env.sh
>> @@ -97,7 +97,7 @@ __save_ebuild_env() {
>>                 GOOD HILITE HOME \
>>                 LAST_E_CMD LAST_E_LEN LD_PRELOAD MISC_FUNCTIONS_ARGS
>> MOPREFIX \
>>                 NOCOLOR NORMAL PKGDIR PKGUSE PKG_LOGDIR PKG_TMPDIR \
>> -               PORTAGE_BASHRCS_SOURCED PORTAGE_COMPRESS \
>> +               PORTAGE_BASHRC_FILES PORTAGE_BASHRCS_SOURCED
>> PORTAGE_COMPRESS \
>>                 PORTAGE_COMPRESS_EXCLUDE_SUFFIXES \
>>                 PORTAGE_DOHTML_UNWARNED_SKIPPED_EXTENSIONS \
>>                 PORTAGE_DOHTML_UNWARNED_SKIPPED_FILES \
>> diff --git a/man/portage.5 b/man/portage.5
>> index e399f0f..309e259 100644
>> --- a/man/portage.5
>> +++ b/man/portage.5
>> @@ -24,6 +24,7 @@ make.defaults
>>  packages
>>  packages.build
>>  package.accept_keywords
>> +package.bashrc
>>  package.keywords
>>  package.mask
>>  package.provided
>> @@ -358,6 +359,31 @@ a '\-'.
>>  A list of packages (one per line) that make up a stage1 tarball.  Really
>> only
>>  useful for stage builders.
>>  .TP
>> +.BR package.bashrc
>> +Per-package bashrc mechanism.  Contains a list of bashrc files to be
>> sourced
>> +before emerging a given atom.  The bashrc files must be stored in
>> bashrc/, in
>> +the profile directory.
>> +
>> +.I Note:
>> +.nf
>> +\- The bashrc files will be sourced after profile.bashrc for the same
>> profile.
>> +\- profile-formats in metadata/layout.conf must contain profile-bashrcs
>> for this
>> +to be enabled.
>> +.fi
>> +
>> +.I Format:
>> +.nf
>> +\- comments begin with # (no inline comments).
>> +\- one atom per line with space-delimited list of bashrc files.
>> +.fi
>> +
>> +.I Example:
>> +.nf
>> +# By setting INSTALL_MASK in bashrc/nostandardconf.conf, we can avoid
>> installing
>> +# the standard configuration and enable another package to install it.
>> +net-misc/dhcp nostardardconf.conf
>> +.fi
>> +.TP
>>  .BR package.provided
>>  A list of packages (one per line) that portage should assume have been
>>  provided.  Useful for porting to non-Linux systems. Basically, it's a
>> @@ -1047,11 +1073,13 @@ The default setting for repoman's --echangelog
>> option.
>>  The cache formats supported in the metadata tree.  There is the old
>> "pms" format
>>  and the newer/faster "md5-dict" format.  Default is to detect dirs.
>>  .TP
>> -.BR profile\-formats " = [pms|portage-1|portage-2]"
>> +.BR profile\-formats " = [pms|portage-1|portage-2|profile-bashrcs]"
>>  Control functionality available to profiles in this repo such as which
>> files
>>  may be dirs, or the syntax available in parent files.  Use "portage-2"
>> if you're
>>  unsure.  The default is "portage-1-compat" mode which is meant to be
>> compatible
>>  with old profiles, but is not allowed to be opted into directly.
>> +Setting profile-bashrcs will enable the per-profile bashrc mechanism
>> +\fBpackage.bashrc\fR.
>>  .RE
>>  .RE
>>
>> diff --git a/pym/portage/package/ebuild/_config/special_env_vars.py
>> b/pym/portage/package/ebuild/_config/special_env_vars.py
>> index 74fedd6..387f4ae 100644
>> --- a/pym/portage/package/ebuild/_config/special_env_vars.py
>> +++ b/pym/portage/package/ebuild/_config/special_env_vars.py
>> @@ -49,7 +49,7 @@ environ_whitelist += [
>>         "FEATURES", "FILESDIR", "HOME", "MERGE_TYPE", "NOCOLOR", "PATH",
>>         "PKGDIR",
>>         "PKGUSE", "PKG_LOGDIR", "PKG_TMPDIR",
>> -       "PORTAGE_ACTUAL_DISTDIR", "PORTAGE_ARCHLIST",
>> +       "PORTAGE_ACTUAL_DISTDIR", "PORTAGE_ARCHLIST",
>> "PORTAGE_BASHRC_FILES",
>>         "PORTAGE_BASHRC", "PM_EBUILD_HOOK_DIR",
>>         "PORTAGE_BINPKG_FILE", "PORTAGE_BINPKG_TAR_OPTS",
>>         "PORTAGE_BINPKG_TMPFILE",
>> @@ -74,7 +74,7 @@ environ_whitelist += [
>>         "PORTAGE_SIGPIPE_STATUS",
>>         "PORTAGE_TMPDIR", "PORTAGE_UPDATE_ENV", "PORTAGE_USERNAME",
>>         "PORTAGE_VERBOSE", "PORTAGE_WORKDIR_MODE",
>> "PORTAGE_XATTR_EXCLUDE",
>> -       "PORTDIR", "PORTDIR_OVERLAY", "PREROOTPATH", "PROFILE_PATHS",
>> +       "PORTDIR", "PORTDIR_OVERLAY", "PREROOTPATH",
>>         "REPLACING_VERSIONS", "REPLACED_BY_VERSION",
>>         "ROOT", "ROOTPATH", "T", "TMP", "TMPDIR",
>>         "USE_EXPAND", "USE_ORDER", "WORKDIR",
>> diff --git a/pym/portage/package/ebuild/config.py
>> b/pym/portage/package/ebuild/config.py
>> index f639e14..183627f 100644
>> --- a/pym/portage/package/ebuild/config.py
>> +++ b/pym/portage/package/ebuild/config.py
>> @@ -316,6 +316,7 @@ class config(object):
>>                         self._accept_restrict =
>> copy.deepcopy(clone._accept_restrict)
>>                         self._paccept_restrict =
>> copy.deepcopy(clone._paccept_restrict)
>>                         self._penvdict = copy.deepcopy(clone._penvdict)
>> +                       self._pbashrcdict =
>> copy.deepcopy(clone._pbashrcdict)
>>                         self._expand_map =
>> copy.deepcopy(clone._expand_map)
>>
>>                 else:
>> @@ -661,6 +662,7 @@ class config(object):
>>                         self._ppropertiesdict =
>> portage.dep.ExtendedAtomDict(dict)
>>                         self._paccept_restrict =
>> portage.dep.ExtendedAtomDict(dict)
>>                         self._penvdict =
>> portage.dep.ExtendedAtomDict(dict)
>> +                       self._pbashrcdict = {}
>>
>>                         self._repo_make_defaults = {}
>>                         for repo in
>> self.repositories.repos_with_profiles():
>> @@ -742,6 +744,25 @@ class config(object):
>>                                 for k, v in penvdict.items():
>>                                         self._penvdict.setdefault(k.cp,
>> {})[k] = v
>>
>> +                               # package.bashrc
>> +                               for profile in profiles_complex:
>> +                                       if not 'profile-bashrcs' in
>> profile.profile_formats:
>> +                                               continue
>> +                                       self._pbashrcdict[profile] = \
>> +
>>  portage.dep.ExtendedAtomDict(dict)
>> +                                       bashrc =
>> grabdict_package(os.path.join(profile.location,
>> +                                               "package.bashrc"),
>> recursive=1, allow_wildcard=True,
>> +
>>  allow_repo=True, verify_eapi=False)
>> +                                       if not bashrc:
>> +                                               continue
>> +
>> +                                       for k, v in bashrc.items():
>> +                                               envfiles =
>> [os.path.join(profile.location,
>> +                                                       "bashrc",
>> +                                                       envname) for
>> envname in v]
>> +
>>  self._pbashrcdict[profile].setdefault(k.cp, {})\
>> +                                                       .setdefault(k,
>> []).extend(envfiles)
>> +
>>                         #getting categories from an external file now
>>                         self.categories = [grabfile(os.path.join(x,
>> "categories")) \
>>                                 for x in
>> locations_manager.profile_and_user_locations]
>> @@ -1501,6 +1522,21 @@ class config(object):
>>                                 for x in penv_matches:
>>                                         self._penv.extend(x)
>>
>> +               bashrc_files = []
>> +
>> +               for profile in self._locations_manager.profiles_complex:
>> +                       bashrc_files.append(os.path.join(profile.location,
>> +                               'profile.bashrc'))
>> +                       if profile in self._pbashrcdict:
>> +                               cpdict =
>> self._pbashrcdict[profile].get(cp)
>> +                               if cpdict:
>> +                                       bashrc_matches = \
>> +
>>  ordered_by_atom_specificity(cpdict, cpv_slot)
>> +                                       for x in bashrc_matches:
>> +                                               bashrc_files.extend(x)
>> +
>> +               self.configdict["BASHRC_FILES"] = bashrc_files
>> +
>>                 protected_pkg_keys = set(pkg_configdict)
>>                 protected_pkg_keys.discard('USE')
>>
>> diff --git a/pym/portage/package/ebuild/doebuild.py
>> b/pym/portage/package/ebuild/doebuild.py
>> index 01707ae..5224775 100644
>> --- a/pym/portage/package/ebuild/doebuild.py
>> +++ b/pym/portage/package/ebuild/doebuild.py
>> @@ -335,7 +335,9 @@ def doebuild_environment(myebuild, mydo, myroot=None,
>> settings=None,
>>         mysettings["ECLASSDIR"]   = mysettings["PORTDIR"]+"/eclass"
>>         mysettings["SANDBOX_LOG"] = mycpv.replace("/", "_-_")
>>
>> -       mysettings["PROFILE_PATHS"] = "\n".join(mysettings.profiles)
>> +       mysettings["PORTAGE_BASHRC_FILES"] = \
>> +               "\n".join(mysettings.configdict["BASHRC_FILES"])
>> +
>>         mysettings["P"]  = mysplit[0]+"-"+mysplit[1]
>>         mysettings["PN"] = mysplit[0]
>>         mysettings["PV"] = mysplit[1]
>> diff --git a/pym/portage/repository/config.py
>> b/pym/portage/repository/config.py
>> index 5e0d055..bef643d 100644
>> --- a/pym/portage/repository/config.py
>> +++ b/pym/portage/repository/config.py
>> @@ -40,7 +40,7 @@ if sys.hexversion >= 0x3000000:
>>  _invalid_path_char_re = re.compile(r'[^a-zA-Z0-9._\-+:/]')
>>
>>  _valid_profile_formats = frozenset(
>> -       ['pms', 'portage-1', 'portage-2'])
>> +       ['pms', 'portage-1', 'portage-2', 'profile-bashrcs'])
>>
>>  _portage1_profiles_allow_directories = frozenset(
>>         ["portage-1-compat", "portage-1", 'portage-2'])
>> --
>> 2.1.0.rc2.206.gedb03e5
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 15959 bytes --]

Re: [gentoo-portage-dev] Re: [PATCH 3/3] package.bashrc: per profile, per-package bashrc mechanism

[Zac Medico]

On 10/06/2014 10:38 AM, Bertrand Simonnet wrote:
> Michal, Zac, did you have time to look at this series of patches ?

I've reviewed the whole series, and it all looks good to me. I think
it's ready to merge.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] Re: [PATCH 3/3] package.bashrc: per profile, per-package bashrc mechanism

[Brian Dolbec]

On Mon, 06 Oct 2014 11:57:45 -0700
Zac Medico <zmedico@gentoo.org> wrote:

> On 10/06/2014 10:38 AM, Bertrand Simonnet wrote:
> > Michal, Zac, did you have time to look at this series of patches ?
> 
> I've reviewed the whole series, and it all looks good to me. I think
> it's ready to merge.

We will be having a team meeting soon to decide recent patches, etc..

Although I think for the 2.2.14 final release we will not include this,
just a few bug fixes.  After 2.2.14 final is released we will likely
merge it so it will be in 9999 for awhile and then the 2.2.15 release.

-- 
Brian Dolbec <dolsen>

Re: [gentoo-portage-dev] Re: [PATCH 3/3] package.bashrc: per profile, per-package bashrc mechanism

[Bertrand Simonnet]

[-- Attachment #1: Type: text/plain, Size: 966 bytes --]

It sounds good.
If you think this will not change dramatically between now and the merge,
I'll cherry-pick this on our portage version while waiting for the 2.2.15.

Let me know if something goes bad before the merge.

Thanks for the review :)
Bertrand

On Mon, Oct 6, 2014 at 2:01 PM, Brian Dolbec <dolsen@gentoo.org> wrote:

> On Mon, 06 Oct 2014 11:57:45 -0700
> Zac Medico <zmedico@gentoo.org> wrote:
>
> > On 10/06/2014 10:38 AM, Bertrand Simonnet wrote:
> > > Michal, Zac, did you have time to look at this series of patches ?
> >
> > I've reviewed the whole series, and it all looks good to me. I think
> > it's ready to merge.
>
> We will be having a team meeting soon to decide recent patches, etc..
>
> Although I think for the 2.2.14 final release we will not include this,
> just a few bug fixes.  After 2.2.14 final is released we will likely
> merge it so it will be in 9999 for awhile and then the 2.2.15 release.
>
> --
> Brian Dolbec <dolsen>
>
>
>

[-- Attachment #2: Type: text/html, Size: 1579 bytes --]

Re: [gentoo-portage-dev] Re: [PATCH 3/3] package.bashrc: per profile, per-package bashrc mechanism

[Zac Medico]

On 10/06/2014 02:01 PM, Brian Dolbec wrote:
> On Mon, 06 Oct 2014 11:57:45 -0700
> Zac Medico <zmedico@gentoo.org> wrote:
> 
>> On 10/06/2014 10:38 AM, Bertrand Simonnet wrote:
>>> Michal, Zac, did you have time to look at this series of patches ?
>>
>> I've reviewed the whole series, and it all looks good to me. I think
>> it's ready to merge.
> 
> We will be having a team meeting soon to decide recent patches, etc..
> 
> Although I think for the 2.2.14 final release we will not include this,
> just a few bug fixes.  After 2.2.14 final is released we will likely
> merge it so it will be in 9999 for awhile and then the 2.2.15 release.
> 

Should we go ahead and merge this now that 2.2.14 is released?
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] Re: [PATCH 3/3] package.bashrc: per profile, per-package bashrc mechanism

[Alexander Berntsen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 24/10/14 04:00, Zac Medico wrote:
> Should we go ahead and merge this now that 2.2.14 is released?
ACK from me.

- -- 
Alexander
bernalex@gentoo.org
https://secure.plaimi.net/~alexander
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlRJ/v0ACgkQRtClrXBQc7XcPAEAqGn2JmtE1NeSlcbYg8xWaRiq
6bnlOF3C+V3lhi6SNFwA/0M1xuuvQL2UtkVGwNev9gCfQjMeigwXXdhMHQSXGhHo
=SbHx
-----END PGP SIGNATURE-----

Re: [gentoo-portage-dev] Re: [PATCH 3/3] package.bashrc: per profile, per-package bashrc mechanism

[Zac Medico]

On 10/24/2014 12:25 AM, Alexander Berntsen wrote:
> On 24/10/14 04:00, Zac Medico wrote:
>> Should we go ahead and merge this now that 2.2.14 is released?
> ACK from me.

Thanks, I've pushed them to git now:

https://github.com/gentoo/portage/commit/60ee4deefb701d532fdd279caa989e7a6f4b8400
https://github.com/gentoo/portage/commit/1d351a59a57e018e9c79a371f0cae21505c2249c
https://github.com/gentoo/portage/commit/803dafc462027d6015721f40513abb5f57dc1178

In the last patch, I made 2 trivial changes:

1) Replace self.configdict["BASHRC_FILES"] with self._pbashrc, since configdict is used
for other unrelated things. Also make it an immutable tuple instead of a list.
2) Test if profile.bashrc exists before adding it to the list, which typically shortens
the list drastically because most profile directories don't have a profile.bashrc.

diff --git a/pym/portage/package/ebuild/config.py b/pym/portage/package/ebuild/config.py
index 33b4753..2ceb122 100644
--- a/pym/portage/package/ebuild/config.py
+++ b/pym/portage/package/ebuild/config.py
@@ -254,6 +254,7 @@ class config(object):
 			self._iuse_implicit_match = clone._iuse_implicit_match
 			self._non_user_variables = clone._non_user_variables
 			self._env_d_blacklist = clone._env_d_blacklist
+			self._pbashrc = clone._pbashrc
 			self._repo_make_defaults = clone._repo_make_defaults
 			self.usemask = clone.usemask
 			self.useforce = clone.useforce
@@ -663,6 +664,7 @@ class config(object):
 			self._paccept_restrict = portage.dep.ExtendedAtomDict(dict)
 			self._penvdict = portage.dep.ExtendedAtomDict(dict)
 			self._pbashrcdict = {}
+			self._pbashrc = ()
 
 			self._repo_make_defaults = {}
 			for repo in self.repositories.repos_with_profiles():
@@ -1531,8 +1533,10 @@ class config(object):
 		bashrc_files = []
 
 		for profile in self._locations_manager.profiles_complex:
-			bashrc_files.append(os.path.join(profile.location,
-				'profile.bashrc'))
+			profile_bashrc = os.path.join(profile.location,
+				'profile.bashrc')
+			if os.path.exists(profile_bashrc):
+				bashrc_files.append(profile_bashrc)
 			if profile in self._pbashrcdict:
 				cpdict = self._pbashrcdict[profile].get(cp)
 				if cpdict:
@@ -1541,7 +1545,7 @@ class config(object):
 					for x in bashrc_matches:
 						bashrc_files.extend(x)
 
-		self.configdict["BASHRC_FILES"] = bashrc_files
+		self._pbashrc = tuple(bashrc_files)
 
 		protected_pkg_keys = set(pkg_configdict)
 		protected_pkg_keys.discard('USE')
diff --git a/pym/portage/package/ebuild/doebuild.py b/pym/portage/package/ebuild/doebuild.py
index 20b179e..544d193 100644
--- a/pym/portage/package/ebuild/doebuild.py
+++ b/pym/portage/package/ebuild/doebuild.py
@@ -337,8 +337,7 @@ def doebuild_environment(myebuild, mydo, myroot=None, settings=None,
 	mysettings["ECLASSDIR"]   = mysettings["PORTDIR"]+"/eclass"
 	mysettings["SANDBOX_LOG"] = mycpv.replace("/", "_-_")
 
-	mysettings["PORTAGE_BASHRC_FILES"] = \
-		"\n".join(mysettings.configdict["BASHRC_FILES"])
+	mysettings["PORTAGE_BASHRC_FILES"] = "\n".join(mysettings._pbashrc)
 
 	mysettings["P"]  = mysplit[0]+"-"+mysplit[1]
 	mysettings["PN"] = mysplit[0]

-- 
Thanks,
Zac

Re: [gentoo-portage-dev] Re: [PATCH 3/3] package.bashrc: per profile, per-package bashrc mechanism

[Alexander Berntsen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 24/10/14 22:57, Zac Medico wrote:
> 1) Replace self.configdict["BASHRC_FILES"] with self._pbashrc,
> since configdict is used for other unrelated things. Also make it
> an immutable tuple instead of a list. 2) Test if profile.bashrc
> exists before adding it to the list, which typically shortens the
> list drastically because most profile directories don't have a 
> profile.bashrc.
Great! Thanks for this. And thank you, Bertrand! And Micha?.

- -- 
Alexander
bernalex@gentoo.org
https://secure.plaimi.net/~alexander
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlRK1ZQACgkQRtClrXBQc7WeEAEArOF6Oy4PiVZmuox9iXL7Iv4j
orPWnIhM6Xpgk4EXTHMA/jBbJ4BdkM4FMVcTs3AGIUbMCJfLSnj1GnlcQPSnTu7I
=nvms
-----END PGP SIGNATURE-----