[gentoo-releng] Enable USE_EXPAND support in catalyst

[gentoo-releng] Enable USE_EXPAND support in catalyst

[Anthony G. Basile]

[-- Attachment #1: Type: text/plain, Size: 115 bytes --]


-- 
Anthony G. Basile, Ph. D.
Chair of Information Technology
D'Youville College
Buffalo, NY 14201
(716) 829-8197

[-- Attachment #2: 0001-modules-generic_stage_target.py-handle-USE_EXPAND-fl.patch --]
[-- Type: text/x-patch, Size: 11192 bytes --]

From d442e3e6579bd47af793d6b57f82283f5b3ec3cf Mon Sep 17 00:00:00 2001
From: "Anthony G. Basile" <blueness@gentoo.org>
Date: Sat, 14 Feb 2015 13:35:57 -0500
Subject: [PATCH] modules/generic_stage_target.py: handle USE_EXPAND flags

Currently catalyst does not handle USE_EXPAND flags.  However, the
recent move of mmx, sse and friends from the global USE flags to the
new CPU_FLAGS_X86 USE_EXPAND flag requires this.

This commit adds code to handle a new syntax for self.setting[x] values
which can now contain dictionaries to objects.  We use these to build
environment variables for USE_EXPAND flags.  Eg. in arch/amd64.py
we set

 self.settings["HOSTUSEEXPAND"]={"CPU_FLAGS_X86":["mmx","sse","sse2"]}

which is later exported in the environtment as:

    clst_HOSTUSEEXPAND="CPU_FLAGS_X86"

and

    clst_CPU_FLAGS_X86="mmx sse sse2"

In general one could have:

  self.settings["A"]={"B":[1,2,3],"C":[4,5,6]}

which would export

 clst_A="B C"

and

 clst_B="1 2 3"
 clst_C="4 5 6"

Signed-off-by: Anthony G. Basile <blueness@gentoo.org>
---
 arch/amd64.py                   | 14 +++++++-------
 arch/x86.py                     | 20 ++++++++++----------
 modules/generic_stage_target.py | 30 ++++++++++++++++++++++++++++++
 targets/stage1/stage1-chroot.sh | 13 ++++++++++++-
 4 files changed, 59 insertions(+), 18 deletions(-)

diff --git a/arch/amd64.py b/arch/amd64.py
index 262b55a..2743e22 100644
--- a/arch/amd64.py
+++ b/arch/amd64.py
@@ -13,7 +13,7 @@ class arch_amd64(generic_amd64):
 		generic_amd64.__init__(self,myspec)
 		self.settings["CFLAGS"]="-O2 -pipe"
 		self.settings["CHOST"]="x86_64-pc-linux-gnu"
-		self.settings["HOSTUSE"]=["mmx","sse","sse2"]
+		self.settings["HOSTUSEEXPAND"]={"CPU_FLAGS_X86":["mmx","sse","sse2"]}
 
 class arch_nocona(generic_amd64):
 	"improved version of Intel Pentium 4 CPU with 64-bit extensions, MMX, SSE, SSE2 and SSE3 support"
@@ -21,7 +21,7 @@ class arch_nocona(generic_amd64):
 		generic_amd64.__init__(self,myspec)
 		self.settings["CFLAGS"]="-O2 -march=nocona -pipe"
 		self.settings["CHOST"]="x86_64-pc-linux-gnu"
-		self.settings["HOSTUSE"]=["mmx","sse","sse2"]
+		self.settings["HOSTUSEEXPAND"]={"CPU_FLAGS_X86":["mmx","sse","sse2"]}
 
 # Requires gcc 4.3 to use this class
 class arch_core2(generic_amd64):
@@ -30,7 +30,7 @@ class arch_core2(generic_amd64):
 		generic_amd64.__init__(self,myspec)
 		self.settings["CFLAGS"]="-O2 -march=core2 -pipe"
 		self.settings["CHOST"]="x86_64-pc-linux-gnu"
-		self.settings["HOSTUSE"]=["mmx","sse","sse2","ssse3"]
+		self.settings["HOSTUSEEXPAND"]={"CPU_FLAGS_X86":["mmx","sse","sse2","ssse3"]}
 
 class arch_k8(generic_amd64):
 	"generic k8, opteron and athlon64 support"
@@ -38,7 +38,7 @@ class arch_k8(generic_amd64):
 		generic_amd64.__init__(self,myspec)
 		self.settings["CFLAGS"]="-O2 -march=k8 -pipe"
 		self.settings["CHOST"]="x86_64-pc-linux-gnu"
-		self.settings["HOSTUSE"]=["mmx","sse","sse2","3dnow"]
+		self.settings["HOSTUSEEXPAND"]={"CPU_FLAGS_X86":["mmx","sse","sse2","3dnow"]}
 
 class arch_k8_sse3(generic_amd64):
 	"improved versions of k8, opteron and athlon64 with SSE3 support"
@@ -46,7 +46,7 @@ class arch_k8_sse3(generic_amd64):
 		generic_amd64.__init__(self,myspec)
 		self.settings["CFLAGS"]="-O2 -march=k8-sse3 -pipe"
 		self.settings["CHOST"]="x86_64-pc-linux-gnu"
-		self.settings["HOSTUSE"]=["mmx","sse","sse2","3dnow"]
+		self.settings["HOSTUSEEXPAND"]={"CPU_FLAGS_X86":["mmx","sse","sse2","3dnow"]}
 
 class arch_amdfam10(generic_amd64):
 	"AMD Family 10h core based CPUs with x86-64 instruction set support"
@@ -54,7 +54,7 @@ class arch_amdfam10(generic_amd64):
 		generic_amd64.__init__(self,myspec)
 		self.settings["CFLAGS"]="-O2 -march=amdfam10 -pipe"
 		self.settings["CHOST"]="x86_64-pc-linux-gnu"
-		self.settings["HOSTUSE"]=["mmx","sse","sse2","3dnow"]
+		self.settings["HOSTUSEEXPAND"]={"CPU_FLAGS_X86":["mmx","sse","sse2","3dnow"]}
 
 class arch_x32(generic_amd64):
 	"builder class for generic x32 (Intel and AMD)"
@@ -62,7 +62,7 @@ class arch_x32(generic_amd64):
 		generic_amd64.__init__(self,myspec)
 		self.settings["CFLAGS"]="-O2 -pipe"
 		self.settings["CHOST"]="x86_64-pc-linux-gnux32"
-		self.settings["HOSTUSE"]=["mmx","sse","sse2"]
+		self.settings["HOSTUSEEXPAND"]={"CPU_FLAGS_X86":["mmx","sse","sse2"]}
 
 def register():
 	"inform main catalyst program of the contents of this plugin"
diff --git a/arch/x86.py b/arch/x86.py
index 0391b79..585dcee 100644
--- a/arch/x86.py
+++ b/arch/x86.py
@@ -54,42 +54,42 @@ class arch_pentium_mmx(generic_x86):
 	def __init__(self,myspec):
 		generic_x86.__init__(self,myspec)
 		self.settings["CFLAGS"]="-O2 -march=pentium-mmx -pipe"
-		self.settings["HOSTUSE"]=["mmx"]
+		self.settings["HOSTUSEEXPAND"]={"CPU_FLAGS_X86":["mmx"]}
 
 class arch_pentium2(generic_x86):
 	"Intel Pentium 2 CPU with MMX support"
 	def __init__(self,myspec):
 		generic_x86.__init__(self,myspec)
 		self.settings["CFLAGS"]="-O2 -march=pentium2 -pipe"
-		self.settings["HOSTUSE"]=["mmx"]
+		self.settings["HOSTUSEEXPAND"]={"CPU_FLAGS_X86":["mmx"]}
 
 class arch_pentium3(generic_x86):
 	"Intel Pentium 3 CPU with MMX and SSE support"
 	def __init__(self,myspec):
 		generic_x86.__init__(self,myspec)
 		self.settings["CFLAGS"]="-O2 -march=pentium3 -pipe"
-		self.settings["HOSTUSE"]=["mmx","sse"]
+		self.settings["HOSTUSEEXPAND"]={"CPU_FLAGS_X86":["mmx","sse"]}
 
 class arch_pentium4(generic_x86):
 	"Intel Pentium 4 CPU with MMX, SSE and SSE2 support"
 	def __init__(self,myspec):
 		generic_x86.__init__(self,myspec)
 		self.settings["CFLAGS"]="-O2 -march=pentium4 -pipe"
-		self.settings["HOSTUSE"]=["mmx","sse","sse2"]
+		self.settings["HOSTUSEEXPAND"]={"CPU_FLAGS_X86":["mmx","sse","sse2"]}
 
 class arch_pentium_m(generic_x86):
 	"Intel Pentium M CPU with MMX, SSE and SSE2 support"
 	def __init__(self,myspec):
 		generic_x86.__init__(self,myspec)
 		self.settings["CFLAGS"]="-O2 -march=pentium-m -pipe"
-		self.settings["HOSTUSE"]=["mmx","sse","sse2"]
+		self.settings["HOSTUSEEXPAND"]={"CPU_FLAGS_X86":["mmx","sse","sse2"]}
 
 class arch_prescott(generic_x86):
 	"improved version of Intel Pentium 4 CPU with MMX, SSE, SSE2 and SSE3 support"
 	def __init__(self,myspec):
 		generic_x86.__init__(self,myspec)
 		self.settings["CFLAGS"]="-O2 -march=prescott -pipe"
-		self.settings["HOSTUSE"]=["mmx","sse","sse2"]
+		self.settings["HOSTUSEEXPAND"]={"CPU_FLAGS_X86":["mmx","sse","sse2"]}
 		self.settings["CHOST"]="i686-pc-linux-gnu"
 
 class arch_k6(generic_x86):
@@ -98,7 +98,7 @@ class arch_k6(generic_x86):
 		generic_x86.__init__(self,myspec)
 		self.settings["CFLAGS"]="-O2 -march=k6 -pipe"
 		self.settings["CHOST"]="i686-pc-linux-gnu"
-		self.settings["HOSTUSE"]=["mmx"]
+		self.settings["HOSTUSEEXPAND"]={"CPU_FLAGS_X86":["mmx"]}
 
 class arch_k6_2(generic_x86):
 	"AMD K6-2 CPU with MMX and 3dNOW! support"
@@ -106,7 +106,7 @@ class arch_k6_2(generic_x86):
 		generic_x86.__init__(self,myspec)
 		self.settings["CFLAGS"]="-O2 -march=k6-2 -pipe"
 		self.settings["CHOST"]="i686-pc-linux-gnu"
-		self.settings["HOSTUSE"]=["mmx","3dnow"]
+		self.settings["HOSTUSEEXPAND"]={"CPU_FLAGS_X86":["mmx","3dnow"]}
 
 class arch_athlon(generic_x86):
 	"AMD Athlon CPU with MMX, 3dNOW!, enhanced 3dNOW! and SSE prefetch support"
@@ -114,7 +114,7 @@ class arch_athlon(generic_x86):
 		generic_x86.__init__(self,myspec)
 		self.settings["CFLAGS"]="-O2 -march=athlon -pipe"
 		self.settings["CHOST"]="i686-pc-linux-gnu"
-		self.settings["HOSTUSE"]=["mmx","3dnow"]
+		self.settings["HOSTUSEEXPAND"]={"CPU_FLAGS_X86":["mmx","3dnow"]}
 
 class arch_athlon_xp(generic_x86):
 	"improved AMD Athlon CPU with MMX, 3dNOW!, enhanced 3dNOW! and full SSE support"
@@ -122,7 +122,7 @@ class arch_athlon_xp(generic_x86):
 		generic_x86.__init__(self,myspec)
 		self.settings["CFLAGS"]="-O2 -march=athlon-xp -pipe"
 		self.settings["CHOST"]="i686-pc-linux-gnu"
-		self.settings["HOSTUSE"]=["mmx","3dnow","sse"]
+		self.settings["HOSTUSEEXPAND"]={"CPU_FLAGS_X86":["mmx","3dnow","sse"]}
 
 def register():
 	"Inform main catalyst program of the contents of this plugin."
diff --git a/modules/generic_stage_target.py b/modules/generic_stage_target.py
index cc24c63..3649223 100644
--- a/modules/generic_stage_target.py
+++ b/modules/generic_stage_target.py
@@ -1073,6 +1073,16 @@ class generic_stage_target(generic_target):
 						"/use will cause portage to ignore"
 					print "\tpackage.use in the profile and portage_confdir. You've been warned!"
 
+			myuseexpandvars={}
+			if "HOSTUSEEXPAND" in self.settings:
+				for hostuseexpand in self.settings["HOSTUSEEXPAND"]:
+					myuseexpandvars.update({hostuseexpand:self.settings["HOSTUSEEXPAND"][hostuseexpand]})
+
+			if myuseexpandvars:
+				myf.write("# These are the USE EXPAND flags that were used in addition to what is provided by the\n# profile used for building.\n")
+				for hostuseexpand in myuseexpandvars:
+					myf.write(hostuseexpand+'="'+string.join(myuseexpandvars[hostuseexpand])+'"\n')
+
 			myf.write('PORTDIR="/usr/portage"\n')
 			myf.write('DISTDIR="${PORTDIR}/distfiles"\n')
 			myf.write('PKGDIR="${PORTDIR}/packages"\n')
@@ -1271,6 +1281,26 @@ class generic_stage_target(generic_target):
 					self.env[varname]="true"
 				else:
 					self.env[varname]="false"
+			# This handles a dictionary of objects just one level deep and no deeper!
+			# Its currently used only for USE_EXPAND flags which are dictionaries of
+			# lists in arch/amd64.py and friends.  If we wanted self.settigs[var]
+			# of any depth, we should make this function recursive.
+			elif type(self.settings[x])==types.DictType:
+				self.env[varname]=string.join(self.settings[x].keys())
+				for y in self.settings[x].keys():
+					varname2="clst_"+string.replace(y,"/","_")
+					varname2=string.replace(varname2,"-","_")
+					varname2=string.replace(varname2,".","_")
+					if type(self.settings[x][y])==types.StringType:
+						self.env[varname2]=self.settings[x][y]
+					elif type(self.settings[x][y])==types.ListType:
+						self.env[varname2]=string.join(self.settings[x][y])
+					elif type(self.settings[x][y])==types.BooleanType:
+						if self.settings[x][y]:
+							self.env[varname]="true"
+						else:
+							self.env[varname]="false"
+
 		if "makeopts" in self.settings:
 			self.env["MAKEOPTS"]=self.settings["makeopts"]
 
diff --git a/targets/stage1/stage1-chroot.sh b/targets/stage1/stage1-chroot.sh
index eccbd3f..f79f360 100755
--- a/targets/stage1/stage1-chroot.sh
+++ b/targets/stage1/stage1-chroot.sh
@@ -54,9 +54,20 @@ run_merge "--oneshot --nodeps sys-apps/baselayout"
 sed -i '/USE="${USE} -build"/d' /etc/portage/make.conf
 
 # Now, we install our packages
-[ -e /etc/portage/make.conf ] && \
+if [ -e /etc/portage/make.conf ]; then
 	echo "USE=\"-* build ${BOOTSTRAP_USE} ${clst_HOSTUSE}\"" \
 	>> /etc/portage/make.conf
+	for useexpand in "${clst_HOSTUSEEXPAND}"; do
+		x="clst_${useexpand}"
+		echo "${useexpand}=\"${!x}\"" \
+		>> /etc/portage/make.conf
+	done
+fi
 run_merge "--oneshot ${clst_buildpkgs}"
 sed -i "/USE=\"-* build ${BOOTSTRAP_USE} ${clst_HOSTUSE}\"/d" \
 	/etc/portage/make.conf
+for useexpand in "${clst_HOSTUSEEXPAND}"; do
+	x="clst_${useexpand}"
+	sed -i "${useexpand}=\"${!x}\"" \
+	/etc/portage/make.conf
+done
-- 
2.0.5

Re: [gentoo-releng] Enable USE_EXPAND support in catalyst

[Matt Turner]

On Sat, Feb 14, 2015 at 11:03 AM, Anthony G. Basile
<basile@opensource.dyc.edu> wrote:
>
> --

You sent this to gentoo-releng@. It should go to the catalyst mailing
list. Also, use git send-email instead of attaching patches.

Re: [gentoo-releng] Enable USE_EXPAND support in catalyst

[Anthony G. Basile]

On 02/14/15 15:22, Matt Turner wrote:
> On Sat, Feb 14, 2015 at 11:03 AM, Anthony G. Basile
> <basile@opensource.dyc.edu> wrote:
>>
>> --
>
> You sent this to gentoo-releng@. It should go to the catalyst mailing
> list. Also, use git send-email instead of attaching patches.
>

My env here doesn't exactly like git send-email.

-- 
Anthony G. Basile, Ph. D.
Chair of Information Technology
D'Youville College
Buffalo, NY 14201
(716) 829-8197

Re: [gentoo-releng] Enable USE_EXPAND support in catalyst

[Rémi Cardona]

Le 14/02/2015 20:03, Anthony G. Basile a écrit :
> 

In the last chunk of modules/generic_stage_target.py:

* type(var) == types.DictType can be replaced with
  isinstance(var, dict).

  Same thing for StringType/str, ListType/list and BooleanType/bool.
  This is how simple types were checked before Python 2.2 (around 2002).

* string.replace has been deprecated for ages, use str.replace directly:

  varname2="clst_"+string.replace(y,"/","_")
  varname2=string.replace(varname2,"-","_")
  varname2=string.replace(varname2,".","_")

  becomes

  varname2 = "clst_" + y.replace("/", "_")
  varname2 = varname2.replace("-", "_")
  varname2 = varname2.replace(".", "_")

* string.join is also deprecated, use ' '.join() instead

* dict objects are iterable (and they return keys). This means that

  string.join(self.settings[x].keys())

  can be written (with the join suggestion above):

  ' '.join(self.settings[x])

* You may want to use dict.items() to get both the key and the
  associated value:

  for y in self.settings[x].keys():

  becomes:

  for y, val in self.settings[x].items():

  which should allow you to replace "self.settings[x][y]" with "val"

Cheers,

Rémi

Re: [gentoo-releng] Enable USE_EXPAND support in catalyst

[Anthony G. Basile]

On 02/15/15 02:54, Rémi Cardona wrote:
> Le 14/02/2015 20:03, Anthony G. Basile a écrit :
> In the last chunk of modules/generic_stage_target.py:
>
> * type(var) == types.DictType can be replaced with
>    isinstance(var, dict).
>
>    Same thing for StringType/str, ListType/list and BooleanType/bool.
>    This is how simple types were checked before Python 2.2 (around 2002).
>
> * string.replace has been deprecated for ages, use str.replace directly:
>
>    varname2="clst_"+string.replace(y,"/","_")
>    varname2=string.replace(varname2,"-","_")
>    varname2=string.replace(varname2,".","_")
>
>    becomes
>
>    varname2 = "clst_" + y.replace("/", "_")
>    varname2 = varname2.replace("-", "_")
>    varname2 = varname2.replace(".", "_")
>
> * string.join is also deprecated, use ' '.join() instead
>
> * dict objects are iterable (and they return keys). This means that
>
>    string.join(self.settings[x].keys())
>
>    can be written (with the join suggestion above):
>
>    ' '.join(self.settings[x])

Looking at that code was like a walk down memory lane.  Those idioms are 
used consistently in many different places throughout the code. I'd 
rather not break from those idioms in this commit because then you get 
mixed code.  If we want to modernize the codebase then we should just 
grep the source tree and do massive commits switching string.replace() 
with <str obj>.replace() etc.

So I'm not disagreeing with you here, but not for this commit.

>
> * You may want to use dict.items() to get both the key and the
>    associated value:
>
>    for y in self.settings[x].keys():
>
>    becomes:
>
>    for y, val in self.settings[x].items():
>
>    which should allow you to replace "self.settings[x][y]" with "val"
>
> Cheers,
>
> Rémi
>

I don't know when d.items() came to python but this one bothers me 
enough that I'll switch --- I never remember it not being there. There 
are a few other places using this idiom in the source, but at least not 
in generic_stage_target.py.  Maybe I'll submit a few cleanup commits, 
but with catalyst 3.x coming, its hard to justify putting the effort 
into it.  There's some pretty scary insanity in 2.x like take a look at 
line 662 of modules/generic_stage_target.py.

@Brian.  What do you think?  Should I spend some time modernizing 2.x or 
just wait for 3.x?

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA