[gentoo-dev] stabilitylevels and such issues.

[gentoo-dev] stabilitylevels and such issues.

[Terje Kvernes]

  I've done some hacking (okay, ugly hacking) with portage lately.
  reading some source and trying to get a grasp at the code.  so far
  I've only patched 'dep_bestmatch' to deal with stability-symlinks in
  the portage tree.

  this is what the patch actually does:

almach freeciv # PORTAGE_LEVEL="STABLE" emerge --pretend freeciv
These are the packages that I would merge, in order.
Calculating dependencies ...done!
[ebuild  N   ] app-games/freeciv-1.12.0-r1 to /

almach freeciv # emerge --pretend freeciv
These are the packages that I would merge, in order.
Calculating dependencies ...done!
[ebuild  N   ] app-games/freeciv-1.12.0-r2 to /

almach / # ls -la /usr/portage/app-games/freeciv/
total 15
drwxr-xr-x    3 root  root   216 Apr 20 01:47 .
drwxr-xr-x   60 root  root  1608 Apr 19 12:20 ..
-rw-r--r--    1 root  root   987 Mar 29 15:52 ChangeLog
drwxr-xr-x    2 root  root   128 Apr 19 12:20 files
-rw-r--r--    1 root  root  1288 Feb  6 23:58 freeciv-1.12.0-r1.ebuild
-rw-r--r--    1 root  root  1321 Mar 29 15:52 freeciv-1.12.0-r2.ebuild
lrwxrwxrwx    1 root  root    24 Apr 20 01:47 freeciv-stable -> freeciv-1.12.0-r1.ebuild

  notice that last symlink.  :)

  this is against 1.9.1, I haven't even looked at any other version of
  portage. 

  now, I don't expect this to get anywhere near the main tree.  don't
  worry about that.  :) 

  I am however curious to how what people think of the idea, and the
  implementation.  it probably shows that I haven't coded python in a
  year or so.  and no, this hasn't been tested overly much, just
  enough for me to feel that it apparently "works for me".  please,
  thump me over the head if there is a reason to do so.

  I know some people want a package.mask to do separate stable
  releases, and they probably know I don't like that one bit.
  personally, I'm going to take a snapshot of my now stable system
  using this scheme.  all I need to do is to make a
  "<package>-sysok"-symlink for each installed package.  this should
  also hopefully allow me to do rollbacks if I need to do so.

  if anyone is wondering on the status of the update-flags I also
  want, well, that'll need quite a bit more work.  it will need to be
  integrated into the datastructure that portage uses to keep track of
  packages internally -- which will take time.

  anyhow, the patch.  I know, I should patch against the portage
  sourcecode directly.  but still.  :)

--- /usr/lib/python2.2/site-packages/portage-clean.py   Sat Apr 20 01:19:57 2002
+++ /usr/lib/python2.2/site-packages/portage.py Sat Apr 20 02:29:04 2002
@@ -2081,6 +2081,21 @@
                        else:
                                return mymatch
                elif not isspecific(mypkgdep):
+                       keys = os.environ.keys()
+                       
+                       if 'PORTAGE_LEVEL' in keys:
+                               if mypkgdep.find("/") > -1:
+                                       level = os.environ['PORTAGE_LEVEL']
+                                       [ dir, name ] = mypkgdep.split("/")
+                                       try:
+                                               ebuild = os.readlink("/usr/portage/" + mypkgdep + "/" + name + "-" + level.lower())
+                                               ebuild = ebuild.replace(".ebuild", "")
+                                               return self.dep_bestmatch("=" + ebuild)
+                                       except OSError:
+                                               print "\n>>> " + mypkgdep + " has no ebuild with label '" + level + "'."
+                                               sys.exit(1)
+
+
                        if not self.hasnode(mypkgdep):
                                return ""
                        mynodes=self.getnode(mypkgdep)[:]


-- 
Terje

Re: [gentoo-dev] stabilitylevels and such issues.

[Mike Payson]

This looks GREAT. Exactly what I had in mind! Obviously PORTAGE_LEVEL should 
be in one of the config files, and you need to be able to override the level 
by specifying a specific version, but excellent for, what, 23 lines of 
code?!?!   

On Friday 19 April 2002 05:56 pm, Terje Kvernes wrote:
>   I've done some hacking (okay, ugly hacking) with portage lately.
>   reading some source and trying to get a grasp at the code.  so far
>   I've only patched 'dep_bestmatch' to deal with stability-symlinks in
>   the portage tree.
>
>   this is what the patch actually does:
>
> almach freeciv # PORTAGE_LEVEL="STABLE" emerge --pretend freeciv
> These are the packages that I would merge, in order.
> Calculating dependencies ...done!
> [ebuild  N   ] app-games/freeciv-1.12.0-r1 to /
>
> almach freeciv # emerge --pretend freeciv
> These are the packages that I would merge, in order.
> Calculating dependencies ...done!
> [ebuild  N   ] app-games/freeciv-1.12.0-r2 to /
>
> almach / # ls -la /usr/portage/app-games/freeciv/
> total 15
> drwxr-xr-x    3 root  root   216 Apr 20 01:47 .
> drwxr-xr-x   60 root  root  1608 Apr 19 12:20 ..
> -rw-r--r--    1 root  root   987 Mar 29 15:52 ChangeLog
> drwxr-xr-x    2 root  root   128 Apr 19 12:20 files
> -rw-r--r--    1 root  root  1288 Feb  6 23:58 freeciv-1.12.0-r1.ebuild
> -rw-r--r--    1 root  root  1321 Mar 29 15:52 freeciv-1.12.0-r2.ebuild
> lrwxrwxrwx    1 root  root    24 Apr 20 01:47 freeciv-stable ->
> freeciv-1.12.0-r1.ebuild
>
>   notice that last symlink.  :)
>
>   this is against 1.9.1, I haven't even looked at any other version of
>   portage.
>
>   now, I don't expect this to get anywhere near the main tree.  don't
>   worry about that.  :)
>
>   I am however curious to how what people think of the idea, and the
>   implementation.  it probably shows that I haven't coded python in a
>   year or so.  and no, this hasn't been tested overly much, just
>   enough for me to feel that it apparently "works for me".  please,
>   thump me over the head if there is a reason to do so.
>
>   I know some people want a package.mask to do separate stable
>   releases, and they probably know I don't like that one bit.
>   personally, I'm going to take a snapshot of my now stable system
>   using this scheme.  all I need to do is to make a
>   "<package>-sysok"-symlink for each installed package.  this should
>   also hopefully allow me to do rollbacks if I need to do so.
>
>   if anyone is wondering on the status of the update-flags I also
>   want, well, that'll need quite a bit more work.  it will need to be
>   integrated into the datastructure that portage uses to keep track of
>   packages internally -- which will take time.
>
>   anyhow, the patch.  I know, I should patch against the portage
>   sourcecode directly.  but still.  :)
>
> --- /usr/lib/python2.2/site-packages/portage-clean.py   Sat Apr 20 01:19:57
> 2002 +++ /usr/lib/python2.2/site-packages/portage.py Sat Apr 20 02:29:04
> 2002 @@ -2081,6 +2081,21 @@
>                         else:
>                                 return mymatch
>                 elif not isspecific(mypkgdep):
> +                       keys = os.environ.keys()
> +
> +                       if 'PORTAGE_LEVEL' in keys:
> +                               if mypkgdep.find("/") > -1:
> +                                       level = os.environ['PORTAGE_LEVEL']
> +                                       [ dir, name ] = mypkgdep.split("/")
> +                                       try:
> +                                               ebuild =
> os.readlink("/usr/portage/" + mypkgdep + "/" + name + "-" + level.lower())
> +                                               ebuild =
> ebuild.replace(".ebuild", "") +                                            
>   return self.dep_bestmatch("=" + ebuild) +                                
>       except OSError:
> +                                               print "\n>>> " + mypkgdep +
> " has no ebuild with label '" + level + "'." +                             
>                  sys.exit(1)
> +
> +
>                         if not self.hasnode(mypkgdep):
>                                 return ""
>                         mynodes=self.getnode(mypkgdep)[:]

Re: [gentoo-dev] stabilitylevels and such issues.

[Terje Kvernes]

Mike Payson <mike@bucky.dawgdayz.com> writes:

> This looks GREAT. 

  thank you.  :)

> Exactly what I had in mind! Obviously PORTAGE_LEVEL should be in one
> of the config files, 

  /etc/make.conf would do nicely.  see the revised patch.  it got a
  bit cleaner as well, once I figured out why I'd broken a shell by
  exporting PORTAGE_LEVEL, and seeing how config worked.  :)

> and you need to be able to override the level by specifying a
> specific version, 

  I think this scheme will require you to do either:

PORTAGE_LEVEL="" emerge <foo>

  or

emerge --level=any emerge <foo>

  the first one already works.

  now, the thing is that now, suddenly, with PORTAGE_LEVEL="stable"
  set in /etc/make.conf, then you can only ask to merge packages of
  that level into the system.  this is somewhat broken when it comes
  to dependencies yet (this is because I've only patched the calls to
  get package "foo", not ">foo" or anything like that.  I'll try to
  get this working during the weekend or so.  but, until then, if you
  emerge a package with the "stable" level, then you're (so far) not
  guaranteed that the dependencies will be stable.

  what I intend to do about that is theoretically simple.  since there
  can only be one package with a given label at any time, check if
  that package is valid for use.  otherwise bail out.

> but excellent for, what, 23 lines of code?!?!

  nope.  I've added 14 lines of code.  there will be more needed to
  handle the dependencies correctly, but still.  

  this patch gets 'PORTAGE_LEVEL' from /etc/make.conf.  apply to a
  clean tree if you want to test it.

--- /usr/lib/python2.2/site-packages/portage-clean.py   Sat Apr 20 01:19:57 2002
+++ /usr/lib/python2.2/site-packages/portage.py Sat Apr 20 12:32:00 2002
@@ -2081,6 +2081,20 @@
                        else:
                                return mymatch
                elif not isspecific(mypkgdep):
+
+                       if settings['PORTAGE_LEVEL']:
+                               if mypkgdep.find("/") > -1:
+                                       level = settings['PORTAGE_LEVEL']
+                                       [ dir, name ] = mypkgdep.split("/")
+                                       try:
+                                               ebuild = os.readlink("/usr/portage/" + mypkgdep + "/" + name + "-" + level.lower())
+                                               ebuild = ebuild.replace(".ebuild", "")
+                                               return self.dep_bestmatch("=" + ebuild)
+                                       except OSError:
+                                               print "\n>>> " + mypkgdep + " has no ebuild with label '" + level + "'."
+                                               sys.exit(1)
+
+
                        if not self.hasnode(mypkgdep):
                                return ""
                        mynodes=self.getnode(mypkgdep)[:]

-- 
Terje

Re: [gentoo-dev] stabilitylevels and such issues.

[Mike Payson]

> > and you need to be able to override the level by specifying a
> > specific version,
>
>   I think this scheme will require you to do either:
>
> PORTAGE_LEVEL="" emerge <foo>
>
>   or
>
> emerge --level=any emerge <foo>
>
>   the first one already works.

How about something like:
  emerge --force foo_1.1beta

with --force (or whatever keyword is chosen) telling it to ignore the 
currently set stability level & install the particular package specified? 

Re: [gentoo-dev] stabilitylevels and such issues.

[Terje Kvernes]

Mike Payson <mike@bucky.dawgdayz.com> writes:

> How about something like:
>   emerge --force foo_1.1beta
> 
> with --force (or whatever keyword is chosen) telling it to ignore
> the currently set stability level & install the particular package
> specified?

  hm, "--force" might eventually get other meanings, with dependencies
  and such.  but I can't really think of any other fitting option.
  "--any-stability-level" is a tad long, but "--force" will probably
  imply "--any-stability-level", and of course, we can use a short
  mapping of it (-A or so).  I'll see about getting this in as well.

  please note that an option like this will _not_ have the exact
  effect you're thinking of.

emerge --force xgammon

  will get the most resent xgammon.  if you specifically want
  xgammon-0.98a you need to do:

emerge --force =xgammon-0.98a

  note the "=".  if you don't give the "="-sign emerge will look for
  the latest version of the package "xgammon-0.98a" and come up short.
  this is also how emerge / portage works today.

-- 
Terje

Re: [gentoo-dev] stabilitylevels and such issues.

[Mike Payson]

On Saturday 20 April 2002 07:12 pm, Terje Kvernes wrote:
> Mike Payson <mike@bucky.dawgdayz.com> writes:
> > How about something like:
> >   emerge --force foo_1.1beta
> >
> > with --force (or whatever keyword is chosen) telling it to ignore
> > the currently set stability level & install the particular package
> > specified?
>
>   hm, "--force" might eventually get other meanings, with dependencies
>   and such.  but I can't really think of any other fitting option.
>   "--any-stability-level" is a tad long, but "--force" will probably
>   imply "--any-stability-level", and of course, we can use a short
>   mapping of it (-A or so).  I'll see about getting this in as well.
>
>   please note that an option like this will _not_ have the exact
>   effect you're thinking of.
>
> emerge --force xgammon
>
>   will get the most resent xgammon.  if you specifically want
>   xgammon-0.98a you need to do:
>
> emerge --force =xgammon-0.98a
>
>   note the "=".  if you don't give the "="-sign emerge will look for
>   the latest version of the package "xgammon-0.98a" and come up short.
>   this is also how emerge / portage works today.

Re: [gentoo-dev] stabilitylevels and such issues.

[Mike Payson]

On Saturday 20 April 2002 07:12 pm, Terje Kvernes wrote:
> Mike Payson <mike@bucky.dawgdayz.com> writes:
> > How about something like:
> >   emerge --force foo_1.1beta
> >
> > with --force (or whatever keyword is chosen) telling it to ignore
> > the currently set stability level & install the particular package
> > specified?
>
>   hm, "--force" might eventually get other meanings, with dependencies
>   and such.  but I can't really think of any other fitting option.
>   "--any-stability-level" is a tad long, but "--force" will probably
>   imply "--any-stability-level", and of course, we can use a short
>   mapping of it (-A or so).  I'll see about getting this in as well.

I agree --force might not be the best term... It is appropriate in the context 
of "force this version, regardless of the default stability level", but as I 
said, other keywords might be more appropriate.

>   please note that an option like this will _not_ have the exact
>   effect you're thinking of.
>
> emerge --force xgammon
>
>   will get the most resent xgammon.  if you specifically want
>   xgammon-0.98a you need to do:
>
> emerge --force =xgammon-0.98a
>
>   note the "=".  if you don't give the "="-sign emerge will look for
>   the latest version of the package "xgammon-0.98a" and come up short.
>   this is also how emerge / portage works today.

That's actually what I meant. Sorry for the confusion.

ps. Sorry about the blank message...

Re: [gentoo-dev] stabilitylevels and such issues.

[Terje Kvernes]

Terje Kvernes <terjekv@math.uio.no> writes:

  [ ... ]

> now, the thing is that now, suddenly, with PORTAGE_LEVEL="stable"
> set in /etc/make.conf, then you can only ask to merge packages of
> that level into the system.  this is somewhat broken when it comes
> to dependencies yet (this is because I've only patched the calls to
> get package "foo", not ">foo" or anything like that.  I'll try to
> get this working during the weekend or so.  but, until then, if you
> emerge a package with the "stable" level, then you're (so far) not
> guaranteed that the dependencies will be stable.

  some simple testing has been done and well, now things seem to work.
  the patch has become a bit longer, and it should now handle
  dependencies.

  a quick example: we want to emerge package cat/foo that depends on
  >=cat/bar-1.0, and we have set a level[1] of "mystate" in
  /etc/make.conf[2].  portage will now look for
  '/usr/portage/cat/bar-mystate'.  if this isn't found, emerge aborts
  at once.  if it is found, the symlink is resolved, and if the
  package that the symlinks points to is new enough, emerge continues.

  now, if you're daring to test this, you need to label a lot of
  packages something so you can set that level.  a very quick and
  dirty hack (yes, it is _that_ quick and ugly) called
  "make_portage_level" can be found here:

  <url: http://terje.kvernes.no/gentoo/utils/ >

  it wants a _file_ containing one package per line on the following
  form:

dev-java/java-config-0.2.2

  and nothing else.  an easy way to build such a list from your
  current setup, and then to call that "mystable" would be something
  like this:

/usr/lib/portage/bin/pkglist > /tmp/packages
make_portage_level /tmp/packages mystable

  a word about levels...  there are some limits to names.  until
  further testing is done, names should match [a-z]+ and nothing
  else.

  again, this stuff isn't tested very well, it probably will break
  something.  you _should_ take a backup of your portage-tree if
  you're even thinking of fiddeling with this stuff.

  but, if anyone is willing to test this, I would be _very_ happy to
  get bugreports.  the patch follows, but patches can also be found
  on my website:  <url: http://terje.kvernes.no/gentoo/patches/ >

--- /usr/lib/python2.2/site-packages/portage-clean.py	Sat Apr 20 01:19:57 2002
+++ /usr/lib/python2.2/site-packages/portage.py	Mon Apr 22 01:33:50 2002
@@ -2003,35 +2003,37 @@
 		"""
 		mypkgdep=self.dep_pkgcat(mypkgdep)
 
-		if (mypkgdep[0]=="="):
-			if mypkgdep[-1]=="*":
-				if not isspecific(mypkgdep[1:-1]):
-					return ""
-				mycatpkg=catpkgsplit(mypkgdep[1:-1])
-				try:
-					mynewver=mycatpkg[2]
-					mynewsplit=string.split(mycatpkg[2],'.')
-					mynewsplit[-1]=`int(mynewsplit[-1])+1`
-				except:
-					return "" 
-				mynodes=[]
-				cmp1=mycatpkg[1:]
-				cmp2=[mycatpkg[1],string.join(mynewsplit,"."),"r0"]
-				for x in self.getnode(mycatpkg[0]+"/"+mycatpkg[1]):
-					if (pkgcmp(x[1][1:],cmp1)>=0) and (pkgcmp(x[1][1:],cmp2)<0):
-						mynodes.append(x)
-				if len(mynodes)==0:
-					return ""
-				bestmatch=mynodes[0]
-				for x in mynodes[1:]:
-					if pkgcmp(x[1][1:],bestmatch[1][1:])>0:
-						bestmatch=x
-				return bestmatch[0]		
-			else:
-				if self.exists_specific(mypkgdep[1:]):
-					return mypkgdep[1:]
+ 		if settings['PORTAGE_LEVEL']:
+			level = settings['PORTAGE_LEVEL']
+			req_package = self.get_base_name(mypkgdep)
+			# in case we get a package without any version information
+			if not isspecific(mypkgdep):
+				level_ebuild = self.get_level_package(mypkgdep, level)
+				[ cat, name ] = mypkgdep.split("/")
+				if not level_ebuild:
+					print "\n>>> " + mypkgdep + " has no ebuild with label '" + level + "'."
+					sys.exit(1)
 				else:
-					return ""
+					return cat + "/" + level_ebuild
+			
+			[req_dir, req_name, req_ver, req_rel] = catpkgsplit(req_package)
+			package = req_dir + "/" + req_name
+			# then get the ebuild that this name relates to.			
+			level_ebuild = self.get_level_package(package, level)
+			if not level_ebuild:
+				print "\n>>> " + package + " has no ebuild with label '" + level + "'."
+				sys.exit(1)
+
+			# check if the package version is high enough.
+			if self.package_version_ok(req_dir + "/" + level_ebuild, req_package) >= 0:
+				return req_dir + "/" + level_ebuild
+			else:
+				print "\n>>> '" + level + "' " + package + " is currently " + level_ebuild + "."
+				print "    Required version is " + mypkgdep + ".  Aborting merge."
+				sys.exit(1)
+
+		if (mypkgdep[0]=="="):
+			self.get_spesific_version(mypkgdep)
 		elif (mypkgdep[0]==">") or (mypkgdep[0]=="<"):
 			if mypkgdep[1]=="=":
 				cmpstr=mypkgdep[0:2]
@@ -2081,17 +2083,93 @@
 			else:
 				return mymatch
 		elif not isspecific(mypkgdep):
-			if not self.hasnode(mypkgdep):
+			return self.get_latest_version(mypkgdep)
+		
+	def get_spesific_version(self, mypkgdep):		
+		if mypkgdep[-1]=="*":
+			if not isspecific(mypkgdep[1:-1]):
 				return ""
-			mynodes=self.getnode(mypkgdep)[:]
-			if len(mynodes)==0:
+			mycatpkg=catpkgsplit(mypkgdep[1:-1])
+			try:
+				mynewver=mycatpkg[2]
+				mynewsplit=string.split(mycatpkg[2],'.')
+				mynewsplit[-1]=`int(mynewsplit[-1])+1`
+			except:
+				return "" 
+			mynodes=[]
+			cmp1=mycatpkg[1:]
+			cmp2=[mycatpkg[1],string.join(mynewsplit,"."),"r0"]
+			for x in self.getnode(mycatpkg[0]+"/"+mycatpkg[1]):
+				if (pkgcmp(x[1][1:],cmp1)>=0) and (pkgcmp(x[1][1:],cmp2)<0):
+					mynodes.append(x)
+					if len(mynodes)==0:
+						return ""
+				bestmatch=mynodes[0]
+				for x in mynodes[1:]:
+					if pkgcmp(x[1][1:],bestmatch[1][1:])>0:
+						bestmatch=x
+				return bestmatch[0]		
+			else:
+				if self.exists_specific(mypkgdep[1:]):
+					return mypkgdep[1:]
+				else:
+					return ""
+
+	def get_latest_version(self, mypkgdep):
+		"""
+		returns the latest version of a package.
+
+		"""
+		if not self.hasnode(mypkgdep):
+			return ""
+		mynodes=self.getnode(mypkgdep)[:]
+		if len(mynodes)==0:
+			return ""
+		bestmatch=mynodes[0]
+		for x in mynodes[1:]:
+			if pkgcmp(x[1][1:],bestmatch[1][1:])>0:
+				bestmatch=x
+				return bestmatch[0]
+
+
+	def get_base_name(self, mypkgdep):
+		"""
+		returns the base name of package, removing *<>=~ at
+		the start, if they exist.
+
+		"""
+		package = mypkgdep
+#		print "\nget_package_name: " + mypkgdep + " -> " + package
+		while package[0] == "=" or package[0] == ">" or package[0] == "<" or package[0] == "~" or package[0] == "*":
+			package = package[1:]
+		return package 
+
+	def get_level_package(self, package, level):
+		"""
+		resolves the symlink package-level and returns package-version. exits if the file 
+		
+		"""
+#		print "\nget_level_package: " + package + " " + level
+		if package.find("/") > -1:
+			[ dir, name ] = package.split("/")
+			try:
+				ebuild = os.readlink("/usr/portage/" + package + "/" + name + "-" + level.lower())
+				ebuild = ebuild.replace(".ebuild", "")
+				return ebuild
+			except OSError:
 				return ""
-			bestmatch=mynodes[0]
-			for x in mynodes[1:]:
-				if pkgcmp(x[1][1:],bestmatch[1][1:])>0:
-					bestmatch=x
-			return bestmatch[0]
 
+	def package_version_ok(self, candidate, requirement):
+		"""
+		package_version_ok() returns 0 if the candidate is an
+		exact match, 1 if the candidate is more recent than
+		the requirement, and -1 if the candidate is older than
+		the requirement.		
+		"""
+
+#		print "package_version_ok: C:" + candidate + " R:" + requirement
+		return pkgcmp(pkgsplit(candidate),pkgsplit(requirement))
+	
 	def dep_nomatch(self,mypkgdep):
 		"""dep_nomatch() has a very specific purpose.  You pass it a dep, like =sys-apps/foo-1.0.
 		Then, it scans the sys-apps/foo category and returns a list of sys-apps/foo packages that



  [1] or snapshot, mask, whatever you want to call it.  you can have
      as many as you want.

  [2] PORTAGE_LEVEL="mystate"

-- 
Terje - who's calling it a night.