[gentoo-catalyst] [PATCH 0/7] Latest pending branch fixes

[gentoo-catalyst] [PATCH 0/7] Latest pending branch fixes

[Brian Dolbec]

From c51b35d3fe4be92c663682e0c1d2dfef7ecd8cfb Mon Sep 17 00:00:00 2001
From: Brian Dolbec <dolsen@gentoo.org>
Date: Fri, 10 Mar 2017 10:53:42 -0800
Subject: [PATCH 0/7] Latest pending branch fixes
To: gentoo-catalyst@lists.gentoo.org

This series of patches fixes several issues.

1) makes the handling of control-c better for cleanup.
2) Fixes an auto-resume problem and cleans up the logic a lot.
3) Fixes the make.conf bleed thruogh of host repo settings.
4) puts all the make.conf write logic into one place
5) Snapshots are now named according to the repo_name setting, so
configurable. 6) with the new defaults for repo, DISTDIR, PKGDIR I was
able to prove it can handle any directory settings.  With these
settings, I was able to set the config to the
old /usr/portage, /usr/portage/distfiles... and everything worked as
expected.  So, we just need to get council to decree the new
locations.  No point in discussing it on the -dev list, they couldn't
come to a consensus.  But there was enough to get some resonable values.

Brian Dolbec (7):
  Move the signal handler into the StageBase class so it can handle
    unbind() cleanup
  base/stagebase.py: Rename unpack_portage resume point to unpack_repo
  base/stagebase.py: Correctly log the correct function name for
    unpack_snapshot()
  base/stagebase.py: Seperate out the writing of the make.conf file
  targets/snapshot.py: Update the code and log messages to use the
    configured repo_name
  defaults.py: Update all repository, DISTDIR, PKGDIR settings to new
    /var paths
  doc/make_subarch_table_guidexml.py: Fix shebang

 bin/catalyst                       |  19 ----
 catalyst/base/stagebase.py         | 221
 +++++++++++++++++++++----------------
 catalyst/defaults.py               |  24 ++--
 catalyst/targets/snapshot.py       |  13 ++-
 doc/make_subarch_table_guidexml.py |   2 +-
 etc/catalyst.conf                  |   2 + 6 files changed, 148
 insertions(+), 133 deletions(-)

-- 
2.12.0



-- 
Brian Dolbec <dolsen>

[gentoo-catalyst] [PATCH 1/7] Move the signal handler into the StageBase class so it can handle unbind() cleanup

[Brian Dolbec]

From 106c588e852927d244df2a9b66d188c60252e31d Mon Sep 17 00:00:00 2001
From: Brian Dolbec <dolsen@gentoo.org>
Date: Mon, 7 Sep 2015 23:15:25 -0700
Subject: [PATCH 1/7] Move the signal handler into the StageBase class so it
 can handle unbind() cleanup
To: gentoo-catalyst@lists.gentoo.org

Not quite complete, still errors on some unmounting
---
 bin/catalyst               | 19 -------------------
 catalyst/base/stagebase.py | 19 +++++++++++++++++++
 2 files changed, 19 insertions(+), 19 deletions(-)

diff --git a/bin/catalyst b/bin/catalyst
index 72a4cb4d..a64cfce8 100755
--- a/bin/catalyst
+++ b/bin/catalyst
@@ -12,25 +12,6 @@ from __future__ import print_function
 
 import sys
 
-# This block ensures that ^C interrupts are handled quietly.
-try:
-	import signal
-
-	def exithandler(_signum, _frame):
-		signal.signal(signal.SIGINT, signal.SIG_IGN)
-		signal.signal(signal.SIGTERM, signal.SIG_IGN)
-		print()
-		sys.exit(1)
-
-	signal.signal(signal.SIGINT, exithandler)
-	signal.signal(signal.SIGTERM, exithandler)
-	signal.signal(signal.SIGPIPE, signal.SIG_DFL)
-
-except KeyboardInterrupt:
-	print()
-	sys.exit(1)
-
-
 from catalyst.main import main
 
 try:
diff --git a/catalyst/base/stagebase.py b/catalyst/base/stagebase.py
index b857a64b..eed14589 100644
--- a/catalyst/base/stagebase.py
+++ b/catalyst/base/stagebase.py
@@ -2,6 +2,7 @@
 import os
 import imp
 import shutil
+import signal
 import sys
 
 from snakeoil import fileutils
@@ -206,6 +207,17 @@ class StageBase(TargetBase, ClearBase, GenBase):
 		if "portage_confdir" in self.settings:
 			file_locate(self.settings, ["portage_confdir"], expand = 0)
 
+
+		# This block ensures that ^C interrupts are handled quietly.
+		try:
+			signal.signal(signal.SIGINT, self.exithandler)
+			signal.signal(signal.SIGTERM, self.exithandler)
+			signal.signal(signal.SIGPIPE, signal.SIG_DFL)
+
+		except KeyboardInterrupt:
+			print()
+			sys.exit(1)
+
 		# Setup our mount points.
 		# initialize our target mounts.
 		self.target_mounts = TARGET_MOUNT_DEFAULTS.copy()
@@ -268,6 +280,13 @@ class StageBase(TargetBase, ClearBase, GenBase):
 			self.env["PORT_LOGDIR"] = self.settings["port_logdir"]
 			self.env["PORT_LOGDIR_CLEAN"] = PORT_LOGDIR_CLEAN
 
+	def exithandler(self, _signum, _frame):
+		signal.signal(signal.SIGINT, signal.SIG_IGN)
+		signal.signal(signal.SIGTERM, signal.SIG_IGN)
+		self.unbind()
+		print()
+		sys.exit(1)
+
 	def override_cbuild(self):
 		if "CBUILD" in self.makeconf:
 			self.settings["CBUILD"] = self.makeconf["CBUILD"]
-- 
2.12.0



-- 
Brian Dolbec <dolsen>

[gentoo-catalyst] [PATCH 6/7] defaults.py: Update all repository, DISTDIR, PKGDIR settings to new /var paths

[Brian Dolbec]

From 926c42c264e2fdb2987e145ae848fccfaa51be39 Mon Sep 17 00:00:00 2001
From: Brian Dolbec <dolsen@gentoo.org>
Date: Thu, 9 Mar 2017 01:24:22 -0800
Subject: [PATCH 6/7] defaults.py: Update all repository, DISTDIR, PKGDIR
 settings to new /var paths
To: gentoo-catalyst@lists.gentoo.org

All settings are still configurable in catalyst.conf.
These are new reasonable settings for a relocated tree system.
A user can still set their own locations during install, or at a later time.

Default catalyst.conf settings will be changed at a later time after additional testing and
the automatic stage building scripts have been updated.
---
 catalyst/defaults.py | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/catalyst/defaults.py b/catalyst/defaults.py
index 84ed2822..947e236c 100644
--- a/catalyst/defaults.py
+++ b/catalyst/defaults.py
@@ -39,24 +39,24 @@ confdefaults={
 	"compressor_options": XATTRS_OPTIONS[TAR],
 	"decomp_opt": DECOMPRESSOR_PROGRAM_OPTIONS[TAR],
 	"decompressor_search_order": DECOMPRESSOR_SEARCH_ORDER,
-	"distdir": "/usr/portage/distfiles",
+	"distdir": "/var/portage/distfiles",
 	"hash_function": "crc32",
 	"icecream": "/var/cache/icecream",
 	'list_xattrs_opt': LIST_XATTRS_OPTIONS[TAR],
-	"local_overlay": "/usr/local/portage",
+	"local_overlay": "/var/portage/repos/local",
 	"port_conf": "/etc/portage",
 	"make_conf": "%(port_conf)s/make.conf",
 	"options": set(),
-	"packagedir": "/usr/portage/packages",
-	"portdir": "/usr/portage",
+	"packagedir": "/var/portage/packages",
+	"portdir": "/var/portage/repos",
 	"port_tmpdir": "/var/tmp/portage",
 	"PythonDir": "./catalyst",
-	"repo_basedir": "/usr",
-	"repo_name": "portage",
+	"repo_basedir": "/var/portage/repos",
+	"repo_name": "gentoo",
 	"sharedir": "/usr/share/catalyst",
 	"shdir": "/usr/share/catalyst/targets/",
 	"snapshot_cache": "/var/tmp/catalyst/snapshot_cache",
-	"snapshot_name": "portage-",
+	"snapshot_name": "%(repo_name)s-",
 	"source_matching": "strict",
 	"storedir": "/var/tmp/catalyst",
 	"target_distdir": "/var/portage/distfiles",
@@ -75,8 +75,8 @@ TARGET_MOUNT_DEFAULTS = {
 	"distdir": "/usr/portage/distfiles",
 	"icecream": "/usr/lib/icecc/bin",
 	"kerncache": "/tmp/kerncache",
-	"packagedir": "/usr/portage/packages",
-	"portdir": "/usr/portage",
+	"packagedir": "/var/portage/packages",
+	"portdir": "/var/portage/repos",
 	"port_tmpdir": "/var/tmp/portage",
 	"port_logdir": "/var/log/portage",
 	"proc": "/proc",
@@ -86,8 +86,8 @@ TARGET_MOUNT_DEFAULTS = {
 SOURCE_MOUNT_DEFAULTS = {
 	"dev": "/dev",
 	"devpts": "/dev/pts",
-	"distdir": "/usr/portage/distfiles",
-	"portdir": "/usr/portage",
+	"distdir": "/var/portage/distfiles",
+	"portdir": "/var/portage/repos",
 	"port_tmpdir": "tmpfs",
 	"proc": "/proc",
 	"shm": "shmfs",
-- 
2.12.0



-- 
Brian Dolbec <dolsen>

Re: [gentoo-catalyst] [PATCH 6/7] defaults.py: Update all repository, DISTDIR, PKGDIR settings to new /var paths

[Zac Medico]

On Fri, Mar 10, 2017 at 11:23 AM, Brian Dolbec <dolsen@gentoo.org> wrote:
> @@ -75,8 +75,8 @@ TARGET_MOUNT_DEFAULTS = {
>         "distdir": "/usr/portage/distfiles",
>         "icecream": "/usr/lib/icecc/bin",
>         "kerncache": "/tmp/kerncache",
> -       "packagedir": "/usr/portage/packages",
> -       "portdir": "/usr/portage",
> +       "packagedir": "/var/portage/packages",
> +       "portdir": "/var/portage/repos",
>         "port_tmpdir": "/var/tmp/portage",
>         "port_logdir": "/var/log/portage",
>         "proc": "/proc",
> @@ -86,8 +86,8 @@ TARGET_MOUNT_DEFAULTS = {
>  SOURCE_MOUNT_DEFAULTS = {
>         "dev": "/dev",
>         "devpts": "/dev/pts",
> -       "distdir": "/usr/portage/distfiles",
> -       "portdir": "/usr/portage",
> +       "distdir": "/var/portage/distfiles",
> +       "portdir": "/var/portage/repos",
>         "port_tmpdir": "tmpfs",
>         "proc": "/proc",
>         "shm": "shmfs",

Shouldn't those portdir settings be /var/portage/repos/gentoo or
something like that?
-- 
Thanks,
Zac

Re: [gentoo-catalyst] [PATCH 6/7] defaults.py: Update all repository, DISTDIR, PKGDIR settings to new /var paths

[Brian Dolbec]

On Fri, 10 Mar 2017 11:51:07 -0800
Zac Medico <zmedico@gentoo.org> wrote:

> On Fri, Mar 10, 2017 at 11:23 AM, Brian Dolbec <dolsen@gentoo.org>
> wrote:
> > @@ -75,8 +75,8 @@ TARGET_MOUNT_DEFAULTS = {
> >         "distdir": "/usr/portage/distfiles",
> >         "icecream": "/usr/lib/icecc/bin",
> >         "kerncache": "/tmp/kerncache",
> > -       "packagedir": "/usr/portage/packages",
> > -       "portdir": "/usr/portage",
> > +       "packagedir": "/var/portage/packages",
> > +       "portdir": "/var/portage/repos",
> >         "port_tmpdir": "/var/tmp/portage",
> >         "port_logdir": "/var/log/portage",
> >         "proc": "/proc",
> > @@ -86,8 +86,8 @@ TARGET_MOUNT_DEFAULTS = {
> >  SOURCE_MOUNT_DEFAULTS = {
> >         "dev": "/dev",
> >         "devpts": "/dev/pts",
> > -       "distdir": "/usr/portage/distfiles",
> > -       "portdir": "/usr/portage",
> > +       "distdir": "/var/portage/distfiles",
> > +       "portdir": "/var/portage/repos",
> >         "port_tmpdir": "tmpfs",
> >         "proc": "/proc",
> >         "shm": "shmfs",  
> 
> Shouldn't those portdir settings be /var/portage/repos/gentoo or
> something like that?

No, that is the base directory for it to unpack or rsync the repo to.
It is a little confusing because of the way the code was written before.
I'll have to get in there and rename that to 'reposdir' to better
reflect the way it is used.

Note, those settings were only temporary for testing that the code was
ready for relocation.  I think I'll change them to /var/gentoo as the
base for now.  Then we can change them when the council has decreed the
final location.

-- 
Brian Dolbec <dolsen>

Re: [gentoo-catalyst] [PATCH 6/7] defaults.py: Update all repository, DISTDIR, PKGDIR settings to new /var paths

[Brian Dolbec]

On Fri, 10 Mar 2017 12:36:06 -0800
Brian Dolbec <dolsen@gentoo.org> wrote:

> On Fri, 10 Mar 2017 11:51:07 -0800
> Zac Medico <zmedico@gentoo.org> wrote:
> 
> > On Fri, Mar 10, 2017 at 11:23 AM, Brian Dolbec <dolsen@gentoo.org>
> > wrote:  
> > > @@ -75,8 +75,8 @@ TARGET_MOUNT_DEFAULTS = {
> > >         "distdir": "/usr/portage/distfiles",
> > >         "icecream": "/usr/lib/icecc/bin",
> > >         "kerncache": "/tmp/kerncache",
> > > -       "packagedir": "/usr/portage/packages",
> > > -       "portdir": "/usr/portage",
> > > +       "packagedir": "/var/portage/packages",
> > > +       "portdir": "/var/portage/repos",
> > >         "port_tmpdir": "/var/tmp/portage",
> > >         "port_logdir": "/var/log/portage",
> > >         "proc": "/proc",
> > > @@ -86,8 +86,8 @@ TARGET_MOUNT_DEFAULTS = {
> > >  SOURCE_MOUNT_DEFAULTS = {
> > >         "dev": "/dev",
> > >         "devpts": "/dev/pts",
> > > -       "distdir": "/usr/portage/distfiles",
> > > -       "portdir": "/usr/portage",
> > > +       "distdir": "/var/portage/distfiles",
> > > +       "portdir": "/var/portage/repos",
> > >         "port_tmpdir": "tmpfs",
> > >         "proc": "/proc",
> > >         "shm": "shmfs",    
> > 
> > Shouldn't those portdir settings be /var/portage/repos/gentoo or
> > something like that?  
> 
> No, that is the base directory for it to unpack or rsync the repo to.
> It is a little confusing because of the way the code was written
> before. I'll have to get in there and rename that to 'reposdir' to
> better reflect the way it is used.
> 
> Note, those settings were only temporary for testing that the code was
> ready for relocation.  I think I'll change them to /var/gentoo as the
> base for now.  Then we can change them when the council has decreed
> the final location.
> 

Also, with these code changes, it completely separates the source host
system source tree used to generate the snapshot and the target
location and name of the snapshot tree and it's new location during
stage generation.  The new stage will end up with the new settings in
its make.conf.  This makes it possible for a host with
traditional /usr/portage or any other location or name to generate a
snapshot with a completely different base repo name.  This will make it
possible for the infra severs generating the snapshot and stages to not
need to be migrated to the new defaults in order to generate the
snapshots and stages with the new settings.  In fact, using an
alternate config, the same servers could generate both old (portage) and
new (gentoo) snapshots for an interim if desired.  Well, I think the
snapshots should be to give time for users to migrate their systems for
the new tree directory name.  That is if this code is used to generate
the daily tree snapshots.

One thing to note is that you must first edit the catalyst.conf to
add/edit the new config variables, then generate a new snapshot before
doing stage runs.

repo_basedir="/usr"
repo_name="portage"
target_distdir="/usr/portage/distfiles"
target_pkgdir="/usr/portage/packages"

Using these settings, creates a snapshot and make.conf with the
traditional locations directory names.  And completely overrides the
new defaults.

Changing the reponame to "gentoo" requires you first generate a
snapshot (the repo_name is used as the target dir and in the final
tarball name).  The stage building process will look for the snapshot
with that same name and specified version.

So, derivative distros could more easily use catalyst to generate their
own stages and snapshots.  Not to mention gentoo change locations and
repo names to something new without needing code changes to do so.  It
would be a simple config edit.
-- 
Brian Dolbec <dolsen>

Re: [gentoo-catalyst] [PATCH 1/7] Move the signal handler into the StageBase class so it can handle unbind() cleanup

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 784 bytes --]

On 10 Mar 2017 11:22, Brian Dolbec wrote:
> Subject: [PATCH 1/7] Move the signal handler into the StageBase class so it
>  can handle unbind() cleanup
> 
> Not quite complete, still errors on some unmounting

i don't think moving signal handling into a random module makes sense.
the main loop should be responsible for this business.  i.e. where the
code is now (or in catalyst._main).

if the only thing you want to handle is clean up of mounts, then we
should be looking at mount namespaces instead.  unlike other namespaces,
support for mount namespaces has been around forever and can't be turned
off in the kernel.

the releng repo deals with this by running everything through unshare.

snakeoil already includes namespace support.  i'll send a patch.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-catalyst] [PATCH 6/7] defaults.py: Update all repository, DISTDIR, PKGDIR settings to new /var paths

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 834 bytes --]

On 10 Mar 2017 11:23, Brian Dolbec wrote:
> -	"distdir": "/usr/portage/distfiles",
> +	"distdir": "/var/portage/distfiles",
> -	"local_overlay": "/usr/local/portage",
> +	"local_overlay": "/var/portage/repos/local",
> -	"packagedir": "/usr/portage/packages",
> +	"packagedir": "/var/portage/packages",
> -	"portdir": "/usr/portage",
> +	"portdir": "/var/portage/repos",
> -	"repo_basedir": "/usr",
> +	"repo_basedir": "/var/portage/repos",

sticking everything under /var/portage doesn't seem like an improvement
to me.  where was the discussion that ended up with these values ?  seems
like it should be:
	repo_basedir:  /var/lib/portage/repos/
	portdir:       /var/lib/portage/repos/gentoo/
	local_overlay: /var/lib/portage/repos/local/
	packagedir:    /var/lib/portage/packages/
	distdir:       /var/cache/portage/distfiles/
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-catalyst] [PATCH 6/7] defaults.py: Update all repository, DISTDIR, PKGDIR settings to new /var paths

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 1898 bytes --]

On Mon, 20 Mar 2017 05:02:18 -0400
Mike Frysinger <vapier@gentoo.org> wrote:

> On 10 Mar 2017 11:23, Brian Dolbec wrote:
> > -	"distdir": "/usr/portage/distfiles",
> > +	"distdir": "/var/portage/distfiles",
> > -	"local_overlay": "/usr/local/portage",
> > +	"local_overlay": "/var/portage/repos/local",
> > -	"packagedir": "/usr/portage/packages",
> > +	"packagedir": "/var/portage/packages",
> > -	"portdir": "/usr/portage",
> > +	"portdir": "/var/portage/repos",
> > -	"repo_basedir": "/usr",
> > +	"repo_basedir": "/var/portage/repos",  
> 
> sticking everything under /var/portage doesn't seem like an
> improvement to me.  where was the discussion that ended up with these
> values ?  seems like it should be:
> 	repo_basedir:  /var/lib/portage/repos/
> 	portdir:       /var/lib/portage/repos/gentoo/
> 	local_overlay: /var/lib/portage/repos/local/
> 	packagedir:    /var/lib/portage/packages/
> 	distdir:       /var/cache/portage/distfiles/
> -mike


your behind the times, I changed it to /var/gentoo as the base in the
pending branch.  The main idea is to keep the path short as possible I
think

They were just temporary anyway.  But that is also what work sets them
under.  Personally I put them under /var/db on my machine, where the
vardb is stored.  But lately, I pretty much just use teh git tree wich
is in a subdir of my user home directory.

The original email thread in -dev a few years ago never could come to
any kind of consensus where the new default should be.  It is going to
have to be a council decision to set the new default.

We'll have to  send a new email to the list with the main options, try
to get them to vote on their preference, just to satisfy council before
they make a choice.

I need to send a couple more patches to the list.  But I need to get a
couple things cleared up first.
-- 
Brian Dolbec <dolsen>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-catalyst] [PATCH 6/7] defaults.py: Update all repository, DISTDIR, PKGDIR settings to new /var paths

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 2181 bytes --]

On 20 Mar 2017 02:21, Brian Dolbec wrote:
> On Mon, 20 Mar 2017 05:02:18 -0400 Mike Frysinger wrote:
> > On 10 Mar 2017 11:23, Brian Dolbec wrote:
> > > -	"distdir": "/usr/portage/distfiles",
> > > +	"distdir": "/var/portage/distfiles",
> > > -	"local_overlay": "/usr/local/portage",
> > > +	"local_overlay": "/var/portage/repos/local",
> > > -	"packagedir": "/usr/portage/packages",
> > > +	"packagedir": "/var/portage/packages",
> > > -	"portdir": "/usr/portage",
> > > +	"portdir": "/var/portage/repos",
> > > -	"repo_basedir": "/usr",
> > > +	"repo_basedir": "/var/portage/repos",  
> > 
> > sticking everything under /var/portage doesn't seem like an
> > improvement to me.  where was the discussion that ended up with these
> > values ?  seems like it should be:
> > 	repo_basedir:  /var/lib/portage/repos/
> > 	portdir:       /var/lib/portage/repos/gentoo/
> > 	local_overlay: /var/lib/portage/repos/local/
> > 	packagedir:    /var/lib/portage/packages/
> > 	distdir:       /var/cache/portage/distfiles/
> 
> your behind the times, I changed it to /var/gentoo as the base in the
> pending branch.  The main idea is to keep the path short as possible I
> think
> 
> They were just temporary anyway.  But that is also what work sets them
> under.  Personally I put them under /var/db on my machine, where the
> vardb is stored.  But lately, I pretty much just use teh git tree wich
> is in a subdir of my user home directory.
> 
> The original email thread in -dev a few years ago never could come to
> any kind of consensus where the new default should be.  It is going to
> have to be a council decision to set the new default.
> 
> We'll have to  send a new email to the list with the main options, try
> to get them to vote on their preference, just to satisfy council before
> they make a choice.
> 
> I need to send a couple more patches to the list.  But I need to get a
> couple things cleared up first.

OK, as long as we don't go merging these patches into master until we've
settled on the desired defaults.  it doesn't make sense to have catalyst
do one thing since it is effectively the default install state.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]