[gentoo-catalyst] More proposed Catalyst changes

[gentoo-catalyst] More proposed Catalyst changes

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 2393 bytes --]

While making changes to catalyst for the tree move I noticed that there
is in my view a serious design flaw in the way the python modules for it
are laid out.

The modules directory has no __init__.py so is not automatically
recognized by python as a directory containing python code...  So, there
is code in the main catalyst script to insert that path to it's modules
so that they are found.

Inside that modules directory is the main catalyst python package
directory.

These two directories are reversed.  The modules directory should be
nested inside the catalyst directory and have an __init__.py file so
that they are properly and automatically inside the catalyst namespace. 

The same is true of the arch directory and it's python files, which it
too should be a sub-package of the catalyst namespace.

The main catalyst start script. 
  I am a firm believer that the start script should be a minimal script
which imports code from the main package and runs it.  I would like to
move the current script into the catalyst directory (some editing
required) and add a minimal start script to a bin directory to avoid
confusion with the catalyst python package dir.  I would also split out
the confdefaults to a separate defaults file, so that any and all
defaults are in one central location.  This would make it easy to edit
for any future changes rather than chasing through all the code.  So all
of the current catalyst python code would be moved to files inside the
catalyst python package directory.


The long term benefit of this restructure is easier maintenance, and the
flexibility to create other front-ends to run it.  I know most/all of
you guys are cli junkies, but with all the working code inside the main
catalyst python package, a web interface could be made for it as well as
gtk, qt, etc. quis.  I'm not saying we would, but the option to is.
From what little I know of the releng teams work.  A web frontend might
be the most logical alternate interface.  It would be setup to run on
one or more of gentoo's build servers.  And with a few more extensions
to the code, be able to provide progress status of the build run.

For examples of this structure, have a look at current gentoolkit's
eclean, enalyze code, layman, mirrorselect, and the emaint rewrite now
included in portage.
-- 
Brian Dolbec <dolsen@gentoo.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-catalyst] More proposed Catalyst changes

[Peter Stuge]

[-- Attachment #1: Type: text/plain, Size: 408 bytes --]

Brian Dolbec wrote:
> I know most/all of you guys are cli junkies, but with all the
> working code inside the main catalyst python package, a web
> interface could be made for it as well as gtk, qt, etc. quis.
> I'm not saying we would, but the option to is.

I think nobody cares at all as long as the CLI stays the same. I'm
all for improving internal structure, so I think you should go for
it.


//Peter

[-- Attachment #2: Type: application/pgp-signature, Size: 190 bytes --]

Re: [gentoo-catalyst] More proposed Catalyst changes

[Brian Dolbec]

[-- Attachment #1.1: Type: text/plain, Size: 1418 bytes --]

On Tue, 2013-01-08 at 00:32 -0800, Brian Dolbec wrote:
> While making changes to catalyst for the tree move I noticed that there
> is in my view a serious design flaw in the way the python modules for it
> are laid out.

I've done quite a bit of the basic restructure.  I still have a number
of files I can shorten the names for, now that they are in aptly named
directories.  There is a lot of "from foo import *" imports to clean up
and specify the actual imports needed.

I have tested the code so far by making a snapshot run:

catalyst -c ./test.conf -s -f testsnap.spec

The code can be run directly from the git checkout with only a few
things needed to set it up

I personally create a "testpath" file (attached) in the base git
directory which I source with:

source ./testpath

just edit it to suit your paths.

I also created a test.conf file which I edited 2 variables to suit:

envscript="/home/brian/Dev/git/catalist/etc/catalystrc"
sharedir="/home/brian/Dev/git/catalyst/targets.sh"


I welcome more testing, beware you are likely to get some tracebacks
which I would like you to send me so I can fix them.

I will do more testing as I get more of the restructure finished, then
begin more thorough testing/debugging.

You can get a git clone of my rewrite work at:
http://dev.gentoo.org/~dolsen/catalyst/

rewrite branch

-- 
Brian Dolbec <dolsen@gentoo.org>

[-- Attachment #1.2: testpath --]
[-- Type: text/plain, Size: 125 bytes --]


export PATH="/home/brian/Dev/git/catalyst/bin:${PATH}"

export PYTHONPATH="/home/brian/Dev/git/catalyst/:${PYTHONPATH}"

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-catalyst] More proposed Catalyst changes

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 5880 bytes --]

On Sat, Jan 12, 2013 at 12:55:58AM -0800, Brian Dolbec wrote:
> You can get a git clone of my rewrite work at:
> http://dev.gentoo.org/~dolsen/catalyst/
> 
> rewrite branch

Ah, it's good to see things looking more Pythonic in Catalyst-land ;).
Here's my initial review to push this forward, although I'm not
particularly well-versed in the Catalyst internals.  This is branch is
getting to the stage where rebasing will get annoying, so I think
merging stuff that looks good into the master branch should happen
sooner, rather than later ;).

321df3b whitespace cleanup

This looks good, but there are also whitespace cleanups in some other
commits (e.g. 2d36d29).  I think we should just run something like:

  for FILE in $(git ls-tree -r --name-only HEAD | grep -v 'bz2$'); do
    sed -i 's/[[:space:]]*$//' "$FILE"
  done

and have done with it ;).

38cd3bb add more configured defaults

List the new settings (distdir, repo_name, packagedir, port_tmpdir,
options, snapshot_name) in the commit message so I don't have to read
the diff.  Probably explain why you think they should be configurable
as well.  `options` is also a pretty ambiguous name.

016704a use the new configured snapshot_name and portdir settings

Rather than a separate add (38cd3bb) and use (016704a), I think it
would be better if new options had their addition and use rolled into
a single commit (e.g. add snapshot_name and use it as one commit, add
portdir and use it as another commit).

This might also be a good place to move the path construction over to
use os.path.join().

fefd501 use the portdir setting rather than hard-coded path

Good to merge.

63a25eb Remove self.mounts and self.mountmap's use of paths for keys and paths. Migrate more hardcoded paths to use settings.

That's a long commit summary ;).  Perhaps the “Migrate…” portion
should go into the commit message body, or that could be split out
into a separate commit.

This might also be a good time to transition from `CatalystError, …`
to `CatalystError(…)` where you're touching lines.

I'm not sure which versions of Python Catalyst is trying to support,
but I'd be happy to see the beginings of a migration to '{}'.format()
for building strings (vs. the current `'' + ''` concatenation)

60919e4 Apply all Matt Turner's has_key() replacement patches

This looks like a job for `rebase` ;).  Cherry-picking will just lead
to redundant commits.

7a909b9 cleanup long lines, improve useage() output formatting slightly

Can we just switch from getopt to argparse (Python ≥2.7)?

af6e6b7 Initial rearrangement of the python directories

Hooray!

5eeb8b1 new minimal start script

I think __version__ and __maintainer__ should live in
catalyst/__init__.py.

267dbb6 add __init__.py's to modules and arch sub-pkgs

Added with blank lines?

299e35d update the module loading paths for the new locations

Can we just drop this import manipulation and use __all__?

e26de08 chmod +x

Squash into 5eeb8b1.

78d8be2 fix catalyst_support import to new location and specify imported modules

Looks good.

2d36d29 move catalyst_support, builder, catalyst_lock out of modules, into the catalyst's base namespace

Yay!  There are some whitespace changes in here too though (see my
comments on 321df3b).

bc66f5d rename the modules subpkg to targets, it better reflects what it contains

Looks good.

a7206bb rename files directory to etc to better reflect the directories contents

Actually, files/ is also used for other things (e.g. built man pages,
see MAN_PAGES in the Makefile).  If we keep the rename to etc/, we
might to just build the man pages under doc/.

da6cbbf rename shell script targets directory to targets.sh so as to not be confused with the new python targets directory

“so as…” should go in the commit message body to get a shorter
summary.

I'm not sold on this.  Since they are data files intended to be used
by the Python modules, why not move them to catalyst/targets/scripts [1]?

f9f18be update gitignore

The Scratch, catalystc, *.geany, and test* entries would appear to be
specific to your local installation and usage.  I'd drop them from the
reroll.

d23d96b revert urllib import from commit 64c16cae70da13de3c55d8555a2e4c5dcdf2fcad since it is not used anywhere in catalyst

64c16cae does a lot more than import urllib, so I think “revert” gives
the wrong impression.  I'd remove the import entirely (instead of
commenting it out), and say:

  catalyst/support.py: Remove unused urllib import

  This was introduced in commit
  64c16cae70da13de3c55d8555a2e4c5dcdf2fcad but is no longer used.

ab2c22a comment out some code introduced in commit b3475906d5f51a21ecaf4ff048002a2f44face52 with an undefined variable "s".  The code must always be hitting the general except: pass block.  Seems useless, if not maybe it'll spit out a true error, so the real problem can be fixed

Multiple sentences in a commit summary!  Also, take advantage of your
version control and just rip the code out instead of commenting it
out.  I looked through b347590 (which is a 408-line monster for
catalyst_support.py), but I'm not clear why Eric was messing with that
code.  At any rate, it looks like a good cantidate for replacement
with subprocess.Popen, although see 1b2597e ;).

95704c7 fix an undefined variable

Oops ;).  I'd mention DESIRED_RLIMIT in the commit message.

1b2597e update todo for a spawn() cleanup.

+lots

I'll see if I can get through 968d818, … sometime later today.

Cheers,
Trevor

[1]: http://docs.python.org/2/distutils/setupscript.html#installing-package-data

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] More proposed Catalyst changes

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 7886 bytes --]

On Thu, Jan 31, 2013 at 01:39:22PM -0500, W. Trevor King wrote:
> I'll see if I can get through 968d818, … sometime later today.

I had a second wind ;).  Here are some comments on the remaining
commits:

968d818 Initial creation of a defaults file. Split out hash and contents to their own classes, files

After this, it's not clear to me what the difference is between
catalyst.support and catalyst.util.  Perhaps they should be merged.

I'd also use catalyst.targets.__all__ instead of coding a list of
targets in the apparently unrelated
catalyst.defaults.valid_build_targets.

a4ef493 re-version to "git-rewrite branch"

Why?

9d752a7 move confdefaults out of main.py

Looks good, except, I'm not sure why you changed from
`confdefaults.keys()` to `list(confdefaults)` in parse_config() (which
should probably be living in catalyst.config anyway).

c303dae some options cleanup, unifying their use, reducing redundancy.

While I like the general thrust of this, I'd be happier with explicit
boolean options instead of a set of boolean options.  For example:

  confdefaults = {
    'autoresume': False,
    'ccache': False,
    …
		}

6aeb5e3 Move LockInUse from support.py to lock.py, fix bad execption raising,  pyflakes cleanup

With respect to the exception raising, see my comments on 63a25eb in
the previous email.  Ahh, it looks like you finish that off with
04068a1.

04068a1 massive pyflakes import cleanup and broken CatalystError calls

A lot of this is great stuff (Hooray no `import *`!).  However, I'd
split the `print_traceback=True` changes out into their own commit (or
just change the default value for CatalystError while you're testing).

It looks like an unrelated change to
catalyst.defaults.required_build_targets snuck into this commit.

The hash_map changes should probably be squashed into the earlier hash
reorganization, rather than going into this commit.

c672887 begin splitting up generic_stage_target into smaller code blocks so snapshot_target does not need to import it since most of it was not used or initialized properly.

I'd shift “so snapshot_target…” into the commit body for a shorter
summary.  Not much to say about the code, since I haven't dug though
it yet.

2b29212 some spacing and comment cleanup, etc.

Looks ok.

28888a7 remove redundant /bin/bash additions in cmd() calls

Looks good.

97802b6 clean out the old spawn functions, use subprocess.call() instead

Hooray.  Since you're doing this, I'd drop the earlier commits that
referenced this change (e.g. d23d96b and 1b2597e).

624eb0d move base stage and target files to thier own sub-pkg

s/thier/their/

The diff looks good.

5c689a8 update module loading for the new python structure, rename snapshot_target to snapshot

I think I like this.  It's good to use __import__.  You're missing a
blank line after your import_module docstring summary [1].

I think that the import_module API should match Python 3's
importlib.import_module [2].

669451d fix options being reset by a config file

See my comments on c303dae.  We should really be using ConfigParser's
defaults here, instead of rolling our own default system.

f56017b fix missed PURGEONLY changes

This should be squashed into the settings work that caused it.

c5eb7f7 fix mounts, mountmap hardcoding removal, use normpath to fix paths, add some debug print statements, remove clear & purge code from support.py,

Split the long commit summary.

Can we use logging instead of print?

The except (Failure loading " + x + " plugin) should be squashed into
the commit that caused it and be restricted to only catch
ImportErrors.

The mountmap fix for 'proc' → '/proc' should be squashed into the
commit that caused it.

-               mypath=self.settings["chroot_path"]
+               #mypath=self.settings["chroot_path"]

Just remove this entirely, don't comment it out.

b2ea321 remove a named parameter in generate_hash() call due to a parameter name change

Move “due to …” into the commit message body.

I think keyword arguments are better, because changes to keywords
usually occur alongside changes to the argument semantics.  A keyword
mismatch is an obvious fix, while changes due to a semantic shift can
be more subtle.

a847803 Use normpath from support

Why are we not using os.path.normpath?

Also, remove instead of commenting out.

6108400 chmod +x all sh scripts so they can run from the git checkout

Looks good,

242c17d rename all target .py file and classes without the "_target" so they are the same as the .sh files and work with teh simplified module loading.

Move “so they …” into the commit message body and s/teh/the/.

I was going to suggest this if you hadn't gotten around to it ;).

ecc653f fix targets.sh path

Squash into the commit that caused it.

d563be8 Comment out a small code block causing TypeError, which also was short circuiting another large code block.  FIXME!!!! This whole class seems overly complicated with TOO MANY nested try:excepts:

Long commit summary.

I agree that a few of these methods should be split into helper
functions for easier reading.

e05cc34 Break out a few more repeated (path1 + path2)'s  to doing it once and using the temp variable. comment out some debug print's

Long commit summary.

os.path.join!  Also, convert debug prints to use the logging module
;).

e0738c0 rename a make.conf key to make_conf due to bash variable name restrictions

Squash into causing commit.

db2af17 fix my broken subprocess call to use Popen due to environment passing and return code needs

Squash into causing commit.

ae61e4a reduce 2 operations into one simpler one

Just switch to ConfigParser ;).

b08a757 extend ParserBase to do variable substitution.

ConfigParser does that to ;).

265c8ed add a "shdir" setting to make moving the bash code around easier and have less hardcoded paths in the bash scripts

Long commit summary.

Your comment claims catalyst runtime executables for both sharedir and
shdir.

dc90e40 migrate all target shell scripts to use the new shdir setting

Squash into 265c8ed.

2fa8968 minor cleanup

Squash into earlier (proposed) global whitespace cleanup and import
cleanups.

d7007a9 fix a bug in relative path use.  Add some unique identifiers to echo statements to make identifying the source of a message easier.

Split into two commits (relative path, unique identifiers).  We should
also try and figure out less arbitrary identifiers.

8b9a1c0 add embedded variable substitiution to default settings

ConfigParser does this.

1259fa6 make shdir a complete path to ease it's use. Add port_config path setting to remove more hard coded paths in code.

Squash shdir stuff into the earlier shdir addition.

Squash port_config into wherever you use it…

923e8a2 remove trailing slash for consistency in variables and remove extra slashes in paths

os.path.join()

55a58ee Add a forced debug print statement in cmd() for better debug output

Python logging.

24d69bd Commit my testpath file with instructions to run the git checkout code directly without being installed.

s/caatalyst/catalyst/

It would also probably be a good idea to add an example test.conf with
relative paths.

b7036f3 update gitignore

As I said earlier, the test.* files your ignoring don't seem to exist
in my checkout.

0fb2609 add archdir to settings

Looks good.

Cheers,
Trevor

[1]: http://www.python.org/dev/peps/pep-0257/#multi-line-docstrings
[2]: http://docs.python.org/dev/library/importlib.html#importlib.import_module

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] More proposed Catalyst changes

[Brian Dolbec]

On Thu, 2013-01-31 at 13:39 -0500, W. Trevor King wrote:
> On Sat, Jan 12, 2013 at 12:55:58AM -0800, Brian Dolbec wrote:
> > You can get a git clone of my rewrite work at:
> > http://dev.gentoo.org/~dolsen/catalyst/
> > 
> > rewrite branch
> 
> Ah, it's good to see things looking more Pythonic in Catalyst-land ;).
> Here's my initial review to push this forward, although I'm not
> particularly well-versed in the Catalyst internals.  This is branch is
> getting to the stage where rebasing will get annoying, so I think
> merging stuff that looks good into the master branch should happen
> sooner, rather than later ;).
> 

There are a few fixes that should be for sure merged into master.  The
rest I intend to rebase them into more meaningful complete commits. 

A couple general answers for questions in this and your next review
email:


 I am halting further changes at this point except for what's needed to
debug everything up to this point.  I will fix/rebase commits to clean
them up as much as possible.  I do think that it will be difficult to
make complete commits for some changes that do not break catalyst until
the remaining commits are also merged.


Thank you for enough info for me to google about commit summary, message
body.  Until now I didn't know about needing to separate them with a
blank line.  I had never come across that info before, so had given up
trying to format them better.  Everything I tried had failed.  Even
googling it now it seems to be everywhere but git's own help.




> 321df3b whitespace cleanup
> 
> This looks good, but there are also whitespace cleanups in some other
> commits (e.g. 2d36d29).  I think we should just run something like:
> 
>   for FILE in $(git ls-tree -r --name-only HEAD | grep -v 'bz2$'); do
>     sed -i 's/[[:space:]]*$//' "$FILE"
>   done
> 
> and have done with it ;).

yeah, that requires way more knowledge about those tools than I know.
For me, I open a file, the editor does the cleanup automatically.  If I
don't separate them when committing, then...  If I didn't edit the
file...

> 
> 38cd3bb add more configured defaults
> 
> List the new settings (distdir, repo_name, packagedir, port_tmpdir,
> options, snapshot_name) in the commit message so I don't have to read
> the diff.  Probably explain why you think they should be configurable
> as well.  `options` is also a pretty ambiguous name.

Now that I know how to blab on in a message body with a brief summary,
yeah :)

"options" was existing and seems comparable to FEATURES for emerge in
make.conf.  Suggestions for a new name?

> 
> 016704a use the new configured snapshot_name and portdir settings
> 
> Rather than a separate add (38cd3bb) and use (016704a), I think it
> would be better if new options had their addition and use rolled into
> a single commit (e.g. add snapshot_name and use it as one commit, add
> portdir and use it as another commit).

I will in the rebase, making new commits.

> 
> This might also be a good place to move the path construction over to
> use os.path.join().

Yes, I agree, but I am still getting accustomed to the code, so was
holding off that for a bit.  Need to debug changes first.  I am also
contemplating stealing the path() from layman which joins a list of
partial paths.  I've noticed it could simplify code in many places that
join more than 2 partial paths.

> 
> fefd501 use the portdir setting rather than hard-coded path
> 
> Good to merge.
> 
> 63a25eb Remove self.mounts and self.mountmap's use of paths for keys
and paths. Migrate more hardcoded paths to use settings.
> 
> That's a long commit summary ;).  Perhaps the “Migrate…” portion
> should go into the commit message body, or that could be split out
> into a separate commit.

will fix

> 
> This might also be a good time to transition from `CatalystError, …`
> to `CatalystError(…)` where you're touching lines.
> 
> I'm not sure which versions of Python Catalyst is trying to support,
> but I'd be happy to see the beginings of a migration to '{}'.format()
> for building strings (vs. the current `'' + ''` concatenation)

yeah, there are a ton of those in the code.

Since portage is python-2.6+ and python-2.5 is pretty close to being
removed from the tree... I intend to make it, 2.6, 2.7, 3.2 +
compatible. 

> 
> 60919e4 Apply all Matt Turner's has_key() replacement patches
> 
> This looks like a job for `rebase` ;).  Cherry-picking will just lead
> to redundant commits.
> 


That will be looked after in my rebase to master again.  Since I had
already made a number of changes in my rewrite branch, I could not apply
his commit cleanly without editing.



> 7a909b9 cleanup long lines, improve useage() output formatting
slightly
> 
> Can we just switch from getopt to argparse (Python ≥2.7)?

Yes, we can also add argparse as dep for 2.6 if they want to maintain
2.6 compatibility.

> 
> af6e6b7 Initial rearrangement of the python directories
> 
> Hooray!
> 
> 5eeb8b1 new minimal start script
> 
> I think __version__ and __maintainer__ should live in
> catalyst/__init__.py.
> 

Can do.  The version can also be set for releases easily using some
methods from gentoolkit when making the release tarball.  The version
could then be left as "git" with a partial hash added of the last commit
when it was merged with.


> 267dbb6 add __init__.py's to modules and arch sub-pkgs
> 
> Added with blank lines?

I cheated at the time and just used my editor to save an existing one to
different places.  So it likely saved it with a blank line.  Instead of
using a terminal and touch.

> 
> 299e35d update the module loading paths for the new locations
> 
> Can we just drop this import manipulation and use __all__?
> 
> e26de08 chmod +x
> 
> Squash into 5eeb8b1.
> 
yup

> a7206bb rename files directory to etc to better reflect the
directories contents
> 
> Actually, files/ is also used for other things (e.g. built man pages,
> see MAN_PAGES in the Makefile).  If we keep the rename to etc/, we
> might to just build the man pages under doc/.
> 
Sounds like a plan to me.

> da6cbbf rename shell script targets directory to targets.sh so as to
not be confused with the new python targets directory
> 
> “so as…” should go in the commit message body to get a shorter
> summary.
> 
> I'm not sold on this.  Since they are data files intended to be used
> by the Python modules, why not move them to catalyst/targets/scripts
[1]?
 
I had been re-thinking that change and was thinking of renaming it back
to targets now that things have been better rearranged.  There is one py
file in stage1.  It is also easier to do now since it is just one edit
and a git mv away.  I've removed that directory name from being hard
coded in the scripts, so now it just means there is one place to edit.


> f9f18be update gitignore
> 
> The Scratch, catalystc, *.geany, and test* entries would appear to be
> specific to your local installation and usage.  I'd drop them from the
> reroll.
> 

They are only for my dev space git repo and me.  They will not be part
of a final ready to merge branch.  But my dev space is also acting as my
backup, so I committed them.  It makes my git status output cleaner.


> d23d96b revert urllib import from commit
64c16cae70da13de3c55d8555a2e4c5dcdf2fcad since it is not used anywhere
in catalyst
> 
> 64c16cae does a lot more than import urllib, so I think “revert” gives
> the wrong impression.  I'd remove the import entirely (instead of
> commenting it out), and say:
> 
>   catalyst/support.py: Remove unused urllib import
> 
>   This was introduced in commit
>   64c16cae70da13de3c55d8555a2e4c5dcdf2fcad but is no longer used.
> 

Actually urllib was never used.  It was added as a fix for a bug.  It
was not a correct fix, so just masked what was causing the bug.  I only
commented it out until I could either find what caused the bug or it
just didn't return due to other changes.  It is often easier to
comment/uncomment during testing runs than to remove/re-add code.


> ab2c22a comment out some code introduced in commit
b3475906d5f51a21ecaf4ff048002a2f44face52 with an undefined variable "s".
The code must always be hitting the general except: pass block.  Seems
useless, if not maybe it'll spit out a true error, so the real problem
can be fixed
> 
> Multiple sentences in a commit summary!  Also, take advantage of your
> version control and just rip the code out instead of commenting it
> out. 

I did that because I did not fully figure out what that whole code block
was suppose to do and refactor it all.  Leaving it commented out leaves
it there for that purpose later which also stands out as a reminder.


 I looked through b347590 (which is a 408-line monster for
> catalyst_support.py), but I'm not clear why Eric was messing with that
> code.  At any rate, it looks like a good cantidate for replacement
> with subprocess.Popen, although see 1b2597e ;).

Yeah, there are a lot of os.system() calls throughout the code to
replace.

> 
> 95704c7 fix an undefined variable
> 
> Oops ;).  I'd mention DESIRED_RLIMIT in the commit message.
> 

yeah.

> 1b2597e update todo for a spawn() cleanup.
> 
> +lots
> 

yeah, there are lots of things to add to the todo.  I just didn't want
to forget about that one.  But if others are going to help, keeping that
up to date will be more important.


> Cheers,
> Trevor
> 
> [1]:
http://docs.python.org/2/distutils/setupscript.html#installing-package-data
> 

Re: [gentoo-catalyst] More proposed Catalyst changes

[Brian Dolbec]

On Thu, 2013-01-31 at 14:46 -0500, W. Trevor King wrote:
> On Thu, Jan 31, 2013 at 01:39:22PM -0500, W. Trevor King wrote:
> > I'll see if I can get through 968d818, … sometime later today.
> 
> I had a second wind ;).  Here are some comments on the remaining
> commits:
> 
> 968d818 Initial creation of a defaults file. Split out hash and
> contents to their own classes, files
> 
> After this, it's not clear to me what the difference is between
> catalyst.support and catalyst.util.  Perhaps they should be merged.

I hadn't looked at util until now.

hmm, I wonder if it would be better to move CatalystError there and
rename it to error.py  There is only a couple error traceback functions
in it.  I was thinking it might be good to define a few more specific
error classes than just the general CatalystError().  But I am not yet
familiar enough with the code to know for certain.

> 
> I'd also use catalyst.targets.__all__ instead of coding a list of
> targets in the apparently unrelated
> catalyst.defaults.valid_build_targets.

yeah, that's part of the old plugin system that was there.  I've
simplified it some, but it needs more work.  That variable is pretty
much useless now if I remember correctly. I was contemplating whether it
would justify using the plugin system I did for emaint.  It would be
easily imported from portage.  But I think it might be more than is
needed for this.

> 
> a4ef493 re-version to "git-rewrite branch"
> 
> Why?
> 

Why not.  It is not intended to be pushed into master.  It also helps to
confirm that you are running the correct code.  I have fixed it so the
code can be run from the git checkout.  That way you can have a
"Production" version installed on the same system.  Now that I am
testing/debugging, I am comparing catalyst operation and results to
current -9999 code.  I am not familiar with using catalyst, so that is
helping me figure out what is wrong.


> 9d752a7 move confdefaults out of main.py
> 
> Looks good, except, I'm not sure why you changed from
> `confdefaults.keys()` to `list(confdefaults)` in parse_config() (which
> should probably be living in catalyst.config anyway).

keeping a separate defaults file can be helpful in importing some info
into different modules while keeping imports to a minimum.  Sometimes it
helps prevents circular import problems.  At this point I opted for a
separate file, to be determined later if a merge is warranted.

As for list(confdefaults), py3 compatibility.  dict.keys() isn't usable
and 2to3 converts it to list(dict)... something about needing to specify
the return type.  So is a preemptive change.   One less thing to change
later.

> 
> c303dae some options cleanup, unifying their use, reducing redundancy.
> 
> While I like the general thrust of this, I'd be happier with explicit
> boolean options instead of a set of boolean options.  For example:
> 
>   confdefaults = {
>     'autoresume': False,
>     'ccache': False,
>     …
> 		}
> 

Yeah, I removed those.  They were capitalized versions of the values in
options.  So, I optimized them into options becoming a set which
eliminates, the duplication and potential problems by changing the value
of one and not the other.  Believe me keeping 2 different lists in sync
can be much more difficult than it seems.  It also makes things much
more difficult to debug.  (the independent booleans like you suggest can
be considered a list, the other is the options list, set,
string...whatever form it is in)  

They are set in default's "options" and updated from a config files's
"options".  So keeping them in "options" makes more sense to me, I just
convert them into a set to eliminate any duplicates.  Member inclusion
equates to True, while not being a member False.

if "ccache" in settings["options"]:

is just as easy to read as

if settings["ccache"]:  or if settings["ccache"] == True: ...

also using member inclusion is faster and prefered compared to other
methods like has_key().  In this case, it is just simpler to use
"options" rather than to individualize them.



> 6aeb5e3 Move LockInUse from support.py to lock.py, fix bad execption
> raising,  pyflakes cleanup
> 
> With respect to the exception raising, see my comments on 63a25eb in
> the previous email.  Ahh, it looks like you finish that off with
> 04068a1.
> 
> 04068a1 massive pyflakes import cleanup and broken CatalystError calls
> 
> A lot of this is great stuff (Hooray no `import *`!).  

Yeah, those import * were hiding a LOT of sins, turning catalyst into a
huge global variable using app for almost everything. 

> However, I'd
> split the `print_traceback=True` changes out into their own commit (or
> just change the default value for CatalystError while you're testing).
> 

I added that to try and limit traceback printing to areas that could
benefit from it for debugging. Reducing noise in others.  It still needs
fine tuning.

> It looks like an unrelated change to
> catalyst.defaults.required_build_targets snuck into this commit.

will look into it


> 28888a7 remove redundant /bin/bash additions in cmd() calls
> 
> Looks good.
> 
> 97802b6 clean out the old spawn functions, use subprocess.call()
> instead
> 
> Hooray.  Since you're doing this, I'd drop the earlier commits that
> referenced this change (e.g. d23d96b and 1b2597e).
> 
> 624eb0d move base stage and target files to thier own sub-pkg
> 
> s/thier/their/
> 
> The diff looks good.
> 
> 5c689a8 update module loading for the new python structure, rename
> snapshot_target to snapshot
> 
> I think I like this.  It's good to use __import__.  You're missing a
> blank line after your import_module docstring summary [1].
> 
> I think that the import_module API should match Python 3's
> importlib.import_module [2].

Yeah, there is more to do in this.  There are other areas that need
changing too.

> 
> 669451d fix options being reset by a config file
> 
> See my comments on c303dae.  We should really be using ConfigParser's
> defaults here, instead of rolling our own default system.
> 

Yes, I agree.


> Can we use logging instead of print?
> 

YES!!!!,  please :D

that's been on my wish list too. 


> -               mypath=self.settings["chroot_path"]
> +               #mypath=self.settings["chroot_path"]
> 
> Just remove this entirely, don't comment it out.

I sometimes do that if I'm unsure what I'm going to do, so makes it
easier to revert and go a different direction/track some changes during
debugging.  Sometimes, I forget to remove it after I've settled on the
change and done the testing.  It will be cleaned up after debugging,
when I make final clean commits to a new branch.


> I think keyword arguments are better, because changes to keywords
> usually occur alongside changes to the argument semantics.  A keyword
> mismatch is an obvious fix, while changes due to a semantic shift can
> be more subtle.

yeah, I was a bit frustrated at that point, debugging code, so chose the
easy way.  /me fixes.

> 
> a847803 Use normpath from support
> 
> Why are we not using os.path.normpath?
> 

I haven't figured that one out either.  os.path.normpath is used in
places.



> d7007a9 fix a bug in relative path use.  Add some unique identifiers
> to echo statements to make identifying the source of a message easier.
> 
> Split into two commits (relative path, unique identifiers).  We should
> also try and figure out less arbitrary identifiers.
> 

yeah, it does need splitting.  It was easy at the time, this is
something for the powers that be (releng team) to have some output
over... It may also be something that gets cleaned once changes &
debugging are done.



> 923e8a2 remove trailing slash for consistency in variables and remove
> extra slashes in paths
> 
> os.path.join()

Yes, for sure.  But after I debug my current changes.  But also most of
those are for the bash side consistency which can not use os.path.join()
and were adding the slashes again at times.

I am not a bash programmer, so if someone good at the bash stuff wants
to work on those... go for it.  I'll try to keep my damage to a minimum.

Also if releng want to recode them in python, that's ok with me ;)


> 24d69bd Commit my testpath file with instructions to run the git
> checkout code directly without being installed.
> 
> s/caatalyst/catalyst/
> 
my bad typing, also is partially caused by a tremor in my hand :(

> It would also probably be a good idea to add an example test.conf with
> relative paths.
> 
ok


> Cheers,
> Trevor
> 
> [1]: http://www.python.org/dev/peps/pep-0257/#multi-line-docstrings
> [2]:
> http://docs.python.org/dev/library/importlib.html#importlib.import_module
> 

Thank you for the review, it has been informative.  And good to keep me
from any blunders.

Re: [gentoo-catalyst] More proposed Catalyst changes

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 8395 bytes --]

On Sat, Feb 02, 2013 at 10:45:19AM -0800, Brian Dolbec wrote:
> I do think that it will be difficult to make complete commits for
> some changes that do not break catalyst until the remaining commits
> are also merged.

This may be true for some of the shifting commits, we'll see after the
initial reroll.

> Thank you for enough info for me to google about commit summary,
> message body.  Until now I didn't know about needing to separate
> them with a blank line.  I had never come across that info before,
> so had given up trying to format them better.  Everything I tried
> had failed.  Even googling it now it seems to be everywhere but
> git's own help.

Actually, it is in Git's help:

  $ git commit --help | grep -B3 blank
  DISCUSSION
    Though not required, it’s a good idea to begin the commit message with
    a single short (less than 50 character) line summarizing the change,
    followed by a blank line and then a more thorough description. The text
    up to the first blank line in a commit message is treated as the commit

> > 321df3b whitespace cleanup
> > 
> > This looks good, but there are also whitespace cleanups in some other
> > commits (e.g. 2d36d29).  I think we should just run something like:
> > 
> >   for FILE in $(git ls-tree -r --name-only HEAD | grep -v 'bz2$'); do
> >     sed -i 's/[[:space:]]*$//' "$FILE"
> >   done
> > 
> > and have done with it ;).
> 
> yeah, that requires way more knowledge about those tools than I know.
> For me, I open a file, the editor does the cleanup automatically.  If I
> don't separate them when committing, then...  If I didn't edit the
> file...

I've pushed this change to the `whitespace` branch on
git://tremily.us/catalyst.git if you want to rebase your changes onto
that.  I can also submit it as a patch to the mailing list if anyone
wants it that way.

> > 38cd3bb add more configured defaults
> > 
> > List the new settings (distdir, repo_name, packagedir, port_tmpdir,
> > options, snapshot_name) in the commit message so I don't have to read
> > the diff.  Probably explain why you think they should be configurable
> > as well.  `options` is also a pretty ambiguous name.
> 
> Now that I know how to blab on in a message body with a brief summary,
> yeah :)
> 
> "options" was existing and seems comparable to FEATURES for emerge in
> make.conf.  Suggestions for a new name?

In a later comment I suggested breaking each of the options variables
out into their own setting.  Then we don't have to worry about what to
call the boolean settings in aggregate ;).

> > This might also be a good place to move the path construction over to
> > use os.path.join().
> 
> Yes, I agree, but I am still getting accustomed to the code, so was
> holding off that for a bit.  Need to debug changes first.  I am also
> contemplating stealing the path() from layman which joins a list of
> partial paths.  I've noticed it could simplify code in many places that
> join more than 2 partial paths.

Doesn't os.path.join already do that?

  $ python2.7 -c "import os; print(os.path.join('a', 'b', 'c'))"
  a/b/c

> > This might also be a good time to transition from `CatalystError, …`
> > to `CatalystError(…)` where you're touching lines.
> > 
> > I'm not sure which versions of Python Catalyst is trying to support,
> > but I'd be happy to see the beginings of a migration to '{}'.format()
> > for building strings (vs. the current `'' + ''` concatenation)
> 
> yeah, there are a ton of those in the code.
> 
> Since portage is python-2.6+ and python-2.5 is pretty close to being
> removed from the tree... I intend to make it, 2.6, 2.7, 3.2 +
> compatible. 

So the CatalystError(…) updates will be fine, and we should use
'{0}'.format(xyz).

> > 60919e4 Apply all Matt Turner's has_key() replacement patches
> > 
> > This looks like a job for `rebase` ;).  Cherry-picking will just lead
> > to redundant commits.
> 
> That will be looked after in my rebase to master again.  Since I had
> already made a number of changes in my rewrite branch, I could not apply
> his commit cleanly without editing.

Right.  What you should have done is rebase onto the new `master`
branch.  Where Matt's changes clashed with yours, Git would stop and
ask to to fix the conflicts.  Then you could update *your* patches so
that they worked ;).  Since Matt's changes got into `master` first, he
gets to keep them the way they originally were.

> > 7a909b9 cleanup long lines, improve useage() output formatting
> slightly
> > 
> > Can we just switch from getopt to argparse (Python ≥2.7)?
> 
> Yes, we can also add argparse as dep for 2.6 if they want to maintain
> 2.6 compatibility.

Ah, that would be much nicer :).

> > 5eeb8b1 new minimal start script
> > 
> > I think __version__ and __maintainer__ should live in
> > catalyst/__init__.py.
> 
> Can do.  The version can also be set for releases easily using some
> methods from gentoolkit when making the release tarball.  The version
> could then be left as "git" with a partial hash added of the last commit
> when it was merged with.

I do something like this in bugs-everywhere [1], but for Catalyst I
think it may be more work than it's worth.

> > 267dbb6 add __init__.py's to modules and arch sub-pkgs
> > 
> > Added with blank lines?
> 
> I cheated at the time and just used my editor to save an existing one to
> different places.  So it likely saved it with a blank line.  Instead of
> using a terminal and touch.

These should probably get copyright blurbs and descriptive docstrings
anyway, so it's probably not worth worrying over the blank lines.

> > da6cbbf rename shell script targets directory to targets.sh so as to
> not be confused with the new python targets directory
> > 
> > “so as…” should go in the commit message body to get a shorter
> > summary.
> > 
> > I'm not sold on this.  Since they are data files intended to be used
> > by the Python modules, why not move them to catalyst/targets/scripts
> [1]?
>  
> I had been re-thinking that change and was thinking of renaming it back
> to targets now that things have been better rearranged.  There is one py
> file in stage1.  It is also easier to do now since it is just one edit
> and a git mv away.  I've removed that directory name from being hard
> coded in the scripts, so now it just means there is one place to edit.

This sounds reasonable to me.

> > f9f18be update gitignore
> > 
> > The Scratch, catalystc, *.geany, and test* entries would appear to be
> > specific to your local installation and usage.  I'd drop them from the
> > reroll.
> > 
> 
> They are only for my dev space git repo and me.  They will not be part
> of a final ready to merge branch.  But my dev space is also acting as my
> backup, so I committed them.  It makes my git status output cleaner.

Fair enough, but it's probably a good idea to put your local changes
in a clearly marked commit:

  $ git commit -am 'FIXME: local changes to gitignore'

> > d23d96b revert urllib import from commit
> 64c16cae70da13de3c55d8555a2e4c5dcdf2fcad since it is not used anywhere
> in catalyst
> > 
> > 64c16cae does a lot more than import urllib, so I think “revert” gives
> > the wrong impression.  I'd remove the import entirely (instead of
> > commenting it out), and say:
> > 
> >   catalyst/support.py: Remove unused urllib import
> > 
> >   This was introduced in commit
> >   64c16cae70da13de3c55d8555a2e4c5dcdf2fcad but is no longer used.
> 
> Actually urllib was never used.  It was added as a fix for a bug.  It
> was not a correct fix, so just masked what was causing the bug.  I only
> commented it out until I could either find what caused the bug or it
> just didn't return due to other changes.  It is often easier to
> comment/uncomment during testing runs than to remove/re-add code.

Sure.  It's strange that it had any effect at all on a bug if it was
never used, but whatever ;).

Cheers,
Trevor

[1]: http://gitorious.org/be/be/blobs/master/libbe/version.py#line21
     http://gitorious.org/be/be/blobs/master/Makefile#line80

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] More proposed Catalyst changes

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 6056 bytes --]

On Sat, Feb 02, 2013 at 12:41:32PM -0800, Brian Dolbec wrote:
> On Thu, 2013-01-31 at 14:46 -0500, W. Trevor King wrote:
> > 968d818 Initial creation of a defaults file. Split out hash and
> > contents to their own classes, files
> > 
> > After this, it's not clear to me what the difference is between
> > catalyst.support and catalyst.util.  Perhaps they should be merged.
> 
> I hadn't looked at util until now.
> 
> hmm, I wonder if it would be better to move CatalystError there and
> rename it to error.py  There is only a couple error traceback functions
> in it.  I was thinking it might be good to define a few more specific
> error classes than just the general CatalystError().  But I am not yet
> familiar enough with the code to know for certain.

This sounds good to me too.

> > a4ef493 re-version to "git-rewrite branch"
> > 
> > Why?
> 
> Why not.  It is not intended to be pushed into master.  It also helps to
> confirm that you are running the correct code.  I have fixed it so the
> code can be run from the git checkout.  That way you can have a
> "Production" version installed on the same system.  Now that I am
> testing/debugging, I am comparing catalyst operation and results to
> current -9999 code.  I am not familiar with using catalyst, so that is
> helping me figure out what is wrong.

A fair enough.  Again, it would be nice if there was a FIXME or
something in the commit message so I knew it wasn't destined for
master ;).

> > 9d752a7 move confdefaults out of main.py
> > 
> > Looks good, except, I'm not sure why you changed from
> > `confdefaults.keys()` to `list(confdefaults)` in parse_config() (which
> > should probably be living in catalyst.config anyway).
> 
> keeping a separate defaults file can be helpful in importing some info
> into different modules while keeping imports to a minimum.  Sometimes it
> helps prevents circular import problems.  At this point I opted for a
> separate file, to be determined later if a merge is warranted.

ok.

> As for list(confdefaults), py3 compatibility.  dict.keys() isn't usable
> and 2to3 converts it to list(dict)... something about needing to specify
> the return type.  So is a preemptive change.   One less thing to change
> later.

Really?

  $ python3.3 -c "a = {1:2, 3:4}; print([x for x in a.keys()])"
  [1, 3]

On the other hand, it might be cleaner to just say:

  for x in confdefaults:

But this should still go into a separate commit.

> > c303dae some options cleanup, unifying their use, reducing redundancy.
> > 
> > While I like the general thrust of this, I'd be happier with explicit
> > boolean options instead of a set of boolean options.  For example:
> > 
> >   confdefaults = {
> >     'autoresume': False,
> >     'ccache': False,
> >     …
> > 		}
> 
> Yeah, I removed those.  They were capitalized versions of the values in
> options.  So, I optimized them into options becoming a set which
> eliminates, the duplication and potential problems by changing the value
> of one and not the other.  Believe me keeping 2 different lists in sync
> can be much more difficult than it seems.  It also makes things much
> more difficult to debug.  (the independent booleans like you suggest can
> be considered a list, the other is the options list, set,
> string...whatever form it is in)  

I think I would do something like:

  import collections as _collections
  import ConfigParser as _configparser

  CONFIG = _configparser.ConfigParser(dict_type=_collections.OrderedDict)
  for setting,value in [
          ('ccache', str(False))]:
      CONFIG.set('DEFAULT', setting, value)

although OrderedDict doesn't exist in 2.6, where we should probably
just fall back to dicts.  Use it with:

  if CONFIG.getboolean('DEFAULT', 'ccache'):
      …

Then there's no duplication to worry about, and you get a
configuration object Python developers will be familiar with.

> also using member inclusion is faster and prefered compared to other
> methods like has_key().  In this case, it is just simpler to use
> "options" rather than to individualize them.

I don't think option lookup speed will have much impact on catalyst
execution speed ;).  And with ConfigParser, individual options will
require no additional coding.

> > Can we use logging instead of print?
> 
> YES!!!!,  please :D
> 
> that's been on my wish list too. 

I can add this if you don't want to.  Let me know if you want me to
base my patch against `master`, or against something in your branch.

> > I think keyword arguments are better, because changes to keywords
> > usually occur alongside changes to the argument semantics.  A keyword
> > mismatch is an obvious fix, while changes due to a semantic shift can
> > be more subtle.
> 
> yeah, I was a bit frustrated at that point, debugging code, so chose the
> easy way.  /me fixes.

;).  I've certainly been there too.

> > 923e8a2 remove trailing slash for consistency in variables and remove
> > extra slashes in paths
> > 
> > os.path.join()
> 
> Yes, for sure.  But after I debug my current changes.  But also most of
> those are for the bash side consistency which can not use os.path.join()
> and were adding the slashes again at times.
> 
> I am not a bash programmer, so if someone good at the bash stuff wants
> to work on those... go for it.  I'll try to keep my damage to a minimum.

I can look into this, but I think I need a better description of the
problem first.  Can you give me an example breakage due to using
os.path.join?

> Also if releng want to recode them in python, that's ok with me ;)

I'd be ok with that too, although I don't feel a pressing need for it
;).

> Thank you for the review, it has been informative.  And good to keep me
> from any blunders.

No problem :D

Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[gentoo-catalyst] patch, fix broken seed stage update

[Brian Dolbec]

The git branch is located at http://dev.gentoo.org/~dolsen/catalyst/
git checkout the rewrite branch.

For those of you that have already cloned it, use --force in your pull.
I've condensed, rearranged the changes into more logical complete
changes.  Still, I wouldn't think you could cherrypick any single commit
in general.  Some fixes could be cherrypicked to apply to current
master, some would likely have to be hand applied due to other changes.

Next on my todo list, fix doc's creation, create a setup.py and make the
code installable via ebuild.  

I think also development should continue in a branch on the main
catalyst repo on g.o.g.o.  Possibly name it catalyst3b so it does not
conflict with the catalyst3 branch started.  I looked at rebasing my
work on it, but decided against it.  There were far too many changes in
master since it was last updated.

There are far too many patches to individually list them in this mail
list.  Please checkout the branch from my repo to review the changes.
It would also be easier to use gitweb to view them online if it was
pushed to the main repo.


Anyway the latest patch...

======================================================================

Fix broken seed stage update...

Strip --usepkg and --buildpkg from emerge options for user defined
update_seed_command.
Add a check for the update_seed option to set the correct update
options.
Fix default seed stage update command to properly update gcc and it's
deps.
Add a seed stage update system command and option.
Add --binpkg-respect-use=y for all cases --usepkg is enabled.


+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

From 9ffa5b8812403bf20f17eba58543fc4b7c04bc33 Mon Sep 17 00:00:00 2001
From: Brian Dolbec <dolsen@gentoo.org>
Date: Mon, 25 Feb 2013 23:31:41 -0800
Subject: [PATCH] Fix broken seed stage update...

Strip --usepkg and --buildpkg from emerge options for user defined update_seed_command.
Add a check for the update_seed option to set the correct update options.
Fix default seed stage update command to properly update gcc and it's deps.
Add a seed stage update system command and option.
Add --binpkg-respect-use=y for all cases --usepkg is enabled.
---
 catalyst/targets/stage1.py          |  3 ++-
 doc/catalyst-spec.5.txt             | 12 +++++++++++-
 targets/stage1/stage1-chroot.sh     | 17 ++++++++++++++---
 targets/support/chroot-functions.sh | 11 ++++++++++-
 4 files changed, 37 insertions(+), 6 deletions(-)

diff --git a/catalyst/targets/stage1.py b/catalyst/targets/stage1.py
index e936929..e067c8c 100644
--- a/catalyst/targets/stage1.py
+++ b/catalyst/targets/stage1.py
@@ -18,7 +18,8 @@ class stage1(StageBase):
 	def __init__(self,spec,addlargs):
 		self.required_values=[]
 		self.valid_values=["chost"]
-		self.valid_values.extend(["update_seed","update_seed_command"])
+		self.valid_values.extend(["update_seed","update_seed_command",
+			"update_seed_system"])
 		StageBase.__init__(self,spec,addlargs)
 
 	def set_stage_path(self):
diff --git a/doc/catalyst-spec.5.txt b/doc/catalyst-spec.5.txt
index 4a6e06c..196bdc3 100644
--- a/doc/catalyst-spec.5.txt
+++ b/doc/catalyst-spec.5.txt
@@ -138,9 +138,19 @@ it should update the seed stage or not (valid values: `yes no`).
 *update_seed_command*::
 This is an optional command to pass to emerge for updating the seed
 stage (example: `--update dev-libs/mpfr dev-libs/mpc dev-libs/gmp`)
-If not specified, catalyst will update gcc deps.
+If not specified, catalyst will update gcc's deps, and rebuild gcc if any of
+it's deps are updated with a new version. Even if it itself is not updated.
+This prevents gcc breakage when it's dependency lib sonames have changed.
 This setting requires enabling update_seed.
 
+*update_seed_system*::
+This is an optional setting supported by stage1 to tell catalyst if
+it should update the seed's system packages or not (valid values: `yes no`).
+This is run after any update_seed_command, updating any remaining upgradable
+system packages.
+This setting requires enabling update_seed.
+
+
 Compilation
 ~~~~~~~~~~~
 
diff --git a/targets/stage1/stage1-chroot.sh b/targets/stage1/stage1-chroot.sh
index 97aef7f..65c2d81 100755
--- a/targets/stage1/stage1-chroot.sh
+++ b/targets/stage1/stage1-chroot.sh
@@ -26,12 +26,23 @@ clst_root_path=/ setup_pkgmgr
 # Update stage3
 if [ -n "${clst_update_seed}" ]; then
 	if [ "${clst_update_seed}" == "yes" ]; then
-		echo "Updating seed stage..."
 		if [ -n "${clst_update_seed_command}" ]; then
-			clst_root_path=/ run_merge "--buildpkg=n ${clst_update_seed_command}"
+			echo "--- Updating seed stage with USER defined update_seed_command"
+			update_cmd=${clst_update_seed_command/--usepkg /}
+			update_cmd=${clst_update_seed_command/--buildpkg /}
+			clst_root_path=/ run_merge "${update_cmd}"
 		else
-			clst_root_path=/ run_merge "--buildpkg=n --update --deep --newuse --onlydeps gcc"
+			echo "--- Updating seed stage with DEFAULT update_seed_command"
+			update_cmd="--update --deep --complete-graph --rebuild-if-new-ver gcc"
+			clst_root_path=/ run_merge ${update_cmd}
 		fi
+		if [ "${clst_update_seed_system}" == "yes" ]; then
+			echo "--- Updating seed stage system packages"
+			update_cmd="--update --deep --complete-graph @system"
+			clst_root_path=/ run_merge ${update_cmd}
+		fi
+		# now reset the emerge options for the target
+		clst_update_seed=no setup_myemergeopts
 	elif [ "${clst_update_seed}" != "no" ]; then
 		echo "Invalid setting for update_seed: ${clst_update_seed}"
 		exit 1
diff --git a/targets/support/chroot-functions.sh b/targets/support/chroot-functions.sh
index 2524b4f..69d2923 100755
--- a/targets/support/chroot-functions.sh
+++ b/targets/support/chroot-functions.sh
@@ -133,9 +133,18 @@ setup_myemergeopts(){
 	then
 		export bootstrap_opts="${bootstrap_opts} -f"
 		export clst_myemergeopts="${clst_myemergeopts} -f"
+	# now intercept normal target options if we're updating the seed
+	# to update the seed we do not want binpkgs that may have links to
+	# sonames no longer installed, due to dependency updates.
+	# this function will be re-run later with clst_update_seed=no
+	elif [ "${clst_update_seed}" == "yes" ]
+	then
+		export clst_myemergeopts="${clst_myemergeopts} --newuse"
+		export bootstrap_opts="${bootstrap_opts} -r"
 	elif [ -n "${clst_PKGCACHE}" ]
 	then
-		export clst_myemergeopts="${clst_myemergeopts} --usepkg --buildpkg --newuse"
+		# if you add --usepkg, then also add --binpkg-respect-use=y
+		export clst_myemergeopts="${clst_myemergeopts} --usepkg --binpkg-respect-use=y --buildpkg --newuse"
 		export bootstrap_opts="${bootstrap_opts} -r"
 	fi
 }
-- 
1.8.1.2

Re: [gentoo-catalyst] patch, fix broken seed stage update

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1617 bytes --]

On Tue, Feb 26, 2013 at 08:20:45AM -0800, Brian Dolbec wrote:
> The git branch is located at http://dev.gentoo.org/~dolsen/catalyst/
> git checkout the rewrite branch.

Did you forget to push?

  $ git remote -v | grep dolsen
  dolsen  http://dev.gentoo.org/~dolsen/catalyst (fetch)
  dolsen  http://dev.gentoo.org/~dolsen/catalyst (push)
  $ git fetch dolsen
  $ git log --pretty=fuller dolsen/rewrite
  commit 0fb26090a730f876993c13e23b6da3effdd5d1ed
  Author:     Brian Dolbec <dolsen@gentoo.org>
  AuthorDate: Thu Jan 24 21:09:05 2013 -0800
  Commit:     Brian Dolbec <dolsen@gentoo.org>
  CommitDate: Thu Jan 24 21:13:01 2013 -0800

      add archdir to settings

> Next on my todo list, fix doc's creation, create a setup.py and make the
> code installable via ebuild.  

For what it's worth, I'd rather see this polished up an merged into
master before you add more work on top, mostly because that would make
it easier for me to help out ;).

> I think also development should continue in a branch on the main
> catalyst repo on g.o.g.o.  Possibly name it catalyst3b so it does not
> conflict with the catalyst3 branch started.  I looked at rebasing my
> work on it, but decided against it.  There were far too many changes in
> master since it was last updated.

Avoiding this is another reason to get fairly incremental portions of
your work merged back into master before they grow too unwieldy.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] patch, fix broken seed stage update

[Brian Dolbec]

On Tue, 2013-02-26 at 11:37 -0500, W. Trevor King wrote:
> On Tue, Feb 26, 2013 at 08:20:45AM -0800, Brian Dolbec wrote:
> > The git branch is located at http://dev.gentoo.org/~dolsen/catalyst/
> > git checkout the rewrite branch.
> 
> Did you forget to push?
> 
>   $ git remote -v | grep dolsen
>   dolsen  http://dev.gentoo.org/~dolsen/catalyst (fetch)
>   dolsen  http://dev.gentoo.org/~dolsen/catalyst (push)
>   $ git fetch dolsen
>   $ git log --pretty=fuller dolsen/rewrite
>   commit 0fb26090a730f876993c13e23b6da3effdd5d1ed
>   Author:     Brian Dolbec <dolsen@gentoo.org>
>   AuthorDate: Thu Jan 24 21:09:05 2013 -0800
>   Commit:     Brian Dolbec <dolsen@gentoo.org>
>   CommitDate: Thu Jan 24 21:13:01 2013 -0800
> 
>       add archdir to settings
> 

BAH!, forgot to ssh in and run git update-server-info on it

try again.

another reason to get a branch going on g.o.g.o

Also I've rebased everything on current master including W. Trevor
King's patches.

Re: [gentoo-catalyst] patch, fix broken seed stage update

[Peter Stuge]

Brian Dolbec wrote:
> BAH!, forgot to ssh in and run git update-server-info on it

Add a hook to do it for you in the future.


//Peter

Re: [gentoo-catalyst] patch, fix broken seed stage update

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 02/26/2013 11:20 AM, Brian Dolbec wrote:
> The git branch is located at http://dev.gentoo.org/~dolsen/catalyst/
> git checkout the rewrite branch.
> 
> For those of you that have already cloned it, use --force in your pull.
> I've condensed, rearranged the changes into more logical complete
> changes.  Still, I wouldn't think you could cherrypick any single commit
> in general.  Some fixes could be cherrypicked to apply to current
> master, some would likely have to be hand applied due to other changes.
> 
> Next on my todo list, fix doc's creation, create a setup.py and make the
> code installable via ebuild.  
> 
> I think also development should continue in a branch on the main
> catalyst repo on g.o.g.o.  Possibly name it catalyst3b so it does not
> conflict with the catalyst3 branch started.  I looked at rebasing my
> work on it, but decided against it.  There were far too many changes in
> master since it was last updated.
In my opinion you can kill the current catalyst 3 branch as nothing uses
it and afaik it doesn't work.  jmbvicetto should be able to
authoritatively agree or not.

- -Zero
> 
> There are far too many patches to individually list them in this mail
> list.  Please checkout the branch from my repo to review the changes.
> It would also be easier to use gitweb to view them online if it was
> pushed to the main repo.
> 
> 
> Anyway the latest patch...
> 
> ======================================================================
> 
> Fix broken seed stage update...
> 
> Strip --usepkg and --buildpkg from emerge options for user defined
> update_seed_command.
> Add a check for the update_seed option to set the correct update
> options.
> Fix default seed stage update command to properly update gcc and it's
> deps.
> Add a seed stage update system command and option.
> Add --binpkg-respect-use=y for all cases --usepkg is enabled.
> 
> 
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 
>>From 9ffa5b8812403bf20f17eba58543fc4b7c04bc33 Mon Sep 17 00:00:00 2001
> From: Brian Dolbec <dolsen@gentoo.org>
> Date: Mon, 25 Feb 2013 23:31:41 -0800
> Subject: [PATCH] Fix broken seed stage update...
> 
> Strip --usepkg and --buildpkg from emerge options for user defined update_seed_command.
> Add a check for the update_seed option to set the correct update options.
> Fix default seed stage update command to properly update gcc and it's deps.
> Add a seed stage update system command and option.
> Add --binpkg-respect-use=y for all cases --usepkg is enabled.
> ---
>  catalyst/targets/stage1.py          |  3 ++-
>  doc/catalyst-spec.5.txt             | 12 +++++++++++-
>  targets/stage1/stage1-chroot.sh     | 17 ++++++++++++++---
>  targets/support/chroot-functions.sh | 11 ++++++++++-
>  4 files changed, 37 insertions(+), 6 deletions(-)
> 
> diff --git a/catalyst/targets/stage1.py b/catalyst/targets/stage1.py
> index e936929..e067c8c 100644
> --- a/catalyst/targets/stage1.py
> +++ b/catalyst/targets/stage1.py
> @@ -18,7 +18,8 @@ class stage1(StageBase):
>  	def __init__(self,spec,addlargs):
>  		self.required_values=[]
>  		self.valid_values=["chost"]
> -		self.valid_values.extend(["update_seed","update_seed_command"])
> +		self.valid_values.extend(["update_seed","update_seed_command",
> +			"update_seed_system"])
>  		StageBase.__init__(self,spec,addlargs)
>  
>  	def set_stage_path(self):
> diff --git a/doc/catalyst-spec.5.txt b/doc/catalyst-spec.5.txt
> index 4a6e06c..196bdc3 100644
> --- a/doc/catalyst-spec.5.txt
> +++ b/doc/catalyst-spec.5.txt
> @@ -138,9 +138,19 @@ it should update the seed stage or not (valid values: `yes no`).
>  *update_seed_command*::
>  This is an optional command to pass to emerge for updating the seed
>  stage (example: `--update dev-libs/mpfr dev-libs/mpc dev-libs/gmp`)
> -If not specified, catalyst will update gcc deps.
> +If not specified, catalyst will update gcc's deps, and rebuild gcc if any of
> +it's deps are updated with a new version. Even if it itself is not updated.
> +This prevents gcc breakage when it's dependency lib sonames have changed.
>  This setting requires enabling update_seed.
>  
> +*update_seed_system*::
> +This is an optional setting supported by stage1 to tell catalyst if
> +it should update the seed's system packages or not (valid values: `yes no`).
> +This is run after any update_seed_command, updating any remaining upgradable
> +system packages.
> +This setting requires enabling update_seed.
> +
> +
>  Compilation
>  ~~~~~~~~~~~
>  
> diff --git a/targets/stage1/stage1-chroot.sh b/targets/stage1/stage1-chroot.sh
> index 97aef7f..65c2d81 100755
> --- a/targets/stage1/stage1-chroot.sh
> +++ b/targets/stage1/stage1-chroot.sh
> @@ -26,12 +26,23 @@ clst_root_path=/ setup_pkgmgr
>  # Update stage3
>  if [ -n "${clst_update_seed}" ]; then
>  	if [ "${clst_update_seed}" == "yes" ]; then
> -		echo "Updating seed stage..."
>  		if [ -n "${clst_update_seed_command}" ]; then
> -			clst_root_path=/ run_merge "--buildpkg=n ${clst_update_seed_command}"
> +			echo "--- Updating seed stage with USER defined update_seed_command"
> +			update_cmd=${clst_update_seed_command/--usepkg /}
> +			update_cmd=${clst_update_seed_command/--buildpkg /}
> +			clst_root_path=/ run_merge "${update_cmd}"
>  		else
> -			clst_root_path=/ run_merge "--buildpkg=n --update --deep --newuse --onlydeps gcc"
> +			echo "--- Updating seed stage with DEFAULT update_seed_command"
> +			update_cmd="--update --deep --complete-graph --rebuild-if-new-ver gcc"
> +			clst_root_path=/ run_merge ${update_cmd}
>  		fi
> +		if [ "${clst_update_seed_system}" == "yes" ]; then
> +			echo "--- Updating seed stage system packages"
> +			update_cmd="--update --deep --complete-graph @system"
> +			clst_root_path=/ run_merge ${update_cmd}
> +		fi
> +		# now reset the emerge options for the target
> +		clst_update_seed=no setup_myemergeopts
>  	elif [ "${clst_update_seed}" != "no" ]; then
>  		echo "Invalid setting for update_seed: ${clst_update_seed}"
>  		exit 1
> diff --git a/targets/support/chroot-functions.sh b/targets/support/chroot-functions.sh
> index 2524b4f..69d2923 100755
> --- a/targets/support/chroot-functions.sh
> +++ b/targets/support/chroot-functions.sh
> @@ -133,9 +133,18 @@ setup_myemergeopts(){
>  	then
>  		export bootstrap_opts="${bootstrap_opts} -f"
>  		export clst_myemergeopts="${clst_myemergeopts} -f"
> +	# now intercept normal target options if we're updating the seed
> +	# to update the seed we do not want binpkgs that may have links to
> +	# sonames no longer installed, due to dependency updates.
> +	# this function will be re-run later with clst_update_seed=no
> +	elif [ "${clst_update_seed}" == "yes" ]
> +	then
> +		export clst_myemergeopts="${clst_myemergeopts} --newuse"
> +		export bootstrap_opts="${bootstrap_opts} -r"
>  	elif [ -n "${clst_PKGCACHE}" ]
>  	then
> -		export clst_myemergeopts="${clst_myemergeopts} --usepkg --buildpkg --newuse"
> +		# if you add --usepkg, then also add --binpkg-respect-use=y
> +		export clst_myemergeopts="${clst_myemergeopts} --usepkg --binpkg-respect-use=y --buildpkg --newuse"
>  		export bootstrap_opts="${bootstrap_opts} -r"
>  	fi
>  }
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJRLPDkAAoJEKXdFCfdEflKLykP/Rx73fH878oLHFunNkfpa929
us+Kcf74QNvTjDUH2uzgofICiJFsnCdgUjmF0+gBDUY8DjxOjMYI4CrbqF6FEJ88
Q+rmN5rkcFvqRaVZrh480U9qPhm8JvYvIzviijBBmOqZDOgFQHpOPJMSeRcMAU/5
7fAT5zbmUGduKtecV/juI6UQimo5/eNuWd/z7BY3qAG/eh4skFK1M0JX2JJKnlAO
m5I5GCDKn32FQhhm5OdW+GFgS6c6MczBGGIEnyTwu7Lv50OC6XRm9xP4uiDjGOqW
A+tPhp1pidhamjCYhW0A+1fv5ZYe6DIZy/b8p/KrgB3oDKmhwWn83u1RhCu3GxMe
6Scg1KlnBJgPVa8b7OiS0lbJlrLAUlcQzJJKhvlrH7mRZ8vAqn4R77s31opuU2JU
UvZdchAYN/ziZG9xR/btrDE8yrpzwjiXj4Sc1BncqDMb2Sq/HLrxxYi0Tuk7UNDz
hQdjRBEaC8e/yvuPtdGqpoq9WojKeMH6DhBHI/FYhOlcqr4vHyPtiD6ioHwd92LK
iBAiAfQ5jjk+pDu1yIbOC5/GchWVV6hKjgz2qZd3dZPQq7YixRVImHR5roBXiKwm
4w9FutyE5slZZgFjIjrkcg1lp0emHI8osKfl9hf+wNvVXmgo8AZrrmidqgESNe8B
iQtoJi1P+d7Mee4cqlrI
=Yseo
-----END PGP SIGNATURE-----

Re: [gentoo-catalyst] patch, fix broken seed stage update

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 9746 bytes --]

On Tue, Feb 26, 2013 at 08:47:14AM -0800, Brian Dolbec wrote:
> Also I've rebased everything on current master

That should make things easier to merge :).

It looks like some of my earlier comments were addressed by this
reroll, but some are still applicable.  Apologies if we'd resolved any
of this earlier and I just missed the reference in my mailbox.

I map my old comments onto the rebased commits below, but the bulk of
the outstanding suggestions revolve around:

* ConfigParser-based configuration
* Argparse-based command line parsing
* Logging-based debugging output
* os.path.join(), normpath(), … for path manipulation

These are mostly “take advantage of Python's standard library”
changes, and I'd be happy to help implement them on top of the current
master if folks feel like that has a chance of getting merged ;).

On Thu, Jan 31, 2013 at 01:39:22PM -0500, W. Trevor King wrote:
> 38cd3bb add more configured defaults
> 
> List the new settings (distdir, repo_name, packagedir, port_tmpdir,
> options, snapshot_name) in the commit message so I don't have to read
> the diff.  Probably explain why you think they should be configurable
> as well.  `options` is also a pretty ambiguous name.

This still applies to 0a25452.  I made a few comments about using
separate boolean options instead of an aggregate `options` set.
Fixing this should be part of the ConfigParser transition.

> 016704a use the new configured snapshot_name and portdir settings
> 
> Rather than a separate add (38cd3bb) and use (016704a), I think it
> would be better if new options had their addition and use rolled into
> a single commit (e.g. add snapshot_name and use it as one commit, add
> portdir and use it as another commit).
> 
> This might also be a good place to move the path construction over to
> use os.path.join().

This still applies to be2f820.

> 63a25eb Remove self.mounts and self.mountmap's use of paths for keys and paths. Migrate more hardcoded paths to use settings.
> 
> That's a long commit summary ;).  Perhaps the “Migrate…” portion
> should go into the commit message body, or that could be split out
> into a separate commit.

This splitting happened in 77eece8, but…

> This might also be a good time to transition from `CatalystError, …`
> to `CatalystError(…)` where you're touching lines.
> 
> I'm not sure which versions of Python Catalyst is trying to support,
> but I'd be happy to see the beginings of a migration to '{}'.format()
> for building strings (vs. the current `'' + ''` concatenation)

… this still applies.

> 7a909b9 cleanup long lines, improve useage() output formatting slightly
> 
> Can we just switch from getopt to argparse (Python ≥2.7)?

Still applies to 7870959.

> 5eeb8b1 new minimal start script
> 
> I think __version__ and __maintainer__ should live in
> catalyst/__init__.py.

Still applies to baae19f.

> 299e35d update the module loading paths for the new locations
> 
> Can we just drop this import manipulation and use __all__?

Still applies to 79f73a3.

> a7206bb rename files directory to etc to better reflect the directories contents
> 
> Actually, files/ is also used for other things (e.g. built man pages,
> see MAN_PAGES in the Makefile).  If we keep the rename to etc/, we
> might to just build the man pages under doc/.

Still applies to 25f6f1b.

> f9f18be update gitignore
> 
> The Scratch, catalystc, *.geany, and test* entries would appear to be
> specific to your local installation and usage.  I'd drop them from the
> reroll.

They're still in 7fdecf4, but it will be easy to drop this before the
merge with master.

On Thu, Jan 31, 2013 at 02:46:53PM -0500, W. Trevor King wrote:
> 968d818 Initial creation of a defaults file. Split out hash and contents to their own classes, files
> 
> After this, it's not clear to me what the difference is between
> catalyst.support and catalyst.util.  Perhaps they should be merged.
> 
> I'd also use catalyst.targets.__all__ instead of coding a list of
> targets in the apparently unrelated
> catalyst.defaults.valid_build_targets.

Still applies to c97dc3d.

> 9d752a7 move confdefaults out of main.py
> 
> Looks good, except, I'm not sure why you changed from
> `confdefaults.keys()` to `list(confdefaults)` in parse_config() (which
> should probably be living in catalyst.config anyway).

Still applies to 0c2302a.  Additional discussion from a sub-thread:

On Sun, Feb 03, 2013 at 07:44:36AM -0500, W. Trevor King wrote:
> On Sat, Feb 02, 2013 at 12:41:32PM -0800, Brian Dolbec wrote:
> > As for list(confdefaults), py3 compatibility.  dict.keys() isn't usable
> > and 2to3 converts it to list(dict)... something about needing to specify
> > the return type.  So is a preemptive change.   One less thing to change
> > later.
> 
> Really?
> 
>   $ python3.3 -c "a = {1:2, 3:4}; print([x for x in a.keys()])"
>   [1, 3]
> 
> On the other hand, it might be cleaner to just say:
> 
>   for x in confdefaults:
> 
> But this should still go into a separate commit.

I still think this is true ;).

On Thu, Jan 31, 2013 at 02:46:53PM -0500, W. Trevor King wrote:
> c303dae some options cleanup, unifying their use, reducing redundancy.
> 
> While I like the general thrust of this, I'd be happier with explicit
> boolean options instead of a set of boolean options.  For example:
> 
>   confdefaults = {
>     'autoresume': False,
>     'ccache': False,
>     …
> 		}

Still applies to ca85cd4.  Like I said above, I'm happy to work up a
ConfigParser-based solution.

> 04068a1 massive pyflakes import cleanup and broken CatalystError calls
> 
> A lot of this is great stuff (Hooray no `import *`!).  However, I'd
> split the `print_traceback=True` changes out into their own commit (or
> just change the default value for CatalystError while you're testing).
> 
> It looks like an unrelated change to
> catalyst.defaults.required_build_targets snuck into this commit.
> 
> The hash_map changes should probably be squashed into the earlier hash
> reorganization, rather than going into this commit.

Still applies to eed08b1.

> 5c689a8 update module loading for the new python structure, rename snapshot_target to snapshot
> 
> I think I like this.  It's good to use __import__.  You're missing a
> blank line after your import_module docstring summary [1].
> 
> I think that the import_module API should match Python 3's
> importlib.import_module [2].

Still applies to still applies to f733b3f.

> 669451d fix options being reset by a config file
> 
> See my comments on c303dae.  We should really be using ConfigParser's
> defaults here, instead of rolling our own default system.

ConfigParser :).

> c5eb7f7 fix mounts, mountmap hardcoding removal, use normpath to fix paths, add some debug print statements, remove clear & purge code from support.py,
> 
> Split the long commit summary.

This was fixed in 9fabaae, but…

> Can we use logging instead of print?
> 
> The except (Failure loading " + x + " plugin) should be squashed into
> the commit that caused it and be restricted to only catch
> ImportErrors.
> 
> The mountmap fix for 'proc' → '/proc' should be squashed into the
> commit that caused it.
> 
> -               mypath=self.settings["chroot_path"]
> +               #mypath=self.settings["chroot_path"]
> 
> Just remove this entirely, don't comment it out.

… these were not.

> a847803 Use normpath from support
> 
> Why are we not using os.path.normpath?
> 
> Also, remove instead of commenting out.

Still applies to be413a4.

> 242c17d rename all target .py file and classes without the "_target" so they are the same as the .sh files and work with teh simplified module loading.
> 
> Move “so they …” into the commit message body and s/teh/the/.

With 6a86874, the commit body has “This is so they are the named the
same as the target…”, which should probably be “This is so they are
named the same as the target…” (removing an extra “the”).

> e05cc34 Break out a few more repeated (path1 + path2)'s  to doing it once and using the temp variable. comment out some debug print's
> 
> Long commit summary.

This is fixed in 4dd2cb2, but…

> os.path.join!  Also, convert debug prints to use the logging module
> ;).

… these are not.

> e0738c0 rename a make.conf key to make_conf due to bash variable name restrictions
> 
> Squash into causing commit.

Still applies to 253dab7.

> ae61e4a reduce 2 operations into one simpler one
> 
> Just switch to ConfigParser ;).

Still applies to 92c6745.

> b08a757 extend ParserBase to do variable substitution.
> 
> ConfigParser does that to ;).

Still applies to 104c387.

> 265c8ed add a "shdir" setting to make moving the bash code around easier and have less hardcoded paths in the bash scripts
> 
> Long commit summary.

This is fixed in b433a82, but…

> Your comment claims catalyst runtime executables for both sharedir and
> shdir.

… this is not.
 
> 55a58ee Add a forced debug print statement in cmd() for better debug output
> 
> Python logging.

Still applies to 3791efc.

> 24d69bd Commit my testpath file with instructions to run the git checkout code directly without being installed.
> 
> s/caatalyst/catalyst/
> 
> It would also probably be a good idea to add an example test.conf with
> relative paths.

Still applies to 786a01c.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] patch, fix broken seed stage update

[Matt Turner]

On Tue, Feb 26, 2013 at 9:29 AM, Rick "Zero_Chaos" Farina
<zerochaos@gentoo.org> wrote:
> In my opinion you can kill the current catalyst 3 branch as nothing uses
> it and afaik it doesn't work.  jmbvicetto should be able to
> authoritatively agree or not.

If Brian's work is to be the basis for future revisions of catalyst,
it should be the master branch. We can branch current master as a
stable branch. The current catalyst2 branch is totally dead, and
overwriting it doesn't cause us to lose anything.

... but the patches should be sent to the list (with git
format-patch/send-email) and reviewed before being committed to
master.

Re: [gentoo-catalyst] patch, fix broken seed stage update

[Brian Dolbec]

On Tue, 2013-02-26 at 13:04 -0500, W. Trevor King wrote:
> On Tue, Feb 26, 2013 at 08:47:14AM -0800, Brian Dolbec wrote:
> > Also I've rebased everything on current master
> 
> That should make things easier to merge :).
> 
> It looks like some of my earlier comments were addressed by this
> reroll, but some are still applicable.  Apologies if we'd resolved any
> of this earlier and I just missed the reference in my mailbox.
> 
> I map my old comments onto the rebased commits below, but the bulk of
> the outstanding suggestions revolve around:
> 
> * ConfigParser-based configuration
> * Argparse-based command line parsing
> * Logging-based debugging output
> * os.path.join(), normpath(), … for path manipulation
> 
> These are mostly “take advantage of Python's standard library”
> changes, and I'd be happy to help implement them on top of the current
> master if folks feel like that has a chance of getting merged ;).
> 

All that is for a TODO list.  What I did was rebase my existing changes,
merging fixes to the commits that made the change..., fix the commit
messages, etc...

Same with the other things you listed below, mostly for a TODO and your
wishlist.  Pretty much all I agree with.


My aim for now is to debug the hell out of it, to stabilize all the
existing changes, BEFORE doing a ton more changes.  That will also give
me more experience in how catalyst is used, possibly ideas how to change
it for the better.  I'll fix the doc's generation and make a setup.py
and new 9999 ebuild.

Plus I need to focus more on another project needed for the git tree
migration.


> Cheers,
> Trevor
> 

Re: [gentoo-catalyst] patch, fix broken seed stage update

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1503 bytes --]

On Tue, Feb 26, 2013 at 05:30:40PM -0800, Brian Dolbec wrote:
> My aim for now is to debug the hell out of it, to stabilize all the
> existing changes, BEFORE doing a ton more changes.  That will also give
> me more experience in how catalyst is used, possibly ideas how to change
> it for the better.  I'll fix the doc's generation and make a setup.py
> and new 9999 ebuild.

I'm not suggesting additional changes on top of yours, I'm suggesting
new ones underneath yours, or alternatives to your current series.  It
seems a shame to spend time testing a work in progress, since you'll
probably want to re-test the final form before merging.  I'm trying to
help polish your series down before we invest a lot of time testing.

> Plus I need to focus more on another project needed for the git tree
> migration.

I have no problem waiting on the catalyst front if it helps push the
Git migration through.  I've been wishing for that for a while :D.

Instead of making my suggested changes on top of your series, I can
also make them underneath your series, and help rebase your series on
top.  That gets the changes I want in, and lets you focus on other
things at the same time ;).

I'm also ok with just waiting until folks have more time (e.g. the
catalyst-spec man page cooked for five months ;).

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] patch, fix broken seed stage update

[Brian Dolbec]

On Tue, 2013-02-26 at 11:39 -0800, Matt Turner wrote:
> On Tue, Feb 26, 2013 at 9:29 AM, Rick "Zero_Chaos" Farina
> <zerochaos@gentoo.org> wrote:
> > In my opinion you can kill the current catalyst 3 branch as nothing uses
> > it and afaik it doesn't work.  jmbvicetto should be able to
> > authoritatively agree or not.
> 
> If Brian's work is to be the basis for future revisions of catalyst,
> it should be the master branch. We can branch current master as a
> stable branch. The current catalyst2 branch is totally dead, and
> overwriting it doesn't cause us to lose anything.
> 

I generally agree that my rewrite should become the master branch, but
it could live in a side branch for a while longer, there are many more
changes needed to clean up the code.  It is likely to be broken at times
some commits by themselves won't be complete, etc...


> ... but the patches should be sent to the list (with git
> format-patch/send-email) and reviewed before being committed to
> master.
> 

I tried using git to send this patch, but it did not work.  I'll look
into how to make it work with email.

But I do question the need to send all patches to this list for review.
Take this patch email for instance.  It was primarily for the last
commit I did to fix the update_seed methods which I found not working
correctly. I also mentioned I had rebased all my other work, condensing
where it made sense...

 Do you see any code suggestions, questions and/or comments in regards
to it... NO

All mails in this thread are about everything else :/

I will continue to spam the list with all my rebased commits, which I
intended to do when I had more time.  I sent this one off this morning
before I went to work.  I know Jorge was interested in the changes.  I'm
back now, so will manually prepare more patch mails.

Re: [gentoo-catalyst] patch, fix broken seed stage update

[Brian Dolbec]

On Tue, 2013-02-26 at 20:40 -0500, W. Trevor King wrote:
> On Tue, Feb 26, 2013 at 05:30:40PM -0800, Brian Dolbec wrote:
> > My aim for now is to debug the hell out of it, to stabilize all the
> > existing changes, BEFORE doing a ton more changes.  That will also give
> > me more experience in how catalyst is used, possibly ideas how to change
> > it for the better.  I'll fix the doc's generation and make a setup.py
> > and new 9999 ebuild.
> 
> I'm not suggesting additional changes on top of yours, I'm suggesting
> new ones underneath yours, or alternatives to your current series.  It
> seems a shame to spend time testing a work in progress, since you'll
> probably want to re-test the final form before merging.  I'm trying to
> help polish your series down before we invest a lot of time testing.
> 

NNNNNNNNNNNNOOOOOOOOOOOOOOOOOOOOOOOO :(  not more rebasing  I spent
several days just tracking down and fixing rebase errors.  Fix a few,
that in turn caused more errors...

You and I both know that the code is likely to be broken at times while
doing many of the code changes needed to fix it.  The original code was
horrible, not all of it, but...   The time for polish is later when the
majority of the changes are done.  I want to move forward, not churn in
circles, rebasing, fixing conflicts, forever waiting for a commit to be
reviewed & merged.  One of the big reasons I started on catalyst was to
remove the hardcoded paths everywhere that was preventing the tree
migration away from /usr/portage.  That has been largely accomplished,
although not really tested.

The reason I wanted to do more testing, is so far I've only done
snapshot, stage1 thru 3.  I've done the major restructure of the python
layout.  I want to get everything working, not 100% bug free, before
moving on.  That will make it easier to figure out where something got
broken while doing the changes.  Call it a midterm exam ;) to see where
we stand.

So I wouldn't mind working in a branch for now, somewhere we can all
participate in pushing commits to in order to achieve our end goal.
Then later when it's mostly done, we can look to moving it to master.
Finish debugging and polishing it.



> Cheers,
> Trevor
> 

Re: [gentoo-catalyst] patch, fix broken seed stage update

[Matt Turner]

On Tue, Feb 26, 2013 at 8:20 AM, Brian Dolbec <dolsen@gentoo.org> wrote:
> --- a/doc/catalyst-spec.5.txt
> +++ b/doc/catalyst-spec.5.txt
> @@ -138,9 +138,19 @@ it should update the seed stage or not (valid values: `yes no`).
>  *update_seed_command*::
>  This is an optional command to pass to emerge for updating the seed
>  stage (example: `--update dev-libs/mpfr dev-libs/mpc dev-libs/gmp`)
> -If not specified, catalyst will update gcc deps.
> +If not specified, catalyst will update gcc's deps, and rebuild gcc if any of
> +it's deps are updated with a new version. Even if it itself is not updated.
> +This prevents gcc breakage when it's dependency lib sonames have changed.
>  This setting requires enabling update_seed.
>
> +*update_seed_system*::
> +This is an optional setting supported by stage1 to tell catalyst if
> +it should update the seed's system packages or not (valid values: `yes no`).
> +This is run after any update_seed_command, updating any remaining upgradable
> +system packages.
> +This setting requires enabling update_seed.

I don't see any need for an update_seed_system option. If someone
wants to run emerge @system then they can do:

update_seed_command: --options @system

Re: [gentoo-catalyst] patch, fix broken seed stage update

[Matt Turner]

On Tue, Feb 26, 2013 at 6:04 PM, Brian Dolbec <dolsen@gentoo.org> wrote:
>  Do you see any code suggestions, questions and/or comments in regards
> to it... NO

Sorry, I've had other things to do today in the 10 intervening hours
since the patch was sent. Have a bit of patience.

Re: [gentoo-catalyst] patch, fix broken seed stage update

[Matt Turner]

On Tue, Feb 26, 2013 at 6:35 PM, Brian Dolbec <dolsen@gentoo.org> wrote:
> On Tue, 2013-02-26 at 20:40 -0500, W. Trevor King wrote:
>> On Tue, Feb 26, 2013 at 05:30:40PM -0800, Brian Dolbec wrote:
>> > My aim for now is to debug the hell out of it, to stabilize all the
>> > existing changes, BEFORE doing a ton more changes.  That will also give
>> > me more experience in how catalyst is used, possibly ideas how to change
>> > it for the better.  I'll fix the doc's generation and make a setup.py
>> > and new 9999 ebuild.
>>
>> I'm not suggesting additional changes on top of yours, I'm suggesting
>> new ones underneath yours, or alternatives to your current series.  It
>> seems a shame to spend time testing a work in progress, since you'll
>> probably want to re-test the final form before merging.  I'm trying to
>> help polish your series down before we invest a lot of time testing.
>>
>
> NNNNNNNNNNNNOOOOOOOOOOOOOOOOOOOOOOOO :(  not more rebasing  I spent
> several days just tracking down and fixing rebase errors.  Fix a few,
> that in turn caused more errors...

Rebasing can be annoying, but isn't inherently difficult. Typically
you don't rebase until you're ready for others to review and commit
them.

I'm curious if you know how to use interactive rebase to apply fix-ups
to commits? If not, I'll be happy to walk you through it. It greatly
simplifies the workflow.

Re: [gentoo-catalyst] patch, fix broken seed stage update

[Brian Dolbec]

On Tue, 2013-02-26 at 18:37 -0800, Matt Turner wrote:
> On Tue, Feb 26, 2013 at 8:20 AM, Brian Dolbec <dolsen@gentoo.org> wrote:
> > --- a/doc/catalyst-spec.5.txt
> > +++ b/doc/catalyst-spec.5.txt
> > @@ -138,9 +138,19 @@ it should update the seed stage or not (valid values: `yes no`).
> >  *update_seed_command*::
> >  This is an optional command to pass to emerge for updating the seed
> >  stage (example: `--update dev-libs/mpfr dev-libs/mpc dev-libs/gmp`)
> > -If not specified, catalyst will update gcc deps.
> > +If not specified, catalyst will update gcc's deps, and rebuild gcc if any of
> > +it's deps are updated with a new version. Even if it itself is not updated.
> > +This prevents gcc breakage when it's dependency lib sonames have changed.
> >  This setting requires enabling update_seed.
> >
> > +*update_seed_system*::
> > +This is an optional setting supported by stage1 to tell catalyst if
> > +it should update the seed's system packages or not (valid values: `yes no`).
> > +This is run after any update_seed_command, updating any remaining upgradable
> > +system packages.
> > +This setting requires enabling update_seed.
> 
> I don't see any need for an update_seed_system option. If someone
> wants to run emerge @system then they can do:
> 
> update_seed_command: --options @system
> 

Well, Jorge last night was saying he wanted -uDN1 @world, but settled
for what was done.

Also if you look at my change for the default update_seed_command.  If
that is just made to @system, the problem I had updating my seed, it
would still be broken.  The problem was that mpc was updated which
changed sonames, so gcc had to be rebuilt or was broken.  Just doing an
emerge -uDN1 @system would not have rebuilt gcc for the new mpc lib.
And most system pkgs can not use EAPI 5 subslots to prevent breakage.

This way gcc will get properly updated, rebuilt if necessary, then if
the user wants, update remaining system pkgs.

It's there, now, only a few lines of code, and for the reasons above,
not as easy to do it all in one command.  We would have to change the
update_seed_command spec option from being a run_merge() to a bash
subshell command.  In that way it can be as complex as the user desires.

Re: [gentoo-catalyst] patch, fix broken seed stage update

[Matt Turner]

On Tue, Feb 26, 2013 at 7:03 PM, Brian Dolbec <dolsen@gentoo.org> wrote:
> On Tue, 2013-02-26 at 18:37 -0800, Matt Turner wrote:
>> On Tue, Feb 26, 2013 at 8:20 AM, Brian Dolbec <dolsen@gentoo.org> wrote:
>> > --- a/doc/catalyst-spec.5.txt
>> > +++ b/doc/catalyst-spec.5.txt
>> > @@ -138,9 +138,19 @@ it should update the seed stage or not (valid values: `yes no`).
>> >  *update_seed_command*::
>> >  This is an optional command to pass to emerge for updating the seed
>> >  stage (example: `--update dev-libs/mpfr dev-libs/mpc dev-libs/gmp`)
>> > -If not specified, catalyst will update gcc deps.
>> > +If not specified, catalyst will update gcc's deps, and rebuild gcc if any of
>> > +it's deps are updated with a new version. Even if it itself is not updated.
>> > +This prevents gcc breakage when it's dependency lib sonames have changed.
>> >  This setting requires enabling update_seed.
>> >
>> > +*update_seed_system*::
>> > +This is an optional setting supported by stage1 to tell catalyst if
>> > +it should update the seed's system packages or not (valid values: `yes no`).
>> > +This is run after any update_seed_command, updating any remaining upgradable
>> > +system packages.
>> > +This setting requires enabling update_seed.
>>
>> I don't see any need for an update_seed_system option. If someone
>> wants to run emerge @system then they can do:
>>
>> update_seed_command: --options @system
>>
>
> Well, Jorge last night was saying he wanted -uDN1 @world, but settled
> for what was done.
>
> Also if you look at my change for the default update_seed_command.  If
> that is just made to @system, the problem I had updating my seed, it
> would still be broken.  The problem was that mpc was updated which
> changed sonames, so gcc had to be rebuilt or was broken.  Just doing an
> emerge -uDN1 @system would not have rebuilt gcc for the new mpc lib.
> And most system pkgs can not use EAPI 5 subslots to prevent breakage.
>
> This way gcc will get properly updated, rebuilt if necessary, then if
> the user wants, update remaining system pkgs.
>
> It's there, now, only a few lines of code, and for the reasons above,
> not as easy to do it all in one command.  We would have to change the
> update_seed_command spec option from being a run_merge() to a bash
> subshell command.  In that way it can be as complex as the user desires.

So from talking on IRC, it seems that the problem that led to this
patch was that a binpkg of gcc linked against an old mpc was used
during stage1, leading to a broken gcc in stage2. Stripping --usepkg
makes sense for this purpose.

Removing --buildpkg I'm not sure about. It seems to me that anything
that is built during the seed-update should be okay to use for the
stage1.

Adding an option for updating @system still seems unnecessary.

Re: [gentoo-catalyst] patch, fix broken seed stage update

[Brian Dolbec]

On Tue, 2013-02-26 at 19:22 -0800, Matt Turner wrote:

> So from talking on IRC, it seems that the problem that led to this
> patch was that a binpkg of gcc linked against an old mpc was used
> during stage1, leading to a broken gcc in stage2. Stripping --usepkg
> makes sense for this purpose.
> 
> Removing --buildpkg I'm not sure about. It seems to me that anything
> that is built during the seed-update should be okay to use for the
> stage1.
> 

Yeah, I thought it should be ok too, but so far opted to leave it out

I thought I'd wait for more feedback.  That's also why I cc'd Zac, since
I didn't know if he was on this list.


> Adding an option for updating @system still seems unnecessary.
> 

 If everyone wants it nuked, is easy todo :)

Re: [gentoo-catalyst] patch, fix broken seed stage update

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1949 bytes --]

On Tue, Feb 26, 2013 at 06:04:54PM -0800, Brian Dolbec wrote:
> I tried using git to send this patch, but it did not work.  I'll look
> into how to make it work with email.
> …
> I will continue to spam the list with all my rebased commits, which I
> intended to do when I had more time.  I sent this one off this morning
> before I went to work.  I know Jorge was interested in the changes.  I'm
> back now, so will manually prepare more patch mails.

It looks like you're pasting diffs into your email client, which makes
it harder for others to apply them with `git am`.  When I'm submitting
patches via email, I usually do something like:

  $ git format-patch --cover-letter -v3 -o my-fix $(git merge-base my-fix master)
  $ emacs my-fix/v3-0000-cover-letter.patch
    … fill in subject and summarize changes since v2 …
  $ git send-email --to gentoo-catalyst@lists.gentoo.org \
      --in-reply-to '<message-id-for-v2@me.invalid>' my-fix/v3-*

Where `my-fix` is the name of my feature branch.  The `-v3` option
requires Git ≥1.8.2 for -v/--reroll-count, before that you had to use
something like `--subject-prefix 'PATCH v3'` [1].

By default, `send-email` will try and send the emails with `sendmail`.
If you don't have a working version of sendmail installed, you can
configure it to use SMTP instead [2,3].

By using the format-patch/send-email pair, you avoid copy/paste errors
and tedium.  You also have author information and commit messages
preserved in a form that `git am` understands.  Wins for everybody ;).

Cheers,
Trevor

[1]: http://kernel.org/pub/software/scm/git/docs/git-format-patch.html
[2]: http://kernel.org/pub/software/scm/git/docs/git-send-email.html
[3]: http://kernel.org/pub/software/scm/git/docs/git-config.html

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[gentoo-catalyst] [PATCH v2] Remove update_seed_command and strengthen update_seed

[W. Trevor King]

From: "W. Trevor King" <wking@tremily.us>

When using `update_seed` to get a 20121213 stage3 from
libmpc.so.2 to libmpc.so.3, stage2 failed with:

  /usr/libexec/gcc/i686-pc-linux-gnu/4.6.3/cc1:
    error while loading shared libraries: libmpc.so.2:
    cannot open shared object file: No such file or directory

This was due to an mpc version bump in the Portage tree that was not
present in the stage3 I used to seed stage1.  Stage1 wasn't
recompiling GCC against the new mpc, so it ended up with GCC linking
against the old mpc.  Heading into stage2, the old mpc (from the seed
stage3) was no longer present, so compilation crashed and burned.

To fix this, we should be extra agressive about rebuilding packages
when their dependencies change in stage1.  The earlier update_seed
command was not catching the bumped mpc, so add:

  --complete-graph --with-bdeps=y --rebuild-if-new-ver

to toughen things up.

This is a general dependency problem (not GCC specific), so I also
replaced the gcc target with:

  @world @system

This leads to a lot of rebuilding, but it should be a stable and
general fix.  With a general fix there's no more need to allow
user-specific overrides with update_seed_command, and that option was
removed.

Based-on-patch-by: Brian Dolbec <dolsen@gentoo.org>
---
I wrote this trimmed-down version up after talking to Brian and Rick
on IRC this morning.  The test run is still running, but I'll post
if/when I get through stage3.

 doc/catalyst-spec.5.txt         | 6 ------
 modules/stage1_target.py        | 2 +-
 targets/stage1/stage1-chroot.sh | 9 +++------
 3 files changed, 4 insertions(+), 13 deletions(-)

diff --git a/doc/catalyst-spec.5.txt b/doc/catalyst-spec.5.txt
index 4a6e06c..a928557 100644
--- a/doc/catalyst-spec.5.txt
+++ b/doc/catalyst-spec.5.txt
@@ -135,12 +135,6 @@ one CD, this defines the layout for the directories under
 This is an optional setting supported by stage1 to tell catalyst if
 it should update the seed stage or not (valid values: `yes no`).
 
-*update_seed_command*::
-This is an optional command to pass to emerge for updating the seed
-stage (example: `--update dev-libs/mpfr dev-libs/mpc dev-libs/gmp`)
-If not specified, catalyst will update gcc deps.
-This setting requires enabling update_seed.
-
 Compilation
 ~~~~~~~~~~~
 
diff --git a/modules/stage1_target.py b/modules/stage1_target.py
index aa43926..7a6c860 100644
--- a/modules/stage1_target.py
+++ b/modules/stage1_target.py
@@ -13,7 +13,7 @@ class stage1_target(generic_stage_target):
 	def __init__(self,spec,addlargs):
 		self.required_values=[]
 		self.valid_values=["chost"]
-		self.valid_values.extend(["update_seed","update_seed_command"])
+		self.valid_values.extend(["update_seed"])
 		generic_stage_target.__init__(self,spec,addlargs)
 
 	def set_stage_path(self):
diff --git a/targets/stage1/stage1-chroot.sh b/targets/stage1/stage1-chroot.sh
index 3f628c2..3dad53d 100644
--- a/targets/stage1/stage1-chroot.sh
+++ b/targets/stage1/stage1-chroot.sh
@@ -26,12 +26,9 @@ clst_root_path=/ setup_pkgmgr
 # Update stage3
 if [ -n "${clst_update_seed}" ]; then
 	if [ "${clst_update_seed}" == "yes" ]; then
-		echo "Updating seed stage..."
-		if [ -n "${clst_update_seed_command}" ]; then
-			clst_root_path=/ run_merge "--buildpkg=n ${clst_update_seed_command}"
-		else
-			clst_root_path=/ run_merge "--buildpkg=n --update --deep --newuse --onlydeps gcc"
-		fi
+		update_cmd="--update --deep --complete-graph --with-bdeps=y --rebuild-if-new-ver @world @system"
+		echo "--- Updating seed stage with: ${update_cmd}"
+		clst_root_path=/ run_merge "${update_cmd}"
 	elif [ "${clst_update_seed}" != "no" ]; then
 		echo "Invalid setting for update_seed: ${clst_update_seed}"
 		exit 1
-- 
1.8.2.rc0.16.g20a599e

Re: [gentoo-catalyst] [PATCH v2] Remove update_seed_command and strengthen update_seed

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 03/08/2013 12:27 PM, W. Trevor King wrote:
> From: "W. Trevor King" <wking@tremily.us>
> 
> When using `update_seed` to get a 20121213 stage3 from
> libmpc.so.2 to libmpc.so.3, stage2 failed with:
> 
>   /usr/libexec/gcc/i686-pc-linux-gnu/4.6.3/cc1:
>     error while loading shared libraries: libmpc.so.2:
>     cannot open shared object file: No such file or directory
> 
> This was due to an mpc version bump in the Portage tree that was not
> present in the stage3 I used to seed stage1.  Stage1 wasn't
> recompiling GCC against the new mpc, so it ended up with GCC linking
> against the old mpc.  Heading into stage2, the old mpc (from the seed
> stage3) was no longer present, so compilation crashed and burned.
> 
> To fix this, we should be extra agressive about rebuilding packages
> when their dependencies change in stage1.  The earlier update_seed
> command was not catching the bumped mpc, so add:
> 
>   --complete-graph --with-bdeps=y --rebuild-if-new-ver
> 
> to toughen things up.
> 
> This is a general dependency problem (not GCC specific), so I also
> replaced the gcc target with:
> 
>   @world @system
> 
> This leads to a lot of rebuilding, but it should be a stable and
> general fix.  With a general fix there's no more need to allow
> user-specific overrides with update_seed_command, and that option was
> removed.
> 
> Based-on-patch-by: Brian Dolbec <dolsen@gentoo.org>
> ---
> I wrote this trimmed-down version up after talking to Brian and Rick
> on IRC this morning.  The test run is still running, but I'll post
> if/when I get through stage3.
> 
>  doc/catalyst-spec.5.txt         | 6 ------
>  modules/stage1_target.py        | 2 +-
>  targets/stage1/stage1-chroot.sh | 9 +++------
>  3 files changed, 4 insertions(+), 13 deletions(-)
> 
> diff --git a/doc/catalyst-spec.5.txt b/doc/catalyst-spec.5.txt
> index 4a6e06c..a928557 100644
> --- a/doc/catalyst-spec.5.txt
> +++ b/doc/catalyst-spec.5.txt
> @@ -135,12 +135,6 @@ one CD, this defines the layout for the directories under
>  This is an optional setting supported by stage1 to tell catalyst if
>  it should update the seed stage or not (valid values: `yes no`).
>  
> -*update_seed_command*::
> -This is an optional command to pass to emerge for updating the seed
> -stage (example: `--update dev-libs/mpfr dev-libs/mpc dev-libs/gmp`)
> -If not specified, catalyst will update gcc deps.
> -This setting requires enabling update_seed.
> -
>  Compilation
>  ~~~~~~~~~~~
>  
> diff --git a/modules/stage1_target.py b/modules/stage1_target.py
> index aa43926..7a6c860 100644
> --- a/modules/stage1_target.py
> +++ b/modules/stage1_target.py
> @@ -13,7 +13,7 @@ class stage1_target(generic_stage_target):
>  	def __init__(self,spec,addlargs):
>  		self.required_values=[]
>  		self.valid_values=["chost"]
> -		self.valid_values.extend(["update_seed","update_seed_command"])
> +		self.valid_values.extend(["update_seed"])
>  		generic_stage_target.__init__(self,spec,addlargs)
>  
>  	def set_stage_path(self):
> diff --git a/targets/stage1/stage1-chroot.sh b/targets/stage1/stage1-chroot.sh
> index 3f628c2..3dad53d 100644
> --- a/targets/stage1/stage1-chroot.sh
> +++ b/targets/stage1/stage1-chroot.sh
> @@ -26,12 +26,9 @@ clst_root_path=/ setup_pkgmgr
>  # Update stage3
>  if [ -n "${clst_update_seed}" ]; then
>  	if [ "${clst_update_seed}" == "yes" ]; then
> -		echo "Updating seed stage..."
> -		if [ -n "${clst_update_seed_command}" ]; then
> -			clst_root_path=/ run_merge "--buildpkg=n ${clst_update_seed_command}"
NACK: we will not be removing update_seed_command. Different people want
different things, gentoo is about choice, if you break it you get to
keep the pieces.

You are more than welcome to add a warning about using
update_seed_command, but it stays.

- -Zero
> -		else
> -			clst_root_path=/ run_merge "--buildpkg=n --update --deep --newuse --onlydeps gcc"
> -		fi
> +		update_cmd="--update --deep --complete-graph --with-bdeps=y --rebuild-if-new-ver @world @system"
> +		echo "--- Updating seed stage with: ${update_cmd}"
> +		clst_root_path=/ run_merge "${update_cmd}"
>  	elif [ "${clst_update_seed}" != "no" ]; then
>  		echo "Invalid setting for update_seed: ${clst_update_seed}"
>  		exit 1
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJROi8ZAAoJEKXdFCfdEflKmDAQALSBVFh6hR4FGR+dQfN3T1Uv
4YLQFk26boO3MdG8/L5J7n9aNh4SGo2NmAzpXPmZnJYFXdbDz5vNtnMb+fOZ2QTU
oSoNRxW6laSY7J5kGXgtIYV4G5m2fYaik87LUuu16bz1GomCsndRuS7fcJZrLMfB
hU3ClOB6BdVeCenvpmxIyg75pZsI2CbSqNXL4MNBocuG/eD6ZfOL17tyvRL1PGdC
X6lv1veu6W6y/UnwuKng3EPqUtUbEh9CNAJZ1vkyyz1FMKtuDoMMrMW2OS/oT5YZ
WELlw/R3XYjyKBM2Ar1bd6jHrTXbUy6buEWY4KFmhx5qY+BRCTeebrz872v4D85C
JAWYJvlinqsyEUnJGtcitgSRx/+A/hweN8s4M+RHHPHDUXQwOedXGnNZU60ZH3o+
Pl9/B/7qBwRCGs0uTXq+GMWFEh4fZP3vMl60xp4GnA9uaxzfN7w0KJ2q1ZxWwGJk
GJYKwTBt5fCA3TDQHHxNaLgAilu7JzMrRIkvaYf6DiXFP9mDUuQOJ8cd/ysUaDa5
QZipSfOw/eQRwzWlirbzHTic8pN5m8GVbYzmdtBVkUGHl8MchytXOwbDEJBW8PJu
4TXtn6KKKFKDDHgPtMHeSzMGQij2n8ivK6QOYT2sPfauNRqHvuW+WgTZi+2EQsK0
UO1kM4BYa2TxuSIkLATT
=lBAs
-----END PGP SIGNATURE-----

[gentoo-catalyst] [PATCH v3] Strengthen update_seed to update @system and @world with dependencies

[W. Trevor King]

From: "W. Trevor King" <wking@tremily.us>

When using `update_seed` to get a 20121213 stage3 from
libmpc.so.2 to libmpc.so.3, stage2 failed with:

  /usr/libexec/gcc/i686-pc-linux-gnu/4.6.3/cc1:
    error while loading shared libraries: libmpc.so.2:
    cannot open shared object file: No such file or directory

This was due to an mpc version bump in the Portage tree that was not
present in the stage3 I used to seed stage1.  Stage1 wasn't
recompiling GCC against the new mpc, so it ended up with GCC linking
against the old mpc.  Heading into stage2, the old mpc (from the seed
stage3) was no longer present, so compilation crashed and burned.

To fix this, we should be extra agressive about rebuilding packages
when their dependencies change in stage1.  The earlier update_seed
command was not catching the bumped mpc, so add:

  --complete-graph --with-bdeps=y --rebuild-if-new-ver

to toughen things up.

This is a general dependency problem (not GCC specific), so I also
replaced the gcc target with:

  @world @system

This leads to a lot of rebuilding, but it should be a stable and
general fix.  With a general fix there should be no more need for
user-specific overrides via update_seed_command, but Gentoo is about
choice, so we keep the option.  Use it at your own risk ;).

Based-on-patch-by: Brian Dolbec <dolsen@gentoo.org>
---
Changes since v2:
* Reinstated update_seed_command, as requested by Zero_Chaos.

 doc/catalyst-spec.5.txt         | 2 +-
 targets/stage1/stage1-chroot.sh | 9 +++------
 2 files changed, 4 insertions(+), 7 deletions(-)

diff --git a/doc/catalyst-spec.5.txt b/doc/catalyst-spec.5.txt
index 4a6e06c..0eb3c2f 100644
--- a/doc/catalyst-spec.5.txt
+++ b/doc/catalyst-spec.5.txt
@@ -138,7 +138,7 @@ it should update the seed stage or not (valid values: `yes no`).
 *update_seed_command*::
 This is an optional command to pass to emerge for updating the seed
 stage (example: `--update dev-libs/mpfr dev-libs/mpc dev-libs/gmp`)
-If not specified, catalyst will update gcc deps.
+If not specified, catalyst will update `@system` and `@world`.
 This setting requires enabling update_seed.
 
 Compilation
diff --git a/targets/stage1/stage1-chroot.sh b/targets/stage1/stage1-chroot.sh
index 3f628c2..0e3859c 100644
--- a/targets/stage1/stage1-chroot.sh
+++ b/targets/stage1/stage1-chroot.sh
@@ -26,12 +26,9 @@ clst_root_path=/ setup_pkgmgr
 # Update stage3
 if [ -n "${clst_update_seed}" ]; then
 	if [ "${clst_update_seed}" == "yes" ]; then
-		echo "Updating seed stage..."
-		if [ -n "${clst_update_seed_command}" ]; then
-			clst_root_path=/ run_merge "--buildpkg=n ${clst_update_seed_command}"
-		else
-			clst_root_path=/ run_merge "--buildpkg=n --update --deep --newuse --onlydeps gcc"
-		fi
+		update_cmd="${clst_update_seed_command:---update --deep --complete-graph --with-bdeps=y --rebuild-if-new-ver @world @system}"
+		echo "--- Updating seed stage with: ${update_cmd}"
+		clst_root_path=/ run_merge "${update_cmd}"
 	elif [ "${clst_update_seed}" != "no" ]; then
 		echo "Invalid setting for update_seed: ${clst_update_seed}"
 		exit 1
-- 
1.8.2.rc0.16.g20a599e

Re: [gentoo-catalyst] [PATCH v3] Strengthen update_seed to update @system and @world with dependencies

[Matt Turner]

On Fri, Mar 8, 2013 at 10:47 AM, W. Trevor King <wking@tremily.us> wrote:
> From: "W. Trevor King" <wking@tremily.us>
>
> When using `update_seed` to get a 20121213 stage3 from
> libmpc.so.2 to libmpc.so.3, stage2 failed with:
>
>   /usr/libexec/gcc/i686-pc-linux-gnu/4.6.3/cc1:
>     error while loading shared libraries: libmpc.so.2:
>     cannot open shared object file: No such file or directory

gcc installed into the stage1 should not be linked against
libmpc.so.2. Something is wrong.

> This was due to an mpc version bump in the Portage tree that was not
> present in the stage3 I used to seed stage1.  Stage1 wasn't
> recompiling GCC against the new mpc, so it ended up with GCC linking
> against the old mpc.

This is really your problem. gcc is built in stage1 and with an
updated mpc built during update_seed. The only way I can think of that
gcc could be linked against libmpc.so.2 in stage1 is if a binpkg of
gcc was used.

> Heading into stage2, the old mpc (from the seed
> stage3) was no longer present, so compilation crashed and burned.
>
> To fix this, we should be extra agressive about rebuilding packages
> when their dependencies change in stage1.

No, we shouldn't, because we don't care about stage1. We simply want
the seed stage to be sufficient to produce a working stage1. By the
time we get to stage3 everything has been rebuilt against the latest
package versions.

> The earlier update_seed
> command was not catching the bumped mpc, so add:

I think this is a false premise. If it is true, we should understand
why and fix it. The current command is emerge --onlydeps gcc which
should be completely sufficient to update mpc.

>
>   --complete-graph --with-bdeps=y --rebuild-if-new-ver
>
> to toughen things up.
>
> This is a general dependency problem (not GCC specific), so I also
> replaced the gcc target with:
>
>   @world @system
>
> This leads to a lot of rebuilding, but it should be a stable and
> general fix.  With a general fix there should be no more need for
> user-specific overrides via update_seed_command, but Gentoo is about
> choice, so we keep the option.  Use it at your own risk ;).
>
> Based-on-patch-by: Brian Dolbec <dolsen@gentoo.org>
> ---
> Changes since v2:
> * Reinstated update_seed_command, as requested by Zero_Chaos.
>
>  doc/catalyst-spec.5.txt         | 2 +-
>  targets/stage1/stage1-chroot.sh | 9 +++------
>  2 files changed, 4 insertions(+), 7 deletions(-)
>
> diff --git a/doc/catalyst-spec.5.txt b/doc/catalyst-spec.5.txt
> index 4a6e06c..0eb3c2f 100644
> --- a/doc/catalyst-spec.5.txt
> +++ b/doc/catalyst-spec.5.txt
> @@ -138,7 +138,7 @@ it should update the seed stage or not (valid values: `yes no`).
>  *update_seed_command*::
>  This is an optional command to pass to emerge for updating the seed
>  stage (example: `--update dev-libs/mpfr dev-libs/mpc dev-libs/gmp`)
> -If not specified, catalyst will update gcc deps.
> +If not specified, catalyst will update `@system` and `@world`.
>  This setting requires enabling update_seed.
>
>  Compilation
> diff --git a/targets/stage1/stage1-chroot.sh b/targets/stage1/stage1-chroot.sh
> index 3f628c2..0e3859c 100644
> --- a/targets/stage1/stage1-chroot.sh
> +++ b/targets/stage1/stage1-chroot.sh
> @@ -26,12 +26,9 @@ clst_root_path=/ setup_pkgmgr
>  # Update stage3
>  if [ -n "${clst_update_seed}" ]; then
>         if [ "${clst_update_seed}" == "yes" ]; then
> -               echo "Updating seed stage..."
> -               if [ -n "${clst_update_seed_command}" ]; then
> -                       clst_root_path=/ run_merge "--buildpkg=n ${clst_update_seed_command}"
> -               else
> -                       clst_root_path=/ run_merge "--buildpkg=n --update --deep --newuse --onlydeps gcc"
> -               fi
> +               update_cmd="${clst_update_seed_command:---update --deep --complete-graph --with-bdeps=y --rebuild-if-new-ver @world @system}"
> +               echo "--- Updating seed stage with: ${update_cmd}"
> +               clst_root_path=/ run_merge "${update_cmd}"
>         elif [ "${clst_update_seed}" != "no" ]; then
>                 echo "Invalid setting for update_seed: ${clst_update_seed}"
>                 exit 1
> --
> 1.8.2.rc0.16.g20a599e

[gentoo-catalyst] Re: [PATCH v3] Strengthen update_seed to update @system and @world with dependencies

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 830 bytes --]

On Fri, Mar 08, 2013 at 12:14:08PM -0800, Matt Turner wrote:
> On Fri, Mar 8, 2013 at 10:47 AM, W. Trevor King <wking@tremily.us> wrote:
> > The earlier update_seed
> > command was not catching the bumped mpc, so add:
> 
> I think this is a false premise. If it is true, we should understand
> why and fix it. The current command is emerge --onlydeps gcc which
> should be completely sufficient to update mpc.

Using catalyst as of d1c2ba9b (create-iso.sh: add usb boot via
isohybrid, 2013-03-05), I can no longer reproduce my original problem.
Next time I'll do a better job preserving my logs, but I'll withdraw
this patch.

Sorry for the noise :p
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[gentoo-catalyst] chmod +x all sh scripts so they can run from the git checkout

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 832 bytes --]

On Thu, Jan 31, 2013 at 02:46:53PM -0500, W. Trevor King wrote:
> 6108400 chmod +x all sh scripts so they can run from the git checkout
> 
> Looks good,

I've been using c988ea3 (a more recent rebased version of this commit
in dolsen/rewrite) for a month or so now, and it's working well.  I'd
also add targets/stage1/build.py to the list of chmodded scripts to
avoid:

  ./tmp/stage1-chroot.sh: line 6: /tmp/build.py: Permission denied
  Your profile seems to be broken.
  Could not build a list of build packages.

The following command should do what's needed:

  $ find targets \( -name '*.sh' -o -name '*.py' \) -exec chmod 755 {} \+

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[gentoo-catalyst] Binary package dependencies and update_seed

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 2622 bytes --]

On Sat, Mar 09, 2013 at 07:10:23AM -0500, W. Trevor King wrote:
> On Fri, Mar 08, 2013 at 12:14:08PM -0800, Matt Turner wrote:
> > On Fri, Mar 8, 2013 at 10:47 AM, W. Trevor King <wking@tremily.us> wrote:
> > > The earlier update_seed
> > > command was not catching the bumped mpc, so add:
> > 
> > I think this is a false premise. If it is true, we should understand
> > why and fix it. The current command is emerge --onlydeps gcc which
> > should be completely sufficient to update mpc.
> 
> Using catalyst as of d1c2ba9b (create-iso.sh: add usb boot via
> isohybrid, 2013-03-05), I can no longer reproduce my original problem.

Ben Kohler (iamben) was working through this problem on #gentoo-releng
recently.  His problem (and probably my original problem as well)
seems to be leftover binary packages from before update_seed was
enabled.  These binary packages matched the version number and USE
flags of the to-be-installed package, but were built against the older
version of mpc.  In order to avoid problems like this, it's probably a
good idea to remove all the cached stuff with something like [1]:

  $ rm -rf --one-file-system /var/tmp/catalyst/{kerncache,packages,tmp}

before building your stage1.

A better approach would involve recording runtime dependency versions
in the binary package metadata, so that Portage could determine
whether a given binary package would work in the target system [2].
Currently, the ebuild RDEPEND is included in Packages, but that does
not always (or even often) specify which package versions were used to
build the package.  For example, sys-devel/gcc lists:

  RDEPEND: sys-libs/zlib virtual/libiconv >=dev-libs/gmp-4.3.2
    >=dev-libs/mpfr-2.4.2 >=dev-libs/mpc-0.8.1

I'd like it to list:

  RDEPEND: =sys-libs/zlib-1.2.7 =virtual/libiconv-0
    =dev-libs/gmp-5.0.2_p1 =dev-libs/mpfr-3.1.1 =dev-libs/mpc-1.0.1

or whatever the versions of those packages were at build time.
Portage could then recurse through this information and only use the
package if versions matched the target system across the whole RDEPEND
tree.  With ABI sub-slots, strict version matching could be loosened
to only require ABI matching.

Cheers,
Trevor

[1]: kerncache probably doesn't matter, but we might as well be
     thorough.
[2]: This is a limitation with the current Portage code, and not a
     catalyst issue.  Maybe I should cross-post this to
     gentoo-portage-dev@?

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] Binary package dependencies and update_seed

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/11/2013 01:09 PM, W. Trevor King wrote:
> On Sat, Mar 09, 2013 at 07:10:23AM -0500, W. Trevor King wrote:
>> On Fri, Mar 08, 2013 at 12:14:08PM -0800, Matt Turner wrote:
>>> On Fri, Mar 8, 2013 at 10:47 AM, W. Trevor King <wking@tremily.us> wrote:
>>>> The earlier update_seed
>>>> command was not catching the bumped mpc, so add:
>>>
>>> I think this is a false premise. If it is true, we should understand
>>> why and fix it. The current command is emerge --onlydeps gcc which
>>> should be completely sufficient to update mpc.
>>
>> Using catalyst as of d1c2ba9b (create-iso.sh: add usb boot via
>> isohybrid, 2013-03-05), I can no longer reproduce my original problem.
> 
> Ben Kohler (iamben) was working through this problem on #gentoo-releng
> recently.  His problem (and probably my original problem as well)
> seems to be leftover binary packages from before update_seed was
> enabled.  These binary packages matched the version number and USE
> flags of the to-be-installed package, but were built against the older
> version of mpc.  In order to avoid problems like this, it's probably a
> good idea to remove all the cached stuff with something like [1]:
> 
>   $ rm -rf --one-file-system /var/tmp/catalyst/{kerncache,packages,tmp}
> 
> before building your stage1.
> 
> A better approach would involve recording runtime dependency versions
> in the binary package metadata, so that Portage could determine
> whether a given binary package would work in the target system [2].
> Currently, the ebuild RDEPEND is included in Packages, but that does
> not always (or even often) specify which package versions were used to
> build the package.  For example, sys-devel/gcc lists:
> 
>   RDEPEND: sys-libs/zlib virtual/libiconv >=dev-libs/gmp-4.3.2
>     >=dev-libs/mpfr-2.4.2 >=dev-libs/mpc-0.8.1
> 
> I'd like it to list:
> 
>   RDEPEND: =sys-libs/zlib-1.2.7 =virtual/libiconv-0
>     =dev-libs/gmp-5.0.2_p1 =dev-libs/mpfr-3.1.1 =dev-libs/mpc-1.0.1

This is called EAPI 5 subslots.  it should not be addressed in catalyst
nor portage, things should simply set subslot deps properly.

- -ZC
> 
> or whatever the versions of those packages were at build time.
> Portage could then recurse through this information and only use the
> package if versions matched the target system across the whole RDEPEND
> tree.  With ABI sub-slots, strict version matching could be loosened
> to only require ABI matching.
> 
> Cheers,
> Trevor
> 
> [1]: kerncache probably doesn't matter, but we might as well be
>      thorough.
> [2]: This is a limitation with the current Portage code, and not a
>      catalyst issue.  Maybe I should cross-post this to
>      gentoo-portage-dev@?
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJRZvVTAAoJEKXdFCfdEflKXzgQAJnhcZX19BnAbfaKitkf0hap
CiewJPTMBqYDtSSviCZkskf9+JQALyegmiIDn7JwEzM9QEz2p9MpCym0KFJetqrW
logbRCNDS8+goqLznaDA1sTHYmvNJnlnI85AIt7GVpqAEq2EVZFStD5s3oyL/aUG
BB8cb1dwDpLfC7BC9InGqO7jpyu0rR4vUWv0K4BOEEtmTXyBrN1JMNbHxG9l8YAm
Vwn666GHM02aXviGay0RXtSqeZhrA3TWhMTOyoxsfZKwriwg9CYRdy3QtUZWMHup
L3FuKMKlobbY69E/GJdGN0YrVMStc6BEe36wPNU+ijmGk40rVnQ/sQhEx1xEz3y1
Vy3P2OryUcCqt9mCleCiseLFY0mZZ68RCuM/AgO8hH4xjqo9qkusWJk4wvuMDGTH
kVo+G8WzNTG22TOkjSp3N1D0Su3AvMK0swOObn4Y3uB/WcfL3gJEs52qDy2Mfdnc
M1s6Up+rIz8Wn1/jDKwCn6T3O4V+Hn+ZWAkWaD13QEqyFVJlJ0eRAE2QEdG6Ju2u
DqQgi+N67yoKCEwsnxkRpExu1mIZX8DwjcJ503VIltIG1pP+SbTPe6AmEjuWwChz
xrqdL1udpxPLnDjArDvwFyFN2BOlP8V/bTSjAIBZCemUk/avDd1hMXf33bQtCtGp
GXI+81lvDVu/VEx5IRFY
=wQVE
-----END PGP SIGNATURE-----

Re: [gentoo-catalyst] Binary package dependencies and update_seed

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1347 bytes --]

On Thu, Apr 11, 2013 at 01:39:31PM -0400, Rick "Zero_Chaos" Farina wrote:
> > For example, sys-devel/gcc lists:
> > 
> >   RDEPEND: sys-libs/zlib virtual/libiconv >=dev-libs/gmp-4.3.2
> >     >=dev-libs/mpfr-2.4.2 >=dev-libs/mpc-0.8.1
> > 
> > I'd like it to list:
> > 
> >   RDEPEND: =sys-libs/zlib-1.2.7 =virtual/libiconv-0
> >     =dev-libs/gmp-5.0.2_p1 =dev-libs/mpfr-3.1.1 =dev-libs/mpc-1.0.1
> 
> This is called EAPI 5 subslots.  it should not be addressed in catalyst
> nor portage, things should simply set subslot deps properly.

I had understood sub-slotting to be a way to list the ABI provided by
a given package (in SLOT) or required by a given package (in *DEPEND),
not a way for a package to record which ABIs it was built against.
For example, GCC builds against >=dev-libs/mpc-0.8.1, but there are
some incompatible ABIs in that range.  We need to know which ABI the
built package actually uses, not which ABIs it could have used.

In any case, we probably still need a warning somewhere though, since
I don't foresee GCC moving to EAPI 5 in the near future ;).  I'm
working up a patch with comre comments in files/catalyst.conf.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] Binary package dependencies and update_seed

[Matt Turner]

On Thu, Apr 11, 2013 at 10:39 AM, Rick "Zero_Chaos" Farina
<zerochaos@gentoo.org> wrote:
> This is called EAPI 5 subslots.  it should not be addressed in catalyst
> nor portage, things should simply set subslot deps properly.

gcc, mpc, gmp, et al are EAPI=0.

Re: [gentoo-catalyst] Binary package dependencies and update_seed

[Matt Turner]

On Thu, Apr 11, 2013 at 10:09 AM, W. Trevor King <wking@tremily.us> wrote:
> In order to avoid problems like this, it's probably a
> good idea to remove all the cached stuff with something like [1]:
>
>   $ rm -rf --one-file-system /var/tmp/catalyst/{kerncache,packages,tmp}
>
> before building your stage1.

Not sure this is the way to go. I'll have to think through the
implications. Consider your stage1 build dies with some stupid tree
breakage that you can fix in your portage snapshot and restart the
stage build. This is going to cause your stage to rebuild everything.

Wouldn't disabling the use of binary packages during the seed stage
update fix this?

Re: [gentoo-catalyst] Binary package dependencies and update_seed

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/11/2013 02:22 PM, Matt Turner wrote:
> On Thu, Apr 11, 2013 at 10:09 AM, W. Trevor King <wking@tremily.us> wrote:
>> In order to avoid problems like this, it's probably a
>> good idea to remove all the cached stuff with something like [1]:
>>
>>   $ rm -rf --one-file-system /var/tmp/catalyst/{kerncache,packages,tmp}
>>
>> before building your stage1.
> 
> Not sure this is the way to go. I'll have to think through the
> implications. Consider your stage1 build dies with some stupid tree
> breakage that you can fix in your portage snapshot and restart the
> stage build. This is going to cause your stage to rebuild everything.
> 
> Wouldn't disabling the use of binary packages during the seed stage
> update fix this?
> 
> 
I'm not sure I'd call rebuilding it every time a "fix" when you just
rejected deleting the bad packages and rebuilding ONCE.

As I said, the only proper fix is to update the toolchain to EAPI5.  We
could live in the dark ages forever, but I really don't think it helps
anyone.

- -ZC
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJRZwajAAoJEKXdFCfdEflKqOcQAKo3MnuT1dZrEudaH2gC1g+f
++iAxgaA3TkwKJ7srhcI1R6Xvoyb4Iv/BNCYcwYQn1Ta0yS682qwFouP17YBDdIN
vqAQViCK39YPwg3pY77IgwMa2KQSrnLIPM6jV05174QR3Og8otzvW3ur9ou8GQgz
KTcRNhl7cPqajV8voq7u4WD05brLpvH2l5rP3UZliavsas3fopioORSdlaRTqicD
MpxK2iXKx+qScwgEcoWelRIMG25U1f0KGcLDDIslfJ+YTZ3/9pRYePXpr8kZlp1O
oegdNTCqH5wD80QoMSocAVahvCOjW/yyBYJFIfzQpzubPsodtVGr0k7hVaPZNZF7
kznW+E0j1HuW/K3hlRfLlhiYbcalXRCsQmzT9Qbenkq1HuR6KsHK3KWe0iipJvgO
lU/TPty6TeXPczK9K7wqo1e5nuE39Z7AUvVIjaIJr1Y9QSWyMVfEQq3pO5x9Ity2
tIBGzYh11r/n3W9Sj5wfqN8GDGugNJmhJS1ytMKuOSRmDEkvOzaaubZBVja92FCQ
DtcY1xLn/edax3ReYpRjLQSxuneKA8lsDCt3WYOedzF5KW2WbgV5XRXh8lS4jG3L
UaayekUNQJImgbJu4hqqsbhnBEEHKHjxSdSaqD1xydE5KPCVSq00dsW2TKyv0sUT
Y5XgK1g1Umxp42brFXG7
=jZeA
-----END PGP SIGNATURE-----

Re: [gentoo-catalyst] Binary package dependencies and update_seed

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1678 bytes --]

On Thu, Apr 11, 2013 at 11:22:54AM -0700, Matt Turner wrote:
> On Thu, Apr 11, 2013 at 10:09 AM, W. Trevor King <wking@tremily.us> wrote:
> > In order to avoid problems like this, it's probably a
> > good idea to remove all the cached stuff with something like [1]:
> >
> >   $ rm -rf --one-file-system /var/tmp/catalyst/{kerncache,packages,tmp}
> >
> > before building your stage1.
> 
> Not sure this is the way to go. I'll have to think through the
> implications. Consider your stage1 build dies with some stupid tree
> breakage that you can fix in your portage snapshot and restart the
> stage build. This is going to cause your stage to rebuild everything.

There shouldn't be a need to clear the caches unless you've changed
the specs or sources.  Presumably you had to change something to fix
whatever killed stage1.  If that was an ebuild problem, it's unlikely
that your fix changed an ABI without also changing the package version
and you should be ok keeping the old binary packages.

> Wouldn't disabling the use of binary packages during the seed stage
> update fix this?

Disabling creation of binary packages during the seed stage update
would help, and disabling the use of binary packages during the stage1
build would help.  However, what if you have built a binary package
for GCC and then you update your tree to something that builds
libmpc.so.3.  In this case, you're in trouble if you use your old GCC
binary packages anywhere based on the new tree.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] Binary package dependencies and update_seed

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1073 bytes --]

On Thu, Apr 11, 2013 at 02:53:23PM -0400, Rick "Zero_Chaos" Farina wrote:
> > Wouldn't disabling the use of binary packages during the seed stage
> > update fix this?
> > 
> > 
> I'm not sure I'd call rebuilding it every time a "fix" when you just
> rejected deleting the bad packages and rebuilding ONCE.
> 
> As I said, the only proper fix is to update the toolchain to EAPI5.  We
> could live in the dark ages forever, but I really don't think it helps
> anyone.

I'd be quite happy to have to toolchain move to EAPI5.  I suspect we
need an interim solution until that happens ;).

I also don't think EAPI5 sub-slots fix this problem, since GCC
wouldn't RDEPEND on a particular sub-slot of mpc, even if both
packages used EAPI5.  RDEPEND describes potential source
compatability, but we need to record the actual ABI linkage (which is
a subset of the source compatability).

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] Binary package dependencies and update_seed

[Matt Turner]

On Thu, Apr 11, 2013 at 11:53 AM, Rick "Zero_Chaos" Farina
<zerochaos@gentoo.org> wrote:
>> Not sure this is the way to go. I'll have to think through the
>> implications. Consider your stage1 build dies with some stupid tree
>> breakage that you can fix in your portage snapshot and restart the
>> stage build. This is going to cause your stage to rebuild everything.
>>
>> Wouldn't disabling the use of binary packages during the seed stage
>> update fix this?
>>
>>
> I'm not sure I'd call rebuilding it every time a "fix" when you just
> rejected deleting the bad packages and rebuilding ONCE.

I'm not sure what you mean.

Updating the seed stage should only ever happen once. If that fails
and you don't have binary packages to fall back to on restarting, I
think that's an okay price to pay. Especially since it happens early
enough in the process and avoids potential problems that are only
noticed at the beginning of stage2.

Oh, and if updating the seed stage fails... it was only rebuilding gcc
dependencies anyway. That's not so bad at all. (I don't care about
custom update commands and isn't something that we can fix)

> As I said, the only proper fix is to update the toolchain to EAPI5.  We
> could live in the dark ages forever, but I really don't think it helps
> anyone.

That seems like a potentially good long term goal, but isn't something
that we can force through.

Re: [gentoo-catalyst] Binary package dependencies and update_seed

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/11/2013 03:03 PM, Matt Turner wrote:
> On Thu, Apr 11, 2013 at 11:53 AM, Rick "Zero_Chaos" Farina
> <zerochaos@gentoo.org> wrote:
>>> Not sure this is the way to go. I'll have to think through the
>>> implications. Consider your stage1 build dies with some stupid tree
>>> breakage that you can fix in your portage snapshot and restart the
>>> stage build. This is going to cause your stage to rebuild everything.
>>>
>>> Wouldn't disabling the use of binary packages during the seed stage
>>> update fix this?
>>>
>>>
>> I'm not sure I'd call rebuilding it every time a "fix" when you just
>> rejected deleting the bad packages and rebuilding ONCE.
> 
> I'm not sure what you mean.
> 
> Updating the seed stage should only ever happen once. If that fails
> and you don't have binary packages to fall back to on restarting, I
> think that's an okay price to pay. Especially since it happens early
> enough in the process and avoids potential problems that are only
> noticed at the beginning of stage2.
> 
> Oh, and if updating the seed stage fails... it was only rebuilding gcc
> dependencies anyway. That's not so bad at all. (I don't care about
> custom update commands and isn't something that we can fix)
> 
>> As I said, the only proper fix is to update the toolchain to EAPI5.  We
>> could live in the dark ages forever, but I really don't think it helps
>> anyone.
> 
> That seems like a potentially good long term goal, but isn't something
> that we can force through.
> 
> 
We already don't BUILD any new packages during update_seed.  Originally
we did, I encountered issues and fixed it in
e7ea409acb52b43e9ea141c57201f9f87673f7ba to prevent building of packages
during update_seed.

Thanks,
Zero
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJRZwxpAAoJEKXdFCfdEflK3qsQAKuklF3DkQ2vJPoYMy2KWo0H
4+YRfH/zgkhnemqgzgbr3eIuY/A1nccIZ89buwua4Gni2gnXdya6gPDkgMTkGvzY
jlHRdSg5+MFgju5OA/peYCoAAz3TU/CP1U8OVJhPZ35R1v3nRVcBlI60jwXCIvqP
LQEur9b2Mdx+p7er3QKypjJNtPNRYxZBoEicKXsIfIcvQRWnsrpnS9xxktGlrukE
+eYTWnAB8gYmjQp2n6ZGT3aTuQoDgomBUTIeTmsAzz+U+MafSItnWBI+4w9s4auV
93ac65u/7Yrey7hLf9V1KP9T1xEGavWgeSy1e2w7a/QAsXdmEpCi1+ai9Bu8j+JU
RL0PkdcWyfEPdgSLAxvhj8LzEovnGn6c74Vu6larOQOOOQWPal+pq6JwJR1zxoVm
WBQsyjPTsBuPoWn4L8vLKwqiHNBPw5jzdpTlADvAm0ET2pQ73fV/gYCHIzwtwPYo
CxHQasxgS/qE/ZD/aoOWzroQ5z6r/8BtOFNhmM5cj2H1rFi/nsKQyZDlR5nA7A1Z
akjQb/aQI+RAERa6acGttmldRkP3dWnjycRJpoVfpg1XJFdwJKSyzfmuU7slFL88
I+2B0e+3Aluog45rGiLcXDHZASdZ/RDRqvIrovLw+hoy5Koo6KRGitjLNWEP6F+f
E7OoClVdEKD5MCVrZL65
=Iqpi
-----END PGP SIGNATURE-----

Re: [gentoo-catalyst] Binary package dependencies and update_seed

[Matt Turner]

On Thu, Apr 11, 2013 at 12:18 PM, Rick "Zero_Chaos" Farina
<zerochaos@gentoo.org> wrote:
> We already don't BUILD any new packages during update_seed.  Originally
> we did, I encountered issues and fixed it in
> e7ea409acb52b43e9ea141c57201f9f87673f7ba to prevent building of packages
> during update_seed.

Right, I see.

So, to make sure that I'm on the same page: is the the problem that
we're using stale packages in stage1 and if so, where did they come
from? A previous stage1 build that didn't do update-seed?

Re: [gentoo-catalyst] Binary package dependencies and update_seed

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1620 bytes --]

On Thu, Apr 11, 2013 at 01:24:19PM -0700, Matt Turner wrote:
> On Thu, Apr 11, 2013 at 12:18 PM, Rick "Zero_Chaos" Farina
> <zerochaos@gentoo.org> wrote:
> > We already don't BUILD any new packages during update_seed.  Originally
> > we did, I encountered issues and fixed it in
> > e7ea409acb52b43e9ea141c57201f9f87673f7ba to prevent building of packages
> > during update_seed.
> 
> Right, I see.
> 
> So, to make sure that I'm on the same page: is the the problem that
> we're using stale packages in stage1 and if so, where did they come
> from? A previous stage1 build that didn't do update-seed?

That's where mine (and presumably iamben's) came from.  However, they
could also (I think) come from a stage1 build that used an older
snapshot.  I'm testing now with a build from:

  subarch: i686
  version_stamp: 2013.1
  target: stage1
  rel_type: default
  profile: default/linux/x86/13.0/desktop
  portage_confdir: /var/tmp/catalyst/portage-conf/default/
  snapshot: 20130208
  source_subpath: default/stage3-i686-20121213
  update_seed: yes

Followed by another build with the same version_stamp but using:

  snapshot: 20130308

The idea is that the first build might produce packages linking
libmpc.so.2, and the second build might reuse those packages, despite
the stabilization of mpc-1.0.1 in the tree.  I'm currently 31/75 on
the first build, so it will be a bit longer before results are in.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] Binary package dependencies and update_seed

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/11/2013 04:24 PM, Matt Turner wrote:
> On Thu, Apr 11, 2013 at 12:18 PM, Rick "Zero_Chaos" Farina
> <zerochaos@gentoo.org> wrote:
>> We already don't BUILD any new packages during update_seed.  Originally
>> we did, I encountered issues and fixed it in
>> e7ea409acb52b43e9ea141c57201f9f87673f7ba to prevent building of packages
>> during update_seed.
> 
> Right, I see.
> 
> So, to make sure that I'm on the same page: is the the problem that
> we're using stale packages in stage1 and if so, where did they come
> from? A previous stage1 build that didn't do update-seed?
> 
> 
- From discussions in irc it sounds like he wasn't using update_seed at
all originally.  That means he encountered the bug which update_seed was
meant to prevent (and built a bad binpkg at that time).  Later he
learned about update_seed, enabled it, and it didn't fix anything
because the bad package was already built.

Perhaps we should just force update_seed to on by default?

- -ZC
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJRZx7zAAoJEKXdFCfdEflK51gP/2yIxWpbgGDvlX5dle07jSGy
g/kGZ+LudmLlRNcAExURpTxMMd5Jgm4WK8qU42BON2rgC+QmuD9opIPWefpYCiwN
BzOvKejhB6cPUO5m5HABoSNUq0fqXtefMfCl9hEbzZfPaOKPz2vpVgzxWGz8uz2v
IGukmkJ91SMUef673UHnk/EyIO6BQKeuTCzcKeZW8ym1r9Ut6w3PvKVqRRSVqSTr
e0VHL5UKLN/4Z9/LBktbmTaNQeDLNuksIan0KORRAUGepBWmBVuUzmgekfDZoKqD
7n6Ig7RYLv8p+8i+XL7SI8PcYdxAP5WqmPnmXaDFhuhMmN8kjapomf/0rUoWeZiP
DRausrPc7XIN+sDSWulkWhwB1buhLnwnZU2NTldvO1fi/P42oeCkpNGGiQnPtmBu
x9x7fhmc0+M35VXBhzWPARxyl/JrAkB+xb9KoAC7uO6iLFb7m55Y6/SPpXuxlK8K
0mF+VfQXSRBIPnJWlzHZS83DKKQ6cHQT9oRLFdBPZJt/C8MMMY6hZPslytj7DX+F
6vNSv8wFBmMOzOvD98FIJ86TtFzbxrZMcFOYAY7I6DqeeJEK8hAT5pTs71l8W9+Y
3vMdvnTwlpIK5gmk3OubERQzCvve03gwRJn03XzSvBdIm41/fg+6gn5/pzLLeG/C
ENqNf6fO0UUP0tphTNb1
=bD7X
-----END PGP SIGNATURE-----

Re: [gentoo-catalyst] Binary package dependencies and update_seed

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 5221 bytes --]

On Thu, Apr 11, 2013 at 04:34:12PM -0400, W. Trevor King wrote:
> On Thu, Apr 11, 2013 at 01:24:19PM -0700, Matt Turner wrote:
> > So, to make sure that I'm on the same page: is the the problem that
> > we're using stale packages in stage1 and if so, where did they come
> > from? A previous stage1 build that didn't do update-seed?
> 
> That's where mine (and presumably iamben's) came from.  However, they
> could also (I think) come from a stage1 build that used an older
> snapshot.  I'm testing now with a build from:
> 
>   subarch: i686
>   version_stamp: 2013.1
>   target: stage1
>   rel_type: default
>   profile: default/linux/x86/13.0/desktop
>   portage_confdir: /var/tmp/catalyst/portage-conf/default/
>   snapshot: 20130208
>   source_subpath: default/stage3-i686-20121213
>   update_seed: yes
> 
> Followed by another build with the same version_stamp but using:
> 
>   snapshot: 20130308
> 
> The idea is that the first build might produce packages linking
> libmpc.so.2, and the second build might reuse those packages, despite
> the stabilization of mpc-1.0.1 in the tree.

So I finished the build and I have a broken stage1 after the tree
update (as I expected).  Here's how it goes:

Clear the caches:

  # rm -rf --one-file-system /tmp/*.log /var/tmp/catalyst/{kerncache,packages,tmp}

Confirm the first spec (using the older tree)

  # cat /var/tmp/catalyst/spec/default-stage1-i686-test.1.spec
  subarch: i686
  version_stamp: 2013.1
  target: stage1
  rel_type: default
  profile: default/linux/x86/13.0/desktop
  portage_confdir: /var/tmp/catalyst/portage-conf/default/
  snapshot: 20130208
  source_subpath: default/stage3-i686-20121213
  update_seed: yes

Build it:

  # python2.7 catalyst -c catalyst.conf -f /var/tmp/catalyst/spec/default-stage1-i686-test.1.spec 2>&1 | tee /tmp/stage1-i686-test.1.log
  …
  >>> Emerging (30 of 38) sys-devel/gcc-4.6.3 for /tmp/stage1root/
  >>> Installing (30 of 38) sys-devel/gcc-4.6.3 to /tmp/stage1root/
  …

As expected, with a tree where mpc-0.8.2 is the latest stable version,
we get dependencies on libmpc.so.2:

  # less /var/tmp/catalyst/tmp/default/stage1-i686-2013.1/tmp/stage1root/usr/libexec/gcc/i686-pc-linux-gnu/4.6.3/cc1 | grep libmpc
   0x00000001 (NEEDED)                     Shared library: [libmpc.so.2]
  # ls -l /var/tmp/catalyst/tmp/default/stage1-i686-2013.1/tmp/stage1root/usr/libexec/gcc/i686-pc-linux-gnu/4.6.3/cc1 
  -rwxr-xr-x 1 root root 10557140 Apr 11 21:24 /var/tmp/catalyst/tmp/default/stage1-i686-2013.1/tmp/stage1root/usr/libexec/gcc/i686-pc-linux-gnu/4.6.3/cc1
  # less /var/tmp/catalyst/packages/default/stage1-i686-2013.1/sys-devel/gcc-4.6.3.tbz2 | grep '/cc1$'
  -rwxr-xr-x root/root  10557140 2013-04-11 21:24 ./usr/libexec/gcc/i686-pc-linux-gnu/4.6.3/cc1

Let's shift that stage out of the way for safe keeping and build
another with a new snapshot:

  # mv /var/tmp/catalyst/tmp/default/stage1-i686-{2013.1,test.1}/
  # diff /var/tmp/catalyst/spec/default-stage1-i686-test.{1,2}.spec
  7c7
  < snapshot: 20130208
  ---
  > snapshot: 20130308
  # python2.7 catalyst -c catalyst.conf -f /var/tmp/catalyst/spec/default-stage1-i686-test.2.spec 2>&1 | tee /tmp/stage1-i686-test.2.log
  …
  >>> Emerging binary (10 of 75) sys-devel/gcc-config-1.7.3 for /tmp/stage1root/
  >>> Installing (10 of 75) sys-devel/gcc-config-1.7.3 to /tmp/stage1root/
  …

Looks like were using the out-of-date binary package.  Let's check to
see we're still linking against the old libmpc:

  # less /var/tmp/catalyst/tmp/default/stage1-i686-2013.1/tmp/stage1root/usr/libexec/gcc/i686-pc-linux-gnu/4.6.3/cc1 | grep libmpc
   0x00000001 (NEEDED)                     Shared library: [libmpc.so.2]
  # ls -l /var/tmp/catalyst/tmp/default/stage1-i686-2013.1/tmp/stage1root/usr/libexec/gcc/i686-pc-linux-gnu/4.6.3/cc1 
  -rwxr-xr-x 1 root root 10557140 Apr 11 21:24 /var/tmp/catalyst/tmp/default/stage1-i686-2013.1/tmp/stage1root/usr/libexec/gcc/i686-pc-linux-gnu/4.6.3/cc1

Yup.  And that mpc version isn't in the new stage1:

  # ls /var/tmp/catalyst/tmp/default/stage1-i686-2013.1/tmp/stage1root/usr/lib/libmpc.so*
  /var/tmp/catalyst/tmp/default/stage1-i686-2013.1/tmp/stage1root/usr/lib/libmpc.so
  /var/tmp/catalyst/tmp/default/stage1-i686-2013.1/tmp/stage1root/usr/lib/libmpc.so.3
  /var/tmp/catalyst/tmp/default/stage1-i686-2013.1/tmp/stage1root/usr/lib/libmpc.so.3.0.0

So, while iamben (I think) and I (definitely) ran into this problem
because we had just turned on update_seed (with out-of-date packeages
from a non-update_seed run), this problem can also crop up due to
snapshot updates (even if you always have update_seed on).  That means
that the only reasonable fixes are:

a. Clear the package cache whenever you update a spec (without bumping
   version_stamp).
b. Migrate the toolchain packages to EAPI5 and use ABI sub-slotting.

Since (b), even if it does happen, is likely to take a while, I think
we should be pushing (a) for now.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] Binary package dependencies and update_seed

[Brian Dolbec]

On Thu, 2013-04-11 at 14:53 -0400, W. Trevor King wrote:
> On Thu, Apr 11, 2013 at 11:22:54AM -0700, Matt Turner wrote:
> > On Thu, Apr 11, 2013 at 10:09 AM, W. Trevor King <wking@tremily.us> wrote:
> > > In order to avoid problems like this, it's probably a
> > > good idea to remove all the cached stuff with something like [1]:
> > >
> > >   $ rm -rf --one-file-system /var/tmp/catalyst/{kerncache,packages,tmp}
> > >
> > > before building your stage1.
> > 
> > Not sure this is the way to go. I'll have to think through the
> > implications. Consider your stage1 build dies with some stupid tree
> > breakage that you can fix in your portage snapshot and restart the
> > stage build. This is going to cause your stage to rebuild everything.
> 
> There shouldn't be a need to clear the caches unless you've changed
> the specs or sources.  Presumably you had to change something to fix
> whatever killed stage1.  If that was an ebuild problem, it's unlikely
> that your fix changed an ABI without also changing the package version
> and you should be ok keeping the old binary packages.
> 
> > Wouldn't disabling the use of binary packages during the seed stage
> > update fix this?
> 
> Disabling creation of binary packages during the seed stage update
> would help, and disabling the use of binary packages during the stage1
> build would help.  However, what if you have built a binary package
> for GCC and then you update your tree to something that builds
> libmpc.so.3.  In this case, you're in trouble if you use your old GCC
> binary packages anywhere based on the new tree.
> 
> Cheers,
> Trevor
> 

OK, When I first discovered the update_seed problem, I discovered
several (potential) bugs in the code that could cause failures.  My main
error was due to using binary pkgs which were built against libmpc.so.2.

I talked with Zac (and he agreed) portage _SHOULD_ be checking
NEEDED.ELF.2 which lists the actual libs it is linked to.  There is an
open bug for it in bugzie (sorry, I'm too tired to look for it).  If a
package manager supports installing binpkgs, then it should ensure it is
not installing a broken pkg due to mismatched lib links.

Everything else that is done in catalyst to avoid that is really just a
workaround until the package manager is fixed.

Blueness recently wrote a utility that scanned NEEDED.ELF.2 and looked
for broken/mismatched lib deps.  Perhaps he could help in getting
portage's binpkg handlers updated to check that the binpkg's lib deps
are met for it to qualify to be used.

[gentoo-catalyst] [PATCH] files/catalyst.conf: Document linking issues with binary packages

[W. Trevor King]

From: "W. Trevor King" <wking@tremily.us>

This gives users a heads up explaining why they might see linking
errors when pkgcache is enabled.  I first saw this when I build a
stage1 without update_seed.  Because my seed stage3 linked against
libmpc.so.2, some of my stage1 files linked against the older mpc.
However, the mpc-1.0.1 built for the stage1 installed libmpc.so.3.
When I tried to use this stage1 to build a stage2, it died with:

  /usr/libexec/gcc/i686-pc-linux-gnu/4.6.3/cc1:
    error while loading shared libraries: libmpc.so.2:
    cannot open shared object file: No such file or directory

To fix this, I enabled update_seed, but binary packages built during
my first pass were used to populate the stage1, so even though I'd
updated the seed stage3 toolchain, I still had a stage1 with cc1
linked against libmpc.so.2.

After clearing the binary package cache, I got a stage1 *built* with
the updated seed stage3, which gave a cc1 linked against libmpc.so.3
(hurray!).

This commit adds a warning in the pkgcache documentation that should
help people understand what might be going wrong if they see similar
linking errors.  For more details, see the thread following
http://thread.gmane.org/gmane.linux.gentoo.catalyst/2137/focus=2193
---
 files/catalyst.conf | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/files/catalyst.conf b/files/catalyst.conf
index b74c546..e285e4f 100644
--- a/files/catalyst.conf
+++ b/files/catalyst.conf
@@ -58,7 +58,11 @@ hash_function="crc32"
 # kerncache = keeps a tbz2 of your built kernel and modules (useful if your
 #	build stops in livecd-stage2)
 # pkgcache = keeps a tbz2 of every built package (useful if your build stops
-#	prematurely)
+#	prematurely.  However, you may see linking problems if the binary
+#	package was built against an older version of a runtime dependency
+#	and the package in question is not using EAPI5's sub-slots.  For
+#	further discussion, see
+#	http://article.gmane.org/gmane.linux.gentoo.catalyst/2224)
 # seedcache = use the build output of a previous target if it exists to speed up
 #	the copy
 # snapcache = cache the snapshot so that it can be bind-mounted into the chroot.
-- 
1.8.2

Re: [gentoo-catalyst] [PATCH] files/catalyst.conf: Document linking issues with binary packages

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/12/2013 11:12 AM, W. Trevor King wrote:
> From: "W. Trevor King" <wking@tremily.us>
> 
> This gives users a heads up explaining why they might see linking
> errors when pkgcache is enabled.  I first saw this when I build a
> stage1 without update_seed.  Because my seed stage3 linked against
> libmpc.so.2, some of my stage1 files linked against the older mpc.
> However, the mpc-1.0.1 built for the stage1 installed libmpc.so.3.
> When I tried to use this stage1 to build a stage2, it died with:
> 
>   /usr/libexec/gcc/i686-pc-linux-gnu/4.6.3/cc1:
>     error while loading shared libraries: libmpc.so.2:
>     cannot open shared object file: No such file or directory
> 
> To fix this, I enabled update_seed, but binary packages built during
> my first pass were used to populate the stage1, so even though I'd
> updated the seed stage3 toolchain, I still had a stage1 with cc1
> linked against libmpc.so.2.
> 
> After clearing the binary package cache, I got a stage1 *built* with
> the updated seed stage3, which gave a cc1 linked against libmpc.so.3
> (hurray!).
> 
> This commit adds a warning in the pkgcache documentation that should
> help people understand what might be going wrong if they see similar
> linking errors.  For more details, see the thread following
> http://thread.gmane.org/gmane.linux.gentoo.catalyst/2137/focus=2193

NACK.  Personally I love this idea but I'm really against having a link
to a ML discussion, no one is going to quit configuring catalyst in the
middle to read a ML discussion...

- -ZC
> ---
>  files/catalyst.conf | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/files/catalyst.conf b/files/catalyst.conf
> index b74c546..e285e4f 100644
> --- a/files/catalyst.conf
> +++ b/files/catalyst.conf
> @@ -58,7 +58,11 @@ hash_function="crc32"
>  # kerncache = keeps a tbz2 of your built kernel and modules (useful if your
>  #	build stops in livecd-stage2)
>  # pkgcache = keeps a tbz2 of every built package (useful if your build stops
> -#	prematurely)
> +#	prematurely.  However, you may see linking problems if the binary
> +#	package was built against an older version of a runtime dependency
> +#	and the package in question is not using EAPI5's sub-slots.  For
> +#	further discussion, see
> +#	http://article.gmane.org/gmane.linux.gentoo.catalyst/2224)
>  # seedcache = use the build output of a previous target if it exists to speed up
>  #	the copy
>  # snapcache = cache the snapshot so that it can be bind-mounted into the chroot.
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJRaCZdAAoJEKXdFCfdEflKpe4QAKsr4Ct1zch9TXFe4E9575HP
YnyScCSgKO8ed9/SfvvNu42GlFAdVbA/awvRqEvSPSrdzTQB0tb0aw5O4daciT98
tLoO6y5YHyOk7bsoEtvmAJVZgHcApGgs9MfjUNkTObS6VzYInBYccFxPOjfefy6C
GoK/hfR1RaUlCPgopaTS8JSWBQyTNjzX7O/CrW4b5RGMMXanX1+X3SVI/75Qg8Ux
8HxcuK5IPRyAUFAw4HIyIPMNAzv/v4FRfbGMBsjYApeE0HP+kMb953VfRJ0jSuNH
DvukXwuwHcR704FUjD2TsN7XG/LdLZB/1jQSv/h2w8XkINQpW5M6J6TJjiGx1CyS
o1CefuyPL9PRzRv7vpN7HsmxPnBkezoLu7xoafzezdrgI6FBM1jXIC2Up4R8GTR7
Hgzf7kiC+HTXB/YQIscSze/5M1Rl+fk+O6ycfn5w9Sy5t3XEB417u55pDRPUiUFi
wtvrbchf7mTfvo95iWeXt3vGbUaA1EbdW4Z95A5HPS8QvlXmnoeN2DHhO58pisMa
WCTTq3JyQxrNwoTHcukaSGsHzKbzOYXkM1X+/UfxCdQIACiHstkJkRSidYnKsIEp
2ZfleRxg4WdrTiqnOofknOoHofkssLHx19s0QYTgIBVL5mirAJhB8Oyz5KPXif9F
QGAqhASyVtzxuIIXKJjE
=Uv3R
-----END PGP SIGNATURE-----

Re: [gentoo-catalyst] [PATCH] files/catalyst.conf: Document linking issues with binary packages

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 2062 bytes --]

On Fri, Apr 12, 2013 at 11:21:01AM -0400, Rick "Zero_Chaos" Farina wrote:
> On 04/12/2013 11:12 AM, W. Trevor King wrote:
> > This commit adds a warning in the pkgcache documentation that should
> > help people understand what might be going wrong if they see similar
> > linking errors.  For more details, see the thread following
> > http://thread.gmane.org/gmane.linux.gentoo.catalyst/2137/focus=2193
> 
> NACK.  Personally I love this idea but I'm really against having a link
> to a ML discussion, no one is going to quit configuring catalyst in the
> middle to read a ML discussion...

Well, that's the commit message, but…

> > diff --git a/files/catalyst.conf b/files/catalyst.conf
> > index b74c546..e285e4f 100644
> > --- a/files/catalyst.conf
> > +++ b/files/catalyst.conf
> > @@ -58,7 +58,11 @@ hash_function="crc32"
> >  # kerncache = keeps a tbz2 of your built kernel and modules (useful if your
> >  #	build stops in livecd-stage2)
> >  # pkgcache = keeps a tbz2 of every built package (useful if your build stops
> > -#	prematurely)
> > +#	prematurely.  However, you may see linking problems if the binary
> > +#	package was built against an older version of a runtime dependency
> > +#	and the package in question is not using EAPI5's sub-slots.  For
> > +#	further discussion, see
> > +#	http://article.gmane.org/gmane.linux.gentoo.catalyst/2224)
> >  # seedcache = use the build output of a previous target if it exists to speed up
> >  #	the copy
> >  # snapcache = cache the snapshot so that it can be bind-mounted into the chroot.

Here I link to the ML from the config file ;).  I'm fine linking
somewhere else (doc/catalyst-conf.1.txt?).  Then config file comments
could move to the new man page (as spec file comments moved to their
own man page).  With more space, we can wax eloquent on the potential
pkgcache issues ;).

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] [PATCH] files/catalyst.conf: Document linking issues with binary packages

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/12/2013 11:33 AM, W. Trevor King wrote:
> On Fri, Apr 12, 2013 at 11:21:01AM -0400, Rick "Zero_Chaos" Farina wrote:
>> On 04/12/2013 11:12 AM, W. Trevor King wrote:
>>> This commit adds a warning in the pkgcache documentation that should
>>> help people understand what might be going wrong if they see similar
>>> linking errors.  For more details, see the thread following
>>> http://thread.gmane.org/gmane.linux.gentoo.catalyst/2137/focus=2193
>>
>> NACK.  Personally I love this idea but I'm really against having a link
>> to a ML discussion, no one is going to quit configuring catalyst in the
>> middle to read a ML discussion...
> 
> Well, that's the commit message, but…
> 
Yeah sorry, a bit distracted, I meant to protest on the below section
>>> diff --git a/files/catalyst.conf b/files/catalyst.conf
>>> index b74c546..e285e4f 100644
>>> --- a/files/catalyst.conf
>>> +++ b/files/catalyst.conf
>>> @@ -58,7 +58,11 @@ hash_function="crc32"
>>>  # kerncache = keeps a tbz2 of your built kernel and modules (useful if your
>>>  #	build stops in livecd-stage2)
>>>  # pkgcache = keeps a tbz2 of every built package (useful if your build stops
>>> -#	prematurely)
>>> +#	prematurely.  However, you may see linking problems if the binary
>>> +#	package was built against an older version of a runtime dependency
>>> +#	and the package in question is not using EAPI5's sub-slots.  For
>>> +#	further discussion, see
>>> +#	http://article.gmane.org/gmane.linux.gentoo.catalyst/2224)
>>>  # seedcache = use the build output of a previous target if it exists to speed up
>>>  #	the copy
>>>  # snapcache = cache the snapshot so that it can be bind-mounted into the chroot.
> 
> Here I link to the ML from the config file ;).  I'm fine linking
> somewhere else (doc/catalyst-conf.1.txt?).  Then config file comments
> could move to the new man page (as spec file comments moved to their
> own man page).  With more space, we can wax eloquent on the potential
> pkgcache issues ;).

Yeah, but more reading is not necessarily more better. It is likely
better if we explain it briefly and not worry about 100% of the details.
the config file isn't meant for long discussions, it's meant for short
recommendations and warnings about features at most.

- -ZC
> 
> Cheers,
> Trevor
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJRaDIXAAoJEKXdFCfdEflKLrYQAJcGiOnYWEQcRvFMiG+B+494
YmyaPyMIS0qaFgN8n5q6wh3AtEM/eXyleff7Bnqulktgb89iNFwCtaXlKsVDygOD
dSRES11nccc+Y96Mve3OujAj0kp7XoVytGRCmsQJsofrtSOWjaebD4UyxQduaWJN
Q01WpgfdTrTMXmfozHs+FrAm0Vo+C/t3mJr/TrPGxXdaHp0ee8tCuh7PAmGvw6hy
lRkmrl6FpsXpytwsuL+ekIKMiCfZ5KIkdzLN3qVil2eZ6YRokxbD7WZsPElLQKud
lTdei62NuIP1+cdTfzqxyv9hF+480OIF40fZh4L9XHlW1aS1BfHGiq6WgNis0nz6
cBZE0/JR+JJr6p2BWZ7imAIldB2vyCgvQuaKEPS59HhIHjwDqfLZtR6B/SmXWiPE
gVLgFipuvNjmBe1swDpAUHpeLYh7DlksYVeoYJbG4V6WLT55Aa7Oa0gGWJUYIzfY
WX9wiifb//b0J0O83C6dLWcL9yyDCPMlL+MW/kjQ4Js2f+aaSfVL62mFp/XFUC2g
p5NrOD5AcJDvExQ0peOZjED+50S2AN/7tlmBJ2OpkKtIAxFzyanwvp167GcjGGXM
wtONk58744rUlq4Na9I81ybxchjaxATwlxx2CIRcutFoqXx7C9boWaddUgYiB0qf
OGmFQH/fuKhOgZI3k1P5
=bjco
-----END PGP SIGNATURE-----

[gentoo-catalyst] [PATCH v2 0/2] pkgcache warning in catalyst-config(5)

[W. Trevor King]

From: "W. Trevor King" <wking@tremily.us>

On Fri, Apr 12, 2013 at 12:11:03PM -0400, Rick "Zero_Chaos" Farina wrote:
> On 04/12/2013 11:33 AM, W. Trevor King wrote:
> > Here I link to the ML from the config file ;).  I'm fine linking
> > somewhere else (doc/catalyst-conf.1.txt?).  Then config file comments
> > could move to the new man page (as spec file comments moved to their
> > own man page).  With more space, we can wax eloquent on the potential
> > pkgcache issues ;).
>
> Yeah, but more reading is not necessarily more better. It is likely
> better if we explain it briefly and not worry about 100% of the details.
> the config file isn't meant for long discussions, it's meant for short
> recommendations and warnings about features at most.

Here's a new man page which breaks the binary package discussion out
into its own section.  This gives me a bit more space to explain
what's going on.  I think that files/catalyst.conf should be stripped
back to only contain things that we thing are really important, since
the detailed information will be in the man page.  What options do
people actually tweak on a regular basis?  What other options do we
want to highlight?

W. Trevor King (2):
  doc/catalyst-config.5.txt: Add man page for catalyst.conf
  doc/catalyst-config.5.txt: Document linking issues with binary
    packages

 doc/catalyst-config.5.txt | 230 ++++++++++++++++++++++++++++++++++++++++++++++
 doc/catalyst-spec.5.txt   |   2 +-
 doc/catalyst.1.txt        |   1 +
 files/.gitignore          |   1 +
 4 files changed, 233 insertions(+), 1 deletion(-)
 create mode 100644 doc/catalyst-config.5.txt

-- 
1.8.2

[gentoo-catalyst] [PATCH v2 1/2] doc/catalyst-config.5.txt: Add man page for catalyst.conf

[W. Trevor King]

From: "W. Trevor King" <wking@tremily.us>

This mostly translates the inline comments from files/catalyst.conf
into asciidoc.  While it's nice to have that as stand-alone
documentation, it also makes it easier to refer to longer descriptions
of any tricky issues.  This gives us a place to distill the collected
wisdom of current users for the benefit of others.
---
 doc/catalyst-config.5.txt | 188 ++++++++++++++++++++++++++++++++++++++++++++++
 doc/catalyst-spec.5.txt   |   2 +-
 doc/catalyst.1.txt        |   1 +
 files/.gitignore          |   1 +
 4 files changed, 191 insertions(+), 1 deletion(-)
 create mode 100644 doc/catalyst-config.5.txt

diff --git a/doc/catalyst-config.5.txt b/doc/catalyst-config.5.txt
new file mode 100644
index 0000000..944f392
--- /dev/null
+++ b/doc/catalyst-config.5.txt
@@ -0,0 +1,188 @@
+CATALYST-CONFIG(5)
+================
+:man source:   catalyst {catalystversion}
+:man manual:   catalyst {catalystversion}
+
+
+NAME
+----
+catalyst-config - Catalyst configuration files
+
+
+SYNOPSIS
+--------
+*catalyst* ['OPTIONS'] *-c* 'FILE'
+
+
+DESCRIPTION
+-----------
+
+*catalyst(1)* reads the configuration file given with `-c` or
+`--config` on the command line.  The file contains keyword-argument
+pairs using a POSIX Shell variable syntax.  Lines starting with `#`
+and empty lines are interpreted as comments.  For example:
+
+---------------------------------
+# /etc/catalyst/catalyst.conf
+digests="md5 sha1 sha512 whirlpool"
+contents="auto"
+distdir="/usr/portage/distfiles"
+envscript="/etc/catalyst/catalystrc"
+options="autoresume bindist kerncache pkgcache seedcache snapcache"
+portdir="/usr/portage"
+sharedir="/usr/lib/catalyst"
+snapshot_cache="/var/tmp/catalyst/snapshot_cache"
+storedir="/var/tmp/catalyst"
+---------------------------------
+
+The possible keywords and their meanings are as follows:
+
+Basic configuration
+~~~~~~~~~~~~~~~~~~~
+
+*digests*::
+Create a `.DIGESTS` file containing the hash output from any of the
+supported options below.  Adding them all may take a long time.
+(example: `md5 sha1 sha512 whirlpool`).  See the *SUPPORTED HASHES*
+section for a list of supported hashes.
+
+*contents*::
+Create a `.CONTENTS` file listing the contents of the file.  If this
+variable is empty, no `.CONTENTS` will be generated at all.  Supported
+values:
++
+--
+auto::
+Strongly recommended
+
+tar-tv::
+Do `tar tvf FILE`
+
+tar-tvz::
+Do `tar tvzf FILE`
+
+tar-tvy::
+Do `tar tvyf FILE`
+
+isoinfo-l::
+Do `isoinfo -l -i FILE`
+
+isoinfo-f::
+Do `isoinfo -f -i FILE`.  'isoinfo-f' is the only option not chosen
+by the automatic algorithm.
+--
+
+*distdir*::
+Distfiles location.  `/usr/portage/distfiles` should work for most
+default installations.
+
+*envscript*::
+Environment script location, which allows users to set options such as
+HTTP proxies, `MAKEOPTS`, `GENTOO_MIRRORS`, or any other environment
+variables needed for building.  The envscript file sets environment
+variables using POSIX shell notation:
++
+---------------------------------
+export FOO="bar"
+---------------------------------
+
+*hash_function*::
+Internal hash function catalyst should use for things like autoresume,
+seedcache, etc.  The default and fastest is `crc32`.  You should not
+ever need to change this unless your OS does not support it.  See the
+*SUPPORTED HASHES* section for a list of supported hashes.
+
+**options*::
+Set different build-time options (example: `autoresume bindist
+kerncache pkgcache seedcache snapcache`).  Supported values:
++
+--
+autoresume::
+Attempt to resume a failed build.  Clear the autoresume flags with the
+`-a` option to the catalyst command line.  `-p` will clear the
+autoresume flags as well as your pkgcache and kerncache.
+
+bindist::
+Enable the `bindist` `USE` flag.  This is recommended if you will be
+redistributing builds, but see the package specific definitions for
+details.
+
+ccache::
+Enable build time ccache support.
+
+distcc::
+Enable distcc support for building. You have to set distcc_hosts in
+your spec file.
+
+icecream::
+Enable icecream compiler cluster support for building.
+
+kerncache::
+Keep a `tbz2` of your built kernel and modules.  This is useful if
+your build dies during `livecd-stage2`.
+
+pkgcache::
+Keep a ``tbz2`` of every built package.  This is useful if your build
+dies prematurely.
+
+seedcache::
+Use the build output of a previous target if it exists to speed up the
+creation of a new stage.  This avoids unpacking the seed tarball.
+
+snapcache::
+Cache the snapshot so that it can be bind-mounted into the chroot.
+WARNING: moving parts of the portage tree from within fsscript *will*
+break your cache.  The cache is unlinked before any empty or rm
+processing, though.
+--
+
+*portdir*::
+Portage tree location.  `/usr/portage/` should work for most default
+installations.
+
+*sharedir*::
+Catalyst runtime script location.  `/usr/lib/catalyst` should work for
+most default installations.  If you are running catalyst from a Git
+checkout, you should change this to point to your checkout directory.
+
+*storedir*::
+Location for built seeds, temporary files, and caches.
+
+*port_logdir*::
+Location for build logs (example: `/var/tmp/catalyst/tmp`).  This dir
+will be automatically cleaned of all logs over 30 days old. If left
+undefined the logs will remain in the build directory as usual and get
+cleaned every time a stage build is restarted.
+
+*var_tmpfs_portage*::
+Set the size of a `/var/tmp/portage` tmpfs in gigabytes (example:
+`16`).  If set, this mounts a tmpfs for `/var/tmp/portage` so building
+takes place in RAM.  This feature requires a pretty large tmpfs
+({open,libre}office needs ~8GB to build).  WARNING: If you use too
+much RAM everything will fail horribly and it is not our fault.
+
+
+SUPPORTED HASHES
+----------------
+Supported hashes: adler32, crc32, crc32b, gost, haval128, haval160,
+haval192, haval224, haval256, md2, md4, md5, ripemd128, ripemd160,
+ripemd256, ripemd320, sha1, sha224, sha256, sha384, sha512, snefru128,
+snefru256, tiger, tiger128, tiger160, whirlpool.
+
+
+FILES
+-----
+An example configuration file can be found at
+`/etc/catalyst/catalyst.conf`.
+
+
+BUGS
+----
+An up-to-date list of Catalyst bugs can always be found listed on the Gentoo
+Linux bug-tracking system at `http://bugs.gentoo.org`.
+
+
+SEE ALSO
+--------
+*catalyst(1)*
+*catalyst-config(5)*
diff --git a/doc/catalyst-spec.5.txt b/doc/catalyst-spec.5.txt
index 4a6e06c..b3e2bf5 100644
--- a/doc/catalyst-spec.5.txt
+++ b/doc/catalyst-spec.5.txt
@@ -496,7 +496,6 @@ specific options for the game.  This is not used on the release media.
 FILES
 -----
 Example specfiles can be found in '/usr/share/doc/catalyst-{catalystversion}/examples'.
-An example configuration file can be found at '/etc/catalyst/catalyst.conf'.
 
 
 SUPPORTED ARCHITECTURES
@@ -516,3 +515,4 @@ Linux bug-tracking system at 'http://bugs.gentoo.org'.
 SEE ALSO
 --------
 *catalyst(1)*
+*catalyst-config(5)*
diff --git a/doc/catalyst.1.txt b/doc/catalyst.1.txt
index 4d51eb0..09dc119 100644
--- a/doc/catalyst.1.txt
+++ b/doc/catalyst.1.txt
@@ -136,6 +136,7 @@ AUTHORS
 SEE ALSO
 --------
 *catalyst-spec(5)*
+*catalyst-config(5)*
 
 Also, a more in-depth examination of Catalyst options and procedures can be found
 linked from the *catalyst* project page, which is located at
diff --git a/files/.gitignore b/files/.gitignore
index 9dbf1f1..7773aeb 100644
--- a/files/.gitignore
+++ b/files/.gitignore
@@ -1,4 +1,5 @@
 catalyst.1
+catalyst-config.5
 catalyst-spec.5
 *.html
 *.css
-- 
1.8.2

[gentoo-catalyst] [PATCH v2 2/2] doc/catalyst-config.5.txt: Document linking issues with binary packages

[W. Trevor King]

From: "W. Trevor King" <wking@tremily.us>

This gives users a heads up explaining why they might see linking
errors when pkgcache is enabled.  I first saw this when I build a
stage1 without update_seed.  Because my seed stage3 linked against
libmpc.so.2, some of my stage1 files linked against the older mpc.
However, the mpc-1.0.1 built for the stage1 installed libmpc.so.3.
When I tried to use this stage1 to build a stage2, it died with:

  /usr/libexec/gcc/i686-pc-linux-gnu/4.6.3/cc1:
    error while loading shared libraries: libmpc.so.2:
    cannot open shared object file: No such file or directory

To fix this, I enabled update_seed, but binary packages built during
my first pass were used to populate the stage1, so even though I'd
updated the seed stage3 toolchain, I still had a stage1 with cc1
linked against libmpc.so.2.

After clearing the binary package cache, I got a stage1 *built* with
the updated seed stage3, which gave a cc1 linked against libmpc.so.3
(hurray!).

This commit adds a warning in the pkgcache documentation that should
help people understand what might be going wrong if they see similar
linking errors.  For more details, see the thread following
http://thread.gmane.org/gmane.linux.gentoo.catalyst/2137/focus=2193
---
 doc/catalyst-config.5.txt | 44 +++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 43 insertions(+), 1 deletion(-)

diff --git a/doc/catalyst-config.5.txt b/doc/catalyst-config.5.txt
index 944f392..61c9aba 100644
--- a/doc/catalyst-config.5.txt
+++ b/doc/catalyst-config.5.txt
@@ -123,7 +123,8 @@ your build dies during `livecd-stage2`.
 
 pkgcache::
 Keep a ``tbz2`` of every built package.  This is useful if your build
-dies prematurely.
+dies prematurely.  However, you may experience linking problems.  See
+the *BINARY PACKAGE DEPENDENCIES* section for details.
 
 seedcache::
 Use the build output of a previous target if it exists to speed up the
@@ -170,6 +171,47 @@ ripemd256, ripemd320, sha1, sha224, sha256, sha384, sha512, snefru128,
 snefru256, tiger, tiger128, tiger160, whirlpool.
 
 
+BINARY PACKAGE DEPENDENCIES
+---------------------------
+This section is only important if you are using binary packages to
+build your stages (by enabling the `pkgcache` option and restarting
+incomplete builds).
+
+Before EAPI-5 introduced ABI sub-slots, the build-time compatibility
+of packages was not recorded.  This leads to problems such as binary
+GCC packages built against mpc-0.8.2 (which installs libmpc.so.2)
+being installed on systems that only have mpc-1.0.1 (which installs
+libmpc.so.3), resulting in:
+
+---------------------------------
+/usr/libexec/gcc/i686-pc-linux-gnu/4.6.3/cc1:
+  error while loading shared libraries: libmpc.so.2:
+  cannot open shared object file: No such file or directory
+---------------------------------
+
+As long as there are packages in your stage that don't use ABI
+sub-slots, you may experience errors like this due to untracked ABI
+missmatches in binary packages.  Packages generated by catalyst builds
+are currently namespaced:
+
+---------------------------------
+.../packages/<rel_type>/<target>-<subarch>-<version_stamp>/Packages
+---------------------------------
+
+so running into these out-of-date packages is unlikely.  You may run
+into problems if:
+
+* you enable `update_seed` in your stage1 spec after a previous run
+  which generated packages linking against out-of-date seed libraries
+  or
+* you update your snapshot and an untracked ABI dependency is bumped
+  without a similar bump in the dependent package.
+
+without also bumping any of the package namespace variables in your
+spec.  If you do make such a change, it's a good idea to clear the
+package cache in question and rebuild the packages from scratch.
+
+
 FILES
 -----
 An example configuration file can be found at
-- 
1.8.2

[gentoo-catalyst] Re: [PATCH v2 1/2] doc/catalyst-config.5.txt: Add man page for catalyst.conf

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 552 bytes --]

On Fri, Apr 12, 2013 at 02:21:42PM -0400, W. Trevor King wrote:
> diff --git a/doc/catalyst-config.5.txt b/doc/catalyst-config.5.txt
> new file mode 100644
> index 0000000..944f392
> --- /dev/null
> +++ b/doc/catalyst-config.5.txt
> @@ -0,0 +1,188 @@
> …
> +SEE ALSO
> +--------
> +*catalyst(1)*
> +*catalyst-config(5)*

Oops.  s/catalyst-config/catalyst-spec/.  Will fix in v3.

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] [PATCH v2 1/2] doc/catalyst-config.5.txt: Add man page for catalyst.conf

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

The worms... the worms they are everywhere.... why did you open the can?

Scroll down

On 04/12/2013 02:21 PM, W. Trevor King wrote:
> From: "W. Trevor King" <wking@tremily.us>
> 
> This mostly translates the inline comments from files/catalyst.conf
> into asciidoc.  While it's nice to have that as stand-alone
> documentation, it also makes it easier to refer to longer descriptions
> of any tricky issues.  This gives us a place to distill the collected
> wisdom of current users for the benefit of others.
> ---
>  doc/catalyst-config.5.txt | 188 ++++++++++++++++++++++++++++++++++++++++++++++
>  doc/catalyst-spec.5.txt   |   2 +-
>  doc/catalyst.1.txt        |   1 +
>  files/.gitignore          |   1 +
>  4 files changed, 191 insertions(+), 1 deletion(-)
>  create mode 100644 doc/catalyst-config.5.txt
> 
> diff --git a/doc/catalyst-config.5.txt b/doc/catalyst-config.5.txt
> new file mode 100644
> index 0000000..944f392
> --- /dev/null
> +++ b/doc/catalyst-config.5.txt
> @@ -0,0 +1,188 @@
> +CATALYST-CONFIG(5)
> +================
> +:man source:   catalyst {catalystversion}
> +:man manual:   catalyst {catalystversion}
> +
> +
> +NAME
> +----
> +catalyst-config - Catalyst configuration files
> +
> +
> +SYNOPSIS
> +--------
> +*catalyst* ['OPTIONS'] *-c* 'FILE'
> +
> +
> +DESCRIPTION
> +-----------
> +
> +*catalyst(1)* reads the configuration file given with `-c` or
> +`--config` on the command line.  The file contains keyword-argument
> +pairs using a POSIX Shell variable syntax.  Lines starting with `#`
> +and empty lines are interpreted as comments.  For example:
> +
> +---------------------------------
> +# /etc/catalyst/catalyst.conf
> +digests="md5 sha1 sha512 whirlpool"
> +contents="auto"
> +distdir="/usr/portage/distfiles"
> +envscript="/etc/catalyst/catalystrc"
> +options="autoresume bindist kerncache pkgcache seedcache snapcache"
> +portdir="/usr/portage"
> +sharedir="/usr/lib/catalyst"
> +snapshot_cache="/var/tmp/catalyst/snapshot_cache"
> +storedir="/var/tmp/catalyst"
> +---------------------------------
> +
> +The possible keywords and their meanings are as follows:
> +
> +Basic configuration
> +~~~~~~~~~~~~~~~~~~~
> +
> +*digests*::
> +Create a `.DIGESTS` file containing the hash output from any of the
> +supported options below.  Adding them all may take a long time.
> +(example: `md5 sha1 sha512 whirlpool`).  See the *SUPPORTED HASHES*
> +section for a list of supported hashes.
> +
> +*contents*::
> +Create a `.CONTENTS` file listing the contents of the file.  If this
> +variable is empty, no `.CONTENTS` will be generated at all.  Supported
> +values:
> ++
> +--
> +auto::
> +Strongly recommended
> +
> +tar-tv::
> +Do `tar tvf FILE`
> +
> +tar-tvz::
> +Do `tar tvzf FILE`
> +
> +tar-tvy::
> +Do `tar tvyf FILE`
> +
> +isoinfo-l::
> +Do `isoinfo -l -i FILE`
> +
> +isoinfo-f::
> +Do `isoinfo -f -i FILE`.  'isoinfo-f' is the only option not chosen
> +by the automatic algorithm.
> +--
> +
> +*distdir*::
> +Distfiles location.  `/usr/portage/distfiles` should work for most
> +default installations.
> +
> +*envscript*::
> +Environment script location, which allows users to set options such as
> +HTTP proxies, `MAKEOPTS`, `GENTOO_MIRRORS`, or any other environment
> +variables needed for building.  The envscript file sets environment
> +variables using POSIX shell notation:
> ++
> +---------------------------------
> +export FOO="bar"
> +---------------------------------
> +
> +*hash_function*::
> +Internal hash function catalyst should use for things like autoresume,
> +seedcache, etc.  The default and fastest is `crc32`.  You should not
> +ever need to change this unless your OS does not support it.  See the
> +*SUPPORTED HASHES* section for a list of supported hashes.
> +
> +**options*::
> +Set different build-time options (example: `autoresume bindist
> +kerncache pkgcache seedcache snapcache`).  Supported values:
> ++
> +--
> +autoresume::
> +Attempt to resume a failed build.  Clear the autoresume flags with the
> +`-a` option to the catalyst command line.  `-p` will clear the
> +autoresume flags as well as your pkgcache and kerncache.
> +
> +bindist::
> +Enable the `bindist` `USE` flag.  This is recommended if you will be
> +redistributing builds, but see the package specific definitions for
> +details.
> +
> +ccache::
> +Enable build time ccache support.
Warning, ccache has been known to cause random build failures and bugs
reported with ccache enabled may be closed invalid.
> +
> +distcc::
> +Enable distcc support for building. You have to set distcc_hosts in
> +your spec file.
> +
> +icecream::
> +Enable icecream compiler cluster support for building.
> +
> +kerncache::
> +Keep a `tbz2` of your built kernel and modules.  This is useful if
> +your build dies during `livecd-stage2`.
kerncache keeps a full dir of everything like seedcache does.
> +
> +pkgcache::
> +Keep a ``tbz2`` of every built package.  This is useful if your build
> +dies prematurely.
Might be easier to just say this enables the binpkg feature of portage...


- -ZC
> +
> +seedcache::
> +Use the build output of a previous target if it exists to speed up the
> +creation of a new stage.  This avoids unpacking the seed tarball.
> +
> +snapcache::
> +Cache the snapshot so that it can be bind-mounted into the chroot.
> +WARNING: moving parts of the portage tree from within fsscript *will*
> +break your cache.  The cache is unlinked before any empty or rm
> +processing, though.
- -, though
> +--
> +
> +*portdir*::
> +Portage tree location.  `/usr/portage/` should work for most default
> +installations.
> +
> +*sharedir*::
> +Catalyst runtime script location.  `/usr/lib/catalyst` should work for
> +most default installations.  If you are running catalyst from a Git
> +checkout, you should change this to point to your checkout directory.
> +
> +*storedir*::
> +Location for built seeds, temporary files, and caches.
mention the default location
> +
> +*port_logdir*::
> +Location for build logs (example: `/var/tmp/catalyst/tmp`).  This dir
> +will be automatically cleaned of all logs over 30 days old. If left
> +undefined the logs will remain in the build directory as usual and get
> +cleaned every time a stage build is restarted.
> +
> +*var_tmpfs_portage*::
> +Set the size of a `/var/tmp/portage` tmpfs in gigabytes (example:
> +`16`).  If set, this mounts a tmpfs for `/var/tmp/portage` so building
> +takes place in RAM.  This feature requires a pretty large tmpfs
> +({open,libre}office needs ~8GB to build).  WARNING: If you use too
> +much RAM everything will fail horribly and it is not our fault.
> +
> +
> +SUPPORTED HASHES
> +----------------
> +Supported hashes: adler32, crc32, crc32b, gost, haval128, haval160,
> +haval192, haval224, haval256, md2, md4, md5, ripemd128, ripemd160,
> +ripemd256, ripemd320, sha1, sha224, sha256, sha384, sha512, snefru128,
> +snefru256, tiger, tiger128, tiger160, whirlpool.
> +
> +
> +FILES
> +-----
> +An example configuration file can be found at
> +`/etc/catalyst/catalyst.conf`.
> +
> +
> +BUGS
> +----
> +An up-to-date list of Catalyst bugs can always be found listed on the Gentoo
> +Linux bug-tracking system at `http://bugs.gentoo.org`.
> +
> +
> +SEE ALSO
> +--------
> +*catalyst(1)*
> +*catalyst-config(5)*
> diff --git a/doc/catalyst-spec.5.txt b/doc/catalyst-spec.5.txt
> index 4a6e06c..b3e2bf5 100644
> --- a/doc/catalyst-spec.5.txt
> +++ b/doc/catalyst-spec.5.txt
> @@ -496,7 +496,6 @@ specific options for the game.  This is not used on the release media.
>  FILES
>  -----
>  Example specfiles can be found in '/usr/share/doc/catalyst-{catalystversion}/examples'.
> -An example configuration file can be found at '/etc/catalyst/catalyst.conf'.
>  
>  
>  SUPPORTED ARCHITECTURES
> @@ -516,3 +515,4 @@ Linux bug-tracking system at 'http://bugs.gentoo.org'.
>  SEE ALSO
>  --------
>  *catalyst(1)*
> +*catalyst-config(5)*
> diff --git a/doc/catalyst.1.txt b/doc/catalyst.1.txt
> index 4d51eb0..09dc119 100644
> --- a/doc/catalyst.1.txt
> +++ b/doc/catalyst.1.txt
> @@ -136,6 +136,7 @@ AUTHORS
>  SEE ALSO
>  --------
>  *catalyst-spec(5)*
> +*catalyst-config(5)*
>  
>  Also, a more in-depth examination of Catalyst options and procedures can be found
>  linked from the *catalyst* project page, which is located at
> diff --git a/files/.gitignore b/files/.gitignore
> index 9dbf1f1..7773aeb 100644
> --- a/files/.gitignore
> +++ b/files/.gitignore
> @@ -1,4 +1,5 @@
>  catalyst.1
> +catalyst-config.5
>  catalyst-spec.5
>  *.html
>  *.css
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJRaFaoAAoJEKXdFCfdEflKC2sP+wS1f4RlU0WAKBzocF1cJKh1
Xd4RaxOruDUkOPvtHNvMMmwgSaZD0iiYmGnUZEG5lYEfIIA9wlqGrYHzpL/Mtbvv
iKTYNPtZycfVvtSE2UEC9BhPQe3+VzjqrB4Rs1utdAfCvgYjRTNiOutcHyyr1kWz
G1su+g2dMX7FFb29gfm0kRoYBdwi8XSep62QLoAvEMGRPkM10jo3sFIqb4RWDG1r
XajCEgNFxvrAy7HV1HOLgFg8Oq7N60aJAZQLc353EMHzmCT0K7t+/2Bg8Rl1AH5j
Gnwj1lol5tvfu76MHDTZeykZFln53S9BoewM0RlHqIT1sfIywwxLl/kh/z5z44K8
at1NvnLAllE/tiR99ovaeTUlMPTigoLuuIOqH5gPAdVETQ3cQEhskSWYlc7wKZcs
SU28aa1xcUwdhPaNhplyNaf+VHp8PJPkHUeeItafFGw8Hn7J2angrS5ouOPjRBHh
MG6usDIjRxhJGgCrV28dXksYxCSocmFEO6npuZe8rGkP6k7uLt3di/wt5lsQWdCq
gNtaiYKrnb1u6oWwZJHZdNF9jU9pjoQjBRYkkbn/3N9ItrxUHO+5+Yz1FFV3q7Bs
B+NuxstRLC1imrsz9nHsp0XMkk9p+Un6xUIRvOP8LRRrmdUvwFooClH6vyPasG7C
WnvzbEV0y+xr02Lmdpv7
=Ly6k
-----END PGP SIGNATURE-----

Re: [gentoo-catalyst] [PATCH v2 1/2] doc/catalyst-config.5.txt: Add man page for catalyst.conf

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1882 bytes --]

On Fri, Apr 12, 2013 at 02:47:05PM -0400, Rick "Zero_Chaos" Farina wrote:
> On 04/12/2013 02:21 PM, W. Trevor King wrote:
> > +ccache::
> > +Enable build time ccache support.
> Warning, ccache has been known to cause random build failures and bugs
> reported with ccache enabled may be closed invalid.

Added for v3.

> > +kerncache::
> > +Keep a `tbz2` of your built kernel and modules.  This is useful if
> > +your build dies during `livecd-stage2`.
> kerncache keeps a full dir of everything like seedcache does.

For v3, I'll use:

  kerncache::
  Keep a copy of the built kernel and modules.  This is useful if your
  build dies during `livecd-stage2`.

> > +pkgcache::
> > +Keep a ``tbz2`` of every built package.  This is useful if your build
> > +dies prematurely.
> Might be easier to just say this enables the binpkg feature of portage...

Do you mean `emerge --usepkg --buildpkg`?  How about:

  pkgcache::
  Enable `--usepkg` and `--buildpkg` for most *emerge(1)* runs.  This is
  useful if your build dies prematurely.  However, you may experience
  linking problems.  See the *BINARY PACKAGE DEPENDENCIES* section for
  details.

> > +snapcache::
> > +Cache the snapshot so that it can be bind-mounted into the chroot.
> > +WARNING: moving parts of the portage tree from within fsscript *will*
> > +break your cache.  The cache is unlinked before any empty or rm
> > +processing, though.
> - -, though

Added to v3.

> > +*storedir*::
> > +Location for built seeds, temporary files, and caches.
> mention the default location

For v3:

  *storedir*::
  Location for built seeds, temporary files, and caches (example:
  `/var/tmp/catalyst`).

Thanks!
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] [PATCH v2 1/2] doc/catalyst-config.5.txt: Add man page for catalyst.conf

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

from my side I think this is all better.

- -Zero

On 04/12/2013 03:05 PM, W. Trevor King wrote:
> On Fri, Apr 12, 2013 at 02:47:05PM -0400, Rick "Zero_Chaos" Farina
> wrote:
>> On 04/12/2013 02:21 PM, W. Trevor King wrote:
>>> +ccache:: +Enable build time ccache support.
>> Warning, ccache has been known to cause random build failures and
>> bugs reported with ccache enabled may be closed invalid.
> 
> Added for v3.
> 
>>> +kerncache:: +Keep a `tbz2` of your built kernel and modules.
>>> This is useful if +your build dies during `livecd-stage2`.
>> kerncache keeps a full dir of everything like seedcache does.
> 
> For v3, I'll use:
> 
> kerncache:: Keep a copy of the built kernel and modules.  This is
> useful if your build dies during `livecd-stage2`.
> 
>>> +pkgcache:: +Keep a ``tbz2`` of every built package.  This is
>>> useful if your build +dies prematurely.
>> Might be easier to just say this enables the binpkg feature of
>> portage...
> 
> Do you mean `emerge --usepkg --buildpkg`?  How about:
> 
> pkgcache:: Enable `--usepkg` and `--buildpkg` for most *emerge(1)*
> runs.  This is useful if your build dies prematurely.  However, you
> may experience linking problems.  See the *BINARY PACKAGE
> DEPENDENCIES* section for details.
> 
>>> +snapcache:: +Cache the snapshot so that it can be bind-mounted
>>> into the chroot. +WARNING: moving parts of the portage tree
>>> from within fsscript *will* +break your cache.  The cache is
>>> unlinked before any empty or rm +processing, though.
>> - -, though
> 
> Added to v3.
> 
>>> +*storedir*:: +Location for built seeds, temporary files, and
>>> caches.
>> mention the default location
> 
> For v3:
> 
> *storedir*:: Location for built seeds, temporary files, and caches
> (example: `/var/tmp/catalyst`).
> 
> Thanks! Trevor
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJRaGC/AAoJEKXdFCfdEflKsY0P/12719rYWUfYwXRwQNL7ejJR
xhm8xgZEnv+36/ssKH5fLyOpaUTMlDA861XbMTiuBTD+tzElef2LTINh9xv3iNwc
5iunUnvy21EhQgjN3fvWr76d2R2ZeN5nSc0srVVpYqctnyl4lRJcSvIT+bHmC72s
2ve665ntVQO6oT4tcIIg2AYhO66J+uODuNC0r2RWkResYH5mkuhKLrqCWPpBSlae
+LHERtEUB22wI1fB7GHMj0UhM/Y4RPeLUl0OpKLVGwL8qjMuraF4TbaCpsj4EQfF
BsiDnFv2xJrlomV6f+I1Agdt06864HwB99L09u+DnBQJLfUHZSYTYcxc3yFKix4U
B+i0H/wdK2hbiomo4GJ45i+c0JcBtcASOTFULgC7QG1WBCp34zXWMQB0q4DAzq0E
kUyV/ZTDDmqic5DrH8Q3gkr7JFiq8qR49y72BmjHYGkSetfh2YHPnTBLqPXe1X/Q
Svuji/eyk/wunbePd1IWRjIgvv7V0Ne9+PAU8rLxl+3I6GifGXerTE7BfvdD97EG
sW/OkvNxSPKwkC1eQFqzuBB40OM6Sirtr6KhOCkLHIU/Nvl85gkVx8PgoIbfvKlb
z/GfE5Iv4jR+PQ/BgcEOy79PTqVEi9N+Iv0z8n4SnUntM/EC641x65Y97kJKpuit
/yg5N8NJ+3VpcSvmxFD+
=Sb7o
-----END PGP SIGNATURE-----

[gentoo-catalyst] [PATCH v3 0/2] pkgcache warning in catalyst-config(5)

[W. Trevor King]

From: "W. Trevor King" <wking@tremily.us>

Changes since v2:
* s/catalyst-config/catalyst-spec/ in catalyst-config.5.txt's “See
  Also” section [1].
* Assorted rewordings as suggested by Zero_Chaos [2] and acknowledged
  in [3].

[1]: http://mid.gmane.org/20130412182752.GD13054@odin.tremily.us
[2]: http://mid.gmane.org/516856A9.5030909@gentoo.org
[3]: http://mid.gmane.org/20130412190524.GE13054@odin.tremily.us

W. Trevor King (2):
  doc/catalyst-config.5.txt: Add man page for catalyst.conf
  doc/catalyst-config.5.txt: Document linking issues with binary
    packages

 doc/catalyst-config.5.txt | 234 ++++++++++++++++++++++++++++++++++++++++++++++
 doc/catalyst-spec.5.txt   |   2 +-
 doc/catalyst.1.txt        |   1 +
 files/.gitignore          |   1 +
 4 files changed, 237 insertions(+), 1 deletion(-)
 create mode 100644 doc/catalyst-config.5.txt

-- 
1.8.2

[gentoo-catalyst] [PATCH v3 1/2] doc/catalyst-config.5.txt: Add man page for catalyst.conf

[W. Trevor King]

From: "W. Trevor King" <wking@tremily.us>

This mostly translates the inline comments from files/catalyst.conf
into asciidoc.  While it's nice to have that as stand-alone
documentation, it also makes it easier to refer to longer descriptions
of any tricky issues.  This gives us a place to distill the collected
wisdom of current users for the benefit of others.
---
 doc/catalyst-config.5.txt | 192 ++++++++++++++++++++++++++++++++++++++++++++++
 doc/catalyst-spec.5.txt   |   2 +-
 doc/catalyst.1.txt        |   1 +
 files/.gitignore          |   1 +
 4 files changed, 195 insertions(+), 1 deletion(-)
 create mode 100644 doc/catalyst-config.5.txt

diff --git a/doc/catalyst-config.5.txt b/doc/catalyst-config.5.txt
new file mode 100644
index 0000000..27bc0bb
--- /dev/null
+++ b/doc/catalyst-config.5.txt
@@ -0,0 +1,192 @@
+CATALYST-CONFIG(5)
+================
+:man source:   catalyst {catalystversion}
+:man manual:   catalyst {catalystversion}
+
+
+NAME
+----
+catalyst-config - Catalyst configuration files
+
+
+SYNOPSIS
+--------
+*catalyst* ['OPTIONS'] *-c* 'FILE'
+
+
+DESCRIPTION
+-----------
+
+*catalyst(1)* reads the configuration file given with `-c` or
+`--config` on the command line.  The file contains keyword-argument
+pairs using a POSIX Shell variable syntax.  Lines starting with `#`
+and empty lines are interpreted as comments.  For example:
+
+---------------------------------
+# /etc/catalyst/catalyst.conf
+digests="md5 sha1 sha512 whirlpool"
+contents="auto"
+distdir="/usr/portage/distfiles"
+envscript="/etc/catalyst/catalystrc"
+options="autoresume bindist kerncache pkgcache seedcache snapcache"
+portdir="/usr/portage"
+sharedir="/usr/lib/catalyst"
+snapshot_cache="/var/tmp/catalyst/snapshot_cache"
+storedir="/var/tmp/catalyst"
+---------------------------------
+
+The possible keywords and their meanings are as follows:
+
+Basic configuration
+~~~~~~~~~~~~~~~~~~~
+
+*digests*::
+Create a `.DIGESTS` file containing the hash output from any of the
+supported options below.  Adding them all may take a long time.
+(example: `md5 sha1 sha512 whirlpool`).  See the *SUPPORTED HASHES*
+section for a list of supported hashes.
+
+*contents*::
+Create a `.CONTENTS` file listing the contents of the file.  If this
+variable is empty, no `.CONTENTS` will be generated at all.  Supported
+values:
++
+--
+auto::
+Strongly recommended
+
+tar-tv::
+Do `tar tvf FILE`
+
+tar-tvz::
+Do `tar tvzf FILE`
+
+tar-tvy::
+Do `tar tvyf FILE`
+
+isoinfo-l::
+Do `isoinfo -l -i FILE`
+
+isoinfo-f::
+Do `isoinfo -f -i FILE`.  'isoinfo-f' is the only option not chosen
+by the automatic algorithm.
+--
+
+*distdir*::
+Distfiles location.  `/usr/portage/distfiles` should work for most
+default installations.
+
+*envscript*::
+Environment script location, which allows users to set options such as
+HTTP proxies, `MAKEOPTS`, `GENTOO_MIRRORS`, or any other environment
+variables needed for building.  The envscript file sets environment
+variables using POSIX shell notation:
++
+---------------------------------
+export FOO="bar"
+---------------------------------
+
+*hash_function*::
+Internal hash function catalyst should use for things like autoresume,
+seedcache, etc.  The default and fastest is `crc32`.  You should not
+ever need to change this unless your OS does not support it.  See the
+*SUPPORTED HASHES* section for a list of supported hashes.
+
+**options*::
+Set different build-time options (example: `autoresume bindist
+kerncache pkgcache seedcache snapcache`).  Supported values:
++
+--
+autoresume::
+Attempt to resume a failed build.  Clear the autoresume flags with the
+`-a` option to the catalyst command line.  `-p` will clear the
+autoresume flags as well as your pkgcache and kerncache.
+
+bindist::
+Enable the `bindist` `USE` flag.  This is recommended if you will be
+redistributing builds, but see the package specific definitions for
+details.
+
+ccache::
+Enable build time ccache support.  WARNING: ccache has been known to
+cause random build failures and bugs reported with ccache enabled may
+be closed invalid.
+
+distcc::
+Enable distcc support for building. You have to set distcc_hosts in
+your spec file.
+
+icecream::
+Enable icecream compiler cluster support for building.
+
+kerncache::
+Keep a copy of the built kernel and modules.  This is useful if your
+build dies during `livecd-stage2`.
+
+pkgcache::
+Enable `--usepkg` and `--buildpkg` for most *emerge(1)* runs.  This is
+useful if your build dies prematurely.  However, you may experience
+linking problems.
+
+seedcache::
+Use the build output of a previous target if it exists to speed up the
+creation of a new stage.  This avoids unpacking the seed tarball.
+
+snapcache::
+Cache the snapshot so that it can be bind-mounted into the chroot.
+WARNING: moving parts of the portage tree from within fsscript *will*
+break your cache.  The cache is unlinked before any empty or rm
+processing.
+--
+
+*portdir*::
+Portage tree location.  `/usr/portage/` should work for most default
+installations.
+
+*sharedir*::
+Catalyst runtime script location.  `/usr/lib/catalyst` should work for
+most default installations.  If you are running catalyst from a Git
+checkout, you should change this to point to your checkout directory.
+
+*storedir*::
+Location for built seeds, temporary files, and caches (example:
+`/var/tmp/catalyst`).
+
+*port_logdir*::
+Location for build logs (example: `/var/tmp/catalyst/tmp`).  This dir
+will be automatically cleaned of all logs over 30 days old. If left
+undefined the logs will remain in the build directory as usual and get
+cleaned every time a stage build is restarted.
+
+*var_tmpfs_portage*::
+Set the size of a `/var/tmp/portage` tmpfs in gigabytes (example:
+`16`).  If set, this mounts a tmpfs for `/var/tmp/portage` so building
+takes place in RAM.  This feature requires a pretty large tmpfs
+({open,libre}office needs ~8GB to build).  WARNING: If you use too
+much RAM everything will fail horribly and it is not our fault.
+
+
+SUPPORTED HASHES
+----------------
+Supported hashes: adler32, crc32, crc32b, gost, haval128, haval160,
+haval192, haval224, haval256, md2, md4, md5, ripemd128, ripemd160,
+ripemd256, ripemd320, sha1, sha224, sha256, sha384, sha512, snefru128,
+snefru256, tiger, tiger128, tiger160, whirlpool.
+
+
+FILES
+-----
+An example configuration file can be found at
+`/etc/catalyst/catalyst.conf`.
+
+
+BUGS
+----
+An up-to-date list of Catalyst bugs can always be found listed on the Gentoo
+Linux bug-tracking system at `http://bugs.gentoo.org`.
+
+
+SEE ALSO
+--------
+*catalyst(1)*
+*catalyst-spec(5)*
diff --git a/doc/catalyst-spec.5.txt b/doc/catalyst-spec.5.txt
index 4a6e06c..b3e2bf5 100644
--- a/doc/catalyst-spec.5.txt
+++ b/doc/catalyst-spec.5.txt
@@ -496,7 +496,6 @@ specific options for the game.  This is not used on the release media.
 FILES
 -----
 Example specfiles can be found in '/usr/share/doc/catalyst-{catalystversion}/examples'.
-An example configuration file can be found at '/etc/catalyst/catalyst.conf'.
 
 
 SUPPORTED ARCHITECTURES
@@ -516,3 +515,4 @@ Linux bug-tracking system at 'http://bugs.gentoo.org'.
 SEE ALSO
 --------
 *catalyst(1)*
+*catalyst-config(5)*
diff --git a/doc/catalyst.1.txt b/doc/catalyst.1.txt
index 4d51eb0..9f70f78 100644
--- a/doc/catalyst.1.txt
+++ b/doc/catalyst.1.txt
@@ -135,6 +135,7 @@ AUTHORS
 
 SEE ALSO
 --------
+*catalyst-config(5)*
 *catalyst-spec(5)*
 
 Also, a more in-depth examination of Catalyst options and procedures can be found
diff --git a/files/.gitignore b/files/.gitignore
index 9dbf1f1..7773aeb 100644
--- a/files/.gitignore
+++ b/files/.gitignore
@@ -1,4 +1,5 @@
 catalyst.1
+catalyst-config.5
 catalyst-spec.5
 *.html
 *.css
-- 
1.8.2

[gentoo-catalyst] [PATCH v3 2/2] doc/catalyst-config.5.txt: Document linking issues with binary packages

[W. Trevor King]

From: "W. Trevor King" <wking@tremily.us>

This gives users a heads up explaining why they might see linking
errors when pkgcache is enabled.  I first saw this when I build a
stage1 without update_seed.  Because my seed stage3 linked against
libmpc.so.2, some of my stage1 files linked against the older mpc.
However, the mpc-1.0.1 built for the stage1 installed libmpc.so.3.
When I tried to use this stage1 to build a stage2, it died with:

  /usr/libexec/gcc/i686-pc-linux-gnu/4.6.3/cc1:
    error while loading shared libraries: libmpc.so.2:
    cannot open shared object file: No such file or directory

To fix this, I enabled update_seed, but binary packages built during
my first pass were used to populate the stage1, so even though I'd
updated the seed stage3 toolchain, I still had a stage1 with cc1
linked against libmpc.so.2.

After clearing the binary package cache, I got a stage1 *built* with
the updated seed stage3, which gave a cc1 linked against libmpc.so.3
(hurray!).

This commit adds a warning in the pkgcache documentation that should
help people understand what might be going wrong if they see similar
linking errors.  For more details, see the thread following
http://thread.gmane.org/gmane.linux.gentoo.catalyst/2137/focus=2193
---
 doc/catalyst-config.5.txt | 44 +++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 43 insertions(+), 1 deletion(-)

diff --git a/doc/catalyst-config.5.txt b/doc/catalyst-config.5.txt
index 27bc0bb..63a015f 100644
--- a/doc/catalyst-config.5.txt
+++ b/doc/catalyst-config.5.txt
@@ -126,7 +126,8 @@ build dies during `livecd-stage2`.
 pkgcache::
 Enable `--usepkg` and `--buildpkg` for most *emerge(1)* runs.  This is
 useful if your build dies prematurely.  However, you may experience
-linking problems.
+linking problems.  See the *BINARY PACKAGE DEPENDENCIES* section for
+details.
 
 seedcache::
 Use the build output of a previous target if it exists to speed up the
@@ -174,6 +175,47 @@ ripemd256, ripemd320, sha1, sha224, sha256, sha384, sha512, snefru128,
 snefru256, tiger, tiger128, tiger160, whirlpool.
 
 
+BINARY PACKAGE DEPENDENCIES
+---------------------------
+This section is only important if you are using binary packages to
+build your stages (by enabling the `pkgcache` option and restarting
+incomplete builds).
+
+Before EAPI-5 introduced ABI sub-slots, the build-time compatibility
+of packages was not recorded.  This leads to problems such as binary
+GCC packages built against mpc-0.8.2 (which installs libmpc.so.2)
+being installed on systems that only have mpc-1.0.1 (which installs
+libmpc.so.3), resulting in:
+
+---------------------------------
+/usr/libexec/gcc/i686-pc-linux-gnu/4.6.3/cc1:
+  error while loading shared libraries: libmpc.so.2:
+  cannot open shared object file: No such file or directory
+---------------------------------
+
+As long as there are packages in your stage that don't use ABI
+sub-slots, you may experience errors like this due to untracked ABI
+missmatches in binary packages.  Packages generated by catalyst builds
+are currently namespaced:
+
+---------------------------------
+.../packages/<rel_type>/<target>-<subarch>-<version_stamp>/Packages
+---------------------------------
+
+so running into these out-of-date packages is unlikely.  You may run
+into problems if:
+
+* you enable `update_seed` in your stage1 spec after a previous run
+  which generated packages linking against out-of-date seed libraries
+  or
+* you update your snapshot and an untracked ABI dependency is bumped
+  without a similar bump in the dependent package.
+
+without also bumping any of the package namespace variables in your
+spec.  If you do make such a change, it's a good idea to clear the
+package cache in question and rebuild the packages from scratch.
+
+
 FILES
 -----
 An example configuration file can be found at
-- 
1.8.2

[gentoo-catalyst] [PATCH 0/2] Blacklisting binary packages

[W. Trevor King]

From: "W. Trevor King" <wking@tremily.us>

The current approach to avoiding problems due to stale binary packages
with untracked ABI dependencies is to disable binpkg use during
troublesome sections of the build (e.g. seed updates).  I think that a
cleaner solution would be to use a configurable spec option
blacklisting binpkgs for troublesome packages.  For example, in a
stage1 with update_seed enabled, the Portage emerge (before the seed
update) has:

  …
  removed '/usr/portage/packages/sys-devel/gcc-4.6.3.tbz2'
  removed '/usr/portage/packages/sys-devel/gcc-config-1.7.3.tbz2'
  emerge --quiet --usepkg --buildpkg --newuse --oneshot --nodeps --update sys-apps/portage
  …

During the seed update, the rest of the packages can use preexisting
binpkgs from earlier builds:

  …
  >>> Emerging binary (1 of 17) sys-devel/gnuconfig-20121010
  >>> Installing (1 of 17) sys-devel/gnuconfig-20121010
  >>> Emerging binary (2 of 17) app-misc/mime-types-9
  >>> Installing (2 of 17) app-misc/mime-types-9
  >>> Emerging (3 of 17) sys-libs/timezone-data-2013a
  >>> Installing (3 of 17) sys-libs/timezone-data-2013a
  …

I'm not sure yet why some packages (e.g. timezone-data) don't use the
prexisting binpkgs, even though they do exist:

  # ls packages/default/stage1-i686-2013.1/sys-libs/timezone-data-201*
  packages/default/stage1-i686-2013.1/sys-libs/timezone-data-2012j.tbz2
  packages/default/stage1-i686-2013.1/sys-libs/timezone-data-2013a.tbz2

Later in the stage1 build, GCC is not installed from a binary package
(because this package was removed earlier):

  …
  >>> Emerging binary (59 of 75) virtual/libc-0 for /tmp/stage1root/
  >>> Installing (59 of 75) virtual/libc-0 to /tmp/stage1root/
  >>> Emerging (60 of 75) sys-devel/gcc-4.6.3 for /tmp/stage1root/
  >>> Installing (60 of 75) sys-devel/gcc-4.6.3 to /tmp/stage1root/
  …

I think this approach is cleaner because it:

* Avoids problems we've already seen by blacklisting troublesome
  packages.
* Allows you to reuse binary packages in the seed update stage, at the
  cost of rebuilding gcc for your stage1root.
* Can be applied to any stage, so you don't get bitten by stale
  binpkgs in stage2, etc. due to a snapshot update.
* Can be overridden by folks who know some blacklisted package will be
  fine for their combination of seed, snapshot, and pkgcache.

W. Trevor King (2):
  spec: Add binpkg_blacklist option for troublesome packages
  Revert "don't build packages during update_seed"

 modules/generic_stage_target.py     | 4 +++-
 targets/stage1/stage1-chroot.sh     | 4 ++--
 targets/support/chroot-functions.sh | 5 +++++
 3 files changed, 10 insertions(+), 3 deletions(-)

-- 
1.8.2

[gentoo-catalyst] [PATCH 1/2] spec: Add binpkg_blacklist option for troublesome packages

[W. Trevor King]

From: "W. Trevor King" <wking@tremily.us>

For packages that don't use EAPI-5's sub-slots to track ABI
dependencies, binary packages may go stale without Portage noticing.
You can use this new option to list packages that may be troublesome.
Listed packages will be removed from the pkgcache, and rebuilt from
scratch for your seed.

If binpkg_blacklist is unset in your spec file, it defaults to
including the following packages:

* sys-devel/gcc: see, for example,
  http://mid.gmane.org/20130412011158.GB17116@odin.tremily.us
* media-libs/mesa: see, for example, #454184
---
 modules/generic_stage_target.py     | 4 +++-
 targets/support/chroot-functions.sh | 5 +++++
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/modules/generic_stage_target.py b/modules/generic_stage_target.py
index 31cb025..4fbbb44 100644
--- a/modules/generic_stage_target.py
+++ b/modules/generic_stage_target.py
@@ -16,7 +16,9 @@ class generic_stage_target(generic_target):
 		self.valid_values.extend(["version_stamp","target","subarch",\
 			"rel_type","profile","snapshot","source_subpath","portage_confdir",\
 			"cflags","cxxflags","ldflags","cbuild","hostuse","portage_overlay",\
-			"distcc_hosts","makeopts","pkgcache_path","kerncache_path"])
+			"distcc_hosts","makeopts","pkgcache_path","kerncache_path",
+			'binpkg_blacklist',
+			])
 
 		self.set_valid_build_kernel_vars(addlargs)
 		generic_target.__init__(self,myspec,addlargs)
diff --git a/targets/support/chroot-functions.sh b/targets/support/chroot-functions.sh
index 35ffbaa..957af9d 100644
--- a/targets/support/chroot-functions.sh
+++ b/targets/support/chroot-functions.sh
@@ -261,6 +261,11 @@ run_merge() {
 	export EPAUSE_IGNORE=0
 	export CONFIG_PROTECT="-*"
 
+	default_binpkg_blacklist="sys-devel/gcc media-libs/mesa"
+	for binpkg in ${clst_binpkg_blacklist-${default_binpkg_blacklist}}; do
+		rm -vf "/usr/portage/packages/${binpkg}"*
+	done
+
 	if [ -n "${clst_VERBOSE}" ]
 	then
 		echo "ROOT=${ROOT} emerge ${clst_myemergeopts} -pt $@" || exit 1
-- 
1.8.2

[gentoo-catalyst] [PATCH 2/2] Revert "don't build packages during update_seed"

[W. Trevor King]

From: "W. Trevor King" <wking@tremily.us>

This reverts commit e7ea409acb52b43e9ea141c57201f9f87673f7ba.

This is a different approach to the same binpkg ABI compatibility
problem that I address in 1ad133e (spec: Add binpkg_blacklist option
for troublesome packages, 2013-04-15).  We only need to have one
approach, and I think binpkg_blacklist is cleaner.
---
 targets/stage1/stage1-chroot.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/targets/stage1/stage1-chroot.sh b/targets/stage1/stage1-chroot.sh
index 3f628c2..462ea28 100644
--- a/targets/stage1/stage1-chroot.sh
+++ b/targets/stage1/stage1-chroot.sh
@@ -28,9 +28,9 @@ if [ -n "${clst_update_seed}" ]; then
 	if [ "${clst_update_seed}" == "yes" ]; then
 		echo "Updating seed stage..."
 		if [ -n "${clst_update_seed_command}" ]; then
-			clst_root_path=/ run_merge "--buildpkg=n ${clst_update_seed_command}"
+			clst_root_path=/ run_merge "${clst_update_seed_command}"
 		else
-			clst_root_path=/ run_merge "--buildpkg=n --update --deep --newuse --onlydeps gcc"
+			clst_root_path=/ run_merge "--update --deep --newuse --onlydeps gcc"
 		fi
 	elif [ "${clst_update_seed}" != "no" ]; then
 		echo "Invalid setting for update_seed: ${clst_update_seed}"
-- 
1.8.2

Re: [gentoo-catalyst] [PATCH 0/2] Blacklisting binary packages

[Matt Turner]

On Tue, Apr 16, 2013 at 12:42 PM, W. Trevor King <wking@tremily.us> wrote:
> From: "W. Trevor King" <wking@tremily.us>

I feel like this is getting out of control.

There seems to be this overwhelming desire to keep applying hack after
hack to make increasingly obscure -- and fundamentally unsupportable
-- configurations work.

Brian mentioned that he talked to Zac and that there was something
that portage should be doing differently about binary packages. Let's
please investigate that a bit further before pursuing other
strategies. I have confidence that if Zac understands and knows about
the problem that he can fix portage.

Brian, Zac: do we have a bug filed? What's the status -- do we have a
good understanding of what's required?

Thanks,
Matt

Re: [gentoo-catalyst] [PATCH 0/2] Blacklisting binary packages

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 2411 bytes --]

On Tue, Apr 16, 2013 at 01:35:40PM -0700, Matt Turner wrote:
> I feel like this is getting out of control.

I'm just proposing an alternative approach.  Feel free to ignore ;).
Also, feel free to tell me to drop this issue for $x weeks until folks
have some more free time ;).

> There seems to be this overwhelming desire to keep applying hack after
> hack to make increasingly obscure -- and fundamentally unsupportable
> -- configurations work.

I was trying to replace the existing hack (from e7ea409, and 6c0a577
through f6ad38) with a simpler, less obscure one.

> Brian mentioned that he talked to Zac and that there was something
> that portage should be doing differently about binary packages. Let's
> please investigate that a bit further before pursuing other
> strategies. I have confidence that if Zac understands and knows about
> the problem that he can fix portage.

That sounds wonderful.  My impression was that neither Portage nor the
toolchain ebuilds were going to be changed to address this problem.
The last I hear on the Portage front was (quoted in [1]):

  10:33 < zmedico> dol-sen: wouldn't it be easier to just migrate
          those pkgs to EAPI 5 + slot-operator?
  10:34 * zmedico doesn't feel like doing extra work when EAPI 5
          already does everything we need
  …
  11:16 < Tommy[D]_> Zero_Chaos: my suggestion: ask the toolchain guys
          about their preferred way (like updating existing eclass,
          using a new eclass, moving code into ebuilds) and when you
          got that, do the needed work, including an EAPI-5 version of
          toolchain ebuilds

If Zac's changed his mind, I'd be happy to wait for the Portage bump
that works around poorly specified ABI dependencies.  However, note
that even some EAPI-5 packages don't use ABI sub-slots to avoid this
issue [2], so unless the Portage work-around deals with such ebuilds
appropriately, we'll still need some sort of hack in catalyst.

Cheers,
Trevor

[1]: http://mid.gmane.org/20130412162540.GB13054@odin.tremily.us
[2]: https://bugs.gentoo.org/show_bug.cgi?id=454184#c18
     (I'll work up the patches Zero_Chaos asks for in #c19 now, but I
     expect this is not an isolated event)

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] [PATCH 0/2] Blacklisting binary packages

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 596 bytes --]

On Tue, Apr 16, 2013 at 03:28:04PM -0700, Zac Medico wrote:
> If you can't get the people maintaining the ebuilds in gentoo-x86 to
> cooperate with migration to EAPI 5, then I think your best option is
> to start an eapi-5 overlay which contains EAPI 5 ports of those
> ebuilds.

Good idea.  I'll look into that after I hit a wall on sub-slotting
udev [1].

Cheers,
Trevor

[1]: https://bugs.gentoo.org/show_bug.cgi?id=466162

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] [PATCH 0/2] Blacklisting binary packages

[Brian Dolbec]

On Tue, 2013-04-16 at 18:53 -0400, W. Trevor King wrote:
> On Tue, Apr 16, 2013 at 03:28:04PM -0700, Zac Medico wrote:
> > If you can't get the people maintaining the ebuilds in gentoo-x86 to
> > cooperate with migration to EAPI 5, then I think your best option is
> > to start an eapi-5 overlay which contains EAPI 5 ports of those
> > ebuilds.
> 
> Good idea.  I'll look into that after I hit a wall on sub-slotting
> udev [1].
> 
> Cheers,
> Trevor
> 
> [1]: https://bugs.gentoo.org/show_bug.cgi?id=466162
> 

I think it should be the other way around.

If they still insist them needing to be EAPI=0 for upgrade capability
for old systems.  Then the EAPI=0 ebuilds should be put in a
Legacy-upgrade overlay.

If udev is troublesome, then try eudev.  You might even be able to get
the devs working on it to make it EAPI 5.

Re: [gentoo-catalyst] [PATCH 0/2] Blacklisting binary packages

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 2680 bytes --]

On Tue, Apr 16, 2013 at 09:18:00PM -0700, Brian Dolbec wrote:
> If udev is troublesome, then try eudev.  You might even be able to get
> the devs working on it to make it EAPI 5.

I think it needs to be the whole udev ecosystem, since mesa (which
caused #454184) only depends on virtual/udev.  We need the ABI
slotting to pass all the way through.  That means:

  gbm? ( virtual/udev:= )

in mesa and something like

  >=sys-fs/udev-197-r8:=[gudev?,…]
  >=sys-apps/systemd-198-r5:=[gudev?,
  kmod? ( >=sys-fs/eudev-1_beta2-r2:=[modutils,…] )
  !kmod? ( >=sys-fs/eudev-1_beta2-r2:=[gudev?,…] )

in virtual/udev-197-r2.  With sub-slotting in the different
implementation ebuilds.  The trouble is that sys-fs/udev installs both
libsystemd-daemon and libudev, which are versioned separately.  This
means you need virtuals for each ABI [1].  With a new virtual/udev-abi
and virtual/systemd-daemon-abi, you'd need to add virtual/udev-abi as
a dependency for mesa (otherwise, only virtual/udev would be rebuilt
on a libudev version bump :p).  It's not clear to me how to handle the
udev multiplexing in virtual/udev-abi.  I suppose there should be
separate versions for both systemd and eudev, but I'm still working
out the details.

However, none of this is really helping get a catalyst release out the
door.  jmbsvicetto's recent patches are dealing with two unrelated
issues: preserved-libs (3b83a6c, 462348d, and f6ad384) and stale
binpkgs (6c0a577).  I'm only taking issue with 6c0a577 (and its
predecessor e7ea409).  Also, 6c0a577 is a small enough change that I'm
fine with reverting that as well if I can talk folks over to my
binpkg-blacklist approach (after the release).  It would have been
nice if the commit message for 6c0a577 explained how it was different
from dol-sen's earlier proposal [2], as I suggested when the patch was
floated on gentoo-catalyst@.  It would also have been nice if we'd
caught the s/;/:/ typo in 462348d during ML review, but I didn't.  Now
that these patches have landed in `master`, I think we should just
roll with them and cut a new release :p.

We will certainly not have a fix for stale binpkgs in the near future,
other than disabling pkgcache.  I'm fine debating the best catalyst
workaround, but if y'all don't agree with me, I'm not going to blow
up ;).

Cheers,
Trevor

[1]: http://wiki.gentoo.org/wiki/Sub-slots_and_Slot-Operators#Multiple_ABIs_for_X.Org
[2]: http://article.gmane.org/gmane.linux.gentoo.catalyst/2166/

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] [PATCH 0/2] Blacklisting binary packages

[Matt Turner]

On Wed, Apr 17, 2013 at 4:30 AM, W. Trevor King <wking@tremily.us> wrote:
> It would also have been nice if we'd
> caught the s/;/:/ typo in 462348d during ML review, but I didn't.

It would, yes, but it's expected that patches sent to the mailing list
have actually been tested. Doing otherwise (and not stating so) is
inexcusable. This has happened a number of times already.

Re: [gentoo-catalyst] [PATCH 0/2] Blacklisting binary packages

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/16/2013 03:42 PM, W. Trevor King wrote:
> From: "W. Trevor King" <wking@tremily.us>
> 
> The current approach to avoiding problems due to stale binary packages
> with untracked ABI dependencies is to disable binpkg use during
> troublesome sections of the build (e.g. seed updates).  I think that a
> cleaner solution would be to use a configurable spec option
> blacklisting binpkgs for troublesome packages.  For example, in a
> stage1 with update_seed enabled, the Portage emerge (before the seed
> update) has:
> 
This needs to remain optional. I'm not going to nack a patch that some
people may find useful but in my personal opinion this is a terrible
solution that should not be used.  We need to find a way to rebuild
packages as needed (like EAPI 5) not force a rebuild everytime.

- -ZC
>   …
>   removed '/usr/portage/packages/sys-devel/gcc-4.6.3.tbz2'
>   removed '/usr/portage/packages/sys-devel/gcc-config-1.7.3.tbz2'
>   emerge --quiet --usepkg --buildpkg --newuse --oneshot --nodeps --update sys-apps/portage
>   …
> 
> During the seed update, the rest of the packages can use preexisting
> binpkgs from earlier builds:
> 
>   …
>   >>> Emerging binary (1 of 17) sys-devel/gnuconfig-20121010
>   >>> Installing (1 of 17) sys-devel/gnuconfig-20121010
>   >>> Emerging binary (2 of 17) app-misc/mime-types-9
>   >>> Installing (2 of 17) app-misc/mime-types-9
>   >>> Emerging (3 of 17) sys-libs/timezone-data-2013a
>   >>> Installing (3 of 17) sys-libs/timezone-data-2013a
>   …
> 
> I'm not sure yet why some packages (e.g. timezone-data) don't use the
> prexisting binpkgs, even though they do exist:
> 
>   # ls packages/default/stage1-i686-2013.1/sys-libs/timezone-data-201*
>   packages/default/stage1-i686-2013.1/sys-libs/timezone-data-2012j.tbz2
>   packages/default/stage1-i686-2013.1/sys-libs/timezone-data-2013a.tbz2
> 
> Later in the stage1 build, GCC is not installed from a binary package
> (because this package was removed earlier):
> 
>   …
>   >>> Emerging binary (59 of 75) virtual/libc-0 for /tmp/stage1root/
>   >>> Installing (59 of 75) virtual/libc-0 to /tmp/stage1root/
>   >>> Emerging (60 of 75) sys-devel/gcc-4.6.3 for /tmp/stage1root/
>   >>> Installing (60 of 75) sys-devel/gcc-4.6.3 to /tmp/stage1root/
>   …
> 
> I think this approach is cleaner because it:
> 
> * Avoids problems we've already seen by blacklisting troublesome
>   packages.
> * Allows you to reuse binary packages in the seed update stage, at the
>   cost of rebuilding gcc for your stage1root.
> * Can be applied to any stage, so you don't get bitten by stale
>   binpkgs in stage2, etc. due to a snapshot update.
> * Can be overridden by folks who know some blacklisted package will be
>   fine for their combination of seed, snapshot, and pkgcache.
> 
> W. Trevor King (2):
>   spec: Add binpkg_blacklist option for troublesome packages
>   Revert "don't build packages during update_seed"
> 
>  modules/generic_stage_target.py     | 4 +++-
>  targets/stage1/stage1-chroot.sh     | 4 ++--
>  targets/support/chroot-functions.sh | 5 +++++
>  3 files changed, 10 insertions(+), 3 deletions(-)
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJRcVCmAAoJEKXdFCfdEflKD40QALjM2FK1L8wiwOQfuov9lMjZ
WO2Qqmyx+asctFdtmpU9QV8GP3j68ZgFM3L/EhWSK0Fi3O9QiEiXXjdnSEH8iqF4
SsMvhCP2l05mBvYYrmpeHKCPmDLxGWB6ppMl/gNLzDOBNSk3VBwXILk5N2PN0QXz
3mnD48QeMd9sAVHKSFOoaY2SsHLQBR6WEH+wdq+bJmrwU8Y949hOksxm65itII+V
1jMpTo5hhBwrdqI04Uk0FQBnqF06zhm8eZO2HhbaFF4P5l+BCahFle+96T3X+ktZ
NN4hw+5woxQOlPe8I2VX/4TdRtRvosdDkop3NnaBENKLu5pTqc0UsuRCw/DQolBg
cgZJ3i6zYDjUTWCGq67nPVZtp3seS9hWJ75s/r8OKumpev+v4WDWt1XbY6UEChXg
4uTSD1lRLV+sXVNJthd5BGQb36mXiVM2muja2edOQfUHp1zxXJD3TklHXenDRw4w
zioui7bsSi0ywI0ftdqff7M9XqlvogGQyduWswnPtoLOrDx32Z+UrAeAfihV9vHy
ywhIb5n/EfBplARkSl+Q7QMemgrB2PEEIQJFLbZaGTHgAtTAwzcc3s5wB7KHIDvb
RJ320R51FgWKgqqO1R6Bpqr2U6lUbE03uQDVlW82AWLM/WTRWgF+Fs18gHhvIo7n
LOCMQuXVOV9HtRhYrfq9
=R5YJ
-----END PGP SIGNATURE-----

Re: [gentoo-catalyst] [PATCH 0/2] Blacklisting binary packages

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 2426 bytes --]

On Fri, Apr 19, 2013 at 10:11:50AM -0400, Rick "Zero_Chaos" Farina wrote:
> On 04/16/2013 03:42 PM, W. Trevor King wrote:
> > From: "W. Trevor King" <wking@tremily.us>
> > 
> > The current approach to avoiding problems due to stale binary packages
> > with untracked ABI dependencies is to disable binpkg use during
> > troublesome sections of the build (e.g. seed updates).  I think that a
> > cleaner solution would be to use a configurable spec option
> > blacklisting binpkgs for troublesome packages.  For example, in a
> > stage1 with update_seed enabled, the Portage emerge (before the seed
> > update) has:
>
> This needs to remain optional.

It is optional.  Just set `binpkg_blacklist` explicitly to an empty
string if you don't want to blacklist the default packages (currently
only two).

> I'm not going to nack a patch that some people may find useful but
> in my personal opinion this is a terrible solution that should not
> be used.  We need to find a way to rebuild packages as needed (like
> EAPI 5) not force a rebuild everytime.

Agreed, but in the absence of one of the following:

* a tree full of EAPI-5+ packages that correctly use ABI sub-slotting
  (if I live long enough to see this, I'll be very happy ;),
* local overlays fixing ABI sub-slotting (maintained by folks without
  much experience in the packages in question?), or
* Portage tweaks to work around packages that don't use EAPI-5+ (or
  that do, but don't use ABI sub-slots appropriately).

I'm fairly confident that none of those are happening in the next six
months, and there seems to be agreement that catalyst needs some sort
of local hack to work around the problem until one of the real
solutions lands.

The `binpkg_blacklist` option lets you name (on a per-package level)
troublesome packages that get ABI sub-slots wrong (or don't even try).
I think that this approach will force *fewer* rebuilds than the
current approach (from e7ea409 and 6c0a577) which blacklists
everything from the seed update.

Another (non-exclusive) possibility is to put a big warning on the
binpkgs option [1] and leave it up to folks to use at their own risk.

Cheers,
Trevor

[1]: http://mid.gmane.org/cover.1366075786.git.wking@tremily.us

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-catalyst] [PATCH 0/2] Blacklisting binary packages

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 04/19/2013 12:18 PM, W. Trevor King wrote:
> On Fri, Apr 19, 2013 at 10:11:50AM -0400, Rick "Zero_Chaos" Farina wrote:
>> On 04/16/2013 03:42 PM, W. Trevor King wrote:
>>> From: "W. Trevor King" <wking@tremily.us>
>>>
>>> The current approach to avoiding problems due to stale binary packages
>>> with untracked ABI dependencies is to disable binpkg use during
>>> troublesome sections of the build (e.g. seed updates).  I think that a
>>> cleaner solution would be to use a configurable spec option
>>> blacklisting binpkgs for troublesome packages.  For example, in a
>>> stage1 with update_seed enabled, the Portage emerge (before the seed
>>> update) has:
>>
>> This needs to remain optional.
> 
> It is optional.  Just set `binpkg_blacklist` explicitly to an empty
> string if you don't want to blacklist the default packages (currently
> only two).
> 
>> I'm not going to nack a patch that some people may find useful but
>> in my personal opinion this is a terrible solution that should not
>> be used.  We need to find a way to rebuild packages as needed (like
>> EAPI 5) not force a rebuild everytime.
> 
> Agreed, but in the absence of one of the following:
> 
> * a tree full of EAPI-5+ packages that correctly use ABI sub-slotting
>   (if I live long enough to see this, I'll be very happy ;),
> * local overlays fixing ABI sub-slotting (maintained by folks without
>   much experience in the packages in question?), or
> * Portage tweaks to work around packages that don't use EAPI-5+ (or
>   that do, but don't use ABI sub-slots appropriately).
> 
> I'm fairly confident that none of those are happening in the next six
> months, and there seems to be agreement that catalyst needs some sort
> of local hack to work around the problem until one of the real
> solutions lands.
> 
> The `binpkg_blacklist` option lets you name (on a per-package level)
> troublesome packages that get ABI sub-slots wrong (or don't even try).
> I think that this approach will force *fewer* rebuilds than the
> current approach (from e7ea409 and 6c0a577) which blacklists
> everything from the seed update.
> 
> Another (non-exclusive) possibility is to put a big warning on the
> binpkgs option [1] and leave it up to folks to use at their own risk.

As I already stated, I have no issue allowing this into the code base as
one option people can use.  It may be a good idea to make this patch
depend on your other patchset which fixes up some of the documentation
and adding more detailed documentation....unless you already did that
and I'm not paying attention.

Thanks,
Rick
> 
> Cheers,
> Trevor
> 
> [1]: http://mid.gmane.org/cover.1366075786.git.wking@tremily.us
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJRcXGOAAoJEKXdFCfdEflKy8sP/jjNADHnnzt2vl8suWHt3LlL
pKuyKwjN9GDdhQJrzLZKYmQZNEFfT+GYUq+Z6zTpuR2fE7+d6SCJZDOOIpjO/cMh
MUMxn+E49rwMo3IPX/ZuaXAAH0OINM2NFg5paX1jorN+WRXmKkvAJQbbRuBAWSBo
OxKz7XsldEjELgCch5/yJH7as8mITFqcheXVMXLOS6xYKCbSHOpnP2s6+KMLlon5
Tw/DNviKvGOGQtLLd4ZWzqic0ElbQUmZIBNpWxODIbfmb2dH46/Xa2pybUOxqm+6
VVhrDxNbEaHNirNxTx3DrWyaeg6Dz/KSEoWHSOYy8DYgg+sIFWrC5RLHg21uWdhU
rGtGnj61hjjuKNsQlzyhFRH7Mxn8fOcktVFfbW1l2Mnvqr0zJbuKeyupt/vpyVHH
bTZL0PBNSPX9YrMQD3wrpW4UxUMBLQCfAfSmNRsb3vtXXjGnI2Q7bmLQnv86Vqau
J6QED7g+AvzqmLXqUU2RpYs3/H5yv1aJNH4yGNrr6vUvQOnoA3nRH5OF2hB+G+p8
JNWSu88IuMqk2h/+KJVGmSEZlBhfsjoW+yUGLncrdQ1nI4ir3IZxjDvXCdLNG31f
Ic8drzEfra9ls4MTCAvFFOY3MCEH16IBqNXodv56FNmA70qmHlXoamyOp5Lwiftv
e6jQsxZ078bxvcQKGAcB
=bjhe
-----END PGP SIGNATURE-----

Re: [gentoo-catalyst] [PATCH 0/2] Blacklisting binary packages

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 999 bytes --]

On Fri, Apr 19, 2013 at 12:32:14PM -0400, Rick "Zero_Chaos" Farina wrote:
> It may be a good idea to make this patch depend on your other
> patchset which fixes up some of the documentation and adding more
> detailed documentation....unless you already did that and I'm not
> paying attention.

That's a good idea.  I'd be happy to rebase either series on top of
the other to flesh out the documentation, but I think I'll wait until
after one of them lands in master.  This fix and the warning message
series seem fairly distinct to me, and having smaller patch series
seems easier to review.

If anyone out there is thinking “I would definately merge this stuff
if it was all in one big series instead of scattered about in topic
branches”, please tell me ;).  I can certainly do that if y'all
prefer.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[gentoo-catalyst] Re: [PATCH v3 0/2] pkgcache warning in catalyst-config(5)

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 881 bytes --]

On Mon, Apr 15, 2013 at 09:33:07PM -0400, W. Trevor King wrote:
> From: "W. Trevor King" <wking@tremily.us>
> 
> Changes since v2:
> * s/catalyst-config/catalyst-spec/ in catalyst-config.5.txt's “See
>   Also” section [1].
> * Assorted rewordings as suggested by Zero_Chaos [2] and acknowledged
>   in [3].
> 
> [1]: http://mid.gmane.org/20130412182752.GD13054@odin.tremily.us
> [2]: http://mid.gmane.org/516856A9.5030909@gentoo.org
> [3]: http://mid.gmane.org/20130412190524.GE13054@odin.tremily.us

This is also in origin/rewrite-on-master but not in origin/master.
It's in:

  git://tremily.us/catalyst.git pkgcache-warning

if anyone wants the patches in non-mailing-list form.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]