[gentoo-catalyst] proper way to deviate in system use flags?

[gentoo-catalyst] proper way to deviate in system use flags?

[Max Arnold]

Hello all!

I'm trying to build my own stage4 with specific packages and use flags.
My steps are: stage3-i686-2007.0 + recent snapshot -> stage1 -> stage2 -> stage3 -> stage4
Specs for stages 1-3 are generic (just copied from releng/specs/2007.0/x86), and stage4
adds portage_confdir, stage4/use and several packages.

Stage4 compiles fine, but when I chroot to it and issue emerge -pvuDN world, portage
wants to remerge several system packages due to changed use flags (for example ipv6, which
is disabled in my stage4/use).

So it is seems that my stage4/use and portage_confdir does not affect system packages
(I guess that catalyst does only --emptytree when emerging stage4).

So, there are my questions:

1. Am I correct in assumption that use flags are stacked during stage4 like this: 
profile -> stage4/use -> package.use (increasing priority from left to right)?
2. What is the proper (and simple to maintain) way to deviate slightly in system use flags?

My guesses:
1. Add 'hostuse' variable to earlier specs (stage3?)
2. Add 'portage_confdir' to earlier specs (but my -ipv6 listed in stage4/use and not in package.use)
3. Create new target (based on stage4 code) and add --newuse somewhere
4. Create my own profile (don't know where to put it and how to maintain during tree updates)

Any suggestions?
-- 
gentoo-catalyst@lists.gentoo.org mailing list

Re: [gentoo-catalyst] proper way to deviate in system use flags?

[Chris Gianelloni]

On Fri, 2008-04-18 at 11:29 +0800, Max Arnold wrote:
> I'm trying to build my own stage4 with specific packages and use flags.
> My steps are: stage3-i686-2007.0 + recent snapshot -> stage1 -> stage2 -> stage3 -> stage4

OK, start from an x86 stage3, though.

> Specs for stages 1-3 are generic (just copied from releng/specs/2007.0/x86), and stage4
> adds portage_confdir, stage4/use and several packages.
> 
> Stage4 compiles fine, but when I chroot to it and issue emerge -pvuDN world, portage
> wants to remerge several system packages due to changed use flags (for example ipv6, which
> is disabled in my stage4/use).
> 
> So it is seems that my stage4/use and portage_confdir does not affect system packages
> (I guess that catalyst does only --emptytree when emerging stage4).

Ehh, not quite.  The USE you set in your spec only affects the stage
build.  It doesn't affect what happens *after* the stage build.  It also
doesn't affect packages built before your current stage build.  One
thing that you did not mention is what version of catalyst that you're
using.  Newer versions of catalyst, like the currently masked
2.0.6_pre17, support writing out spec_prefix/use to make.conf so the
changes "stick" in your stage.

> So, there are my questions:
> 
> 1. Am I correct in assumption that use flags are stacked during stage4 like this: 
> profile -> stage4/use -> package.use (increasing priority from left to right)?

That's over-simplified, but yes.

> 2. What is the proper (and simple to maintain) way to deviate slightly in system use flags?

The way that you're doing it works fine if you're planning on shipping a
stage4 all the time.  Were I doing this myself, I'd make a new profile
with the desired changes, but that's just me.

> My guesses:
> 1. Add 'hostuse' variable to earlier specs (stage3?)

Already done in newer catalyst versions...

> 2. Add 'portage_confdir' to earlier specs (but my -ipv6 listed in stage4/use and not in package.use)

You should always use the same portage_confdir across all stages.  In
the future, catalyst will check for this and will refuse to run if you
didn't use the same snapshot and portage_confdir on your stages.  This
should hopefully reduce the number of "human errors" made when people
are using catalyst.  ;]

> 3. Create new target (based on stage4 code) and add --newuse somewhere

It wouldn't matter.  You'd need --newuse --deep.  We specifically have
not done this, as the stage4 code shares most of its code with the
livecd/stage1 and livecd-stage2 code, which was originally written with
the Minimal/Universal InstallCD in mind and doesn't require rebuilding
the system packages to match the CD's USE.

The best and most correct solution is to make your USE changes in a
profile and use that profile to build your stages.

> 4. Create my own profile (don't know where to put it and how to maintain during tree updates)

This is what you should do.  Simply stick it in $portdir/profiles where
you'd like it before running your snapshot.  To keep "emerge --sync"
from wiping it out, add
PORTAGE_RSYNC_EXTRA_OPTS="--exclude=profiles/myprofilename" to
your /etc/make.conf on your host/build system.

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Games Developer
-- 
gentoo-catalyst@lists.gentoo.org mailing list

Re: [gentoo-catalyst] proper way to deviate in system use flags?

[Max Arnold]

On Fri, Apr 18, 2008 at 03:12:08PM -0700, Chris Gianelloni wrote:
> > My steps are: stage3-i686-2007.0 + recent snapshot -> stage1 -> stage2 -> stage3 -> stage4
> 
> OK, start from an x86 stage3, though.

Is it critical?  I'm shipping only stage4 and it will be deployed on fixed 
hardware configuration.

> > So it is seems that my stage4/use and portage_confdir does not affect system packages
> > (I guess that catalyst does only --emptytree when emerging stage4).
> 
> Ehh, not quite.  The USE you set in your spec only affects the stage
> build.  It doesn't affect what happens *after* the stage build.  It also
> doesn't affect packages built before your current stage build.  One
> thing that you did not mention is what version of catalyst that you're
> using.  Newer versions of catalyst, like the currently masked
> 2.0.6_pre17, support writing out spec_prefix/use to make.conf so the
> changes "stick" in your stage.

I'm using catalyst-2.0.5, but at least on stage4 it puts my stage4/use into make.conf
and copies content of portage_confdir to /etc/portage.  I've just verified it in
resulting stage4 tarball.

> > So, there are my questions:
> > 
> > 1. Am I correct in assumption that use flags are stacked during stage4 like this: 
> > profile -> stage4/use -> package.use (increasing priority from left to right)?
> 
> That's over-simplified, but yes.

The reason I'm asking this is a phrase "If you set the portage configuration dir: package.use 
doesn't work how you think it should in catalyst. It's a known bug, but one that won't 
likely be fixed for some time. Basically, if you use $target/use, then it will ignore 
package.use settings." here: http://gentoo-wiki.com/HOWTO_build_a_LiveCD
Is that already fixed?

> > 2. What is the proper (and simple to maintain) way to deviate slightly in system use flags?
> 
> The way that you're doing it works fine if you're planning on shipping a
> stage4 all the time.  Were I doing this myself, I'd make a new profile
> with the desired changes, but that's just me.
> 
> > My guesses:
> > 1. Add 'hostuse' variable to earlier specs (stage3?)
> 
> Already done in newer catalyst versions...

2.0.5 uses it: generic_stage_target.py adds hostuse to valid_values list, and
stage4_chroot.sh mentions $clst_HOSTUSE.   But is it safe to simply define hostuse
in stage spec?  Is it additive (I see x86.py assigns some values to this variable) or
arch specific value will be simply overridden and it is better to do not redefine it?

> > 2. Add 'portage_confdir' to earlier specs (but my -ipv6 listed in stage4/use and not in package.use)
> 
> You should always use the same portage_confdir across all stages.  In
> the future, catalyst will check for this and will refuse to run if you
> didn't use the same snapshot and portage_confdir on your stages.  This
> should hopefully reduce the number of "human errors" made when people
> are using catalyst.  ;]
 
> > 3. Create new target (based on stage4 code) and add --newuse somewhere
> 
> It wouldn't matter.  You'd need --newuse --deep.  We specifically have
> not done this, as the stage4 code shares most of its code with the
> livecd/stage1 and livecd-stage2 code, which was originally written with
> the Minimal/Universal InstallCD in mind and doesn't require rebuilding
> the system packages to match the CD's USE.
> 
> The best and most correct solution is to make your USE changes in a
> profile and use that profile to build your stages.
> 
> > 4. Create my own profile (don't know where to put it and how to maintain during tree updates)
> 
> This is what you should do.  Simply stick it in $portdir/profiles where
> you'd like it before running your snapshot.  To keep "emerge --sync"
> from wiping it out, add
> PORTAGE_RSYNC_EXTRA_OPTS="--exclude=profiles/myprofilename" to
> your /etc/make.conf on your host/build system.

Can be custom profile added to portage_overlay on all stages istead of putting it to tree?
BTW, what is the difference between profiles/default/linux and profiles/default-linux ?

So for now I'll try to upgrade catalyst, add portage_confdir to all stages, leave only
stage4/use (for additional world packages) and experiment with profiles.

I also have some more general questions, but I'll ask them in another thread.

Thank you!
-- 
gentoo-catalyst@lists.gentoo.org mailing list

Re: [gentoo-catalyst] proper way to deviate in system use flags?

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 4674 bytes --]

On Sat, 2008-04-19 at 11:19 +0800, Max Arnold wrote:
> On Fri, Apr 18, 2008 at 03:12:08PM -0700, Chris Gianelloni wrote:
> > > My steps are: stage3-i686-2007.0 + recent snapshot -> stage1 -> stage2 -> stage3 -> stage4
> > 
> > OK, start from an x86 stage3, though.
> 
> Is it critical?  I'm shipping only stage4 and it will be deployed on fixed 
> hardware configuration.

If you're building a stage1, you should always use a generic stage3 as
your seed.  It likely isn't critical for you, but it's good practice to
get used to doing.

> > > So it is seems that my stage4/use and portage_confdir does not affect system packages
> > > (I guess that catalyst does only --emptytree when emerging stage4).
> > 
> > Ehh, not quite.  The USE you set in your spec only affects the stage
> > build.  It doesn't affect what happens *after* the stage build.  It also
> > doesn't affect packages built before your current stage build.  One
> > thing that you did not mention is what version of catalyst that you're
> > using.  Newer versions of catalyst, like the currently masked
> > 2.0.6_pre17, support writing out spec_prefix/use to make.conf so the
> > changes "stick" in your stage.
> 
> I'm using catalyst-2.0.5, but at least on stage4 it puts my stage4/use into make.conf
> and copies content of portage_confdir to /etc/portage.  I've just verified it in
> resulting stage4 tarball.

Sure, but none of your earlier stages did this correctly and even the
stage4 target doesn't get it quite right.  Upgrade your catalyst
version.

> > > So, there are my questions:
> > > 
> > > 1. Am I correct in assumption that use flags are stacked during stage4 like this: 
> > > profile -> stage4/use -> package.use (increasing priority from left to right)?
> > 
> > That's over-simplified, but yes.
> 
> The reason I'm asking this is a phrase "If you set the portage configuration dir: package.use 
> doesn't work how you think it should in catalyst. It's a known bug, but one that won't 
> likely be fixed for some time. Basically, if you use $target/use, then it will ignore 
> package.use settings." here: http://gentoo-wiki.com/HOWTO_build_a_LiveCD
> Is that already fixed?

It should be *only* in newer catalyst, not the current stable.

> > > 2. What is the proper (and simple to maintain) way to deviate slightly in system use flags?
> > 
> > The way that you're doing it works fine if you're planning on shipping a
> > stage4 all the time.  Were I doing this myself, I'd make a new profile
> > with the desired changes, but that's just me.
> > 
> > > My guesses:
> > > 1. Add 'hostuse' variable to earlier specs (stage3?)
> > 
> > Already done in newer catalyst versions...
> 
> 2.0.5 uses it: generic_stage_target.py adds hostuse to valid_values list, and
> stage4_chroot.sh mentions $clst_HOSTUSE.   But is it safe to simply define hostuse
> in stage spec?  Is it additive (I see x86.py assigns some values to this variable) or
> arch specific value will be simply overridden and it is better to do not redefine it?

You cannot redefine it.  It is defined exactly once.  HOSTUSE is
literally *only* for subarchitecture-specific USE which controls CPU
capabilities/features.  It is only designed to be used internally, so if
you try to do anything with it, I cannot guarantee what will happen.
> > > 4. Create my own profile (don't know where to put it and how to maintain during tree updates)
> > 
> > This is what you should do.  Simply stick it in $portdir/profiles where
> > you'd like it before running your snapshot.  To keep "emerge --sync"
> > from wiping it out, add
> > PORTAGE_RSYNC_EXTRA_OPTS="--exclude=profiles/myprofilename" to
> > your /etc/make.conf on your host/build system.
> 
> Can be custom profile added to portage_overlay on all stages istead of putting it to tree?

Yes, it can, but it is safer to put it into your tree that you're using
for your snapshots.  I suggest keeping a separate portage tree from
"/usr/portage" for this, so you don't have to worry about accidentally
hosing it.

> BTW, what is the difference between profiles/default/linux and profiles/default-linux ?

Well, profiles/default/linux is the new location and default-linux will
be going away, so that makes it rather simple.  ;]

> So for now I'll try to upgrade catalyst, add portage_confdir to all stages, leave only
> stage4/use (for additional world packages) and experiment with profiles.

Sounds great!

> I also have some more general questions, but I'll ask them in another thread.

OK.

> Thank you!
-- 
Chris Gianelloni
Release Engineering Strategic Lead
Games Developer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]