[gentoo-dev] ironing out release tarballs

[gentoo-dev] ironing out release tarballs

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1607 bytes --]

background:
everyone wants @system to be slim, but most people want the initial stage
tarball that we release and you install Gentoo from to not be completely
sparse.  we've got a bug for this topic:
https://bugs.gentoo.org/393445

items to sort out:
- should the list of packages be in catalyst or profile-stacked content
-> imo it should be entirely in the profile

- should the packages list be in a new packages.default, or should we create a
  new set to hold it, or should we just go with @profile ?
-> @profile has the advantage of already existing.  we have to be careful so as
   to make it difficult to uninstall packages that the user does not actually
   want.

- if the packages aren't in @profile, should they be seeded in @world ?
-> imo yes as  we don't want all the default packages getting depcleaned as
   soon as you start using the new install.  if they're in @profile, then this
   is a moot point (assuming depclean does not clean out @profile).

- should stage3 be @system only, or @system+@profile, or 
  @system+@profile+packages.default ?
-> this depends on the previous discussion a bit.  today, stage3's are 
   @system, but imo @system+@profile is reasonable.  see next question too.

- should we release stage4's instead of stage3's ?
-> if we keep stage3 as @system-only, then we'd build stage4's which would add
   @profile/whatever
-> downside is that we've been training the world to download & install stage3
   for almost 15 years
-> imo as long as the default @profile is kept slim, adjusting the definition of
   a stage3 is OK
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] ironing out release tarballs

[Alexis Ballier]

On Thu, 15 Oct 2015 11:34:22 -0400
Mike Frysinger <vapier@gentoo.org> wrote:

> background:
> everyone wants @system to be slim, but most people want the initial
> stage tarball that we release and you install Gentoo from to not be
> completely sparse.  we've got a bug for this topic:
> https://bugs.gentoo.org/393445
> 
> items to sort out:
> - should the list of packages be in catalyst or profile-stacked
> content -> imo it should be entirely in the profile  

fully agree

> - should the packages list be in a new packages.default, or should we
> create a new set to hold it, or should we just go with @profile ?
> -> @profile has the advantage of already existing.  we have to be
> careful so as to make it difficult to uninstall packages that the
> user does not actually want.
> 
> - if the packages aren't in @profile, should they be seeded in
> @world ? -> imo yes as  we don't want all the default packages
> getting depcleaned as soon as you start using the new install.  if
> they're in @profile, then this is a moot point (assuming depclean
> does not clean out @profile).

some kind of 'world' file in profiles like the 'packages' one that is
just used to populate world file after (or just before) stage3 build ?

not sure if sets provide the same flexibility: i can imagine iputils in
that set, but also another embedded profile with
busybox[make-symlinks], or the bsds

> - should stage3 be @system only, or @system+@profile, or 
>   @system+@profile+packages.default ?
> -> this depends on the previous discussion a bit.  today, stage3's
> are @system, but imo @system+@profile is reasonable.  see next
> question too.
> 
> - should we release stage4's instead of stage3's ?
> -> if we keep stage3 as @system-only, then we'd build stage4's which
> would add @profile/whatever
> -> downside is that we've been training the world to download &
> install stage3 for almost 15 years
> -> imo as long as the default @profile is kept slim, adjusting the
> definition of a stage3 is OK

i also think it's better to adjust stage3 if it is kept relatively slim,
but i'm pretty sure there'll be demand for @system only stages

Alexis.

Re: [gentoo-dev] ironing out release tarballs

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1629 bytes --]

On 15 Oct 2015 19:01, Alexis Ballier wrote:
> On Thu, 15 Oct 2015 11:34:22 -0400 Mike Frysinger wrote:
> > - should the packages list be in a new packages.default, or should we
> > create a new set to hold it, or should we just go with @profile ?
> > -> @profile has the advantage of already existing.  we have to be
> > careful so as to make it difficult to uninstall packages that the
> > user does not actually want.
> > 
> > - if the packages aren't in @profile, should they be seeded in
> > @world ? -> imo yes as  we don't want all the default packages
> > getting depcleaned as soon as you start using the new install.  if
> > they're in @profile, then this is a moot point (assuming depclean
> > does not clean out @profile).
> 
> some kind of 'world' file in profiles like the 'packages' one that is
> just used to populate world file after (or just before) stage3 build ?

i suggested the name "packages.default" originally to convey the notion
that it's just the default set of packages (you'd find in a release).
keeping the "packages." prefix seemed to be better namespace wise.

doesn't require a PMS update because only releng tools (catalyst) would
read it.  set integration would have to conform to PMS.

> not sure if sets provide the same flexibility: i can imagine iputils in
> that set, but also another embedded profile with
> busybox[make-symlinks], or the bsds

i'm not sure putting USE flag qualifiers makes sense as the next time the
package updates, the USE flags will change.  if profiles want to default
USE flags, the existing package.use makes more sense imo.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] ironing out release tarballs

[Rich Freeman]

On Thu, Oct 15, 2015 at 11:34 AM, Mike Frysinger <vapier@gentoo.org> wrote:
>
> items to sort out:
> - should the list of packages be in catalyst or profile-stacked content
> -> imo it should be entirely in the profile

++

This would be really nice to combine with mix-ins so that instead of
special cases we could just use additional profiles (without the
normal cost of additional profiles), but absent that the approach you
have below seems fine.

>
> - should the packages list be in a new packages.default, or should we create a
>   new set to hold it, or should we just go with @profile ?
> -> @profile has the advantage of already existing.  we have to be careful so as
>    to make it difficult to uninstall packages that the user does not actually
>    want.

I would be interested in use cases for @profile OTHER than convenience
packages (sshd, screen, etc).

Again, this is a case where having more profiles would get rid of the
need to have a compromise.  Just make it @profile, and be sure to have
a profile available that doesn't have any packages beyond @system.

However, if some profiles really do need to install fairly critical
packages then maybe we should also have a packages.default in addition
to @profile.

>
> - if the packages aren't in @profile, should they be seeded in @world ?
> -> imo yes as  we don't want all the default packages getting depcleaned as
>    soon as you start using the new install.  if they're in @profile, then this
>    is a moot point (assuming depclean does not clean out @profile).

If we end up with @system+@profile+packages.default then I'd say that
packages.default gets seeded in @world.  @profile becomes difficult to
uninstall.  This should be the solution if some profiles really do
need to add essential packages as well as convenience packages, but
the essential packages aren't assumed dependencies.

If we end up with just @system+@profile then I'd say that packages in
@profile get seeded in @world, and thus nothing special needs to be
done to uninstall them.  This should be done if there aren't essential
profile packages.

>
> - should stage3 be @system only, or @system+@profile, or
>   @system+@profile+packages.default ?
> -> this depends on the previous discussion a bit.  today, stage3's are
>    @system, but imo @system+@profile is reasonable.  see next question too.

Agree it depends on the previous outcome.

>
> - should we release stage4's instead of stage3's ?
> -> if we keep stage3 as @system-only, then we'd build stage4's which would add
>    @profile/whatever
> -> downside is that we've been training the world to download & install stage3
>    for almost 15 years
> -> imo as long as the default @profile is kept slim, adjusting the definition of
>    a stage3 is OK

I don't have a strong opinion on this.  I don't see the need to
maintain separate stage3s in addition to the stage4s, so we're just
arguing semantics.

I think the real question I have is what would the @profile set be
used for OTHER than convenience packages?  While I did mention mix-ins
as being a better long-term solution I'm not suggesting that we should
hold off on a reasonable interim solution until we work that out.
Without mix-in support we don't really want to add more profiles,
which is the other way to go with this.

-- 
Rich

Re: [gentoo-dev] ironing out release tarballs

[Zac Medico]

On 10/15/2015 08:34 AM, Mike Frysinger wrote:
> background:
> everyone wants @system to be slim, but most people want the initial stage
> tarball that we release and you install Gentoo from to not be completely
> sparse.  we've got a bug for this topic:
> https://bugs.gentoo.org/393445
> 
> items to sort out:
> - should the list of packages be in catalyst or profile-stacked content
> -> imo it should be entirely in the profile
> 
> - should the packages list be in a new packages.default, or should we create a
>   new set to hold it, or should we just go with @profile ?
> -> @profile has the advantage of already existing.  we have to be careful so as
>    to make it difficult to uninstall packages that the user does not actually
>    want.

In portage, the current meaning of @profile is very similar to @system,
except that it implies that members specify dependencies completely
(allowing for optimal parallelization) [1]. The @profile set is only
enabled for profiles from repositories that have "profile-formats =
profile-set" set in metadata/layout.conf. It's an extension which is not
covered by PMS.

> - if the packages aren't in @profile, should they be seeded in @world ?
> -> imo yes as  we don't want all the default packages getting depcleaned as
>    soon as you start using the new install.  if they're in @profile, then this
>    is a moot point (assuming depclean does not clean out @profile).

In portage, @world = @profile + @selected + @system, which means that
@profile is protected from depclean since it's a part of @world.

[1] https://bugs.gentoo.org/show_bug.cgi?id=532224
-- 
Thanks,
Zac

Re: [gentoo-dev] ironing out release tarballs

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 15/10/15 03:15 PM, Zac Medico wrote:
> On 10/15/2015 08:34 AM, Mike Frysinger wrote:
>> background: everyone wants @system to be slim, but most people
>> want the initial stage tarball that we release and you install
>> Gentoo from to not be completely sparse.  we've got a bug for
>> this topic: https://bugs.gentoo.org/393445
>> 
>> items to sort out: - should the list of packages be in catalyst
>> or profile-stacked content -> imo it should be entirely in the
>> profile
>> 
>> - should the packages list be in a new packages.default, or
>> should we create a new set to hold it, or should we just go
>> with @profile ? -> @profile has the advantage of already
>> existing.  we have to be careful so as to make it difficult to
>> uninstall packages that the user does not actually want.
> 
> In portage, the current meaning of @profile is very similar to
> @system, except that it implies that members specify dependencies
> completely (allowing for optimal parallelization) [1]. The
> @profile set is only enabled for profiles from repositories that
> have "profile-formats = profile-set" set in metadata/layout.conf.
> It's an extension which is not covered by PMS.
> 
>> - if the packages aren't in @profile, should they be seeded in
>> @world ? -> imo yes as  we don't want all the default packages
>> getting depcleaned as soon as you start using the new install.
>> if they're in @profile, then this is a moot point (assuming
>> depclean does not clean out @profile).
> 
> In portage, @world = @profile + @selected + @system, which means
> that @profile is protected from depclean since it's a part of
> @world.
> 
> [1] https://bugs.gentoo.org/show_bug.cgi?id=532224
> 


So just to clarify..  if we start adding these packages that are
removed from @system into @profile, what do we gain here?  They'll
exist in the stage3's (which is one of the goals right?), and
they'll be included in @world without entries in
/var/lib/portage/world (so end-users will have them on their
systems)..  Seems like everything would be the same as if they were
in @system, except 'emerge -e @system' wouldn't rebuild them..? Do
we get the advantage(s) we were looking for, going this route?


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iF4EAREIAAYFAlYf/rUACgkQAJxUfCtlWe3dWQEAlo+oBK+uyzRf+fzF2o17skMS
0438JShMlObzWOkgZYYA/R65hZUl7enVItRWvzqPSP0qfKLjmXjCWcJiuepBBoRl
=1fJ6
-----END PGP SIGNATURE-----

Re: [gentoo-dev] ironing out release tarballs

[Anthony G. Basile]

For some reason I didn't receive the original email from Mike.  I'll 
answer via Rich's email. Hopefully I won't be misinterpreting anything.

On 10/15/15 1:43 PM, Rich Freeman wrote:
> On Thu, Oct 15, 2015 at 11:34 AM, Mike Frysinger <vapier@gentoo.org> wrote:
>> items to sort out:
>> - should the list of packages be in catalyst or profile-stacked content
>> -> imo it should be entirely in the profile
> ++
>
> This would be really nice to combine with mix-ins so that instead of
> special cases we could just use additional profiles (without the
> normal cost of additional profiles), but absent that the approach you
> have below seems fine.

I assume you're talking about the list of packages in each stage.  I 
would like them entirely in the profile.  This gives the profile 
maintainer control and I need that for some of the more exotic stuff I 
work on.
>
>> - should the packages list be in a new packages.default, or should we create a
>>    new set to hold it, or should we just go with @profile ?
>> -> @profile has the advantage of already existing.  we have to be careful so as
>>     to make it difficult to uninstall packages that the user does not actually
>>     want.
> I would be interested in use cases for @profile OTHER than convenience
> packages (sshd, screen, etc).
>
> Again, this is a case where having more profiles would get rid of the
> need to have a compromise.  Just make it @profile, and be sure to have
> a profile available that doesn't have any packages beyond @system.
>
> However, if some profiles really do need to install fairly critical
> packages then maybe we should also have a packages.default in addition
> to @profile.

Why would we need a new packages.default or @newset?  @profile should be 
enough.  I guess I'm not sure how packages.default would work 
differently than @profile?  What would it add?

>
>> - if the packages aren't in @profile, should they be seeded in @world ?
>> -> imo yes as  we don't want all the default packages getting depcleaned as
>>     soon as you start using the new install.  if they're in @profile, then this
>>     is a moot point (assuming depclean does not clean out @profile).
> If we end up with @system+@profile+packages.default then I'd say that
> packages.default gets seeded in @world.  @profile becomes difficult to
> uninstall.  This should be the solution if some profiles really do
> need to add essential packages as well as convenience packages, but
> the essential packages aren't assumed dependencies.
>
> If we end up with just @system+@profile then I'd say that packages in
> @profile get seeded in @world, and thus nothing special needs to be
> done to uninstall them.  This should be done if there aren't essential
> profile packages.

i'm happy with having them in @profile and making sure that depclean 
doesn't clear @profile.

>
>> - should stage3 be @system only, or @system+@profile, or
>>    @system+@profile+packages.default ?
>> -> this depends on the previous discussion a bit.  today, stage3's are
>>     @system, but imo @system+@profile is reasonable.  see next question too.
> Agree it depends on the previous outcome.

Answer to this and next.  stage3 should be @system+@profile (again I'm 
sure what packages.default would add).   I see little value in a stage3 
which is @system only followed by a stage4 which is @system+@profile.
>
>> - should we release stage4's instead of stage3's ?
>> -> if we keep stage3 as @system-only, then we'd build stage4's which would add
>>     @profile/whatever
>> -> downside is that we've been training the world to download & install stage3
>>     for almost 15 years
>> -> imo as long as the default @profile is kept slim, adjusting the definition of
>>     a stage3 is OK

I'm in agreement with your last statement, let's just adjust the 
definition of stage3 and keep @profile slim.  Its a lot of work to fix 
up our documentation to read stage4 instead of stage3 with little gain 
in my opinion.  And users could be confused.

If we really needed a stage with just @system, we could add some 
catalyst flag that produced a stage2.5 instead of a stage3.  So a 
typical run could be stage1 -> stage2 -> stage3 (where stage3 means 
@system+@profile) and optionally stage1 -> stage2 -> stage2.5 -> stage3.

> I don't have a strong opinion on this.  I don't see the need to
> maintain separate stage3s in addition to the stage4s, so we're just
> arguing semantics.
>
> I think the real question I have is what would the @profile set be
> used for OTHER than convenience packages?  While I did mention mix-ins
> as being a better long-term solution I'm not suggesting that we should
> hold off on a reasonable interim solution until we work that out.
> Without mix-in support we don't really want to add more profiles,
> which is the other way to go with this.
>


-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] ironing out release tarballs

[Anthony G. Basile]

On 10/15/15 3:15 PM, Zac Medico wrote:
>
> In portage, @world = @profile + @selected + @system, which means that
> @profile is protected from depclean since it's a part of @world.
>
> [1] https://bugs.gentoo.org/show_bug.cgi?id=532224

I thought so but wasn't sure and was about to test.  Both @system and 
@profile are controlled via the packages file in portage and stack.  
Packages in @system lead with an * while @profile don't. @system 
packages have incomplete dependency specifications while @profile have 
full.  This affects more than just emerge's ability to parallelize, no?

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] ironing out release tarballs

[Zac Medico]

On 10/15/2015 12:29 PM, Ian Stakenvicius wrote:
> On 15/10/15 03:15 PM, Zac Medico wrote:
>> On 10/15/2015 08:34 AM, Mike Frysinger wrote:
>>> background: everyone wants @system to be slim, but most people
>>> want the initial stage tarball that we release and you install
>>> Gentoo from to not be completely sparse.  we've got a bug for
>>> this topic: https://bugs.gentoo.org/393445
>>>
>>> items to sort out: - should the list of packages be in catalyst
>>> or profile-stacked content -> imo it should be entirely in the
>>> profile
>>>
>>> - should the packages list be in a new packages.default, or
>>> should we create a new set to hold it, or should we just go
>>> with @profile ? -> @profile has the advantage of already
>>> existing.  we have to be careful so as to make it difficult to
>>> uninstall packages that the user does not actually want.
> 
>> In portage, the current meaning of @profile is very similar to
>> @system, except that it implies that members specify dependencies
>> completely (allowing for optimal parallelization) [1]. The
>> @profile set is only enabled for profiles from repositories that
>> have "profile-formats = profile-set" set in metadata/layout.conf.
>> It's an extension which is not covered by PMS.
> 
>>> - if the packages aren't in @profile, should they be seeded in
>>> @world ? -> imo yes as  we don't want all the default packages
>>> getting depcleaned as soon as you start using the new install.
>>> if they're in @profile, then this is a moot point (assuming
>>> depclean does not clean out @profile).
> 
>> In portage, @world = @profile + @selected + @system, which means
>> that @profile is protected from depclean since it's a part of
>> @world.
> 
>> [1] https://bugs.gentoo.org/show_bug.cgi?id=532224
> 
> 
> 
> So just to clarify..  if we start adding these packages that are
> removed from @system into @profile, what do we gain here?  They'll
> exist in the stage3's (which is one of the goals right?), and
> they'll be included in @world without entries in
> /var/lib/portage/world (so end-users will have them on their
> systems)..  Seems like everything would be the same as if they were
> in @system, except 'emerge -e @system' wouldn't rebuild them..? Do
> we get the advantage(s) we were looking for, going this route?


What's probably desired is to create a stage3 profile which adds
whatever extra stuff you want to @system, and to use the stage3 profile
for to build stage3. After the stage3 is built, catalyst could set some
other profile if we don't want users to have the stage3 profile by default.
-- 
Thanks,
Zac

Re: [gentoo-dev] ironing out release tarballs

[Zac Medico]

On 10/15/2015 01:06 PM, Anthony G. Basile wrote:
> On 10/15/15 3:15 PM, Zac Medico wrote:
>>
>> In portage, @world = @profile + @selected + @system, which means that
>> @profile is protected from depclean since it's a part of @world.
>>
>> [1] https://bugs.gentoo.org/show_bug.cgi?id=532224
> 
> I thought so but wasn't sure and was about to test.  Both @system and
> @profile are controlled via the packages file in portage and stack. 
> Packages in @system lead with an * while @profile don't. @system
> packages have incomplete dependency specifications while @profile have
> full.

Right.

> This affects more than just emerge's ability to parallelize, no?

Having complete dependency specifications could be useful for some other
things, but allowing for more aggressive parallelization is one of the
most obvious advantage.
-- 
Thanks,
Zac

Re: [gentoo-dev] ironing out release tarballs

[Anthony G. Basile]

On 10/15/15 4:14 PM, Zac Medico wrote:
> On 10/15/2015 01:06 PM, Anthony G. Basile wrote:
>> On 10/15/15 3:15 PM, Zac Medico wrote:
>>> In portage, @world = @profile + @selected + @system, which means that
>>> @profile is protected from depclean since it's a part of @world.
>>>
>>> [1] https://bugs.gentoo.org/show_bug.cgi?id=532224
>> I thought so but wasn't sure and was about to test.  Both @system and
>> @profile are controlled via the packages file in portage and stack.
>> Packages in @system lead with an * while @profile don't. @system
>> packages have incomplete dependency specifications while @profile have
>> full.
> Right.
>
>> This affects more than just emerge's ability to parallelize, no?
> Having complete dependency specifications could be useful for some other
> things, but allowing for more aggressive parallelization is one of the
> most obvious advantage.

Okay, good because that fits my understanding of how we'd be dividing 
@system from @profile.

@system = the bare set that we need in an stage3 in order to build 
another stage3 via the catalyst process.  so the way this works is that 
you unpack a stage3, chroot into it, and then do a `ROOT=/some/new/root 
emerge @system` to prepare a pristine new root.  that root then seeds 
your stage2 at which point your rebuild your toolchain.  then that seeds 
your new stage3 in which you rebuild @system.

So that defines @system from the point of view of using a current stage3 
to give birth to a next generation stage3.  But that may not be what you 
want to release, eg. do you need any networking stuff in there?  This 
gave birth to the idea of a stage4 which would have the added goodies 
needed for an end user to grow a system from our release tarball.  
vapier is suggesting using @profile for the extra needed beyond @system 
for the release.  So at all points except the very end, you just use 
@system for building because that's all you need, and then finally you 
produce an @system+@profile for release.

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] ironing out release tarballs

[Rich Freeman]

On Thu, Oct 15, 2015 at 4:10 PM, Zac Medico <zmedico@gentoo.org> wrote:
> What's probably desired is to create a stage3 profile which adds
> whatever extra stuff you want to @system, and to use the stage3 profile
> for to build stage3. After the stage3 is built, catalyst could set some
> other profile if we don't want users to have the stage3 profile by default.

Unless we seed @selected with the extra stuff, it will get removed the
first time the user runs --depclean, which probably isn't what we
want.

I'm fine with going the multi-profile route, but it seems to me that
we still need some kind of initial @selected set.

Basically the goal here is to give the user a bunch of useful stuff by
default (specified in some way in the profile), but make it easy for
the user to remove it if they wish.

A side goal is to do this without adding 14 more profiles, because we
don't have mixins.

-- 
Rich

Re: [gentoo-dev] ironing out release tarballs

[Zac Medico]

On 10/15/2015 01:29 PM, Anthony G. Basile wrote:
> On 10/15/15 4:14 PM, Zac Medico wrote:
>> On 10/15/2015 01:06 PM, Anthony G. Basile wrote:
>>> On 10/15/15 3:15 PM, Zac Medico wrote:
>>>> In portage, @world = @profile + @selected + @system, which means that
>>>> @profile is protected from depclean since it's a part of @world.
>>>>
>>>> [1] https://bugs.gentoo.org/show_bug.cgi?id=532224
>>> I thought so but wasn't sure and was about to test.  Both @system and
>>> @profile are controlled via the packages file in portage and stack.
>>> Packages in @system lead with an * while @profile don't. @system
>>> packages have incomplete dependency specifications while @profile have
>>> full.
>> Right.
>>
>>> This affects more than just emerge's ability to parallelize, no?
>> Having complete dependency specifications could be useful for some other
>> things, but allowing for more aggressive parallelization is one of the
>> most obvious advantage.
> 
> Okay, good because that fits my understanding of how we'd be dividing
> @system from @profile.
> 
> @system = the bare set that we need in an stage3 in order to build
> another stage3 via the catalyst process.  so the way this works is that
> you unpack a stage3, chroot into it, and then do a `ROOT=/some/new/root
> emerge @system` to prepare a pristine new root.  that root then seeds
> your stage2 at which point your rebuild your toolchain.  then that seeds
> your new stage3 in which you rebuild @system.
> 
> So that defines @system from the point of view of using a current stage3
> to give birth to a next generation stage3.  But that may not be what you
> want to release, eg. do you need any networking stuff in there?  This
> gave birth to the idea of a stage4 which would have the added goodies
> needed for an end user to grow a system from our release tarball. 
> vapier is suggesting using @profile for the extra needed beyond @system
> for the release.  So at all points except the very end, you just use
> @system for building because that's all you need, and then finally you
> produce an @system+@profile for release.
> 

What you're talking about essentially results in a @world set which
varies depending on the context. I'm not so sure that's a good idea. I'm
inclined to suggest that you just use a different profile for each
context (like the stage3 profile that I suggested in my reply to Ian).
-- 
Thanks,
Zac

Re: [gentoo-dev] ironing out release tarballs

[Zac Medico]

On 10/15/2015 01:45 PM, Rich Freeman wrote:
> On Thu, Oct 15, 2015 at 4:10 PM, Zac Medico <zmedico@gentoo.org> wrote:
>> What's probably desired is to create a stage3 profile which adds
>> whatever extra stuff you want to @system, and to use the stage3 profile
>> for to build stage3. After the stage3 is built, catalyst could set some
>> other profile if we don't want users to have the stage3 profile by default.
> 
> Unless we seed @selected with the extra stuff, it will get removed the
> first time the user runs --depclean, which probably isn't what we
> want.
> 
> I'm fine with going the multi-profile route, but it seems to me that
> we still need some kind of initial @selected set.
> 
> Basically the goal here is to give the user a bunch of useful stuff by
> default (specified in some way in the profile), but make it easy for
> the user to remove it if they wish.
> 
> A side goal is to do this without adding 14 more profiles, because we
> don't have mixins.
> 

Given the goals, having catalyst seed /var/lib/portage/world seems
pretty reasonable to me.
-- 
Thanks,
Zac

Re: [gentoo-dev] ironing out release tarballs

[Rich Freeman]

On Thu, Oct 15, 2015 at 4:57 PM, Zac Medico <zmedico@gentoo.org> wrote:
> Given the goals, having catalyst seed /var/lib/portage/world seems
> pretty reasonable to me.

Then the question becomes how.  Does it diff @profile between the two
profiles and put the extra stuff in @selected?  Or, does the profile
just contain a special file containing the stuff that gets seeded?
That is really the gist of the two approaches, and if you just have a
special file full of stuff that gets seeded you really don't need
another profile, which is nice since profiles are a PITA right now.

-- 
Rich

Re: [gentoo-dev] ironing out release tarballs

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 2101 bytes --]

On 15 Oct 2015 12:15, Zac Medico wrote:
> On 10/15/2015 08:34 AM, Mike Frysinger wrote:
> > background:
> > everyone wants @system to be slim, but most people want the initial stage
> > tarball that we release and you install Gentoo from to not be completely
> > sparse.  we've got a bug for this topic:
> > https://bugs.gentoo.org/393445
> > 
> > items to sort out:
> > - should the list of packages be in catalyst or profile-stacked content
> > -> imo it should be entirely in the profile
> > 
> > - should the packages list be in a new packages.default, or should we create a
> >   new set to hold it, or should we just go with @profile ?
> > -> @profile has the advantage of already existing.  we have to be careful so as
> >    to make it difficult to uninstall packages that the user does not actually
> >    want.
> 
> In portage, the current meaning of @profile is very similar to @system,
> except that it implies that members specify dependencies completely
> (allowing for optimal parallelization) [1]. The @profile set is only
> enabled for profiles from repositories that have "profile-formats =
> profile-set" set in metadata/layout.conf. It's an extension which is not
> covered by PMS.

layout.conf isn't covered by PMS, nor are sets, and packages file format is
compatible with PMS.  so i think we are OK here.

> > - if the packages aren't in @profile, should they be seeded in @world ?
> > -> imo yes as  we don't want all the default packages getting depcleaned as
> >    soon as you start using the new install.  if they're in @profile, then this
> >    is a moot point (assuming depclean does not clean out @profile).
> 
> In portage, @world = @profile + @selected + @system, which means that
> @profile is protected from depclean since it's a part of @world.

so if iputils is in @profile, and i do:
	emerge -C iputils
i don't get the ugly warning about it being in @system, but if i do:
	emerge @world
iputils comes back.  i think that's OK actually since people can do:
	emerge @selected
which has the classic @world meaning.
-mike

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] ironing out release tarballs

[Zac Medico]

On 10/15/2015 02:03 PM, Rich Freeman wrote:
> On Thu, Oct 15, 2015 at 4:57 PM, Zac Medico <zmedico@gentoo.org> wrote:
>> Given the goals, having catalyst seed /var/lib/portage/world seems
>> pretty reasonable to me.
> 
> Then the question becomes how.  Does it diff @profile between the two
> profiles and put the extra stuff in @selected?  Or, does the profile
> just contain a special file containing the stuff that gets seeded?
> That is really the gist of the two approaches, and if you just have a
> special file full of stuff that gets seeded you really don't need
> another profile, which is nice since profiles are a PITA right now.
> 

We already have packages.build which is used to build stage1, so
introducing a packages.stage3 that's used to seed @selected (aka
/var/lib/portage/world) for stage3 seems reasonable.
-- 
Thanks,
Zac

Re: [gentoo-dev] ironing out release tarballs

[Zac Medico]

On 10/15/2015 02:13 PM, Mike Frysinger wrote:
> On 15 Oct 2015 12:15, Zac Medico wrote:
>> On 10/15/2015 08:34 AM, Mike Frysinger wrote:
>>> background:
>>> everyone wants @system to be slim, but most people want the initial stage
>>> tarball that we release and you install Gentoo from to not be completely
>>> sparse.  we've got a bug for this topic:
>>> https://bugs.gentoo.org/393445
>>>
>>> items to sort out:
>>> - should the list of packages be in catalyst or profile-stacked content
>>> -> imo it should be entirely in the profile
>>>
>>> - should the packages list be in a new packages.default, or should we create a
>>>   new set to hold it, or should we just go with @profile ?
>>> -> @profile has the advantage of already existing.  we have to be careful so as
>>>    to make it difficult to uninstall packages that the user does not actually
>>>    want.
>>
>> In portage, the current meaning of @profile is very similar to @system,
>> except that it implies that members specify dependencies completely
>> (allowing for optimal parallelization) [1]. The @profile set is only
>> enabled for profiles from repositories that have "profile-formats =
>> profile-set" set in metadata/layout.conf. It's an extension which is not
>> covered by PMS.
> 
> layout.conf isn't covered by PMS, nor are sets, and packages file format is
> compatible with PMS.  so i think we are OK here.
> 
>>> - if the packages aren't in @profile, should they be seeded in @world ?
>>> -> imo yes as  we don't want all the default packages getting depcleaned as
>>>    soon as you start using the new install.  if they're in @profile, then this
>>>    is a moot point (assuming depclean does not clean out @profile).
>>
>> In portage, @world = @profile + @selected + @system, which means that
>> @profile is protected from depclean since it's a part of @world.
> 
> so if iputils is in @profile, and i do:
> 	emerge -C iputils
> i don't get the ugly warning about it being in @system, but if i do:
> 	emerge @world
> iputils comes back.  i think that's OK actually since people can do:
> 	emerge @selected
> which has the classic @world meaning.
> -mike
> 

People need to be able to use @world without it pulling in unwanted
packages, since it's the only way to properly do a full update of all
relevant packages (relevant packages being those that would not be
removed by depclean).
-- 
Thanks,
Zac

Re: [gentoo-dev] ironing out release tarballs

[Rich Freeman]

On Thu, Oct 15, 2015 at 5:15 PM, Zac Medico <zmedico@gentoo.org> wrote:
> On 10/15/2015 02:03 PM, Rich Freeman wrote:
>> On Thu, Oct 15, 2015 at 4:57 PM, Zac Medico <zmedico@gentoo.org> wrote:
>>> Given the goals, having catalyst seed /var/lib/portage/world seems
>>> pretty reasonable to me.
>>
>> Then the question becomes how.  Does it diff @profile between the two
>> profiles and put the extra stuff in @selected?  Or, does the profile
>> just contain a special file containing the stuff that gets seeded?
>> That is really the gist of the two approaches, and if you just have a
>> special file full of stuff that gets seeded you really don't need
>> another profile, which is nice since profiles are a PITA right now.
>>
>
> We already have packages.build which is used to build stage1, so
> introducing a packages.stage3 that's used to seed @selected (aka
> /var/lib/portage/world) for stage3 seems reasonable.

Seems reasonable to me, and the nice thing is that it doesn't change
the behavior of anything at all besides catalyst, so other than
starting out with a non-empty /var/lib/portage/world users won't see a
thing happen.

I won't bikeshed on the name of the file.  Whatever seems reasonable
to the catalyst team works fine for me.

This then conserves @profile for stuff that is more essential to the
profile itself, such as a fancy firmware loader for an arm box or
whatever (in our future luxury world where we can spare new profiles
for specific boards and such).  Of course, it wouldn't hurt to
standardize on how such sets work if we're going to start using them
seriously if that isn't in PMS.  However, I see all of that as
off-topic for the present discussion other than the desire to not
interfere with it.

-- 
Rich

Re: [gentoo-dev] ironing out release tarballs

[Anthony G. Basile]

On 10/15/15 5:20 PM, Zac Medico wrote:
> On 10/15/2015 02:13 PM, Mike Frysinger wrote:
>
> so if iputils is in @profile, and i do:
> 	emerge -C iputils
> i don't get the ugly warning about it being in @system, but if i do:
> 	emerge @world
> iputils comes back.  i think that's OK actually since people can do:
> 	emerge @selected
> which has the classic @world meaning.
> -mike
>
> People need to be able to use @world without it pulling in unwanted
> packages, since it's the only way to properly do a full update of all
> relevant packages (relevant packages being those that would not be
> removed by depclean).
But that's true of @system packages now.  If a user does `emerge -C 
iputils` they get a warning about it being in the system set and when 
they do `emerge @world` it comes back.  The only change in moving it to 
@profile is the warning.  Also, I think the other change is during the 
building of the /tmp/stage1root during stage1, @system won't include 
iputils and this saves on the number of packages built.

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] ironing out release tarballs

[Zac Medico]

On 10/15/2015 02:51 PM, Anthony G. Basile wrote:
> On 10/15/15 5:20 PM, Zac Medico wrote:
>> On 10/15/2015 02:13 PM, Mike Frysinger wrote:
>>
>> so if iputils is in @profile, and i do:
>>     emerge -C iputils
>> i don't get the ugly warning about it being in @system, but if i do:
>>     emerge @world
>> iputils comes back.  i think that's OK actually since people can do:
>>     emerge @selected
>> which has the classic @world meaning.
>> -mike
>>
>> People need to be able to use @world without it pulling in unwanted
>> packages, since it's the only way to properly do a full update of all
>> relevant packages (relevant packages being those that would not be
>> removed by depclean).
> But that's true of @system packages now.  If a user does `emerge -C
> iputils` they get a warning about it being in the system set and when
> they do `emerge @world` it comes back. 

I think it's obvious that such packages don't belong in @system, unless
we're talking about a profile where iputils is absolutely required for
some reason.

> The only change in moving it to @profile is the warning.

What's the point of getting rid of the warning if the package is going
to get pulled back in on the next @world update? Either way, the end
result it that the user has to go through some hoops
(/etc/portage/profile overrides) if they don't want that package
installed again.

>  Also, I think the other change is during the
> building of the /tmp/stage1root during stage1, @system won't include
> iputils and this saves on the number of packages built.

The last time I checked, @system was not used for stage1. Catalyst used
package.build instead. Has that changed?
-- 
Thanks,
Zac

Re: [gentoo-dev] ironing out release tarballs

[Rich Freeman]

On Thu, Oct 15, 2015 at 6:00 PM, Zac Medico <zmedico@gentoo.org> wrote:
> On 10/15/2015 02:51 PM, Anthony G. Basile wrote:
>
>> The only change in moving it to @profile is the warning.
>
> What's the point of getting rid of the warning if the package is going
> to get pulled back in on the next @world update? Either way, the end
> result it that the user has to go through some hoops
> (/etc/portage/profile overrides) if they don't want that package
> installed again.

++

I think just pre-seeding the world file is a simpler solution.  Users
can just uninstall the files normally then.


-- 
Rich

Re: [gentoo-dev] ironing out release tarballs

[Anthony G. Basile]

On 10/15/15 6:00 PM, Zac Medico wrote:
> .
> The last time I checked, @system was not used for stage1. Catalyst used
> package.build instead. Has that changed?
Sorry, yes it is.  I forgot how that file was interpreted.  I was 
thinking when I wrote my previous emails that it contained a list of 
packages in addition to @system, but its actually the full stage1 list, 
and packages.build files stack, so that (for example) my

default/linux/uclibc/packages.build

builds on top of the list in

default/linux/packages.build

For the full stage1 list of uclibc stages.

Anyhow, I like the direction Rich and you are going in with 
packages.stage3.  Can we call it something else though, like 
packages.world.  :)

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] ironing out release tarballs

[Alexis Ballier]

On Thu, 15 Oct 2015 18:15:42 -0400
Rich Freeman <rich0@gentoo.org> wrote:

> On Thu, Oct 15, 2015 at 6:00 PM, Zac Medico <zmedico@gentoo.org>
> wrote:
> > On 10/15/2015 02:51 PM, Anthony G. Basile wrote:
> >  
> >> The only change in moving it to @profile is the warning.  
> >
> > What's the point of getting rid of the warning if the package is
> > going to get pulled back in on the next @world update? Either way,
> > the end result it that the user has to go through some hoops
> > (/etc/portage/profile overrides) if they don't want that package
> > installed again.  
> 
> ++
> 
> I think just pre-seeding the world file is a simpler solution.  Users
> can just uninstall the files normally then.


+1


pre-seeding world means 'emerge -C package' gets rid of it, the
@profile idea makes it completely different to uninstall it

Re: [gentoo-dev] ironing out release tarballs

[Alexis Ballier]

On Thu, 15 Oct 2015 13:39:40 -0400
Mike Frysinger <vapier@gentoo.org> wrote:

> On 15 Oct 2015 19:01, Alexis Ballier wrote:
> > On Thu, 15 Oct 2015 11:34:22 -0400 Mike Frysinger wrote:  
> > > - should the packages list be in a new packages.default, or
> > > should we create a new set to hold it, or should we just go with
> > > @profile ? -> @profile has the advantage of already existing.  we
> > > have to be careful so as to make it difficult to uninstall
> > > packages that the user does not actually want.
> > > 
> > > - if the packages aren't in @profile, should they be seeded in
> > > @world ? -> imo yes as  we don't want all the default packages
> > > getting depcleaned as soon as you start using the new install.  if
> > > they're in @profile, then this is a moot point (assuming depclean
> > > does not clean out @profile).  
> > 
> > some kind of 'world' file in profiles like the 'packages' one that
> > is just used to populate world file after (or just before) stage3
> > build ?  
> 
> i suggested the name "packages.default" originally to convey the
> notion that it's just the default set of packages (you'd find in a
> release). keeping the "packages." prefix seemed to be better
> namespace wise.

makes sense

> doesn't require a PMS update because only releng tools (catalyst)
> would read it.  set integration would have to conform to PMS.

if it's just used by catalyst to pre-seed world then indeed pms
doesn't have anything to do with it, but if it's meant to be some set
that profiles add to 'world' set dynamically, then interoperability is
probably desired


> > not sure if sets provide the same flexibility: i can imagine
> > iputils in that set, but also another embedded profile with
> > busybox[make-symlinks], or the bsds  
> 
> i'm not sure putting USE flag qualifiers makes sense as the next time
> the package updates, the USE flags will change.  if profiles want to
> default USE flags, the existing package.use makes more sense imo.


yes, I wasn't talking about useflags at all, and classical ways are
more than enough to deal with them

what I was wondering is how they'd stack: we'd likely want to have a
default set in profiles/base/ that covers 90+% of cases but still leave
room for subprofiles to remove what they replace by something else

anyway, now that I understand the 'packages.default' is the same syntax
as 'packages', I think this is already handled :)

Alexis.

Re: [gentoo-dev] ironing out release tarballs

[Rich Freeman]

On Fri, Oct 16, 2015 at 2:45 AM, Alexis Ballier <aballier@gentoo.org> wrote:
>
> if it's just used by catalyst to pre-seed world then indeed pms
> doesn't have anything to do with it, but if it's meant to be some set
> that profiles add to 'world' set dynamically, then interoperability is
> probably desired
>

I'd suggest that we probably don't want anything dynamic here.

I think most of our users would appreciate a friendly "here, I went
ahead and pre-loaded screen for you but feel free to drop it" on the
first install.  They could review that world file and see what
is/isn't pre-loaded and just delete the whole file if they want a
blank slate.

On the other hand, three years into using their system they probably
wouldn't appreciate if portage just went and installed screen for
them, because it was trying to be helpful and we decided that new
installs should now include screen.

-- 
Rich