[gentoo-dev] new profile layout with flavors and mix-ins

[gentoo-dev] new profile layout with flavors and mix-ins

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 2132 bytes --]

All,

I'm moving to a new thread since the discussion has moved away from just
a sub profile for no-multilib.

On Wed, Jul 02, 2014 at 09:30:50AM -0400, Rich Freeman wrote:
> On Wed, Jun 25, 2014 at 4:01 PM, Andreas K. Huettel
> <dilfridge@gentoo.org> wrote:
> > Am Mittwoch 25 Juni 2014, 15:11:40 schrieb Rich Freeman:
> >> On Wed, Jun 25, 2014 at 2:44 PM, Michał Górny <mgorny@gentoo.org> wrote:
> >> > Long story short, doing anything to Gentoo profiles is utter pain
> >> > and comes with random breakage guarantee. Therefore, I'm asking -- nuke
> >> > those damn profiles, and start over! The current situation is
> >> > completely unmaintainable.
> >>
> >> ++
> >>
> >> But, would it make sense to just go the Funtoo route with "mix-ins."
> > ++
> >
> > this is what we've been just discussing on the irc channel
> 
> So, not wanting this to die on the vine.
> 
> If we did the mix-in approach, would we just follow the example of Funtoo?
> 
> They use an arch profile, a stability profile (~arch vs arch), a
> "flavor" profile (core, minimal, desktop), and then users can layer as
> much other stuff on top of that as they want (gnome, kde, multimedia,
> etc).

I think this could work for us as well, or something similar anyway.

For those who are curious, I am including the link to the flavors and
mix-ins descriptions from the funtoo site. [1]

> 
> Do we want to do things the same way?
> 
> Some things to think about include multilib (just another arch?),
> systemd, and usr-merge.  I'm not saying that we need to implement any
> of that stuff completely - but when planning the profile layout we
> should at least consider whether it will handle things like this in
> the future.  Should some types of profiles be only additive?  Etc...

I see systemd and multilib as mix-ins, like the ones you mentioned
above.

It is funny that you mention usr-merge. I don't want to get into a big
debate on it on this thread, so for now I'll just say that I don't see
that as affecting the profile structure at all.

William

[1] http://www.funtoo.org/Flavors_and_Mix-ins

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] new profile layout with flavors and mix-ins

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 3166 bytes --]

Dnia 2014-07-02, o godz. 10:44:16
William Hubbs <williamh@gentoo.org> napisał(a):

> All,
> 
> I'm moving to a new thread since the discussion has moved away from just
> a sub profile for no-multilib.
> 
> On Wed, Jul 02, 2014 at 09:30:50AM -0400, Rich Freeman wrote:
> > On Wed, Jun 25, 2014 at 4:01 PM, Andreas K. Huettel
> > <dilfridge@gentoo.org> wrote:
> > > Am Mittwoch 25 Juni 2014, 15:11:40 schrieb Rich Freeman:
> > >> On Wed, Jun 25, 2014 at 2:44 PM, Michał Górny <mgorny@gentoo.org> wrote:
> > >> > Long story short, doing anything to Gentoo profiles is utter pain
> > >> > and comes with random breakage guarantee. Therefore, I'm asking -- nuke
> > >> > those damn profiles, and start over! The current situation is
> > >> > completely unmaintainable.
> > >>
> > >> ++
> > >>
> > >> But, would it make sense to just go the Funtoo route with "mix-ins."
> > > ++
> > >
> > > this is what we've been just discussing on the irc channel
> > 
> > So, not wanting this to die on the vine.
> > 
> > If we did the mix-in approach, would we just follow the example of Funtoo?
> > 
> > They use an arch profile, a stability profile (~arch vs arch), a
> > "flavor" profile (core, minimal, desktop), and then users can layer as
> > much other stuff on top of that as they want (gnome, kde, multimedia,
> > etc).
> 
> I think this could work for us as well, or something similar anyway.
> 
> For those who are curious, I am including the link to the flavors and
> mix-ins descriptions from the funtoo site. [1]

It's not that easy. As you can see on that site, they're supporting
much less variants than Gentoo does. In particular, they don't seem to
support non-GNU/Linux at all. No Prefix, no Hardened, no FreeBSD.

I was thinking about modularization a bit and the main issue is
handling intersecting profiles. As you can see in Funtoo, it already
starts with the 'build' flavor -- they're pretty much applying a cheap
hack (ACCEPT_KEYWORDS="${ARCH}") but such hacks can't cover all our
needs.

A simple example is CHOST. The value of CHOST depends on the arch,
often ABI, kernel, libc. Of course, we could hack this around by
creating some intermediate variables and merging them afterwards.
But not everything can be hacked around like this.

I don't feel like we ought to vote on something like this without
understanding most of the current profiles. And I'm afraid there are
only few people who have any idea about the current profile
structure...

> > Do we want to do things the same way?
> > 
> > Some things to think about include multilib (just another arch?),
> > systemd, and usr-merge.  I'm not saying that we need to implement any
> > of that stuff completely - but when planning the profile layout we
> > should at least consider whether it will handle things like this in
> > the future.  Should some types of profiles be only additive?  Etc...
> 
> I see systemd and multilib as mix-ins, like the ones you mentioned
> above.

Multilib won't work as Funtoo-style mix-in for the simple reason it
relies on heavily on the architecture in use.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] new profile layout with flavors and mix-ins

[Rich Freeman]

On Wed, Jul 2, 2014 at 1:54 PM, Michał Górny <mgorny@gentoo.org> wrote:
> I don't feel like we ought to vote on something like this without
> understanding most of the current profiles. And I'm afraid there are
> only few people who have any idea about the current profile
> structure...

No argument there.

We may very well still end up with something hierarchical, but we can
at least limit that to the parts of the profile where it matters.
Maybe x86/BSD and amd64/Linux and amd64/Linux-hardened need to be
interdependent.  However, that still gets rid of need to deal with
desktop environments, init systems, arguments over what belongs in
@system, and so on.  We could have a blocker mechanism to keep people
from mixing systemd with BSD, or we could just let people shoot
themselves in the foot.

Sounds like a good time to start reverse engineering the profiles...

Rich

Re: [gentoo-dev] new profile layout with flavors and mix-ins

[Anthony G. Basile]

On 07/02/14 14:10, Rich Freeman wrote:
> On Wed, Jul 2, 2014 at 1:54 PM, Michał Górny <mgorny@gentoo.org> wrote:
>> I don't feel like we ought to vote on something like this without
>> understanding most of the current profiles. And I'm afraid there are
>> only few people who have any idea about the current profile
>> structure...
> No argument there.
>
> We may very well still end up with something hierarchical, but we can
> at least limit that to the parts of the profile where it matters.
> Maybe x86/BSD and amd64/Linux and amd64/Linux-hardened need to be
> interdependent.  However, that still gets rid of need to deal with
> desktop environments, init systems, arguments over what belongs in
> @system, and so on.  We could have a blocker mechanism to keep people
> from mixing systemd with BSD, or we could just let people shoot
> themselves in the foot.
>
> Sounds like a good time to start reverse engineering the profiles...
>
> Rich
>

The way the profiles stack via the parent file makes them difficult to 
work with if they get to any significant depth.  Here, for example, is 
the stacking for default/linux/mips/13.0/mipsel/multilib/n32

/usr/portage/profiles/base
/usr/portage/profiles/default/linux
*/usr/portage/profiles/arch/base
*/usr/portage/profiles/arch/mips
*/usr/portage/profiles/default/linux/mips
/usr/portage/profiles/releases
/usr/portage/profiles/releases/13.0
/usr/portage/profiles/default/linux/mips/13.0
*/usr/portage/profiles/arch/base
*/usr/portage/profiles/arch/mips
*/usr/portage/profiles/arch/mips/mipsel
/usr/portage/profiles/default/linux/mips/13.0/mipsel
*/usr/portage/profiles/arch/base
*/usr/portage/profiles/arch/mips
*/usr/portage/profiles/arch/mips/mipsel
/usr/portage/profiles/arch/mips/mipsel/mips64el
/usr/portage/profiles/features/multilib
/usr/portage/profiles/arch/mips/mipsel/mips64el/multilib
/usr/portage/profiles/arch/mips/mipsel/mips64el/multilib/n32
/usr/portage/profiles/default/linux/mips/13.0/mipsel/multilib/n32


I put asterisks there to point out a pattern that gets pulled in 
repeatedly.  Sometimes this isn't a problem but sometimes this leads to 
asserting something, then reverting it, then asserting it again. In the 
ancient selinux profiles (circa 2009) this meant you couldn't have a 
no-mutlilib amd64 system.  Shallow profiles avoid this.  Also "features" 
avoid this (the closest thing we have to mix-ins) provided they operate 
on a set of flags/packages orthogonal to the rest of the stack.  You 
then have  shallow base and you can add as many features as you like in, 
in any order, confident that one will not clobber stuff from another 
since each feature is well separated.

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] new profile layout with flavors and mix-ins

[Rich Freeman]

On Wed, Jul 2, 2014 at 2:32 PM, Anthony G. Basile <blueness@gentoo.org> wrote:
> Shallow profiles avoid this.  Also "features" avoid this (the
> closest thing we have to mix-ins) provided they operate on a set of
> flags/packages orthogonal to the rest of the stack.  You then have  shallow
> base and you can add as many features as you like in, in any order,
> confident that one will not clobber stuff from another since each feature is
> well separated.

That was my thinking as well.   If we had categories of profiles and
set up the rules to try to keep things orthogonal then we're going to
be better off.  Also, things like feature profiles should try to be
additive in nature.

Rich

Re: [gentoo-dev] new profile layout with flavors and mix-ins

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/02/2014 02:10 PM, Rich Freeman wrote:
> On Wed, Jul 2, 2014 at 1:54 PM, Michał Górny <mgorny@gentoo.org> wrote:
>> I don't feel like we ought to vote on something like this without
>> understanding most of the current profiles. And I'm afraid there are
>> only few people who have any idea about the current profile
>> structure...
> 
> No argument there.
> 
> We may very well still end up with something hierarchical, but we can
> at least limit that to the parts of the profile where it matters.
> Maybe x86/BSD and amd64/Linux and amd64/Linux-hardened need to be
> interdependent.  However, that still gets rid of need to deal with
> desktop environments, init systems, arguments over what belongs in
> @system, and so on.  We could have a blocker mechanism to keep people
> from mixing systemd with BSD, or we could just let people shoot
> themselves in the foot.
> 
> Sounds like a good time to start reverse engineering the profiles...
> 
I've talked to the funtoo devs a few times about stealing their profile
idea.  A few things need to be done to make this happen, mainly eselect,
catalyst, and repoman support.  I can do the eselect and catalyst
changes myself, however, I'll require some help for the repoman support
most likely.  I'll prototype this locally, see what I can make work, and
then see about making it usable for others to test.

- -Zero
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJTtFI/AAoJEKXdFCfdEflK7GYP/0GS+Sh4odpHNMUfNfyU+q7F
HpNuJ/OSnnNBRol4dDjcvhE3RFxIqU1DkC1atsl19K14vJXk+JBSKw8fgV4J6ous
0uOYqN472Noy7NxzmrbfrlR1PePG1b878ZpOMbGce1dgMfamoDwv48IjfcSwakcv
6HSy09xfr5O8xG+cSBqyVmaiDxsPHPQwv1s+/JUpcadBr4AmD7qc7yTZ5QjKJoIg
o1eL/j//KiYxQywoQ1udqXwx/beBgYMQB4R5Vu4d1BBtIttnJdTHCZdAv6Uoi686
x4VWuvBPDaILojyhIYqpoMbKAu/c/AFRFkpFZXFVl5kWyolw/YuBg4RL21qecDvp
9qyoQFCzIWCSzXl747O4iFYjS1ACbMSUXOSDEecvnYuCsjvI+bpJT2pWVlAxIMUH
79IgKTu0+1LETl/lEeMN8ccrG8R2hhrYCxxyTw1nZfevN9SPC88RwEIcuBwVFHC3
NC7fpxKzDFy2+0ixBnLCQA7ICnvV5crF2Tt4/bg9Hbp2UP/QMmhxIwUbceRSDXc5
3c8nqsUGavLX8K9IE3IlI5ZYutmYJgJwfqX8yA6wdOeZgPvH59gypnBYFKJjO2DQ
JZE+lVZ4vXLmgw1WTAdRUYFhLEqesioCAVH7Bm7gUnccCeWE4PFHr15BGhdh/Mw9
L88DkCnfAbLcSHRJS0Cx
=iBrl
-----END PGP SIGNATURE-----

Re: [gentoo-dev] new profile layout with flavors and mix-ins

[Anthony G. Basile]

On 07/02/14 14:41, Rick "Zero_Chaos" Farina wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 07/02/2014 02:10 PM, Rich Freeman wrote:
>> On Wed, Jul 2, 2014 at 1:54 PM, Michał Górny <mgorny@gentoo.org> wrote:
>>> I don't feel like we ought to vote on something like this without
>>> understanding most of the current profiles. And I'm afraid there are
>>> only few people who have any idea about the current profile
>>> structure...
>> No argument there.
>>
>> We may very well still end up with something hierarchical, but we can
>> at least limit that to the parts of the profile where it matters.
>> Maybe x86/BSD and amd64/Linux and amd64/Linux-hardened need to be
>> interdependent.  However, that still gets rid of need to deal with
>> desktop environments, init systems, arguments over what belongs in
>> @system, and so on.  We could have a blocker mechanism to keep people
>> from mixing systemd with BSD, or we could just let people shoot
>> themselves in the foot.
>>
>> Sounds like a good time to start reverse engineering the profiles...
>>
> I've talked to the funtoo devs a few times about stealing their profile
> idea.  A few things need to be done to make this happen, mainly eselect,
> catalyst, and repoman support.  I can do the eselect and catalyst
> changes myself, however, I'll require some help for the repoman support
> most likely.  I'll prototype this locally, see what I can make work, and
> then see about making it usable for others to test.
>
> - -Zero
> -----BEGIN PGP SIGNATURE-----
>

I don't know how to get from here to there.  The problem isn't just 
constructing an alternative profile tree.  We could even have 
/usr/portage/profiles-r2 and switch between the two on demand.  The 
problem is there's a lot of memory with flags and masks and these only 
make sense in the context of the current stacking profiles. 
Disentangling this information and bringing it over to profiles-r2 is 
going to be work.

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] new profile layout with flavors and mix-ins

[Rick "Zero_Chaos" Farina]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/02/2014 03:07 PM, Anthony G. Basile wrote:
> On 07/02/14 14:41, Rick "Zero_Chaos" Farina wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> On 07/02/2014 02:10 PM, Rich Freeman wrote:
>>> On Wed, Jul 2, 2014 at 1:54 PM, Michał Górny <mgorny@gentoo.org> wrote:
>>>> I don't feel like we ought to vote on something like this without
>>>> understanding most of the current profiles. And I'm afraid there are
>>>> only few people who have any idea about the current profile
>>>> structure...
>>> No argument there.
>>>
>>> We may very well still end up with something hierarchical, but we can
>>> at least limit that to the parts of the profile where it matters.
>>> Maybe x86/BSD and amd64/Linux and amd64/Linux-hardened need to be
>>> interdependent.  However, that still gets rid of need to deal with
>>> desktop environments, init systems, arguments over what belongs in
>>> @system, and so on.  We could have a blocker mechanism to keep people
>>> from mixing systemd with BSD, or we could just let people shoot
>>> themselves in the foot.
>>>
>>> Sounds like a good time to start reverse engineering the profiles...
>>>
>> I've talked to the funtoo devs a few times about stealing their profile
>> idea.  A few things need to be done to make this happen, mainly eselect,
>> catalyst, and repoman support.  I can do the eselect and catalyst
>> changes myself, however, I'll require some help for the repoman support
>> most likely.  I'll prototype this locally, see what I can make work, and
>> then see about making it usable for others to test.
>>
>> - -Zero
>> -----BEGIN PGP SIGNATURE-----
>>
> 
> I don't know how to get from here to there.  The problem isn't just
> constructing an alternative profile tree.  We could even have
> /usr/portage/profiles-r2 and switch between the two on demand.  The
> problem is there's a lot of memory with flags and masks and these only
> make sense in the context of the current stacking profiles.
> Disentangling this information and bringing it over to profiles-r2 is
> going to be work.
> 
I agree entirely.  Right now the mixin style profiles are not supported
in gentoo _at_all_, and if no one ever works on that, then we will never
support it.  I will work on the first step in a long road, that's all
I'm talking about here.

Thanks,
Zero
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJTtFtTAAoJEKXdFCfdEflKDk0P/j6F/7ivY2HkbRV3Y60YiahE
IbzwrZ7wQCap5sAELOtQaXTyTHuyTLkTrAqqM/r+/8a0/Ebm67yEALbzE4IOMAFL
DoiS7mg0FZeq2Bm8AGKtYwmBHAiIYbGWX+XgxuFbwpHz4uilsVhQH9+e+rFlu6SN
22BDmW6tQSDL5a3C2Coqoq/KVH/heHu43w6/TuZmVVByLcABHrEG/KOnLRHSGfwF
ps+R3UaMmFA9w6VGqGxlOXuIbF+cp//xOTfqmetYfe5kkOsls/ZAKou1MMCrCSI7
j6oJbH5q5Gn5W6vxctf8PJPI8R5NWZBOFwvDgkeJ9Zcsqhsinb19QQFVkG6HOrt1
2nSTOP9U/5d0wYn9P7cNgQsXjg8P2KwHfPXG8qZ8jx1CSM0T2bY4csjALgqAmvpz
ATa9XbpGmcRMoldqiVZEVN2EqL1NWVXo0i490Y51noiLpNSiSO2XT4lbIj1fae+i
yxt6RuJ04QltsQKKzH3swCqgFBUZmNZe2h0p0dt+GFOl9Pg0oa3QX3eGR0msgmEy
Qti0JRZwQJrHkDCSzZP0OAI2RZDj5XxmhS1brTTvb7LnxFecWR760LfQ/bzJgqno
M800gZKrJ9AjzC8lvzqWQrd39W4VYhcDn19S5zHtBPb3KK/4jC9J1TVwwgzbW7jb
z6ytQ2biVKhkBPrec5tf
=2WzO
-----END PGP SIGNATURE-----

Re: [gentoo-dev] new profile layout with flavors and mix-ins

[Rich Freeman]

On Wed, Jul 2, 2014 at 3:19 PM, Rick "Zero_Chaos" Farina
<zerochaos@gentoo.org> wrote:
>
> On 07/02/2014 03:07 PM, Anthony G. Basile wrote:
>>
>> I don't know how to get from here to there.  The problem isn't just
>> constructing an alternative profile tree.  We could even have
>> /usr/portage/profiles-r2 and switch between the two on demand.  The
>> problem is there's a lot of memory with flags and masks and these only
>> make sense in the context of the current stacking profiles.
>> Disentangling this information and bringing it over to profiles-r2 is
>> going to be work.
>>
> I agree entirely.  Right now the mixin style profiles are not supported
> in gentoo _at_all_, and if no one ever works on that, then we will never
> support it.  I will work on the first step in a long road, that's all
> I'm talking about here.

I agree.  I think getting support in all the tools for multiple
profiles is a good start.  Volunteers can then begin working on
profiles that dis-entangle things.

A first step might be to just rip out the desktop/etc categories,
leaving the mess of arch/hardened/kernel/etc.  Then we try to keep
peeling back the layers.  We don't have to get there in one step, nor
do we have to release anything to the general public before it is
finished.

We might not end up with completely flat profiles - the goal is to end
up in a better place than we are now, and perhaps we can make more
progress later.

Rich

Re: Re: [gentoo-dev] new profile layout with flavors and mix-ins

[Andreas K. Huettel]

Am Mittwoch 02 Juli 2014, 15:07:12 schrieb Anthony G. Basile:

> 
> I don't know how to get from here to there.  The problem isn't just
> constructing an alternative profile tree.  We could even have
> /usr/portage/profiles-r2 and switch between the two on demand.  The
> problem is there's a lot of memory with flags and masks and these only
> make sense in the context of the current stacking profiles.
> Disentangling this information and bringing it over to profiles-r2 is
> going to be work.

Crazy idea: 
* introduce a change that makes portage look at a new filename for 
inheritance, i.e. existing "parent" files are disregarded and a new filename 
is introduced ("inherits" ?)
* in most dirs that file will not exist -> no inheritance
* new profile specs will have new main directories that pull in the resulting 
flat structure piece by piece

This means that existing files can (carefully) be re-used in the transition.

-- 
Andreas K. Huettel
Gentoo Linux developer
kde, council

Re: [gentoo-dev] new profile layout with flavors and mix-ins

[Tom Wijsman]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, 02 Jul 2014 14:41:03 -0400
"Rick \"Zero_Chaos\" Farina" <zerochaos@gentoo.org> wrote:

> I've talked to the funtoo devs a few times about stealing their
> profile idea.  A few things need to be done to make this happen,
> mainly eselect, catalyst, and repoman support.  I can do the eselect
> and catalyst changes myself, however, I'll require some help for the
> repoman support most likely.  I'll prototype this locally, see what I
> can make work, and then see about making it usable for others to test.

Did the Funtoo devs tell you that they don't run repoman because of the
explosive set of possible combinations that flavors & mix-ins introduce?

Having it run over them should be easy; but having it run within a
reasonable time when scaling up is going to be quite painful, ...

It sounds like a trade-off between better profiles and better repoman.

- -- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBAgAGBQJTteLDAAoJEPWZc8roOL/QXbIIAJpM1QZhfu51deklcxIi+6PJ
6yxNc1CA6eJfhQouGYR2QgEHUI4YASEz0invFpXbv4IM/NYY/JSX/e+EOoykqcub
eN9ZIFcUOdnuP7GnG+tVOe0oMJ0jhj7yEB/+Iifz63qPlL04gMqZDYnMpz2EIrUB
2SmiFKaxWaJBv/nFoVF2ErWuNjVTMwZE8cP7Eob2HO+G2N2tLbRVMf0cJPOYHGGt
00Y+4c8EPnWvdcnHN0ei14FyTYKb+eRnvGsN5xqTnMw6oG/bGhEiHtIkdMGfrzfq
UeG5cPGfxll7u8m9isA22c+lKdtnLZ9EIQtPs7Jxg09hSW3HqiNgkA5b/o/m+1A=
=ROh0
-----END PGP SIGNATURE-----

Re: [gentoo-dev] new profile layout with flavors and mix-ins

[Rich Freeman]

On Thu, Jul 3, 2014 at 7:09 PM, Tom Wijsman <TomWij@gentoo.org> wrote:
> Did the Funtoo devs tell you that they don't run repoman because of the
> explosive set of possible combinations that flavors & mix-ins introduce?
>
> Having it run over them should be easy; but having it run within a
> reasonable time when scaling up is going to be quite painful, ...

I don't think we need to check all the combinations.

To keep things manageable I think we'd want to define some structure
around the profiles and what different kinds of profiles should and
shouldn't do.  If a profile just pulls in some packages and sets some
application-oriented USE flags we shouldn't really need to worry about
conflicts - if there are any users will either learn not to run them
in combination or at least they won't end up with a completely cripped
system if some non-core applications have issues.    Messing with ABIs
or system-wide configuration settings should be reserved for specific
types of profiles which wouldn't be run in combinations unless
carefully controlled.  We should definitely avoid the mess of
inheritance that leads to settings being toggled back and forth.

Even if we still end up being stuck with a kernel/arch/subarch
hierarchy we'll at least be able to ditch layering on things like
kde/gnome further down and open the door to having more variety in our
profiles at that level.  Something like x32 coming along is fairly
rare, even if it is really interesting (and this stuff is one of
Gentoo's strong points, actually).  There is probably a lot more
interest in having more application-level mix-ins in the main tree, or
especially in overlays.  The reason we avoid them is that today they'd
lead to 300 more profiles on top of what is already a big mess.  If
that part of the tree gets flattened there is no reason somebody could
have a minimal profile that doesn't stick openssh in @system, or a
bunch of server profiles for various use cases, or some profiles for
clusters, config-managed boxes, etc.

So, if we come up with a decent strategy, deciding what repoman does
and doesn't have to worry about would be part of it.  We obviously
can't check all the combos, but that doesn't mean that we can't make
sure that linux/x86/foo doesn't break.

Rich

Re: [gentoo-dev] new profile layout with flavors and mix-ins

[Joshua Kinard]

On 07/02/2014 13:54, Michał Górny wrote:
> Dnia 2014-07-02, o godz. 10:44:16
[snip]
> 
> I don't feel like we ought to vote on something like this without
> understanding most of the current profiles. And I'm afraid there are
> only few people who have any idea about the current profile
> structure...
> 

I am going to throw this out there and see what people think.  Maybe it's
insane, maybe it's not, maybe it's a mix of insane and not-insane.

Years ago, before we had the current stacking profile design (we were
discussing the current design, actually), I kinda conjured up this "building
blocks" like approach for a profile design.  Basically building on how most
of a Gentoo system is pieced together in /etc/make.conf (or wherever that
file lives now).  A new variable, $PROFILE, would be defined that contains a
colon-separated list of profile "blocks" that specifies the nature of the
system.  Kinda like how $PATH is built.  This idea never got any traction,
though, and may still not.

Quoting from an e-mail I sent months ago to blueness after reading one of
his blog posts, here's what I worked out years ago with some updated
thinking a few months ago:

----------------------

An idea I had at the time, but which never gained any traction, was to use a
similar profile layout as we have now, but not treat it like a sequence of
directory structures.  Instead, the idea was to have it modeled more like
how $PATH is built up from discrete components.

So instead of something like this:

    /usr/portage/profiles/base
    /usr/portage/profiles/default/linux
    /usr/portage/profiles/arch/base
    /usr/portage/profiles/features/multilib
    /usr/portage/profiles/features/multilib/lib32
    /usr/portage/profiles/arch/amd64
    /usr/portage/profiles/releases
    /usr/portage/profiles/eapi-5-files
    /usr/portage/profiles/releases/13.0
    /usr/portage/profiles/hardened/linux
    /usr/portage/profiles/hardened/linux/amd64
    /usr/portage/profiles/targets/desktop
    /usr/portage/profiles/hardened/linux/amd64/destkop

You would have a layout in /usr/portage/profiles look like this instead
(btw, this is from long-term memory, so some things might be off from my
original idea way back when):

   arch/               base/            desktops/
     |-amd64/                             |-e17/
     |-arm/                               |-kde/
     |-mips/                              |-gnome/
     |-sparc/                             |-xfce/
     |-x86/

   eapi/             kernel/            libc/
     |-3/              |-freebsd/         |-glibc
     |-4/              |-linux/           |-uclibc
     |-5/                                 |-musl
     |-6/                                 |-freebsd

   options/          releases/          roles/
     |-hardened/       |-11.0/            |-desktop/
     |-multilib/       |-12.0/            |-firewall/
                       |-13.0/            |-ids/
                       |-freebsd-9.2/     |-router/
                                          |-server/

   servers/          subarch/
     |-dns/            |-mips-o32
     |-file/           |-mips-n32
     |-mail/           |-sparc-v8
     |-www/            |-amd64-x32

The idea being that, in /etc/make.conf (or wherever that file is now), you'd
define $PROFILE like this:

linux-mips o32 uclibc server:
PROFILE="base:kernel/linux:arch/mips:subarch/mips-o32:libc/uclibc:roles/server:releases/13.0"

linux-amd64/x86_64/x64 glibc hardened 13.0 KDE desktop:
PROFILE="base:kernel/linux:arch/amd64:libc/glibc:roles/desktop:options/hardened:desktops/kde:releases/13.0"

freebsd-amd64 9.2 firewall:
PROFILE="base:kernel/freebsd:arch/amd64:libc/freebsd:roles/firewall:releases/freebsd-9.2"

And so on.  The goal was to have profiles/ be extremely flat, with limited
nesting only for categorization purposes.  Each component would contain all
of the information specific to that component, and rely on OOP-like
inheritance to override parent-level variables only within that component
(although, <arch> and <kernel> would have to be a little bit more broad in
scope).

PROFILE also had a set order for the first few pieces, if only to make
parsing and building the profile stack saner for the Portage developers:
   PROFILE="base:<KERNEL>:<ARCH>[:<SUBARCH>]:<LIBC>:<EVERYTHING_ELSE>"

base - Core Gentoo/Portage/whatever information/vars/foobar/kittens

kernel - Specifies the OS kernel.  At the time, only Linux existed, but
         I *think* Debian was eyeballing kFreeBSD at the time.  So I *think*
         I accounted for it back then.

arch - Machine arch-specific generic information -- doesn't handle
       lower-level things like ISAs/ABIs/Endian, etc.

subarch - Defines items specific given to given subarch of a main arch.
          Items under this directory would carry their parent arch's
          name for clarity only.  Again, at the time, MIPS would have been
          the only real user of this, and, back then, it would've dealt
          with SGI-machine-specific packages and USE flags, such as
          differentiating an ip32 machine from an ip27 machine or a
          cobalt.  Now that amd64/x86_64 is considering its own form of
          n32 (x32), that would go here, and I imagine arm would
          make heavy use of it as well.

libc - Defines the main C library used.  A bit redundant for *BSD, because
       they really only have one, but might help if we ever add k*BSD
       support (such as freebsd/glibc-based or such).  For Linux...could be
       tricky, since there are so many libc possibilities there.  I feel it
       fits better getting defined after <SUBARCH>, especially in the case
       of MIPS.

desktops - Settings for specific desktops if "roles/desktop" is specified
           in $PROFILE.

eapi - EAPI didn't exist back when this topic was visited.  Basically,
       everything was EAPI=0 and there was only Portage (Paludis nor
       pkgcore had been invented yet).

options - Was a different name back then.  Basically, hardened was the only
          alternate mode available back then.  Multilib hadn't even been
          thought of, because MIPS was the only arch that even had to worry
          about it, and most people were both scared and confused by the
          thought of multiple ABIs.

releases - I don't think this existed back then.  How it'd operate
           nowadays, I have no idea.  Probably would contain
           version-specific maskings for specific @system packages
           to facilitate upgrading, and help us if we ever tried to cut
           actual, fixed release points again (like what FreeBSD does
           w/ -RELEASE-x.y)

roles - Just thought this one up, as back then, we only had desktop and
        server.  It'd basically define a minimal set of recommended
        packages necessary to qualify that role.  I.e., desktop/kde
        would install KDE, Qt, and other related packages, while ids/
        would install say, snort and tcpdump for packet inspection.

servers - Like "desktops" above, defines minimal settings for specific
          server types.  So "servers/www" would drag in apache, and
          "servers/dns" would attempt to pull in BIND.  Obviously,
          there would need to be a way to override the core software
          element for a given server role -- Maybe USE flags could be
          modified to handle this (so that -apache +lighthttpd swaps
          apache out for lighthttpd).  Needs a lot more thinking on
          this one.

----------------------

Thoughts?

-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
4096R/D25D95E3 2011-03-28

"The past tempts us, the present confuses us, the future frightens us.  And
our lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

Re: [gentoo-dev] new profile layout with flavors and mix-ins

[Michael Haubenwallner]

On 07/03/2014 08:18 AM, Joshua Kinard wrote:
> On 07/02/2014 13:54, Michał Górny wrote:
>> Dnia 2014-07-02, o godz. 10:44:16
> [snip]
>>
>> I don't feel like we ought to vote on something like this without
>> understanding most of the current profiles. And I'm afraid there are
>> only few people who have any idea about the current profile
>> structure...
>>
> 
> I am going to throw this out there and see what people think.  Maybe it's
> insane, maybe it's not, maybe it's a mix of insane and not-insane.
> 
> Years ago, before we had the current stacking profile design (we were
> discussing the current design, actually), I kinda conjured up this "building
> blocks" like approach for a profile design.

> The idea being that, in /etc/make.conf (or wherever that file is now), you'd
> define $PROFILE like this:
> 
> linux-mips o32 uclibc server:
> PROFILE="base:kernel/linux:arch/mips:subarch/mips-o32:libc/uclibc:roles/server:releases/13.0"

What about making /etc/portage/make.profile a directory rather than a symlink,
having /etc/portage/make.profile/parent to reference all the flavours?

Tools that need to respect the /current/ profile should work without any change, and
tools that need to respect the /available/ profiles (repoman) already do have a list
of profiles to respect (profiles/profiles.desc).

So the only missing thing would be the eselect profile module to manage entries of
/etc/portage/make.profile/parent, maybe using /usr/portage/profiles/profiles.desc
as the source for available flavours.

my 2 cents
/haubi/

Re: [gentoo-dev] new profile layout with flavors and mix-ins

[Joshua Kinard]

On 07/03/2014 03:00, Michael Haubenwallner wrote:
> 
> On 07/03/2014 08:18 AM, Joshua Kinard wrote:
>> On 07/02/2014 13:54, Michał Górny wrote:
>>> Dnia 2014-07-02, o godz. 10:44:16
>> [snip]
>>>
>>> I don't feel like we ought to vote on something like this without
>>> understanding most of the current profiles. And I'm afraid there are
>>> only few people who have any idea about the current profile
>>> structure...
>>>
>>
>> I am going to throw this out there and see what people think.  Maybe it's
>> insane, maybe it's not, maybe it's a mix of insane and not-insane.
>>
>> Years ago, before we had the current stacking profile design (we were
>> discussing the current design, actually), I kinda conjured up this "building
>> blocks" like approach for a profile design.
> 
>> The idea being that, in /etc/make.conf (or wherever that file is now), you'd
>> define $PROFILE like this:
>>
>> linux-mips o32 uclibc server:
>> PROFILE="base:kernel/linux:arch/mips:subarch/mips-o32:libc/uclibc:roles/server:releases/13.0"
> 
> What about making /etc/portage/make.profile a directory rather than a symlink,
> having /etc/portage/make.profile/parent to reference all the flavours?
> 
> Tools that need to respect the /current/ profile should work without any change, and
> tools that need to respect the /available/ profiles (repoman) already do have a list
> of profiles to respect (profiles/profiles.desc).
> 
> So the only missing thing would be the eselect profile module to manage entries of
> /etc/portage/make.profile/parent, maybe using /usr/portage/profiles/profiles.desc
> as the source for available flavours.
> 
> my 2 cents
> /haubi/

That's the thing, make.profile technically *is* a directory -- a symlink to
one.  The original design of make.profile was to specify generic, base
settings for a given profile and keep that in the tree.  Things like default
CHOST, default ARCH, default <VARIABLE>, etc.  make.conf then overrides
in-tree settings with settings specific to your system.  So making
/etc/make.profile an actual directory disconnects it from the tree, which I
don't think will work very well for Portage, since it won't know what your
currently-chosen profile is.

-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
4096R/D25D95E3 2011-03-28

"The past tempts us, the present confuses us, the future frightens us.  And
our lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

Re: [gentoo-dev] new profile layout with flavors and mix-ins

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 03/07/14 04:47 AM, Joshua Kinard wrote:
> On 07/03/2014 03:00, Michael Haubenwallner wrote:
>> 
>> On 07/03/2014 08:18 AM, Joshua Kinard wrote:
>>> On 07/02/2014 13:54, Michał Górny wrote:
>>>> Dnia 2014-07-02, o godz. 10:44:16
>>> [snip]
>>>> 
>>>> I don't feel like we ought to vote on something like this
>>>> without understanding most of the current profiles. And I'm
>>>> afraid there are only few people who have any idea about the
>>>> current profile structure...
>>>> 
>>> 
>>> I am going to throw this out there and see what people think.
>>> Maybe it's insane, maybe it's not, maybe it's a mix of insane
>>> and not-insane.
>>> 
>>> Years ago, before we had the current stacking profile design
>>> (we were discussing the current design, actually), I kinda
>>> conjured up this "building blocks" like approach for a profile
>>> design.
>> 
>>> The idea being that, in /etc/make.conf (or wherever that file
>>> is now), you'd define $PROFILE like this:
>>> 
>>> linux-mips o32 uclibc server: 
>>> PROFILE="base:kernel/linux:arch/mips:subarch/mips-o32:libc/uclibc:roles/server:releases/13.0"
>>
>>
>>> 
What about making /etc/portage/make.profile a directory rather than a
symlink,
>> having /etc/portage/make.profile/parent to reference all the
>> flavours?
>> 
>> Tools that need to respect the /current/ profile should work
>> without any change, and tools that need to respect the
>> /available/ profiles (repoman) already do have a list of profiles
>> to respect (profiles/profiles.desc).
>> 
>> So the only missing thing would be the eselect profile module to
>> manage entries of /etc/portage/make.profile/parent, maybe using
>> /usr/portage/profiles/profiles.desc as the source for available
>> flavours.
>> 
>> my 2 cents /haubi/
> 
> That's the thing, make.profile technically *is* a directory -- a
> symlink to one.  The original design of make.profile was to specify
> generic, base settings for a given profile and keep that in the
> tree.  Things like default CHOST, default ARCH, default <VARIABLE>,
> etc.  make.conf then overrides in-tree settings with settings
> specific to your system.  So making /etc/make.profile an actual
> directory disconnects it from the tree, which I don't think will
> work very well for Portage, since it won't know what your 
> currently-chosen profile is.
> 

I did a test where I dropped my make.profile symlink, made a directory
of the same name, and populated a 'parent' file in that directory with
paths to various /usr/portage/profiles/ bits which made up my previous
main profile.  Everything continued to work as expected.

I expect for portage proper, we could accomodate mixins or whatever
new structure method we want simply by adjusting 'eselect profile' to
list the various possible combinations and adjust an
/etc/make.profile/parent file accordingly.

(We probably also need to ensure portage warns or errors appropriately
when one of these inherit lines in the parent file no longer points to
a directory, the same as it does now with the symlink points to
nothing; I didn't test but I expect that error would not be pretty or
easily understood right now)




-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iF4EAREIAAYFAlO1f4IACgkQ2ugaI38ACPB1wgEApznnssegz2ImYIq6fAeR2oGa
RX0TNT6bThDcypkn0skA/j/w33cWekomOQanCKIspuOWxXgOAeMxMWlnhsORZfj7
=Qwj0
-----END PGP SIGNATURE-----

[gentoo-dev] Re: new profile layout with flavors and mix-ins

[Duncan]

Michael Haubenwallner posted on Thu, 03 Jul 2014 09:00:18 +0200 as
excerpted:

> What about making /etc/portage/make.profile a directory rather than a
> symlink, having /etc/portage/make.profile/parent to reference all the
> flavours?

We /almost/ have that already.  I've long thought that /etc/portage/
profile should be the "real" profile instead of an override of the real 
profile, in which case it would have a parent file as you suggested, 
which could have multiple listed parents.

Then /etc/portage/make.profile could be done away with, or more likely at 
least for a time, for compatibility reasons simply be a symlink
-> profile.

Then users would simply directly modify their /etc/portage/profile 
profile, including its parent file, as desired, instead of having the 
/etc/portage/make.profile symlink, with /etc/portage/profile being yet 
another layer on top of that.

For all I know that might actually work right now as I've not tried it, 
but the portage (5) manpage does specifically say /etc/portage/profile 
supports all profile files EXCEPT parent, so unless the documentation is 
wrong...

Assuming the documentation is correct, however, "all" we'd need to do on 
the user side would be to make portage and the other tools treat the 
parent file in /etc/portage/profile just as they do in other profile dirs, 
create an eselect module or equivalent to help manage that parent file, 
and update the documentation including the handbook accordingly.

Updating other tree related tools would be required as well, of course, 
but as already pointed out, the real work would probably be in designing 
and setting up the new "flatter" profile tree layout and finding the 
appropriate new-layout location for all the existing settings.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[gentoo-dev] Re: new profile layout with flavors and mix-ins

[Martin Vaeth]

Michael Haubenwallner <haubi@gentoo.org> wrote:
>
>> The idea being that, in /etc/make.conf (or wherever
>> that file is now), you'd define $PROFILE like this:
>>
>> linux-mips o32 uclibc server:
>> PROFILE="base:kernel/linux:arch/mips:[...]"
>
> What about making /etc/portage/make.profile a directory
> rather than a symlink, having /etc/portage/make.profile/parent
> to reference all the flavours?

/etc/portage/make.profile already *is* a directory
(only on most user systems it is currently implemented as a symlink
to a directory from the portage tree, but there is no technical
requirement to do that).

I can assure you (from some requests I got as a maintainer of eix
to implement this correctly) that already quite some people
use such an approach:

They specify in the "parent" file (either in their
/etc/portage/make.profile or in /etc/portage/profiles/...)
all sort of additions to profiles which they have written themselves
for various tasks.

So, technically, all this is already covered by portage and
existing tools; no new magic "PROFILE" variable or similar things
have to implemented, and no tools (like eix) need to be changed.

It might, however, be appropriate to reorganize the profile structure
to support this approach better.

> So the only missing thing would be the eselect profile module
> to manage entries

Yes, if you want a user interface for convenient handling of
such a new profiles style, this would need to be written.
However, such a user interface ("eselect profile"?) is perhaps
overkill: The user should be able to handle a single configuration
file "parent" if it is described in the documentation.
The documentation would need to be updated, of course:
Currently, users have to read pms or similar things to understand
what the "parent" file is for...
Also a news item would be appropriate for such a major change, of course.

Re: [gentoo-dev] new profile layout with flavors and mix-ins

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 3519 bytes --]

Dnia 2014-07-03, o godz. 02:18:23
Joshua Kinard <kumba@gentoo.org> napisał(a):

> [...]
>
> And so on.  The goal was to have profiles/ be extremely flat, with limited
> nesting only for categorization purposes.  Each component would contain all
> of the information specific to that component, and rely on OOP-like
> inheritance to override parent-level variables only within that component
> (although, <arch> and <kernel> would have to be a little bit more broad in
> scope).

Another sub-thread, the same question: how do you handle information
that needs to be applied to the intersection of the profiles? CHOST for
amd64 FreeBSD, for example.

> PROFILE also had a set order for the first few pieces, if only to make
> parsing and building the profile stack saner for the Portage developers:
>    PROFILE="base:<KERNEL>:<ARCH>[:<SUBARCH>]:<LIBC>:<EVERYTHING_ELSE>"
> 
> base - Core Gentoo/Portage/whatever information/vars/foobar/kittens

I'm against plain 'base' since it quickly becomes a mess. I would
prefer having flavor-specific bases, like for example arch/base to mask
stuff available only in 1/2 arches and revert it in another arch/.

> kernel - Specifies the OS kernel.  At the time, only Linux existed, but
>          I *think* Debian was eyeballing kFreeBSD at the time.  So I *think*
>          I accounted for it back then.

What about kernel-ABI intersection? What about Prefix?

> arch - Machine arch-specific generic information -- doesn't handle
>        lower-level things like ISAs/ABIs/Endian, etc.
> 
> subarch - Defines items specific given to given subarch of a main arch.
>           Items under this directory would carry their parent arch's
>           name for clarity only.  Again, at the time, MIPS would have been
>           the only real user of this, and, back then, it would've dealt
>           with SGI-machine-specific packages and USE flags, such as
>           differentiating an ip32 machine from an ip27 machine or a
>           cobalt.  Now that amd64/x86_64 is considering its own form of
>           n32 (x32), that would go here, and I imagine arm would
>           make heavy use of it as well.

This is a no-go for multilib support. We need to work on a consistent
arch profile tree, not more splitting.

> libc - Defines the main C library used.  A bit redundant for *BSD, because
>        they really only have one, but might help if we ever add k*BSD
>        support (such as freebsd/glibc-based or such).  For Linux...could be
>        tricky, since there are so many libc possibilities there.  I feel it
>        fits better getting defined after <SUBARCH>, especially in the case
>        of MIPS.

I think you need to handle kernel & libc in the same group.

> eapi - EAPI didn't exist back when this topic was visited.  Basically,
>        everything was EAPI=0 and there was only Portage (Paludis nor
>        pkgcore had been invented yet).

And what is this supposed to do?

> releases - I don't think this existed back then.  How it'd operate
>            nowadays, I have no idea.  Probably would contain
>            version-specific maskings for specific @system packages
>            to facilitate upgrading, and help us if we ever tried to cut
>            actual, fixed release points again (like what FreeBSD does
>            w/ -RELEASE-x.y)

I don't think releases would make any sense without heavy intersection
with other profiles.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] new profile layout with flavors and mix-ins

[Joshua Kinard]

On 07/03/2014 03:32, Michał Górny wrote:
> Dnia 2014-07-03, o godz. 02:18:23
> Joshua Kinard <kumba@gentoo.org> napisał(a):
> 
>> [...]
>>
>> And so on.  The goal was to have profiles/ be extremely flat, with limited
>> nesting only for categorization purposes.  Each component would contain all
>> of the information specific to that component, and rely on OOP-like
>> inheritance to override parent-level variables only within that component
>> (although, <arch> and <kernel> would have to be a little bit more broad in
>> scope).
> 
> Another sub-thread, the same question: how do you handle information
> that needs to be applied to the intersection of the profiles? CHOST for
> amd64 FreeBSD, for example.

Well, as I stated, I thought the bulk of this up *years* ago (at least 6-7
years ago, probably longer).  The issues you're raising now weren't issues
back then.  I only updated it slightly based on what I know now about how we
do things and what things are supported in Gentoo.  I am only proposing it
as a base from which others to think about and either build upon or enhance,
or strike it down.


>> PROFILE also had a set order for the first few pieces, if only to make
>> parsing and building the profile stack saner for the Portage developers:
>>    PROFILE="base:<KERNEL>:<ARCH>[:<SUBARCH>]:<LIBC>:<EVERYTHING_ELSE>"
>>
>> base - Core Gentoo/Portage/whatever information/vars/foobar/kittens
> 
> I'm against plain 'base' since it quickly becomes a mess. I would
> prefer having flavor-specific bases, like for example arch/base to mask
> stuff available only in 1/2 arches and revert it in another arch/.

The only reason I put a 'base' folder in was so that where was some kind of
anchor point for Portage to build off of.  I didn't put any thought into
what it might actually contain.  For all intents and purposes, it might just
contain empty variable declarations, kinda like an abstract base class in an
OOP language.  By itself, it's unusable and must be overridden, but it's the
point from which everything derives.

Maybe that's not needed now in current Portage.  Dunno.


>> kernel - Specifies the OS kernel.  At the time, only Linux existed, but
>>          I *think* Debian was eyeballing kFreeBSD at the time.  So I *think*
>>          I accounted for it back then.
> 
> What about kernel-ABI intersection? What about Prefix?

I know very little about Prefix.  Any new profile system we create should
focus only what is in-tree *first*, then once we work the bugs and kinks
out, we can figure out how Prefixes will be incorporated.

But that said, what little I do know about Prefix is that it's Portage
ported to other OSes, like Solaris, MacOS, Interix, IRIX, etc.  I guess an
out-of-tree Prefix overlay would just define missing bits in its profiles/*
folder.  I.e., for Solaris, you'd have a kernel/solaris, libc/solaris-libc, etc.

I'm not the best at OOP-designs, but if we set the in-tree stuff up right,
it should be trivial for out-of-tree overlays to extend and override to
support things, without going down the endlessly-nested directory snafu that
the current profile system is.  In-tree should be as flat as possible,
Prefix and other external overlays can extend it out however far they want.

As for kernel-ABI intersection, can you expand on that a little more?  A
given kernel can support multiple ABIs simultaneously.  Using MIPS as the
classic example, a 32-bit MIPS kernel only does o32, but a 64-bit kernel
does n64 natively, and you can enable o32 and n32 support individually
within the configuration.  And MIPS has even more obscure ABIs that aren't
well-known, either.


>> arch - Machine arch-specific generic information -- doesn't handle
>>        lower-level things like ISAs/ABIs/Endian, etc.
>>
>> subarch - Defines items specific given to given subarch of a main arch.
>>           Items under this directory would carry their parent arch's
>>           name for clarity only.  Again, at the time, MIPS would have been
>>           the only real user of this, and, back then, it would've dealt
>>           with SGI-machine-specific packages and USE flags, such as
>>           differentiating an ip32 machine from an ip27 machine or a
>>           cobalt.  Now that amd64/x86_64 is considering its own form of
>>           n32 (x32), that would go here, and I imagine arm would
>>           make heavy use of it as well.
> 
> This is a no-go for multilib support. We need to work on a consistent
> arch profile tree, not more splitting.

Well, it'll have to be nested somehow.  A given arch can handle multiple
ABIs, so there should at least be a top-level arch/ folder that defines very
high-level, very generic things specific to a given arch.  How we fold the
multilib and multiple ABI stuff in probably needs a lot of thought.  It may
not be possible with this proposed idea, so, if not, we'll need to think of
something else.

Also keep in mind that with MIPS, we up the fun fun with the ISA
(Instruction Set Architecture) mess (mips1 to mips4, mips32r2, mips64r2,
etc), which are also mostly independent of the ABI (meaning, n32/n64 on
mips3 or greater, but o32 only for mips1 and mips2.  No idea about mips32r*
-- other MIPS devs will have to speak for those).


>> libc - Defines the main C library used.  A bit redundant for *BSD, because
>>        they really only have one, but might help if we ever add k*BSD
>>        support (such as freebsd/glibc-based or such).  For Linux...could be
>>        tricky, since there are so many libc possibilities there.  I feel it
>>        fits better getting defined after <SUBARCH>, especially in the case
>>        of MIPS.
> 
> I think you need to handle kernel & libc in the same group.

If this was BSD, maybe.  But Linux the kernel is independent from the
userland libc.  You can have Linux/uclibc, Linux/glibc, and even Linux/musl,
among others.  And as Debian has shown, kFreeBSD marries a FreeBSD kernel
with a GNU glibc-based userland (which probably chafes diehard BSD users).

Hence the purpose of me breaking them up that way was to make individual
elements as discrete as possible.  I.e., the old UNIX philosophy of having
lots of little things that each do one thing very, very well.  Not a few big
things that try to do many things poorly or halfway (like the IOC3 chip in
certain SGI systems, but I digress).

And, by organizing them one directory deep, it even adheres to our current
Portage philosophy of <category>/<package>, which also would keep
/usr/portage/profiles somewhat clean (once we got rid of the old stacking
system many years down the road).


>> eapi - EAPI didn't exist back when this topic was visited.  Basically,
>>        everything was EAPI=0 and there was only Portage (Paludis nor
>>        pkgcore had been invented yet).
> 
> And what is this supposed to do?

I dunno, I threw it in there just in case there was some nit about EAPIs
that I didn't know about that could be defined at a profile level.  If it's
not relevant, then ignore it.


>> releases - I don't think this existed back then.  How it'd operate
>>            nowadays, I have no idea.  Probably would contain
>>            version-specific maskings for specific @system packages
>>            to facilitate upgrading, and help us if we ever tried to cut
>>            actual, fixed release points again (like what FreeBSD does
>>            w/ -RELEASE-x.y)
> 
> I don't think releases would make any sense without heavy intersection
> with other profiles.

Possibly.  It's just another thing to think about, though.  Arguably, if we
ever migrate the tree to git, setting release-specific branches is better
done there, though.  But, it's just building on how we currently do
release-specific profiles in the tree.  Maybe it's not need anymore either.

Brainstorming doesn't mean everything I am proposing is right.  It might all
be wrong.  Some of it might be right.  Maybe it inspires someone to think of
something better.  But if I don't put it out there, it'll never get
discussed in the first place.

-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
4096R/D25D95E3 2011-03-28

"The past tempts us, the present confuses us, the future frightens us.  And
our lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic