[gentoo-dev] Questions on overlays, repositories and PMS

[gentoo-dev] Questions on overlays, repositories and PMS

[Michael Lienhardt]

I started refactoring my solver to make it more modular, to fix some details w.r.t. the PMS and to manage different repositories.
I thus have several questions on how multiple repositories work in portage.

1. My understanding was that /etc/portage/repos.conf replaced the PORTDIR_OVERLAY variable, however this variable is still documented (e.g. in https://devmanual.gentoo.org/general-concepts/overlay/index.html).
Was my intuition right?
Or in other word, it is enough to only look at /etc/portage/repos.conf?
In general, an overlay is a repository, i.e., a valid tree layout for the PMS, right (as stated in https://devmanual.gentoo.org/general-concepts/overlay/index.html)?

2. the PMS states that any valid repository has a profiles folder which can contain profiles and a package.mask file.
  - can the profiles in a repository different from DEFAULT be selected?
  - is the package.mask file apply only on the packages of that repository, or on every packages of every repositories listed in /etc/portage/repos.conf?

3. many repositories do not have an eclass folder, and miss many (optional) configuration files in the profiles folder (like arch.list, categories):
  - is such information implicitly inherited from the DEFAULT repository (even though https://wiki.gentoo.org/wiki//etc/portage/repos.conf states that it is not)?
      the brother overlay (https://github.com/stefan-langenmaier/brother-overlay) does not specify any masters
  - when the eclass folder, profiles/arch.list and such are present, is the data from the DEFAULT repository still implicitly inherited?
  - when the eclass folder, profiles/arch.list and such are present, are they visible globally (i.e., a package from another repository can use a keyword of the arch.list and inherit from one of the eclass)?

4. is the "masters" attribute in /etc/portage/repos.conf make the repository inherit other data than the eclasses?

5. since every repos can have a profiles/categories file, is the file /etc/portage/categories obsolete (or should it be)?


Best Regards,
Michael Lienhardt

Re: [gentoo-dev] Questions on overlays, repositories and PMS

[Vadim A. Misbakh-Soloviov]

> Or in other word, it is enough to only look at /etc/portage/repos.conf?
No

> In general, an overlay is a repository, i.e., a valid tree layout for the
Yes
 
> - can the profiles in a repository different from DEFAULT be selected?
Yes

> - is the package.mask file apply only on the packages of that repository, or on every packages of
> every repositories listed in /etc/portage/repos.conf?
Actually, I can't remember the correct answer right now, but definitelly it have the effect on repos, that states this repo as master.

> is such information implicitly inherited from the DEFAULT repository (even
> though https://wiki.gentoo.org/wiki//etc/portage/repos.conf states that it
> is not)?
Usually, that info is inerited from `master` repo of the current repo (that is stated in the layout conf file)

> the brother overlay (https://github.com/stefan-langenmaier/brother-overlay) does not specify
> any masters
Eeeerm?
https://github.com/stefan-langenmaier/brother-overlay/blob/master/metadata/layout.conf#L1

> - when the eclass folder, profiles/arch.list and such are
> present, is the data from the DEFAULT repository still implicitly
> inherited?
I still insist on inheritance from master repo.

> - when the eclass folder, profiles/arch.list and such are
> present, are they visible globally (i.e., a package from another repository
> can use a keyword of the arch.list and inherit from one of the eclass)?
AFAIRC, depends on the repos relative priority.

> 4. is the "masters" attribute in /etc/portage/repos.conf make the repository
> inherit other data than the eclasses?
Yes, but that attribut is usually not recommended for general use.

> 5. since every repos can have a profiles/categories file, is the file
> /etc/portage/categories obsolete (or should it be)?
Why?

Re: [gentoo-dev] Questions on overlays, repositories and PMS

[Michael Lienhardt]

Thank you for your answers :).
I have few followup questions.

Il 23/02/2018 19:09, Vadim A. Misbakh-Soloviov ha scritto:
>> Or in other word, it is enough to only look at /etc/portage/repos.conf?
> No

What else should I look for?

>> - is the package.mask file apply only on the packages of that repository, or on every packages of
>> every repositories listed in /etc/portage/repos.conf?
> Actually, I can't remember the correct answer right now, but definitelly it have the effect on repos, that states this repo as master.

If you get the correct answer, I'm really interested.

Also, what does mean "that states this repo as master"?
  - every repo that have a profiles/package.mask file is implicitly the master of every other repository?
  - there can be only one repo with a profiles/package.mask file?
  - other possible meaning?

>> is such information implicitly inherited from the DEFAULT repository (even
>> though https://wiki.gentoo.org/wiki//etc/portage/repos.conf states that it
>> is not)?
> Usually, that info is inerited from `master` repo of the current repo (that is stated in the layout conf file)

You're right, I forgot about the metadata/layout.conf

This makes me think: a repository can have four different names:
  - the name of the file in /etc/portage/repos.conf/
  - the name specified in the section of the /etc/portage/repos.conf/
  - the name in the metadata/layout.conf
  - the name in the profiles/repo_name

All these names should be equal, but what happens if they aren't?
In particular, which of the four names is used to resolve the masters dependency?

>> 4. is the "masters" attribute in /etc/portage/repos.conf make the repository
>> inherit other data than the eclasses?
> Yes, but that attribute is usually not recommended for general use.

I guess the masters attribute in /etc/portage/repos.conf overrides the one in the metadata/layout.conf, and that's why it is not recommended, right?
Which other data is inherited?

Also, what happen when some inherited data is also declared locally?
  - for instance when a master and the local repository both share the same eclass?
  - or if the profiles/categories is inherited but also declared locally: does the local declaration override the one from the masters, or all the category lists are merged together?

>> 5. since every repos can have a profiles/categories file, is the file
>> /etc/portage/categories obsolete (or should it be)?
> Why?

Because this file is useless if every repository declare its own categories (that's what I understand from the documentation https://wiki.gentoo.org/wiki//etc/portage/categories).
So I guess it is not deprecated.

My experience is that it is far more maintainable to have the repositories not depending on unreliable external data (the /etc/portage/categories file) to work.
Hence, the only case I can see where it is not necessary for a repository to have its own local profiles/categories file (or having it inherited from some master) is when the repository is user-defined and not distributed to anyone.
But in this case too, it makes more sense to me to have repository-specific data declared locally in the repository itself.


Best,
Michael Lienhardt

Re: [gentoo-dev] Questions on overlays, repositories and PMS

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 3082 bytes --]

On Fri, Feb 23, 2018 at 12:36 PM, Michael Lienhardt <
michael.lienhardt@laposte.net> wrote:

> I started refactoring my solver to make it more modular, to fix some
> details w.r.t. the PMS and to manage different repositories.
> I thus have several questions on how multiple repositories work in portage.
>
> 1. My understanding was that /etc/portage/repos.conf replaced the
> PORTDIR_OVERLAY variable, however this variable is still documented (e.g.
> in https://devmanual.gentoo.org/general-concepts/overlay/index.html).
> Was my intuition right?
> Or in other word, it is enough to only look at /etc/portage/repos.conf?
> In general, an overlay is a repository, i.e., a valid tree layout for the
> PMS, right (as stated in https://devmanual.gentoo.org/g
> eneral-concepts/overlay/index.html)?
>
> 2. the PMS states that any valid repository has a profiles folder which
> can contain profiles and a package.mask file.
>  - can the profiles in a repository different from DEFAULT be selected?
>  - is the package.mask file apply only on the packages of that repository,
> or on every packages of every repositories listed in
> /etc/portage/repos.conf?
>
> 3. many repositories do not have an eclass folder, and miss many
> (optional) configuration files in the profiles folder (like arch.list,
> categories):
>  - is such information implicitly inherited from the DEFAULT repository
> (even though https://wiki.gentoo.org/wiki//etc/portage/repos.conf states
> that it is not)?
>      the brother overlay (https://github.com/stefan-lan
> genmaier/brother-overlay) does not specify any masters
>  - when the eclass folder, profiles/arch.list and such are present, is the
> data from the DEFAULT repository still implicitly inherited?
>  - when the eclass folder, profiles/arch.list and such are present, are
> they visible globally (i.e., a package from another repository can use a
> keyword of the arch.list and inherit from one of the eclass)?
>
> 4. is the "masters" attribute in /etc/portage/repos.conf make the
> repository inherit other data than the eclasses?
>
> 5. since every repos can have a profiles/categories file, is the file
> /etc/portage/categories obsolete (or should it be)?
>
>
My general observation is that Gentoo is not successful as an organization
about deprecating and removing things. One area where Gentoo has done well
is in EAPI and in PMS itself, with mostly-clear versioning and standards
and whatnot. But in general if something worked 15 years ago, it probably
still works today (doubly so for sys-apps/portage).

There is a different question when building a tool like yours if it is
worth the effort to support things that are 15 years old and are possibly
not used (particularly in cases where functionality was replaced). I'd
recommend starting with the basic implementation and adding support for the
'older' formats when users ask for them; but this is mostly a trade-off in
efforts. If your goal is to build a "100% compatible" tool then you will
probably need to support these edge cases.

-A


>
> Best Regards,
> Michael Lienhardt
>
>
>

[-- Attachment #2: Type: text/html, Size: 4133 bytes --]

Re: [gentoo-dev] Questions on overlays, repositories and PMS

[Michael Lienhardt]

Il 23/02/2018 20:37, Alec Warner ha scritto:
> My general observation is that Gentoo is not successful as an organization about deprecating and removing things. One area where Gentoo has done well is in EAPI and in PMS itself, with mostly-clear versioning and standards and whatnot. But in general if something worked 15 years ago, it probably still works today (doubly so for sys-apps/portage).
> 
> There is a different question when building a tool like yours if it is worth the effort to support things that are 15 years old and are possibly not used (particularly in cases where functionality was replaced). I'd recommend starting with the basic implementation and adding support for the 'older' formats when users ask for them; but this is mostly a trade-off in efforts. If your goal is to build 
> a "100% compatible" tool then you will probably need to support these edge cases.

You have a very good point.
I'd like to be complete (it's a side effect of working in formal methods), but it's quite unrealistic as I am the only developer in this project, and it's true that there are few technical design choices that were made in portage that I'd be happier not to implement.
I'd like to implement the /etc/portage/repos.conf system to remove as many hard coded references to /usr/portage in my code as possible.
Moreover, the /etc/portage/repos.conf system looks nice, modular with explicit dependencies and it almost unifies all the repositories (I don't really understand the need of a DEFAULT section).

If possible, I'd rather avoid implementing things that are deprecated, but like you pointed out, few are (portage seems to be always expanding with new/alternative functionalities).
The ones that are, like the /etc/portage/package.keywords file, seem to be still used (I've got a request to support it in my get_installation.sh script).
Additionally, there are two systems that I did not want to implement but had to: the IUSE_IMPLICIT and USE_EXPAND.
I didn't find any good documentation on these systems (nor the PMS nor https://dev.gentoo.org/%7Ezmedico/portage/doc/man/portage.5.html are very clear on the subject -- the PMS is still clearer), I tested a lot and looked at the portage implementation...
I don't understand the reason to implement these systems with bash variables expanded with prefixes, while many of the USE flag manipulation is done with dedicated files (use.*, package.use.*).
It really felt like an old design choice kept there because it worked, but which could be simplified.

On a similar topic, does anyone still have USE-related variables in his /etc/env.d folder? (https://wiki.gentoo.org/wiki/USE_ORDER)
It seems to me that portage's current effort is to have all configuration files in /etc/portage or in the profile.

Best,
Michael Lienhardt

Re: [gentoo-dev] Questions on overlays, repositories and PMS

[Roy Bamford]

[-- Attachment #1: Type: text/plain, Size: 3365 bytes --]

On 2018.02.24 01:32, Michael Lienhardt wrote:
> 
> 
> Il 23/02/2018 20:37, Alec Warner ha scritto:
> > My general observation is that Gentoo is not successful as an
> organization about deprecating and removing things. One area where
> Gentoo has done well is in EAPI and in PMS itself, with mostly-clear
> versioning and standards and whatnot. But in general if something
> worked 15 years ago, it probably still works today (doubly so for
> sys-apps/portage).
> > 
> > There is a different question when building a tool like yours if it
> is worth the effort to support things that are 15 years old and are
> possibly not used (particularly in cases where functionality was
> replaced). I'd recommend starting with the basic implementation and
> adding support for the 'older' formats when users ask for them; but
> this is mostly a trade-off in efforts. If your goal is to build 
> > a "100% compatible" tool then you will probably need to support
> these edge cases.
> 
> You have a very good point.
> I'd like to be complete (it's a side effect of working in formal
> methods), but it's quite unrealistic as I am the only developer in
> this project, and it's true that there are few technical design
> choices that were made in portage that I'd be happier not to
> implement.
> I'd like to implement the /etc/portage/repos.conf system to remove as
> many hard coded references to /usr/portage in my code as possible.
> Moreover, the /etc/portage/repos.conf system looks nice, modular with
> explicit dependencies and it almost unifies all the repositories (I
> don't really understand the need of a DEFAULT section).
> 
> If possible, I'd rather avoid implementing things that are deprecated,
> but like you pointed out, few are (portage seems to be always
> expanding with new/alternative functionalities).
> The ones that are, like the /etc/portage/package.keywords file, seem
> to be still used (I've got a request to support it in my
> get_installation.sh script).
> Additionally, there are two systems that I did not want to implement
> but had to: the IUSE_IMPLICIT and USE_EXPAND.
> I didn't find any good documentation on these systems (nor the PMS nor
> https://dev.gentoo.org/%7Ezmedico/portage/doc/man/portage.5.html are
> very clear on the subject -- the PMS is still clearer), I tested a lot
> and looked at the portage implementation...
> I don't understand the reason to implement these systems with bash
> variables expanded with prefixes, while many of the USE flag
> manipulation is done with dedicated files (use.*, package.use.*).
> It really felt like an old design choice kept there because it worked,
> but which could be simplified.
> 
> On a similar topic, does anyone still have USE-related variables in
> his /etc/env.d folder? (https://wiki.gentoo.org/wiki/USE_ORDER)
> It seems to me that portage's current effort is to have all
> configuration files in /etc/portage or in the profile.
> 
> Best,
> Michael Lienhardt
> 
> 
> 
Michael,

'Support' can be as simple as nagging the user to move with the 
times and failing. 

I suspect that many older systems (including mine) are not updated
because it still works.

crossdev users may be familiar with this approach. 

-- 
Regards,

Roy Bamford
(Neddyseagoon) a member of
elections
gentoo-ops
forum-mods

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Questions on overlays, repositories and PMS

[Brian Dolbec]

On Fri, 23 Feb 2018 18:36:07 +0100
Michael Lienhardt <michael.lienhardt@laposte.net> wrote:

> I started refactoring my solver to make it more modular, to fix some
> details w.r.t. the PMS and to manage different repositories. I thus
> have several questions on how multiple repositories work in portage.
> 
> 1. My understanding was that /etc/portage/repos.conf replaced the
> PORTDIR_OVERLAY variable, however this variable is still documented
> (e.g. in
> https://devmanual.gentoo.org/general-concepts/overlay/index.html).
> Was my intuition right? Or in other word, it is enough to only look
> at /etc/portage/repos.conf? In general, an overlay is a repository,
> i.e., a valid tree layout for the PMS, right (as stated in
> https://devmanual.gentoo.org/general-concepts/overlay/index.html)?
> 

Yes, you should not have to look at the old make.conf variables, yes,
they do work, but mostly for backward compatibility with some old third
party tools.  Those really need to be updated.  I forget now how many
there are, but if I remember correctly only 2 or 3 are left.  And yes,
we have been slow to get them updated or replaced.


> 2. the PMS states that any valid repository has a profiles folder
> which can contain profiles and a package.mask file.
>   - can the profiles in a repository different from DEFAULT be
> selected?
>   - is the package.mask file apply only on the packages of that
> repository, or on every packages of every repositories listed
> in /etc/portage/repos.conf?
> 
> 3. many repositories do not have an eclass folder, and miss many
> (optional) configuration files in the profiles folder (like
> arch.list, categories):
>   - is such information implicitly inherited from the DEFAULT
> repository (even though
> https://wiki.gentoo.org/wiki//etc/portage/repos.conf states that it
> is not)? the brother overlay
> (https://github.com/stefan-langenmaier/brother-overlay) does not
> specify any masters
>   - when the eclass folder, profiles/arch.list and such are present,
> is the data from the DEFAULT repository still implicitly inherited?
>   - when the eclass folder, profiles/arch.list and such are present,
> are they visible globally (i.e., a package from another repository
> can use a keyword of the arch.list and inherit from one of the
> eclass)?
> 
> 4. is the "masters" attribute in /etc/portage/repos.conf make the
> repository inherit other data than the eclasses?
> 
> 5. since every repos can have a profiles/categories file, is the
> file /etc/portage/categories obsolete (or should it be)?
> 
> 
> Best Regards,
> Michael Lienhardt
> 
> 



-- 
Brian Dolbec <dolsen>