[gentoo-dev] rfc: moving default location of portage tree (was: [gentoo-project] Call for agenda items - Council meeting 2018-07-29)

[gentoo-dev] rfc: moving default location of portage tree (was: [gentoo-project] Call for agenda items - Council meeting 2018-07-29)

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 2928 bytes --]

[Moving the thread back from -project to -dev.]

>>>>> On Fri, 13 Jul 2018, Ulrich Mueller wrote:

>>>>> On Fri, 13 Jul 2018, Rich Freeman wrote:
>> Well, /var/lib/<something> is 3 right there.  If 5 is no good then you
>> only have one left.  We could just make it /var/lib/repos which seems
>> non-compliant.

> FHS 3.0 says: "An application (or a group of inter-related
> applications) must use a subdirectory of /var/lib for its data".
> Certainly /var/lib/repos is a subdirectory of /var/lib? So why would
> it be non-compliant? And if it was, do we care about non-compliance at
> the third directory level? The important part is that we move it out
> of /usr, and IMHO we should care to get the /var/{lib,cache,db} part
> somewhat right.

In the same section 5.8.1 (emphasis is mine):
"This hierarchy holds state information pertaining to an application
or the system. State information is data that programs modify while
they run, and that _pertains_to_one_specific_host._ Users must never
need to modify files in /var/lib to configure a package's operation,
and _the_specific_file_hierarchy_ used to store the data _must_not_be_
_exposed_ to regular users."

"Pertains to one specific host" doesn't seem to apply to the Gentoo
repository though. Also, there is that strange requirement that the
file hierarchy must not be exposed to users. At least for the
make.profile link we rely on a well defined hierarchy, and certainly
expose it to users. The same is true for license information in
/usr/portage/licenses. So I would conclude that using /var/lib does
not comply with the FHS.

It was mentioned that all three directories (ebuild repository, binary
packages, distfiles) have some characteristics of a cache. However, I
think this is much more true for distfiles than for the other two,
which cannot be easily restored to a previous state.

I therefore suggest the following scheme:

/usr/portage -> /var/db/repos/gentoo

This follows the existing usage in eselect-repositories. Note that
/var/db is not specified by the FHS (though it is mentioned in a
footnote [1]) but used by all current BSD variants. Plus, we already
have /var/db/pkgs and (as pointed out by antarus) we won't get rid of
it anytime soon.

/usr/portage/packages -> /var/db/binpkgs

This keeps binary packages along with ebuilds. As suggested by mgorny,
rename the "packages" directory to "binpkgs", because it is more
specific and avoids confusion with "pkgs".

/usr/portage/distfiles -> /var/cache/distfiles

Omitting the "portage" subdirectory level here, as distfiles aren't
specific to any package manager. We could use an intermediate
"package-manager" directory level, but then again, it would have only
one single subdirectory. The FHS isn't very specific about /var/cache
but mentions /var/cache/fonts as an example which appears to be
similar.

Ulrich

[1] https://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#ftn.idm236086558032

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] rfc: moving default location of portage tree (was: [gentoo-project] Call for agenda items - Council meeting 2018-07-29)

[Kristian Fiskerstrand]

[-- Attachment #1.1: Type: text/plain, Size: 276 bytes --]

On 07/18/2018 11:55 AM, Ulrich Mueller wrote:
> I therefore suggest the following scheme:

The full scheme looks good to me

-- 
Kristian Fiskerstrand
OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] rfc: moving default location of portage tree (was: [gentoo-project] Call for agenda items - Council meeting 2018-07-29)

[Rich Freeman]

On Wed, Jul 18, 2018 at 5:55 AM Ulrich Mueller <ulm@gentoo.org> wrote:
>
> "Pertains to one specific host" doesn't seem to apply to the Gentoo
> repository though.

Sure it does.  The state of the package repository on a Gentoo host
doesn't affect any other host.

Sure, that state is synced from someplace that many other hosts also
sync from, so the state of THAT version of the repository does affect
many hosts.

However, if you do an emerge --sync on one host, and then an hour
later do an emerge --sync on another host, the state of either local
repository won't affect the other.

When they talk about things that pertain to multiple hosts they're
talking about situations where a single path is often mounted across
many hosts simultaneously, eg using NFS.  An example of this given in
FHS is /var/mail.

Now, you could share the repository across multiple hosts, but that is
not a typical configuration.  Really, just about anything on the
filesystem could potentially be shared across multiple hosts.

> Also, there is that strange requirement that the
> file hierarchy must not be exposed to users. At least for the
> make.profile link we rely on a well defined hierarchy, and certainly
> expose it to users.

That isn't true at all.  When you run eselect profile you have no idea
what it is looking at.

The fact that some of our users like to poke around the internals of
the package manager doesn't change the fact that it has interfaces
that don't require direct modification.

> The same is true for license information in
> /usr/portage/licenses.

You have a fair point there.  Honestly, I don't really see this as a
good reason on its own to abandon /var/lib, which is where other
distros seem to stick stuff like this.  Also, you don't need to poke
around that directory to determine what license a package uses - just
to read the text of the license itself (which could arguably just be
stored on our webserver unless we're actually redistributing something
in the repository under that license, which is unlikely in general
since patches/etc are probably fair use).

> This follows the existing usage in eselect-repositories. Note that
> /var/db is not specified by the FHS (though it is mentioned in a
> footnote [1]) but used by all current BSD variants. Plus, we already
> have /var/db/pkgs and (as pointed out by antarus) we won't get rid of
> it anytime soon.

So, you reject one path on the basis of small conflicts with FHS, and
then just toss FHS out the window entirely to use a path not specified
at all?

If you want to appeal to what BSD is doing then /usr/portage belongs
right where it is today.  Oh, and most of our packages should be
installing to /usr/local as well.

What other linux distro even has /var/db at all?

> /usr/portage/packages -> /var/db/binpkgs

Why not put this in /var/cache?  It is completely reproducible and
exists to save build time.  That's what a cache is.

Or put it in /var/lib.  Certainly your objections above don't apply to
binpkgs.

-- 
Rich

Re: [gentoo-dev] rfc: moving default location of portage tree (was: [gentoo-project] Call for agenda items - Council meeting 2018-07-29)

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 3947 bytes --]

>>>>> On Wed, 18 Jul 2018, Rich Freeman wrote:

> On Wed, Jul 18, 2018 at 5:55 AM Ulrich Mueller <ulm@gentoo.org> wrote:
>> Also, there is that strange requirement that the
>> file hierarchy must not be exposed to users. At least for the
>> make.profile link we rely on a well defined hierarchy, and certainly
>> expose it to users.

> That isn't true at all.  When you run eselect profile you have no idea
> what it is looking at.

If I run eselect profile, it shows a list of pathnames like
default/linux/amd64/17.1/desktop. If that doesn't expose the "specific
file hierarchy" then I wonder what could ever qualify?

Also eselect profile is a tool for convenience only, and nobody is
forced to use it. The make.profile symlink and its target are
mentioned in our documentation and users can set it manually.

> The fact that some of our users like to poke around the internals of
> the package manager doesn't change the fact that it has interfaces
> that don't require direct modification.

Hopefully we will keep having such users who want to understand the
inner workings, instead of being satisfied with a black box. :)
Ebuild repositories are human readable text, and we shouldn't move
them to a hidden location.

>> The same is true for license information in
>> /usr/portage/licenses.

> You have a fair point there.  Honestly, I don't really see this as a
> good reason on its own to abandon /var/lib, which is where other
> distros seem to stick stuff like this.  Also, you don't need to poke
> around that directory to determine what license a package uses - just
> to read the text of the license itself (which could arguably just be
> stored on our webserver unless we're actually redistributing something
> in the repository under that license, which is unlikely in general
> since patches/etc are probably fair use).

I totally disagree here. We keep local copies of licenses for good
reasons, and storing them on our webserver is no substitute.

>> This follows the existing usage in eselect-repositories. Note that
>> /var/db is not specified by the FHS (though it is mentioned in a
>> footnote [1]) but used by all current BSD variants. Plus, we already
>> have /var/db/pkgs and (as pointed out by antarus) we won't get rid of
>> it anytime soon.

> So, you reject one path on the basis of small conflicts with FHS, and
> then just toss FHS out the window entirely to use a path not specified
> at all?

No, the argument is that we already use /var/db/pkg, and grouping
other related things with it seems natural.

> If you want to appeal to what BSD is doing then /usr/portage belongs
> right where it is today.  Oh, and most of our packages should be
> installing to /usr/local as well.

As I said before, the two important parts are the move from /usr to
/var, and that distfiles and packages are moved out of the repository.

Apart from this (quoting the devmanual): "Gentoo does not consider
the Filesystem Hierarchy Standard to be an authoritative standard,
although much of our policy coincides with it." And the discussion so
far has shown that the FHS wasn't designed for our use case. We can
still use it as a guideline, but maybe we shouldn't jump through
hoops, only to make it completely fit.

> What other linux distro even has /var/db at all?

Many (most?) of those that build from source. Also, Gentoo is not only
a Linux distro.

>> /usr/portage/packages -> /var/db/binpkgs

> Why not put this in /var/cache?  It is completely reproducible and
> exists to save build time.  That's what a cache is.

*shrug* I don't have a strong opinion about this. To me, binpkgs look
less cache-like than distfiles, but more cache-like than ebuild
repositories. So /var/cache/binpkgs would work for me as well.

The only thing that seems certain is that regardless of what we will
do, we cannot make everybody happy. We shouldn't take that as an
excuse to do nothing, because /usr/portage is no good solution either.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] rfc: moving default location of portage tree (was: [gentoo-project] Call for agenda items - Council meeting 2018-07-29)

[Rich Freeman]

On Wed, Jul 18, 2018 at 9:30 AM Ulrich Mueller <ulm@gentoo.org> wrote:
>
> >>>>> On Wed, 18 Jul 2018, Rich Freeman wrote:
>
> > On Wed, Jul 18, 2018 at 5:55 AM Ulrich Mueller <ulm@gentoo.org> wrote:
> >> Also, there is that strange requirement that the
> >> file hierarchy must not be exposed to users. At least for the
> >> make.profile link we rely on a well defined hierarchy, and certainly
> >> expose it to users.
>
> > That isn't true at all.  When you run eselect profile you have no idea
> > what it is looking at.
>
> If I run eselect profile, it shows a list of pathnames like
> default/linux/amd64/17.1/desktop. If that doesn't expose the "specific
> file hierarchy" then I wonder what could ever qualify?

It lists profile names.  The fact that our profiles are implemented
using paths is an implementation detail.  And you would have no idea
that they're stored in /usr or /var from the output of eselect.  If we
modified PMS so that profiles were stored in a mysql database the
current eselect output could work just fine.

> Also eselect profile is a tool for convenience only, and nobody is
> forced to use it. The make.profile symlink and its target are
> mentioned in our documentation and users can set it manually.

So is /var/lib/portage/world.  Are you planning to move that to /etc?
(I wouldn't object to that, actually.)

Personally I think that a lot of this comes from the standpoint of a
typical distro where command lines are things that should be hidden as
deeply in the gnome menus as possible.  But, my point is that if you
want that kind of experience on Gentoo it is certainly possible
insofar as profiles are concerned.

> Hopefully we will keep having such users who want to understand the
> inner workings, instead of being satisfied with a black box. :)
> Ebuild repositories are human readable text, and we shouldn't move
> them to a hidden location.

There is nothing hidden about /var/lib.  And I also prefer editing
text files for the most part.  My point is that FHS wasn't really
written with Gentoo as its main target, and that is likely the reason
for the bit about hiding paths.

>
> >> The same is true for license information in
> >> /usr/portage/licenses.
>
> > You have a fair point there.  Honestly, I don't really see this as a
> > good reason on its own to abandon /var/lib, which is where other
> > distros seem to stick stuff like this.  Also, you don't need to poke
> > around that directory to determine what license a package uses - just
> > to read the text of the license itself (which could arguably just be
> > stored on our webserver unless we're actually redistributing something
> > in the repository under that license, which is unlikely in general
> > since patches/etc are probably fair use).
>
> I totally disagree here. We keep local copies of licenses for good
> reasons, and storing them on our webserver is no substitute.

I'm not sure what you're disagreeing with.  I'm not advocating for not
storing them locally.  I'm just saying it could be done, and I did say
that you had a fair point.

> No, the argument is that we already use /var/db/pkg, and grouping
> other related things with it seems natural.

Perhaps we should be trying to move away from using /var/db, and not
doubling down?

> Apart from this (quoting the devmanual): "Gentoo does not consider
> the Filesystem Hierarchy Standard to be an authoritative standard,
> although much of our policy coincides with it." And the discussion so
> far has shown that the FHS wasn't designed for our use case. We can
> still use it as a guideline, but maybe we shouldn't jump through
> hoops, only to make it completely fit.

Sure, I'd agree with that, but why not use /var/lib then?  Your only
argument against it is based on the FHS, which you freely concede
wasn't written to fit our use case.

> The only thing that seems certain is that regardless of what we will
> do, we cannot make everybody happy. We shouldn't take that as an
> excuse to do nothing, because /usr/portage is no good solution either.

Well, the one thing I like about all these proposals is that it will
result in Gentoo systems with a wide mixture of locations for these
things, which will guarantee that any script that hard-codes paths
will break.  That means that those who exercise their choice to do
things in a more sane way than our defaults will probably run into
less trouble...

-- 
Rich

Re: [gentoo-dev] rfc: moving default location of portage tree (was: [gentoo-project] Call for agenda items - Council meeting 2018-07-29)

[Christopher Head]

[-- Attachment #1: Type: text/plain, Size: 781 bytes --]

On July 18, 2018 2:55:55 AM PDT, Ulrich Mueller <ulm@gentoo.org> wrote:
>It was mentioned that all three directories (ebuild repository, binary
>packages, distfiles) have some characteristics of a cache. However, I
>think this is much more true for distfiles than for the other two,
>which cannot be easily restored to a previous state.

Neither can a Web browser’s cache, if the remote page has changed, yet those are still called caches. Surely a cache is something that can safely be discarded without negative impact (which, for most users, the ebuild tree can be, since for most users, getting back today’s tree rather than last week’s is not a negative impact), not something that can be restored to exactly its prior state automatically.

-- 
Christopher Head

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 244 bytes --]

[gentoo-dev] Re: rfc: moving default location of portage tree

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1844 bytes --]

>>>>> On Wed, 18 Jul 2018, Christopher Head wrote:

> On July 18, 2018 2:55:55 AM PDT, Ulrich Mueller <ulm@gentoo.org> wrote:
>> It was mentioned that all three directories (ebuild repository,
>> binary packages, distfiles) have some characteristics of a cache.
>> However, I think this is much more true for distfiles than for the
>> other two, which cannot be easily restored to a previous state.

> Neither can a Web browser’s cache, if the remote page has changed,
> yet those are still called caches. Surely a cache is something that
> can safely be discarded without negative impact

In my understanding, a cache is typically an open collection of items.
Some subset of them can be deleted without much negative consequence,
and there may also be surplus items that are no longer necessary and
will be expired at some later time in order to reclaim disk space.

Nothing of this is true for an ebuild repository, which is a closed
collection of files: A single file cannot be discarded without
invalidating the whole repository. Also there cannot be any stray
files which would be expired later. Same as above, a single stray file
will invalidate all.

(A collection of binary packages may qualify as a cache though, by
this definition.)

> (which, for most users, the ebuild tree can be, since for most
> users, getting back today’s tree rather than last week’s is not a
> negative impact), not something that can be restored to exactly its
> prior state automatically.

Consider a production system that is only updated at certain well
defined times. Its system administrator may still want to install one
additional package, or flip a use flag for another. In this scenario,
the gentoo repository is a precious resource, and restoring it to a
different state would not be acceptable.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] rfc: moving default location of portage tree (was: [gentoo-project] Call for agenda items - Council meeting 2018-07-29)

[Chí-Thanh Christopher Nguyễn]

Ulrich Mueller schrieb:
> Users must never
> need to modify files in /var/lib to configure a package's operation,
> and _the_specific_file_hierarchy_ used to store the data _must_not_be_
> _exposed_ to regular users."

One small note, while it is never needed to modify, skel.ebuild would then be 
a file that is meant to be accessed by users in /var/lib if your proposal is 
realized.


Best regards,
Chí-Thanh Christopher Nguyễn

Re: [gentoo-dev] rfc: moving default location of portage tree (was: [gentoo-project] Call for agenda items - Council meeting 2018-07-29)

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1789 bytes --]

>>>>> On Thu, 19 Jul 2018, Chí-Thanh Christopher Nguyễn wrote:

>> Users must never need to modify files in /var/lib to configure a
>> package's operation, and _the_specific_file_hierarchy_ used to
>> store the data _must_not_be_ _exposed_ to regular users."

> One small note, while it is never needed to modify, skel.ebuild
> would then be a file that is meant to be accessed by users in
> /var/lib if your proposal is realized.

That's one of the reasons why the proposal prefers /var/db. The other
reason is existing usage in eselect-repository.

>>>>> On Thu, 19 Jul 2018, Ulrich Mueller wrote:

> In my understanding, a cache is typically an open collection of items.
> Some subset of them can be deleted without much negative consequence,
> and there may also be surplus items that are no longer necessary and
> will be expired at some later time in order to reclaim disk space.

> Nothing of this is true for an ebuild repository, which is a closed
> collection of files: A single file cannot be discarded without
> invalidating the whole repository. Also there cannot be any stray
> files which would be expired later. Same as above, a single stray file
> will invalidate all.

> (A collection of binary packages may qualify as a cache though, by
> this definition.)

So, considering all the feedback from mailing list and IRC:

   /usr/portage           -> /var/db/repos/gentoo
   /usr/portage/distfiles -> /var/cache{,/gentoo}/distfiles
   /usr/portage/packages  -> /var/cache{,/gentoo}/binpkgs

Open question: Should we have the additional "gentoo" path component
for the ones in /var/cache? The tradeoff is between a path that is
easier to type, or slightly easier usage if someone wants to NFS mount
distfiles and binpkgs.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] rfc: moving default location of portage tree (was: [gentoo-project] Call for agenda items - Council meeting 2018-07-29)

[Michał Górny]

Dnia 27 lipca 2018 10:32:17 CEST, Ulrich Mueller <ulm@gentoo.org> napisał(a):
>>>>>> On Thu, 19 Jul 2018, Chí-Thanh Christopher Nguyễn wrote:
>
>>> Users must never need to modify files in /var/lib to configure a
>>> package's operation, and _the_specific_file_hierarchy_ used to
>>> store the data _must_not_be_ _exposed_ to regular users."
>
>> One small note, while it is never needed to modify, skel.ebuild
>> would then be a file that is meant to be accessed by users in
>> /var/lib if your proposal is realized.
>
>That's one of the reasons why the proposal prefers /var/db. The other
>reason is existing usage in eselect-repository.
>
>>>>>> On Thu, 19 Jul 2018, Ulrich Mueller wrote:
>
>> In my understanding, a cache is typically an open collection of
>items.
>> Some subset of them can be deleted without much negative consequence,
>> and there may also be surplus items that are no longer necessary and
>> will be expired at some later time in order to reclaim disk space.
>
>> Nothing of this is true for an ebuild repository, which is a closed
>> collection of files: A single file cannot be discarded without
>> invalidating the whole repository. Also there cannot be any stray
>> files which would be expired later. Same as above, a single stray
>file
>> will invalidate all.
>
>> (A collection of binary packages may qualify as a cache though, by
>> this definition.)
>
>So, considering all the feedback from mailing list and IRC:
>
>   /usr/portage           -> /var/db/repos/gentoo
>   /usr/portage/distfiles -> /var/cache{,/gentoo}/distfiles
>   /usr/portage/packages  -> /var/cache{,/gentoo}/binpkgs
>
>Open question: Should we have the additional "gentoo" path component
>for the ones in /var/cache? The tradeoff is between a path that is
>easier to type, or slightly easier usage if someone wants to NFS mount
>distfiles and binpkgs.

Note that NFS is not exactly clear cut here since binpkgs are not portable to different hosts, so you can have multiple variants of it.

>
>Ulrich


-- 
Best regards,
Michał Górny (by phone)

Re: [gentoo-dev] rfc: moving default location of portage tree (was: [gentoo-project] Call for agenda items - Council meeting 2018-07-29)

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1186 bytes --]

On Fri, Jul 27, 2018 at 10:32:17AM +0200, Ulrich Mueller wrote:
> So, considering all the feedback from mailing list and IRC:
> 
>    /usr/portage           -> /var/db/repos/gentoo
>    /usr/portage/distfiles -> /var/cache{,/gentoo}/distfiles
>    /usr/portage/packages  -> /var/cache{,/gentoo}/binpkgs
> 
> Open question: Should we have the additional "gentoo" path component
> for the ones in /var/cache? The tradeoff is between a path that is
> easier to type, or slightly easier usage if someone wants to NFS mount
> distfiles and binpkgs.

Section 5.5.2 describes the directory structure of /var/cache. These
paths are all optional [1].

/var/cache/fonts
/var/cache/man
/var/cache/www
/var/cache/<package>

Gentoo isn't a package, so I don't think /var/cache/gentoo/* is
appropriate. Here is my proposal:

/usr/portage           -> /var/db/repos/gentoo
/usr/portage/distfiles -> /var/cache/portage/distfiles
/usr/portage/packages  -> /var/cache/portage/binpkgs

I'm not 100% comfortable with /var/db, but I don't have any better
suggestion either.

William

[1] http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#varcacheApplicationCacheData

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] rfc: moving default location of portage tree (was: [gentoo-project] Call for agenda items - Council meeting 2018-07-29)

[Corentin “Nado” Pazdera]

July 27, 2018 4:07 PM, "William Hubbs" <williamh@gentoo.org> wrote:

> On Fri, Jul 27, 2018 at 10:32:17AM +0200, Ulrich Mueller wrote:
> 
>> So, considering all the feedback from mailing list and IRC:
>> 
>> /usr/portage -> /var/db/repos/gentoo
>> /usr/portage/distfiles -> /var/cache{,/gentoo}/distfiles
>> /usr/portage/packages -> /var/cache{,/gentoo}/binpkgs
>> 
>> Open question: Should we have the additional "gentoo" path component
>> for the ones in /var/cache? The tradeoff is between a path that is
>> easier to type, or slightly easier usage if someone wants to NFS mount
>> distfiles and binpkgs.
> 
> Section 5.5.2 describes the directory structure of /var/cache. These
> paths are all optional [1].
> 
> /var/cache/fonts
> /var/cache/man
> /var/cache/www
> /var/cache/<package>
> 
> Gentoo isn't a package, so I don't think /var/cache/gentoo/* is
> appropriate. Here is my proposal:
> 
> /usr/portage -> /var/db/repos/gentoo
> /usr/portage/distfiles -> /var/cache/portage/distfiles
> /usr/portage/packages -> /var/cache/portage/binpkgs
> 
> I'm not 100% comfortable with /var/db, but I don't have any better
> suggestion either.
> 
> William
> 
> [1] http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#varcacheApplicationCacheData

From the same source
"No other requirements are made on the data format of the cache directories."
And as you have quoted it, everything under /var/cache is optional.

So anything which doesn't conflict with another package seems fine according to FHS.

--
Corentin “Nado” Pazdera

Re: [gentoo-dev] rfc: moving default location of portage tree (was: [gentoo-project] Call for agenda items - Council meeting 2018-07-29)

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1407 bytes --]

>>>>> On Fri, 27 Jul 2018, Corentin “Nado” Pazdera wrote:

> July 27, 2018 4:07 PM, "William Hubbs" <williamh@gentoo.org> wrote:

>> Section 5.5.2 describes the directory structure of /var/cache.
>> These paths are all optional [1].
>> 
>> /var/cache/fonts
>> /var/cache/man
>> /var/cache/www
>> /var/cache/<package>
>> 
>> Gentoo isn't a package, so I don't think /var/cache/gentoo/* is
>> appropriate. Here is my proposal:
>> 
>> /usr/portage -> /var/db/repos/gentoo
>> /usr/portage/distfiles -> /var/cache/portage/distfiles
>> /usr/portage/packages -> /var/cache/portage/binpkgs
>> 
>> I'm not 100% comfortable with /var/db, but I don't have any better
>> suggestion either.
>> 
>> [1] http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#varcacheApplicationCacheData

> From the same source
> "No other requirements are made on the data format of the cache
> directories."
> And as you have quoted it, everything under /var/cache is optional.

> So anything which doesn't conflict with another package seems fine
> according to FHS.

That's how I would read it, too. We could of course invent a package
name (like "package-manager" for virtual/package-manager) but it seems
cumbersome, and I don't see any benefit of it.

There also is /var/cache/fonts, so the FHS itself lists an example of
a directory that's not named after a specific package.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] rfc: moving default location of portage tree (was: [gentoo-project] Call for agenda items - Council meeting 2018-07-29)

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 1838 bytes --]

On Fri, 27 Jul 2018 16:31:15 +0200
Ulrich Mueller <ulm@gentoo.org> wrote:

> >>>>> On Fri, 27 Jul 2018, Corentin “Nado” Pazdera wrote:  
> 
> > July 27, 2018 4:07 PM, "William Hubbs" <williamh@gentoo.org>
> > wrote:  
> 
> >> Section 5.5.2 describes the directory structure of /var/cache.
> >> These paths are all optional [1].
> >> 
> >> /var/cache/fonts
> >> /var/cache/man
> >> /var/cache/www
> >> /var/cache/<package>
> >> 
> >> Gentoo isn't a package, so I don't think /var/cache/gentoo/* is
> >> appropriate. Here is my proposal:
> >> 
> >> /usr/portage -> /var/db/repos/gentoo
> >> /usr/portage/distfiles -> /var/cache/portage/distfiles
> >> /usr/portage/packages -> /var/cache/portage/binpkgs
> >> 
> >> I'm not 100% comfortable with /var/db, but I don't have any better
> >> suggestion either.
> >> 
> >> [1]
> >> http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#varcacheApplicationCacheData  
> 
> > From the same source
> > "No other requirements are made on the data format of the cache
> > directories."
> > And as you have quoted it, everything under /var/cache is
> > optional.  
> 
> > So anything which doesn't conflict with another package seems fine
> > according to FHS.  
> 
> That's how I would read it, too. We could of course invent a package
> name (like "package-manager" for virtual/package-manager) but it seems
> cumbersome, and I don't see any benefit of it.
> 
> There also is /var/cache/fonts, so the FHS itself lists an example of
> a directory that's not named after a specific package.
> 
> Ulrich

/var/db/repos/gentoo
/var/cache/distfiles
/var/cache/binpkgs

Works for me, just please keep "portage" out of it, after all distfiles
are not restricted to portage use only, and neither are binpkgs.  There
is alternate binpkg installers.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] rfc: moving default location of portage tree (was: [gentoo-project] Call for agenda items - Council meeting 2018-07-29)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2222 bytes --]

W dniu pią, 27.07.2018 o godzinie 08∶06 -0700, użytkownik Brian Dolbec
napisał:
> On Fri, 27 Jul 2018 16:31:15 +0200
> Ulrich Mueller <ulm@gentoo.org> wrote:
> 
> > > > > > > On Fri, 27 Jul 2018, Corentin “Nado” Pazdera wrote:  
> > > July 27, 2018 4:07 PM, "William Hubbs" <williamh@gentoo.org>
> > > wrote:  
> > > > Section 5.5.2 describes the directory structure of /var/cache.
> > > > These paths are all optional [1].
> > > > 
> > > > /var/cache/fonts
> > > > /var/cache/man
> > > > /var/cache/www
> > > > /var/cache/<package>
> > > > 
> > > > Gentoo isn't a package, so I don't think /var/cache/gentoo/* is
> > > > appropriate. Here is my proposal:
> > > > 
> > > > /usr/portage -> /var/db/repos/gentoo
> > > > /usr/portage/distfiles -> /var/cache/portage/distfiles
> > > > /usr/portage/packages -> /var/cache/portage/binpkgs
> > > > 
> > > > I'm not 100% comfortable with /var/db, but I don't have any better
> > > > suggestion either.
> > > > 
> > > > [1]
> > > > http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#varcacheApplicationCacheData  
> > > From the same source
> > > "No other requirements are made on the data format of the cache
> > > directories."
> > > And as you have quoted it, everything under /var/cache is
> > > optional.  
> > > So anything which doesn't conflict with another package seems fine
> > > according to FHS.  
> > 
> > That's how I would read it, too. We could of course invent a package
> > name (like "package-manager" for virtual/package-manager) but it seems
> > cumbersome, and I don't see any benefit of it.
> > 
> > There also is /var/cache/fonts, so the FHS itself lists an example of
> > a directory that's not named after a specific package.
> > 
> > Ulrich
> 
> /var/db/repos/gentoo
> /var/cache/distfiles
> /var/cache/binpkgs
> 
> Works for me, just please keep "portage" out of it, after all distfiles
> are not restricted to portage use only, and neither are binpkgs.  There
> is alternate binpkg installers.

Well, technically speaking this specific binary package format is
Portage-specific.  But I don't think we need to go into that kind of
nuances.

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: [gentoo-dev] rfc: moving default location of portage tree (was: [gentoo-project] Call for agenda items - Council meeting 2018-07-29)

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 2584 bytes --]

[Proxying a message from Anders Thomson <andersthomson888@gmail.com>]

Hi Ulrich,
As non-devs aren't allowed to post to gentoo-dev, I was hoping that you would
proxy this question/comment for me.

While on the subject of changing defaults, could we consider changing the
(default) location of the pkg db? Roughly everything in /usr (and /(s)bin) is
managed by the package manager, and it would be handy if the db of installed
bits is close to the bits themselves . Keeping it in /var/ makes /var
opinionated on the current set of installed packages and thus another thing to
backup etc of you want "just the os/system stuff, not the applications'
databases". along the same vein, if you want /var to be on another storage
device (NAS/SAN for large databases), you get the OS's pkg db with it. If for
any reason a boot fails to get /var mounted, you're without the pkg db.

Something along the lines of /usr/lib/pkg ?

Best,
/Anders

On Fri, Jul 27, 2018 at 4:31 PM, Ulrich Mueller <ulm@gentoo.org> wrote:

    >>>>> On Fri, 27 Jul 2018, Corentin “Nado” Pazdera wrote:
   
    > July 27, 2018 4:07 PM, "William Hubbs" <williamh@gentoo.org> wrote:
   
    >> Section 5.5.2 describes the directory structure of /var/cache.
    >> These paths are all optional [1].
    >>
    >> /var/cache/fonts
    >> /var/cache/man
    >> /var/cache/www
    >> /var/cache/<package>
    >>
    >> Gentoo isn't a package, so I don't think /var/cache/gentoo/* is
    >> appropriate. Here is my proposal:
    >>
    >> /usr/portage -> /var/db/repos/gentoo
    >> /usr/portage/distfiles -> /var/cache/portage/distfiles
    >> /usr/portage/packages -> /var/cache/portage/binpkgs
    >>
    >> I'm not 100% comfortable with /var/db, but I don't have any better
    >> suggestion either.
    >>
    >> [1] http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#varcacheApplicationCacheData
   
    > From the same source
    > "No other requirements are made on the data format of the cache
    > directories."
    > And as you have quoted it, everything under /var/cache is optional.
   
    > So anything which doesn't conflict with another package seems fine
    > according to FHS.
   
    That's how I would read it, too. We could of course invent a package
    name (like "package-manager" for virtual/package-manager) but it seems
    cumbersome, and I don't see any benefit of it.
   
    There also is /var/cache/fonts, so the FHS itself lists an example of
    a directory that's not named after a specific package.
   
    Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] rfc: moving default location of portage tree (was: [gentoo-project] Call for agenda items - Council meeting 2018-07-29)

[Matt Turner]

On Fri, Jul 27, 2018 at 1:40 AM, Michał Górny <mgorny@gentoo.org> wrote:
> Dnia 27 lipca 2018 10:32:17 CEST, Ulrich Mueller <ulm@gentoo.org> napisał(a):
>>>>>>> On Thu, 19 Jul 2018, Chí-Thanh Christopher Nguyễn wrote:
>>
>>>> Users must never need to modify files in /var/lib to configure a
>>>> package's operation, and _the_specific_file_hierarchy_ used to
>>>> store the data _must_not_be_ _exposed_ to regular users."
>>
>>> One small note, while it is never needed to modify, skel.ebuild
>>> would then be a file that is meant to be accessed by users in
>>> /var/lib if your proposal is realized.
>>
>>That's one of the reasons why the proposal prefers /var/db. The other
>>reason is existing usage in eselect-repository.
>>
>>>>>>> On Thu, 19 Jul 2018, Ulrich Mueller wrote:
>>
>>> In my understanding, a cache is typically an open collection of
>>items.
>>> Some subset of them can be deleted without much negative consequence,
>>> and there may also be surplus items that are no longer necessary and
>>> will be expired at some later time in order to reclaim disk space.
>>
>>> Nothing of this is true for an ebuild repository, which is a closed
>>> collection of files: A single file cannot be discarded without
>>> invalidating the whole repository. Also there cannot be any stray
>>> files which would be expired later. Same as above, a single stray
>>file
>>> will invalidate all.
>>
>>> (A collection of binary packages may qualify as a cache though, by
>>> this definition.)
>>
>>So, considering all the feedback from mailing list and IRC:
>>
>>   /usr/portage           -> /var/db/repos/gentoo
>>   /usr/portage/distfiles -> /var/cache{,/gentoo}/distfiles
>>   /usr/portage/packages  -> /var/cache{,/gentoo}/binpkgs
>>
>>Open question: Should we have the additional "gentoo" path component
>>for the ones in /var/cache? The tradeoff is between a path that is
>>easier to type, or slightly easier usage if someone wants to NFS mount
>>distfiles and binpkgs.
>
> Note that NFS is not exactly clear cut here since binpkgs are not portable to different hosts, so you can have multiple variants of it.

True, but trivially solvable by configuring like-hosts to share
packages. Skylake packages go here, Sandybridge packages go here, etc.
This is what I do.

Re: [gentoo-dev] rfc: moving default location of portage tree (was: [gentoo-project] Call for agenda items - Council meeting 2018-07-29)

[Matt Turner]

On Fri, Jul 27, 2018 at 1:32 AM, Ulrich Mueller <ulm@gentoo.org> wrote:
>>>>>> On Thu, 19 Jul 2018, Chí-Thanh Christopher Nguyễn wrote:
>
>>> Users must never need to modify files in /var/lib to configure a
>>> package's operation, and _the_specific_file_hierarchy_ used to
>>> store the data _must_not_be_ _exposed_ to regular users."
>
>> One small note, while it is never needed to modify, skel.ebuild
>> would then be a file that is meant to be accessed by users in
>> /var/lib if your proposal is realized.
>
> That's one of the reasons why the proposal prefers /var/db. The other
> reason is existing usage in eselect-repository.
>
>>>>>> On Thu, 19 Jul 2018, Ulrich Mueller wrote:
>
>> In my understanding, a cache is typically an open collection of items.
>> Some subset of them can be deleted without much negative consequence,
>> and there may also be surplus items that are no longer necessary and
>> will be expired at some later time in order to reclaim disk space.
>
>> Nothing of this is true for an ebuild repository, which is a closed
>> collection of files: A single file cannot be discarded without
>> invalidating the whole repository. Also there cannot be any stray
>> files which would be expired later. Same as above, a single stray file
>> will invalidate all.
>
>> (A collection of binary packages may qualify as a cache though, by
>> this definition.)
>
> So, considering all the feedback from mailing list and IRC:
>
>    /usr/portage           -> /var/db/repos/gentoo
>    /usr/portage/distfiles -> /var/cache{,/gentoo}/distfiles
>    /usr/portage/packages  -> /var/cache{,/gentoo}/binpkgs
>
> Open question: Should we have the additional "gentoo" path component
> for the ones in /var/cache? The tradeoff is between a path that is
> easier to type, or slightly easier usage if someone wants to NFS mount
> distfiles and binpkgs.

That proposal has by vote of support. No strong preference on whether
to include gentoo/ or not. It's one NFS mount vs two so not a big deal
either way.

Re: [gentoo-dev] rfc: moving default location of portage tree (was: [gentoo-project] Call for agenda items - Council meeting 2018-07-29)

[Rich Freeman]

On Sun, Jul 29, 2018 at 3:56 PM Matt Turner <mattst88@gentoo.org> wrote:
>
> On Fri, Jul 27, 2018 at 1:32 AM, Ulrich Mueller <ulm@gentoo.org> wrote:
> >
> > So, considering all the feedback from mailing list and IRC:
> >
> >    /usr/portage           -> /var/db/repos/gentoo
> >    /usr/portage/distfiles -> /var/cache{,/gentoo}/distfiles
> >    /usr/portage/packages  -> /var/cache{,/gentoo}/binpkgs
> >
> > Open question: Should we have the additional "gentoo" path component
> > for the ones in /var/cache? The tradeoff is between a path that is
> > easier to type, or slightly easier usage if someone wants to NFS mount
> > distfiles and binpkgs.
>
> That proposal has by vote of support. No strong preference on whether
> to include gentoo/ or not. It's one NFS mount vs two so not a big deal
> either way.
>

Why not stick the repos in /var/repos and not /var/db/repos?  If we're
just making up paths, why not make up a shorter one?  I don't think
any other linux distros use /var/db...

-- 
Rich

Re: [gentoo-dev] rfc: moving default location of portage tree (was: [gentoo-project] Call for agenda items - Council meeting 2018-07-29)

[M. J. Everitt]

[-- Attachment #1.1: Type: text/plain, Size: 264 bytes --]

On 29/07/18 21:01, Rich Freeman wrote:
> <snip>
>
> Why not stick the repos in /var/repos and not /var/db/repos?  If we're
> just making up paths, why not make up a shorter one?  I don't think
> any other linux distros use /var/db...
>
*BSD I believe ..


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] rfc: moving default location of portage tree

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 1356 bytes --]

>>>>> On Fri, 27 Jul 2018, Brian Dolbec wrote:

> On Fri, 27 Jul 2018 16:31:15 +0200
> Ulrich Mueller <ulm@gentoo.org> wrote:

>> >>>>> On Fri, 27 Jul 2018, Corentin “Nado” Pazdera wrote:
>>
>> > From the same source
>> > "No other requirements are made on the data format of the cache
>> > directories."
>> > And as you have quoted it, everything under /var/cache is
>> > optional.
>>
>> > So anything which doesn't conflict with another package seems
>> > fine according to FHS.
>>
>> That's how I would read it, too. We could of course invent a
>> package name (like "package-manager" for virtual/package-manager)
>> but it seems cumbersome, and I don't see any benefit of it.
>>
>> There also is /var/cache/fonts, so the FHS itself lists an example
>> of a directory that's not named after a specific package.

> /var/db/repos/gentoo
> /var/cache/distfiles
> /var/cache/binpkgs

For the record, these three paths have been approved in today's
Council meeting.

> Works for me, just please keep "portage" out of it, after all
> distfiles are not restricted to portage use only, and neither are
> binpkgs. There is alternate binpkg installers.

By another Council vote, snapshot names should be changed from
portage-YYYYMMDD.tar.{bz2,xz} to gentoo-YYYYMMDD.tar.{bz2,xz}.

So, no "portage" any more. ;)

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] rfc: moving default location of portage tree (was: [gentoo-project] Call for agenda items - Council meeting 2018-07-29)

[Matt Turner]

On Sun, Jul 29, 2018 at 1:01 PM, Rich Freeman <rich0@gentoo.org> wrote:
> On Sun, Jul 29, 2018 at 3:56 PM Matt Turner <mattst88@gentoo.org> wrote:
>>
>> On Fri, Jul 27, 2018 at 1:32 AM, Ulrich Mueller <ulm@gentoo.org> wrote:
>> >
>> > So, considering all the feedback from mailing list and IRC:
>> >
>> >    /usr/portage           -> /var/db/repos/gentoo
>> >    /usr/portage/distfiles -> /var/cache{,/gentoo}/distfiles
>> >    /usr/portage/packages  -> /var/cache{,/gentoo}/binpkgs
>> >
>> > Open question: Should we have the additional "gentoo" path component
>> > for the ones in /var/cache? The tradeoff is between a path that is
>> > easier to type, or slightly easier usage if someone wants to NFS mount
>> > distfiles and binpkgs.
>>
>> That proposal has by vote of support. No strong preference on whether
>> to include gentoo/ or not. It's one NFS mount vs two so not a big deal
>> either way.
>>
>
> Why not stick the repos in /var/repos and not /var/db/repos?  If we're
> just making up paths, why not make up a shorter one?  I don't think
> any other linux distros use /var/db...

That would be fine with me as well :)