[gentoo-dev] rfc: location of portage tree

[gentoo-dev] rfc: location of portage tree

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 538 bytes --]

All,

I know this has come up before, but I don't really recall what the
specific objections were.

IMO the portage directory doesn't belong under /usr at all.
I was chatting with another developer who uses
/var/cache/portage/{tree,distfiles}, and I'm thinking about switching my
default setup to do this.

I realize that historically the portage tree has been installed under
/usr, but Can we consider changing this default for new installations
and providing instructions for users for how to get the portage tree out
of /usr?
William


[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] rfc: location of portage tree

[Aaron W. Swenson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 03/27/2012 03:05 PM, William Hubbs wrote:
> All,
> 
> I know this has come up before, but I don't really recall what the 
> specific objections were.
> 
> IMO the portage directory doesn't belong under /usr at all. I was
> chatting with another developer who uses 
> /var/cache/portage/{tree,distfiles}, and I'm thinking about
> switching my default setup to do this.
> 
> I realize that historically the portage tree has been installed
> under /usr, but Can we consider changing this default for new
> installations and providing instructions for users for how to get
> the portage tree out of /usr? William
> 

But, that'd violate the spirit of usrmove!

Seriously, I don't have a strong opinion on it either way. It should
be placed in /var as a way to kind of hint that the files there
shouldn't be edited.

- - Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iF4EAREIAAYFAk9yEU4ACgkQVxOqA9G7/aCdsgD9Hz1KgBVosuGa3RM9uwzzSoou
CnmD3nXj4iBT6cDBY2oA/iThMycgi+Y0lBhr+N7TMWQJwvqgTjzpxg/wQ7wVDF49
=NN8U
-----END PGP SIGNATURE-----

Re: [gentoo-dev] rfc: location of portage tree

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 27/03/12 03:05 PM, William Hubbs wrote:
> All,
> 
> I know this has come up before, but I don't really recall what the 
> specific objections were.
> 
> IMO the portage directory doesn't belong under /usr at all. I was
> chatting with another developer who uses 
> /var/cache/portage/{tree,distfiles}, and I'm thinking about
> switching my default setup to do this.
> 
> I realize that historically the portage tree has been installed
> under /usr, but Can we consider changing this default for new
> installations and providing instructions for users for how to get
> the portage tree out of /usr? William
> 

IIRC, 'cache' can be a volatile storage area, that is, anything in it
can be removed.  One's system is b0rked (or at least, portage is) if
/path/to/portage/profiles goes missing.  I wholeheartedly agree that
distfiles should be moved to /var , but I think the portage tree
shouldn't be there..

(at least, shouldn't be in /var/cache/ ; maybe /var/lib/ ?  of course
then we're colliding with the existing use of /var/lib/portage ...)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)

iF4EAREIAAYFAk9yEmYACgkQAJxUfCtlWe0FNAEAyD6zMS/R7P0kltN6J84kAOkM
5LHcznZRWnn6WFyy4CIA+wXNkzDQ5Pim/hqxHylSILlmUUkb+96KvkjX/mmO03eU
=VVCn
-----END PGP SIGNATURE-----

Re: [gentoo-dev] rfc: location of portage tree

[Kent Fredric]

On 28 March 2012 08:05, William Hubbs <williamh@gentoo.org> wrote:
> All,
>
> I know this has come up before, but I don't really recall what the
> specific objections were.
>
> IMO the portage directory doesn't belong under /usr at all.
> I was chatting with another developer who uses
> /var/cache/portage/{tree,distfiles}, and I'm thinking about switching my
> default setup to do this.
>
> I realize that historically the portage tree has been installed under
> /usr, but Can we consider changing this default for new installations
> and providing instructions for users for how to get the portage tree out
> of /usr?
> William
>

I think I'd rather something closer to paludis's notion, don't assume
its "portage", assume its a repository instead.


/var/cache/repositories/gentoo/*
/var/cache/repositories/perl-experimental/*
/var/cache/distfiles/*
/var/cache/packages/*

Or something along those lines. ( And definitely with the default
locations for distfiles and pkg's outside the repository tree instead
of inside it )

-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

Re: [gentoo-dev] rfc: location of portage tree

[Kent Fredric]

>
> /var/cache/repositories/gentoo/*
> /var/cache/repositories/perl-experimental/*
> /var/cache/distfiles/*
> /var/cache/packages/*
>


Actually, now I think of it, repositories /might/ be suitable for
being under /db/
the repository does sort of function like a database, the tools we use
to access it treats it like one. .

And we already have /var/db/pkg   , why not /var/db/repositories  beside it?

/var/db/pkg
/var/db/repositories/gentoo/*
/var/db/repositories/perl-experimental/*
/var/db/repositories/sunrise/*
/var/cache/distfiles
/var/db/binpkg/


-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

Re: [gentoo-dev] rfc: location of portage tree

[Krzysztof Pawlik]

[-- Attachment #1: Type: text/plain, Size: 1462 bytes --]

On 27/03/12 21:17, Ian Stakenvicius wrote:
> On 27/03/12 03:05 PM, William Hubbs wrote:
>> All,
> 
>> I know this has come up before, but I don't really recall what the 
>> specific objections were.
> 
>> IMO the portage directory doesn't belong under /usr at all. I was
>> chatting with another developer who uses 
>> /var/cache/portage/{tree,distfiles}, and I'm thinking about
>> switching my default setup to do this.
> 
>> I realize that historically the portage tree has been installed
>> under /usr, but Can we consider changing this default for new
>> installations and providing instructions for users for how to get
>> the portage tree out of /usr? William
> 
> 
> IIRC, 'cache' can be a volatile storage area, that is, anything in it
> can be removed.  One's system is b0rked (or at least, portage is) if
> /path/to/portage/profiles goes missing.  I wholeheartedly agree that
> distfiles should be moved to /var , but I think the portage tree
> shouldn't be there..
> 
> (at least, shouldn't be in /var/cache/ ; maybe /var/lib/ ?  of course
> then we're colliding with the existing use of /var/lib/portage ...)

Portage tree is a kind of database (I know, I know -- long shot), so maybe
/var/db/portage for the tree and /var/cache/portage/distfiles (or drop portage
from that path) for distfiles?

-- 
Krzysztof Pawlik  <nelchael at gentoo.org>  key id: 0xF6A80E46
desktop-misc, java, vim, kernel, python, apache...


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 554 bytes --]

Re: [gentoo-dev] rfc: location of portage tree

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 523 bytes --]

On Wed, Mar 28, 2012 at 08:25:58AM +1300, Kent Fredric wrote:
> On 28 March 2012 08:05, William Hubbs <williamh@gentoo.org> wrote:
> /var/cache/repositories/gentoo/*
> /var/cache/repositories/perl-experimental/*
> /var/cache/distfiles/*
> /var/cache/packages/*
 
These sub directories are all portage related, so it is best to put them
 under /var/cache/portage.
 Look in /var/cache on your system; most of the directories in there (at
 least on my system) are named for the program that uses them.

William


[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] rfc: location of portage tree

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 668 bytes --]

On Wed, Mar 28, 2012 at 08:29:50AM +1300, Kent Fredric wrote:
> >
> > /var/cache/repositories/gentoo/*
> > /var/cache/repositories/perl-experimental/*
> > /var/cache/distfiles/*
> > /var/cache/packages/*
> >
> 
> 
> Actually, now I think of it, repositories /might/ be suitable for
> being under /db/
> the repository does sort of function like a database, the tools we use
> to access it treats it like one. .
> 
> And we already have /var/db/pkg   , why not /var/db/repositories  beside it?

I disagree with this, because the repositories can be recovered by doing
an emerge --sync, but if you rm -rf /var/db/pkg you hose your system.

William


[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] rfc: location of portage tree

[Alec Warner]

On Tue, Mar 27, 2012 at 12:40 PM, William Hubbs <williamh@gentoo.org> wrote:
> On Wed, Mar 28, 2012 at 08:25:58AM +1300, Kent Fredric wrote:
>> On 28 March 2012 08:05, William Hubbs <williamh@gentoo.org> wrote:
>> /var/cache/repositories/gentoo/*
>> /var/cache/repositories/perl-experimental/*
>> /var/cache/distfiles/*
>> /var/cache/packages/*
>
> These sub directories are all portage related, so it is best to put them
>  under /var/cache/portage.
>  Look in /var/cache on your system; most of the directories in there (at
>  least on my system) are named for the program that uses them.

The gentoo-x86 ebuild tree is not necessarily portage related.
However I think we should paint the bike shed '/srv/tree'

-A

>
> William
>

Re: [gentoo-dev] rfc: location of portage tree

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1094 bytes --]

On Tue, Mar 27, 2012 at 12:47:10PM -0700, Alec Warner wrote:
> On Tue, Mar 27, 2012 at 12:40 PM, William Hubbs <williamh@gentoo.org> wrote:
> > On Wed, Mar 28, 2012 at 08:25:58AM +1300, Kent Fredric wrote:
> >> On 28 March 2012 08:05, William Hubbs <williamh@gentoo.org> wrote:
> >> /var/cache/repositories/gentoo/*
> >> /var/cache/repositories/perl-experimental/*
> >> /var/cache/distfiles/*
> >> /var/cache/packages/*
> >
> > These sub directories are all portage related, so it is best to put them
> >  under /var/cache/portage.
> >  Look in /var/cache on your system; most of the directories in there (at
> >  least on my system) are named for the program that uses them.
> 
> The gentoo-x86 ebuild tree is not necessarily portage related.
> However I think we should paint the bike shed '/srv/tree'
heh ;-)

What I was wanting to discuss mainly was that /usr/portage isn't right;
I think we need to move that out of the /usr directory.

I'm not sure what the new default should be, nor how the default should
be decided. Maybe we just let Zac pick one?

William


[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] rfc: location of portage tree

[Aaron W. Swenson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 03/27/2012 03:47 PM, Alec Warner wrote:
> On Tue, Mar 27, 2012 at 12:40 PM, William Hubbs
> <williamh@gentoo.org> wrote:
>> On Wed, Mar 28, 2012 at 08:25:58AM +1300, Kent Fredric wrote:
>>> On 28 March 2012 08:05, William Hubbs <williamh@gentoo.org>
>>> wrote: /var/cache/repositories/gentoo/* 
>>> /var/cache/repositories/perl-experimental/* 
>>> /var/cache/distfiles/* /var/cache/packages/*
>> 
>> These sub directories are all portage related, so it is best to
>> put them under /var/cache/portage. Look in /var/cache on your
>> system; most of the directories in there (at least on my system)
>> are named for the program that uses them.
> 
> The gentoo-x86 ebuild tree is not necessarily portage related. 
> However I think we should paint the bike shed '/srv/tree'
> 
> -A

/var/cache/{ebuilds,distfiles,eclasses,profiles}

Or we can just call it Portage.

We call it the "Portage tree", just like we call it gentoo-x86 but
that isn't what it only contains, in several places, both in official
docs and unofficial docs, tweets, pins, notes, stickies....

/var/cache/portage is my vote.

Further, Portage is the official package manager. So, it make more
sense to say "Paludis is compatible with the Portage tree" rather than
"Portage is compatible with the Paludis tree." Portage is the
reference implementation. Whether or not there are other managers are
out there is moot.

- - Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iF4EAREIAAYFAk9yHCcACgkQVxOqA9G7/aAuqgD+Is2OypsU+vhJH4IF0zL0O8N9
OYqCDTbP+gJYy02l1UUA/3geAO62WjiT56Hftq3qIreknkr+3vHA3KpyEZPtiXxj
=tfB4
-----END PGP SIGNATURE-----

Re: [gentoo-dev] rfc: location of portage tree

[Richard Yao]

[-- Attachment #1: Type: text/plain, Size: 1199 bytes --]

On 03/27/12 15:13, Aaron W. Swenson wrote:
> On 03/27/2012 03:05 PM, William Hubbs wrote:
>> All,
> 
>> I know this has come up before, but I don't really recall what the 
>> specific objections were.
> 
>> IMO the portage directory doesn't belong under /usr at all. I was
>> chatting with another developer who uses 
>> /var/cache/portage/{tree,distfiles}, and I'm thinking about
>> switching my default setup to do this.
> 
>> I realize that historically the portage tree has been installed
>> under /usr, but Can we consider changing this default for new
>> installations and providing instructions for users for how to get
>> the portage tree out of /usr? William
> 
> 
> But, that'd violate the spirit of usrmove!
> 
> Seriously, I don't have a strong opinion on it either way. It should
> be placed in /var as a way to kind of hint that the files there
> shouldn't be edited.
> 
> - Aaron
> 

To be honest, the location should not matter. As long as make.conf sets
PORTAGE_DIR correctly, we can put it anywhere. With that said,
/var/portage might better reflect the variable nature of the tree, but I
don't think that would imply that it should not be edited.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: [gentoo-dev] rfc: location of portage tree

[Kent Fredric]

On 28 March 2012 08:47, Alec Warner <antarus@gentoo.org> wrote:
> The gentoo-x86 ebuild tree is not necessarily portage related.
> However I think we should paint the bike shed '/srv/tree'

I for one never developed any love for /srv  , its always seemed like
an unwanted bit of poo left behind by an unloved gremlin.



-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

Re: [gentoo-dev] rfc: location of portage tree

[Kent Fredric]

On 28 March 2012 08:59, William Hubbs <williamh@gentoo.org> wrote:
> What I was wanting to discuss mainly was that /usr/portage isn't right;
> I think we need to move that out of the /usr directory.
>
> I'm not sure what the new default should be, nor how the default should
> be decided. Maybe we just let Zac pick one?

While we're talking about things that probably don't belong in /usr/ :

src ,  esp /usr/src/linux and friends.

I know its not likely to ever change, but its always felt very very
wrong to be under /usr

I've always sort of treated /usr as if it was this big space of
"untouchable, except by package manager", and /usr/src/linux sort of
violates that sense  ( as does /usr/local/ to an extent , but its
slightly different )



-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

Re: [gentoo-dev] rfc: location of portage tree

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 27/03/12 04:08 PM, Kent Fredric wrote:
> On 28 March 2012 08:59, William Hubbs <williamh@gentoo.org> wrote:
>> What I was wanting to discuss mainly was that /usr/portage isn't
>> right; I think we need to move that out of the /usr directory.
>> 
>> I'm not sure what the new default should be, nor how the default
>> should be decided. Maybe we just let Zac pick one?
> 
> While we're talking about things that probably don't belong in
> /usr/ :
> 
> src ,  esp /usr/src/linux and friends.
> 
> I know its not likely to ever change, but its always felt very
> very wrong to be under /usr
> 
> I've always sort of treated /usr as if it was this big space of 
> "untouchable, except by package manager", and /usr/src/linux sort
> of violates that sense  ( as does /usr/local/ to an extent , but
> its slightly different )
> 

Remember that these things pre-dated package managers, though, right?
 (of course i'm showing my own age here..) :D

To further the bikeshed:

- ---Quote: FHS 2.3---
/var/cache
Application cache data. Such data are locally generated as a result of
time-consuming I/O or calculation. The application must be able to
regenerate or restore the data. The cached files can be deleted
without loss of data.

..ok, I guess 'emerge --sync' is a valid regeneration method for the
cache.  I withdraw my objection to moving /usr/portage into /var/cache
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)

iF4EAREIAAYFAk9yIfEACgkQAJxUfCtlWe0NMQEAgGugDZWZS5EfB3rn3oUOU7Vf
wYgYo3Oflgd4EqzjH20BAJkk2l/dXX0yAw6NZEmB9VuSfwgbQUQe/wetoMbbc5BR
=4AZd
-----END PGP SIGNATURE-----

Re: [gentoo-dev] rfc: location of portage tree

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 2706 bytes --]

On Wed, 2012-03-28 at 08:25 +1300, Kent Fredric wrote:
> On 28 March 2012 08:05, William Hubbs <williamh@gentoo.org> wrote:
> > All,
> >
> > I know this has come up before, but I don't really recall what the
> > specific objections were.
> >
> > IMO the portage directory doesn't belong under /usr at all.
> > I was chatting with another developer who uses
> > /var/cache/portage/{tree,distfiles}, and I'm thinking about switching my
> > default setup to do this.
> >
> > I realize that historically the portage tree has been installed under
> > /usr, but Can we consider changing this default for new installations
> > and providing instructions for users for how to get the portage tree out
> > of /usr?
> > William
> >
> 
> I think I'd rather something closer to paludis's notion, don't assume
> its "portage", assume its a repository instead.
> 
> 
> /var/cache/repositories/gentoo/*
> /var/cache/repositories/perl-experimental/*
> /var/cache/distfiles/*
> /var/cache/packages/*
> 
> Or something along those lines. ( And definitely with the default
> locations for distfiles and pkg's outside the repository tree instead
> of inside it )
> 

I am very much in favor of moving all overlays and the main tree into a
common repos, or repositories directory in /var somewhere, maybe even
just /var/repos/gentoo, /var/repos/sunrise,... I see no need to put them
under cache/, etc.. Although under /var/db/ would be the one I'd prefer
from the choices.  

 Layman currently uses /var/lib/layman/overlay-name.  It would be best I
feel to place them in one common location.  I also feel the main tree
should be stored as the same name as it's repo_name value.

If it is done in some fashion like that.  The package managers could
also be modified to automatically scan the base directory for valid
repositories without the need to have them specifically configured in
make.conf. Not to say that they cannot be set in make.conf, but would be
required for anything outside of that base dir.

While we're discussing repos location and naming in general, I recently
came upon a layman overlay (via bug 408897) that is listed as haskell in
layman, but it's repo_name value is gentoo-haskell.  This was for the
glep 42 news reporting feature I just added to layman-2.0.  I had to
patch layman to get around this issue of the mismatched names by getting
the correct name from portage which is needed for the portage
news-reporting function that layman will do after an add/sync
operation. 

Is this something we should specify for them to match?
My thinking is that they should, at the very least, for consistency.

-- 
Brian Dolbec <dolsen@gentoo.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] rfc: location of portage tree

[Alex Alexander]

[-- Attachment #1: Type: text/plain, Size: 1264 bytes --]

On Tue, Mar 27, 2012 at 02:05:54PM -0500, William Hubbs wrote:
> All,
> 
> I know this has come up before, but I don't really recall what the
> specific objections were.
> 
> IMO the portage directory doesn't belong under /usr at all.
> I was chatting with another developer who uses
> /var/cache/portage/{tree,distfiles}, and I'm thinking about switching my
> default setup to do this.
> 
> I realize that historically the portage tree has been installed under
> /usr, but Can we consider changing this default for new installations
> and providing instructions for users for how to get the portage tree out
> of /usr?
> William

If/when this happens, we should also consider improving the internal
structure of the portage folder. At the moment we just throw everything
in it, which is not very user friendly. I recommend creating a subfolder
for the actual tree, keeping distfiles and packages out.

For example, my /usr/portage/ on this system looks like this:

portage/
	tree/
	profiles/ -> tree/profiles/
	distfiles/
	packages/
	layman/

it is a big improvement over the current
distfiles-and-packages-mixed-with-tree-while-layman-wanders state :)
-- 
Alex Alexander | wired
+ Gentoo Linux Developer
++ www.linuxized.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] rfc: location of portage tree

[Kent Fredric]

On 28 March 2012 20:46, Alex Alexander <wired@gentoo.org> wrote:
> For example, my /usr/portage/ on this system looks like this:
>
> portage/
>        tree/
>        profiles/ -> tree/profiles/
>        distfiles/
>        packages/
>        layman/
>
> it is a big improvement over the current
> distfiles-and-packages-mixed-with-tree-while-layman-wanders state :)
> --


I'd rather the gentoo tree be classed as the same tier as other
sources, ie, more like:

> portage/
>        profiles/ -> repositories/gentoo/profiles/
>        distfiles/
>        packages/
>        repositories/gentoo/
>        repositories/sunrise/

At least that way the notion of overlays is less of a "3rd class
citzens, filth, scum" comparison., and ::gentoo being the "master
repository"  is just a configuration convention, not something that is
a fixed design constraint.

Fwiw, I've also long despised the layout of the distfiles directory
being a flat hierarchy, it makes the directory a festering pit of
hellspawn over time on any filesystem that doesn't have dirindex.  (
I've seriously had "ls" take up to a minute to run in that directory,
and if I've ever made the mistake of trying to tab compete something
in there .... /usr/portage/distfiles/foo<tab>  is my normal muscle
memory response, and then it sits there doing nothing for a minute and
it would have been faster to just finish typing it myself =_= )

Though I don't have any solution better than a "break it into 26
subdirs by first letter" .


-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

http://kent-fredric.fox.geek.nz

[gentoo-dev] Re: rfc: location of portage tree

[Duncan]

Brian Dolbec posted on Wed, 28 Mar 2012 00:00:02 -0700 as excerpted:

> Layman currently uses /var/lib/layman/overlay-name.  It would be best I
> feel to place them in one common location.  I also feel the main tree
> should be stored as the same name as it's repo_name value.
> 
> If it is done in some fashion like that.  The package managers could
> also be modified to automatically scan the base directory for valid
> repositories without the need to have them specifically configured in
> make.conf. Not to say that they cannot be set in make.conf, but would be
> required for anything outside of that base dir.

If auto-repo-scanning is added, I'd suggest also adding a make.conf 
OVERLAY_DISABLE value, so it's possible to temporarily disable an overlay 
without removing it (as it is now by simply removing that path from the 
PORTDIR_OVERLAY var, or commenting/moving that line out of the var-
assignment quotes).

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] rfc: location of portage tree

[Piotr Szymaniak]

[-- Attachment #1: Type: text/plain, Size: 1515 bytes --]

On Wed, Mar 28, 2012 at 10:24:56PM +1300, Kent Fredric wrote:
> Fwiw, I've also long despised the layout of the distfiles directory
> being a flat hierarchy, it makes the directory a festering pit of
> hellspawn over time on any filesystem that doesn't have dirindex.  (
> I've seriously had "ls" take up to a minute to run in that directory,
> and if I've ever made the mistake of trying to tab compete something
> in there .... /usr/portage/distfiles/foo<tab>  is my normal muscle
> memory response, and then it sits there doing nothing for a minute and
> it would have been faster to just finish typing it myself =_= )
> 
> Though I don't have any solution better than a "break it into 26
> subdirs by first letter" .

Just use categories from repos?

/usr/portage/distfiles/sys-devel/gcc-1.2.tar.bz2
/usr/portage/distfiles/sys-libs/glibc-2.3.tar.bz2
/usr/portage/distfiles/sys-libs/zlib-3.4.tar.bz2
/usr/portage/distfiles/zomg-soft/zomgawesomesoft-5.3.1.tar.xz
(from zomg repo with custom zomg-soft category ;)

Btw. what would happen if, ie. mc package - well, two different
packages, one from app-misc, one from sci-libs - but lets say they have
a brand new release 5.0 and there's mc-5.0.tar.bz2 for both of them?


Piotr Szymaniak.
-- 
 - Jeden hamburger na dziesiec ci zaszkodzi. Jeden moj stary przyjaciel
to sprawdzil.  Zjadal dziewiec hamburgerow i byl idealnie zdrowy, a gdy
probowal zjesc dziesiatego, z miejsca dostawal torsji.
  -- Graham Masterton, "Night Warriors"

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] rfc: location of portage tree

[Ian Whyman]

> Just use categories from repos?

I've always thought splitting distfiles by category would make a huge
amount of sense.

Re: [gentoo-dev] rfc: location of portage tree

[Kent Fredric]

>
> Just use categories from repos?
>
> /usr/portage/distfiles/sys-devel/gcc-1.2.tar.bz2
> /usr/portage/distfiles/sys-libs/glibc-2.3.tar.bz2
> /usr/portage/distfiles/sys-libs/zlib-3.4.tar.bz2
> /usr/portage/distfiles/zomg-soft/zomgawesomesoft-5.3.1.tar.xz
> (from zomg repo with custom zomg-soft category ;)
>
> Btw. what would happen if, ie. mc package - well, two different
> packages, one from app-misc, one from sci-libs - but lets say they have
> a brand new release 5.0 and there's mc-5.0.tar.bz2 for both of them?
>

Yeah, as admittedly rare as that might be, thats why I didn't suggest
grouping by category =)



-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

http://kent-fredric.fox.geek.nz

Re: [gentoo-dev] rfc: location of portage tree

[Richard Yao]

[-- Attachment #1: Type: text/plain, Size: 1491 bytes --]

On 03/28/12 10:24, Kent Fredric wrote:
>>
>> Just use categories from repos?
>>
>> /usr/portage/distfiles/sys-devel/gcc-1.2.tar.bz2
>> /usr/portage/distfiles/sys-libs/glibc-2.3.tar.bz2
>> /usr/portage/distfiles/sys-libs/zlib-3.4.tar.bz2
>> /usr/portage/distfiles/zomg-soft/zomgawesomesoft-5.3.1.tar.xz
>> (from zomg repo with custom zomg-soft category ;)
>>
>> Btw. what would happen if, ie. mc package - well, two different
>> packages, one from app-misc, one from sci-libs - but lets say they have
>> a brand new release 5.0 and there's mc-5.0.tar.bz2 for both of them?
>>
> 
> Yeah, as admittedly rare as that might be, thats why I didn't suggest
> grouping by category =)

This could cause problems for people using crossdev, because it relies
on overlays to work. If crossdev were to use symlinks, using
`eclean-dist -df` to remove things that are not needed by the main tree
could delete the targets of the symlinks. Hard links would work around
this, but then the distfiles for everything would need to be in the same
file system and that file system would need to support hard links.

The general sentiment that I have seen from Gentoo developers on IRC is
that overlays are bad and that they are meant for things that will
eventually be merged into the main tree. With that in mind, I am not
convinced that this is a problem worth fixing. The overlay owner is
supposed to prepare his things for inclusion into the main tree, so he
should handle it.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: [gentoo-dev] rfc: location of portage tree

[Kent Fredric]

On 28 March 2012 23:04, Piotr Szymaniak <szarpaj@grubelek.pl> wrote:
> Just use categories from repos?
>
> /usr/portage/distfiles/sys-devel/gcc-1.2.tar.bz2
> /usr/portage/distfiles/sys-libs/glibc-2.3.tar.bz2
> /usr/portage/distfiles/sys-libs/zlib-3.4.tar.bz2
> /usr/portage/distfiles/zomg-soft/zomgawesomesoft-5.3.1.tar.xz
> (from zomg repo with custom zomg-soft category ;)
>
> Btw. what would happen if, ie. mc package - well, two different
> packages, one from app-misc, one from sci-libs - but lets say they have
> a brand new release 5.0 and there's mc-5.0.tar.bz2 for both of them?
>


AAaactually, there's a good way of doing it which could /mostly/ be
category based.

You could in a future EAPI define a variable, say DISTDIRPREFIX , the
value of which would *default* to being $CATEGORY, but could be
overridden on a per-package basis if it was deemed nessecary to do so.

So perhaps in the case of something big like Spidermonkey, which might
need a copy of Xulrunner or firefox's source code , the package
developers could set DISTDIRPREFIX to be something like "x-mozilla"
and set that same value for xulrunner, firefox, and seamonkey, and
then the "right thing" would always just work.

And if you *really* wanted to be friendly, you could test after
sourcing the ebuild if DISTDIRPREFIX was still $CATEGORY, and if not,
create a symlink

so  you'd have

dev-lang/js185-1.0.0.tar.gz   -> x-mozilla/js185-1.0.0.tar.gz   # spidermonkey
net-libs/firefox-4.0.1.source.tar.bz2  ->
x-mozilla/firefox-4.0.1.source.tar.bz2  # xulrunner
net-libs/xulrunner-2.0-patches-1.8.tar.bz2  ->
x-mozilla/xulrunner-2.0-patches-1.8.tar.bz2 # xulrunner
www-client/firefox-10.0-patches-0.5.tar.xz  ->
x-mozilla/firefox-10.0-patches-0.5.tar.xz  # seamonkey
www-client/firefox-11.0-patches-0.4.tar.xz ->
x-mozilla/firefox-11.0-patches-0.4.tar.xz # firefox
www-client/firefox-11.0.source.tar.bz2 ->
x-mozilla/firefox-11.0.source.tar.bz2 # firefox
www-client/seamonkey-2.7.1.source.tar.bz2 ->
x-mozilla/seamonkey-2.7.1.source.tar.bz2 # seamonkey
www-client/seamonkey-2.7-patches-03.tar.xz ->
x-mozilla/seamonkey-2.7-patches-03.tar.xz # seamonkey
x-mozilla/firefox-10.0-patches-0.5.tar.xz
x-mozilla/firefox-11.0-patches-0.4.tar.xz
x-mozilla/firefox-11.0.source.tar.bz2
x-mozilla/firefox-4.0.1.source.tar.bz2
x-mozilla/js185-1.0.0.tar.gz
x-mozilla/seamonkey-2.7.1.source.tar.bz2
x-mozilla/seamonkey-2.7-patches-03.tar.xz
x-mozilla/xulrunner-2.0-patches-1.8.tar.bz2

-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3,
3 ) for ( 9,8,0,7,1,6,5,4,3,2 );"

http://kent-fredric.fox.geek.nz

Re: [gentoo-dev] rfc: location of portage tree

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 459 bytes --]

On Wed, 28 Mar 2012 10:42:26 -0400
Richard Yao <ryao@cs.stonybrook.edu> wrote:
> The general sentiment that I have seen from Gentoo developers on IRC
> is that overlays are bad and that they are meant for things that will
> eventually be merged into the main tree.

What they should really be saying is that Portage is bad at overlays
but the Summer of Code projects to fix it will solve all of that this
time around, honest.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] rfc: location of portage tree

[Richard Yao]

[-- Attachment #1: Type: text/plain, Size: 1789 bytes --]

On 03/28/12 10:42, Richard Yao wrote:
> On 03/28/12 10:24, Kent Fredric wrote:
>>>
>>> Just use categories from repos?
>>>
>>> /usr/portage/distfiles/sys-devel/gcc-1.2.tar.bz2
>>> /usr/portage/distfiles/sys-libs/glibc-2.3.tar.bz2
>>> /usr/portage/distfiles/sys-libs/zlib-3.4.tar.bz2
>>> /usr/portage/distfiles/zomg-soft/zomgawesomesoft-5.3.1.tar.xz
>>> (from zomg repo with custom zomg-soft category ;)
>>>
>>> Btw. what would happen if, ie. mc package - well, two different
>>> packages, one from app-misc, one from sci-libs - but lets say they have
>>> a brand new release 5.0 and there's mc-5.0.tar.bz2 for both of them?
>>>
>>
>> Yeah, as admittedly rare as that might be, thats why I didn't suggest
>> grouping by category =)
> 
> This could cause problems for people using crossdev, because it relies
> on overlays to work. If crossdev were to use symlinks, using
> `eclean-dist -df` to remove things that are not needed by the main tree
> could delete the targets of the symlinks. Hard links would work around
> this, but then the distfiles for everything would need to be in the same
> file system and that file system would need to support hard links.
> 
> The general sentiment that I have seen from Gentoo developers on IRC is
> that overlays are bad and that they are meant for things that will
> eventually be merged into the main tree. With that in mind, I am not
> convinced that this is a problem worth fixing. The overlay owner is
> supposed to prepare his things for inclusion into the main tree, so he
> should handle it.
> 

On second thought, I guess this would be okay if you let the overlay
specify its own DISTFILES location within its directory tree to override
the main tree's location. That way crossdev won't be affected.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: [gentoo-dev] rfc: location of portage tree

[Marc Schiffbauer]

[-- Attachment #1: Type: text/plain, Size: 1768 bytes --]

* Aaron W. Swenson schrieb am 27.03.12 um 21:59 Uhr:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
> 
> On 03/27/2012 03:47 PM, Alec Warner wrote:
> > On Tue, Mar 27, 2012 at 12:40 PM, William Hubbs
> > <williamh@gentoo.org> wrote:
> >> On Wed, Mar 28, 2012 at 08:25:58AM +1300, Kent Fredric wrote:
> >>> On 28 March 2012 08:05, William Hubbs <williamh@gentoo.org>
> >>> wrote: /var/cache/repositories/gentoo/* 
> >>> /var/cache/repositories/perl-experimental/* 
> >>> /var/cache/distfiles/* /var/cache/packages/*
> >> 
> >> These sub directories are all portage related, so it is best to
> >> put them under /var/cache/portage. Look in /var/cache on your
> >> system; most of the directories in there (at least on my system)
> >> are named for the program that uses them.
> > 
> > The gentoo-x86 ebuild tree is not necessarily portage related. 
> > However I think we should paint the bike shed '/srv/tree'
> > 
> > -A
> 
> /var/cache/{ebuilds,distfiles,eclasses,profiles}
> 
> Or we can just call it Portage.
> 
> We call it the "Portage tree", just like we call it gentoo-x86 but
> that isn't what it only contains, in several places, both in official
> docs and unofficial docs, tweets, pins, notes, stickies....
> 
> /var/cache/portage is my vote.

+1

I like the idea of one directory because I wthink lots of people do
have that stuff in a dedicated filesystem which today is mounted on
/usr/portage. It would only have to be mounted to /var/cache/portage
and this people were done with "migration".

Having several directories will make it much harder to make "the
portage stuff" be in its own fs. (be it several fs or symlinks ...)

-Marc
-- 
0x35A64134 - 8AAC 5F46 83B4 DB70 8317  3723 296C 6CCA 35A6 4134

[-- Attachment #2: Type: application/pgp-signature, Size: 190 bytes --]

Re: [gentoo-dev] rfc: location of portage tree

[Richard Yao]

[-- Attachment #1: Type: text/plain, Size: 1570 bytes --]

On 03/27/12 15:59, Aaron W. Swenson wrote:
> On 03/27/2012 03:47 PM, Alec Warner wrote:
>> On Tue, Mar 27, 2012 at 12:40 PM, William Hubbs
>> <williamh@gentoo.org> wrote:
>>> On Wed, Mar 28, 2012 at 08:25:58AM +1300, Kent Fredric wrote:
>>>> On 28 March 2012 08:05, William Hubbs <williamh@gentoo.org>
>>>> wrote: /var/cache/repositories/gentoo/* 
>>>> /var/cache/repositories/perl-experimental/* 
>>>> /var/cache/distfiles/* /var/cache/packages/*
>>>
>>> These sub directories are all portage related, so it is best to
>>> put them under /var/cache/portage. Look in /var/cache on your
>>> system; most of the directories in there (at least on my system)
>>> are named for the program that uses them.
> 
>> The gentoo-x86 ebuild tree is not necessarily portage related. 
>> However I think we should paint the bike shed '/srv/tree'
> 
>> -A
> 
> /var/cache/{ebuilds,distfiles,eclasses,profiles}
> 
> Or we can just call it Portage.
> 
> We call it the "Portage tree", just like we call it gentoo-x86 but
> that isn't what it only contains, in several places, both in official
> docs and unofficial docs, tweets, pins, notes, stickies....
> 
> /var/cache/portage is my vote.
> 
> Further, Portage is the official package manager. So, it make more
> sense to say "Paludis is compatible with the Portage tree" rather than
> "Portage is compatible with the Paludis tree." Portage is the
> reference implementation. Whether or not there are other managers are
> out there is moot.
> 
> - Aaron
> 

Or we could just use /var/portage.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: [gentoo-dev] rfc: location of portage tree

[Alec Warner]

On Wed, Mar 28, 2012 at 3:04 AM, Piotr Szymaniak <szarpaj@grubelek.pl> wrote:
> On Wed, Mar 28, 2012 at 10:24:56PM +1300, Kent Fredric wrote:
>> Fwiw, I've also long despised the layout of the distfiles directory
>> being a flat hierarchy, it makes the directory a festering pit of
>> hellspawn over time on any filesystem that doesn't have dirindex.  (
>> I've seriously had "ls" take up to a minute to run in that directory,
>> and if I've ever made the mistake of trying to tab compete something
>> in there .... /usr/portage/distfiles/foo<tab>  is my normal muscle
>> memory response, and then it sits there doing nothing for a minute and
>> it would have been faster to just finish typing it myself =_= )
>>
>> Though I don't have any solution better than a "break it into 26
>> subdirs by first letter" .
>
> Just use categories from repos?
>
> /usr/portage/distfiles/sys-devel/gcc-1.2.tar.bz2
> /usr/portage/distfiles/sys-libs/glibc-2.3.tar.bz2
> /usr/portage/distfiles/sys-libs/zlib-3.4.tar.bz2
> /usr/portage/distfiles/zomg-soft/zomgawesomesoft-5.3.1.tar.xz
> (from zomg repo with custom zomg-soft category ;)
>
> Btw. what would happen if, ie. mc package - well, two different
> packages, one from app-misc, one from sci-libs - but lets say they have
> a brand new release 5.0 and there's mc-5.0.tar.bz2 for both of them?

In the current system it is likely you will get a manifest error for
one of the packages and then someone will have to rename their
distfile.

-A

>
>
> Piotr Szymaniak.
> --
>  - Jeden hamburger na dziesiec ci zaszkodzi. Jeden moj stary przyjaciel
> to sprawdzil.  Zjadal dziewiec hamburgerow i byl idealnie zdrowy, a gdy
> probowal zjesc dziesiatego, z miejsca dostawal torsji.
>  -- Graham Masterton, "Night Warriors"

Re: [gentoo-dev] rfc: location of portage tree

[Aaron W. Swenson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 03/27/2012 03:05 PM, William Hubbs wrote:
> All,
> 
> I know this has come up before, but I don't really recall what the 
> specific objections were.
> 
> IMO the portage directory doesn't belong under /usr at all. I was
> chatting with another developer who uses 
> /var/cache/portage/{tree,distfiles}, and I'm thinking about
> switching my default setup to do this.
> 
> I realize that historically the portage tree has been installed
> under /usr, but Can we consider changing this default for new
> installations and providing instructions for users for how to get
> the portage tree out of /usr? William
> 

So, we're all getting way off topic and discussing reorganizing the
whole enchilada.

How about we all agree or disagree on the primary point: The Portage
tree doesn't belong in /usr.

I believe that it does belong under /var/cache/.

We can go a bit further and make it /var/cache/gentoo-repos/portage/.

That way Layman and friends can all make the move there quite simply
without major infrastructure changes.

The Portage PMS on it's next release would just do a 'mkdir
/var/cache/gentoo-repos/portage/ && sync && rm -rf /usr/portage &&
echo "Portage has moved"' on its next 'emerge --sync' while still
looking in both locations for packages.

(After looking at overlays, if /usr/portage exists, check there first,
if not found look in /var/cache/gentoo-repos/).

Other PMSs can then continue to use /usr/portage until they catch up.
It also allows 'emerge --sync' on older versions of the Portage PMS or
whatever the other PMSs use to continue working without breaking
everything.

We can continue forward with restructuring the tree in later stages,
but we can't move the tree and break compatibility in one go. There
must be stages to the restructuring. The first step is moving it to
the proper top/sub level directory.

So, I'm proposing we use /var/cache/gentoo-repos/portage/ as the
location of the official tree.

- - Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iF4EAREIAAYFAk9zW9oACgkQVxOqA9G7/aBU9gD9FnT7EOl6HZ8HJS0pWJyYJm6G
50VtLCyN8Rt6MBmkB1IBAIVH5tX0IEMe4frJ3tQmdYmqAJNhEwoX/UE/+e3Ihq2u
=oiG3
-----END PGP SIGNATURE-----

Re: [gentoo-dev] rfc: location of portage tree

[Christoph Mende]

On Wed, Mar 28, 2012 at 8:43 PM, Aaron W. Swenson <titanofold@gentoo.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 03/27/2012 03:05 PM, William Hubbs wrote:
>> All,
>>
>> I know this has come up before, but I don't really recall what the
>> specific objections were.
>>
>> IMO the portage directory doesn't belong under /usr at all. I was
>> chatting with another developer who uses
>> /var/cache/portage/{tree,distfiles}, and I'm thinking about
>> switching my default setup to do this.
>>
>> I realize that historically the portage tree has been installed
>> under /usr, but Can we consider changing this default for new
>> installations and providing instructions for users for how to get
>> the portage tree out of /usr? William
>>
>
> So, we're all getting way off topic and discussing reorganizing the
> whole enchilada.
>
> How about we all agree or disagree on the primary point: The Portage
> tree doesn't belong in /usr.
>
> I believe that it does belong under /var/cache/.

I believe it's /var/lib/<name>. Here's what FHS says:
/var/cache is intended for cached data from applications. Such data is
locally generated as a result of time-consuming I/O or calculation.
The application must be able to regenerate or restore the data. Unlike
/var/spool, the cached files can be deleted without data loss.

And:
/var/lib/<name> is the location that must be used for all distribution
packaging support.

Re: [gentoo-dev] rfc: location of portage tree

[Rich Freeman]

On Wed, Mar 28, 2012 at 2:53 PM, Christoph Mende <angelos@gentoo.org> wrote:
>
> I believe it's /var/lib/<name>. Here's what FHS says:
> /var/cache is intended for cached data from applications. Such data is
> locally generated as a result of time-consuming I/O or calculation.
> The application must be able to regenerate or restore the data. Unlike
> /var/spool, the cached files can be deleted without data loss.
>

I can do rm -rf /usr/portage ; mkdir /usr/portage ; emerge --sync and
it will work just fine, I think.

That really does point to cache.  The only thing different from a
browser cache is that portage doesn't automatically refresh it.

distfiles and packages are the same (well, depending on where you get
your binpackages from, that might or might not be a cache per-se - if
you're just using FEATURES=buildpkg then you can do an emerge -e world
and get it back).

> And:
> /var/lib/<name> is the location that must be used for all distribution
> packaging support.
>

I think that things like the local list of installed packages belongs
in this category.  It is a bit debatable how the tree fits into this.

However, this really is bikeshedding.  Sure, /usr isn't ideal, but
unless we actually start supporting some use case where it doesn't
work so well in the future, I doubt we'll ever see it move.  Plus,
there is even a case for keeping it in /usr in the Fedora-envisioned
/usr-is-ro world.  You could have a complete installation and a
portage tree that it was generated from all snapshotted there.  Sure,
maybe /usr/lib or /usr/share might make more sense then, but again, I
don't see it changing unless it actually results in a real benefit to
users.

Rich

Re: [gentoo-dev] rfc: location of portage tree

[Aaron W. Swenson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 03/28/2012 02:53 PM, Christoph Mende wrote:
> On Wed, Mar 28, 2012 at 8:43 PM, Aaron W. Swenson
> <titanofold@gentoo.org> wrote:
>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
>> 
>> On 03/27/2012 03:05 PM, William Hubbs wrote:
>>> All,
>>> 
>>> I know this has come up before, but I don't really recall what
>>> the specific objections were.
>>> 
>>> IMO the portage directory doesn't belong under /usr at all. I
>>> was chatting with another developer who uses 
>>> /var/cache/portage/{tree,distfiles}, and I'm thinking about 
>>> switching my default setup to do this.
>>> 
>>> I realize that historically the portage tree has been
>>> installed under /usr, but Can we consider changing this default
>>> for new installations and providing instructions for users for
>>> how to get the portage tree out of /usr? William
>>> 
>> 
>> So, we're all getting way off topic and discussing reorganizing
>> the whole enchilada.
>> 
>> How about we all agree or disagree on the primary point: The
>> Portage tree doesn't belong in /usr.
>> 
>> I believe that it does belong under /var/cache/.
> 
> I believe it's /var/lib/<name>. Here's what FHS says: /var/cache is
> intended for cached data from applications. Such data is locally
> generated as a result of time-consuming I/O or calculation. The
> application must be able to regenerate or restore the data. Unlike 
> /var/spool, the cached files can be deleted without data loss.
> 
> And: /var/lib/<name> is the location that must be used for all
> distribution packaging support.
> 

'Support' is the keyword here. The repositories are regenerated given
machinesan 'emerge --sync' and can be considered as temporary as the
packages themselves are impermanent. Further, the repository isn't
required to persist. If somebody really wanted to be hard on our
infrastructure, they could do an 'emerge --sync' at boot to repopulate
/var/cache/gentoo-repos/.

Portage PMS already does the right thing and uses /var/lib/ for the
appropriate use, config and world, things that need to persist between
reboots.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iF4EAREIAAYFAk9zZKMACgkQVxOqA9G7/aCcUAD+JEnR5dE1S7QNUr+8zNFzh/kR
hlnZUXopuQSrMhxjxYEA/AlT2I5p4KEiGybgDELTnVNqOHNKlpO5RepBMjhII1Yy
=sjCv
-----END PGP SIGNATURE-----

Re: [gentoo-dev] rfc: location of portage tree

[Zac Medico]

On 03/28/2012 11:43 AM, Aaron W. Swenson wrote:
> The Portage PMS on it's next release would just do a 'mkdir
> /var/cache/gentoo-repos/portage/&&  sync&&  rm -rf /usr/portage&&
> echo "Portage has moved"' on its next 'emerge --sync' while still
> looking in both locations for packages.
>
> (After looking at overlays, if /usr/portage exists, check there first,
> if not found look in /var/cache/gentoo-repos/).

My preferred migration approach would be to change the default location 
for new installs, and keep the existing location for existing installed 
systems. I'd do that by automatically generating a config update for 
make.conf, so that systems using the old defaults will continue to use them.
-- 
Thanks,
Zac

Re: [gentoo-dev] rfc: location of portage tree

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 2818 bytes --]

On Wed, 2012-03-28 at 14:43 -0400, Aaron W. Swenson wrote:
> So, we're all getting way off topic and discussing reorganizing the
> whole enchilada.
> 
> How about we all agree or disagree on the primary point: The Portage
> tree doesn't belong in /usr.
> 
> I believe that it does belong under /var/cache/.
> 
> We can go a bit further and make it /var/cache/gentoo-repos/portage/.

a little too convoluted.  It should be simpler... see example below


> 
> That way Layman and friends can all make the move there quite simply
> without major infrastructure changes.

Layman, portage, pkgcore have all been able to have them elsewhere.  It
won't break anything.  It is only a config value change.  Coding not
required.  So it is easy to do that now.  We are arguing about the
default location

> 
> The Portage PMS on it's next release would just do a 'mkdir
> /var/cache/gentoo-repos/portage/ && sync && rm -rf /usr/portage &&
> echo "Portage has moved"' on its next 'emerge --sync' while still
> looking in both locations for packages.
> 

It would be quite easy for simple use cases (the majority of users), to
create a migration script that users could use which would read the
current config values, then migrate them and update the config values.
But that would be entirely optional.  If a user wants to keep it at the
current location it would not break anything.  The only thing that would
be required is to set the correct variables in make.conf to override the
new defaults to maintain the current locations.


> (After looking at overlays, if /usr/portage exists, check there first,
> if not found look in /var/cache/gentoo-repos/).
> 
> Other PMSs can then continue to use /usr/portage until they catch up.
> It also allows 'emerge --sync' on older versions of the Portage PMS or
> whatever the other PMSs use to continue working without breaking
> everything.
> 
> We can continue forward with restructuring the tree in later stages,
> but we can't move the tree and break compatibility in one go. There
> must be stages to the restructuring. The first step is moving it to
> the proper top/sub level directory.
> 

I fail to see the complexity that you seem to think is involved to
accomplish this.

> So, I'm proposing we use /var/cache/gentoo-repos/portage/ as the
> location of the official tree.
> 
> - Aaron

to keep everything under one directory like some would prefer...

I propose we name that dir, "gentoo"  simple, to the point.

then to sum up several other posts.

/var/{db,cache,}/gentoo/repositories/gentoo
/var/{db,cache,}/gentoo/repositories/local
/var/{db,cache,}/gentoo/repositories/{overlay of choice}
/var/{db,cache,}/gentoo/distfiles
/var/{db,cache,}/gentoo/packages


-- 
Brian Dolbec <dolsen@gentoo.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] rfc: location of portage tree

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 28/03/12 03:04 PM, Rich Freeman wrote:
> On Wed, Mar 28, 2012 at 2:53 PM, Christoph Mende
> <angelos@gentoo.org> wrote:
>> 
>> I believe it's /var/lib/<name>. Here's what FHS says: /var/cache
>> is intended for cached data from applications. Such data is 
>> locally generated as a result of time-consuming I/O or
>> calculation. The application must be able to regenerate or
>> restore the data. Unlike /var/spool, the cached files can be
>> deleted without data loss.
>> 
> 
> I can do rm -rf /usr/portage ; mkdir /usr/portage ; emerge --sync
> and it will work just fine, I think.

It does, i tried it yesterday.

> 
> That really does point to cache.  The only thing different from a 
> browser cache is that portage doesn't automatically refresh it.
> 

Although, we could always make emerge do an automatic --sync if, say,
/path/to/portage/profiles doesn't exist.  :)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)

iF4EAREIAAYFAk9zbCwACgkQAJxUfCtlWe2abgEAl8eapp2DQOYJx6RAcl6Ei/iN
9L4e7tG9maNTryI6lKMBAOEqAdgWrKWx2UJ3+g7oBNFc5G7Lu+yk3deZZFN4zBjU
=sluw
-----END PGP SIGNATURE-----

Re: [gentoo-dev] rfc: location of portage tree

[Robin H. Johnson]

On Wed, Mar 28, 2012 at 12:04:46PM +0200, Piotr Szymaniak wrote:
> On Wed, Mar 28, 2012 at 10:24:56PM +1300, Kent Fredric wrote:
> > Fwiw, I've also long despised the layout of the distfiles directory
> > being a flat hierarchy, it makes the directory a festering pit of
> > hellspawn over time on any filesystem that doesn't have dirindex.  (
> > I've seriously had "ls" take up to a minute to run in that directory,
> > and if I've ever made the mistake of trying to tab compete something
> > in there .... /usr/portage/distfiles/foo<tab>  is my normal muscle
> > memory response, and then it sits there doing nothing for a minute and
> > it would have been faster to just finish typing it myself =_= )
> > 
> > Though I don't have any solution better than a "break it into 26
> > subdirs by first letter" .
> 
> Just use categories from repos?
No, please don't.

Right now, the Manifests in the tree reference 45790 distfiles.
Of that, 4262 are used by more than one package, and 197 are used by
more than one category.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-dev] rfc: location of portage tree

[Dale]

Marc Schiffbauer wrote:
> * Aaron W. Swenson schrieb am 27.03.12 um 21:59 Uhr:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> On 03/27/2012 03:47 PM, Alec Warner wrote:
>>> On Tue, Mar 27, 2012 at 12:40 PM, William Hubbs

>> /var/cache/{ebuilds,distfiles,eclasses,profiles}
>>
>> Or we can just call it Portage.
>>
>> We call it the "Portage tree", just like we call it gentoo-x86 but
>> that isn't what it only contains, in several places, both in official
>> docs and unofficial docs, tweets, pins, notes, stickies....
>>
>> /var/cache/portage is my vote.
> 
> +1
> 
> I like the idea of one directory because I wthink lots of people do
> have that stuff in a dedicated filesystem which today is mounted on
> /usr/portage. It would only have to be mounted to /var/cache/portage
> and this people were done with "migration".
> 
> Having several directories will make it much harder to make "the
> portage stuff" be in its own fs. (be it several fs or symlinks ...)
> 
> -Marc


As a lowly user, I would like it on /var but could careless about the
directory though the above would work fine.  Reason, I have /var on its
own partition already.  I also have /usr/portage on its own too.  Since
the /usr/portage has lots of ever changing files and CAN get fragmented
a lot, this solves a lot of issues since a lot of things in /var are in
the same boat.  A user could use a file system that is better at this
sort of thing and have only one partition to handle it all.

Back to my hole.  Twice now.

Dale

:-)  :-)

-- 
I am only responsible for what I said ... Not for what you understood or
how you interpreted my words!

Miss the compile output?  Hint:
EMERGE_DEFAULT_OPTS="--quiet-build=n"

[gentoo-dev] Re: rfc: location of portage tree

[Duncan]

Ian Stakenvicius posted on Wed, 28 Mar 2012 15:53:16 -0400 as excerpted:

> Although, we could always make emerge do an automatic --sync if, say,
> /path/to/portage/profiles doesn't exist.  :)

Ugh, no.  Some (many?) of us have a separate portage tree partition, and 
occasionally accidentally do an emerge <something> without it mounted.  
Having portage decide that it should automatically start downloading a 
new tree directly onto the filesystem containing the mountpoint is *NOT* 
ideal!

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: rfc: location of portage tree

[Dale]

Duncan wrote:
> Ian Stakenvicius posted on Wed, 28 Mar 2012 15:53:16 -0400 as excerpted:
> 
>> Although, we could always make emerge do an automatic --sync if, say,
>> /path/to/portage/profiles doesn't exist.  :)
> 
> Ugh, no.  Some (many?) of us have a separate portage tree partition, and 
> occasionally accidentally do an emerge <something> without it mounted.  
> Having portage decide that it should automatically start downloading a 
> new tree directly onto the filesystem containing the mountpoint is *NOT* 
> ideal!
> 

+1

Dale

:-)  :-)

-- 
I am only responsible for what I said ... Not for what you understood or
how you interpreted my words!

Miss the compile output?  Hint:
EMERGE_DEFAULT_OPTS="--quiet-build=n"

Re: [gentoo-dev] rfc: location of portage tree

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1447 bytes --]

On 29 March 2012 07:43, Aaron W. Swenson <titanofold@gentoo.org> wrote:

> So, we're all getting way off topic and discussing reorganizing the
> whole enchilada.
>
> How about we all agree or disagree on the primary point: The Portage
> tree doesn't belong in /usr.
>

+1


> I believe that it does belong under /var/cache/.
>
> =0  # Not sure , semantically it doesn't make sense as its not behaving as
a caching mechanism of any kind and would rather  /var/portage or
/var/lib/portage or something in that direction over /var/cache . I'd even
prefer /var/lib/repositories/portage over /var/cache/portage/



> We can go a bit further and make it /var/cache/gentoo-repos/portage/.
>
> That way Layman and friends can all make the move there quite simply
> without major infrastructure changes.
>
> The Portage PMS on it's next release would just do a 'mkdir
> /var/cache/gentoo-repos/portage/ && sync && rm -rf /usr/portage &&
> echo "Portage has moved"' on its next 'emerge --sync' while still
> looking in both locations for packages.
>
> I'd rather this change not be automatic, and should be driven by ENV
variables, and the new layout be a default layout for new systems, and
write an e-news article describing the default change and how to migrate to
the new layout for people who want to.


-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3, 3 )
for ( 9,8,0,7,1,6,5,4,3,2 );"

http://kent-fredric.fox.geek.nz

[-- Attachment #2: Type: text/html, Size: 2246 bytes --]

Re: [gentoo-dev] rfc: location of portage tree

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1018 bytes --]

On 29 March 2012 08:21, Aaron W. Swenson <titanofold@gentoo.org> wrote:

>
>
> 'Support' is the keyword here. The repositories are regenerated given
> machinesan 'emerge --sync' and can be considered as temporary as the
> packages themselves are impermanent. Further, the repository isn't
> required to persist. If somebody really wanted to be hard on our
> infrastructure, they could do an 'emerge --sync' at boot to repopulate
> /var/cache/gentoo-repos/.
>
>
Though of course, if anybody has custom stuff in say, /usr/portage/local/
which they make by hand, nuking /usr/portage will make you *Very*
unpopular.

As will I be if I have /usr/portage/distfiles under /usr/portage/  and you
nuke /usr/portage including distfiles.

I could download distfiles again, but sorry, bandwidth is not free in every
country, and neither is the time wasted by redownloading it all.

-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3, 3 )
for ( 9,8,0,7,1,6,5,4,3,2 );"

http://kent-fredric.fox.geek.nz

[-- Attachment #2: Type: text/html, Size: 1461 bytes --]

Re: [gentoo-dev] rfc: location of portage tree

[Alec Warner]

On Thu, Mar 29, 2012 at 2:26 AM, Kent Fredric <kentfredric@gmail.com> wrote:
> On 29 March 2012 08:21, Aaron W. Swenson <titanofold@gentoo.org> wrote:
>>
>>
>>
>> 'Support' is the keyword here. The repositories are regenerated given
>> machinesan 'emerge --sync' and can be considered as temporary as the
>> packages themselves are impermanent. Further, the repository isn't
>> required to persist. If somebody really wanted to be hard on our
>> infrastructure, they could do an 'emerge --sync' at boot to repopulate
>> /var/cache/gentoo-repos/.
>>
>
> Though of course, if anybody has custom stuff in say, /usr/portage/local/
> which they make by hand, nuking /usr/portage will make you *Very* unpopular.
>
> As will I be if I have /usr/portage/distfiles under /usr/portage/  and you
> nuke /usr/portage including distfiles.
>
> I could download distfiles again, but sorry, bandwidth is not free in every
> country, and neither is the time wasted by redownloading it all.

Zac's migration plan doesn't involve moving data at all, merely
changing the default for new installs. I think this is a pretty simple
migration plan provided you are ok with it taking a decade. It will be
hard on doc writers who instead of getting to write /usr/portage
everywhere will likely have to write $PORTDIR or $(portageq env
PORTDIR) instead.

-A

>
>
> --
> Kent
>
> perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3, 3 )
> for ( 9,8,0,7,1,6,5,4,3,2 );"
>
> http://kent-fredric.fox.geek.nz

Re: [gentoo-dev] rfc: location of portage tree

[Roy Bamford]

On 2012.03.28 08:46, Alex Alexander wrote:
> On Tue, Mar 27, 2012 at 02:05:54PM -0500, William Hubbs wrote:
> > All,
> > 
> > I know this has come up before, but I don't really recall what the
> > specific objections were.
> > 
> > IMO the portage directory doesn't belong under /usr at all.

[snip]

> > William
> 
> If/when this happens, we should also consider improving the internal
> structure of the portage folder. At the moment we just throw
> everything
> in it, which is not very user friendly. I recommend creating a
> subfolder
> for the actual tree, keeping distfiles and packages out.
> 
> For example, my /usr/portage/ on this system looks like this:
> 
> portage/
> 	tree/
> 	profiles/ -> tree/profiles/
> 	distfiles/
> 	packages/
> 	layman/
> 
> it is a big improvement over the current
> distfiles-and-packages-mixed-with-tree-while-layman-wanders state :)
> -- 
> Alex Alexander | wired
> + Gentoo Linux Developer
> ++ www.linuxized.com
> 

Lets move packages/ out of there.  I share /usr/portage over NFS to 
several different arches.  Sharing /usr/portage/packages is a really 
bad idea in that set up. As they all run ~arch, they all build packages 
so I can downgrade quickly.

-- 
Regards,

Roy Bamford
(Neddyseagoon) a member of
elections
gentoo-ops
forum-mods
trustees

Re: [gentoo-dev] rfc: location of portage tree

[Roy Bamford]

On 2012.03.28 20:04, Rich Freeman wrote:
> On Wed, Mar 28, 2012 at 2:53 PM, Christoph Mende <angelos@gentoo.org>
> wrote:
> >
> > I believe it's /var/lib/<name>. Here's what FHS says:
> > /var/cache is intended for cached data from applications. Such data
> is
> > locally generated as a result of time-consuming I/O or calculation.
> > The application must be able to regenerate or restore the data.
> Unlike
> > /var/spool, the cached files can be deleted without data loss.
> >
> 
> I can do rm -rf /usr/portage ; mkdir /usr/portage ; emerge --sync and
> it will work just fine, I think.

That's pretty much what happened in a stage1 or stage2 install.
Its not cache though as you don't get back the same data as was 
deleted.  

Think 6 month old install.

> 
> That really does point to cache.  The only thing different from a
> browser cache is that portage doesn't automatically refresh it.
> 
> distfiles and packages are the same (well, depending on where you get
> your binpackages from, that might or might not be a cache per-se - if
> you're just using FEATURES=buildpkg then you can do an emerge -e 
> world
> and get it back).
Nope.  

If you have just done 
rm -rf /usr/portage ; mkdir /usr/portage ; emerge --sync, 
then   emerge -e world  gets you the equivelent of emerge --sync && 
emerge world -uDN

Even if you haven't fetched a new tree, you have lost all your old 
binary packages, which you were keeping in case of a broken ~arch 
upgrade that needs to be reverted in a hurry. e.g. one of the nice big 
shiny packages that emerge -e world just updated for you.

[snip]

> 
> Rich
> 
> 
> 

-- 
Regards,

Roy Bamford
(Neddyseagoon) a member of
elections
gentoo-ops
forum-mods
trustees

Re: [gentoo-dev] rfc: location of portage tree

[Walter Dnes]

On Thu, Mar 29, 2012 at 10:26:22PM +1300, Kent Fredric wrote

> Though of course, if anybody has custom stuff in say, /usr/portage/local/
> which they make by hand, nuking /usr/portage will make you *Very*
> unpopular.

  http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?full=1#book_part3_chap5
in the install handbook gives "/usr/local/portage" as an example overlay
directory.  I thought it was implicit that one shouldn't edit or create
files in /usr/portage because they may be overwritten by the system e.g.
during an "emerge --sync".  Maybe the manual needs to state this
explicitly.  Also, /usr/local is the "standard" place to keep one's own
software and/or global customizations that aren't handled by the package
manager, but don't belong in one user's home directory.

-- 
Walter Dnes <waltdnes@waltdnes.org>

Re: [gentoo-dev] rfc: location of portage tree

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1027 bytes --]

On 30 March 2012 17:08, Walter Dnes <waltdnes@waltdnes.org> wrote:

> in the install handbook gives "/usr/local/portage" as an example overlay
> directory.  I thought it was implicit that one shouldn't edit or create
> files in /usr/portage because they may be overwritten by the system e.g.
> during an "emerge --sync".  Maybe the manual needs to state this
> explicitly.  Also, /usr/local is the "standard" place to keep one's own
> software and/or global customizations that aren't handled by the package
> manager, but don't belong in one user's home directory.
>

Yeah, I don't have that layout /now/, but there was a time where one of my
systems had stuff in there, for whatever reason, and I can't recall
deciding to put it there myself. But that said, I *have* been using Gentoo
since one of the 2004.x releases ... Ah the good old days of having the
choice of NPTL ;)


-- 
Kent

perl -e  "print substr( \"edrgmaM  SPA NOcomil.ic\\@tfrken\", \$_ * 3, 3 )
for ( 9,8,0,7,1,6,5,4,3,2 );"

http://kent-fredric.fox.geek.nz

[-- Attachment #2: Type: text/html, Size: 1462 bytes --]

Re: [gentoo-dev] rfc: location of portage tree

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 714 bytes --]

On 29-03-2012 22:12:40 +0100, Roy Bamford wrote:
> > For example, my /usr/portage/ on this system looks like this:
> > 
> > portage/
> > 	tree/
> > 	profiles/ -> tree/profiles/
> > 	distfiles/
> > 	packages/
> > 	layman/
> > 
> > it is a big improvement over the current
> > distfiles-and-packages-mixed-with-tree-while-layman-wanders state :)
> 
> Lets move packages/ out of there.  I share /usr/portage over NFS to 
> several different arches.  Sharing /usr/portage/packages is a really 
> bad idea in that set up. As they all run ~arch, they all build packages 
> so I can downgrade quickly.

I always use packages/CHOST for that reason.


-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[gentoo-dev] Re: rfc: location of portage tree

[Duncan]

Walter Dnes posted on Fri, 30 Mar 2012 00:08:08 -0400 as excerpted:

> On Thu, Mar 29, 2012 at 10:26:22PM +1300, Kent Fredric wrote
> 
>> Though of course, if anybody has custom stuff in say,
>> /usr/portage/local/
>> which they make by hand, nuking /usr/portage will make you *Very*
>> unpopular.
> 
>   http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?
full=1#book_part3_chap5
> in the install handbook gives "/usr/local/portage" as an example overlay
> directory.  I thought it was implicit that one shouldn't edit or create
> files in /usr/portage because they may be overwritten by the system e.g.
> during an "emerge --sync".  Maybe the manual needs to state this
> explicitly.  Also, /usr/local is the "standard" place to keep one's own
> software and/or global customizations that aren't handled by the package
> manager, but don't belong in one user's home directory.

FWIW:

$>>grep PORTDIR\\\|DISTDIR /etc/portage/make/fs
PORTDIR=/p
PORTDIR_OVERLAY=/l/p
DISTDIR=${PORTDIR}/src

$>>grep exclude /etc/portage/make/net$>>cat /etc/portage/make/
rsync.exclude 
# / on the left anchors (like regex ^), / on the right indicates dirs only

/layman/
/src/
/use.defaults

PORTAGE_RSYNC_EXTRA_OPTS="--exclude-from='/etc/portage/make/
rsync.exclude'"

$>>cat /etc/portage/make/rsync.exclude 
# / on the left anchors (like regex ^), / on the right indicates dirs only

/layman/
/src/
/use.defaults

------------------------------------------

(/etc/portage/make/fs and the corresponding net file are sourced by
/etc/portage/make/master, which in turn is sourced by /etc/make.conf, so 
those settings appear in portage, even tho they're actually in files in a 
particular subdir of /etc/portage.)

So I have PORTDIR set to /p, with the layman and src subdirs in it, and 
portage's rsync command set to exclude those subdirs.

FWIW, /usr/portage is a symlink to /p, just in case.

As Kent said, if that arrangement gets nuked despite my rsync --exclude 
settings, the person responsible certainly won't be particularly popular 
here (tho nothing irreplaceable would be lost, here, I'd simply have to 
adjust things and try again).

Fortunately, our portage devs appear to be a bit more sane than to try 
that and Zac at least isn't even proposing it.  His proposal is to simply 
change the default location for new users, which is fine by me. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] rfc: location of portage tree

[Graham Murray]

"Walter Dnes" <waltdnes@waltdnes.org> writes:

> On Thu, Mar 29, 2012 at 10:26:22PM +1300, Kent Fredric wrote
>
>> Though of course, if anybody has custom stuff in say, /usr/portage/local/
>> which they make by hand, nuking /usr/portage will make you *Very*
>> unpopular.
>
>   http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?full=1#book_part3_chap5
> in the install handbook gives "/usr/local/portage" as an example overlay
> directory.  I thought it was implicit that one shouldn't edit or create
> files in /usr/portage because they may be overwritten by the system e.g.
> during an "emerge --sync".  Maybe the manual needs to state this
> explicitly.  Also, /usr/local is the "standard" place to keep one's own
> software and/or global customizations that aren't handled by the package
> manager, but don't belong in one user's home directory.

Where using /usr/portage/local is useful is for 'site local'
packages. Where one system syncs externally and also has all of the
locally generated/edited packages in /usr/portage/local, and the other
systems share this site local repository simply by running "emerge
--sync" to the 'master' system.