[gentoo-dev] DRAFT GLEP: MULTI-REPO

[gentoo-dev] DRAFT GLEP: MULTI-REPO

[Andrew Muraco]

[-- Attachment #1: Type: text/plain, Size: 413 bytes --]

Attached is a draft of a glep for formalizing multiple-repository support

This is far from ideal in many ways, but i'm too tired and I drank too 
much caffine to be sane.

Also, i have no clue how to use docutils so i just tried my best.
(** is my way of doing another level of bullets, please let me know how 
to properly do this)
Comments, objections, anything consructive is welcome.

Thanks,

Andrew Muraco


[-- Attachment #2: glep-multi-repo-draft.txt --]
[-- Type: text/plain, Size: 4460 bytes --]

GLEP: XX
Title: Multiple Repository Support in Portage
Version: $Revision: 1.0 $
Author: Andrew Muraco <tuxp3@leetworks.com>
Last-Modified: $Date: 2005/12/17 03:13:10 $
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 17-Dec-2005
Post-History: 17-Dec-2005

Abstract
========
To implement a functional and expandable method for Portage to support multiple repositories.

Motivation
==========
Multiple Repository support is needed, this GLEP is to address this need.

Specification
=============

Portage will make use of two (2) ways to address repositories:
*	A User-defined name, which is likely to be used as a convinance in most situations - this will be referred to as REPO_NAME in this GLEP
*	A hard-coded repository-id which will be found in the repository tree at: metadata/repo_id - this will be referred to as REPO_ID in this GLEP
Both names will contain no spaces, and only standard characters [TODO: references]

Repositories
------------

Each repository will contain:
*	the repo name in metadata/repo_id
*	repo information such as maintainer of the repo, notes on who hosts it, etc will be contained metadata/repo_info
*	unique packages.mask which will only apply to ebuilds within that specific repo.

The REPO_ID must match the name that will be used for rsync
Therefore, rsync://MyServer.tdl/REPO_ID/

/etc/portage/*
-------------

In order to provide users with the current set of options and extend them so they can be customized to each repository, the structure of /etc/portage will remain similar with these changes:
*	/etc/portage/REPO_NAME/* will be the location of repository-specific portage files.
*	/etc/portage/ will continue to function over all repos
** ex) =sys-devel/gcc-4 -* in /etc/portage/package.keywords would use the latest gcc-4 regardless of what tree it comes from.

The following new files will be added to /etc/portage:
*	/etc/portage/repositories.perfer - will contain each REPO_NAME in order of preferance, higher is more perfered. (Each REPO_NAME will be on a seperate line)
**	In the absence of this file portage should use repositories in alphabetical order.
*	/etc/portage/REPO_NAME/repository.id - contains the specific REPO_ID which REPO_NAME applies to.
*/etc/portage/REPO_NAME/repository.conf - will contain any repository-specific options, which can include, but is not limited to, FEATURES="" C[XX]FLAGS="".
**	This will also include a new variable; OPTIONS="" of which is similar to FEATURES, but modifies the way portage will handle that specific repository. A few examples of options which could be useful: 
***	EXCLUDESYNC - Prevents portage from doing a sync on this repo.
***	EXCLUDEUPDATE - Prevents portage from using ebuilds in this repo as updates for packages which currently reside in a different repo.
***	EXCLUSIVEUPDATE - forces any update to any package which is from this repository to a newer version which resides inside of this repo. 
***	et al.

All of the repository rsync URIs will be stored in /etc/make.conf
SYNC="rsync://myfavoriterepo.org/myportage \ rsync://rsync.namerica.gentoo.org/gentoo-portage" 

The Tree: /usr/portage -> /var/repositories/REPO_ID/
----------------------
The repository tree will need to be moved, each repository will have its own folder: /var/repositories/REPO_ID/.

For compatibility reasons, /usr/portage will be treated as /var/repositories/gentoo-portage


Ebuilds
-------

Ebuilds will now be able to have dependencies based on packages from specific repositories.

*	DEP Atoms now support the following format: =REPO_ID:SLOTNUM:CAT/EBUILD-X.Y.Z
**	Ex1) >=MyRepo:2:sys-devel/gcc-4.0
**	Ex2) <gentoo-portage::kde-base/kdelibs-3.5
**	Ex3 ::foo/bar
Dependency atoms that lack the new format (::) or do not have a REPO_ID will then just use any ebuild which fulfills the requirements.

/var/db/pkg/
------------

Along with other information about the building of packages, the specific repo which provided the ebuild will be recorded.
/var/db/pkg/cat/package/REPOSITORY will contain REPO_ID

If no REPOSITORY file is found, then gentoo-portage is assumed.

Backwards Compatibility
=======================
/usr/portage will be treated as /var/repositories/gentoo-portage so it would be possible to function with no changes after the upgrade.


Packages in /var/db that lack REPOSITORY will be assumed to be gentoo-portage.
References
==========

[TODO]

Copyright
=========

This document has been placed in the public domain.

Re: [gentoo-dev] DRAFT GLEP: MULTI-REPO

[Andrew Muraco]

I apologize for the caps in the subject.

Andrew Muraco
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] draft glep: multi-repo

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 11317 bytes --]

Pardon bluntness; don't mean offense, just specifically picking the 
hell out of this proposal to make a point (lucky you) :)

On Sat, Dec 17, 2005 at 03:17:44AM -0500, Andrew Muraco wrote:
> Attached is a draft of a glep for formalizing multiple-repository support

I appreciate trying to chip in, but frankly this glep needs a lot more 
thought put into it.  

Further, I _really_ do not see the point of glepping this either.  Puking 
up proposals due to folks making noise is a waste of time- don't 
document/propose just because folks are making noise- do it for 
large scale changes, or conflict, not because someone requires a 
glep/spec before they're willing to listen to the _developers_ of a 
project about how to integrate a new feature into _their_ project.


> This is far from ideal in many ways, but i'm too tired and I drank too 
> much caffine to be sane.
> 
> Comments, objections, anything consructive is welcome.

Inlined...


> Abstract
> ========
> To implement a functional and expandable method for Portage to support multiple repositories.
> 
> Motivation
> ==========
> Multiple Repository support is needed, this GLEP is to address this need.

define multiple repository.  We _have_ multi repo already
(binpkg and portdir, let alone overlays).


> Specification
> =============
> 
> Portage will make use of two (2) ways to address repositories:
> *	A User-defined name, which is likely to be used as a convinance in most situations - this will be referred to as REPO_NAME in this GLEP
> *	A hard-coded repository-id which will be found in the repository tree at: metadata/repo_id - this will be referred to as REPO_ID in this GLEP
> Both names will contain no spaces, and only standard characters [TODO: references]

Portage externally will use user defined, internally it will do it's 
own thing.


> Repositories
> ------------
> 
> Each repository will contain:
> *	the repo name in metadata/repo_id
> *	repo information such as maintainer of the repo, notes on who hosts it, etc will be contained metadata/repo_info
> *	unique packages.mask which will only apply to ebuilds within that specific repo.
> 
> The REPO_ID must match the name that will be used for rsync
> Therefore, rsync://MyServer.tdl/REPO_ID/

No.  It's arbitrary, and invalid to assume rsync is the only sync uri 
that's going to be used- this isn't even getting into _remote_ repos.

*ANY* unique id tagged into a repo is just that, a magic constant in 
it's metadata.  Just that.  No mandates about SYNC, file layout, etc, 
will fly that bind to the metadata id.


> /etc/portage/*
> -------------
> 
> In order to provide users with the current set of options and extend them so they can be customized to each repository, the structure of /etc/portage 
> will remain similar with these changes:
> *	/etc/portage/REPO_NAME/* will be the location of repository-specific portage files.
> *	/etc/portage/ will continue to function over all repos
> ** ex) =sys-devel/gcc-4 -* in /etc/portage/package.keywords would use the latest gcc-4 regardless of what tree it comes from.
> 
> The following new files will be added to /etc/portage:
> *	/etc/portage/repositories.perfer - will contain each REPO_NAME in order of preferance, higher is more perfered. (Each REPO_NAME will be on a seperate line)

yuck.  This is a bit of a waste for a single file.


> **	In the absence of this file portage should use repositories in alphabetical order.

directory returned ordering, not alpha- no ordering == set of 
repositories, thus don't try to induce a fallback ordering.


> *	/etc/portage/REPO_NAME/repository.id - contains the specific REPO_ID which REPO_NAME applies to.

This is going to induce more slowdown for any config instantiation- 
more directories/files to scan.  Iow, python -c 'import portage' 
(which instantiates a config obj), is going to get slower, which will 
piss off the slower system folk even further (hell, even with 1.5ghz 
and decent IO it still is a 10s import for me uncached).


> */etc/portage/REPO_NAME/repository.conf - will contain any repository-specific options, which can include, but is not limited to, FEATURES="" C[XX]FLAGS="".
> **	This will also include a new variable; OPTIONS="" of which is similar to FEATURES, but modifies the way portage will handle that specific repository. 
>       A few examples of options which could be useful: 
This seems a bit arbitrary.

> ***	EXCLUDESYNC - Prevents portage from doing a sync on this repo.

And how are you going to specify the sync method for that repo?


> ***	EXCLUDEUPDATE - Prevents portage from using ebuilds in this repo as updates for packages which currently reside in a different repo.
> ***	EXCLUSIVEUPDATE - forces any update to any package which is from this repository to a newer version which resides inside of this repo. 

And this is implemented how?  Why is it specifying resolver 
directives as repo attributes?

When is this forced -U going to be triggered?  Sync?  Next random 
emerge call?  How is this going to be bound to the resolver, let alone 
the question of how build configuration data is going to be bound to 
the actual build?  yes, leading question there. :)

Where in this proposal, is it extending similar capabilities to 
binpkgs?


> ***	et al.
> 
> All of the repository rsync URIs will be stored in /etc/make.conf
> SYNC="rsync://myfavoriterepo.org/myportage \ rsync://rsync.namerica.gentoo.org/gentoo-portage" 

No. No no no no no.... at least this explains the metadata ID 
being part of SYNC comment thought :)


> The Tree: /usr/portage -> /var/repositories/REPO_ID/
> ----------------------
> The repository tree will need to be moved, each repository will have its own folder: /var/repositories/REPO_ID/.

This directly castrates the user's ability to store the tree wherever 
they want, loss of that long standing capability is an issue.  No go 
there.


> For compatibility reasons, /usr/portage will be treated as /var/repositories/gentoo-portage

Building hacks into portage isn't going to fly; no special cases.  The 
need for this is a sign that the forced FHS compliant path (instead of 
letting the user do whatever they want) is a bad idea.

This beast can be accomplished without sacrificing our existing 
flexibility.


> Ebuilds
> -------
> 
> Ebuilds will now be able to have dependencies based on packages from specific repositories.
> 
> *	DEP Atoms now support the following format: =REPO_ID:SLOTNUM:CAT/EBUILD-X.Y.Z
> **	Ex1) >=MyRepo:2:sys-devel/gcc-4.0
> **	Ex2) <gentoo-portage::kde-base/kdelibs-3.5
> **	Ex3 ::foo/bar
> Dependency atoms that lack the new format (::) or do not have a REPO_ID will then just use any ebuild which fulfills the requirements.

Backwards compatibility?  Protection for portage version that see this 
and aren't capable of handling it?  Why are you proposing the use/slot 
additions be changed so slot is prefixed rather then postfixed?


> Backwards Compatibility
> =======================
> /usr/portage will be treated as /var/repositories/gentoo-portage so it would be possible to function with no changes after the upgrade.

This is a bit short... see above.  lot more to it, especially since 
you're trying to use stable.

Glep doesn't provide any way for knowing what repo's are active... 
nor what repo's are overlaying each other, nor extending the 
capabilties to anything beyond ebuild trees.  Bintree?  VDB? (yes, 
extending slaving capabilities to vdb has uses).

The shortcomings I'm pointing at are historical- you're trying to 
shoehorn into the existing portage configuration, thus I'm pointing 
out the failings of it :)

Basically... this glep is exactly opposite of what I'm after, and what 
I've already written in saviour.  Saviour's configuration *currently* 
is ini based, although that isn't a requirement (pluggable config parser 
is there, although marienz is working on making my initial prototype 
not suck).

Make.conf support will be there (backwards compatibility); 
just will require a config parser written to convert make.conf format 
into the internal config definitions.

The existing stable configuration is inherintly single master repo 
set, single configuration, single domain, single root, single sync.  
There's no groupping in make.conf (nor aux files), thus it's pretty 
fricking hard to try and make it groupped.  Further, no control over 
the individual objects- no way to specify that a repo is remote fex.

All the make.conf support would do for saviour crap is translate it 
into a single domain, single repo + overlays, etc.

If folks need more power (seperate caches, seperate syncs for repos, N 
domains/roots...), they need to use a more powerful config format.

Via ini, you have repo definitions, defining the repo class to use, 
location (if applicable), host (if applicable), cache reference (cache 
instance to use, defined via it's own configuration item), slaving, 
multiplexing together, package.* specifications (state the 
filepath)...  etc.  Plus, user label (section label for that repo 
definition).  

The framework in place allows for a helluva lot more crazy stuff 
(your own repo classes, say a strict R->L union of repos), I'm just 
rambling off the obvious targets.

The configuration parser/handler actually addresses all of this crap 
on it's own without forcing us to introduce hacks to try and shoehorn 
N repo's into the current single repo config.

Old doc of details/intentions, 
http://dev.gentoo.org/~ferringb/portage/3.0/dev-notes/framework/config.txt 
.

It's generalized and flexible- prior to screaming off with his head 
for the ini usage, please read through 
http://dev.gentoo.org/~ferringb/portage/3.0/dev-notes ; there are a 
_lot_ of good reasons, jotting down of intention, layout/design of 
portage 3, etc.

Do the reading prior to the screaming ;)

Either way... here's how it *should* proceed from where I'm sitting.  
Introduce full PORTDIR capabilities to all repos available- vdb, binpkg, 
portdir/portdir_overlay.  That right there is a major request people 
have been pushing for, and in doing so the framework gets hammered 
on/debugged.

Introducing resolver level constraints (match strictly from this repo) 
requires atom extension among other things, and introduces it's own 
class of problems.  Introduce the capabilities while treating existing 
repos as a single repo set, then extend it with what's needed for true 
stand alone repos.

Either way, while discussion of standalone repos is good, I'm going to 
kindly remind y'all that glep42 just needs to pass in a repo id- it's 
not blocked by anything but minor api changes to portageq so that the 
portage devs aren't forced to fix someone elses proposal down the 
line (in the process breaking crap for users when we do so).

True stand alone repository capabilities aren't required/bound to 
glep42, all that's required out of glep42 is that the syncing repo id
be used now (even if it may seem superfluous).  Iow, news *should* 
work regardless if it's an overlay or a stand alone repo.

Please keep that in mind.
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] draft glep: multi-repo

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 754 bytes --]

On Sat, 17 Dec 2005 15:15:48 -0800 Brian Harring <ferringb@gentoo.org>
wrote:
| True stand alone repository capabilities aren't required/bound to 
| glep42, all that's required out of glep42 is that the syncing repo id
| be used now (even if it may seem superfluous).  Iow, news *should* 
| work regardless if it's an overlay or a stand alone repo.

Not quite true. I also need to know how to handle Display-If-Installed
and Display-If-Profile headers with multiple repositories. Hence why
I'd prefer to leave it as "something that can easily be implemented in
the future"...

-- 
Ciaran McCreesh : Gentoo Developer (I can kill you with my brain)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] draft glep: multi-repo

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 945 bytes --]

On Sat, Dec 17, 2005 at 11:25:58PM +0000, Ciaran McCreesh wrote:
> On Sat, 17 Dec 2005 15:15:48 -0800 Brian Harring <ferringb@gentoo.org>
> wrote:
> | True stand alone repository capabilities aren't required/bound to 
> | glep42, all that's required out of glep42 is that the syncing repo id
> | be used now (even if it may seem superfluous).  Iow, news *should* 
> | work regardless if it's an overlay or a stand alone repo.
> 
> Not quite true. I also need to know how to handle Display-If-Installed
> and Display-If-Profile headers with multiple repositories. Hence why
> I'd prefer to leave it as "something that can easily be implemented in
> the future"...

News is repo specific.  What I stated in the glep42 thread email 
covers this already; zmedico already adress Display-If-Profile 
(namely, a simple portageq extension).

What remaining straw men are there for ignoring the portage 
developers requests?
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] draft glep: multi-repo

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 644 bytes --]

On Sat, 17 Dec 2005 16:14:15 -0800 Brian Harring <ferringb@gentoo.org>
wrote:
| What remaining straw men are there for ignoring the portage 
| developers requests?

Asking you to specify how multiple repositories will work before I try
to extend the GLEP to support multiple repositories is hardly a straw
man argument. *shrug* But if you prefer, I'll change the GLEP to
support multiple repositories the way I'd like to see them done rather
than the way you'd like.

-- 
Ciaran McCreesh : Gentoo Developer (I can kill you with my brain)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] draft glep: multi-repo

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 2222 bytes --]

On Sun, Dec 18, 2005 at 12:24:48AM +0000, Ciaran McCreesh wrote:
> On Sat, 17 Dec 2005 16:14:15 -0800 Brian Harring <ferringb@gentoo.org>
> wrote:
> | What remaining straw men are there for ignoring the portage 
> | developers requests?
> 
> Asking you to specify how multiple repositories will work before I try
> to extend the GLEP to support multiple repositories is hardly a straw
> man argument.

Do you need to know how every bit of a car works to drive it?  No.

We're simply asking you to tag in a repo_id to your calls to portage.  
It's bloody simple, just need to know what you explicitly need from a 
news client standpoint.

Stop being a stubborn mule, and realize that you're trying to shove a 
feature into portage that *we* have to maintain, including your built 
in design flaw for N repos.

If you can't accept our criticism and suggestions for your glep, tough 
cookies, we're the ones stuck maintaining it, and the users are the 
one stuck when we expand portage functionality (thus breaking your 
implementation).


> *shrug* But if you prefer, I'll change the GLEP to
> support multiple repositories the way I'd like to see them done rather
> than the way you'd like.

Ciaranm, cut the word games and stupid threats.

If you can't work with others, and actually understand that they may 
not agree with your views (let alone be willing to have you dump a 
mess on them), I suggest you leave the distro and go work on your 
social skills.

Propose the glep however you want.

As long as the glep is around, I'm going to do the same you do- point 
out the flaws in it.  If you're unwilling to even nail down what is 
needed so that all parties are happy, that's fine- I would expect the 
council to realize the glep (heavily affecting the portage group, 
since server and client side mods fall on our head) needs to be worked 
out further and thus reject it.

Or, you stop debating the fact portage group might have a say in this, 
and start discussing what needs to be done so progress is made.

Balls in your court, you know we view it as unacceptable right now, if 
you're not willing to work with us there isn't much that can be done.

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] draft glep: multi-repo

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 422 bytes --]

On Sat, 17 Dec 2005 17:01:59 -0800 Brian Harring <ferringb@gentoo.org>
wrote:
| Do you need to know how every bit of a car works to drive it?  No.

No, but I do need to know whether it has manual or automatic gears, and
where the steering wheel is.

-- 
Ciaran McCreesh : Gentoo Developer (I can kill you with my brain)
Mail            : ciaranm at gentoo.org
Web             : http://dev.gentoo.org/~ciaranm


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] draft glep: multi-repo

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 812 bytes --]

On Sun, Dec 18, 2005 at 01:08:31AM +0000, Ciaran McCreesh wrote:
> On Sat, 17 Dec 2005 17:01:59 -0800 Brian Harring <ferringb@gentoo.org>
> wrote:
> | Do you need to know how every bit of a car works to drive it?  No.
> 
> No, but I do need to know whether it has manual or automatic gears, and
> where the steering wheel is.

Stop spamming the list with idiot snarky responses, and discuss the 
issues at hand.

Bluntly, others may back down from your shit but I'm not going 
to.  A half baked proposal that is *broken* from the start being 
dumped on the mess that is portage is not acceptable.

If you're not going to fix your glep, state so, stop wasting others 
time.  Just take it to the council and have them ixnay it on the same 
issues people have continually pointed out.
~harring


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]