From: Brian Harring <ferringb@gentoo.org>
To: gentoo-dev@lists.gentoo.org
Subject: Re: [gentoo-dev] draft glep: multi-repo
Date: Sat, 17 Dec 2005 15:15:48 -0800 [thread overview]
Message-ID: <20051217231548.GB22142@nightcrawler.e-centre.net> (raw)
In-Reply-To: <43A3C9A8.9050608@leetworks.com>
[-- Attachment #1: Type: text/plain, Size: 11317 bytes --]
Pardon bluntness; don't mean offense, just specifically picking the
hell out of this proposal to make a point (lucky you) :)
On Sat, Dec 17, 2005 at 03:17:44AM -0500, Andrew Muraco wrote:
> Attached is a draft of a glep for formalizing multiple-repository support
I appreciate trying to chip in, but frankly this glep needs a lot more
thought put into it.
Further, I _really_ do not see the point of glepping this either. Puking
up proposals due to folks making noise is a waste of time- don't
document/propose just because folks are making noise- do it for
large scale changes, or conflict, not because someone requires a
glep/spec before they're willing to listen to the _developers_ of a
project about how to integrate a new feature into _their_ project.
> This is far from ideal in many ways, but i'm too tired and I drank too
> much caffine to be sane.
>
> Comments, objections, anything consructive is welcome.
Inlined...
> Abstract
> ========
> To implement a functional and expandable method for Portage to support multiple repositories.
>
> Motivation
> ==========
> Multiple Repository support is needed, this GLEP is to address this need.
define multiple repository. We _have_ multi repo already
(binpkg and portdir, let alone overlays).
> Specification
> =============
>
> Portage will make use of two (2) ways to address repositories:
> * A User-defined name, which is likely to be used as a convinance in most situations - this will be referred to as REPO_NAME in this GLEP
> * A hard-coded repository-id which will be found in the repository tree at: metadata/repo_id - this will be referred to as REPO_ID in this GLEP
> Both names will contain no spaces, and only standard characters [TODO: references]
Portage externally will use user defined, internally it will do it's
own thing.
> Repositories
> ------------
>
> Each repository will contain:
> * the repo name in metadata/repo_id
> * repo information such as maintainer of the repo, notes on who hosts it, etc will be contained metadata/repo_info
> * unique packages.mask which will only apply to ebuilds within that specific repo.
>
> The REPO_ID must match the name that will be used for rsync
> Therefore, rsync://MyServer.tdl/REPO_ID/
No. It's arbitrary, and invalid to assume rsync is the only sync uri
that's going to be used- this isn't even getting into _remote_ repos.
*ANY* unique id tagged into a repo is just that, a magic constant in
it's metadata. Just that. No mandates about SYNC, file layout, etc,
will fly that bind to the metadata id.
> /etc/portage/*
> -------------
>
> In order to provide users with the current set of options and extend them so they can be customized to each repository, the structure of /etc/portage
> will remain similar with these changes:
> * /etc/portage/REPO_NAME/* will be the location of repository-specific portage files.
> * /etc/portage/ will continue to function over all repos
> ** ex) =sys-devel/gcc-4 -* in /etc/portage/package.keywords would use the latest gcc-4 regardless of what tree it comes from.
>
> The following new files will be added to /etc/portage:
> * /etc/portage/repositories.perfer - will contain each REPO_NAME in order of preferance, higher is more perfered. (Each REPO_NAME will be on a seperate line)
yuck. This is a bit of a waste for a single file.
> ** In the absence of this file portage should use repositories in alphabetical order.
directory returned ordering, not alpha- no ordering == set of
repositories, thus don't try to induce a fallback ordering.
> * /etc/portage/REPO_NAME/repository.id - contains the specific REPO_ID which REPO_NAME applies to.
This is going to induce more slowdown for any config instantiation-
more directories/files to scan. Iow, python -c 'import portage'
(which instantiates a config obj), is going to get slower, which will
piss off the slower system folk even further (hell, even with 1.5ghz
and decent IO it still is a 10s import for me uncached).
> */etc/portage/REPO_NAME/repository.conf - will contain any repository-specific options, which can include, but is not limited to, FEATURES="" C[XX]FLAGS="".
> ** This will also include a new variable; OPTIONS="" of which is similar to FEATURES, but modifies the way portage will handle that specific repository.
> A few examples of options which could be useful:
This seems a bit arbitrary.
> *** EXCLUDESYNC - Prevents portage from doing a sync on this repo.
And how are you going to specify the sync method for that repo?
> *** EXCLUDEUPDATE - Prevents portage from using ebuilds in this repo as updates for packages which currently reside in a different repo.
> *** EXCLUSIVEUPDATE - forces any update to any package which is from this repository to a newer version which resides inside of this repo.
And this is implemented how? Why is it specifying resolver
directives as repo attributes?
When is this forced -U going to be triggered? Sync? Next random
emerge call? How is this going to be bound to the resolver, let alone
the question of how build configuration data is going to be bound to
the actual build? yes, leading question there. :)
Where in this proposal, is it extending similar capabilities to
binpkgs?
> *** et al.
>
> All of the repository rsync URIs will be stored in /etc/make.conf
> SYNC="rsync://myfavoriterepo.org/myportage \ rsync://rsync.namerica.gentoo.org/gentoo-portage"
No. No no no no no.... at least this explains the metadata ID
being part of SYNC comment thought :)
> The Tree: /usr/portage -> /var/repositories/REPO_ID/
> ----------------------
> The repository tree will need to be moved, each repository will have its own folder: /var/repositories/REPO_ID/.
This directly castrates the user's ability to store the tree wherever
they want, loss of that long standing capability is an issue. No go
there.
> For compatibility reasons, /usr/portage will be treated as /var/repositories/gentoo-portage
Building hacks into portage isn't going to fly; no special cases. The
need for this is a sign that the forced FHS compliant path (instead of
letting the user do whatever they want) is a bad idea.
This beast can be accomplished without sacrificing our existing
flexibility.
> Ebuilds
> -------
>
> Ebuilds will now be able to have dependencies based on packages from specific repositories.
>
> * DEP Atoms now support the following format: =REPO_ID:SLOTNUM:CAT/EBUILD-X.Y.Z
> ** Ex1) >=MyRepo:2:sys-devel/gcc-4.0
> ** Ex2) <gentoo-portage::kde-base/kdelibs-3.5
> ** Ex3 ::foo/bar
> Dependency atoms that lack the new format (::) or do not have a REPO_ID will then just use any ebuild which fulfills the requirements.
Backwards compatibility? Protection for portage version that see this
and aren't capable of handling it? Why are you proposing the use/slot
additions be changed so slot is prefixed rather then postfixed?
> Backwards Compatibility
> =======================
> /usr/portage will be treated as /var/repositories/gentoo-portage so it would be possible to function with no changes after the upgrade.
This is a bit short... see above. lot more to it, especially since
you're trying to use stable.
Glep doesn't provide any way for knowing what repo's are active...
nor what repo's are overlaying each other, nor extending the
capabilties to anything beyond ebuild trees. Bintree? VDB? (yes,
extending slaving capabilities to vdb has uses).
The shortcomings I'm pointing at are historical- you're trying to
shoehorn into the existing portage configuration, thus I'm pointing
out the failings of it :)
Basically... this glep is exactly opposite of what I'm after, and what
I've already written in saviour. Saviour's configuration *currently*
is ini based, although that isn't a requirement (pluggable config parser
is there, although marienz is working on making my initial prototype
not suck).
Make.conf support will be there (backwards compatibility);
just will require a config parser written to convert make.conf format
into the internal config definitions.
The existing stable configuration is inherintly single master repo
set, single configuration, single domain, single root, single sync.
There's no groupping in make.conf (nor aux files), thus it's pretty
fricking hard to try and make it groupped. Further, no control over
the individual objects- no way to specify that a repo is remote fex.
All the make.conf support would do for saviour crap is translate it
into a single domain, single repo + overlays, etc.
If folks need more power (seperate caches, seperate syncs for repos, N
domains/roots...), they need to use a more powerful config format.
Via ini, you have repo definitions, defining the repo class to use,
location (if applicable), host (if applicable), cache reference (cache
instance to use, defined via it's own configuration item), slaving,
multiplexing together, package.* specifications (state the
filepath)... etc. Plus, user label (section label for that repo
definition).
The framework in place allows for a helluva lot more crazy stuff
(your own repo classes, say a strict R->L union of repos), I'm just
rambling off the obvious targets.
The configuration parser/handler actually addresses all of this crap
on it's own without forcing us to introduce hacks to try and shoehorn
N repo's into the current single repo config.
Old doc of details/intentions,
http://dev.gentoo.org/~ferringb/portage/3.0/dev-notes/framework/config.txt
.
It's generalized and flexible- prior to screaming off with his head
for the ini usage, please read through
http://dev.gentoo.org/~ferringb/portage/3.0/dev-notes ; there are a
_lot_ of good reasons, jotting down of intention, layout/design of
portage 3, etc.
Do the reading prior to the screaming ;)
Either way... here's how it *should* proceed from where I'm sitting.
Introduce full PORTDIR capabilities to all repos available- vdb, binpkg,
portdir/portdir_overlay. That right there is a major request people
have been pushing for, and in doing so the framework gets hammered
on/debugged.
Introducing resolver level constraints (match strictly from this repo)
requires atom extension among other things, and introduces it's own
class of problems. Introduce the capabilities while treating existing
repos as a single repo set, then extend it with what's needed for true
stand alone repos.
Either way, while discussion of standalone repos is good, I'm going to
kindly remind y'all that glep42 just needs to pass in a repo id- it's
not blocked by anything but minor api changes to portageq so that the
portage devs aren't forced to fix someone elses proposal down the
line (in the process breaking crap for users when we do so).
True stand alone repository capabilities aren't required/bound to
glep42, all that's required out of glep42 is that the syncing repo id
be used now (even if it may seem superfluous). Iow, news *should*
work regardless if it's an overlay or a stand alone repo.
Please keep that in mind.
~harring
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
next prev parent reply other threads:[~2005-12-17 23:18 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2005-12-17 8:17 [gentoo-dev] DRAFT GLEP: MULTI-REPO Andrew Muraco
2005-12-17 8:19 ` Andrew Muraco
2005-12-17 23:15 ` Brian Harring [this message]
2005-12-17 23:25 ` [gentoo-dev] draft glep: multi-repo Ciaran McCreesh
2005-12-18 0:14 ` Brian Harring
2005-12-18 0:24 ` Ciaran McCreesh
2005-12-18 1:01 ` Brian Harring
2005-12-18 1:08 ` Ciaran McCreesh
2005-12-18 1:31 ` Brian Harring
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20051217231548.GB22142@nightcrawler.e-centre.net \
--to=ferringb@gentoo.org \
--cc=gentoo-dev@lists.gentoo.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox