From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from lists.gentoo.org ([140.105.134.102] helo=robin.gentoo.org) by nuthatch.gentoo.org with esmtp (Exim 4.54) id 1EnlKC-0000hd-2g for garchives@archives.gentoo.org; Sat, 17 Dec 2005 23:18:48 +0000 Received: from robin.gentoo.org (localhost [127.0.0.1]) by robin.gentoo.org (8.13.5/8.13.5) with SMTP id jBHNI4Qq021472; Sat, 17 Dec 2005 23:18:04 GMT Received: from cubert.e-centre.net (morbo.e-centre.net [66.154.82.3]) by robin.gentoo.org (8.13.5/8.13.5) with ESMTP id jBHNGCEZ024743 for ; Sat, 17 Dec 2005 23:16:12 GMT Received: from [10.3.1.19] (helo=barracuda2.stayonline.net) by cubert.e-centre.net with esmtp (Exim 4.50) id 1EnlHe-00018D-UC for gentoo-dev@lists.gentoo.org; Sat, 17 Dec 2005 18:16:11 -0500 X-ASG-Debug-ID: 1134861369-26896-65-0 X-Barracuda-URL: http://10.3.1.19:8000/cgi-bin/mark.cgi Received: from et-pdx-2.site.stayonline.net (unknown [65.200.64.131]) by barracuda2.stayonline.net (Spam Firewall) with ESMTP id B8BB1C97D1 for ; Sat, 17 Dec 2005 18:16:09 -0500 (EST) Received: from nightcrawler ([172.16.1.202]) by et-pdx-2.site.stayonline.net (8.12.6/8.12.6) with ESMTP id jBHNFv5j020282 for ; Sat, 17 Dec 2005 23:15:58 GMT Date: Sat, 17 Dec 2005 15:15:48 -0800 From: Brian Harring To: gentoo-dev@lists.gentoo.org X-ASG-Orig-Subj: Re: [gentoo-dev] draft glep: multi-repo Subject: Re: [gentoo-dev] draft glep: multi-repo Message-ID: <20051217231548.GB22142@nightcrawler.e-centre.net> References: <43A3C9A8.9050608@leetworks.com> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@gentoo.org Reply-to: gentoo-dev@lists.gentoo.org Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="jq0ap7NbKX2Kqbes" Content-Disposition: inline In-Reply-To: <43A3C9A8.9050608@leetworks.com> User-Agent: Mutt/1.5.11 X-Virus-Scanned: by Barracuda Spam Firewall at stayonline.net X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=4.0 tests= X-Barracuda-Spam-Report: Code version 3.02, rules version 3.0.6391 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- X-Archives-Salt: e0a81994-067b-4afb-9160-92769466d108 X-Archives-Hash: 8b5fbe54125eae6d8b500c9213b4e09a --jq0ap7NbKX2Kqbes Content-Type: text/plain; charset=utf8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Pardon bluntness; don't mean offense, just specifically picking the=20 hell out of this proposal to make a point (lucky you) :) On Sat, Dec 17, 2005 at 03:17:44AM -0500, Andrew Muraco wrote: > Attached is a draft of a glep for formalizing multiple-repository support I appreciate trying to chip in, but frankly this glep needs a lot more=20 thought put into it. =20 Further, I _really_ do not see the point of glepping this either. Puking= =20 up proposals due to folks making noise is a waste of time- don't=20 document/propose just because folks are making noise- do it for=20 large scale changes, or conflict, not because someone requires a=20 glep/spec before they're willing to listen to the _developers_ of a=20 project about how to integrate a new feature into _their_ project. > This is far from ideal in many ways, but i'm too tired and I drank too=20 > much caffine to be sane. >=20 > Comments, objections, anything consructive is welcome. Inlined... > Abstract > =3D=3D=3D=3D=3D=3D=3D=3D > To implement a functional and expandable method for Portage to support mu= ltiple repositories. >=20 > Motivation > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > Multiple Repository support is needed, this GLEP is to address this need. define multiple repository. We _have_ multi repo already (binpkg and portdir, let alone overlays). > Specification > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >=20 > Portage will make use of two (2) ways to address repositories: > * A User-defined name, which is likely to be used as a convinance in most= situations - this will be referred to as REPO_NAME in this GLEP > * A hard-coded repository-id which will be found in the repository tree a= t: metadata/repo_id - this will be referred to as REPO_ID in this GLEP > Both names will contain no spaces, and only standard characters [TODO: re= ferences] Portage externally will use user defined, internally it will do it's=20 own thing. > Repositories > ------------ >=20 > Each repository will contain: > * the repo name in metadata/repo_id > * repo information such as maintainer of the repo, notes on who hosts it,= etc will be contained metadata/repo_info > * unique packages.mask which will only apply to ebuilds within that speci= fic repo. >=20 > The REPO_ID must match the name that will be used for rsync > Therefore, rsync://MyServer.tdl/REPO_ID/ No. It's arbitrary, and invalid to assume rsync is the only sync uri=20 that's going to be used- this isn't even getting into _remote_ repos. *ANY* unique id tagged into a repo is just that, a magic constant in=20 it's metadata. Just that. No mandates about SYNC, file layout, etc,=20 will fly that bind to the metadata id. > /etc/portage/* > ------------- >=20 > In order to provide users with the current set of options and extend them= so they can be customized to each repository, the structure of /etc/portag= e=20 > will remain similar with these changes: > * /etc/portage/REPO_NAME/* will be the location of repository-specific po= rtage files. > * /etc/portage/ will continue to function over all repos > ** ex) =3Dsys-devel/gcc-4 -* in /etc/portage/package.keywords would use t= he latest gcc-4 regardless of what tree it comes from. >=20 > The following new files will be added to /etc/portage: > * /etc/portage/repositories.perfer - will contain each REPO_NAME in order= of preferance, higher is more perfered. (Each REPO_NAME will be on a seper= ate line) yuck. This is a bit of a waste for a single file. > ** In the absence of this file portage should use repositories in alphabe= tical order. directory returned ordering, not alpha- no ordering =3D=3D set of=20 repositories, thus don't try to induce a fallback ordering. > * /etc/portage/REPO_NAME/repository.id - contains the specific REPO_ID wh= ich REPO_NAME applies to. This is going to induce more slowdown for any config instantiation-=20 more directories/files to scan. Iow, python -c 'import portage'=20 (which instantiates a config obj), is going to get slower, which will=20 piss off the slower system folk even further (hell, even with 1.5ghz=20 and decent IO it still is a 10s import for me uncached). > */etc/portage/REPO_NAME/repository.conf - will contain any repository-spe= cific options, which can include, but is not limited to, FEATURES=3D"" C[XX= ]FLAGS=3D"". > ** This will also include a new variable; OPTIONS=3D"" of which is simila= r to FEATURES, but modifies the way portage will handle that specific repos= itory.=20 > A few examples of options which could be useful:=20 This seems a bit arbitrary. > *** EXCLUDESYNC - Prevents portage from doing a sync on this repo. And how are you going to specify the sync method for that repo? > *** EXCLUDEUPDATE - Prevents portage from using ebuilds in this repo as u= pdates for packages which currently reside in a different repo. > *** EXCLUSIVEUPDATE - forces any update to any package which is from this= repository to a newer version which resides inside of this repo.=20 And this is implemented how? Why is it specifying resolver=20 directives as repo attributes? When is this forced -U going to be triggered? Sync? Next random=20 emerge call? How is this going to be bound to the resolver, let alone=20 the question of how build configuration data is going to be bound to=20 the actual build? yes, leading question there. :) Where in this proposal, is it extending similar capabilities to=20 binpkgs? > *** et al. >=20 > All of the repository rsync URIs will be stored in /etc/make.conf > SYNC=3D"rsync://myfavoriterepo.org/myportage \ rsync://rsync.namerica.gen= too.org/gentoo-portage"=20 No. No no no no no.... at least this explains the metadata ID=20 being part of SYNC comment thought :) > The Tree: /usr/portage -> /var/repositories/REPO_ID/ > ---------------------- > The repository tree will need to be moved, each repository will have its = own folder: /var/repositories/REPO_ID/. This directly castrates the user's ability to store the tree wherever=20 they want, loss of that long standing capability is an issue. No go=20 there. > For compatibility reasons, /usr/portage will be treated as /var/repositor= ies/gentoo-portage Building hacks into portage isn't going to fly; no special cases. The=20 need for this is a sign that the forced FHS compliant path (instead of=20 letting the user do whatever they want) is a bad idea. This beast can be accomplished without sacrificing our existing=20 flexibility. > Ebuilds > ------- >=20 > Ebuilds will now be able to have dependencies based on packages from spec= ific repositories. >=20 > * DEP Atoms now support the following format: =3DREPO_ID:SLOTNUM:CAT/EBUI= LD-X.Y.Z > ** Ex1) >=3DMyRepo:2:sys-devel/gcc-4.0 > ** Ex2) ** Ex3 ::foo/bar > Dependency atoms that lack the new format (::) or do not have a REPO_ID w= ill then just use any ebuild which fulfills the requirements. Backwards compatibility? Protection for portage version that see this=20 and aren't capable of handling it? Why are you proposing the use/slot=20 additions be changed so slot is prefixed rather then postfixed? > Backwards Compatibility > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > /usr/portage will be treated as /var/repositories/gentoo-portage so it wo= uld be possible to function with no changes after the upgrade. This is a bit short... see above. lot more to it, especially since=20 you're trying to use stable. Glep doesn't provide any way for knowing what repo's are active...=20 nor what repo's are overlaying each other, nor extending the=20 capabilties to anything beyond ebuild trees. Bintree? VDB? (yes,=20 extending slaving capabilities to vdb has uses). The shortcomings I'm pointing at are historical- you're trying to=20 shoehorn into the existing portage configuration, thus I'm pointing=20 out the failings of it :) Basically... this glep is exactly opposite of what I'm after, and what=20 I've already written in saviour. Saviour's configuration *currently*=20 is ini based, although that isn't a requirement (pluggable config parser=20 is there, although marienz is working on making my initial prototype=20 not suck). Make.conf support will be there (backwards compatibility);=20 just will require a config parser written to convert make.conf format=20 into the internal config definitions. The existing stable configuration is inherintly single master repo=20 set, single configuration, single domain, single root, single sync. =20 There's no groupping in make.conf (nor aux files), thus it's pretty=20 fricking hard to try and make it groupped. Further, no control over=20 the individual objects- no way to specify that a repo is remote fex. All the make.conf support would do for saviour crap is translate it=20 into a single domain, single repo + overlays, etc. If folks need more power (seperate caches, seperate syncs for repos, N=20 domains/roots...), they need to use a more powerful config format. Via ini, you have repo definitions, defining the repo class to use,=20 location (if applicable), host (if applicable), cache reference (cache=20 instance to use, defined via it's own configuration item), slaving,=20 multiplexing together, package.* specifications (state the=20 filepath)... etc. Plus, user label (section label for that repo=20 definition). =20 The framework in place allows for a helluva lot more crazy stuff=20 (your own repo classes, say a strict R->L union of repos), I'm just=20 rambling off the obvious targets. The configuration parser/handler actually addresses all of this crap=20 on it's own without forcing us to introduce hacks to try and shoehorn=20 N repo's into the current single repo config. Old doc of details/intentions,=20 http://dev.gentoo.org/~ferringb/portage/3.0/dev-notes/framework/config.txt= =20 =2E It's generalized and flexible- prior to screaming off with his head=20 for the ini usage, please read through=20 http://dev.gentoo.org/~ferringb/portage/3.0/dev-notes ; there are a=20 _lot_ of good reasons, jotting down of intention, layout/design of=20 portage 3, etc. Do the reading prior to the screaming ;) Either way... here's how it *should* proceed from where I'm sitting. =20 Introduce full PORTDIR capabilities to all repos available- vdb, binpkg,=20 portdir/portdir_overlay. That right there is a major request people=20 have been pushing for, and in doing so the framework gets hammered=20 on/debugged. Introducing resolver level constraints (match strictly from this repo)=20 requires atom extension among other things, and introduces it's own=20 class of problems. Introduce the capabilities while treating existing=20 repos as a single repo set, then extend it with what's needed for true=20 stand alone repos. Either way, while discussion of standalone repos is good, I'm going to=20 kindly remind y'all that glep42 just needs to pass in a repo id- it's=20 not blocked by anything but minor api changes to portageq so that the=20 portage devs aren't forced to fix someone elses proposal down the=20 line (in the process breaking crap for users when we do so). True stand alone repository capabilities aren't required/bound to=20 glep42, all that's required out of glep42 is that the syncing repo id be used now (even if it may seem superfluous). Iow, news *should*=20 work regardless if it's an overlay or a stand alone repo. Please keep that in mind. ~harring --jq0ap7NbKX2Kqbes Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFDpJwkvdBxRoA3VU0RApIUAJ9MnHhx2DUBJcN6ex/E0Azs1PT+lgCeMCHA XY/6WOHSMFHTXLYBezhsSk8= =YfdB -----END PGP SIGNATURE----- --jq0ap7NbKX2Kqbes-- -- gentoo-dev@gentoo.org mailing list