From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from lists.gentoo.org ([140.105.134.102] helo=robin.gentoo.org) by nuthatch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1GTkfb-0004Gc-R7 for garchives@archives.gentoo.org; Sat, 30 Sep 2006 19:38:44 +0000 Received: from robin.gentoo.org (localhost [127.0.0.1]) by robin.gentoo.org (8.13.8/8.13.6) with SMTP id k8UJbLhR012541; Sat, 30 Sep 2006 19:37:21 GMT Received: from rwcrmhc12.comcast.net (rwcrmhc12.comcast.net [216.148.227.152]) by robin.gentoo.org (8.13.8/8.13.6) with ESMTP id k8UJYYUS021813 for ; Sat, 30 Sep 2006 19:34:34 GMT Received: from seldon (c-24-21-135-117.hsd1.or.comcast.net[24.21.135.117]) by comcast.net (rwcrmhc12) with SMTP id <20060930193432m1200oq12fe>; Sat, 30 Sep 2006 19:34:33 +0000 Date: Sat, 30 Sep 2006 12:34:31 -0700 From: Brian Harring To: gentoo-dev@lists.gentoo.org Subject: Re: [gentoo-dev] RFC about another *DEPEND variable Message-ID: <20060930193430.GA9496@seldon> References: <45126B07.6030403@gentoo.org> <200609270224.42169.vapier@gentoo.org> <20060927075422.GA7684@seldon> <200609301401.09089.vapier@gentoo.org> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="bg08WKrSYDhXBjb5" Content-Disposition: inline In-Reply-To: <200609301401.09089.vapier@gentoo.org> User-Agent: Mutt/1.5.13 (2006-08-11) X-Archives-Salt: 39bd3d42-7d28-4d34-8f64-e5fe54f34bdc X-Archives-Hash: d3f45270a2e8eba6a43e837d66604873 --bg08WKrSYDhXBjb5 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Sep 30, 2006 at 02:01:08PM -0400, Mike Frysinger wrote: > On Wednesday 27 September 2006 03:54, Brian Harring wrote: > > > > Bleh, this is getting back to exactly my point that it's unbounded > > > > resolution. To support this, every step of execution would require > > > > scanning for dangling nodes to punt; aside from invalidating -p's > > > > output it *still* is a collision-protect hit. > > > > > > when the package upgrade detects an ABI change you recalculate the > > > packages that need to be rebuilt ... > > > > Reiterating, this screws over any form of up front calculation; > > dialup users/per minute connections can't rely on emerge -f to > > actually fetch all involved sources, -p results ain't guranteed > > valid, parallelization must now lock at each threads merge on the off > > chance a recalc is forced. >=20 > it does indeed prevent full up front calculation. we can always make the= =20 > behavior configurable; revdep on the fly or allow people to break it up. = the=20 > key being that their system will still continue to function as the ABI li= b=20 > has been preserved automagically Folks aren't that daft; you make it an optional component, it means a=20 *proper* solution will never be pushed because the duct tape is in=20 place already. If that's what folks want, sure, but what you're proposing is just=20 sliding NEEDED in as the defacto solution without labeling it as such. > > > dangling nodes get recorded in the new package and considering these = old > > > files are not detrimental to the health of the system, you can do suc= h a > > > cleanup once at the end of the emerge > > > > It's not orphaning files, but your scheme doesn't work for any form of > > interruption; failed builds, user intervention, etc, they all can > > leave the lib stuck in the new contents. >=20 > huh ? i outlined in a previous e-mail how by simply ordering the operati= ons=20 > sanely, there is no race condition Re-read your emails, and mine please. The scenario I pointed out was=20 ctrl+c'ing the merge while it's doing a rebuild for lib1 to lib2. Now how does portage know that it needs to complete that upgrade for=20 the next emerge action? How does it know to even scan for lib1, let=20 alone punting? It's *not* simple, and I keep pointing out this issue (and you're=20 missing it every damn time). Please address it, or point to where you=20 have (gone over the thread and still not seeing anything even=20 remotely touching this flaw). > > Dealing with that possibility means the manager has to maintain on > > disk a list of the refcount of each dangling libs to decrement, > > unmerge has to modify said list, etc. >=20 > which is already being done in the NEEDED file ... funny how unpainless i= t is=20 > to generate that file First of all, unpainless is painful. Which is apt, actually. Familiar with old style virtuals? That whole, "you have to walk the=20 whole vdb to collect all old style virtuals" ? This is the same damn thing. There is no refcount maintained via NEEDED, just a list of libs a=20 binary uses; you _still_ have to walk the damn vdb to build the=20 refcount list. Now either you're forcing a fairly huge mapping in memory, or you're=20 forcing a scan of the vdb for every pkg operation that has NEEDED=20 entries it will install. So no, NEEDED doesn't cover this, and my point still stands. > > > > It also involves changing vdb nodes from "installed and usable" to > > > > "installed/usable" or "lingering" > > > > > > no ... the old versions get merged into the new one as their existenc= e is > > > purely hidden > > > > How exactly are they going to be hidden? A new file? If so, > > backwards compatibility for vdb for transitioning in such a support > > has to be addressed. >=20 > purely hidden from the standpoint of any new package trying to use it ...= =20 > since you're only installing the runtime ABI library, you cannot link aga= inst=20 > it or utilize it any way other than existing files. Except for dlopen, but thats again besides my original point; how do=20 you note to portage that the lib is one to watch so it can be punted? You're suggesting it get shoved into contents, and for portage to=20 identify it, it has to do a revdep mapping on the fly to find it. =20 This *sucks* massively, both from a speed and complexity standpoint;=20 further, the lib isn't hidden from pkg ops (say quickpkging, or=20 binpkging) in any way so the cruft spreads. That's surprisingly minor=20 in comparison to implications of relying on refcounting to identify=20 the lib to punt. If the lib to punt isn't tracked in some fashion, the only algo=20 you're left with is attempting to find all libs that have a refcount=20 of zero, and punt those- obviously, this is going to screw up just=20 about every single dlopen based linkage, literally, suck an algo=20 won't spot that python dlopen's it's modules, and would think the=20 refcount was zero (thus the so could get booted). The response there is to add blacklists, directories to *not* inspect,=20 which is a further hack to try and make this tank fly. *OR*, you're stuck maintaining a seperated list of libs to punt,=20 rather then just trying to slide the lib into existing CONTENTS. > > > > Tracking BINCOMPAT should *not* be that hard. > > > > > > it's one more thing to keep track of and considering all of the > > > possibilities i outlined, a single maintainer can easily lose his san= ity > > > ... or you force wasted rebuilds on people when it's only needed in s= ome > > > circumstances > > > > How exactly is this forcing wasted rebuilds? You're stating that > > maintainers are going to bump it willy nilly. As I said, it is a key > > that would be bumped *as needed*, and would only affected pkgs that > > specified that node as a binding dep (specially marked atom). >=20 > no, i mean for example scenarios where a package provides more than one= =20 > library (say libfoo and libbar) and only one of those changes ABI values = (say=20 > libfoo.0 -> libfoo.1). if another package links against just libbar, you= 've=20 > got a pointless rebuild. If one changes it's version, it's a fair bet that any consumer of that=20 pkg is linked to more then just that lib; as such they would be=20 rebuilt *anyways*. > > Seriously, maintainers ought to know *now* when they're forcing a > > revdep on folks systems, I'm not seeing the massive overhead from > > pushing that info into a var, nor am I seeing mass forced useless > > rebuilds. >=20 > there's a ton of things maintainers ought to know ... pretty soon our pac= kage=20 > maintainers are going to have to be gods if they want to write an ebuild = as=20 > they're going to have to know every detail about the package inside and o= ut Sorry, but if a developer is bumping a pkg and doesn't even look to=20 see if the damn thing is potentially going to break others via soname=20 changes, that maintainer is being an idiot. Being aware of a high level detail like "hey, my package installs a=20 library, and they changed the soname" isn't being god like, it's=20 being at least haphazardly concerned about QA of the tree. Worth noting this exact scenario is already laid out in the quizes=20 too; 'sposed to know the affects of stabilizing a library. > > Realistically, just the same as the NEEDED solution has holes, the > > maintaining dev can screw up and miss a BINCOMPAT bump. Difference is > > that they can do something for BINCOMPAT; hell, QA checks can be > > automated to catch missing BINCOMPAT bumps. >=20 > what's the difference ? in my scenario they dont have to do anything bec= ause=20 > the bump would have been caught automatically ? Your solution has holes up the ying yang for actual=20 implementation/handling of it due to the fact it's forcing the=20 resolver to be an effective recalc at each step, and relies on duct=20 taping metadata handling to try and exempt bits of lingering data. You move that data up front, the implementation is actually sane, it's=20 deterministic (thus *representative* to the user) and is more=20 powerful. > > KEYWORDed arches bit, bit unlikely that the underlying arch is > > actually going to screw with the soname, thus I'd want actual examples > > backing it up. >=20 > how about libc.so from glibc and libgcc.so from gcc ? or would you like = other=20 > packages ? Considering such a change would be internal ABI, NEEDED doesn't=20 actually fix anything; if it were a soname change, level playing=20 ground again. Admittedly, BINCOMPAT breaks down there due to not being fine grained=20 enough if it's internal changes; that however doesn't make NEEDED a=20 good solution, just means that my alternative has a hole. > > Besides, again, for keywording, the dev *should* be compiling it, so > > there is a way to catch it :P. A revbump isn't going to break things > > unless the dev is introducing something intrusive, minor version bumps > > (1.1.0 to 1.1.1) shouldn't be again, introducing anything intrusive; > > regardless, dev should know the high level affects of their change >=20 > uhh, there is no such requirement at this time for revbumping on differen= t=20 > arches ... currently we say the maintainer tests for his arches and leave= s=20 > all the others as ~arch Reiterating; devs should know the high level affects of their changes. =20 If it's going to change the soname version, they should know from the=20 get go- unless it's an arch specific breakage (which I still posit is=20 the rare/corner case), they will *see* it for their arch and bump it=20 already. They keywording comment above is specific to moving from ~arch to arch=20 also; for new bumps, the ~arch keywords carry forward, but see the=20 paragraph above re: it. > > Inducing a revdep-rebuild is definitely one of those high level > > changes they should be aware of *prior* to the bump. >=20 > and yet we look at our history of so many packages being bumped with ABI= =20 > changes and users having broken-as-shit systems ... i'm accounting for th= e=20 > worst; not hoping for the best ABI changes aren't fixed by NEEDED. Presume you meant soname changes. Stating that things are broken doesn't make NEEDED automagically=20 better either; *both* enable a way to fix it, so you need to justify=20 the "accounting for the worst; not hoping for the best". > > > or dig > > > through the source code line by line and hope to catch all such cases= by > > > hand/eye ? > > > > Bit of FUD here, although I spose I'll just point out that if so's > > change as massively as you're implying above, the affects on -p are > > that much worse. >=20 > awesome, mark something you disagree with as FUD, problem solved. my poi= nt is=20 > that you can never know completely for sure the behavior of a package wit= hout=20 > digging through the entire source code. It's FUD due to the fact NEEDED suffers the same fucking issue. The=20 irony is that BINCOMPAT would at least give a knob to mark it as a=20 breakage, NEEDED cannot. > > bind the checks in as a QA measure, enabled via FEATURES=3Dstricter, > > used to maintain a metadata var. >=20 > a lot of things fail already with FEATURES=3Dstricter ... too bad we dont= have=20 > per-package env var support as then maintainers could just flag all their= own=20 > packages as stricter without wanting to shoot themselves due to everyone= =20 > else's package failures Take it up with your fellow portage devs then. It was shot down=20 internally multiple times due to the affects it has on the python=20 side. If you want to get into per package features, split a thread and I'll=20 let marius argue over it. > > Literally, pkg x version y forced a rebuild. revdep is annoying, but > > stats would be useful to actually evaluate the seperate proposals; > > related, getting stats for how often the various updaters were > > required to be ran for a particular pkgs upgrade would be useful in > > evaluating that particular gap NEEDED has. >=20 > any openssl version bump where the numerical value changes ... only the l= etter=20 > updates preserve ABI compat (0.9.7[a-z] are compat) >=20 > expat-1.x -> expat-2.x >=20 > iirc, tcl-8.3.x -> tcl-8.4.x >=20 > readline-4.x -> readline-5.x >=20 > ncurses-4.x -> ncurses-5.x Was looking for stats comparing breakage to a subset of the tree; iow,=20 out of N packages, how many actually induce a rebuild? Yes it's=20 slightly nasty generating those stats (tinderbox?), but it would be=20 useful. Frankly, if you don't believe me or think my points are correct about=20 how a NEEDED based solution is going to suck, zac/jason/genone/anyone=20 else. They're going to point out the same flaws. ~harring --bg08WKrSYDhXBjb5 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) iD8DBQFFHsbGsiLx3HvNzgcRApygAJ48X822HuFq0vM3mU5DG28ntez92ACglFim +4afwB5RoNUFMUBi7VcLFTM= =XaMp -----END PGP SIGNATURE----- --bg08WKrSYDhXBjb5-- -- gentoo-dev@gentoo.org mailing list