[gentoo-dev] RFC about another *DEPEND variable

[gentoo-dev] RFC about another *DEPEND variable

[Alin Nastac]

[-- Attachment #1: Type: text/plain, Size: 1282 bytes --]

I'm annoyed about impossibility to fix a certain class of breakages
(other than reemerging the failing package). I am referring to the
breakages occurred when foo has been upgraded, but the bar package
cannot work with it because it was build against the old foo version.

We all had to run revdep-rebuild once in awhile, for fixing dynamic
linkage problems, but unfortunately revdep-rebuild cannot fix another
kind of incompatibility, namely dynamic version check implemented in
some packages.
For instance, the recent openssl-0.9.8* update broke dev-libs/neon (and
consequently subversion) because neon library isn't happy just by
linking with libssl.so.0.9.7 but also check the libssl version when
loads the ssl library. Another example is the subtle dependency between
the pppd version and pppd plugins (net-dialup/pptpd needs to be rebuild
every time you change PV of the net-dialup/ppp because pptpd builds a
plugin for the ppp daemon).

I reckon this could be solved by yet another *DEPEND variable. The only
atoms accepted by this variable would be "CATEGORY/PN". Every time when
a package gets updated from PV1 to PV2 (distinct versions, revisions
will not count), portage will automatically re-emerge those packages
that depend on it.

Thoughts?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Luca Barbato]

Alin Nastac wrote:
> I reckon this could be solved by yet another *DEPEND variable. The only
> atoms accepted by this variable would be "CATEGORY/PN". Every time when
> a package gets updated from PV1 to PV2 (distinct versions, revisions
> will not count), portage will automatically re-emerge those packages
> that depend on it.
> 
> Thoughts?
> 

It would require revdep resolution on emerge... how painful would be?

lu

-- 

Luca Barbato

Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC about another *DEPEND variable

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 2330 bytes --]

On Thu, Sep 21, 2006 at 01:38:59PM +0200, Luca Barbato wrote:
> Alin Nastac wrote:
> >I reckon this could be solved by yet another *DEPEND variable. The only
> >atoms accepted by this variable would be "CATEGORY/PN". Every time when
> >a package gets updated from PV1 to PV2 (distinct versions, revisions
> >will not count), portage will automatically re-emerge those packages
> >that depend on it.
> >
> >Thoughts?
> >
> 
> It would require revdep resolution on emerge... how painful would be?

Off the top of my head, adding revdeps (period) for unmerge 
functionality (fex) is the harder part; slipping something of this 
sort in is just a logic tweak.

The problem with this proposal however is that it's attempting 
automatic revdep based off of version; _any_ non-rev version bump is 
way too rebuild happy.

Proposal I was pushing a while back was addition of a metadata key; 
it's not exactly .so version, but pretty close- a _manually_ 
maintained counter var in the ebuild that represents the abi 
compatibility for that packages versions.

Example would be openssl-0.9.7, you stick BINCOMPAT=0.9.7 in it, 
in openssl-0.9.8, you stick BINCOMPAT=0.9.8 in it, for a replace op 
the resolver sees that it's breaking compatibility and knows to 
rebuild any revdeps.

Why have the explicit var?  Because 0.9.7a through 0.9.7c may all be 
compatible, but 0.9.7d isn't.  If you force an automatic algo that 
tries to (effectively) guess, you get a lot of rebuilds through a,b,c, 
end result being folks likely update less because it becomes a bigger 
pain in the ass.

There is one flaw with this though; packages can provide both 
libraries _and_ binaries.  Our dependencies don't represent whether 
the dep is actual linkage, or just commandline consuming, so (using 
the openssl example) any package that invokes openssl via the 
commandline to do a few simple chksum ops gets rebuilt, despite the 
fact it wasn't affected by linkage change ups.

So... short version, at least what jstubbs/marius/myself hashed out in 
irc a long while back, need binding dependencies and actual tracking 
of the lib breakage in the metadata.

Alternative to shoving an extra key in would be extending slot rules, 
but that can be somewhat ugly.

My 2 cents, for what its worth.
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Alin Nastac]

[-- Attachment #1: Type: text/plain, Size: 946 bytes --]

Luca Barbato wrote:
> Alin Nastac wrote:
>> I reckon this could be solved by yet another *DEPEND variable. The only
>> atoms accepted by this variable would be "CATEGORY/PN". Every time when
>> a package gets updated from PV1 to PV2 (distinct versions, revisions
>> will not count), portage will automatically re-emerge those packages
>> that depend on it.
>>
>> Thoughts?
>>
>
> It would require revdep resolution on emerge... how painful would be?
>
I don't know anything about portage intricacies, but I guess it would be
fairly easy to have a map of CATEGORY1/PN1 -> CATEGORY2/PN2 key-values,
where package 2 depends on package 1 (package 2 is the one that defines
the xxxDEPEND variable). In order to add such (key, value) in the map
the following assumptions must be satisfied:
   - package 2 must be installed (mandatory)
   - package 1 must be a direct or indirect RDEPEND dependency of the
package 2 (optional).



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 720 bytes --]

On Thursday 21 September 2006 06:35, Alin Nastac wrote:
> For instance, the recent openssl-0.9.8* update broke dev-libs/neon (and
> consequently subversion) because neon library isn't happy just by
> linking with libssl.so.0.9.7 but also check the libssl version when
> loads the ssl library. Another example is the subtle dependency between
> the pppd version and pppd plugins (net-dialup/pptpd needs to be rebuild
> every time you change PV of the net-dialup/ppp because pptpd builds a
> plugin for the ppp daemon).

i guess you're referring to the plugins eh ?  this was proposed some time ago 
by eradicator but i dont know where that development track left off (my guess 
is nowhere significant)
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 660 bytes --]

On Thursday 21 September 2006 07:59, Brian Harring wrote:
> Why have the explicit var?  Because 0.9.7a through 0.9.7c may all be
> compatible, but 0.9.7d isn't.  If you force an automatic algo that
> tries to (effectively) guess, you get a lot of rebuilds through a,b,c,
> end result being folks likely update less because it becomes a bigger
> pain in the ass.

if it isnt compatible then it shouldnt have the same SONAME, simple as 
that ... that is after all the point of encoding the ABI version number into 
the SONAME

forcing devs to maintain a manual var which is basically duplicating the 
SONAME smells like maintenance nightmare
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 1359 bytes --]

On Thu, Sep 21, 2006 at 09:52:27AM -0400, Mike Frysinger wrote:
> On Thursday 21 September 2006 07:59, Brian Harring wrote:
> > Why have the explicit var?  Because 0.9.7a through 0.9.7c may all be
> > compatible, but 0.9.7d isn't.  If you force an automatic algo that
> > tries to (effectively) guess, you get a lot of rebuilds through a,b,c,
> > end result being folks likely update less because it becomes a bigger
> > pain in the ass.
> 
> if it isnt compatible then it shouldnt have the same SONAME, simple as 
> that ... that is after all the point of encoding the ABI version number into 
> the SONAME
> 
> forcing devs to maintain a manual var which is basically duplicating the 
> SONAME smells like maintenance nightmare

I agree; while I'm labeling it ABI, includes both bad soname handling 
and seperate sonames.

Re: forcing devs... the request was to fold revdep-rebuild into 
resolution; in other words, 3 options
1) situation gets ignored, stays as is
2) all packages are somehow fixed (ultra restrictive deps) to never 
require revdep-rebuild
3) revdep-rebulid capabilities get inline into resolution.

Feel free to point out a 4th option if I'm missing it, but for the 
request, that's what exists afaict; meanwhile, stating that pkgs are 
being stupid, while true, doesn't actually solve the issue :)

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Donnie Berkholz]

Mike Frysinger wrote:
> On Thursday 21 September 2006 07:59, Brian Harring wrote:
>> Why have the explicit var?  Because 0.9.7a through 0.9.7c may all be
>> compatible, but 0.9.7d isn't.  If you force an automatic algo that
>> tries to (effectively) guess, you get a lot of rebuilds through a,b,c,
>> end result being folks likely update less because it becomes a bigger
>> pain in the ass.
> 
> if it isnt compatible then it shouldnt have the same SONAME, simple as 
> that ... that is after all the point of encoding the ABI version number into 
> the SONAME
> 
> forcing devs to maintain a manual var which is basically duplicating the 
> SONAME smells like maintenance nightmare

Not adding it into the ebuild means that it's impossible to show in 
advance what packages will actually be installed, because you don't know 
whether the sover will bump.

Thanks,
Donnie
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC about another *DEPEND variable

[Alin Nastac]

[-- Attachment #1: Type: text/plain, Size: 909 bytes --]

Brian Harring wrote:
> On Thu, Sep 21, 2006 at 01:38:59PM +0200, Luca Barbato wrote:
>   
> There is one flaw with this though; packages can provide both 
> libraries _and_ binaries.  Our dependencies don't represent whether 
> the dep is actual linkage, or just commandline consuming, so (using 
> the openssl example) any package that invokes openssl via the 
> commandline to do a few simple chksum ops gets rebuilt, despite the 
> fact it wasn't affected by linkage change ups.
>   
I like BINCOMPAT proposal but it solves only half of the problem. You
assume that all dependent packages cares about binary compatibility.
Why not using a BDEPEND var in those dependent packages affected by the
BINCOMPAT values of their dependencies?

For instance, I would set the following:
 - in net-dialup/ppp ebuild:     BINCOMPAT=${PV}
 - in net-dialup/pptpd ebuild:   BDEPEND="net-dialup/ppp"



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 709 bytes --]

On Thursday 21 September 2006 10:14, Donnie Berkholz wrote:
> Not adding it into the ebuild means that it's impossible to show in
> advance what packages will actually be installed, because you don't know
> whether the sover will bump.

sometimes you dont know as the ABI bump may be arch or feature specific ... 
pathological case i agree ;)

you would certainly know between the time you merge the new one and unmerge 
the old one though ... but now we tread into the territory of "portage should 
not unmerge ABI libraries until all consumers have been destroyed" which is 
better than the current craptastic situation of "you gotta revdep-rebuild 
after the fact with a broken system"
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 959 bytes --]

On Thursday 21 September 2006 10:04, Brian Harring wrote:
> I agree; while I'm labeling it ABI, includes both bad soname handling
> and seperate sonames.

those people should be smacked (for the interest of disclosure, i have 
violated the "bad soname" rule for the sake of following upstream)

> Feel free to point out a 4th option if I'm missing it, but for the
> request, that's what exists afaict; meanwhile, stating that pkgs are
> being stupid, while true, doesn't actually solve the issue :)

4) portage maintains a list of ABI SONAMEs in use and does not unmerge the 
library until all consumers are gone

i'm referring to the specific file of course, not anything else in the 
package ... so integrating the hack eutils.eclass:preserve_old_lib() into 
portage so it isnt a hack (not a knock against the current implementation 
here; it's always going to be a hack until portage manages proper unmerging 
of the ABI library)
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1312 bytes --]

On Thursday 21 September 2006 10:38, Alin Nastac wrote:
> Brian Harring wrote:
> > There is one flaw with this though; packages can provide both
> > libraries _and_ binaries.  Our dependencies don't represent whether
> > the dep is actual linkage, or just commandline consuming, so (using
> > the openssl example) any package that invokes openssl via the
> > commandline to do a few simple chksum ops gets rebuilt, despite the
> > fact it wasn't affected by linkage change ups.
>
> I like BINCOMPAT proposal but it solves only half of the problem. You
> assume that all dependent packages cares about binary compatibility.
> Why not using a BDEPEND var in those dependent packages affected by the
> BINCOMPAT values of their dependencies?
>
> For instance, I would set the following:
>  - in net-dialup/ppp ebuild:     BINCOMPAT=${PV}
>  - in net-dialup/pptpd ebuild:   BDEPEND="net-dialup/ppp"

i think you're merging the two issues you brought up ... there is binary ABI 
issues (which should not require a new DEPEND variable as portage can extract 
that information out at runtime) and there is runtime plugin issues with 
packages using dlopen() (which portage cannot extract as the dependency 
cannot ever be extracted)

or did i not read your original e-mail incorrectly ?
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 1477 bytes --]

On Thu, Sep 21, 2006 at 05:38:08PM +0300, Alin Nastac wrote:
> Brian Harring wrote:
> > On Thu, Sep 21, 2006 at 01:38:59PM +0200, Luca Barbato wrote:
> >   
> > There is one flaw with this though; packages can provide both 
> > libraries _and_ binaries.  Our dependencies don't represent whether 
> > the dep is actual linkage, or just commandline consuming, so (using 
> > the openssl example) any package that invokes openssl via the 
> > commandline to do a few simple chksum ops gets rebuilt, despite the 
> > fact it wasn't affected by linkage change ups.
> >   
> I like BINCOMPAT proposal but it solves only half of the problem. You
> assume that all dependent packages cares about binary compatibility.
> Why not using a BDEPEND var in those dependent packages affected by the
> BINCOMPAT values of their dependencies?
> 
> For instance, I would set the following:
>  - in net-dialup/ppp ebuild:     BINCOMPAT=${PV}
>  - in net-dialup/pptpd ebuild:   BDEPEND="net-dialup/ppp"

BDEPEND was actually a seperate proposal/idea, intention there was to 
have that be the deps that *must* be CHOST (gcc would be an example); 
bits that are used to actually build the pkg, not data it consumes in 
building (headers would be data).

Meanwhile, for this I don't see the point in using a seperate metadata 
key.  Overload DEPEND and add a marker char that is used to indicate 
that a particular dependency is 'binding', ie, it is linkage.

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Donnie Berkholz]

Mike Frysinger wrote:
> On Thursday 21 September 2006 10:14, Donnie Berkholz wrote:
>> Not adding it into the ebuild means that it's impossible to show in
>> advance what packages will actually be installed, because you don't know
>> whether the sover will bump.
> 
> sometimes you dont know as the ABI bump may be arch or feature specific ... 
> pathological case i agree ;)
> 
> you would certainly know between the time you merge the new one and unmerge 
> the old one though ... but now we tread into the territory of "portage should 
> not unmerge ABI libraries until all consumers have been destroyed" which is 
> better than the current craptastic situation of "you gotta revdep-rebuild 
> after the fact with a broken system"

Yes, I agree with you. For example, take expat. The maintainers have 
refused to allow both versions to exist simultaneously on a system 
because it apparently causes more breakage than just breaking every app 
on your system by removing .so.0.

Thanks,
Donnie
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC about another *DEPEND variable

[Duncan Coutts]

On Thu, 2006-09-21 at 09:52 -0400, Mike Frysinger wrote:
> On Thursday 21 September 2006 07:59, Brian Harring wrote:
> > Why have the explicit var?  Because 0.9.7a through 0.9.7c may all be
> > compatible, but 0.9.7d isn't.  If you force an automatic algo that
> > tries to (effectively) guess, you get a lot of rebuilds through a,b,c,
> > end result being folks likely update less because it becomes a bigger
> > pain in the ass.
> 
> if it isnt compatible then it shouldnt have the same SONAME, simple as 
> that ... that is after all the point of encoding the ABI version number into 
> the SONAME
> 
> forcing devs to maintain a manual var which is basically duplicating the 
> SONAME smells like maintenance nightmare

There are various kinds of ABI. Certainly for C libs the SONAME is
probably the most common. If we go for some kind of ABI reverse deps
feature I would beg for something a little more general - though
certainly with SONAME as a common case.

Other languages have similar problems. For example Python has
incompatible ABIs for each major release 2.2, 2.3 etc. They have a
similar solution to the revdep-rebuild: python-updater.

As lead of the Haskell team my interest in this is that our primary
Haskell compiler GHC has ABI incompatibility between versions too. We've
made a ghc-updater similar in style to the python one but this is
clearly an unsatisfactory situation. It's more acute for us as even
minor revisions are ABI-incompatible.

So for it's something like:
# for C:
ABI=${SONAME}

# for python
ABI=${PY_PV}

# for haskell:
ABI=${GHC_PV}

paludis has something going in this direction but I don't think it'd
quite solve the python/ghc abi issue. It was aimed more at cases like
mips with it's multiple abis.

If we do go in this direction it'd be great to be able to slot on the
ABI and still have dependencies resolved correctly. For example imagine
having parallel python-2.3 and 2.4 installations with some libs
installed for both. Crucially, deps need to be resolved to the version
of a lib with the right ABI.

debian solves this problem in an ad-hoc way by tacking extra components
into the package name:

pyfoo-py22.deb which deps on pybar-py22.deb
pyfoo-py23.deb which deps on pybar-py23.deb

so all can be installed at once and deps are resolved within the right
ABI. Now this is obviously not in the Gentoo tradition. We much prefer
cleaner solutions like SLOTing. I would love to see this extended to
allow us to SLOT on the abi and then correctly resolve based on that
abi.

If we just SLOTed at the moment we'd get the sitation where
dev-python/bar built for py 2.2 would be considered ok to satisfy a
dependency of dev-python/foo that is being built for py 2.3.

We want some kind of extra component to be able to resolve on:

dev-python/foo-1.0.ebuild:

SLOT="${PV}-${PY_PV}"
ABI="${PY_PV}"
DEPEND="dev-python/bar @ ${PY_PV}"

Actually for Haskell the situation is even more fun; we have multiple
haskell implementations, so we would like to install a lib and SLOT upon
and correctly resolve deps for multiple haskell compilers.

Fun stuff. :-)

If portage people are interested in moving in this direction we have an
experimental emerge-compatible mini dep-resolver which might be useful
to prototype an extended ABI/SLOTing system.

Duncan Coutts

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC about another *DEPEND variable

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 734 bytes --]

On Thursday 21 September 2006 10:54, Donnie Berkholz wrote:
> Yes, I agree with you. For example, take expat. The maintainers have
> refused to allow both versions to exist simultaneously on a system
> because it apparently causes more breakage than just breaking every app
> on your system by removing .so.0.

that is the exact case portage should be handling for you

it would go "oh hey, check out libexpat.so.0 ... some things seem to want 
it ... HEY USER, you need to rebuild: xxxxxxxx" ... once all the packages 
still consuming libexpat.so.0 are rebuilt, portage could silently trim it 
from the system

complicated ?  not really, scanelf can produce all this information in an 
easily digestable format
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 1694 bytes --]

On Thu, Sep 21, 2006 at 10:43:11AM -0400, Mike Frysinger wrote:
> On Thursday 21 September 2006 10:04, Brian Harring wrote:
> > I agree; while I'm labeling it ABI, includes both bad soname handling
> > and seperate sonames.
> 
> those people should be smacked (for the interest of disclosure, i have 
> violated the "bad soname" rule for the sake of following upstream)
> 
> > Feel free to point out a 4th option if I'm missing it, but for the
> > request, that's what exists afaict; meanwhile, stating that pkgs are
> > being stupid, while true, doesn't actually solve the issue :)
> 
> 4) portage maintains a list of ABI SONAMEs in use and does not unmerge the 
> library until all consumers are gone
> 
> i'm referring to the specific file of course, not anything else in the 
> package ... so integrating the hack eutils.eclass:preserve_old_lib() into 
> portage so it isnt a hack (not a knock against the current implementation 
> here; it's always going to be a hack until portage manages proper unmerging 
> of the ABI library)

The reason folks aren't talking about using NEEDED is that NEEDED data 
is generated _after_ building; getting the info into the resolver 
up front allows for a helluva lot more options, and makes stuff like 
ensuring you have all sources required downloaded *prior* actually 
simple to do, rather then inserting recalculating hacks into the 
resolver.

Clarifying the 'recalculating', what you're suggesting is effectivelly 
unbounded resolution, re-calculating at each step.  That route is 
*very* nasty since you can't gurantee up front the resolution will 
work, let alone ensuring the bugger doesn't go cyclic.
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Simon Stelling]

Duncan Coutts wrote:
> So for it's something like:
> # for C:
> ABI=${SONAME}
> 
> # for python
> ABI=${PY_PV}
> 
> # for haskell:
> ABI=${GHC_PV}
> 
> paludis has something going in this direction but I don't think it'd
> quite solve the python/ghc abi issue. It was aimed more at cases like
> mips with it's multiple abis.

It's all about multilib and has (except for the unfortunate name)
nothing to do with this issue.

-- 
Kind Regards,

Simon Stelling
Gentoo/AMD64 developer
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC about another *DEPEND variable

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 521 bytes --]

On Thursday 21 September 2006 10:56, Duncan Coutts wrote:
> If we do go in this direction it'd be great to be able to slot on the
> ABI and still have dependencies resolved correctly. For example imagine
> having parallel python-2.3 and 2.4 installations with some libs
> installed for both. Crucially, deps need to be resolved to the version
> of a lib with the right ABI.

ugh, no ... we are not a binary distribution so we should not have to worry 
about ABI baggage

we SLOT based upon API, not ABI
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Duncan Coutts]

On Thu, 2006-09-21 at 11:11 -0400, Mike Frysinger wrote:
> On Thursday 21 September 2006 10:56, Duncan Coutts wrote:
> > If we do go in this direction it'd be great to be able to slot on the
> > ABI and still have dependencies resolved correctly. For example imagine
> > having parallel python-2.3 and 2.4 installations with some libs
> > installed for both. Crucially, deps need to be resolved to the version
> > of a lib with the right ABI.
> 
> ugh, no ... we are not a binary distribution so we should not have to worry 
> about ABI baggage

So we can't ever install two versions of python or ghc at once? That
seems a shame.

> we SLOT based upon API, not ABI

Here's another example; I'm not sure if it passes the ABI/API test:

We would like to support 3 Haskell implementations:
  * GHC which compiles to native code (ELF binaries & static .a libs)
  * Hugs which is an interpreter so installation is .hs source files
  * YHC which compiles to portable bytecode

A single Haskell library is likely to work with all three
implementations. So that's API.

Once installed however each implementation is very different. So that's
incompatible ABI.

This could be 'solved' by having dev-haskell/foo-ghc,
dev-haskell/foo-hugs, dev-haskell/foo-yhc, but that's obviously not the
Gentoo way (though it's pretty much what debian does).

These multiple impls is pretty similar to multiple versions of the same
compiler.

So my point is, I don't think it can be simply dismissed as ABI nonsense
that we don't have to deal with. Being able to SLOT on the compiler
flavour (and possibly version) would allow us to do useful things that
we cannot currently do.

Duncan

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC about another *DEPEND variable

[Alin Nastac]

[-- Attachment #1: Type: text/plain, Size: 1077 bytes --]

Brian Harring wrote:
> BDEPEND was actually a seperate proposal/idea, intention there was to 
> have that be the deps that *must* be CHOST (gcc would be an example); 
> bits that are used to actually build the pkg, not data it consumes in 
> building (headers would be data).
>   
Well, until now I didn't thought at the build compatibility.
My concern was only the runtime compatibility.
> Meanwhile, for this I don't see the point in using a seperate metadata 
> key.  Overload DEPEND and add a marker char that is used to indicate 
> that a particular dependency is 'binding', ie, it is linkage.
>   
Lets suppose we use & as 'binding' dependency marker. What sense would
DEPEND="&net-dialup/ppp" have in a context of an ebuild. It certainly
don't specify the necessity of package rebuild whenever net-dialup/ppp
version is changed.
Unless you save the specific compatibility version of the net-dialup/ppp
used by net-dialup/pptpd for building the package, I don't see how can
it help me.
Judging after /var/db/pkg content, I have no such information.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Alin Nastac]

[-- Attachment #1: Type: text/plain, Size: 782 bytes --]

Mike Frysinger wrote:
> i think you're merging the two issues you brought up ... there is binary ABI 
> issues (which should not require a new DEPEND variable as portage can extract 
> that information out at runtime) and there is runtime plugin issues with 
> packages using dlopen() (which portage cannot extract as the dependency 
> cannot ever be extracted)
>   
Okay, maybe I hijacked BINCOMPAT purpose. As I said in a previous reply,
my original message was about runtime compatibility, not compilation
compatibility.
I want to be able to save in a package metadata that it was build using
some version (as in compatibility version, not necessarily PV) of
another package and I want emerge to automatically rebuild my package
whenever this version is changed.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Luca Barbato]

Duncan Coutts wrote:

> 
> So my point is, I don't think it can be simply dismissed as ABI nonsense
> that we don't have to deal with. Being able to SLOT on the compiler
> flavour (and possibly version) would allow us to do useful things that
> we cannot currently do.

what about making them build what you want depending on useflags?

lu

-- 

Luca Barbato

Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC about another *DEPEND variable

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 1403 bytes --]

On Thu, Sep 21, 2006 at 08:15:48PM +0300, Alin Nastac wrote:
> Brian Harring wrote:
> > BDEPEND was actually a seperate proposal/idea, intention there was to 
> > have that be the deps that *must* be CHOST (gcc would be an example); 
> > bits that are used to actually build the pkg, not data it consumes in 
> > building (headers would be data).
> >   
> Well, until now I didn't thought at the build compatibility.
> My concern was only the runtime compatibility.
> > Meanwhile, for this I don't see the point in using a seperate metadata 
> > key.  Overload DEPEND and add a marker char that is used to indicate 
> > that a particular dependency is 'binding', ie, it is linkage.
> >   
> Lets suppose we use & as 'binding' dependency marker. What sense would
> DEPEND="&net-dialup/ppp" have in a context of an ebuild. It certainly
> don't specify the necessity of package rebuild whenever net-dialup/ppp
> version is changed.
>
> Unless you save the specific compatibility version of the net-dialup/ppp
> used by net-dialup/pptpd for building the package, I don't see how can
> it help me.
> Judging after /var/db/pkg content, I have no such information.

Any such implementation would require storing some extra data in the 
vdb....

For this, would just walk the *DEPEND collecting 'binding' 
dependencies, and storing their BINCOMPAT in a simple mapping.

~harring


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Duncan Coutts]

On Thu, 2006-09-21 at 20:27 +0200, Luca Barbato wrote:
> Duncan Coutts wrote:
> 
> > 
> > So my point is, I don't think it can be simply dismissed as ABI nonsense
> > that we don't have to deal with. Being able to SLOT on the compiler
> > flavour (and possibly version) would allow us to do useful things that
> > we cannot currently do.
> 
> what about making them build what you want depending on useflags?

Aye, for the implementation flavours that's probably the way to go once
we have use-deps. We'll have to hold off on multiple versions of the
same compiler though.

It might get a bit hairy though :-)

DEPEND="ghc?  ( dev-haskell/foo @ ghc )
	hugs? ( dev-haskell/foo @ hugs )
	yhc? ( dev-haskell/foo @ yhc )
	jhc? ( dev-haskell/foo @ jhc )"

(I've not looked up what the use-dep syntax is, I'm just guessing)

Duncan

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC about another *DEPEND variable

[Luca Barbato]

Duncan Coutts wrote:
> On Thu, 2006-09-21 at 20:27 +0200, Luca Barbato wrote:
>> Duncan Coutts wrote:
>>
>>> So my point is, I don't think it can be simply dismissed as ABI nonsense
>>> that we don't have to deal with. Being able to SLOT on the compiler
>>> flavour (and possibly version) would allow us to do useful things that
>>> we cannot currently do.
>> what about making them build what you want depending on useflags?
> 
> Aye, for the implementation flavours that's probably the way to go once
> we have use-deps. We'll have to hold off on multiple versions of the
> same compiler though.
> 
> It might get a bit hairy though :-)
> 
> DEPEND="ghc?  ( dev-haskell/foo @ ghc )
> 	hugs? ( dev-haskell/foo @ hugs )
> 	yhc? ( dev-haskell/foo @ yhc )
> 	jhc? ( dev-haskell/foo @ jhc )"
> 
> (I've not looked up what the use-dep syntax is, I'm just guessing)

as now, you can handle that with an eclass that takes deps, checks those 
deps with certain use if the use has been enabled and dies if those 
criterions aren't meet (asking you to falling back to hc-updater to keep 
  those applications in shape)

pretty hacky and itchy but that's probably the best you can do w/out 
use-deps. (PS can you slot on use now?)

lu

-- 

Luca Barbato

Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC about another *DEPEND variable

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 464 bytes --]

On Thursday 21 September 2006 13:15, Alin Nastac wrote:
> Unless you save the specific compatibility version of the net-dialup/ppp
> used by net-dialup/pptpd for building the package, I don't see how can
> it help me.
> Judging after /var/db/pkg content, I have no such information.

it is all there right now actually :)

check out the NEEDED file ... combine that with CONTENTS of other packages, 
and you have all the binding info you want ...
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 779 bytes --]

On Thursday 21 September 2006 11:41, Duncan Coutts wrote:
> On Thu, 2006-09-21 at 11:11 -0400, Mike Frysinger wrote:
> > On Thursday 21 September 2006 10:56, Duncan Coutts wrote:
> > > If we do go in this direction it'd be great to be able to slot on the
> > > ABI and still have dependencies resolved correctly. For example imagine
> > > having parallel python-2.3 and 2.4 installations with some libs
> > > installed for both. Crucially, deps need to be resolved to the version
> > > of a lib with the right ABI.
> >
> > ugh, no ... we are not a binary distribution so we should not have to
> > worry about ABI baggage
>
> So we can't ever install two versions of python or ghc at once? That
> seems a shame.

that's an issue for the python or ghc maintainers to address
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1551 bytes --]

On Thursday 21 September 2006 11:08, Brian Harring wrote:
> On Thu, Sep 21, 2006 at 10:43:11AM -0400, Mike Frysinger wrote:
> > i'm referring to the specific file of course, not anything else in the
> > package ... so integrating the hack eutils.eclass:preserve_old_lib() into
> > portage so it isnt a hack (not a knock against the current implementation
> > here; it's always going to be a hack until portage manages proper
> > unmerging of the ABI library)
>
> The reason folks aren't talking about using NEEDED is that NEEDED data
> is generated _after_ building;

as well it should be ... trying to enumerate the linking ABI possibilities 
before hand is not doable and not worth wasting the time of maintainers when 
it can be automated

> getting the info into the resolver 
> up front allows for a helluva lot more options, and makes stuff like
> ensuring you have all sources required downloaded *prior* actually
> simple to do, rather then inserting recalculating hacks into the
> resolver.

rather than integrating it all into the resolver in a one-pass system, a two 
pass system would work:
 - build all the packages requested
 - after each package, if an ABI library is being removed, check to see if it 
is needed by any other package.  if it is needed, record it and preserve it 
and move on
 - once all the packages requested have been merged, you start the second 
phase and calculate everything that needs to be rebuilt.  as ABI libs are no 
longer needed on a system, portage can scrub them out
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Duncan Coutts]

On Sat, 2006-09-23 at 06:13 -0400, Mike Frysinger wrote: 
> On Thursday 21 September 2006 11:41, Duncan Coutts wrote:
> > On Thu, 2006-09-21 at 11:11 -0400, Mike Frysinger wrote:
> > > On Thursday 21 September 2006 10:56, Duncan Coutts wrote:
> > > > If we do go in this direction it'd be great to be able to slot on the
> > > > ABI and still have dependencies resolved correctly. For example imagine
> > > > having parallel python-2.3 and 2.4 installations with some libs
> > > > installed for both. Crucially, deps need to be resolved to the version
> > > > of a lib with the right ABI.
> > >
> > > ugh, no ... we are not a binary distribution so we should not have to
> > > worry about ABI baggage
> >
> > So we can't ever install two versions of python or ghc at once? That
> > seems a shame.
> 
> that's an issue for the python or ghc maintainers to address

Yes and as ghc maintainer for Gentoo I'd love to be able to do it. It is
easy to install more than one version of ghc at once as it uses
versioned directories etc. The problem is that we cannot correctly
resolve dependencies with the current versions of portage. Hence we
cannot effectively do it.

I was worried from your ABI/API comments that you meant that we should
never be allowed to do it.

As I said before, if we were to SLOT ghc and the various dev-haskell/*
libs on the ghc version then we would run into this problem:

assume dev-haskell/foo needs dev-haskell/bar

supposing dev-haskell/bar was installed and sloted for ghc-6.2
now I install ghc-6.4 too
now I emerge dev-haskell/foo.

portage will use the existing installation of dev-haskell/bar even
though it was installed for another version of ghc. Therefore the build
will break (think of haskell libs of having a SONAME that is the version
of ghc they were built with). What needs to happen is to install
dev-haskell/bar sloted for ghc-6.4 and use that.

Currently there is no way to explain these dependencies to portage.

I was hoping that with this talk of tracking reverse deps and such like
that we could move in a direction where this kind of dependency could be
supported. Even if we can't slot on the ghc version, automatically
rebuilding for the latest ghc version would be a great improvement -
like the suggestion to rebuild dependent libs when the SONAME changes.

Duncan

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC about another *DEPEND variable

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 266 bytes --]

On Saturday 23 September 2006 06:35, Duncan Coutts wrote:
> I was worried from your ABI/API comments that you meant that we should
> never be allowed to do it.

i was commenting on the more general case; SLOTing something that wasnt meant 
to be SLOTed
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 3152 bytes --]

On Sat, Sep 23, 2006 at 06:20:44AM -0400, Mike Frysinger wrote:
> On Thursday 21 September 2006 11:08, Brian Harring wrote:
> > On Thu, Sep 21, 2006 at 10:43:11AM -0400, Mike Frysinger wrote:
> > > i'm referring to the specific file of course, not anything else in the
> > > package ... so integrating the hack eutils.eclass:preserve_old_lib() into
> > > portage so it isnt a hack (not a knock against the current implementation
> > > here; it's always going to be a hack until portage manages proper
> > > unmerging of the ABI library)
> >
> > The reason folks aren't talking about using NEEDED is that NEEDED data
> > is generated _after_ building;
> 
> as well it should be ... trying to enumerate the linking ABI possibilities 
> before hand is not doable and not worth wasting the time of maintainers when 
> it can be automated
> 
> > getting the info into the resolver 
> > up front allows for a helluva lot more options, and makes stuff like
> > ensuring you have all sources required downloaded *prior* actually
> > simple to do, rather then inserting recalculating hacks into the
> > resolver.
> 
> rather than integrating it all into the resolver in a one-pass system, a two 
> pass system would work:
>  - build all the packages requested
>  - after each package, if an ABI library is being removed, check to see if it 
> is needed by any other package.  if it is needed, record it and preserve it 
> and move on

You're assuming that after the merge of the pkg that breaks 
compatibility, building is actually _still_ possible for the depends.

We don't classify our deps as actual build depends vs link depends; as 
such trying to (essentially) "patch things up after" allow for the 
scenario where merging breaks the toolchain, thus building isn't 
possible.

Because we don't classify the type of build time dep, that means each 
DEPEND match must have it's run time depends (RDEPEND) satisfied prior 
to being usable as a DEPEND; that right there punches a whole in the 
"delay till the end" approach, and is a good example of what I mean 
when I say "this is unbounded resolution"; the only way to solve it is 
to redo the resolution between finished building and merging the pkg 
to determine if it will break any required DEPENDS for later steps.

>  - once all the packages requested have been merged, you start the second 
> phase and calculate everything that needs to be rebuilt.  as ABI libs are no 
> longer needed on a system, portage can scrub them out

"as ABI libs are no longer needed on a system", phrasing of that 
implies you're suggesting that portage should leave the older package 
in place till it's updated all revdeps, then wipe it.

Which is fairly nasty, especially if you factor in the user potential 
of ctrl+c'ing it.  Finally, if I'm interpretting your statement there 
correctly, still isn't guranteed to work- just having the lib around 
doesn't mean that the older package is left in a working state with 
the new partially merged over it, only way that would work is if the 
two were slotted (indicating they could coexist on disk).
~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1150 bytes --]

On Saturday 23 September 2006 09:14, Brian Harring wrote:
> You're assuming that after the merge of the pkg that breaks
> compatibility, building is actually _still_ possible for the depends.

of course i am; i just said that portage would make sure to not unmerge any 
ABI lib still in use

> We don't classify our deps as actual build depends vs link depends; as
> such trying to (essentially) "patch things up after" allow for the
> scenario where merging breaks the toolchain, thus building isn't
> possible.

huh ?  RDEPEND is linktime ... see my statement above

> >  - once all the packages requested have been merged, you start the second
> > phase and calculate everything that needs to be rebuilt.  as ABI libs are
> > no longer needed on a system, portage can scrub them out
>
> "as ABI libs are no longer needed on a system", phrasing of that
> implies you're suggesting that portage should leave the older package
> in place till it's updated all revdeps, then wipe it.

no i am not; read my previous e-mails where i said it would leave behind the 1 
ABI lib required ... aka whatever is encoded in SONAME
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 860 bytes --]

On Saturday 23 September 2006 09:50, Mike Frysinger wrote:
> On Saturday 23 September 2006 09:14, Brian Harring wrote:
> > We don't classify our deps as actual build depends vs link depends; as
> > such trying to (essentially) "patch things up after" allow for the
> > scenario where merging breaks the toolchain, thus building isn't
> > possible.
>
> huh ?  RDEPEND is linktime ... see my statement above

lemme rephrase ... you keep the ABI SONAME lib around until all packages are 
done with it; this does not affect any other package (including upgrades to 
the same package) as the ABI filename is unique in the filesystem

it is not used at linktime because it does not provide a linkable file ... it 
merely provides the library used at runtime

so you can upgrade on the fly as the old ABI lib does not conflict with 
anything
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Alin Nastac]

[-- Attachment #1: Type: text/plain, Size: 984 bytes --]

Mike Frysinger wrote:
> On Thursday 21 September 2006 13:15, Alin Nastac wrote:
>   
>> Unless you save the specific compatibility version of the net-dialup/ppp
>> used by net-dialup/pptpd for building the package, I don't see how can
>> it help me.
>> Judging after /var/db/pkg content, I have no such information.
>>     
>
> it is all there right now actually :)
>
> check out the NEEDED file ... combine that with CONTENTS of other packages, 
> and you have all the binding info you want ...
I see only libraries in NEEDED and it is probably generated
automatically. There is no way for the automatic tools to discover the
dependency between pptpd and ppp version.

Besides, even if I would have somehow /usr/lib/ppp/2.4.3 in NEEDED file
of the pptpd, the amount of computation needed to discover which package
offers such thing would be prohibitive. The reciprocal operation (find
which packages use the old path before upgrade) would also be prohibitive.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 3868 bytes --]

On Sat, Sep 23, 2006 at 09:50:12AM -0400, Mike Frysinger wrote:
> On Saturday 23 September 2006 09:14, Brian Harring wrote:
> > You're assuming that after the merge of the pkg that breaks
> > compatibility, building is actually _still_ possible for the depends.
> 
> of course i am; i just said that portage would make sure to not unmerge any 
> ABI lib still in use

dlopen?

How does this fix openssl horkage? (bad soname handling)

Also... what do we do for python/perl (*-updater scripts in general) 
where a change in a pkg state means we have to rebuild the revdeps?

What you're suggesting works for strictly linkage; still think this 
shouuld work for the general problem rather then just one subset.

> > We don't classify our deps as actual build depends vs link depends; as
> > such trying to (essentially) "patch things up after" allow for the
> > scenario where merging breaks the toolchain, thus building isn't
> > possible.
> 
> huh ?  RDEPEND is linktime ... see my statement above

RDEPEND is execution requirements; to use the binary, this is what 
needs to be in the graph.

Clarifying my statement; we don't break our DEPEND down into "this is 
what is executed in building the package" (toolchain), vs "this is the 
crap the binaries we build against need avail to be linked against", 
literally what winds up as -l args.

If punting the old lib (as I assumed), means we would potentially be 
making certain DEPEND atoms unusable if they're required in an 
execution context (rather then just winding up as a -l arg).

So... ignore that bit since you're talking about lingering files.


> > >  - once all the packages requested have been merged, you start the second
> > > phase and calculate everything that needs to be rebuilt.  as ABI libs are
> > > no longer needed on a system, portage can scrub them out
> >
> > "as ABI libs are no longer needed on a system", phrasing of that
> > implies you're suggesting that portage should leave the older package
> > in place till it's updated all revdeps, then wipe it.
> 
> no i am not; read my previous e-mails where i said it would leave behind the 1 
> ABI lib required ... aka whatever is encoded in SONAME

Yeah, missed the "presvered" (woot for 5am wakeup).

In that case, wouldn't mind a response to the "what about ctrl+c 
during the run?"  The potential for orphaning there sucks; recording 
the old library in the new version sucks also since it complicates the 
merge process, that lib *must* be removed else it's a potential 
collision-protect minefield.

Finally, even if the lib is temporarily left behind, this solution 
doesn't gurantee the library actually would *work* still- it only can 
work if the lib is standalone from the rest of the pkg and doesn't 
rely on any external data from the pkg.

Example would be pkg foobar that internally has libconvience, used by 
it's libs but not externally linked, contains (oddly enough) 
convience bits shared across foobars libraries.  

libconvience is *not* to be externally linked against, consumers must 
access the other libs (say libfoo); any soname bumps to libfoo, the 
old version gets broke in the process despite due to it linking 
internally against an unversioned so.

Granted, semi retarded, but gnomes libegg comes to mind as a potential 
case of this.

Basically trying to point out that what you're proposing only works in 
a subset of the cases revdep must deal with, and that revdep itself 
doesn't deal with *all* situations as is; hence BINCOMPAT as a way to 
try and shift it under maintainers control.

Maintainence of it *should* be pretty simple also; for sane upstream 
soname handling, you just bump it with the majors; for the rest, its a 
knob that can be fiddled to at least give up front warning of the 
issue.

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 865 bytes --]

On Saturday 23 September 2006 10:24, Alin Nastac wrote:
> I see only libraries in NEEDED and it is probably generated
> automatically. There is no way for the automatic tools to discover the
> dependency between pptpd and ppp version.

that gets back to ABI versus dynamic plugins ... we already know we'll need a 
new DEPEND to track dlopen-ed plugins

> Besides, even if I would have somehow /usr/lib/ppp/2.4.3 in NEEDED file
> of the pptpd, the amount of computation needed to discover which package
> offers such thing would be prohibitive. The reciprocal operation (find
> which packages use the old path before upgrade) would also be prohibitive.

no it wouldnt ... when you merge a package, you record all the SONAME's it 
provides:
scanelf -qRS "${D}" > SONAME

in fact, running `scanelf -qlpRS` doesnt take that long on my machine
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 1420 bytes --]

On Sat, Sep 23, 2006 at 10:34:03AM -0400, Mike Frysinger wrote:
> On Saturday 23 September 2006 10:24, Alin Nastac wrote:
> > I see only libraries in NEEDED and it is probably generated
> > automatically. There is no way for the automatic tools to discover the
> > dependency between pptpd and ppp version.
> 
> that gets back to ABI versus dynamic plugins ... we already know we'll need a 
> new DEPEND to track dlopen-ed plugins
> 
> > Besides, even if I would have somehow /usr/lib/ppp/2.4.3 in NEEDED file
> > of the pptpd, the amount of computation needed to discover which package
> > offers such thing would be prohibitive. The reciprocal operation (find
> > which packages use the old path before upgrade) would also be prohibitive.
> 
> no it wouldnt ... when you merge a package, you record all the SONAME's it 
> provides:
> scanelf -qRS "${D}" > SONAME
> 
> in fact, running `scanelf -qlpRS` doesnt take that long on my machine

Flush the cache... Makes a world of difference.  Additionally, he is 
talking about what is *done* with that data after the fact, iow other 
words walking the entire vdb to find all affected pkgs.  Can pretty 
much gurantee after a build that data isn't likely to be available 
(part of the reason portageq calls during building are slow).

Could collapse that into a simple mapping, but that introduces 
backwards compatibility issues...

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 458 bytes --]

On Saturday 23 September 2006 10:53, Brian Harring wrote:
> Flush the cache... Makes a world of difference.

i was running from a cold cache, thanks

> Additionally, he is 
> talking about what is *done* with that data after the fact, iow other
> words walking the entire vdb to find all affected pkgs.

locating dependencies between NEEDED and SONAMES files is not that time 
consuming especially when you consider how slow portage is now
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1510 bytes --]

On Thu, 21 Sep 2006 11:01:40 -0400 Mike Frysinger <vapier@gentoo.org>
wrote:
| On Thursday 21 September 2006 10:54, Donnie Berkholz wrote:
| > Yes, I agree with you. For example, take expat. The maintainers have
| > refused to allow both versions to exist simultaneously on a system
| > because it apparently causes more breakage than just breaking every
| > app on your system by removing .so.0.
| 
| that is the exact case portage should be handling for you
| 
| it would go "oh hey, check out libexpat.so.0 ... some things seem to
| want it ... HEY USER, you need to rebuild: xxxxxxxx" ... once all the
| packages still consuming libexpat.so.0 are rebuilt, portage could
| silently trim it from the system
| 
| complicated ?  not really, scanelf can produce all this information
| in an easily digestable format

How would it know what other files are required? For example, if
libexpat.so.0 were to rely upon /usr/share/expat-0/config , how would
the package manager know not to clobber that file? Or are you
suggesting leaving (or reparenting, if you prefer) all a package's
files, not just the .so files?

Or a related question: what proportion of breakages will be fixed
merely by keeping .so files and nothing else around? Will implementing
this prevent enough breakages to make it worthwhile?

-- 
Ciaran McCreesh
Mail                : ciaranm at ciaranm.org
Web                 : http://ciaranm.org/
as-needed is broken : http://ciaranm.org/show_post.pl?post_id=13


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1083 bytes --]

On Saturday 23 September 2006 22:36, Ciaran McCreesh wrote:
> How would it know what other files are required? For example, if
> libexpat.so.0 were to rely upon /usr/share/expat-0/config , how would
> the package manager know not to clobber that file? Or are you
> suggesting leaving (or reparenting, if you prefer) all a package's
> files, not just the .so files?

i'm talking about just libexpat.so.0 ... keep the file around long enough to 
update the system

> Or a related question: what proportion of breakages will be fixed
> merely by keeping .so files and nothing else around? Will implementing
> this prevent enough breakages to make it worthwhile?

of course it will when you look at core things like openssl, acl, openldap, 
etc...  if you remove anyone one of these packages completely, you risk 
breaking everything (coreutils and many system packages use libacl) or all 
your network capabilities (sshd/ssh no longer usuable; now you require 
on-site service to fix ... hopefully you have packages cached cause wget no 
longer works either)
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 4498 bytes --]

On Saturday 23 September 2006 10:30, Brian Harring wrote:
> dlopen?

we already said that this will need a new depend variable

> How does this fix openssl horkage? (bad soname handling)

it wont, but such things are broken regardless outside of Gentoo ... and 
trying to accommodate something that happens every three blue moons at the 
cost of developer time is not worth it

> Also... what do we do for python/perl (*-updater scripts in general)
> where a change in a pkg state means we have to rebuild the revdeps?

my solution does not address this, but what you're proposing would over 
address this ... how do you know when a package needs to be rebuilt (a perl 
module) or rebuilding is a waste of time (a package installs perl scripts 
that execute `perl` and nothing else)

> What you're suggesting works for strictly linkage; still think this
> shouuld work for the general problem rather then just one subset.

yes, i am addressing what i see to be the most critical issue and the easiest 
to break a user's system

> Clarifying my statement; we don't break our DEPEND down into "this is
> what is executed in building the package" (toolchain),

DEPEND

> vs "this is the 
> crap the binaries we build against need avail to be linked against",
> literally what winds up as -l args.

RDEPEND

> If punting the old lib (as I assumed), means we would potentially be
> making certain DEPEND atoms unusable if they're required in an
> execution context (rather then just winding up as a -l arg).

no ... lemme give a perfect example.

user has openssl-0.9.7j installed and they upgrade to openssl-0.9.8c ... the 
old version provided SONAME files libcrypto.so.0.9.7 and libssl.so.0.9.7 
while the new one provides SONAME files libcrypto.so.0.9.8 and 
libssl.so.0.9.8

everything from openssl-0.9.7j is unmerged except for the two files 
libcrypto.so.0.9.7 and libssl.so.0.9.7.  openssl-0.9.8c, being a different 
ABI, does not provide these files thus there is no clash.

portage keeps track of libcrypto.so.0.9.7 and libssl.so.0.9.7 and once no more 
packages have NEEDED entries for these, will silently clean them out

openssl is odd in that it encodes .x.y.z version into the ABI ... if we use 
the more common example with say expat, older versions install 
libexpat.so.0.1, libexpat.so.0.2, libexpat.so.0.3, etc... but the ABI SONAME 
is still just libexpat.so.0.  when the next major version of expat comes out, 
the SONAME is bumped to libexpat.so.1 and portage needs to keep around the 
last libexpat.so.0

> In that case, wouldn't mind a response to the "what about ctrl+c
> during the run?"  The potential for orphaning there sucks; recording
> the old library in the new version sucks also since it complicates the
> merge process, that lib *must* be removed else it's a potential
> collision-protect minefield.

- portage merges new version 2 to $ROOT
- system now has version 1 and version 2 in $ROOT
- portage starts to clean out version 1 from $ROOT
- do not fully trim version 1's vdb until version 2 has been updated with the 
new information

so ctrl+c at any point and so what ?  you dont remove old files until new 
files are fully placed and you can resume at any point

> Finally, even if the lib is temporarily left behind, this solution
> doesn't gurantee the library actually would *work* still- it only can
> work if the lib is standalone from the rest of the pkg and doesn't
> rely on any external data from the pkg.

we're talking about preserving the system long enough to rebuild things; we're 
not talking about keeping the system forever in a sane state.

i would guess that this corner case is not the norm and thus we can ignore it 
as the situation is still a lot better than what we have now:
 $ foo
 foo: error while loading shared libraries: libbar.so.1: cannot open shared 
object file: No such file or directory

awesome

> Basically trying to point out that what you're proposing only works in
> a subset of the cases revdep must deal with, and that revdep itself
> doesn't deal with *all* situations as is; hence BINCOMPAT as a way to
> try and shift it under maintainers control.

revdep-rebuild doesnt take into consideration any of the issues you raised

and forcing maintainers to always track BINCOMPAT is unwieldy ... a single 
package provides multiple SONAMEs ?  bitrot ?  SONAME is dynamic based upon 
architecture or USE flags ?
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 8846 bytes --]

On Sat, Sep 23, 2006 at 11:31:12PM -0400, Mike Frysinger wrote:
> On Saturday 23 September 2006 10:30, Brian Harring wrote:
> > dlopen?
> 
> we already said that this will need a new depend variable
> 
> > How does this fix openssl horkage? (bad soname handling)
> 
> it wont, but such things are broken regardless outside of Gentoo ... and 
> trying to accommodate something that happens every three blue moons at the 
> cost of developer time is not worth it

Bad soname handling is just *part* of what BINCOMPAT could do; it's 
not the sole reason for it's existance, as such it's not quite right 
dismissing it just because it addresses a rarity the NEEDED approach 
doesn't. :)

> > Also... what do we do for python/perl (*-updater scripts in general)
> > where a change in a pkg state means we have to rebuild the revdeps?
> 
> my solution does not address this, but what you're proposing would over 
> address this ... how do you know when a package needs to be rebuilt (a perl 
> module) or rebuilding is a waste of time (a package installs perl scripts 
> that execute `perl` and nothing else)

Original email stated that 'binding deps' would be required for that, 
marking deps in some way such that it indicates they're sensitive to 
changes in BINCOMPAT of the match.

*IF* we actually had that in place it would enable detecting and 
rebuilding c++ code whenever gcc pulls its next c++ abi change with 
appropriate deps in place (iow, kill off the implicit system deps).

Back to your example, if it's just a caller of it, it's not binding; 
now if it were a perl module that sticks its modules into the perl 
installation, yes, binding (it needs to rebuild to merge to the new 
location).


> > What you're suggesting works for strictly linkage; still think this
> > shouuld work for the general problem rather then just one subset.
> 
> yes, i am addressing what i see to be the most critical issue and the easiest 
> to break a user's system
> 
> > Clarifying my statement; we don't break our DEPEND down into "this is
> > what is executed in building the package" (toolchain),
> 
> DEPEND
> 
> > vs "this is the 
> > crap the binaries we build against need avail to be linked against",
> > literally what winds up as -l args.
> 
> RDEPEND

If that were the case, why do we have libraries listed in DEPEND then?

DEPEND is (and always has been) "this is the crap I need merged to be 
able to build an install image of myself", RDEPEND is (and always has 
been) "this is the crap I need on the fs to actually run my 
binaries/libs" and PDEPEND is around to cover up portages resolver, 
but moreso the trees mostly whacked deps.

This is why eradicator, solar, and you were discussing splitting link 
depends out of DEPEND for saner CHOST/CTARGET support around a year 
back also.

Semantics at this point, but RDEPEND does *not* need to be merged to 
build a pkg, only DEPEND; it's never been any other way.


> > If punting the old lib (as I assumed), means we would potentially be
> > making certain DEPEND atoms unusable if they're required in an
> > execution context (rather then just winding up as a -l arg).
> 
> no ... lemme give a perfect example.
> 
> user has openssl-0.9.7j installed and they upgrade to openssl-0.9.8c ... the 
> old version provided SONAME files libcrypto.so.0.9.7 and libssl.so.0.9.7 
> while the new one provides SONAME files libcrypto.so.0.9.8 and 
> libssl.so.0.9.8
> 
> everything from openssl-0.9.7j is unmerged except for the two files 
> libcrypto.so.0.9.7 and libssl.so.0.9.7.  openssl-0.9.8c, being a different 
> ABI, does not provide these files thus there is no clash.
> 
> portage keeps track of libcrypto.so.0.9.7 and libssl.so.0.9.7 and once no more 
> packages have NEEDED entries for these, will silently clean them out
> 
> openssl is odd in that it encodes .x.y.z version into the ABI ... if we use 
> the more common example with say expat, older versions install 
> libexpat.so.0.1, libexpat.so.0.2, libexpat.so.0.3, etc... but the ABI SONAME 
> is still just libexpat.so.0.  when the next major version of expat comes out, 
> the SONAME is bumped to libexpat.so.1 and portage needs to keep around the 
> last libexpat.so.0
> 
> > In that case, wouldn't mind a response to the "what about ctrl+c
> > during the run?"  The potential for orphaning there sucks; recording
> > the old library in the new version sucks also since it complicates the
> > merge process, that lib *must* be removed else it's a potential
> > collision-protect minefield.
> 
> - portage merges new version 2 to $ROOT
> - system now has version 1 and version 2 in $ROOT
> - portage starts to clean out version 1 from $ROOT
> - do not fully trim version 1's vdb until version 2 has been updated with the 
> new information
> 
> so ctrl+c at any point and so what ?  you dont remove old files until new 
> files are fully placed and you can resume at any point

Bleh, this is getting back to exactly my point that it's unbounded 
resolution.  To support this, every step of execution would require 
scanning for dangling nodes to punt; aside from invalidating -p's 
output it *still* is a collision-protect hit.

It also involves changing vdb nodes from "installed and usable" to 
"installed/usable" or "lingering", which makes things messier for
1) resolver
2) for --info
3) for has_version
4) for built_with_use
5) glsa scanning

There are more, but the point is that there isn't any concept of "half 
merged" packages for portage; introducing it gums things up pretty 
heavily, beyond the support NEEDED requires to actually mangle the 
half merged nodes.


> > Finally, even if the lib is temporarily left behind, this solution
> > doesn't gurantee the library actually would *work* still- it only can
> > work if the lib is standalone from the rest of the pkg and doesn't
> > rely on any external data from the pkg.
> 
> we're talking about preserving the system long enough to rebuild things; we're 
> not talking about keeping the system forever in a sane state.
> 
> i would guess that this corner case is not the norm and thus we can ignore it 
> as the situation is still a lot better than what we have now:
>  $ foo
>  foo: error while loading shared libraries: libbar.so.1: cannot open shared 
> object file: No such file or directory
> 
> awesome

So... NEEDED doesn't support bad soname handling, doesn't have a graph 
based gurantee that the remaining bits actually would work (bits 
above), and requires making merging *massively* slower (a NEEDED scan 
per vdb change effectively), finally emerge -p results no longer being 
accurate to what will be undertaken.

Ways to speed it up, but frankly NEEDED just plain sucks.  Its the 
route towards openembeddeds resolution, rather then up front 
calculation portage/rpm resolvers/apt all do.


> > Basically trying to point out that what you're proposing only works in
> > a subset of the cases revdep must deal with, and that revdep itself
> > doesn't deal with *all* situations as is; hence BINCOMPAT as a way to
> > try and shift it under maintainers control.
> 
> revdep-rebuild doesnt take into consideration any of the issues you raised
> 
> and forcing maintainers to always track BINCOMPAT is unwieldy ... a single 
> package provides multiple SONAMEs ?  bitrot ?  SONAME is dynamic based upon 
> architecture or USE flags ?

Tracking BINCOMPAT should *not* be that hard.  You hit a major 
version?  Bump the bincompat.  Not that fricking hard really; if 
dealing in a package that is pissy about it's so handling, well, you 
ought to know your inducing wide spread breakage for a rename already- 
so again, not that hard.

The only case that is slightly funky is if the so doesn't change name, 
but changes abi- BINCOMPAT at least enables notifying the manager that 
a rebuild is needed, NEEDED based approach is incapable of it.

Hell, automate a tool to determine if it's a BINCOMPAT bump via NEEDED 
data (folks should be compiling the pkg anyways), point is it's mainly 
common sense for maintainenance of it.


The NEEDED approach doesn't really fix much frankly; from a complexity 
standpoint, leaving it in revdep-rebuild is *far* better then trying 
to integrate the crap you're requesting, both from an implementation 
and from a long standing maintainence view (unbounded resolution is 
*really* not the direction you want to go).

The solution (imo) should be one that enables fixing the *class* of 
problem without making matters worse.

Yes, if the solution can be automated without flinging poo into the 
code, I'm for it; that said I know what the implementation is going to 
have to look like, and it's *nasty*.

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 2991 bytes --]

On Monday 25 September 2006 14:16, Brian Harring wrote:
> Bad soname handling is just *part* of what BINCOMPAT could do; it's 
> not the sole reason for it's existance, as such it's not quite right
> dismissing it just because it addresses a rarity the NEEDED approach
> doesn't. :)

i dismiss it as being a real advantage and/or argument for taking BINCOMPAT 
over an automated NEEDED scan

as i said, if you have changed ABI without an ABI bump, then the upstream 
package maintainer is screwing everyone who uses the package, not just 
Gentoo ... so perhaps we should be talking to them to get the situation 
resolved for all consumers, not just Gentoo

> *IF* we actually had that in place it would enable detecting and
> rebuilding c++ code whenever gcc pulls its next c++ abi change with
> appropriate deps in place (iow, kill off the implicit system deps).

scanelf already does this ... the only time you need to rebuild is when the 
ABI breaks ... aka libstdc++.so.5 -> libstdc++.so.6

> If that were the case, why do we have libraries listed in DEPEND then?

because i have a short memory

> Bleh, this is getting back to exactly my point that it's unbounded
> resolution.  To support this, every step of execution would require
> scanning for dangling nodes to punt; aside from invalidating -p's
> output it *still* is a collision-protect hit.

when the package upgrade detects an ABI change you recalculate the packages 
that need to be rebuilt ...

dangling nodes get recorded in the new package and considering these old files 
are not detrimental to the health of the system, you can do such a cleanup 
once at the end of the emerge

> It also involves changing vdb nodes from "installed and usable" to
> "installed/usable" or "lingering"

no ... the old versions get merged into the new one as their existence is 
purely hidden

> Tracking BINCOMPAT should *not* be that hard.

it's one more thing to keep track of and considering all of the possibilities 
i outlined, a single maintainer can easily lose his sanity ... or you force 
wasted rebuilds on people when it's only needed in some circumstances

> Hell, automate a tool to determine if it's a BINCOMPAT bump via NEEDED
> data (folks should be compiling the pkg anyways), point is it's mainly
> common sense for maintainenance of it.

and when are you going to run that tool ?  when you bump the package ?  so now 
the maintainer has to test it on all the KEYWORDed arches with all the USE 
combos, or de-KEYWORD it and make the arch maintainers test it ?  or dig 
through the source code line by line and hope to catch all such cases by 
hand/eye ?

> Yes, if the solution can be automated without flinging poo into the
> code, I'm for it; that said I know what the implementation is going to
> have to look like, and it's *nasty*.

i dont think either of our solutions are satisfactory for all presented cases; 
a hybrid model would be needed
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 7820 bytes --]

On Wed, Sep 27, 2006 at 02:24:41AM -0400, Mike Frysinger wrote:
> On Monday 25 September 2006 14:16, Brian Harring wrote:
> > Bad soname handling is just *part* of what BINCOMPAT could do; it's 
> > not the sole reason for it's existance, as such it's not quite right
> > dismissing it just because it addresses a rarity the NEEDED approach
> > doesn't. :)
> 
> i dismiss it as being a real advantage and/or argument for taking BINCOMPAT 
> over an automated NEEDED scan
> 
> as i said, if you have changed ABI without an ABI bump, then the upstream 
> package maintainer is screwing everyone who uses the package, not just 
> Gentoo ... so perhaps we should be talking to them to get the situation 
> resolved for all consumers, not just Gentoo

Happens however; afaik, we also weren't monkey patching ssl in the 
past to preserve the abi either so it still is valid (although rare if 
upstream is behaving) scenario.


> > *IF* we actually had that in place it would enable detecting and
> > rebuilding c++ code whenever gcc pulls its next c++ abi change with
> > appropriate deps in place (iow, kill off the implicit system deps).
> 
> scanelf already does this ... the only time you need to rebuild is when the 
> ABI breaks ... aka libstdc++.so.5 -> libstdc++.so.6
> 
> > If that were the case, why do we have libraries listed in DEPEND then?
> 
> because i have a short memory
> 
> > Bleh, this is getting back to exactly my point that it's unbounded
> > resolution.  To support this, every step of execution would require
> > scanning for dangling nodes to punt; aside from invalidating -p's
> > output it *still* is a collision-protect hit.
> 
> when the package upgrade detects an ABI change you recalculate the packages 
> that need to be rebuilt ...

Reiterating, this screws over any form of up front calculation; 
dialup users/per minute connections can't rely on emerge -f to 
actually fetch all involved sources, -p results ain't guranteed 
valid, parallelization must now lock at each threads merge on the off 
chance a recalc is forced.

This is a rather huge step *backwards*; it actually puts us at a 
similar level to autopackages 'resolver'.  The approach would have to 
solve it perfectly imo for it to even be realistically considered.

For your proposal to fly, it must address this somehow imo which 
isn't possible from where I'm sitting.


> dangling nodes get recorded in the new package and considering these old files 
> are not detrimental to the health of the system, you can do such a cleanup 
> once at the end of the emerge

It's not orphaning files, but your scheme doesn't work for any form of 
interruption; failed builds, user intervention, etc, they all can 
leave the lib stuck in the new contents.

Dealing with that possibility means the manager has to maintain on 
disk a list of the refcount of each dangling libs to decrement, 
unmerge has to modify said list, etc.

Further nastyness in short.


> > It also involves changing vdb nodes from "installed and usable" to
> > "installed/usable" or "lingering"
> 
> no ... the old versions get merged into the new one as their existence is 
> purely hidden

How exactly are they going to be hidden?  A new file?  If so, 
backwards compatibility for vdb for transitioning in such a support 
has to be addressed.


> > Tracking BINCOMPAT should *not* be that hard.
> 
> it's one more thing to keep track of and considering all of the possibilities 
> i outlined, a single maintainer can easily lose his sanity ... or you force 
> wasted rebuilds on people when it's only needed in some circumstances

How exactly is this forcing wasted rebuilds?  You're stating that 
maintainers are going to bump it willy nilly.  As I said, it is a key 
that would be bumped *as needed*, and would only affected pkgs that 
specified that node as a binding dep (specially marked atom).

Seriously, maintainers ought to know *now* when they're forcing a 
revdep on folks systems, I'm not seeing the massive overhead from 
pushing that info into a var, nor am I seeing mass forced useless 
rebuilds.

What I *am* seeing is one cluster fuck done to resolution and some 
nasty side affects you're glossing over.  Invalidating -p's output 
opens up some nasty issues.

To be fair, one claimed con could be that it is one more knob 
maintainers have to fool with, although counter is that they should 
already know about the issue (see dev quiz about "what should be done 
when bumping a lib" for an example).


> > Hell, automate a tool to determine if it's a BINCOMPAT bump via NEEDED
> > data (folks should be compiling the pkg anyways), point is it's mainly
> > common sense for maintainenance of it.
> 
> and when are you going to run that tool ?  when you bump the package ?

One would hope that when bumping a package, the dev actually attempts 
to *compile* the damn thing.  Theres your opportunity to run a check, 
hell integrate it as a feature into portage that does the check for 
soname change.  Gurantee that mod is far less intrusive, and far less 
error prone then what you're proposing.


> so now 
> the maintainer has to test it on all the KEYWORDed arches with all the USE 
> combos, or de-KEYWORD it and make the arch maintainers test it ?

Realistically, just the same as the NEEDED solution has holes, the 
maintaining dev can screw up and miss a BINCOMPAT bump.  Difference is 
that they can do something for BINCOMPAT; hell, QA checks can be 
automated to catch missing BINCOMPAT bumps.  

Meanwhile, use flag affects: use deps reduce this heavily (toggle a 
flag off, graph is incomplete and the resolver 'fixes' it), so that's 
actually solvable *without* bincompat.

KEYWORDed arches bit, bit unlikely that the underlying arch is 
actually going to screw with the soname, thus I'd want actual examples 
backing it up.

Besides, again, for keywording, the dev *should* be compiling it, so 
there is a way to catch it :P.  A revbump isn't going to break things 
unless the dev is introducing something intrusive, minor version bumps 
(1.1.0 to 1.1.1) shouldn't be again, introducing anything intrusive; 
regardless, dev should know the high level affects of their change

Inducing a revdep-rebuild is definitely one of those high level 
changes they should be aware of *prior* to the bump.


> or dig 
> through the source code line by line and hope to catch all such cases by 
> hand/eye ?

Bit of FUD here, although I spose I'll just point out that if so's 
change as massively as you're implying above, the affects on -p are 
that much worse.


> > Yes, if the solution can be automated without flinging poo into the
> > code, I'm for it; that said I know what the implementation is going to
> > have to look like, and it's *nasty*.
> 
> i dont think either of our solutions are satisfactory for all presented cases; 
> a hybrid model would be needed

bind the checks in as a QA measure, enabled via FEATURES=stricter, 
used to maintain a metadata var.

Keep in mind I'm not claiming that implementing the binding deps 
modification is all that simple for portage; what I'm pushing here is 
a general solution that can be abused for more then just soname, 
*while* not cluster fucking the resolver execution to hell and 
preserving bounded resolution.

Finally, anyone got a general list of revdep events for the tree?  
Literally, pkg x version y forced a rebuild.  revdep is annoying, but 
stats would be useful to actually evaluate the seperate proposals; 
related, getting stats for how often the various updaters were 
required to be ran for a particular pkgs upgrade would be useful in 
evaluating that particular gap NEEDED has.

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 7271 bytes --]

On Wednesday 27 September 2006 03:54, Brian Harring wrote:
> On Wed, Sep 27, 2006 at 02:24:41AM -0400, Mike Frysinger wrote:
> > as i said, if you have changed ABI without an ABI bump, then the upstream
> > package maintainer is screwing everyone who uses the package, not just
> > Gentoo ... so perhaps we should be talking to them to get the situation
> > resolved for all consumers, not just Gentoo
>
> Happens however; afaik, we also weren't monkey patching ssl in the
> past to preserve the abi either so it still is valid (although rare if
> upstream is behaving) scenario.

we've never monkey patched ssl so i dont really know what you're referring to

> > > Bleh, this is getting back to exactly my point that it's unbounded
> > > resolution.  To support this, every step of execution would require
> > > scanning for dangling nodes to punt; aside from invalidating -p's
> > > output it *still* is a collision-protect hit.
> >
> > when the package upgrade detects an ABI change you recalculate the
> > packages that need to be rebuilt ...
>
> Reiterating, this screws over any form of up front calculation;
> dialup users/per minute connections can't rely on emerge -f to
> actually fetch all involved sources, -p results ain't guranteed
> valid, parallelization must now lock at each threads merge on the off
> chance a recalc is forced.

it does indeed prevent full up front calculation.  we can always make the 
behavior configurable; revdep on the fly or allow people to break it up.  the 
key being that their system will still continue to function as the ABI lib 
has been preserved automagically

> > dangling nodes get recorded in the new package and considering these old
> > files are not detrimental to the health of the system, you can do such a
> > cleanup once at the end of the emerge
>
> It's not orphaning files, but your scheme doesn't work for any form of
> interruption; failed builds, user intervention, etc, they all can
> leave the lib stuck in the new contents.

huh ?  i outlined in a previous e-mail how by simply ordering the operations 
sanely, there is no race condition

> Dealing with that possibility means the manager has to maintain on
> disk a list of the refcount of each dangling libs to decrement,
> unmerge has to modify said list, etc.

which is already being done in the NEEDED file ... funny how unpainless it is 
to generate that file

> > > It also involves changing vdb nodes from "installed and usable" to
> > > "installed/usable" or "lingering"
> >
> > no ... the old versions get merged into the new one as their existence is
> > purely hidden
>
> How exactly are they going to be hidden?  A new file?  If so,
> backwards compatibility for vdb for transitioning in such a support
> has to be addressed.

purely hidden from the standpoint of any new package trying to use it ... 
since you're only installing the runtime ABI library, you cannot link against 
it or utilize it any way other than existing files

> > > Tracking BINCOMPAT should *not* be that hard.
> >
> > it's one more thing to keep track of and considering all of the
> > possibilities i outlined, a single maintainer can easily lose his sanity
> > ... or you force wasted rebuilds on people when it's only needed in some
> > circumstances
>
> How exactly is this forcing wasted rebuilds?  You're stating that
> maintainers are going to bump it willy nilly.  As I said, it is a key
> that would be bumped *as needed*, and would only affected pkgs that
> specified that node as a binding dep (specially marked atom).

no, i mean for example scenarios where a package provides more than one 
library (say libfoo and libbar) and only one of those changes ABI values (say 
libfoo.0 -> libfoo.1).  if another package links against just libbar, you've 
got a pointless rebuild.

> Seriously, maintainers ought to know *now* when they're forcing a
> revdep on folks systems, I'm not seeing the massive overhead from
> pushing that info into a var, nor am I seeing mass forced useless
> rebuilds.

there's a ton of things maintainers ought to know ... pretty soon our package 
maintainers are going to have to be gods if they want to write an ebuild as 
they're going to have to know every detail about the package inside and out

> Realistically, just the same as the NEEDED solution has holes, the
> maintaining dev can screw up and miss a BINCOMPAT bump.  Difference is
> that they can do something for BINCOMPAT; hell, QA checks can be
> automated to catch missing BINCOMPAT bumps.

what's the difference ?  in my scenario they dont have to do anything because 
the bump would have been caught automatically ?

> KEYWORDed arches bit, bit unlikely that the underlying arch is
> actually going to screw with the soname, thus I'd want actual examples
> backing it up.

how about libc.so from glibc and libgcc.so from gcc ?  or would you like other 
packages ?

> Besides, again, for keywording, the dev *should* be compiling it, so
> there is a way to catch it :P.  A revbump isn't going to break things
> unless the dev is introducing something intrusive, minor version bumps
> (1.1.0 to 1.1.1) shouldn't be again, introducing anything intrusive;
> regardless, dev should know the high level affects of their change

uhh, there is no such requirement at this time for revbumping on different 
arches ... currently we say the maintainer tests for his arches and leaves 
all the others as ~arch

> Inducing a revdep-rebuild is definitely one of those high level
> changes they should be aware of *prior* to the bump.

and yet we look at our history of so many packages being bumped with ABI 
changes and users having broken-as-shit systems ... i'm accounting for the 
worst; not hoping for the best

> > or dig
> > through the source code line by line and hope to catch all such cases by
> > hand/eye ?
>
> Bit of FUD here, although I spose I'll just point out that if so's
> change as massively as you're implying above, the affects on -p are
> that much worse.

awesome, mark something you disagree with as FUD, problem solved.  my point is 
that you can never know completely for sure the behavior of a package without 
digging through the entire source code.

> bind the checks in as a QA measure, enabled via FEATURES=stricter,
> used to maintain a metadata var.

a lot of things fail already with FEATURES=stricter ... too bad we dont have 
per-package env var support as then maintainers could just flag all their own 
packages as stricter without wanting to shoot themselves due to everyone 
else's package failures

> Literally, pkg x version y forced a rebuild.  revdep is annoying, but
> stats would be useful to actually evaluate the seperate proposals;
> related, getting stats for how often the various updaters were
> required to be ran for a particular pkgs upgrade would be useful in
> evaluating that particular gap NEEDED has.

any openssl version bump where the numerical value changes ... only the letter 
updates preserve ABI compat (0.9.7[a-z] are compat)

expat-1.x -> expat-2.x

iirc, tcl-8.3.x -> tcl-8.4.x

readline-4.x -> readline-5.x

ncurses-4.x -> ncurses-5.x
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 12725 bytes --]

On Sat, Sep 30, 2006 at 02:01:08PM -0400, Mike Frysinger wrote:
> On Wednesday 27 September 2006 03:54, Brian Harring wrote:
> > > > Bleh, this is getting back to exactly my point that it's unbounded
> > > > resolution.  To support this, every step of execution would require
> > > > scanning for dangling nodes to punt; aside from invalidating -p's
> > > > output it *still* is a collision-protect hit.
> > >
> > > when the package upgrade detects an ABI change you recalculate the
> > > packages that need to be rebuilt ...
> >
> > Reiterating, this screws over any form of up front calculation;
> > dialup users/per minute connections can't rely on emerge -f to
> > actually fetch all involved sources, -p results ain't guranteed
> > valid, parallelization must now lock at each threads merge on the off
> > chance a recalc is forced.
> 
> it does indeed prevent full up front calculation.  we can always make the 
> behavior configurable; revdep on the fly or allow people to break it up.  the 
> key being that their system will still continue to function as the ABI lib 
> has been preserved automagically

Folks aren't that daft; you make it an optional component, it means a 
*proper* solution will never be pushed because the duct tape is in 
place already.

If that's what folks want, sure, but what you're proposing is just 
sliding NEEDED in as the defacto solution without labeling it as such.


> > > dangling nodes get recorded in the new package and considering these old
> > > files are not detrimental to the health of the system, you can do such a
> > > cleanup once at the end of the emerge
> >
> > It's not orphaning files, but your scheme doesn't work for any form of
> > interruption; failed builds, user intervention, etc, they all can
> > leave the lib stuck in the new contents.
> 
> huh ?  i outlined in a previous e-mail how by simply ordering the operations 
> sanely, there is no race condition

Re-read your emails, and mine please.  The scenario I pointed out was 
ctrl+c'ing the merge while it's doing a rebuild for lib1 to lib2.

Now how does portage know that it needs to complete that upgrade for 
the next emerge action?  How does it know to even scan for lib1, let 
alone punting?

It's *not* simple, and I keep pointing out this issue (and you're 
missing it every damn time).  Please address it, or point to where you 
have (gone over the thread and still not seeing anything even 
remotely touching this flaw).


> > Dealing with that possibility means the manager has to maintain on
> > disk a list of the refcount of each dangling libs to decrement,
> > unmerge has to modify said list, etc.
> 
> which is already being done in the NEEDED file ... funny how unpainless it is 
> to generate that file

First of all, unpainless is painful.  Which is apt, actually.

Familiar with old style virtuals?  That whole, "you have to walk the 
whole vdb to collect all old style virtuals" ?

This is the same damn thing.

There is no refcount maintained via NEEDED, just a list of libs a 
binary uses; you _still_ have to walk the damn vdb to build the 
refcount list.

Now either you're forcing a fairly huge mapping in memory, or you're 
forcing a scan of the vdb for every pkg operation that has NEEDED 
entries it will install.

So no, NEEDED doesn't cover this, and my point still stands.


> > > > It also involves changing vdb nodes from "installed and usable" to
> > > > "installed/usable" or "lingering"
> > >
> > > no ... the old versions get merged into the new one as their existence is
> > > purely hidden
> >
> > How exactly are they going to be hidden?  A new file?  If so,
> > backwards compatibility for vdb for transitioning in such a support
> > has to be addressed.
> 
> purely hidden from the standpoint of any new package trying to use it ... 
> since you're only installing the runtime ABI library, you cannot link against 
> it or utilize it any way other than existing files.

Except for dlopen, but thats again besides my original point; how do 
you note to portage that the lib is one to watch so it can be punted?

You're suggesting it get shoved into contents, and for portage to 
identify it, it has to do a revdep mapping on the fly to find it.  

This *sucks* massively, both from a speed and complexity standpoint; 
further, the lib isn't hidden from pkg ops (say quickpkging, or 
binpkging) in any way so the cruft spreads.  That's surprisingly minor 
in comparison to implications of relying on refcounting to identify 
the lib to punt.

If the lib to punt isn't tracked in some fashion, the only algo 
you're left with is attempting to find all libs that have a refcount 
of zero, and punt those- obviously, this is going to screw up just 
about every single dlopen based linkage, literally, suck an algo 
won't spot that python dlopen's it's modules, and would think the 
refcount was zero (thus the so could get booted).

The response there is to add blacklists, directories to *not* inspect, 
which is a further hack to try and make this tank fly.

*OR*, you're stuck maintaining a seperated list of libs to punt, 
rather then just trying to slide the lib into existing CONTENTS.


> > > > Tracking BINCOMPAT should *not* be that hard.
> > >
> > > it's one more thing to keep track of and considering all of the
> > > possibilities i outlined, a single maintainer can easily lose his sanity
> > > ... or you force wasted rebuilds on people when it's only needed in some
> > > circumstances
> >
> > How exactly is this forcing wasted rebuilds?  You're stating that
> > maintainers are going to bump it willy nilly.  As I said, it is a key
> > that would be bumped *as needed*, and would only affected pkgs that
> > specified that node as a binding dep (specially marked atom).
> 
> no, i mean for example scenarios where a package provides more than one 
> library (say libfoo and libbar) and only one of those changes ABI values (say 
> libfoo.0 -> libfoo.1).  if another package links against just libbar, you've 
> got a pointless rebuild.

If one changes it's version, it's a fair bet that any consumer of that 
pkg is linked to more then just that lib; as such they would be 
rebuilt *anyways*.


> > Seriously, maintainers ought to know *now* when they're forcing a
> > revdep on folks systems, I'm not seeing the massive overhead from
> > pushing that info into a var, nor am I seeing mass forced useless
> > rebuilds.
> 
> there's a ton of things maintainers ought to know ... pretty soon our package 
> maintainers are going to have to be gods if they want to write an ebuild as 
> they're going to have to know every detail about the package inside and out

Sorry, but if a developer is bumping a pkg and doesn't even look to 
see if the damn thing is potentially going to break others via soname 
changes, that maintainer is being an idiot.

Being aware of a high level detail like "hey, my package installs a 
library, and they changed the soname" isn't being god like, it's 
being at least haphazardly concerned about QA of the tree.

Worth noting this exact scenario is already laid out in the quizes 
too; 'sposed to know the affects of stabilizing a library.


> > Realistically, just the same as the NEEDED solution has holes, the
> > maintaining dev can screw up and miss a BINCOMPAT bump.  Difference is
> > that they can do something for BINCOMPAT; hell, QA checks can be
> > automated to catch missing BINCOMPAT bumps.
> 
> what's the difference ?  in my scenario they dont have to do anything because 
> the bump would have been caught automatically ?

Your solution has holes up the ying yang for actual 
implementation/handling of it due to the fact it's forcing the 
resolver to be an effective recalc at each step, and relies on duct 
taping metadata handling to try and exempt bits of lingering data.

You move that data up front, the implementation is actually sane, it's 
deterministic (thus *representative* to the user) and is more 
powerful.


> > KEYWORDed arches bit, bit unlikely that the underlying arch is
> > actually going to screw with the soname, thus I'd want actual examples
> > backing it up.
> 
> how about libc.so from glibc and libgcc.so from gcc ?  or would you like other 
> packages ?

Considering such a change would be internal ABI, NEEDED doesn't 
actually fix anything; if it were a soname change, level playing 
ground again.

Admittedly, BINCOMPAT breaks down there due to not being fine grained 
enough if it's internal changes; that however doesn't make NEEDED a 
good solution, just means that my alternative has a hole.


> > Besides, again, for keywording, the dev *should* be compiling it, so
> > there is a way to catch it :P.  A revbump isn't going to break things
> > unless the dev is introducing something intrusive, minor version bumps
> > (1.1.0 to 1.1.1) shouldn't be again, introducing anything intrusive;
> > regardless, dev should know the high level affects of their change
> 
> uhh, there is no such requirement at this time for revbumping on different 
> arches ... currently we say the maintainer tests for his arches and leaves 
> all the others as ~arch

Reiterating; devs should know the high level affects of their changes.  

If it's going to change the soname version, they should know from the 
get go- unless it's an arch specific breakage (which I still posit is 
the rare/corner case), they will *see* it for their arch and bump it 
already.

They keywording comment above is specific to moving from ~arch to arch 
also; for new bumps, the ~arch keywords carry forward, but see the 
paragraph above re: it.


> > Inducing a revdep-rebuild is definitely one of those high level
> > changes they should be aware of *prior* to the bump.
> 
> and yet we look at our history of so many packages being bumped with ABI 
> changes and users having broken-as-shit systems ... i'm accounting for the 
> worst; not hoping for the best

ABI changes aren't fixed by NEEDED.  Presume you meant soname changes.

Stating that things are broken doesn't make NEEDED automagically 
better either; *both* enable a way to fix it, so you need to justify 
the "accounting for the worst; not hoping for the best".


> > > or dig
> > > through the source code line by line and hope to catch all such cases by
> > > hand/eye ?
> >
> > Bit of FUD here, although I spose I'll just point out that if so's
> > change as massively as you're implying above, the affects on -p are
> > that much worse.
> 
> awesome, mark something you disagree with as FUD, problem solved.  my point is 
> that you can never know completely for sure the behavior of a package without 
> digging through the entire source code.

It's FUD due to the fact NEEDED suffers the same fucking issue.  The 
irony is that BINCOMPAT would at least give a knob to mark it as a 
breakage, NEEDED cannot.


> > bind the checks in as a QA measure, enabled via FEATURES=stricter,
> > used to maintain a metadata var.
> 
> a lot of things fail already with FEATURES=stricter ... too bad we dont have 
> per-package env var support as then maintainers could just flag all their own 
> packages as stricter without wanting to shoot themselves due to everyone 
> else's package failures

Take it up with your fellow portage devs then.  It was shot down 
internally multiple times due to the affects it has on the python 
side.

If you want to get into per package features, split a thread and I'll 
let marius argue over it.


> > Literally, pkg x version y forced a rebuild.  revdep is annoying, but
> > stats would be useful to actually evaluate the seperate proposals;
> > related, getting stats for how often the various updaters were
> > required to be ran for a particular pkgs upgrade would be useful in
> > evaluating that particular gap NEEDED has.
> 
> any openssl version bump where the numerical value changes ... only the letter 
> updates preserve ABI compat (0.9.7[a-z] are compat)
> 
> expat-1.x -> expat-2.x
> 
> iirc, tcl-8.3.x -> tcl-8.4.x
> 
> readline-4.x -> readline-5.x
> 
> ncurses-4.x -> ncurses-5.x

Was looking for stats comparing breakage to a subset of the tree; iow, 
out of N packages, how many actually induce a rebuild?  Yes it's 
slightly nasty generating those stats (tinderbox?), but it would be 
useful.


Frankly, if you don't believe me or think my points are correct about 
how a NEEDED based solution is going to suck, zac/jason/genone/anyone 
else.  They're going to point out the same flaws.

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC about another *DEPEND variable

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 6114 bytes --]

On Saturday 30 September 2006 15:34, Brian Harring wrote:
> If that's what folks want, sure, but what you're proposing is just
> sliding NEEDED in as the defacto solution without labeling it as such.

no idea what this means

> Re-read your emails, and mine please.  The scenario I pointed out was
> ctrl+c'ing the merge while it's doing a rebuild for lib1 to lib2.

then let me turn it around ... how exactly does your solution account for all 
these little details ?  i dont recall you outlining anything ...

> > > How exactly is this forcing wasted rebuilds?  You're stating that
> > > maintainers are going to bump it willy nilly.  As I said, it is a key
> > > that would be bumped *as needed*, and would only affected pkgs that
> > > specified that node as a binding dep (specially marked atom).
> >
> > no, i mean for example scenarios where a package provides more than one
> > library (say libfoo and libbar) and only one of those changes ABI values
> > (say libfoo.0 -> libfoo.1).  if another package links against just
> > libbar, you've got a pointless rebuild.
>
> If one changes it's version, it's a fair bet that any consumer of that
> pkg is linked to more then just that lib; as such they would be
> rebuilt *anyways*.

it isnt guaranteed ... you asked for a pointless rebuild and i gave you 
one ... one that is not falsely flagged when reviewing the NEEDED list

the other example is where the ABI changes for one arch but not for any 
other ... what do you do, force ABI rebuild for everyone ?  ok, maybe that 
arch was x86 so we say screw you to the smaller arch teams ... but what if 
that arch was a smaller arch team ?

> Sorry, but if a developer is bumping a pkg and doesn't even look to
> see if the damn thing is potentially going to break others via soname
> changes, that maintainer is being an idiot.

and yet it does happen and again, we're looking at this from different 
angles ... you'd rather assume the developer is 100% competent and never gets 
things wrong; i'd rather assume nothing and let the details be discovered 
automatically.

in the end, the people using Gentoo are the ones that suffer and i'm tired of 
seeing systems broken beyond usuability because a developer unleashed a 
package without realizing the consequences of it

> > > Realistically, just the same as the NEEDED solution has holes, the
> > > maintaining dev can screw up and miss a BINCOMPAT bump.  Difference is
> > > that they can do something for BINCOMPAT; hell, QA checks can be
> > > automated to catch missing BINCOMPAT bumps.

i bet a post-emerge would make this painless and automate the QA step ... 
after src_install(), you run something like `scanelf -qsRS ${D}` and save the 
results in vdb/SONAME ... and then on subsequent installs, you run it again 
and if the SONAME changes but BINCOMPAT did not, you bail with a die message 
telling the user to go file a bug and preventing the merge which would have 
broken his system

> > > KEYWORDed arches bit, bit unlikely that the underlying arch is
> > > actually going to screw with the soname, thus I'd want actual examples
> > > backing it up.
> >
> > how about libc.so from glibc and libgcc.so from gcc ?  or would you like
> > other packages ?
>
> Considering such a change would be internal ABI, NEEDED doesn't
> actually fix anything; if it were a soname change, level playing
> ground again.

it is a SONAME change, why else would i mention this

> Reiterating; devs should know the high level affects of their changes.

reiterating: you're hoping for the best; i'm looking at our history and 
assuming that devs will make mistakes as everyone does

> If it's going to change the soname version, they should know from the
> get go- unless it's an arch specific breakage (which I still posit is
> the rare/corner case), they will *see* it for their arch and bump it
> already.

maybe, but it's still a possibility which analyzing of SONAME would catch

> Stating that things are broken doesn't make NEEDED automagically
> better either; *both* enable a way to fix it, so you need to justify
> the "accounting for the worst; not hoping for the best".

justify how ?  would you like me to dig up the bugs where devs bumped packages 
into stable and the SONAME change broke a ton of people's systems ?  has it 
never happened to you ?

> > > > or dig
> > > > through the source code line by line and hope to catch all such cases
> > > > by hand/eye ?
> > >
> > > Bit of FUD here, although I spose I'll just point out that if so's
> > > change as massively as you're implying above, the affects on -p are
> > > that much worse.
> >
> > awesome, mark something you disagree with as FUD, problem solved.  my
> > point is that you can never know completely for sure the behavior of a
> > package without digging through the entire source code.
>
> It's FUD due to the fact NEEDED suffers the same fucking issue.  The
> irony is that BINCOMPAT would at least give a knob to mark it as a
> breakage, NEEDED cannot.

and then you get angry ?  try to have a discussion without getting all anxious 
as that only wastes time

there's a few issues here, not just one ...
 - ABI change w/no SONAME bump - impossible to catch without reading source; 
BINCOMPAT would allow for it; NEEDED would not
 - ABI change w/SONAME bump - BINCOMPAT would not catch; NEEDED would

> Frankly, if you don't believe me or think my points are correct about
> how a NEEDED based solution is going to suck, zac/jason/genone/anyone
> else.  They're going to point out the same flaws.

if you dont want to hash out ideas on a development list but rather just 
declare one right and one wrong, then feel free to leave.  if you're confused 
about my intentions here then let me set you straight.  i see advantages to 
both sides and i'm going to argue out problems until we have something good 
to push back into portage.  i dont really care if the final solution is "my 
idea" or not; such personal ties is just stupid and helps no one.
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]