[gentoo-dev] making revdep-rebuild (partially) obsolete

[gentoo-dev] making revdep-rebuild (partially) obsolete

[Enrico Weigelt]

Hi folks,


just a little braindump, how revdep-rebuild could be made
(partially) obsolete in future:

Today it might happen that on an library update an old .so file
gets unmerged while still used by somebody - that's what revdep-
rebuild scans for. While it should catch those cases, we still
have some downtime for certain packages (in bad cases, when it
broke somewhere deep in the dependency chain, rebuild might take
quite a lot of time).

The main problem IMHO is that portage doesn't record which libraries
some package links in, so it doesn't know which ones have to be
protected from unmerge (unless explicitly stated somewhere).
So I'd propose to add record that information. On next merge,
this information can be used for an automatic library-protect.
This would also record which libraries have been protected from
removal and for whom. Subsequent merges will update this that,
and once all importers have been unmerged, depclean can clean
up the leftover dirt.


What do you think about this idea ?


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-dev] making revdep-rebuild (partially) obsolete

[Mike Gilbert]

On 12/31/2010 03:42 PM, Enrico Weigelt wrote:
> What do you think about this idea ?

I think you should check out the preserve-libs feature in portage-2.2.

Re: [gentoo-dev] making revdep-rebuild (partially) obsolete

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 785 bytes --]

On Fri, 31 Dec 2010 21:42:32 +0100
Enrico Weigelt <weigelt@metux.de> wrote:

> The main problem IMHO is that portage doesn't record which libraries
> some package links in, so it doesn't know which ones have to be
> protected from unmerge (unless explicitly stated somewhere).
> So I'd propose to add record that information. On next merge,
> this information can be used for an automatic library-protect.
> This would also record which libraries have been protected from
> removal and for whom. Subsequent merges will update this that,
> and once all importers have been unmerged, depclean can clean
> up the leftover dirt.
> 
> What do you think about this idea ?

You mean what do we think about portage-2.2 and preserved-libs?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] making revdep-rebuild (partially) obsolete

[dev-random]

> ...
> Subsequent merges will update this that,
> ...

Subsequent merges may happen after a long while. Old, possibly
vulunerable library will still be used, and most likely unseen by
admin.

Re: [gentoo-dev] making revdep-rebuild (partially) obsolete

[Zac Medico]

On 12/31/2010 12:42 PM, Enrico Weigelt wrote:
> The main problem IMHO is that portage doesn't record which libraries
> some package links in, so it doesn't know which ones have to be
> protected from unmerge (unless explicitly stated somewhere).
> So I'd propose to add record that information. On next merge,
> this information can be used for an automatic library-protect.
> This would also record which libraries have been protected from
> removal and for whom. Subsequent merges will update this that,
> and once all importers have been unmerged, depclean can clean
> up the leftover dirt.

As other's have mentioned, this is already implemented in portage-2.2
with FEATURES=preserve-libs and @preserved-rebuild.

However, before this feature is unmasked, I think that it's critical to
implement bug 192319 [1] (abi-slot-deps) in order to ensure that reverse
dependencies of preserved libraries are rebuilt at the earliest possible
opportunity since this will minimize the possibility of symbol
collisions [2].

[1] http://bugs.gentoo.org/show_bug.cgi?id=192319
[2]
http://blog.flameeyes.eu/2008/06/30/a-few-risks-i-see-related-to-the-new-portage-2-2-preserve-libs-behaviour
-- 
Thanks,
Zac

Re: [gentoo-dev] making revdep-rebuild (partially) obsolete

[Enrico Weigelt]

* Micha?? Górny <mgorny@gentoo.org> schrieb:

> > What do you think about this idea ?
> 
> You mean what do we think about portage-2.2 and preserved-libs?

Well, I'm still using portage-2.1, so I wans't aware of whats going
on there. For now it seems the preservation is still done explicitly
(preserve_old_lib calls in certain ebuilds ?). My proposal is to
record the necessary information (eg. which so some executable/so
is linked against) automatically - does portage-2.2 do that ?

BTW: several blog/maillist postings talked about the problem that
even on recompile, older library versions could be linked in even
on recompile. Somebody suggested to move away preserved libs to
another directly (which is then added to ld.so.conf). What do you
think about that ?

Another approach could be building everything in an separate,
minimal sysroot or chroot. (I admit, I have no idea how complex
it would be to implement that in portage - my Briegel buildsystem
does always does this)


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-dev] making revdep-rebuild (partially) obsolete

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 286 bytes --]

On Saturday, January 01, 2011 23:09:11 Enrico Weigelt wrote:
> BTW: several blog/maillist postings talked about the problem that
> even on recompile, older library versions could be linked in even
> on recompile.

you'll need to provide an actual example.  i have yet to see one.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] making revdep-rebuild (partially) obsolete

[Zac Medico]

On 01/01/2011 08:09 PM, Enrico Weigelt wrote:
> Well, I'm still using portage-2.1, so I wans't aware of whats going
> on there. For now it seems the preservation is still done explicitly
> (preserve_old_lib calls in certain ebuilds ?). My proposal is to
> record the necessary information (eg. which so some executable/so
> is linked against) automatically - does portage-2.2 do that ?

Yes, portage-2.2 uses /var/db/pkg/*/*/NEEDED.ELF.3 for that.
-- 
Thanks,
Zac

Re: [gentoo-dev] making revdep-rebuild (partially) obsolete

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 651 bytes --]

On Sunday, January 02, 2011 00:17:03 Zac Medico wrote:
> On 01/01/2011 08:09 PM, Enrico Weigelt wrote:
> > Well, I'm still using portage-2.1, so I wans't aware of whats going
> > on there. For now it seems the preservation is still done explicitly
> > (preserve_old_lib calls in certain ebuilds ?). My proposal is to
> > record the necessary information (eg. which so some executable/so
> > is linked against) automatically - does portage-2.2 do that ?
> 
> Yes, portage-2.2 uses /var/db/pkg/*/*/NEEDED.ELF.3 for that.

although portage has long been generating the NEEDED files in vdb.  even 
stable portage generates these files.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[gentoo-dev] Re: making revdep-rebuild (partially) obsolete

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 1222 bytes --]

On Sun, 2 Jan 2011 00:08:34 -0500
Mike Frysinger <vapier@gentoo.org> wrote:

> On Saturday, January 01, 2011 23:09:11 Enrico Weigelt wrote:
> > BTW: several blog/maillist postings talked about the problem that
> > even on recompile, older library versions could be linked in even
> > on recompile.
> 
> you'll need to provide an actual example.  i have yet to see one.

Not exactly the same thing, but the recent spidermonkey bump changed the lib
from libjs.so to libmozjs.so.  mediatomb kept finding the old libjs on
rebuild.  I had to uninstall mediatomb to dump the library and reinstall it.

There's not much we can do about stuff like that, but it's not an uncommon
occurrence.  I usually run into a sticky preserved lib or two every couple
months.

I still think it's much better than the current portage 2.1 behavior which
results in a broken mediatomb.  I'll take the rare possibility of a symbol
collision over a guaranteed broken package any day.


-- 
fonts, gcc-porting,                  it makes no sense how it makes no sense
toolchain, wxwidgets                           but i'll take it free anytime
@ gentoo.org                EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: making revdep-rebuild (partially) obsolete

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1404 bytes --]

On Sunday, January 02, 2011 02:31:06 Ryan Hill wrote:
> On Sun, 2 Jan 2011 00:08:34 -0500 Mike Frysinger wrote:
> > On Saturday, January 01, 2011 23:09:11 Enrico Weigelt wrote:
> > > BTW: several blog/maillist postings talked about the problem that
> > > even on recompile, older library versions could be linked in even
> > > on recompile.
> > 
> > you'll need to provide an actual example.  i have yet to see one.
> 
> Not exactly the same thing, but the recent spidermonkey bump changed the
> lib from libjs.so to libmozjs.so.  mediatomb kept finding the old libjs on
> rebuild.  I had to uninstall mediatomb to dump the library and reinstall
> it.
> 
> There's not much we can do about stuff like that, but it's not an uncommon
> occurrence.  I usually run into a sticky preserved lib or two every couple
> months.
> 
> I still think it's much better than the current portage 2.1 behavior which
> results in a broken mediatomb.  I'll take the rare possibility of a symbol
> collision over a guaranteed broken package any day.

yes, this is a case where linking would pick up the old library.  but that is 
*only* because the SONAME is stupid and is the same thing as the link name.  
imo, these packages are pretty much broken and should get their SONAME fixed.  
a simple change as making the SONAME "libjs.so.0" instead of "libjs.so" would 
make this a non-issue.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] making revdep-rebuild (partially) obsolete

[Enrico Weigelt]

* Mike Frysinger <vapier@gentoo.org> schrieb:

> although portage has long been generating the NEEDED files in vdb.  even 
> stable portage generates these files.

Ah, okay, I wasn't aware of that.

What's the difference between NEEEDED and NEEDED.2 ? Multiarch ?


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------