[gentoo-dev] Over using preserve_old_lib, don't do that

[gentoo-dev] Over using preserve_old_lib, don't do that

[Samuli Suominen]

I've recently stumbled upon several packages unnecessarily using old
preserve_old_lib feature from eutils.eclass, namely:

libgnomekbd
libproxy

And then users hit issues like this:

https://bugs.gentoo.org/show_bug.cgi?id=326517#c5

Please only use the preserve_old_lib function in case of breaking:

- base-system
- toolchain
- or the library is so widely used that revdep-rebuilding the system
would be nearly impossible without having the old lib around since
revdep-rebuild (emerge) can't get the emerge ordering right (because of
overlinking and lack of asneeded by default, namely jpeg and png comes
into mind)

It's a hack, not a solution

Thanks

Re: [gentoo-dev] Over using preserve_old_lib, don't do that

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 279 bytes --]

On 7/2/10 7:51 AM, Samuli Suominen wrote:
> It's a hack, not a solution

Should we make repoman issue a warning about it?

It already warns about using make -j1 as a workaround for upstream
issues. The new warning could be on the same level (yellow, not red).

Paweł


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Over using preserve_old_lib, don't do that

[Samuli Suominen]

On 07/02/2010 08:51 AM, Samuli Suominen wrote:
> I've recently stumbled upon several packages unnecessarily using old
> preserve_old_lib feature from eutils.eclass, namely:
> 
> libgnomekbd
> libproxy
> 
> And then users hit issues like this:
> 
> https://bugs.gentoo.org/show_bug.cgi?id=326517#c5
> 
> Please only use the preserve_old_lib function in case of breaking:
> 
> - base-system
> - toolchain
> - or the library is so widely used that revdep-rebuilding the system
> would be nearly impossible without having the old lib around since
> revdep-rebuild (emerge) can't get the emerge ordering right (because of
> overlinking and lack of asneeded by default, namely jpeg and png comes
> into mind)

Futhermore the old library gets injected to binpkgs as a file of the new
package, see recent bug:

http://bugs.gentoo.org/326275
http://bugs.gentoo.org/show_bug.cgi?id=326275#c7

Re: [gentoo-dev] Over using preserve_old_lib, don't do that

[Samuli Suominen]

On 07/02/2010 08:54 AM, "Paweł Hajdan, Jr." wrote:
> On 7/2/10 7:51 AM, Samuli Suominen wrote:
>> It's a hack, not a solution
> 
> Should we make repoman issue a warning about it?
> 
> It already warns about using make -j1 as a workaround for upstream
> issues. The new warning could be on the same level (yellow, not red).

That sounds good (and easily doable), small reminder doesn't hurt

Re: [gentoo-dev] Over using preserve_old_lib, don't do that

[Samuli Suominen]

On 07/02/2010 09:03 AM, Samuli Suominen wrote:
> On 07/02/2010 08:54 AM, "Paweł Hajdan, Jr." wrote:
>> On 7/2/10 7:51 AM, Samuli Suominen wrote:
>>> It's a hack, not a solution
>>
>> Should we make repoman issue a warning about it?
>>
>> It already warns about using make -j1 as a workaround for upstream
>> issues. The new warning could be on the same level (yellow, not red).
> 
> That sounds good (and easily doable), small reminder doesn't hurt
> 

https://bugs.gentoo.org/show_bug.cgi?id=326553

Re: [gentoo-dev] Over using preserve_old_lib, don't do that

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1158 bytes --]

On Friday, July 02, 2010 01:51:25 Samuli Suominen wrote:
> I've recently stumbled upon several packages unnecessarily using old
> preserve_old_lib feature from eutils.eclass, namely:
> 
> libgnomekbd
> libproxy
> 
> And then users hit issues like this:
> 
> https://bugs.gentoo.org/show_bug.cgi?id=326517#c5
> 
> Please only use the preserve_old_lib function in case of breaking:
> - or the library is so widely used that revdep-rebuilding the system
> would be nearly impossible without having the old lib around since
> revdep-rebuild (emerge) can't get the emerge ordering right (because of
> overlinking and lack of asneeded by default, namely jpeg and png comes
> into mind)

i dont know how critical libgnomekbd is to the general running of GNOME, but 
if all GNOME apps break because of it, then that is a valid use case.  
upgrading a library that the whole desktop env cascades upon should not result 
in the inability to log in (i.e. being forced to work in a Linux console shell 
to recover things).

that bug report looks to me like the user not reading the output where they 
have to delete the file themselves.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Over using preserve_old_lib, don't do that

[Gilles Dartiguelongue]

Le vendredi 02 juillet 2010 à 08:51 +0300, Samuli Suominen a écrit :
> I've recently stumbled upon several packages unnecessarily using old
> preserve_old_lib feature from eutils.eclass, namely:
> 
> libgnomekbd
> libproxy
> 
> And then users hit issues like this:
> 
> https://bugs.gentoo.org/show_bug.cgi?id=326517#c5

     1. How is it different than preserved-libs feature from
        portage-2.2 ? The issue that I have with you argument is that
        you encourage breaking user apps at build time instead of
        leaving the user some time to run revdep-rebuild or any other
        tool needed when user wishes so.
     2. We cannot (and do not in gnome herd) support users who did not
        complete revdep-rebuild. We help them get out of build problem
        if possible and if the bug still exists by then, we try to fix
        it but otherwise it's just closed invalid.

-- 
Gilles Dartiguelongue <eva@gentoo.org>
Gentoo

Re: [gentoo-dev] Over using preserve_old_lib, don't do that

[Mike Auty]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Sorry I'm a bit late to the thread,

Just to add that empathy preserves libemapthy in this manner too.

On 05/07/10 17:40, Gilles Dartiguelongue wrote:
> 
>      1. How is it different than preserved-libs feature from
>         portage-2.2 ? The issue that I have with you argument is that
>         you encourage breaking user apps at build time instead of
>         leaving the user some time to run revdep-rebuild or any other
>         tool needed when user wishes so.

This is different from preserve-libs because FEATURES="-preserve-libs"
doesn't stop these calls from keeping old libraries around.  Also, once
preserve-libs has been used, a normal revdev-rebuild won't spot these
issues, and cruft-checkers can't find them because they're classed as
part of the new package.

I understand we have users who want this feature, and also that we
advise everybody to read through every single elog message ever made.
Having said that, I personally know how to run revdep-rebuild, and I do
it often so that when I'm upgrading 100+ packages in one go, I don't
then have to sit around reading through every elog message to make sure
that a library I didn't ask for doesn't accidentally get left on my
system for all time.

I can live with this for in places where it causes massive breakage
(openssl/libpng/libjpg), because it's genuinely useful, but I think it
should be restricted to such important packages, or at least disabled by
FEATURES="-preserve-libs".

Ideally, these calls should either adhere to FEATURES="-preserve-libs",
or there should be a tool that can identify which files portage has
preserved, and allow easy rebuilding of dependent packages, and removal.
 At the moment, I'm having to manually grep ebuilds, ls the libraries
and run revdep-rebuild over them one at a time...

Mike  5:)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)

iEYEARECAAYFAkw+y6oACgkQu7rWomwgFXoehQCgsrbUBRorY6J4rBmASh16t1eP
YzoAnAhAi7kWd/bI9xhUh8UHMFfCR5xY
=OOj9
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Over using preserve_old_lib, don't do that

[Gilles Dartiguelongue]

Le jeudi 15 juillet 2010 à 09:49 +0100, Mike Auty a écrit :
[...]
> I can live with this for in places where it causes massive breakage
> (openssl/libpng/libjpg), because it's genuinely useful, but I think it
> should be restricted to such important packages, or at least disabled by
> FEATURES="-preserve-libs".
> 
> Ideally, these calls should either adhere to FEATURES="-preserve-libs",
> or there should be a tool that can identify which files portage has
> preserved, and allow easy rebuilding of dependent packages, and removal.
>  At the moment, I'm having to manually grep ebuilds, ls the libraries
> and run revdep-rebuild over them one at a time...

These sound like very good ideas to me.

-- 
Gilles Dartiguelongue <eva@gentoo.org>
Gentoo

[gentoo-dev] Re: Over using preserve_old_lib, don't do that

[Duncan]

Gilles Dartiguelongue posted on Thu, 15 Jul 2010 11:09:39 +0200 as
excerpted:

> Le jeudi 15 juillet 2010 à 09:49 +0100, Mike Auty a écrit : [...]
>> I can live with this for in places where it causes massive breakage
>> (openssl/libpng/libjpg), because it's genuinely useful, but I think it
>> should be restricted to such important packages, or at least disabled
>> by FEATURES="-preserve-libs".
>> 
>> Ideally, these calls should either adhere to FEATURES="-preserve-libs",
>> or there should be a tool that can identify which files portage has
>> preserved, and allow easy rebuilding of dependent packages, and
>> removal.
>>  At the moment, I'm having to manually grep ebuilds, ls the libraries
>> and run revdep-rebuild over them one at a time...
> 
> These sound like very good ideas to me.

++

If I have FEATURE=-preserve-libs, that's what I want.  Exceptions should 
be limited to what will break the toolchain (including revdep-rebuild 
here, since that's what's normally used to get out of the situation) 
itself.

If there was a way to handle it so a general revdep-rebuild run would 
still detect the preserved library as missing and do the necessary 
rebuilds, it'd be one thing, but if the libraries are there, it figures 
things are OK unless you've fed it that specific library, thereby making a 
general revdep-rebuld run useless at the very task it was designed to fix.

Talking about which...  What about creating an eutils (or whatever) 
function to handle the critical preservations, having it build a 
centralized list of them somewhere, and having a revdep-rebuild mode that 
will treat that list as if it had been fed in with --library on the 
command line?  Make revdep-rebuild able to run this mode either on its 
own, or as part of an otherwise general run, and then you can have 
packages (or the package-manager itself, if it uses the list as well) 
preserve libs as they wish, without interfering with the ability of revdep-
rebuild to detect and resolve the issues in a normal run.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Over using preserve_old_lib, don't do that

[Maciej Mrozowski]

On Thursday 15 of July 2010 12:14:29 Duncan wrote:
> Gilles Dartiguelongue posted on Thu, 15 Jul 2010 11:09:39 +0200 as
> 
> excerpted:
> > Le jeudi 15 juillet 2010 à 09:49 +0100, Mike Auty a écrit : [...]
> > 
> >> I can live with this for in places where it causes massive breakage
> >> (openssl/libpng/libjpg), because it's genuinely useful, but I think it
> >> should be restricted to such important packages, or at least disabled
> >> by FEATURES="-preserve-libs".
> >> 
> >> Ideally, these calls should either adhere to FEATURES="-preserve-libs",
> >> or there should be a tool that can identify which files portage has
> >> preserved, and allow easy rebuilding of dependent packages, and
> >> removal.
> >> 
> >>  At the moment, I'm having to manually grep ebuilds, ls the libraries
> >> 
> >> and run revdep-rebuild over them one at a time...
> > 
> > These sound like very good ideas to me.
> 
> ++
> 
> If I have FEATURE=-preserve-libs, that's what I want.  Exceptions should
> be limited to what will break the toolchain (including revdep-rebuild
> here, since that's what's normally used to get out of the situation)
> itself.
> 
> If there was a way to handle it so a general revdep-rebuild run would
> still detect the preserved library as missing and do the necessary
> rebuilds, it'd be one thing, but if the libraries are there, it figures
> things are OK unless you've fed it that specific library, thereby making a
> general revdep-rebuld run useless at the very task it was designed to fix.
> 
> Talking about which...  What about creating an eutils (or whatever)
> function to handle the critical preservations, having it build a
> centralized list of them somewhere, and having a revdep-rebuild mode that
> will treat that list as if it had been fed in with --library on the
> command line?  Make revdep-rebuild able to run this mode either on its
> own, or as part of an otherwise general run, and then you can have
> packages (or the package-manager itself, if it uses the list as well)
> preserve libs as they wish, without interfering with the ability of revdep-
> rebuild to detect and resolve the issues in a normal run.

And what about using portage 2.2 and be done with it. I don't see the point in 
reinventing the wheel yet again.

Imho, revdep-rebuild and all 'misc' tools requiring users' good will like 
python-updater should be obsolete and phased out in favour of package manager 
controlled mechanisms.

-- 
regards
MM

Re: [gentoo-dev] Re: Over using preserve_old_lib, don't do that

[Mike Auty]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 15/07/10 14:57, Maciej Mrozowski wrote:
> And what about using portage 2.2 and be done with it. I don't see the point in 
> reinventing the wheel yet again.

I'm using portage-2.2 and have been since it first came out.  I find the
@set notation invaluable.  I didn't like the preserve-libs feature
however, so I set FEATURES="-preserve-libs".  Unfortunately the eutils
function only checks for the presence of preserve-libs to drop out and
let portage handle it, not the absence of it.  So there's no way to get
that function *not* to leave these extraneous files around, even with 2.2.

As I said, that's fine, and I'm happy with that for extreme situations
(toolchain breaking or libpng/jpg size changes), but I'm not for most of
the other packages.

If portage offers a way to turn off functionality (like preserving
libraries), it should actually turn it off, rather than sometimes turn
it off...

> Imho, revdep-rebuild and all 'misc' tools requiring users' good will like 
> python-updater should be obsolete and phased out in favour of package manager 
> controlled mechanisms.

You're just moving around the good will, but it's still required.
Instead of typing "revdep-rebuild preserved-libs", you have to type
"emerge @preserved-libs", but neither of those solutions is automatic.

Also, if you feel that way, why not request that preserve-libs be made
mandatory in portage-2.2?  If the changes are big enough, and
not-well-tested enough that they warrant making the feature optional,
then why not ensure that the simpler fallback tools to correct problems
(like revdep-rebuild) can still do the job...

Mike 5:)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)

iEYEARECAAYFAkw/GgcACgkQu7rWomwgFXrY5QCeJha63SB9lpl1lLhgq9CqePj8
QsQAniLZpr0RymqtQlXAJVdoCa9eEEjW
=5a5g
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Re: Over using preserve_old_lib, don't do that

[Jeremy Olexa]

On Thu, 15 Jul 2010 15:24:08 +0100, Mike Auty <ikelos@gentoo.org>
wrote:
> <snip a bunch of stuff>
> If portage offers a way to turn off functionality (like preserving
> libraries), it should actually turn it off, rather than sometimes turn
> it off...

Yes, you are referring to
https://bugs.gentoo.org/show_bug.cgi?id=326275 I do believe this was
already mentioned in the thread. :)

On my build host, I have made preserve_old_lib() do nothing because I
expect my build host to always have the proper linking and it can
tolerate temp breakage as long as my client gets the proper packages in
the end.

-Jeremy

Re: [gentoo-dev] Over using preserve_old_lib, don't do that

[Markus Hauschild]

On Thu, Jul 15, 2010 at 10:49 AM, Mike Auty <ikelos@gentoo.org> wrote:
> Ideally, these calls should either adhere to FEATURES="-preserve-libs",
> or there should be a tool that can identify which files portage has
> preserved, and allow easy rebuilding of dependent packages, and removal.
>  At the moment, I'm having to manually grep ebuilds, ls the libraries
> and run revdep-rebuild over them one at a time...

Such a tool which can at least identify files that remain on the
system from preserved-libs or similar would be really useful in my
opinion!

[gentoo-dev] Re: Over using preserve_old_lib, don't do that

[Duncan]

Maciej Mrozowski posted on Thu, 15 Jul 2010 15:57:01 +0200 as excerpted:

> On Thursday 15 of July 2010 12:14:29 Duncan wrote:
>> 
>> If I have FEATURE=-preserve-libs, that's what I want.  Exceptions
>> should be limited to what will break the toolchain (including
>> revdep-rebuild here, since that's what's normally used to get out of
>> the situation) itself.
>> 
>> If there was a way to handle it so a general revdep-rebuild run would
>> still detect the preserved library as missing and do the necessary
>> rebuilds, it'd be one thing, but if the libraries are there, it figures
>> things are OK unless you've fed it that specific library, thereby
>> making a general revdep-rebuld run useless at the very task it was
>> designed to fix.
>> 
>> Talking about which...  What about creating an eutils (or whatever)
>> function to handle the critical preservations, having it build a
>> centralized list of them somewhere, and having a revdep-rebuild mode
>> that will treat that list as if it had been fed in with --library on
>> the command line?  Make revdep-rebuild able to run this mode either on
>> its own, or as part of an otherwise general run, and then you can have
>> packages (or the package-manager itself, if it uses the list as well)
>> preserve libs as they wish, without interfering with the ability of
>> revdep- rebuild to detect and resolve the issues in a normal run.
> 
> And what about using portage 2.2 and be done with it. I don't see the
> point in reinventing the wheel yet again.

As with Mike A, I've been running portage 2.2 for some time, and in fact, 
don't even have a world file any longer, as everything is in far more 
organized and easier to admin sets (thus bug #298298 on --depclean).

But preserved-libs caused me some serious problems and I feature-disabled 
it.  (If a package owns a file, it should do so consistently, everywhere 
it's installed, building the package should create it, and I should be 
able to dig its version as installed out of the binpkg tarball.)

> Imho, revdep-rebuild and all 'misc' tools requiring users' good will
> like python-updater should be obsolete and phased out in favour of
> package manager controlled mechanisms.

Ugh.  Automated can be useful, but don't get rid of the tools that allow 
one to do it step-by-step.  They're way too useful when something bugs out.

In my case, I always run emerge --update --deep --newuse when updating 
@system and @world, and always revdep-rebuild and --depclean afterward.  
While I do read the output portage spits out at the end of its emerges,  
having packages adopt earlier versions of their libraries and forcing me 
to do an additional library specific revdep-rebuild is forcing additional 
steps that the revdep-rebuild general run would normally find and fix 
entirely on its own -- only the adoption handling breaks that 
functionality.

For toolchain dependencies, that breakage is acceptable as it avoids worse 
alternatives.  But it shouldn't be happening for anything else (and I'd 
not make the exception Mike does for big-breakage libs, either, if the 
toolchain, including revdep-rebuild, works, it can fix it).

FWIW, since I'm running --as-needed in my ldflags and have lafilefixer 
post-install-hooked, I generally have far fewer rebuilds to do that those 
running without that.  Thus, the big breakages aren't nearly as big as 
they'd be otherwise, and with the exception of toolchain dependencies, 
that revdep-rebuild run at the end tends to be far less hassle than trying 
to cope with preserved-libs, and with breakage when libraries aren't in 
the binpkgs equery belongs says they're supposed to be in, etc.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Over using preserve_old_lib, don't do that

[Maciej Mrozowski]

On Thursday 15 of July 2010 16:24:08 Mike Auty wrote:
> On 15/07/10 14:57, Maciej Mrozowski wrote:
> > And what about using portage 2.2 and be done with it. I don't see the
> > point in reinventing the wheel yet again.
> 
> I'm using portage-2.2 and have been since it first came out.  I find the
> @set notation invaluable.

From my perspective sets notation is actually worthless and confusing (but 
that's another topic).
IIRC I've already spoke with portage team to think about making it emerge 
option instead or at least decouple it from sets semantics if possible (or it 
was about @live-rebuild? hmm). I think there are some preserved-related bugs 
(false positives) that need to be fixed first. Otherwise I'd welcome it being 
stable. I've already forgot what revdep-rebuild is thanks to portage-2.2

-- 
regards
MM

Re: [gentoo-dev] Over using preserve_old_lib, don't do that

[Jim Ramsay]

On Thu, Jul 15, 2010 at 07:28:40PM +0200, Markus Hauschild wrote:
> On Thu, Jul 15, 2010 at 10:49 AM, Mike Auty <ikelos@gentoo.org> wrote:
> > Ideally, these calls should either adhere to FEATURES="-preserve-libs",
> > or there should be a tool that can identify which files portage has
> > preserved, and allow easy rebuilding of dependent packages, and removal.
> >  At the moment, I'm having to manually grep ebuilds, ls the libraries
> > and run revdep-rebuild over them one at a time...
> 
> Such a tool which can at least identify files that remain on the
> system from preserved-libs or similar would be really useful in my
> opinion!

What if preserve_old_libs did the following:

  mv ${oldlib} ${oldlib}.preserved
  ln -s ${oldlib}.preserved ${oldlib}

Cons: 2 files to delete once you're done revdep-rebuilding

Pros: Easy to tell at a glance which libs are the preserved libs.

-- 
Jim Ramsay