[gentoo-dev] RANT: Upgrade icu and KDE at once

[gentoo-dev] RANT: Upgrade icu and KDE at once

[Jörg Schaible]

Well, here we go again! Again an update of Gentoo stable where emerge tries 
to upgrade icu and KDE in one run (and this time additionally libreoffice).

Other essential libraries (e.g. libpng) can always be updated, because the 
ebuild preserves the old shared libs and let me run revdep-rebuild against 
the old ones afterwards. Not so icu. The ebuild happily removes any trace of 
the old shared libs with the result that half of the stuff that is 
*required* to build kdelibs is now broken. The build aborts and leaves 
behind a broken system. And this happened now not for the first time!

The lengthly procedure is now to hard mask the new icu version, update your 
system (incl. automatic icu downgrade, kde and libreoffice upgrade), drop 
the mask and upgrade anything again, run revdep-rebuild and compile again a 
lot of big KDE packages (and this time inclding libreoffice).

The most annoying fact is, that none of this would have been necessary with 
portage 2.2, but maybe we have to wait for 2.1.11.500 before 2.2 gets 
stable...

- Jörg

Re: [gentoo-dev] RANT: Upgrade icu and KDE at once

[Rich Freeman]

On Tue, Apr 30, 2013 at 12:51 PM, Jörg Schaible <joerg.schaible@gmx.de> wrote:
> The most annoying fact is, that none of this would have been necessary with
> portage 2.2, but maybe we have to wait for 2.1.11.500 before 2.2 gets
> stable...

Actually, @preserved-rebuild is supported in the current stable
portage.  It just isn't the default or encouraged.  I'm running stable
portage and had no issues with the icu upgrade.  It even triggered
slot operator dependency rebuilds automatically for about a half-dozen
packages.

I did get the icu and KDE upgrades on separate days.

Rich

Re: [gentoo-dev] RANT: Upgrade icu and KDE at once

[Michael Mol]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

On 04/30/2013 01:06 PM, Rich Freeman wrote:
> On Tue, Apr 30, 2013 at 12:51 PM, Jörg Schaible <joerg.schaible@gmx.de> wrote:
>> The most annoying fact is, that none of this would have been necessary with
>> portage 2.2, but maybe we have to wait for 2.1.11.500 before 2.2 gets
>> stable...
> 
> Actually, @preserved-rebuild is supported in the current stable
> portage.  It just isn't the default or encouraged.  I'm running stable
> portage and had no issues with the icu upgrade.  It even triggered
> slot operator dependency rebuilds automatically for about a half-dozen
> packages.
> 
> I did get the icu and KDE upgrades on separate days.

Indeed, the libicu update this time around went much, much cleaner than
any previous time. (At least, it did for me.)

I'm not running full KDE, though. The biggest quirk I hit, I described
in bug #468022


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 555 bytes --]

Re: [gentoo-dev] RANT: Upgrade icu and KDE at once

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 2089 bytes --]

On Tue, 30 Apr 2013 18:51:06 +0200
Jörg Schaible <joerg.schaible@gmx.de> wrote:

> Well, here we go again! Again an update of Gentoo stable where emerge
> tries to upgrade icu and KDE in one run (and this time additionally
> libreoffice).

If you don't want that to happen, use package sets and exclusion.

> Other essential libraries (e.g. libpng) can always be updated,
> because the ebuild preserves the old shared libs and let me run
> revdep-rebuild against the old ones afterwards.

I haven't ran revdep-rebuild for a year, you can set
FEATURES="preserve-libs" which will preserve any libs, once libs are
being preserved you can then get rid of them by doing an `emerge
@preserved-rebuild` whenever you feel like as opposed to when
something breaks. It's not a default yet, but will probably be so...

> The ebuild happily removes any trace of the old shared libs with the
> result that half of the stuff that is *required* to build kdelibs is
> now broken. The build aborts and leaves behind a broken system. And
> this happened now not for the first time!

Packages such as kdelibs are huge for no good reason, either look for
lighter alternatives or don't upgrade them often if they bother you.

> The lengthly procedure is now to hard mask the new icu version,
> update your system (incl. automatic icu downgrade, kde and
> libreoffice upgrade)

Why? You shouldn't need to do the downgrade here.

> Drop the mask and upgrade anything again, run
> revdep-rebuild and compile again a lot of big KDE packages (and this
> time inclding libreoffice).

This is why I avoid huge packages where I can.

> The most annoying fact is, that none of this would have been
> necessary with portage 2.2, but maybe we have to wait for 2.1.11.500
> before 2.2 gets stable...

Features and fixes are backported, don't wait for that stabilization.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] RANT: Upgrade icu and KDE at once

[Rich Freeman]

On Tue, Apr 30, 2013 at 1:24 PM, Tom Wijsman <TomWij@gentoo.org> wrote:
> I haven't ran revdep-rebuild for a year, you can set
> FEATURES="preserve-libs" which will preserve any libs, once libs are
> being preserved you can then get rid of them by doing an `emerge
> @preserved-rebuild` whenever you feel like as opposed to when
> something breaks. It's not a default yet, but will probably be so...

I believe there have been concerns with @preserved-rebuild sometimes
causing problems.

I think there is probably still a place for it, even in a world where
everybody uses slot operator dependencies.  Upgrading a large
collection of packages takes time, and it might even fail midway.  It
would still be nice to have the system maintain the old libraries
until the entire process is completed.

I'm sure there are one-offs that I'm not thinking of, but I think
we're better off finding a solution for those one-offs than giving up
the benefits of deferred library removal.  For security items an
update to glsa-check to find vulnerable lingering remains of removed
packages might help address concerns.

Rich

Re: [gentoo-dev] RANT: Upgrade icu and KDE at once

[Zac Medico]

On Tue, Apr 30, 2013 at 9:51 AM, Jörg Schaible <joerg.schaible@gmx.de> wrote:
> The most annoying fact is, that none of this would have been necessary with
> portage 2.2, but maybe we have to wait for 2.1.11.500 before 2.2 gets
> stable...

Since portage-2.1.11.20 [1], you can do this:

   echo 'FEATURES="${FEATURES} preserve-libs"' >> /etc/portage/make.conf

[1] http://blogs.gentoo.org/zmedico/2012/09/21/preserve-libs-available-in-portage-2-1/

[gentoo-dev] Re: RANT: Upgrade icu and KDE at once

[Jörg Schaible]

Hi Zac,

Zac Medico wrote:

> On Tue, Apr 30, 2013 at 9:51 AM, Jörg Schaible <joerg.schaible@gmx.de>
> wrote:
>> The most annoying fact is, that none of this would have been necessary
>> with portage 2.2, but maybe we have to wait for 2.1.11.500 before 2.2
>> gets stable...
> 
> Since portage-2.1.11.20 [1], you can do this:
> 
>    echo 'FEATURES="${FEATURES} preserve-libs"' >> /etc/portage/make.conf
> 
> [1]
> [http://blogs.gentoo.org/zmedico/2012/09/21/preserve-libs-available-in-portage-2-1/

That announcement slipped somehow my awareness. Indeed an upgrade of a 
different machine with preserve-libs added to FEATURES went fine. Still, I 
wonder what prevents portage-2.2 form going stable, I have one machine where 
I use that one for years without any flaws and a lot of benefits.

- Jörg

Re: [gentoo-dev] RANT: Upgrade icu and KDE at once

[Peter Stuge]

Jörg Schaible wrote:
> icu. The ebuild happily removes any trace of the old shared libs
> with the result that half of the stuff that is *required* to build
> kdelibs is now broken. The build aborts and leaves behind a broken
> system. And this happened now not for the first time!

Tom Wijsman wrote:
> > Well, here we go again! Again an update of Gentoo stable where emerge
> > tries to upgrade icu and KDE in one run (and this time additionally
> > libreoffice).
> 
> If you don't want that to happen, use package sets and exclusion.

Wait - using default settings results in emerge leaving a broken
system, and your response is "take non-default actions to avoid that" ?


//Peter

Re: [gentoo-dev] RANT: Upgrade icu and KDE at once

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 832 bytes --]

On Wed, 1 May 2013 11:20:37 +0200
Peter Stuge <peter@stuge.se> wrote:

> Tom Wijsman wrote:
> > > Well, here we go again! Again an update of Gentoo stable where
> > > emerge tries to upgrade icu and KDE in one run (and this time
> > > additionally libreoffice).
> > 
> > If you don't want that to happen, use package sets and exclusion.
> 
> Wait - using default settings results in emerge leaving a broken
> system, and your response is "take non-default actions to avoid
> that" ?

"in one run", I see no mention of "broken" in what I quoted here;
don't pull it out of its context, it was just a suggestion.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] RANT: Upgrade icu and KDE at once

[Peter Stuge]

Peter Stuge wrote:
> Jörg Schaible wrote:
> > icu. The ebuild happily removes any trace of the old shared libs
> > with the result that half of the stuff that is *required* to build
> > kdelibs is now broken. The build aborts and leaves behind a broken
> > system. And this happened now not for the first time!

Tom Wijsman wrote:
> > > If you don't want that to happen, use package sets and exclusion.
> > 
> > Wait - using default settings results in emerge leaving a broken
> > system, and your response is "take non-default actions to avoid
> > that" ?
> 
> "in one run", I see no mention of "broken" in what I quoted here;
> don't pull it out of its context, it was just a suggestion.

The way I understood Jörg was that the greater context is that files
needed in later steps of an emerge were removed by earlier steps.

Even if this doesn't happen very regularly I think it's a problem
that it can happen at all. I guess the solution is known, but not
there yet?


//Peter

Re: [gentoo-dev] RANT: Upgrade icu and KDE at once

[Mike Gilbert]

On Tue, Apr 30, 2013 at 2:53 PM, Zac Medico <zmedico@gentoo.org> wrote:
> On Tue, Apr 30, 2013 at 9:51 AM, Jörg Schaible <joerg.schaible@gmx.de> wrote:
>> The most annoying fact is, that none of this would have been necessary with
>> portage 2.2, but maybe we have to wait for 2.1.11.500 before 2.2 gets
>> stable...
>
> Since portage-2.1.11.20 [1], you can do this:
>
>    echo 'FEATURES="${FEATURES} preserve-libs"' >> /etc/portage/make.conf
>
> [1] http://blogs.gentoo.org/zmedico/2012/09/21/preserve-libs-available-in-portage-2-1/
>

I think it is time to consider enabling this by default. Hopefully any
ABI bumps will be accompanied by a subslot / slot-operator migration
at this point.

Re: [gentoo-dev] Re: RANT: Upgrade icu and KDE at once

[Zac Medico]

On 05/01/2013 02:07 AM, Jörg Schaible wrote:
> Hi Zac,
> 
> Zac Medico wrote:
> 
>> On Tue, Apr 30, 2013 at 9:51 AM, Jörg Schaible <joerg.schaible@gmx.de>
>> wrote:
>>> The most annoying fact is, that none of this would have been necessary
>>> with portage 2.2, but maybe we have to wait for 2.1.11.500 before 2.2
>>> gets stable...
>>
>> Since portage-2.1.11.20 [1], you can do this:
>>
>>    echo 'FEATURES="${FEATURES} preserve-libs"' >> /etc/portage/make.conf
>>
>> [1]
>> [http://blogs.gentoo.org/zmedico/2012/09/21/preserve-libs-available-in-portage-2-1/
> 
> That announcement slipped somehow my awareness. Indeed an upgrade of a 
> different machine with preserve-libs added to FEATURES went fine. Still, I 
> wonder what prevents portage-2.2 form going stable, I have one machine where 
> I use that one for years without any flaws and a lot of benefits.

I think it's more useful to talk about specific features and their
readiness to be enabled by default in stable, rather then when
"everything in portage-2.2" should go stable. Which features in
portage-2.2 are you using that are ready for stable?

Not that the difference between portage-2.1 and portage-2.2 is just the
constants that you can see in this commit:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=92ce3fcbf2c6d791151afc6edbbb18a530db12e2
-- 
Thanks,
Zac

Re: [gentoo-dev] RANT: Upgrade icu and KDE at once

[Zac Medico]

On 05/01/2013 07:46 AM, Mike Gilbert wrote:
> On Tue, Apr 30, 2013 at 2:53 PM, Zac Medico <zmedico@gentoo.org> wrote:
>> On Tue, Apr 30, 2013 at 9:51 AM, Jörg Schaible <joerg.schaible@gmx.de> wrote:
>>> The most annoying fact is, that none of this would have been necessary with
>>> portage 2.2, but maybe we have to wait for 2.1.11.500 before 2.2 gets
>>> stable...
>>
>> Since portage-2.1.11.20 [1], you can do this:
>>
>>    echo 'FEATURES="${FEATURES} preserve-libs"' >> /etc/portage/make.conf
>>
>> [1] http://blogs.gentoo.org/zmedico/2012/09/21/preserve-libs-available-in-portage-2-1/
>>
> 
> I think it is time to consider enabling this by default. Hopefully any
> ABI bumps will be accompanied by a subslot / slot-operator migration
> at this point.

Yeah, I'm pretty happy with the slot-operator adoption, so it feels like
it's about time to enable preserve-libs by default in stable. I know
that this feature has been questioned by some, especially by people
involved with Paludis (which doesn't implement preserve-libs). Maybe it
would be a good idea to get an opinion from the council on whether or
not it should be enabled by default in stable portage.
-- 
Thanks,
Zac

Re: [gentoo-dev] RANT: Upgrade icu and KDE at once

[Mike Gilbert]

On Wed, May 1, 2013 at 11:01 AM, Zac Medico <zmedico@gentoo.org> wrote:
> On 05/01/2013 07:46 AM, Mike Gilbert wrote:
>> On Tue, Apr 30, 2013 at 2:53 PM, Zac Medico <zmedico@gentoo.org> wrote:
>>> On Tue, Apr 30, 2013 at 9:51 AM, Jörg Schaible <joerg.schaible@gmx.de> wrote:
>>>> The most annoying fact is, that none of this would have been necessary with
>>>> portage 2.2, but maybe we have to wait for 2.1.11.500 before 2.2 gets
>>>> stable...
>>>
>>> Since portage-2.1.11.20 [1], you can do this:
>>>
>>>    echo 'FEATURES="${FEATURES} preserve-libs"' >> /etc/portage/make.conf
>>>
>>> [1] http://blogs.gentoo.org/zmedico/2012/09/21/preserve-libs-available-in-portage-2-1/
>>>
>>
>> I think it is time to consider enabling this by default. Hopefully any
>> ABI bumps will be accompanied by a subslot / slot-operator migration
>> at this point.
>
> Yeah, I'm pretty happy with the slot-operator adoption, so it feels like
> it's about time to enable preserve-libs by default in stable. I know
> that this feature has been questioned by some, especially by people
> involved with Paludis (which doesn't implement preserve-libs). Maybe it
> would be a good idea to get an opinion from the council on whether or
> not it should be enabled by default in stable portage.

I have requested that this be added to the agenda for the next council
meeting in a couple of weeks.

[gentoo-dev] Re: RANT: Upgrade icu and KDE at once

[Duncan]

Zac Medico posted on Wed, 01 May 2013 08:01:45 -0700 as excerpted:

> On 05/01/2013 07:46 AM, Mike Gilbert wrote:

>> I think it is time to consider enabling [preserve-libs] by default.
>> Hopefully any ABI bumps will be accompanied by a
>> subslot / slot-operator migration at this point.
> 
> Yeah, I'm pretty happy with the slot-operator adoption, so it feels like
> it's about time to enable preserve-libs by default in stable. I know
> that this feature has been questioned by some, especially by people
> involved with Paludis (which doesn't implement preserve-libs). Maybe it
> would be a good idea to get an opinion from the council on whether or
> not it should be enabled by default in stable portage.

FWIW I've been running 2.2 (and won't touch paludis with a 3 metre pole) 
for some time, but turned preserved-libs off here because it simply 
complicates things for me.  After some early issues with "too much magic" 
re preserved-libs (possibly long since fixed but I wouldn't know as I 
have the feature turned off), I originally would rather let the upgrades 
happen as they always did and simply run revdep-rebuild afterward, and 
preserved-libs interfered with that as the libs were still there so 
revdep-rebuild didn't find anything to rebuild.

Of course with sub-slots "doing it the 'correct' way" revdep-rebuild 
isn't finding so much to rebuild anymore, anyway, so like most people I 
think, I'm a big subslots fan, but that doesn't mean I trust preserved-
libs any more than I did.

But I've no objection to preserved-libs becoming the default and while 
it's not for me personally, I'm cautiously in favor of the idea as a 
default, as long as it remains a togglable feature.

While I /am/ cautiously in favor, I definitely believe running it by 
council is a good idea, as it should help put to bed any remaining 
controversy over the idea.  Neither the formal "speak now or forever hold 
your peace" aspect nor the CYA and "it's voted and settled now, don't 
reopen the topic unless there's GOOD reason" a council blessing provides 
can be a bad thing. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: RANT: Upgrade icu and KDE at once

[Tom Wijsman]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

On Thu, 2 May 2013 03:38:24 +0000 (UTC)
Duncan <1i5t5.duncan@cox.net> wrote:

> After some early issues with "too much magic" re preserved-libs

Why is it magic? It is well explained what it does (eg. man make.conf).

> I originally would rather let the upgrades happen as
> they always did and simply run revdep-rebuild afterward

You know that if you enable preserve-libs that you have to instead run
`emerge @preserved-rebuild`, which has a much shorter runtime.

> and preserved-libs interfered with that as the libs were still there
> so revdep-rebuild didn't find anything to rebuild.

`emerge @preserved-rebuild` will find and rebuild them.

-- 
With kind regards,

Tom Wijsman (TomWij)
Gentoo Developer

E-mail address  : TomWij@gentoo.org
GPG Public Key  : 6D34E57D
GPG Fingerprint : C165 AF18 AB4C 400B C3D2  ABF0 95B2 1FCD 6D34 E57D

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[gentoo-dev] Re: RANT: Upgrade icu and KDE at once

[Duncan]

Tom Wijsman posted on Thu, 02 May 2013 07:09:10 +0200 as excerpted:

> Duncan <1i5t5.duncan@cox.net> wrote:
> 
>> After some early issues with "too much magic" re preserved-libs
> 
> Why is it magic? It is well explained what it does (eg. man make.conf).
> 
>> I originally would rather let the upgrades happen as they always did
>> and simply run revdep-rebuild afterward
> 
> You know that if you enable preserve-libs that you have to instead run
> `emerge @preserved-rebuild`, which has a much shorter runtime.
> 
>> and preserved-libs interfered with that as the libs were still there so
>> revdep-rebuild didn't find anything to rebuild.
> 
> `emerge @preserved-rebuild` will find and rebuild them.

To those who say I'm too verbose, this is what happens when I try to 
shortcut things... I invariably end up spelling it all out anyway in a 
followup, when someone doesn't understand the shortcut and needs the full 
explanation...  It happens frequently enough that I had learned to avoid 
it by covering what I could the first time around.  Now with some 
friendly urging, I'm trying to unlearn that, but... Just sayin'...

Yes.  I understand all about @preserved-libs and was in fact an early 
feature tester... which is actually likely part of the problem.

The problem, which as I said earlier I expect has long since been fixed, 
was in the "magic" bit of old libraries being temporarily reassigned as 
owned by the newer versions that replaced them, even tho they weren't 
actually part of that version of the package at all, but a previous 
version.  What happened was that the emerge @preserved-libs failed for 
some reason, leaving some of these stale libs that had been artificially 
reassigned as owned by the new version, now owned by no one and 
untracked.  But because they were still there, revdep-rebuild wouldn't 
detect the problem, and because they were now orphaned, I couldn't find 
them by looking at the files owned by the new versions, so I was a bit 
stuck.  Fortunately in that instance I had only updated a few packages 
that I had to go manually comparing old binpkg contents with new binpkg 
contents to track down the orphan libs to delete manually, after which 
revdep-rebuild worked as it should, but I decided that was "too much 
magic" to rely on in the future, when the old remove it and let revdep-
rebuild detect and handle the resulting rebuilds was a well proven method 
that "just worked".

"Magic" in this case being defined simply as too much hassle for me to 
figure out what it did manually and fix things up when it screwed up, 
when there was a well tested tool that might be a bit slower, but that 
uses a method known to be /very/ reliable at finding and fixing the sort 
of library-dependency issues it dealt with.

Meanwhile, revdep-rebuild isn't /that/ slow.  As I found out by 
experience, it's MUCH faster than rooting out problems manually when 
@preserved-libs fails its "magic" for whatever reason.

Actually, most of the wait on revdep-rebuild on conventional/legacy 
"spinning rust" systems is due to the I/O latency of actually reading in 
all the files it scans.  Which is quite convenient, since it only needs 
run after a normally heavily CPU bound world update... such that an I/O 
bound process conveniently runnable at the same time won't significantly 
slow down either one! =:^)  For those with sufficient memory (@16 gigs I 
rarely zero-out free memory and dump cache), simply start the emerge 
update, then in another VT or terminal window, start an initial revdep-
rebuild --pretend.

By the time the update is done, the revdep-rebuild has generally finished 
as well, so all the files it scans are in cache.  And with a hot cache, 
revdep-rebuild runs **MUCH** faster the second time around, after the 
update's done so any needed rebuilds can actually be detected and the 
rebuild done. =:^)

I generally run the emerge --jobs --update..., then run the revdep-
rebuild in parallel, since the first is generally CPU bound and the 
second disk bound.  Then when the update is done, I run revdep-rebuild 
for-real this time, while running etc-update and emerge --ask --depclean 
(serially) in the other terminal.  By the time the etc-update and depclean 
are done, the (hot-cached) revdep-rebuild is generally done too (thanks 
to being run routinely so there's never a backlog, and thanks to lddflags 
as-needed and now subslots as well, taking away most of the work revdep-
rebuild used to need to do), so it really didn't take me much more time 
at all, since otherwise I'd have been waiting first on the update, and 
then would have been managing the etc-update and waiting on the depclean.


Meanwhile, I've finally decided it's time to leave the legacy spinning-
rust world behind (for the frequently accessed stuff like the OS, much 
of /home, and the tree, anyway) and should be upgrading to SSDs soon.  
With luck, that'll solve much of the current spinning rust latency 
bottlenecks, leaving me an entirely new set of bottlenecks to learn to to 
deal with and attempt to optimize around...  (I've tried btrfs already, I 
think I'll try f2fs on the SSDs, while the spinning rust I'll be copying 
over from is still current enough to serve as a good backup in case 
anything goes wrong...)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman