[gentoo-dev] Is removing old EAPIs worth the churn?

[gentoo-dev] Is removing old EAPIs worth the churn?

[Matt Turner]

EAPI 2 removal bug: https://bugs.gentoo.org/648050

It seems like tons of churn to update old stable ebuilds to a new
EAPI, just for its own sake. Take https://bugs.gentoo.org/648154 for
example. New ebuild added with EAPI 6 bumped from EAPI 2. Otherwise
functionally identical. Now asking arch teams to retest and
restabilize. Multiply by 100 or more.

In the end we might get to delete some code from portage or an eclass?
Does this seem worth it?

Re: [gentoo-dev] Is removing old EAPIs worth the churn?

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1431 bytes --]

On Mon, 5 Mar 2018 17:52:54 -0800
Matt Turner <mattst88@gentoo.org> wrote:

> EAPI 2 removal bug: https://bugs.gentoo.org/648050
> 
> It seems like tons of churn to update old stable ebuilds to a new
> EAPI, just for its own sake. Take https://bugs.gentoo.org/648154 for
> example. New ebuild added with EAPI 6 bumped from EAPI 2. Otherwise
> functionally identical. Now asking arch teams to retest and
> restabilize. Multiply by 100 or more.
> 
> In the end we might get to delete some code from portage or an eclass?
> Does this seem worth it?
> 

New EAPI's don't just do nothing, some for example, add more power to
users.

EAPI6 is an especially significant example due to eapply_user becoming
standardized.

And with Perl packages at least, incrementing EAPI results in actual
changes driven by the eclass:

- Changes the names of various control variables
- Makes perl tests on by default, and parallel by default ( previously
  required opt-in )
- Disables the legacy code path that kills the '.packlist' files, which
  are actually useful to some tools, and was the wrong place to kill
  those files in the first place.

Incrementing EAPI also functions as an indicator that legacy approaches
and interfaces are moved away from, which can also signify an
improvement in ebuild quality.

In short, while it looks superficially useless, I'd argue that there's
a lot of nuanced benefits.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Is removing old EAPIs worth the churn?

[Dirkjan Ochtman]

[-- Attachment #1: Type: text/plain, Size: 950 bytes --]

On Tue, Mar 6, 2018 at 2:52 AM, Matt Turner <mattst88@gentoo.org> wrote:

> In the end we might get to delete some code from portage or an eclass?
> Does this seem worth it?
>

To add to some of the points Kent made, I think so, yes: this means freeing
us from the cognitive overhead of thinking about older EAPIs at all, having
to remember the differences between those old EAPIs and the newer ones.
Removing that complexity from developer's minds as well as from our tooling
means our tooling will be we can spend our time and energy on other things,
which will be a win for the distribution.

Yes, the long tail is painful: these are probably the packages that haven't
been upgraded before for a reason, but actually putting the finishing
touches on the EAPI migration will be worth it in the end.

It might have been even better if Andreas or someone else had announced
this project/discussed the trade-offs before starting it.

Cheers,

Dirkjan

[-- Attachment #2: Type: text/html, Size: 1372 bytes --]

Re: [gentoo-dev] Is removing old EAPIs worth the churn?

[Michał Górny]

W dniu pon, 05.03.2018 o godzinie 17∶52 -0800, użytkownik Matt Turner
napisał:
> EAPI 2 removal bug: https://bugs.gentoo.org/648050
> 
> It seems like tons of churn to update old stable ebuilds to a new
> EAPI, just for its own sake. Take https://bugs.gentoo.org/648154 for
> example. New ebuild added with EAPI 6 bumped from EAPI 2. Otherwise
> functionally identical. Now asking arch teams to retest and
> restabilize. Multiply by 100 or more.
> 
> In the end we might get to delete some code from portage or an eclass?
> Does this seem worth it?

Removing any code from package managers is unlikely, so that is not
an argument.

Removing code from eclasses is actually beneficial. Please remember that
EAPI changes are not isolated to the PMS, and eclasses also use them to
kill deprecated features and make other breaking changes.

Both PMS and eclasses add new features in new EAPIs. Others have already
bought some examples. To examples others have already mentioned you
could add new econf options [1]. As a result, bumping EAPI frequently
*improves* the quality of the package and sometimes fixes bugs (reported
or unreported).

Most importantly, killing old EAPIs makes Gentoo easier for
contributors. You may think it's normal to expect Gentoo developers
to know 7 EAPIs and the differences between them. It might be normal for
people who are around long enough to see most of them.
But e.g. in proxy-maint we don't want to teach people about 7 EAPIs
if only 1 or 2 of them are really relevant to what they're doing.

Finally, old EAPIs are simply more prone to mistakes. When the majority
of 'current' Gentoo packages, i.e. the packages Gentoo developers
usually touch, use new EAPIs, it is *really easy* to miss that some old
package is using an ancient EAPI and start scratching your head why
something does not work.

Killing old EAPIs proactively -- like many other proactive efforts --
has the advantage of making the work of others easier. If I end up
having to fix a dozen old packages because of some reverse dependency,
I'd really find it preferable that someone already did the EAPI bump for
me and I wouldn't have to focus at the same time on fixing some issue,
bumping EAPI, fixing tests and doing all the other long-overdue
maintenance tasks.

[1]:https://dev.gentoo.org/~ulm/pms/head/pms.html#x1-13700012.3.8

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] Is removing old EAPIs worth the churn?

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 1836 bytes --]

Am Dienstag, 6. März 2018, 02:52:54 CET schrieb Matt Turner:
> EAPI 2 removal bug: https://bugs.gentoo.org/648050
> 
> It seems like tons of churn to update old stable ebuilds to a new
> EAPI, just for its own sake. Take https://bugs.gentoo.org/648154 for
> example. New ebuild added with EAPI 6 bumped from EAPI 2. Otherwise
> functionally identical. Now asking arch teams to retest and
> restabilize. Multiply by 100 or more.

OK so here's my personal opinion: 

Is it worth the effort? Yes, see below.
Is it a high priority task? No.

Is it really that much effort? Well, we're even in the case of EAPI=2 talking 
about only 400 ebuilds of 35000 in total. That's roughly 1% of the tree. And 
I'd strongly suspect that even without the EAPI update it would make very much 
sense to check these 400 old ebuilds and test whether they still work as 
intended. 

What do we gain? 

* Mainly, less stuff to memorize. I'll be throwing a party on the day when the 
last EAPI=0 ebuild is gone. (In the retirement home, probably.)

* Also, it's not just having a bigger number, but also useful features...

Why now EAPI=2? 

* EAPI=3 is nearly gone (27 ebuilds left, scheme & java please get a move! :)
* EAPI=2 is the one with the next-least ebuilds.

While it would be very nice to remove EAPI=0, let's go for easier targets 
first; the number of EAPI=0 ebuilds will decrease organically in the meantime.

[Interestingly, as long as no specific efforts are made, the number of ebuilds 
in all deprecated EAPI decreases roughly equally and exponentially. That means
the probability of any old ebuild to be removed within a certain time interval 
is a constant as function of time...]

-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, perl, libreoffice, comrel)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] Is removing old EAPIs worth the churn?

[Rich Freeman]

On Tue, Mar 6, 2018 at 4:17 PM, Andreas K. Huettel <dilfridge@gentoo.org> wrote:
>
> Is it worth the effort? Yes, see below.
> Is it a high priority task? No.
>

It sounds like all that has been done is to log a tracker and create
some bugs.  That is hardly a major burden on anybody.  If it nudges
people to bump the EAPI when they're doing other work so much the
better, but there doesn't seem to be a drop-dead date yet.

If devs don't want to think about EAPI cleanup they don't have to right now.

The delta as you get to more recent EAPIs hasn't been quite as extreme
as in the early ones, so this should hopefully become simpler as we go
along...

-- 
Rich

Re: [gentoo-dev] Is removing old EAPIs worth the churn?

[Matt Turner]

On Mon, Mar 5, 2018 at 9:56 PM, Kent Fredric <kentnl@gentoo.org> wrote:
> On Mon, 5 Mar 2018 17:52:54 -0800
> And with Perl packages at least, incrementing EAPI results in actual
> changes driven by the eclass:
>
> - Changes the names of various control variables
> - Makes perl tests on by default, and parallel by default ( previously
>   required opt-in )
> - Disables the legacy code path that kills the '.packlist' files, which
>   are actually useful to some tools, and was the wrong place to kill
>   those files in the first place.

Sounds like you're not bumping EAPI for its own sake.

Re: [gentoo-dev] Is removing old EAPIs worth the churn?

[Matt Turner]

On Tue, Mar 6, 2018 at 1:00 AM, Dirkjan Ochtman <djc@gentoo.org> wrote:
> On Tue, Mar 6, 2018 at 2:52 AM, Matt Turner <mattst88@gentoo.org> wrote:
>>
>> In the end we might get to delete some code from portage or an eclass?
>> Does this seem worth it?
>
>
> To add to some of the points Kent made, I think so, yes: this means freeing
> us from the cognitive overhead of thinking about older EAPIs at all, having
> to remember the differences between those old EAPIs and the newer ones.
> Removing that complexity from developer's minds as well as from our tooling
> means our tooling will be we can spend our time and energy on other things,
> which will be a win for the distribution.
>
> Yes, the long tail is painful: these are probably the packages that haven't
> been upgraded before for a reason, but actually putting the finishing
> touches on the EAPI migration will be worth it in the end.

The long tail of packages were packages that by definition didn't
require any maintenance or add to our collective cognitive overhead.
:)

Re: [gentoo-dev] Is removing old EAPIs worth the churn?

[Matt Turner]

On Tue, Mar 6, 2018 at 1:17 PM, Andreas K. Huettel <dilfridge@gentoo.org> wrote:
> Am Dienstag, 6. März 2018, 02:52:54 CET schrieb Matt Turner:
>> EAPI 2 removal bug: https://bugs.gentoo.org/648050
>>
>> It seems like tons of churn to update old stable ebuilds to a new
>> EAPI, just for its own sake. Take https://bugs.gentoo.org/648154 for
>> example. New ebuild added with EAPI 6 bumped from EAPI 2. Otherwise
>> functionally identical. Now asking arch teams to retest and
>> restabilize. Multiply by 100 or more.
>
> OK so here's my personal opinion:
>
> Is it worth the effort? Yes, see below.
> Is it a high priority task? No.
>
> Is it really that much effort? Well, we're even in the case of EAPI=2 talking
> about only 400 ebuilds of 35000 in total. That's roughly 1% of the tree. And
> I'd strongly suspect that even without the EAPI update it would make very much
> sense to check these 400 old ebuilds and test whether they still work as
> intended.

There are plenty of reported bugs to look at. No need to go searching
for more :)

> What do we gain?
>
> * Mainly, less stuff to memorize. I'll be throwing a party on the day when the
> last EAPI=0 ebuild is gone. (In the retirement home, probably.)

This is the argument made by others about the cognitive overhead of
remembering all the EAPI differences. If the packages are untouched
for ages and don't require maintenance, my claim is that there is no
cognitive load to begin with.

> * Also, it's not just having a bigger number, but also useful features...
>
> Why now EAPI=2?
>
> * EAPI=3 is nearly gone (27 ebuilds left, scheme & java please get a move! :)
> * EAPI=2 is the one with the next-least ebuilds.
>
> While it would be very nice to remove EAPI=0, let's go for easier targets
> first; the number of EAPI=0 ebuilds will decrease organically in the meantime.
>
> [Interestingly, as long as no specific efforts are made, the number of ebuilds
> in all deprecated EAPI decreases roughly equally and exponentially. That means
> the probability of any old ebuild to be removed within a certain time interval
> is a constant as function of time...]

This is a great point in favor of *not* bothering to proactively bump EAPI.

Re: [gentoo-dev] Is removing old EAPIs worth the churn?

[Matt Turner]

On Tue, Mar 6, 2018 at 1:35 PM, Rich Freeman <rich0@gentoo.org> wrote:
> On Tue, Mar 6, 2018 at 4:17 PM, Andreas K. Huettel <dilfridge@gentoo.org> wrote:
>>
>> Is it worth the effort? Yes, see below.
>> Is it a high priority task? No.
>>
>
> It sounds like all that has been done is to log a tracker and create
> some bugs.  That is hardly a major burden on anybody.  If it nudges
> people to bump the EAPI when they're doing other work so much the
> better, but there doesn't seem to be a drop-dead date yet.
>
> If devs don't want to think about EAPI cleanup they don't have to right now.

No, not true. Look at the blocking bugs. We're asking arch teams to
retest and restabilize ebuilds whose only difference is the EAPI bump.

Re: [gentoo-dev] Is removing old EAPIs worth the churn?

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 1117 bytes --]

On Tue, Mar 6, 2018 at 5:22 PM, Matt Turner <mattst88@gentoo.org> wrote:

> On Tue, Mar 6, 2018 at 1:35 PM, Rich Freeman <rich0@gentoo.org> wrote:
> > On Tue, Mar 6, 2018 at 4:17 PM, Andreas K. Huettel <dilfridge@gentoo.org>
> wrote:
> >>
> >> Is it worth the effort? Yes, see below.
> >> Is it a high priority task? No.
> >>
> >
> > It sounds like all that has been done is to log a tracker and create
> > some bugs.  That is hardly a major burden on anybody.  If it nudges
> > people to bump the EAPI when they're doing other work so much the
> > better, but there doesn't seem to be a drop-dead date yet.
> >
> > If devs don't want to think about EAPI cleanup they don't have to right
> now.
>
> No, not true. Look at the blocking bugs. We're asking arch teams to
> retest and restabilize ebuilds whose only difference is the EAPI bump.
>
>
Ultimate the arch teams are supposed to test the ebuild (that it works), so
when we change the EAPI of the ebuild re-testing is required.
If we only cared about the ebuild (that it worked or not) we could use
automation to cut down on the humans involved in this work.

-A

[-- Attachment #2: Type: text/html, Size: 1807 bytes --]

Re: [gentoo-dev] Is removing old EAPIs worth the churn?

[Matt Turner]

On Tue, Mar 6, 2018 at 2:27 PM, Alec Warner <antarus@gentoo.org> wrote:
> On Tue, Mar 6, 2018 at 5:22 PM, Matt Turner <mattst88@gentoo.org> wrote:
>> On Tue, Mar 6, 2018 at 1:35 PM, Rich Freeman <rich0@gentoo.org> wrote:
>> > On Tue, Mar 6, 2018 at 4:17 PM, Andreas K. Huettel
>> > <dilfridge@gentoo.org> wrote:
>> >>
>> >> Is it worth the effort? Yes, see below.
>> >> Is it a high priority task? No.
>> >>
>> >
>> > It sounds like all that has been done is to log a tracker and create
>> > some bugs.  That is hardly a major burden on anybody.  If it nudges
>> > people to bump the EAPI when they're doing other work so much the
>> > better, but there doesn't seem to be a drop-dead date yet.
>> >
>> > If devs don't want to think about EAPI cleanup they don't have to right
>> > now.
>>
>> No, not true. Look at the blocking bugs. We're asking arch teams to
>> retest and restabilize ebuilds whose only difference is the EAPI bump.
>>
>
> Ultimate the arch teams are supposed to test the ebuild (that it works), so
> when we change the EAPI of the ebuild re-testing is required.

Of course, but that's not the point...

Re: [gentoo-dev] Is removing old EAPIs worth the churn?

[Rich Freeman]

On Tue, Mar 6, 2018 at 6:39 PM, Matt Turner <mattst88@gentoo.org> wrote:
> On Tue, Mar 6, 2018 at 2:27 PM, Alec Warner <antarus@gentoo.org> wrote:
>> On Tue, Mar 6, 2018 at 5:22 PM, Matt Turner <mattst88@gentoo.org> wrote:
>>> On Tue, Mar 6, 2018 at 1:35 PM, Rich Freeman <rich0@gentoo.org> wrote:
>>> > On Tue, Mar 6, 2018 at 4:17 PM, Andreas K. Huettel
>>> > <dilfridge@gentoo.org> wrote:
>>> >>
>>> >> Is it worth the effort? Yes, see below.
>>> >> Is it a high priority task? No.
>>> >>
>>> >
>>> > It sounds like all that has been done is to log a tracker and create
>>> > some bugs.  That is hardly a major burden on anybody.  If it nudges
>>> > people to bump the EAPI when they're doing other work so much the
>>> > better, but there doesn't seem to be a drop-dead date yet.
>>> >
>>> > If devs don't want to think about EAPI cleanup they don't have to right
>>> > now.
>>>
>>> No, not true. Look at the blocking bugs. We're asking arch teams to
>>> retest and restabilize ebuilds whose only difference is the EAPI bump.
>>>
>>
>> Ultimate the arch teams are supposed to test the ebuild (that it works), so
>> when we change the EAPI of the ebuild re-testing is required.
>
> Of course, but that's not the point...
>

Strictly speaking nobody is forcing the arch teams to test any of
these bumps either.  They are as free to choose to work on those bugs
as anybody else.

-- 
Rich

Re: [gentoo-dev] Is removing old EAPIs worth the churn?

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 1991 bytes --]

> > 
> > * Mainly, less stuff to memorize. I'll be throwing a party on the day when
> > the last EAPI=0 ebuild is gone. (In the retirement home, probably.)
> 
> This is the argument made by others about the cognitive overhead of
> remembering all the EAPI differences. If the packages are untouched
> for ages and don't require maintenance, my claim is that there is no
> cognitive load to begin with.

That stops the moment something could use a USE- or slot operator dependency 
(because of a tree-wide change that also affects the old package). Then the 
EAPI bump suddenly becomes urgent (and a minor QA-like commit becomes a major 
change).

And no bugs probably just means no users that could file bugs. We really need 
something like gentoostats...

> > [Interestingly, as long as no specific efforts are made, the number of
> > ebuilds in all deprecated EAPI decreases roughly equally and
> > exponentially. That means the probability of any old ebuild to be removed
> > within a certain time interval is a constant as function of time...]
> 
> This is a great point in favor of *not* bothering to proactively bump EAPI.

Not really. It's an asymptotic curve. *If* you want to have it gone 
*completely* at some point, you have to start actively working on that. The 
only question is when, earlier or later... 

I'd guess having an EAPI approach <~ 1% of the tree is probably as good as any 
other criterion.

[[What is interesting but offtopic is that the exponential decay is the same 
for a long timeframe, see [*] third panel... the downward slope of the white 
line for EAPI=0 barely changes, and the other EAPI run parallel to it as long 
as nothing special happens... This means that Gentoo isn't dying after all, 
since the same developer activity takes place!]]

[*] https://www.akhuettel.de/~huettel/plots/eapi.php

-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, perl, libreoffice, comrel)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] Is removing old EAPIs worth the churn?

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1089 bytes --]

On Tue, 6 Mar 2018 14:13:11 -0800
Matt Turner <mattst88@gentoo.org> wrote:

> The long tail of packages were packages that by definition didn't
> require any maintenance or add to our collective cognitive overhead.
> :)

In some cases, they "didn't require maintenance" because nobody was
using them.

And because nobody was using them, nobody realised they were broken.

Subsequently, the "has old EAPI" is a good indicator in some cases of
"this package has not seen love in a while, and is probably broken"

Revisiting old packages like this root out real problems, like "oh,
this is so broken, it silently installed but didn't run at all for 3
years...."

Which won't become apparent until some poor user attempts to use it,
and at that point, you *hope* they file a bug.

So the resulting re-stabilization ends up forcibly expanding the test
audience, reaffirming "this package that nobody has glanced at for 5+
years still works on every platform we said it did", which is probably
an important thing to be able to state when a package is deemed
'stable'.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]