[gentoo-dev] stabilizing libraries without testing reverse deps

[gentoo-dev] stabilizing libraries without testing reverse deps

[hasufell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

It seems this happens more frequently these days, so I'd like to
remind people to check stable reverse deps before stabilizing a
library, especially when this is a non-maintainer stablereq.

Arch teams do not test them, so this is the business of the maintainer
or the dev who requested stabilization.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSSJ5vAAoJEFpvPKfnPDWz9IAH/iFs7a87ylffzSBALk7FqmPw
voO1ubXaneNsjAGErZpj2XZlmZHuWHGrbPbvJllHKoKTWcXV1hUwYMYgUWwEF8Th
Ypco+MM/kAlcaB8vPVOjV+k3h3ogqcz7xUhkl9qShMgyKJrLS4gwlXavEpFQi0Mq
u/uqIwW2XN8tfrPCbB/FG4A6GqggIDb6ce9UCCjyYuqF3IUOKlj2LMz1SMBI7/ZQ
fVLrk9DqYzgns5Q0cQmCN+Fk+/2ypBdhybLNCpUD0HfhP8U05S5Cp1VEN9XOczDs
EAM/3pXRG6rZZspQmFIPjiTiQMFvxlF1kexpp1QQ/mqiWfRbjU4j7hHkc4//npE=
=w1hs
-----END PGP SIGNATURE-----

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 1571 bytes --]

On 9/29/13 2:41 PM, hasufell wrote:
> It seems this happens more frequently these days, so I'd like to
> remind people to check stable reverse deps before stabilizing a
> library, especially when this is a non-maintainer stablereq.

+1 to the reminder. It would be great to hear about specific examples of
problems happening more frequently recently. FWIW I didn't notice such
tendency, but that doesn't mean it's not happening.

> Arch teams do not test them, so this is the business of the maintainer
> or the dev who requested stabilization.

This is new to me.

Do you have anything official to point to so that you could back your claim?

See e.g.
<http://www.gentoo.org/proj/en/base/x86/arch-testers-faq.xml#steptest>:
"If the package is a library, emerge a couple of packages that use the
library to ensure they still work with the new version (best option: all
that depend on it and have a stable version)."

I also created a tool
(<http://git.overlays.gentoo.org/gitweb/?p=proj/arch-tools.git;a=blob;f=reverse-dependencies.py;hb=HEAD>)
to assist arch teams in testing reverse dependencies.

As far as I know the person opening the bug does not guarantee anything.
Furthermore, the bugs I've filed using an automated tool explicitly ask
for maintainer's opinion before adding arches. They are only filed for
packages with no open bugs by the way. The person stabilizing the
package is ultimately responsible for breakages - otherwise why would we
have dedicated arch teams instead of letting anyone stabilize anything?

Paweł


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[Thomas Kahle]

[-- Attachment #1: Type: text/plain, Size: 583 bytes --]

On 09/29/2013 11:41 PM, hasufell wrote:
> It seems this happens more frequently these days, so I'd like to
> remind people to check stable reverse deps before stabilizing a
> library, especially when this is a non-maintainer stablereq.
> 
> Arch teams do not test them, so this is the business of the maintainer
> or the dev who requested stabilization.

Huh.  This must be new.  When I was still arch-testing, reverse deps
were checked.  app-portage/tatt (which is sort of maintainer-needed now)
has support for these tests too.

Cheers,
Thomas

-- 
Thomas Kahle


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 555 bytes --]

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[Andreas K. Huettel]

Am Sonntag, 29. September 2013, 23:41:03 schrieb hasufell:
> It seems this happens more frequently these days, so I'd like to
> remind people to check stable reverse deps before stabilizing a
> library, especially when this is a non-maintainer stablereq.
> 
> Arch teams do not test them, so this is the business of the maintainer
> or the dev who requested stabilization.

Arch testing includes testing of reverse deps. 
If that's not the case, arch teams are not doing their job.

-- 

Andreas K. Huettel
Gentoo Linux developer 
dilfridge@gentoo.org
http://www.akhuettel.de/

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[hasufell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/30/2013 12:54 AM, Andreas K. Huettel wrote:
> Am Sonntag, 29. September 2013, 23:41:03 schrieb hasufell:
>> It seems this happens more frequently these days, so I'd like to 
>> remind people to check stable reverse deps before stabilizing a 
>> library, especially when this is a non-maintainer stablereq.
>> 
>> Arch teams do not test them, so this is the business of the
>> maintainer or the dev who requested stabilization.
> 
> Arch testing includes testing of reverse deps. If that's not the
> case, arch teams are not doing their job.
> 

I'd have to search the irc logs, but afair I was told so by ago.

CCing him if I am wrong.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSSMJoAAoJEFpvPKfnPDWzKM0IAJE9RHUsSJKS9owZbuAdfvPh
3SSQDmXOP3WqG71Q0tLh0kYLH5IkSBktqypEgIBTyXbEiak8BH9tyR9xCSi6sEtb
toId4i9xCPZuHc5nLl6Yllc4uHQpkHziRGChESxWrGUv/+BVfrs9DhYA17yy3JYc
R1QGANf82v4hH85zsN9ITbJK07YkEzOsdBTbKZpxMeku6+X70KWMe51TImXDhOup
fVNTL5uss5BhHpbhVB4PInlTpSTBGUVbiNQRmdmxs+oj/IKfOQZZaWa4ax1d7gmI
hDunq5ZjsetaU3o7bDLzantSyLEX5fMYbOrZA/HTAKSX6jLDykmLRkrov39/E8M=
=RK65
-----END PGP SIGNATURE-----

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[Rich Freeman]

On Sun, Sep 29, 2013 at 8:14 PM, hasufell <hasufell@gentoo.org> wrote:
> On 09/30/2013 12:54 AM, Andreas K. Huettel wrote:
>> Am Sonntag, 29. September 2013, 23:41:03 schrieb hasufell:
>>> It seems this happens more frequently these days, so I'd like to
>>> remind people to check stable reverse deps before stabilizing a
>>> library, especially when this is a non-maintainer stablereq.
>>>
>>> Arch teams do not test them, so this is the business of the
>>> maintainer or the dev who requested stabilization.
>>
>> Arch testing includes testing of reverse deps. If that's not the
>> case, arch teams are not doing their job.
>>
>
> I'd have to search the irc logs, but afair I was told so by ago.
>
> CCing him if I am wrong.

If you aren't wrong, that might be why Ago is about the only one
stabilizing libraries...  :)

It probably makes sense for arch teams to test a few reverse
dependencies as the FAQ suggests.  If we want them all tested, then it
would make a lot more sense to have a tinderbox or other automated
testing tools of some sort.  Even then, we won't get much more than
compile testing, or whatever test suites the packages happen to come
with.

I haven't really seen any sign of widespread breakage though.  I'm
sure some issues slip through, but short of having automated testing I
doubt we'll ever do better than that.

Rich

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 735 bytes --]

On 9/29/13 7:41 PM, Rich Freeman wrote:
> Even then, we won't get much more than compile testing, or whatever
> test suites the packages happen to come with.

That's right.

I think we can rely on the time packages spend in ~arch to catch the
issues that wouldn't come up with compile and test phases.

Also, when stable is not far behind testing, the testing that ~arch
packages get is more applicable to stable (because there are less
differences). This is one of the main reason for my push to stabilize
more packages (always with proper testing), and I've also seen bugs
being fixed in ~arch but the fixes are not always pushed to stable
(which can be addressed by stabilizing more packages in general).

Paweł


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[Peter Volkov]

В Пн, 30/09/2013 в 00:54 +0200, Andreas K. Huettel пишет:
> Am Sonntag, 29. September 2013, 23:41:03 schrieb hasufell:
> > It seems this happens more frequently these days, so I'd like to
> > remind people to check stable reverse deps before stabilizing a
> > library, especially when this is a non-maintainer stablereq.
> > 
> > Arch teams do not test them, so this is the business of the maintainer
> > or the dev who requested stabilization.
> 
> Arch testing includes testing of reverse deps. 
> If that's not the case, arch teams are not doing their job.

I think it's good idea if maintainers and arch team developers will work
in tandem and help each other by both checking reverse deps. That said
the question here is who is in charge for stable tree stability? Stable
tree is the arch team's primary responsibility. That is why only they
are allowed to touch stable keywords after all. So arch team developer
should never commit anything if at least few reverse deps were checked.
Last but not least, I guess that average developer has lot's of things
in package.keywords, so it's not sane to expect developer to do stable
tree checks.

--
Peter.

[gentoo-dev] Re: stabilizing libraries without testing reverse deps

[Agostino Sarubbo]

This is a delicate point.

If you look at the policy, it says to test few rdeps.

The arch tester is in charge to test the packages on his architecture. These type of failures are _not_ architecture dependant.

So, instead of have 10 ATs that are testing the same rdeps, seems logic that the maintainer could do it one time.

What do you think?

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[Sergey Popov]

[-- Attachment #1: Type: text/plain, Size: 685 bytes --]

30.09.2013 01:41, hasufell пишет:
> Arch teams do not test them, so this is the business of the maintainer
> or the dev who requested stabilization.
> 

I hope you are kidding, cause when i was joining to arch teams, i was
taught to test reverse dependencies of libraries.

Of course, maintainer should test this also when bringing new version to
tree to prevent breakages. But stabilizing is different story and those
who putting apropriate version to stable should be 100% sure that it
would not break revdeps.

-- 
Best regards, Sergey Popov
Gentoo developer
Gentoo Desktop Effects project lead
Gentoo Qt project lead
Gentoo Proxy maintainers project lead


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 555 bytes --]

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[Markos Chandras]

On 09/29/2013 10:41 PM, hasufell wrote:
> Arch teams do not test them, so this is the business of the maintainer
> or the dev who requested stabilization.
> 

That is definitely not true. We always trained Arch Testers to test
reverse dependencies as well.

-- 
Regards,
Markos Chandras - Gentoo Linux Developer
http://dev.gentoo.org/~hwoarang

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[hasufell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/30/2013 09:22 AM, Markos Chandras wrote:
> On 09/29/2013 10:41 PM, hasufell wrote:
>> Arch teams do not test them, so this is the business of the
>> maintainer or the dev who requested stabilization.
>> 
> 
> That is definitely not true. We always trained Arch Testers to
> test reverse dependencies as well.
> 

Then something is wrong:

https://bugs.gentoo.org/show_bug.cgi?id=464536
https://bugs.gentoo.org/show_bug.cgi?id=470554

for the first bug:
net-libs/ortp media-libs/mediastreamer and net-voip/linphone
are from the same upstream and actually have to be bumped and
stabilized TOGETHER, because it is very likely that they break
otherwise. And that's exactly what happened. The maintainer was
probably aware of it, but didn't respond, so arch testers went ahead
and did not test reverse deps.

for the second bug:
we now have a stable net-libs/libosip that cannot be installed when
you want to install stable net-libs/libeXosip... that is not a good
spot. Those libraries again should have been stabilized TOGETHER.

To me it seems one relies on the other to handle this and in the end
no one does?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSSVYfAAoJEFpvPKfnPDWzPm0H+wU2G0iWQOT1cF02UPLUNFYF
GniIti6UBxV1iHmMBBLuKLzQYsSQtYQLSE7vTrbjUE2LyY6GlcfTOFiJOZG3Kar7
SNpH6jKkFY0XQnB7IaUcEKNJ+7esXZt9C3OvLJ/IqLcrQrqNsggGyKjoHwE7DoCG
RRAt+P18iU+XCOWgqGDCCzv6Ocez1aGhQ83geimTJB5aap1dQiI96S3obN6hu+8F
m9aXZ/nqBYLcrKQvjTWRtGX9Vz0WLN/F3vnfz7LcV5plz33tAn1rHSoe7okI5LQr
KP84dT+t6ixv8UHX8uPc2hFd3XvxVNKgduyvpLf7NnWN2A3D+JToM6zCJ64Xz5Q=
=J4jo
-----END PGP SIGNATURE-----

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[Chí-Thanh Christopher Nguyễn]

hasufell schrieb:
> https://bugs.gentoo.org/show_bug.cgi?id=464536
> https://bugs.gentoo.org/show_bug.cgi?id=470554
>
> for the first bug:
> net-libs/ortp media-libs/mediastreamer and net-voip/linphone
> are from the same upstream and actually have to be bumped and
> stabilized TOGETHER, because it is very likely that they break
> otherwise. And that's exactly what happened. The maintainer was
> probably aware of it, but didn't respond, so arch testers went ahead
> and did not test reverse deps.

I already replied in the second bug but let me reiterate again. What I
wrote in October 2012 to this list[1] is basically still true.

Following retirements, there is nobody in voip team who is interested in
these packages any more. Nobody in voip requested that these packages go
stable. When I read bugzilla reports that version bumps (typically done
as drive-by commits by outside developers) break consumers, then I
sometimes update the dependencies to account for that. When I saw the
second stabilization bug, I added the blocker, but the stabilization
proceeded anyway due to technical issues with the robo-stable scripts.

The ffmpeg-1.0 (and libav-9) situation got especially bad. For example,
a ptlib ebuild was committed which introduced libav-9 compatibility, but
also broke *every* *single* *consumer* of the package[2]! I have only
little time to dedicate to voip packages, and cleaning up the mess that
other developers leave is not a good way to use this time.

> To me it seems one relies on the other to handle this and in the end
> no one does?

We have one user, Andrew Savchenko, who expressed interest to proxy
maintain linphone and its dependencies via the voip overlay. I have
offered to commit the ebuilds for him to g-x86. Unfortunately it was a
lengthy process to allow him access to the overlay but that was cleared
10 days ago.

Once he starts pushing new ebuilds to the overlay, I will add him as
proxy maintainer in metadata.xml.


Best regards,
Chí-Thanh Christopher Nguyễn


[1] http://thread.gmane.org/gmane.linux.gentoo.devel/80638
[2] https://bugs.gentoo.org/show_bug.cgi?id=474742

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[hasufell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/30/2013 01:45 PM, Chí-Thanh Christopher Nguyễn wrote:
> hasufell schrieb:
>> https://bugs.gentoo.org/show_bug.cgi?id=464536 
>> https://bugs.gentoo.org/show_bug.cgi?id=470554
>> 
>> for the first bug: net-libs/ortp media-libs/mediastreamer and
>> net-voip/linphone are from the same upstream and actually have to
>> be bumped and stabilized TOGETHER, because it is very likely that
>> they break otherwise. And that's exactly what happened. The
>> maintainer was probably aware of it, but didn't respond, so arch
>> testers went ahead and did not test reverse deps.
> 
> I already replied in the second bug but let me reiterate again.
> What I wrote in October 2012 to this list[1] is basically still
> true.
> 
> Following retirements, there is nobody in voip team who is
> interested in these packages any more. Nobody in voip requested
> that these packages go stable. When I read bugzilla reports that
> version bumps (typically done as drive-by commits by outside
> developers) break consumers, then I sometimes update the
> dependencies to account for that. When I saw the second
> stabilization bug, I added the blocker, but the stabilization 
> proceeded anyway due to technical issues with the robo-stable
> scripts.
> 
> The ffmpeg-1.0 (and libav-9) situation got especially bad. For
> example, a ptlib ebuild was committed which introduced libav-9
> compatibility, but also broke *every* *single* *consumer* of the
> package[2]! I have only little time to dedicate to voip packages,
> and cleaning up the mess that other developers leave is not a good
> way to use this time.
> 
>> To me it seems one relies on the other to handle this and in the
>> end no one does?
> 
> We have one user, Andrew Savchenko, who expressed interest to
> proxy maintain linphone and its dependencies via the voip overlay.
> I have offered to commit the ebuilds for him to g-x86.
> Unfortunately it was a lengthy process to allow him access to the
> overlay but that was cleared 10 days ago.
> 
> Once he starts pushing new ebuilds to the overlay, I will add him
> as proxy maintainer in metadata.xml.
> 
> 
> Best regards, Chí-Thanh Christopher Nguyễn
> 
> 
> [1] http://thread.gmane.org/gmane.linux.gentoo.devel/80638 [2]
> https://bugs.gentoo.org/show_bug.cgi?id=474742
> 
> 

Yeah... I mean, I noticed. And this isn't critique to the team for the
packages being outdated. No bump is better than a shitty bump imo.

However... the fact still stands: arch team did not realize that it
breaks linphone. It could have been discussed on that bug how to
proceed because of the severity of the ffmpeg situation. But there was
no discussion and it broke a package in stable arch 3 months ago or so
and is unresolved til now.

That's not the way we should do things. If we know this is going to
happen we have to do something and in the worst case that is dropping
back to ~arch, tree-cleaning or hardmasking.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSSWd8AAoJEFpvPKfnPDWzBwsH/RNPbDqHwi3zO685i+3BIoYn
T2u6JpdMeXdyco1UaYcTxZcp46oHjP40XG6wNWv73O+EydtwxR2Gl5Z/3VS1liXx
QdQ2eBLR0R5aqFYJYzX6vj2iEDypeMoz7ka/0CiaS/uL1ygrEkJ2GB1X4TE/td8p
58OjsD0T8+wo3OZphUu+tyALI2v3AS9J60U9DEv9nTZ55kWVHVpmClxDJudgHmFv
RriqcViFapFf54fnVXtOUzw8l7to5+bhiSM4Ne5cRvfu+cdn06w7p4iMEIiwshYS
OHUqG3K5peDW6BiO/T8Vg1hZuc/BMtDrg9Z8mj8vvZIC4axaesfKRUtykgijXUc=
=fuyE
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Re: stabilizing libraries without testing reverse deps

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 3225 bytes --]

On 9/29/13 11:14 PM, Agostino Sarubbo wrote:
> If you look at the policy, it says to test few rdeps.

And I think this is right. If you'd like things to be done in a
different way, discussing them is OK, but "unilaterally" just skipping
that is not OK.

> The arch tester is in charge to test the packages on his architecture.
> These type of failures are _not_ architecture dependant.

It depends. It certainly can be package-version-dependent, and this can
vary by arch, even between x86 and amd64.

> So, instead of have 10 ATs that are testing the same rdeps, seems logic
> that the maintainer could do it one time.
> 
> What do you think?

It's not always the maintainer who files the stabilization bug. It can
be a user, or it can be me using automated scripts (which for the first
bug submission ask for maintainer's OK).

One possible option is for arch teams to refuse to do stabilization bugs
not explicitly approved by maintainers. If everybody agrees that such
approval also means testing reverse dependencies, it may be a move in
direction you're saying.

Now you've used an argument of scale: 10 ATs doing the "same thing"
(which btw I rarely see so many AT reports in one bug) vs. just 1
maintainer doing that.

I think that's a too local view. When you take the whole stabilization
queue into account (at least 80-120 bugs, below that it's not worth it
to me) you can save a lot of time through batching. Here's how:

1. On a fully stable system (even before testing packages to be
stabilized) emerge the reverse dependencies to make sure they currently
work (sometimes they're already broken in the stable tree, and some
packages to be stabilized actually fix these breakages). This can easily
take an overnight emerge, so it's good to put as many packages there as
possible to use that time effectively (and emerge's --keep-going option
is very helpful).

2. Emerge the packages to be tested. Usually another overnight emerge.

3. revdep-rebuild and all usual actions after updating the system.

4. Explicitly re-emerge the reverse dependencies again and compare the
results with run from (1). This would usually also take a long time on
my machine, so it's good to batch as many packages as possible. File
bugs for any regressions and do not stabilize packages introducing the
breakages.

As you see, the arch team member can do the all ~100 packages in about
3-4 days, and most of the process doesn't require human input. Most of
the time is spent reviewing the bugs (I have a script for that called
bugzilla-viewer.py which also displays related bugs and repoman output)
and reviewing the error messages for packages to be tested and reverse
dependencies. This is more effective the more packages you have, so arch
team member has a great advantage here being able to batch ~100 packages
instead of asking several maintainers to do their tests.

Of course it's good when maintainers also do their own testing in
stable, but I wouldn't block stabilization on that. IMHO instead of
trying to put the blame/duty on either maintainers or arch teams, the
right question to ask is what's the best process to do stabilizations well.

Paweł


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 629 bytes --]

On 9/30/13 3:44 AM, hasufell wrote:
> https://bugs.gentoo.org/show_bug.cgi?id=470554
> [...]
> for the second bug:
> we now have a stable net-libs/libosip that cannot be installed when
> you want to install stable net-libs/libeXosip... that is not a good
> spot. Those libraries again should have been stabilized TOGETHER.
> 
> To me it seems one relies on the other to handle this and in the end
> no one does?

Applying the maintainer timeout might be the problematic part there.

Note that the arch teams should still look at the blocking bug, and e.g.
bugzilla-viewer.py displays that prominently.

Paweł


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[Patrick Lauer]

On 09/30/2013 07:45 PM, Chí-Thanh Christopher Nguyễn wrote:
> hasufell schrieb:
>> https://bugs.gentoo.org/show_bug.cgi?id=464536
>> https://bugs.gentoo.org/show_bug.cgi?id=470554
>>
>> for the first bug:
>> net-libs/ortp media-libs/mediastreamer and net-voip/linphone
>> are from the same upstream and actually have to be bumped and
>> stabilized TOGETHER, because it is very likely that they break
>> otherwise. And that's exactly what happened. The maintainer was
>> probably aware of it, but didn't respond, so arch testers went ahead
>> and did not test reverse deps.
> 
> I already replied in the second bug but let me reiterate again. What I
> wrote in October 2012 to this list[1] is basically still true.
> 
> Following retirements, there is nobody in voip team who is interested in
> these packages any more. Nobody in voip requested that these packages go
> stable. When I read bugzilla reports that version bumps (typically done
> as drive-by commits by outside developers) break consumers, then I
> sometimes update the dependencies to account for that. When I saw the
> second stabilization bug, I added the blocker, but the stabilization
> proceeded anyway due to technical issues with the robo-stable scripts.
> 

> due to technical issues with the robo-stable scripts.

> due to technical issues with the robo-stable scripts.


let me summarize my response as "WAT"


Maybe we should just have a cronjob that just does all that automatically?

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[Markos Chandras]

On 09/30/2013 11:44 AM, hasufell wrote:
> On 09/30/2013 09:22 AM, Markos Chandras wrote:
>> On 09/29/2013 10:41 PM, hasufell wrote:
>>> Arch teams do not test them, so this is the business of the
>>> maintainer or the dev who requested stabilization.
>>>
> 
>> That is definitely not true. We always trained Arch Testers to
>> test reverse dependencies as well.
> 
> 
> Then something is wrong:
> 
> https://bugs.gentoo.org/show_bug.cgi?id=464536
> https://bugs.gentoo.org/show_bug.cgi?id=470554
> 
> for the first bug:
> net-libs/ortp media-libs/mediastreamer and net-voip/linphone
> are from the same upstream and actually have to be bumped and
> stabilized TOGETHER, because it is very likely that they break
> otherwise. And that's exactly what happened. The maintainer was
> probably aware of it, but didn't respond, so arch testers went ahead
> and did not test reverse deps.
> 
The fact that an AT may or may not do his job properly does not mean
that we should delegate the problem to the maintainer. I am not saying
that the maintainer should not test his packages. But when it comes to
stabilizations it's the job of ATs and Arch developers to decide what
does and what doesn't belong to the stable tree.

-- 
Regards,
Markos Chandras - Gentoo Linux Developer
http://dev.gentoo.org/~hwoarang

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 736 bytes --]

On Mon, Sep 30, 2013 at 12:40:04AM +0200, Thomas Kahle wrote:
> On 09/29/2013 11:41 PM, hasufell wrote:
> > It seems this happens more frequently these days, so I'd like to
> > remind people to check stable reverse deps before stabilizing a
> > library, especially when this is a non-maintainer stablereq.
> > 
> > Arch teams do not test them, so this is the business of the maintainer
> > or the dev who requested stabilization.
> 
> Huh.  This must be new.  When I was still arch-testing, reverse deps
> were checked.  app-portage/tatt (which is sort of maintainer-needed now)
> has support for these tests too.

Agreed. I was always told that it is up to the arch teams to test the
reverse deps.

William

William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[Rich Freeman]

On Mon, Sep 30, 2013 at 12:58 PM, William Hubbs <williamh@gentoo.org> wrote:
>
> Agreed. I was always told that it is up to the arch teams to test the
> reverse deps.

While I think this makes the most sense in general, I think
maintainers do have a role.

If some package has 75 reverse dependencies, and 1 of them tends to
break every time there is an upgrade, then it makes more sense for the
maintainer to stay on top of that and coordinate stabilization.  The
arch team isn't going to remember that, and unless they test all 75
they're not going to spot the 1 that breaks.

Now, in cases like the VOIP ones where there are only 3 reverse deps
it really does make sense for the arch teams to be expected to spot
issues, especially now that tools exist to make this a bit easier...

Rich

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[Jeroen Roovers]

On Sun, 29 Sep 2013 23:41:03 +0200
hasufell <hasufell@gentoo.org> wrote:

> Arch teams do not test them

When "arch teams" do not test them, there is something wrong with "arch
teams". Being a member of one, I assure you that is not what *I* do.


     jer

Re: [gentoo-dev] Re: stabilizing libraries without testing reverse deps

[Jeroen Roovers]

On Mon, 30 Sep 2013 08:14:29 +0200
Agostino Sarubbo <ago@gentoo.org> wrote:

> These type of failures are _not_ architecture dependant.

This is wrong.

Libraries behave differently on different architectures because the
compiled code is actually different. Different architectures use
different ways to access and manipulate memory. Libraries are
linked/loaded differently on different architectures.

If there is architecture specific code in a library, there is a really
good chance that the internal test suite does not cover it because
upstream doesn't have the hardware to test it.

If there are generic code paths in a library that gets used when no
architecture-specific code is available for a specific architecture,
those might go untested upstream as well as on the package maintainer's
system, since some (optional?) alternative architecture specific
optimisation code paths might be (automatically) selected in either
case.

I could go on...

> So, instead of have 10 ATs that are testing the same rdeps, seems
> logic that the maintainer could do it one time.

Your logic is flawed.

> What do you think?

I think that when you set out to help every minor architecture get
stable, you didn't know what you were getting into.


     jer

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[Jeroen Roovers]

On Tue, 01 Oct 2013 00:23:16 +0800
Patrick Lauer <patrick@gentoo.org> wrote:

> On 09/30/2013 07:45 PM, Chí-Thanh Christopher Nguyễn wrote:
> > due to technical issues with the robo-stable scripts.
> 
> > due to technical issues with the robo-stable scripts.
> 
> let me summarize my response as "WAT"

I call, and raise you a "THIS".

> Maybe we should just have a cronjob that just does all that
> automatically?

Having been doing stabilisation for nearly eight years, I have often
*thought* about ways to automate architecture testing/stabilisation.

I have invested in tools and scripts that help me run certain steps
along the way. I have never invested time or effort in automating
the process entirely, because it will cause failures.

Actually looking at stabilisation targets (the relevance USE flags, code
in the actual ebuilds, DEPENDs), assessing the targets' importance (for
instance, in checking what RDEPENDs on them), and testing accordingly,
is the only way to ensure that a stabilisation request is sane and
doable, and only then can you execute it.

Since we have had automated stabilisation requests filed, it has
become even more important to not automate stabilisation itself.

Not automating the entire process involves more manual steps, and most
of the time it isn't very rewarding, but it does get the job done of
catching out nearly all of the superficial bugs (like, does this
library actually work with stable RDEPENDs).


     jer

Re: [gentoo-dev] Re: stabilizing libraries without testing reverse deps

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 693 bytes --]

On 10/1/13 7:13 AM, Jeroen Roovers wrote:
> On Mon, 30 Sep 2013 08:14:29 +0200
> Agostino Sarubbo <ago@gentoo.org> wrote:
>> What do you think?
> I think that when you set out to help every minor architecture get
> stable, you didn't know what you were getting into.

+1

I think it's great to see your work there, Agostino. If I can have a
suggestion though, I'd much rather see x86 and amd64 being done with
thorough testing and other arches mostly on as needed / best effort
basis (which doesn't mean less testing but basically larger delays would
be acceptable - or not keeping as many packages in the stable tree, not
sure how many people actually use them).

Paweł



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[Peter Stuge]

hasufell wrote:
> No bump is better than a shitty bump imo.

I agree, but I think the problem is basically that many people
consider it impossible for "newer" to ever be shitty.

Even if they are intimately familiar with the details of a package
upstream they may still not be capable of determining what is shitty
in the context of a distribution.

I guess most stabilization requesters as well as actual stabilizers
are even less familiar with upstream details, so determining what is
shitty and not is really hard.. :\


//Peter

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 2902 bytes --]

On 2 October 2013 08:51, Peter Stuge <peter@stuge.se> wrote:

> I agree, but I think the problem is basically that many people
> consider it impossible for "newer" to ever be shitty.
>
> Even if they are intimately familiar with the details of a package
> upstream they may still not be capable of determining what is shitty
> in the context of a distribution.
>
> I guess most stabilization requesters as well as actual stabilizers
> are even less familiar with upstream details, so determining what is
> shitty and not is really hard.. :\
>


The other really annoying thing you have to consider, is that most people
out there are using all (~ARCH) or all (ARCH) keywording, not a mix of the
two.

^ This has the fun side effect of meaning packages that are (~ARCH) and
have (ARCH) dependents, where the package that is currently (~ARCH) is
pending stabilization,  has likely had nobody test it at all except for
arch testers.

So if you're relying on the presence of filed bugs to give some sort of
coverage metric, you're going to be out of luck from time to time. For
instance, that fun bug where stabilising a version of libraw broke the
things depending upon it that were already stable.

Its ironic really, thats essentially a hidden bug that exists as a result
of having two tiers of testing.

https://bugs.gentoo.org/show_bug.cgi?id=458516

I've been long wishing there was a FEATURE I could turn on that would just
report installs to a database somewhere, showing
success/fail/succcess+tests/fail+tests , with dependencies, useflags, and
other relevant context, so you'd at least have a dataset of *success*
rates, and you could do more heavy testing on things where there were fail
results, or an absense of results.

CPAN has a comparable service that leverages end users reporting test runs
on code while they install it, and some end users, given this power, go so
out and set up mass automated testing boxes, the utility of which I find
useful on a daily basis, because a user is far more likely to get an
automated success/fail report sent to a server, and far *less* likely to
want to set time aside to go through the rigmarole of filing a bug report.

Some users are also inclined to just try a few versions either side, and
never file a bug report, or try twiddling USE flags or disabling
problematic FEATURES to find a version that works for them, and you may
never see a bug report for that.

An automated "X combination failed" report at very least lets you know a
datapoint where a failure occurred.

I'm not saying we should make any automated decision making *based* on
those reports, but it would be far more useful to have a list of known
failure cases up front than to ask a tester to try be lucky and find it by
looking for it.

( After all, bugs often arise when you're not looking for them, as opposed
to when you are, and some bugs, when looked for, vanish )

-- 
Kent

[-- Attachment #2: Type: text/html, Size: 3869 bytes --]

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[yac]

[-- Attachment #1: Type: text/plain, Size: 3665 bytes --]

On Wed, 2 Oct 2013 10:12:00 +1300
Kent Fredric <kentfredric@gmail.com> wrote:

> On 2 October 2013 08:51, Peter Stuge <peter@stuge.se> wrote:
> 
> > I agree, but I think the problem is basically that many people
> > consider it impossible for "newer" to ever be shitty.
> >
> > Even if they are intimately familiar with the details of a package
> > upstream they may still not be capable of determining what is shitty
> > in the context of a distribution.
> >
> > I guess most stabilization requesters as well as actual stabilizers
> > are even less familiar with upstream details, so determining what is
> > shitty and not is really hard.. :\
> >
> 
> 
> The other really annoying thing you have to consider, is that most
> people out there are using all (~ARCH) or all (ARCH) keywording, not
> a mix of the two.
> 
> ^ This has the fun side effect of meaning packages that are (~ARCH)
> and have (ARCH) dependents, where the package that is currently
> (~ARCH) is pending stabilization,  has likely had nobody test it at
> all except for arch testers.

AFAIK this moot since users running testing are expected to expect
breakage regardless of what the breakage is, as long as it caused by
testing package.

btw, I'm one of those running mixed stable/testing system and based on
my limited experience I believe this is a rare scenario.

> So if you're relying on the presence of filed bugs to give some sort
> of coverage metric, you're going to be out of luck from time to time.
> For instance, that fun bug where stabilising a version of libraw
> broke the things depending upon it that were already stable.
> 
> Its ironic really, thats essentially a hidden bug that exists as a
> result of having two tiers of testing.
> 
> https://bugs.gentoo.org/show_bug.cgi?id=458516
> 
> I've been long wishing there was a FEATURE I could turn on that would
> just report installs to a database somewhere, showing
> success/fail/succcess+tests/fail+tests , with dependencies, useflags,
> and other relevant context, so you'd at least have a dataset of
> *success* rates, and you could do more heavy testing on things where
> there were fail results, or an absense of results.

+1
 
> CPAN has a comparable service that leverages end users reporting test
> runs on code while they install it, and some end users, given this
> power, go so out and set up mass automated testing boxes, the utility
> of which I find useful on a daily basis, because a user is far more
> likely to get an automated success/fail report sent to a server, and
> far *less* likely to want to set time aside to go through the
> rigmarole of filing a bug report.
> 
> Some users are also inclined to just try a few versions either side,
> and never file a bug report, or try twiddling USE flags or disabling
> problematic FEATURES to find a version that works for them, and you
> may never see a bug report for that.
> 
> An automated "X combination failed" report at very least lets you
> know a datapoint where a failure occurred.

I'd be cautious about involving users in this as it very often happens
to myself that something breaks, I get mad and then figure it was my
own fault (eg. messing with cflags I shouldn't mess with)
 
> I'm not saying we should make any automated decision making *based* on
> those reports, but it would be far more useful to have a list of known
> failure cases up front than to ask a tester to try be lucky and find
> it by looking for it.
> 
> ( After all, bugs often arise when you're not looking for them, as
> opposed to when you are, and some bugs, when looked for, vanish )
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[hasufell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/02/2013 06:43 PM, yac wrote:
>> So if you're relying on the presence of filed bugs to give some
>> sort of coverage metric, you're going to be out of luck from time
>> to time. For instance, that fun bug where stabilising a version
>> of libraw broke the things depending upon it that were already
>> stable.
>> 
>> Its ironic really, thats essentially a hidden bug that exists as
>> a result of having two tiers of testing.
>> 
>> https://bugs.gentoo.org/show_bug.cgi?id=458516
>> 
>> I've been long wishing there was a FEATURE I could turn on that
>> would just report installs to a database somewhere, showing 
>> success/fail/succcess+tests/fail+tests , with dependencies,
>> useflags, and other relevant context, so you'd at least have a
>> dataset of *success* rates, and you could do more heavy testing
>> on things where there were fail results, or an absense of
>> results.
> 
> +1
> 

yo

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSTFimAAoJEFpvPKfnPDWzzqIH/2Sfn7fbNGmZxGWKDRKNPowQ
7FyUeMWtjZ/ghL9J7erp9f293QBh2iLzgsVGqij1++LPul6BAYaAqX3HcktB3zNJ
8AEjLh9ied1497EtXiCrQGmL35dg2kyvtnxfp8/OAxz80CaJKO+TQoGBnn8gjBQF
1kzeIJ02aWuU40143B0pQ2dS8vjD0tYViWxy6QQr7YUdAChtkSsbchnVUA0oqnX4
fQXXA1pfwyDqKpKereDM5PqrGWumhXhaEwEzoSUCm7ozIW0mtFxdMhBabFWcJFMK
lcTNWKyMfRCzoWsAu31jSC/nYMyIDFnS1VCQCk10pb6rFVkEpaQtW1MSyjRdBkI=
=/IqK
-----END PGP SIGNATURE-----

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 2123 bytes --]

On 3 October 2013 05:43, yac <yac@gentoo.org> wrote:

> I'd be cautious about involving users in this as it very often happens
> to myself that something breaks, I get mad and then figure it was my
> own fault (eg. messing with cflags I shouldn't mess with)
>

That does happen from time to time on the CPAN Testers network, the great
thing about this is though, that there exists a test report, user err or
not.

And it is the job of the person who manually looks into bug reports to
determine if they're a clear indication of a real bug being exhibited, or
if its a user configuration problem.

And if a CPAN Author wants, they can contact a sender and ask for more
context if the CPAN author feels it relevant.

This is basically an inversion of bug reporting mechanics in cases of
trivial accidents user side that might otherwise simply not get reported,
because it becomes the developers duty to follow up on a failed build if
they think its relevant.

Also, it helps with the bug reporting process, and helps users diagnose
their own bugs, because they can, upon seeing a test/build failure, check
the bug tracker *and* the report matrix, and see if other users are also
exihibiting the failure, to help them get an idea of what is causing the
failure, by comparing reported variables.

Then, a user can say "Oh look, see, there's a boat load of tests failing on
x86 with only this specific USE flag, there's no open bug for that, so I'll
open one, and give the maintainer a lot of context on what looks to be the
problem by linking directly to a handful of failed reports".

This, would also be incidentally convenient for that old "ANSI Term Escape
codes" problem, because the report/log submission would be automated (
giving us full control over how reports are performed ), and we could have
a custom interface for displaying such reports that handled any submitted
escape codes graciously, decoupling us from needing to find a way to handle
that problem in bugzilla.

Just the unfortunate part here is there are no off-the-shelf products for
this problem space, so somebody would have to develop one.

-- 
Kent

[-- Attachment #2: Type: text/html, Size: 2681 bytes --]

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 485 bytes --]

On 10/1/13 2:12 PM, Kent Fredric wrote:
> So if you're relying on the presence of filed bugs to give some sort of
> coverage metric, you're going to be out of luck from time to time. For
> instance, that fun bug where stabilising a version of libraw broke the
> things depending upon it that were already stable.

That's why arch testers and arch developers are expected to test reverse
dependencies.

Even then, no amount of testing guarantees lack of problems.

Paweł


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1255 bytes --]

On 4 October 2013 05:11, "Paweł Hajdan, Jr." <phajdan.jr@gentoo.org> wrote:

> Even then, no amount of testing guarantees lack of problems.


Indeed, but which is a better assurance, "5 testers tested this combination
and nothing bad happened", or "5000 people tested this combination and
nothing bad happened".

Now, if you were to see "no people have successfully built combination X",
that in itself is interesting, even if you don't have actual failure
reports of that combination.

Also, if "5 testers tested this combination and nothing bad happened" is
combined with "however, we have 200 similar installation failures reported
for this combination", you've got some context for research you need to do
to understand why those failures exist ( even if none of them managed to
file a bug report ).

Essentially, I'm saying we need to lower the thresholds to providing
reliable feedback about what is happening with packages in the field, ie:
Diego's smoke boxes are very very useful, but thats *one* person. Imagine
if we can get 500+ people running similar smoke operations with a
manageable feedback system.

That would give us fare more assurance than the arch testers are likely to
be able to provide.


-- 
Kent

[-- Attachment #2: Type: text/html, Size: 1829 bytes --]

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 1465 bytes --]

On 10/3/13 3:30 PM, Kent Fredric wrote:
> Now, if you were to see "no people have successfully built combination X",
> that in itself is interesting, even if you don't have actual failure
> reports of that combination.
> 
> Also, if "5 testers tested this combination and nothing bad happened" is
> combined with "however, we have 200 similar installation failures reported
> for this combination", you've got some context for research you need to do
> to understand why those failures exist ( even if none of them managed to
> file a bug report ).
> 
> Essentially, I'm saying we need to lower the thresholds to providing
> reliable feedback about what is happening with packages in the field, ie:
> Diego's smoke boxes are very very useful, but thats *one* person. Imagine
> if we can get 500+ people running similar smoke operations with a
> manageable feedback system.

Oh totally, I was not dismissing benefits of that. It'd be great.

There is only one small detail: someone would need to create it.

Then, based on how other stats-related efforts in Gentoo turned out,
it's not that obvious to me how big the coverage would actually be.

When I think about using Gentoo in any production environment, I'm
pretty sure one has to do his own testing and staging. We try to keep
things reasonably sane in Gentoo stable, but even what you're describing
above won't cover _everything_, and this is mostly what I'm saying.

Paweł



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: [gentoo-dev] stabilizing libraries without testing reverse deps

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 933 bytes --]

On 4 October 2013 17:03, "Paweł Hajdan, Jr." <phajdan.jr@gentoo.org> wrote:

> but even what you're describing
> above won't cover _everything_, and this is mostly what I'm saying.
>


Yeh, its a given that it won't cover /all/ scenarios. Its obviously not
intended to /replace/ arch testers, just supplementary context.

Also, something else that I forgot to mention, is the real benefit you see
with automated test reporting, is the turn around time.

Within 24-48 hours of the changes hitting the rsync mirrors, you'll have a
batch of results showing you if the recent change broke the build or not.

And that could greatly improve the effective stability of (~ARCH) for those
who use it, as long as devs check the report stats.

And the effective stability of (~ARCH) increases the rate at which (~ARCH)
can afford to move, and increased fluidity and movement there would be
rather helpful.


-- 
Kent

[-- Attachment #2: Type: text/html, Size: 1499 bytes --]