[gentoo-dev] Disturbing state of arch testing in Gentoo

[gentoo-dev] Disturbing state of arch testing in Gentoo

[Michał Górny]

Hi, everyone.

Arch testing's relying on automation a lot these days.  Not saying
that's bad, if it improves the state of affairs.  However, I have some
concerns, based on what I've seen lately.

On top of that, it seems that most of it still relies on proprietary
software and we have no clue how *exactly* it works, and it's really,
really hard to get a straight answer.

So, my questions are:

1. Is "runtime testing required" field being respected?  Obviously not
every package can be (sufficiently) tested via FEATURES=test, so we've
added that fields.  However, if arch testers just ignore it and push
things stable based on pure build testing...

2. How are kernels being tested?  Given the speed with which new gentoo-
sources stablereqs are handled, I really feel like "arch testing" there
means "checking if sources install", and have little to do with working
kernels.

3. How does the automation handle packages that aren't trivially
installable?  I recall that in the past stablereqs were stalled for
months without a single comment because automation couldn't figure out
how to proceed, and nobody bothered reporting a problem.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] Disturbing state of arch testing in Gentoo

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 2175 bytes --]



> On 6 Nov 2022, at 08:15, Michał Górny <mgorny@gentoo.org> wrote:
> 
> Hi, everyone.
> 
> Arch testing's relying on automation a lot these days.  Not saying
> that's bad, if it improves the state of affairs.  However, I have some
> concerns, based on what I've seen lately.

Thanks for starting this discussion, I think others have felt this way too.

> 
> On top of that, it seems that most of it still relies on proprietary
> software and we have no clue how *exactly* it works, and it's really,
> really hard to get a straight answer.
> 

arthurzam, jsmolic, and I are using https://github.com/arthurzam/tattoo.

> So, my questions are:
> 
> 1. Is "runtime testing required" field being respected?  Obviously not
> every package can be (sufficiently) tested via FEATURES=test, so we've
> added that fields.  However, if arch testers just ignore it and push
> things stable based on pure build testing...

Not right now. We discussed it on #gentoo-dev maybe 2 months ago
or so but concluded we needed nattka support to fix up our automation.

That had two parts:
1. https://github.com/projg2/nattka/issues/72 & https://github.com/projg2/nattka/pull/73 (done)
2. https://github.com/arthurzam/tattoo/issues/1 (not done)

> 
> 2. How are kernels being tested?  Given the speed with which new gentoo-
> sources stablereqs are handled, I really feel like "arch testing" there
> means "checking if sources install", and have little to do with working
> kernels.
> 

I usually blacklist gentoo-sources because I can't actually test it, or
if I do stable it, I've tried to run them for real.

For gentoo-kernel, I run src_test.

> 3. How does the automation handle packages that aren't trivially
> installable?  I recall that in the past stablereqs were stalled for
> months without a single comment because automation couldn't figure out
> how to proceed, and nobody bothered reporting a problem.

It doesn't, really. I think at the very least we need to try do world
upgrades after adding the package list to package.accept_keywords,
as stuff like LLVM and GNOME bugs won't work without it (blockers etc).

Best,
sam

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 358 bytes --]

Re: [gentoo-dev] Disturbing state of arch testing in Gentoo

[Arthur Zamarin]

[-- Attachment #1.1: Type: text/plain, Size: 1433 bytes --]

On 06/11/2022 10.34, Sam James wrote:
> 
> ...
> 
> That had two parts:
> 1. https://github.com/projg2/nattka/issues/72 & https://github.com/projg2/nattka/pull/73 (done)
> 2. https://github.com/arthurzam/tattoo/issues/1 (not done)

I was waiting for nattka-0.4 (which returns the field value) and was
hoping to wait until it becomes stable, so it will be able to upgrade
nattka on all devboxes easily and just use new API. Seeing this
conversation, made me understand that it is desired to do it ASAP, so I
added it now with a little ugly code around (import guards to check that
nattka supports the new field).

So currently tattoo skips all bugs marked with Manual testing.


I also plan to create a small dashboard showing special cases of Arch
Testing bugs, such as:

1. Bugs with only one arch remaining
2. Bugs blocked by blocker bug
3. Bugs without any info and activity (not blocked, untouched, not done)

This is in hopes to have a more organized and priority bugs, to mitigate
cases when something somewhere failed, and we lose the bug until pinged.

My current plan is to have a script that generated a HTML file that I
will host on ~dev space, and will have a scheduled process to regenerate
the HTML. I'm still planning the solution and how to schedule it, so no
promises :)

-- 
Arthur Zamarin
arthurzam@gentoo.org
Gentoo Linux developer (Python, Arch Teams, pkgcore stack, GURU)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Disturbing state of arch testing in Gentoo

[Piotr Karbowski]

Hi,

On 06/11/2022 09.15, Michał Górny wrote:
> On top of that, it seems that most of it still relies on proprietary
> software and we have no clue how*exactly*  it works, and it's really,
> really hard to get a straight answer.

I never understood how it become socially acceptable in open source 
project that Gentoo is that one of the important automation that does 
tinderboxing is closed source and gate kept, meaning no one can on their 
own reproduce the failures, which is utterly annoying when your normal 
testing just works, yet their testing methodology is a secret.

I would be in favour of stepping up the social contract and actually 
prohibiting this kind of things, we had that before too, the nattka you 
mgorny wrote is replacement for old bugzilla bot that was ... 
closedsource and perished, though nattka now have way more features than 
the old thing ever had.

-- Piotr.

Re: [gentoo-dev] Disturbing state of arch testing in Gentoo

[Oskari Pirhonen]

[-- Attachment #1: Type: text/plain, Size: 1014 bytes --]

On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote:
> I would be in favour of stepping up the social contract and actually 
> prohibiting this kind of things, we had that before too, the nattka you 
> mgorny wrote is replacement for old bugzilla bot that was ... 
> closedsource and perished, though nattka now have way more features than 
> the old thing ever had.

As a user, I think it would be really cool if there was a requirement
that all infra and infra-adjacent stuff was free software.

I feel like I've read that Debian already has something like this. While
doing some quick searches I didn't find a full-on requirement, but all
their infra bits I did find were powered by free software. The most
relevant ones being buildd [1] and debci [2]. Additionally, the debci
docs has inctructions on reproducing tests yourself [3] which is a nice
extra IMO.

[1] https://buildd.debian.org
[2] https://ci.debian.net
[3] https://ci.debian.net/doc/file.MAINTAINERS.html

- Oskari

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [gentoo-dev] Disturbing state of arch testing in Gentoo

[Joonas Niilola]

[-- Attachment #1.1: Type: text/plain, Size: 1599 bytes --]

On 7.11.2022 8.07, Oskari Pirhonen wrote:
> On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote:
>> I would be in favour of stepping up the social contract and actually 
>> prohibiting this kind of things, we had that before too, the nattka you 
>> mgorny wrote is replacement for old bugzilla bot that was ... 
>> closedsource and perished, though nattka now have way more features than 
>> the old thing ever had.
> 
> As a user, I think it would be really cool if there was a requirement
> that all infra and infra-adjacent stuff was free software.
> 
> I feel like I've read that Debian already has something like this. While
> doing some quick searches I didn't find a full-on requirement, but all
> their infra bits I did find were powered by free software. The most
> relevant ones being buildd [1] and debci [2]. Additionally, the debci
> docs has inctructions on reproducing tests yourself [3] which is a nice
> extra IMO.
> 
> [1] https://buildd.debian.org
> [2] https://ci.debian.net
> [3] https://ci.debian.net/doc/file.MAINTAINERS.html
> 
> - Oskari

I _believe_ ago's tinderbox isn't being paid by the GF _anymore_ due to
this reason, but he keeps it running with his own expenses. I don't mind
this as long as the results are desirable and not phony. I still see a
lot of value in most of ago's work.

It is unfortunate we don't get to see the engine behind and copy it,
since I'd be really interested in using his automated bug search /
report tool. Even tattoo (https://github.com/arthurzam/tattoo) lacks
that at the moment.

-- juippis

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] Disturbing state of arch testing in Gentoo

[John Helmert III]

[-- Attachment #1: Type: text/plain, Size: 1845 bytes --]

On Mon, Nov 07, 2022 at 08:26:15AM +0200, Joonas Niilola wrote:
> On 7.11.2022 8.07, Oskari Pirhonen wrote:
> > On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote:
> >> I would be in favour of stepping up the social contract and actually 
> >> prohibiting this kind of things, we had that before too, the nattka you 
> >> mgorny wrote is replacement for old bugzilla bot that was ... 
> >> closedsource and perished, though nattka now have way more features than 
> >> the old thing ever had.
> > 
> > As a user, I think it would be really cool if there was a requirement
> > that all infra and infra-adjacent stuff was free software.
> > 
> > I feel like I've read that Debian already has something like this. While
> > doing some quick searches I didn't find a full-on requirement, but all
> > their infra bits I did find were powered by free software. The most
> > relevant ones being buildd [1] and debci [2]. Additionally, the debci
> > docs has inctructions on reproducing tests yourself [3] which is a nice
> > extra IMO.
> > 
> > [1] https://buildd.debian.org
> > [2] https://ci.debian.net
> > [3] https://ci.debian.net/doc/file.MAINTAINERS.html
> > 
> > - Oskari
> 
> I _believe_ ago's tinderbox isn't being paid by the GF _anymore_ due to
> this reason, but he keeps it running with his own expenses. I don't mind
> this as long as the results are desirable and not phony. I still see a
> lot of value in most of ago's work.

He's not using infra's AWS resources anymore, but he's still using the
devbox.amd64.dev.gentoo.org VM.

> It is unfortunate we don't get to see the engine behind and copy it,
> since I'd be really interested in using his automated bug search /
> report tool. Even tattoo (https://github.com/arthurzam/tattoo) lacks
> that at the moment.
> 
> -- juippis




[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [gentoo-dev] Disturbing state of arch testing in Gentoo

[Oskari Pirhonen]

[-- Attachment #1: Type: text/plain, Size: 880 bytes --]

On Mon, Nov 07, 2022 at 08:26:15 +0200, Joonas Niilola wrote:
> I _believe_ ago's tinderbox isn't being paid by the GF _anymore_ due to
> this reason, but he keeps it running with his own expenses. I don't mind
> this as long as the results are desirable and not phony. I still see a
> lot of value in most of ago's work.
> 

Don't get me wrong, I really appreaciate the work that all of you do to
keep my favorite distro running. And maintaining the level of quality
that I've grown to expect from Gentoo over the years too :)

> It is unfortunate we don't get to see the engine behind and copy it,
> since I'd be really interested in using his automated bug search /
> report tool. Even tattoo (https://github.com/arthurzam/tattoo) lacks
> that at the moment.
> 

Perhaps he would be open to the idea of releasing some parts of it? (pun
intended)

- Oskari

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [gentoo-dev] Disturbing state of arch testing in Gentoo

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 1092 bytes --]



> On 7 Nov 2022, at 06:07, Oskari Pirhonen <xxc3ncoredxx@gmail.com> wrote:
> 
> On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote:
>> I would be in favour of stepping up the social contract and actually
>> prohibiting this kind of things, we had that before too, the nattka you
>> mgorny wrote is replacement for old bugzilla bot that was ...
>> closedsource and perished, though nattka now have way more features than
>> the old thing ever had.
> 
> As a user, I think it would be really cool if there was a requirement
> that all infra and infra-adjacent stuff was free software.
> 
> I feel like I've read that Debian already has something like this. While
> doing some quick searches I didn't find a full-on requirement, but all
> their infra bits I did find were powered by free software. The most
> relevant ones being buildd [1] and debci [2]. Additionally, the debci
> docs has inctructions on reproducing tests yourself [3] which is a nice
> extra IMO.

Gentoo has https://www.gentoo.org/get-started/philosophy/social-contract.html.

Best,
sam

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 358 bytes --]

Re: [gentoo-dev] Disturbing state of arch testing in Gentoo

[Rich Freeman]

On Mon, Nov 7, 2022 at 6:16 PM Sam James <sam@gentoo.org> wrote:
>
> > On 7 Nov 2022, at 06:07, Oskari Pirhonen <xxc3ncoredxx@gmail.com> wrote:
> >
> > On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote:
> >> I would be in favour of stepping up the social contract and actually
> >> prohibiting this kind of things, we had that before too, the nattka you
> >> mgorny wrote is replacement for old bugzilla bot that was ...
> >> closedsource and perished, though nattka now have way more features than
> >> the old thing ever had.
> >
> > As a user, I think it would be really cool if there was a requirement
> > that all infra and infra-adjacent stuff was free software.
> >
> > I feel like I've read that Debian already has something like this. While
> > doing some quick searches I didn't find a full-on requirement, but all
> > their infra bits I did find were powered by free software. The most
> > relevant ones being buildd [1] and debci [2]. Additionally, the debci
> > docs has inctructions on reproducing tests yourself [3] which is a nice
> > extra IMO.
>
> Gentoo has https://www.gentoo.org/get-started/philosophy/social-contract.html.

I feel like something like a dev-run tinderbox is a bit out of the
scope of that.

Suppose I file a bug against a package, pointing out some issue in it.
How do you know I didn't use some proprietary static code analysis
tool to discover that error?  Does it even really matter?  The bug
speaks for itself.  It is like worrying about whether somebody who
filed a bug was running Windows or another proprietary OS or browser
on their desktop.

Well, a tinderbox is just an automated process for doing just that.
We don't require any dev to use a proprietary tinderbox before
committing.  It is something that individual devs choose to use for
themselves, automating the testing workflow and possibly the
submission of bugs.

I think the key is something that was brought up earlier in the
thread: is this causing problems?  If somebody is running some tool
against the repository and automatically filing bugs, and those bugs
are not useful/actionable and waste the time of volunteers, then that
is a problem.  Proprietary tools do contribute to this since they can
generate results that are harder to reproduce, but if they are clear
and accurate and actionable it could still be a net-positive.

Of course if somebody wants to contribute to 100% FOSS tinderbox
efforts that would be even better.  Perhaps if our 100% FOSS tinderbox
efforts addressed our needs very well, then nobody would want to
bother with the proprietary reports, or generating them.  IMO it would
be better to create the FOSS solution before abandoning the
proprietary one.  Doing otherwise is basically burning bridges - it
can be motivating in a sense but not really ideal.  I'd love to have a
100% FOSS solution around all of this, but I appreciate what has been
created and can hardly criticize volunteers for failing to make it
happen, especially since I haven't contributed to that myself.

-- 
Rich

Re: [gentoo-dev] Disturbing state of arch testing in Gentoo

[Sam James]

[-- Attachment #1: Type: text/plain, Size: 1911 bytes --]



> On 8 Nov 2022, at 00:23, Rich Freeman <rich0@gentoo.org> wrote:
> 
> On Mon, Nov 7, 2022 at 6:16 PM Sam James <sam@gentoo.org> wrote:
>> 
>>> On 7 Nov 2022, at 06:07, Oskari Pirhonen <xxc3ncoredxx@gmail.com> wrote:
>>> 
>>> On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote:
>>>> I would be in favour of stepping up the social contract and actually
>>>> prohibiting this kind of things, we had that before too, the nattka you
>>>> mgorny wrote is replacement for old bugzilla bot that was ...
>>>> closedsource and perished, though nattka now have way more features than
>>>> the old thing ever had.
>>> 
>>> As a user, I think it would be really cool if there was a requirement
>>> that all infra and infra-adjacent stuff was free software.
>>> 
>>> I feel like I've read that Debian already has something like this. While
>>> doing some quick searches I didn't find a full-on requirement, but all
>>> their infra bits I did find were powered by free software. The most
>>> relevant ones being buildd [1] and debci [2]. Additionally, the debci
>>> docs has inctructions on reproducing tests yourself [3] which is a nice
>>> extra IMO.
>> 
>> Gentoo has https://www.gentoo.org/get-started/philosophy/social-contract.html.
> 
> I feel like something like a dev-run tinderbox is a bit out of the
> scope of that.

I intentionally didn't comment on the scope for now, but I'm glad you did.

> 
> Suppose I file a bug against a package, pointing out some issue in it.
> How do you know I didn't use some proprietary static code analysis
> tool to discover that error?  Does it even really matter?  The bug
> speaks for itself.  It is like worrying about whether somebody who
> filed a bug was running Windows or another proprietary OS or browser
> on their desktop.
> 

It matters if someone can't then reproduce the bug which happens
somewhat often here.

[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 358 bytes --]

Re: [gentoo-dev] Disturbing state of arch testing in Gentoo

[John Helmert III]

[-- Attachment #1: Type: text/plain, Size: 3424 bytes --]

On Mon, Nov 07, 2022 at 07:23:33PM -0500, Rich Freeman wrote:
> On Mon, Nov 7, 2022 at 6:16 PM Sam James <sam@gentoo.org> wrote:
> >
> > > On 7 Nov 2022, at 06:07, Oskari Pirhonen <xxc3ncoredxx@gmail.com> wrote:
> > >
> > > On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote:
> > >> I would be in favour of stepping up the social contract and actually
> > >> prohibiting this kind of things, we had that before too, the nattka you
> > >> mgorny wrote is replacement for old bugzilla bot that was ...
> > >> closedsource and perished, though nattka now have way more features than
> > >> the old thing ever had.
> > >
> > > As a user, I think it would be really cool if there was a requirement
> > > that all infra and infra-adjacent stuff was free software.
> > >
> > > I feel like I've read that Debian already has something like this. While
> > > doing some quick searches I didn't find a full-on requirement, but all
> > > their infra bits I did find were powered by free software. The most
> > > relevant ones being buildd [1] and debci [2]. Additionally, the debci
> > > docs has inctructions on reproducing tests yourself [3] which is a nice
> > > extra IMO.
> >
> > Gentoo has https://www.gentoo.org/get-started/philosophy/social-contract.html.
> 
> I feel like something like a dev-run tinderbox is a bit out of the
> scope of that.
> 
> Suppose I file a bug against a package, pointing out some issue in it.
> How do you know I didn't use some proprietary static code analysis
> tool to discover that error?  Does it even really matter?  The bug
> speaks for itself.  It is like worrying about whether somebody who
> filed a bug was running Windows or another proprietary OS or browser
> on their desktop.
> 
> Well, a tinderbox is just an automated process for doing just that.
> We don't require any dev to use a proprietary tinderbox before
> committing.  It is something that individual devs choose to use for
> themselves, automating the testing workflow and possibly the
> submission of bugs.
> 
> I think the key is something that was brought up earlier in the
> thread: is this causing problems?  If somebody is running some tool
> against the repository and automatically filing bugs, and those bugs
> are not useful/actionable and waste the time of volunteers, then that
> is a problem.  Proprietary tools do contribute to this since they can
> generate results that are harder to reproduce, but if they are clear
> and accurate and actionable it could still be a net-positive.

In some cases, yes, this is exactly the problem. This was one of the
bugs reported in the now-deleted issue tracking repository on Github.

> Of course if somebody wants to contribute to 100% FOSS tinderbox
> efforts that would be even better.  Perhaps if our 100% FOSS tinderbox
> efforts addressed our needs very well, then nobody would want to
> bother with the proprietary reports, or generating them.  IMO it would
> be better to create the FOSS solution before abandoning the
> proprietary one.  Doing otherwise is basically burning bridges - it
> can be motivating in a sense but not really ideal.  I'd love to have a
> 100% FOSS solution around all of this, but I appreciate what has been
> created and can hardly criticize volunteers for failing to make it
> happen, especially since I haven't contributed to that myself.
> 
> -- 
> Rich
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [gentoo-dev] Disturbing state of arch testing in Gentoo

[Rich Freeman]

On Mon, Nov 7, 2022 at 7:34 PM John Helmert III <ajak@gentoo.org> wrote:
>
> On Mon, Nov 07, 2022 at 07:23:33PM -0500, Rich Freeman wrote:
> > Proprietary tools do contribute to this since they can
> > generate results that are harder to reproduce, but if they are clear
> > and accurate and actionable it could still be a net-positive.
>
> In some cases, yes, this is exactly the problem. This was one of the
> bugs reported in the now-deleted issue tracking repository on Github.
>

This was hinted at earlier in the thread.  My goal wasn't to say
whether it was or was not an issue, but bring the focus more on the
tangible impact, as I think that will probably help to make this less
about philosophy and more about impact.  I think that is more likely
to create action (whether that is a policy change or improvement to
the tooling).

-- 
Rich

Re: [gentoo-dev] Disturbing state of arch testing in Gentoo

[Joonas Niilola]

[-- Attachment #1.1: Type: text/plain, Size: 895 bytes --]

On 8.11.2022 2.23, Rich Freeman wrote:
> 
> Of course if somebody wants to contribute to 100% FOSS tinderbox
> efforts that would be even better.  Perhaps if our 100% FOSS tinderbox
> efforts addressed our needs very well, then nobody would want to
> bother with the proprietary reports, or generating them.  IMO it would
> be better to create the FOSS solution before abandoning the
> proprietary one.  Doing otherwise is basically burning bridges - it
> can be motivating in a sense but not really ideal.  I'd love to have a
> 100% FOSS solution around all of this, but I appreciate what has been
> created and can hardly criticize volunteers for failing to make it
> happen, especially since I haven't contributed to that myself.
> 

https://gitweb.gentoo.org/proj/tinderbox-cluster.git/
https://wiki.gentoo.org/wiki/Project:Tinderbox-cluster

Hopefully soon.

-- juippis

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] Disturbing state of arch testing in Gentoo

[Michał Górny]

On Mon, 2022-11-07 at 19:23 -0500, Rich Freeman wrote:
> On Mon, Nov 7, 2022 at 6:16 PM Sam James <sam@gentoo.org> wrote:
> > 
> > > On 7 Nov 2022, at 06:07, Oskari Pirhonen <xxc3ncoredxx@gmail.com> wrote:
> > > 
> > > On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote:
> > > > I would be in favour of stepping up the social contract and actually
> > > > prohibiting this kind of things, we had that before too, the nattka you
> > > > mgorny wrote is replacement for old bugzilla bot that was ...
> > > > closedsource and perished, though nattka now have way more features than
> > > > the old thing ever had.
> > > 
> > > As a user, I think it would be really cool if there was a requirement
> > > that all infra and infra-adjacent stuff was free software.
> > > 
> > > I feel like I've read that Debian already has something like this. While
> > > doing some quick searches I didn't find a full-on requirement, but all
> > > their infra bits I did find were powered by free software. The most
> > > relevant ones being buildd [1] and debci [2]. Additionally, the debci
> > > docs has inctructions on reproducing tests yourself [3] which is a nice
> > > extra IMO.
> > 
> > Gentoo has https://www.gentoo.org/get-started/philosophy/social-contract.html.
> 
> [...]
> 
> I think the key is something that was brought up earlier in the
> thread: is this causing problems?

We're talking about handling arch testing and not tinderboxing
in general but yes, this is causing problems.

If someone's running automation that takes care of a significant portion
of arch testing, it effectively leads to monopolized arch testing. 
Other arch testers don't need to do anything, so they eventually stop
paying attention and everyone assumes "X will take care of it anyway".

Now, the first problem is the bus factor.  If X stops doing arch
testing, requests pile up.  It takes time before others resume their
work.  If the software used to do the automation was proprietary, others
have to start over.  Of course, this is better now that we have
an alternative.

The second problem is that we don't really know *how* things are
processed.  As I've said, it happened to me before that stablereqs were
ignored for months.  My guess is that the automation couldn't figure out
how to process them, so it skipped them, silently.  I still don't know
what was the problem, or how to avoid it in the future.  If the code was
public, I could try figuring it out and perhaps even fixing it.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] Disturbing state of arch testing in Gentoo

[Agostino Sarubbo]

[-- Attachment #1: Type: text/plain, Size: 431 bytes --]

On martedì 8 novembre 2022 14:26:18 CET Michał Górny wrote:
> If the code was
> public, I could try figuring it out and perhaps even fixing it.

Stable requests are handled by many people. o, since your requests were 
ignored by all members and sam said that him, arthurzam, jsmolic are using 
tattoo, my guess is that you have already everything in place.

Quoting the relevant part from sam's email:



Agostino 

[-- Attachment #2: Type: text/html, Size: 1013 bytes --]

Re: [gentoo-dev] Disturbing state of arch testing in Gentoo

[John Helmert III]

[-- Attachment #1: Type: text/plain, Size: 1901 bytes --]

On Sun, Nov 06, 2022 at 09:15:40AM +0100, Michał Górny wrote:
> Hi, everyone.
> 
> Arch testing's relying on automation a lot these days.  Not saying
> that's bad, if it improves the state of affairs.  However, I have some
> concerns, based on what I've seen lately.
> 
> On top of that, it seems that most of it still relies on proprietary
> software and we have no clue how *exactly* it works, and it's really,
> really hard to get a straight answer.

As far as I can tell, there's ONE person relying completely on a
proprietary arch testing system.

Ago, could you comment on this? What's blocking you from open sourcing
your software?

I'll also point out that you removed the Github repository that you
used to tell people to report issues with your CI at, while there were
several outstanding issues. Why? I have two (of three) issue titles in
my browser history, but as I recall you never touched them:

2. Release source code
3. More information on the bug

[1] https://github.com/asarubbo/ci

> So, my questions are:
> 
> 1. Is "runtime testing required" field being respected?  Obviously not
> every package can be (sufficiently) tested via FEATURES=test, so we've
> added that fields.  However, if arch testers just ignore it and push
> things stable based on pure build testing...
> 
> 2. How are kernels being tested?  Given the speed with which new gentoo-
> sources stablereqs are handled, I really feel like "arch testing" there
> means "checking if sources install", and have little to do with working
> kernels.
> 
> 3. How does the automation handle packages that aren't trivially
> installable?  I recall that in the past stablereqs were stalled for
> months without a single comment because automation couldn't figure out
> how to proceed, and nobody bothered reporting a problem.
> 
> -- 
> Best regards,
> Michał Górny
> 
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [gentoo-dev] Disturbing state of arch testing in Gentoo

[Agostino Sarubbo]

[-- Attachment #1: Type: text/plain, Size: 1168 bytes --]

On domenica 6 novembre 2022 14:27:40 CET John Helmert III wrote:
> As far as I can tell, there's ONE person relying completely on a
> proprietary arch testing system.
> 
> Ago, could you comment on this? What's blocking you from open sourcing
> your software?

Hi,

I already answered in the previous post:

"I still use getatoms.py to fetch 'doable' stablereqs (it is on my todo 
to switch to nattka). And I have a script the **simply** does emerge over the list of 
the packages. There is nothing obscure in it."

I'm working in arch testing since 2009. In the past I relied on scripts done by someone else 
and every time there was an issue I got no response.
At a certain point I decided to make my own script in language I know so I can edit it when 
is needed.


Since few years we allow self stabilization from maintainer. Do we know how and with 
what they test? No because it is not required.
The requirement for test is that the package you are testing works as expected.

https://wiki.gentoo.org/wiki/Project:AMD64_Arch_Testers#Arch_tester.27s_policy[1]

Agostino

--------
[1] https://wiki.gentoo.org/wiki/Project:AMD64_Arch_Testers#Arch_tester.27s_policy

[-- Attachment #2: Type: text/html, Size: 2762 bytes --]

Re: [gentoo-dev] Disturbing state of arch testing in Gentoo

[John Helmert III]

[-- Attachment #1: Type: text/plain, Size: 1753 bytes --]

On Sun, Nov 06, 2022 at 08:03:16PM +0100, Agostino Sarubbo wrote:
> On domenica 6 novembre 2022 14:27:40 CET John Helmert III wrote:
> > As far as I can tell, there's ONE person relying completely on a
> > proprietary arch testing system.
> > 
> > Ago, could you comment on this? What's blocking you from open sourcing
> > your software?
> 
> Hi,
> 
> I already answered in the previous post:
> 
> "I still use getatoms.py to fetch 'doable' stablereqs (it is on my todo 
> to switch to nattka). And I have a script the **simply** does emerge over the list of 
> the packages. There is nothing obscure in it."
> 
> I'm working in arch testing since 2009. In the past I relied on scripts done by someone else 
> and every time there was an issue I got no response.

And so you force that frustration on everyone else? Why?

> At a certain point I decided to make my own script in language I know so I can edit it when 
> is needed.

None of this blocks you from open sourcing it. Is your reason for not
open-sourcing your automation really that "There is nothing obscure in
it"?

You also ignored my other question:

"I'll also point out that you removed the Github repository that you
used to tell people to report issues with your CI at, while there were
several outstanding issues. Why?"

> Since few years we allow self stabilization from maintainer. Do we know how and with 
> what they test? No because it is not required.
> The requirement for test is that the package you are testing works as expected.
> 
> https://wiki.gentoo.org/wiki/Project:AMD64_Arch_Testers#Arch_tester.27s_policy[1]
> 
> Agostino
> 
> --------
> [1] https://wiki.gentoo.org/wiki/Project:AMD64_Arch_Testers#Arch_tester.27s_policy

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [gentoo-dev] Disturbing state of arch testing in Gentoo

[Agostino Sarubbo]

[-- Attachment #1: Type: text/plain, Size: 2086 bytes --]

On domenica 6 novembre 2022 09:15:40 CET Michał Górny wrote:
> On top of that, it seems that most of it still relies on proprietary
> software and we have no clue how *exactly* it works, and it's really,
> really hard to get a straight answer.

I'm speaking for myself. I still use getatoms.py to fetch 'doable' stablereqs (it is on my todo 
to switch to nattka). And I have a script the *simply* does emerge over the list of the 
packages.
There is nothing obscure in it.


> So, my questions are:
> 
> 1. Is "runtime testing required" field being respected?  Obviously not
> every package can be (sufficiently) tested via FEATURES=test, so we've
> added that fields.  However, if arch testers just ignore it and push
> things stable based on pure build testing...

sam already provided the right answer. In addition, when we introduced runtime_testing 
and package_list fields we requested support in pybugz:

https://github.com/williamh/pybugz/issues/105[1]

There is no trace (into the github ticket) about runtime testing field because I discussed/
requested over irc.

> 2. How are kernels being tested?  Given the speed with which new gentoo-
> sources stablereqs are handled, I really feel like "arch testing" there
> means "checking if sources install", and have little to do with working
> kernels.

For amd64, I boot into the new kernel to verify that at least it boots. For other 'exotic' 
arches, since there is a lack of hardware, the rule was to verify that at least it builds (install 
if we want to use the right word).
If you think that build only is not appropriate, I can skip kernel stablereqs.


> 3. How does the automation handle packages that aren't trivially
> installable?  I recall that in the past stablereqs were stalled for
> months without a single comment because automation couldn't figure out
> how to proceed, and nobody bothered reporting a problem.

I skip them from automation and from time to time I handle it manually.


Agostino



--------
[1] https://github.com/williamh/pybugz/issues/105

[-- Attachment #2: Type: text/html, Size: 4276 bytes --]

Re: [gentoo-dev] Disturbing state of arch testing in Gentoo

[Agostino Sarubbo]

[-- Attachment #1: Type: text/plain, Size: 335 bytes --]

Hi,

Whatever outside the arch testing (like tinderbox) is off topic here since it 
is a completely different argument.

To make John Helmert III happy, I just switched to tatt; so my actual 
workflow is tatt + nattka and there is nothing more.

If there are unanswered questions about the arch testing, please let me 
know.

Agostino

[-- Attachment #2: Type: text/html, Size: 833 bytes --]

Re: [gentoo-dev] Disturbing state of arch testing in Gentoo

[John Helmert III]

[-- Attachment #1: Type: text/plain, Size: 454 bytes --]

On Tue, Nov 08, 2022 at 09:43:19AM +0100, Agostino Sarubbo wrote:
> Hi,
> 
> Whatever outside the arch testing (like tinderbox) is off topic here since it 
> is a completely different argument.
> 
> To make John Helmert III happy, I just switched to tatt; so my actual 
> workflow is tatt + nattka and there is nothing more.
> 
> If there are unanswered questions about the arch testing, please let me 
> know.
> 
> Agostino

Great, thanks!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]