* [gentoo-dev] Disturbing state of arch testing in Gentoo
@ 2022-11-06 8:15 Michał Górny
2022-11-06 8:34 ` Sam James
` (4 more replies)
0 siblings, 5 replies; 22+ messages in thread
From: Michał Górny @ 2022-11-06 8:15 UTC (permalink / raw
To: gentoo-dev
Hi, everyone.
Arch testing's relying on automation a lot these days. Not saying
that's bad, if it improves the state of affairs. However, I have some
concerns, based on what I've seen lately.
On top of that, it seems that most of it still relies on proprietary
software and we have no clue how *exactly* it works, and it's really,
really hard to get a straight answer.
So, my questions are:
1. Is "runtime testing required" field being respected? Obviously not
every package can be (sufficiently) tested via FEATURES=test, so we've
added that fields. However, if arch testers just ignore it and push
things stable based on pure build testing...
2. How are kernels being tested? Given the speed with which new gentoo-
sources stablereqs are handled, I really feel like "arch testing" there
means "checking if sources install", and have little to do with working
kernels.
3. How does the automation handle packages that aren't trivially
installable? I recall that in the past stablereqs were stalled for
months without a single comment because automation couldn't figure out
how to proceed, and nobody bothered reporting a problem.
--
Best regards,
Michał Górny
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
2022-11-06 8:15 [gentoo-dev] Disturbing state of arch testing in Gentoo Michał Górny
@ 2022-11-06 8:34 ` Sam James
2022-11-08 4:58 ` Arthur Zamarin
2022-11-06 10:37 ` Piotr Karbowski
` (3 subsequent siblings)
4 siblings, 1 reply; 22+ messages in thread
From: Sam James @ 2022-11-06 8:34 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 2175 bytes --]
> On 6 Nov 2022, at 08:15, Michał Górny <mgorny@gentoo.org> wrote:
>
> Hi, everyone.
>
> Arch testing's relying on automation a lot these days. Not saying
> that's bad, if it improves the state of affairs. However, I have some
> concerns, based on what I've seen lately.
Thanks for starting this discussion, I think others have felt this way too.
>
> On top of that, it seems that most of it still relies on proprietary
> software and we have no clue how *exactly* it works, and it's really,
> really hard to get a straight answer.
>
arthurzam, jsmolic, and I are using https://github.com/arthurzam/tattoo.
> So, my questions are:
>
> 1. Is "runtime testing required" field being respected? Obviously not
> every package can be (sufficiently) tested via FEATURES=test, so we've
> added that fields. However, if arch testers just ignore it and push
> things stable based on pure build testing...
Not right now. We discussed it on #gentoo-dev maybe 2 months ago
or so but concluded we needed nattka support to fix up our automation.
That had two parts:
1. https://github.com/projg2/nattka/issues/72 & https://github.com/projg2/nattka/pull/73 (done)
2. https://github.com/arthurzam/tattoo/issues/1 (not done)
>
> 2. How are kernels being tested? Given the speed with which new gentoo-
> sources stablereqs are handled, I really feel like "arch testing" there
> means "checking if sources install", and have little to do with working
> kernels.
>
I usually blacklist gentoo-sources because I can't actually test it, or
if I do stable it, I've tried to run them for real.
For gentoo-kernel, I run src_test.
> 3. How does the automation handle packages that aren't trivially
> installable? I recall that in the past stablereqs were stalled for
> months without a single comment because automation couldn't figure out
> how to proceed, and nobody bothered reporting a problem.
It doesn't, really. I think at the very least we need to try do world
upgrades after adding the package list to package.accept_keywords,
as stuff like LLVM and GNOME bugs won't work without it (blockers etc).
Best,
sam
[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 358 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
2022-11-06 8:15 [gentoo-dev] Disturbing state of arch testing in Gentoo Michał Górny
2022-11-06 8:34 ` Sam James
@ 2022-11-06 10:37 ` Piotr Karbowski
2022-11-07 6:07 ` Oskari Pirhonen
2022-11-06 13:27 ` John Helmert III
` (2 subsequent siblings)
4 siblings, 1 reply; 22+ messages in thread
From: Piotr Karbowski @ 2022-11-06 10:37 UTC (permalink / raw
To: gentoo-dev
Hi,
On 06/11/2022 09.15, Michał Górny wrote:
> On top of that, it seems that most of it still relies on proprietary
> software and we have no clue how*exactly* it works, and it's really,
> really hard to get a straight answer.
I never understood how it become socially acceptable in open source
project that Gentoo is that one of the important automation that does
tinderboxing is closed source and gate kept, meaning no one can on their
own reproduce the failures, which is utterly annoying when your normal
testing just works, yet their testing methodology is a secret.
I would be in favour of stepping up the social contract and actually
prohibiting this kind of things, we had that before too, the nattka you
mgorny wrote is replacement for old bugzilla bot that was ...
closedsource and perished, though nattka now have way more features than
the old thing ever had.
-- Piotr.
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
2022-11-06 8:15 [gentoo-dev] Disturbing state of arch testing in Gentoo Michał Górny
2022-11-06 8:34 ` Sam James
2022-11-06 10:37 ` Piotr Karbowski
@ 2022-11-06 13:27 ` John Helmert III
2022-11-06 19:03 ` Agostino Sarubbo
2022-11-06 18:50 ` Agostino Sarubbo
2022-11-08 8:43 ` Agostino Sarubbo
4 siblings, 1 reply; 22+ messages in thread
From: John Helmert III @ 2022-11-06 13:27 UTC (permalink / raw
To: ago; +Cc: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 1901 bytes --]
On Sun, Nov 06, 2022 at 09:15:40AM +0100, Michał Górny wrote:
> Hi, everyone.
>
> Arch testing's relying on automation a lot these days. Not saying
> that's bad, if it improves the state of affairs. However, I have some
> concerns, based on what I've seen lately.
>
> On top of that, it seems that most of it still relies on proprietary
> software and we have no clue how *exactly* it works, and it's really,
> really hard to get a straight answer.
As far as I can tell, there's ONE person relying completely on a
proprietary arch testing system.
Ago, could you comment on this? What's blocking you from open sourcing
your software?
I'll also point out that you removed the Github repository that you
used to tell people to report issues with your CI at, while there were
several outstanding issues. Why? I have two (of three) issue titles in
my browser history, but as I recall you never touched them:
2. Release source code
3. More information on the bug
[1] https://github.com/asarubbo/ci
> So, my questions are:
>
> 1. Is "runtime testing required" field being respected? Obviously not
> every package can be (sufficiently) tested via FEATURES=test, so we've
> added that fields. However, if arch testers just ignore it and push
> things stable based on pure build testing...
>
> 2. How are kernels being tested? Given the speed with which new gentoo-
> sources stablereqs are handled, I really feel like "arch testing" there
> means "checking if sources install", and have little to do with working
> kernels.
>
> 3. How does the automation handle packages that aren't trivially
> installable? I recall that in the past stablereqs were stalled for
> months without a single comment because automation couldn't figure out
> how to proceed, and nobody bothered reporting a problem.
>
> --
> Best regards,
> Michał Górny
>
>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
2022-11-06 8:15 [gentoo-dev] Disturbing state of arch testing in Gentoo Michał Górny
` (2 preceding siblings ...)
2022-11-06 13:27 ` John Helmert III
@ 2022-11-06 18:50 ` Agostino Sarubbo
2022-11-08 8:43 ` Agostino Sarubbo
4 siblings, 0 replies; 22+ messages in thread
From: Agostino Sarubbo @ 2022-11-06 18:50 UTC (permalink / raw
To: gentoo-dev; +Cc: Michał Górny
[-- Attachment #1: Type: text/plain, Size: 2086 bytes --]
On domenica 6 novembre 2022 09:15:40 CET Michał Górny wrote:
> On top of that, it seems that most of it still relies on proprietary
> software and we have no clue how *exactly* it works, and it's really,
> really hard to get a straight answer.
I'm speaking for myself. I still use getatoms.py to fetch 'doable' stablereqs (it is on my todo
to switch to nattka). And I have a script the *simply* does emerge over the list of the
packages.
There is nothing obscure in it.
> So, my questions are:
>
> 1. Is "runtime testing required" field being respected? Obviously not
> every package can be (sufficiently) tested via FEATURES=test, so we've
> added that fields. However, if arch testers just ignore it and push
> things stable based on pure build testing...
sam already provided the right answer. In addition, when we introduced runtime_testing
and package_list fields we requested support in pybugz:
https://github.com/williamh/pybugz/issues/105[1]
There is no trace (into the github ticket) about runtime testing field because I discussed/
requested over irc.
> 2. How are kernels being tested? Given the speed with which new gentoo-
> sources stablereqs are handled, I really feel like "arch testing" there
> means "checking if sources install", and have little to do with working
> kernels.
For amd64, I boot into the new kernel to verify that at least it boots. For other 'exotic'
arches, since there is a lack of hardware, the rule was to verify that at least it builds (install
if we want to use the right word).
If you think that build only is not appropriate, I can skip kernel stablereqs.
> 3. How does the automation handle packages that aren't trivially
> installable? I recall that in the past stablereqs were stalled for
> months without a single comment because automation couldn't figure out
> how to proceed, and nobody bothered reporting a problem.
I skip them from automation and from time to time I handle it manually.
Agostino
--------
[1] https://github.com/williamh/pybugz/issues/105
[-- Attachment #2: Type: text/html, Size: 4276 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
2022-11-06 13:27 ` John Helmert III
@ 2022-11-06 19:03 ` Agostino Sarubbo
2022-11-06 19:15 ` John Helmert III
0 siblings, 1 reply; 22+ messages in thread
From: Agostino Sarubbo @ 2022-11-06 19:03 UTC (permalink / raw
To: John Helmert III; +Cc: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 1168 bytes --]
On domenica 6 novembre 2022 14:27:40 CET John Helmert III wrote:
> As far as I can tell, there's ONE person relying completely on a
> proprietary arch testing system.
>
> Ago, could you comment on this? What's blocking you from open sourcing
> your software?
Hi,
I already answered in the previous post:
"I still use getatoms.py to fetch 'doable' stablereqs (it is on my todo
to switch to nattka). And I have a script the **simply** does emerge over the list of
the packages. There is nothing obscure in it."
I'm working in arch testing since 2009. In the past I relied on scripts done by someone else
and every time there was an issue I got no response.
At a certain point I decided to make my own script in language I know so I can edit it when
is needed.
Since few years we allow self stabilization from maintainer. Do we know how and with
what they test? No because it is not required.
The requirement for test is that the package you are testing works as expected.
https://wiki.gentoo.org/wiki/Project:AMD64_Arch_Testers#Arch_tester.27s_policy[1]
Agostino
--------
[1] https://wiki.gentoo.org/wiki/Project:AMD64_Arch_Testers#Arch_tester.27s_policy
[-- Attachment #2: Type: text/html, Size: 2762 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
2022-11-06 19:03 ` Agostino Sarubbo
@ 2022-11-06 19:15 ` John Helmert III
0 siblings, 0 replies; 22+ messages in thread
From: John Helmert III @ 2022-11-06 19:15 UTC (permalink / raw
To: ago; +Cc: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 1753 bytes --]
On Sun, Nov 06, 2022 at 08:03:16PM +0100, Agostino Sarubbo wrote:
> On domenica 6 novembre 2022 14:27:40 CET John Helmert III wrote:
> > As far as I can tell, there's ONE person relying completely on a
> > proprietary arch testing system.
> >
> > Ago, could you comment on this? What's blocking you from open sourcing
> > your software?
>
> Hi,
>
> I already answered in the previous post:
>
> "I still use getatoms.py to fetch 'doable' stablereqs (it is on my todo
> to switch to nattka). And I have a script the **simply** does emerge over the list of
> the packages. There is nothing obscure in it."
>
> I'm working in arch testing since 2009. In the past I relied on scripts done by someone else
> and every time there was an issue I got no response.
And so you force that frustration on everyone else? Why?
> At a certain point I decided to make my own script in language I know so I can edit it when
> is needed.
None of this blocks you from open sourcing it. Is your reason for not
open-sourcing your automation really that "There is nothing obscure in
it"?
You also ignored my other question:
"I'll also point out that you removed the Github repository that you
used to tell people to report issues with your CI at, while there were
several outstanding issues. Why?"
> Since few years we allow self stabilization from maintainer. Do we know how and with
> what they test? No because it is not required.
> The requirement for test is that the package you are testing works as expected.
>
> https://wiki.gentoo.org/wiki/Project:AMD64_Arch_Testers#Arch_tester.27s_policy[1]
>
> Agostino
>
> --------
> [1] https://wiki.gentoo.org/wiki/Project:AMD64_Arch_Testers#Arch_tester.27s_policy
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
2022-11-06 10:37 ` Piotr Karbowski
@ 2022-11-07 6:07 ` Oskari Pirhonen
2022-11-07 6:26 ` Joonas Niilola
2022-11-07 23:16 ` Sam James
0 siblings, 2 replies; 22+ messages in thread
From: Oskari Pirhonen @ 2022-11-07 6:07 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 1014 bytes --]
On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote:
> I would be in favour of stepping up the social contract and actually
> prohibiting this kind of things, we had that before too, the nattka you
> mgorny wrote is replacement for old bugzilla bot that was ...
> closedsource and perished, though nattka now have way more features than
> the old thing ever had.
As a user, I think it would be really cool if there was a requirement
that all infra and infra-adjacent stuff was free software.
I feel like I've read that Debian already has something like this. While
doing some quick searches I didn't find a full-on requirement, but all
their infra bits I did find were powered by free software. The most
relevant ones being buildd [1] and debci [2]. Additionally, the debci
docs has inctructions on reproducing tests yourself [3] which is a nice
extra IMO.
[1] https://buildd.debian.org
[2] https://ci.debian.net
[3] https://ci.debian.net/doc/file.MAINTAINERS.html
- Oskari
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
2022-11-07 6:07 ` Oskari Pirhonen
@ 2022-11-07 6:26 ` Joonas Niilola
2022-11-07 15:13 ` John Helmert III
2022-11-08 7:40 ` Oskari Pirhonen
2022-11-07 23:16 ` Sam James
1 sibling, 2 replies; 22+ messages in thread
From: Joonas Niilola @ 2022-11-07 6:26 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1.1: Type: text/plain, Size: 1599 bytes --]
On 7.11.2022 8.07, Oskari Pirhonen wrote:
> On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote:
>> I would be in favour of stepping up the social contract and actually
>> prohibiting this kind of things, we had that before too, the nattka you
>> mgorny wrote is replacement for old bugzilla bot that was ...
>> closedsource and perished, though nattka now have way more features than
>> the old thing ever had.
>
> As a user, I think it would be really cool if there was a requirement
> that all infra and infra-adjacent stuff was free software.
>
> I feel like I've read that Debian already has something like this. While
> doing some quick searches I didn't find a full-on requirement, but all
> their infra bits I did find were powered by free software. The most
> relevant ones being buildd [1] and debci [2]. Additionally, the debci
> docs has inctructions on reproducing tests yourself [3] which is a nice
> extra IMO.
>
> [1] https://buildd.debian.org
> [2] https://ci.debian.net
> [3] https://ci.debian.net/doc/file.MAINTAINERS.html
>
> - Oskari
I _believe_ ago's tinderbox isn't being paid by the GF _anymore_ due to
this reason, but he keeps it running with his own expenses. I don't mind
this as long as the results are desirable and not phony. I still see a
lot of value in most of ago's work.
It is unfortunate we don't get to see the engine behind and copy it,
since I'd be really interested in using his automated bug search /
report tool. Even tattoo (https://github.com/arthurzam/tattoo) lacks
that at the moment.
-- juippis
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 618 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
2022-11-07 6:26 ` Joonas Niilola
@ 2022-11-07 15:13 ` John Helmert III
2022-11-08 7:40 ` Oskari Pirhonen
1 sibling, 0 replies; 22+ messages in thread
From: John Helmert III @ 2022-11-07 15:13 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 1845 bytes --]
On Mon, Nov 07, 2022 at 08:26:15AM +0200, Joonas Niilola wrote:
> On 7.11.2022 8.07, Oskari Pirhonen wrote:
> > On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote:
> >> I would be in favour of stepping up the social contract and actually
> >> prohibiting this kind of things, we had that before too, the nattka you
> >> mgorny wrote is replacement for old bugzilla bot that was ...
> >> closedsource and perished, though nattka now have way more features than
> >> the old thing ever had.
> >
> > As a user, I think it would be really cool if there was a requirement
> > that all infra and infra-adjacent stuff was free software.
> >
> > I feel like I've read that Debian already has something like this. While
> > doing some quick searches I didn't find a full-on requirement, but all
> > their infra bits I did find were powered by free software. The most
> > relevant ones being buildd [1] and debci [2]. Additionally, the debci
> > docs has inctructions on reproducing tests yourself [3] which is a nice
> > extra IMO.
> >
> > [1] https://buildd.debian.org
> > [2] https://ci.debian.net
> > [3] https://ci.debian.net/doc/file.MAINTAINERS.html
> >
> > - Oskari
>
> I _believe_ ago's tinderbox isn't being paid by the GF _anymore_ due to
> this reason, but he keeps it running with his own expenses. I don't mind
> this as long as the results are desirable and not phony. I still see a
> lot of value in most of ago's work.
He's not using infra's AWS resources anymore, but he's still using the
devbox.amd64.dev.gentoo.org VM.
> It is unfortunate we don't get to see the engine behind and copy it,
> since I'd be really interested in using his automated bug search /
> report tool. Even tattoo (https://github.com/arthurzam/tattoo) lacks
> that at the moment.
>
> -- juippis
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
2022-11-07 6:07 ` Oskari Pirhonen
2022-11-07 6:26 ` Joonas Niilola
@ 2022-11-07 23:16 ` Sam James
2022-11-08 0:23 ` Rich Freeman
1 sibling, 1 reply; 22+ messages in thread
From: Sam James @ 2022-11-07 23:16 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 1092 bytes --]
> On 7 Nov 2022, at 06:07, Oskari Pirhonen <xxc3ncoredxx@gmail.com> wrote:
>
> On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote:
>> I would be in favour of stepping up the social contract and actually
>> prohibiting this kind of things, we had that before too, the nattka you
>> mgorny wrote is replacement for old bugzilla bot that was ...
>> closedsource and perished, though nattka now have way more features than
>> the old thing ever had.
>
> As a user, I think it would be really cool if there was a requirement
> that all infra and infra-adjacent stuff was free software.
>
> I feel like I've read that Debian already has something like this. While
> doing some quick searches I didn't find a full-on requirement, but all
> their infra bits I did find were powered by free software. The most
> relevant ones being buildd [1] and debci [2]. Additionally, the debci
> docs has inctructions on reproducing tests yourself [3] which is a nice
> extra IMO.
Gentoo has https://www.gentoo.org/get-started/philosophy/social-contract.html.
Best,
sam
[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 358 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
2022-11-07 23:16 ` Sam James
@ 2022-11-08 0:23 ` Rich Freeman
2022-11-08 0:30 ` Sam James
` (3 more replies)
0 siblings, 4 replies; 22+ messages in thread
From: Rich Freeman @ 2022-11-08 0:23 UTC (permalink / raw
To: gentoo-dev
On Mon, Nov 7, 2022 at 6:16 PM Sam James <sam@gentoo.org> wrote:
>
> > On 7 Nov 2022, at 06:07, Oskari Pirhonen <xxc3ncoredxx@gmail.com> wrote:
> >
> > On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote:
> >> I would be in favour of stepping up the social contract and actually
> >> prohibiting this kind of things, we had that before too, the nattka you
> >> mgorny wrote is replacement for old bugzilla bot that was ...
> >> closedsource and perished, though nattka now have way more features than
> >> the old thing ever had.
> >
> > As a user, I think it would be really cool if there was a requirement
> > that all infra and infra-adjacent stuff was free software.
> >
> > I feel like I've read that Debian already has something like this. While
> > doing some quick searches I didn't find a full-on requirement, but all
> > their infra bits I did find were powered by free software. The most
> > relevant ones being buildd [1] and debci [2]. Additionally, the debci
> > docs has inctructions on reproducing tests yourself [3] which is a nice
> > extra IMO.
>
> Gentoo has https://www.gentoo.org/get-started/philosophy/social-contract.html.
I feel like something like a dev-run tinderbox is a bit out of the
scope of that.
Suppose I file a bug against a package, pointing out some issue in it.
How do you know I didn't use some proprietary static code analysis
tool to discover that error? Does it even really matter? The bug
speaks for itself. It is like worrying about whether somebody who
filed a bug was running Windows or another proprietary OS or browser
on their desktop.
Well, a tinderbox is just an automated process for doing just that.
We don't require any dev to use a proprietary tinderbox before
committing. It is something that individual devs choose to use for
themselves, automating the testing workflow and possibly the
submission of bugs.
I think the key is something that was brought up earlier in the
thread: is this causing problems? If somebody is running some tool
against the repository and automatically filing bugs, and those bugs
are not useful/actionable and waste the time of volunteers, then that
is a problem. Proprietary tools do contribute to this since they can
generate results that are harder to reproduce, but if they are clear
and accurate and actionable it could still be a net-positive.
Of course if somebody wants to contribute to 100% FOSS tinderbox
efforts that would be even better. Perhaps if our 100% FOSS tinderbox
efforts addressed our needs very well, then nobody would want to
bother with the proprietary reports, or generating them. IMO it would
be better to create the FOSS solution before abandoning the
proprietary one. Doing otherwise is basically burning bridges - it
can be motivating in a sense but not really ideal. I'd love to have a
100% FOSS solution around all of this, but I appreciate what has been
created and can hardly criticize volunteers for failing to make it
happen, especially since I haven't contributed to that myself.
--
Rich
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
2022-11-08 0:23 ` Rich Freeman
@ 2022-11-08 0:30 ` Sam James
2022-11-08 0:34 ` John Helmert III
` (2 subsequent siblings)
3 siblings, 0 replies; 22+ messages in thread
From: Sam James @ 2022-11-08 0:30 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 1911 bytes --]
> On 8 Nov 2022, at 00:23, Rich Freeman <rich0@gentoo.org> wrote:
>
> On Mon, Nov 7, 2022 at 6:16 PM Sam James <sam@gentoo.org> wrote:
>>
>>> On 7 Nov 2022, at 06:07, Oskari Pirhonen <xxc3ncoredxx@gmail.com> wrote:
>>>
>>> On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote:
>>>> I would be in favour of stepping up the social contract and actually
>>>> prohibiting this kind of things, we had that before too, the nattka you
>>>> mgorny wrote is replacement for old bugzilla bot that was ...
>>>> closedsource and perished, though nattka now have way more features than
>>>> the old thing ever had.
>>>
>>> As a user, I think it would be really cool if there was a requirement
>>> that all infra and infra-adjacent stuff was free software.
>>>
>>> I feel like I've read that Debian already has something like this. While
>>> doing some quick searches I didn't find a full-on requirement, but all
>>> their infra bits I did find were powered by free software. The most
>>> relevant ones being buildd [1] and debci [2]. Additionally, the debci
>>> docs has inctructions on reproducing tests yourself [3] which is a nice
>>> extra IMO.
>>
>> Gentoo has https://www.gentoo.org/get-started/philosophy/social-contract.html.
>
> I feel like something like a dev-run tinderbox is a bit out of the
> scope of that.
I intentionally didn't comment on the scope for now, but I'm glad you did.
>
> Suppose I file a bug against a package, pointing out some issue in it.
> How do you know I didn't use some proprietary static code analysis
> tool to discover that error? Does it even really matter? The bug
> speaks for itself. It is like worrying about whether somebody who
> filed a bug was running Windows or another proprietary OS or browser
> on their desktop.
>
It matters if someone can't then reproduce the bug which happens
somewhat often here.
[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 358 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
2022-11-08 0:23 ` Rich Freeman
2022-11-08 0:30 ` Sam James
@ 2022-11-08 0:34 ` John Helmert III
2022-11-08 0:54 ` Rich Freeman
2022-11-08 6:06 ` Joonas Niilola
2022-11-08 13:26 ` Michał Górny
3 siblings, 1 reply; 22+ messages in thread
From: John Helmert III @ 2022-11-08 0:34 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 3424 bytes --]
On Mon, Nov 07, 2022 at 07:23:33PM -0500, Rich Freeman wrote:
> On Mon, Nov 7, 2022 at 6:16 PM Sam James <sam@gentoo.org> wrote:
> >
> > > On 7 Nov 2022, at 06:07, Oskari Pirhonen <xxc3ncoredxx@gmail.com> wrote:
> > >
> > > On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote:
> > >> I would be in favour of stepping up the social contract and actually
> > >> prohibiting this kind of things, we had that before too, the nattka you
> > >> mgorny wrote is replacement for old bugzilla bot that was ...
> > >> closedsource and perished, though nattka now have way more features than
> > >> the old thing ever had.
> > >
> > > As a user, I think it would be really cool if there was a requirement
> > > that all infra and infra-adjacent stuff was free software.
> > >
> > > I feel like I've read that Debian already has something like this. While
> > > doing some quick searches I didn't find a full-on requirement, but all
> > > their infra bits I did find were powered by free software. The most
> > > relevant ones being buildd [1] and debci [2]. Additionally, the debci
> > > docs has inctructions on reproducing tests yourself [3] which is a nice
> > > extra IMO.
> >
> > Gentoo has https://www.gentoo.org/get-started/philosophy/social-contract.html.
>
> I feel like something like a dev-run tinderbox is a bit out of the
> scope of that.
>
> Suppose I file a bug against a package, pointing out some issue in it.
> How do you know I didn't use some proprietary static code analysis
> tool to discover that error? Does it even really matter? The bug
> speaks for itself. It is like worrying about whether somebody who
> filed a bug was running Windows or another proprietary OS or browser
> on their desktop.
>
> Well, a tinderbox is just an automated process for doing just that.
> We don't require any dev to use a proprietary tinderbox before
> committing. It is something that individual devs choose to use for
> themselves, automating the testing workflow and possibly the
> submission of bugs.
>
> I think the key is something that was brought up earlier in the
> thread: is this causing problems? If somebody is running some tool
> against the repository and automatically filing bugs, and those bugs
> are not useful/actionable and waste the time of volunteers, then that
> is a problem. Proprietary tools do contribute to this since they can
> generate results that are harder to reproduce, but if they are clear
> and accurate and actionable it could still be a net-positive.
In some cases, yes, this is exactly the problem. This was one of the
bugs reported in the now-deleted issue tracking repository on Github.
> Of course if somebody wants to contribute to 100% FOSS tinderbox
> efforts that would be even better. Perhaps if our 100% FOSS tinderbox
> efforts addressed our needs very well, then nobody would want to
> bother with the proprietary reports, or generating them. IMO it would
> be better to create the FOSS solution before abandoning the
> proprietary one. Doing otherwise is basically burning bridges - it
> can be motivating in a sense but not really ideal. I'd love to have a
> 100% FOSS solution around all of this, but I appreciate what has been
> created and can hardly criticize volunteers for failing to make it
> happen, especially since I haven't contributed to that myself.
>
> --
> Rich
>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
2022-11-08 0:34 ` John Helmert III
@ 2022-11-08 0:54 ` Rich Freeman
0 siblings, 0 replies; 22+ messages in thread
From: Rich Freeman @ 2022-11-08 0:54 UTC (permalink / raw
To: gentoo-dev
On Mon, Nov 7, 2022 at 7:34 PM John Helmert III <ajak@gentoo.org> wrote:
>
> On Mon, Nov 07, 2022 at 07:23:33PM -0500, Rich Freeman wrote:
> > Proprietary tools do contribute to this since they can
> > generate results that are harder to reproduce, but if they are clear
> > and accurate and actionable it could still be a net-positive.
>
> In some cases, yes, this is exactly the problem. This was one of the
> bugs reported in the now-deleted issue tracking repository on Github.
>
This was hinted at earlier in the thread. My goal wasn't to say
whether it was or was not an issue, but bring the focus more on the
tangible impact, as I think that will probably help to make this less
about philosophy and more about impact. I think that is more likely
to create action (whether that is a policy change or improvement to
the tooling).
--
Rich
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
2022-11-06 8:34 ` Sam James
@ 2022-11-08 4:58 ` Arthur Zamarin
0 siblings, 0 replies; 22+ messages in thread
From: Arthur Zamarin @ 2022-11-08 4:58 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1.1: Type: text/plain, Size: 1433 bytes --]
On 06/11/2022 10.34, Sam James wrote:
>
> ...
>
> That had two parts:
> 1. https://github.com/projg2/nattka/issues/72 & https://github.com/projg2/nattka/pull/73 (done)
> 2. https://github.com/arthurzam/tattoo/issues/1 (not done)
I was waiting for nattka-0.4 (which returns the field value) and was
hoping to wait until it becomes stable, so it will be able to upgrade
nattka on all devboxes easily and just use new API. Seeing this
conversation, made me understand that it is desired to do it ASAP, so I
added it now with a little ugly code around (import guards to check that
nattka supports the new field).
So currently tattoo skips all bugs marked with Manual testing.
I also plan to create a small dashboard showing special cases of Arch
Testing bugs, such as:
1. Bugs with only one arch remaining
2. Bugs blocked by blocker bug
3. Bugs without any info and activity (not blocked, untouched, not done)
This is in hopes to have a more organized and priority bugs, to mitigate
cases when something somewhere failed, and we lose the bug until pinged.
My current plan is to have a script that generated a HTML file that I
will host on ~dev space, and will have a scheduled process to regenerate
the HTML. I'm still planning the solution and how to schedule it, so no
promises :)
--
Arthur Zamarin
arthurzam@gentoo.org
Gentoo Linux developer (Python, Arch Teams, pkgcore stack, GURU)
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
2022-11-08 0:23 ` Rich Freeman
2022-11-08 0:30 ` Sam James
2022-11-08 0:34 ` John Helmert III
@ 2022-11-08 6:06 ` Joonas Niilola
2022-11-08 13:26 ` Michał Górny
3 siblings, 0 replies; 22+ messages in thread
From: Joonas Niilola @ 2022-11-08 6:06 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1.1: Type: text/plain, Size: 895 bytes --]
On 8.11.2022 2.23, Rich Freeman wrote:
>
> Of course if somebody wants to contribute to 100% FOSS tinderbox
> efforts that would be even better. Perhaps if our 100% FOSS tinderbox
> efforts addressed our needs very well, then nobody would want to
> bother with the proprietary reports, or generating them. IMO it would
> be better to create the FOSS solution before abandoning the
> proprietary one. Doing otherwise is basically burning bridges - it
> can be motivating in a sense but not really ideal. I'd love to have a
> 100% FOSS solution around all of this, but I appreciate what has been
> created and can hardly criticize volunteers for failing to make it
> happen, especially since I haven't contributed to that myself.
>
https://gitweb.gentoo.org/proj/tinderbox-cluster.git/
https://wiki.gentoo.org/wiki/Project:Tinderbox-cluster
Hopefully soon.
-- juippis
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 618 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
2022-11-07 6:26 ` Joonas Niilola
2022-11-07 15:13 ` John Helmert III
@ 2022-11-08 7:40 ` Oskari Pirhonen
1 sibling, 0 replies; 22+ messages in thread
From: Oskari Pirhonen @ 2022-11-08 7:40 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 880 bytes --]
On Mon, Nov 07, 2022 at 08:26:15 +0200, Joonas Niilola wrote:
> I _believe_ ago's tinderbox isn't being paid by the GF _anymore_ due to
> this reason, but he keeps it running with his own expenses. I don't mind
> this as long as the results are desirable and not phony. I still see a
> lot of value in most of ago's work.
>
Don't get me wrong, I really appreaciate the work that all of you do to
keep my favorite distro running. And maintaining the level of quality
that I've grown to expect from Gentoo over the years too :)
> It is unfortunate we don't get to see the engine behind and copy it,
> since I'd be really interested in using his automated bug search /
> report tool. Even tattoo (https://github.com/arthurzam/tattoo) lacks
> that at the moment.
>
Perhaps he would be open to the idea of releasing some parts of it? (pun
intended)
- Oskari
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
2022-11-06 8:15 [gentoo-dev] Disturbing state of arch testing in Gentoo Michał Górny
` (3 preceding siblings ...)
2022-11-06 18:50 ` Agostino Sarubbo
@ 2022-11-08 8:43 ` Agostino Sarubbo
2022-11-08 15:18 ` John Helmert III
4 siblings, 1 reply; 22+ messages in thread
From: Agostino Sarubbo @ 2022-11-08 8:43 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 335 bytes --]
Hi,
Whatever outside the arch testing (like tinderbox) is off topic here since it
is a completely different argument.
To make John Helmert III happy, I just switched to tatt; so my actual
workflow is tatt + nattka and there is nothing more.
If there are unanswered questions about the arch testing, please let me
know.
Agostino
[-- Attachment #2: Type: text/html, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
2022-11-08 0:23 ` Rich Freeman
` (2 preceding siblings ...)
2022-11-08 6:06 ` Joonas Niilola
@ 2022-11-08 13:26 ` Michał Górny
2022-11-08 13:55 ` Agostino Sarubbo
3 siblings, 1 reply; 22+ messages in thread
From: Michał Górny @ 2022-11-08 13:26 UTC (permalink / raw
To: gentoo-dev
On Mon, 2022-11-07 at 19:23 -0500, Rich Freeman wrote:
> On Mon, Nov 7, 2022 at 6:16 PM Sam James <sam@gentoo.org> wrote:
> >
> > > On 7 Nov 2022, at 06:07, Oskari Pirhonen <xxc3ncoredxx@gmail.com> wrote:
> > >
> > > On Sun, Nov 06, 2022 at 11:37:24 +0100, Piotr Karbowski wrote:
> > > > I would be in favour of stepping up the social contract and actually
> > > > prohibiting this kind of things, we had that before too, the nattka you
> > > > mgorny wrote is replacement for old bugzilla bot that was ...
> > > > closedsource and perished, though nattka now have way more features than
> > > > the old thing ever had.
> > >
> > > As a user, I think it would be really cool if there was a requirement
> > > that all infra and infra-adjacent stuff was free software.
> > >
> > > I feel like I've read that Debian already has something like this. While
> > > doing some quick searches I didn't find a full-on requirement, but all
> > > their infra bits I did find were powered by free software. The most
> > > relevant ones being buildd [1] and debci [2]. Additionally, the debci
> > > docs has inctructions on reproducing tests yourself [3] which is a nice
> > > extra IMO.
> >
> > Gentoo has https://www.gentoo.org/get-started/philosophy/social-contract.html.
>
> [...]
>
> I think the key is something that was brought up earlier in the
> thread: is this causing problems?
We're talking about handling arch testing and not tinderboxing
in general but yes, this is causing problems.
If someone's running automation that takes care of a significant portion
of arch testing, it effectively leads to monopolized arch testing.
Other arch testers don't need to do anything, so they eventually stop
paying attention and everyone assumes "X will take care of it anyway".
Now, the first problem is the bus factor. If X stops doing arch
testing, requests pile up. It takes time before others resume their
work. If the software used to do the automation was proprietary, others
have to start over. Of course, this is better now that we have
an alternative.
The second problem is that we don't really know *how* things are
processed. As I've said, it happened to me before that stablereqs were
ignored for months. My guess is that the automation couldn't figure out
how to process them, so it skipped them, silently. I still don't know
what was the problem, or how to avoid it in the future. If the code was
public, I could try figuring it out and perhaps even fixing it.
--
Best regards,
Michał Górny
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
2022-11-08 13:26 ` Michał Górny
@ 2022-11-08 13:55 ` Agostino Sarubbo
0 siblings, 0 replies; 22+ messages in thread
From: Agostino Sarubbo @ 2022-11-08 13:55 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 431 bytes --]
On martedì 8 novembre 2022 14:26:18 CET Michał Górny wrote:
> If the code was
> public, I could try figuring it out and perhaps even fixing it.
Stable requests are handled by many people. o, since your requests were
ignored by all members and sam said that him, arthurzam, jsmolic are using
tattoo, my guess is that you have already everything in place.
Quoting the relevant part from sam's email:
Agostino
[-- Attachment #2: Type: text/html, Size: 1013 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [gentoo-dev] Disturbing state of arch testing in Gentoo
2022-11-08 8:43 ` Agostino Sarubbo
@ 2022-11-08 15:18 ` John Helmert III
0 siblings, 0 replies; 22+ messages in thread
From: John Helmert III @ 2022-11-08 15:18 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 454 bytes --]
On Tue, Nov 08, 2022 at 09:43:19AM +0100, Agostino Sarubbo wrote:
> Hi,
>
> Whatever outside the arch testing (like tinderbox) is off topic here since it
> is a completely different argument.
>
> To make John Helmert III happy, I just switched to tatt; so my actual
> workflow is tatt + nattka and there is nothing more.
>
> If there are unanswered questions about the arch testing, please let me
> know.
>
> Agostino
Great, thanks!
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2022-11-08 15:18 UTC | newest]
Thread overview: 22+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-11-06 8:15 [gentoo-dev] Disturbing state of arch testing in Gentoo Michał Górny
2022-11-06 8:34 ` Sam James
2022-11-08 4:58 ` Arthur Zamarin
2022-11-06 10:37 ` Piotr Karbowski
2022-11-07 6:07 ` Oskari Pirhonen
2022-11-07 6:26 ` Joonas Niilola
2022-11-07 15:13 ` John Helmert III
2022-11-08 7:40 ` Oskari Pirhonen
2022-11-07 23:16 ` Sam James
2022-11-08 0:23 ` Rich Freeman
2022-11-08 0:30 ` Sam James
2022-11-08 0:34 ` John Helmert III
2022-11-08 0:54 ` Rich Freeman
2022-11-08 6:06 ` Joonas Niilola
2022-11-08 13:26 ` Michał Górny
2022-11-08 13:55 ` Agostino Sarubbo
2022-11-06 13:27 ` John Helmert III
2022-11-06 19:03 ` Agostino Sarubbo
2022-11-06 19:15 ` John Helmert III
2022-11-06 18:50 ` Agostino Sarubbo
2022-11-08 8:43 ` Agostino Sarubbo
2022-11-08 15:18 ` John Helmert III
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox