[gentoo-dev] Improving the stabilisation process - part 1

[gentoo-dev] Improving the stabilisation process - part 1

[Michael Palimaka]

As I am sure everyone is aware by now, stabilisation requests on many
architectures take a long time to be actioned. There are many factors
contributing to this, but today I'd like to address three specific
problems that unnecessarily delay developers actioning those requests.:

1. There's no easy way to pick atoms out of a stabilisation bug.
Currently, atoms can appear, in a varying formations, in multiple
locations: in the bug title, spread across multiple comments, in an
attachment, [...].

2. There's no way to determine if a stabilisation request is valid (will
pass repoman) until someone actually tries it.

3. There's no easy way to identify the level of testing required for any
given request.

To address this, I am proposing some changes to Bugzilla, some
automation, and some improved tools.


Bugzilla changes
================

I'd like to introduce two new fields:

1. "Runtime testing required" - a drop-down list with three values:
        * (unset) - the default. The stabilising developer will use
their best judgement as to what testing is required
        * Yes - the maintainer is specifically requesting runtime testing
        * No - the maintainer is happy to stabilise with only a build
test (for example, trivial revision bumps, libraries with good test
coverage, or otherwise at the maintainer's discretion)

2. "Atoms to stabilise" - a textarea containing a list of fully
qualified atoms. Each atom may optionally be followed by a
whitespace-delimited list of architectures for which that atom should be
stabilised. If an atom is not followed by any list of architectures, it
is assumed it should be stabilised for all architectures CC'd on the bug.

Example atom list from a bug with amd64, arm, and x86 in CC:

=app-foo/bar-1.2.3           # will be stabilised on amd64, arm, and x86
=app-foo/baz-2.3.4 amd64 x86 # will be stabilised on only amd64 and x86

I'd also like to introduce a flag called "repoman-tested". This will be
used identify stabilisation requests that have been automatically
repoman checked by a bot (described below).

To prevent cluttering the Bugzilla interface, I suggest resurrecting the
"Stabilisation" component and having these new fields appear only for
bugs filed there and in "Security: Vulnerabilities" (so not to disrupt
the security team's workflow).


Automation
==========

It's easy to forget to check that all the required dependencies are in
stable before filing a stabilisation test, but this wastes the actioning
developer's time. I have prepared a bot that repoman checks the list of
atoms and flags the bug appropriately. This allows easy filtering out of
broken requests.


Improved tools
==============

To support the above, I am also preparing some tools to streamline the
process further:

* A script to, given a bug, produce an architecture-specific list of
atoms ready to feed into package.accept_keywords / emerge

* An updated version of app-portage/tatt (allowing easy testing of USE
flag combinations, reverse dependencies, committing keyword changes etc.)

* Your idea here


Too long; didn't read
=====================

We add a new box to Bugzilla to hold atoms on stabilisation request bugs
so it's easy to extract programmatically.

We add a new box to Bugzilla to indicate if a stabilisation request
requires runtime testing or not to better focus effort.

If your stabilisation request is invalid a bot will let you know and it
will be easy to skip your request until you fix it.


This will also help pave the way for future enhancements such as
triggering automated build and QA tests so that the human only has to do
runtime testing.

Thoughts?

Re: [gentoo-dev] Improving the stabilisation process - part 1

[M. J. Everitt]

[-- Attachment #1: Type: text/plain, Size: 4086 bytes --]

On 24/11/16 19:41, Michael Palimaka wrote:
> As I am sure everyone is aware by now, stabilisation requests on many
> architectures take a long time to be actioned. There are many factors
> contributing to this, but today I'd like to address three specific
> problems that unnecessarily delay developers actioning those requests.:
>
> 1. There's no easy way to pick atoms out of a stabilisation bug.
> Currently, atoms can appear, in a varying formations, in multiple
> locations: in the bug title, spread across multiple comments, in an
> attachment, [...].
>
> 2. There's no way to determine if a stabilisation request is valid (will
> pass repoman) until someone actually tries it.
>
> 3. There's no easy way to identify the level of testing required for any
> given request.
>
> To address this, I am proposing some changes to Bugzilla, some
> automation, and some improved tools.
>
>
> Bugzilla changes
> ================
>
> I'd like to introduce two new fields:
>
> 1. "Runtime testing required" - a drop-down list with three values:
>         * (unset) - the default. The stabilising developer will use
> their best judgement as to what testing is required
>         * Yes - the maintainer is specifically requesting runtime testing
>         * No - the maintainer is happy to stabilise with only a build
> test (for example, trivial revision bumps, libraries with good test
> coverage, or otherwise at the maintainer's discretion)
>
> 2. "Atoms to stabilise" - a textarea containing a list of fully
> qualified atoms. Each atom may optionally be followed by a
> whitespace-delimited list of architectures for which that atom should be
> stabilised. If an atom is not followed by any list of architectures, it
> is assumed it should be stabilised for all architectures CC'd on the bug.
>
> Example atom list from a bug with amd64, arm, and x86 in CC:
>
> =app-foo/bar-1.2.3           # will be stabilised on amd64, arm, and x86
> =app-foo/baz-2.3.4 amd64 x86 # will be stabilised on only amd64 and x86
>
> I'd also like to introduce a flag called "repoman-tested". This will be
> used identify stabilisation requests that have been automatically
> repoman checked by a bot (described below).
>
> To prevent cluttering the Bugzilla interface, I suggest resurrecting the
> "Stabilisation" component and having these new fields appear only for
> bugs filed there and in "Security: Vulnerabilities" (so not to disrupt
> the security team's workflow).
>
>
> Automation
> ==========
>
> It's easy to forget to check that all the required dependencies are in
> stable before filing a stabilisation test, but this wastes the actioning
> developer's time. I have prepared a bot that repoman checks the list of
> atoms and flags the bug appropriately. This allows easy filtering out of
> broken requests.
>
>
> Improved tools
> ==============
>
> To support the above, I am also preparing some tools to streamline the
> process further:
>
> * A script to, given a bug, produce an architecture-specific list of
> atoms ready to feed into package.accept_keywords / emerge
>
> * An updated version of app-portage/tatt (allowing easy testing of USE
> flag combinations, reverse dependencies, committing keyword changes etc.)
>
> * Your idea here
>
>
> Too long; didn't read
> =====================
>
> We add a new box to Bugzilla to hold atoms on stabilisation request bugs
> so it's easy to extract programmatically.
>
> We add a new box to Bugzilla to indicate if a stabilisation request
> requires runtime testing or not to better focus effort.
>
> If your stabilisation request is invalid a bot will let you know and it
> will be easy to skip your request until you fix it.
>
>
> This will also help pave the way for future enhancements such as
> triggering automated build and QA tests so that the human only has to do
> runtime testing.
>
> Thoughts?
>
+1

Doing a great job there, kensington :]

Will grab a paint-brush for the 'shed, soon as my own have completed
erection....


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Improving the stabilisation process - part 1

[Kent Fredric]

[-- Attachment #1.1: Type: text/plain, Size: 967 bytes --]

On Fri, 25 Nov 2016 06:41:20 +1100
Michael Palimaka <kensington@gentoo.org> wrote:

> Example atom list from a bug with amd64, arm, and x86 in CC:
> 
> =app-foo/bar-1.2.3           # will be stabilised on amd64, arm, and x86
> =app-foo/baz-2.3.4 amd64 x86 # will be stabilised on only amd64 and x86

I was doing this in the past, but there's a reason I stopped:

Bugzilla-enforced wordwrap ( at least, this is very strict on the bugzilla-email relay )

> =app-foo/bar-1.2.3           # will be stabilised on amd64, arm, and x86

Could become more like 

> =app-foo/bar-1.2.3           # will be stabilised on amd64, 
> arm, and x86


A literal example of an email I received for a bug I filed ( via an alias ) is attached.


The example I give did not render poorly on bugzilla
( https://bugs.gentoo.org/show_bug.cgi?id=599550 ), but I suspect people
who do AT probably prefer their notification emails about needed AT not
to be useless.


[-- Attachment #1.2: [Bug 599550] New: dev-perl_DateTime-1.360.0 and deps (re-)keyword request for stable arches --]
[-- Type: text/plain, Size: 4419 bytes --]

Return-Path: <bugzilla-daemon@gentoo.org>
X-Original-To: perl@gentoo.org
Delivered-To: perl@gentoo.org
Received: from gannet.gentoo.org (unknown [IPv6:2607:fcc0:4:ffff::4])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.gentoo.org (Postfix) with ESMTPS id CF7393416A0
	for <perl@gentoo.org>; Sat, 12 Nov 2016 00:39:12 +0000 (UTC)
Received: by gannet.gentoo.org (Postfix, from userid 60095)
	id 9A72D4061; Sat, 12 Nov 2016 00:39:12 +0000 (UTC)
From: bugzilla-daemon@gentoo.org
To: perl@gentoo.org
Subject: [Bug 599550] New: dev-perl/DateTime-1.360.0 and deps (re-)keyword
 request for stable arches
Date: Sat, 12 Nov 2016 00:39:07 +0000
Reply-To: DO NOT REPLY <devnull@localhost.invalid>
X-Bugzilla-Reason: AssignedTo
X-Bugzilla-Type: new
X-Bugzilla-Watch-Reason: None
X-Bugzilla-Product: Gentoo Linux
X-Bugzilla-Component: Current packages
X-Bugzilla-Version: unspecified
X-Bugzilla-Keywords: KEYWORDREQ
X-Bugzilla-Severity: normal
X-Bugzilla-Who: kentnl@gentoo.org
X-Bugzilla-Status: CONFIRMED
X-Bugzilla-Resolution: 
X-Bugzilla-Priority: Normal
X-Bugzilla-Assigned-To: perl@gentoo.org
X-Bugzilla-Target-Milestone: ---
X-Bugzilla-Flags: 
X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform
 op_sys bug_status keywords bug_severity priority component assigned_to
 reporter cc target_milestone
X-Bugzilla-Reporter: kentnl@gentoo.org
X-Bugzilla-CC: alpha@gentoo.org, arm@gentoo.org, hppa@gentoo.org,
 ia64@gentoo.org, ppc64@gentoo.org, ppc@gentoo.org, sparc@gentoo.org
Message-ID: <bug-599550-7560@http.bugs.gentoo.org/>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Bugzilla-URL: http://bugs.gentoo.org/
Auto-Submitted: auto-generated
MIME-Version: 1.0

DO NOT REPLY TO THIS EMAIL. Also, do not reply via email to the person
whose email is mentioned below. To comment on this bug, please visit:

https://bugs.gentoo.org/show_bug.cgi?id=3D599550

            Bug ID: 599550
           Summary: dev-perl/DateTime-1.360.0 and deps (re-)keyword
                    request for stable arches
           Product: Gentoo Linux
           Version: unspecified
          Hardware: All
                OS: Linux
            Status: CONFIRMED
          Keywords: KEYWORDREQ
          Severity: normal
          Priority: Normal
         Component: Current packages
          Assignee: perl@gentoo.org
          Reporter: kentnl@gentoo.org
                CC: alpha@gentoo.org, arm@gentoo.org, hppa@gentoo.org,
                    ia64@gentoo.org, ppc64@gentoo.org, ppc@gentoo.org,
                    sparc@gentoo.org
  Target Milestone: ---

DateTime has acquired sparkling new dependencies which require keywording.

Please (re-)keyword for the following targets as necessary

~alpha ~arm ~hppa ~ia64 ~ppc ~ppc64 ~sparc

The Packages as follows:

=3Ddev-perl/DateTime-1.360.0              # ~alpha ~arm ~hppa ~ia64 ~ppc ~p=
pc64
~sparc
=3Ddev-perl/Specio-0.310.0                # ~alpha ~arm ~hppa ~ia64 ~ppc ~p=
pc64
~sparc
=3Ddev-perl/Test-Needs-0.2.2              # ~alpha ~arm ~hppa ~ia64 ~ppc ~p=
pc64
~sparc

=3Ddev-perl/Devel-LexAlias-0.50.0         # ~alpha            ~ia64      ~p=
pc64
~sparc
=3Ddev-perl/Devel-Caller-2.60.0           # ~alpha            ~ia64      ~p=
pc64
~sparc
=3Ddev-perl/B-Hooks-EndOfScope-0.200.0    # ~alpha            ~ia64      ~p=
pc64
~sparc
=3Ddev-perl/Eval-Closure-0.130.0          # ~alpha            ~ia64      ~p=
pc64
~sparc

=3Ddev-perl/Variable-Magic-0.600.0        #                   ~ia64      ~p=
pc64
~sparc
=3Ddev-perl/Role-Tiny-2.0.1               #                   ~ia64      ~p=
pc64
~sparc
=3Ddev-perl/namespace-autoclean-0.280.0   #                   ~ia64      ~p=
pc64
~sparc

=3Ddev-perl/Perl-Tidy-20160302.0.0        #                   ~ia64=20=20=
=20=20=20=20=20=20=20=20=20=20
~sparc
=3Ddev-perl/CPAN-Meta-Check-0.13.0        #                   ~ia64=20=20=
=20=20=20=20=20=20=20=20=20=20
~sparc
=3Ddev-perl/PadWalker-2.200.0             #                   ~ia64=20=20=
=20=20=20=20=20=20=20=20=20=20
~sparc
=3Ddev-perl/Sub-Identify-0.120.0          #                   ~ia64=20=20=
=20=20=20=20=20=20=20=20=20=20
~sparc

--=20
You are receiving this mail because:
You are the assignee for the bug.=

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Improving the stabilisation process - part 1

[Kent Fredric]

[-- Attachment #1.1: Type: text/plain, Size: 362 bytes --]

On Fri, 25 Nov 2016 17:54:41 +1300
Kent Fredric <kentnl@gentoo.org> wrote:

> Content-Transfer-Encoding: quoted-printable

Ugh. I attached that so wrong and it is unreadable.

How does one "reply to X" and "attach other email Y" in claws without cocking it
up entirely.

"message/rfc822" as an attachment encoding type? Yay.

Lets try that again.


[-- Attachment #1.2: Type: message/rfc822, Size: 4294 bytes --]

From: bugzilla-daemon@gentoo.org
To: perl@gentoo.org
Subject: [Bug 599550] New: dev-perl/DateTime-1.360.0 and deps (re-)keyword request for stable arches
Date: Sat, 12 Nov 2016 00:39:07 +0000
Message-ID: <bug-599550-7560@http.bugs.gentoo.org/>

DO NOT REPLY TO THIS EMAIL. Also, do not reply via email to the person
whose email is mentioned below. To comment on this bug, please visit:

https://bugs.gentoo.org/show_bug.cgi?id=599550

            Bug ID: 599550
           Summary: dev-perl/DateTime-1.360.0 and deps (re-)keyword
                    request for stable arches
           Product: Gentoo Linux
           Version: unspecified
          Hardware: All
                OS: Linux
            Status: CONFIRMED
          Keywords: KEYWORDREQ
          Severity: normal
          Priority: Normal
         Component: Current packages
          Assignee: perl@gentoo.org
          Reporter: kentnl@gentoo.org
                CC: alpha@gentoo.org, arm@gentoo.org, hppa@gentoo.org,
                    ia64@gentoo.org, ppc64@gentoo.org, ppc@gentoo.org,
                    sparc@gentoo.org
  Target Milestone: ---

DateTime has acquired sparkling new dependencies which require keywording.

Please (re-)keyword for the following targets as necessary

~alpha ~arm ~hppa ~ia64 ~ppc ~ppc64 ~sparc

The Packages as follows:

=dev-perl/DateTime-1.360.0              # ~alpha ~arm ~hppa ~ia64 ~ppc ~ppc64
~sparc
=dev-perl/Specio-0.310.0                # ~alpha ~arm ~hppa ~ia64 ~ppc ~ppc64
~sparc
=dev-perl/Test-Needs-0.2.2              # ~alpha ~arm ~hppa ~ia64 ~ppc ~ppc64
~sparc

=dev-perl/Devel-LexAlias-0.50.0         # ~alpha            ~ia64      ~ppc64
~sparc
=dev-perl/Devel-Caller-2.60.0           # ~alpha            ~ia64      ~ppc64
~sparc
=dev-perl/B-Hooks-EndOfScope-0.200.0    # ~alpha            ~ia64      ~ppc64
~sparc
=dev-perl/Eval-Closure-0.130.0          # ~alpha            ~ia64      ~ppc64
~sparc

=dev-perl/Variable-Magic-0.600.0        #                   ~ia64      ~ppc64
~sparc
=dev-perl/Role-Tiny-2.0.1               #                   ~ia64      ~ppc64
~sparc
=dev-perl/namespace-autoclean-0.280.0   #                   ~ia64      ~ppc64
~sparc

=dev-perl/Perl-Tidy-20160302.0.0        #                   ~ia64            
~sparc
=dev-perl/CPAN-Meta-Check-0.13.0        #                   ~ia64            
~sparc
=dev-perl/PadWalker-2.200.0             #                   ~ia64            
~sparc
=dev-perl/Sub-Identify-0.120.0          #                   ~ia64            
~sparc

-- 
You are receiving this mail because:
You are the assignee for the bug.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Improving the stabilisation process - part 1

[Jason Zaman]

On Fri, Nov 25, 2016 at 05:54:41PM +1300, Kent Fredric wrote:
> On Fri, 25 Nov 2016 06:41:20 +1100
> Michael Palimaka <kensington@gentoo.org> wrote:
> 
> > Example atom list from a bug with amd64, arm, and x86 in CC:
> > 
> > =app-foo/bar-1.2.3           # will be stabilised on amd64, arm, and x86
> > =app-foo/baz-2.3.4 amd64 x86 # will be stabilised on only amd64 and x86
> 
> I was doing this in the past, but there's a reason I stopped:
> 
> Bugzilla-enforced wordwrap ( at least, this is very strict on the bugzilla-email relay )
> 
> > =app-foo/bar-1.2.3           # will be stabilised on amd64, arm, and x86
> 
> Could become more like 
> 
> > =app-foo/bar-1.2.3           # will be stabilised on amd64, 
> > arm, and x86

One way would be to use a plain text attachment with a standardized
filename. If there are updates to the list then the new should obsolete
the old and the script can pull non-obsoleted ones.
The problem then is how do you search for them properly? put them all
in the title anyway? then its duplicated.

When i do big lists of packages the title is eg "XFCE Stabilization for
Nov 2016" which is not duplicated and okay.
For single packages the title is just "cat/pkg-1.2.3 stablereq" so then
we duplicate anyway?

-- Jason

> A literal example of an email I received for a bug I filed ( via an alias ) is attached.
> 
> The example I give did not render poorly on bugzilla
> ( https://bugs.gentoo.org/show_bug.cgi?id=599550 ), but I suspect people
> who do AT probably prefer their notification emails about needed AT not
> to be useless.
> 

Re: [gentoo-dev] Improving the stabilisation process - part 1

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1732 bytes --]

On Fri, 25 Nov 2016 14:40:36 +0800
Jason Zaman <perfinion@gentoo.org> wrote:

> One way would be to use a plain text attachment with a standardized
> filename. If there are updates to the list then the new should obsolete
> the old and the script can pull non-obsoleted ones.
> The problem then is how do you search for them properly? put them all
> in the title anyway? then its duplicated.
> 
> When i do big lists of packages the title is eg "XFCE Stabilization for
> Nov 2016" which is not duplicated and okay.
> For single packages the title is just "cat/pkg-1.2.3 stablereq" so then
> we duplicate anyway?

If you standardized the format / filename, you could have some process
that monitors bugzilla and indexes its contents and computes overlaps,
and provides per-arch aggregates grouped by bugs.

Then the "tester" would get it in the format:

=cat/pn-VERSION  #4145 #678345
=cat/pn-VERSION  #4145
=cat/pn-VERSION  #678345

Having it in a standardized form would also potentially make it repoman-checkable
via automation:

Just like our current QA bots do, a QA Bot could fetch the keywording/stabilization
"recipie" and locally apply the requisite changes, and then do a basic repoman sanity
check to ensure coherence.

But at this point we're dove tailing good ideas into a fundamentally limited system
where we probably should have some dedicated interface (and tools) to automate and
verify these things.

Automating the *tests* is a contentious idea to some extent, but I doubt we'd see opposition
for the idea of "throw repoman at it at least and make sure it doesnt' barf" 

*That* much could be done reasonably with little effort from the existing stuff we have as it is.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[gentoo-dev] Re: Improving the stabilisation process - part 1

[Michael Palimaka]

On 25/11/16 18:51, Kent Fredric wrote:
> On Fri, 25 Nov 2016 14:40:36 +0800
> Jason Zaman <perfinion@gentoo.org> wrote:
> 
>> One way would be to use a plain text attachment with a standardized
>> filename. If there are updates to the list then the new should obsolete
>> the old and the script can pull non-obsoleted ones.
>> The problem then is how do you search for them properly? put them all
>> in the title anyway? then its duplicated.
>>
>> When i do big lists of packages the title is eg "XFCE Stabilization for
>> Nov 2016" which is not duplicated and okay.
>> For single packages the title is just "cat/pkg-1.2.3 stablereq" so then
>> we duplicate anyway?
> 
> If you standardized the format / filename, you could have some process
> that monitors bugzilla and indexes its contents and computes overlaps,
> and provides per-arch aggregates grouped by bugs.
> 
> Then the "tester" would get it in the format:
> 
> =cat/pn-VERSION  #4145 #678345
> =cat/pn-VERSION  #4145
> =cat/pn-VERSION  #678345
> 
> Having it in a standardized form would also potentially make it repoman-checkable
> via automation:
> 
> Just like our current QA bots do, a QA Bot could fetch the keywording/stabilization
> "recipie" and locally apply the requisite changes, and then do a basic repoman sanity
> check to ensure coherence.

I'm not sure what you mean. A standardised format and a bot to repoman
check lists in this format is part of my original post.

> But at this point we're dove tailing good ideas into a fundamentally limited system
> where we probably should have some dedicated interface (and tools) to automate and
> verify these things.

A dedicated portal for handling this type of thing would of course be
preferable, and work is ongoing on such a tool. However, completing and
rolling out such a system could take a very long time. Improvements to
the stabilisation process via Bugzilla can be rolled out in very little
time.

Let's not block incremental progress in favour of a perfect solution
that doesn't exist yet.

> 
> Automating the *tests* is a contentious idea to some extent, but I doubt we'd see opposition
> for the idea of "throw repoman at it at least and make sure it doesnt' barf" 

That depends on how you define 'test'. This proposal (part 1) deals
exclusively with repoman, primarily to make sure that we don't waste
time working on requests where the maintained failed to include all
required atoms.

A future proposal will certainly involve fully-automated build testing
of all requests, allowing effort to be better focused on runtime
testing. I'm unsure how that could be controversial.

Re: [gentoo-dev] Re: Improving the stabilisation process - part 1

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 2804 bytes --]

On Fri, 25 Nov 2016 20:05:55 +1100
Michael Palimaka <kensington@gentoo.org> wrote:

> I'm not sure what you mean. A standardised format and a bot to repoman
> check lists in this format is part of my original post.

Ah. I didn't take enough care reading it and mostly replied on the earlier part.

Good to see we reached similar conclusions through different avenues.

(NB: I find "too long didn't read" coming at the bottom of a post not too productive
because people who feel an article was too long to read tend to not read to the bottom
just in case somebody put a TL;DR there. I always find that data to be better presented
head-first, in a "read this first to get the general gist of it, then keep reading if you care
about details" format, that way you hit your target audience sooner)


> > But at this point we're dove tailing good ideas into a fundamentally limited system
> > where we probably should have some dedicated interface (and tools) to automate and
> > verify these things.  
> 
> A dedicated portal for handling this type of thing would of course be
> preferable, and work is ongoing on such a tool. However, completing and
> rolling out such a system could take a very long time. Improvements to
> the stabilisation process via Bugzilla can be rolled out in very little
> time.
> 
> Let's not block incremental progress in favour of a perfect solution
> that doesn't exist yet.

Agreed. I'm fine with this as a concept as an initial staging.

> > 
> > Automating the *tests* is a contentious idea to some extent, but I doubt we'd see opposition
> > for the idea of "throw repoman at it at least and make sure it doesnt' barf"   
> 
> That depends on how you define 'test'. This proposal (part 1) deals
> exclusively with repoman, primarily to make sure that we don't waste
> time working on requests where the maintained failed to include all
> required atoms.
> 
> A future proposal will certainly involve fully-automated build testing
> of all requests, allowing effort to be better focused on runtime
> testing. I'm unsure how that could be controversial.

I'm not saying *I* have a problem with this, but I've gotten the over-all vibe
from some people that "autostabilizing all the things is bad" and I wanted to demark
this action as being somuch the opposite of that that even those people are unlikely to complain.

Because the subset of tests that only prove *repoman* integrity can't ever be
imagined sufficient in themselves to mark anything as "stable" or "working",
and so it forces people somewhere to do due diligence manually.

Hence, the general vibe is the fear that the abundance of automation will usurp
due diligence as people use that automation as the *only* indicator of "do we flip the flag" or not.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[gentoo-dev] Re: Improving the stabilisation process - part 1

[Michael Palimaka]

On 25/11/16 17:40, Jason Zaman wrote:
> On Fri, Nov 25, 2016 at 05:54:41PM +1300, Kent Fredric wrote:
>> On Fri, 25 Nov 2016 06:41:20 +1100
>> Michael Palimaka <kensington@gentoo.org> wrote:
>>
>>> Example atom list from a bug with amd64, arm, and x86 in CC:
>>>
>>> =app-foo/bar-1.2.3           # will be stabilised on amd64, arm, and x86
>>> =app-foo/baz-2.3.4 amd64 x86 # will be stabilised on only amd64 and x86
>>
>> I was doing this in the past, but there's a reason I stopped:
>>
>> Bugzilla-enforced wordwrap ( at least, this is very strict on the bugzilla-email relay )
>>
>>> =app-foo/bar-1.2.3           # will be stabilised on amd64, arm, and x86
>>
>> Could become more like 
>>
>>> =app-foo/bar-1.2.3           # will be stabilised on amd64, 
>>> arm, and x86
> 
> One way would be to use a plain text attachment with a standardized
> filename. If there are updates to the list then the new should obsolete
> the old and the script can pull non-obsoleted ones.
> The problem then is how do you search for them properly? put them all
> in the title anyway? then its duplicated.

While I personally think the additional field is adequate, I'm more than
happy to support attachments as well. Suggestions for the standardised
filename? Alternatively, it would easy to add a flag to the interface to
mark an attachment as a stabilisation list.

> When i do big lists of packages the title is eg "XFCE Stabilization for
> Nov 2016" which is not duplicated and okay.
> For single packages the title is just "cat/pkg-1.2.3 stablereq" so then
> we duplicate anyway?

Yes, every atom should be referenced either in the special field (or as
per above in an attachment), even if that means duplicating the title.
It's a very minor effort on the part of the person filing the bug, but
it will guarantee that we can always find the atom list in the same
place every time.

Of course, we can't *force* anyone to use the new field if they don't
want to. They might find themselves wondering why their request ends up
on the bottom of the queue however. :-)

[gentoo-dev] Re: Improving the stabilisation process - part 1

[Michael Palimaka]

On 25/11/16 15:54, Kent Fredric wrote:
> On Fri, 25 Nov 2016 06:41:20 +1100
> Michael Palimaka <kensington@gentoo.org> wrote:
> 
>> Example atom list from a bug with amd64, arm, and x86 in CC:
>>
>> =app-foo/bar-1.2.3           # will be stabilised on amd64, arm, and x86
>> =app-foo/baz-2.3.4 amd64 x86 # will be stabilised on only amd64 and x86
> 
> I was doing this in the past, but there's a reason I stopped:
> 
> Bugzilla-enforced wordwrap ( at least, this is very strict on the bugzilla-email relay )
> 
>> =app-foo/bar-1.2.3           # will be stabilised on amd64, arm, and x86
> 
> Could become more like 
> 
>> =app-foo/bar-1.2.3           # will be stabilised on amd64, 
>> arm, and x86
> 
> 
> A literal example of an email I received for a bug I filed ( via an alias ) is attached.
> 
> 
> The example I give did not render poorly on bugzilla
> ( https://bugs.gentoo.org/show_bug.cgi?id=599550 ), but I suspect people
> who do AT probably prefer their notification emails about needed AT not
> to be useless.
> 

Does it matter if it gets wrapped a bit strangely in an email? Nobody
should be actioning a stabilisation request based on the contents of a
bugmail, as it doesn't necessarily reflect the most up-to-date information.

Re: [gentoo-dev] Improving the stabilisation process - part 1

[Jason Zaman]

On Fri, Nov 25, 2016 at 06:41:20AM +1100, Michael Palimaka wrote:
> As I am sure everyone is aware by now, stabilisation requests on many
> architectures take a long time to be actioned. There are many factors
> contributing to this, but today I'd like to address three specific
> problems that unnecessarily delay developers actioning those requests.:
> 
> 1. There's no easy way to pick atoms out of a stabilisation bug.
> Currently, atoms can appear, in a varying formations, in multiple
> locations: in the bug title, spread across multiple comments, in an
> attachment, [...].
> 
> 2. There's no way to determine if a stabilisation request is valid (will
> pass repoman) until someone actually tries it.
> 
> 3. There's no easy way to identify the level of testing required for any
> given request.
> 
> To address this, I am proposing some changes to Bugzilla, some
> automation, and some improved tools.
> 
> 
> Bugzilla changes
> ================
> 
> I'd like to introduce two new fields:
> 
> 1. "Runtime testing required" - a drop-down list with three values:
>         * (unset) - the default. The stabilising developer will use
> their best judgement as to what testing is required
>         * Yes - the maintainer is specifically requesting runtime testing
>         * No - the maintainer is happy to stabilise with only a build
> test (for example, trivial revision bumps, libraries with good test
> coverage, or otherwise at the maintainer's discretion)
> 
> 2. "Atoms to stabilise" - a textarea containing a list of fully
> qualified atoms. Each atom may optionally be followed by a
> whitespace-delimited list of architectures for which that atom should be
> stabilised. If an atom is not followed by any list of architectures, it
> is assumed it should be stabilised for all architectures CC'd on the bug.
> 
> Example atom list from a bug with amd64, arm, and x86 in CC:
> 
> =app-foo/bar-1.2.3           # will be stabilised on amd64, arm, and x86
> =app-foo/baz-2.3.4 amd64 x86 # will be stabilised on only amd64 and x86
> 
> I'd also like to introduce a flag called "repoman-tested". This will be
> used identify stabilisation requests that have been automatically
> repoman checked by a bot (described below).
> 
> To prevent cluttering the Bugzilla interface, I suggest resurrecting the
> "Stabilisation" component and having these new fields appear only for
> bugs filed there and in "Security: Vulnerabilities" (so not to disrupt
> the security team's workflow).
> 
> 
> Automation
> ==========
> 
> It's easy to forget to check that all the required dependencies are in
> stable before filing a stabilisation test, but this wastes the actioning
> developer's time. I have prepared a bot that repoman checks the list of
> atoms and flags the bug appropriately. This allows easy filtering out of
> broken requests.

This part would need to take into account DEPENDS ON in the bug too.
When I file my xfce lists I dep on the gnome stable bug since I need
the gtk stuff and if that isnt taken into account, repoman would just
die.
> 
> Improved tools
> ==============
> 
> To support the above, I am also preparing some tools to streamline the
> process further:
> 
> * A script to, given a bug, produce an architecture-specific list of
> atoms ready to feed into package.accept_keywords / emerge
> 
> * An updated version of app-portage/tatt (allowing easy testing of USE
> flag combinations, reverse dependencies, committing keyword changes etc.)
> 
> * Your idea here
> 
> 
> Too long; didn't read
> =====================
> 
> We add a new box to Bugzilla to hold atoms on stabilisation request bugs
> so it's easy to extract programmatically.
> 
> We add a new box to Bugzilla to indicate if a stabilisation request
> requires runtime testing or not to better focus effort.
> 
> If your stabilisation request is invalid a bot will let you know and it
> will be easy to skip your request until you fix it.
> 
> 
> This will also help pave the way for future enhancements such as
> triggering automated build and QA tests so that the human only has to do
> runtime testing.
> 
> Thoughts?

Everything here sounds pretty good to me!

I just realized there is another rare issue that we may have to take
into account. Some sets of packages *must* be stablized in lockstep. For
regular sets of packages like eg xfce if you leave off a bunch of the
extras its no big deal.
Eg for SELinux, the policy packages must all be stabilized all at
once because they depend on each other (I think perl is like this too?).

We would need a way for maintainers to ask for testing without actually
committing. The maintainer can wait till everything is done and commit
everything at once himself. Some flag to make the tatt script skip the
step would be enough I think.

-- Jason

[gentoo-dev] Re: Improving the stabilisation process - part 1

[Michael Palimaka]

On 25/11/16 17:51, Jason Zaman wrote:
>> Automation
>> ==========
>>
>> It's easy to forget to check that all the required dependencies are in
>> stable before filing a stabilisation test, but this wastes the actioning
>> developer's time. I have prepared a bot that repoman checks the list of
>> atoms and flags the bug appropriately. This allows easy filtering out of
>> broken requests.
> 
> This part would need to take into account DEPENDS ON in the bug too.
> When I file my xfce lists I dep on the gnome stable bug since I need
> the gtk stuff and if that isnt taken into account, repoman would just
> die.

The bot will either try its best to take into account bug dependencies,
or otherwise just give up and skip such a bug.

> I just realized there is another rare issue that we may have to take
> into account. Some sets of packages *must* be stablized in lockstep. For
> regular sets of packages like eg xfce if you leave off a bunch of the
> extras its no big deal.
> Eg for SELinux, the policy packages must all be stabilized all at
> once because they depend on each other (I think perl is like this too?).
> 
> We would need a way for maintainers to ask for testing without actually
> committing. The maintainer can wait till everything is done and commit
> everything at once himself. Some flag to make the tatt script skip the
> step would be enough I think.

Do you have a bug number handy so I can double check how the process
normally looks?

Re: [gentoo-dev] Re: Improving the stabilisation process - part 1

[Jason Zaman]

On Fri, Nov 25, 2016 at 08:27:12PM +1100, Michael Palimaka wrote:
> On 25/11/16 17:51, Jason Zaman wrote:
> >> Automation
> >> ==========
> >>
> >> It's easy to forget to check that all the required dependencies are in
> >> stable before filing a stabilisation test, but this wastes the actioning
> >> developer's time. I have prepared a bot that repoman checks the list of
> >> atoms and flags the bug appropriately. This allows easy filtering out of
> >> broken requests.
> > 
> > This part would need to take into account DEPENDS ON in the bug too.
> > When I file my xfce lists I dep on the gnome stable bug since I need
> > the gtk stuff and if that isnt taken into account, repoman would just
> > die.
> 
> The bot will either try its best to take into account bug dependencies,
> or otherwise just give up and skip such a bug.
> 
> > I just realized there is another rare issue that we may have to take
> > into account. Some sets of packages *must* be stablized in lockstep. For
> > regular sets of packages like eg xfce if you leave off a bunch of the
> > extras its no big deal.
> > Eg for SELinux, the policy packages must all be stabilized all at
> > once because they depend on each other (I think perl is like this too?).
> > 
> > We would need a way for maintainers to ask for testing without actually
> > committing. The maintainer can wait till everything is done and commit
> > everything at once himself. Some flag to make the tatt script skip the
> > step would be enough I think.
> 
> Do you have a bug number handy so I can double check how the process
> normally looks?

SwifT and I both run stable machines so we always just stabilize SELinux
stuff ourselves so dont have a bug number, no.

its pretty much just:
cd sec-policy/
ekeyword amd64 x86 */*-2.20151208-r6.ebuild
git add .
repoman full
repoman commit -m 'sec-policy: Stabilize selinux policy 2.20151208-r6'

every single selinux policy package {,R}DEPENDs on
=sec-policy/selinux-base-policy-${PVR}
which in turn deps on 
=sec-policy/selinux-base-${PVR}
and we always do everything with the same revision at the same time
otherwise you get lots of weirdness. For us, each arch could be done
separately (no point tho), the main part is that committing the packages
in a random order or even alphabetical doesnt really work.

How aware of deps do we want to make such a tool? Having it figure out
the right order seems like a lot of work. Or we could force the script
to follow the order defined in the package list and then its up to the
maintainer. Then also if there is a failure either abort completely or
continue with the rest that it can (a. la --keep-going).

-- Jason

[gentoo-dev] Re: Improving the stabilisation process - part 1

[Michael Palimaka]

On 25/11/16 21:00, Jason Zaman wrote:
> On Fri, Nov 25, 2016 at 08:27:12PM +1100, Michael Palimaka wrote:
>> On 25/11/16 17:51, Jason Zaman wrote:
>>>> Automation
>>>> ==========
>>>>
>>>> It's easy to forget to check that all the required dependencies are in
>>>> stable before filing a stabilisation test, but this wastes the actioning
>>>> developer's time. I have prepared a bot that repoman checks the list of
>>>> atoms and flags the bug appropriately. This allows easy filtering out of
>>>> broken requests.
>>>
>>> This part would need to take into account DEPENDS ON in the bug too.
>>> When I file my xfce lists I dep on the gnome stable bug since I need
>>> the gtk stuff and if that isnt taken into account, repoman would just
>>> die.
>>
>> The bot will either try its best to take into account bug dependencies,
>> or otherwise just give up and skip such a bug.
>>
>>> I just realized there is another rare issue that we may have to take
>>> into account. Some sets of packages *must* be stablized in lockstep. For
>>> regular sets of packages like eg xfce if you leave off a bunch of the
>>> extras its no big deal.
>>> Eg for SELinux, the policy packages must all be stabilized all at
>>> once because they depend on each other (I think perl is like this too?).
>>>
>>> We would need a way for maintainers to ask for testing without actually
>>> committing. The maintainer can wait till everything is done and commit
>>> everything at once himself. Some flag to make the tatt script skip the
>>> step would be enough I think.
>>
>> Do you have a bug number handy so I can double check how the process
>> normally looks?
> 
> SwifT and I both run stable machines so we always just stabilize SELinux
> stuff ourselves so dont have a bug number, no.
> 
> its pretty much just:
> cd sec-policy/
> ekeyword amd64 x86 */*-2.20151208-r6.ebuild
> git add .
> repoman full
> repoman commit -m 'sec-policy: Stabilize selinux policy 2.20151208-r6'
> 
> every single selinux policy package {,R}DEPENDs on
> =sec-policy/selinux-base-policy-${PVR}
> which in turn deps on 
> =sec-policy/selinux-base-${PVR}
> and we always do everything with the same revision at the same time
> otherwise you get lots of weirdness. For us, each arch could be done
> separately (no point tho), the main part is that committing the packages
> in a random order or even alphabetical doesnt really work.
> 
> How aware of deps do we want to make such a tool? Having it figure out
> the right order seems like a lot of work. Or we could force the script
> to follow the order defined in the package list and then its up to the
> maintainer. Then also if there is a failure either abort completely or
> continue with the rest that it can (a. la --keep-going).

Most tools I've seen address this by keywording the entire list of
packages first, then doing the commit(s). tatt splits the entire process
into multiple stages (build the package, build revdeps, keyword the
ebuilds, git commit, ...) which can be run individually to maximise
control to the person running it.

That said, not everything needs a fancy tool. Your SELinux process looks
pretty good the way it is.

Re: [gentoo-dev] Improving the stabilisation process - part 1

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 2270 bytes --]

On Fri, Nov 25, 2016 at 06:41:20AM +1100, Michael Palimaka wrote:
> As I am sure everyone is aware by now, stabilisation requests on many
> architectures take a long time to be actioned. There are many factors
> contributing to this, but today I'd like to address three specific
> problems that unnecessarily delay developers actioning those requests.:
>
> 1. There's no easy way to pick atoms out of a stabilisation bug.
> Currently, atoms can appear, in a varying formations, in multiple
> locations: in the bug title, spread across multiple comments, in an
> attachment, [...].
> 
> 2. There's no way to determine if a stabilisation request is valid (will
> pass repoman) until someone actually tries it.
> 
> 3. There's no easy way to identify the level of testing required for any
> given request.
> 
> To address this, I am proposing some changes to Bugzilla, some
> automation, and some improved tools.
> 
> 
> Bugzilla changes
> ================
> 
> I'd like to introduce two new fields:
> 
> 1. "Runtime testing required" - a drop-down list with three values:
>         * (unset) - the default. The stabilising developer will use
> their best judgement as to what testing is required
>         * Yes - the maintainer is specifically requesting runtime testing
>         * No - the maintainer is happy to stabilise with only a build
> test (for example, trivial revision bumps, libraries with good test
> coverage, or otherwise at the maintainer's discretion)
> 
> 2. "Atoms to stabilise" - a textarea containing a list of fully
> qualified atoms. Each atom may optionally be followed by a
> whitespace-delimited list of architectures for which that atom should be
> stabilised. If an atom is not followed by any list of architectures, it
> is assumed it should be stabilised for all architectures CC'd on the bug.
> 
> Example atom list from a bug with amd64, arm, and x86 in CC:
> 
> =app-foo/bar-1.2.3           # will be stabilised on amd64, arm, and x86
> =app-foo/baz-2.3.4 amd64 x86 # will be stabilised on only amd64 and x86
 
Listing the architectures seems redundant if they are also in the cc:
field. In your example above, why would you need arm in the cc: field
 for app-foo/bas-2.3.4?

 William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 163 bytes --]

[gentoo-dev] Re: Improving the stabilisation process - part 1

[Michael Palimaka]

On 27/11/16 10:32, William Hubbs wrote:
> On Fri, Nov 25, 2016 at 06:41:20AM +1100, Michael Palimaka wrote:
>> As I am sure everyone is aware by now, stabilisation requests on many
>> architectures take a long time to be actioned. There are many factors
>> contributing to this, but today I'd like to address three specific
>> problems that unnecessarily delay developers actioning those requests.:
>>
>> 1. There's no easy way to pick atoms out of a stabilisation bug.
>> Currently, atoms can appear, in a varying formations, in multiple
>> locations: in the bug title, spread across multiple comments, in an
>> attachment, [...].
>>
>> 2. There's no way to determine if a stabilisation request is valid (will
>> pass repoman) until someone actually tries it.
>>
>> 3. There's no easy way to identify the level of testing required for any
>> given request.
>>
>> To address this, I am proposing some changes to Bugzilla, some
>> automation, and some improved tools.
>>
>>
>> Bugzilla changes
>> ================
>>
>> I'd like to introduce two new fields:
>>
>> 1. "Runtime testing required" - a drop-down list with three values:
>>         * (unset) - the default. The stabilising developer will use
>> their best judgement as to what testing is required
>>         * Yes - the maintainer is specifically requesting runtime testing
>>         * No - the maintainer is happy to stabilise with only a build
>> test (for example, trivial revision bumps, libraries with good test
>> coverage, or otherwise at the maintainer's discretion)
>>
>> 2. "Atoms to stabilise" - a textarea containing a list of fully
>> qualified atoms. Each atom may optionally be followed by a
>> whitespace-delimited list of architectures for which that atom should be
>> stabilised. If an atom is not followed by any list of architectures, it
>> is assumed it should be stabilised for all architectures CC'd on the bug.
>>
>> Example atom list from a bug with amd64, arm, and x86 in CC:
>>
>> =app-foo/bar-1.2.3           # will be stabilised on amd64, arm, and x86
>> =app-foo/baz-2.3.4 amd64 x86 # will be stabilised on only amd64 and x86
>  
> Listing the architectures seems redundant if they are also in the cc:
> field. In your example above, why would you need arm in the cc: field
>  for app-foo/bas-2.3.4?

This is to handle the case where multiple packages are stabilised in a
single bug and each package needs stabilising on a different set of
architectures.

Re: [gentoo-dev] Improving the stabilisation process - part 1

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1401 bytes --]

On Sat, 26 Nov 2016 17:32:02 -0600
William Hubbs <williamh@gentoo.org> wrote:

> Listing the architectures seems redundant if they are also in the cc:
> field. In your example above, why would you need arm in the cc: field
>  for app-foo/bas-2.3.4?

Often, the required work for "lists of keywords/stabilizations" has inconsistency.

e.g: Often you'll have only a few packages that need actioned on one arch
and you'll need a dozen or so that need actioned on others.

Hence, breaking them down to have keywords on the atom lines allows for a quick
grep of what's affected.

For example, https://bugs.gentoo.org/show_bug.cgi?id=599550#c0

Where I sorted everything to minimise how tall each list would be.

Only 3 packages needed actioned to satisfy ~arm, ~hppa and ~ppc

7 packages needed action for ~alpha

10 packages needed action for ~ppc64

and 14 packages needed action for ~ia64 and ~sparc

People who process such lists of course are free to ignore this metadata.

But I'd imagine having a short checklist of "tweak these and it should work" 
to be helpful.

*especially* given the current default behaviour of `ekeyword`, where calling

  ekeyword ~foo

Will unintentionally downgrade stable to ~arch, even though the point was to
change from unkeyworded to ~arch.

Surely though, we could fix that?

   ekeyword --as-needed ~foo PKGHERE

Plz? 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]