[gentoo-dev] Improving the stabilisation process - part 1

[Michael Palimaka <kensington@gentoo.org>]

As I am sure everyone is aware by now, stabilisation requests on many
architectures take a long time to be actioned. There are many factors
contributing to this, but today I'd like to address three specific
problems that unnecessarily delay developers actioning those requests.:

1. There's no easy way to pick atoms out of a stabilisation bug.
Currently, atoms can appear, in a varying formations, in multiple
locations: in the bug title, spread across multiple comments, in an
attachment, [...].

2. There's no way to determine if a stabilisation request is valid (will
pass repoman) until someone actually tries it.

3. There's no easy way to identify the level of testing required for any
given request.

To address this, I am proposing some changes to Bugzilla, some
automation, and some improved tools.


Bugzilla changes
================

I'd like to introduce two new fields:

1. "Runtime testing required" - a drop-down list with three values:
        * (unset) - the default. The stabilising developer will use
their best judgement as to what testing is required
        * Yes - the maintainer is specifically requesting runtime testing
        * No - the maintainer is happy to stabilise with only a build
test (for example, trivial revision bumps, libraries with good test
coverage, or otherwise at the maintainer's discretion)

2. "Atoms to stabilise" - a textarea containing a list of fully
qualified atoms. Each atom may optionally be followed by a
whitespace-delimited list of architectures for which that atom should be
stabilised. If an atom is not followed by any list of architectures, it
is assumed it should be stabilised for all architectures CC'd on the bug.

Example atom list from a bug with amd64, arm, and x86 in CC:

=app-foo/bar-1.2.3           # will be stabilised on amd64, arm, and x86
=app-foo/baz-2.3.4 amd64 x86 # will be stabilised on only amd64 and x86

I'd also like to introduce a flag called "repoman-tested". This will be
used identify stabilisation requests that have been automatically
repoman checked by a bot (described below).

To prevent cluttering the Bugzilla interface, I suggest resurrecting the
"Stabilisation" component and having these new fields appear only for
bugs filed there and in "Security: Vulnerabilities" (so not to disrupt
the security team's workflow).


Automation
==========

It's easy to forget to check that all the required dependencies are in
stable before filing a stabilisation test, but this wastes the actioning
developer's time. I have prepared a bot that repoman checks the list of
atoms and flags the bug appropriately. This allows easy filtering out of
broken requests.


Improved tools
==============

To support the above, I am also preparing some tools to streamline the
process further:

* A script to, given a bug, produce an architecture-specific list of
atoms ready to feed into package.accept_keywords / emerge

* An updated version of app-portage/tatt (allowing easy testing of USE
flag combinations, reverse dependencies, committing keyword changes etc.)

* Your idea here


Too long; didn't read
=====================

We add a new box to Bugzilla to hold atoms on stabilisation request bugs
so it's easy to extract programmatically.

We add a new box to Bugzilla to indicate if a stabilisation request
requires runtime testing or not to better focus effort.

If your stabilisation request is invalid a bot will let you know and it
will be easy to skip your request until you fix it.


This will also help pave the way for future enhancements such as
triggering automated build and QA tests so that the human only has to do
runtime testing.

Thoughts?