From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 5D610139694 for ; Mon, 24 Jul 2017 21:22:46 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 5F0A9E0D12; Mon, 24 Jul 2017 21:22:40 +0000 (UTC) Received: from smtp.gentoo.org (woodpecker.gentoo.org [IPv6:2001:470:ea4a:1:5054:ff:fec7:86e4]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 05639E0BE9 for ; Mon, 24 Jul 2017 21:22:39 +0000 (UTC) Received: from sf (host81-147-13-134.range81-147.btcentralplus.com [81.147.13.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: slyfox) by smtp.gentoo.org (Postfix) with ESMTPSA id 05C95341748; Mon, 24 Jul 2017 21:22:28 +0000 (UTC) Date: Mon, 24 Jul 2017 22:22:23 +0100 From: Sergei Trofimovich To: gentoo-dev@lists.gentoo.org, wg-stable@gentoo.org, arch-leads@gentoo.org, alpha@gentoo.org, amd64@gentoo.org, amd64-fbsd@gentoo.org, arm@gentoo.org, arm64@gentoo.org, hppa@gentoo.org, ia64@gentoo.org, m68k@gentoo.org, mips@gentoo.org, ppc@gentoo.org, ppc64@gentoo.org, s390@gentoo.org, sh@gentoo.org, sparc@gentoo.org, x86@gentoo.org, x86-fbsd@gentoo.org Subject: [gentoo-dev] [RFC] Future of gentoo's stable and unstable trees: what are your thoughts? Message-ID: <20170724222223.6d359e47@sf> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.31; x86_64-pc-linux-gnu) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/jQKRPHCgTgamf+WMIihTF9O"; protocol="application/pgp-signature" X-Archives-Salt: 78b98bae-e85c-444b-9596-db9787ca0dc1 X-Archives-Hash: 0aba06faccfa7c5a9648745a3639932b --Sig_/jQKRPHCgTgamf+WMIihTF9O Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable TL;DR;TL;DR: ------------ This email seeks for one step towards less toil tied to gentoo's keywording/stabilization process. I've CCed a few groups who might be interested in making this area better: - gentoo-dev@ as it affects most devs (and non-devs!) - wg-stable@ as it overlaps quite a bit with efforts staged on STABLEREQ (should I join? :) - arch-leads@ as it directly helps (or breaks everything for) arch teams - all individual arches for wider visibility TL;DR ----- I see the problem of lagging stable and unstable trees as: 1. lack of automation 2. lack of manpower The PROPOSAL to solve the '1. automation' part is to draft a new GLEP. If there is any interest (I assume there is!) I'll start one. Let's call that fufure 'life of KEYWORDS'. It will cover: - Update on GLEP-40 ("x86 stabilization can do only x86@ team") to allow package maintainers to do ARCH=3Dx86 stabilization. It will be an arch-agnostic way: each arch will have minimal requirements to setup environment suitable for stabilization and keywording: CFLAGS to have, hardware required, whatever else is practical. - Formalize list of stable arches as such (will be covered by 'arches.desc' GLEP) - Formalize what is a "stable arch". In short: - arch is marked as such in 'arches.desc' - performs most of STABLEREQ/KEYWORDREQ or gives rationale why progress can't be easily done before 90-days timeout - Formalize and automate process of dropping keywords for timed out STABLEREQ/KEYWORDREQ requests. - Automate process of restoring dropped KEYWORDS due to bumps adding new unkeyworded dependencies. repoman already complains about those. What is left is to grab them in batches time to time and handle those as if those were KEYWORDREQ. - File more automated STABLEREQs to rely less on lazy maintainers (I am example of lazy maintainer not siling STABLEREQs enough). - Formalize which STABLEREQ/KEYWORDREQ can be done automatically by arch teams (or maybe anyone else having the hardware!). In short: anything not marked as "Runtime testing required" on bugzilla and not having any blocker bugs. The proposal to solve the '2. manpower' part is: - Write more docs and make stabilisation process easier for everyone. Important detail: the list is not in set in stone but rather a guideline of things to cover. Please feel free to propose other topics, questions, concerns, likes or any other actionable feedback! Story mode ---------- Hi all! As you might suspect arch testing (an important process part of gentoo's health). Recently arch testing became a point of contentions among gentoo developers. The non-exhaustive list of complaints is: - Minor (usually understaffed) arch ${A} is slow and does not process STABLEREQ/KEYWORDREQ packages for many months. Lag forces maintainers to keep more old versions in the tree prohibiting removal of not really-maintained packages. In theory policy allows dropping stable keywords from such packages but in practice people frequently do not do it because it's a large and tedious tree-wide change. - Packages on major arch (amd64 or x86) are not stabilized frequently enough. Packages fall through the cracks in bugzilla, STABLEREQs don't get filed by maintainers. - What are the actual problems? ----------------------------- - Arch testing is complicated, repetitive and (somethat) unrewarding. People only notice when things don't work. - Arch testing is complicated: each architecture (or OS) has it's own quirks. It's hard for a developer to join empty arch team as documentation on specifics is frequently lacking. Some example questions could arise: - When keywording for ARCH=3Dppc64 should we test on both powerpc64 and powerpc64le or only one of them? - Does -Wl,--as-needed work on ia64? - There seem to be a little of coordination between arch teams which tools to use to handle stabilization pipeline. Some examples: - Q: How to grab a list for keywording? A: use 'getatoms.py' from https://wiki.gentoo.org/wiki/Package_testing#= Tools but there is a few caveats. - Q: How to keyword a package before stabilization? A: $ ekeyword ~ia64 Caveat: can easily drop a keyword from ia64 down to ~ia64 for a package or two causing tree breakage. - Q: How to mark a package as explicitly not working or an ${arch}? A: Use KEYWORDS=3D"-${arch}" but bots and ekeyword ignore it. - Q: How to mark a single package version as broken for a particular arch? - Q: What is the format of commit message for arch commit? Should package version be there? Should it be one commit per package? Per bug? Per arch? A: Everyone does it their own way. What can we do about it? ------------------------ The short answer is: we need more automation and more manpower. Both are related: in an ideal world robots would do all package testing for us. People would only need to add new packages into system. More detailed plan (actions and changes) ---------------------------------------- The plan is to get feedback from gentoo community and form a new GLEP that will supersede GLEP-40. We will need input from many active members: arch teams, wg-stable team and gentoo devs. The questions that will be covered in the new GLEP: 1. Q: What is a supported stable arch? A: - Arch is marked 'stable' in 'profiles.desc' (and upcoming 'arches.desc') - Arch that has at least one team member willing to do STABLEREQ/KEYWORDREQ processing - The STABLEREQ/KEYWORDREQ request timeout is 90 days After timeout expires all keywords for requested package will be demoted. To avoid tedious manual de-keywording we will need tooling for that. 2. Q: How to make arch testing faster and easier? A: - KEYWORDREQ/STABLEREQ bugs not marked as "runtime testing required" will be automatically tested and keyworded. [handwave] automated tinderbox setup would help a lot to now upfront what fails to built, fails tests. 3. Q: How to programmatically get "readiness" status for a particular package? Say, how many STABLE-blocking bugs the package has? A: Suggestions welcome! Maybe add "package list" field for bugs? 4. Q: How to push more packages into STABLE? A: File automatic STABLEREQ bugs more aggressively if no known bugs exist for a package version. The rough workflow is the following: - Grab a list of candidates for stabilization (will need additional tooling) - File STABLEREQ against maintainer only first. - Maintainer will have a 2-week timeout to either proceed with request: - add "runtime testing required fields", CC relevant arches to start stabilization - or set blocking bugs against STABLEREQ to stop stabilization - After 2-week timeout tooling automatically CCes arches and stabilization happens (ideally with minimal manual work) - Profit! 5. Q: What do you think of all that? =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D Can this proposal make a difference and make gentoo better and easier to work with? Does it try to attack the right thing? Does it completely miss the point? Does it sound fun? Please do share your thoughts! Thank you! -- Sergei --Sig_/jQKRPHCgTgamf+WMIihTF9O Content-Type: application/pgp-signature Content-Description: Цифровая подпись OpenPGP -----BEGIN PGP SIGNATURE----- iF0EARECAB0WIQSZKa0VG5avZRlY01hxoe52YR/zqgUCWXZlDwAKCRBxoe52YR/z qpmqAJ9D+KzEq3QzrEM4hWoL3MO7fLbq3QCePXdchsL+r4RMFhNfWHVN2L3bsTI= =R9/P -----END PGP SIGNATURE----- --Sig_/jQKRPHCgTgamf+WMIihTF9O--