From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <gentoo-dev+bounces-98414-garchives=archives.gentoo.org@lists.gentoo.org>
Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (2048 bits))
	(No client certificate requested)
	by finch.gentoo.org (Postfix) with ESMTPS id AF2CA158020
	for <garchives@archives.gentoo.org>; Sun,  6 Nov 2022 08:15:47 +0000 (UTC)
Received: from pigeon.gentoo.org (localhost [127.0.0.1])
	by pigeon.gentoo.org (Postfix) with SMTP id 7A8F1E0858;
	Sun,  6 Nov 2022 08:15:44 +0000 (UTC)
Received: from smtp.gentoo.org (woodpecker.gentoo.org [140.211.166.183])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits))
	(No client certificate requested)
	by pigeon.gentoo.org (Postfix) with ESMTPS id 3F403E0826
	for <gentoo-dev@lists.gentoo.org>; Sun,  6 Nov 2022 08:15:44 +0000 (UTC)
Message-ID: <2bd8402513dd47b06aa38a0d49e1c29880c1aad4.camel@gentoo.org>
Subject: [gentoo-dev] Disturbing state of arch testing in Gentoo
From: =?UTF-8?Q?Micha=C5=82_G=C3=B3rny?= <mgorny@gentoo.org>
To: gentoo-dev@lists.gentoo.org
Date: Sun, 06 Nov 2022 09:15:40 +0100
Organization: Gentoo
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.44.4 
Precedence: bulk
List-Post: <mailto:gentoo-dev@lists.gentoo.org>
List-Help: <mailto:gentoo-dev+help@lists.gentoo.org>
List-Unsubscribe: <mailto:gentoo-dev+unsubscribe@lists.gentoo.org>
List-Subscribe: <mailto:gentoo-dev+subscribe@lists.gentoo.org>
List-Id: Gentoo Linux mail <gentoo-dev.gentoo.org>
X-BeenThere: gentoo-dev@lists.gentoo.org
Reply-to: gentoo-dev@lists.gentoo.org
X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply
MIME-Version: 1.0
X-Archives-Salt: fd5f04f7-40fd-4c55-94ae-213001db944e
X-Archives-Hash: b9a014a18091cac95368e15233fe02d2

Hi, everyone.

Arch testing's relying on automation a lot these days.  Not saying
that's bad, if it improves the state of affairs.  However, I have some
concerns, based on what I've seen lately.

On top of that, it seems that most of it still relies on proprietary
software and we have no clue how *exactly* it works, and it's really,
really hard to get a straight answer.

So, my questions are:

1. Is "runtime testing required" field being respected?  Obviously not
every package can be (sufficiently) tested via FEATURES=3Dtest, so we've
added that fields.  However, if arch testers just ignore it and push
things stable based on pure build testing...

2. How are kernels being tested?  Given the speed with which new gentoo-
sources stablereqs are handled, I really feel like "arch testing" there
means "checking if sources install", and have little to do with working
kernels.

3. How does the automation handle packages that aren't trivially
installable?  I recall that in the past stablereqs were stalled for
months without a single comment because automation couldn't figure out
how to proceed, and nobody bothered reporting a problem.

--=20
Best regards,
Micha=C5=82 G=C3=B3rny