From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from lists.gentoo.org ([140.105.134.102] helo=robin.gentoo.org) by nuthatch.gentoo.org with esmtp (Exim 4.62) (envelope-from <gentoo-dev+bounces-23260-garchives=archives.gentoo.org@gentoo.org>) id 1Hj1rt-0007te-Ii for garchives@archives.gentoo.org; Tue, 01 May 2007 23:34:50 +0000 Received: from robin.gentoo.org (localhost [127.0.0.1]) by robin.gentoo.org (8.14.0/8.14.0) with SMTP id l41NXmtK006492; Tue, 1 May 2007 23:33:48 GMT Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) by robin.gentoo.org (8.14.0/8.14.0) with ESMTP id l41NVrJc004209 for <gentoo-dev@lists.gentoo.org>; Tue, 1 May 2007 23:31:53 GMT Received: from localhost (localhost [127.0.0.1]) by smtp.gentoo.org (Postfix) with ESMTP id 05FF0644B3 for <gentoo-dev@lists.gentoo.org>; Tue, 1 May 2007 23:31:53 +0000 (UTC) X-Virus-Scanned: amavisd-new at gentoo.org X-Spam-Score: -0.854 X-Spam-Level: X-Spam-Status: No, score=-0.854 required=5.5 tests=[AWL=-0.854] Received: from smtp.gentoo.org ([127.0.0.1]) by localhost (smtp.gentoo.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ev27puqZgskK for <gentoo-dev@lists.gentoo.org>; Tue, 1 May 2007 23:31:51 +0000 (UTC) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.171]) by smtp.gentoo.org (Postfix) with ESMTP id 7330C6446D for <gentoo-dev@gentoo.org>; Tue, 1 May 2007 23:31:46 +0000 (UTC) Received: from [82.82.178.19] (helo=sheridan.genone.homeip.net) by mrelayeu.kundenserver.de (node=mrelayeu4) with ESMTP (Nemesis), id 0ML21M-1Hj1ov1bJw-0003Bf; Wed, 02 May 2007 01:31:45 +0200 Date: Wed, 2 May 2007 01:32:20 +0200 From: Marius Mauch <genone@gentoo.org> To: gentoo-dev@lists.gentoo.org Subject: Re: [gentoo-dev] tests Message-ID: <20070502013220.7a3ae9a4@sheridan.genone.homeip.net> In-Reply-To: <200705011508.57220.peper@gentoo.org> References: <200705011508.57220.peper@gentoo.org> Organization: Gentoo X-Mailer: Claws Mail 2.9.1 (GTK+ 2.10.11; x86_64-pc-linux-gnu) Precedence: bulk List-Post: <mailto:gentoo-dev@lists.gentoo.org> List-Help: <mailto:gentoo-dev+help@gentoo.org> List-Unsubscribe: <mailto:gentoo-dev+unsubscribe@gentoo.org> List-Subscribe: <mailto:gentoo-dev+subscribe@gentoo.org> List-Id: Gentoo Linux mail <gentoo-dev.gentoo.org> X-BeenThere: gentoo-dev@gentoo.org Reply-to: gentoo-dev@lists.gentoo.org Mime-Version: 1.0 Content-Type: multipart/signed; boundary=Sig_Myx54YMPutcl_bDWxhuUN1Y; protocol="application/pgp-signature"; micalg=PGP-SHA1 X-Provags-ID: V01U2FsdGVkX1//PDmBwEW/2YPwwo8IHIpeRVxDUlhXqK/Uu7t B082OlHPwkzeypRdH4rzA0KfMZVO9JWX+U6PZqHnPOxTq7xU3X VoSl+NFuuyN55sxMxQGdg== X-Archives-Salt: 151c1069-3237-46a0-be2d-56883a424e1f X-Archives-Hash: 7d0a404f19455139116c6f3449e1a3eb --Sig_Myx54YMPutcl_bDWxhuUN1Y Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Tue, 1 May 2007 15:08:56 +0200 Piotr Jaroszy=C5=84ski <peper@gentoo.org> wrote: > Hello, >=20 > There was some discussion about forcing/not forcing tests in EAPI-1, > but there was clearly no compromise. Imho, tests are very important > and thus I want to discuss them a little more, but in more sensible > fashion. >=20 > Firstly each test can be(not all categories are mutually exclusive): > - not existant > - non-functional > - not runnable from ebuild > - useful but unreasonable resource-wise > - useful and reasonable resource-wise > - necessary > - known to partially fail but with a way of skipping failing tests > - known to partially fail but with no easy way of skipping failing > tests Is that list comprehensive? I'd approach it a bit different: Before creating fixed classification groups I'd first identify the attributes of tests that should be used for those classifications. a) cost (in terms of runtime, resource usage, additional deps) b) effectiveness (does a failing/working test mean the package is broken/working?) c) importance (is there a realistic chance for the test to be useful?) d) correctness (does the test match the implementation? overlaps a bit with effectiveness) e) others? Each of these needs to be considered if we want to find a good compromise of which tests to run and which not. A test with high cost can still be worth running if effectiveness, correctness and importance are also high, on the other hand a test with little effectiveness, correctness and/or importance probably isn't worth running even with zero cost. Now the tricky question is how to actually measure those attributes. > Secondly we must answer the question how precisely we want to > distinguish them, so users/dev can choose which categories of tests > they want to run. What comes to mind is: > - run all tests > - run only necessary tests > - run only reasonable tests > - don't run tests at all > Again, is that list comprehensive? Problem is that terms like "reasonable" or "necessary" are quite subjective (regarding both humans and machines), and in this special context even "all" could be interpreted in different ways (btw, could someone give some real examples for packages with "necessary" tests?). So I think a more fine grained classification is needed that can be adopted for specific use cases (e.g. the mips+embedded profiles might want different defaults than the amd64+desktop profiles). Marius --=20 Public Key at http://www.genone.de/info/gpg-key.pub In the beginning, there was nothing. And God said, 'Let there be Light.' And there was still nothing, but you could see a bit better. --Sig_Myx54YMPutcl_bDWxhuUN1Y Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.3 (GNU/Linux) iD8DBQFGN84IWzrL1pM7SNcRAp/CAJ9MGIvwdsxilSdn0sEJvKtH8ykrXgCfZU+k kGLrgYGB+XVRFsbsBVUJVpk= =XYvM -----END PGP SIGNATURE----- --Sig_Myx54YMPutcl_bDWxhuUN1Y-- -- gentoo-dev@gentoo.org mailing list