From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from lists.gentoo.org ([140.105.134.102] helo=robin.gentoo.org) by nuthatch.gentoo.org with esmtp (Exim 4.62) (envelope-from ) id 1Hiw2c-000325-8c for garchives@archives.gentoo.org; Tue, 01 May 2007 17:21:30 +0000 Received: from robin.gentoo.org (localhost [127.0.0.1]) by robin.gentoo.org (8.14.0/8.14.0) with SMTP id l41HKYvl030391; Tue, 1 May 2007 17:20:34 GMT Received: from griffon26.kfk4ever.com (griffon26.demon.nl [82.161.57.89]) by robin.gentoo.org (8.14.0/8.14.0) with ESMTP id l41HIdlF028080 for ; Tue, 1 May 2007 17:18:40 GMT Received: by griffon26.kfk4ever.com (Postfix, from userid 1000) id D571C3ABE; Tue, 1 May 2007 19:18:28 +0200 (CEST) Date: Tue, 1 May 2007 19:18:28 +0200 From: Maurice van der Pot To: gentoo-dev@lists.gentoo.org Subject: Re: [gentoo-dev] tests Message-ID: <20070501171828.GL10636@kfk4ever.com> Mail-Followup-To: gentoo-dev@lists.gentoo.org References: <200705011508.57220.peper@gentoo.org> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="4OpS+d6oOtUQaRm1" Content-Disposition: inline In-Reply-To: <200705011508.57220.peper@gentoo.org> X-PGP-Key: http://www.kfk4ever.com/~griffon26/pubkey.asc X-URL: http://www.kfk4ever.com/ X-Archives-Salt: 42793906-117f-4a8e-a09f-d955feb2434c X-Archives-Hash: 3ab7daf21f069b93afbe1581b4254bec --4OpS+d6oOtUQaRm1 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, May 01, 2007 at 03:08:56PM +0200, Piotr Jaroszy=C5=84ski wrote: > Firstly each test can be(not all categories are mutually exclusive): > - not existant > - non-functional > - not runnable from ebuild > - useful but unreasonable resource-wise > - useful and reasonable resource-wise > - necessary > - known to partially fail but with a way of skipping failing tests > - known to partially fail but with no easy way of skipping failing tests > Is that list comprehensive? Isn't it easier to list a set of boolean properties of _individual_ tests? =20 - We don't need "non existent".=20 - Non-functional and known to partially fail come down to "known to fail" for individual tests. =20 - I have no idea what "not runnable from ebuild" is. =20 - Unreasonable could be "resource hungry" or "needs additional deps" (those are two different things). - If a test is "necessary" I don't see why we should allow it to be skipped. - And about skipping failing tests. There is always a way to skip failing tests: not running any of them. It's just the granularity that is different, but the user doesn't care about that. > Secondly we must answer the question how precisely we want to distinguish= =20 > them, so users/dev can choose which categories of tests they want to run.= =20 > What comes to mind is: > - run all tests > - run only necessary tests > - run only reasonable tests > - don't run tests at all > Again, is that list comprehensive? I'd say, let the user decide based on the properties, fex: run known to fail : no run resource hungry: yes run additional deps: no if ( (known to fail =3D=3D false || run known to fail =3D=3D true) && (resource hungry =3D=3D false || run resource hungry =3D=3D true) && (additional deps =3D=3D false || run additional deps =3D=3D true) )=20 { run the test } So for each test/set of tests and for each possible reason not to run it, either the reason does not apply to this test or the user explicitly said to run it anyway. You don't see the "way of skipping failing tests" in this last part. This is because if the decision is made not to run a test, the smallest set that can be skipped (possibly all tests for an ebuild) are skipped. > Please don't post solutions unless we figure out which options we really = want=20 > to deliver. I'm sorry if this counts as a solution, but I wasn't sure the list you gave was the kind of list we need and I didn't know another way of explaining the use of the properties I listed. Regards, Maurice. --=20 Maurice van der Pot Gentoo Linux Developer griffon26@gentoo.org http://www.gentoo.org Creator of BiteMe! griffon26@kfk4ever.com http://www.kfk4ever.com --4OpS+d6oOtUQaRm1 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.3 (GNU/Linux) iD8DBQFGN3ZkMGnpIbeahxwRAopXAKCP0qudFBrF8988Zv3M8/uY1gu3pwCggsup iiRTnFLQUxjR0XDe7ah9dH0= =4Ly1 -----END PGP SIGNATURE----- --4OpS+d6oOtUQaRm1-- -- gentoo-dev@gentoo.org mailing list