From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from lists.gentoo.org ([140.105.134.102] helo=robin.gentoo.org) by nuthatch.gentoo.org with esmtp (Exim 4.62) (envelope-from ) id 1Hj2Rq-0007tJ-R9 for garchives@archives.gentoo.org; Wed, 02 May 2007 00:11:59 +0000 Received: from robin.gentoo.org (localhost [127.0.0.1]) by robin.gentoo.org (8.14.0/8.14.0) with SMTP id l420AGoQ007176; Wed, 2 May 2007 00:10:16 GMT Received: from mail.isohunt.com (b01.ext.isohunt.com [208.71.112.51]) by robin.gentoo.org (8.14.0/8.14.0) with ESMTP id l42087Gw004451 for ; Wed, 2 May 2007 00:08:08 GMT Received: (qmail 14319 invoked from network); 2 May 2007 00:08:07 -0000 Received: from Unknown (HELO curie.orbis-terrarum.net) (24.81.201.182) (smtp-auth username robbat2@isohunt.com, mechanism login) by mail.isohunt.com (qpsmtpd/0.33-dev on beta01) with (AES256-SHA encrypted) ESMTPSA; Wed, 02 May 2007 00:08:07 +0000 Received: (qmail 17120 invoked by uid 10000); 1 May 2007 17:08:06 -0700 Date: Tue, 1 May 2007 17:08:06 -0700 From: "Robin H. Johnson" To: gentoo-dev@lists.gentoo.org Subject: Re: [gentoo-dev] tests Message-ID: <20070502000806.GL7846@curie-int.orbis-terrarum.net> References: <200705011508.57220.peper@gentoo.org> <20070502013220.7a3ae9a4@sheridan.genone.homeip.net> <1178063216.1136.6.camel@athena.fprintf.net> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="MRqhvNSJsMAynmoC" Content-Disposition: inline In-Reply-To: <1178063216.1136.6.camel@athena.fprintf.net> User-Agent: Mutt/1.5.15 (2007-04-06) X-Archives-Salt: 958079cd-d168-4872-8898-30929040b804 X-Archives-Hash: 2c004a0432cef537b0903dfbadb5df08 --MRqhvNSJsMAynmoC Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, May 01, 2007 at 07:46:56PM -0400, Daniel Gryniewicz wrote: > On Wed, 2007-05-02 at 01:32 +0200, Marius Mauch wrote: > > I'd approach it a bit different: Before creating fixed classification > > groups I'd first identify the attributes of tests that should be used > > for those classifications. > > a) cost (in terms of runtime, resource usage, additional deps) > > b) effectiveness (does a failing/working test mean the package is > > broken/working?) > > c) importance (is there a realistic chance for the test to be useful?) > > d) correctness (does the test match the implementation? overlaps a bit > > with effectiveness) > > e) others? > There is one serious problem with this: Who's going to do the work to > figure all this out for the 11,000 odd packages in the tree? This seems > like a *huge* amount of work, work that I have no plan on doing for the > 100-odd packages I (help) maintain, let alone the 4-10 different > versions of each package. I highly doubt other maintainers want to do > this kind of work either. This wouldn't be an instant transition, and a lot of packages would be covered under the 'importance' side. Using crypto packages as an example, the costs are low (compare known input+output pairs), the effectiveness is high, the importance is high (witness the checksum problems caused in the tree some months ago), and the correctness is very high. The mysql testcases on the other hand have a low effectiveness, there have been lots of cases where they break due to userpriv or sandbox and high cost. For the packages I maintain, I'd definitely implement test stuff for the crypto and system-admin packages where feasible, but for a lot of others I wouldn't bother - the cost/benefit ratio is not high enough. --=20 Robin Hugh Johnson Gentoo Linux Developer & Council Member E-Mail : robbat2@gentoo.org GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 --MRqhvNSJsMAynmoC Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.2 (GNU/Linux) Comment: Robbat2 @ Orbis-Terrarum Networks - The text below is a digital signature. If it doesn't make any sense to you, ignore it. iD8DBQFGN9ZmPpIsIjIzwiwRAnSSAKCK257YSPjXrVoV5y3agw2EQ3exRQCgslSP o/AePlnj28SsgZSwtskjUcw= =sKHo -----END PGP SIGNATURE----- --MRqhvNSJsMAynmoC-- -- gentoo-dev@gentoo.org mailing list