From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1QKDOF-0002Gu-EM for garchives@archives.gentoo.org; Wed, 11 May 2011 17:40:03 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 3ACF91C033; Wed, 11 May 2011 17:39:49 +0000 (UTC) Received: from mail-pw0-f53.google.com (mail-pw0-f53.google.com [209.85.160.53]) by pigeon.gentoo.org (Postfix) with ESMTP id 54D7D1C011 for ; Wed, 11 May 2011 17:39:12 +0000 (UTC) Received: by pwj5 with SMTP id 5so525507pwj.40 for ; Wed, 11 May 2011 10:39:11 -0700 (PDT) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Received: by 10.142.135.4 with SMTP id i4mr4067800wfd.99.1305135551146; Wed, 11 May 2011 10:39:11 -0700 (PDT) Sender: antarus@scriptkitty.com Received: by 10.143.8.3 with HTTP; Wed, 11 May 2011 10:39:11 -0700 (PDT) In-Reply-To: <4DCA8B27.8030200@bonyari.com> References: <4DC99C77.1050105@gentoo.org> <4DCA8B27.8030200@bonyari.com> Date: Wed, 11 May 2011 10:39:11 -0700 X-Google-Sender-Auth: i5DxdaimdNr39UXwmRxKNXJhP4k Message-ID: Subject: Re: [gentoo-dev] Automatic testing on Gentoo From: Alec Warner To: gentoo-dev@lists.gentoo.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Archives-Salt: X-Archives-Hash: 4f21b5e4c4946a3814e1094704f49e00 On Wed, May 11, 2011 at 6:12 AM, Jack Morgan wrote: > > > On 05/10/2011 01:13 PM, Jorge Manuel B. S. Vicetto wrote: >> Hi. >> >> Another issue that was raised in the discussion with the arch teams, >> even though it predates the arch teams resources thread as we've talked >> about it on FOSDEM 2011 and even before, is getting more automatic >> testing done on Gentoo. >> >> I'm bcc'ing a few teams on this thread as it involves them and hopefully >> might interest them as well. >> >> Both Release Engineering and QA teams would like to have more automatic >> testing to find breakages and to help track "when" things break and more >> importantly *why* they break. >> >> To avoid misunderstandings, we already have testing and even automated >> testing being done on Gentoo. The "first line" of testing is done by >> developers using repoman and or the PM's QA tools. We also have >> individual developers and the QA team hopefully checking commits and >> everyone testing their packages. >> >> Furtermore, the current weekly automatic stage building has helped >> identify some issues with the tree. The tinderbox work done by Patrick >> and Diego, as well as others, has also helped finding broken packages >> and or identifying packages affected by major changes before they hit >> the tree. The use of repoman, pcheck and or paludis quality assurance >> tools in the past and present to generate reports about tree issues, >> like Michael's (mr_bones) emails have also helped identifying and >> addressing issues. >> >> Recently, we've got a new site to check the results of some tests >> http://qa-reports.gentoo.org/ with the possibility to add more scripts >> to provide / run even more tests. >> >> So, why "more testing"? For starters, more *automatic* testing. Then >> more testing as reports from testing can help greatly in identifying >> when things break and why they break. As someone that looks over the >> automatic stage building for amd64 and x86, and that has to talk to >> teams / developers when things break, having more, more in depth and >> regular automatic testing would help my (releng) job. The work for the >> live-dvd would also be easier if the builds were "automated" and the job >> wasn't "restarted" every N months. Furthermore, creating a framework for >> developers to be able to schedule testing for proposed changes, in >> particular for substantial changes, might (should?) help improve the >> user's experience. >> >> I hope you agree with "more testing" by now, but what testing? It's good >> to test something, but "what" do we want to test and "how" do we want to >> test? >> >> >> I think we should try to have at least the following categories of tests= : >> >> =C2=A0* Portage (overlays?) QA tests >> =C2=A0 =C2=A0 =C2=A0 tests with the existing QA tools to check the consi= stency of >> dependencies and the quality of ebuilds / eclasses. These are almost separate. I assume your intent was 'lets automate pcheck & co. runs of gentoo-x86 and if we get that working we can add overlays from layman' which sounds fine to me ;) >> >> =C2=A0* (on demand?) package (stable / unstable) revdep rebuild (tinderb= ox) >> =C2=A0 =C2=A0 =C2=A0 framework to schedule testing of proposed changes a= nd check their impact I'd be curious what the load is here. We are adopting an on-demand testing infrastructure at work. Right now we have a continuous build but it is time-delta based and not event-based so it groups changes together which makes it hard to find what broke things. At work we only submit a few changes a day though, so we need a very small infrastructure to test each change. Gentoo has way more commits (at least one every few minutes on average, and then there are huge commits like KDE stablization...) What I'd recommend here is essentially some kind of control field in the commit itself (commitmsg?) that controls exactly what tests are done for that commit. >> >> =C2=A0* Weekly (?) stable / unstable stage / ISO arch builds >> =C2=A0 =C2=A0 =C2=A0 the automatic stage building, including new specs f= or the testing tree >> as we currently only test the stable tree. I'm curious if you constantly build unstable..do you plan on fixing it? My understanding of Gentoo is that in ~arch something is always slightly broken and thats OK. I worry that ~arch builds may just end up being noise because they don't build properly due to the high velocity of changes. >> >> =C2=A0* (schedule?) specific tailored stage4 builds >> =C2=A0 =C2=A0 =C2=A0 testing of specific tailored "real world" images (w= eb server, intranet >> server, generic desktop, GNOME desktop, KDE desktop, etc). Again it would be interesting to have some kind of control field in my commits so when KDE is stable I could trigger a build of the 'KDE stage4' or whatnot. If we ever finish this gentoo-stats project it would be interesting to see what users are actually using as well. Do users use the defaults? Are the stage4's we are testing actually relevant? >> >> =C2=A0* Bi-Weekly (?) stable / unstable AMD64/X86 LiveDVD builds >> =C2=A0 =C2=A0 =C2=A0 automatic creation of the live-DVD to test a very b= road set of packages >> >> =C2=A0* automated testing of built stage / CD / LiveDVD (KVM guest?) (CL= I / >> GUI / log parsing ?) >> =C2=A0 =C2=A0 =C2=A0 framework to test the built stages / install media = and ensure it works >> as expected I think testing that the liveDVD we just built boots is a decent test (and probably not to difficult to write.) Testing that 'everything on the DVD works' is likely more of a challenge and I'm not sure it buys us anything. Do we often find that we release LiveDVDs with broken software? >> >> >> I don't have a framework for conducting some of these tests, including >> the stage/iso validation, but some of them can use the existing tools >> like the stage building and the tree QA tests. >> >> Do you have any suggestions about the automatic testing? Do you know of >> other tests or tools that we can and should use to improve QA on Gentoo? > > You might take a look at autotest from kernel.org. It's a Python based > framework for automating testing. It's specific towards kernel testing, > but could be modified for your needs. Autotest would likely require a branch and a fair bit of work to be used for OS qualification. We use it for OS qualification at work (Goobuntu@Google) While I hesitate to say 'roll your own' if you can get something working in 1-2 months I can certainly see it being easier to maintain than autotest...there really is not a killer feature that autotest has. The reporting / graphing is pretty bad, it uses ssh for everything and basically keeps long-running connections open (might be fine if you are using kvm..but not over the WAN), the API is terrible and requires all kinds of horrible-ness to use...I could go on ; > > > > > -- > Jack Morgan > Pub 4096R/761D8E0A 2010-09-13 Jack Morgan > Fingerprint =3D DD42 EA48 D701 D520 C2CD 55BE BF53 C69B 761D 8E0A > >