On 05/10/2011 01:13 PM, Jorge Manuel B. S. Vicetto wrote:
> Hi.
> 
> Another issue that was raised in the discussion with the arch teams,
> even though it predates the arch teams resources thread as we've talked
> about it on FOSDEM 2011 and even before, is getting more automatic
> testing done on Gentoo.
> 
> I'm bcc'ing a few teams on this thread as it involves them and hopefully
> might interest them as well.
> 
> Both Release Engineering and QA teams would like to have more automatic
> testing to find breakages and to help track "when" things break and more
> importantly *why* they break.
> 
> To avoid misunderstandings, we already have testing and even automated
> testing being done on Gentoo. The "first line" of testing is done by
> developers using repoman and or the PM's QA tools. We also have
> individual developers and the QA team hopefully checking commits and
> everyone testing their packages.
> 
> Furtermore, the current weekly automatic stage building has helped
> identify some issues with the tree. The tinderbox work done by Patrick
> and Diego, as well as others, has also helped finding broken packages
> and or identifying packages affected by major changes before they hit
> the tree. The use of repoman, pcheck and or paludis quality assurance
> tools in the past and present to generate reports about tree issues,
> like Michael's (mr_bones) emails have also helped identifying and
> addressing issues.
> 
> Recently, we've got a new site to check the results of some tests
> http://qa-reports.gentoo.org/ with the possibility to add more scripts
> to provide / run even more tests.
> 
> So, why "more testing"? For starters, more *automatic* testing. Then
> more testing as reports from testing can help greatly in identifying
> when things break and why they break. As someone that looks over the
> automatic stage building for amd64 and x86, and that has to talk to
> teams / developers when things break, having more, more in depth and
> regular automatic testing would help my (releng) job. The work for the
> live-dvd would also be easier if the builds were "automated" and the job
> wasn't "restarted" every N months. Furthermore, creating a framework for
> developers to be able to schedule testing for proposed changes, in
> particular for substantial changes, might (should?) help improve the
> user's experience.
> 
> I hope you agree with "more testing" by now, but what testing? It's good
> to test something, but "what" do we want to test and "how" do we want to
> test?
> 
> 
> I think we should try to have at least the following categories of tests:
> 
>  * Portage (overlays?) QA tests
> 	tests with the existing QA tools to check the consistency of
> dependencies and the quality of ebuilds / eclasses.
> 
>  * (on demand?) package (stable / unstable) revdep rebuild (tinderbox)
> 	framework to schedule testing of proposed changes and check their impact
> 
>  * Weekly (?) stable / unstable stage / ISO arch builds
> 	the automatic stage building, including new specs for the testing tree
> as we currently only test the stable tree.
> 
>  * (schedule?) specific tailored stage4 builds
> 	testing of specific tailored "real world" images (web server, intranet
> server, generic desktop, GNOME desktop, KDE desktop, etc).
> 
>  * Bi-Weekly (?) stable / unstable AMD64/X86 LiveDVD builds
> 	automatic creation of the live-DVD to test a very broad set of packages
> 
>  * automated testing of built stage / CD / LiveDVD (KVM guest?) (CLI /
> GUI / log parsing ?)
> 	framework to test the built stages / install media and ensure it works
> as expected
> 
> 
> I don't have a framework for conducting some of these tests, including
> the stage/iso validation, but some of them can use the existing tools
> like the stage building and the tree QA tests.
> 
> Do you have any suggestions about the automatic testing? Do you know of
> other tests or tools that we can and should use to improve QA on Gentoo?

You might take a look at autotest from kernel.org. It's a Python based
framework for automating testing. It's specific towards kernel testing,
but could be modified for your needs.




-- 
Jack Morgan
Pub 4096R/761D8E0A 2010-09-13 Jack Morgan <jack@bonayri.com>
Fingerprint = DD42 EA48 D701 D520 C2CD 55BE BF53 C69B 761D 8E0A