[gentoo-releng] Checklists for testers and QA

[gentoo-releng] Checklists for testers and QA

[Chris Gianelloni]

Well, I'm sure all of you are still reading over the summary and meeting
logs, but when you're done, I'd like to get everyone's involvement in
helping create the three checklists we plan on distributing.  The lists
are Beta, QA, and Release.  For the most part, they will overlap quite
heavily, but I still see the need for keeping them separate.

In the case of Beta and Release, we'll also want to ask people for
specific information about their environment (hardware, software, etc.)
that will help us to troubleshoot the issues and focus more energy on
the worst problems.

I'll start us off:

- Which media did you download? (Minimal, Universal, LiveCD, LiveDVD)
- Did the media pass MD5/SHA1 checks? (If you didn't check, say so.)

- Does the media boot on your system?
- If not, what errors were reported?
- Did you use any command line options?
- If so, which ones?

- After boot, were all of your devices detected?
- If not, what was not detected?
- What driver does this device use?
- Is the driver on the media? (Say "don't know" if you're unsure.)

- Did previous releases work where this one is not?

- What chipset is your motherboard? IDE? SATA?
- Do you use external controllers for SATA/SAS/SCSI/etc.?
- If so, which ones?
- How much RAM is in your system?
- Did you try to boot with "docache" enabled?
- What video card(s) do you have?
- Was it properly detected?
- Did X start properly?

This should be a pretty decent start and give everyone an idea of what
we would be looking for with these checklists.  What things do you think
need to be checked for and what information do you think we would need
to properly diagnose any issues?

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Games Developer
-- 
gentoo-releng@lists.gentoo.org mailing list

Re: [gentoo-releng] Checklists for testers and QA

[Chris Gianelloni]

On Thu, 2008-01-24 at 16:38 -0800, Chris Gianelloni wrote:
> In the case of Beta and Release, we'll also want to ask people for
> specific information about their environment (hardware, software, etc.)
> that will help us to troubleshoot the issues and focus more energy on
> the worst problems.

So nobody has *any* ideas, at all for this?

It's been a week and there hasn't been a *single* response.

This is everyone's chance to give their input on what we should have
checked on the release media.  If you do not speak up now, it isn't
likely that it will get checked.

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Games Developer
-- 
gentoo-releng@lists.gentoo.org mailing list

Re: [gentoo-releng] Checklists for testers and QA

[Denys Fedoryshchenko]

Hi

I can test on some recent hardware. Mostly Intel motherboards, such as DP35DP.
Also i have some interesting cases like Sony Vaio tiny laptop (without 
embedded CD-ROM, so it is USB, a bit non-standart environment), installing 
gentoo in qemu.
Probably i will be able to test Gentoo also in local shop who is selling 
Apple products, on Apple hardware.

I am very interested in remote gentoo installations, so would like to test 
new gentoo release for such cases. But probably i am a bit late for this?

Or i understand wrong question?

On Thu, 31 Jan 2008 17:40:06 -0800, Chris Gianelloni wrote
> On Thu, 2008-01-24 at 16:38 -0800, Chris Gianelloni wrote:
> > In the case of Beta and Release, we'll also want to ask people for
> > specific information about their environment (hardware, software, etc.)
> > that will help us to troubleshoot the issues and focus more energy on
> > the worst problems.
> 
> So nobody has *any* ideas, at all for this?
> 
> It's been a week and there hasn't been a *single* response.
> 
> This is everyone's chance to give their input on what we should have
> checked on the release media.  If you do not speak up now, it isn't
> likely that it will get checked.
> 
> -- 
> Chris Gianelloni
> Release Engineering Strategic Lead
> Games Developer
> -- 
> gentoo-releng@lists.gentoo.org mailing list


--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.

-- 
gentoo-releng@lists.gentoo.org mailing list

Re: [gentoo-releng] Checklists for testers and QA

[M. Edward (Ed) Borasky]

Chris Gianelloni wrote:
> On Thu, 2008-01-24 at 16:38 -0800, Chris Gianelloni wrote:
>> In the case of Beta and Release, we'll also want to ask people for
>> specific information about their environment (hardware, software, etc.)
>> that will help us to troubleshoot the issues and focus more energy on
>> the worst problems.
> 
> So nobody has *any* ideas, at all for this?
> 
> It's been a week and there hasn't been a *single* response.
> 
> This is everyone's chance to give their input on what we should have
> checked on the release media.  If you do not speak up now, it isn't
> likely that it will get checked.

I have a 4 GB Athlon64 X2, so I'm planning on testing on that. One thing 
I might also test is building LiveCDs and LiveDVDs via Catalyst.

By the way, I've got the 2.6.24 kernel and so far it's absolutely 
flawless on my machine, including the latest "nvidia-drivers" ebuild.
> 

-- 
gentoo-releng@lists.gentoo.org mailing list

Re: [gentoo-releng] Checklists for testers and QA

[Andrew Gaffney]

M. Edward (Ed) Borasky wrote:
> Chris Gianelloni wrote:
>> On Thu, 2008-01-24 at 16:38 -0800, Chris Gianelloni wrote:
>>> In the case of Beta and Release, we'll also want to ask people for
>>> specific information about their environment (hardware, software, etc.)
>>> that will help us to troubleshoot the issues and focus more energy on
>>> the worst problems.
>>
>> So nobody has *any* ideas, at all for this?
>>
>> It's been a week and there hasn't been a *single* response.
>>
>> This is everyone's chance to give their input on what we should have
>> checked on the release media.  If you do not speak up now, it isn't
>> likely that it will get checked.
> 
> I have a 4 GB Athlon64 X2, so I'm planning on testing on that. One thing 
> I might also test is building LiveCDs and LiveDVDs via Catalyst.
> 
> By the way, I've got the 2.6.24 kernel and so far it's absolutely 
> flawless on my machine, including the latest "nvidia-drivers" ebuild.

You guys misunderstand. He's not asking for testers. He's asking for things to 
add to the standard list of questions that testers should provide answers to 
when filing a bug report for a beta release.

-- 
Andrew Gaffney                                 http://dev.gentoo.org/~agaffney/
Gentoo Linux Developer             Catalyst/Installer + x86 release coordinator
-- 
gentoo-releng@lists.gentoo.org mailing list

Re: [gentoo-releng] Checklists for testers and QA

[John Alberts]

On Jan 31, 2008 7:40 PM, Chris Gianelloni <wolf31o2@gentoo.org> wrote:
> So nobody has *any* ideas, at all for this?
>
> It's been a week and there hasn't been a *single* response.

I think the list of questions you have suggested is fine.  The only
thing I would add is a comments section at the end of the survey.


-- 
John Alberts
-- 
gentoo-releng@lists.gentoo.org mailing list

Re: [gentoo-releng] Checklists for testers and QA

[John Alberts]

Also, I think instead of asking the user what type of hardware they
have, they should be asked to attach the output of lspci, lsusb, and
cat /proc/cpuinfo.
I think someone else previously mentioned using a program called
hwreport.  I just gave it a try.  It's only 11k and it's in portage.
It basically just dumps info from various important entries under
/proc and creates a tarball.  So, after I installed it, I ran '#
hwreport reportname' and it instantly created a reportname.tar.bz2
file.  This could be attached with the survey.
Of course, ideally it would be easier on the folks trying to sort out
this information if there was a script to parse they surveys, extract
the tarball and upload the info to a database, so you can just view
the compiled info.

John Alberts


On Jan 31, 2008 9:06 PM, John Alberts <john.m.alberts@gmail.com> wrote:
> On Jan 31, 2008 7:40 PM, Chris Gianelloni <wolf31o2@gentoo.org> wrote:
> > So nobody has *any* ideas, at all for this?
> >
> > It's been a week and there hasn't been a *single* response.
>
> I think the list of questions you have suggested is fine.  The only
> thing I would add is a comments section at the end of the survey.
>
>
> --
> John Alberts
>



-- 
John Alberts
-- 
gentoo-releng@lists.gentoo.org mailing list

Re: [gentoo-releng] Checklists for testers and QA

[Rémi Cardona]

Chris Gianelloni a écrit :
> On Thu, 2008-01-24 at 16:38 -0800, Chris Gianelloni wrote:
>> In the case of Beta and Release, we'll also want to ask people for
>> specific information about their environment (hardware, software, etc.)
>> that will help us to troubleshoot the issues and focus more energy on
>> the worst problems.
> 
> So nobody has *any* ideas, at all for this?

I'd say the list you've provided is already a really good one. We're 
talking about a live media here, one that hopefully you won't be using 
for a very long time (between 2 and 48 hours, I'd say).

So really, the live media needs basic support for basic hardware : disc 
controllers and network devices. That's what the list should be focusing 
on, and that's exactly what your list focuses on :)

Let's go with what you've come up with and we'll see if there are major 
pitfalls along the way.

Cheers,

Rémi
-- 
gentoo-releng@lists.gentoo.org mailing list

Re: [gentoo-releng] Checklists for testers and QA

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 1418 bytes --]

On Thu, 2008-01-31 at 21:06 -0600, John Alberts wrote:
> On Jan 31, 2008 7:40 PM, Chris Gianelloni <wolf31o2@gentoo.org> wrote:
> > So nobody has *any* ideas, at all for this?
> >
> > It's been a week and there hasn't been a *single* response.
> 
> I think the list of questions you have suggested is fine.  The only
> thing I would add is a comments section at the end of the survey.

No, it isn't.  That list was what I could come up with in about 30
seconds without even thinking about it.  I intentionally went light on
my own ideas to *try* to get people to actually give their input, since
everybody constantly complains about how their voice isn't heard by
Release Engineering and that I'm always off doing whatever I like.
People also complain about the quality.  Well, this is people's chance
to suggest anything, openly.  Though this has always been the case,
nobody has ever really exercised it, so we decided to announce it again.
At any rate, nobody is saying anything, which really just reminds me why
I didn't bother to ask before... ;]

Is it really that nobody cares or is everybody just being lazy and
saying "yeah, that's good enough" when I intentionally didn't list lots
of things that I'd came up with when writing the original list.  I
honestly don't know which it is here.

/me is confused

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Games Developer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-releng] Checklists for testers and QA

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 681 bytes --]

On Thu, 2008-01-31 at 21:22 -0600, John Alberts wrote:
> Also, I think instead of asking the user what type of hardware they
> have, they should be asked to attach the output of lspci, lsusb, and
> cat /proc/cpuinfo.

Hardware information is fine and dandy, but having some piece of
critical software not working that stops your installation is just as
bad as missing/broken hardware support.

I'd like to try to come up with some other tests.  Realize that bugs in
the Beta could be resolved before the final snapshot is made.  The more
we get fixed, the higher the quality of the release.

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Games Developer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-releng] Re: Checklists for testers and QA

[Christian Faulhammer]

[-- Attachment #1.1: Type: text/plain, Size: 1834 bytes --]

Hi,

Chris Gianelloni <wolf31o2-aBrp7R+bbdUdnm+yROfE0A@public.gmane.org>:
> I'll start us off:
> 
> - Which media did you download? (Minimal, Universal, LiveCD, LiveDVD)
> - Did the media pass MD5/SHA1 checks? (If you didn't check, say so.)
> 
> - Does the media boot on your system?
> - If not, what errors were reported?
> - Did you use any command line options?
> - If so, which ones?
> 
> - After boot, were all of your devices detected?
> - If not, what was not detected?
> - What driver does this device use?
> - Is the driver on the media? (Say "don't know" if you're unsure.)
> 
> - Did previous releases work where this one is not?
> 
> - What chipset is your motherboard? IDE? SATA?
> - Do you use external controllers for SATA/SAS/SCSI/etc.?
> - If so, which ones?
> - How much RAM is in your system?
> - Did you try to boot with "docache" enabled?
> - What video card(s) do you have?
> - Was it properly detected?
> - Did X start properly?
 
> This should be a pretty decent start and give everyone an idea of what
> we would be looking for with these checklists.  What things do you
> think need to be checked for and what information do you think we
> would need to properly diagnose any issues?

 About hardware information.  I proposed app-admin/hwreport for that,
which gathers information from lspci and friends (I attach one to this
mail) and tars it up.  Some more questions (I never got the call on
-core, btw...):

* Did you try a lot of the software on the live CD/DVD?  Did they work
fine?

* Are all links on the desktop working (terminal, Handbook etc.)?

* Was splash image displayed correctly?

V-Li



-- 
Christian Faulhammer, Gentoo Lisp project
<URL:http://www.gentoo.org/proj/en/lisp/>, #gentoo-lisp on FreeNode

<URL:http://www.faulhammer.org/>

[-- Attachment #1.2: test.tar.bz2 --]
[-- Type: application/x-bzip, Size: 10780 bytes --]

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-releng] Checklists for testers and QA

[Rémi Cardona]

Chris Gianelloni a écrit :
> Is it really that nobody cares or is everybody just being lazy and
> saying "yeah, that's good enough" when I intentionally didn't list lots
> of things that I'd came up with when writing the original list.  I
> honestly don't know which it is here.

Do we know how many people actually use the software on the live media? 
Do they care if it's not 100% functional?

<personal experience> When I install Gentoo, I'll use which ever live 
CD/DVD/USB I can find, if not, I'll download the minimal install CD and 
the only pieces of software I'll use is dhcpcd and a web browser to read 
the handbook. I'll occasionally use screen and ssh{,d} if the box is too 
far from my desk.

So once you release a beta, that's what I'll be trying out first and 
foremost.

But, wouldn't it be interesting to find out the usage patterns for our 
users before making a big list of things to check?

Maybe not enough users do care to make it block QA and releng's time 
when making new releases.

> /me is confused

Don't be :)

Cheers,

Rémi
-- 
gentoo-releng@lists.gentoo.org mailing list

Re: [gentoo-releng] Checklists for testers and QA

[Markus Hauschild]

Hi,
I think I've come up with some more questions:
* did you use the graphical installer or did you install manually (and why)
* did everything with the graphical installer work as expected (if
used, and explain what did not work as expected)

--
Cheers, Markus
-- 
gentoo-releng@lists.gentoo.org mailing list

RE: [gentoo-releng] Checklists for testers and QA

[Nelson Batalha]

[-- Attachment #1: Type: text/plain, Size: 2572 bytes --]


 
-Does your computer run other OS/distributions without the same problems? If so, please list them (optionally along with drivers/modifications you installed/made in it).


For the LiveCD/DVD:

-Did you start all applications? In any case, are any of them failing? If so, what is reported when you run them on the command line?
 
> Subject: [gentoo-releng] Checklists for testers and QA> From: wolf31o2@gentoo.org> To: gentoo-dev-announce@lists.gentoo.org> CC: gentoo-releng@lists.gentoo.org> Date: Thu, 24 Jan 2008 16:38:22 -0800> > Well, I'm sure all of you are still reading over the summary and meeting> logs, but when you're done, I'd like to get everyone's involvement in> helping create the three checklists we plan on distributing. The lists> are Beta, QA, and Release. For the most part, they will overlap quite> heavily, but I still see the need for keeping them separate.> > In the case of Beta and Release, we'll also want to ask people for> specific information about their environment (hardware, software, etc.)> that will help us to troubleshoot the issues and focus more energy on> the worst problems.> > I'll start us off:> > - Which media did you download? (Minimal, Universal, LiveCD, LiveDVD)> - Did the media pass MD5/SHA1 checks? (If you didn't check, say so.)> > - Does the media boot on your system?> - If not, what errors were reported?> - Did you use any command line options?> - If so, which ones?> > - After boot, were all of your devices detected?> - If not, what was not detected?> - What driver does this device use?> - Is the driver on the media? (Say "don't know" if you're unsure.)> > - Did previous releases work where this one is not?> > - What chipset is your motherboard? IDE? SATA?> - Do you use external controllers for SATA/SAS/SCSI/etc.?> - If so, which ones?> - How much RAM is in your system?> - Did you try to boot with "docache" enabled?> - What video card(s) do you have?> - Was it properly detected?> - Did X start properly?> > This should be a pretty decent start and give everyone an idea of what> we would be looking for with these checklists. What things do you think> need to be checked for and what information do you think we would need> to properly diagnose any issues?> > -- > Chris Gianelloni> Release Engineering Strategic Lead> Games Developer> -- > gentoo-releng@lists.gentoo.org mailing list> 
_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE!
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

[-- Attachment #2: Type: text/html, Size: 3398 bytes --]

Re: [gentoo-releng] Checklists for testers and QA

[Pacho Ramos]

El jue, 31-01-2008 a las 17:40 -0800, Chris Gianelloni escribió:
> On Thu, 2008-01-24 at 16:38 -0800, Chris Gianelloni wrote:
> > In the case of Beta and Release, we'll also want to ask people for
> > specific information about their environment (hardware, software, etc.)
> > that will help us to troubleshoot the issues and focus more energy on
> > the worst problems.
> 
> So nobody has *any* ideas, at all for this?
> 
> It's been a week and there hasn't been a *single* response.
> 
> This is everyone's chance to give their input on what we should have
> checked on the release media.  If you do not speak up now, it isn't
> likely that it will get checked.
> 
> -- 
> Chris Gianelloni
> Release Engineering Strategic Lead
> Games Developer

I don't know what could I say:

1. On my laptop I still use ipw3945 driver as I am unable to make
iwl3945 work with my 3945 card

2. On my desktop I have to stay in nvidia-drivers-100.14.19 because
169.07 and 169.09 makes my monitor be set to unproper resolutions with
fullscreen games

But I dubt gentoo could do something with nvidia problem, and I know
that a lot of people also have problems with iwlwifi driver even with
other distributions :-/



-- 
gentoo-releng@lists.gentoo.org mailing list

Re: [gentoo-releng] Checklists for testers and QA

[M. Edward (Ed) Borasky]

Rémi Cardona wrote:
> Do we know how many people actually use the software on the live media? 
> Do they care if it's not 100% functional?

I don't know about new users, but I suspect many veteran users use 
Gentoo media only in the extremely unlikely event that they've blasted 
their system to the point where it's necessary, or when they've acquired 
a new box. The other major use case, of course, is giving media to 
friends who are unfamiliar with Gentoo.

There have been occasions where I would boot the Gentoo (and others') 
LiveDVD on my Windows workstation at my day job and use it as a Linux 
workstation, but VMware Server pretty much eliminated that use case. 
Corporate IT policy forbids me from running beta software, at least that 
of vendors, because of licensing issues, but that may not apply to open 
source betas.

> But, wouldn't it be interesting to find out the usage patterns for our 
> users before making a big list of things to check?
> 
> Maybe not enough users do care to make it block QA and releng's time 
> when making new releases.

I can't speak for either QA or releng, but you may have a point. Veteran 
users most likely will have the tools and motivation to file accurate 
bugs without a checklist, and newbies probably will be looking for 
something simpler, like,

a. Does it come up to a desktop, find my network, can I browse the web, 
read email and edit a document in OpenOffice?

b. Why is the start button in the upper left corner?

c. Does the wireless work? Does the sound work? Where's the &$^#& virus 
scanner?

d. Well, I've had enough of Linux -- did it trash my Windows install, or 
can I eject the CD and give it to my 13-year-old gamer cousin and get 
back to work?

:)
-- 
gentoo-releng@lists.gentoo.org mailing list

Re: [gentoo-releng] Re: Checklists for testers and QA

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 912 bytes --]

On Fri, 2008-02-01 at 09:49 +0100, Christian Faulhammer wrote:
> * Did you try a lot of the software on the live CD/DVD?  Did they work
> fine?

This is good, but I'd prefer we list specific pieces of software, since
the CD/DVD is also used for networkless installs, the software needs to
work on the CD/DVD for it to work on their machines.

> * Are all links on the desktop working (terminal, Handbook etc.)?

Good one.  Thanks!

> * Was splash image displayed correctly?

I'll add this one if we have a splash.  Current catalyst/genkernel
doesn't really support the new splash stuff properly, so I'll either
need to hack in the code for it in a very short time, or I'll need to
drop the splash.  I'm definitely going to dump it for the minimal CD, if
only to reduce space usage further.

Thanks for the ideas!

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Games Developer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-releng] Checklists for testers and QA

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 2308 bytes --]

On Fri, 2008-02-01 at 09:58 +0100, Rémi Cardona wrote:
> Chris Gianelloni a écrit :
> > Is it really that nobody cares or is everybody just being lazy and
> > saying "yeah, that's good enough" when I intentionally didn't list lots
> > of things that I'd came up with when writing the original list.  I
> > honestly don't know which it is here.
> 
> Do we know how many people actually use the software on the live media? 

Yes.  Every single person who uses the Installer uses the software on
the disk.

> Do they care if it's not 100% functional?

Yes, since anything non-functional ends up non-functional on their
machines.

Now, could we focus less on whether checking the software is a valid use
of time or not and instead look at *what* we should test within that
software?

> <personal experience> When I install Gentoo, I'll use which ever live 
> CD/DVD/USB I can find, if not, I'll download the minimal install CD and 
> the only pieces of software I'll use is dhcpcd and a web browser to read 
> the handbook. I'll occasionally use screen and ssh{,d} if the box is too 
> far from my desk.

We don't really care what you use.  I mean no offense by this, but it
doesn't help *this* conversation any.

> So once you release a beta, that's what I'll be trying out first and 
> foremost.

Sure.  That's the absolute minimal amount of testing that we normally
perform.  All we gain is knowing it works on your specific hardware and
not much else, since you're planning on testing the parts we test
extensively.

> But, wouldn't it be interesting to find out the usage patterns for our 
> users before making a big list of things to check?

Not really.  The quality should be as perfect as possible on everything,
not just the major things.  The whole point here is to get community
involvement, which means many more eyes so we can cover much more
ground.

> Maybe not enough users do care to make it block QA and releng's time 
> when making new releases.

We *want* to know what to test.  That's the whole point.  We've only
been testing certain things due to time constraints.  If we had a more
comprehensive list, it would make a better release.  Plain and simple...

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Games Developer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-releng] Re: [gentoo-dev-announce] Checklists for testers and QA

[Vlastimil Babka]

[-- Attachment #1: Type: text/plain, Size: 809 bytes --]

Chris Gianelloni wrote:
> This should be a pretty decent start and give everyone an idea of what
> we would be looking for with these checklists.  What things do you think
> need to be checked for and what information do you think we would need
> to properly diagnose any issues?

Well quite obvious questions but not mentioned in the e-mail:

- what installation method did you use? (manual/installer)
- if manual, did following the instructions exactly work, or were any 
extra steps needed or some steps performed differently? any missing 
information? also which optional parts you performed, which not (I guess 
that might be about doc issues but still)
- if installer, what method (networkless/advanced), what settings
- if installer, did all work or failed, where, post the log?

VB


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]