[gentoo-dev] Re: [gentoo-osx] Package testing -- Automated initiative

[gentoo-dev] Re: [gentoo-osx] Package testing -- Automated initiative

[Kito]

Adding -dev to CC: in case someone has any meaningful input or has  
already tackled this problem...

On Aug 14, 2005, at 8:21 AM, Grobian wrote:

> Introduction
> ============
>
> Recently, once again we were confronted with a package marked as
> ppc-macos stable, while it didn't compile at all, let alone run.   
> It is
> believed more of these packages are in portage, and need to be  
> found and
> fixed.  Keeping the cause of why they are marked stable up to another
> discussion, and out of the scope of this discussion, I will focus  
> on how
> to track these packages down and report them to us.
>
> In the secondary line, all 'unstable' packages, marked ~ppc-macos  
> should
> be tested as well, since they can be faulty as well.  Since for OSX  
> much
> is in ~ppc-macos, many users consider it a normal procedure to  
> switch to
> the unstable side of portage, hence some extra need for careful  
> testing
> of ~ppc-macos also.
>
>
> Proposed Global Structure
> =========================
>
> Testing should be done on a regular basis, both push and pull based.
> This means that the testing machine would start testing packages  
> itself
> if it is out of work, and on the other hand starts testing packages as
> soon as they are being added/changed in CVS.  It may need no great
> imagination to see that the latter 'push-based' activity has priority
> over the 'pull-based' work.

I'm not sure direct interaction with CVS would be needed, usually  
only takes a short time for cvs commits to hit the rsync mirrors  
(hence the volatile nature of the tree)

>
> Starting over, will for the test machine mean that it starts cleaning
> out its world file.  Cleaning this file out to a bare minimum is an
> important aspect of getting a test environment that reflects the
> situation on new user's machines.  If an ebuild uses a package without
> having it in it's DEPEND, this may get noticed only when starting on a
> clean machine.  This, however, will add a big delay in testing as many
> packages will need to be built prior the right package can be  
> installed.
>
> The testing machine will have a queue file, which it reads packages to
> emerge from.  If the queue file is empty, i.e. when there is no push
> based work, the machine will generate work by starting to compile
> uncompiled packages, or emptying the tree.
>
> Because ~ppc-macos and ppc-macos packages interfere with each other  
> -- a ~ppc-macos package overwrites a ppc-macos package -- both  
> stable and unstable have to be dealt with separately, i.e. they  
> should both have their own environment either via two separate  
> machines, or through the use of a chroot jail.

I think seperate chroots are definitely the way to go. We can just  
store a 'pristine' chroot in iso or dmg or whatever on the build  
server and copy when needed.

>
>
> Queues
> ------
>
> In order not to drag in a full DBMS (in the end Portage already is  
> one) queues are just simple flat files consisting of absolute  
> package names, one per line.  Table wise locking granularity is  
> handled by the OS as one process opens the file in write mode.   
> Consumers -- the testing box in this case -- read the first line  
> and delete it, while producers simply add one line (or more) to the  
> end of the file.
>
> The queue itself, is more a set than a list.  This means that  
> packages that are in the queue, should be unique.  If a package is  
> added that is already in the queue, it is dropped such that the  
> original queue position of the package is maintained.

Maybe a 'proper' dbms wouldn't be such a bad idea, could also store  
build logs, timestamps, etc. there and make it easier for multiple  
build hosts to push/pull from a centralized server.

>
>
> CVS Producer
> ------------
>
> To catch up automatically with changes made to the tree, it is  
> necessary to act upon any commit to the tree for an ebuild file.  A  
> possibility to do this would be via processing of CVS commit  
> messages, sent out as email by the CVS server.  It is a task of the  
> producer to find out whether the ebuild found applies to the  
> testing machine (ppc-macos) and add the package/ebuild to the queue.
>
>
> Consumer (testing process)
> --------------------------
>
> The test machine reads a line from the queue, and basically  
> executes 'emerge ${PACKAGE}'.  However, before doing this, first it  
> figures out which use flags can be used (emerge -pv) and which  
> dependencies will be pulled (emerge -pt).  If portage returns the  
> message all ebuilds that could satisfy X have been masked, the  
> emerge is cancelled, the line is removed from the queue and an  
> email message will be sent out.
>
> All dependencies are put in the right order and emerged as normal  
> packages, that is: all dependencies are pushed at the front of the  
> queue, thereby keeping uniqueness of the queue and removing  
> duplicates that appear later on in the queue.  After this, the  
> consumer is restarted and reads again from the queue.  This should  
> result in usually merging only one package at a time, and as such  
> quite isolated cases, which should improve the error email  
> notification service.
>
> Compile testing a package is supposed to be a thorough test that  
> tries all possible combinations of the package's USE flags.  As  
> this might be somewhat endless as some packages are rather big and  
> have zillions of USE flags, it may be necessary to have a special  
> "don't do it" file.
> Since all dependencies were put at the front of the queue, there  
> should normally be no dependencies that the package pulls.
> If compilation fails for a certain USE-flag combination this is  
> reported by sending out an email, and compilation of the next USE- 
> flag combination is attempted.
>
> When everything goes fine, no email notification is being sent  
> out.  A convenient log structure would, however, make it possible  
> to see which packages and USE-flag combinations successfully passed  
> through. Providing this log via a web-page would be a useful  
> thing.  Again backing this with a DBMS to allow easy searching,  
> versioning and stuff is considered to be overhead, though crafting  
> logs in SQL's "INSERT INTO" format might enable another machine to  
> display the output data. Perhaps the communication methods needs a  
> section on itself.
>
>
> Recap and Conclusion
> ====================
>
> By setting up a testing system, it is possible to greatly improve  
> the Quality of Service of the portage tree for an architecture by  
> exhaustive testing of both packages already in there, as well as  
> packages added or modified.  Automated testing should not release  
> developers from testing themselves, but should help in pointing out  
> problems that may arise on moving grounds such as portage where  
> packages are constantly updated and dependencies might get broken.
>
>
> ToDo
> ====
>
> - Not only check dependencies of the respective package, but also  
> consider packages that depend on the respective package, thus  
> rebuilding all packages that depend on the package to check if  
> anything is broken by the update.
> - Is there a gleptomaniac in the room?  This would be useful for  
> x86 also, of course.  In that case it may be necessary to make sure  
> the packages are split over multiple machines.
> - The message system needs more customisation options, especially  
> backing things by a DBMS would allow for many nice bugzilla-like  
> preferences for email generation as well as web-based versioned  
> info/report pages
> - To make the system even bigger, a central DBMS powered server  
> might take a leading role and ... {editor note: wait, stop it right  
> now, you're going too fast right now}
>
>
> By The Way
> ==========
>
> - Kito offers his lil' chico as machine for this automated testing  
> initiative.
> - Comments are welcome, as well as expressions of worry on my  
> mental state.
> - Implementation of described system will need some better  
> specified system and needs some coding (the dirty work) in some  
> language...
>
>
> -- 
> Fabian Groffen
> eBuild && Porting
> Gentoo for Mac OS X
>
> -- 
> gentoo-osx@gentoo.org mailing list
>
>

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: [gentoo-osx] Package testing -- Automated initiative

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1580 bytes --]

On Sun, Aug 14, 2005 at 09:00:19AM -0500, Kito wrote:
> Adding -dev to CC: in case someone has any meaningful input or has  
> already tackled this problem...
We do already have an application capable of doing tinderboxing for
packages. It's one of the abilities of Catalyst.

So all you would need to do is hook up your queue/control system to
call Catalyst for actually doing the build env. setup and actual build.

I've been playing with Catalyst's tinderbox myself recently, running
multiple instances on a 128-node x86 cluster (1/node). There are one or
two things I'd like to change in Catalyst, as they currently make it
slightly dangerous for your system. If for some reason an ebuild goes
wrong, the bind mounts allow it access into your core system, namely
/dev and $DISTDIR and it really hurts to have a bunch of /dev erased on
a box that you don't have quick physical access to (I toasted a few
cluster nodes due to a bug in some scripts this way, but luckily I could
force a remote reinstall).

> >Queues
> >------
[snip]
I'm fully up with this idea.

> >CVS Producer
> >------------
How about using the gentoo-cvs commits mailing list?

> >Consumer (testing process)
> >--------------------------
Catalyst's tinderbox goes here. Combining it with a package cache, you
can easily test just what you need.

-- 
Robin Hugh Johnson
E-Mail     : robbat2@orbis-terrarum.net
Home Page  : http://www.orbis-terrarum.net/?l=people.robbat2
ICQ#       : 30269588 or 41961639
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 241 bytes --]

Re: [gentoo-dev] Re: [gentoo-osx] Package testing -- Automated initiative

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 867 bytes --]

On Sun, 2005-08-14 at 14:22 -0700, Robin H. Johnson wrote:
> multiple instances on a 128-node x86 cluster (1/node). There are one or
> two things I'd like to change in Catalyst, as they currently make it
> slightly dangerous for your system. If for some reason an ebuild goes
> wrong, the bind mounts allow it access into your core system, namely
> /dev and $DISTDIR and it really hurts to have a bunch of /dev erased on
> a box that you don't have quick physical access to (I toasted a few
> cluster nodes due to a bug in some scripts this way, but luckily I could
> force a remote reinstall).

I accept patches.  *grin*

By the way, I am working to get catalyst running on OSX, so version 2.0
will definite suit your needs when it is released.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Re: [gentoo-osx] Package testing -- Automated initiative

[Grobian]

Chris Gianelloni wrote:
> By the way, I am working to get catalyst running on OSX, so version 2.0
> will definite suit your needs when it is released.

If you need help on OSX specific things, be sure to contact us...


-- 
Fabian Groffen
eBuild && Porting
Gentoo for Mac OS X
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: [gentoo-osx] Package testing -- Automated initiative

[Kito]

On Aug 15, 2005, at 10:39 AM, Grobian wrote:

>
>
> Chris Gianelloni wrote:
>
>> By the way, I am working to get catalyst running on OSX, so  
>> version 2.0
>> will definite suit your needs when it is released.

Very cool. I had 1.x nearly working a while back...haven't looked at  
2.0 yet.

>>
>
> If you need help on OSX specific things, be sure to contact us...
>

Indeed.

>
> -- 
> Fabian Groffen
> eBuild && Porting
> Gentoo for Mac OS X
> -- 
> gentoo-dev@gentoo.org mailing list
>
>

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: [gentoo-osx] Package testing -- Automated initiative

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 888 bytes --]

On Mon, 2005-08-15 at 11:16 -0500, Kito wrote:
> On Aug 15, 2005, at 10:39 AM, Grobian wrote:
> 
> >
> >
> > Chris Gianelloni wrote:
> >
> >> By the way, I am working to get catalyst running on OSX, so  
> >> version 2.0
> >> will definite suit your needs when it is released.
> 
> Very cool. I had 1.x nearly working a while back...haven't looked at  
> 2.0 yet.

I managed to get it to work once I made some dirty hacks to account for
uname being different, and removing dependencies on /proc... once inside
the chroot, it's Linux anyway, so none of the BSDism's are an issue.

Note: I am talking about being able to *run* catalyst on OSX, not build
OSX targets with catalyst.  I'm sure that support would be something we
added later as the project matured.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Re: [gentoo-osx] Package testing -- Automated initiative

[Kito]

On Aug 15, 2005, at 12:07 PM, Chris Gianelloni wrote:

> I managed to get it to work once I made some dirty hacks to account  
> for
> uname being different, and removing dependencies on /proc... once  
> inside
> the chroot, it's Linux anyway, so none of the BSDism's are an issue.

Ok, let me know if you want/need an account on our little dev machine  
for testing anything.

>
> Note: I am talking about being able to *run* catalyst on OSX, not  
> build
> OSX targets with catalyst.  I'm sure that support would be  
> something we
> added later as the project matured.

Yeah of course, I wasn't expecting you to work miracles ;)

For the package testing stuff, I should have a stage1 tarball done in  
the coming weeks(months?) that could be used to do proper chroot'd  
builds for Darwin/OS X.

>
> -- 
> Chris Gianelloni
> Release Engineering - Strategic Lead/QA Manager
> Games - Developer
> Gentoo Linux
>

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: [gentoo-osx] Package testing -- Automated initiative

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 1107 bytes --]

On Mon, 2005-08-15 at 12:23 -0500, Kito wrote:
> On Aug 15, 2005, at 12:07 PM, Chris Gianelloni wrote:
> 
> > I managed to get it to work once I made some dirty hacks to account  
> > for
> > uname being different, and removing dependencies on /proc... once  
> > inside
> > the chroot, it's Linux anyway, so none of the BSDism's are an issue.
> 
> Ok, let me know if you want/need an account on our little dev machine  
> for testing anything.
> 
> >
> > Note: I am talking about being able to *run* catalyst on OSX, not  
> > build
> > OSX targets with catalyst.  I'm sure that support would be  
> > something we
> > added later as the project matured.
> 
> Yeah of course, I wasn't expecting you to work miracles ;)
> 
> For the package testing stuff, I should have a stage1 tarball done in  
> the coming weeks(months?) that could be used to do proper chroot'd  
> builds for Darwin/OS X.

Tinderbox requires a stage3 tarball...

Just thought I would let you know...

-- 
Chris Gianelloni
Release Engineering - Strategic Lead/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]