From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([69.77.167.62] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1K8brb-0006WH-9F for garchives@archives.gentoo.org; Tue, 17 Jun 2008 14:08:47 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 8BEEFE0358; Tue, 17 Jun 2008 14:08:46 +0000 (UTC) Received: from kyron.neuralbs.com (modemcable086.140-70-69.static.videotron.ca [69.70.140.86]) by pigeon.gentoo.org (Postfix) with ESMTP id 68C19E03E2 for ; Tue, 17 Jun 2008 14:08:46 +0000 (UTC) Received: from [192.168.1.2] (kyron.neuralbs.com [192.168.1.2]) by kyron.neuralbs.com (Postfix) with ESMTP id 37F1F561887 for ; Tue, 17 Jun 2008 10:08:46 -0400 (EDT) Message-ID: <4857C56E.5080502@neuralbs.com> Date: Tue, 17 Jun 2008 10:08:46 -0400 From: Eric Thibodeau Organization: Neural Bucket Solutions Inc. User-Agent: Thunderbird 2.0.0.14 (X11/20080525) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-soc@lists.gentoo.org Reply-to: gentoo-soc@lists.gentoo.org MIME-Version: 1.0 To: gentoo-soc@lists.gentoo.org Subject: Re: [gentoo-soc] Progress report - AutotuA (formerly "Automate it All") References: <8b4c83ad0806152358ydc58abwe57a9e87bb342bc3@mail.gmail.com> In-Reply-To: <8b4c83ad0806152358ydc58abwe57a9e87bb342bc3@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Archives-Salt: 27812135-4a77-4b6c-b799-8dbcc9008748 X-Archives-Hash: e2df1065182c06b44527b926140578c0 Dude...sounds to me like you need a PBS.... Nirbheek Chauhan wrote: > It's only been a little more than a week since I started working on > the project (due to personal reasons), and my time spent to work ratio > is extremely bad, so I'm sorry but the progress isn't as much as I had > hoped. > > The idea has undergone significant changes in the time passed, and > thanks to Patrick's guidance (and constant cluebats), I now have a far > more clear-cut idea of how the whole thing will come together. I > wonder whether I should describe the project blueprint that we've come > up with, the path that led to it, or what all code I have written. I > suppose the progress of the code written cannot be judged unless one > knows the whole plan, and the path taken to come up with the plan is > largely irrelevant :) > > The general idea has changed somewhat from the abstract: > > As before, there is a master server which acts a storage area, manages > all the slaves and does various bookkeeping. This part will be written > in Django. > > The concept of the slave has changed radically to allow for a less > steep learning curve. The project described "jobs" which consist of > executables stored on the master-server which could be fetched and run > by the slaves. We thought of ways in which we could describe > dependencies between the slaves, and the most obvious answer to me was > an XML format (much to the disgust of Patrick). > > However, there were numerous problems with such an approach (least of > which was the overhead involved with parsing XML and the jing deps on > the Django side for it). The most serious of these was the fact that > learning a new XML format and writing custom executables (scripts or > otherwise) which communicate with the server via the Slave's bindings > has an *extremely* steep learning curve, and will cause chaos. The > project is useless if no one ends up using it, or it gets too > complicated to use. > > The solution came to me in the form of a "Doh!" moment as I was > cycling back to my room. The answer was -- "jobuilds". Bash scripts > are easily adaptable, easy to understand and use (for Gentoo devs), > and their parsing is well-understood. For the second time in my life, > I appreciated the ingenuity of the inventors of the ebuild format. > > > Jobuilds: > ---------- > A jobuild is the smallest possible "quantum" of work. A job consists > of a root jobuild which has dependencies on other jobuilds, and all > these taken together form a job. The format of a jobuild is: > http://pastebin.osuosl.org/8355 > > - The four phases are to be run (by default) in the chroot where the > job will take place. > - SRC_URI are programs: test suites etc which are required by the > jobuild (does not include the deps which will be pulled in by emerge > in the chroot). > - PORTCONF_URI are tarballs which will contain portage config files > (/etc/portage/* /etc/make.conf etc) > - DEPEND are other jobuilds on which this jobuild _hard_ depends, ie > they must be completed in the same chroot (example: Test Amarok > depends on Build KDE which depends on Build X) > - SIDEPEND are SuperImpose Depends, all we need to know is that those > jobuilds completed successfully *somewhere* so that further > distribution of work is possible. (example: testing if all the > packages that import gnome2.eclass still work after some changes to > it) > > SRC_URI will be downloaded before entering the chroot, stored in a > tarballs folder, and hardlinked (if on the same device), or bind > mounted inside the chroot. > > To counter the problem of recursive QA checking, the jobuild format > will be *extremely* simple. That means no EAPI, no eclasses, no SLOTS, > minimal versioning (xxx.yyy), no fancy depends (except perhaps ||). > Built-in functions such as unpack() etc will be provided of course. > > The loss of utility from there not being eclasses will be offset > through the concept of "Template jobuilds" (similar in concept to how > Django handles Templates[1]). However, I am open to including eclasses > in the design (who doesn't love them? :) if enough reasons can be > given. > > NOTE: It will be highly recommended that the autotua work folder be on > the same device. I've assumed this to be true to allow a number of > optimisations, but I will keep (slower) fallbacks in case that is not > true. > > > The Tree: > ----------- > Obviously the jobuilds will be stored in a structured format similar > to the portage tree :^) > And following the tradition of being completely unimaginative, it > shall be called the "Jobtage tree". > The structure is as follows: > > ${user}/ > ${user}.asc > ${jobuild_name}/ > ${jobuild_name}-${ver}.jobuild > Manifest > > The tree will be stored in bzr, with an overlays/ directory in .bzrignore > jobuilds will not be manifested, and will only be signed with the > maintainer's gpg key > SRC_URI and PORTCONF_URI will be Manifested (probably the same way in portage) > > To further offset the problem of QA in this tree (mentioned in > "Jobuild" above), when Jobs are created/committed/uploaded on the > server (the details of that are in the next section), the whole > depgraph is validated, details about that stored as metadata, and the > Job itself is attached to **that specific revision** of the Jobtage > tree. This prevents breakages due to future changes made to the > jobuilds it depends on. If the maintainer wishes to update the > attached revision (for say a bugfix in a depending jobuild), he can > force a re-validation at anytime before a Job is accepted by a Slave. > Whenever a Slave accepts a Job, it syncs with the revision of the tree > it's attached to. > > The other solution to this problem could've been to trigger a reverse > depgraph validation whenever a commit was made to the tree. The > problems with that approach are: > - Load on the server increases exponentially with jobuilds > - Raises the question of what the next action should be -- revert the > (potentially critical) commit or mark (potentially hundreds of) > jobuilds as broken? > - Makes Jobs fragile -- a job might be fine when you upload it, but > horribly broken 4 hours later. > > > Slaves: > -------- > The slave can pull a list of Jobs that it can do from the master > server. A Job will consist of metadata about it: > http://pastebin.osuosl.org/8358 . The actual data is then gathered > from the jobuild(s), the chroot is prepared, etc etc and work begins. > Slave reports back to the master server after every jobuild is > complete with data and receives updates (if any) about the Job > (updates might consist of changing depends due to SIDEPENDs). > > Obviously the Slave has to parse jobuilds. And so the concepts should > be similar to Portage. However, I am drawing inspiration from the > pkgcore[2] codebase, simplifying the extremely versatile code to suit > my needs (which is another reason for my slow progress -- it's not > easy to understand a work of art ;) > > > Actual Progress aka "No more hand-waving": > ------------------------------------------------------ > Now follows my *real* progress w.r.t the code. > > I'm currently working on the slave, and am concentrating on the things > that don't depend on the part of parsing the jobuilds (have a general > idea how it's done, haven't fleshed out the details). Currently I've > implemented an OO interface (in Python of course) to a Job() object > accessed via Jobs(), a Syncer() object (jobtage), a Fetchable object > and a Fetcher (stage3 etc). Total code comes out to 167+70+38+30 = > ~300 lines ;p > > This week I'll start on chroot preparation and iron out the kinks in > that, followed by the Jobuild() object, the jobuild parser > (jobuild.sh), and the bridge connecting them. The #pkgcore guys are > really helpful and nice so I'll have good help for this part :) > > Next week (end of the month) will (hopefully) see a working slave > which accepts Jobs from some magical source and runs them. > > I'll begin work on the Master server the week after that, specifically > the backend work and the details of the communication between the > Master and Slaves. Frontend prettyfication will take place towards the > end. > > > 1. http://www.djangobook.com/en/1.0/chapter04/ -- not the exact > format, only the idea of "Reverse Inheritance" > 2. http://www.pkgcore.org/ > > PS: Another reason why progress is slow is because the Slave portion > has become much more sophisticated than what I had originally > intended. The original idea had (maintainer-made) executables doing > all the work (causing a steep learning curve) with the Slave just > being an API wrapper to talk to the master server. All of that work is > now shifted into the Slave and abstracted for the maintainer to use in > a familiar way. > > -- gentoo-soc@lists.gentoo.org mailing list