From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([69.77.167.62] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1K88fq-0005Pl-UA for garchives@archives.gentoo.org; Mon, 16 Jun 2008 06:58:43 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 4C5F2E04AE; Mon, 16 Jun 2008 06:58:42 +0000 (UTC) Received: from ti-out-0910.google.com (ti-out-0910.google.com [209.85.142.190]) by pigeon.gentoo.org (Postfix) with ESMTP id 7CDE5E04AE for ; Mon, 16 Jun 2008 06:58:41 +0000 (UTC) Received: by ti-out-0910.google.com with SMTP id u5so1586712tia.10 for ; Sun, 15 Jun 2008 23:58:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type:content-transfer-encoding :content-disposition; bh=56euZVPeZRtgyDC3GuAkqgnEE3bxBmJU3XrXhdHK0Gc=; b=R5c1SXn42aYuPwqIEeY9Mdw/y11iKAj9Zb70GmoFzViG2LxInFkYnwhQfBICZSSFmh fgTiq2OsUTZRbv5/gqckdCH9x7iFfLVSGGQKbbWBl5zcjTYhemoI6RDgjaWGRnJaBZyX 1xwTkDe8xa8SzMmTTzsINMjn7hSEX5kHqSz9I= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type :content-transfer-encoding:content-disposition; b=hwebaZGHSZVjd3qszB312WgsQj4uN9aSUAXuIbu5tZe6HemK4UU1Z6pLGHEmIDu+/k QSq7qt1SC5DbDZpUu2fWg+ZQCikMejBLh3PEFWPOID0TF1KoYW1FNljrX5kcPnHsUgdp BlYUJOd5NAIx4i1LkvoI4/UbkAqaricu/Ht98= Received: by 10.110.68.4 with SMTP id q4mr4102670tia.41.1213599517776; Sun, 15 Jun 2008 23:58:37 -0700 (PDT) Received: by 10.110.17.2 with HTTP; Sun, 15 Jun 2008 23:58:37 -0700 (PDT) Message-ID: <8b4c83ad0806152358ydc58abwe57a9e87bb342bc3@mail.gmail.com> Date: Mon, 16 Jun 2008 12:28:37 +0530 From: "Nirbheek Chauhan" To: gentoo-soc@lists.gentoo.org Subject: [gentoo-soc] Progress report - AutotuA (formerly "Automate it All") Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-soc@lists.gentoo.org Reply-to: gentoo-soc@lists.gentoo.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Archives-Salt: 2cad276b-c654-4764-9b03-0cf5ee7954c8 X-Archives-Hash: 7b8cf5c6033d78f8afb3d5919b81560b It's only been a little more than a week since I started working on the project (due to personal reasons), and my time spent to work ratio is extremely bad, so I'm sorry but the progress isn't as much as I had hoped. The idea has undergone significant changes in the time passed, and thanks to Patrick's guidance (and constant cluebats), I now have a far more clear-cut idea of how the whole thing will come together. I wonder whether I should describe the project blueprint that we've come up with, the path that led to it, or what all code I have written. I suppose the progress of the code written cannot be judged unless one knows the whole plan, and the path taken to come up with the plan is largely irrelevant :) The general idea has changed somewhat from the abstract: As before, there is a master server which acts a storage area, manages all the slaves and does various bookkeeping. This part will be written in Django. The concept of the slave has changed radically to allow for a less steep learning curve. The project described "jobs" which consist of executables stored on the master-server which could be fetched and run by the slaves. We thought of ways in which we could describe dependencies between the slaves, and the most obvious answer to me was an XML format (much to the disgust of Patrick). However, there were numerous problems with such an approach (least of which was the overhead involved with parsing XML and the jing deps on the Django side for it). The most serious of these was the fact that learning a new XML format and writing custom executables (scripts or otherwise) which communicate with the server via the Slave's bindings has an *extremely* steep learning curve, and will cause chaos. The project is useless if no one ends up using it, or it gets too complicated to use. The solution came to me in the form of a "Doh!" moment as I was cycling back to my room. The answer was -- "jobuilds". Bash scripts are easily adaptable, easy to understand and use (for Gentoo devs), and their parsing is well-understood. For the second time in my life, I appreciated the ingenuity of the inventors of the ebuild format. Jobuilds: ---------- A jobuild is the smallest possible "quantum" of work. A job consists of a root jobuild which has dependencies on other jobuilds, and all these taken together form a job. The format of a jobuild is: http://pastebin.osuosl.org/8355 - The four phases are to be run (by default) in the chroot where the job will take place. - SRC_URI are programs: test suites etc which are required by the jobuild (does not include the deps which will be pulled in by emerge in the chroot). - PORTCONF_URI are tarballs which will contain portage config files (/etc/portage/* /etc/make.conf etc) - DEPEND are other jobuilds on which this jobuild _hard_ depends, ie they must be completed in the same chroot (example: Test Amarok depends on Build KDE which depends on Build X) - SIDEPEND are SuperImpose Depends, all we need to know is that those jobuilds completed successfully *somewhere* so that further distribution of work is possible. (example: testing if all the packages that import gnome2.eclass still work after some changes to it) SRC_URI will be downloaded before entering the chroot, stored in a tarballs folder, and hardlinked (if on the same device), or bind mounted inside the chroot. To counter the problem of recursive QA checking, the jobuild format will be *extremely* simple. That means no EAPI, no eclasses, no SLOTS, minimal versioning (xxx.yyy), no fancy depends (except perhaps ||). Built-in functions such as unpack() etc will be provided of course. The loss of utility from there not being eclasses will be offset through the concept of "Template jobuilds" (similar in concept to how Django handles Templates[1]). However, I am open to including eclasses in the design (who doesn't love them? :) if enough reasons can be given. NOTE: It will be highly recommended that the autotua work folder be on the same device. I've assumed this to be true to allow a number of optimisations, but I will keep (slower) fallbacks in case that is not true. The Tree: ----------- Obviously the jobuilds will be stored in a structured format similar to the portage tree :^) And following the tradition of being completely unimaginative, it shall be called the "Jobtage tree". The structure is as follows: ${user}/ ${user}.asc ${jobuild_name}/ ${jobuild_name}-${ver}.jobuild Manifest The tree will be stored in bzr, with an overlays/ directory in .bzrignore jobuilds will not be manifested, and will only be signed with the maintainer's gpg key SRC_URI and PORTCONF_URI will be Manifested (probably the same way in portage) To further offset the problem of QA in this tree (mentioned in "Jobuild" above), when Jobs are created/committed/uploaded on the server (the details of that are in the next section), the whole depgraph is validated, details about that stored as metadata, and the Job itself is attached to **that specific revision** of the Jobtage tree. This prevents breakages due to future changes made to the jobuilds it depends on. If the maintainer wishes to update the attached revision (for say a bugfix in a depending jobuild), he can force a re-validation at anytime before a Job is accepted by a Slave. Whenever a Slave accepts a Job, it syncs with the revision of the tree it's attached to. The other solution to this problem could've been to trigger a reverse depgraph validation whenever a commit was made to the tree. The problems with that approach are: - Load on the server increases exponentially with jobuilds - Raises the question of what the next action should be -- revert the (potentially critical) commit or mark (potentially hundreds of) jobuilds as broken? - Makes Jobs fragile -- a job might be fine when you upload it, but horribly broken 4 hours later. Slaves: -------- The slave can pull a list of Jobs that it can do from the master server. A Job will consist of metadata about it: http://pastebin.osuosl.org/8358 . The actual data is then gathered from the jobuild(s), the chroot is prepared, etc etc and work begins. Slave reports back to the master server after every jobuild is complete with data and receives updates (if any) about the Job (updates might consist of changing depends due to SIDEPENDs). Obviously the Slave has to parse jobuilds. And so the concepts should be similar to Portage. However, I am drawing inspiration from the pkgcore[2] codebase, simplifying the extremely versatile code to suit my needs (which is another reason for my slow progress -- it's not easy to understand a work of art ;) Actual Progress aka "No more hand-waving": ------------------------------------------------------ Now follows my *real* progress w.r.t the code. I'm currently working on the slave, and am concentrating on the things that don't depend on the part of parsing the jobuilds (have a general idea how it's done, haven't fleshed out the details). Currently I've implemented an OO interface (in Python of course) to a Job() object accessed via Jobs(), a Syncer() object (jobtage), a Fetchable object and a Fetcher (stage3 etc). Total code comes out to 167+70+38+30 = ~300 lines ;p This week I'll start on chroot preparation and iron out the kinks in that, followed by the Jobuild() object, the jobuild parser (jobuild.sh), and the bridge connecting them. The #pkgcore guys are really helpful and nice so I'll have good help for this part :) Next week (end of the month) will (hopefully) see a working slave which accepts Jobs from some magical source and runs them. I'll begin work on the Master server the week after that, specifically the backend work and the details of the communication between the Master and Slaves. Frontend prettyfication will take place towards the end. 1. http://www.djangobook.com/en/1.0/chapter04/ -- not the exact format, only the idea of "Reverse Inheritance" 2. http://www.pkgcore.org/ PS: Another reason why progress is slow is because the Slave portion has become much more sophisticated than what I had originally intended. The original idea had (maintainer-made) executables doing all the work (causing a steep learning curve) with the Slave just being an API wrapper to talk to the master server. All of that work is now shifted into the Slave and abstracted for the maintainer to use in a familiar way. -- ~Nirbheek Chauhan -- gentoo-soc@lists.gentoo.org mailing list