[gentoo-dev] [GSoC status] Web-based image builder

[gentoo-dev] [GSoC status] Web-based image builder

[Eitan Mosenkis]

This is my first weekly status report, so I'll introduce my project a
little.  I'm building a web-based application in PHP (with MySQL) that
will allow the user to build a customized image of a linux
installation.  The intent is to support various different
architectures/profiles, using the binary packages currently found on
tinderbox.dev.gentoo.org.  Users should be able to produce tarballs
or, I hope, ISO images, etc.

Also, I'd like to note before I get down to the status part, that if
anyone would like to volunteer to do some web or graphic design for
the project, whoever ends up using it (or their eyes) will probably
thank you.  I'm a much better web programmer than designer.

As of now, I've made local copies of three different package sets to
work with, default-linux/amd64, hardened/x86, and embedded/beagle.
I'm working with solar to get some more packages added to tinderbox so
I can test things more completely, but as of now my backend can
generate a complete target environment for any of the above profiles
and attempt to emerge system (the only one where enough binpkgs are
available to actually complete this is the amd64).  The backend polls
the database for requested builds and builds them as requested,
sending detailed logs back to the database.

The frontend is coming along nicely, currently featuring user login,
build request (as of now just choosing a name and selecting the
profile), and log viewer.

All comments, questions, suggestions, etc. are welcome.

Re: [gentoo-dev] [GSoC status] Web-based image builder

[Krzysiek Pawlik]

[-- Attachment #1: Type: text/plain, Size: 346 bytes --]

Eitan Mosenkis wrote:
> All comments, questions, suggestions, etc. are welcome.

Interesting - could you point me/us at some design document or provide more
detailed description of how the whole process works?

-- 
Krzysiek Pawlik   <nelchael at gentoo.org>   key id: 0xBC555551
desktop-misc, java, apache, ppc, vim, kernel, python...


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 261 bytes --]

Re: [gentoo-dev] [GSoC status] Web-based image builder

[Eitan Mosenkis]

[-- Attachment #1: Type: text/plain, Size: 592 bytes --]

Yes, I posted a sort of mock-up at http://eitan.mosenkis.net/soc/, including
a description/features list, though I can't promise that it will remain
accurate.

On Fri, Jun 19, 2009 at 6:15 PM, Krzysiek Pawlik <nelchael@gentoo.org>wrote:

> Eitan Mosenkis wrote:
> > All comments, questions, suggestions, etc. are welcome.
>
> Interesting - could you point me/us at some design document or provide more
> detailed description of how the whole process works?
>
> --
> Krzysiek Pawlik   <nelchael at gentoo.org>   key id: 0xBC555551
> desktop-misc, java, apache, ppc, vim, kernel, python...
>
>

[-- Attachment #2: Type: text/html, Size: 1048 bytes --]

[gentoo-dev] Re: [GSoC status] Web-based image builder

[Eitan Mosenkis]

This week has been much slower with the image builder project.  I've
wrinkled out several issues and generally worked on usability and
other frontend aspects.  The backend (and frontend) currently handle
building a complete tar/gz image and letting the user download it with
the user's choice of profile (as in /etc/make.profile), based on
available binpkgs repositories, the base system (emerge system), and
an arbitrary set of packages (selected from a list derived from the
Packages file of the binpkgs repo).  The frontend now also supports
new user registration.  The front end offers fairly complete logs of
the build process, including converting ANSI colors to HTML
equivalents so logs can be colored too.

Things to do next week (and beyond):
-Invitation-only user registration system
-Management of disk space, meaning a good system that can be called by
cron to delete old images, logs, etc. that can be assumed to be no
longer needed.
-Work towards finalizing the database structure
-Addition of new options/features in frontend and implementation of
said options in backend
-Making the frontend more user-friendly

I'd also like to ask for input from anyone interested, especially
anyone who thinks they will use this system once it's public, so I
created a forum topic [1] for suggestions, requests, etc.

Anyone who would be interested in testing out this system if/when a
testing server becomes available, please also email me off-list.

Eitan

[1] http://forums.gentoo.org/viewtopic-p-5830406.html

[gentoo-dev] Re: [GSoC status] Web-based image builder

[Eitan Mosenkis]

This week has gone pretty well and I'm hoping to get a testing server
up soon, though it looks like it will be running the backend in a
virtual machine due to the root privileges requirement.  I've added
support for several more output formats: ext2 image, jffs2, and two
different styles of bootable ISO - one that's just a standard weekly
Gentoo minimal CD with a tarball of the system, and one that replaces
the minimal CD's environment with the generated system (the CDs are at
a fairly primitive stage, but I did succeed in booting both of them
with qemu).  I also changed around the frontend so that rather than
configuring each build individually, configuration is done on reusable
configurations, which can then be used any number of times to request
actual builds.  I also added an invitations system to allow a more
closed environment, especially for testing before it's ready for
release.
Both the frontend and backend are now set up to use modules (presently
I'm only worrying about modules for Gentoo/portage) so that in the
future, support can be added for other distributions or package
managers.  I also did some work to provide more high-level functions
for these modules so they will be easy to write and modify.  The
portage backend now caches the generated image after the base system
has been installed, providing a huge reduction in the time it takes to
generate an image for large profiles once the cache exists.
Lastly, the backend can now upload completed images to the frontend
for a case when they are on separate machines and it is not desirable
to have users download their finished images directly from the backend
machines on which they were compiled.

To do:
-Make the frontend more user-friendly (particularly the selection of
packages to install)
-Allow editing of configurations after they are initially completed
-Wrinkle out the rest of the issues with separating frontend and backend
-Make sure no race conditions exist when using more than one backend
to serve one frontend
-Figure out a (safe) system for updating the package repositories,
particularly with backend and frontend split and with multiple
backends
-Perhaps more important of all, provide more options to configure the
image produced (see http://etherpad.com/BJkgXcMkwJ for current list of
options, http://forums.gentoo.org/viewtopic-p-5830406.html if you have
suggestions)

Thanks to those who have been giving support and offering suggestions so far.

Eudyptula

[gentoo-dev] Re: [GSoC status] Web-based image builder

[Eitan Mosenkis]

According to my original time line, which I've largely ignored since I
started coding, I'm about a week ahead of schedule, though I've also
left out things that I'd planned on finishing by now and implemented
things .  This is how my week went:
Sunday: Copied the client-side searching of available packages from
the mockup I did for my proposal
Monday: Made modifications to allow the frontend to function when the
package repositories that backend is using are not available to it;
wrote script for taking into account updates to package repositories;
made final changes to allow multiple backends to serve one frontend
Tuesday: Added a logo and favicon; began major restructuring of
configuration wizard to be object-oriented and provide a simple API
for defining configuration items/steps
Wednesday: Did most of the conversion of the searchable package
selection interface into a very configurable class available through
the frontend API
Thursday: Finished the package selection class, began separating the
code that bundles the final image from the code that builds it
Friday: Finished bundler separation in backend, added metadata for
better handling of frontend modules, changed system configuration file
to have default values and be easier to work with

To be done:
-Get Ingenue ready for deployment on a real server - check that
authentication is present wherever necessary, add administrative
functions and the capability to automatically clean up stale data
-Add more build options, as listed on the features list [0]
-Do some testing, preferably with real users on a testing server (get
a testing server up and running)

More distant:
-Spice up the frontend and add some documentation to it
-Create a live development ebuild for the project
-Have the project generate images containing itself to run as backend slaves

As always, feedback welcome.

[0] http://etherpad.com/BJkgXcMkwJ

[gentoo-dev] Re: [GSoC status] Web-based image builder

[Eitan Mosenkis]

This week was a little less productive than usual.  I got the front
end to be much more usable, allowing configurations to be modified by
the user at will and making it possible to view the settings of a
configuration.  I also added a validity check on configurations before
the build stage, making sure the backend won't waste time trying to
build an image with packages that are no longer available, etc.  I
broke the backend into smaller parts just to keep things better
organized and fixed various bugs that had made their way into the
backend while I've been focusing on the frontend.  I also added the
option to set a timezone, choose udev or static-dev, and to choose
from preconfigured sets of packages, such as "HTTP Servers" and
"Desktop Environments".  Lastly, I updated the welcome page to list
popular builds, recent completed builds, and recently downloaded
builds.

To do:
-Continue to expand the options available for configuring the build
-Extend the API used by frontend modules to also work for options
universal to all modules
-Add privacy options so users can choose whether to share the builds
and configurations they have created

In preparation for offering adding/removing init scripts, I'd
appreciate if people could execute the following and email it to me,
along with what arch they're running:
equery belongs /etc/init.d/*

Thanks.

[gentoo-dev] Re: [GSoC status] Web-based image builder

[Eitan Mosenkis]

I forgot to post a status report since last Friday was pretty hectic.
In short form, last week I:
-Finished the logging mechanisms so that the logs will track things
besides just command execution.
-Wrote a command-line wrapper in C to an ISO manipulation library to
make things far more efficient and avoid any number of difficulties
with generating bootable ISOs from the Gentoo autobuilds
-Stopped generating the base system with emerge system (which was very
slow and not really complete) and moved to using stage3 tarballs
instead
-Cache a list of packages and init scripts present in the stage3s in
the database
-Quite a bit of clean-up of the backend to make the logs less messy
and the automatic notification emails actually usable
In the last two days, I:
-Cleaned up some pointless links in the frontend
-Rehashed the configuration wizard so I can reuse the HTML form
classes in other contexts
-Tweaked the package selection mechanism for efficiency
-Added the option to remove init scripts enabled by default and
packages present in the stage3

Ideally I'd like to make the system available for testing ASAP, but
since the system needs root access to run, getting a server to run it
on is a little tricky.

I'd like to ask for some input here.  I intend to make an option
available whereby instead of using an entire stage3 system, the user
can choose to have all packages not absolutely necessary for a working
system uninstalled.  Please respond off-list with the following: the
make.profile you use (or arch) and a list of packages absolutely
necessary for the base system.

[gentoo-dev] Re: [gentoo-soc] Re: [GSoC status] Web-based image builder

[Sérgio Almeida]

[-- Attachment #1: Type: text/plain, Size: 554 bytes --]

Hello,

On Tue, 2009-07-28 at 18:11 -0400, Eitan Mosenkis wrote:
> Ideally I'd like to make the system available for testing ASAP, but
> since the system needs root access to run, getting a server to run it
> on is a little tricky.
> 

I can help with a server with root access. I can make a gentoo vmware
machine on a quad-core with 1GB of dedicated RAM just for you and your
testings. The machine has 100/100MB connection to internet. What else do
you need?

Cheers,
Sérgio
-- 
Sérgio Almeida - mephx.x@gmail.com
mephx @ freenode


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[gentoo-dev] Re: [GSoC status] Web-based image builder

[Eitan Mosenkis]

This week is my last for SoC this summer, so unless I do something
groundbreaking tomorrow, this will be my last status report.  I was
hoping to be able to show off a working server with Ingenue running on
it by this time, but I'm still waiting for all the difficulties with
getting a root-access server to get ironed out.  The project is in
pretty good shape, though.  I have one important change to make to the
frontend tomorrow (just moving some options from the middle of the
config process where they don't belong to the end) and some minuscule
tweaks to the database structure to allow for better extendibility in
the future.  Some usage or hacking instructions are also on the to-do
list.  For anyone interested in testing it out on their own computer
or looking at the source code, I've finally created an ebuild [0].
I'm putting some basic usage instructions at the end of this message.
My biggest accomplishment of late was adapting the backend to work in
stages such that if it is stopped at any time, it will automatically
start where it left off next time it is started (except for places
where some command fails if it has already been run - fixing that is
on the to-do list).  This is incredibly helpful when troubleshooting
things later in the build process since I can skip up to 10 minutes of
building prior to the point where problems happen.  The status page is
also a little more detailed now as it shows approximately how much of
the build process has been completed.
I also implemented a handful of new options so that most of the parts
of the Gentoo install guide that seem worthwhile (and possible) to me
have been implemented [1].  I also created an update script to help
keeping stage3 tarballs and other external files up to date (currently
only FTP is supported since it makes it easy to get a directory
listing along with file size and mtime).  I fixed quite a large number
of bugs that crept into the backend while I was working on other parts
of the project.  One of the most important things I spent my time on
this week was quality of code.  I moved a lot of code from place to
place to minimize repeating the same code and to make the flow more
clear.  With a few exceptions, I think the code is in great shape for
continued development.  Hopefully I'll be able to deploy it soon and
people will be able to make good use of it.  Towards that end, I had
great success in building an image which, with around a dozen
commands, I was able to install on a qemu machine and run Ingenue on
to replicate itself again.
Lastly, I'd like to thank my mentor killerx, along with all the others
who have helped me with this project - solar, zmedico, robbat2,
agaffney, and others.  I've really enjoyed working with the Gentoo
community (though I had to break the brief habit of reading everything
that went by on #gentoo since it caused something of a skydive in my
productivity).  I'm looking forward to continuing to improve Ingenue
and otherwise contribute to Gentoo once I get adjusted to college life
a little bit (my next non-Ingenue project is getting
pam-face-authentication [2] into portage - the ebuild is pretty
outdated).

[0] http://git.overlays.gentoo.org/gitweb/?p=proj/ingenue.git;a=blob_plain;f=ingenue-9999.ebuild;hb=HEAD
[1] http://git.overlays.gentoo.org/gitweb/?p=proj/ingenue.git;a=blob_plain;f=gentoo-steps;hb=HEAD
[2] http://code.google.com/p/pam-face-authentication/

1) Edit shared/config.php (the ebuild installs in /usr/share/ingenue -
probably not the right place, but I'll fix that when I figure out what
the right place is) to your liking.  It's fairly well commented
already and most things can be left commented to use intelligent
defaults.
2) Make sure that you have a Gentoo packages directory (with Packages
file) at the location specified in the config file (you can offer more
than one profile by putting the repos (or symlinking them) as
subdirectories of the directory in the config - it searches
recursively for Packages files)
3) Configure cache/conf to download stage3 images for the appropriate
archs (follow the example of the entries already there) - also install
CDs if you want them and a portage snapshot unless you want to spend a
long time copying your local portage tree file-by-file
4) Run setup.php and update_cache.php
5) Configure your web server to make the frontend directory available
and navigate to it (currently, it is necessary to use Apache with
mod_rewrite and htaccess files enabled or in some other way redirect
all traffic to index.php)
6) Run update_cache.php to update to newer stage3's, CDs, portage
snapshots (it checks filename/modtime to avoid downloading the same
versions repeatedly)
7) Whenever your package repos are updated, run update_gentoo_profiles
to update the database with the changes
8) Report bugs! (I hope there aren't too many, but it's been somewhat
difficult to test comprehensively)