[gentoo-dev] The new Portage Job System

[Dan Armak <danarmak@gentoo.org>]

Hi all,

In accordance with drobbins' wishes, the discussion of the new Portage job 
system moves to gentoo-dev. This is a quick summary of the discussion on 
#gentoo yesterday.

The idea as of now is this: (my version)


Intro:
--------
Under profiles/jobs, or in a similar location, will be a job dir (Possibly 
profiles/default/jobs, and the references may then be redirected through 
/etc/make.profile). Developers can place jobs in it for Portage to execute.

Jobs can request to be run once, or many times based on a condition. Portage 
will keep track of jobs that have been run in e.g. /var/db/jobs.


Helper files:
----------------
Jobs can have misc files, a separate dir should be given to these i.e. 
profiles/jobs/files.


Generic scripts:
--------------------
Provide a dir profiles/jobs/scripts and populate it with *generic* scripts. 
Under profiles/jobs put job spec files which, in their first few (commented 
out) lines, have info about when to run them, and then have a #! line that 
specifies one of these generic scripts as the interpreter. For example, a job 
spec for moving a package would be called move-koffice.job, and begin:

# Job spec description: move the koffice package from kde-apps to app-office
# Run Once
# Author: Dan Armak <danarmak@gentoo.org>
# $Header: $
#! /usr/portage/profiles/jobs/scripts/movepackage.sh
move koffice kde-apps app-office
...
and so on, from here script-specific info would follow. This is the best 
model IMHO.
Of course, under this model a job file can still be a script in its own 
right, and use none of the generic scripts.


The Portage-handled header: (I don't like this but have no better idea)
--------------------------------------------------------------------------------------
It can specify:
1. Name
2. Revision
3. Author
4. Date
5. Condition for running:
The simplest method: jobs are sorted into 3 groups, in three separate dirs:
1. Run once as soon as possible and never run again.
2. Run on-demand only. For example, jobs to configure a program after merging.
3. Run whenever all jobs are run, i.e. after emerge rsync. The job (i.e. 
script) itself will decide whether to take any action.

Note that this is cluttering and not very elegant. Jobs should be able to run 
many times, than tell portage not to run them any more. But, making portage 
evaluate a special-syntax run condition is such a bother. Please give new 
ideas for this.


When to run - options:
-----------------------------
1. The jobs will be executed after each run of emerge, including emerge rsync.

2. Before each run of emerge, to accommodate the following scenario: change 
in cvs requires job to run before any emerges.

3. Before/after runs of ebuild as well. (Probably not).

4. Provide "emerge jobs" command to run on demand.


Ideas/needs for jobs:
---------------------------
1. Move a package to a different category, update /var/db/pkg if the package 
was already installed. This is what we're doing with kde/gnome.

2. Output some info to the user. For example about an important new change in 
Portage/emerge/new KDE version etc., for the users who don't read the cvs 
logs or gentoo-dev.

3. Merge a new version of sys-apps/portage when this is needed for some 
ebuilds' new versions to work, like in the \" story. A user who got the new 
ebuilds with non-escaped quotes but didn't update portage would be in big 
trouble.

4. After merging a package, output usage intructions. Deprecate the 
pkg_postinst usage, emerge outputs info after it anyway.

5. As above, run a configuration program.

6. Manage a bug report system. A user who got an error running one of uor 
ebuilds would get this job executed.
...


Please add ideas!

-- 

Dan Armak
Gentoo Linux Developer, Desktop Team
Matan, Israel