[gentoo-dev] The new Portage Job System

[gentoo-dev] The new Portage Job System

[Dan Armak]

Hi all,

In accordance with drobbins' wishes, the discussion of the new Portage job 
system moves to gentoo-dev. This is a quick summary of the discussion on 
#gentoo yesterday.

The idea as of now is this: (my version)


Intro:
--------
Under profiles/jobs, or in a similar location, will be a job dir (Possibly 
profiles/default/jobs, and the references may then be redirected through 
/etc/make.profile). Developers can place jobs in it for Portage to execute.

Jobs can request to be run once, or many times based on a condition. Portage 
will keep track of jobs that have been run in e.g. /var/db/jobs.


Helper files:
----------------
Jobs can have misc files, a separate dir should be given to these i.e. 
profiles/jobs/files.


Generic scripts:
--------------------
Provide a dir profiles/jobs/scripts and populate it with *generic* scripts. 
Under profiles/jobs put job spec files which, in their first few (commented 
out) lines, have info about when to run them, and then have a #! line that 
specifies one of these generic scripts as the interpreter. For example, a job 
spec for moving a package would be called move-koffice.job, and begin:

# Job spec description: move the koffice package from kde-apps to app-office
# Run Once
# Author: Dan Armak <danarmak@gentoo.org>
# $Header: $
#! /usr/portage/profiles/jobs/scripts/movepackage.sh
move koffice kde-apps app-office
...
and so on, from here script-specific info would follow. This is the best 
model IMHO.
Of course, under this model a job file can still be a script in its own 
right, and use none of the generic scripts.


The Portage-handled header: (I don't like this but have no better idea)
--------------------------------------------------------------------------------------
It can specify:
1. Name
2. Revision
3. Author
4. Date
5. Condition for running:
The simplest method: jobs are sorted into 3 groups, in three separate dirs:
1. Run once as soon as possible and never run again.
2. Run on-demand only. For example, jobs to configure a program after merging.
3. Run whenever all jobs are run, i.e. after emerge rsync. The job (i.e. 
script) itself will decide whether to take any action.

Note that this is cluttering and not very elegant. Jobs should be able to run 
many times, than tell portage not to run them any more. But, making portage 
evaluate a special-syntax run condition is such a bother. Please give new 
ideas for this.


When to run - options:
-----------------------------
1. The jobs will be executed after each run of emerge, including emerge rsync.

2. Before each run of emerge, to accommodate the following scenario: change 
in cvs requires job to run before any emerges.

3. Before/after runs of ebuild as well. (Probably not).

4. Provide "emerge jobs" command to run on demand.


Ideas/needs for jobs:
---------------------------
1. Move a package to a different category, update /var/db/pkg if the package 
was already installed. This is what we're doing with kde/gnome.

2. Output some info to the user. For example about an important new change in 
Portage/emerge/new KDE version etc., for the users who don't read the cvs 
logs or gentoo-dev.

3. Merge a new version of sys-apps/portage when this is needed for some 
ebuilds' new versions to work, like in the \" story. A user who got the new 
ebuilds with non-escaped quotes but didn't update portage would be in big 
trouble.

4. After merging a package, output usage intructions. Deprecate the 
pkg_postinst usage, emerge outputs info after it anyway.

5. As above, run a configuration program.

6. Manage a bug report system. A user who got an error running one of uor 
ebuilds would get this job executed.
...


Please add ideas!

-- 

Dan Armak
Gentoo Linux Developer, Desktop Team
Matan, Israel

Re: [gentoo-dev] The new Portage Job System

[Mikael Hallendal]

Den 11 Aug 2001 12:05:01 +0300 skrev Dan Armak:
> Hi all,

Hi!

As I was in the discussion on IRC #gentoo I agree that we should have
something like this, I do howevery disagree on some of the stuff
below...

> Under profiles/jobs, or in a similar location, will be a job dir (Possibly 
> profiles/default/jobs, and the references may then be redirected through 
> /etc/make.profile). Developers can place jobs in it for Portage to execute.

Should it really be possible to redirect this? I guess it should always
be in the same place in portage and should be redirected by redirecting
the portage-tree.

> Helper files:
> ----------------
> Jobs can have misc files, a separate dir should be given to these i.e. 
> profiles/jobs/files.

Agreed.

> Generic scripts:
> --------------------
> Provide a dir profiles/jobs/scripts and populate it with *generic* scripts. 
> Under profiles/jobs put job spec files which, in their first few (commented 
> out) lines, have info about when to run them, and then have a #! line that 
> specifies one of these generic scripts as the interpreter. For example, a job 
> spec for moving a package would be called move-koffice.job, and begin:
> 
> # Job spec description: move the koffice package from kde-apps to app-office
> # Run Once
> # Author: Dan Armak <danarmak@gentoo.org>
> # $Header: $
> #! /usr/portage/profiles/jobs/scripts/movepackage.sh
> move koffice kde-apps app-office
> ...
> and so on, from here script-specific info would follow. This is the best 
> model IMHO.
> Of course, under this model a job file can still be a script in its own 
> right, and use none of the generic scripts.

I really don't like this approach. This way a wrong written script might
corrupt a users database. I think it's better if we have a few scripts
(i.e. those we have found out we need) and then we use the job-files for
giving parameters to the script. Like in the case of moving a package we
could have something like:

In job-file:

type: rellocate
author: Mikael Hallendal <hallski@gentoo.org>
version: $Header$
control-file: koffice-to-app-office.control

the control-file is then in files/

portage takes a look at type, recognizes it as a rellocate-type of job.
It excecutes:
'/usr/lib/portage/bin/rellocate
/usr/portage/profiles/jobs/files/koffice-to-app-office.control'

Only the system-team may add/approve scripts into portage so that we get
some control of what goes in. This way a badly written job-file will
result in an error when excecuting the script. A badly written
control-file might do more damage but I think this is unavoidable and
the control-file should have a really easy syntax.

> The Portage-handled header: (I don't like this but have no better 
> ideas)

Hmm, I'm not sure I follow you here. What is this for?

> When to run - options:
> -----------------------------
> 1. The jobs will be executed after each run of emerge, including emerge rsync.

What types of jobs would be needed to run after each emerge?

> 2. Before each run of emerge, to accommodate the following scenario: change 
> in cvs requires job to run before any emerges.

How will this be of any use. If something changed in the cvs it will
break the users tree at the same time he gets the job-descr. the actuall
job wouldn't run until next rsync.

> 4. Provide "emerge jobs" command to run on demand.

I think that we should have a 'emerge jobs'-kinda a tool which should be
used by developers which don't use 'emerge rsync'. I do however think
that jobs for users should be run by 'emerge rsync' and no otherwise.

> Ideas/needs for jobs:
> ---------------------------
> 1. Move a package to a different category, update /var/db/pkg if the package 
> was already installed. This is what we're doing with kde/gnome.

This is indeed a very much needed.

> 2. Output some info to the user. For example about an important new change in 
> Portage/emerge/new KDE version etc., for the users who don't read the cvs 
> logs or gentoo-dev.

This is also a good feature. But should only be run after an emerge
rsync.

> 3. Merge a new version of sys-apps/portage when this is needed for some 
> ebuilds' new versions to work, like in the \" story. A user who got the new 
> ebuilds with non-escaped quotes but didn't update portage would be in big 
> trouble.

Perhaps this should be handled either by (2), ie. give the user a
message about upgrading there portage-system. Or by actually doing
inside of 'emerge rsync'. The problem with automatically do stuff is
that there are people (me for one) that don't like when things happend
by themself. I like to have control of when I do update a package.
 
> 4. After merging a package, output usage intructions. Deprecate the 
> pkg_postinst usage, emerge outputs info after it anyway.
>
> 5. As above, run a configuration program.
> 
> 6. Manage a bug report system. A user who got an error running one of uor 
> ebuilds would get this job executed.
> ...

4-5 should be handled inside the ebuild (imho). I see no reason why it
should be in the jobs-system instead. If a package needs to be
configured afterwords it should be done in pkg_postinst (). Same for
giving the user a notice about the package.

6: would be a nice feature, but I don't think that it should be a job,
it should be in portage system itself. The system should give a question
if it fails to merge an ebuild asking if the user would like to report
the bug. If so it starts the routines for doing so and puts the results
in /var/db/bug-reports or something, then when doing emerge rsync
portage looks in /var/db/bug-reports and sends them somewhere.

Regards,
Mikael Hallendal

Re: [gentoo-dev] The new Portage Job System

[Dan Armak]

On Saturday 11 August 2001 16:37, you wrote:
> Den 11 Aug 2001 12:05:01 +0300 skrev Dan Armak:
> > Hi all,
>
> Hi!
>
> As I was in the discussion on IRC #gentoo I agree that we should have
> something like this, I do howevery disagree on some of the stuff
> below...
>
> > Under profiles/jobs, or in a similar location, will be a job dir
> > (Possibly profiles/default/jobs, and the references may then be
> > redirected through /etc/make.profile). Developers can place jobs in it
> > for Portage to execute.
>
> Should it really be possible to redirect this? I guess it should always
> be in the same place in portage and should be redirected by redirecting
> the portage-tree.
No, you didn't understand. My use of "redirect" was ambigous. 
/etc/make.profile is a symlink to /usr/portage/profiles/deafult (check your 
system, it's there). So I said a user might redirect references to 
/etc/make.profile with the same result as if he used 
/usr/portage/profiles/default directly. It's not a very important point.


>
> > Helper files:
> > ----------------
> > Jobs can have misc files, a separate dir should be given to these i.e.
> > profiles/jobs/files.
>
> Agreed.
>
> > Generic scripts:
> > --------------------
> > Provide a dir profiles/jobs/scripts and populate it with *generic*
> > scripts. Under profiles/jobs put job spec files which, in their first few
> > (commented out) lines, have info about when to run them, and then have a
> > #! line that specifies one of these generic scripts as the interpreter.
> > For example, a job spec for moving a package would be called
> > move-koffice.job, and begin:
> >
> > # Job spec description: move the koffice package from kde-apps to
> > app-office # Run Once
> > # Author: Dan Armak <danarmak@gentoo.org>
> > # $Header: $
> > #! /usr/portage/profiles/jobs/scripts/movepackage.sh
> > move koffice kde-apps app-office
> > ...
> > and so on, from here script-specific info would follow. This is the best
> > model IMHO.
> > Of course, under this model a job file can still be a script in its own
> > right, and use none of the generic scripts.
>
> I really don't like this approach. This way a wrong written script might
> corrupt a users database. I think it's better if we have a few scripts
> (i.e. those we have found out we need) and then we use the job-files for
> giving parameters to the script. Like in the case of moving a package we
> could have something like:
>
> In job-file:
>
> type: rellocate
> author: Mikael Hallendal <hallski@gentoo.org>
> version: $Header$
> control-file: koffice-to-app-office.control
>
> the control-file is then in files/
>
> portage takes a look at type, recognizes it as a rellocate-type of job.
> It excecutes:
> '/usr/lib/portage/bin/rellocate
> /usr/portage/profiles/jobs/files/koffice-to-app-office.control'
>
> Only the system-team may add/approve scripts into portage so that we get
> some control of what goes in. This way a badly written job-file will
> result in an error when excecuting the script. A badly written
> control-file might do more damage but I think this is unavoidable and
> the control-file should have a really easy syntax.
That's just what I meant!! The "generic" scripts are what you want trhe 
system team to approve, I agree with that. Then, any developer may add what I 
called "job spec files", which you called control-files - that's just what I  
meant. We just said the same thing in different terms.

>
> > The Portage-handled header: (I don't like this but have no better
> > ideas)
>
> Hmm, I'm not sure I follow you here. What is this for?
I'm not sure I followed your comments either, I've tried to answer them:

>
> > When to run - options:
> > -----------------------------
> > 1. The jobs will be executed after each run of emerge, including emerge
> > rsync.
>
> What types of jobs would be needed to run after each emerge?
I don't knwo - 

>
> > 2. Before each run of emerge, to accommodate the following scenario:
> > change in cvs requires job to run before any emerges.
>
> How will this be of any use. If something changed in the cvs it will
> break the users tree at the same time he gets the job-descr. the actuall
> job wouldn't run until next rsync.
I don't follow, sorry.

>
> > 4. Provide "emerge jobs" command to run on demand.
>
> I think that we should have a 'emerge jobs'-kinda a tool which should be
> used by developers which don't use 'emerge rsync'. I do however think
> that jobs for users should be run by 'emerge rsync' and no otherwise.
No, I didn't mean "emerge jobs" to download new jobs. I meant "emergre jobs" 
to execute currently present jobs, or a specific job. For example, a job that 
runs a configure script or interface for some package, and you want to 
reconfigure it. You don't want to remerge it, so you run the job yourself.

>
> > Ideas/needs for jobs:
> > ---------------------------
> > 1. Move a package to a different category, update /var/db/pkg if the
> > package was already installed. This is what we're doing with kde/gnome.
>
> This is indeed a very much needed.
>
> > 2. Output some info to the user. For example about an important new
> > change in Portage/emerge/new KDE version etc., for the users who don't
> > read the cvs logs or gentoo-dev.
>
> This is also a good feature. But should only be run after an emerge
> rsync.
A job can specify when it should be run.

>
> > 3. Merge a new version of sys-apps/portage when this is needed for some
> > ebuilds' new versions to work, like in the \" story. A user who got the
> > new ebuilds with non-escaped quotes but didn't update portage would be in
> > big trouble.
>
> Perhaps this should be handled either by (2), ie. give the user a
> message about upgrading there portage-system. Or by actually doing
> inside of 'emerge rsync'. The problem with automatically do stuff is
> that there are people (me for one) that don't like when things happend
> by themself. I like to have control of when I do update a package.
You can change the automatic/manual control of these things of course. A user 
who only uses Portage to emerge new apps he needs doesn't need to keep track 
of updates to Portage itself, just emerge rsync once in a while.

>
> > 4. After merging a package, output usage intructions. Deprecate the
> > pkg_postinst usage, emerge outputs info after it anyway.
> >
> > 5. As above, run a configuration program.
> >
> > 6. Manage a bug report system. A user who got an error running one of uor
> > ebuilds would get this job executed.
> > ...
>
> 4-5 should be handled inside the ebuild (imho). I see no reason why it
> should be in the jobs-system instead. If a package needs to be
> configured afterwords it should be done in pkg_postinst (). Same for
> giving the user a notice about the package.
Well, pkg_postinst comes before annoying portage messages that move it off 
the screen. Also currently ebuilds are supposed to be always non-interactive, 
not even e.g. displaying a help message with most but only with echo. Jobs 
can be interactive.

>
> 6: would be a nice feature, but I don't think that it should be a job,
> it should be in portage system itself. The system should give a question
> if it fails to merge an ebuild asking if the user would like to report
> the bug. If so it starts the routines for doing so and puts the results
> in /var/db/bug-reports or something, then when doing emerge rsync
> portage looks in /var/db/bug-reports and sends them somewhere.

I was only making suggestions, trying to think up ideas. I'm not saying all 
these things should be made jobs, or only them.




-- 

Dan Armak
Gentoo Linux Developer, Desktop Team
Matan, Israel

Re: [gentoo-dev] The new Portage Job System

[Daniel Robbins]

On Sat, Aug 11, 2001 at 12:05:01PM +0300, Dan Armak wrote:
> Hi all,
> 
> In accordance with drobbins' wishes, the discussion of the new Portage job 
> system moves to gentoo-dev. This is a quick summary of the discussion on 
> #gentoo yesterday.

IMHO, this is overkill.  We just need a way of executing certain jobs once;
specifically, jobs that update the /var/db/pkg structure so that moved packages
aren't remerged.  I don't want to create something huge to do this.

-- 
Daniel Robbins					<drobbins@gentoo.org>
Chief Architect/President			http://www.gentoo.org 
Gentoo Technologies, Inc.			

Re: [gentoo-dev] The new Portage Job System

[Dan Armak]

On Saturday 11 August 2001 21:11, you wrote:
> On Sat, Aug 11, 2001 at 12:05:01PM +0300, Dan Armak wrote:
> > Hi all,
> >
> > In accordance with drobbins' wishes, the discussion of the new Portage
> > job system moves to gentoo-dev. This is a quick summary of the discussion
> > on #gentoo yesterday.
>
> IMHO, this is overkill.  We just need a way of executing certain jobs once;
> specifically, jobs that update the /var/db/pkg structure so that moved
> packages aren't remerged.  I don't want to create something huge to do
> this.

Well, you're welcome to use any part of what I proposed. All I care about is 
the kde package moving, not the job system itself.


-- 

Dan Armak
Gentoo Linux Developer, Desktop Team
Matan, Israel

Re: [gentoo-dev] The new Portage Job System

[Dan Armak]

On Saturday 11 August 2001 21:11, you wrote:
> On Sat, Aug 11, 2001 at 12:05:01PM +0300, Dan Armak wrote:
> > Hi all,
> >
> > In accordance with drobbins' wishes, the discussion of the new Portage
> > job system moves to gentoo-dev. This is a quick summary of the discussion
> > on #gentoo yesterday.
>
> IMHO, this is overkill.  We just need a way of executing certain jobs once;
> specifically, jobs that update the /var/db/pkg structure so that moved
> packages aren't remerged.  I don't want to create something huge to do
> this.

drobbins, I think you're right. An important point is that we need to 
complete the kde/gnome move quickly. I have already copied the kde packages 
to their new locations, but kept the copies in the old olocations also. The 
system is therefore in a limbo.

Emerging all of kde isn't a problem, but since I changed several ebuilds to 
depend on the new locations, emerging them would remerge kde.

Worse, after remerging kde, /var/db/pkg would have e.g. both kde-apps/foo and 
kde-base/foo registered. Then, when my script was run on such a database, 
there would be problems and I'd have to make special provisions in the script.

The point is that we need either to complete the move quickly - even if it 
means not using jobs, or not iplementing jobs fully - or to backtrack and 
return kde to its original state until we can get the job system operational.



-- 

Dan Armak
Gentoo Linux Developer, Desktop Team
Matan, Israel