[gentoo-dev] Phase invariancy and exclusivity requirements

[gentoo-dev] Phase invariancy and exclusivity requirements

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 2164 bytes --]

What specifically are the phase invariancy and exclusivity requirements
for ebuilds? Currently PMS doesn't have anything to say about this;
clearly it needs to, since existing ebuilds fairly obviously do have
invariancy and exclusivity requirements.

Note that we're only discussing package manager related things here. If
the user manually upgrades libc / switches compiler / whatever whilst a
package manager is busy, there's nothing we can do.

Is the following set sufficient? Is the following set the least
restrictive correct solution?

* No syncing whilst anything else is going on.

* Variancy is any package manager action that modifies ROOT in a way
that isn't merely a simple addition of something that doesn't alter
other packages. This includes any non-default pkg_(pre|post)(inst|rm),
merging to ROOT and unmerging from ROOT.

* As an exception, changes to DISTDIR do not count as variancy. [1]

* pkg_setup does not introduce variancy. [2]

* Any pkg_ function that is not the default must be run exclusively. [3]

* No variancy may be introduced at any point between a package's
pkg_setup run up to the point that it is merged, except for any
variancy introduced by that package.

* There must be no variancy between a package's pkg_setup and a
package's pkg_postinst, except for any variancy introduced by that
package.

[1]: This allows background fetching. It means DISTDIR can be added to
by other processes at any point. It doesn't mean that a package's ${A}
can be nuked randomly.

[2]: Because otherwise a failed install would result in a damaged
system, and an install would temporarily damage a system until
complete. Adding a user isn't variancy by our definition, since when
combined with the exclusivity requirements it doesn't alter any part of
other packages.

[3]: Weird stuff happens if, for example, two package's pkg_postinsts
are run at the same time, since ebuilds do no ROOT locking. I'm fairly
sure the exclusivity needs to be shared amongst all pkg_ phases (think
package one doing a useradd fred in pkg_setup and package two doing it
in pkg_postinst).

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Phase invariancy and exclusivity requirements

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1453 bytes --]

On Fri, 9 Nov 2007 22:40:08 +0000
Ciaran McCreesh <ciaran.mccreesh@blueyonder.co.uk> wrote:
> Is the following set sufficient? Is the following set the least
> restrictive correct solution?

... to explain the implications of these...

Say we have packages a, b and c, and none of them have any
dependencies. One valid solution to the build ordering is as follows:

* Install a
* Install b
* Install c

One of many solutions that is *not* valid is:

* Start doing a, b and c in parallel. Install them as they become
ready, doing only one merge at once.

Another that is not valid is:

* Start doing a, b and c in parallel, but don't merge them.
* Merge a.
* Merge b.
* Merge c.

One that is valid is:

* Build binary packages for a, b and c in parallel.
* Merge a's binary.
* Merge b's binary.
* Merge c's binary.

Another trickier situation. Say a-1 is installed, and a-2 and b are
targets, and b deps upon a (any version). By the rules given, this is
allowed:

* Build binary packages for a-2 and b in parallel.
* Merge a-2's binary (and clean a-1).
* Merge b's binary.

The situation becomes a whole lot more fun when, for example, we have
ten packages with interdependencies, and we only want to build at most
three things at once. That's why it pretty much has to be defined in
terms of invariancies and exclusivities rather than by listing a small
set of permitted algorithms.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Phase invariancy and exclusivity requirements

[Marijn Schouten (hkBst)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ciaran McCreesh wrote:
> On Fri, 9 Nov 2007 22:40:08 +0000
> Ciaran McCreesh <ciaran.mccreesh@blueyonder.co.uk> wrote:
>> Is the following set sufficient? Is the following set the least
>> restrictive correct solution?
> 
> ... to explain the implications of these...
> 
> Say we have packages a, b and c, and none of them have any
> dependencies. One valid solution to the build ordering is as follows:
> 
> * Install a
> * Install b
> * Install c
> 
> One of many solutions that is *not* valid is:
> 
> * Start doing a, b and c in parallel. Install them as they become
> ready, doing only one merge at once.
> 
> Another that is not valid is:
> 
> * Start doing a, b and c in parallel, but don't merge them.
> * Merge a.
> * Merge b.
> * Merge c.
> 
> One that is valid is:
> 
> * Build binary packages for a, b and c in parallel.
> * Merge a's binary.
> * Merge b's binary.
> * Merge c's binary.

What exactly is the difference between this valid situation and the previous
invalid one?

Marijn

- --
Marijn Schouten (hkBst), Gentoo Lisp project, Gentoo ML
<http://www.gentoo.org/proj/en/lisp/>, #gentoo-{lisp,ml} on FreeNode
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHOEaGp/VmCx0OL2wRAlShAKCNohJzGNppNM7LFgHT/ID/9AyVjwCeJhlM
vGHuzGLLa/+Oyj1t2T1KTP4=
=TKhb
-----END PGP SIGNATURE-----
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Phase invariancy and exclusivity requirements

[Piotr Jaroszyński]

On Monday 12 of November 2007 13:26:46 Marijn Schouten (hkBst) wrote:
> What exactly is the difference between this valid situation and the
> previous invalid one?

between | | are things that can be done in parallel.

invalid:
a_pkg_setup b_pkg_setup a_build b_build | a_merge | b_merge

valid:
a_pkg_setup b_pkg_setup a_build_binary b_build_binary | a_binary_pkg_setup | 
a_binary_merge | b_binary_pkg_setup | b_binary_merge

Note that pkg_setup is run twice for the second case, so when the merge order 
is a then b, b_pkg_setup is aware of the changes that a_merge did, which is 
not the case in first situation.

-- 
Best Regards,
Piotr Jaroszyński
--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Phase invariancy and exclusivity requirements

[Fernando J. Pereda]

[-- Attachment #1: Type: text/plain, Size: 1522 bytes --]

On Mon, Nov 12, 2007 at 01:26:46PM +0100, Marijn Schouten (hkBst) wrote:
> Ciaran McCreesh wrote:
> > On Fri, 9 Nov 2007 22:40:08 +0000
> > Ciaran McCreesh <ciaran.mccreesh@blueyonder.co.uk> wrote:
> >> Is the following set sufficient? Is the following set the least
> >> restrictive correct solution?
> > 
> > ... to explain the implications of these...
> > 
> > Say we have packages a, b and c, and none of them have any
> > dependencies. One valid solution to the build ordering is as follows:
> > 
> > * Install a
> > * Install b
> > * Install c
> > 
> > One of many solutions that is *not* valid is:
> > 
> > * Start doing a, b and c in parallel. Install them as they become
> > ready, doing only one merge at once.
> > 
> > Another that is not valid is:
> > 
> > * Start doing a, b and c in parallel, but don't merge them.
> > * Merge a.
> > * Merge b.
> > * Merge c.
> > 
> > One that is valid is:
> > 
> > * Build binary packages for a, b and c in parallel.
> > * Merge a's binary.
> > * Merge b's binary.
> > * Merge c's binary.
> 
> What exactly is the difference between this valid situation and the previous
> invalid one?

The state of the environment when pkg_setup is run. In the first
situation you can't trust it (it is racy and unpredictable among other
things). In the second one, you can.

That's the first thing that I can think of, there might be others.

- ferdy

-- 
Fernando J. Pereda Garcimartín
20BB BDC3 761A 4781 E6ED  ED0B 0A48 5B0C 60BD 28D4

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Phase invariancy and exclusivity requirements

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 663 bytes --]

On Mon, 12 Nov 2007 13:26:46 +0100
"Marijn Schouten (hkBst)" <hkBst@gentoo.org> wrote:
> What exactly is the difference between this valid situation and the
> previous invalid one?

It's basically down to whether pkg_setup has to be run with the same
system state as pkg_preinst / pkg_postinst. If arbitrary changes to the
live system are permitted between pkg_setup and pkg_preinst then the
rules I gave are too strict. But I suspect that arbitrary changes will
cause breakages to quite a few packages, and given that there're fairly
nice solutions even with the stricter requirements, I'd say that the
safe rules are better.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]