[gentoo-dev] RFC: persistence of eclasses for installed packages

[gentoo-dev] RFC: persistence of eclasses for installed packages

[Zac Medico]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi everyone,

As many of you probably know already, bug 46223 [1] prevents the
proper uninstallation of a package when one or more of the eclasses
that it inherits are missing from the live portage tree.

There are essentially two ways to solve this problem:

Approach #1: Reuse the saved environment which contains a mixture of
eclasses and parts of the package manager (such as portage's
ebuild.sh) that performed the installation.

Approach #2: Save copies of the raw eclasses and use them, together
with the saved ebuild, to recreate the environment.

A major advantage of approach #1 is that it can potentially provide
complete restoration of the install-time environment.  The major
disadvantage is that the saved environment needs to be purified so
that parts of the package manager that did the install do not
interfere with the package manager that does the uninstall.  The
purification process could potentially be complex or error prone,
making it difficult to maintain or unreliable.

Simplicity is the major advantage approach #2.  The eclasses that
are used by a package could be stored in eclasses.tbz2 (much like
enviroment.bz2 is currently stored for each installed package), and
the environment could be recreated from scratch by using the ebuild
and eclasses in the normal inherit process.  This wouldn't
necessarily allow access to the complete build-time or install-time
environment, but that feature is not currently available anyway.  If
we save the raw eclasses for each installed package, we can easily
gain the ability to remove eclasses from the portage tree and make
incompatible api changes when necessary, without the complexity of
environment purification.

What do people think about these two approaches?  Personally, I
would prefer approach #2 for the sake of simplicity and
maintainability.  The sooner that we start storing eclasses.tbz2 for
each installed package, the sooner that we will be able to have more
freedom with the eclasses in the live portage tree.

Zac

[1] http://bugs.gentoo.org/show_bug.cgi?id=46223

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFFTF58/ejvha5XGaMRApOeAJ9CncD89mRUy/mhyVZnhFjmazjYYACfYhdy
4qLqBySCLvAyuNm58bqXZnY=
=Ailg
-----END PGP SIGNATURE-----
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC: persistence of eclasses for installed packages

[Petteri Räty]

[-- Attachment #1: Type: text/plain, Size: 521 bytes --]

Zac Medico kirjoitti:
> 
> What do people think about these two approaches?  Personally, I
> would prefer approach #2 for the sake of simplicity and
> maintainability.  The sooner that we start storing eclasses.tbz2 for
> each installed package, the sooner that we will be able to have more
> freedom with the eclasses in the live portage tree.

One thing that comes to mind is that how do we handle the case where the
old version of the eclass has a major bug in pkg_postrm for example.

Regards,
Petteri


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [gentoo-dev] RFC: persistence of eclasses for installed packages

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 648 bytes --]

On Saturday 04 November 2006 06:14, Petteri Räty wrote:
> Zac Medico kirjoitti:
> > What do people think about these two approaches?  Personally, I
> > would prefer approach #2 for the sake of simplicity and
> > maintainability.  The sooner that we start storing eclasses.tbz2 for
> > each installed package, the sooner that we will be able to have more
> > freedom with the eclasses in the live portage tree.
>
> One thing that comes to mind is that how do we handle the case where the
> old version of the eclass has a major bug in pkg_postrm for example.

that would be the only thing keeping me from saying LETS DO #2 NOW ;)
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] RFC: persistence of eclasses for installed packages

[Zac Medico]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Petteri Räty wrote:
> Zac Medico kirjoitti:
>> What do people think about these two approaches?  Personally, I
>> would prefer approach #2 for the sake of simplicity and
>> maintainability.  The sooner that we start storing eclasses.tbz2 for
>> each installed package, the sooner that we will be able to have more
>> freedom with the eclasses in the live portage tree.
> 
> One thing that comes to mind is that how do we handle the case where the
> old version of the eclass has a major bug in pkg_postrm for example.

I suppose we could check the live tree for the required eclasses and
use them if they are all available.  Perhaps, in that case, we
should use the live ebuild too if it is available. In cases where
something isn't available in the live tree we could fall back to the
saved files as a last resort.  We'd have to maintain api
compatibility, but at least there would still be a reasonable chance
for the user to do a normal uninstall after some eclasses have been
removed.

Zac
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFFTIK3/ejvha5XGaMRAiRTAKCV31alUOZjVaC5qh3iWNBlEuW/wwCdFxiz
TgJ4hqxMap9thhy1VIEQrOk=
=o5pR
-----END PGP SIGNATURE-----
-- 
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: RFC: persistence of eclasses for installed packages

[Duncan]

Zac Medico <zmedico@gentoo.org> posted 454C82B8.9080504@gentoo.org,
excerpted below, on  Sat, 04 Nov 2006 04:08:24 -0800:

> Petteri Räty wrote:
>> Zac Medico kirjoitti:
>>> The sooner that we start storing eclasses.tbz2 for each installed
>>> package, the sooner that we will be able to have more freedom with the
>>> eclasses in the live portage tree.
>> 
>> One thing that comes to mind is that how do we handle the case where
>> the old version of the eclass has a major bug in pkg_postrm for
>> example.
> 
> I suppose we could check the live tree for the required eclasses and use
> them if they are all available.  Perhaps, in that case, we should use
> the live ebuild too if it is available. In cases where something isn't
> available in the live tree we could fall back to the saved files as a
> last resort.  We'd have to maintain api compatibility, but at least
> there would still be a reasonable chance for the user to do a normal
> uninstall after some eclasses have been removed.

What about some way of marking priority, along with versioning of some
sort?  Default to using the stored versions unless there's something
in-tree that says it's higher priority for version X.

Ideally, the need for out-ranking the stored version wouldn't come along
that often, and the checked policies could be centralized in a file in
base-profile or the like for speed, preferably with comments indicating the
dates the bad version was "live", and an expiry, say 30 days after bump of
all affected packages for unstable, 180 days after bump of affected
packages for stable.

I'm thinking something along the lines of package.mask, but probably not
cascaded, only in the base profile.  Note that this would /only/ contain
the supersede /policies/, including a pointer to the appropriate
ebuild/eclass/elib/whatever. The idea being to keep the incremental data
portage has to parse down to a minimum -- one file -- unless it mentions
an ebuild/eclass/elib (when/if they are implemented) used by the package
being removed.  (It'd actually have to be one file per repository, where
multiple repositories are supported and used, of course.)  Only if
something of interest were mentioned in the central file would the tree
version of the e-whatever file need to be parsed, thus keeping speed up
and complications from parsing some stuff from the live tree and some from
the saved files to a minimum, using the saved files, and thus the
environment as it was at the time of install, by default.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC: persistence of eclasses for installed packages

[Francesco Riosa]

Zac Medico ha scritto:
> Petteri Räty wrote:
>> Zac Medico kirjoitti:
>>> What do people think about these two approaches?  Personally, I
>>> would prefer approach #2 for the sake of simplicity and
>>> maintainability.  The sooner that we start storing eclasses.tbz2 for
>>> each installed package, the sooner that we will be able to have more
>>> freedom with the eclasses in the live portage tree.
>> One thing that comes to mind is that how do we handle the case where the
>> old version of the eclass has a major bug in pkg_postrm for example.
> 
> I suppose we could check the live tree for the required eclasses and
> use them if they are all available.  Perhaps, in that case, we
> should use the live ebuild too if it is available. In cases where
> something isn't available in the live tree we could fall back to the
> saved files as a last resort.  We'd have to maintain api
> compatibility, but at least there would still be a reasonable chance
> for the user to do a normal uninstall after some eclasses have been
> removed.
> 
> Zac

could it be the other way around ? Use the saved and if it fail try to
use the current?

Reason for this is that otherwise you need to maintain in the tree code
to un-merge very old packages, needlessy bloating the current, plus
experience teach this is very error prone.

There is a problem however with this approach, you can't fix bug where
the (original ebuild) exit code is "success" but the behaviour is "damn
bugged".
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC: persistence of eclasses for installed packages

[Stephen Bennett]

On Sat, 04 Nov 2006 13:02:19 +0000
Francesco Riosa <vivo@gentoo.org> wrote:

> There is a problem however with this approach, you can't fix bug where
> the (original ebuild) exit code is "success" but the behaviour is
> "damn bugged".

You can, but perhaps not in a nice, automatic, or particularly elegant
way. Just send out a (glep42) news item saying "The stored version of
foo.eclass has a serious bug. Please remove it from your installed
package database and re-install any packages which used it."
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC: persistence of eclasses for installed packages

[Yuri Vasilevski]

On Sat, 4 Nov 2006 06:46:02 -0500
Mike Frysinger <vapier@gentoo.org> wrote:

> On Saturday 04 November 2006 06:14, Petteri Räty wrote:
> > Zac Medico kirjoitti:
> > > What do people think about these two approaches?  Personally, I
> > > would prefer approach #2 for the sake of simplicity and
> > > maintainability.  The sooner that we start storing eclasses.tbz2
> > > for each installed package, the sooner that we will be able to
> > > have more freedom with the eclasses in the live portage tree.
> >
> > One thing that comes to mind is that how do we handle the case
> > where the old version of the eclass has a major bug in pkg_postrm
> > for example.

I think it is also possible that changes into a eclass introduce a new
unexpected bug, so this shouldn't be the only reason to stop doing this.

How about introducing a variable OBSOLETES="list of cvs revisions",
this way it can be automatically detected if the cached ebuild/eclass
can be used or if it needs to be ignored and the newer version from
portage that obsoletes it has to be used?

> that would be the only thing keeping me from saying LETS DO #2 NOW ;)

I'd vote, lets do #2 in any case.

Yuri.

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC: persistence of eclasses for installed packages

[Zac Medico]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Francesco Riosa wrote:
> could it be the other way around ? Use the saved and if it fail try to
> use the current?
> 
> Reason for this is that otherwise you need to maintain in the tree code
> to un-merge very old packages, needlessy bloating the current, plus
> experience teach this is very error prone.
> 
> There is a problem however with this approach, you can't fix bug where
> the (original ebuild) exit code is "success" but the behaviour is "damn
> bugged".

It gets a little messy if we have to check exit codes like that.  If
pkg_prerm() fails with the saved ebuild/eclasses, then we substitute
everything that we can from the live tree and then try again?  What
about if pkg_prerm() succeeds with the saved ebuild/eclasses, but
then  pkg_postrm() fails?  I suppose we would substitute everything
that we can from the live tree at that point.  It's a little messy
but I guess we can do that if that's how people think it should behave.

Anyway, portage-2.1.2 is nearing a final release and it may be the
stable version of portage for 2007.0, so it might be a good idea to
make it start saving eclasses.tbz2 for installed packages now (even
if it doesn't actually use them for anything).  I imagine that it
will just be a bzipped tarball containing an "eclass" directory with
all of the inherited eclasses inside.

Zac
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFFTTa9/ejvha5XGaMRAmI+AJ0URJ9FEssxQsT/qEvnRqJkUppQBACg1Isa
76JGdGN7DzmFBwBndd6DujA=
=dsoy
-----END PGP SIGNATURE-----
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC: persistence of eclasses for installed packages

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 7601 bytes --]

On Sat, Nov 04, 2006 at 01:33:49AM -0800, Zac Medico wrote:
> Hi everyone,
> 
> As many of you probably know already, bug 46223 [1] prevents the
> proper uninstallation of a package when one or more of the eclasses
> that it inherits are missing from the live portage tree.
> 
> There are essentially two ways to solve this problem:
> 
> Approach #1: Reuse the saved environment which contains a mixture of
> eclasses and parts of the package manager (such as portage's
> ebuild.sh) that performed the installation.

That's not technically true; what is required is stripping the manager 
*out* of the env, leaving in only things that are part of the ebuild 
env (eapi crap basically).

There are a few exceptions to this, but those exceptions are covering 
unversioned changes in the ebuild api (cue EAPI); example being the 
use function going silent after .50-r11; since older ebuilds relied on 
use echoing the flag to stdout if it was enabled, you have to use the 
ebuild envs copy of it to avoid breaking older installed ebuilds 
(something portage ignored, and broke I might add).


> Approach #2: Save copies of the raw eclasses and use them, together
> with the saved ebuild, to recreate the environment.
>
> A major advantage of approach #1 is that it can potentially provide
> complete restoration of the install-time environment.  The major
> disadvantage is that the saved environment needs to be purified so
> that parts of the package manager that did the install do not
> interfere with the package manager that does the uninstall.
> The
> purification process could potentially be complex or error prone,
> making it difficult to maintain or unreliable.

The complexity comes down to tracking what the manager provides, and 
knowing the internal vars to strip (DISTDIR is a system specific 
setting, should be pulled from the local system everytime)...

This actually is a *good* thing; 

1) forces ebuild.sh functionality to be written nonsucky, namely being 
aware of the side affects of function invocations; specifically, not 
bleeding vars (use local like a mad man).

2) actually allows distdir/portdir to change between phases if the 
user (typically a dev) is running phases manually for testing, and 
happens to change something in between phases (happens, and is 
annoying).

So... it requires having at least an audited ebuild.sh, which is 
pretty simple (good thing to do anyways), and a bash side that isn't 
fragile.

The *one* spot I'd define as tricky is stripping functions out of the 
environment dump, since sans pkgcores filter-env, no good method 
existed for it till recently; <bash-3.2 would choke on certain 
functions (try f-x() { :; }; declare -f f-x on 3.1 sometime), which's 
--read-function blows, and the grep approach is a total no go (here 
ops can cause a fair bit of hell for line based grep).

That said, with bash 3.2 doing a selective dump of the environment is 
now doable completely in pure bash, far simpler implementation.

So, error prone?  Not really in my experience; any bugs I've hit over 
the last 2 years of running ebd off and on have always been related 
to trying to match filter-env to bash parsing rules; with bash 3.2, 
can do it in about 10 lines of shell give or take.


> Simplicity is the major advantage approach #2.  The eclasses that
> are used by a package could be stored in eclasses.tbz2 (much like
> enviroment.bz2 is currently stored for each installed package), and
> the environment could be recreated from scratch by using the ebuild
> and eclasses in the normal inherit process.

Addressed below...


> This wouldn't
> necessarily allow access to the complete build-time or install-time
> environment, but that feature is not currently available anyway.

Was at one point.  Bit curious what happened to those changes since I 
went through and fixed all of that crap about a year back (it *may* 
have stayed only in the portage-2.1_alpha branch).


> If
> we save the raw eclasses for each installed package, we can easily
> gain the ability to remove eclasses from the portage tree and make
> incompatible api changes when necessary, without the complexity of
> environment purification.

That data isn't in the vdb *now*, and there is no way to force folks 
to upgrade *now*.

This isn't one of the normal "wait six months, and then make them go 
through a bit of hell", this is "break their ability to use all 
preexisting binpkgs and make it far easier for folks to have pkgs 
that cannot be uninstalled".

You cannot screw with the existing eclasses in a backwards 
incompatible way without running a real risk of breaking vdb/binpkg 
ebuilds (built ebuilds essentially); re-read glep33 and 
experiment locally if in doubt.


> The sooner that we start storing eclasses.tbz2 for
> each installed package, the sooner that we will be able to have more
> freedom with the eclasses in the live portage tree.

Reiterating it again since this is massively wrong; read glep33.  
Existing eclasses pretty much are stuck in the tree until it's decided 
to cause a fair bit of hell for folks lagging behind on their $PKG_MANAGER 
updates.

That is the sole reason that glep33 required moving all new eclasses 
to eclass2, a location that existing managers do not know about; keeps 
folks who have crappy managers from touching the dynamic eclasses and 
winding up broke, while removing the static restriction on the new 
eclasses.


> What do people think about these two approaches?  Personally, I
> would prefer approach #2 for the sake of simplicity and
> maintainability.  

One thing that hasn't yet been mentioned is that ebuilds are written 
to expect their environment to be accessible through the phases; folks 
*expect* that if they set something in src_compile, it'll be there in 
src_install.

Re-sourcing the ebuild/eclass for every phase means that it's now 
possible for the ebuild/eclass to inadvertantly corrupt the stored 
phase state; extension of this problem is that profile.env and friends 
get sourced *after* the ebuilds phase dump is restored.

Short version; look at java-2 eclass, and note that they're using a 
user specific hook to restore the environment prior to every single 
phase execution.  They're forced to do this because portage assumes it 
can rebuild the environment via re-executing code that is intended to 
*generate* an initial environment.

You run from the (existing) saved env, it works now for vdb, for 
binpkgs, enables dynamic eclasses, and wipes out the entire class of 
issues from the manager/initial env screwing up the ebuilds expected 
stated.  Approach #2 doesn't.

Finally, portage already has an implementation of proper environemnt 
saving/restoration- portage-2.1_alpha* holds EBD, and pkgcore holds 
the current version of it.  Ripping it out of pkgcore and pushing 
in the updates is fairly straight forward work (rote), and carries 
with it a 2x regen speedup while fixing the issues from above.

Already got pkgcores cache, might as well rip out more bits ;)

Not after the regen speed up via daemonizing ebuild.sh?  Fair 
enough, the code for proper env saving/restoration isn't bound to it; 
can rip that out without issue.

So... basically either pull a NIH, or nudge portage the remaining 
steps to gain the full support (harsh, but the environment data is 
there and has been for a long while, it's just a matter of using the 
work I've already done to finish the job).

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC: persistence of eclasses for installed packages

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 2964 bytes --]

On Sat, Nov 04, 2006 at 04:56:31PM -0800, Zac Medico wrote:
> Francesco Riosa wrote:
> > could it be the other way around ? Use the saved and if it fail try to
> > use the current?
> > 
> > Reason for this is that otherwise you need to maintain in the tree code
> > to un-merge very old packages, needlessy bloating the current, plus
> > experience teach this is very error prone.
> > 
> > There is a problem however with this approach, you can't fix bug where
> > the (original ebuild) exit code is "success" but the behaviour is "damn
> > bugged".
> 
> It gets a little messy if we have to check exit codes like that.

Can't rely on exit codes coming back from phases anyways; 
ebuild/eclass devs don't force it to be 0 for success- last time I 
tried checking exit code, just wound up making things bail out a lot 
due to false positives.

> If pkg_prerm() fails with the saved ebuild/eclasses, then we substitute
> everything that we can from the live tree and then try again?  What
> about if pkg_prerm() succeeds with the saved ebuild/eclasses, but
> then  pkg_postrm() fails?  I suppose we would substitute everything
> that we can from the live tree at that point.  It's a little messy
> but I guess we can do that if that's how people think it should behave.

This approach should *never* be tried; reasons-

You're going to have fork for every search attempt to protect the 
original env. from getting crap from failing eclasses

Even if you find a combination that doesn't explode, there is no 
gurantee it'll actually run completely; whats nasty is if it 
*partially* runs; (pre|pkg)(inst|rm) are not tracked in any fashion, 
so any changes they do t the livefs isn't tracked.

Further, ebuilds/eclasses *do* pair prerm and postrm changes; swapping 
in a different phase func their requires ebuild/eclass devs to always 
rely on the same state vars between those phases (not going to 
happen).

This situation gets massively worse when (yes when) dyanmic eclasses 
(able to be removed, api can change) hits, since at least with the 
current static eclasses folks are forced to keep things compatible.  
Remove that restriction, and this proposed approach becomes a much 
worse minefield- end result there is that dynamic eclasses cannot be 
implemented until whats proposed above is backed out.


> Anyway, portage-2.1.2 is nearing a final release and it may be the
> stable version of portage for 2007.0, so it might be a good idea to
> make it start saving eclasses.tbz2 for installed packages now (even
> if it doesn't actually use them for anything).  I imagine that it
> will just be a bzipped tarball containing an "eclass" directory with
> all of the inherited eclasses inside.

Env handling isn't something that should be rushed (this includes 
slipping in eclasses.tbz2, which I posit will not be used); the data 
needed has been there for a *long* time already, use it.

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]