[gentoo-dev] RFC: emerge snapshots

[gentoo-dev] RFC: emerge snapshots

[Kalin KOZHUHAROV]

Hi all.

During the last many months, more than once an idea occured in my mind, so I
decided to share it.


2006-01-25T01:34 kalin $ dd if=/dev/brain of=gentoo-dev bs=1 count=3292

Do you think it will be good to have something like a snapshot of the
installed packages?
Something that will help when something unexpectedly breaks and you cannot
fix it bu shuffling around its dependencies.

Because s--t does happen; sometimes.

Right now vmware-workstaion is broken and I am failing to understand why for
the last few hours. But that is just an example, forget about it.

So how can we get that thing? Let's call it "emerge snapshot" not to be
confused with portage snapshots used during install.

One way is to employ a version control system, say subversion, to keep the
list of installed packages. It has to be updated during emerge.
Also there should be a way to insert markers like "OK, tested and working",
"going to try this fuzzy foo install, mark!" All these should be timestamped.

So after the "_BANG_ my box is b0rked", it will be easy to retrace your
steps by just doing `emerge --log 3` to see the last three labels/log entries:

3544	2006-01-01T12:12	OK, tested and working
3545	2006-01-01T12:12	going to try this fuzzy foo install, mark
3546	2006-01-01T12:12	yay, new modular X is in ~arch (-:

then `emerge --revert -r3544` and everything is smooth and runnig again.

The "behind the scenes" actions to be taken by portage is to unmerge
everything that was merged after -r3544

That was the easy part.

Now what if some config file is b0rked?

Solution 0 (slow, disk space):

_before_ upgrading a package, or even before installing the same package in
a new SLOT, do `quickpkg $PACKAGE`

Solution 1 (better): Anybody?


Some notes to consider:

AFAIK, at present portage leaves most (all?) files that are in
CONFIG_PROTECT on unmerge... is there any easy way to trace which files are
left over in the system and delete them? Parsing the logs might help, but
unless it is automated it will be error prone and cumbersome.

Now imagine this timeline:
r100
emerge foo
r101
emerge --unmerge foo
r102

Is there anything else to be taken care so that the system @r100 and @102 is
the same? (sans system logs, user created files and the like)

I guess all the tools and info are already in portage and by writing a bunch
of scripts (to parse the logs mainly) it will be feasible to make it work.
But if it is automated, this will give _HUGE_ advantages:

1. Users will be instructed to revert to a good point when something breaks.
Probably an automated reporter can be hacked up

2. Bugs can easily be confirmed if there is enough CPU power
(Give me your broken "emerge snapshot" and the one before that)

3. devs and trying-to-be-devs, testers will be able to test easily if
package foo-2.3.4 does break something without the need of things like
vmware (snapshots)

4. Any n00b will have the "Ctrl+Z", "Recycle bin", "Trash bin", ":undo"
whatever to their whole system

5. (dangerous) Experimenting on semi-production servers will be easy!

6. Given the above 5, Gentoo user base will grow enourmously:

"You have the flexibility, now you have stability. No need to dream, welcome
to reality: it's called Gentoo!" (from the 2006.3 release campaign :-)

I know it is far from a GLEP, but let me first see how people feel about it.
Even if not implemented as a whole (at first) it will greatly improve our
Gentoo life.

Any comments? It is an RFC after all :-)

2006-01-25T02:07 kalin $
-- 
|[ ~~~~~~~~~~~~~~~~~~~~~~ ]|
+-> http://ThinRope.net/ <-+
|[ ______________________ ]|
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC: emerge snapshots

[Marius Mauch]

[-- Attachment #1: Type: text/plain, Size: 1121 bytes --]

On Wed, 25 Jan 2006 02:12:38 +0900
Kalin KOZHUHAROV <kalin@thinrope.net> wrote:

> Hi all.
> 
> During the last many months, more than once an idea occured in my
> mind, so I decided to share it.
> 
> 
> 2006-01-25T01:34 kalin $ dd if=/dev/brain of=gentoo-dev bs=1
> count=3292
> 
> Do you think it will be good to have something like a snapshot of the
> installed packages?
[snip]

It sounds like you want to build a complete system backup and recovery
solution into portage, which IMO is way beyond the scope of a package
manager.
Now AFAIK saviour can support multiple modular vdb backends, so you
might be able to plug such a system into it, but a) that is a rather
long time ahead and b) noone inside the portage team will probably work
on such a feature, so you'll have to write that code yourself.
Note that I'm not questioning the gain of this, just that it won't get
done by us.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC: emerge snapshots

[Wernfried Haas]

[-- Attachment #1: Type: text/plain, Size: 542 bytes --]

2 (already implemented) things you may find useful (unless you know
them already of course):
- adding buildpkg to your FEATURES builds binary packages, which makes 
  it faster to revert to older versions if the new one cause
  problems.
- dispatch-conf does a great job for the configuration files,
  especially when up- and downgrading packages.

HTH,
	Wernfried

-- 
Wernfried Haas (amne) - amne at gentoo dot org
Gentoo Forums: http://forums.gentoo.org
IRC: #gentoo-forums on freenode - email: forum-mods at gentoo dot org

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re[2]: [gentoo-dev] RFC: emerge snapshots

[Jakub Moc]

[-- Attachment #1: Type: text/plain, Size: 580 bytes --]


24.1.2006, 19:36:49, Wernfried Haas wrote:

> 2 (already implemented) things you may find useful (unless you know
> them already of course):
> - adding buildpkg to your FEATURES builds binary packages, which makes 
>   it faster to revert to older versions if the new one cause
>   problems.
> - dispatch-conf does a great job for the configuration files,
>   especially when up- and downgrading packages.

... and quickpkg makes a quick backup of a package incl. configuration files
and modifications you've made since it has been installed.



-- 

jakub


[-- Attachment #2: Type: application/pgp-signature, Size: 183 bytes --]

Re: [gentoo-dev] RFC: emerge snapshots

[A. Khattri]

On Tue, 24 Jan 2006, Wernfried Haas wrote:

> - adding buildpkg to your FEATURES builds binary packages, which makes
>   it faster to revert to older versions if the new one cause
>   problems.

You could also use quickpkg, i.e. write a script that interates through
world and runs quickpkg on each atom. Quickpkg will drop a binary package
in PKGDIR (which is set in /etc/make.conf). If you make a backup of PKGDIR
and your world file, you have a crude packages backup which can be used
to rebuild the box if need be.


-- 

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC: emerge snapshots

[Francesco Riosa]

A. Khattri wrote:
> On Tue, 24 Jan 2006, Wernfried Haas wrote:
> 
>> - adding buildpkg to your FEATURES builds binary packages, which makes
>>   it faster to revert to older versions if the new one cause
>>   problems.
> 
> You could also use quickpkg, i.e. write a script that interates through
> world and runs quickpkg on each atom. Quickpkg will drop a binary package
> in PKGDIR (which is set in /etc/make.conf). If you make a backup of PKGDIR
> and your world file, you have a crude packages backup which can be used
> to rebuild the box if need be.
> 

Better create the list from /var/db/pkg/* , world file only own the file
explicitly merged, leaving out any dependencies (fex libraries).

The backup of the world file is still (and more) needed
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC: emerge snapshots

[A. Khattri]

On Tue, 24 Jan 2006, Francesco Riosa wrote:

> Better create the list from /var/db/pkg/* , world file only own the file
> explicitly merged, leaving out any dependencies (fex libraries).

Yes you're right - that is what I did last time I played with this.


-- 

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC: emerge snapshots

[Sven Vermeulen]

[-- Attachment #1: Type: text/plain, Size: 995 bytes --]

On Wed, Jan 25, 2006 at 02:12:38AM +0900, Kalin KOZHUHAROV wrote:
> r100
> emerge foo
> r101
> emerge --unmerge foo
> r102

You're like asking to put your entire system under a versioning system
control. Sounds like fun, but also like a lot of diskspace usage on your
versioning server.

$ ${VCTOOL} commit --label=r100 --recursive /
$ emerge foo
$ ${VCTOOL} commit --label=r101 --recursive /
...
$ ${VCTOOL} checkout --label=r100 --recursive /

Using the FEATURES="buildpkg" helps a lot, especially if you also take
snapshots of your portage tree (since any breakage due to an "emerge -uDN
world" can hopefully be fixed by restoring the backup portage tree and run
"emerge -uDN world" again).

If you then put your configuration files under versioning control, you
should be set.

Wkr,
      Sven Vermeulen

-- 
  Gentoo Foundation Trustee          |  http://foundation.gentoo.org
  Gentoo Council Member  

  The Gentoo Project   <<< http://www.gentoo.org >>>

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-dev] Re: RFC: emerge snapshots

[Duncan]

Sven Vermeulen posted <20060126170040.GG7201@gentoo.org>, excerpted below,
 on Thu, 26 Jan 2006 18:00:40 +0100:

>  (since any breakage due to an "emerge -uDN
> world" can hopefully be fixed by restoring the backup portage tree and run
> "emerge -uDN world" again)

Neither here nor there, but for some reason, I prefer "emerge -NuD world".
=8^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman in
http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html


-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: RFC: emerge snapshots

[Stephen Bennett]

On Fri, 27 Jan 2006 16:08:40 -0700
Duncan <1i5t5.duncan@cox.net> wrote:

> Neither here nor there, but for some reason, I prefer "emerge -NuD
> world". =8^)

Especially since you just said yourself it was irrelevant, I'm at a loss
to see quite why you felt the need to tell us this.
-- 
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: Re: RFC: emerge snapshots

[Duncan]

Stephen Bennett posted <20060127233132.464db991@localhost>, excerpted
below,  on Fri, 27 Jan 2006 23:31:32 +0000:

> On Fri, 27 Jan 2006 16:08:40 -0700
> Duncan <1i5t5.duncan@cox.net> wrote:
> 
>> Neither here nor there, but for some reason, I prefer "emerge -NuD
>> world". =8^)
> 
> Especially since you just said yourself it was irrelevant, I'm at a loss
> to see quite why you felt the need to tell us this.

Perhaps for the same reason you replied to me, continuing what you are
complaining about as irrelevant... and now, I'm replying to you for much
the same reason, and will anyone reply to this for the same reason, ending
up with how many irrelevant posts in the process of complaining about one,
a short one at that?

Or, to put it differently, for much the same reason some seem to find
goats and <location> conspiracies worth remarking about here.  It's an
attempt at adding a bit of humor to what otherwise might get too dry
and technical to be tolerable.

... Just answering the question asked of me...

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman in
http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html


-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: Re: RFC: emerge snapshots

[Marcelo Góes]

On 1/27/06, Duncan <1i5t5.duncan@cox.net> wrote:
> attempt at adding a bit of humor to what otherwise might get too dry
> and technical to be tolerable.

This list _is_ for technical discussion. Please keep it that way.
Thanks.
--
Marcelo Góes
marcelogoes@gmail.com
vanquirius@gentoo.org

-- 
gentoo-dev@gentoo.org mailing list