[gentoo-dev] i have an idea ! (erescue)

[gentoo-dev] i have an idea ! (erescue)

[Mike Frysinger]

one advantage that other binary based package managers have over Gentoo is 
ease of recovery from broken core packages ... break your gcc ?  no problem !  
simply do `apt-get install gcc` or `rpm -i gcc` or whatever

my proposal is to implement a new utility (called 'erescue' for lack of a 
better name) that is written in C and designed to be statically linked ... 
then next time you break a core system package which cannot be recovered by 
simply running `emerge` a few times, you run `erescue <broken package>`

for example, when i broke binutils in unstable with a gcc4 patch, i noticed 
that it's hard for users to *easily* recover from this ... we developers end 
up scrambling to build a bunch of binary packages for a variety of compatible 
compiler/libc combinations so the user can just wget the file and run `emerge 
binutils.tbz2` and be on their way

the packages that would be eligible for an 'erescue' package would be just 
about everything when you do `USE=-* emerge system -ep` ... i'm sure we can 
trim many of those out though :)  maybe even create a new USE flag for some 
of these core packages so that we can trim out more files

the idea would be to create very bare min packages so that the user can simply 
'rescue' themselves ... after that, they it's up to them to re-emerge the 
package to apply all their fun ricer-optimizations as they see fit

i dont think it'd be too hard to integrate this 'rescue set' into a catalyst 
target so that it'll become part of our normal release schedule of stage 
tarballs
-mike
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] i have an idea ! (erescue)

[Tom Wesley]

[-- Attachment #1: Type: text/plain, Size: 1897 bytes --]

On Sun, 2005-05-15 at 17:18 -0400, Mike Frysinger wrote:
> one advantage that other binary based package managers have over Gentoo is 
> ease of recovery from broken core packages ... break your gcc ?  no problem !  
> simply do `apt-get install gcc` or `rpm -i gcc` or whatever
> 
> my proposal is to implement a new utility (called 'erescue' for lack of a 
> better name) that is written in C and designed to be statically linked ... 
> then next time you break a core system package which cannot be recovered by 
> simply running `emerge` a few times, you run `erescue <broken package>`
> 
> for example, when i broke binutils in unstable with a gcc4 patch, i noticed 
> that it's hard for users to *easily* recover from this ... we developers end 
> up scrambling to build a bunch of binary packages for a variety of compatible 
> compiler/libc combinations so the user can just wget the file and run `emerge 
> binutils.tbz2` and be on their way
> 
> the packages that would be eligible for an 'erescue' package would be just 
> about everything when you do `USE=-* emerge system -ep` ... i'm sure we can 
> trim many of those out though :)  maybe even create a new USE flag for some 
> of these core packages so that we can trim out more files
> 
> the idea would be to create very bare min packages so that the user can simply 
> 'rescue' themselves ... after that, they it's up to them to re-emerge the 
> package to apply all their fun ricer-optimizations as they see fit
> 
> i dont think it'd be too hard to integrate this 'rescue set' into a catalyst 
> target so that it'll become part of our normal release schedule of stage 
> tarballs
> -mike

This would really help me at work, as I generally have to partimage my
workstation PC before updates that might break stuff.

I nominate "ecockup-reverse" as the name.

-- 
Tom Wesley <tom@tomaw.org>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] i have an idea ! (erescue)

[Henrik Brix Andersen]

[-- Attachment #1: Type: text/plain, Size: 904 bytes --]

On Sun, 2005-05-15 at 17:18 -0400, Mike Frysinger wrote:
> my proposal is to implement a new utility (called 'erescue' for lack of a 
> better name) that is written in C and designed to be statically linked ... 
> then next time you break a core system package which cannot be recovered by 
> simply running `emerge` a few times, you run `erescue <broken package>`
> 
> for example, when i broke binutils in unstable with a gcc4 patch, i noticed 
> that it's hard for users to *easily* recover from this ... we developers end 
> up scrambling to build a bunch of binary packages for a variety of compatible 
> compiler/libc combinations so the user can just wget the file and run `emerge 
> binutils.tbz2` and be on their way

Having been in a similar situation a couple of times myself, I think
that's a fantastic idea.

./Brix
-- 
Henrik Brix Andersen <brix@gentoo.org>
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] i have an idea ! (erescue)

[Carlos Silva]

[-- Attachment #1: Type: text/plain, Size: 1711 bytes --]

On Sun, 2005-05-15 at 17:18 -0400, Mike Frysinger wrote:
> one advantage that other binary based package managers have over Gentoo is 
> ease of recovery from broken core packages ... break your gcc ?  no problem !  
> simply do `apt-get install gcc` or `rpm -i gcc` or whatever
> 
> my proposal is to implement a new utility (called 'erescue' for lack of a 
> better name) that is written in C and designed to be statically linked ... 
> then next time you break a core system package which cannot be recovered by 
> simply running `emerge` a few times, you run `erescue <broken package>`
> 
> for example, when i broke binutils in unstable with a gcc4 patch, i noticed 
> that it's hard for users to *easily* recover from this ... we developers end 
> up scrambling to build a bunch of binary packages for a variety of compatible 
> compiler/libc combinations so the user can just wget the file and run `emerge 
> binutils.tbz2` and be on their way
> 
> the packages that would be eligible for an 'erescue' package would be just 
> about everything when you do `USE=-* emerge system -ep` ... i'm sure we can 
> trim many of those out though :)  maybe even create a new USE flag for some 
> of these core packages so that we can trim out more files
> 
> the idea would be to create very bare min packages so that the user can simply 
> 'rescue' themselves ... after that, they it's up to them to re-emerge the 
> package to apply all their fun ricer-optimizations as they see fit
> 
> i dont think it'd be too hard to integrate this 'rescue set' into a catalyst 
> target so that it'll become part of our normal release schedule of stage 
> tarballs
> -mike

I like the ideia :)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] i have an idea ! (erescue)

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 1014 bytes --]

On Sun, 2005-05-15 at 22:24 +0100, Carlos Silva wrote:
> > i dont think it'd be too hard to integrate this 'rescue set' into a catalyst 
> > target so that it'll become part of our normal release schedule of stage 
> > tarballs

Weren't we just talking about people proposing work for other projects
without consulting them... ;]

Anyway, catalyst 2.0, which is what the current HEAD version of catalyst
will become once it is released, has a "stage4" target, which is capable
of adding or subtracting packages from a stage3, or rebuilding them with
new USE flags.

Added with the recently added portage 2.0.51.22, which does proper USE
flag checking when using both --usepkg and --newuse, we can easily
create a rescue tarball.  I don't think that catalyst2 will be ready in
time for the next release, however, as we were not planning on rocking
the boat quite that much for the release.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] i have an idea ! (erescue)

[Krzysiek Pawlik]

Mike Frysinger wrote:
> [...]

Use `quickpkg` before dangerous updates/merges. If something brakes -
untar the package.

-- 
Krzysiek 'Nelchael' Pawlik     RLU #322999    krzysiek.pawlik@people.pl
gentoo base system - kernel 2.6.11-ck8                   GPG:0x7E226904
http://fatcat.ftj.agh.edu.pl/~nelchael/
Never forget: 2 + 2 = 5 for extremely large values of 2.
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] i have an idea ! (erescue)

[Donnie Berkholz]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Krzysiek Pawlik wrote:
> Use `quickpkg` before dangerous updates/merges. If something brakes -
> untar the package.

Doesn't work too well when tar's broken too. =)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFCh84IXVaO67S1rtsRAibiAKD6i0twE/06iCEqe+uiHa8ht3tUVQCgoYTo
A+8VR9nD42r+NlH3M4tVFUE=
=+Fva
-----END PGP SIGNATURE-----
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] i have an idea ! (erescue)

[Krzysiek Pawlik]

Donnie Berkholz wrote:
>> Use `quickpkg` before dangerous updates/merges. If something brakes -
>> untar the package.
> 
> Doesn't work too well when tar's broken too. =)

Use static tar na bzip2 ;) Seriously: I'm for erescue (or whatever name
will be chosen).

-- 
Krzysiek 'Nelchael' Pawlik     RLU #322999    krzysiek.pawlik@people.pl
gentoo base system - kernel 2.6.11-ck8                   GPG:0x7E226904
http://fatcat.ftj.agh.edu.pl/~nelchael/
According to my calculations the problem doesn't exist.
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] i have an idea ! (erescue)

[Ryan]

Heres the easiest way to do your "erescue"

dd if=/dev/hda of=backup.iso 

"Three days later: AAAHHH! I blew up the /usr dir!"

mkdir /backup
mount -o loop backup.iso /backup
cp -f -r -a /backup/usr /

Pretty simple no?

And there you have it.  Backup shouldnt really be the burdon of
developers.  This should be up to the user in my opinion.  If you break
something, then its your fault for not having a backup.  Of coarse this
is just ONE way to backup. There are a bazillion ways to do it.  The
choice is up to you.  Besides, if you are running the unstable branch
and you DONT have a backup? You deserve what you get for not having
backed up :P

Donnie Berkholz wrote:

> Krzysiek Pawlik wrote:
>
> >Use `quickpkg` before dangerous updates/merges. If something brakes -
> >untar the package.
>
>
> Doesn't work too well when tar's broken too. =)

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] i have an idea ! (erescue)

[Sami Samhuri]

[-- Attachment #1: Type: text/plain, Size: 408 bytes --]

* On Sun May-15-2005 at 04:48:04 PM -0600, Ryan said:
[...]
> something, then its your fault for not having a backup.  Of coarse this
> is just ONE way to backup. There are a bazillion ways to do it.  The
> choice is up to you.  Besides, if you are running the unstable branch

And if Gentoo provides yet another choice for one to use in recovery
then that sounds good to me. :)

-- 
Sami Samhuri

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] i have an idea ! (erescue)

[david stanek]

[-- Attachment #1: Type: text/plain, Size: 754 bytes --]

On Sun, May 15, 2005 at 03:32:40PM -0700, Donnie Berkholz wrote:
>
> Krzysiek Pawlik wrote:
> > Use `quickpkg` before dangerous updates/merges. If something brakes -
> > untar the package.
> 
> Doesn't work too well when tar's broken too. =)

How about statically linking a version of tar with portage. This way
tar cannot be broken. Add an option to emerge, --backup or something
similar, that will automatically run quickpkg. Then make a erescue
executable that can parse the command line to figure out which
package the user wants to be rescued from, and execute the new tar
command.

-- 
David Stanek
www.roninds.net

GPG keyID #6272EDAF on http://pgp.mit.edu
Key fingerprint = 8BAA 7E11 8856 E148 6833  655A 92E2 3E00 6272 EDAF

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] i have an idea ! (erescue)

[Mike Frysinger]

On Sunday 15 May 2005 07:41 pm, david stanek wrote:
> On Sun, May 15, 2005 at 03:32:40PM -0700, Donnie Berkholz wrote:
> > Krzysiek Pawlik wrote:
> > > Use `quickpkg` before dangerous updates/merges. If something brakes -
> > > untar the package.
> >
> > Doesn't work too well when tar's broken too. =)
>
> How about statically linking a version of tar with portage.

USE=static emerge tar

also, busybox has tar built into it

regardless, all these are side roads
-mike
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] i have an idea ! (erescue)

[John Myers]

[-- Attachment #1: Type: text/plain, Size: 268 bytes --]

On Sunday 15 May 2005 16:41, david stanek wrote:
> Add an option to emerge, --backup or something
> similar, that will automatically run quickpkg.

If you set FEATURES="buildpkg", portage automatically makes binary packages 
for you. No need to add new support.

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] i have an idea ! (erescue)

[David Stanek]

[-- Attachment #1: Type: text/plain, Size: 699 bytes --]

On Sun, May 15, 2005 at 06:12:13PM -0700, John Myers wrote:
> On Sunday 15 May 2005 16:41, david stanek wrote:
> > Add an option to emerge, --backup or something
> > similar, that will automatically run quickpkg.
> 
> If you set FEATURES="buildpkg", portage automatically makes binary packages 
> for you. No need to add new support.

That would build a binary package for the potentially broken
package. What it would need to do is build a binary package of the
existing merged package. So a user can recover from a botched
upgrade.

-- 
David Stanek
www.roninds.net

GPG keyID #6272EDAF on http://pgp.mit.edu
Key fingerprint = 8BAA 7E11 8856 E148 6833  655A 92E2 3E00 6272 EDAF

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] i have an idea ! (erescue)

[David Stanek]

[-- Attachment #1: Type: text/plain, Size: 1080 bytes --]

On Sun, May 15, 2005 at 09:56:54PM -0400, David Stanek wrote:
> On Sun, May 15, 2005 at 06:12:13PM -0700, John Myers wrote:
> > On Sunday 15 May 2005 16:41, david stanek wrote:
> > > Add an option to emerge, --backup or something
> > > similar, that will automatically run quickpkg.
> > 
> > If you set FEATURES="buildpkg", portage automatically makes binary packages 
> > for you. No need to add new support.
> 
> That would build a binary package for the potentially broken
> package. What it would need to do is build a binary package of the
> existing merged package. So a user can recover from a botched
> upgrade.

Actually a very simple safe-emerge script can be easily written. It
could first do an 'emerge --buildpkg' on the existing merged package,
if that package was already merged.

The other part would be to create a statically linked version of tar
that can be used in a recovery situation.

-- 
David Stanek
www.roninds.net

GPG keyID #6272EDAF on http://pgp.mit.edu
Key fingerprint = 8BAA 7E11 8856 E148 6833  655A 92E2 3E00 6272 EDAF

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-dev] Re: i have an idea ! (erescue)

[Duncan]

David Stanek posted <20050516015654.GA7890@goliath.hsd1.oh.comcast.net>,
excerpted below,  on Sun, 15 May 2005 21:56:54 -0400:

> On Sun, May 15, 2005 at 06:12:13PM -0700, John Myers wrote:
>> On Sunday 15 May 2005 16:41, david stanek wrote:
>> > Add an option to emerge, --backup or something similar, that will
>> > automatically run quickpkg.
>> 
>> If you set FEATURES="buildpkg", portage automatically makes binary
>> packages for you. No need to add new support.
> 
> That would build a binary package for the potentially broken package.
> What it would need to do is build a binary package of the existing
> merged package. So a user can recover from a botched upgrade.

I think Myers' point was that if "buildpkg" is set, the system soon builds
up a history of /multiple/ versions back of any particular package.  Thus,
if any new emerge or remerge borks the system (hmm, interesting term,
that, particularly at this moment, but then, this isn't supposed to be a
political list so enough of that, just a nod to the heritage...), it's
normally fairly easy to quickly recover to a fully working system by just
emerging the binpkg, going back as necessary to a known working version,
taking a step back if it was a remerge and you just borked the previously
working current version.

That of course brings me to /my/ point, that buildpkg is a very useful
thing to have, certainly with toolchain packages, even if the system is
space or otherwise limited enough that it's not a desirable option overall.

Thus, my suggestion.  Why not create a second feature, toolchain-buildpkg,
I'm calling it here for purposes of developing the suggestion, that's on
by default, as contrasted to the normal buildpkg being off by default. 
The Gentoo Handbook would then of course be modified to cover it and
mention why Gentoo recommends that it stay on.  Portage would then always
buildpkg anything rescue-critical, including portage itself, gcc,
binutils, coreutils, python, etc.  Don't forget glibc (which would of
course need put in place from a LiveCD, one's "emergency" copy of the root
partition, etc).

As mentioned here, tar and bzip2 would be special cases, because while
one can untar a package directly over the current filesystem if portage
(or python) itself breaks, without tar and bzip2, one is somewhat hosed
unless one uses the resources of a backup rootfs or liveCD.  Perhaps
building them static by default would be a good idea, as well as including
them in the toolchain-buildpkg list.

With such a feature, and with it on by default, another make.conf
parameter would then be useful as well.  Call it binver-depth, and set
binver-depth=3 in make.globals.  Then, when three versions of the binpkg
are reached, it would delete the oldest one as it created a new one, thus
leaving two presumably known working backups at all times, even if the
newest version it just created fails.  0 would special-case to unlimited,
of course, since we already have the feature as a toggle.  As a bonus, one
could then make the normal buildpkg feature follow binver-depth as well,
since the code would already be there, or create two separate binver-depth
vars so users can control toolchain binver-depth separately from normal
binver-depth, if desired.

I believe this solution has the merit of requiring the least new code,
since we already have the normal buildpkg feature in place and tested. 
The only new code would be that to control the toolchain package option
separate from the others, which should be easy to add, and the rotate
functionality (or make it a cron script similar to logrotate), which would
be nice but isn't an absolute "must".

Optionally, we could simply change the default of the buildpkg feature to
"on", along with appropriate coverage in the usual places (GWN, the forums
and user list, the Gentoo home page, and naturally, the Gentoo Handbook,
for new users).  That wouldn't require /any/ new portage code, altho
again, a binpkg-rotate cron script would be rather useful.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman in
http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html


-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: i have an idea ! (erescue)

[Jan Kundrát]

[-- Attachment #1: Type: text/plain, Size: 1226 bytes --]

Duncan wrote:
> Thus, my suggestion.  Why not create a second feature, toolchain-buildpkg,
> I'm calling it here for purposes of developing the suggestion, that's on
> by default, as contrasted to the normal buildpkg being off by default. 
> The Gentoo Handbook would then of course be modified to cover it and
> mention why Gentoo recommends that it stay on.  Portage would then always
> buildpkg anything rescue-critical, including portage itself, gcc,
> binutils, coreutils, python, etc.  Don't forget glibc (which would of
> course need put in place from a LiveCD, one's "emergency" copy of the root
> partition, etc).

What is a probability that user running *stable* branch would bork his
box by upgrades?

Aren't those running ~ARCH expected to make backups? Is it such a
problem to reboot their *testing* boxes from some CD and restore backups
from such environment? (I know about special cases like machines w/o
CD-ROM/netboot, but I think their owners are skilled enough to find
other solution which better suits their needs.)

Why bother with such a feature which won't be used by most of people,
instead of doing something "more useful"? Just my 2 cents, though.

-jkt

-- 
cd /local/pub && more beer > /dev/mouth

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-dev] Re: Re: i have an idea ! (erescue)

[Duncan]

Jan Kundrát posted <428881F2.9000004@flaska.net>, excerpted below,  on
Mon, 16 May 2005 13:20:18 +0200:

> What is a probability that user running *stable* branch would bork his
> box by upgrades?

Well, I suppose that depends on what sort of CFLAGS they run, and whether
anybody else has been hair-brained enough to decide to try them and then
filed a bug that got them filterflagged in the ebuild, yet, or if they
bypassed the filterflags stuff (as I occasionally do, but on ~arch and
with a working partition and three levels of backup on two separately
bootable disks). <g>

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman in
http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html


-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: Re: i have an idea ! (erescue)

[Jan Kundrát]

[-- Attachment #1: Type: text/plain, Size: 786 bytes --]

Duncan wrote:
> Jan Kundrát posted <428881F2.9000004@flaska.net>, excerpted below,  on
> Mon, 16 May 2005 13:20:18 +0200:
> 
> 
>>What is a probability that user running *stable* branch would bork his
>>box by upgrades?
> 
> 
> Well, I suppose that depends on what sort of CFLAGS they run, and whether
> anybody else has been hair-brained enough to decide to try them and then
> filed a bug that got them filterflagged in the ebuild, yet, or if they
> bypassed the filterflags stuff (as I occasionally do, but on ~arch and
> with a working partition and three levels of backup on two separately
> bootable disks). <g>
> 

Well, I meant wise users :-P. There is not so many ricers using stable
branch, IMHO...

-jkt

-- 
cd /local/pub && more beer > /dev/mouth


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Re: i have an idea ! (erescue)

[Thomas de Grenier de Latour]

On Mon, 16 May 2005 03:24:21 -0700
Duncan <1i5t5.duncan@cox.net> wrote:

> Thus, my suggestion.  Why not create a second feature,
> toolchain-buildpkg, I'm calling it here for purposes of
> developing the suggestion, that's on by default, as contrasted
> to the normal buildpkg being off by default. 

There exists a "buildsyspkg" flag already, which is supposed to
build packages for ebuilds that are in the system group. It's
probably a bit broken tho (won't catch system packages
dependencies that are not explicitly in your profile's system
group, and also i don't think the version in portage resolves
virtuals).

If you're using portage HEAD, you can play with this patch too:
http://tdegreni.free.fr/gentoo/portage-20050323--custom_buildpkg.patch
It deprecates "buildsyspkg", and instead introduces a new
variable for user-defined policy about building binaries. Using it
with BUILD_PKGS="*" in make.conf gives a behavior similar to
the buildpkg FEATURES flag (keep binaries of everything), and
other rules are available too. For instance, i use:
BUILD_PKGS="system !sys-kernel kde-base kde-misc \
  app-office/openoffice app-office/openoffice-ximian x11-libs/qt"
which makes it build binaries for system packages (but the ones
from sys-kernel), and also for kde and openoffice stuffs (because
they are slow to build so i don't want to revert them from
sources if they break). This patch should handle system virtuals
fine (although the code organisation is not really nice), but
still doesn't track system dependencies.

> With such a feature, and with it on by default, another
> make.conf parameter would then be useful as well.  Call it
> binver-depth, and set binver-depth=3 in make.globals.  Then,
> when three versions of the binpkg are reached, it would delete
> the oldest one as it created a new one, thus leaving two
> presumably known working backups at all times, even if the
> newest version it just created fails.

Better to keep that out of emerge imho, there exists some /usr/
portage/packages cleanup scripts already that you can add to your
crontab. An example is:
https://bugs.gentoo.org/attachment.cgi?id=53397 
(it doesn't have your "keep the latest 3 versions" policy tho,
but that could be added - I think i see how to do it, only taking
care of slotted packaged needs a bit of care). And there are
others similar scripts on f.g.o I think.

-- 
TGL.
-- 
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: Re: i have an idea ! (erescue)

[Duncan]

Thomas de Grenier de Latour posted <20050516133315.04696000@eusebe>,
excerpted below,  on Mon, 16 May 2005 13:33:15 +0200:

> There exists a "buildsyspkg" flag already, which is supposed to
> build packages for ebuilds that are in the system group. It's
> probably a bit broken tho (won't catch system packages
> dependencies that are not explicitly in your profile's system
> group, and also i don't think the version in portage resolves
> virtuals).

Wow!  "Before they call, I will answer, and while they are yet speaking, I
will hear." =8^)

Or, to put it another way, "Great minds think alike!"  <g>

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman in
http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html


-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] i have an idea ! (erescue)

[Olivier Crête]

[-- Attachment #1: Type: text/plain, Size: 952 bytes --]

On Sun, 2005-15-05 at 17:18 -0400, Mike Frysinger wrote:
> for example, when i broke binutils in unstable with a gcc4 patch, i noticed 
> that it's hard for users to *easily* recover from this ... we developers end 
> up scrambling to build a bunch of binary packages for a variety of compatible 
> compiler/libc combinations so the user can just wget the file and run `emerge 
> binutils.tbz2` and be on their way

I had exactly the same problem recently and I had a hard time finding a
fitting tbz2 since the package was part of the stages and not the GRP.

Just having packages built as part of a stage3 uploaded somewhere
as .tbz2s on the mirrors would already be a good first step and isn't
much more work for the release teams. 

Would erescue just be static "tar xjvf ${file} -C /" or do you want
something more intelligent that would also update the portage db ?


-- 
Olivier Crête
tester@gentoo.org
x86 Security Liaison

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] i have an idea ! (erescue)

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 879 bytes --]

On Sun, 2005-05-15 at 17:31 -0400, Olivier Crête wrote:
> I had exactly the same problem recently and I had a hard time finding a
> fitting tbz2 since the package was part of the stages and not the GRP.
> 
> Just having packages built as part of a stage3 uploaded somewhere
> as .tbz2s on the mirrors would already be a good first step and isn't
> much more work for the release teams.

...and you speak for the mirror admins and their space constraints,
correct?

This would add quite a bit of space to the mirrors.  The average stage3
tarball is about 100MB.  So you can assume that the packages would equal
about the same.  Multiply this by the number of releasing arches, and
possibly even subarches, and you have an additional 1GB just from
x86/ppc.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] i have an idea ! (erescue)

[Pete Ezzo]

On 5/16/05, Chris Gianelloni <wolf31o2@gentoo.org> wrote:
> This would add quite a bit of space to the mirrors.  The average stage3
> tarball is about 100MB.  So you can assume that the packages would equal
> about the same.  Multiply this by the number of releasing arches, and
> possibly even subarches, and you have an additional 1GB just from
> x86/ppc.

i don't believe this was meant to be a full replace anything that
breaks kind of system here.  just a simple rescue program that would
get you very basic stuff, like portage, python, the toolchain.  for
any arch they could be optimized to the lowest common denominator,
like i386... and then once they are installed and the user's system
works again, they use the again working portage or compiler to rebuild
the exact version they want.  it isn't a huge substantial hit, and the
rescue packages don't have to be the latest version in question, they
can be updated once a release, or once a year, leaving a fairly small
footprint.

i also like the idea of it being wholly seperate from portage, which
neatly eliminates several points of failure, and keeps the portage
code a bit easier to control.

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] i have an idea ! (erescue)

[Olivier Crête]

[-- Attachment #1: Type: text/plain, Size: 1440 bytes --]

On Mon, 2005-16-05 at 09:57 -0400, Chris Gianelloni wrote:
> This would add quite a bit of space to the mirrors.  The average stage3
> tarball is about 100MB.  So you can assume that the packages would equal
> about the same.  Multiply this by the number of releasing arches, and
> possibly even subarches, and you have an additional 1GB just from
> x86/ppc.

I would tend to believe that the content of a stage1 or stage2 would be
enough and just for the majors architectures (those that have a
stage1).. Anyways people will rebuild said packages once that's done,
right? That's not much for the mirrors.. Around 168 megs for the content
of a stage1 or 300 megs for the stage2.. 

tester@mirror releases $ find . -name stage1-*-2005.0.tar.bz2 |xargs du -c
21084   ./amd64/2005.0/stages/stage1-amd64-2005.0.tar.bz2
17292   ./sparc/2005.0/sparc32/stages/stage1-sparc-2005.0.tar.bz2
20520   ./sparc/2005.0/sparc64/stages/stage1-sparc64-2005.0.tar.bz2
18012   ./x86/2005.0/stages/hardened/2.4/stage1-x86-hardened-2.4-2005.0.tar.bz2
18252   ./x86/2005.0/stages/hardened/2.6/stage1-x86-hardened-2.6-2005.0.tar.bz2
16424   ./x86/2005.0/stages/x86/stage1-x86-2005.0.tar.bz2
18720   ./ppc64/2005.0/stages/stage1-ppc64-2005.0.tar.bz2
18756   ./alpha/2005.0/stages/stage1-alpha-2005.0.tar.bz2
19468   ./ia64/2005.0/stages/stage1-ia64-2005.0.tar.bz2
168528  total

-- 
Olivier Crête
tester@gentoo.org
x86 Security Liaison

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] i have an idea ! (erescue)

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 1162 bytes --]

On Mon, 2005-05-16 at 20:48 -0400, Olivier Crête wrote:
> I would tend to believe that the content of a stage1 or stage2 would be
> enough and just for the majors architectures (those that have a
> stage1).. Anyways people will rebuild said packages once that's done,
> right? That's not much for the mirrors.. Around 168 megs for the content
> of a stage1 or 300 megs for the stage2..

Honestly, a stage1 is not a good starting point.  In fact, there should
*never* be anything provided that is directly laid onto the live
filesystem.

A much better approach would be for there to be a rescue build,
completely independent of the stages, since it doesn't need to mirror
them in any way.  It should be extracted (self-extracted?) to something
like /rescue and executed from there, being completely self-contained.
This keeps it from stomping on system files and breaking
collision-protect or doing anything else nasty like hosing configuration
files (ever made the mistake of extracting a stage onto a live
filesystem?) when unpacked.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] i have an idea ! (erescue)

[Donnie Berkholz]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Chris Gianelloni wrote:
> A much better approach would be for there to be a rescue build,
> completely independent of the stages, since it doesn't need to mirror
> them in any way.  It should be extracted (self-extracted?) to something
> like /rescue and executed from there, being completely self-contained.
> This keeps it from stomping on system files and breaking
> collision-protect or doing anything else nasty like hosing configuration
> files (ever made the mistake of extracting a stage onto a live
> filesystem?) when unpacked.

This sounds a lot like saying, use an initrd, but when you pivot roots
to the live filesystem, leave it mounted somewhere.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFCiirqXVaO67S1rtsRAgHyAJ49adhPpYhwHqUeFNMw4I6h+GUyDwCfbwMB
rgA0GKbYsBjd8K9I7X2pyE8=
=tZSe
-----END PGP SIGNATURE-----
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] i have an idea ! (erescue)

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 1267 bytes --]

On Tue, 2005-05-17 at 10:33 -0700, Donnie Berkholz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Chris Gianelloni wrote:
> > A much better approach would be for there to be a rescue build,
> > completely independent of the stages, since it doesn't need to mirror
> > them in any way.  It should be extracted (self-extracted?) to something
> > like /rescue and executed from there, being completely self-contained.
> > This keeps it from stomping on system files and breaking
> > collision-protect or doing anything else nasty like hosing configuration
> > files (ever made the mistake of extracting a stage onto a live
> > filesystem?) when unpacked.
> 
> This sounds a lot like saying, use an initrd, but when you pivot roots
> to the live filesystem, leave it mounted somewhere.

Kinda... This wouldn't require a reboot, though.  When the user is done,
they simply rm -rf /rescue and their system is clean again.

> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.1 (GNU/Linux)
> 
> iD8DBQFCiirqXVaO67S1rtsRAgHyAJ49adhPpYhwHqUeFNMw4I6h+GUyDwCfbwMB
> rgA0GKbYsBjd8K9I7X2pyE8=
> =tZSe
> -----END PGP SIGNATURE-----
-- 
Chris Gianelloni
Release Engineering - Strategic Lead/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] i have an idea ! (erescue) ro-overlays

[Jim Northrup]

I had smashing success migratingraid volumes to a new motherboard by 
building a readonly loopback boot-cd rootfs volume, and using

cp -sr /mnt/rescue /mnt/newroot

before building stage2,3;  with minor /etc grumbling, the system 
bootstrapped flawlessly while still borrowing a few sensitive static 
utils where combinations of kernel, gcc and libc could otherwise wreak havoc

I'm very happy with new GUID-based volume mounting and more stable raid 
tools, but a CF-based or initrd root available when /lib goes to hell is 
an absolute must for supporting fault tolerance.


Chris Gianelloni wrote:

>On Tue, 2005-05-17 at 10:33 -0700, Donnie Berkholz wrote:
>  
>
>>-----BEGIN PGP SIGNED MESSAGE-----
>>Hash: SHA1
>>
>>Chris Gianelloni wrote:
>>    
>>
>>>A much better approach would be for there to be a rescue build,
>>>completely independent of the stages, since it doesn't need to mirror
>>>them in any way.  It should be extracted (self-extracted?) to something
>>>like /rescue and executed from there, being completely self-contained.
>>>This keeps it from stomping on system files and breaking
>>>collision-protect or doing anything else nasty like hosing configuration
>>>files (ever made the mistake of extracting a stage onto a live
>>>filesystem?) when unpacked.
>>>      
>>>
>>This sounds a lot like saying, use an initrd, but when you pivot roots
>>to the live filesystem, leave it mounted somewhere.
>>    
>>
>
>Kinda... This wouldn't require a reboot, though.  When the user is done,
>they simply rm -rf /rescue and their system is clean again.
>  
>
   
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] i have an idea ! (erescue) ro-overlays

[Mike Frysinger]

On Tuesday 24 May 2005 05:11 pm, Jim Northrup wrote:
> but a CF-based or initrd root available when /lib goes to hell is
> an absolute must for supporting fault tolerance.

do you mean like the disk underneath /lib is blown to crap or a bad glibc is 
merged ?

if the latter, then the new busybox can help ... just boot with init=/bin/bb 
and you should have plenty of tools to recover with
-mike
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] i have an idea ! (erescue) ro-overlays

[Jim Northrup]

Mike Frysinger wrote:

>On Tuesday 24 May 2005 05:11 pm, Jim Northrup wrote:
>  
>
>>but a CF-based or initrd root available when /lib goes to hell is
>>an absolute must for supporting fault tolerance.
>>    
>>
>
>do you mean like the disk underneath /lib is blown to crap or a bad glibc is 
>merged ?
>
>if the latter, then the new busybox can help ... just boot with init=/bin/bb 
>and you should have plenty of tools to recover with
>-mike
>  
>
great advice.

I've learned whenever md or devfs rework occurs, or the **/sbin tools 
get rev'd, its foolish to proceed without a rescue nearby.

of course bb is a space-saver, but i find myself turning the room upside 
down for full-static versions of tar, nc and fileutils

Jim
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] i have an idea ! (erescue) ro-overlays

[Mike Frysinger]

On Tuesday 24 May 2005 06:34 pm, Jim Northrup wrote:
> of course bb is a space-saver, but i find myself turning the room upside
> down for full-static versions of tar, nc and fileutils

bb is static and it supports tar, nc, and many fileutils ;)
-mike
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] i have an idea ! (erescue) ro-overlays

[Stuart Longland]

[-- Attachment #1: Type: text/plain, Size: 814 bytes --]

Mike Frysinger wrote:
> On Tuesday 24 May 2005 06:34 pm, Jim Northrup wrote:
> 
>>of course bb is a space-saver, but i find myself turning the room upside
>>down for full-static versions of tar, nc and fileutils
> 
> 
> bb is static and it supports tar, nc, and many fileutils ;)
> -mike

If only a little crippled in functionality compared to their full
equivalents... :-)

Although for what BusyBox is... it's a great tool.
-- 
+-------------------------------------------------------------+
| Stuart Longland -oOo- http://stuartl.longlandclan.hopto.org |
| Atomic Linux Project     -oOo-    http://atomicl.berlios.de |
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - |
| I haven't lost my mind - it's backed up on a tape somewhere |
+-------------------------------------------------------------+

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: [gentoo-dev] i have an idea ! (erescue) ro-overlays

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 673 bytes --]

On Tue, 2005-05-24 at 14:11 -0700, Jim Northrup wrote:
> I'm very happy with new GUID-based volume mounting and more stable raid 
> tools, but a CF-based or initrd root available when /lib goes to hell is 
> an absolute must for supporting fault tolerance.

If you use genkernel to build your kernel, then you will have a usable
initrd with lvm2/evms/dmraid (via --lvm2, --evms, or --dmraid)
capabilities and tools for rescuing your system.

This is only good for filesystem rescue, though.  It won't help you if
you emerge a bad copy of binutils or gcc.

-- 
Chris Gianelloni
Release Engineering - Strategic Lead/QA Manager
Games - Developer
Gentoo Linux

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] i have an idea ! (erescue) ro-overlays

[James Northrup]

overall I'm quite pleased with genkernel and have relegated much  
tedium to its functions over time.  perhaps it's a worthy mule for  
more responsibility.

I have mirror volumes which have survived almost 8 years with 2nd and  
third generation drives, motherboard, and architecture (32->64 bit).

in those years, the newer revs which don't jump up and bite me in the  
ass probably go unnoticed..

abstractly speaking, the clearest working example of what breaks is  
oft-times a recent kernel on a recent install disk.  slopping an  
install disk on a modern hard-drive consumes but a gnat's real-estate.

using a symlink foundation does a pretty good job of allowing emerge  
to over-write the static known-good binaries with dynamics, but  
occasionally the gcc and/or libc is a repeatable failure and having  
the ro overlay handy allows wholesale excision of the broken  
installs, esp on young architectures.

for whatever reasons, I pack my cd-rom drive bays with hard drives,  
and install with a cd-rom hanging off the side of the case tethered  
by its cables... right about the time its a bootable system the cd- 
rom comes off and the box is tucked into some crawl-space or other  
behind desks, shelves, etc. and hopefully forgotten.



On May 25, 2005, at 6:32 AM, Chris Gianelloni wrote:

> On Tue, 2005-05-24 at 14:11 -0700, Jim Northrup wrote:
>
>> I'm very happy with new GUID-based volume mounting and more stable  
>> raid
>> tools, but a CF-based or initrd root available when /lib goes to  
>> hell is
>> an absolute must for supporting fault tolerance.
>>
>
> If you use genkernel to build your kernel, then you will have a usable
> initrd with lvm2/evms/dmraid (via --lvm2, --evms, or --dmraid)
> capabilities and tools for rescuing your system.
>
> This is only good for filesystem rescue, though.  It won't help you if
> you emerge a bad copy of binutils or gcc.
>
> -- 
> Chris Gianelloni
> Release Engineering - Strategic Lead/QA Manager
> Games - Developer
> Gentoo Linux
>

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] i have an idea ! (erescue)

[Sami Samhuri]

[-- Attachment #1: Type: text/plain, Size: 615 bytes --]

* On Sun May-15-2005 at 05:18:06 PM -0400, Mike Frysinger said:
[...]
> my proposal is to implement a new utility (called 'erescue' for lack of a 
> better name) that is written in C and designed to be statically linked ... 
> then next time you break a core system package which cannot be recovered by 
> simply running `emerge` a few times, you run `erescue <broken package>`

Everyone who is saying that Portage can already sort of handle this
seems to be missing one important point. If Python is broken then emerge
won't work. The proposed erescue would still work in that case.

-- 
Sami Samhuri

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] i have an idea ! (erescue)

[David Stanek]

[-- Attachment #1: Type: text/plain, Size: 960 bytes --]

On Sun, May 15, 2005 at 09:07:15PM -0700, Sami Samhuri wrote:
> * On Sun May-15-2005 at 05:18:06 PM -0400, Mike Frysinger said:
> [...]
> > my proposal is to implement a new utility (called 'erescue' for lack of a 
> > better name) that is written in C and designed to be statically linked ... 
> > then next time you break a core system package which cannot be recovered by 
> > simply running `emerge` a few times, you run `erescue <broken package>`
> 
> Everyone who is saying that Portage can already sort of handle this
> seems to be missing one important point. If Python is broken then emerge
> won't work. The proposed erescue would still work in that case.

If erescue is a statically built binary that basically untars a
backed up copy of a package, why would it depend on Python?


-- 
David Stanek
www.roninds.net

GPG keyID #6272EDAF on http://pgp.mit.edu
Key fingerprint = 8BAA 7E11 8856 E148 6833  655A 92E2 3E00 6272 EDAF

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] i have an idea ! (erescue)

[Colin Kingsley]

On 5/16/05, David Stanek <dstanek@dstanek.com> wrote:
> If erescue is a statically built binary that basically untars a
> backed up copy of a package, why would it depend on Python?

It won't. Thats the whole point.

Colin

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] i have an idea ! (erescue)

[Chris Bainbridge]

I've been in this position more than once, and had to go through the
bootcd+binaries (thanks to http://dev.gentoo.org/~avenj/bins/)
restore. Argh. I've often thought that atomic updates and rollback
within portage would be useful - maybe it could just be done as a
layer over subversion for Gentoo updated binary packages. Or maybe
rebuilding from source is preferable. Anyway, it would be very useful
to be able to revert to a known good state with a single command.

-- 
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: i have an idea ! (erescue)

[R Hill]

Mike Frysinger wrote:
> my proposal is to implement a new utility (called 'erescue' for lack of a 
> better name) that is written in C and designed to be statically linked ... 
> then next time you break a core system package which cannot be recovered by 
> simply running `emerge` a few times, you run `erescue <broken package>`

that'd be cool. i found myself wishing for something like this after 
taking up breaking glibc as a hobby.  it would have saved me a lot of 
chroot building.

i wonder if i could suggest a small repository of default configuration 
files to go along with this idea?  that's another thing i've found 
myself looking for from time to time.  i know there's other ways to get 
the original config for whatever, but they're inconsistent and sometimes 
difficult.

--de

-- 
gentoo-dev@gentoo.org mailing list