[gentoo-amd64] Anybody else having problems with audio/vidio apps and glibc-2.5?

[gentoo-amd64] Anybody else having problems with audio/vidio apps and glibc-2.5?

[Duncan]

I've been having all /sorts/ of problems with formerly stable audio and
video apps crashing recently.  The pattern is a crash at launch most (but
not all) of the time, often with some memory error.  However, if it
/does/ start and works more than a few minutes, it's fully stable and
will play for hours without issue.  xmms, kaffeine, amarok, all affected.

I didn't notice it until the upgrade to kde-3.5.5, which was my first big
set of apps built using the experimental CFLAG -ftree-vectorize as
discussed here a month or so ago, so I thought it was KDE. However, after
recompiling a bunch of stuff several different times/ways, nothing seemed
to be working.

Then I chanced across some ongoing discussion about nptl/linux-threads in
glibc-2.5 and forward on the dev list, while I was taking a break from
troubleshooting, and the thought occurred to me that glibc had been
upgraded at about the same time.

VWALLA!  I try to downgrade to glibc-2.4-r4, and get hit with its sanity
downgrade blocker.  It won't let me do it.  So a quick reboot to my backup
image (still on glibc-2.4-r3) and a quick ROOT=<backup> (which is main
working, since I'm no /on/ backup) export later, I'm emerging glibc-2.4-r4
(which I have binpkged, thanks to FEATURES=buildpkg) over top of what I'm
now convinced is a bad glibc-2.5.

Sure enough, reboot back to my main/working image again, now with
glibc-2.4-r4 once again, and **NO MORE CRASHES!!**

So... kde-3.5.5 with -ftree-vectorize is back in the clear.  The problem
is either glibc-2.5 itself, or -ftree-vectorize with it.  I haven't
figured out which yet, but I thought I'd post this both as a heads-up to
others and a question to see if anyone else has run into similar issues. 
I'll probably followup after I figure out which of those is the culprit or
if it's the combination.

Meanwhile a potentially useful trick to keep up your sleeve, just in case
you ever find yourself needing to downgrade glibc but the glibc ebuild
failing to let you do so.  Reboot to your emergency image, be that a
LiveCD or a backup set of partitions on your hard drive, mount your normal
working filesystem image, set ROOT= to point portage at the normal system
(not the backup), and /then/ do your glibc downgrade.  Then boot back to
your regular system and hope the downgrade works, as it did here. =8^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Anybody else having problems with audio/vidio apps and glibc-2.5?

[Hemmann, Volker Armin]

On Monday 23 October 2006 19:22, Duncan wrote:
> I've been having all /sorts/ of problems with formerly stable audio and
> video apps crashing recently.  The pattern is a crash at launch most (but
> not all) of the time, often with some memory error.  However, if it
> /does/ start and works more than a few minutes, it's fully stable and
> will play for hours without issue.  xmms, kaffeine, amarok, all affected.

nope

>
> I didn't notice it until the upgrade to kde-3.5.5, which was my first big
> set of apps built using the experimental CFLAG -ftree-vectorize as
> discussed here a month or so ago, so I thought it was KDE. However, after
> recompiling a bunch of stuff several different times/ways, nothing seemed
> to be working.
>

which I am using for some time now. I get konqueror crashes once in a while. 
Nothing else. (I can make crash konqueror, when I move the mouse over a mpeg 
and wait for the popup - bang, segfault).

> Then I chanced across some ongoing discussion about nptl/linux-threads in
> glibc-2.5 and forward on the dev list, while I was taking a break from
> troubleshooting, and the thought occurred to me that glibc had been
> upgraded at about the same time.
>
> VWALLA!  I try to downgrade to glibc-2.4-r4, and get hit with its sanity
> downgrade blocker.  It won't let me do it.  So a quick reboot to my backup
> image (still on glibc-2.4-r3) and a quick ROOT=<backup> (which is main
> working, since I'm no /on/ backup) export later, I'm emerging glibc-2.4-r4
> (which I have binpkged, thanks to FEATURES=buildpkg) over top of what I'm
> now convinced is a bad glibc-2.5.

DO NEVER downgrade glibc! I have been there, bitten by it very, very hard. A 
nonbooting system, because not even udev runs, is a big problem (I solved it, 
but it cost me time, sweat and tears).


> Meanwhile a potentially useful trick to keep up your sleeve, just in case
> you ever find yourself needing to downgrade glibc but the glibc ebuild
> failing to let you do so.  Reboot to your emergency image, be that a
> LiveCD or a backup set of partitions on your hard drive, mount your normal
> working filesystem image, set ROOT= to point portage at the normal system
> (not the backup), and /then/ do your glibc downgrade.  Then boot back to
> your regular system and hope the downgrade works, as it did here. =8^)
>

and rebuilt EVERYTHING that was built against the new glibc! Every single app, 
every lib, everything. You miss something and you will suffer.
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Anybody else having problems with audio/vidio apps and glibc-2.5?

[Jan Jitse Venselaar]

[-- Attachment #1: Type: text/plain, Size: 872 bytes --]

On Monday 23 October 2006 19:22, Duncan wrote:
> So... kde-3.5.5 with -ftree-vectorize is back in the clear.  The problem
> is either glibc-2.5 itself, or -ftree-vectorize with it.  I haven't
> figured out which yet, but I thought I'd post this both as a heads-up to
> others and a question to see if anyone else has run into similar issues.
> I'll probably followup after I figure out which of those is the culprit or
> if it's the combination.
You patched your glibc ebuild to not strip -ftree-vectorize ?
If so, I'd say that means trouble. I had some trouble with segfaulting apps 
when I tried a rebuild with this flag. I did not rebuild glibc with this flag 
however. I'd say -ftree-vectorize with glibc means big trouble.

FWIW, I have glibc-2.5, everything built with gcc-4.1.1 but nothing fancy in 
the cflags, and everything runs fine.

Jan Jitse

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-amd64] Re: Anybody else having problems with audio/vidio apps and glibc-2.5?

[Duncan]

"Hemmann, Volker Armin" <volker.armin.hemmann@tu-clausthal.de> posted
200610232110.58523.volker.armin.hemmann@tu-clausthal.de, excerpted below,
on  Mon, 23 Oct 2006 21:10:58 +0200:

> DO NEVER downgrade glibc! I have been there, bitten by it very, very
> hard. A nonbooting system, because not even udev runs, is a big problem
> (I solved it, but it cost me time, sweat and tears).

That's what a backup image, a snapshot of the system taken periodically
when everything is known to be working, is useful for. =8^)

As I said, I had to boot to the backup to allow me to do the downgrade
to my main system using ROOT=, but it worked.

> and rebuilt EVERYTHING that was built against the new glibc! Every
> single app, every lib, everything. You miss something and you will
> suffer.

Actually, I was expecting issues, but haven't had any so far but one,
easily fixed.  As you mentioned, the latest kernel didn't want to boot, so
there again I backed up a couple notches and got a working one, but
everything else I had merged since then, notably including all of KDE
3.5.5, merged against glibc-2.5, seems to be working great against
glibc-2.4-r4, which as it happens was merged only a couple weeks earlier
-- actually that's probably my saving grace or I'd likely have had more
issues.

As it is, booting 2.6.18 with glibc-2.4-r4 works great, better than
2.6.19-rc2 with glibc-2.5 actually, due to the crashes that started the
whole thing on the latter combo.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-amd64@gentoo.org mailing list

[gentoo-amd64] Re: Anybody else having problems with audio/vidio apps and glibc-2.5?

[Duncan]

Jan Jitse Venselaar <janjitse@gmail.com> posted
200610232131.04118.janjitse@gmail.com, excerpted below, on  Mon, 23 Oct
2006 21:30:58 +0200:

> You patched your glibc ebuild to not strip -ftree-vectorize ?
> If so, I'd say that means trouble. I had some trouble with segfaulting apps 
> when I tried a rebuild with this flag. I did not rebuild glibc with this flag 
> however. I'd say -ftree-vectorize with glibc means big trouble.
> 
> FWIW, I have glibc-2.5, everything built with gcc-4.1.1 but nothing fancy in 
> the cflags, and everything runs fine.

No, I didn't patch the ebuild.  That's one of the things I hadn't double
checked yet when I posted, but indeed, it seems that -ftree-vectorize
(and pretty much everything else for that matter) is stripped by the glibc
ebuild. Thus, it's almost certainly glibc-2.5 itself, not any CFLAGS, that
caused my pain, tho it very well could be glibc-2.5 in /combination/ with
other software compiled with certain CFLAGS.  I'll have to recompile
glibc-2.5 again and try it again to be sure, but that's my working
hypothesis at this point.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: Anybody else having problems with audio/vidio apps and glibc-2.5?

[Hemmann, Volker Armin]

On Tuesday 24 October 2006 00:12, Duncan wrote:

>
> > and rebuilt EVERYTHING that was built against the new glibc! Every
> > single app, every lib, everything. You miss something and you will
> > suffer.
>
> Actually, I was expecting issues, but haven't had any so far but one,
> easily fixed.  As you mentioned, the latest kernel didn't want to boot, so
> there again I backed up a couple notches and got a working one, but
> everything else I had merged since then, notably including all of KDE
> 3.5.5, merged against glibc-2.5, seems to be working great against
> glibc-2.4-r4, which as it happens was merged only a couple weeks earlier
> -- actually that's probably my saving grace or I'd likely have had more
> issues.

I had used some pre 2.5 glibcs. Updating to 2.5 did not work, because of 
gcc-config problems - but one of the devs said, I should remove 
overlay-software.. and come back after that. The onl overlay software was the 
glibc.. so I downgraded - and everything fell apart.

As I said, no boot, because udev did not work. Half of the apps did not work. 
Lots and lots of problems. I got it booting and halfway working, than read, 
that it may be gcc-config related. It was.. so I did not had to rebuilt 
everything, just emerged the 2.5 glibc.

But I will never ever again downgrade a glibc...
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: Anybody else having problems with audio/vidio apps and glibc-2.5?

[Jamie]

<snip>
>
> That's what a backup image, a snapshot of the system taken periodically
> when everything is known to be working, is useful for. =8^)
>
<snip>

A good piece of advice and one that I really should follow.
What is the best way to take an image of the Gentoo install?
In my case my Gentoo install resides on /dev/hda2 (boot) ; /dev/hda3
(swap) and /dev/hda5 (root) - is it possible to use something like dd to
image these partitions to a disk somewhere else on my network? Is this a
good/bad/non-workable idea?

I would welcome any constructive advise on this as the many hours of
building a Gentoo box is not something I want to do too often so being
able to image prior to any major upgrades would be a great safety net...

Cheers

Jamie
(back to Gentoo after a short period with Ubuntu!)

-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: Anybody else having problems with audio/vidio apps and glibc-2.5?

[Hemmann, Volker Armin]

On Tuesday 24 October 2006 01:48, Jamie wrote:
> <snip>
>
> > That's what a backup image, a snapshot of the system taken periodically
> > when everything is known to be working, is useful for. =8^)
>
> <snip>
>
> A good piece of advice and one that I really should follow.
> What is the best way to take an image of the Gentoo install?
> In my case my Gentoo install resides on /dev/hda2 (boot) ; /dev/hda3
> (swap) and /dev/hda5 (root) - is it possible to use something like dd to
> image these partitions to a disk somewhere else on my network? Is this a
> good/bad/non-workable idea?

get a tape drive (dlt is great), tar you system onto the tape drive. 

If something happens, boot a livecd, and untar from the tape into the 
partitions....
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: Anybody else having problems with audio/vidio apps and glibc-2.5?

[Jamie]

> On Tuesday 24 October 2006 01:48, Jamie wrote:
>> <snip>
>>
>> > That's what a backup image, a snapshot of the system taken
>> periodically
>> > when everything is known to be working, is useful for. =8^)
>>
>> <snip>
>>
>> A good piece of advice and one that I really should follow.
>> What is the best way to take an image of the Gentoo install?
>> In my case my Gentoo install resides on /dev/hda2 (boot) ; /dev/hda3
>> (swap) and /dev/hda5 (root) - is it possible to use something like dd to
>> image these partitions to a disk somewhere else on my network? Is this a
>> good/bad/non-workable idea?
>
> get a tape drive (dlt is great), tar you system onto the tape drive.
>
> If something happens, boot a livecd, and untar from the tape into the
> partitions....
> --
> gentoo-amd64@gentoo.org mailing list

A great idea, but a DLT tape drive is well over $1000 here in New Zealand,
and every tape drive I have seen is far more expensive than disk.
My ideal solution would be to somehow back up to a drive on another
machine on the network (assuming that is possible).

-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: Anybody else having problems with audio/vidio apps and glibc-2.5?

[Joe Menola]

On Monday 23 October 2006 8:26 pm, Jamie wrote:
> A great idea, but a DLT tape drive is well over $1000 here in New Zealand,
> and every tape drive I have seen is far more expensive than disk.
> My ideal solution would be to somehow back up to a drive on another
> machine on the network (assuming that is possible).

Patimage works well, if you don't mind unmounting your partitions. 
Tar is also a cheap alternative. The folloing example commands will tar your 
root partition and exclude any mounts embedded within:

mkdir /mnt/tmp
mount -o bind / /mnt/tmp
cd /mnt/tmp
tar  -cvvpjf <wherever>/backup.tbz2 .

-jm
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: Anybody else having problems with audio/vidio apps and glibc-2.5?

[Richard Fish]

On 10/23/06, Jamie <gentoo@ihug.co.nz> wrote:
> A great idea, but a DLT tape drive is well over $1000 here in New Zealand,
> and every tape drive I have seen is far more expensive than disk.
> My ideal solution would be to somehow back up to a drive on another
> machine on the network (assuming that is possible).

I use dar and USB2 disks for my backups.  Faster, cheaper, and with
more capacity than most tape drives.

-Richard
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: Anybody else having problems with audio/vidio apps and glibc-2.5?

[Nuitari]

> A great idea, but a DLT tape drive is well over $1000 here in New Zealand,
> and every tape drive I have seen is far more expensive than disk.
> My ideal solution would be to somehow back up to a drive on another
> machine on the network (assuming that is possible).

If you only need backups at specific times, the easiest is to set up a 
Mirror (raid1) RAID and split the raid when the backup is needed.

That way if the updated system is non working you can easily switch the 
hard drives and use the backup and readd the non working one back to the 
raid.


-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: Anybody else having problems with audio/vidio apps and glibc-2.5?

[Hemmann, Volker Armin]

> >
> > get a tape drive (dlt is great), tar you system onto the tape drive.
> >
> > If something happens, boot a livecd, and untar from the tape into the
> > partitions....

> A great idea, but a DLT tape drive is well over $1000 here in New Zealand,
> and every tape drive I have seen is far more expensive than disk.

that is, what ebay is great for ;)

also, tape drives are way more reliable than harddisks. I don't trust them 
anymore - got bitten too much times.

Hm, that said, it is time for a backup...
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: Anybody else having problems with audio/vidio apps and glibc-2.5?

[Simon Stelling]

Jamie wrote:
> I would welcome any constructive advise on this as the many hours of
> building a Gentoo box is not something I want to do too often so being
> able to image prior to any major upgrades would be a great safety net...

Here's my low cost backup method: rsync -aux

1. Get two drives which are both capable of holding all your data, 
ideally two equally large ones.
2. Partition them the same way
3. Whenever you feel like backing up your data on eg. sda1, mount sdb1 
and rsync -aux <mountpoint>. The -x in the rsync command makes sure it 
doesn't try to copy /dev or /proc. -u speeds up the things a bit if 
you're impatient.
4. Now your harddrive broke. Buy a new one. (If you bought a Maxtor it 
is likely that it breaks before guarantee runs out so you get one for 
free!) Meanwhile, swap the disks and continue to use your system.

I've been doing this for the past two years or so. It may not be the 
perfect backup system, but it is damn cheap and easy in maintenance. 
Plus it has the advantage that you can continue using your system 
immediately, when you would have to wait for your vendor to ship a new 
disk with every other method.

-- 
Kind Regards,

Simon Stelling
Gentoo/AMD64 developer
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: Anybody else having problems with audio/vidio apps and glibc-2.5?

[Peter Humphrey]

On Tuesday 24 October 2006 01:58, Richard Fish wrote:

> I use dar and USB2 disks for my backups.  Faster, cheaper, and with
> more capacity than most tape drives.

I hadn't spotted dar before. Can you explain the use of the dar32 and dar64 
USE flags? The use.desc description doesn't help much.

-- 
Rgds
Peter
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: Anybody else having problems with audio/vidio apps and glibc-2.5?

[Andrei Slavoiu]

--- Duncan <1i5t5.duncan@cox.net> wrote:
> As you mentioned, the latest kernel
> didn't want to boot, so
> there again I backed up a couple notches and got a
> working one, but
> everything else I had merged since then, notably
> including all of KDE
> 3.5.5, merged against glibc-2.5, seems to be working
> great against
> glibc-2.4-r4, which as it happens was merged only a
> couple weeks earlier
> -- actually that's probably my saving grace or I'd
> likely have had more
> issues.
Probably the explanation that KDE did not break is
that it does not directly link to glibc. C++ programs
usually only link to libstdc++ which in turn is linked
to glibc, so you would only have problems with C++
applications if you recompile gcc with the new glibc
and then downgrade.

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Anybody else having problems with audio/vidio apps and glibc-2.5?

[Piotr Pruszczak]

> 
> FWIW, I have glibc-2.5, everything built with gcc-4.1.1 but nothing fancy in 
> the cflags, and everything runs fine.
> 
> Jan Jitse

I can confirm, with new glibc 2.5 my audio apps work fine (even thouse 
which didn't worked stable before..)

-- 
Piotr
-- 
gentoo-amd64@gentoo.org mailing list

[gentoo-amd64] Backup techniques Was: audio/vidio apps and glibc-2.5?

[Duncan]

"Jamie" <gentoo@ihug.co.nz> posted
4049.210.55.22.193.1161647305.squirrel@drgnfire.is-a-geek.com, excerpted
below, on  Tue, 24 Oct 2006 12:48:25 +1300:

> A good piece of advice and one that I really should follow. What is the
> best way to take an image of the Gentoo install? In my case my Gentoo
> install resides on /dev/hda2 (boot) ; /dev/hda3 (swap) and /dev/hda5
> (root) - is it possible to use something like dd to image these partitions
> to a disk somewhere else on my network? Is this a good/bad/non-workable
> idea?
> 
> I would welcome any constructive advise on this as the many hours of
> building a Gentoo box is not something I want to do too often so being
> able to image prior to any major upgrades would be a great safety net...

There are of course several different alternatives.  I've found what works
for me, and it has been decently tested in recovery situations.

#1 rule for me was that it needed to be relatively convenient, or I'd not
use it regularly.

#2, as you, hard disks seem to be the most cost effective here for nearly
whole-system backup, and are reasonably reliable if you treat them right. 
I do backup choice bits multiple times, often multiple disk backup
entries, but also burned to CD/DVD, particularly where the data isn't
likely to be changing over long periods.

#3 works as a result of 1 and 2:  I'm able to boot directly into my backup
system and resume from there using it as my main system.  Something like
tape backup simply doesn't work that way.  Neither do emergency boot
LiveCDs and the like, tho they have their place as a "worst case situation"
solution, if my main backups fail.

#4, my system scales.  I used it as multiple partitions on a single hard
drive and/or a second hard drive for years, including recovering from two
partial hard drive failures.  The last one was caused by overheating after
my A/C went out. (Phoenix temps can reach 50C/122F outdoor, so after the
A/C went out I figure inside temps like reached 60C/140F, possibly higher.
The CPUs' thermal protection kicked in, but the hard drive understandably
just wasn't the same after that.)  I decided to switch to a RAID system
for additional reliability, and my backup system scaled right along with
it.  I'm now running 4 x 300 gig Seagate SATA drives, slightly slower than
some but a full five year warrantee, using kernel RAID, additional details
below.

Here's how it works.  I have my system divided up into several protection
levels, depending on the content.  

0: Temporary or easily redownloadable content need not be backed up at all.

This includes /tmp, /var/tmp, the portage tree, /usr/src, and my ccache.
All these can be kept in separate directories on the same "partition",
using either config settings (for ccache and the portage tree) or symlinks
(/tmp, /var/tmp).

On a single drive, this will be a single partition, not duplicated.  On a
RAID system, this works very well as RAID-0/striped.  Striped RAID is very
fast, which happens to fit the use of much of this data quite well, but if
one disk goes down, it takes the entire RAID-0 with it. That's just fine,
as this is temporary data anyway, either throw-away or easily
redownloadable/recoverable.  IOW, this is the /perfect/ case for
RAID-0/striped.

1: boot area (MBR and /boot, with GRUB and kernels).

On a single drive, you should keep a removable (ISO, thumb
drive, zip drive, whatever) around with at least GRUB/LILO and a kernel or
two, in case the hard drive's boot area and/or /boot fail.  If you have
two drives, keep a tested/working GRUB and /boot with fairly current
kernels on both drives.  In a RAID installation, RAID-1/mirrored /boot,
and install and test-boot from GRUB on all drives in the mirror set.  Note
that RAID-1/mirroring will provide reasonable protection against physical
drive failure, but it will **NOT** protect against "sysadmin fat-finger
syndrome", since it's still only a single copy of /boot.  Thus, you'll
still want to keep a removable media backup boot solution available.  For
ease of update, a USB thumb-drive is probably the most convenient (128MB
or even 64MB should work), altho a CDRW/DVDRW would work, as would a
bootable zip drive and disk, altho they are somewhat passe by now.

Again, this will likely be a single "partition" or unpartitioned RAID-1
device.

Everything else: data that likely needs some level of redundancy
protection.

On a RAID system, it will be RAID protected, but likely at
RAID-5, 6 or 10.  (Here I use RAID-6.  Again, more detail below.)  For
higher value data, however, just the redundancy of the RAID itself isn't
enough, due to the previously mentioned "sysadmin fat-finger syndrome",
or tho we don't have to worry about it so much on *ix, malware or the
like.  Single and double disk systems will want to backup some but not all
of this data.

This "everything else" can be further divided into three levels, system,
high-value data, mid-value data, with multiple partitions accordingly.

mid-value data:  This consists of stuff like /var/log.  It may also
include content known retrievable from the net, but at some cost in time
and trouble, and other system/purpose specific data that's not extremely
difficult to replace but would require enough time or expense to do so
that you can't rightly call it "temporary".

On a single or double disk system, mid-value data will likely be treated
for purposes of backup much like temporary data -- ignore it.  The hassle
of backing it up is likely to be more work than the value of the recovered
data.  However, on RAID systems, there's a distinction between the two in
that mid-value data will be considered valuable enough to protect it with
the redundancy of RAID, while temporary data is throw-away even at that
level.  Even so, the redundancy of the RAID will normally be considered
safety enough -- no additional backup efforts worth it.  That remains true
despite the risk of "sysadmin fat-finger syndrome", since the risk is
relatively low compared to the value of the data and the hassle of
additional backup arrangements.

A single operational copy of mid-value data will be all that's tracked,
but as the amount and filesystem layout of this data will vary greatly
across individual purposes and installations, it may be one partition,
many, or one or more logical volumes managed with LVM2 or similar.  (More
on that below.)

System data:  This is the critical operational system.

For least trouble managing it, the goal in Gentoo terms should be to
include everything installed by portage, together with the portage
database (in /var/db), and all system level configuration, all on the same
(single) partition, BUT CRITICALLY, TO MAKE THREE IMAGES, three separate
partitions, imaging essentially similar data.

>From experience, I know keeping everything installed by portage together
with the portage database and all system level config data, is the
simplest to manage.  While it's /possible/ to split off for example /var
and /usr, that's needless additional complexity.  The idea here is that
everything should stay in sync.  If you lose /usr, the portage database of
what's installed that's located in /var/db won't be very useful. 
Similarly if / goes bye bye and you boot from "rootbak1", which has older
binaries in /bin and an older /etc configuration, but still matched
against the new /usr.  That's no good and only causes headaches -- I've
had it happen!  Thus, keep the portage installation database in /var/db in
sync with /usr, /, and /etc, by keeping everything on the same
"partition", and you save yourself VAST amounts of trouble!  <<  That last
sentence is worth rereading.  I KNOW from experience!  FWIW, I'm running
10 gig "system image" partitions, giving me plenty of room as I'm only
using 1.5 gig according to df, so 5 gig system partitions should be fine
on most systems, I'd think.

As I mentioned above, ideally you have a minimum of THREE system images,
identically sized.  One will be your "working" system.  The other two (and
additionals if desired) are alternating backup snapshots.  Here, I have no
set schedule for system backups, though every three-ish months might be a
reasonable goal.  When I think everything's working particularly well,
I'll just erase the oldest backup and copy the working system image over,
creating a new snapshot.  Two backups minimum is safest, as that means
even if disaster strikes when I'm in the middle of one, and I lose the
main working image before the backup is complete, I still have a known
working second backup I can boot to.  No sweat!  Of course more system
images lets you have more backups, effectively letting you take snapshots
more frequently as anything over say 9 months old will likely be nearly as
much work to update as it would be to install from scratch, so there's a
practical limit to how old you want to go, and more images simply means
you can take snapshots more frequently within that limit.

It's worth point out again what I mentioned as #3 requirement/advantage
above.  With my system, any of the system snapshots is pretty much self
contained and ready to boot, just as it was when the snapshot was copied
over.  No fiddling around restoring stuff if you are up against a deadline
and have to have a working system NOW.  If the normal/working system image
fails, you simply boot one of your snapshots and continue where you left
off.  You have exactly the same set of applications, with exactly the same
installation specific configuration, as you did when you took the
snapshot, so you are immediately up and running, only possibly having to
think back on how it was configured back then if you changed something
since then.

On a single disk system, you may be able to get away with two system
images, but you lose the additional safety of the second backup.  On a
dual disk system, if space permits, you almost certainly want at least one
backup image on each disk, the main/working image and one backup on
presumably the bigger disk, the other backup on the other disk, thus
maintaining a fully working system if either disk goes out.  On a RAID
system, you'll probably want your images in RAID-5, -6, or -10, /possibly/
RAID-1/mirrored if you have only a two-disk RAID or need the extreme
redundancy.

Taking a paragraph out to discuss RAID for a moment.  RAID-0 as mentioned
is striped for speed, with speed scaling as more devices are added, up to
the speed of the buses they are attached to, but offers no redundancy, so
any disk out and you lose it all. RAID-1 as mentioned is mirrored for
redundancy -- the data is simply copied to each device in the RAID, and
the RAID can continue to function as long as even a single device
survives.  RAID-1 (kernel implemented, anyway) is slow writing as the data
must be written once to each device, but slightly faster than single-disk
for reading under single tasking scenarios, faster still under
multi-tasking i/o scenarios as the kernel will split the tasks between
devices. RAID 4,5,6 are variations on parity RAID. RAID-4 is devices minus
1 for data, with that one device dedicated to parity protection.  If any
data device goes down, the parity device can reconstruct the data, altho
it's slower. Read-speed is comparable to the number of data devices in a
striped array, but write-speed is bottlenecked by the parity device which
is always written, which is the reason RAID-4 isn't commonly used any
more.  RAID-5 is RAID-4 but with the parity stripe alternating between
devices, thus eliminating the write bottleneck of RAID-4.  Both read and
write speeds compare to a striped array of one less device.  RAID-6 is
double-parity, again alternating stripes.  Thus, a four-device RAID-6 array
is comparable in speed to a two-way striped array.  Any TWO devices can
die in a RAID-6, and it can still recover or even continue to operate
altho at reduced speed.  RAID-10 is a combination of RAID-0 and RAID-1,
mirrored and striped. There's also RAID-0+1 which is the reverse
combination but less efficient.  I can never keep straight which is which,
however, but that's fine as I'm not using it and remember the concept and
that if I DO find a use for it, that I need to look it up again.  Speeds
are typically half the speed of a striped array of the same size, but with
symmetric mirroring redundancy.  RAID-10 is typically used in large
arrays, where its efficiency is far better than RAID-5 or -6, while RAID-5
or -6 are typically used in smaller arrays or where total byte capacity is
more important than speed.  RAID-10 also maintains degraded (one or more
bad devices but still operational) speed better than RAID-5 and -6 do, so
is the better choice where continued operation at full speed even as
individual devices fail is a must.

As I mentioned, for my little four-disk RAID, RAID-6 seemed the best
compromise of redundancy and speed for my needs, so that's what I chose to
implement my "everything else" data protection on.  The practical effect
is a two-way striped array, in which I could lose any two of the four
drives and still have the data on the RAID intact.

Below I cover partitioned RAID vs. LVM2 managed volumes.  Here, I'll
simply mention that I have my system images implemented as partitioned
RAID, (mdd partitioned, vs. md and lvm2), as it *significantly*
decomplicates booting and recovery issues.

That leaves high-value data:  Installation specific data (such as /home
and /usr/local) for which the redundancy of RAID is NOT sufficient
protection.

As with system images, here you'll want backup images of the data even if
it's on RAID, to protect against "sysadmin fat-finger syndrome" among
other things.  Actual configuration and number of images will vary by
purpose and site, but the idea is always keep at least two copies of
every "partition". Single disk systems will want at least second partition
copies, and will likely want removable backup copies of the most critical
stuff.  Dual disk systems will likely want to keep one image of at least
the most critical stuff on each disk, just in case, thus lessening the
need for removable media backup (altho not eliminating it entirely for the
super-critical stuff).  Likewise for RAID systems -- a second image
protected by RAID is a must, but that and the RAID protection may well be
considered enough given the lowered risk and the hassle factor of
removable media backups, except for the most critical stuff, of course. 
In practice, it's likely that what would be DVDs worth of data needing
some sort of backup can be reduced to CDs or thumb-drives worth of data at
the critical value level worth backup beyond the second or third on-disk
image.

Here, I didn't worry much about number of partitions as I chose to manage
everything using LVM2 (Logical Volume Management), which maintains most of
the advantages of partitions (with an exception of direct bootability,
thus the lower suitability for bootable system images), but is VASTLY more
flexible.  It's out of scope to detail LVM2 further here, save to note that
even for single disk systems it could be worth the trouble to research it.

Now, a few more notes on implementation and practicality issues.

For general system management, I find mc aka midnight commander, an
extremely valuable tool in my sysadmin toolbox.  If you aren't familiar
with it yet but are interested in an ncurses based
file-manager/viewer/editor complete with scriptable macros and syntax
highlighting, it's certainly worth trying.  app-misc/mc .

As with most of my sysadmin tasks, I use mc when I'm working on my
partition images.  Its copy and move mode maintain attributes by default
and just do "The Right Thing" (TM) when it comes to symlinks, socket
and device files, and other assorted filesystem irregularities of the *ix
world.  What could be simpler than simply copying existing files from the
working location to the backup/snapshot location?  Using mc's two window
layout, it's that easy -- at least after the issue in the next paragraph
is dealt with, anyway.  Sure, one can use the archive switch and etc on a
command line copy, or do impressive stuff using tar, but for me, simply
copying it with mc accomplishes my objective with the least fuss and muss.

So what's that issue I mentioned?  Well, simply this.  Particularly for
the system image (/ fs with /etc and much of /usr and /var) upon which
everything else is mounted, simply copying won't /normally/ stop at the
system image (aka partition) boundary as we'd like in this case.  There's
also the issue of trying to copy the raw device files in /dev with udev
mounted there, and other similar things to worry about.

As it turns out, there's a relatively easy way around that.  Simply use
the mount --bind option to mount a view of the root filesystem (only,
without everything mounted on it recursively) elsewhere, and copy /that/
view of the filesystem, instead of trying to copy the normal one.  This is
easiest accomplished with an entry in fstab similar to the following:

# Dev/Part    MntPnt           Type    MntOpt        D F
# for mount --bind, --rbind, and --move              ###
# /old/dir    /new/dir         none    bind          0 0
/.            /mnt/rootbind    none    bind,noauto   0 0

Now, when I go to create a system image snapshot, I can simply
"mount /mnt/rootbind", and then simply copy (using mc, naturally)
everything from there to my new image snapshot.  Note that the dev/part
is listed as /.  Since there's already an entry for the root filesystem
mounted at /, I can't use that or mount gets confused.  Of course, /.
is self referencing, and this little trick keeps mount happy.  I had fun
trying to figure it out. =8^)

One more caveat:  A very few files may be read-locked under normal
operation and thus not able to be copied directly.  I've never run into
that here, but from what I understand, a typical example might be a
database file.  MySQL users and others running databases might want to pay
attention here, but as I said, I've never run into issues, and those
shouldn't be on the system image, but rather belong in the
high-value data section, likely in their own dedicated partition, which
should make special-casing that particular imaging task not too difficult.

OK, now a word about fstab management.  What I've found works best is to
keep three separate fstabs (or one for each system image), call them
fstab.working, fstab.bak1, and fstab.bak2, if you wish.  They will be
essentially the same except that the root filesystem will be switched
around with the (noauto optioned, so they don't automount) two backup
system image snapshots. Then on the working image, make fstab a symlink
pointing to fstab.working.  When you take a snapshot image, before you
unmount after the copying, simply point the fstab symlink at the
appropriate fstab.bak? file.  Each system image then has a copy of all
three files at the time the snapshot was taken, with the symlink adjusted
to point at the right fstab.* file so everything just works when that
image is booted.

Finally, a bit more detail on the lvm vs partitioned RAID choices I made
and why.  First, it's worthwhile to note that a lot of the documentation
still says (incorrectly) that Linux kernel RAID doesn't handle partitions.
Linux kernel RAID (and the corresponding mdadm, don't know about the older
raidtools) has worked with partitioned RAID devices since at least early
in the 2.6 cycle, and I think sometime in the 2.5 development kernel cycle.

Second, there's an important difference between the way Linux handles LVM2
and the way it handles RAID, including partitioned RAID.  The RAID device
drivers are entirely contained within the kernel (provided they are
configured as built-in, not modules), and can be configured on the kernel
command line (thus from grub), so one can boot RAID, including partitioned
RAID, directly.  Not so with LVM(both 1 and 2), which require userspace
configuration to work, thus complicating the root on LVM2 boot scenario
dramatically.  While it's still possible to use an initrd/initramfs
containing the LVM2 config for at least the root device, that's VASTLY
more complex than simply tweaking the kernel command line in grub.  Of
course, should things go wrong with LVM2, fixing the problem is likely to
be VASTLY more complex as well, while with RAID including partitioned
RAID, if it's fixable, it's fixable from GRUB by altering the kernel
command line or in the event of a wrongly configured kernel, simply
booting back to the old kernel that handled it correctly.

Thus, for ease of boot maintenance and recovery, I chose NOT to use LVM2
on my bootable system rootfs images.  Once that's up and mounted, I have
the full system with all the usual tools available to me to recover
or reconfigure additional RAID devices, or troubleshoot LVM2, if either
one is necessary.  Thus, no big deal having my mid-value and high-value
data on LVM2, where it's easy to manage the volumes than it would be
partitions.  I just don't want the additional complexity of having to
worry about LVM2 on my main root filesystem, whether working image or
backup images, since the purpose is to have all of them bootable, and
managing that is VASTLY less complex with direct kernel boot than it would
be if I had to worry about an initramfs to load LVM2.

So, after all the above, here as an example is how I have my system
configured (roughly):

For each drive, /dev/sd[a-d], same sized partitions on each of the four
300 gig SATA drives:

GRUB on the MBR

/dev/sdX1: partition forming part of a RAID-1 /dev/md0 for /boot

/dev/sdX2: partition forming part of a partitioned RAID-6 /dev/md_d1

/dev/sdX3: partition for swap, all four partitions pri=1, so the
           kernel stripes them automatically when it activates them

/dev/sdX4: extended partition, reserved/ignore

/dev/sdX5: partition forming part of a partitioned RAID=0 /dev/md_d2
           however I didn't actually need this one partitioned

/dev/sdX6 and beyond: ~100 gig remains free and unpartitioned on each 300
gig drive, to be allocated later as needed.

The partitions on the RAID-6 /dev/md_d1 are as follows:

/dev/md_d1p1: working system image
/dev/md_d1p2: /mnt/bak1 (first backup system image)
/dev/md_d1p3: /mnt/bak2 (second backup system image)
/dev/md_d1p4: reserved as an LVM2 physical volume group, vgraid6

vgraid6 has the following logical volumes:

/dev/vgraid6/home
/dev/vgraid6/homebk   (the backup /home image)
/dev/vgraid6/local    (this is /usr/local)
/dev/vgraid6/localbk
/dev/vgraid6/log      (/var/log, raided but not worth a backup image)
/dev/vgraid6/mail
/dev/vgraid6/mailbk
/dev/vgraid6/mm       (multimedia)
/dev/vgraid6/mmbk
/dev/vgraid6/news     (raided but no backup, again, mid-value data)
/dev/vgraid6/pkg      (portages package dir for FEATURES=buildpkg)
/dev/vgraid6/pkgbk

The RAID-0/striped /dev/md_d2p1 (the only partition I'm using)
mounts at /mnt/str (for striped).  Subdirs:

/mnt/str/ccache
/mnt/str/portage      (the portage tree, including distdir but not pkgdir)
/mnt/str/usrsrc       (for /usr/src/linux)

It would have /tmp as well except that with 8 gigs of memory, I mount /tmp
as tmpfs, maximum size 5 gig.  /var/tmp symlinks /tmp, and all portage's
compiles use the tmpfs for scratch space, dramatically speeding up merges!
=8^)

Obviously, I've spent quite some time devising a solution that fits my
needs.  There's only one big thing I'd change about my layout, and I
already accounted for that as described above.  I actually have only two
system images, working and a single backup.  Next time I reorganize
(remember, I've got a third of the space remaining on the drives, a
hundred gigs each, to do so), I'll create a third system image.  I'll
probably do something different with swap as well, since I only made those
partitions four gigs each and with 8 gigs of memory and the fact that
in-kernel suspend to disk writes to a single device/partition only, that's
not enough to properly suspend, even if the kernel's implementation /does/
work right (last I checked, back when I had only a gig of RAM, it didn't,
but there's some dramatic changes going on in that department so it might
now -- except I error out due to lack of room in the partition!).

While the above is my current RAID layout, I was using something very
similar back when I had only two disks, as described.  The layout now is a
bit more sophisticated, however.  In particular, I had /usr and /var on
their own partitions back then, but as I said, that lead to serious
problems when one died and they got out of sync with each other, thus the
warnings about that and the current layout folding those back into the
single system image.  The idea is solid and has demonstrated itself over
two partial drive failures, as I mentioned, before I decided I wanted the
reliability of RAID.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Backup techniques Was: audio/vidio apps and glibc-2.5?

[Rob Lesslie]

There is another school of thought, which is that the time taken to
create and maintain this excellent, but overly elaborate, system for
the home user would be greater than the time taken to simply
re-install.  I am not interested in a maintaining such a comprehensive
system at home - I am interested in using my computer to get things
done :)

Work, however, has different requirements (which I won't go into here)
but your method strikes me as being much more suited to that
environment.  It provides much greater to scope to recover from
failures quickly and easily, and I can spend less time doing actual
work - a double win :>p

I fully agree that non-replaceable user data such as movies, music,
personal documents, photos, work etc should be backed up fully, both
to defend against hardware failure and stupid user syndrome.  Backing
up of the entire system to this extent can be a waste of time for many
users.  To clairfy, I am only talking about sensible requirments for a
HOME user.  Personally, I have a shell script which uses rsync to
backup my home folder (and some other bits) to another box accross my
LAN.  This has got me through one disk failure already :).  It IS a
hassle to rebuild a system, but less so than to manage an overly
complex system.

Just my opinion as a humble Gentoo user.
-- 
Rob Lesslie
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Backup techniques Was: audio/vidio apps and glibc-2.5?

[Peter Humphrey]

On Tuesday 24 October 2006 13:04, Duncan wrote:

> For least trouble managing it, the goal in Gentoo terms should be to
> include everything installed by portage, together with the portage
> database (in /var/db), ...

Personally, I don't bother backing-up /var/db since it's easily recreated 
with emerge --metadata.

-- 
Rgds
Peter
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Backup techniques Was: audio/vidio apps and glibc-2.5?

[Bo Ørsted Andresen]

[-- Attachment #1: Type: text/plain, Size: 667 bytes --]

On Tuesday 24 October 2006 16:01, Peter Humphrey wrote:
> On Tuesday 24 October 2006 13:04, Duncan wrote:
> > For least trouble managing it, the goal in Gentoo terms should be to
> > include everything installed by portage, together with the portage
> > database (in /var/db), ...
>
> Personally, I don't bother backing-up /var/db since it's easily recreated
> with emerge --metadata.

/var/db/pkg contains info about what is installed. If you lose it in the best 
case `emerge -e world` will regenerate it. In the worst case you may need to 
reinstall. It has nothing to do with the metadata which are located in 
$PORTDIR/metadata.

-- 
Bo Andresen

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-amd64] Re: Anybody else having problems with audio/vidio apps and glibc-2.5?

[Bob Sanders]

Jamie, mused, then expounded:
> 
> A good piece of advice and one that I really should follow.
> What is the best way to take an image of the Gentoo install?
> In my case my Gentoo install resides on /dev/hda2 (boot) ; /dev/hda3
> (swap) and /dev/hda5 (root) - is it possible to use something like dd to
> image these partitions to a disk somewhere else on my network? Is this a
> good/bad/non-workable idea?
>

I use rsnapshot to automatically backup a few servers to save
the data onto a raid 5 disk array.  I have it set to take daily and weekly
snapshots of the data.

As to the root partitions, I run xfs so that I can attach a USB or
Firewire hard drive and do an xfs dump/restore, which copies the
partition I'm after -

      xfsdump -l 0 - /sourcedisk | xfsrestore - /targetdisk/

The nice thing about xfsdump/xfsrestore is the partitons don't have to be
the same size.  The backup partition just has to be as big or bigger than
the original.  But I generally don't make backup partitions, just use it
when swapping out hard drives.

After the copy, I run xfs_check or xfs_repair to verify the new partition.

As far as worrying about the root partition, I don't.  Typically, I
can have a functional Gentoo system back to where it's usable for
it's main function in about 2 hrs.  Desktop and nice things by the
next day.  I learned a long time ago not to worry about the operating
system and just be concerned about the important things - the data.
The os can always be re-installed.
 
Bob 
-  
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: Anybody else having problems with audio/vidio apps and glibc-2.5?

[Richard Fish]

On 10/24/06, Peter Humphrey <prh@gotadsl.co.uk> wrote:
> On Tuesday 24 October 2006 01:58, Richard Fish wrote:
>
> > I use dar and USB2 disks for my backups.  Faster, cheaper, and with
> > more capacity than most tape drives.
>
> I hadn't spotted dar before. Can you explain the use of the dar32 and dar64
> USE flags? The use.desc description doesn't help much.

By default dar uses an arbitrary-precision integer implementation to
handle archives of basically any size.  But it takes more memory and
CPU cycles to use, so you can compile dar with dar32 or dar64 to use
32-bit or 64-bit integers in place of the infinint.  If you use dar32,
you cannot create or restore from any archive larger than 4G, or
backup files larger than 4G, etc.

I personally use dar64, as that *easily* handles all my data.

http://dar.linux.free.fr/doc/Limitations.html

-Richard
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: Anybody else having problems with audio/vidio apps and glibc-2.5?

[Peter Humphrey]

On Tuesday 24 October 2006 17:25, Richard Fish wrote:

> By default dar uses an arbitrary-precision integer implementation to
> handle archives of basically any size.  But it takes more memory and
> CPU cycles to use, so you can compile dar with dar32 or dar64 to use
> 32-bit or 64-bit integers in place of the infinint.  If you use dar32,
> you cannot create or restore from any archive larger than 4G, or
> backup files larger than 4G, etc.

I see - thanks!

> I personally use dar64, as that *easily* handles all my data.

I'll do the same. I'm not sure yet whether I'll switch from tar yet, but 
I'll give dar a try. Thanks for the info.

-- 
Rgds
Peter
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Backup techniques Was: audio/vidio apps and glibc-2.5?

[Peter Humphrey]

On Tuesday 24 October 2006 14:18, Bo Ørsted Andresen wrote:

> /var/db/pkg contains info about what is installed. If you lose it in the
> best case `emerge -e world` will regenerate it. In the worst case you may
> need to reinstall. It has nothing to do with the metadata which are
> located in $PORTDIR/metadata.

Ah. I stand corrected.

-- 
Rgds
Peter

-- 
gentoo-amd64@gentoo.org mailing list