[gentoo-amd64] Root on Raid and LVM

[gentoo-amd64] Root on Raid and LVM

[Chris S]

Greetings,

This is not an really an AMD64 issue, although the system is an nForce4 
SLI with nv sata.

In short, I'm trying to get my Gentoo system built on an lvm group on 
top of raid5, but it won't go.

I am using a genkernel --lvm2 created initrd but when it searches for 
raid volumes it find nothing! (I did also have --dmraid but I think 
that's for those "sudo hardware" onboard raid controllers)
However, if I boot to the kernel directly without an initrd, it DOES 
find the raid devices (but then fails, naturally, because it can't find 
lvm volumes).

Essentially I have:
sda5, b5, c5, d5 all on a linux software raid 5 array (/dev/md5) - 
partition types are all "fd".
I then have an lvm group called "system" under that, and then volumes 
"slash" and "home" under that.

Have tried 2.6.13 gentoo-sources and 2.6.14 mm-sources.

I've read about a new bootloader option, lvmraid, but I have yet to try 
adding "lvmraid=/dev/md5" to grub's kernel line.
However, it seems strange that the initrd is not picking up the raid 
devices.

Any thoughts?

Cheers,
-c
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Root on Raid and LVM

[Chris Smart]

Chris S wrote:

> I've read about a new bootloader option, lvmraid, but I have yet to 
> try adding "lvmraid=/dev/md5" to grub's kernel line.
> However, it seems strange that the initrd is not picking up the raid 
> devices. 

This got me booting - but it is strange! The initrd still says that it 
can't find any software raid devices, but now it CAN find the lvm 
volumes and it boots all the way. Once I log in I see that I have all my 
raid devices started, and all the right things are mounted.

There is an issue however, that after it finds the lvm group and start 
booting, I get "rm - can't remove /sys/blah*" spewing across the screen 
for about 2 minutes, then it goes and boots just fine.

Has anyone heard of these issues?

I assume lvmraid option is new as I had this working without it 
previously. Perhaps Gentoo devs have changed the way all this works?

Cheers,
-c
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Root on Raid and LVM - Solved

[Chris Smart]

Chris Smart wrote:

> Has anyone heard of these issues?
>
> I assume lvmraid option is new as I had this working without it 
> previously. Perhaps Gentoo devs have changed the way all this works?

Further to this,  I re-emerged baselayout and added a few other things 
to grub.conf kernel line (along with the lvmraid entry), namely 
"gentoo=nodevfs udev".

Now when Gentoo boots I see what I was expecting - it no longer says 
"Scanning for Software Raid - Found None", now it says:
"Detected real_root as an md device, filling node."
Then it detects the LVM partitions and it's all good! Also no longer get 
the "rm /sys/blah*" errors.

Only one issue now, and that is that upon shutdown, Gentoo cannot stop 
/dev/md5 because it is in use.
Is there a fix for this? Perhaps in the order of the shutdown?

Cheers,
Chris
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Root on Raid and LVM

[Florian D.]

> This got me booting - but it is strange! The initrd still says that it 
> can't find any software raid devices, but now it CAN find the lvm 
> volumes and it boots all the way. Once I log in I see that I have all my 
> raid devices started, and all the right things are mounted.

hi chris,
i cannot understand, how you got things working without an extra /boot 
partition. the system needs initrd to start the LVM, right? but if 
initrd itself is on the LVM...? very interesting ;-)

cheers, flo
-- 
gentoo-amd64@gentoo.org mailing list

[gentoo-amd64] Re: Root on Raid and LVM

[Duncan]

Florian D. posted <433FDF0A.4090306@gmx.at>, excerpted below,  on Sun, 02
Oct 2005 15:22:18 +0200:

> i cannot understand, how you got things working without an extra /boot 
> partition. the system needs initrd to start the LVM, right? but if 
> initrd itself is on the LVM...? very interesting ;-)

He didn't say he didn't have an extra boot.  He only enumerated the LVs
(logical volumes) on the MD/RAID, which he said was on sda5, sdb5, etc,
all partition 5, on /dev/md5.

The usual way to handle that, if you are booting from the RAID array
as well, is to create a small raid1 array out of small partitions toward
the front of each physical disk, for /boot. Being raid1 (mirrored), the
same data is imaged to all disks in the raid1 identically.  Both GRUB and
LILO can read individual disk partitions belonging to the mirror array as
if they were stand-alone partitions, so can boot from any of them, altho
they do boot from only one at a time.  The kernel and initrd are on this
boot array (so mirrored to each physical disk in the array), and can then
be loaded.  Root and other LVs are then created on the main VG (volume
group (or multiple groups) taking the rest of the disks, to be loaded from
the initrd.  Once /they/ are loaded, and the pivot_root to the main-root
on the main VG is accomplished, the raid1 array containing /boot can be
mounted, if desired.  (As usual, Gentoo keeps /boot unmounted by default.)

...  As some may guess, I've been studying this stuff recently!  =8^)  I
don't have my own RAID setup yet, but probably will by late this week.  (I
plan to go pickup the drives probably Tue or Wed.)

Of course, he didn't actually say /boot is on raid1, either.  It may be on
a conventional disk, perhaps manually backed up.  There is some clue that
there are extra partitions, however, and probably extra RAID volumes as
well, since both the partitions and the md's mentioned were #5.  Now, #5
is the first secondary partition in the extended partition, so there
wouldn't /have/ to be a 1-4.  However, there's likely at least an sdX1 on
all the disks, which would form the raid1 for /boot.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman in
http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html


-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: Root on Raid and LVM

[Florian D.]

Duncan wrote:
> He didn't say he didn't have an extra boot.  He only enumerated the LVs
> (logical volumes) on the MD/RAID, which he said was on sda5, sdb5, etc,
> all partition 5, on /dev/md5.
yes, i also realized that afterwards..
> 
> The usual way to handle that, if you are booting from the RAID array
> as well, is to create a small raid1 array out of small partitions toward
> the front of each physical disk, for /boot. Being raid1 (mirrored), the
> same data is imaged to all disks in the raid1 identically.  Both GRUB and
> LILO can read individual disk partitions belonging to the mirror array as
> if they were stand-alone partitions, so can boot from any of them, altho
> they do boot from only one at a time.
i have a similar solution with a raid5 for /opt /usr /home, etc. and a 
small / on raid1. but i think grub is better in this situation, because 
with the grub shell, in case of a failed disk, the changed 
identification of the subsequent disks can easily be corrected. i don´t 
know how lilo handles this.

> ...  As some may guess, I've been studying this stuff recently!  =8^)  I
> don't have my own RAID setup yet, but probably will by late this week.  (I
> plan to go pickup the drives probably Tue or Wed.)
good luck. when there is no email from you for some days, then we know 
sth. went wrong ;-)
-- 
gentoo-amd64@gentoo.org mailing list

Re: [gentoo-amd64] Re: Root on Raid and LVM

[Sebastian Redl]

Florian D. wrote:

> i have a similar solution with a raid5 for /opt /usr /home, etc. and a 
> small / on raid1. but i think grub is better in this situation, 
> because with the grub shell, in case of a failed disk, the changed 
> identification of the subsequent disks can easily be corrected. i 
> don´t know how lilo handles this.

My RAID setup (two disks) looks like this:
/boot -> RAID 1
/var -> RAID 1
/var/tmp/portage -> RAID 0 (no use wasting space there, and it might 
even speed up compilation a bit)
/usr -> RAID 0 (nothing in /usr that can't be simply reinstalled, thus 
again no reason to waste space)
/ -> RAID 1

Sebastian Redl
-- 
gentoo-amd64@gentoo.org mailing list

[gentoo-amd64] Re: Re: Root on Raid and LVM

[Duncan]

Florian D. posted <4368B219.1070903@gmx.at>, excerpted below,  on Wed, 02
Nov 2005 13:33:29 +0100:

>> ...  As some may guess, I've been studying this stuff recently!  =8^)  I
>> don't have my own RAID setup yet, but probably will by late this week.  (I
>> plan to go pickup the drives probably Tue or Wed.)
> good luck. when there is no email from you for some days, then we know 
> sth. went wrong ;-)

=8^)  I have the drives... four Seagate SATA 300 gigs.  I got them
yesterday and installed them this AM.  I haven't done anything with them
yet, so they're just empty unconfigured drives still, but the BIOS and
kernel see them, so I've gotten a good start!  =8^)

I've now got this full tower case stuffed to the gills! =8^)  The four
SATA drives, my current main drive, a 250 gig Maxtor PATA, on PATA primary
master, a DVD writer on PATA primary slave, an old 36 gig Maxtor on PATA
secondary master, and a CD writer on PATA secondary slave.  Round that out
with a legacy 3.5" floppy, and I've got 9 drives in this thing, with > 1.4
TB of hard drive capacity (pre-raid mirror/parity discounting)!

The 250 gig still has  ~100 gig not even formatted yet, but my AC went out
this summer and it got hot and developed some bad blocks.  With them all
marked and one 4 gig partition written off entirely, it has proved
decently stable, but I've been worried about it, thus the current upgrade
job.  I've been considering RAID and drives are now cheap enough to do it,
so I decided to go for it!

I'll be setting up three RAID sets on the four SATAs, and will be using
the Maxtors as emergency boot and backups.  A small 4-way raid1 will have
/boot.  The majority of the disks, 200+ gig of each of the four, will be a
raid6 (raid5 but with two parity copies... any two drives can go down, one
out is supposed to maintain speed fairly well, two out will drop speed
substantially but will still function) carrying root, /usr, /var, /home,
and my dedicated mail, news, and media LVs.  A medium size raid0
will complete the roundup, containing /tmp (which is $PORTAGE_TMPDIR) and
a dedicated ccache LV.  I'm currently running 1G memory and no swap, but 
I think I'll reactivate swap as well while I'm at it, with a small
non-raid swap partition on each of the four.  The kernel stripes swap on
its own, so there's little reason for me to put it on raid, and one of
the reasons I'm reactivating swap is to try out the developing kernel SMP
suspend features. =8^)  Having swap on raid isn't likely to work too well,
when I'm trying to load memory images out of swap coming out of suspend!

I'll use my existing Maxtor PATAs, as I said, for an emergency boot
installation and off-RAID online backups, and plan on using LVM2's
snapshotting capacities to maintain probably up to two working snapshots
(yes, I'm aware of the write penalty of doing so) of root and /usr, taken
before any major upgrades, such as to a new baselayout or a new version of
KDE. Once I verify functionality of the new upgrades, I'll make
necessary config changes (like switching fstabs), then write off the
snapshot to the non-raid emergency backup, so I can remove it and avoid
the write penalty of the COW (copy on write) snapshot.

Currently, I have a working copy and a backup copy of both root and /usr
on each of my Maxtors, therefore, four OS area snapshots total.  That has
proven to be a good strategy, when I got those badblocks due to the bad
A/C and consequent overheating, but after having used it, I decided it
needed a bit of modification, mainly in that I needed a better way to keep
the snapshots updated so they didn't get massively outdated.  The RAID
gives me better reliability on the one side (and the Seagates with their 5
year warrantee are nice in that department as well), and lvm2's
snapshotting should come in handy as an intermediate stage from the other
angle.

As I mentioned, I still have ~100 gig unpartitioned on the 250 gig.  I'll
maintain that practice on the raid, but the dynamic expansion capabilities
of lvm2 will provide significant additional flexibility in that area.

Well, I've got a quite a bit of work to do still, soo...

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman in
http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html


-- 
gentoo-amd64@gentoo.org mailing list