[gentoo-server] RAID5 vs. RAID0+1

[gentoo-server] RAID5 vs. RAID0+1

[Christian Spoo]

[-- Attachment #1.1: Type: text/plain, Size: 353 bytes --]

Hi,

I need to set up a RAID box of about 470GB disk space accessible via
GBit LAN.

The whole thing should have good performance but must be reliable as
well. Which RAID mode would you recommend, 5, 0+1 or maybe any other?
How about the time needed for rebuilding such arrays in case of disk
failure?


Thanks in advance,

Christian Spoo

[-- Attachment #1.2: mail.vcf --]
[-- Type: text/x-vcard, Size: 281 bytes --]

begin:vcard
fn:Christian Spoo
n:Spoo;Christian
adr;quoted-printable:;;Am Kaiser 26;W=C3=BCrselen;NRW;52146;Deutschland
email;internet:mail@christian-spoo.info
tel;fax:02405/475071
tel;home:02405/493466
tel;cell:0176/61055475
x-mozilla-html:FALSE
version:2.1
end:vcard


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 888 bytes --]

Re: [gentoo-server] RAID5 vs. RAID0+1

[Ronan Mullally]

Hi Christian,

On Fri, 6 Oct 2006, Christian Spoo wrote:

> I need to set up a RAID box of about 470GB disk space accessible via
> GBit LAN.
>
> The whole thing should have good performance but must be reliable as
> well. Which RAID mode would you recommend, 5, 0+1 or maybe any other?
> How about the time needed for rebuilding such arrays in case of disk
> failure?

You forgot to mention one of the key criteria - cost.  If you've got the
budget for it then RAID 0+1 (aka RAID 10) is always going to out-perform
RAID 5.

As general rule of thumb, "which RAID should I use" questions can always
be boiled down to:

 "Speed, Reliability, Low Cost, pick any two."


-Ronan

-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] RAID5 vs. RAID0+1

[Mike Williams]

On Friday 06 October 2006 17:30, Christian Spoo wrote:
> I need to set up a RAID box of about 470GB disk space accessible via
> GBit LAN.
>
> The whole thing should have good performance but must be reliable as
> well. Which RAID mode would you recommend, 5, 0+1 or maybe any other?
> How about the time needed for rebuilding such arrays in case of disk
> failure?

Firstly, forget 0+1, use RAID10.

Define "good performance". Reads or writes?

RAID5 is fast for reads, slow for writes, and you lose the capacity of 1 disk.
RAID10 is *fast* for reads *and* writes, but you lose the capacity of half 
your disks.

RAID5 can live with the failure of one drive, but takes a large performance 
hit and all your redundancy is gone until a new one is synced up which is 
hard and time consuming to do.
RAID10 can, in theory, lose half of the disks and continue with little to no 
slow down. Rebuilds are easier than RAID5 as it's a straight bit for bit 
copy.

RAID6 adds a second disk worth of redundancy, but slows writes down further.

As a bad benchmark I had to rebuild a 4 200GB disk RAID5 array the other 
evening on a P3 1400, took ~140minutes.
On at least 2 occasions I've had another disk die while doing a RAID rebuild, 
the stress was too much for it, for this reason I won't use RAID5 again 
unless there is a very good reason for it (i.e. need for redundancy is 
minimal, and space is more important).

-- 
Mike Williams
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] RAID5 vs. RAID0+1

[Richard Broersma Jr]

> Firstly, forget 0+1, use RAID10.

I don't mean to hijack the tread but I have a related question.

I see from reading the mdadm man page that a RAID10 array can be created directly from individual
drives.  I assume this gives better performance than creating two raid1 arrays and then using
raid0 to attach the two raid1 meta devices.  Is this the case?

Also, I notice when building new kernels that there are no kernel modules for RAID10.  I haven't
yet tested this myself (although I have (4) 300GB on hand to start testing in the next few weeks),
but would this create a problem when trying to create/mount a RAID10 meta device?


> RAID5 is fast for reads, slow for writes, and you lose the capacity of 1 disk.
> RAID10 is *fast* for reads *and* writes, but you lose the capacity of half 
> your disks.

I had a terrible experience with my array when it was configured as raid5.  It worked well for
samba shares with lots of reads and few writes, but when I tried to use it in a heavy write
environment,  the performance was terrible and the array would break and individual drive would
become out of sync.  mdadm would of course automatically re-sync the drives once the writes
completed.


-- 
gentoo-server@gentoo.org mailing list

RE: [gentoo-server] RAID5 vs. RAID0+1

[Jesse, Rich]

http://www.baarf.com

Pay special attention to the whitepaper "Is RAID 5 Really a Bargain?" by
Cary Millsap, also on the site.

It's a slow crusade, but gaining strength.  And it's about time I
joined, too.

GL!

Rich

-----Original Message-----
From: Christian Spoo [mailto:mail@christian-spoo.info] 
Sent: Friday, October 06, 2006 11:30 AM
To: gentoo-server@lists.gentoo.org
Subject: [gentoo-server] RAID5 vs. RAID0+1

Hi,

I need to set up a RAID box of about 470GB disk space accessible via
GBit LAN.

The whole thing should have good performance but must be reliable as
well. Which RAID mode would you recommend, 5, 0+1 or maybe any other?
How about the time needed for rebuilding such arrays in case of disk
failure?


Thanks in advance,

Christian Spoo

-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] RAID5 vs. RAID0+1

[Mike Williams]

On Friday 06 October 2006 18:20, Richard Broersma Jr wrote:
> I see from reading the mdadm man page that a RAID10 array can be created
> directly from individual drives.  I assume this gives better performance
> than creating two raid1 arrays and then using raid0 to attach the two raid1
> meta devices.  Is this the case?

Yup. I use it on 4 200GB drives (which 1 died a couple weeks ago, but I 
haven't got round to replacing) at home.
Can't say whether you'd get any better performance, but it's a whole lot 
easier to create and manage!

> Also, I notice when building new kernels that there are no kernel modules
> for RAID10.  I haven't yet tested this myself (although I have (4) 300GB on
> hand to start testing in the next few weeks), but would this create a
> problem when trying to create/mount a RAID10 meta device?

You'll need the kernel driver regardless.
What kernel? I was using RAID10 way back on 2.6.11.10 as a standard part of 
the kernel, I even found a bug which was fixed by a nice man from Suse within 
2 days (kernel.org bug#5181, hdf is the one which has properly failed now, I 
never did replace any after that bug).

> I had a terrible experience with my array when it was configured as raid5. 
> It worked well for samba shares with lots of reads and few writes, but when
> I tried to use it in a heavy write environment,  the performance was
> terrible and the array would break and individual drive would become out of
> sync.  mdadm would of course automatically re-sync the drives once the
> writes completed.

Heh, I use RAID5 on the fileservers at work, and the backup box at our colo. 
The DBA is forever moaning at me that it's too slow for the DB backups, the 
fileservers are fine.

-- 
Mike Williams
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] RAID5 vs. RAID0+1

[Karl Hiramoto]

Christian Spoo wrote:
> Hi,
>
> I need to set up a RAID box of about 470GB disk space accessible via
> GBit LAN.
>
> The whole thing should have good performance but must be reliable as
> well. Which RAID mode would you recommend, 5, 0+1 or maybe any other?
> How about the time needed for rebuilding such arrays in case of disk
> failure?
>
>
> Thanks in advance,
>
> Christian Spoo
>   
I work with about 10 systems that have RAID5  all have between 300GB and
1.5TB of space.

The slowest old systems with 3 or 4 disks  IDE 200GB  can rebuild raid5
at  about 25MB/sec.

Newer faster systems with 3 or 4 SATA 500GB disks can rebuild raid5 at
about 60MB/sec.

The RAID5 storage servers can always saturate a 100mbps LAN connection
in my case. Doing a copy over a gbit LAN, of large multimedia files from
one RAID5 to another RAID5 server over NFSv3, I see about 30MB/sec
max.    Sorry i don't have exact benchmarks to show you, only what I
recall seeing in the last few months.

For your needs, assuming you have to buy new disks,  2x 500GB disks in
RAID1  and your done.



-- 

--
Karl Hiramoto  http://karl.hiramoto.org/

-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] RAID5 vs. RAID0+1

[kashani]

Jesse, Rich wrote:
> http://www.baarf.com
> 
> Pay special attention to the whitepaper "Is RAID 5 Really a Bargain?" by
> Cary Millsap, also on the site.
> 
> It's a slow crusade, but gaining strength.  And it's about time I
> joined, too.

	baarf assumes you're doing db work in which case they are absolutely 
correct. I did a recent transition from five drives in a RAID5 to six 
drives in RAID10 on our Mysql servers and the change was well worth it. 
Roughly 30% increase in simple writes like an scp from one db to the 
other. We rolled a number of other changes into the db so it's hard to 
tell what was disk and what was proper tuning in Mysql at this point.

	However if you're running a file share that is primarily reads, aka 
just about every file share in the world, RAID5 will work just fine and 
give you much more space. Assuming you're got a limited budget and 
you're doing general file serving I'd err on the side of more space. If 
you're running specialized applications like email, video editing, 
databases then you may want to look at other things besides RAID5.

kashani
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] RAID5 vs. RAID0+1

[Richard Broersma Jr]

> > Also, I notice when building new kernels that there are no kernel modules
> > for RAID10.  I haven't yet tested this myself (although I have (4) 300GB on
> > hand to start testing in the next few weeks), but would this create a
> > problem when trying to create/mount a RAID10 meta device?
> 
> You'll need the kernel driver regardless.
> What kernel? I was using RAID10 way back on 2.6.11.10 as a standard part of 
> the kernel, I even found a bug which was fixed by a nice man from Suse within 
> 2 days (kernel.org bug#5181, hdf is the one which has properly failed now, I 
> never did replace any after that bug).

Your are correct, RAID10 was a kernel option.  I don't know why I missed it before?  Thanks for
straitening me out.

Regards,

Richard Broersma Jr.

-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] RAID5 vs. RAID0+1

[Christian Spoo]

[-- Attachment #1.1: Type: text/plain, Size: 1096 bytes --]

First, thanks for all your answers!

kashani schrieb:
>     However if you're running a file share that is primarily reads, aka
> just about every file share in the world, RAID5 will work just fine and
> give you much more space. Assuming you're got a limited budget and
> you're doing general file serving I'd err on the side of more space. If
> you're running specialized applications like email, video editing,
> databases then you may want to look at other things besides RAID5.
> 
> kashani

That's just the point. The server will be used to store bigger files
(lots of digital photos, videos, etc.) and publish them. Concerning
Ronan's last sentence I'd stick to speed and reliability as the most
important points. At least with the load of several users that'll access
the files at the same time RAID5 would probably be not enough. Another
point is rebuild time in which RAID10 would beat RAID5.

I will give RAID10 a try. That means a configuration of 4 250GB disks
with a total of 500GB space which fits comfortably our needs. Later I
will describe my experiences.

[-- Attachment #1.2: mail.vcf --]
[-- Type: text/x-vcard, Size: 281 bytes --]

begin:vcard
fn:Christian Spoo
n:Spoo;Christian
adr;quoted-printable:;;Am Kaiser 26;W=C3=BCrselen;NRW;52146;Deutschland
email;internet:mail@christian-spoo.info
tel;fax:02405/475071
tel;home:02405/493466
tel;cell:0176/61055475
x-mozilla-html:FALSE
version:2.1
end:vcard


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 888 bytes --]

Re: [gentoo-server] RAID5 vs. RAID0+1

[kashani]

Christian Spoo wrote:

> That's just the point. The server will be used to store bigger files
> (lots of digital photos, videos, etc.) and publish them. Concerning
> Ronan's last sentence I'd stick to speed and reliability as the most
> important points. At least with the load of several users that'll access

heh. I work for a video and photo sharing site and a good friend of mine 
works for an HD post production house. His I/O requirements are at least 
an order of magnitude higher than mine. Publishing vs editing, and it 
really depends on how one defines these things, are two very different 
workloads. I tend to have much more random reads than he does and have 
to partition data and disks to get around that.

That said in the four disk range RAID 10 in a single box is a pretty 
good deal. The cost starts to get ugly as your arrays get bigger and 
start having to pay for the shelf or enclosure.

1 shelf + (16 - 2 (hot spare & parity)) x 500GB x RAID 5 = 7TB usable
1 shelf + (16/2) x 500GB x RAID10 = 4TB usable

Assuming a total cost of $10k per, 28TB at RAID 5 you'd pay $40k and at 
RAID10 you'd pay $70k

In a single box your only have to worry about drive costs.

(3-1) x 250 = 500 GB
(2/1) x 250 = 500 GB

Assuming drives are $100 then it's the difference of $300 vs $400 which 
is a much better ratio than the shelf example. At this level going with 
better performance for that cost difference makes sense.

This concludes today's lecture in system administration titled 
"Economics: Bane of Performance" :-)

kashani
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] RAID5 vs. RAID0+1

[Christian Spoo]

[-- Attachment #1.1: Type: text/plain, Size: 419 bytes --]

> Assuming drives are $100 then it's the difference of $300 vs $400 which
> is a much better ratio than the shelf example. At this level going with
> better performance for that cost difference makes sense.

I totally agree since the 500GB are enough capacity to save at least the
work of half a year. Older things are backed up and removed from the
server. A whole rack would be real overkill, at least for us.

[-- Attachment #1.2: mail.vcf --]
[-- Type: text/x-vcard, Size: 281 bytes --]

begin:vcard
fn:Christian Spoo
n:Spoo;Christian
adr;quoted-printable:;;Am Kaiser 26;W=C3=BCrselen;NRW;52146;Deutschland
email;internet:mail@christian-spoo.info
tel;fax:02405/475071
tel;home:02405/493466
tel;cell:0176/61055475
x-mozilla-html:FALSE
version:2.1
end:vcard


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 888 bytes --]

Re: [gentoo-server] RAID5 vs. RAID0+1

[Richard Broersma Jr]

I thought it might be nice to recap the characteristics of the possible RAID configurations. 
Please feel free to correct any points that are incorrect or inaccurate.

I was thinking of comparing the characteristics of various RAID configurations using 4 250G disks.

(4) Disk RAID1:

Total Array Capacity:    250G
Reading Throughput:      <= 4 x a single drive read cap.
                         1 file can be read simultaneous 4 times
                         or 4 separate file can be read simultaneously
Write Throughput:        <= a single drive write cap.
Fault tolerance:         can survive 3 disk failures


(4) Disk RAID0:

Total Array Capacity:    1000G
Reading Throughput:      <<= 4 x a single drive read cap.
                         1 file can be read at-most 1 time
                         or 4 separate file can be read simultaneously
Write Throughput:        <= 4 x a single drive write cap.
Fault tolerance:         can survive 0 disk failures


(4) Disk RAID5:

Total Array Capacity:    750G
Reading Throughput:      <= 2 x a single drive read cap. (similar to RAID0)
                         1 file can be read at-most 1 time
                         or 4 separate file can be read simultaneously
Write Throughput:        <= 1 x a single drive write cap.
Fault tolerance:         can survive 1 disk failures


(4) Disk RAID10 --Layout=n(near):
functions similar to a RAID1&0 but more closely
follows the pattern of a RAID1 i.e. better read preformance
than write preformance


Total Array Capacity:    500G
Reading Throughput:      <= 4 x a single drive read cap. (similar to RAID0)
                         1 file can be read at-most 2 times
                         or 4 separate file can be read simultaneously
Write Throughput:        <= 2 x a single drive write cap.
Fault tolerance:         can survive at most 2 disk failures


(4) Disk RAID10 --Layout=o(offset):
functions similar to a RAID1&0 but more closely
follows the pattern of a RAID0 i.e. better write preformance
than read preformance


Total Array Capacity:    500G
Reading Throughput:      <= 4 x a single drive read cap. (similar to RAID0)
                         1 file can be read at-most 2 times
                         or 4 separate file can be read simultaneously
Write Throughput:        <= 2 x a single drive write cap.
Fault tolerance:         can survive at most 2 disk failures


(4) Disk RAID10 --Layout=f(far):
functions similar to a RAID1&0 but more closely
follows the pattern of a RAID1 i.e. better read preformance
than write preformance
I am not sure sure how this differs from 
--Layout=n(near)


Total Array Capacity:    500G
Reading Throughput:      <= 4 x a single drive read cap. (similar to RAID0)
                         1 file can be read at-most 2 times
                         or 4 separate file can be read simultaneously
Write Throughput:        <= 2 x a single drive write cap.
Fault tolerance:         can survive at most 2 disk failures


Regards,

Richard Broersma Jr.
-- 
gentoo-server@gentoo.org mailing list

Re: [gentoo-server] RAID5 vs. RAID0+1

[Francisco Olarte Sanz]

On Saturday 07 October 2006 02:38, Richard Broersma Jr wrote:
> I thought it might be nice to recap the characteristics of the possible
> RAID configurations. Please feel free to correct any points that are
> incorrect or inaccurate.

> (4) Disk RAID5:
>
> Total Array Capacity:    750G
> Reading Throughput:      <= 2 x a single drive read cap. (similar to RAID0)
>                          1 file can be read at-most 1 time
>                          or 4 separate file can be read simultaneously
> Write Throughput:        <= 1 x a single drive write cap.
> Fault tolerance:         can survive 1 disk failures

	I'm not sure of the raid 5 figures. For very big writes, i.e., lot's of full 
stripes, you'll get up to 3x write performance ( if your controller can write 
to all disk in paralell and is smart enough to calculate parity in memory ), 
but for small writes ( i.e, directory entries or DB updates ), you'll get 
about 0.5x ( exact figure is difficult, but for a single sector update a raid 
5 has to make 2 reads ( old sector, old parity ), then recalculate new parity 
as and then two writes AFTER them ( new sector, new parity ). Also read 
performance should be <= 3x if you give <= 4x for raid 0.

F.O.S.
-- 
gentoo-server@gentoo.org mailing list