From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 4F6011381F3 for ; Sat, 22 Jun 2013 23:18:04 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 0AD85E0A43; Sat, 22 Jun 2013 23:17:59 +0000 (UTC) Received: from mail-ve0-f171.google.com (mail-ve0-f171.google.com [209.85.128.171]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 32658E0A10 for ; Sat, 22 Jun 2013 23:17:57 +0000 (UTC) Received: by mail-ve0-f171.google.com with SMTP id b10so7599691vea.30 for ; Sat, 22 Jun 2013 16:17:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=gN+i6OUoXgbQ+hgNEBAUY96flpotLmsSIWF1MrKLGy8=; b=Bc16JU6utMhcP7HEL9q5Sb8AbVJgbgtAMcxKFm4K+HppvUR6d9xXj97XhNY3jHpxAM 8n2PXriq6HCrko4z3YQgq2qm+n5JPJACdY07lTb2BWi/HIAB/VccTu7qDClUl8zRGzTK DBNgj3N/e//Cr4wtHNgMihVXNHrZWC2qSSU9edHIRohv2R8wXDyiwxDmViGMC7o1IU6d wgFlyg0hgfc+3gWpBv32zkkU96u+s83Stc0b/vsrQZiinl0ItVth2aNWj4LEI5ST57Ic YMONm6W94gwUP1DvhUhUzap9wXFB+RdcGKPgohRExKtCPyj+ES/FKpYplZnsdgy5NAjR Bxgw== Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-amd64@lists.gentoo.org Reply-to: gentoo-amd64@lists.gentoo.org MIME-Version: 1.0 X-Received: by 10.220.104.135 with SMTP id p7mr8366080vco.10.1371943077191; Sat, 22 Jun 2013 16:17:57 -0700 (PDT) Received: by 10.58.161.237 with HTTP; Sat, 22 Jun 2013 16:17:57 -0700 (PDT) Received: by 10.58.161.237 with HTTP; Sat, 22 Jun 2013 16:17:57 -0700 (PDT) In-Reply-To: References: Date: Sat, 22 Jun 2013 16:17:57 -0700 Message-ID: Subject: Re: [gentoo-amd64] Re: Is my RAID performance bad possibly due to starting sector value? From: Matthew Marlowe To: gentoo-amd64@lists.gentoo.org Content-Type: multipart/alternative; boundary=047d7b3436bac4b37104dfc6631e X-Gm-Message-State: ALoCoQmO/1s+B7dsUKLEkBFEFhhmY6STNF5hI9xiNP0xwn/EBXz6pRH/l4dQ0tC9cOIa3cb1h4I6 X-Archives-Salt: e1780674-82f8-4c13-962b-1f2fa45ac07f X-Archives-Hash: 35a03cfb077b83f946208ac40e4eea0b --047d7b3436bac4b37104dfc6631e Content-Type: text/plain; charset=ISO-8859-1 I would recommend that anyone concerned about mdadm software raid performance on gentoo test via tools like bonnie++ before putting any data on the drives and separate from data into different sets/volumes. I did testing two years ago watching read, write burst and sustained rates, file ops per second, etc.... Ended up getting 7 2tb enterprise data drives Disk 1 is os, no raid Disk 2-5 are data, raid 10 Disk 6-7 are backups and to test/scratch space, raid 0 On Jun 22, 2013 4:04 PM, "Mark Knecht" wrote: > On Fri, Jun 21, 2013 at 3:28 AM, Rich Freeman wrote: > > On Fri, Jun 21, 2013 at 3:31 AM, Duncan <1i5t5.duncan@cox.net> wrote: > >> So with 4k block sizes on a 5-device raid6, you'd have 20k stripes, 12k > >> in data across three devices, and 8k of parity across the other two > >> devices. > > > > With mdadm on a 5-device raid6 with 512K chunks you have 1.5M in a > > stripe, not 20k. If you modify one block it needs to read all 1.5M, > > or it needs to read at least the old chunk on the single drive to be > > modified and both old parity chunks (which on such a small array is 3 > > disks either way). > > > > Hi Rich, > I've been rereading everyone's posts as well as trying to collect > my own thoughts. One question I have at this point, being that you and > I seem to be the two non-RAID1 users (but not necessarily devotees) at > this time, is what chunk size, stride & stripe width with you are > using? Are you currently using 512K chunks on your RAID5? If so that's > potentially quite different than my 16K chunk RAID6. The more I read > through this thread and other things on the web the more I am > concerned that 16K chunks has possibly forced far more IO operations > that really makes sense for performance. Unfortunately there's no easy > way to me to really test this right now as the RAID6 uses the whole > drive. However for every 512K I want to get off the drive you might > need 1 chuck whereas I'm going to need what, 32 chunks? That's got to > be a lot more IO operations on my machine isn't it? > > For clarity, I'm a 16K chunk, stride of 4K, stripe of 12K: > > c2RAID6 ~ # tune2fs -l /dev/md3 | grep RAID > Filesystem volume name: RAID6root > RAID stride: 4 > RAID stripe width: 12 > c2RAID6 ~ # > > c2RAID6 ~ # cat /proc/mdstat > Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] > md3 : active raid6 sdb3[9] sdf3[5] sde3[6] sdd3[7] sdc3[8] > 1452264480 blocks super 1.2 level 6, 16k chunk, algorithm 2 [5/5] > [UUUUU] > > unused devices: > c2RAID6 ~ # > > As I understand one of your earlier responses I think you are using > 4K sector drives, which again has that extra level of complexity in > terms of creating the partitions initially, but after that should be > fairly straight forward to use. (I think) That said there are > trade-offs between RAID5 & RAID6 but have you measured speeds using > anything like the dd method I posted yesterday, or any other way that > we could compare? > > As I think Duncan asked about storage usage requirements in another > part of this thread I'll just document it here. The machine serves > main 3 purposes for me: > > 1) It's my day in, day out desktop. I run almostly totally Gentoo > 64-bit stable unless I need to keyword a package to get what I need. > Over time I tend to let my keyworded packages go stable if they are > working for me. The overall storage requirements for this, including > my home directory, typically don't run over 50GB. > > 2) The machine runs 3 Windows VMs every day - 2 Win 7 & 1 Win XP. > Total storage for the basic VMs is about 150GB. XP is just for things > like NetFlix. These 3 VMs typically have allocated 9 cores allocated > to them (6+2+1) leaving 3 for Gentoo to run the hardware, etc. The 6 > core VM is often using 80-100% of its CPUs sustained for times. (hours > to days.) It's doing a lot of stock market math... > > 3) More recently, and really the reason to consolidate into a single > RAID of any type, I have about 900GB of mp4s which has been on an > external USB drive, and backed up to a second USB drive. However this > is mostly storage. We watch most of this video on the TV using the > second copy drive hooked directly to the TV or copied onto Kindles. > I've been having to keep multiple backups of this outside the machine > (poor man's RAID1 - two separate USB drives hooked up one at a time!) > ;-) I'd rather just keep it safe on the RAID 6, That said, I've not > yet put it on the RAID6 as I have these performance issues I'd like to > solve first. (If possible. Duncan is making me worry that they cannot > be solved...) > > Lastly, even if I completely buy into Duncan's well formed reasons > about why RAID1 might be faster, using 500GB drives I see no single > RAID solution for me other than RAID5/6. The real RAID1/RAID6 > comparison from a storage standpoint would be a (conceptual) 3-drive > RAID6 vs 3 drive RAID1. Both create 500GB of storage and can > (conceptually) lose 2 drives and still recover data. However adding > another drive to the RAID1 gains you more speed but no storage (buying > into Duncan's points) vs adding storage to the RAID6 and probably > reducing speed. As I need storage what other choices do I have? > > Answering myself, take the 5 drives, create two RAIDS - a 500GB > 2-drive RAID1 for the system + VMs, and then a 3-drive RAID5 for video > data maybe? I don't know... > > Or buy more hardware and do a 2 drive SSD RAID1 for the system, or > a hardware RAID controller, etc. The options explode if I start buying > more hardware. > > Also, THANKS TO EVERYONE for the continued conversation. > > Cheers, > Mark > > --047d7b3436bac4b37104dfc6631e Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

I would recommend that anyone concerned about mdadm software= raid performance on gentoo test via tools like bonnie++ before putting any= data on the drives and separate from data into different sets/volumes.

I did testing two years ago watching read, write burst and s= ustained rates, file ops per second, etc.... Ended up getting 7 2tb enterpr= ise data drives
Disk 1 is os, no raid
Disk 2-5 are data, raid 10
Disk 6-7 are backups and to test/scratch space, raid 0

On Jun 22, 2013 4:04 PM, "Mark Knecht"= <markknecht@gmail.com> w= rote:
On Fri, Jun 21, 2013 at 3:28 AM, Rich Freeman <rich0@gentoo.org> wrote:
> On Fri, Jun 21, 2013 at 3:31 AM, Duncan <1i5t5.duncan@cox.net> wrote:
>> So with 4k block sizes on a 5-device raid6, you'd have 20k str= ipes, 12k
>> in data across three devices, and 8k of parity across the other tw= o
>> devices.
>
> With mdadm on a 5-device raid6 with 512K chunks you have 1.5M in a
> stripe, not 20k. =A0If you modify one block it needs to read all 1.5M,=
> or it needs to read at least the old chunk on the single drive to be > modified and both old parity chunks (which on such a small array is 3<= br> > disks either way).
>

Hi Rich,
=A0 =A0I've been rereading everyone's posts as well as trying to co= llect
my own thoughts. One question I have at this point, being that you and
I seem to be the two non-RAID1 users (but not necessarily devotees) at
this time, is what chunk size, stride & stripe width with you are
using? Are you currently using 512K chunks on your RAID5? If so that's<= br> potentially quite different than my 16K chunk RAID6. The more I read
through this thread and other things on the web the more I am
concerned that 16K chunks has possibly forced far more IO operations
that really makes sense for performance. Unfortunately there's no easy<= br> way to me to really test this right now as the RAID6 uses the whole
drive. However for every 512K I want to get off the drive you might
need 1 chuck whereas I'm going to need what, 32 chunks? That's got = to
be a lot more IO operations on my machine isn't it?

=A0 =A0For clarity, I'm a 16K chunk, stride of 4K, stripe of 12K:

c2RAID6 ~ # tune2fs -l /dev/md3 | grep RAID
Filesystem volume name: =A0 RAID6root
RAID stride: =A0 =A0 =A0 =A0 =A0 =A0 =A04
RAID stripe width: =A0 =A0 =A0 =A012
c2RAID6 ~ #

c2RAID6 ~ # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] md3 : active raid6 sdb3[9] sdf3[5] sde3[6] sdd3[7] sdc3[8]
=A0 =A0 =A0 1452264480 blocks super 1.2 level 6, 16k chunk, algorithm 2 [5/= 5] [UUUUU]

unused devices: <none>
c2RAID6 ~ #

=A0 =A0As I understand one of your earlier responses I think you are using<= br> 4K sector drives, which again has that extra level of complexity in
terms of creating the partitions initially, but after that should be
fairly straight forward to use. (I think) That said there are
trade-offs between RAID5 & RAID6 but have you measured speeds using
anything like the dd method I posted yesterday, or any other way that
we could compare?

=A0 =A0As I think Duncan asked about storage usage requirements in another<= br> part of this thread I'll just document it here. The machine serves
main 3 purposes for me:

1) It's my day in, day out desktop. I run almostly totally Gentoo
64-bit stable unless I need to keyword a package to get what I need.
Over time I tend to let my keyworded packages go stable if they are
working for me. The overall storage requirements for this, including
my home directory, typically don't run over 50GB.

2) The machine runs 3 Windows VMs every day - 2 Win 7 & 1 Win XP.
Total storage for the basic VMs is about 150GB. XP is just for things
like NetFlix. These 3 VMs typically have allocated 9 cores allocated
to them (6+2+1) leaving 3 for Gentoo to run the hardware, etc. The 6
core VM is often using 80-100% of its CPUs sustained for times. (hours
to days.) It's doing a lot of stock market math...

3) More recently, and really the reason to consolidate into a single
RAID of any type, I have about 900GB of mp4s which has been on an
external USB drive, and backed up to a second USB drive. However this
is mostly storage. We watch most of this video on the TV using the
second copy drive hooked directly to the TV or copied onto Kindles.
I've been having to keep multiple backups of this outside the machine (poor man's RAID1 - two separate USB drives hooked up one at a time!) ;-) I'd rather just keep it safe on the RAID 6, That said, I've not=
yet put it on the RAID6 as I have these performance issues I'd like to<= br> solve first. (If possible. Duncan is making me worry that they cannot
be solved...)

=A0 =A0Lastly, even if I completely buy into Duncan's well formed reaso= ns
about why RAID1 might be faster, using 500GB drives I see no single
RAID solution for me other than RAID5/6. The real RAID1/RAID6
comparison from a storage standpoint would be a (conceptual) 3-drive
RAID6 vs 3 drive RAID1. Both create 500GB of storage and can
(conceptually) lose 2 drives and still recover data. However adding
another drive to the RAID1 gains you more speed but no storage (buying
into Duncan's points) vs adding storage to the RAID6 and probably
reducing speed. As I need storage what other choices do I have?

=A0 =A0Answering myself, take the 5 drives, create two RAIDS - a 500GB
2-drive RAID1 for the system + VMs, and then a 3-drive RAID5 for video
data maybe? I don't know...

=A0 =A0Or buy more hardware and do a 2 drive SSD RAID1 for the system, or a hardware RAID controller, etc. The options explode if I start buying
more hardware.

=A0 =A0Also, THANKS TO EVERYONE for the continued conversation.

Cheers,
Mark

--047d7b3436bac4b37104dfc6631e--