From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1PzRQ3-0004Un-1V for garchives@archives.gentoo.org; Tue, 15 Mar 2011 10:24:03 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id BFA08E0123; Tue, 15 Mar 2011 10:22:27 +0000 (UTC) Received: from outbound.icp-qv1-irony-out4.iinet.net.au (outbound.icp-qv1-irony-out4.iinet.net.au [203.59.1.104]) by pigeon.gentoo.org (Postfix) with ESMTP id 14B91E027D for ; Tue, 15 Mar 2011 10:22:26 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvsEAB/bfk18qVOk/2dsb2JhbACmDnfCaoViBIUuin4a X-IronPort-AV: E=Sophos;i="4.62,321,1297008000"; d="scan'208";a="642239487" Received: from unknown (HELO moriah.localdomain) ([124.169.83.164]) by outbound.icp-qv1-irony-out4.iinet.net.au with ESMTP; 15 Mar 2011 18:22:16 +0800 Received: from localhost (localhost [127.0.0.1]) by moriah.localdomain (Postfix) with ESMTP id 071681FEDAF4 for ; Tue, 15 Mar 2011 18:22:16 +0800 (WST) X-Virus-Scanned: amavisd-new at localdomain Received: from moriah.localdomain ([127.0.0.1]) by localhost (moriah.localdomain [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YaRzFUGcNuH9 for ; Tue, 15 Mar 2011 18:22:07 +0800 (WST) Received: from [192.168.44.2] (rattus [192.168.44.2]) by moriah.localdomain (Postfix) with ESMTP id 7ED671F5B81E for ; Tue, 15 Mar 2011 18:22:07 +0800 (WST) Subject: Re: [gentoo-user] terrible performance with btrfs on LVM2 using a WD 2TB green drive From: William Kenworthy To: gentoo-user@lists.gentoo.org In-Reply-To: <201103142350.49670.matt@deploylinux.net> References: <1300167466.30677.32.camel@troll> <201103142350.49670.matt@deploylinux.net> Content-Type: text/plain; charset="ISO-8859-1" Organization: Home in Perth! Date: Tue, 15 Mar 2011 18:22:06 +0800 Message-ID: <1300184527.12835.14.camel@rattus> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org Mime-Version: 1.0 X-Mailer: Evolution 2.32.2 Content-Transfer-Encoding: 7bit X-Archives-Salt: X-Archives-Hash: 4eb6742279bf4690270d5f41c9af08cc On Mon, 2011-03-14 at 23:50 -0700, Matthew Marlowe wrote: > > My problem is that LVM2 is not supported in parted which is the > > recommended tool to deal with this. > > > > I suspect I only need to map the individual PE to a particular start > > sector on each drive, not btrfs, but then there is stripe/block sizes to > > consider as well ... WD also are recommending 1mb sector boundaries for > > best performance - I can see a reinstall coming up :) > > > > I have on my workstation: > 2 WD 2TB Black Drives > 5 WD 2TB RE4 Drives > > Some notes: > - The black drives have horrible reliability, poor sector remapping, and have > certain standard drive features to make them unusable in raid. I would not > buy them again. I'm not sure how similar the green drives are. > - Many of the recent WD drives have a tendency to power down/up frequently > which can reduce drive lifetime (research it and ensure it is set > appropriately for your needs). > - Due to reliability concerns, you'll may need to run smartd to give adequate > pre-failure warnings > > Anyhow, in my config I have: > > 1 RE4 Drive as Server Boot Disk > 4 RE4 Drives in SW RAID10 (extremely good performance and reliability) > 2 Black Drives in LVM RAID0 for disk-to-disk backups (thats about all I trust > them with). > > When I setup the LVM RAID0, I used the following commands to get good > performance: > fdisk (remove all partitions, you don't need them for lvm) > pvcreate --dataalignmentoffset 7s /dev/sdd > pvcreate --dataalignmentoffset 7s /dev/sdf > vgcreate -s 64M -M 2 vgArchive /dev/sdd /dev/sdf > lvcreate -i 2 -l 100%FREE -I 256 -n lvArchive -r auto vgArchive > mkfs.ext4 -c -b 4096 -E stride=64,stripe_width=128 -j -i 1048576 -L > /archive /dev/vgArchive/lvArchive > > I may have the ext4 stride/stripe settings wrong above, I didn't have my > normal notes when I selected them - but the rest of the config I scrounged > from other blogs and seemed to make sense (the --dataalignmentoffset 7s) seems > to be the key. > > My RAID10 drives are configured slightly different w/ 1 partition that starts > on sector 2048 if I remember and extends to the end of the drive. > > The 4 Disk SW RAID10 array gives me 255MB/s reads, 135MB/s block writes, and > 98MB/s rewrites (old test, may need to rerun for latest changes/etc). > > LVM 2 Disk RAID0 gives 303MB/s reads, 190MB/s block writes, and 102MB/s > rewrites (test ran last week). > > Regards, > Matt Thanks Matthew, some good ideas here. I have other partitions on the disks such as swap and rescue so LVM doesnt get all the space. I have steered away from striping as I have lost an occasional disk over the years and worry that a stripe will take out a larger block of data than a linear BOD but your performance numbers look ... great! As the stripe size is hard to change after creation it looks like I'll have to migrate the data and recreate from scratch to get the best out of the hardware. In the short term, I'll just do some shuffling and delete then readd the LVM partition on the green drive to the volume group which should improve the performance a lot. If I am reading it right, I have to get the disk partitioning right first, them make sure the PV is also created at the right boundaries on the LVM. Then I will see how to tune btrfs which I am becoming quite sold on - solid, and online fsck is better than reiserfs which is just as solid, but you have to take offline to check - not that either corrupt often. BillK