[gentoo-user] terrible performance with btrfs on LVM2 using a WD 2TB green drive

[gentoo-user] terrible performance with btrfs on LVM2 using a WD 2TB green drive

[Bill Kenworthy]

I have recently added a WD 2TB green drive to two systems and am finding
terrible performance with btrfs on an LVM using these drives.

I just saw on the mythtv list about the sector size problem these drives
have where they have poor performance unless you can map the partitions
onto certain sector boundaries.

My problem is that LVM2 is not supported in parted which is the
recommended tool to deal with this.

How can I map an lvm with 4 or more disks, only one of which is a "WD
Green drive" and containing multiple reiserfs and btrfs file systems for
best performance.  As these drives look useful in a data centre because
of their power savings, I presume someone must have solved this already!

I suspect I only need to map the individual PE to a particular start
sector on each drive, not btrfs, but then there is stripe/block sizes to
consider as well ... WD also are recommending 1mb sector boundaries for
best performance - I can see a reinstall coming up :)

Is there an application that can analyze a disk for settings and
automatically list the best/recommended settings?  This should be
scriptable anyway so I may end up going this way if someone hasn't gone
there first.

Yesterday, a backup using dirvish that used to take 3-5 minutes to a
reiserfs partition on lvm on an older disk took nearly 19 minutes to the
same server, same lvm but on a btrfs partition on the new WD drive the
lvm was extended onto.  Delete performance is even worse  :(

BillK

Re: [gentoo-user] terrible performance with btrfs on LVM2 using a WD 2TB green drive

[Matthew Marlowe]

> My problem is that LVM2 is not supported in parted which is the
> recommended tool to deal with this.
> 
> I suspect I only need to map the individual PE to a particular start
> sector on each drive, not btrfs, but then there is stripe/block sizes to
> consider as well ... WD also are recommending 1mb sector boundaries for
> best performance - I can see a reinstall coming up :)
>
 
I have on my workstation:
    2 WD 2TB Black Drives
    5 WD 2TB RE4 Drives

Some notes:
- The black drives have horrible reliability, poor sector remapping, and have 
certain standard drive features to make them unusable in raid.  I would not 
buy them again. I'm not sure how similar the green drives are.
- Many of the recent WD drives have a tendency to power down/up frequently 
which can reduce drive lifetime (research it and ensure it is set 
appropriately for your needs).
- Due to reliability concerns, you'll may need to run smartd to give adequate 
pre-failure warnings

Anyhow, in my config I have:

1 RE4 Drive as Server Boot Disk
4 RE4 Drives in SW RAID10 (extremely good performance and reliability)
2 Black Drives in LVM RAID0 for disk-to-disk backups (thats about all I trust 
them with).

When I setup the LVM RAID0, I used the following commands to get good 
performance:
      fdisk (remove all partitions, you don't need them for lvm)
     pvcreate --dataalignmentoffset 7s /dev/sdd
     pvcreate --dataalignmentoffset 7s /dev/sdf
     vgcreate -s 64M -M 2 vgArchive /dev/sdd /dev/sdf
     lvcreate -i 2 -l 100%FREE -I 256 -n lvArchive -r auto vgArchive
    mkfs.ext4 -c -b 4096 -E stride=64,stripe_width=128 -j -i 1048576 -L 
/archive /dev/vgArchive/lvArchive

I may have the ext4 stride/stripe settings wrong above, I didn't have my 
normal notes when I selected them - but the rest of the config I scrounged 
from other blogs and seemed to make sense (the --dataalignmentoffset 7s) seems 
to be the key.

My RAID10 drives are configured slightly different w/ 1 partition that starts 
on sector 2048 if I remember and extends to the end of the drive.

The 4 Disk SW RAID10 array gives me 255MB/s reads, 135MB/s block writes, and 
98MB/s rewrites (old test, may need to rerun for latest changes/etc).

LVM 2 Disk RAID0 gives 303MB/s reads, 190MB/s block writes, and 102MB/s 
rewrites (test ran last week).  

Regards,
Matt
-- 
Matthew Marlowe    /  858-400-7430  /    DeployLinux Consulting, Inc
  Professional Linux Hosting and Systems Administration Services
              www.deploylinux.net   *   matt@deploylinux.net
                             'MattM' @ irc.freenode.net
       

Re: [gentoo-user] terrible performance with btrfs on LVM2 using a WD 2TB green drive

[William Kenworthy]

On Mon, 2011-03-14 at 23:50 -0700, Matthew Marlowe wrote:
> > My problem is that LVM2 is not supported in parted which is the
> > recommended tool to deal with this.
> > 
> > I suspect I only need to map the individual PE to a particular start
> > sector on each drive, not btrfs, but then there is stripe/block sizes to
> > consider as well ... WD also are recommending 1mb sector boundaries for
> > best performance - I can see a reinstall coming up :)
> >
>  
> I have on my workstation:
>     2 WD 2TB Black Drives
>     5 WD 2TB RE4 Drives
> 
> Some notes:
> - The black drives have horrible reliability, poor sector remapping, and have 
> certain standard drive features to make them unusable in raid.  I would not 
> buy them again. I'm not sure how similar the green drives are.
> - Many of the recent WD drives have a tendency to power down/up frequently 
> which can reduce drive lifetime (research it and ensure it is set 
> appropriately for your needs).
> - Due to reliability concerns, you'll may need to run smartd to give adequate 
> pre-failure warnings
> 
> Anyhow, in my config I have:
> 
> 1 RE4 Drive as Server Boot Disk
> 4 RE4 Drives in SW RAID10 (extremely good performance and reliability)
> 2 Black Drives in LVM RAID0 for disk-to-disk backups (thats about all I trust 
> them with).
> 
> When I setup the LVM RAID0, I used the following commands to get good 
> performance:
>       fdisk (remove all partitions, you don't need them for lvm)
>      pvcreate --dataalignmentoffset 7s /dev/sdd
>      pvcreate --dataalignmentoffset 7s /dev/sdf
>      vgcreate -s 64M -M 2 vgArchive /dev/sdd /dev/sdf
>      lvcreate -i 2 -l 100%FREE -I 256 -n lvArchive -r auto vgArchive
>     mkfs.ext4 -c -b 4096 -E stride=64,stripe_width=128 -j -i 1048576 -L 
> /archive /dev/vgArchive/lvArchive
> 
> I may have the ext4 stride/stripe settings wrong above, I didn't have my 
> normal notes when I selected them - but the rest of the config I scrounged 
> from other blogs and seemed to make sense (the --dataalignmentoffset 7s) seems 
> to be the key.
> 
> My RAID10 drives are configured slightly different w/ 1 partition that starts 
> on sector 2048 if I remember and extends to the end of the drive.
> 
> The 4 Disk SW RAID10 array gives me 255MB/s reads, 135MB/s block writes, and 
> 98MB/s rewrites (old test, may need to rerun for latest changes/etc).
> 
> LVM 2 Disk RAID0 gives 303MB/s reads, 190MB/s block writes, and 102MB/s 
> rewrites (test ran last week).  
> 
> Regards,
> Matt

Thanks Matthew,
 some good ideas here.  I have other partitions on the disks such as
swap and rescue so LVM doesnt get all the space.  I have steered away
from striping as I have lost an occasional disk over the years and worry
that a stripe will take out a larger block of data than a linear BOD but
your performance numbers look ... great!

As the stripe size is hard to change after creation it looks like I'll
have to migrate the data and recreate from scratch to get the best out
of the hardware.

In the short term, I'll just do some shuffling and delete then readd the
LVM partition on the green drive to the volume group which should
improve the performance a lot.  If I am reading it right, I have to get
the disk partitioning right first, them make sure the PV is also created
at the right boundaries on the LVM.  Then I will see how to tune btrfs
which I am becoming quite sold on - solid, and online fsck is better
than reiserfs which is just as solid, but you have to take offline to
check - not that either corrupt often.

BillK

Re: [gentoo-user] terrible performance with btrfs on LVM2 using a WD 2TB green drive

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 933 bytes --]

Am 15.03.2011 07:50, schrieb Matthew Marlowe:
> 
>> My problem is that LVM2 is not supported in parted which is the
>> recommended tool to deal with this.
>>
>> I suspect I only need to map the individual PE to a particular start
>> sector on each drive, not btrfs, but then there is stripe/block sizes to
>> consider as well ... WD also are recommending 1mb sector boundaries for
>> best performance - I can see a reinstall coming up :)
>>
>  
> I have on my workstation:
>     2 WD 2TB Black Drives
>     5 WD 2TB RE4 Drives
> 
> Some notes:
> - The black drives have horrible reliability, poor sector remapping, and have 
> certain standard drive features to make them unusable in raid.  I would not 
> buy them again. I'm not sure how similar the green drives are.

Green drives also seem to be affected:
http://doug.warner.fm/d/blog/2009/11/Western-Digital-15TB-Green-Drives-Not-your-Linux-Software-RAID


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] terrible performance with btrfs on LVM2 using a WD 2TB green drive

[Joost Roeleveld]

On Wednesday 16 March 2011 09:53:37 Florian Philipp wrote:
> Am 15.03.2011 07:50, schrieb Matthew Marlowe:
> >> My problem is that LVM2 is not supported in parted which is the
> >> recommended tool to deal with this.
> >> 
> >> I suspect I only need to map the individual PE to a particular start
> >> sector on each drive, not btrfs, but then there is stripe/block sizes
> >> to
> >> consider as well ... WD also are recommending 1mb sector boundaries
> >> for
> >> best performance - I can see a reinstall coming up :)
> > 
> > I have on my workstation:
> >     2 WD 2TB Black Drives
> >     5 WD 2TB RE4 Drives
> > 
> > Some notes:
> > - The black drives have horrible reliability, poor sector remapping, and
> > have certain standard drive features to make them unusable in raid.  I
> > would not buy them again. I'm not sure how similar the green drives
> > are.
> 
> Green drives also seem to be affected:
> http://doug.warner.fm/d/blog/2009/11/Western-Digital-15TB-Green-Drives-Not-y
> our-Linux-Software-RAID

I have 6 Green drives (WDC WD15EARS) in a RAID5 and I have not seen any 
issues.
Only issue I saw was 1 drive with "reallocated" sectors as also mentioned by 
one of the commenters on that page.
Replaced that drive and no further problems so far.

I did, however, spent time to align the sectors correctly for the Raid-
partitions, striping, LVM blocksize and the mkfs-statements.
Without those, performance was really bad.

I would prefer to see proper support for 4K-sector-size drives.

--
Joost

[gentoo-user] Re: terrible performance with btrfs on LVM2 using a WD 2TB green drive

[James]

Bill Kenworthy <billk <at> iinet.net.au> writes:


> I have recently added a WD 2TB green drive to two systems and am finding
> terrible performance with btrfs on an LVM using these drives.

Hmmmm, I've been contemplating btrfs on some new installs
with the eventual goal of a CEPH network file system, on top
of ext4 and/or btrfs. Is this issue just with the drivers, or 
related to btrfs too? 

How is btrfs , from the "driver's seat"?

Does this aberration you detail, extend to the Green Seagate 2T drives?

> I just saw on the mythtv list about the sector size problem these drives

Can you give a precise link to this list or can I follow it (read only)
via net news?  (if so what is the news group name)?

I'd like to read up on this issue, as I have several gentoo installs 
(very soon) that will have  RAID1 using (2) ST 2T green drives.

Alternative RAID configs and performance/reliability results is
also of keen interest to many; so thanks for posting about these
issues.


James

Re: [gentoo-user] Re: terrible performance with btrfs on LVM2 using a WD 2TB green drive

[Volker Armin Hemmann]

On Tuesday 15 March 2011 15:26:44 James wrote:
> Bill Kenworthy <billk <at> iinet.net.au> writes:
> > I have recently added a WD 2TB green drive to two systems and am finding
> > terrible performance with btrfs on an LVM using these drives.
> 
> Hmmmm, I've been contemplating btrfs on some new installs
> with the eventual goal of a CEPH network file system, on top
> of ext4 and/or btrfs. Is this issue just with the drivers, or
> related to btrfs too?
> 
> How is btrfs , from the "driver's seat"?
> 
> Does this aberration you detail, extend to the Green Seagate 2T drives?
> 
> > I just saw on the mythtv list about the sector size problem these drives
> 
> Can you give a precise link to this list or can I follow it (read only)
> via net news?  (if so what is the news group name)?
> 
> I'd like to read up on this issue, as I have several gentoo installs
> (very soon) that will have  RAID1 using (2) ST 2T green drives.
> 
> Alternative RAID configs and performance/reliability results is
> also of keen interest to many; so thanks for posting about these
> issues.
> 

just google for 4k sectors and or adf drives. There is plenty of material out 
there.

And you don't need parted. You can do the correct alignment with good old 
fdisk.

Re: [gentoo-user] Re: terrible performance with btrfs on LVM2 using a WD 2TB green drive

[Bill Kenworthy]

On Tue, 2011-03-15 at 15:26 +0000, James wrote:
> Bill Kenworthy <billk <at> iinet.net.au> writes:
> 
> 
> > I have recently added a WD 2TB green drive to two systems and am finding
> > terrible performance with btrfs on an LVM using these drives.
> 
> Hmmmm, I've been contemplating btrfs on some new installs
> with the eventual goal of a CEPH network file system, on top
> of ext4 and/or btrfs. Is this issue just with the drivers, or 
> related to btrfs too? 
> 
same plan as mine :)

> How is btrfs , from the "driver's seat"?
> 
Some early corruption that was due to a failing (old/ancient) HD -
easily fixed using online fsck until I realised it was caused by the
hardware and removed the drive (actually it finally died!).  I have had
one instance of a more serious corruption caused by some bug in btrfs
that needed the latest btrfsck from git to fix offline - later kernels
do not suffer from this apparently.

Otherwise I have been hammering it as a dirvish backup server and except
for speed its fine - and I wont be able to confirm that the speed issue
is btrfs's fault until I get the partitioning correct - and dirvish is
somewhat unique in what it does so it could be a special case.

> Does this aberration you detail, extend to the Green Seagate 2T drives?
> 
No idea :)

> > I just saw on the mythtv list about the sector size problem these drives
> 
> Can you give a precise link to this list or can I follow it (read only)
> via net news?  (if so what is the news group name)?
google for adf and 4k sectors

> 
> I'd like to read up on this issue, as I have several gentoo installs 
> (very soon) that will have  RAID1 using (2) ST 2T green drives.
> 
> Alternative RAID configs and performance/reliability results is
> also of keen interest to many; so thanks for posting about these
> issues.
> 
> 
> James
> 
> 

Read up about raid and the model of green drive you are contemplating -
not all are compatible from what I have read.

Billk

[gentoo-user] Re: terrible performance with btrfs on LVM2 using a WD 2TB green drive

[james]

Bill Kenworthy <billk <at> iinet.net.au> writes:

> > How is btrfs , from the "driver's seat"?

>  I have had  one instance of a more serious corruption caused by 
> some bug in btrfs that needed the latest btrfsck from git to fix 
> offline - later kernels do not suffer from this apparently.

Interesting...
BTRFS


> Otherwise I have been hammering it as a dirvish backup server and except
> for speed its fine - and I wont be able to confirm that the speed issue
> is btrfs's fault until I get the partitioning correct - and dirvish is
> somewhat unique in what it does so it could be a special case.

I found the 4k sectors problem.
Weird.

btrfs should be
fast....


thanks
for all
the responses,


james

Re: [gentoo-user] Re: terrible performance with btrfs on LVM2 using a WD 2TB green drive

[Bill Kenworthy]

On Wed, 2011-03-16 at 08:29 +0800, Bill Kenworthy wrote:
> On Tue, 2011-03-15 at 15:26 +0000, James wrote:
> > Bill Kenworthy <billk <at> iinet.net.au> writes:
> > 
> > 
> > > I have recently added a WD 2TB green drive to two systems and am finding
> > > terrible performance with btrfs on an LVM using these drives.
> > 

Just some feedback now Ive fixed it.  Moved all the valuable data off
the lv, dropped the rest and shrunk the vg off the disk.  Recreated the
partitions using fdisk and restored the lv and btrfs.  Only the initial
partitioning and the pv were set up specifically 4k aware.

As a simple benchmark using dirvish to backup in the same fashion as I
was previously, time went from 18-19 minutes down to just over 3 minutes
- ~6 time speedup, same times as on older disks using reiserfs.  I feel
reiserfs is a little faster at this, but not by much and no solid
figures as too much has changed.

I'm happy!

BillK

Re: [gentoo-user] Re: terrible performance with btrfs on LVM2 using a WD 2TB green drive

[Helmut Jarausch]

On 03/15/2011 04:26:44 PM, James wrote:
> Bill Kenworthy <billk <at> iinet.net.au> writes:
> 
> 
> Hmmmm, I've been contemplating btrfs on some new installs
> with the eventual goal of a CEPH network file system, on top
> of ext4 and/or btrfs. Is this issue just with the drivers, or 
> related to btrfs too? 
> 

But beware of the bug I've reported last Saturday which
is revealed by  /bin/cp  if you have  >= coreutils-8.10
installed.
Only the 2.6.38 kernel (from today) has fixed this for btrfs
but AFAIK not for ext4, yet.

Helmut.


-- 
Helmut Jarausch
Lehrstuhl fuer Numerische Mathematik
RWTH - Aachen University
D 52056 Aachen, Germany

Re: [gentoo-user] terrible performance with btrfs on LVM2 using a WD 2TB green drive

[Volker Armin Hemmann]

On Tuesday 15 March 2011 13:37:46 Bill Kenworthy wrote:
> I have recently added a WD 2TB green drive to two systems and am finding
> terrible performance with btrfs on an LVM using these drives.
> 
> I just saw on the mythtv list about the sector size problem these drives
> have where they have poor performance unless you can map the partitions
> onto certain sector boundaries.
> 
> My problem is that LVM2 is not supported in parted which is the
> recommended tool to deal with this.
> 
use google. fdisk is fine.