[gentoo-cluster] Performance problems with domU on drbd

[gentoo-cluster] Performance problems with domU on drbd

[Reto Gantenbein]

Hello everybody

After 3 days messing around I'm at my wits' end. I have two Sun X4200M2
servers which I would like to setup as Xen dom0s with a drbd-mirrored
file system for paravirtualized domU images.

I setup an LVM2 on top of DRBD to create the domU volumes and to be able
to backup the domUs with LVM snapshot.

I was able to configure this setup and make it run, but when accessing
the block-frontend device in domU this is terrible slow. E.g. copying a
5G file via scp and gigabit Ethernet onto the domU leads to a lot of
stalls during transfer. The peak transfer rate is about 5MB/s every 1-2
minutes but generally it's about a few hundred KB/s. When copying the
same file onto the DRBD volume mounted directly in dom0 it works with
the expected 30-40MB/s. Another test was to sync the portage tree but
also this takes maybe about 20 minutes or spontaneously raises some
timeout errors. Sometimes after a while the disk fully detaches from the
domU, so every call of a binary in domU leads to a "Input/output error".
I guess the domU kernel adds this troubles, but why? There are no
statements in any log files. E.g. /var/log/xen/xend-debug.log doesn't
notify the detached block device.

I use Gentoo Linux and tried so far with xen-3.1.2 and xen-3.2.0,
xen-sources-2.6.18-r8, xen-sources-2.6.20-r6 and xen-sources-2.6.22,
drbd-8.0.7, drbd-8.0.8 and drbd-8.0.11.

The longer the more I expect a general problem with Xen and this setup,
a kernel problem or a misconfiguration. Attached are the most important
configurations for this setup.

Are there any hints or other experiences for such a setup? All ideas are
welcome.

Thanks for reading and giving any hints...
Cheers Reto Gantenbein


################ Configurations #################################

/etc/drbd.conf
,----------------------------------------------------------------
| global {
|     usage-count no;
|     dialog-refresh 1;
| }
|
| common {
|     startup {
|         wfc-timeout 60;
|         degr-wfc-timeout 60;
|     }
|
|     disk {
|	  on-io-error detach;
|         # also tried 'pass-on' with same behavior
|     }
|
|     syncer {
|         rate 500M;
|         al-extents 80;
|     }
| }
|
| resource "gentoo" {
|     protocol C;
|     net {
|         # required for xen live migration
|         allow-two-primaries;
|	  after-sb-0pri discard-zero-changes;
|  	  after-sb-1pri discard-secondary;
|    }
|
|    on vserver03 {
|	  device 	/dev/drbd1;
|      	  disk 		/dev/xen-vg/gentoo;
|         address 	10.0.0.1:7789;
| 	  meta-disk 	internal;
|    }
|
|    on vserver04 {
|    	 device 	/dev/drbd1;
| 	 disk 		/dev/xen-vg/gentoo;
|	 address 	10.0.0.2:7789;
|	 meta-disk 	internal;
|    }
| }
`----------------------------------------------------------------

/etc/xen/gentoo
,----------------------------------------------------------------
| name = "gentoo"
|
| kernel = "/cluster/xen/kernel/vmlinuz-gentoo-domU"
| extra = ""
| root = "/dev/hda1"
|
| memory = 4096
| vcpus = 3
| disk = [ 'drbd:gentoo,hda1,w' ]
| vif = [ 'mac=00:40:a6:00:45:01,bridge=xenbr-core',
|         'mac=00:40:a6:00:45:02,bridge=xenbr-public' ]
`----------------------------------------------------------------
-- 
gentoo-cluster@lists.gentoo.org mailing list

Re: [gentoo-cluster] Performance problems with domU on drbd

[Michael Marineau]

On Fri, Feb 15, 2008 at 7:38 AM, Reto Gantenbein
<reto.gantenbein@id.unibe.ch> wrote:
> Hello everybody
>
>  After 3 days messing around I'm at my wits' end. I have two Sun X4200M2
>  servers which I would like to setup as Xen dom0s with a drbd-mirrored
>  file system for paravirtualized domU images.
>
>  I setup an LVM2 on top of DRBD to create the domU volumes and to be able
>  to backup the domUs with LVM snapshot.
>
>  I was able to configure this setup and make it run, but when accessing
>  the block-frontend device in domU this is terrible slow. E.g. copying a
>  5G file via scp and gigabit Ethernet onto the domU leads to a lot of
>  stalls during transfer. The peak transfer rate is about 5MB/s every 1-2
>  minutes but generally it's about a few hundred KB/s. When copying the
>  same file onto the DRBD volume mounted directly in dom0 it works with
>  the expected 30-40MB/s. Another test was to sync the portage tree but
>  also this takes maybe about 20 minutes or spontaneously raises some
>  timeout errors. Sometimes after a while the disk fully detaches from the
>  domU, so every call of a binary in domU leads to a "Input/output error".
>  I guess the domU kernel adds this troubles, but why? There are no
>  statements in any log files. E.g. /var/log/xen/xend-debug.log doesn't
>  notify the detached block device.
>
>  I use Gentoo Linux and tried so far with xen-3.1.2 and xen-3.2.0,
>  xen-sources-2.6.18-r8, xen-sources-2.6.20-r6 and xen-sources-2.6.22,
>  drbd-8.0.7, drbd-8.0.8 and drbd-8.0.11.
>
>  The longer the more I expect a general problem with Xen and this setup,
>  a kernel problem or a misconfiguration. Attached are the most important
>  configurations for this setup.
>
>  Are there any hints or other experiences for such a setup? All ideas are
>  welcome.
>
>  Thanks for reading and giving any hints...
>  Cheers Reto Gantenbein

I couldn't quite tell from the above but do you see the bad
performance inside domU only when using scp/rsync or is it just as
slow when you copy files locally?

-- 
Michael Marineau
Oregon State University
mike@marineau.org
-- 
gentoo-cluster@lists.gentoo.org mailing list

Re: [gentoo-cluster] Performance problems with domU on drbd

[Reto Gantenbein]

Hi Michael

Thanks for your answer. I was mainly checking the stability with copying
from a remote location. A short dd for filling the /tmp partitions with
zeros didn't show up any problems.

After changing some drbd parameters i did start a update world in domU
on Friday. It crashed after a few hours with XFS errors. Now I'm doing a
clean install of a virtual machine with ext3 as file system. I will
report again later how it emerged.

Cheers
Reto


Michael Marineau wrote:
> 
> I couldn't quite tell from the above but do you see the bad
> performance inside domU only when using scp/rsync or is it just as
> slow when you copy files locally?
> 
-- 
gentoo-cluster@lists.gentoo.org mailing list

Re: [gentoo-cluster] Performance problems with domU on drbd

[Reto Gantenbein]

Hi

I solved the problem. It was neither xen nor drbd, but only my
stupidity. You were right when pointing to the direction of a networking
problem. The thing that happened was, I made a test-server, copied the
working domU to it and forgot to change the mac-address of the domU.
Therefore I had all the time two machines with same mac-address what
lead to the poor network performance and corrupted transfers.

Thanks again for replying.

Cheers
Reto


Michael Marineau wrote:
> 
> I couldn't quite tell from the above but do you see the bad
> performance inside domU only when using scp/rsync or is it just as
> slow when you copy files locally?
> 
-- 
gentoo-cluster@lists.gentoo.org mailing list