[gentoo-cluster] Performance problems with domU on drbd

[Reto Gantenbein <reto.gantenbein@id.unibe.ch>]

Hello everybody

After 3 days messing around I'm at my wits' end. I have two Sun X4200M2
servers which I would like to setup as Xen dom0s with a drbd-mirrored
file system for paravirtualized domU images.

I setup an LVM2 on top of DRBD to create the domU volumes and to be able
to backup the domUs with LVM snapshot.

I was able to configure this setup and make it run, but when accessing
the block-frontend device in domU this is terrible slow. E.g. copying a
5G file via scp and gigabit Ethernet onto the domU leads to a lot of
stalls during transfer. The peak transfer rate is about 5MB/s every 1-2
minutes but generally it's about a few hundred KB/s. When copying the
same file onto the DRBD volume mounted directly in dom0 it works with
the expected 30-40MB/s. Another test was to sync the portage tree but
also this takes maybe about 20 minutes or spontaneously raises some
timeout errors. Sometimes after a while the disk fully detaches from the
domU, so every call of a binary in domU leads to a "Input/output error".
I guess the domU kernel adds this troubles, but why? There are no
statements in any log files. E.g. /var/log/xen/xend-debug.log doesn't
notify the detached block device.

I use Gentoo Linux and tried so far with xen-3.1.2 and xen-3.2.0,
xen-sources-2.6.18-r8, xen-sources-2.6.20-r6 and xen-sources-2.6.22,
drbd-8.0.7, drbd-8.0.8 and drbd-8.0.11.

The longer the more I expect a general problem with Xen and this setup,
a kernel problem or a misconfiguration. Attached are the most important
configurations for this setup.

Are there any hints or other experiences for such a setup? All ideas are
welcome.

Thanks for reading and giving any hints...
Cheers Reto Gantenbein


################ Configurations #################################

/etc/drbd.conf
,----------------------------------------------------------------
| global {
|     usage-count no;
|     dialog-refresh 1;
| }
|
| common {
|     startup {
|         wfc-timeout 60;
|         degr-wfc-timeout 60;
|     }
|
|     disk {
|	  on-io-error detach;
|         # also tried 'pass-on' with same behavior
|     }
|
|     syncer {
|         rate 500M;
|         al-extents 80;
|     }
| }
|
| resource "gentoo" {
|     protocol C;
|     net {
|         # required for xen live migration
|         allow-two-primaries;
|	  after-sb-0pri discard-zero-changes;
|  	  after-sb-1pri discard-secondary;
|    }
|
|    on vserver03 {
|	  device 	/dev/drbd1;
|      	  disk 		/dev/xen-vg/gentoo;
|         address 	10.0.0.1:7789;
| 	  meta-disk 	internal;
|    }
|
|    on vserver04 {
|    	 device 	/dev/drbd1;
| 	 disk 		/dev/xen-vg/gentoo;
|	 address 	10.0.0.2:7789;
|	 meta-disk 	internal;
|    }
| }
`----------------------------------------------------------------

/etc/xen/gentoo
,----------------------------------------------------------------
| name = "gentoo"
|
| kernel = "/cluster/xen/kernel/vmlinuz-gentoo-domU"
| extra = ""
| root = "/dev/hda1"
|
| memory = 4096
| vcpus = 3
| disk = [ 'drbd:gentoo,hda1,w' ]
| vif = [ 'mac=00:40:a6:00:45:01,bridge=xenbr-core',
|         'mac=00:40:a6:00:45:02,bridge=xenbr-public' ]
`----------------------------------------------------------------
-- 
gentoo-cluster@lists.gentoo.org mailing list