[gentoo-user] NFS Server Tuning

[gentoo-user] NFS Server Tuning

[Stefan Onken]

Hello,

I am running a x86 gentoo box as a nfs server. As a filesystem I am 
using XFS on a 3ware Raid system. 

The 3ware systems seems to be quite quick, although access via NFS 
seems to be very slow. Any ideas how I can improve speed ? I was 
expected a speed only limited by the 100 Mbit network. Now, the 
speed is so slow, that reading/writing at the same time is nearly 
impossible.

Any ideas how to improve the speed ?

hdparm on the gentoo box machine:
backup3 ~ # hdparm -tT /dev/sdb
/dev/sdb:
Timing cached reads:   4108 MB in  2.00 seconds = 2054.34 MB/sec
Timing buffered disk reads:  232 MB in  3.01 seconds =  77.09 MB/sec

/etc/exports on the gentoo box:
/mnt/backup/ 192.168.1.0/24
(rw,no_root_squash,async,no_subtree_check)

I am not even able to run a bonnie benchmark on the client. It seems 
to hang while doing rewriting. Running bonnie on the nfs server 
gives a fast result.

/etc/fstab on the client: 
192.168.1.3:/mnt/backup         /mnt/backup     nfs     
rw,users,async   0

http://pastebin.com/m72ae9d47

Any ideas ?

-- 
www.stonki.de:    the more I see, the more I know.......
www.proftpd.de:   Deutsche ProFTPD Dokumentation
www.krename.net:  Der Batch Renamer für KDE
www.kbarcode.net: Die Barcode Solution für KDE
--
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] NFS Server Tuning

[Arttu V.]

On 1/24/08, Stefan Onken <Support@stonki.de> wrote:
>
> Any ideas how to improve the speed ?

noatime?

-- 
Arttu V.
-- 
gentoo-user@lists.gentoo.org mailing list

RE: [gentoo-user] NFS Server Tuning

[Marzan, Richard non Unisys]

Man fstab & man nfs{d} &  man mount and search for sync write options. Wsize and rsize=8192 and wsize=8192 might work but I think they are deprecated. Althought, async or sync might still be used. Use these options when mouting your nfs share or make them permanent in your fstab file. Gentoo-wiki has a good guide on this. I would have given you a good set of options but I'm not able to access the inet. 
> -----Original Message-----
> From: Stefan Onken [mailto:Support@stonki.de]
> Sent: Thursday, January 24, 2008 2:18 PM
> To: gentoo-user@lists.gentoo.org
> Subject: [gentoo-user] NFS Server Tuning
> 
> Hello,
> 
> I am running a x86 gentoo box as a nfs server. As a filesystem I am
> using XFS on a 3ware Raid system.
> 
> The 3ware systems seems to be quite quick, although access via NFS
> seems to be very slow. Any ideas how I can improve speed ? I was
> expected a speed only limited by the 100 Mbit network. Now, the
> speed is so slow, that reading/writing at the same time is nearly
> impossible.
> 
> Any ideas how to improve the speed ?
> 
> hdparm on the gentoo box machine:
> backup3 ~ # hdparm -tT /dev/sdb
> /dev/sdb:
> Timing cached reads:   4108 MB in  2.00 seconds = 2054.34 MB/sec
> Timing buffered disk reads:  232 MB in  3.01 seconds =  77.09 MB/sec
> 
> /etc/exports on the gentoo box:
> /mnt/backup/ 192.168.1.0/24
> (rw,no_root_squash,async,no_subtree_check)
> 
> I am not even able to run a bonnie benchmark on the client. It seems
> to hang while doing rewriting. Running bonnie on the nfs server
> gives a fast result.
> 
> /etc/fstab on the client:
> 192.168.1.3:/mnt/backup         /mnt/backup     nfs
> rw,users,async   0
> 
> http://pastebin.com/m72ae9d47
> 
> Any ideas ?
> 
> --
> www.stonki.de:    the more I see, the more I know.......
> www.proftpd.de:   Deutsche ProFTPD Dokumentation
> www.krename.net:  Der Batch Renamer für KDE
> www.kbarcode.net: Die Barcode Solution für KDE
> --
> gentoo-user@lists.gentoo.org mailing list

--
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] NFS Server Tuning

[Dan Farrell]

On Thu, 24 Jan 2008 22:58:54 +0200
"Arttu V." <arttuv69@gmail.com> wrote:

> On 1/24/08, Stefan Onken <Support@stonki.de> wrote:
> >
> > Any ideas how to improve the speed ?
> 
> noatime?
> 

I wouldn't expect that to help too much.  Async is the #1 speed
improvement on my network; I get disc access speeds of 11.5 mb/s on
mine, which effectively maxes out the network.    In /etc/exports on
the server I have: 

/mnt/storage 192.168.0.0/16(rw,async,no_subtree_check,no_root_squash)

and as the client (from `mount`): 

nfs:/mnt/storage on /home/media/storage type
	nfs(rw,rsize=65536,wsize=65536,soft,timeo=300,addr=192.168.1.88)

/etc/fstab on the client looks like: 

nfs:/mnt/storage /home/media/storage    nfs
		rsize=65536,wsize=65536,rw,async,soft,timeo=300 0 0


Of these options, rsize,wsize,and async are reputed to effect
performance.  However, I do not see much of an effect between different
rsize and wsize settings.  I believe that over an uncongested 100T
network it probably doesn't matter too much what rsize and wsize are.  
On a different share (same server) mounted async without [r|w]size set,
performance (write, this time) was 11.2mb/s, roughly the same.
Furthermore, I'm not sure these values are even valid.
http://www.linuxdocs.org/HOWTOs/NFS-HOWTO/performance.html said that
nfs3 goes only to 32768.  

wdelay and no_wdelay might have an effect, depending on your
application.  I no longer tweak those values.

There are some network performance tweaks as well; their effect wasn't
particularly noticable to me, but look
http://www.linuxdocs.org/HOWTOs/NFS-HOWTO/performance.html for more
information on "5.4. Memory Limits on the Input Queue", "5.3. Number of
Instances of NFSD", and "5.5. Overflow of Fragmented Packets" were
interesting to me.  

Finally, NFS4 is reputed to be much faster in certain cases.  


Hope that helps.  I would be very interested in your findings.  

Be well, 

	Dan Farrell
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] NFS Server Tuning

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1338 bytes --]


On Thu, 2008-01-24 at 19:19 -0600, Dan Farrell wrote:

[...]
> and as the client (from `mount`): 
> 
> nfs:/mnt/storage on /home/media/storage type
> 	nfs(rw,rsize=65536,wsize=65536,soft,timeo=300,addr=192.168.1.88)
> 
> /etc/fstab on the client looks like: 
> 
> nfs:/mnt/storage /home/media/storage    nfs
> 		rsize=65536,wsize=65536,rw,async,soft,timeo=300 0 0
> 
> 
> Of these options, rsize,wsize,and async are reputed to effect
> performance.  However, I do not see much of an effect between different
> rsize and wsize settings.  I believe that over an uncongested 100T
> network it probably doesn't matter too much what rsize and wsize are.  
> On a different share (same server) mounted async without [r|w]size set,
> performance (write, this time) was 11.2mb/s, roughly the same.
> Furthermore, I'm not sure these values are even valid.
> http://www.linuxdocs.org/HOWTOs/NFS-HOWTO/performance.html said that
> nfs3 goes only to 32768.  
[...]

As far as I remember, rsize and wsize are negotiated between client and
server. Those mount options just set an upper limit which is certainly
not what you want. I'm even wondering that those settings are accepted
at all! Normally, unsigned 16bit integer has a range from 0 to 65535. If
you ask me, that's an off-by-one error just waiting to happen...

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] NFS Server Tuning

[Stroller]

On 25 Jan 2008, at 22:40, Florian Philipp wrote:
> On Thu, 2008-01-24 at 19:19 -0600, Dan Farrell wrote:
>
> [...]
>> and as the client (from `mount`):
>>
>> nfs:/mnt/storage on /home/media/storage type
>> 	nfs(rw,rsize=65536,wsize=65536,soft,timeo=300,addr=192.168.1.88)
>>
>> /etc/fstab on the client looks like:
>>
>> nfs:/mnt/storage /home/media/storage    nfs
>> 		rsize=65536,wsize=65536,rw,async,soft,timeo=300 0 0
>>
>>
>> Of these options, rsize,wsize,and async are reputed to effect
>> performance.  ...
> [...]
>
> As far as I remember, rsize and wsize are negotiated between client  
> and
> server. Those mount options just set an upper limit which is certainly
> not what you want. I'm even wondering that those settings are accepted
> at all! Normally, unsigned 16bit integer has a range from 0 to  
> 65535. If
> you ask me, that's an off-by-one error just waiting to happen...

This seems to suggest that 32768 is the largest figure that can be  
specified for rsize,wsize:
   http://www.mythtv.org/wiki/index.php/ 
Optimizing_Performance#NFS_servers

Stroller.


-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] NFS Server Tuning

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1724 bytes --]


On Sat, 2008-01-26 at 10:29 +0000, Stroller wrote:
> On 25 Jan 2008, at 22:40, Florian Philipp wrote:
> > On Thu, 2008-01-24 at 19:19 -0600, Dan Farrell wrote:
> >
> > [...]
> >> and as the client (from `mount`):
> >>
> >> nfs:/mnt/storage on /home/media/storage type
> >> 	nfs(rw,rsize=65536,wsize=65536,soft,timeo=300,addr=192.168.1.88)
> >>
> >> /etc/fstab on the client looks like:
> >>
> >> nfs:/mnt/storage /home/media/storage    nfs
> >> 		rsize=65536,wsize=65536,rw,async,soft,timeo=300 0 0
> >>
> >>
> >> Of these options, rsize,wsize,and async are reputed to effect
> >> performance.  ...
> > [...]
> >
> > As far as I remember, rsize and wsize are negotiated between client  
> > and
> > server. Those mount options just set an upper limit which is certainly
> > not what you want. I'm even wondering that those settings are accepted
> > at all! Normally, unsigned 16bit integer has a range from 0 to  
> > 65535. If
> > you ask me, that's an off-by-one error just waiting to happen...
> 
> This seems to suggest that 32768 is the largest figure that can be  
> specified for rsize,wsize:
>    http://www.mythtv.org/wiki/index.php/ 
> Optimizing_Performance#NFS_servers
> 
> Stroller.
> 
> 

Ehmm, yes. NFS-docs approve this.

From a programmer's perspective this number is still odd because it's
one more than can fit into signed 16bit int and and 32767 less than
unsigned 16bit int... maybe they had other reasons.

Well, although neither info- nor man-pages mention it, I've found an old
man-page [1] which states that these values default to 1024, therefore
setting it to 32768 seems the best choice. 

[1] http://www.trinler.de/de/linux/man.html?command=nfs

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] NFS Server Tuning

[Eric Martin]

Stefan Onken wrote:
> Hello,
>
> I am running a x86 gentoo box as a nfs server. As a filesystem I am 
> using XFS on a 3ware Raid system. 
>
> The 3ware systems seems to be quite quick, although access via NFS 
> seems to be very slow. Any ideas how I can improve speed ? I was 
> expected a speed only limited by the 100 Mbit network. Now, the 
> speed is so slow, that reading/writing at the same time is nearly 
> impossible.
>
> Any ideas how to improve the speed ?
>
> hdparm on the gentoo box machine:
> backup3 ~ # hdparm -tT /dev/sdb
> /dev/sdb:
> Timing cached reads:   4108 MB in  2.00 seconds = 2054.34 MB/sec
> Timing buffered disk reads:  232 MB in  3.01 seconds =  77.09 MB/sec
>
> /etc/exports on the gentoo box:
> /mnt/backup/ 192.168.1.0/24
> (rw,no_root_squash,async,no_subtree_check)
>
> I am not even able to run a bonnie benchmark on the client. It seems 
> to hang while doing rewriting. Running bonnie on the nfs server 
> gives a fast result.
>
> /etc/fstab on the client: 
> 192.168.1.3:/mnt/backup         /mnt/backup     nfs     
> rw,users,async   0
>
> http://pastebin.com/m72ae9d47
>
> Any ideas ?
>   
I turned on tcp (vs. udp for nfs and I'm much happier.  I have a mixed 
cat5 (100Mbps) vs 802.11G(54Mbps) network.  YMMV but it's worth a shot.  
I'm streaming mp3s over it, my home directories and mythtv.

HTH
-eric
-- 
gentoo-user@lists.gentoo.org mailing list