[gentoo-user] NFS vs. jumbo frames

[gentoo-user] NFS vs. jumbo frames

[Matthias Bethke]

[-- Attachment #1: Type: text/plain, Size: 1530 bytes --]

I've been fiddling with this for some days and can't but assume it's a
bug in one of the Gentoo patches to either the kernel or NFS tools:
Basically, NFS locking breaks as soon as I enable jumbo frames on both
server and client.
  touch foobar
  flock foobar ls
works fine in my NFS-mounted home with an MTU of 1500. An MTU of 9000 is
great for general net throughput so I wanted to use it on both the
server and the clients, but the above sequence hangs indefinitely when I
try. I'm aware flock() isn't supposed to work correctly with NFS anyway,
but all kinds of stuff depends on it at least pretending to.
The strange thing is, SuSE 10.1 as a client works fine with jumbo
frames, just my Gentoo box doesn't. I tried enabling nfs_debug with
sysctl and sniffing the wire with tcpdump and wireshark but with my
pretty basic knowledge of NFS workings I didn't spot anything
conspicuous other than that
  lookup(msbethke/foobar)
  nfs_update_inode(0:18/3424742 ct=1 info=0x6)
  nfs_fhget(0:18/1081970 ct=1)
  permission(0:18/1081970), mask=0x4, res=0 
seems to be the exchange after which the hang occurs.
Our server is running 2.6.18-hardened-r6 and nfs-utils-1.0.12. The
clients are mostly SuSE 10.1 boxes with kernel 2.6.16.21-0.21-smp and
nfs-utils-1.0.7-36 while my workstation has 2.6.20-gentoo-r6 (was
linux-2.6.19-gentoo-r5 before) and the same ns-utils as the server.

-- 
I prefer encrypted and signed messages. KeyID: FAC37665
Fingerprint: 8C16  3F0m A6FC DF0D 19B0  8DEF 48D9 1700 FAC3 7665

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] NFS vs. jumbo frames

[Uwe Thiem]

On 23 April 2007, Matthias Bethke wrote:
> I've been fiddling with this for some days and can't but assume it's a
> bug in one of the Gentoo patches to either the kernel or NFS tools:
> Basically, NFS locking breaks as soon as I enable jumbo frames on both
> server and client.
>   touch foobar
>   flock foobar ls
> works fine in my NFS-mounted home with an MTU of 1500. An MTU of 9000 is
> great for general net throughput so I wanted to use it on both the
> server and the clients, but the above sequence hangs indefinitely when I
> try.

Just curious: What kind of network (layer 2) is this that allows an MTU of 
9000?

Uwe

-- 
The Informal Linux Group Namibia:
http://www.linux.org.na
SysEx (Pty) Ltd.:
http://www.SysEx.com.na
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] NFS vs. jumbo frames

[Tony Stohne]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Uwe Thiem said the following on 2007-04-23 17:53:
> Just curious: What kind of network (layer 2) is this that allows an MTU of 
> 9000?
> 
> Uwe
> 
It sounds like Gigabit Ethernet to me.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (MingW32)

iD8DBQFGLPN6JDzv6DN+QUkRAqfuAKDQsjPMRMMnnSBKeOTynrB8vsC9sACfce+m
T5AGW8nM3NTlg66jHxpzZJk=
=2WDI
-----END PGP SIGNATURE-----
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] NFS vs. jumbo frames

[kashani]

Tony Stohne wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Uwe Thiem said the following on 2007-04-23 17:53:
>> Just curious: What kind of network (layer 2) is this that allows an MTU of 
>> 9000?
>>
>> Uwe
>>
> It sounds like Gigabit Ethernet to me.

Keep in mind that not all fastE or gigE switches support jumbo frames. 
Additionally not all cards support jumbo frames either though you can 
certainly set them to an MTU of 9000 and watch things break.

To the original poster, I'd do some googling and verify that all the 
network cards and switches involved can do jumbo frames and that it is 
enabled on each device as needed.

kashani
-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: NFS vs. jumbo frames

[ames]

kashani <kashani-list <at> badapple.net> writes:


> >> Just curious: What kind of network (layer 2) is this that allows an MTU of 
> >> 9000?
> >> Uwe

> > It sounds like Gigabit Ethernet to me.

> Keep in mind that not all fastE or gigE switches support jumbo frames. 
> Additionally not all cards support jumbo frames either though you can 
> certainly set them to an MTU of 9000 and watch things break.

> To the original poster, I'd do some googling and verify that all the 
> network cards and switches involved can do jumbo frames and that it is 
> enabled on each device as needed.

> kashani



Does NFS have any negotiations to determine if jumbo frames can work
between 2 system, then use a smaller mtu if a larger (jumbo) mtu
is not suppported between devices?


James




-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] NFS vs. jumbo frames

[Tony Stohne]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

kashani said the following on 2007-04-23 20:11:
> Keep in mind that not all fastE or gigE switches support jumbo frames. 
> Additionally not all cards support jumbo frames either though you can 
> certainly set them to an MTU of 9000 and watch things break.
> 
> To the original poster, I'd do some googling and verify that all the 
> network cards and switches involved can do jumbo frames and that it is 
> enabled on each device as needed.
> 
Good point, on both comments!
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (MingW32)

iD8DBQFGLPzoJDzv6DN+QUkRArESAJ99y7HrbMdv/0QxZEsETlJpD63d9QCfWpGT
YEGKnn0hz9HT5bye2c15AqU=
=n2C9
-----END PGP SIGNATURE-----
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Re: NFS vs. jumbo frames

[Uwe Thiem]

On 23 April 2007, ames wrote:
> kashani <kashani-list <at> badapple.net> writes:
> > >> Just curious: What kind of network (layer 2) is this that allows an
> > >> MTU of 9000?
> > >> Uwe
> > >
> > > It sounds like Gigabit Ethernet to me.
> >
> > Keep in mind that not all fastE or gigE switches support jumbo frames.
> > Additionally not all cards support jumbo frames either though you can
> > certainly set them to an MTU of 9000 and watch things break.
> >
> > To the original poster, I'd do some googling and verify that all the
> > network cards and switches involved can do jumbo frames and that it is
> > enabled on each device as needed.
> >
> > kashani
>
> Does NFS have any negotiations to determine if jumbo frames can work
> between 2 system, then use a smaller mtu if a larger (jumbo) mtu
> is not suppported between devices?

Don't stare at NFS.  It's too high a layer in the TCP/IP stack.  And yes, it 
can deal with large packets. You can use NFS with localhost (127.0.0.1), 
right? That one usually has an MTU of 16,436.

The real issues with MTUs occur at layer 2 (ethernet or whatever you are 
using), IP (fragmentation and de-fragmentation) and ICMP (MTU discovery).

Uwe

-- 
The Informal Linux Group Namibia:
http://www.linux.org.na
SysEx (Pty) Ltd.:
http://www.SysEx.com.na
-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: NFS vs. jumbo frames

[Francesco Talamona]

On Monday 23 April 2007, kashani wrote:
> Tony Stohne wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Uwe Thiem said the following on 2007-04-23 17:53:
> >> Just curious: What kind of network (layer 2) is this that allows
> >> an MTU of 9000?
> >>
> >> Uwe
> >
> > It sounds like Gigabit Ethernet to me.
>
> Keep in mind that not all fastE or gigE switches support jumbo
> frames. Additionally not all cards support jumbo frames either though
> you can certainly set them to an MTU of 9000 and watch things break.
Some cards do support jumbo frames, but up to values lower than 9000, 
for example I set up a NFS over a gbit link with jumbo frames with an 
MTU of 7200 because this was the lower common.

> To the original poster, I'd do some googling and verify that all the
> network cards and switches involved can do jumbo frames and that it
> is enabled on each device as needed.
>
> kashani

Based on my experience I would add to verify also the upper MTU value 
really supported.

Ciao
	Francesco

-- 
Linux Version 2.6.20-gentoo-r6, Compiled #2 PREEMPT Fri Apr 20 17:31:11 
CEST 2007
One 2.2GHz AMD Athlon 64 Processor, 2GB RAM, 4408.86 Bogomips Total
aemaeth

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Re: NFS vs. jumbo frames

[Fabio Correa]

You can also fiddle with the rsize, wsize NFS mount parameters.


-- 
Fabio A. Correa D.

Physics Dept, Universidad Nacional, Bogota, Colombia
facorread@gmail.com
ffaaccdd@yahoo.co.uk         facorread@unal.edu.co
My webpage and OpenPGP key at http://facorread.150m.com
My alexandria.cc address is not available anymore.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] NFS vs. jumbo frames

[Matthias Bethke]

[-- Attachment #1: Type: text/plain, Size: 1595 bytes --]

Hi kashani,
on Monday, 2007-04-23 at 11:11:40, you wrote:
> >It sounds like Gigabit Ethernet to me.

Yes, that's it.

> Keep in mind that not all fastE or gigE switches support jumbo frames. 
> Additionally not all cards support jumbo frames either though you can 
> certainly set them to an MTU of 9000 and watch things break.

I had that problem before with the Server's onboard Broadcom chip;
fortunately it just breaks completely when you up the MTU :) Now I
installed an Intel 82545GM card that officially supports jumbo frames
and that I haven't heard anyone complain about. The clients all have the
same 82547EI onboard chip.

> To the original poster, I'd do some googling and verify that all the 
> network cards and switches involved can do jumbo frames and that it is 
> enabled on each device as needed.

Check. The switches are HP ProCurve 2824 supporting up to 9216 bytes per
frame, and I checked the config several times. Jumbo frames are enabled
on all ports, and it's a rather basic config anyway, no VLANs 'n stuff,
no voice LAN features, just switching. And for everything else but NFS
locking it does work fine. A plain netcat from /dev/zero to /dev/null
goes from some 35 MB/s at an MTU of 1500 to over 80, ssh does very well,
and even NFS file operations other than locking work. 
I have googled for quite a while but can't find a thing.
Anyone here using NFS and GigE+jumbo frames with Gentoo?

cheers!
	Matthias
-- 
I prefer encrypted and signed messages. KeyID: FAC37665
Fingerprint: 8C16 3F0A A6FC DF0D 19B0  8DEF 48D9 1700 FAC3 7665

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Re: NFS vs. jumbo frames

[Matthias Bethke]

[-- Attachment #1: Type: text/plain, Size: 492 bytes --]

Hi Francesco,
on Monday, 2007-04-23 at 21:58:18, you wrote:
> Based on my experience I would add to verify also the upper MTU value 
> really supported.

According to Documentation/networking/e1000.txt, the adapters should all
support 16K frames. The limiting factor would be the switch's 9K limit,
but I've stayed below that as well.

cheers!
	Matthias
-- 
I prefer encrypted and signed messages. KeyID: FAC37665
Fingerprint: 8C16 3F0A A6FC DF0D 19B0  8DEF 48D9 1700 FAC3 7665

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] NFS vs. jumbo frames

[Matthias Bethke]

[-- Attachment #1: Type: text/plain, Size: 797 bytes --]

On Tuesday, 2007-04-24 at 15:38:12, I wrote:
> I have googled for quite a while but can't find a thing.
> Anyone here using NFS and GigE+jumbo frames with Gentoo?

Just to follow up for the archives' sake: this seems to be an old and
frustrating problem, I've run into a few messages dating back to 2002 of
people with similar problems. Like here:
http://lists.us.dell.com/pipermail/linux-poweredge/2002-December/005568.html
and a more recent one on Sun hardware:
http://www.opensolaris.org/jive/thread.jspa?messageID=74750

I've switched back to MTU 1500 for now and if I find the time I'll ask
for news on this on some kernel list.

cheers!
  Matthias
-- 
I prefer encrypted and signed messages. KeyID: FAC37665
Fingerprint: 8C16 3F0A A6FC DF0D 19B0  8DEF 48D9 1700 FAC3 7665

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] NFS vs. jumbo frames

[kashani]

Matthias Bethke wrote:
> On Tuesday, 2007-04-24 at 15:38:12, I wrote:
>> I have googled for quite a while but can't find a thing.
>> Anyone here using NFS and GigE+jumbo frames with Gentoo?
> 
> Just to follow up for the archives' sake: this seems to be an old and
> frustrating problem, I've run into a few messages dating back to 2002 of
> people with similar problems. Like here:
> http://lists.us.dell.com/pipermail/linux-poweredge/2002-December/005568.html
> and a more recent one on Sun hardware:
> http://www.opensolaris.org/jive/thread.jspa?messageID=74750
> 
> I've switched back to MTU 1500 for now and if I find the time I'll ask
> for news on this on some kernel list.

Neither of these cases should have any relation to your problem since 
they talking about NFS on a 2.4 kernel or over UDP in NFSv2.

kashani
-- 
gentoo-user@gentoo.org mailing list