[gentoo-user] weird 1000baseT problem

[gentoo-user] weird 1000baseT problem

[Uwe Thiem]

Hi folks,

this message is rather lengthy. If you don't feel like reading all of it 
please don't bother to answer. You'll need the whole lot to get the 
picture. ;-)

I have run into a weird network problem with 1Gb NICs. It involves these two 
boxes:

Box A
P4 2.8Ghz HT
512GB ram
Tigon Gb NIC (module tg3)
IDE drives

Box B
Xeon 2.6Ghz HT
512GB ram
Intel Pro/1000 Gb NIC (module e1000)
SCSI RAID5

The two of them are connected by a cross-over cable. So nothing else is on 
that network, kinda peer-to-peer connection. Both boxes are running *exactly* 
the same gentoo software. I emerged it on one box, tarred it up, copied it 
over to the other one and made the config changes like IP addresses, names 
and such. Kernel is 2.6.12-gentoo-r6. Of course, box B loads the SCSI 
modules. All file transfers I am talking about are done with a file 
"all.tar.bz2" of the size of 1088MB. Both boxes are idle otherwise. Neither 
box runs services like FTP or HTTP. So I have to resort to other protocols to 
transfer files. Both do run NFS and SSH.

Case 1:
I log into A and NFS mount B's /tmp on A's /mnt/floppy and cd to /tmp. 
"cp /mnt/floppy/all.tar.bz2  ." (receiving on A) as well as "cp 
all.tar.bz2 /mnt/floppy" (sending from A) result in a sustained transfer rate 
of 2xMB/s. That's to be expected because it involves an IDE drive on A, and 
that's about the limit of current IDE drives (though 1Gb NICs can transfer 
data at about 4 to 5 times that rate). It also confirms that both Gb NICs are 
performing though it doesn't confirm they are getting near their theoretical 
limits (the latter unimportant in this case).

Case 2:
I log into A and sftp into B. "get all.tar.bz2" (receiving on A) transfers the 
file at 2xMB/s, same as in case 1. CPU utilisation is up to 40-50% due to 
encryption. Still, encryption does not slow down the transfer rate by any 
significant amount. This can be expected with the CPUs involved.

Case 3:
I log into A and sftp into B. "put all.tar.bz2" (sending from A) transfers the 
file at 3.7MB/s!!!!! This is far slower than on a 100baseT network where I 
get transfer rates of about 10MB/s with the network being the bottleneck 
rather than the harddisks. CPU utilisation is down to about 10%, indicating 
that something else than encryption is throttling the transfer. This is odd!

Case 4:
I log into B and try to NFS mount A's /tmp to B's /mnt/floppy. It returns with 
an RPC timeout. So I can't do the "cp" test from B.

Case 5:
I log into B and sftp into A. It sits there for about 10 seconds before 
presenting me with a password prompt. ???? After, I get transfer rates close 
to case 2 and case 3, just the other way round. 

I am puzzled. First I thought that the Gb NIC on box A is somehow kaput but 
case 1 surely shows it is performing. What the heck is going on here? I would 
be deeply indebted to any person on this list that could shed some light on 
this. Any hint what to investigate would be highly appreciated. Really. This 
has troubled me for the last three days and I would go as far as ship you a 
Windhoek Lager. ;-)

Uwe

-- 
95% of all programmers rate themselves among the top 5% of all software 
developers. - Linus Torvalds

http://www.uwix.iway.na (last updated: 20.06.2004)
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] weird 1000baseT problem

[Ow Mun Heng]

On Thu, 2005-08-11 at 20:40 +0100, Uwe Thiem wrote:

> Case 5:
> I log into B and sftp into A. It sits there for about 10 seconds before 
> presenting me with a password prompt. ???? After, I get transfer rates close 
> to case 2 and case 3, just the other way round. 


The issues with the slow logon is most likely due to some DNS lookups or
something. I've had this before, (can't remember what happened but
managed to fix it).

I believe your SSH sessions will also be hung for 10 secs?

-- 
Ow Mun Heng
Gentoo/Linux on DELL D600 1.4Ghz 1.5GB RAM
98% Microsoft(tm) Free!! 
Neuromancer 09:39:26 up 10:55, 7 users, load average: 0.55, 0.42, 0.72 


-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] weird 1000baseT problem

[Mark Knecht]

Right - I saw this a few weeks ago when I took a new Myth frontend
machine to my dad's house and had my DNS server as the top server in
/etc/resolv.conf instead of the ones he should use on his network.

On 8/11/05, Ow Mun Heng <Ow.Mun.Heng@wdc.com> wrote:
> On Thu, 2005-08-11 at 20:40 +0100, Uwe Thiem wrote:
> 
> > Case 5:
> > I log into B and sftp into A. It sits there for about 10 seconds before
> > presenting me with a password prompt. ???? After, I get transfer rates close
> > to case 2 and case 3, just the other way round.
> 
> 
> The issues with the slow logon is most likely due to some DNS lookups or
> something. I've had this before, (can't remember what happened but
> managed to fix it).
> 
> I believe your SSH sessions will also be hung for 10 secs?
> 
> --
> Ow Mun Heng
> Gentoo/Linux on DELL D600 1.4Ghz 1.5GB RAM
> 98% Microsoft(tm) Free!!
> Neuromancer 09:39:26 up 10:55, 7 users, load average: 0.55, 0.42, 0.72
> 
> 
> --
> gentoo-user@gentoo.org mailing list
> 
>

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] weird 1000baseT problem

[Bob Sanders]

On Thu, 11 Aug 2005 20:40:12 +0100
Uwe Thiem <uwix@iway.na> wrote:

> 
> I am puzzled. First I thought that the Gb NIC on box A is somehow kaput but 
> case 1 surely shows it is performing. What the heck is going on here? I would 
> be deeply indebted to any person on this list that could shed some light on 
> this. Any hint what to investigate would be highly appreciated. Really. This 
> has troubled me for the last three days and I would go as far as ship you a 
> Windhoek Lager. ;-)
> 

The long timeout before password is probably DNS not working the port properly.
Just a guess.

Have you tried - scp, in both directions?

And which nfs?  V3, V4?  I suggest V4, if not.

Have you run top and netstat -rn on both boxes to see what they think the
routing is?

Bob
-  
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] weird 1000baseT problem

[Uwe Thiem]

On 12 August 2005 02:40, Ow Mun Heng wrote:
> On Thu, 2005-08-11 at 20:40 +0100, Uwe Thiem wrote:
> > Case 5:
> > I log into B and sftp into A. It sits there for about 10 seconds before
> > presenting me with a password prompt. ???? After, I get transfer rates
> > close to case 2 and case 3, just the other way round.
>
> The issues with the slow logon is most likely due to some DNS lookups or
> something. I've had this before, (can't remember what happened but
> managed to fix it).
>
> I believe your SSH sessions will also be hung for 10 secs?

Yup, the delay occurs with both ssh and sftp.

Has nothing to do with DNS. It also occurs when using IP addresses.

Uwe

-- 
95% of all programmers rate themselves among the top 5% of all software 
developers. - Linus Torvalds

http://www.uwix.iway.na (last updated: 20.06.2004)
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] weird 1000baseT problem

[Uwe Thiem]

On 12 August 2005 04:13, Bob Sanders wrote:
> On Thu, 11 Aug 2005 20:40:12 +0100
>
> Uwe Thiem <uwix@iway.na> wrote:
> > I am puzzled. First I thought that the Gb NIC on box A is somehow kaput
> > but case 1 surely shows it is performing. What the heck is going on here?
> > I would be deeply indebted to any person on this list that could shed
> > some light on this. Any hint what to investigate would be highly
> > appreciated. Really. This has troubled me for the last three days and I
> > would go as far as ship you a Windhoek Lager. ;-)
>
> The long timeout before password is probably DNS not working the port
> properly. Just a guess.

Nope. Same behaviour when using IP addresses.

>
> Have you tried - scp, in both directions?

Yes, did it now. Surprisingly it transfers data at 2xB/s both ways. Quite 
different from sftp.

>
> And which nfs?  V3, V4?  I suggest V4, if not.

V3 is what nfsstat says. Hmm. I've enabled both, V3 and V4, server- and 
client-side. How do I force it to use V4?

>
> Have you run top and netstat -rn on both boxes to see what they think the
> routing is?

Routes are alright. Both boxes have a route to the class C network 
(192.168.254.0/24) they are using.

Uwe

-- 
95% of all programmers rate themselves among the top 5% of all software 
developers. - Linus Torvalds

http://www.uwix.iway.na (last updated: 20.06.2004)
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] weird 1000baseT problem

[Mark Humphrey]

Uwe Thiem wrote:

>On 12 August 2005 04:13, Bob Sanders wrote:
>  
>
>>On Thu, 11 Aug 2005 20:40:12 +0100
>>
>>Uwe Thiem <uwix@iway.na> wrote:
>>    
>>
>>>I am puzzled. First I thought that the Gb NIC on box A is somehow kaput
>>>but case 1 surely shows it is performing. What the heck is going on here?
>>>I would be deeply indebted to any person on this list that could shed
>>>some light on this. Any hint what to investigate would be highly
>>>appreciated. Really. This has troubled me for the last three days and I
>>>would go as far as ship you a Windhoek Lager. ;-)
>>>      
>>>
It would probably cost 10 times the price of the lager for shipping.
Maybe you should make it a case? I heard that Namibia Breweries is due
to stop making Heineken and it's going to SAB? :-)

>>The long timeout before password is probably DNS not working the port
>>properly. Just a guess.
>>    
>>
>
>Nope. Same behaviour when using IP addresses.
>
>  
>
>>Have you tried - scp, in both directions?
>>    
>>
>
>Yes, did it now. Surprisingly it transfers data at 2xB/s both ways. Quite 
>different from sftp.
>
>  
>
>>And which nfs?  V3, V4?  I suggest V4, if not.
>>    
>>
>
>V3 is what nfsstat says. Hmm. I've enabled both, V3 and V4, server- and 
>client-side. How do I force it to use V4?
>
>  
>
>>Have you run top and netstat -rn on both boxes to see what they think the
>>routing is?
>>    
>>
>
>Routes are alright. Both boxes have a route to the class C network 
>(192.168.254.0/24) they are using.
>
>Uwe
>
>  
>



Email Disclaimer
http://www.aplitec.co.za/emaildisclaimer.htm

-- 
gentoo-user@gentoo.org mailing list

[completely OT]Re: [gentoo-user] weird 1000baseT problem

[Uwe Thiem]

On 12 August 2005 12:23, Mark Humphrey wrote:
> Uwe Thiem wrote:
> >On 12 August 2005 04:13, Bob Sanders wrote:
> >>On Thu, 11 Aug 2005 20:40:12 +0100
> >>
> >>Uwe Thiem <uwix@iway.na> wrote:
> >>>I am puzzled. First I thought that the Gb NIC on box A is somehow kaput
> >>>but case 1 surely shows it is performing. What the heck is going on
> >>> here? I would be deeply indebted to any person on this list that could
> >>> shed some light on this. Any hint what to investigate would be highly
> >>> appreciated. Really. This has troubled me for the last three days and I
> >>> would go as far as ship you a Windhoek Lager. ;-)
>
> It would probably cost 10 times the price of the lager for shipping.
> Maybe you should make it a case? I heard that Namibia Breweries is due
> to stop making Heineken and it's going to SAB? :-)

Sure shipping would cost more than the lager. But what the heck!

No way Namibia Breweries is going to SAB!!!! SAB might be the second largest 
brewery in the world but they still make junk beer. Even Pilsner Urquell has 
become less good since SAB bought them.

As for Heineken, I don't know whether they are going to stop making it, and I 
don't care. It isn't really good beer. BTW, there is a funny little story 
about it. When Namibia Breweries started to make Heineken, it turned out 
better than the original. So the good boys from Heineken got their collective 
butts over to Namibia to study the brewing process here. Afterwards, they 
changed their brewing back home according to what they learnt here. :-)

Uwe

-- 
95% of all programmers rate themselves among the top 5% of all software 
developers. - Linus Torvalds

http://www.uwix.iway.na (last updated: 20.06.2004)
-- 
gentoo-user@gentoo.org mailing list