[gentoo-server] Interesting Iptables issue.

[gentoo-server] Interesting Iptables issue.

[Nestor Camacho III]

I am wondering if anyone has come into this really unique problem...

Short story...

I had a gateway box on different hardware, that finally kicked the
bucket. I purchased new hardware rebuilt gentoo on it and I was  able to
get on the internet as I used to with the old box....however... with one
issue that has gotten the best of me... I have spent hours recompiling
kernel options and iptables. Google'ing and reading as much as I can on
the issue. I have sniffed the traffic both on the gateway server and on
the laptop(s) that I have experienced the issue on.

Now, what the problem is...I vpn (over ssl, to a Juniper device) to my
job. What I am seeing is when I finally connect I can ping hosts
internal to my work network, but when I try to initiate a connection
(ssh, http, rdp, etc) I get no where. It just hangs on trying to
establish the connections.

Upon sniffing the traffic I see that I start to generate duplicate
acks/packets and the connection fails.

I have tried already enableing and disabling things with ethtool to no
avail.

Now the kicker! I boot up on the same computer using Ubuntu live cd and
import the same firewall rules and everything works as it should!

It is an elusive issue and I know that I am not doing it justice in the
email but any nod in the right direction would be greatly appreciated.

Nes++
-- 
gentoo-server@lists.gentoo.org mailing list

Re: [gentoo-server] Interesting Iptables issue.

[Andrew Cowie]

[-- Attachment #1: Type: text/plain, Size: 1128 bytes --]

On Tue, 2008-01-08 at 23:50 -0500, Nestor Camacho III wrote:
> Now the kicker! I boot up on the same computer using Ubuntu live cd and
> import the same firewall rules and everything works as it should!

Doing "it works on one system and not on another" means you're going to
need to

a) compare the kernel .config files to find out what options their
kernel has in that yours doesn't,

b) compare the versions of the user space software being run and figure
out what might be different.

etc.

If you discover a discrepancy and wonder how to enable/disable it for
Gentoo, I imagine someone 'round these parts might be able to help. But
I'm afraid you're going to have to narrow the problem down a fair bit
before anyone can do so.

Best of luck,

AfC
Sydney

-- 
Andrew Frederick Cowie

We are an operations engineering consultancy focusing on strategy,
organizational architecture, systems review, and change management
procedures: enabling successful use of open source in mission
critical enterprises, worldwide.

http://www.operationaldynamics.com/

Sydney   New York   Toronto   London

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-server] Interesting Iptables issue.

[Nestor Camacho III]

Yeah, Andrew, I did all of that too. I even reverted to the same kernel
version that I was using on the box before it died (I had my .config
backed up) only things that I changed were things concerning the sata
drives that I was now using and the change in memory and cpu architecture.

Nes++

Andrew Cowie wrote:
> On Tue, 2008-01-08 at 23:50 -0500, Nestor Camacho III wrote:
>   
>> Now the kicker! I boot up on the same computer using Ubuntu live cd and
>> import the same firewall rules and everything works as it should!
>>     
>
> Doing "it works on one system and not on another" means you're going to
> need to
>
> a) compare the kernel .config files to find out what options their
> kernel has in that yours doesn't,
>
> b) compare the versions of the user space software being run and figure
> out what might be different.
>
> etc.
>
> If you discover a discrepancy and wonder how to enable/disable it for
> Gentoo, I imagine someone 'round these parts might be able to help. But
> I'm afraid you're going to have to narrow the problem down a fair bit
> before anyone can do so.
>
> Best of luck,
>
> AfC
> Sydney
>
>   
-- 
gentoo-server@lists.gentoo.org mailing list

Re: [gentoo-server] Interesting Iptables issue.

[Lindsay Haisley]

Try adding the following to your iptables rules:

iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

This sounds very much like the problem I had setting up a Gentoo
firewall here.  The man page for iptables says:

   TCPMSS
       This target allows to alter the MSS value of TCP SYN packets,  to  con-
       trol  the maximum size for that connection (usually limiting it to your
       outgoing interface's MTU minus 40).  Of course, it can only be used  in
       conjunction with -p tcp.  It is only valid in the mangle table.
       This  target  is  used to overcome criminally braindead ISPs or servers
       which block ICMP Fragmentation Needed packets.  The  symptoms  of  this
       problem are that everything works fine from your Linux firewall/router,
       but machines behind it can never exchange large packets:
        1) Web browsers connect, then hang with no data received.
        2) Small mail works fine, but large emails hang.
        3) ssh works fine, but scp hangs after initial handshaking.
       Workaround: activate this option and add a rule to your  firewall  con-
       figuration like:
        iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
                    -j TCPMSS --clamp-mss-to-pmtu

This fix worked for me.  Issues of MTU and MTU discovery are complex.
I'm still trying to understand fully what the issues are here.

On Tue, 2008-01-08 at 23:50 -0500, Nestor Camacho III wrote:
> I am wondering if anyone has come into this really unique problem...
> 
> Short story...
> 
> I had a gateway box on different hardware, that finally kicked the
> bucket. I purchased new hardware rebuilt gentoo on it and I was  able to
> get on the internet as I used to with the old box....however... with one
> issue that has gotten the best of me... I have spent hours recompiling
> kernel options and iptables. Google'ing and reading as much as I can on
> the issue. I have sniffed the traffic both on the gateway server and on
> the laptop(s) that I have experienced the issue on.
> 
> Now, what the problem is...I vpn (over ssl, to a Juniper device) to my
> job. What I am seeing is when I finally connect I can ping hosts
> internal to my work network, but when I try to initiate a connection
> (ssh, http, rdp, etc) I get no where. It just hangs on trying to
> establish the connections.
> 
> Upon sniffing the traffic I see that I start to generate duplicate
> acks/packets and the connection fails.
> 
> I have tried already enableing and disabling things with ethtool to no
> avail.
> 
> Now the kicker! I boot up on the same computer using Ubuntu live cd and
> import the same firewall rules and everything works as it should!
> 
> It is an elusive issue and I know that I am not doing it justice in the
> email but any nod in the right direction would be greatly appreciated.
> 
> Nes++

-- 
gentoo-server@lists.gentoo.org mailing list

Re: [gentoo-server] Interesting Iptables issue.

[Nestor Camacho III]

Lindsay, thanks for the tip, unfortunately it did not work :( :)

Nes++

Lindsay Haisley wrote:
> Try adding the following to your iptables rules:
>
> iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
>
> This sounds very much like the problem I had setting up a Gentoo
> firewall here.  The man page for iptables says:
>
>    TCPMSS
>        This target allows to alter the MSS value of TCP SYN packets,  to  con-
>        trol  the maximum size for that connection (usually limiting it to your
>        outgoing interface's MTU minus 40).  Of course, it can only be used  in
>        conjunction with -p tcp.  It is only valid in the mangle table.
>        This  target  is  used to overcome criminally braindead ISPs or servers
>        which block ICMP Fragmentation Needed packets.  The  symptoms  of  this
>        problem are that everything works fine from your Linux firewall/router,
>        but machines behind it can never exchange large packets:
>         1) Web browsers connect, then hang with no data received.
>         2) Small mail works fine, but large emails hang.
>         3) ssh works fine, but scp hangs after initial handshaking.
>        Workaround: activate this option and add a rule to your  firewall  con-
>        figuration like:
>         iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
>                     -j TCPMSS --clamp-mss-to-pmtu
>
> This fix worked for me.  Issues of MTU and MTU discovery are complex.
> I'm still trying to understand fully what the issues are here.
>
> On Tue, 2008-01-08 at 23:50 -0500, Nestor Camacho III wrote:
>   
>> I am wondering if anyone has come into this really unique problem...
>>
>> Short story...
>>
>> I had a gateway box on different hardware, that finally kicked the
>> bucket. I purchased new hardware rebuilt gentoo on it and I was  able to
>> get on the internet as I used to with the old box....however... with one
>> issue that has gotten the best of me... I have spent hours recompiling
>> kernel options and iptables. Google'ing and reading as much as I can on
>> the issue. I have sniffed the traffic both on the gateway server and on
>> the laptop(s) that I have experienced the issue on.
>>
>> Now, what the problem is...I vpn (over ssl, to a Juniper device) to my
>> job. What I am seeing is when I finally connect I can ping hosts
>> internal to my work network, but when I try to initiate a connection
>> (ssh, http, rdp, etc) I get no where. It just hangs on trying to
>> establish the connections.
>>
>> Upon sniffing the traffic I see that I start to generate duplicate
>> acks/packets and the connection fails.
>>
>> I have tried already enableing and disabling things with ethtool to no
>> avail.
>>
>> Now the kicker! I boot up on the same computer using Ubuntu live cd and
>> import the same firewall rules and everything works as it should!
>>
>> It is an elusive issue and I know that I am not doing it justice in the
>> email but any nod in the right direction would be greatly appreciated.
>>
>> Nes++
>>     
>
>   
-- 
gentoo-server@lists.gentoo.org mailing list

Re: [gentoo-server] Interesting Iptables issue.

[Nestor Camacho III]

Lindsay, although your original fix did not work for me, you did put me
on right track. After reading your email that you had an MTU issue I
went back and took a look at what my MTU settings were for all of my
NIC's... I had 1500 from all but one... the internet facing one (546 was
the setting).

I hard set it to 1500 and guess what... ? It works now every time no
longer is it sporadic. I went back to Ubuntu to see what the nic came up
as and it maintained an MTU of 1500, I dug around and saw that the
settings it is getting from the dhcp server bumps the MTU down, but it
seems that Ubuntu ignores this setting from my ISP, where as gentoo
accepts it. I am sure that it is just a matter of hard codding it in
/etc/conf.d/net going forward...

But thanks again, I never even thought to look at the MTU out of all the
many things I looked at ;).

I want to say thanks to you and everyone else that gave their input.

Nes++

Lindsay Haisley wrote:
> Try adding the following to your iptables rules:
>
> iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
>
> This sounds very much like the problem I had setting up a Gentoo
> firewall here.  The man page for iptables says:
>
>    TCPMSS
>        This target allows to alter the MSS value of TCP SYN packets,  to  con-
>        trol  the maximum size for that connection (usually limiting it to your
>        outgoing interface's MTU minus 40).  Of course, it can only be used  in
>        conjunction with -p tcp.  It is only valid in the mangle table.
>        This  target  is  used to overcome criminally braindead ISPs or servers
>        which block ICMP Fragmentation Needed packets.  The  symptoms  of  this
>        problem are that everything works fine from your Linux firewall/router,
>        but machines behind it can never exchange large packets:
>         1) Web browsers connect, then hang with no data received.
>         2) Small mail works fine, but large emails hang.
>         3) ssh works fine, but scp hangs after initial handshaking.
>        Workaround: activate this option and add a rule to your  firewall  con-
>        figuration like:
>         iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
>                     -j TCPMSS --clamp-mss-to-pmtu
>
> This fix worked for me.  Issues of MTU and MTU discovery are complex.
> I'm still trying to understand fully what the issues are here.
>
> On Tue, 2008-01-08 at 23:50 -0500, Nestor Camacho III wrote:
>   
>> I am wondering if anyone has come into this really unique problem...
>>
>> Short story...
>>
>> I had a gateway box on different hardware, that finally kicked the
>> bucket. I purchased new hardware rebuilt gentoo on it and I was  able to
>> get on the internet as I used to with the old box....however... with one
>> issue that has gotten the best of me... I have spent hours recompiling
>> kernel options and iptables. Google'ing and reading as much as I can on
>> the issue. I have sniffed the traffic both on the gateway server and on
>> the laptop(s) that I have experienced the issue on.
>>
>> Now, what the problem is...I vpn (over ssl, to a Juniper device) to my
>> job. What I am seeing is when I finally connect I can ping hosts
>> internal to my work network, but when I try to initiate a connection
>> (ssh, http, rdp, etc) I get no where. It just hangs on trying to
>> establish the connections.
>>
>> Upon sniffing the traffic I see that I start to generate duplicate
>> acks/packets and the connection fails.
>>
>> I have tried already enableing and disabling things with ethtool to no
>> avail.
>>
>> Now the kicker! I boot up on the same computer using Ubuntu live cd and
>> import the same firewall rules and everything works as it should!
>>
>> It is an elusive issue and I know that I am not doing it justice in the
>> email but any nod in the right direction would be greatly appreciated.
>>
>> Nes++
>>     
>
>   
-- 
gentoo-server@lists.gentoo.org mailing list

Re: [gentoo-server] Interesting Iptables issue.

[Oliver Schad]

[-- Attachment #1: Type: text/plain, Size: 717 bytes --]

Am Mittwoch, 9. Januar 2008 05:50 schrieb mir Nestor Camacho III:
> Now, what the problem is...I vpn (over ssl, to a Juniper device) to my
> job. What I am seeing is when I finally connect I can ping hosts
> internal to my work network, but when I try to initiate a connection
> (ssh, http, rdp, etc) I get no where. It just hangs on trying to
> establish the connections.
[...]
> Now the kicker! I boot up on the same computer using Ubuntu live cd and
> import the same firewall rules and everything works as it should!

Same routing table, same interface configurations (ip, netmask, mtu), same 
packet filter config, same vpn client version, same vpn client config, same 
vpn gateway?

Regards
Oli

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-server] Interesting Iptables issue.

[Lindsay Haisley]

This still sounds very much like an MTU problem, especially with the
unresponded ACKs.  You might explore clamping the MTU through the VPN.
You didn't say what VPN client you're using, but if you use a robust VPN
system such as OpenVPN you have control over the MTU of packets sent
through the tunnel.

In OpenVPN, the --tun-mtu, --fragment and --mssfix options are available
to help tune packet size through your tunnel.

I highly recommend OpenVPN.  It's easy to set up, robust, secure and
runs on both Unix-like systems (Linux, BSD, Mac OS-X) and Windows.

On Wed, 2008-01-09 at 20:00 +0100, Oliver Schad wrote:
> Am Mittwoch, 9. Januar 2008 05:50 schrieb mir Nestor Camacho III:
> > Now, what the problem is...I vpn (over ssl, to a Juniper device) to my
> > job. What I am seeing is when I finally connect I can ping hosts
> > internal to my work network, but when I try to initiate a connection
> > (ssh, http, rdp, etc) I get no where. It just hangs on trying to
> > establish the connections.
> [...]
> > Now the kicker! I boot up on the same computer using Ubuntu live cd and
> > import the same firewall rules and everything works as it should!
> 
> Same routing table, same interface configurations (ip, netmask, mtu), same 
> packet filter config, same vpn client version, same vpn client config, same 
> vpn gateway?
> 
> Regards
> Oli

-- 
Lindsay Haisley       | "In an open world,    |     PGP public key
FMP Computer Services |    who needs Windows  |      available at
512-259-1190          |      or Gates"        | http://pubkeys.fmp.com
http://www.fmp.com    |                       |

-- 
gentoo-server@lists.gentoo.org mailing list

Re: [gentoo-server] Interesting Iptables issue.

[Oliver Schad]

[-- Attachment #1: Type: text/plain, Size: 347 bytes --]

Am Mittwoch, 9. Januar 2008 20:25 schrieb mir Lindsay Haisley:
> In OpenVPN, the --tun-mtu, --fragment and --mssfix options are available
> to help tune packet size through your tunnel.

You need these options only with braindead packet filter configurations. I 
suggest you should fix the packet filter configuration first.

Regards
Oli

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-server] Interesting Iptables issue.

[Oliver Schad]

[-- Attachment #1: Type: text/plain, Size: 312 bytes --]

Am Mittwoch, 9. Januar 2008 20:25 schrieb mir Lindsay Haisley:
> This still sounds very much like an MTU problem, especially with the
> unresponded ACKs.  

If the three way handshake works it could be a MTU problem. If the three way 
handshake doesn't work it is definitly no MTU problem.

Regards
Oli

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]