[gentoo-user] Proxy server problem

[gentoo-user] Proxy server problem

[Grant]

I set up squid on a remote system so I can browse the internet from
that IP address.  It works but it stalls frequently.  I had similar
results with ziproxy.  I went over this with the squid list but we got
nowhere as it seems to be some kind of a system or network problem.

http://squid-web-proxy-cache.1019090.n4.nabble.com/squid-3-3-5-hangs-the-entire-system-td4660893.html

Can anyone here help me figure out what is wrong?  I'm not sure where to start.

- Grant

Re: [gentoo-user] Proxy server problem

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 953 bytes --]

On Saturday 24 Aug 2013 10:47:53 Grant wrote:
> I set up squid on a remote system so I can browse the internet from
> that IP address.  It works but it stalls frequently.  I had similar
> results with ziproxy.  I went over this with the squid list but we got
> nowhere as it seems to be some kind of a system or network problem.
> 
> http://squid-web-proxy-cache.1019090.n4.nabble.com/squid-3-3-5-hangs-the-en
> tire-system-td4660893.html
> 
> Can anyone here help me figure out what is wrong?  I'm not sure where to
> start.
> 
> - Grant

Just a quick pointer in case it applies to you:  if you tunnel into the proxy 
machine (using ssh, VPN, proxychains and what not) you would suffer from 
packet fragmentation, which could quickly snowball.  In this case try reducing 
your mtu to lower values, than the default ethernet 1500 byte packets, to 
cater for the overhead of the larger tunnelling headers.

HTH.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-user] Proxy server problem

[Grant]

>> I set up squid on a remote system so I can browse the internet from
>> that IP address.  It works but it stalls frequently.  I had similar
>> results with ziproxy.  I went over this with the squid list but we got
>> nowhere as it seems to be some kind of a system or network problem.
>>
>> http://squid-web-proxy-cache.1019090.n4.nabble.com/squid-3-3-5-hangs-the-en
>> tire-system-td4660893.html
>>
>> Can anyone here help me figure out what is wrong?  I'm not sure where to
>> start.
>>
>> - Grant
>
> Just a quick pointer in case it applies to you:  if you tunnel into the proxy
> machine (using ssh, VPN, proxychains and what not) you would suffer from
> packet fragmentation, which could quickly snowball.  In this case try reducing
> your mtu to lower values, than the default ethernet 1500 byte packets, to
> cater for the overhead of the larger tunnelling headers.

I've tried disconnecting from my SSH tunnel and changing the mtu on my
laptop and on the remote proxy server via ifconfig and there is some
kind of an improvement but I can't narrow it down.  I've tried mtu
down to 1000 on both systems but the proxy server still stalls
sometimes.  Any tips for narrowing this down further?

- Grant

Re: [gentoo-user] Proxy server problem

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 2761 bytes --]

On Saturday 24 Aug 2013 14:23:26 Grant wrote:
> >> I set up squid on a remote system so I can browse the internet from
> >> that IP address.  It works but it stalls frequently.  I had similar
> >> results with ziproxy.  I went over this with the squid list but we got
> >> nowhere as it seems to be some kind of a system or network problem.
> >> 
> >> http://squid-web-proxy-cache.1019090.n4.nabble.com/squid-3-3-5-hangs-the
> >> -en tire-system-td4660893.html
> >> 
> >> Can anyone here help me figure out what is wrong?  I'm not sure where to
> >> start.
> >> 
> >> - Grant
> > 
> > Just a quick pointer in case it applies to you:  if you tunnel into the
> > proxy machine (using ssh, VPN, proxychains and what not) you would
> > suffer from packet fragmentation, which could quickly snowball.  In this
> > case try reducing your mtu to lower values, than the default ethernet
> > 1500 byte packets, to cater for the overhead of the larger tunnelling
> > headers.
> 
> I've tried disconnecting from my SSH tunnel and changing the mtu on my
> laptop and on the remote proxy server via ifconfig and there is some
> kind of an improvement but I can't narrow it down.  I've tried mtu
> down to 1000 on both systems but the proxy server still stalls
> sometimes.  Any tips for narrowing this down further?
> 
> - Grant

Now that you mentioned using ssh, I don't think that you can improve this.  An 
mtu at 1000 bytes is lower than I thought might have helped.  The problem is 
caused by stacking tcp packets (tcp within tcp) each of which is using its own 
timeout for failed fragments.

The problem is explained here (tcp meltdown):

  http://sites.inka.de/~W1011/devel/tcp-tcp.html

and here (useful relevant references to other works are also made):

  http://publications.lib.chalmers.se/records/fulltext/123799.pdf


There are some suggested solutions like increasing buffer size, but I don't 
know this might work in a real world use case.  You can experiment with 
different buffer sizes as suggested here and see if it makes a difference:

   http://www.cyberciti.biz/faq/linux-tcp-tuning/


If the interruptions are not acceptable to you, you could consider using a 
different tunnel method.  A network layer VPN, like IPSec (you can use 
StrongSwan which also offers IKEv2 and MOBIKE for your laptop, or ipsec-tools 
with racoon for IKEv1 only) should work without such problems.  You will be 
tunnelling tcp in udp packets.  If you tunnel to your home router you will 
need to configure an IPSec tunnel mode connection, otherwise you would use an 
IPSec transport mode connection directly to your server after you allow IP 
protocol 50 packets through your router.

HTH.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-user] Proxy server problem

[Grant]

>> >> I set up squid on a remote system so I can browse the internet from
>> >> that IP address.  It works but it stalls frequently.  I had similar
>> >> results with ziproxy.  I went over this with the squid list but we got
>> >> nowhere as it seems to be some kind of a system or network problem.
>> >>
>> >> http://squid-web-proxy-cache.1019090.n4.nabble.com/squid-3-3-5-hangs-the
>> >> -en tire-system-td4660893.html
>> >>
>> >> Can anyone here help me figure out what is wrong?  I'm not sure where to
>> >> start.
>> >>
>> >> - Grant
>> >
>> > Just a quick pointer in case it applies to you:  if you tunnel into the
>> > proxy machine (using ssh, VPN, proxychains and what not) you would
>> > suffer from packet fragmentation, which could quickly snowball.  In this
>> > case try reducing your mtu to lower values, than the default ethernet
>> > 1500 byte packets, to cater for the overhead of the larger tunnelling
>> > headers.
>>
>> I've tried disconnecting from my SSH tunnel and changing the mtu on my
>> laptop and on the remote proxy server via ifconfig and there is some
>> kind of an improvement but I can't narrow it down.  I've tried mtu
>> down to 1000 on both systems but the proxy server still stalls
>> sometimes.  Any tips for narrowing this down further?
>>
>> - Grant
>
> Now that you mentioned using ssh, I don't think that you can improve this.  An
> mtu at 1000 bytes is lower than I thought might have helped.  The problem is
> caused by stacking tcp packets (tcp within tcp) each of which is using its own
> timeout for failed fragments.

I think I may have misunderstood you.  I do SSH into the machine
running squid, but I don't tunnel through that connection in order to
use the proxy.  I connect to the remote squid instance directly via my
browser and I also happen to SSH into the same machine to run
commands.  Do any of your recommendations apply in this scenario?

- Grant


> The problem is explained here (tcp meltdown):
>
>   http://sites.inka.de/~W1011/devel/tcp-tcp.html
>
> and here (useful relevant references to other works are also made):
>
>   http://publications.lib.chalmers.se/records/fulltext/123799.pdf
>
>
> There are some suggested solutions like increasing buffer size, but I don't
> know this might work in a real world use case.  You can experiment with
> different buffer sizes as suggested here and see if it makes a difference:
>
>    http://www.cyberciti.biz/faq/linux-tcp-tuning/
>
>
> If the interruptions are not acceptable to you, you could consider using a
> different tunnel method.  A network layer VPN, like IPSec (you can use
> StrongSwan which also offers IKEv2 and MOBIKE for your laptop, or ipsec-tools
> with racoon for IKEv1 only) should work without such problems.  You will be
> tunnelling tcp in udp packets.  If you tunnel to your home router you will
> need to configure an IPSec tunnel mode connection, otherwise you would use an
> IPSec transport mode connection directly to your server after you allow IP
> protocol 50 packets through your router.

Re: [gentoo-user] Proxy server problem

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 3042 bytes --]

On Sunday 25 Aug 2013 11:13:07 Grant wrote:
> >> >> I set up squid on a remote system so I can browse the internet from
> >> >> that IP address.  It works but it stalls frequently.  I had similar
> >> >> results with ziproxy.  I went over this with the squid list but we
> >> >> got nowhere as it seems to be some kind of a system or network
> >> >> problem.
> >> >> 
> >> >> http://squid-web-proxy-cache.1019090.n4.nabble.com/squid-3-3-5-hangs-> >> >> the -en tire-system-td4660893.html
> >> >> 
> >> >> Can anyone here help me figure out what is wrong?  I'm not sure where
> >> >> to start.
> >> >> 
> >> >> - Grant
> >> > 
> >> > Just a quick pointer in case it applies to you:  if you tunnel into
> >> > the proxy machine (using ssh, VPN, proxychains and what not) you
> >> > would suffer from packet fragmentation, which could quickly snowball.
> >> >  In this case try reducing your mtu to lower values, than the default
> >> > ethernet 1500 byte packets, to cater for the overhead of the larger
> >> > tunnelling headers.
> >> 
> >> I've tried disconnecting from my SSH tunnel and changing the mtu on my
> >> laptop and on the remote proxy server via ifconfig and there is some
> >> kind of an improvement but I can't narrow it down.  I've tried mtu
> >> down to 1000 on both systems but the proxy server still stalls
> >> sometimes.  Any tips for narrowing this down further?
> >> 
> >> - Grant
> > 
> > Now that you mentioned using ssh, I don't think that you can improve
> > this.  An mtu at 1000 bytes is lower than I thought might have helped. 
> > The problem is caused by stacking tcp packets (tcp within tcp) each of
> > which is using its own timeout for failed fragments.
> 
> I think I may have misunderstood you.  I do SSH into the machine
> running squid, but I don't tunnel through that connection in order to
> use the proxy.  I connect to the remote squid instance directly via my
> browser and I also happen to SSH into the same machine to run
> commands.  Do any of your recommendations apply in this scenario?

Ahh!  I misunderstood your set up too.  The problem I described only applies 
to setting up an application layer tunnel (e.g. SSH) and running tcp within 
that secure tunnel.  In your case, you establish a direct connection to your 
server over http or https and a separate connection using SSH.  No tunnelling 
involved and no tcp stacking.

Sorry for the bum steer.

Is this stalling problem happening when you just browse the internet, visiting 
websites, or do you get it when you are downloading large files such as 
videos, or music?  If it is the former, then I am not sure what causes it.  If 
it is the latter, then this may be relevant to http timeout settings. 

I don't have in depth knowledge of this, other than increasing timeout and 
data rates from default values on a hosted server to avoid this problem 
(apache's mod_reqtimeout).  Eventually, I ended up using ftp to download large 
files, instead of http.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-user] Proxy server problem

[Grant]

>> >> >> I set up squid on a remote system so I can browse the internet from
>> >> >> that IP address.  It works but it stalls frequently.  I had similar
>> >> >> results with ziproxy.  I went over this with the squid list but we
>> >> >> got nowhere as it seems to be some kind of a system or network
>> >> >> problem.
>> >> >>
>> >> >> http://squid-web-proxy-cache.1019090.n4.nabble.com/squid-3-3-5-hangs-> >> >> the -en tire-system-td4660893.html
>> >> >>
>> >> >> Can anyone here help me figure out what is wrong?  I'm not sure where
>> >> >> to start.
>
> Is this stalling problem happening when you just browse the internet, visiting
> websites, or do you get it when you are downloading large files such as
> videos, or music?  If it is the former, then I am not sure what causes it.  If
> it is the latter, then this may be relevant to http timeout settings.

Actually it happens when I'm just browsing the internet.  How can a
problem of this nature be debugged?

- Grant

Re: [gentoo-user] Proxy server problem

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 1275 bytes --]

On Aug 25, 2013 11:38 PM, "Grant" <emailgrant@gmail.com> wrote:
>
> >> >> >> I set up squid on a remote system so I can browse the internet
from
> >> >> >> that IP address.  It works but it stalls frequently.  I had
similar
> >> >> >> results with ziproxy.  I went over this with the squid list but
we
> >> >> >> got nowhere as it seems to be some kind of a system or network
> >> >> >> problem.
> >> >> >>
> >> >> >>
http://squid-web-proxy-cache.1019090.n4.nabble.com/squid-3-3-5-hangs-> >>
>> the -en tire-system-td4660893.html
> >> >> >>
> >> >> >> Can anyone here help me figure out what is wrong?  I'm not sure
where
> >> >> >> to start.
> >
> > Is this stalling problem happening when you just browse the internet,
visiting
> > websites, or do you get it when you are downloading large files such as
> > videos, or music?  If it is the former, then I am not sure what causes
it.  If
> > it is the latter, then this may be relevant to http timeout settings.
>
> Actually it happens when I'm just browsing the internet.  How can a
> problem of this nature be debugged?
>
> - Grant
>

After reading your description on the squid mailing list, I'm suspicious
that you might've run out of TCP buffers.

Please post the results of:

sysctl -A | egrep "(mem)|(tcp)"

Rgds,
--

[-- Attachment #2: Type: text/html, Size: 1945 bytes --]

Re: [gentoo-user] Proxy server problem

[Grant]

>> >> >> >> I set up squid on a remote system so I can browse the internet
>> >> >> >> from
>> >> >> >> that IP address.  It works but it stalls frequently.  I had
>> >> >> >> similar
>> >> >> >> results with ziproxy.  I went over this with the squid list but
>> >> >> >> we
>> >> >> >> got nowhere as it seems to be some kind of a system or network
>> >> >> >> problem.
>> >> >> >>
>> >> >> >>
>> >> >> >> http://squid-web-proxy-cache.1019090.n4.nabble.com/squid-3-3-5-hangs-> >> >>
>> >> >> >> the -en tire-system-td4660893.html
>> >> >> >>
>> >> >> >> Can anyone here help me figure out what is wrong?  I'm not sure
>> >> >> >> where
>> >> >> >> to start.
>> >
>> > Is this stalling problem happening when you just browse the internet,
>> > visiting
>> > websites, or do you get it when you are downloading large files such as
>> > videos, or music?  If it is the former, then I am not sure what causes
>> > it.  If
>> > it is the latter, then this may be relevant to http timeout settings.
>>
>> Actually it happens when I'm just browsing the internet.  How can a
>> problem of this nature be debugged?
>>
>> - Grant
>>
>
> After reading your description on the squid mailing list, I'm suspicious
> that you might've run out of TCP buffers.
>
> Please post the results of:
>
> sysctl -A | egrep "(mem)|(tcp)"

Hi Pandu, thank you for having a look.  These commands were run during a stall.

Proxy Server:

# sysctl -A | egrep "(mem)|(tcp)"
fs.nfs.nlm_tcpport = 0
net.core.optmem_max = 20480
net.core.rmem_default = 212992
net.core.rmem_max = 212992
net.core.wmem_default = 212992
net.core.wmem_max = 212992
net.ipv4.igmp_max_memberships = 20
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_adv_win_scale = 1
net.ipv4.tcp_allowed_congestion_control = cubic reno
net.ipv4.tcp_app_win = 31
net.ipv4.tcp_available_congestion_control = cubic reno
net.ipv4.tcp_base_mss = 512
net.ipv4.tcp_challenge_ack_limit = 100
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_cookie_size = 0
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_early_retrans = 2
net.ipv4.tcp_ecn = 2
net.ipv4.tcp_fack = 1
net.ipv4.tcp_fastopen = 0
net.ipv4.tcp_fastopen_key = ad627831-f01340bb-a649b04d-35ba63cd
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_frto = 2
net.ipv4.tcp_frto_response = 0
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_limit_output_bytes = 131072
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_max_orphans = 8192
net.ipv4.tcp_max_ssthresh = 0
net.ipv4.tcp_max_syn_backlog = 128
net.ipv4.tcp_max_tw_buckets = 8192
net.ipv4.tcp_mem = 46323 61766 92646
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_no_metrics_save = 0
net.ipv4.tcp_orphan_retries = 0
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_retrans_collapse = 1
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_rmem = 4096 87380 6291456
net.ipv4.tcp_sack = 1
net.ipv4.tcp_slow_start_after_idle = 1
net.ipv4.tcp_stdurg = 0
net.ipv4.tcp_syn_retries = 6
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_thin_dupack = 0
net.ipv4.tcp_thin_linear_timeouts = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_tso_win_divisor = 3
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_wmem = 4096 16384 4194304
net.ipv4.tcp_workaround_signed_windows = 0
net.ipv4.udp_mem = 46521 62030 93042
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
net.netfilter.nf_conntrack_tcp_be_liberal = 0
net.netfilter.nf_conntrack_tcp_loose = 1
net.netfilter.nf_conntrack_tcp_max_retrans = 3
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 432000
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
vm.lowmem_reserve_ratio = 256 256 32
vm.overcommit_memory = 0

Client:

# sysctl -A | egrep "(mem)|(tcp)"
fs.nfs.nfs_callback_tcpport = 0
fs.nfs.nlm_tcpport = 0
net.core.optmem_max = 20480
net.core.rmem_default = 212992
net.core.rmem_max = 212992
net.core.wmem_default = 212992
net.core.wmem_max = 212992
net.ipv4.igmp_max_memberships = 20
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_adv_win_scale = 1
net.ipv4.tcp_allowed_congestion_control = cubic reno
net.ipv4.tcp_app_win = 31
net.ipv4.tcp_available_congestion_control = cubic reno
net.ipv4.tcp_base_mss = 512
net.ipv4.tcp_challenge_ack_limit = 100
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_early_retrans = 3
net.ipv4.tcp_ecn = 2
net.ipv4.tcp_fack = 1
net.ipv4.tcp_fastopen = 0
net.ipv4.tcp_fastopen_key = af77bb2f-38636dab-7fd80263-df37cea3
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_frto = 2
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_limit_output_bytes = 131072
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_max_orphans = 16384
net.ipv4.tcp_max_ssthresh = 0
net.ipv4.tcp_max_syn_backlog = 128
net.ipv4.tcp_max_tw_buckets = 16384
net.ipv4.tcp_mem = 78105 104140 156210
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_no_metrics_save = 0
net.ipv4.tcp_orphan_retries = 0
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_retrans_collapse = 1
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_rmem = 4096 87380 6291456
net.ipv4.tcp_sack = 1
net.ipv4.tcp_slow_start_after_idle = 1
net.ipv4.tcp_stdurg = 0
net.ipv4.tcp_syn_retries = 6
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_thin_dupack = 0
net.ipv4.tcp_thin_linear_timeouts = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_tso_win_divisor = 3
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_wmem = 4096 16384 4194304
net.ipv4.tcp_workaround_signed_windows = 0
net.ipv4.udp_mem = 78360 104483 156720
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
net.netfilter.nf_conntrack_tcp_be_liberal = 0
net.netfilter.nf_conntrack_tcp_loose = 1
net.netfilter.nf_conntrack_tcp_max_retrans = 3
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 432000
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
vm.lowmem_reserve_ratio = 256 256 32
vm.overcommit_memory = 0

- Grant