[gentoo-user] Gentoo router: Conntrack table full

[gentoo-user] Gentoo router: Conntrack table full

[Dan Cowsill]

Hi folks,

Today I had some really serious problems with my Gentoo router.  I
could ping it, and all the network connections were in place and
functional, but no outside access.  I looked into it and found that
the syslog was flooded with this:


Mar 22 21:25:55 localhost kernel: nf_conntrack: table full, dropping packet.
Mar 22 21:26:00 localhost kernel: printk: 11 messages suppressed.
Mar 22 21:26:00 localhost kernel: nf_conntrack: table full, dropping packet.
Mar 22 21:26:05 localhost kernel: printk: 16 messages suppressed.


These messages spanned a full 20 hours of the log.  I understand that
conntrack is the connection tracking system that iptables uses.  I
also understand that its maximum is something on the order of 65000
simultaneous connections.  For a simple home network, I think we can
agree that I would probably never approach this number of connections
with normal use.

So my question is this:  what could have caused the router's
connection tracker to overflow?
-- 
Dan Cowsill
http://www.danthehat.net
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Gentoo router: Conntrack table full

[Andrey Falko]

On Sat, Mar 22, 2008 at 11:16 PM, Dan Cowsill <danthehat@gmail.com> wrote:
> Hi folks,
>
>  Today I had some really serious problems with my Gentoo router.  I
>  could ping it, and all the network connections were in place and
>  functional, but no outside access.  I looked into it and found that
>  the syslog was flooded with this:
>
>
>  Mar 22 21:25:55 localhost kernel: nf_conntrack: table full, dropping packet.
>  Mar 22 21:26:00 localhost kernel: printk: 11 messages suppressed.
>  Mar 22 21:26:00 localhost kernel: nf_conntrack: table full, dropping packet.
>  Mar 22 21:26:05 localhost kernel: printk: 16 messages suppressed.
>
>
>  These messages spanned a full 20 hours of the log.  I understand that
>  conntrack is the connection tracking system that iptables uses.  I
>  also understand that its maximum is something on the order of 65000
>  simultaneous connections.  For a simple home network, I think we can
>  agree that I would probably never approach this number of connections
>  with normal use.
>
>  So my question is this:  what could have caused the router's
>  connection tracker to overflow?
>  --
>  Dan Cowsill
>  http://www.danthehat.net
>  --
>  gentoo-user@lists.gentoo.org mailing list
>
>

What type of 'net services do you run between your home network and
the outside? Is there a possibility that someone out have put a denial
of service attack on you?
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Gentoo router: Conntrack table full

[Dan Cowsill]

On Sat, Mar 22, 2008 at 11:22 PM, Andrey Falko <ma3oxuct@gmail.com> wrote:
>
> On Sat, Mar 22, 2008 at 11:16 PM, Dan Cowsill <danthehat@gmail.com> wrote:
>  > Hi folks,
>  >
>  >  Today I had some really serious problems with my Gentoo router.  I
>  >  could ping it, and all the network connections were in place and
>  >  functional, but no outside access.  I looked into it and found that
>  >  the syslog was flooded with this:
>  >
>  >
>  >  Mar 22 21:25:55 localhost kernel: nf_conntrack: table full, dropping packet.
>  >  Mar 22 21:26:00 localhost kernel: printk: 11 messages suppressed.
>  >  Mar 22 21:26:00 localhost kernel: nf_conntrack: table full, dropping packet.
>  >  Mar 22 21:26:05 localhost kernel: printk: 16 messages suppressed.
>  >
>  >
>  >  These messages spanned a full 20 hours of the log.  I understand that
>  >  conntrack is the connection tracking system that iptables uses.  I
>  >  also understand that its maximum is something on the order of 65000
>  >  simultaneous connections.  For a simple home network, I think we can
>  >  agree that I would probably never approach this number of connections
>  >  with normal use.
>  >
>  >  So my question is this:  what could have caused the router's
>  >  connection tracker to overflow?
>  >  --
>  >  Dan Cowsill
>  >  http://www.danthehat.net
>  >  --
>  >  gentoo-user@lists.gentoo.org mailing list
>  >
>  >
>
>  What type of 'net services do you run between your home network and
>  the outside? Is there a possibility that someone out have put a denial
>  of service attack on you?
>  --
>  gentoo-user@lists.gentoo.org mailing list
>
>

I have SSH to a server, two open ports for bit torrent connections and
a few ranges for DCC transfers from irc.

The possibility of a DoS attack is pretty real, I imagine.  Is there
any way I could be sure?

-- 
Dan Cowsill
http://www.danthehat.net
-- 
gentoo-user@lists.gentoo.org mailing list

Re: [gentoo-user] Gentoo router: Conntrack table full

[Michal 'vorner' Vaner]

[-- Attachment #1: Type: text/plain, Size: 700 bytes --]

Hello

On Sat, Mar 22, 2008 at 11:26:16PM -0400, Dan Cowsill wrote:
> On Sat, Mar 22, 2008 at 11:22 PM, Andrey Falko <ma3oxuct@gmail.com> wrote:
> I have SSH to a server, two open ports for bit torrent connections and
> a few ranges for DCC transfers from irc.

Torrents can sometimes open thousands of connections (yes, it is very
aggressive way of transferring data). And if you have something like
Skype, they a full table is not something unusual.

So IMO it could be DoS, but it can be some application that doesn't know
what is polite behaviour.

-- 
This is a terroristic email. It will explode in 10 minutes, 
if you do not close it in the meantime.

Michal 'vorner' Vaner

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] Gentoo router: Conntrack table full

[Mike Williams]

On Sunday 23 March 2008 03:16:16 Dan Cowsill wrote:
>  I
> also understand that its maximum is something on the order of 65000
> simultaneous connections.

That's a significant understatement.
The default limit is based on how much RAM you have, and is set very 
conservatively.
/proc/sys/net/ipv4/netfilter/ip_conntrack_max sets how many connections you 
can track.

You should also 
drop /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established 
significantly. Connections can hang around for weeks, unless properly closed.

On the production linux firewalls I maintain they were happily handling 
~50-60k connections until I dropped ip_conntrack_tcp_timeout_established to 
432000 seconds when the conntrack table dropped to ~30k. I could drop it a 
lot lower, but the machines cope with absolutely no issues.

Personally, I'd drop ip_conntrack_tcp_timeout_established to about a day, or 
even less, as connections won't time out if traffic continues to pass.

-- 
Mike Williams
--
gentoo-user@lists.gentoo.org mailing list

RE: [gentoo-user] Gentoo router: Conntrack table full

[Adam Carter]

The contents of the connection table is in
/proc/net/ip_conntrack

Example:
tcp      6 65 TIME_WAIT src=192.168.1.4 dst=20.x.y.40 sport=4986 dport=80 src=207.46.109.40 dst=192.168.1.4 sport=80 dport=4986 [ASSURED] mark=0 use=1

So go nuts with grep/awk/sed/sort/uniq etc to find what is consuming all the connections.

+1 for what Mike said about dropping the timeouts to something more sensible. FWIW Checkpoint uses a default TCP timer of 1 hour. Use 4 hours to be conservative.
--
gentoo-user@lists.gentoo.org mailing list