From: Kerin Millar <kerframil@fastmail.co.uk>
To: gentoo-server@lists.gentoo.org
Subject: Re: [gentoo-server] DoS Analysis and Prevemption
Date: Sun, 28 Jul 2013 15:01:30 +0100 [thread overview]
Message-ID: <51F5243A.2080208@fastmail.co.uk> (raw)
In-Reply-To: <CA+qvzFOR1RJZUmDET-_3Sq86vD-hnX+95JTP9P0Q+Pa4Gu4Sqg@mail.gmail.com>
On 15/04/2013 16:07, Christian Parpart wrote:
> Hey all,
>
> we hit some nice traffic last night that took our main gateway down.
> Pacemaker was configured to failover to our second one, but that one
> died aswell.
>
> In a little post-analysis, I found the following in the logs:
>
> Apr 14 21:42:11 cesar1 kernel: [27613652.439846] BUG: soft lockup -
> CPU#4 stuck for 22s! [swapper/4:0]
> Apr 14 21:42:11 cesar1 kernel: [27613652.440319] Stack:
> Apr 14 21:42:11 cesar1 kernel: [27613652.440446] Call Trace:
> Apr 14 21:42:11 cesar1 kernel: [27613652.440595] <IRQ>
> Apr 14 21:42:12 cesar1 kernel: [27613652.440828] <EOI>
> Apr 14 21:42:12 cesar1 kernel: [27613652.440979] Code: c1 51 da 03 81 48
> c7 c2 4e da 03 81 e9 dd fe ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90
> 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 <89> c2
> Apr 14 21:42:12 cesar1 CRON[13599]: nss_ldap: could not connect to any
> LDAP server as cn=admin,dc=rz,dc=dawanda,dc=com - Can't contact LDAP server
> Apr 14 21:42:12 cesar1 CRON[13599]: nss_ldap: could not search LDAP
> server - Server is unavailable
> Apr 14 21:42:24 cesar1 crmd: [7287]: ERROR: process_lrm_event: LRM
> operation management-gateway-ip1_stop_0 (917) Timed Out (timeout=20000ms)
> Apr 14 21:42:48 cesar1 kernel: [27613688.611501] BUG: soft lockup -
> CPU#7 stuck for 22s! [named:32166]
> Apr 14 21:42:48 cesar1 kernel: [27613688.611914] Stack:
> Apr 14 21:42:48 cesar1 kernel: [27613688.612036] Call Trace:
> Apr 14 21:42:48 cesar1 kernel: [27613688.612200] <IRQ>
> Apr 14 21:42:48 cesar1 kernel: [27613688.612408] <EOI>
> Apr 14 21:42:48 cesar1 kernel: [27613688.612626] Code: c1 51 da 03 81 48
> c7 c2 4e da 03 81 e9 dd fe ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90
> 55 b8 00 00 01 00 48 89 e5 f0 0f c1 07 <89> c2
> Apr 14 21:42:55 cesar1 kernel: [27613695.946295] BUG: soft lockup -
> CPU#0 stuck for 21s! [ksoftirqd/0:3]
> Apr 14 21:42:55 cesar1 kernel: [27613695.946785] Stack:
> Apr 14 21:42:55 cesar1 kernel: [27613695.946917] Call Trace:
> Apr 14 21:42:55 cesar1 kernel: [27613695.947137] Code: c4 00 00 81 a8 44
> e0 ff ff ff 01 00 00 48 63 80 44 e0 ff ff a9 00 ff ff 07 74 36 65 48 8b
> 04 25 c8 c4 00 00 83 a8 44 e0 ff ff 01 <5d> c3
>
> We're using irqbalance to not only hit the first CPU for ethernet card
> hardware interrupts when traffic comes in (learned from last much more
> intensive DDoS).
To use irqbalance is wise. You could also try using receive packet
steering [1] [2]:
#!/bin/bash
iface='eth*'
flow=16384
echo $flow > /proc/sys/net/core/rps_sock_flow_entries
queues=(/sys/class/net/${iface}/queues/rx-*)
for rx in "${queues[@]}"; do
echo $(sed -e 's/0/f/g' < $rx/rps_cpus) > $rx/rps_cpus
echo $flow > $rx/rps_flow_cnt
done
I have found this to be beneficial on systems running networking
applications that are subject to a high load, but not for systems that
are simply forwarding packets and processing them entirely in kernel space.
> However, since this not helped, I'd like to find out what else we can
> do. Our gateway has to do NAT and has a few other iptables rules it
> needs in order to run OpenStack behind,
> so I can't just drop it.
>
> Regarding the logs, I can see, that something caused the CPU cores to
> get stuck for a number of different processes.
> Has anyone ever encountered such error messages I quoted above or knows
I used to encounter them but they cleared up at some point during the
3.4 (longterm) kernel series. If you also use the 3.4 series, I would
advise upgrading if running < 3.4.51. If you are not using a longterm
kernel, consider doing so unless there is a feature in a later kernel
that you cannot do without. My experience of the later 'stable' kernels
lately is that they have a tendency to introduce serious regressions.
> other things one might want to do in order to prevent hugh unsocialized
> incoming traffic from bringing a Linux node down?
If you can, talk with your upstream to see if there is a way in which
such traffic can be throttled there.
Be sure to use good quality NICs. In particular, it should support
multiqueue and adjustable interrupt coalescing (preferably on a dynamic
basis). For what it's worth, I'm using Intel 82576 based cards for busy
hosts. These support dynamic interrupt throttling. Even without such a
feature, some cards will allow their behaviour to be altered via ethtool
-C. Google will turn up a lot of information on this topic.
I should add that the stability of the driver is of paramount
performance. Though my Intel cards have been solid, the igb driver
bundled with the 3.4 kernel is not, which took me a long time to figure
out. I now use a local ebuild to compile the igb driver from upstream.
Not only did it improve performance, but it resolved all stability
issues that I had experienced up until then.
In the event that you are also using the igb driver, ensure that it is
configured optimally for multiqueue. Here's an example for the upstream
driver (my NIC has 4 ports, each with 8 queues):
# cat /etc/modprobe.d/igb.conf
options igb RSS=8,8,8,8
Enable I/OAT if your hardware supports it. Some hardware will support it
but fail to expose a BIOS option to enable it, in which case you can try
using dca_force [3] (YMMV). Similarly, make use of x2APIC if supported,
but do not make use of the IOMMU provided by Intel as of Nehalem (boot
with intel_iommu=off if in doubt).
Consider fine-tuning sysctl.conf, especially those pertaining to buffer
sizes/limits. I would consider this essential if operating at gigabit
speeds or higher. Examples are widespread, such as in section 3.1 of the
Mellanox performance tuning guide [4].
--Kerin
[1] https://lwn.net/Articles/361440/
[2] http://thread.gmane.org/gmane.linux.network/179883/focus=179976
[3] https://github.com/ice799/dca_force
[4]
http://www.mellanox.com/related-docs/prod_software/Performance_Tuning_Guide_for_Mellanox_Network_Adapters_rev_1_0.pdf
prev parent reply other threads:[~2013-07-28 14:01 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-04-15 15:07 [gentoo-server] DoS Analysis and Prevemption Christian Parpart
2013-07-28 14:01 ` Kerin Millar [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=51F5243A.2080208@fastmail.co.uk \
--to=kerframil@fastmail.co.uk \
--cc=gentoo-server@lists.gentoo.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox