From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 9CF0D138330 for ; Tue, 20 Sep 2016 14:53:51 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id AA467E0BED; Tue, 20 Sep 2016 14:53:44 +0000 (UTC) Received: from mail-qk0-f173.google.com (mail-qk0-f173.google.com [209.85.220.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 1CC6EE07FD for ; Tue, 20 Sep 2016 14:53:43 +0000 (UTC) Received: by mail-qk0-f173.google.com with SMTP id w204so17342437qka.0 for ; Tue, 20 Sep 2016 07:53:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=3KMMzc8bKMVJWBbYGj5LaohWOVsR6CSOJV1LPCE/Xxw=; b=PG2diOG3NTDTnRtXo0o8th5XArbAiyPpNvOYVSFBVm8B0EzNRXwLhzcuaW1VzvgWT8 ofHs5OTh7SJvJMpPNbKtYEjomt/2ClL/DjJqfEf7Sm1EGPPHCMXKn97MUjXdZ8Sz3aqK H6d94whnqgzP6nX83oSGBnhqR9LgXPa9vKmtf78mV68Lmca3D6XJl9Xvd9h6U4wZ0jBr TJyGRgzUY9vsbDfQzICW0P/TL9dLtOkDwcdCn0ZHqgggJovl/3kFyBURzHhAZv9rvVNI BXPbZWRiFeBZRIF9j8hmUGiODF5foibMm/aBcH+UkcVIYkpov4lAv3kn9e72zyw3saBl xNCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=3KMMzc8bKMVJWBbYGj5LaohWOVsR6CSOJV1LPCE/Xxw=; b=J9KyQ/sNGQkY/DZ8FhfOtWicN5kDeMTyA1ttl9gp+rysbZN42UJ5Z+tD70ISB+1III OikkkYdskM32ZP6uHdu8LKVQqm28NNOKedQXsGXpE8Gt55ch8nVjbS5a9flbJocy3UVy x6YoHwaWp/ZNFSQsc8yr0zZy0LrDYAxF+tqObpJLpL+Ph5iNQLdzCYhRer5trmrE86DO O2H1LTQwZudWlVH4Y2iNBFQ758pgYXNZ9l64Bth1qnrPqXm0AH8a38cGlaKuI+z3u7Gm OL4PsM2DKRAeA+LoRKQo+vx0GjBCDG1wa8G+Z5EsMSQCCOLCy/wHEftNOywnrw2r674w zR7w== X-Gm-Message-State: AE9vXwPXwuUeTCqO/TROdnv3XfLZhEqC7YRGhGnmJdh9pbo68fF/aDo1qbdS+t+8PdvzyOjV8iCJptzsch/Mpw== X-Received: by 10.55.201.209 with SMTP id m78mr26685498qkl.308.1474383222024; Tue, 20 Sep 2016 07:53:42 -0700 (PDT) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org MIME-Version: 1.0 Received: by 10.140.38.179 with HTTP; Tue, 20 Sep 2016 07:53:41 -0700 (PDT) In-Reply-To: <501ED693-3FD7-4C58-B76D-3A811C948DFF@antarean.org> References: <501ED693-3FD7-4C58-B76D-3A811C948DFF@antarean.org> From: Grant Date: Tue, 20 Sep 2016 07:53:41 -0700 Message-ID: Subject: Re: [gentoo-user] Re: TCP Queuing problem To: Gentoo mailing list Content-Type: text/plain; charset=UTF-8 X-Archives-Salt: 1c4c9195-81e9-4b63-8771-39c89255d9b5 X-Archives-Hash: ff813781d730b161c25835280eeb75ff >>>>> My web server's response time for http requests skyrockets every >>>>> weekday between about 9am and 5pm. I've gone over my munin graphs >>and >>>>> the only one that really correlates well with the slowdown is "TCP >>>>> Queuing". It looks like I normally have about 400 packets per >>second >>>>> graphed as "direct copy from queue" in munin throughout the day, >>but 2 >>>>> to 3.5 times that many are periodically graphed during work hours. >>I >>>>> don't see the same pattern at all from the graph of all traffic on >>my >>>>> network interface which actually peaks over the weekend. TCP >>Queuing >>>>> doesn't rise above 400 packets per second all weekend. This is >>>>> consistent week after week. >>>>> >>>>> My two employees come into work during the hours in question, and >>they >>>>> certainly make frequent requests of the web server while at work, >>but >>>>> if their volume of requests were the cause of the problem then that >>>>> would be reflected in the graph of web server requests but it is >>not. >>>>> I do run a small MTU on the systems at work due to the config of >>the >>>>> modem/router we have there. >>>>> >>>>> Is this a recognizable problem to anyone? >>>> >>>> >>>> I'm in the midst of this. Are there certain attacks I should check >>for? >>> >>> >>> It looks like the TCP Queuing spike itself was due to imapproxy which >>> I've now disabled. I'll post more info as I gather it. >> >> >>imapproxy was clearly affecting the TCP Queuing graph in munin but I >>still ended up with a massive TCP Queuing spike today and >>corresponding http response time issues long after I disabled >>imapproxy. Graph attached. I'm puzzled. >> >>- Grant > > Things to check for: > Torrent or other distributed downloads. > Download program with multiple download threads There sure shouldn't be anything like that running either on the server or in the office. Is there a good way to find out? Maybe something that would clearly indicate it? > Maybe another proxy running? Esp. as you saw this also with imapproxy. nginx acts as a reverse proxy to apache2 but that's a pretty common config. Nothing else that I know of. - Grant