From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 44548138330 for ; Tue, 20 Sep 2016 18:06:19 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 96FC5E0B8B; Tue, 20 Sep 2016 18:06:12 +0000 (UTC) Received: from gw2.antarean.org (gw2.antarean.org [141.105.125.208]) by pigeon.gentoo.org (Postfix) with ESMTP id E5064E0B60 for ; Tue, 20 Sep 2016 18:06:09 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by gw2.antarean.org (Postfix) with ESMTP id 83CC31249EC for ; Tue, 20 Sep 2016 18:02:30 +0000 () X-Virus-Scanned: amavisd-new at antarean.org Received: from gw2.antarean.org ([127.0.0.1]) by localhost (gw2.antarean.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id k0Q36VxfCMkQ for ; Tue, 20 Sep 2016 18:02:28 +0000 (%Z) Received: from data.antarean.org (localhost [127.0.0.1]) by gw2.antarean.org (Postfix) with ESMTP id D348B1240FE for ; Tue, 20 Sep 2016 18:02:28 +0000 () Received: from lan016.nl.antarean.org (lan016.nl.antarean.org [10.20.13.116]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by data.antarean.org (Postfix) with ESMTPSA id D75E64C for ; Tue, 20 Sep 2016 20:02:50 +0200 (CEST) In-Reply-To: References: <501ED693-3FD7-4C58-B76D-3A811C948DFF@antarean.org> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Subject: Re: [gentoo-user] Re: TCP Queuing problem From: "J. Roeleveld" Date: Tue, 20 Sep 2016 18:06:05 +0000 To: gentoo-user@lists.gentoo.org Message-ID: <243473F4-252D-4A88-B946-891190C3A24B@antarean.org> X-Archives-Salt: 07094a1c-7862-4fa0-a40a-a40c3d7853c0 X-Archives-Hash: 47f9d491387e638fc9c1ba6bc32d4ccc On September 20, 2016 4:53:41 PM GMT+02:00, Grant wrote: >>>>>> My web server's response time for http requests skyrockets every >>>>>> weekday between about 9am and 5pm. I've gone over my munin >graphs >>>and >>>>>> the only one that really correlates well with the slowdown is >"TCP >>>>>> Queuing". It looks like I normally have about 400 packets per >>>second >>>>>> graphed as "direct copy from queue" in munin throughout the day, >>>but 2 >>>>>> to 3.5 times that many are periodically graphed during work >hours. >>>I >>>>>> don't see the same pattern at all from the graph of all traffic >on >>>my >>>>>> network interface which actually peaks over the weekend. TCP >>>Queuing >>>>>> doesn't rise above 400 packets per second all weekend. This is >>>>>> consistent week after week. >>>>>> >>>>>> My two employees come into work during the hours in question, and >>>they >>>>>> certainly make frequent requests of the web server while at work, >>>but >>>>>> if their volume of requests were the cause of the problem then >that >>>>>> would be reflected in the graph of web server requests but it is >>>not. >>>>>> I do run a small MTU on the systems at work due to the config of >>>the >>>>>> modem/router we have there. >>>>>> >>>>>> Is this a recognizable problem to anyone? >>>>> >>>>> >>>>> I'm in the midst of this. Are there certain attacks I should >check >>>for? >>>> >>>> >>>> It looks like the TCP Queuing spike itself was due to imapproxy >which >>>> I've now disabled. I'll post more info as I gather it. >>> >>> >>>imapproxy was clearly affecting the TCP Queuing graph in munin but I >>>still ended up with a massive TCP Queuing spike today and >>>corresponding http response time issues long after I disabled >>>imapproxy. Graph attached. I'm puzzled. >>> >>>- Grant >> >> Things to check for: >> Torrent or other distributed downloads. >> Download program with multiple download threads > > >There sure shouldn't be anything like that running either on the >server or in the office. Is there a good way to find out? Maybe >something that would clearly indicate it? > > >> Maybe another proxy running? Esp. as you saw this also with >imapproxy. > > >nginx acts as a reverse proxy to apache2 but that's a pretty common >config. Nothing else that I know of. > >- Grant Any way to find out between which hosts/servers those connections are for? That might help in locating the cause. Eg. which of your desktops/laptops inside your network and where they are trying to connect to. -- Joost -- Sent from my Android device with K-9 Mail. Please excuse my brevity.