From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id A0987138330 for ; Tue, 20 Sep 2016 19:53:07 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 7E63CE0BA7; Tue, 20 Sep 2016 19:52:59 +0000 (UTC) Received: from mail-qk0-f180.google.com (mail-qk0-f180.google.com [209.85.220.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id EC7CAE0B86 for ; Tue, 20 Sep 2016 19:52:58 +0000 (UTC) Received: by mail-qk0-f180.google.com with SMTP id n185so26182008qke.1 for ; Tue, 20 Sep 2016 12:52:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=5GLE5Sd7qHzOU1ObrQzwl74AF7pcx9H92uRf896qvBU=; b=LFFj6D6YD4xy5xE7xTmQxYUtHjAukayPvIjTo7X3jjvyqYiQfj5etVMyR4orCDL5we DM9NKsc94NH82OLeqCZ/T+rRuPyiYsp+3eVh68QtotMhKEpevHZacQoUJXrnJo8AEGoB 14qdnsVnfGlvmK8wX4x7UtnLbaTMVL/c67HMsfG81rrP/mz2z3xaHhli3yhNI5cCLdro stVeu8K/doIgukLIeo40znzkO1c+4UQVdXn7aZUyuzNyDFM7ilH4Pg9THaQi41pWucDI C/q+u9/y6zpuDJZJiVwp7ExJyBdQRMoVm5tImIwB39wX2LfNWskUxpDeD2IsDNNnQpYF JjrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=5GLE5Sd7qHzOU1ObrQzwl74AF7pcx9H92uRf896qvBU=; b=Atx9sfL/okojLQSRW7HVVoRR0LDNk/gYUh3gQOizpvi+zM/oM8NPLWduzk2oKaevPg sryfZ1p4ykdpJX+JtIFpE6n6+aBdhsC938M871ZWd+7k9YHcihgWFortDhNnpzdZkQMC /bwIHRC4XFdph2nhJTe9mbo0Rgb3tXgFnuvcHrrTVAvNP516B5SrwOsQdzOoP/PcFIFX ZH94MVe8ifq21AxWfEOmkvs378f/RcQHx04+oIJG5sUVbBhHbddlgv+Y3wKvWmYQ/00q Z9e8LYZ9GZKfU40c/Bm/zqHBlc+M4HdamvE0xfD8whsAqt4K9Fkq6y9ClIXbXBdp4ilV Ae5A== X-Gm-Message-State: AE9vXwMnFS7yTvY2ejp8JmiQXwfHVBMJC89+EgC90XE/qZVc5GfIgzToaE/4mEt3GdmwWLDgDJUp4Jt2OMFgcQ== X-Received: by 10.55.201.209 with SMTP id m78mr28246427qkl.308.1474401178019; Tue, 20 Sep 2016 12:52:58 -0700 (PDT) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org MIME-Version: 1.0 Received: by 10.140.38.179 with HTTP; Tue, 20 Sep 2016 12:52:57 -0700 (PDT) In-Reply-To: <243473F4-252D-4A88-B946-891190C3A24B@antarean.org> References: <501ED693-3FD7-4C58-B76D-3A811C948DFF@antarean.org> <243473F4-252D-4A88-B946-891190C3A24B@antarean.org> From: Grant Date: Tue, 20 Sep 2016 12:52:57 -0700 Message-ID: Subject: Re: [gentoo-user] Re: TCP Queuing problem To: Gentoo mailing list Content-Type: text/plain; charset=UTF-8 X-Archives-Salt: e107af1f-b067-431d-a8cf-e698a1b5bb9e X-Archives-Hash: 590091ac3c380f138f7cab4dcf5b27e4 >>>>>>> My web server's response time for http requests skyrockets every >>>>>>> weekday between about 9am and 5pm. I've gone over my munin >>graphs >>>>and >>>>>>> the only one that really correlates well with the slowdown is >>"TCP >>>>>>> Queuing". It looks like I normally have about 400 packets per >>>>second >>>>>>> graphed as "direct copy from queue" in munin throughout the day, >>>>but 2 >>>>>>> to 3.5 times that many are periodically graphed during work >>hours. >>>>I >>>>>>> don't see the same pattern at all from the graph of all traffic >>on >>>>my >>>>>>> network interface which actually peaks over the weekend. TCP >>>>Queuing >>>>>>> doesn't rise above 400 packets per second all weekend. This is >>>>>>> consistent week after week. >>>>>>> >>>>>>> My two employees come into work during the hours in question, and >>>>they >>>>>>> certainly make frequent requests of the web server while at work, >>>>but >>>>>>> if their volume of requests were the cause of the problem then >>that >>>>>>> would be reflected in the graph of web server requests but it is >>>>not. >>>>>>> I do run a small MTU on the systems at work due to the config of >>>>the >>>>>>> modem/router we have there. >>>>>>> >>>>>>> Is this a recognizable problem to anyone? >>>>>> >>>>>> >>>>>> I'm in the midst of this. Are there certain attacks I should >>check >>>>for? >>>>> >>>>> >>>>> It looks like the TCP Queuing spike itself was due to imapproxy >>which >>>>> I've now disabled. I'll post more info as I gather it. >>>> >>>> >>>>imapproxy was clearly affecting the TCP Queuing graph in munin but I >>>>still ended up with a massive TCP Queuing spike today and >>>>corresponding http response time issues long after I disabled >>>>imapproxy. Graph attached. I'm puzzled. >>>> >>>>- Grant >>> >>> Things to check for: >>> Torrent or other distributed downloads. >>> Download program with multiple download threads >> >> >>There sure shouldn't be anything like that running either on the >>server or in the office. Is there a good way to find out? Maybe >>something that would clearly indicate it? >> >> >>> Maybe another proxy running? Esp. as you saw this also with >>imapproxy. >> >> >>nginx acts as a reverse proxy to apache2 but that's a pretty common >>config. Nothing else that I know of. >> >>- Grant > > Any way to find out between which hosts/servers those connections are for? > That might help in locating the cause. > > Eg. which of your desktops/laptops inside your network and where they are trying to connect to. The spikes are taking place on my remote server but they seem to roughly coincide with user activity within my own network. My technical knowledge of networking internals is weak. Does anyone know which tool will tell me more about the connections that are causing the TCP Queuing spikes? - Grant