From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id A7CC9138330 for ; Thu, 22 Sep 2016 16:58:49 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 46C5BE08C5; Thu, 22 Sep 2016 16:58:43 +0000 (UTC) Received: from mail-wm0-f49.google.com (mail-wm0-f49.google.com [74.125.82.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 0B652E07F7 for ; Thu, 22 Sep 2016 16:58:41 +0000 (UTC) Received: by mail-wm0-f49.google.com with SMTP id w84so264304668wmg.1 for ; Thu, 22 Sep 2016 09:58:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding; bh=47JXa87Ptwm+la7/pPAMUDNtupwIx/j+Yf6G+dgq/6I=; b=lx8WQkbQJWbgdkdp7S1Jmb9YR+fMEC7Zgfud9JtJarfgxXob139CbGwyvM+AJdgQ8j LN+GsvqYFWf3vQ9Re92XaypaqhL9svTSlCZ6BYttkAUBMzrIyOSCV1fIXkaNDCTi5WWP j9sWJDCbsUSoX2xT8hs7ILhz6LhUAFkYhoRvG3cf7UKDPSc8Xj2BhCO8xOnvhyA6grrh RFV7tvNsgtcjkw537xEBBnUs2g2ZeTQu79llhcMkNyT7eKCko0tSksoGLWtAUN1er7VQ 1HNk8dr8Lz+t0+gncSBBnvrDYsOn0bpt44gEWNETfdifOmGDNdaL+Zrk2vS50/3WAFV6 nMGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=47JXa87Ptwm+la7/pPAMUDNtupwIx/j+Yf6G+dgq/6I=; b=b6wYeGaacbkFCWOmgr7NsxkC++mNgZEVEMHafAXcJbRu64JlqvSlGAKx9tkcHdjZqv n9MJKpXGOCtmFofcM3zs8/dH3qjrHK0k7OQV7+i37Z5SgAA9VyRptjSX5+LCFh97qURr QlObXq/ogl/FgblymilZdj8AAb8CZMplRlO/Vps7BhDWkps6FGiEkiVs6Cbvoxa+cPfB MSjd4DSbHJFn7YSilUmCgPGv1Q6ZBzadDnAhEBmAGsV9GUKsABq5vAGR3xV+ZoGQcpZm q7W+oDLTa2yiOmGxgJkRkj6QkesMKJRS5LwBfRR9pbqIBZmx5/W76ImPswCp4hZK6ZHC WtDw== X-Gm-Message-State: AE9vXwMFFxsFQKYnKKXpEv6azEgcOT25RFqt57Muldt6jmOljBXBC9KtAW/B4D/tjZDhZw== X-Received: by 10.194.23.39 with SMTP id j7mr3400081wjf.4.1474563520184; Thu, 22 Sep 2016 09:58:40 -0700 (PDT) Received: from [192.168.178.21] (p4FC107C2.dip0.t-ipconnect.de. [79.193.7.194]) by smtp.googlemail.com with ESMTPSA id vh6sm3009000wjb.0.2016.09.22.09.58.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 22 Sep 2016 09:58:39 -0700 (PDT) Subject: Re: [gentoo-user] Re: TCP Queuing problem To: gentoo-user@lists.gentoo.org References: <501ED693-3FD7-4C58-B76D-3A811C948DFF@antarean.org> <243473F4-252D-4A88-B946-891190C3A24B@antarean.org> From: Volker Armin Hemmann Message-ID: <57E40DBE.3090603@googlemail.com> Date: Thu, 22 Sep 2016 18:58:38 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.7.1 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Archives-Salt: eb4c8f62-cf1c-4a12-8c79-8b439fca837c X-Archives-Hash: ce993c631816333b23d7661ebc87c2a8 Am 20.09.2016 um 21:52 schrieb Grant: >>>>>>>> My web server's response time for http requests skyrockets every >>>>>>>> weekday between about 9am and 5pm. I've gone over my munin >>> graphs >>>>> and >>>>>>>> the only one that really correlates well with the slowdown is >>> "TCP >>>>>>>> Queuing". It looks like I normally have about 400 packets per >>>>> second >>>>>>>> graphed as "direct copy from queue" in munin throughout the day, >>>>> but 2 >>>>>>>> to 3.5 times that many are periodically graphed during work >>> hours. >>>>> I >>>>>>>> don't see the same pattern at all from the graph of all traffic >>> on >>>>> my >>>>>>>> network interface which actually peaks over the weekend. TCP >>>>> Queuing >>>>>>>> doesn't rise above 400 packets per second all weekend. This is >>>>>>>> consistent week after week. >>>>>>>> >>>>>>>> My two employees come into work during the hours in question, and >>>>> they >>>>>>>> certainly make frequent requests of the web server while at work, >>>>> but >>>>>>>> if their volume of requests were the cause of the problem then >>> that >>>>>>>> would be reflected in the graph of web server requests but it is >>>>> not. >>>>>>>> I do run a small MTU on the systems at work due to the config of >>>>> the >>>>>>>> modem/router we have there. >>>>>>>> >>>>>>>> Is this a recognizable problem to anyone? >>>>>>> >>>>>>> I'm in the midst of this. Are there certain attacks I should >>> check >>>>> for? >>>>>> >>>>>> It looks like the TCP Queuing spike itself was due to imapproxy >>> which >>>>>> I've now disabled. I'll post more info as I gather it. >>>>> >>>>> imapproxy was clearly affecting the TCP Queuing graph in munin but I >>>>> still ended up with a massive TCP Queuing spike today and >>>>> corresponding http response time issues long after I disabled >>>>> imapproxy. Graph attached. I'm puzzled. >>>>> >>>>> - Grant >>>> Things to check for: >>>> Torrent or other distributed downloads. >>>> Download program with multiple download threads >>> >>> There sure shouldn't be anything like that running either on the >>> server or in the office. Is there a good way to find out? Maybe >>> something that would clearly indicate it? >>> >>> >>>> Maybe another proxy running? Esp. as you saw this also with >>> imapproxy. >>> >>> >>> nginx acts as a reverse proxy to apache2 but that's a pretty common >>> config. Nothing else that I know of. >>> >>> - Grant >> Any way to find out between which hosts/servers those connections are for? >> That might help in locating the cause. >> >> Eg. which of your desktops/laptops inside your network and where they are trying to connect to. > > The spikes are taking place on my remote server but they seem to > roughly coincide with user activity within my own network. My > technical knowledge of networking internals is weak. Does anyone know > which tool will tell me more about the connections that are causing > the TCP Queuing spikes? > > - Grant > > wireshark or whatever it is called at the moment?