From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 8F122138330 for ; Sat, 1 Oct 2016 09:57:22 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id F421F21C09E; Sat, 1 Oct 2016 09:57:09 +0000 (UTC) Received: from mail-qk0-f171.google.com (mail-qk0-f171.google.com [209.85.220.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id AA22F21C04E for ; Sat, 1 Oct 2016 09:57:09 +0000 (UTC) Received: by mail-qk0-f171.google.com with SMTP id n189so54498336qke.0 for ; Sat, 01 Oct 2016 02:57:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=ruFKX5KqZDGIs9RDt1mg4XcmrI6gAbQPADzfn6sPt0U=; b=gs/AXF5HCxomMOCsgBlLSa97s5cAU8CFxUxT9NL4BQlUZGJwPJHZ/245ztDtE75x7H KOTix9stSRxgtKz3XcoasFk3fgNjFylz2JlqPOh8xA8zD3bbEQS64Z4dwtjh2ik0mmMe nRxxOuWxh3GrE2wLKMX6i1AKfGL6McO/U70BnzbyJmXnuL+lEcW1/maX5NI6oEMXYBfx sFBZ94oVtlQ9lZOgriFVkV+/m7OxQeXmnK247ty74cmXfUzvkZLY585wuEnDQhvHyWI3 c1jpRPGMNPxakTvpgrNcxuIiyDmGIotmzDCfUW4EgwZRnNOfEmpfOvL4YM53SYTWExS6 SauA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=ruFKX5KqZDGIs9RDt1mg4XcmrI6gAbQPADzfn6sPt0U=; b=b5iDzPBRi68dnY6WwxLCRyygopoTglpujSAPsqIZiepUiyfMpyudO9f0V9j7Xpa786 773px7RSZWiEbSaoxj8q40VsZ99Y/tsvpoyh160rhgicenA9gbjxr2Lk2VJwtlmq8clL Ux3aa06VrKHinJgq/j9630GOonhCXpdWFCobuD6PcK2qELnxNc5JKjWPKMs8+W5dfFRp 3sWaZe4MjK/0wbHcLYzR3hRlO5iu/s7TiLVGUT/3Oukk3qDmEGY9WZ0kwxirY9j7ZmlK MrqiEmZ0hXRurA/kofn7CsYA+oihX9gqM9sxF9ToxKI7LwfM8k5+HIMdlbZYiLqRIDFz tceQ== X-Gm-Message-State: AA6/9Rn201MemE4Oq0fdbpKzKQc1hWxHMxmMAQ6bWz3aZjJ6Zd0lgQX3F0Uqwf2F7lQQu0KXBsIEaz3h43NnFA== X-Received: by 10.55.43.22 with SMTP id r22mr11237328qkh.308.1475315828527; Sat, 01 Oct 2016 02:57:08 -0700 (PDT) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org MIME-Version: 1.0 Received: by 10.140.38.179 with HTTP; Sat, 1 Oct 2016 02:57:07 -0700 (PDT) In-Reply-To: References: <20160921060118.759d94ba@jupiter.sol.kaishome.de> <20160921212913.352eae72@jupiter.sol.kaishome.de> <20160921214138.3ca8a146@jupiter.sol.kaishome.de> <20160921221839.29ceae91@jupiter.sol.kaishome.de> <20160922090628.73d6a3b1@jupiter.sol.kaishome.de> From: Grant Date: Sat, 1 Oct 2016 02:57:07 -0700 Message-ID: Subject: Re: [gentoo-user] Re: TCP Queuing problem To: Gentoo mailing list Content-Type: text/plain; charset=UTF-8 X-Archives-Salt: b27c56d8-682e-4d14-b469-a5bf124692d5 X-Archives-Hash: 74cc04875684aa14a71b67e453e4d9fe > It turned out this was a combination of two problems which made it > much more difficult to figure out. > > First of all I didn't have enough apache2 processes. That seems like > it should have been obvious but it wasn't for two reasons. Firstly, > my apache2 processes are always idle or nearly idle, even when traffic > levels are high. But it must be the case that each request made to > nginx which is then handed off to apache2 monopolizes an apache2 > process even though my backend application server is the one using all > the CPU instead of apache2. The other thing that made it difficult to > track down was the way munin graphs apache2 processes. On my graph, > busy and free processes only appeared as tiny dots at the bottom > because apache2's ServerLimit is drawn on the same graph which is many > times greater than the number of busy and free processes. It would be > better to draw MaxClients instead of ServerLimit since I think > MaxClients is more likely to be tuned. It at least appears in the > default config file on Gentoo. Since busy and free apache2 processes > were virtually invisible on the munin graph, I wasn't able to > correlate their ebb and flow with my server's response times. > > Once I fixed the apache2 problem, I was sure I had it nailed. That's > when I emailed here a few days ago to say I think I got it. But it > turned out there was another problem and that was Odoo (formerly known > as OpenERP) which is also running in a reverse proxy configuration > behind nginx. Whenever someone uses Odoo on my server, it absolutely > destroys performance for my non-Odoo website. That would have been > really easy to test and I did test stopping the odoo service early on, > but I ruled it out when the problem persisted after stopping Odoo > which I now realize must have been because of the apache2 problem. The root of the Odoo problem was that I didn't have keepalive enabled between the nginx reverse proxy server and the Odoo server. nginx enables keepalive by default for the client side (HTTP/1.1) but not for the upstream side (HTTP/1.0). I still see TCP Queuing spikes in munin with Odoo usage, but they no longer slow down the apache2/nginx reverse proxy running my main site. - Grant