From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1RXSho-0002Yw-KS for garchives@archives.gentoo.org; Mon, 05 Dec 2011 07:11:17 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 06D3821C024; Mon, 5 Dec 2011 07:11:03 +0000 (UTC) Received: from mail-fx0-f53.google.com (mail-fx0-f53.google.com [209.85.161.53]) by pigeon.gentoo.org (Postfix) with ESMTP id 62CA421C01F for ; Mon, 5 Dec 2011 07:10:10 +0000 (UTC) Received: by faai28 with SMTP id i28so2498511faa.40 for ; Sun, 04 Dec 2011 23:10:09 -0800 (PST) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org MIME-Version: 1.0 Received: by 10.227.207.146 with SMTP id fy18mr18228749wbb.14.1323069009382; Sun, 04 Dec 2011 23:10:09 -0800 (PST) Sender: antarus@scriptkitty.com Received: by 10.227.206.197 with HTTP; Sun, 4 Dec 2011 23:10:09 -0800 (PST) Date: Sun, 4 Dec 2011 23:10:09 -0800 X-Google-Sender-Auth: H-CkemUhXKmKt88SkjyYzUWQus8 Message-ID: Subject: [gentoo-dev] sources.gentoo.org instability From: Alec Warner To: Gentoo Dev Content-Type: text/plain; charset=UTF-8 X-Archives-Salt: 7d119169-d5a0-4f49-bd05-36c210137a70 X-Archives-Hash: ac699dbfa77d815132025ac45da026b4 Hello, For a while sources.gentoo.org has been puttering along and its health has slowly declined. We migrated it to some newer shiny hardware in an attempt to mitigate the problem but that did not pan out. 90% (or more) of sources.gentoo.org traffic is crawler bots and not actual humans. That being said; if we cannot serve requests to the bots within our timeouts we serve 500's instead which is never really what we want (particularly when we spent 20s of CPU to calculate 80% of the response only to see the client timeout :/.) The majority of the expensive requests are related to package.mask and use.local.desc queries by crawlers. Like crawling the entire 13000 rev history for package.mask (or similar.) While it is likely we will monkey patch viewvc to be less wasteful; in the meantime I have removed use.local.desc from sources.gentoo.org (and also anoncvs, because they share the same repo.) I hope this is a short term (order of weeks) hack. -A