From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 28FEA13800E for ; Fri, 3 Aug 2012 12:08:37 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id C02F5E07AF for ; Fri, 3 Aug 2012 12:08:36 +0000 (UTC) Received: from mail-ee0-f53.google.com (mail-ee0-f53.google.com [74.125.83.53]) by pigeon.gentoo.org (Postfix) with ESMTP id BA556E05EB for ; Fri, 3 Aug 2012 09:33:12 +0000 (UTC) Received: by eekc4 with SMTP id c4so99513eek.40 for ; Fri, 03 Aug 2012 02:33:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=date:from:to:subject:message-id:references:mime-version :content-type:content-disposition:content-transfer-encoding :in-reply-to:user-agent:x-gm-message-state; bh=cXhy52vX+MGmLAXB4pP7w+TbJSxW43o3oeTgsADYHFg=; b=J94+mFYxFN60j0sKrnKj5ywpJmdG19H6A3tf5eQUQ35mrbUg/Fi7w2vpcR7z/kIRhb 5hM+sylQg78GQ3f8CW8gnvrpCyzQntCA2SoTLe44P1ahJZOfJmLwLYjdgFnWrpLLoyvw UAiqAgcoNzgtgPxcS+Z41afJJaKSy4++zgsf5q81zSSR3iT5oVCJ6uasx1RK/7OpYj3F QqOOxSJBxNwuu1286P9vHpc/3l88X6FfnNSZRjF+HiH/Gdc5IZPOGwN6eoJjSUZNw64Q SWc75GVkOLzoaqwBdKbN7Kvn7F77NM9tMGE/BUr2bk08CuECwFU4Z/pb0c028Dttdeo0 BI4A== Received: by 10.14.207.137 with SMTP id n9mr1315967eeo.34.1343986391281; Fri, 03 Aug 2012 02:33:11 -0700 (PDT) Received: from 3-0.hurrikane.de (ns2.hurrikane.de. [2001:4dd0:ff00:8cb6::2]) by mx.google.com with ESMTPS id g42sm24111183eem.14.2012.08.03.02.33.10 (version=SSLv3 cipher=OTHER); Fri, 03 Aug 2012 02:33:10 -0700 (PDT) Date: Fri, 3 Aug 2012 11:33:04 +0200 From: W-Mark Kubacki To: gentoo-portage-dev@lists.gentoo.org Subject: Re: [gentoo-portage-dev] [PATCH] portage: HTTP if-modified-since and compression Message-ID: <20120803093304.GA26033@3-0.hurrikane.de> References: <5019D1B7.5050807@gentoo.org> <5019E686.5030407@gentoo.org> <1343957371.28838.64.camel@big_daddy.dol-sen.ca> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-portage-dev@lists.gentoo.org Reply-to: gentoo-portage-dev@lists.gentoo.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <1343957371.28838.64.camel@big_daddy.dol-sen.ca> User-Agent: Mutt/1.5.21 (2010-09-15) X-Gm-Message-State: ALoCoQm1p18S4A2bkA9Uj/Y7K4o/XjaTFHcgItmVo0unx+H/Htaw60vulzPxp9q0XgIo1FV7se5C X-Archives-Salt: 3524fe86-c2bd-4c74-a3ef-87f8e1fbe6ac X-Archives-Hash: 7dc363a58ccfbe907df4cf6812d26ac8 On Thu, Aug 02, 2012 at 06:29:31PM -0700, Brian Dolbec wrote: > On Thu, 2012-08-02 at 21:57 +0200, Mark Kubacki wrote: > > > > Regarding functionality – there is still some room for more > > optimizations and more features. For example, if the local copy is no > > older than x seconds then there's no need to contact any remote > > server. Expect patches. > > > > Mark, I did similar for the layman-2.0 code which has been running with > the header info for quite a while now. After it had been running for a > good amount of time I put in a request to infra for some usage stats. > > The If-Modified-Since header does make a big difference for layman. > Now I just really need to make a good blog post with a few graphs of the > data. > You can view the results on this bug if your interested: > https://bugs.gentoo.org/show_bug.cgi?id=398465 > Brian, thanks for the stats and the pointer to layman. I guess we both see the opportunity to share some experiences and code. Layman can benefit from adding compression and I need to integrate your notices about Py2/Py3 compatibility. If it is okay with Zac I will refactor and improve the URL-fetching some more. Following redirects, a proper auth-handler and 'identificator' comes to mind. You could copy the final handlers, then. Portage's 'emerge' currently contacts remote hosts whenever it is run and this adds a noticeable delay. In the best case even the 304 (not modified) responses are avoid wherever possible. So in the end success of Portage's caching will not be measurable by a 200-to-304 ratio. -- Mark [1] http://trac.nginx.org/nginx/ticket/93 – discussion about unintuitive but valid handling of the If-Modified-Since header