Re: [gentoo-portage-dev] [PATCH] portage: HTTP if-modified-since and compression

[W-Mark Kubacki <wmark@hurrikane.de>]

On Thu, Aug 02, 2012 at 06:29:31PM -0700, Brian Dolbec wrote:
> On Thu, 2012-08-02 at 21:57 +0200, Mark Kubacki wrote:
> > 
> > Regarding functionality – there is still some room for more
> > optimizations and more features. For example, if the local copy is no
> > older than x seconds then there's no need to contact any remote
> > server. Expect patches.
> > 
> 
> Mark, I did similar for the layman-2.0 code which has been running with
> the header info for quite a while now.  After it had been running for a
> good amount of time I put in a request to infra for some usage stats.
> 
> The If-Modified-Since header does make a big difference for layman.
> Now I just really need to make a good blog post with a few graphs of the
> data.
> You can view the results on this bug if your interested:
> https://bugs.gentoo.org/show_bug.cgi?id=398465
> 

Brian, thanks for the stats and the pointer to layman. I guess we both 
see the opportunity to share some experiences and code. Layman can 
benefit from adding compression and I need to integrate your notices 
about Py2/Py3 compatibility.

If it is okay with Zac I will refactor and improve the URL-fetching some 
more. Following redirects, a proper auth-handler and 'identificator' 
comes to mind. You could copy the final handlers, then.

Portage's 'emerge' currently contacts remote hosts whenever it is run 
and this adds a noticeable delay. In the best case even the 304 (not 
modified) responses are avoid wherever possible. So in the end success 
of Portage's caching will not be measurable by a 200-to-304 ratio.

-- 
Mark

[1] http://trac.nginx.org/nginx/ticket/93 – discussion about unintuitive 
but valid handling of the If-Modified-Since header