From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from pigeon.gentoo.org ([208.92.234.80] helo=lists.gentoo.org) by finch.gentoo.org with esmtp (Exim 4.60) (envelope-from ) id 1RPDj8-00030Y-LT for garchives@archives.gentoo.org; Sat, 12 Nov 2011 13:34:34 +0000 Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 9C91DE0439; Sat, 12 Nov 2011 13:34:25 +0000 (UTC) Received: from svr-us4.tirtonadi.com (svr-us4.tirtonadi.com [69.65.43.212]) by pigeon.gentoo.org (Postfix) with ESMTP id B84B7E0330 for ; Sat, 12 Nov 2011 13:33:32 +0000 (UTC) Received: from mail-bw0-f53.google.com ([209.85.214.53]) by svr-us4.tirtonadi.com with esmtpsa (TLSv1:RC4-SHA:128) (Exim 4.69) (envelope-from ) id 1RPDi7-000fcm-FM for gentoo-user@lists.gentoo.org; Sat, 12 Nov 2011 20:33:31 +0700 Received: by bkaq10 with SMTP id q10so5326541bka.40 for ; Sat, 12 Nov 2011 05:33:28 -0800 (PST) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org MIME-Version: 1.0 Received: by 10.205.129.12 with SMTP id hg12mr4347352bkc.113.1321104808350; Sat, 12 Nov 2011 05:33:28 -0800 (PST) Received: by 10.223.74.16 with HTTP; Sat, 12 Nov 2011 05:33:28 -0800 (PST) Received: by 10.223.74.16 with HTTP; Sat, 12 Nov 2011 05:33:28 -0800 (PST) In-Reply-To: <4EBE7139.5000504@binarywings.net> References: <4EBE38F3.2000005@binarywings.net> <201111121155.41045.michaelkintzios@gmail.com> <4EBE7139.5000504@binarywings.net> Date: Sat, 12 Nov 2011 20:33:28 +0700 Message-ID: Subject: Re: [gentoo-user] The LIGHTEST web server (just for serving files)? From: Pandu Poluan To: gentoo-user@lists.gentoo.org Content-Type: multipart/alternative; boundary=0015173febced0165b04b189aefb X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - svr-us4.tirtonadi.com X-AntiAbuse: Original Domain - lists.gentoo.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - poluan.info X-Archives-Salt: ddd7d6f7-a6e6-4ecd-b4de-3994f8029b04 X-Archives-Hash: fb05fab9aa2c4a941b0dda271b8f870d --0015173febced0165b04b189aefb Content-Type: text/plain; charset=UTF-8 On Nov 12, 2011 8:20 PM, "Florian Philipp" wrote: > > Am 12.11.2011 13:40, schrieb Pandu Poluan: > > > > During my drive home, something hit my brain: why not have the 'master' > > server share the distfiles dir via NFS? > > > > So, the question now becomes: what's the drawback/benefit of NFS-sharing > > vs HTTP-sharing? The scenario is back-end LAN at the office, thus, a > > trusted network by definition. > > > > Rgds, > > > > How exactly had you planned to share distfiles? You didn't want to > mirror everything from the offical mirrors, did you? I'm not perfectly > sure how portage handles a mirror that occasionally returns 404 errors > but I think I've seen it fall back to the official mirrors in that case. Yes, portage (at least, 2.2) automatically use the next mirror in the list. > Anyway, making educated guesses about what should be on your own mirror > is probably a bit ineffective unless you have a very homogeneous > environment. > > What I think you /should/ have wanted is a proxy specifically configured > to cache very large files. net-proxy/http-replicator has been made > specifically for Gentoo distfiles. > I had planned on having a script peruse the log file, looking for which box got a 404, and 1 hour later try to move the file using scp from that box into the common local subrepo. But http-replicator sounds mighty better :-) > NFS has the advantage that it doesn't duplicate distfiles locally on all > machines. It is also easier to set up. Disadvantages? I'm unsure how > portage will handle cases when two machines fetch the same file at the > same time. I can always stagger the time my boxes fetch the distfiles. That should prevent locking problems. Rgds, --0015173febced0165b04b189aefb Content-Type: text/html; charset=UTF-8


On Nov 12, 2011 8:20 PM, "Florian Philipp" <lists@binarywings.net> wrote:
>
> Am 12.11.2011 13:40, schrieb Pandu Poluan:
> >
> > During my drive home, something hit my brain: why not have the 'master'
> > server share the distfiles dir via NFS?
> >
> > So, the question now becomes: what's the drawback/benefit of NFS-sharing
> > vs HTTP-sharing? The scenario is back-end LAN at the office, thus, a
> > trusted network by definition.
> >
> > Rgds,
> >
>
> How exactly had you planned to share distfiles? You didn't want to
> mirror everything from the offical mirrors, did you? I'm not perfectly
> sure how portage handles a mirror that occasionally returns 404 errors
> but I think I've seen it fall back to the official mirrors in that case.

Yes, portage (at least, 2.2) automatically use the next mirror in the list.

> Anyway, making educated guesses about what should be on your own mirror
> is probably a bit ineffective unless you have a very homogeneous
> environment.
>
> What I think you /should/ have wanted is a proxy specifically configured
> to cache very large files. net-proxy/http-replicator has been made
> specifically for Gentoo distfiles.
>

I had planned on having a script peruse the log file, looking for which box got a 404, and 1 hour later try to move the file using scp from that box into the common local subrepo.

But http-replicator sounds mighty better :-)

> NFS has the advantage that it doesn't duplicate distfiles locally on all
> machines. It is also easier to set up. Disadvantages? I'm unsure how
> portage will handle cases when two machines fetch the same file at the
> same time.

I can always stagger the time my boxes fetch the distfiles. That should prevent locking problems.

Rgds,

--0015173febced0165b04b189aefb--