[gentoo-user] The LIGHTEST web server (just for serving files) ?

[gentoo-user] The LIGHTEST web server (just for serving files) ?

[Pandu Poluan]

What is the *LIGHTEST* web server package you know for gentoo?

I just want to serve the distfiles, so no CGI / PHP /
whathaveyouscripting support is needed.

Preferably, with logging so I can see which packages I missed, but not
necessary.

Rgds,
-- 
FdS Pandu E Poluan
~ IT Optimizer ~

 • LOPSA Member #15248
 • Blog : http://pepoluan.tumblr.com
 • Linked-In : http://id.linkedin.com/in/pepoluan

Re: [gentoo-user] The LIGHTEST web server (just for serving files) ?

[Nilesh Govindarajan]

On Sat 12 Nov 2011 01:54:10 PM IST, Pandu Poluan wrote:
> What is the *LIGHTEST* web server package you know for gentoo?
>
> I just want to serve the distfiles, so no CGI / PHP /
> whathaveyouscripting support is needed.
>
> Preferably, with logging so I can see which packages I missed, but not
> necessary.
>
> Rgds,

nginx.
You can disable fastcgi/etc using use flags.

-- 
Nilesh Govindarajan
http://nileshgr.com

Re: [gentoo-user] The LIGHTEST web server (just for serving files) ?

[Pandu Poluan]

On Sat, Nov 12, 2011 at 15:46, Nilesh Govindarajan <contact@nileshgr.com> wrote:
> On Sat 12 Nov 2011 01:54:10 PM IST, Pandu Poluan wrote:
>> What is the *LIGHTEST* web server package you know for gentoo?
>>
>> I just want to serve the distfiles, so no CGI / PHP /
>> whathaveyouscripting support is needed.
>>
>> Preferably, with logging so I can see which packages I missed, but not
>> necessary.
>>
>> Rgds,
>
> nginx.
> You can disable fastcgi/etc using use flags.
>

What about www-servers/fnord ?

Its website[1] claims that its binaries are less than 20 kB[2]

[1] http://www.fefe.de/fnord/
[2] http://www.fefe.de/fnord/others.html

Rgds,
-- 
FdS Pandu E Poluan
~ IT Optimizer ~

 • LOPSA Member #15248
 • Blog : http://pepoluan.tumblr.com
 • Linked-In : http://id.linkedin.com/in/pepoluan

Re: [gentoo-user] The LIGHTEST web server (just for serving files) ?

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 2321 bytes --]

Am 12.11.2011 10:01, schrieb Pandu Poluan:
> On Sat, Nov 12, 2011 at 15:46, Nilesh Govindarajan <contact@nileshgr.com> wrote:
>> On Sat 12 Nov 2011 01:54:10 PM IST, Pandu Poluan wrote:
>>> What is the *LIGHTEST* web server package you know for gentoo?
>>>
>>> I just want to serve the distfiles, so no CGI / PHP /
>>> whathaveyouscripting support is needed.
>>>
>>> Preferably, with logging so I can see which packages I missed, but not
>>> necessary.
>>>
>>> Rgds,
>>
>> nginx.
>> You can disable fastcgi/etc using use flags.
>>
> 
> What about www-servers/fnord ?
> 
> Its website[1] claims that its binaries are less than 20 kB[2]
> 
> [1] http://www.fefe.de/fnord/
> [2] http://www.fefe.de/fnord/others.html
> 
> Rgds,

Talking about small: `/bin/busybox httpd`

I goes without saying that this is not meant to be web-facing.

Most things are explained in the --help output but the config file
format is only explained in a source code comment. See below:

httpd.conf has the following format:

H:/serverroot     # define the server root. It will override -h
A:172.20.         # Allow address from 172.20.0.0/16
A:10.0.0.0/25     # Allow any address from 10.0.0.0-10.0.0.127
A:10.0.0.0/255.255.255.128  # Allow any address that previous set
A:127.0.0.1       # Allow local loopback connections
D:*               # Deny from other IP connections
E404:/path/e404.html # /path/e404.html is the 404 (not found) error page
I:index.html      # Show index.html when a directory is requested

P:/url:[http://]hostname[:port]/new/path
          # When /urlXXXXXX is requested, reverse proxy
          # it to http://hostname[:port]/new/pathXXXXXX

/cgi-bin:foo:bar
          # Require user foo, pwd bar on urls starting with /cgi-bin/
/adm:admin:setup
          # Require user admin, pwd setup on urls starting with /adm/
adm:toor:PaSsWd  # or user toor, pwd PaSsWd on urls starting with /adm/
.au:audio/basic   # additional mime type for audio.au files
*.php:/path/php   # run xxx.php through an interpreter

A/D may be as a/d or allow/deny - only first char matters.
Deny/Allow IP logic:
- Default is to allow all (Allow all (A:*) is a no-op).
- Deny rules take precedence over allow rules.
- "Deny all" rule (D:*) is applied last.

Regards,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] The LIGHTEST web server (just for serving files)?

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 3297 bytes --]

On Saturday 12 Nov 2011 09:14:27 Florian Philipp wrote:
> Am 12.11.2011 10:01, schrieb Pandu Poluan:
> > On Sat, Nov 12, 2011 at 15:46, Nilesh Govindarajan <contact@nileshgr.com> 
wrote:
> >> On Sat 12 Nov 2011 01:54:10 PM IST, Pandu Poluan wrote:
> >>> What is the *LIGHTEST* web server package you know for gentoo?
> >>> 
> >>> I just want to serve the distfiles, so no CGI / PHP /
> >>> whathaveyouscripting support is needed.
> >>> 
> >>> Preferably, with logging so I can see which packages I missed, but not
> >>> necessary.
> >>> 
> >>> Rgds,
> >> 
> >> nginx.
> >> You can disable fastcgi/etc using use flags.
> > 
> > What about www-servers/fnord ?
> > 
> > Its website[1] claims that its binaries are less than 20 kB[2]
> > 
> > [1] http://www.fefe.de/fnord/
> > [2] http://www.fefe.de/fnord/others.html
> > 
> > Rgds,
> 
> Talking about small: `/bin/busybox httpd`
> 
> I goes without saying that this is not meant to be web-facing.
> 
> Most things are explained in the --help output but the config file
> format is only explained in a source code comment. See below:
> 
> httpd.conf has the following format:
> 
> H:/serverroot     # define the server root. It will override -h
> A:172.20.         # Allow address from 172.20.0.0/16
> A:10.0.0.0/25     # Allow any address from 10.0.0.0-10.0.0.127
> A:10.0.0.0/255.255.255.128  # Allow any address that previous set
> A:127.0.0.1       # Allow local loopback connections
> D:*               # Deny from other IP connections
> E404:/path/e404.html # /path/e404.html is the 404 (not found) error page
> I:index.html      # Show index.html when a directory is requested
> 
> P:/url:[http://]hostname[:port]/new/path
>           # When /urlXXXXXX is requested, reverse proxy
>           # it to http://hostname[:port]/new/pathXXXXXX
> 
> /cgi-bin:foo:bar
>           # Require user foo, pwd bar on urls starting with /cgi-bin/
> /adm:admin:setup
>           # Require user admin, pwd setup on urls starting with /adm/
> adm:toor:PaSsWd  # or user toor, pwd PaSsWd on urls starting with /adm/
> .au:audio/basic   # additional mime type for audio.au files
> *.php:/path/php   # run xxx.php through an interpreter
> 
> A/D may be as a/d or allow/deny - only first char matters.
> Deny/Allow IP logic:
> - Default is to allow all (Allow all (A:*) is a no-op).
> - Deny rules take precedence over allow rules.
> - "Deny all" rule (D:*) is applied last.

I've been using boa just for this purpose for years:

* www-servers/boa
     Available versions:  
		~	0.94.14_rc21 "~x86 ~sparc ~mips ~ppc ~amd64" [doc]
     Homepage:            http://www.boa.org/
     Description:         A very small and very fast http daemon.

It can be easily locked down for internet facing roles.

I've also used thttpd (you can throttle its bandwidth if that's important in 
your network), but it's probably more than required for this purpose:

* www-servers/thttpd
     Available versions:  
			2.25b-r7 "amd64 ~hppa ~mips ppc sparc x86 ~x86-fbsd" [static]
		~	2.25b-r8 "~amd64 ~hppa ~mips ~ppc ~sparc ~x86 ~x86-fbsd" 
[static]
     Homepage:            http://www.acme.com/software/thttpd/
     Description:         Small and fast multiplexing webserver.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] The LIGHTEST web server (just for serving files)?

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 1257 bytes --]

On Nov 12, 2011 7:00 PM, "Mick" <michaelkintzios@gmail.com> wrote:
>
> I've been using boa just for this purpose for years:
>
> * www-servers/boa
>     Available versions:
>                ~       0.94.14_rc21 "~x86 ~sparc ~mips ~ppc ~amd64" [doc]
>     Homepage:            http://www.boa.org/
>     Description:         A very small and very fast http daemon.
>
> It can be easily locked down for internet facing roles.
>
> I've also used thttpd (you can throttle its bandwidth if that's important
in
> your network), but it's probably more than required for this purpose:
>
> * www-servers/thttpd
>     Available versions:
>                        2.25b-r7 "amd64 ~hppa ~mips ppc sparc x86
~x86-fbsd" [static]
>                ~       2.25b-r8 "~amd64 ~hppa ~mips ~ppc ~sparc ~x86
~x86-fbsd"
> [static]
>     Homepage:            http://www.acme.com/software/thttpd/
>     Description:         Small and fast multiplexing webserver.

Thanks for all the input!

During my drive home, something hit my brain: why not have the 'master'
server share the distfiles dir via NFS?

So, the question now becomes: what's the drawback/benefit of NFS-sharing vs
HTTP-sharing? The scenario is back-end LAN at the office, thus, a trusted
network by definition.

Rgds,

[-- Attachment #2: Type: text/html, Size: 1745 bytes --]

Re: [gentoo-user] The LIGHTEST web server (just for serving files)?

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 1921 bytes --]

On Saturday 12 Nov 2011 12:40:08 Pandu Poluan wrote:
> On Nov 12, 2011 7:00 PM, "Mick" <michaelkintzios@gmail.com> wrote:
> > I've been using boa just for this purpose for years:
> > 
> > * www-servers/boa
> > 
> >     Available versions:
> >                ~       0.94.14_rc21 "~x86 ~sparc ~mips ~ppc ~amd64" [doc]
> >     
> >     Homepage:            http://www.boa.org/
> >     Description:         A very small and very fast http daemon.
> > 
> > It can be easily locked down for internet facing roles.
> > 
> > I've also used thttpd (you can throttle its bandwidth if that's important
> 
> in
> 
> > your network), but it's probably more than required for this purpose:
> > 
> > * www-servers/thttpd
> > 
> >     Available versions:
> >                        2.25b-r7 "amd64 ~hppa ~mips ppc sparc x86
> 
> ~x86-fbsd" [static]
> 
> >                ~       2.25b-r8 "~amd64 ~hppa ~mips ~ppc ~sparc ~x86
> 
> ~x86-fbsd"
> 
> > [static]
> > 
> >     Homepage:            http://www.acme.com/software/thttpd/
> >     Description:         Small and fast multiplexing webserver.
> 
> Thanks for all the input!
> 
> During my drive home, something hit my brain: why not have the 'master'
> server share the distfiles dir via NFS?
> 
> So, the question now becomes: what's the drawback/benefit of NFS-sharing vs
> HTTP-sharing? The scenario is back-end LAN at the office, thus, a trusted
> network by definition.

HTTP is not really 'sharing'.  It is just 'copying'.  Clients download the 
distfiles from the home server to minimise load on the gentoo mirrors.  
Following the download a client machine will have a local copy of said distfile 
in the client://usr/distfile.

With NFS there is only one copy of the file, on the server, shared by other 
clients in the LAN.

In my case the server is not always on, so NFS would not be appropriate.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] The LIGHTEST web server (just for serving files)?

[YoYo Siska]

On Sat, Nov 12, 2011 at 07:40:08PM +0700, Pandu Poluan wrote:
> On Nov 12, 2011 7:00 PM, "Mick" <michaelkintzios@gmail.com> wrote:
> >
> > I've been using boa just for this purpose for years:
> >
> > * www-servers/boa
> >     Available versions:
> >                ~       0.94.14_rc21 "~x86 ~sparc ~mips ~ppc ~amd64" [doc]
> >     Homepage:            http://www.boa.org/
> >     Description:         A very small and very fast http daemon.
> >
> > It can be easily locked down for internet facing roles.
> >
> > I've also used thttpd (you can throttle its bandwidth if that's important
> in
> > your network), but it's probably more than required for this purpose:
> >
> > * www-servers/thttpd
> >     Available versions:
> >                        2.25b-r7 "amd64 ~hppa ~mips ppc sparc x86
> ~x86-fbsd" [static]
> >                ~       2.25b-r8 "~amd64 ~hppa ~mips ~ppc ~sparc ~x86
> ~x86-fbsd"
> > [static]
> >     Homepage:            http://www.acme.com/software/thttpd/
> >     Description:         Small and fast multiplexing webserver.
> 
> Thanks for all the input!
> 
> During my drive home, something hit my brain: why not have the 'master'
> server share the distfiles dir via NFS?
> 
> So, the question now becomes: what's the drawback/benefit of NFS-sharing vs
> HTTP-sharing? The scenario is back-end LAN at the office, thus, a trusted
> network by definition.

NFS doesn't like when it looses connection to the server. The only
problems I had ever with NFS were because I forgot to unmout it before a
server restart or when I  took a computer (laptop) off to another
network...
Otherwise it works well, esp. when mounted ro on the clients, however
for distfiles it might make sense to allow the clients download and save
tarballs that are not there yet ;), though I never used it with many
computer emerging/downloading same same stuff, so can't say if locking
etc works correctly...

And with NFS the clients won't duplicate the files in their own
distfiles directories ;)

yoyo

Re: [gentoo-user] The LIGHTEST web server (just for serving files)?

[Stéphane Guedon]

[-- Attachment #1: Type: Text/Plain, Size: 2721 bytes --]

On Saturday 12 November 2011 14:01:54 Mick wrote:
> On Saturday 12 Nov 2011 12:40:08 Pandu Poluan wrote:
> > On Nov 12, 2011 7:00 PM, "Mick" <michaelkintzios@gmail.com> wrote:
> > > I've been using boa just for this purpose for years:
> > > 
> > > * www-servers/boa
> > > 
> > >     Available versions:
> > >                ~       0.94.14_rc21 "~x86 ~sparc ~mips ~ppc ~amd64"
> > >                [doc]
> > >     
> > >     Homepage:            http://www.boa.org/
> > >     Description:         A very small and very fast http daemon.
> > > 
> > > It can be easily locked down for internet facing roles.
> > > 
> > > I've also used thttpd (you can throttle its bandwidth if that's
> > > important
> > 
> > in
> > 
> > > your network), but it's probably more than required for this purpose:
> > > 
> > > * www-servers/thttpd
> > > 
> > >     Available versions:
> > >                        2.25b-r7 "amd64 ~hppa ~mips ppc sparc x86
> > 
> > ~x86-fbsd" [static]
> > 
> > >                ~       2.25b-r8 "~amd64 ~hppa ~mips ~ppc ~sparc ~x86
> > 
> > ~x86-fbsd"
> > 
> > > [static]
> > > 
> > >     Homepage:            http://www.acme.com/software/thttpd/
> > >     Description:         Small and fast multiplexing webserver.
> > 
> > Thanks for all the input!
> > 
> > During my drive home, something hit my brain: why not have the 'master'
> > server share the distfiles dir via NFS?
> > 
> > So, the question now becomes: what's the drawback/benefit of NFS-sharing
> > vs HTTP-sharing? The scenario is back-end LAN at the office, thus, a
> > trusted network by definition.
> 
> HTTP is not really 'sharing'.  It is just 'copying'.  Clients download the
> distfiles from the home server to minimise load on the gentoo mirrors.
> Following the download a client machine will have a local copy of said
> distfile in the client://usr/distfile.
> 
> With NFS there is only one copy of the file, on the server, shared by other
> clients in the LAN.
> 
> In my case the server is not always on, so NFS would not be appropriate.

With NFS, there is only one distfiles repo. So, when someone emerge an unknown 
program, the distfiles become available to your whole network (if shared of 
course).

I share /mnt/portage (the portage tree on a reiserfs) and /mnt/distfiles (on a 
ext4 fs). The tree is synced only once also ! It reduce server load (bandwitch 
on gentoo master server) and your global diskspace used (because repo are 
mutualized).

-- 
Stéphane Guedon
page web : http://www.22decembre.eu/
carte de visite : http://www.22decembre.eu/downloads/Stephane-Guedon.vcf
clé publique gpg : http://www.22decembre.eu/downloads/Stephane-Guedon.asc

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-user] The LIGHTEST web server (just for serving files)?

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1251 bytes --]

Am 12.11.2011 13:40, schrieb Pandu Poluan:
> 
> During my drive home, something hit my brain: why not have the 'master'
> server share the distfiles dir via NFS?
> 
> So, the question now becomes: what's the drawback/benefit of NFS-sharing
> vs HTTP-sharing? The scenario is back-end LAN at the office, thus, a
> trusted network by definition.
> 
> Rgds,
> 

How exactly had you planned to share distfiles? You didn't want to
mirror everything from the offical mirrors, did you? I'm not perfectly
sure how portage handles a mirror that occasionally returns 404 errors
but I think I've seen it fall back to the official mirrors in that case.
Anyway, making educated guesses about what should be on your own mirror
is probably a bit ineffective unless you have a very homogeneous
environment.

What I think you /should/ have wanted is a proxy specifically configured
to cache very large files. net-proxy/http-replicator has been made
specifically for Gentoo distfiles.

NFS has the advantage that it doesn't duplicate distfiles locally on all
machines. It is also easier to set up. Disadvantages? I'm unsure how
portage will handle cases when two machines fetch the same file at the
same time.

Regards,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] The LIGHTEST web server (just for serving files)?

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 2086 bytes --]

On Nov 12, 2011 8:05 PM, "Mick" <michaelkintzios@gmail.com> wrote:
>
> On Saturday 12 Nov 2011 12:40:08 Pandu Poluan wrote:
> > On Nov 12, 2011 7:00 PM, "Mick" <michaelkintzios@gmail.com> wrote:
> > > I've been using boa just for this purpose for years:
> > >
> > > * www-servers/boa
> > >
> > >     Available versions:
> > >                ~       0.94.14_rc21 "~x86 ~sparc ~mips ~ppc ~amd64"
[doc]
> > >
> > >     Homepage:            http://www.boa.org/
> > >     Description:         A very small and very fast http daemon.
> > >
> > > It can be easily locked down for internet facing roles.
> > >
> > > I've also used thttpd (you can throttle its bandwidth if that's
important
> >
> > in
> >
> > > your network), but it's probably more than required for this purpose:
> > >
> > > * www-servers/thttpd
> > >
> > >     Available versions:
> > >                        2.25b-r7 "amd64 ~hppa ~mips ppc sparc x86
> >
> > ~x86-fbsd" [static]
> >
> > >                ~       2.25b-r8 "~amd64 ~hppa ~mips ~ppc ~sparc ~x86
> >
> > ~x86-fbsd"
> >
> > > [static]
> > >
> > >     Homepage:            http://www.acme.com/software/thttpd/
> > >     Description:         Small and fast multiplexing webserver.
> >
> > Thanks for all the input!
> >
> > During my drive home, something hit my brain: why not have the 'master'
> > server share the distfiles dir via NFS?
> >
> > So, the question now becomes: what's the drawback/benefit of
NFS-sharing vs
> > HTTP-sharing? The scenario is back-end LAN at the office, thus, a
trusted
> > network by definition.
>
> HTTP is not really 'sharing'.  It is just 'copying'.  Clients download the
> distfiles from the home server to minimise load on the gentoo mirrors.

Yeah, should've put quotes around the sharing part.

> Following the download a client machine will have a local copy of said
distfile
> in the client://usr/distfile.
>
> With NFS there is only one copy of the file, on the server, shared by
other
> clients in the LAN.
>
> In my case the server is not always on, so NFS would not be appropriate.

Hmmm, you have a point there.

Rgds,

[-- Attachment #2: Type: text/html, Size: 3127 bytes --]

Re: [gentoo-user] The LIGHTEST web server (just for serving files)?

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 2264 bytes --]

On Nov 12, 2011 8:16 PM, "YoYo Siska" <yoyo@gl.ksp.sk> wrote:
>
> On Sat, Nov 12, 2011 at 07:40:08PM +0700, Pandu Poluan wrote:
> > On Nov 12, 2011 7:00 PM, "Mick" <michaelkintzios@gmail.com> wrote:
> > >
> > > I've been using boa just for this purpose for years:
> > >
> > > * www-servers/boa
> > >     Available versions:
> > >                ~       0.94.14_rc21 "~x86 ~sparc ~mips ~ppc ~amd64"
[doc]
> > >     Homepage:            http://www.boa.org/
> > >     Description:         A very small and very fast http daemon.
> > >
> > > It can be easily locked down for internet facing roles.
> > >
> > > I've also used thttpd (you can throttle its bandwidth if that's
important
> > in
> > > your network), but it's probably more than required for this purpose:
> > >
> > > * www-servers/thttpd
> > >     Available versions:
> > >                        2.25b-r7 "amd64 ~hppa ~mips ppc sparc x86
> > ~x86-fbsd" [static]
> > >                ~       2.25b-r8 "~amd64 ~hppa ~mips ~ppc ~sparc ~x86
> > ~x86-fbsd"
> > > [static]
> > >     Homepage:            http://www.acme.com/software/thttpd/
> > >     Description:         Small and fast multiplexing webserver.
> >
> > Thanks for all the input!
> >
> > During my drive home, something hit my brain: why not have the 'master'
> > server share the distfiles dir via NFS?
> >
> > So, the question now becomes: what's the drawback/benefit of
NFS-sharing vs
> > HTTP-sharing? The scenario is back-end LAN at the office, thus, a
trusted
> > network by definition.
>
> NFS doesn't like when it looses connection to the server. The only
> problems I had ever with NFS were because I forgot to unmout it before a
> server restart or when I  took a computer (laptop) off to another
> network...
> Otherwise it works well, esp. when mounted ro on the clients, however
> for distfiles it might make sense to allow the clients download and save
> tarballs that are not there yet ;), though I never used it with many
> computer emerging/downloading same same stuff, so can't say if locking
> etc works correctly...
>
> And with NFS the clients won't duplicate the files in their own
> distfiles directories ;)

Yes, that would be beneficial. But if NFS is as finicky as that, what's a
better way to share directories?

Rgds,

[-- Attachment #2: Type: text/html, Size: 3196 bytes --]

Re: [gentoo-user] The LIGHTEST web server (just for serving files)?

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 1729 bytes --]

On Nov 12, 2011 8:20 PM, "Florian Philipp" <lists@binarywings.net> wrote:
>
> Am 12.11.2011 13:40, schrieb Pandu Poluan:
> >
> > During my drive home, something hit my brain: why not have the 'master'
> > server share the distfiles dir via NFS?
> >
> > So, the question now becomes: what's the drawback/benefit of NFS-sharing
> > vs HTTP-sharing? The scenario is back-end LAN at the office, thus, a
> > trusted network by definition.
> >
> > Rgds,
> >
>
> How exactly had you planned to share distfiles? You didn't want to
> mirror everything from the offical mirrors, did you? I'm not perfectly
> sure how portage handles a mirror that occasionally returns 404 errors
> but I think I've seen it fall back to the official mirrors in that case.

Yes, portage (at least, 2.2) automatically use the next mirror in the list.

> Anyway, making educated guesses about what should be on your own mirror
> is probably a bit ineffective unless you have a very homogeneous
> environment.
>
> What I think you /should/ have wanted is a proxy specifically configured
> to cache very large files. net-proxy/http-replicator has been made
> specifically for Gentoo distfiles.
>

I had planned on having a script peruse the log file, looking for which box
got a 404, and 1 hour later try to move the file using scp from that box
into the common local subrepo.

But http-replicator sounds mighty better :-)

> NFS has the advantage that it doesn't duplicate distfiles locally on all
> machines. It is also easier to set up. Disadvantages? I'm unsure how
> portage will handle cases when two machines fetch the same file at the
> same time.

I can always stagger the time my boxes fetch the distfiles. That should
prevent locking problems.

Rgds,

[-- Attachment #2: Type: text/html, Size: 2105 bytes --]

Re: [gentoo-user] The LIGHTEST web server (just for serving files) ?

[Ciprian Dorin Craciun]

On Sat, Nov 12, 2011 at 10:24, Pandu Poluan <pandu@poluan.info> wrote:
> What is the *LIGHTEST* web server package you know for gentoo?
>
> I just want to serve the distfiles, so no CGI / PHP /
> whathaveyouscripting support is needed.
>
> Preferably, with logging so I can see which packages I missed, but not
> necessary.
>
> Rgds,


    We use for such purpose http://linux.bytesex.org/misc/webfs.html .
It features both (reasonable) performance (threaded), security
(chroot, SSL) and simplicity (can be statically linked, can be
controlled solely from arguments).

    For example (or add -d for debugging):
~~~~
webfsd -F -p 80 -r .
~~~~

    Ciprian.

[gentoo-user] Re: The LIGHTEST web server (just for serving files)?

[Holger Hoffstaette]

On Sat, 12 Nov 2011 14:14:33 +0100, Florian Philipp wrote:

> Disadvantages? I'm unsure how portage will handle cases when two
> machines fetch the same file at the same time.

In my experience concurrent/overlapping fetches work just fine, thanks to
portage's lock file. The second potential downloader just waits or does
something else in the meantime.

-h

Re: [gentoo-user] The LIGHTEST web server (just for serving files)?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 964 bytes --]

On Sat, 12 Nov 2011 19:40:08 +0700, Pandu Poluan wrote:

> During my drive home, something hit my brain: why not have the 'master'
> server share the distfiles dir via NFS?

No reason at all, I've been doing it for years without a single
problem.

> So, the question now becomes: what's the drawback/benefit of
> NFS-sharing vs HTTP-sharing? The scenario is back-end LAN at the
> office, thus, a trusted network by definition.

The benefit is that everything is centralised. With an HTTP proxy, you
still have to download from the server to each client. The only drawback
that I experience is that if several packages use the same, large source
file, as so many of the KDE packages do, you are repeatedly pulling the
same file over the network, which is a little slower.


-- 
Neil Bothwick

Keyboard: (n.) a device used by programmers to write software for a mouse
or joystick and by operators for playing games such as 'word processing.'

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] The LIGHTEST web server (just for serving files)?

[William Kenworthy]

Its been solved in the past ... designed for just this purpose:

moriah ~ # esearch http-replicator
[ Results for search key : http-replicator ]
[ Applications found : 1 ]

*  net-proxy/http-replicator
      Latest version available: 3.0-r2
      Latest version installed: 3.0-r2
      Size of downloaded files: 38 kB
      Homepage:    http://sourceforge.net/projects/http-replicator
      Description: Proxy cache for Gentoo packages
      License:     GPL-2


moriah ~ #

I chain them together (two levels, avoiding expensive download costs) so
the remote site doesnt have it in its cache, upstream is the master
cache, which if it doesnt have it will fetch from the repo.  You can
specify what port it runs on, and then use the http_proxy entry in
make.conf to point the clients to it thus avoiding port 80 and any
existing webserver. Handles concurrent fetches transparently. Overall, I
have found it preferable to NFS which has been a bit flaky at times in
the past.

Recommended!


BillK




On Sat, 2011-11-12 at 22:01 +0000, Neil Bothwick wrote:
> On Sat, 12 Nov 2011 19:40:08 +0700, Pandu Poluan wrote:
> 
> > During my drive home, something hit my brain: why not have the 'master'
> > server share the distfiles dir via NFS?
> 
> No reason at all, I've been doing it for years without a single
> problem.
> 
> > So, the question now becomes: what's the drawback/benefit of
> > NFS-sharing vs HTTP-sharing? The scenario is back-end LAN at the
> > office, thus, a trusted network by definition.
> 
> The benefit is that everything is centralised. With an HTTP proxy, you
> still have to download from the server to each client. The only drawback
> that I experience is that if several packages use the same, large source
> file, as so many of the KDE packages do, you are repeatedly pulling the
> same file over the network, which is a little slower.
> 
> 

Re: [gentoo-user] The LIGHTEST web server (just for serving files)?

[J. Roeleveld]

On Sat, November 12, 2011 2:11 pm, YoYo Siska wrote:
> On Sat, Nov 12, 2011 at 07:40:08PM +0700, Pandu Poluan wrote:
>> During my drive home, something hit my brain: why not have the 'master'
>> server share the distfiles dir via NFS?
>>
>> So, the question now becomes: what's the drawback/benefit of NFS-sharing
>> vs
>> HTTP-sharing? The scenario is back-end LAN at the office, thus, a
>> trusted
>> network by definition.
>
> NFS doesn't like when it looses connection to the server. The only
> problems I had ever with NFS were because I forgot to unmout it before a
> server restart or when I  took a computer (laptop) off to another
> network...

NFS-shares can work, but these need to be umounted before network goes.
If server goes, problems can occur there as well.
But that is true with any server/client filesharing. (CIFS/Samba, for
instance)

> Otherwise it works well, esp. when mounted ro on the clients, however
> for distfiles it might make sense to allow the clients download and save
> tarballs that are not there yet ;), though I never used it with many
> computer emerging/downloading same same stuff, so can't say if locking
> etc works correctly...

Locking works correctly, have had 5 machines share the same NFS-shared
distfiles and all downloading the source-files.

> And with NFS the clients won't duplicate the files in their own
> distfiles directories ;)

Big plus, for me :)

--
Joost

Re: [gentoo-user] The LIGHTEST web server (just for serving files)?

[microcai]

[-- Attachment #1: Type: text/plain, Size: 1645 bytes --]


http://code.google.com/p/bashttpd/

run with systemd or xinetd



于 2011年11月14日 18:05, J. Roeleveld 写道:
> On Sat, November 12, 2011 2:11 pm, YoYo Siska wrote:
>> On Sat, Nov 12, 2011 at 07:40:08PM +0700, Pandu Poluan wrote:
>>> During my drive home, something hit my brain: why not have the 'master'
>>> server share the distfiles dir via NFS?
>>>
>>> So, the question now becomes: what's the drawback/benefit of NFS-sharing
>>> vs
>>> HTTP-sharing? The scenario is back-end LAN at the office, thus, a
>>> trusted
>>> network by definition.
>>
>> NFS doesn't like when it looses connection to the server. The only
>> problems I had ever with NFS were because I forgot to unmout it before a
>> server restart or when I  took a computer (laptop) off to another
>> network...
> 
> NFS-shares can work, but these need to be umounted before network goes.
> If server goes, problems can occur there as well.
> But that is true with any server/client filesharing. (CIFS/Samba, for
> instance)
> 
>> Otherwise it works well, esp. when mounted ro on the clients, however
>> for distfiles it might make sense to allow the clients download and save
>> tarballs that are not there yet ;), though I never used it with many
>> computer emerging/downloading same same stuff, so can't say if locking
>> etc works correctly...
> 
> Locking works correctly, have had 5 machines share the same NFS-shared
> distfiles and all downloading the source-files.
> 
>> And with NFS the clients won't duplicate the files in their own
>> distfiles directories ;)
> 
> Big plus, for me :)
> 
> --
> Joost
> 
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-user] The LIGHTEST web server (just for serving files)?

[Michael Mol]

Isn't there a kernelland HTTP server? ISTR seeing the option. I don't
know anything about it, though.

On Mon, Nov 14, 2011 at 5:10 AM, microcai <microcai@fedoraproject.org> wrote:
>
> http://code.google.com/p/bashttpd/
>
> run with systemd or xinetd
>
>
>
> 于 2011年11月14日 18:05, J. Roeleveld 写道:
>> On Sat, November 12, 2011 2:11 pm, YoYo Siska wrote:
>>> On Sat, Nov 12, 2011 at 07:40:08PM +0700, Pandu Poluan wrote:
>>>> During my drive home, something hit my brain: why not have the 'master'
>>>> server share the distfiles dir via NFS?
>>>>
>>>> So, the question now becomes: what's the drawback/benefit of NFS-sharing
>>>> vs
>>>> HTTP-sharing? The scenario is back-end LAN at the office, thus, a
>>>> trusted
>>>> network by definition.
>>>
>>> NFS doesn't like when it looses connection to the server. The only
>>> problems I had ever with NFS were because I forgot to unmout it before a
>>> server restart or when I  took a computer (laptop) off to another
>>> network...
>>
>> NFS-shares can work, but these need to be umounted before network goes.
>> If server goes, problems can occur there as well.
>> But that is true with any server/client filesharing. (CIFS/Samba, for
>> instance)
>>
>>> Otherwise it works well, esp. when mounted ro on the clients, however
>>> for distfiles it might make sense to allow the clients download and save
>>> tarballs that are not there yet ;), though I never used it with many
>>> computer emerging/downloading same same stuff, so can't say if locking
>>> etc works correctly...
>>
>> Locking works correctly, have had 5 machines share the same NFS-shared
>> distfiles and all downloading the source-files.
>>
>>> And with NFS the clients won't duplicate the files in their own
>>> distfiles directories ;)
>>
>> Big plus, for me :)
>>
>> --
>> Joost
>>
>>
>
>
>



-- 
:wq

Re: [gentoo-user] The LIGHTEST web server (just for serving files)?

[v_2e]

  There is a very small web server called "thttpd" which is very
lightweight and lets start serving files very quickly.
  It runs on my home router machine with an old Pentium CPU and several
megabytes of RAM and seems to consume about 500 kb of it.

  Regards,
    Vladimir

On Mon, 14 Nov 2011 09:36:22 -0500
Michael Mol <mikemol@gmail.com> wrote:

> Isn't there a kernelland HTTP server? ISTR seeing the option. I don't
> know anything about it, though.
> 
> On Mon, Nov 14, 2011 at 5:10 AM, microcai
> <microcai@fedoraproject.org> wrote:
> >
> > http://code.google.com/p/bashttpd/
> >
> > run with systemd or xinetd
> >
> >
> >
> > 于 2011年11月14日 18:05, J. Roeleveld 写道:
> >> On Sat, November 12, 2011 2:11 pm, YoYo Siska wrote:
> >>> On Sat, Nov 12, 2011 at 07:40:08PM +0700, Pandu Poluan wrote:
> >>>> During my drive home, something hit my brain: why not have the
> >>>> 'master' server share the distfiles dir via NFS?
> >>>>
> >>>> So, the question now becomes: what's the drawback/benefit of
> >>>> NFS-sharing vs
> >>>> HTTP-sharing? The scenario is back-end LAN at the office, thus, a
> >>>> trusted
> >>>> network by definition.
> >>>
> >>> NFS doesn't like when it looses connection to the server. The only
> >>> problems I had ever with NFS were because I forgot to unmout it
> >>> before a server restart or when I  took a computer (laptop) off
> >>> to another network...
> >>
> >> NFS-shares can work, but these need to be umounted before network
> >> goes. If server goes, problems can occur there as well.
> >> But that is true with any server/client filesharing. (CIFS/Samba,
> >> for instance)
> >>
> >>> Otherwise it works well, esp. when mounted ro on the clients,
> >>> however for distfiles it might make sense to allow the clients
> >>> download and save tarballs that are not there yet ;), though I
> >>> never used it with many computer emerging/downloading same same
> >>> stuff, so can't say if locking etc works correctly...
> >>
> >> Locking works correctly, have had 5 machines share the same
> >> NFS-shared distfiles and all downloading the source-files.
> >>
> >>> And with NFS the clients won't duplicate the files in their own
> >>> distfiles directories ;)
> >>
> >> Big plus, for me :)
> >>
> >> --
> >> Joost
> >>
> >>
> >
> >
> >
> 
> 
> 
> -- 
> :wq
> 
> 


----- 
 <v_2e@ukr.net>

[gentoo-user] Re: The LIGHTEST web server (just for serving files)?

[Steven J Long]

Michael Mol wrote:

> Isn't there a kernelland HTTP server? ISTR seeing the option. I don't
> know anything about it, though.
> 
Yeah there was; as I recall it got removed a while back.

Google got me:
http://en.wikipedia.org/wiki/TUX_web_server
and khttpd at: http://www.fenrus.demon.nl/

..both of which appear dead. I couldn't find any mention of http in my
kernel config either.

We use lighttpd for our dev stuff; I guess it's that, nginx or thttpd,
last of which doesn't do fastCGI, so might be the best for this
purpose.

http://en.wikipedia.org/wiki/Comparison_of_web_server_software
..might prove helpful.
-- 
#friendly-coders -- We're friendly, but we're not /that/ friendly ;-)