[gentoo-user] An alternative to http-replicator

[gentoo-user] An alternative to http-replicator

[Walter Dnes]

  We want to be "kinder and gentler" to the Gentoo mirrors.  I have 2
machines; my main machine and a 1999 450 mhz PIII as "hot backup".  I
try to keep the backup up to date with apps duplicating the main
machine, so that if the main machine dies, I can restore some data from
backups and be running in a couple of hours.

  The procedure for "emerge --rsync" using the main machine as server is
simple.  I'll now present my solution for /usr/portage/distfiles.  This
obviously works best for machines with a similar set of apps.  Run
emerge update on the server machine first.  It'll get the tarballs it
needs, and they'll be sitting in /usr/portage/distfiles, waiting to be
served out to other machines in your LAN that need the same tarballs.

  I use the "boa" webserver.  It's lighweight, simple to configure, and
you can run multiple instances simultaneously.  That is the best way to
separate privileges for different clients.

  *IMPORTANT*  Gentoo's emerge command always looks in the "distfiles"
folder below whatever URL it's given for a mirror.  E.g. if I specify
"http://www.bad.example.com" as the mirror to use, emerge will look in
"http://www.bad.example.com/distfiles".  The "Alias" declaration in
boa.conf handles this.  The secondary machine's /etc/make.conf has the
line...

GENTOO_MIRRORS="http://192.168.123.252:1024 ftp://ftp.ndlug.nd.edu/pub/gentoo/ http://mirror.datapipe.net/gentoo"

so it'll try my main machine (192.168.123.252 on port 1024) first.  I
invoke the portage server dedicated boa with the command...
boa -c /root/.boa/portage/
which implies that the config file to use is /root/.boa/portage/boa.conf

My /root/.boa/portage/boa.conf currently looks like so...

Port 1024
Listen 192.168.123.252
User nobody
Group nogroup
ErrorLog /var/log/boa/portage/error_log
AccessLog /var/log/boa/portage/access_log
DocumentRoot /usr/portage/distfiles/
UseLocaltime
DirectoryMaker /usr/lib/boa/boa_indexer
KeepAliveMax 1000
KeepAliveTimeout 10
MimeTypes /etc/boa/mime.types
DefaultType text/plain
Alias /distfiles /usr/portage/distfiles

  Note that in addition to using a dedicated I/O port, the "portage
instance" of boa also has its own log files.

-- 
Walter Dnes <waltdnes@waltdnes.org> In linux /sbin/init is Job #1
My musings on technology and security at http://tech_sec.blog.ca
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] An alternative to http-replicator

[Anielkis Herrera Gonzalez]

try torpage http://www.kroon.co.za/torpage.php
-- 
________________________________________________________
           Ing. Anielkis Herrera González
             Desarrollador de Nova LNX
                 Linux User #377809

        Universidad de las Ciencias Informáticas
                        Cuba
________________________________________________________

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] An alternative to http-replicator

[Michael Weyershäuser]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I don't really see the need for such elaborate setups. I just export
/usr/portage/distfiles via NFS on the server and mount it on all other
boxes...
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEilHV6q4f+IV6B/wRAplLAJ42ZqDtFziHj8FUMX7kNtvDebGZIgCfVEwl
LWe+m+T6khyrTc42tJFIvE4=
=WtNo
-----END PGP SIGNATURE-----
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] An alternative to http-replicator

[Teresa and Dale]

Michael Weyershäuser wrote:

> I don't really see the need for such elaborate setups. I just export
> /usr/portage/distfiles via NFS on the server and mount it on all other
> boxes...


That is a good way to do it.  I was going to before but I had http thing
set up already.  If you think about it and have all the boxes on the
same network, it would work like a charm.

Dale
:-)  :-)
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] An alternative to http-replicator

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 550 bytes --]

On Sat, 10 Jun 2006 07:00:05 +0200, Michael Weyershäuser wrote:

> I don't really see the need for such elaborate setups. I just export
> /usr/portage/distfiles via NFS on the server and mount it on all other
> boxes...

I've been doing it that way for years. there were some problems with
simultaneous fetches at first, but since Portage implemented file locks
for NFS, it has been totally trans[parent and reliable, with no need for
any extra software.


-- 
Neil Bothwick

Windows Multitasking - screwing up several things at once

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] An alternative to http-replicator

[William Kenworthy]

Used to do that - heaps of problems when using multiple machines and
multiple users.  http-replicator insulates you from that.  It also
allows you to remove nfs if you dont need it for anything else - also
simplifies and makes the system (all machines) more reliable.  nfs is ok
in its place, but it is certainly not secure or trouble free!

BillK

On Sat, 2006-06-10 at 07:00 +0200, Michael Weyershäuser wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> I don't really see the need for such elaborate setups. I just export
> /usr/portage/distfiles via NFS on the server and mount it on all other
> boxes...
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.2.2 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
> 
> iD8DBQFEilHV6q4f+IV6B/wRAplLAJ42ZqDtFziHj8FUMX7kNtvDebGZIgCfVEwl
> LWe+m+T6khyrTc42tJFIvE4=
> =WtNo
> -----END PGP SIGNATURE-----
-- 
William Kenworthy <billk@iinet.net.au>
Home!
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] An alternative to http-replicator

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 576 bytes --]

On Sun, 11 Jun 2006 10:37:08 +0800, William Kenworthy wrote:

> Used to do that - heaps of problems when using multiple machines and
> multiple users.

There used to be problems when trying to download the same file from two
computers at once, but NFS-aware file locking in portage fixed that quite
a long time ago.

> > I don't really see the need for such elaborate setups. I just export
> > /usr/portage/distfiles via NFS on the server and mount it on all other
> > boxes...


-- 
Neil Bothwick

In plumbing, a straight flush is better than a full house.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] An alternative to http-replicator

[Jeremy Olexa]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Walter Dnes wrote:
> <snip>

Much simplier:
http://gentoo-wiki.com/HOWTO_Using_a_shared_portage_via_NFS

- --
Jeremy Olexa
(olexa@cs.umn.edu)
Office: EE/CS 1-201
CS/IT Systems Staff
University of Minnesota

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEimIBFN7pD9kMi/URAsYvAJwL3Fl5YIsQyhMDcy0UZS/lqOJ7GgCfTc3l
N3fCAdmjCiyllzHj+dB9Bls=
=ACbf
-----END PGP SIGNATURE-----
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] An alternative to http-replicator

[Michael Weyershäuser]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jeremy Olexa wrote:
> 
> Much simplier:
> http://gentoo-wiki.com/HOWTO_Using_a_shared_portage_via_NFS

I have used that setup for some time and must say that sharing
portage over NFS is slow like hell...
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)

iD8DBQFEioqu6q4f+IV6B/wRAljdAJ9VsOH58DCuPOUDSTBRhgMvLwqHMwCeMySr
og8sOr+1r/yf/8Vq7iI3FCo=
=qi11
-----END PGP SIGNATURE-----
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] An alternative to http-replicator

[Mick]

On 10/06/06, Michael Weyershäuser <thedude0001@gmx.de> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Jeremy Olexa wrote:
> >
> > Much simplier:
> > http://gentoo-wiki.com/HOWTO_Using_a_shared_portage_via_NFS
>
> I have used that setup for some time and must say that sharing
> portage over NFS is slow like hell...

What's the pros/cons of mounting portage over NFS Vs http-replicator?
-- 
Regards,
Mick

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] An alternative to http-replicator

[Bob Sanders]

On Sat, 10 Jun 2006 10:25:25 +0000
Mick <michaelkintzios@gmail.com> wrote:

> 
> What's the pros/cons of mounting portage over NFS Vs http-replicator?


If you only have one architecture and one system type or one system that
can be a superset of the others, nfs will serve you fine.

If you have multiple architectures, the packages release at different
times and sometimes different revs.  For this http-replicator is a 
better choice.

For example - I run x86, amd64, and power pc.  Thus, need a broader
spectrum of packages.

Or if you run desktops and servers (different sets of software) and don't
have a common set of USE flags - use say, lighttpd, php, and mysql on the
server but not on the desktop.  Or more likely, use postfix, sasl, tinydns,
and procmail on the server, but not the desktop (assumes the desktop uses
LDAP or POP).  Then http-replicator would be a better choice.

Bob
-  
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] An alternative to http-replicator

[Teresa and Dale]

Bob Sanders wrote:

>On Sat, 10 Jun 2006 10:25:25 +0000
>Mick <michaelkintzios@gmail.com> wrote:
>
>  
>
>>What's the pros/cons of mounting portage over NFS Vs http-replicator?
>>    
>>
>
>
>If you only have one architecture and one system type or one system that
>can be a superset of the others, nfs will serve you fine.
>
>If you have multiple architectures, the packages release at different
>times and sometimes different revs.  For this http-replicator is a 
>better choice.
>
>For example - I run x86, amd64, and power pc.  Thus, need a broader
>spectrum of packages.
>
>Or if you run desktops and servers (different sets of software) and don't
>have a common set of USE flags - use say, lighttpd, php, and mysql on the
>server but not on the desktop.  Or more likely, use postfix, sasl, tinydns,
>and procmail on the server, but not the desktop (assumes the desktop uses
>LDAP or POP).  Then http-replicator would be a better choice.
>
>Bob
>-  
>  
>

Could a person use NFS for the distfles then use rsync for the snapshot
or sync part??  I have used the rsync before and it works fine,
especially when I was on a 26K dial-up which sucked.

Dale
:-)  :-)
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] An alternative to http-replicator

[Jeremy Olexa]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Michael Weyershäuser wrote:
> Jeremy Olexa wrote:
>>> Much simplier:
>>> http://gentoo-wiki.com/HOWTO_Using_a_shared_portage_via_NFS
> 
> I have used that setup for some time and must say that sharing
> portage over NFS is slow like hell...

I really only do it for the distfiles. There isn't really a reason to
share the whole tree for me. But "downloading" the distfiles from a NFS
share is much faster than the internet. ;-)

- --
Jeremy Olexa
(olexa@cs.umn.edu)
Office: EE/CS 1-201
CS/IT Systems Staff
University of Minnesota

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEit5HFN7pD9kMi/URAitQAJ0WqvXnQlOE0AojjeLerRa+OGqMDQCfXpds
2tv6JrpZjhodVy0vGGB74UY=
=Nfkl
-----END PGP SIGNATURE-----
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] An alternative to http-replicator

[Andrew Frink]

[-- Attachment #1: Type: text/plain, Size: 800 bytes --]

On 6/10/06, Michael Weyershäuser <thedude0001@gmx.de> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Jeremy Olexa wrote:
> >
> > Much simplier:
> > http://gentoo-wiki.com/HOWTO_Using_a_shared_portage_via_NFS
>
> I have used that setup for some time and must say that sharing
> portage over NFS is slow like hell...
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.3 (GNU/Linux)
>
> iD8DBQFEioqu6q4f+IV6B/wRAljdAJ9VsOH58DCuPOUDSTBRhgMvLwqHMwCeMySr
> og8sOr+1r/yf/8Vq7iI3FCo=
> =qi11
> -----END PGP SIGNATURE-----
> --
> gentoo-user@gentoo.org mailing list






Michael Weyershäuser,
I've only seen problems with that setup with very very large tarbars when it
tries to do the md5sum of them, other than that its fine fore me, using NFS3
or NFS4 i forget which right now

Cynyr

[-- Attachment #2: Type: text/html, Size: 1410 bytes --]

Re: [gentoo-user] An alternative to http-replicator

[Uwe Thiem]

On 10 June 2006 10:02, Michael Weyershäuser wrote:
> Jeremy Olexa wrote:
> > Much simplier:
> > http://gentoo-wiki.com/HOWTO_Using_a_shared_portage_via_NFS
>
> I have used that setup for some time and must say that sharing
> portage over NFS is slow like hell...

Not here. 100baseT network is quite performant. If you've got a 1000baseT, it 
can actually outperform a local harddrive.

Uwe

-- 
Mark Twain: I rather decline two drinks than a German adjective.

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] An alternative to http-replicator

[Walter Dnes]

On Sat, Jun 10, 2006 at 01:09:05AM -0500, Jeremy Olexa wrote

> Much simplier:
> http://gentoo-wiki.com/HOWTO_Using_a_shared_portage_via_NFS

  My approach requires 2 emerges (boa and rsyncd) and their config files
on the server plus inserting the server as the preferred mirror in 2
lines in /etc/make.conf on the client(s).

  NFS requires building NFS server support on the server (reboot
required) and NFS client support on the client(s) plus setting up lines
in the associated config files.  It also requires enabling and running
portmapper on port 111 (the linux version of Windows' port 139) on the
server and punching a few holes in iptables.  For details see...
http://gentoo-wiki.com/HOWTO_Share_Directories_via_NFS

  In terms of programs to configure and run, it's a wash.  But I'll
pick http security (especially boa) over portmap/nfs security any day.

  rsync and nfs both work for "emerge --sync" because we know that those
files are supposed to be *ABSOLUTELY* identical for all machines at any
given time.  Unless your server has *EVERY PACKAGE THAT EVERY CLIENT
NEEDS*, you run into a problem when emerging packages.  If a required
tarball isn't present, emerge downloads the missing file (tarball, etc)
to the distfiles directory.  But, but, but...  the clients' distfiles
directories are nfs mounts on the server.  That implies that *THE
CLIENTS WILL NEED WRITE PERMISSION ON THE SERVER*!!!!

  My approach is to make *LOCAL* copies on the client from mirrors.  The
local server just happens to be the first mirror on the list.  If it
doesn't have a required file, no problem, check the next mirror in the
list.  The clients never have to write on the local server.

-- 
Walter Dnes <waltdnes@waltdnes.org> In linux /sbin/init is Job #1
My musings on technology and security at http://tech_sec.blog.ca
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] An alternative to http-replicator

[Bob Sanders]

On Sat, 10 Jun 2006 22:43:03 -0400
"Walter Dnes" <waltdnes@waltdnes.org> wrote:


>   My approach requires 2 emerges (boa and rsyncd) and their config files
> on the server plus inserting the server as the preferred mirror in 2
> lines in /etc/make.conf on the client(s).
> 

That's close to what I do at work.  Only I run a full Gentoo mirror because
I need multiple architectures - x86, amd64, ia64, mips.  Also, there are
multiple users, and it's necessary to insure the LiveCDs and snapshots get
transfered automatically.

At home, http-replicator work fine for the small set of systems.  No NFS
required for either setup.

Bob
-  
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] An alternative to http-replicator

[Mick]

On 11/06/06, Walter Dnes <waltdnes@waltdnes.org> wrote:

[snip . . . ]
>   My approach is to make *LOCAL* copies on the client from mirrors.  The
> local server just happens to be the first mirror on the list.  If it
> doesn't have a required file, no problem, check the next mirror in the
> list.  The clients never have to write on the local server.

Thanks Walter for a clear explanation of your system.  How does boa
compares with http-replicator in terms of functionality/security?

PS.  Other than not running portmapper is there a way of securing it?
-- 
Regards,
Mick
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] An alternative to http-replicator

[Walter Dnes]

On Sun, Jun 11, 2006 at 08:18:11AM +0000, Mick wrote

> How does boa compares with http-replicator in terms of
> functionality/security?

  boa is a lean+mean+fast webserver.  See http://www.boa.org for details.

> PS.  Other than not running portmapper is there a way of securing it?

  It has to be accessed by all machines that want to nfs-mount on the
server, so they can ask it what port nfs is running on.  Restrict access
to only the clients that need to nfs-mount on the server.  The usual
method is iptables.  For "defense in depth", you can also use inet.d and
hosts.allow/hosts.deny.

  I'm not an nfs expert.  Is it possible to force nfs to a specific port
on both the client and server such that they'll talk without handshaking
via portmapper first?

-- 
Walter Dnes <waltdnes@waltdnes.org> In linux /sbin/init is Job #1
My musings on technology and security at http://tech_sec.blog.ca
-- 
gentoo-user@gentoo.org mailing list