[gentoo-user] Personal Gentoo mirror

[gentoo-user] Personal Gentoo mirror

[Devraj Mukherjee]

Hi everyone,

I am interested in creating a Gentoo mirror for our organisation. 
However the issue I have is getting an initial snapshot of the distfiles.

RSYNCing the files from a current server is going to take us awefully 
long and our ADSL connection is not really going to stand it :)

Can we get this sent on DVD or something from a current mirror and RSYNC 
the differences?

Any suggestions are welcome.

Kind regards,
Devraj
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Personal Gentoo mirror

[John Jolet]

I was asked a similar question today at work.  we have some new  
gentoo servers that do not have outbound access to the net.  We'd  
like to have a local mirror.  Is there a howto?

On Oct 5, 2005, at 9:21 PM, Devraj Mukherjee wrote:

> Hi everyone,
>
> I am interested in creating a Gentoo mirror for our organisation.  
> However the issue I have is getting an initial snapshot of the  
> distfiles.
>
> RSYNCing the files from a current server is going to take us  
> awefully long and our ADSL connection is not really going to stand  
> it :)
>
> Can we get this sent on DVD or something from a current mirror and  
> RSYNC the differences?
>
> Any suggestions are welcome.
>
> Kind regards,
> Devraj
> -- 
> gentoo-user@gentoo.org mailing list
>
>

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Personal Gentoo mirror

[Nick Rout]

Why would you want ALL of distfiles? A lot is old & redundant. 

Also why would you rsync it? Thats a lot of cpu grinding.



I suggest setting up one of these:

http://gentoo-wiki.com/HOWTO_Download_Cache_for_LAN-Http-Replicator

If someone really wants every source file ever made, this will get it
eventually.


On Thu, 06 Oct 2005 12:21:16 +1000
Devraj Mukherjee wrote:

> Hi everyone,
> 
> I am interested in creating a Gentoo mirror for our organisation. 
> However the issue I have is getting an initial snapshot of the distfiles.
> 
> RSYNCing the files from a current server is going to take us awefully 
> long and our ADSL connection is not really going to stand it :)
> 
> Can we get this sent on DVD or something from a current mirror and RSYNC 
> the differences?
> 
> Any suggestions are welcome.
> 
> Kind regards,
> Devraj
> -- 
> gentoo-user@gentoo.org mailing list

-- 
Nick Rout <nick@rout.co.nz>

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Personal Gentoo mirror

[Rumen Yotov]

[-- Attachment #1: Type: text/plain, Size: 1001 bytes --]

Hi,
On Wed, 5 Oct 2005 21:28:04 -0500
John Jolet <john@jolet.net> wrote:

> I was asked a similar question today at work.  we have some new  
> gentoo servers that do not have outbound access to the net.  We'd  
> like to have a local mirror.  Is there a howto?
> 
Check on www.gentoo.org, there is/was a doc about Howto mirror gentoo.
HTH.Rumen
> On Oct 5, 2005, at 9:21 PM, Devraj Mukherjee wrote:
> 
> > Hi everyone,
> >
> > I am interested in creating a Gentoo mirror for our organisation.  
> > However the issue I have is getting an initial snapshot of the  
> > distfiles.
> >
> > RSYNCing the files from a current server is going to take us  
> > awefully long and our ADSL connection is not really going to stand  
> > it :)
> >
> > Can we get this sent on DVD or something from a current mirror and  
> > RSYNC the differences?
> >
> > Any suggestions are welcome.
> >
> > Kind regards,
> > Devraj
> > -- 
> > gentoo-user@gentoo.org mailing list
> >
> >
> 

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] Personal Gentoo mirror

[kashani]

John Jolet wrote:
> I was asked a similar question today at work.  we have some new  gentoo 
> servers that do not have outbound access to the net.  We'd  like to have 
> a local mirror.  Is there a howto?

I run a proxy server for the servers without access. That might be 
easier than having a local mirror. Once the proxy is set add this to the 
other machines' /etc/make.conf

http_proxy="proxyhost.yourdomain.com:3128"

kashani
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Personal Gentoo mirror

[Dave Nebinger]

On Thursday 06 October 2005 12:38 am, kashani wrote:
> John Jolet wrote:
> > I was asked a similar question today at work.  we have some new  gentoo
> > servers that do not have outbound access to the net.  We'd  like to have
> > a local mirror.  Is there a howto?
>
> I run a proxy server for the servers without access. That might be
> easier than having a local mirror. Once the proxy is set add this to the
> other machines' /etc/make.conf
>
> http_proxy="proxyhost.yourdomain.com:3128"
>
> kashani

Do a google search for http-replicator.  It's what I use and it works very 
well.

Each gentoo box has it's /etc/make.conf pointed at http-replicator running on 
a gentoo box on the gateway.  They each operate independently requesting 
files, but they are only retrieved from external sites once.  First hit 
downloads at regular speeds, other hits (from remote boxes) are as fast as 
the internal network speeds.

Comes with a script to add to /etc/cron.monthly to clean up old files.

I don't remember there being an ebuild in portage, but I've got one in my 
overlay I'd be willing to share.

Dave
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Personal Gentoo mirror

[William Kenworthy]

I'll also jump on board this - http-replicator works great.  You build
it from your existing distfile cache.  You do not need the large mirror
files as many, or rather most are unlikely ever to be accessed.  Build
it with what you already have, and just top it up as you go - if the
file isnt already in your cache, its added when you download it.

BillK


On Thu, 2005-10-06 at 16:09 +1300, Nick Rout wrote:
> Why would you want ALL of distfiles? A lot is old & redundant. 
> 
> Also why would you rsync it? Thats a lot of cpu grinding.
> 
> 
> 
> I suggest setting up one of these:
> 
> http://gentoo-wiki.com/HOWTO_Download_Cache_for_LAN-Http-Replicator
> 
> If someone really wants every source file ever made, this will get it
> eventually.

-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: Personal Gentoo mirror

[James]

William Kenworthy <billk <at> iinet.net.au> writes:


> I'll also jump on board this - http-replicator works great.  You build
> it from your existing distfile cache.  You do not need the large mirror
> files as many, or rather most are unlikely ever to be accessed.  Build
> it with what you already have, and just top it up as you go - if the
> file isnt already in your cache, its added when you download it.

> > http://gentoo-wiki.com/HOWTO_Download_Cache_for_LAN-Http-Replicator

What mechanisms can you employ to either limit the size of the stored files
or prune out the oldest files or the files that are completely obsolete
as new sources files exist in the distfile cache?

Is the distfile cache the same this as the CCACHE cache in the 
make.conf file? If so then, CCACHE_SIZE="3G" could be used to 
limit the cache size to 3 gigs, but, what mechanisms, short of 
a script can be used to prune out old duplicate sources?

I guess the download cache is the complemetary companion to setting
 up your own rsync mirror, as both allow for multiple systems to be 
updated and synced off of an internal system and only one system 
syncs or downloads new ebuilds/distfiles)?

I set this up both before, but I recall having to use aspects of 
several documents. In the end, I abandoned syncing off of a single 
system, as syncing update were sporadically successful, and the 
scripts and cron entries were not amicable.  I meant to get back to 
this problem to sort it all out.

It'd be much simpler if a single document explained all the methods 
and options and scripts, in a clear, consise manner to use a single 
system for syncing,cache-ing, updateding, and pruning of disk space.

tmpwatch, distcleaner, distclean, yacleaner, repcacheman, dailysync.sh
just to mention a few script possibilities .

and the 4 basic URLs I looked at to try and sort things out:
http://www.gentoo.org/doc/en/rsync.xml?style=printable
http://gentoo-wiki.com/HOWTO_Local_Rsync_Mirror.
http://gentoo-wiki.com/Http_proxy
http://gentoo-wiki.com/HOWTO_Download_Cache_for_LAN-Http-Replicator

USE this URL to automate the updating of Gentoo workstations:
http://www.panhorst.com/glcu/
was yet another approach.


Any comments or suggestions as to the development of a single,
concise document, would be appreciated. 

James



-- 
gentoo-user@gentoo.org mailing list