[gentoo-user] GENTOO_MIRRORS

[gentoo-user] GENTOO_MIRRORS

[James]

Hello,

Noodling around I found a gentoo server at:
http://open-systems.ufl.edu/mirrors/gentoo/

I'm not sure if I can use this in my make.conf file?

If so, what would the setting look like?

Hopes are that is closer than my current mirrors.

Is there an a method/software to test which mirrors
are the fastest?

I have  this setting on some gentoo systems:
SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage"

Should it be added to all my gentoo systems and how does 
the 'SYNC' setting affect/interact with specific 
GENTOO_MIRRORS settings?

'mirrorselect' detail documentation seems to elude me,
and I'm not even sure this tools is useful to evaluate
the closest/fastest mirror or rsync host. Hell, I'm
not even sure what the main/functional difference is
between a mirror and a rsync host?

confused on optimizing gentoo downloads....

James



-- 
gentoo-user@gentoo.org mailing list

RE: [gentoo-user] GENTOO_MIRRORS

[Dave Nebinger]

> Noodling around I found a gentoo server at:
> http://open-systems.ufl.edu/mirrors/gentoo/
> 
> I'm not sure if I can use this in my make.conf file?

If it's complete you should be able to use it.  If it's not a publicized
mirror then there could be many reasons why.  For example, they might not
sync the mirror as frequently as other sites or have a limitation on the
number of concurrent users.  Could also be meant only as an internal sync
mirror (I use an internal mirror to sync my internal gentoo systems
against).

> If so, what would the setting look like?

Add the URL for everything up to the "distfiles" directory.  If distfiles is
in the /mirrors/gentoo directory, then you can add the URL as you've listed
above to GENTOO_MIRRORS line.

> Hopes are that is closer than my current mirrors.
> 
> Is there an a method/software to test which mirrors
> are the fastest?

The handbook recommends using mirrorselect.  Use that to identify your
closest mirrors.  As a word of advice I would try to time the mirrorselect
run with when you plan on actually syncing the system; a mirror may appear
to be slow during the day when you run mirrorselect but actually be faster
when syncing at 3am.

 
> I have  this setting on some gentoo systems:
> SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage"
> 
> Should it be added to all my gentoo systems and how does
> the 'SYNC' setting affect/interact with specific
> GENTOO_MIRRORS settings?

SYNC is used to define the rsync server to get the portage mirror from.  You
probably do not want to use any value in there (portage will pick the
appropriate one at runtime).  The sync mirror and the gentoo mirrors serve
different things, namely the portage data versus the distfiles.

> 'mirrorselect' detail documentation seems to elude me,
> and I'm not even sure this tools is useful to evaluate
> the closest/fastest mirror or rsync host. Hell, I'm
> not even sure what the main/functional difference is
> between a mirror and a rsync host?

Rsync is used to remotely sync a filesystem.  The portage tree, for example,
is not a single file to be downloaded but a remote filesystem that needs to
be copied to your system.  Rsync is optimized to only pull remote files that
are different from those in your local /usr/portage directory.

Mirrorselect is used to find the closest/fastest distfiles mirror.  The
distfiles are the actual source package files that are downloaded to build
the packages on your system.

> confused on optimizing gentoo downloads....

The first thing you can do to significantly optimize internal gentoo
downloads is to set up your own mirror system.  I use that for my systems
and it's pretty easy to set up.  The gentoo wiki has all of the details.
Basically I've got a server system that:

a) at 1am runs emerge --sync to sync portage against the public mirrors.
b) runs rsyncd to make the server's portage tree available to the internal
gentoo systems.
c) runs http-replicator as a download proxy.  All internal gentoo systems
use the server as the proxy - actual internet downloads occur only once,
after that the cached copy is returned.  There is only a delay for the
initial download, the rest are almost instantaneous.

I'm sure you'll get recommendations for using NFS as the portage tree,
synced by a primary system and exported to the other internal systems.
While this will work, I personally don't like it because a) NFS is a huge
network hog and b) if the serving system is not available the remote gentoo
systems don't have the portage tree available.  However, if disk space is an
issue for you this might be the way to go.



-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: GENTOO_MIRRORS

[James]

Dave Nebinger <dnebinger <at> joat.com> writes:


> Add the URL for everything up to the "distfiles" directory.  If distfiles is
> in the /mirrors/gentoo directory, then you can add the URL as you've listed
> above to GENTOO_MIRRORS line.


Thanks for all of the information.


James



-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: GENTOO_MIRRORS

[James]

Dave Nebinger <dnebinger <at> joat.com> writes:
 
> The first thing you can do to significantly optimize internal gentoo
> downloads is to set up your own mirror system.  I use that for my systems
> and it's pretty easy to set up.  The gentoo wiki has all of the details.
> Basically I've got a server system that:

> a) at 1am runs emerge --sync to sync portage against the public mirrors.
OK I edited /etc/crontab to this:

30 1 * * * root emerge sync

b) runs rsyncd to make the server's portage tree available to the internal
gentoo systems.

OK following these pages:
http://gentoo-wiki.com/HOWTO_Local_Rsync_Mirror.
ON the internal server I 'emerge rsync'

Then edited the /etc/rsync.conf file as specified
and ran rc-update.

http://gentoo-wiki.com/Http_proxy

On each system I added these entries to the /etc/make.conf file:
GENTOO_MIRRORS="192.168.2.9"
SYNC="rsync://192.168.2.9"
So that they both sync and download files from the internal server




> c) runs http-replicator as a download proxy.  All internal gentoo systems
> use the server as the proxy - actual internet downloads occur only once,
> after that the cached copy is returned.  There is only a delay for the
> initial download, the rest are almost instantaneous.

Well this is a problem. 

Following 
http://gentoo-wiki.com/HOWTO_Download_Cache_for_LAN-Http-Replicator

I get:
emerge http-replicator
Calculating dependencies
!!! All ebuilds that could satisfy "http-replicator" have been masked.


-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: GENTOO_MIRRORS

[James]

Dave Nebinger <dnebinger <at> joat.com> writes:
 
> The first thing you can do to significantly optimize internal gentoo
> downloads is to set up your own mirror system.  I use that for my systems
> and it's pretty easy to set up.  The gentoo wiki has all of the details.
> Basically I've got a server system that:

> a) at 1am runs emerge --sync to sync portage against the public mirrors.
OK I edited /etc/crontab to this:

30 1 * * * root emerge sync

b) runs rsyncd to make the server's portage tree available to the internal
gentoo systems.

OK following these pages:
http://gentoo-wiki.com/HOWTO_Local_Rsync_Mirror.
ON the internal server I 'emerge rsync'

Then edited the /etc/rsync.conf file as specified
and ran rc-update.

http://gentoo-wiki.com/Http_proxy

On each system I added these entries to the /etc/make.conf file:
GENTOO_MIRRORS="192.168.2.9"
SYNC="rsync://192.168.2.9"
So that they both sync and download files from the internal server




> c) runs http-replicator as a download proxy.  All internal gentoo systems
> use the server as the proxy - actual internet downloads occur only once,
> after that the cached copy is returned.  There is only a delay for the
> initial download, the rest are almost instantaneous.

Well this is a problem. 

Following 
http://gentoo-wiki.com/HOWTO_Download_Cache_for_LAN-Http-Replicator

I get:
emerge http-replicator
Calculating dependencies
!!! All ebuilds that could satisfy "http-replicator" have been masked.


Running an 'emerge sync' from one of the test clients gives this error:

crystal james # emerge sync
>>> starting rsync with rsync://192.168.2.9...
>>> checking server timestamp ...
Welcome to My Local Rsync Mirror!!
@ERROR: Unknown module 'metadata'
rsync: connection unexpectedly closed (83 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(189)
>>> retry ...

Not sure how to proceed from here


James

-- 
gentoo-user@gentoo.org mailing list