[gentoo-user] Mirroring part of tinderbox.dev.gentoo.org

[gentoo-user] Mirroring part of tinderbox.dev.gentoo.org

[Noah McNallie]

[-- Attachment #1: Type: text/plain, Size: 764 bytes --]

I'm trying to mirror 'http://tinderbox.dev.gentoo.org/default-linux/sparc'
because I run a sparcv9 machine that is quite old and not good for
compiling. This is the only gentoo sparc binary repo that I know of and I'd
like to have a local copy that I would make available on a webserver.

I'd like to know the best way to mirror it. I have been trying with the
following command:

'wget -mkN -np http://tinderbox.dev.gentoo.org/default-linux/sparc/'

-m to mirror -N to check timestamps and only update and -np not to access
the parent directory.

This will get the files but the second time around it does not update to
only changed files but tries to grab everything again.

Could someone point me in the direction for mirroring this lovely repo?

Noah McNallie

[-- Attachment #2: Type: text/html, Size: 1039 bytes --]

Re: [gentoo-user] Mirroring part of tinderbox.dev.gentoo.org

[Alan McKinnon]

On 23/10/2013 20:40, Noah McNallie wrote:
> I'm trying to mirror
> 'http://tinderbox.dev.gentoo.org/default-linux/sparc' because I run a
> sparcv9 machine that is quite old and not good for compiling. This is
> the only gentoo sparc binary repo that I know of and I'd like to have a
> local copy that I would make available on a webserver.
> 
> I'd like to know the best way to mirror it. I have been trying with the
> following command:
> 
> 'wget -mkN -np http://tinderbox.dev.gentoo.org/default-linux/sparc/'
> 
> -m to mirror -N to check timestamps and only update and -np not to
> access the parent directory.
> 
> This will get the files but the second time around it does not update to
> only changed files but tries to grab everything again.
> 
> Could someone point me in the direction for mirroring this lovely repo?
> 
> Noah McNallie


You are downloading tbz files, not html files, so every file downloaded
has this in the output to the console:


Reusing existing connection to tinderbox.dev.gentoo.org:80.
HTTP request sent, awaiting response... 200 OK
Length: 104635 (102K) [application/octet-stream]
Last-modified header missing -- time-stamps turned off.
                                ^^^^^^^^^^^^^^^^^^^^^^

This leaves wget only one option - it cannot confirm that the file is
unchanged, so it has to download it newly just to be sure. I don't know
of any option to wget to assume that existing local files with the _same
size_ as remote files must be identical and to ignore them. That would
indeed be very dodgy and unsafe.

rsync was developed to amongst other things work around this kind of
problem - the protocol transmits the information needed to make this
decision instead of trying to rely on HTML headers. tinderbox.dev is
also not running rsyncd :-(

Re-downloading everything everytime seems to be your only option. I
don;t imagine that repo changes all that much with time though, why
don't you just re-sync infrequently, like once a week maybe?

-- 
Alan McKinnon
alan.mckinnon@gmail.com