Re: [gentoo-user] emerge --sync issue on only one comp on LAN

[Alan McKinnon <alan.mckinnon@gmail.com>]

On 03/10/2013 18:57, Paul Hartman wrote:
> On Wed, Oct 2, 2013 at 4:51 PM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
>> On 02/10/2013 19:37, Paul Hartman wrote:
>>> On Tue, Oct 1, 2013 at 10:45 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
>>>> On 01/10/2013 17:17, Greg Turner wrote:


[snip]


>> You don't sync very often, right?
> 
> I usually sync manually daily or every other day if I'm busy and don't
> get a chance. I assumed there was some mass change to ebuild headers
> or license text or something which caused everything in the tree to
> get touched this week.
> 
> My local portage tree is on a fast SSD in an 8-core box with 32GB of
> RAM and a 100mbit internet connection, so the bottleneck hopefully is
> not on my side of the transaction. ;)
> 
> Let's do some more trials. Between yesterday and today, I have synced
> with my normal mirror, but I'm syncing with your server again now:
> 
> Number of files: 174410
> Number of files transferred: 17372
> Total file size: 306.28M bytes
> Total transferred file size: 22.32M bytes
> Literal data: 22.32M bytes
> Matched data: 0 bytes
> File list size: 4.31M
> File list generation time: 379.920 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 382.35K
> Total bytes received: 15.71M
> 
> sent 382.35K bytes  received 15.71M bytes  29.33K bytes/sec
> total size is 306.28M  speedup is 19.04
> 
> Now I'm immediately doing another sync, first deleting timestamp.chk
> to force it to sync again. There should be zero files to transfer
> (except the timestamp file).
> 
> Number of files: 174410
> Number of files transferred: 1
> Total file size: 306.28M bytes
> Total transferred file size: 32 bytes
> Literal data: 32 bytes
> Matched data: 0 bytes
> File list size: 4.31M
> File list generation time: 28.612 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 183
> Total bytes received: 4.31M
> 
> sent 183 bytes  received 4.31M bytes  128.75K bytes/sec
> total size is 306.28M  speedup is 71.01
> 
> 
> Now I'm switching back to my beloved mirror.steadfast.net and running
> another sync.
> 
> Number of files: 174409
> Number of files transferred: 17364
> Total file size: 306.30M bytes
> Total transferred file size: 21.74M bytes
> Literal data: 21.74M bytes
> Matched data: 0 bytes
> File list size: 4.39M
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 355.23K
> Total bytes received: 15.67M
> 
> sent 355.23K bytes  received 15.67M bytes  191.93K bytes/sec
> total size is 306.30M  speedup is 19.11
> 
> 
> Interestingly it transferred almost the same number of files as my
> first sync with yours. Comparing timestamps, your server's latest
> update is about 5 hours older than Steadfast's, so things must be
> changing frequently in portage these days! 17k changes in 5 hours...


My guess is the metadata. I'll have to do some checks on that mirror,
IIRC it syncs every 6 hours, most likely steadfast syncs more often.

One of the things about running a mirror is, it's very much set it up
once and forget all about it evermore. Which is great and all, but users
tend to spot problems long before the sysadmins do :-)


> 
> My ping to your server is 300ms, my ping to steadfast is 18ms. I don't
> know anything about how rsync works behind the curtain, if a higher
> latency would cause the file list generation to be slower, or if that
> is a measurement of server performance or something else.


I don't think latency is much of a factor but let me re-read some FAQs
before commenting further.

300ms is totally normal from here to eu, uk and us - we're in deepest
darkest Africa where hyenas prowl the streets[1] - and all traffic goes
over undersea cable with *lots* of repeaters


> 
> Total sync times from my log:
> 
> 1380814364: >>> Starting rsync with rsync://196.4.160.12/gentoo-portage
> 1380814916: === Sync completed with rsync://196.4.160.12/gentoo-portage
> (first sync, 17k files updated, 552 seconds)
> 
> 1380815150: >>> Starting rsync with rsync://196.4.160.12/gentoo-portage
> 1380815188: === Sync completed with rsync://196.4.160.12/gentoo-portage
> (sync with no updates except timestamp.chk, 38 seconds)
> 
> 1380815292: >>> Starting rsync with rsync://208.100.4.53/gentoo-portage
> 1380815375: === Sync completed with rsync://208.100.4.53/gentoo-portage
> (re-sync with steadfast, 17k files updated, 83 seconds)
> 
> 1380816062: >>> Starting rsync with rsync://208.100.4.53/gentoo-portage
> 1380816074: === Sync completed with rsync://208.100.4.53/gentoo-portage
> (sync with no updates except timestamp.chk, 12 seconds)
> 
> 
> HTH and thanks for the mirror :)
> Paul
> 

[1] Literally. True's bob, I kid you not. A 6 month old brown hyena this
week wandered into the suburb where I live - it must have got separated
from it's mother and walked 15 miles in the dark to get here



-- 
Alan McKinnon
alan.mckinnon@gmail.com