Re: [gentoo-user] emerge --sync issue on only one comp on LAN

[Paul Hartman <paul.hartman+gentoo@gmail.com>]

On Wed, Oct 2, 2013 at 4:51 PM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> On 02/10/2013 19:37, Paul Hartman wrote:
>> On Tue, Oct 1, 2013 at 10:45 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
>>> On 01/10/2013 17:17, Greg Turner wrote:
>>>> Rsync mirrors don't grow on trees, man.  People pay good money to
>>>> provide that service to us.  You should seriously be embarrassed to
>>>> have posted this.
>>>
>>> Really?
>>>
>>>
>>> Then you can all use mine with the greatest of pleasure:
>>>
>>> SYNC="rsync://ftp.is.co.za/gentoo-portage"
>>>
>>>
>>> I have the NetOps team BEGGING me weekly to try and generate more
>>> traffic out of our network going international. The in-out ratio on the
>>> peering links is seriously screwed and they badly want something to even
>>> it out a bit :-)
>>
>> Challenge accepted. :)
>>
>> Here's my sync summary  (I'm in the USA):
>>
>> Number of files: 174305
>> Number of files transferred: 50913
>                                ^^^^^
>> Total file size: 305.99M bytes
>> Total transferred file size: 73.98M bytes
>> Literal data: 73.98M bytes
>> Matched data: 0 bytes
>> File list size: 4.31M
>> File list generation time: 343.526 seconds
>                              ^^^^^^^^^^^^^^^
>> File list transfer time: 0.000 seconds
>> Total bytes sent: 1.12M
>> Total bytes received: 41.37M
>>
>> sent 1.12M bytes  received 41.37M bytes  64.33K bytes/sec
>> total size is 305.99M  speedup is 7.20
>
>
> You don't sync very often, right?

I usually sync manually daily or every other day if I'm busy and don't
get a chance. I assumed there was some mass change to ebuild headers
or license text or something which caused everything in the tree to
get touched this week.

My local portage tree is on a fast SSD in an 8-core box with 32GB of
RAM and a 100mbit internet connection, so the bottleneck hopefully is
not on my side of the transaction. ;)

Let's do some more trials. Between yesterday and today, I have synced
with my normal mirror, but I'm syncing with your server again now:

Number of files: 174410
Number of files transferred: 17372
Total file size: 306.28M bytes
Total transferred file size: 22.32M bytes
Literal data: 22.32M bytes
Matched data: 0 bytes
File list size: 4.31M
File list generation time: 379.920 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 382.35K
Total bytes received: 15.71M

sent 382.35K bytes  received 15.71M bytes  29.33K bytes/sec
total size is 306.28M  speedup is 19.04

Now I'm immediately doing another sync, first deleting timestamp.chk
to force it to sync again. There should be zero files to transfer
(except the timestamp file).

Number of files: 174410
Number of files transferred: 1
Total file size: 306.28M bytes
Total transferred file size: 32 bytes
Literal data: 32 bytes
Matched data: 0 bytes
File list size: 4.31M
File list generation time: 28.612 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 183
Total bytes received: 4.31M

sent 183 bytes  received 4.31M bytes  128.75K bytes/sec
total size is 306.28M  speedup is 71.01


Now I'm switching back to my beloved mirror.steadfast.net and running
another sync.

Number of files: 174409
Number of files transferred: 17364
Total file size: 306.30M bytes
Total transferred file size: 21.74M bytes
Literal data: 21.74M bytes
Matched data: 0 bytes
File list size: 4.39M
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 355.23K
Total bytes received: 15.67M

sent 355.23K bytes  received 15.67M bytes  191.93K bytes/sec
total size is 306.30M  speedup is 19.11


Interestingly it transferred almost the same number of files as my
first sync with yours. Comparing timestamps, your server's latest
update is about 5 hours older than Steadfast's, so things must be
changing frequently in portage these days! 17k changes in 5 hours...

My ping to your server is 300ms, my ping to steadfast is 18ms. I don't
know anything about how rsync works behind the curtain, if a higher
latency would cause the file list generation to be slower, or if that
is a measurement of server performance or something else.

Total sync times from my log:

1380814364: >>> Starting rsync with rsync://196.4.160.12/gentoo-portage
1380814916: === Sync completed with rsync://196.4.160.12/gentoo-portage
(first sync, 17k files updated, 552 seconds)

1380815150: >>> Starting rsync with rsync://196.4.160.12/gentoo-portage
1380815188: === Sync completed with rsync://196.4.160.12/gentoo-portage
(sync with no updates except timestamp.chk, 38 seconds)

1380815292: >>> Starting rsync with rsync://208.100.4.53/gentoo-portage
1380815375: === Sync completed with rsync://208.100.4.53/gentoo-portage
(re-sync with steadfast, 17k files updated, 83 seconds)

1380816062: >>> Starting rsync with rsync://208.100.4.53/gentoo-portage
1380816074: === Sync completed with rsync://208.100.4.53/gentoo-portage
(sync with no updates except timestamp.chk, 12 seconds)


HTH and thanks for the mirror :)
Paul