[gentoo-user] emerge --sync issue on only one comp on LAN

[gentoo-user] emerge --sync issue on only one comp on LAN

[Bruce Hill]

There are 3 (or more) computers which sync (sometimes daily) on my LAN at
work: server, router, and workstation. server has issues almost all the time
getting a rsync server (for lack of better way to state it). All three comps
have the exact same SYNC in make.conf:

mingdao@server ~ $ grep SYNC /etc/make.conf
SYNC="rsync://rsync.us.gentoo.org/gentoo-portage"

mingdao@router ~ $ grep SYNC /etc/portage/make.conf
SYNC="rsync://rsync.us.gentoo.org/gentoo-portage"

mingdao@workstation ~ $ grep SYNC /etc/portage/make.conf
SYNC="rsync://rsync.us.gentoo.org/gentoo-portage"

When I first noticed, it was that with the same command (alias ud) run on each
computer, server *always* took a good deal longer than workstation and router.

Here's the output from the emerge --sync portion of my "ud" alias today:

alias ud='eix-sync && emerge -aDjuv --changed-use @world && dispatch-conf && emerge -a --depclean && revdep-rebuild -i && clear && exit'

server ~ # ud
 * Running emerge --sync
>>> Synchronization of repository 'gentoo' located in '/usr/portage'...
>>> Starting rsync with rsync://128.61.111.9/gentoo-portage...
>>> Checking server timestamp ...
timed out
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [Receiver=3.0.9]
>>> Retrying...


>>> Starting retry 1 of 9 with rsync://209.59.138.21/gentoo-portage
>>> Checking server timestamp ...
timed out
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [Receiver=3.0.9]
>>> Retrying...


>>> Starting retry 2 of 9 with rsync://128.175.60.112/gentoo-portage
>>> Checking server timestamp ...
timed out
rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [Receiver=3.0.9]
>>> Retrying...


>>> Starting retry 3 of 9 with rsync://129.21.171.98/gentoo-portage
>>> Checking server timestamp ...

receiving incremental file list
timestamp.chk

Number of files: 1
Number of files transferred: 1
Total file size: 32 bytes
Total transferred file size: 32 bytes
Literal data: 32 bytes
Matched data: 0 bytes
File list size: 27
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 98
Total bytes received: 140

sent 98 bytes  received 140 bytes  158.67 bytes/sec
total size is 32  speedup is 0.13

receiving incremental file list




router ~ # ud
 * Running emerge --sync
>>> Synchronization of repository 'gentoo' located in '/usr/portage'...
>>> Starting rsync with rsync://208.100.4.53/gentoo-portage...
>>> Checking server timestamp ...
 
 Welcome! This is a gentoo-portage and CentOS mirror, hosted by 
 Steadfast Networks!

 http://steadfast.net

   Hostname:     mirror.steadfast.net & rsync11.us.gentoo.org
   IP Addresses: 208.100.4.53 & 2607:f128:1:3::2
   Location:     Chicago, IL, US
   Bandwidth:    1000 Mbps
   Hardware:     Dual Opteron 2212, 8 GB RAM
   User Limit:   40

 If you experience any trouble with this mirror, please contact 
 mirror@steadfast.net.

receiving incremental file list
timestamp.chk

Number of files: 1
Number of files transferred: 1
Total file size: 32 bytes
Total transferred file size: 32 bytes
Literal data: 32 bytes
Matched data: 0 bytes
File list size: 27
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 98
Total bytes received: 570

sent 98 bytes  received 570 bytes  445.33 bytes/sec
total size is 32  speedup is 0.05
 
 Welcome! This is a gentoo-portage and CentOS mirror, hosted by 
 Steadfast Networks!

 http://steadfast.net

   Hostname:     mirror.steadfast.net & rsync11.us.gentoo.org
   IP Addresses: 208.100.4.53 & 2607:f128:1:3::2
   Location:     Chicago, IL, US
   Bandwidth:    1000 Mbps
   Hardware:     Dual Opteron 2212, 8 GB RAM
   User Limit:   40

 If you experience any trouble with this mirror, please contact 
 mirror@steadfast.net.

receiving incremental file list




workstation ~ # ud
 * Running emerge --sync
>>> Synchronization of repository 'gentoo' located in '/usr/portage'...
>>> Starting rsync with rsync://209.221.142.124/gentoo-portage...
>>> Checking server timestamp ...
Gentoo Portage/CPAN rsync mirror
Server: gentoo.llarian.net & cpan.llarian.net
IP(s): 209.221.142.124 & 2001:5d8:11::13
Bandwidth: 1000Mbps via multiple carriers
User Limit: 30
Location: Seattle, WA, USA
Admin Contact: Dylan Vanderhoof <llarian@llarian.net>


receiving incremental file list
timestamp.chk

Number of files: 1
Number of files transferred: 1
Total file size: 32 bytes
Total transferred file size: 32 bytes
Literal data: 32 bytes
Matched data: 0 bytes
File list size: 27
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 98
Total bytes received: 392

sent 98 bytes  received 392 bytes  326.67 bytes/sec
total size is 32  speedup is 0.07
Gentoo Portage/CPAN rsync mirror
Server: gentoo.llarian.net & cpan.llarian.net
IP(s): 209.221.142.124 & 2001:5d8:11::13
Bandwidth: 1000Mbps via multiple carriers
User Limit: 30
Location: Seattle, WA, USA
Admin Contact: Dylan Vanderhoof <llarian@llarian.net>


receiving incremental file list




Can anyone see something obvious, or tell me where to begin checking or
something to change?

Cheers,
Bruce
-- 
Happy Penguin Computers               >')
126 Fenco Drive                       ( \
Tupelo, MS 38801                       ^^
support@happypenguincomputers.com
662-269-2706 662-205-6424
http://happypenguincomputers.com/

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

Don't top-post: http://en.wikipedia.org/wiki/Top_post#Top-posting

Re: [gentoo-user] emerge --sync issue on only one comp on LAN

[Alan McKinnon]

On 01/10/2013 16:20, Bruce Hill wrote:
> There are 3 (or more) computers which sync (sometimes daily) on my LAN at
> work: server, router, and workstation. server has issues almost all the time
> getting a rsync server (for lack of better way to state it). All three comps
> have the exact same SYNC in make.conf:
> 
> mingdao@server ~ $ grep SYNC /etc/make.conf
> SYNC="rsync://rsync.us.gentoo.org/gentoo-portage"
> 
> mingdao@router ~ $ grep SYNC /etc/portage/make.conf
> SYNC="rsync://rsync.us.gentoo.org/gentoo-portage"
> 
> mingdao@workstation ~ $ grep SYNC /etc/portage/make.conf
> SYNC="rsync://rsync.us.gentoo.org/gentoo-portage"


Most likely obvious cause:

You have outbound firewall rules on your server/router affecting the server.















> 
> When I first noticed, it was that with the same command (alias ud) run on each
> computer, server *always* took a good deal longer than workstation and router.
> 
> Here's the output from the emerge --sync portion of my "ud" alias today:
> 
> alias ud='eix-sync && emerge -aDjuv --changed-use @world && dispatch-conf && emerge -a --depclean && revdep-rebuild -i && clear && exit'
> 
> server ~ # ud
>  * Running emerge --sync
>>>> Synchronization of repository 'gentoo' located in '/usr/portage'...
>>>> Starting rsync with rsync://128.61.111.9/gentoo-portage...
>>>> Checking server timestamp ...
> timed out
> rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [Receiver=3.0.9]
>>>> Retrying...
> 
> 
>>>> Starting retry 1 of 9 with rsync://209.59.138.21/gentoo-portage
>>>> Checking server timestamp ...
> timed out
> rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [Receiver=3.0.9]
>>>> Retrying...
> 
> 
>>>> Starting retry 2 of 9 with rsync://128.175.60.112/gentoo-portage
>>>> Checking server timestamp ...
> timed out
> rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [Receiver=3.0.9]
>>>> Retrying...
> 
> 
>>>> Starting retry 3 of 9 with rsync://129.21.171.98/gentoo-portage
>>>> Checking server timestamp ...
> 
> receiving incremental file list
> timestamp.chk
> 
> Number of files: 1
> Number of files transferred: 1
> Total file size: 32 bytes
> Total transferred file size: 32 bytes
> Literal data: 32 bytes
> Matched data: 0 bytes
> File list size: 27
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 98
> Total bytes received: 140
> 
> sent 98 bytes  received 140 bytes  158.67 bytes/sec
> total size is 32  speedup is 0.13
> 
> receiving incremental file list
> 
> 
> 
> 
> router ~ # ud
>  * Running emerge --sync
>>>> Synchronization of repository 'gentoo' located in '/usr/portage'...
>>>> Starting rsync with rsync://208.100.4.53/gentoo-portage...
>>>> Checking server timestamp ...
>  
>  Welcome! This is a gentoo-portage and CentOS mirror, hosted by 
>  Steadfast Networks!
> 
>  http://steadfast.net
> 
>    Hostname:     mirror.steadfast.net & rsync11.us.gentoo.org
>    IP Addresses: 208.100.4.53 & 2607:f128:1:3::2
>    Location:     Chicago, IL, US
>    Bandwidth:    1000 Mbps
>    Hardware:     Dual Opteron 2212, 8 GB RAM
>    User Limit:   40
> 
>  If you experience any trouble with this mirror, please contact 
>  mirror@steadfast.net.
> 
> receiving incremental file list
> timestamp.chk
> 
> Number of files: 1
> Number of files transferred: 1
> Total file size: 32 bytes
> Total transferred file size: 32 bytes
> Literal data: 32 bytes
> Matched data: 0 bytes
> File list size: 27
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 98
> Total bytes received: 570
> 
> sent 98 bytes  received 570 bytes  445.33 bytes/sec
> total size is 32  speedup is 0.05
>  
>  Welcome! This is a gentoo-portage and CentOS mirror, hosted by 
>  Steadfast Networks!
> 
>  http://steadfast.net
> 
>    Hostname:     mirror.steadfast.net & rsync11.us.gentoo.org
>    IP Addresses: 208.100.4.53 & 2607:f128:1:3::2
>    Location:     Chicago, IL, US
>    Bandwidth:    1000 Mbps
>    Hardware:     Dual Opteron 2212, 8 GB RAM
>    User Limit:   40
> 
>  If you experience any trouble with this mirror, please contact 
>  mirror@steadfast.net.
> 
> receiving incremental file list
> 
> 
> 
> 
> workstation ~ # ud
>  * Running emerge --sync
>>>> Synchronization of repository 'gentoo' located in '/usr/portage'...
>>>> Starting rsync with rsync://209.221.142.124/gentoo-portage...
>>>> Checking server timestamp ...
> Gentoo Portage/CPAN rsync mirror
> Server: gentoo.llarian.net & cpan.llarian.net
> IP(s): 209.221.142.124 & 2001:5d8:11::13
> Bandwidth: 1000Mbps via multiple carriers
> User Limit: 30
> Location: Seattle, WA, USA
> Admin Contact: Dylan Vanderhoof <llarian@llarian.net>
> 
> 
> receiving incremental file list
> timestamp.chk
> 
> Number of files: 1
> Number of files transferred: 1
> Total file size: 32 bytes
> Total transferred file size: 32 bytes
> Literal data: 32 bytes
> Matched data: 0 bytes
> File list size: 27
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 98
> Total bytes received: 392
> 
> sent 98 bytes  received 392 bytes  326.67 bytes/sec
> total size is 32  speedup is 0.07
> Gentoo Portage/CPAN rsync mirror
> Server: gentoo.llarian.net & cpan.llarian.net
> IP(s): 209.221.142.124 & 2001:5d8:11::13
> Bandwidth: 1000Mbps via multiple carriers
> User Limit: 30
> Location: Seattle, WA, USA
> Admin Contact: Dylan Vanderhoof <llarian@llarian.net>
> 
> 
> receiving incremental file list
> 
> 
> 
> 
> Can anyone see something obvious, or tell me where to begin checking or
> something to change?
> 
> Cheers,
> Bruce
> 


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] emerge --sync issue on only one comp on LAN

[Greg Turner]

On Tue, Oct 1, 2013 at 7:20 AM, Bruce Hill
<daddy@happypenguincomputers.com> wrote:
> There are 3 (or more) computers which sync (sometimes daily) on my LAN at
> work: server, router, and workstation. server has issues almost all the time
> getting a rsync server (for lack of better way to state it). All three comps
> have the exact same SYNC in make.conf:

Seriously?  Your problem is that an incredible build-up of bad karma
has polluted your network.  You are selfishly and pointlessly wasting
the rsync.us.gentoo.org mirror network's resources, and your own
bandwidth as well.

Run rsyncd somewhere.  Sync the other two systems to it.  If the
server has problems with outbound connectivity, use the router, I
guess.

Rsync mirrors don't grow on trees, man.  People pay good money to
provide that service to us.  You should seriously be embarrassed to
have posted this.

-gmt

Re: [gentoo-user] emerge --sync issue on only one comp on LAN

[Alan McKinnon]

On 01/10/2013 17:17, Greg Turner wrote:
> On Tue, Oct 1, 2013 at 7:20 AM, Bruce Hill
> <daddy@happypenguincomputers.com> wrote:
>> There are 3 (or more) computers which sync (sometimes daily) on my LAN at
>> work: server, router, and workstation. server has issues almost all the time
>> getting a rsync server (for lack of better way to state it). All three comps
>> have the exact same SYNC in make.conf:
> 
> Seriously?  Your problem is that an incredible build-up of bad karma
> has polluted your network.  You are selfishly and pointlessly wasting
> the rsync.us.gentoo.org mirror network's resources, and your own
> bandwidth as well.
> 
> Run rsyncd somewhere.  Sync the other two systems to it.  If the
> server has problems with outbound connectivity, use the router, I
> guess.
> 
> Rsync mirrors don't grow on trees, man.  People pay good money to
> provide that service to us.  You should seriously be embarrassed to
> have posted this.



Really?


Then you can all use mine with the greatest of pleasure:

SYNC="rsync://ftp.is.co.za/gentoo-portage"


I have the NetOps team BEGGING me weekly to try and generate more
traffic out of our network going international. The in-out ratio on the
peering links is seriously screwed and they badly want something to even
it out a bit :-)




-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] emerge --sync issue on only one comp on LAN

[Paul Hartman]

On Tue, Oct 1, 2013 at 10:45 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> On 01/10/2013 17:17, Greg Turner wrote:
>> Rsync mirrors don't grow on trees, man.  People pay good money to
>> provide that service to us.  You should seriously be embarrassed to
>> have posted this.
>
> Really?
>
>
> Then you can all use mine with the greatest of pleasure:
>
> SYNC="rsync://ftp.is.co.za/gentoo-portage"
>
>
> I have the NetOps team BEGGING me weekly to try and generate more
> traffic out of our network going international. The in-out ratio on the
> peering links is seriously screwed and they badly want something to even
> it out a bit :-)

Challenge accepted. :)

Here's my sync summary  (I'm in the USA):

Number of files: 174305
Number of files transferred: 50913
Total file size: 305.99M bytes
Total transferred file size: 73.98M bytes
Literal data: 73.98M bytes
Matched data: 0 bytes
File list size: 4.31M
File list generation time: 343.526 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 1.12M
Total bytes received: 41.37M

sent 1.12M bytes  received 41.37M bytes  64.33K bytes/sec
total size is 305.99M  speedup is 7.20

Re: [gentoo-user] emerge --sync issue on only one comp on LAN

[Alan McKinnon]

On 02/10/2013 19:37, Paul Hartman wrote:
> On Tue, Oct 1, 2013 at 10:45 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
>> On 01/10/2013 17:17, Greg Turner wrote:
>>> Rsync mirrors don't grow on trees, man.  People pay good money to
>>> provide that service to us.  You should seriously be embarrassed to
>>> have posted this.
>>
>> Really?
>>
>>
>> Then you can all use mine with the greatest of pleasure:
>>
>> SYNC="rsync://ftp.is.co.za/gentoo-portage"
>>
>>
>> I have the NetOps team BEGGING me weekly to try and generate more
>> traffic out of our network going international. The in-out ratio on the
>> peering links is seriously screwed and they badly want something to even
>> it out a bit :-)
> 
> Challenge accepted. :)
> 
> Here's my sync summary  (I'm in the USA):
> 
> Number of files: 174305
> Number of files transferred: 50913
                               ^^^^^
> Total file size: 305.99M bytes
> Total transferred file size: 73.98M bytes
> Literal data: 73.98M bytes
> Matched data: 0 bytes
> File list size: 4.31M
> File list generation time: 343.526 seconds
                             ^^^^^^^^^^^^^^^
> File list transfer time: 0.000 seconds
> Total bytes sent: 1.12M
> Total bytes received: 41.37M
> 
> sent 1.12M bytes  received 41.37M bytes  64.33K bytes/sec
> total size is 305.99M  speedup is 7.20


You don't sync very often, right?


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] emerge --sync issue on only one comp on LAN

[Paul Hartman]

On Wed, Oct 2, 2013 at 4:51 PM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> On 02/10/2013 19:37, Paul Hartman wrote:
>> On Tue, Oct 1, 2013 at 10:45 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
>>> On 01/10/2013 17:17, Greg Turner wrote:
>>>> Rsync mirrors don't grow on trees, man.  People pay good money to
>>>> provide that service to us.  You should seriously be embarrassed to
>>>> have posted this.
>>>
>>> Really?
>>>
>>>
>>> Then you can all use mine with the greatest of pleasure:
>>>
>>> SYNC="rsync://ftp.is.co.za/gentoo-portage"
>>>
>>>
>>> I have the NetOps team BEGGING me weekly to try and generate more
>>> traffic out of our network going international. The in-out ratio on the
>>> peering links is seriously screwed and they badly want something to even
>>> it out a bit :-)
>>
>> Challenge accepted. :)
>>
>> Here's my sync summary  (I'm in the USA):
>>
>> Number of files: 174305
>> Number of files transferred: 50913
>                                ^^^^^
>> Total file size: 305.99M bytes
>> Total transferred file size: 73.98M bytes
>> Literal data: 73.98M bytes
>> Matched data: 0 bytes
>> File list size: 4.31M
>> File list generation time: 343.526 seconds
>                              ^^^^^^^^^^^^^^^
>> File list transfer time: 0.000 seconds
>> Total bytes sent: 1.12M
>> Total bytes received: 41.37M
>>
>> sent 1.12M bytes  received 41.37M bytes  64.33K bytes/sec
>> total size is 305.99M  speedup is 7.20
>
>
> You don't sync very often, right?

I usually sync manually daily or every other day if I'm busy and don't
get a chance. I assumed there was some mass change to ebuild headers
or license text or something which caused everything in the tree to
get touched this week.

My local portage tree is on a fast SSD in an 8-core box with 32GB of
RAM and a 100mbit internet connection, so the bottleneck hopefully is
not on my side of the transaction. ;)

Let's do some more trials. Between yesterday and today, I have synced
with my normal mirror, but I'm syncing with your server again now:

Number of files: 174410
Number of files transferred: 17372
Total file size: 306.28M bytes
Total transferred file size: 22.32M bytes
Literal data: 22.32M bytes
Matched data: 0 bytes
File list size: 4.31M
File list generation time: 379.920 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 382.35K
Total bytes received: 15.71M

sent 382.35K bytes  received 15.71M bytes  29.33K bytes/sec
total size is 306.28M  speedup is 19.04

Now I'm immediately doing another sync, first deleting timestamp.chk
to force it to sync again. There should be zero files to transfer
(except the timestamp file).

Number of files: 174410
Number of files transferred: 1
Total file size: 306.28M bytes
Total transferred file size: 32 bytes
Literal data: 32 bytes
Matched data: 0 bytes
File list size: 4.31M
File list generation time: 28.612 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 183
Total bytes received: 4.31M

sent 183 bytes  received 4.31M bytes  128.75K bytes/sec
total size is 306.28M  speedup is 71.01


Now I'm switching back to my beloved mirror.steadfast.net and running
another sync.

Number of files: 174409
Number of files transferred: 17364
Total file size: 306.30M bytes
Total transferred file size: 21.74M bytes
Literal data: 21.74M bytes
Matched data: 0 bytes
File list size: 4.39M
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 355.23K
Total bytes received: 15.67M

sent 355.23K bytes  received 15.67M bytes  191.93K bytes/sec
total size is 306.30M  speedup is 19.11


Interestingly it transferred almost the same number of files as my
first sync with yours. Comparing timestamps, your server's latest
update is about 5 hours older than Steadfast's, so things must be
changing frequently in portage these days! 17k changes in 5 hours...

My ping to your server is 300ms, my ping to steadfast is 18ms. I don't
know anything about how rsync works behind the curtain, if a higher
latency would cause the file list generation to be slower, or if that
is a measurement of server performance or something else.

Total sync times from my log:

1380814364: >>> Starting rsync with rsync://196.4.160.12/gentoo-portage
1380814916: === Sync completed with rsync://196.4.160.12/gentoo-portage
(first sync, 17k files updated, 552 seconds)

1380815150: >>> Starting rsync with rsync://196.4.160.12/gentoo-portage
1380815188: === Sync completed with rsync://196.4.160.12/gentoo-portage
(sync with no updates except timestamp.chk, 38 seconds)

1380815292: >>> Starting rsync with rsync://208.100.4.53/gentoo-portage
1380815375: === Sync completed with rsync://208.100.4.53/gentoo-portage
(re-sync with steadfast, 17k files updated, 83 seconds)

1380816062: >>> Starting rsync with rsync://208.100.4.53/gentoo-portage
1380816074: === Sync completed with rsync://208.100.4.53/gentoo-portage
(sync with no updates except timestamp.chk, 12 seconds)


HTH and thanks for the mirror :)
Paul

Re: [gentoo-user] emerge --sync issue on only one comp on LAN

[Alan McKinnon]

On 03/10/2013 18:57, Paul Hartman wrote:
> On Wed, Oct 2, 2013 at 4:51 PM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
>> On 02/10/2013 19:37, Paul Hartman wrote:
>>> On Tue, Oct 1, 2013 at 10:45 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
>>>> On 01/10/2013 17:17, Greg Turner wrote:


[snip]


>> You don't sync very often, right?
> 
> I usually sync manually daily or every other day if I'm busy and don't
> get a chance. I assumed there was some mass change to ebuild headers
> or license text or something which caused everything in the tree to
> get touched this week.
> 
> My local portage tree is on a fast SSD in an 8-core box with 32GB of
> RAM and a 100mbit internet connection, so the bottleneck hopefully is
> not on my side of the transaction. ;)
> 
> Let's do some more trials. Between yesterday and today, I have synced
> with my normal mirror, but I'm syncing with your server again now:
> 
> Number of files: 174410
> Number of files transferred: 17372
> Total file size: 306.28M bytes
> Total transferred file size: 22.32M bytes
> Literal data: 22.32M bytes
> Matched data: 0 bytes
> File list size: 4.31M
> File list generation time: 379.920 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 382.35K
> Total bytes received: 15.71M
> 
> sent 382.35K bytes  received 15.71M bytes  29.33K bytes/sec
> total size is 306.28M  speedup is 19.04
> 
> Now I'm immediately doing another sync, first deleting timestamp.chk
> to force it to sync again. There should be zero files to transfer
> (except the timestamp file).
> 
> Number of files: 174410
> Number of files transferred: 1
> Total file size: 306.28M bytes
> Total transferred file size: 32 bytes
> Literal data: 32 bytes
> Matched data: 0 bytes
> File list size: 4.31M
> File list generation time: 28.612 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 183
> Total bytes received: 4.31M
> 
> sent 183 bytes  received 4.31M bytes  128.75K bytes/sec
> total size is 306.28M  speedup is 71.01
> 
> 
> Now I'm switching back to my beloved mirror.steadfast.net and running
> another sync.
> 
> Number of files: 174409
> Number of files transferred: 17364
> Total file size: 306.30M bytes
> Total transferred file size: 21.74M bytes
> Literal data: 21.74M bytes
> Matched data: 0 bytes
> File list size: 4.39M
> File list generation time: 0.001 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 355.23K
> Total bytes received: 15.67M
> 
> sent 355.23K bytes  received 15.67M bytes  191.93K bytes/sec
> total size is 306.30M  speedup is 19.11
> 
> 
> Interestingly it transferred almost the same number of files as my
> first sync with yours. Comparing timestamps, your server's latest
> update is about 5 hours older than Steadfast's, so things must be
> changing frequently in portage these days! 17k changes in 5 hours...


My guess is the metadata. I'll have to do some checks on that mirror,
IIRC it syncs every 6 hours, most likely steadfast syncs more often.

One of the things about running a mirror is, it's very much set it up
once and forget all about it evermore. Which is great and all, but users
tend to spot problems long before the sysadmins do :-)


> 
> My ping to your server is 300ms, my ping to steadfast is 18ms. I don't
> know anything about how rsync works behind the curtain, if a higher
> latency would cause the file list generation to be slower, or if that
> is a measurement of server performance or something else.


I don't think latency is much of a factor but let me re-read some FAQs
before commenting further.

300ms is totally normal from here to eu, uk and us - we're in deepest
darkest Africa where hyenas prowl the streets[1] - and all traffic goes
over undersea cable with *lots* of repeaters


> 
> Total sync times from my log:
> 
> 1380814364: >>> Starting rsync with rsync://196.4.160.12/gentoo-portage
> 1380814916: === Sync completed with rsync://196.4.160.12/gentoo-portage
> (first sync, 17k files updated, 552 seconds)
> 
> 1380815150: >>> Starting rsync with rsync://196.4.160.12/gentoo-portage
> 1380815188: === Sync completed with rsync://196.4.160.12/gentoo-portage
> (sync with no updates except timestamp.chk, 38 seconds)
> 
> 1380815292: >>> Starting rsync with rsync://208.100.4.53/gentoo-portage
> 1380815375: === Sync completed with rsync://208.100.4.53/gentoo-portage
> (re-sync with steadfast, 17k files updated, 83 seconds)
> 
> 1380816062: >>> Starting rsync with rsync://208.100.4.53/gentoo-portage
> 1380816074: === Sync completed with rsync://208.100.4.53/gentoo-portage
> (sync with no updates except timestamp.chk, 12 seconds)
> 
> 
> HTH and thanks for the mirror :)
> Paul
> 

[1] Literally. True's bob, I kid you not. A 6 month old brown hyena this
week wandered into the suburb where I live - it must have got separated
from it's mother and walked 15 miles in the dark to get here



-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] emerge --sync issue on only one comp on LAN

[Bruce Hill]

On Tue, Oct 01, 2013 at 05:45:13PM +0200, Alan McKinnon wrote:
> 
> Then you can all use mine with the greatest of pleasure:
> 
> SYNC="rsync://ftp.is.co.za/gentoo-portage"
> 
> 
> I have the NetOps team BEGGING me weekly to try and generate more
> traffic out of our network going international. The in-out ratio on the
> peering links is seriously screwed and they badly want something to even
> it out a bit :-)

Just for giggles I tried it too, from the US. Slower than turtles mating. ;)
-- 
Happy Penguin Computers               >')
126 Fenco Drive                       ( \
Tupelo, MS 38801                       ^^
support@happypenguincomputers.com
662-269-2706 662-205-6424
http://happypenguincomputers.com/

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

Don't top-post: http://en.wikipedia.org/wiki/Top_post#Top-posting

Re: [gentoo-user] emerge --sync issue on only one comp on LAN

[Alan McKinnon]

On 02/10/2013 20:48, Bruce Hill wrote:
> On Tue, Oct 01, 2013 at 05:45:13PM +0200, Alan McKinnon wrote:
>>
>> Then you can all use mine with the greatest of pleasure:
>>
>> SYNC="rsync://ftp.is.co.za/gentoo-portage"
>>
>>
>> I have the NetOps team BEGGING me weekly to try and generate more
>> traffic out of our network going international. The in-out ratio on the
>> peering links is seriously screwed and they badly want something to even
>> it out a bit :-)
> 
> Just for giggles I tried it too, from the US. Slower than turtles mating. ;)
> 


Give the poor thing a break - it's not dedicated to Gentoo :-)

There's about 11TB of mirrors on that host = full copies of every major
distro, most of the popular middle teir ones and full copies of all 3 BSDs

The general public keep it busy - I blame them :-)


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] emerge --sync issue on only one comp on LAN

[Bruce Hill]

On Wed, Oct 02, 2013 at 11:54:30PM +0200, Alan McKinnon wrote:
> On 02/10/2013 20:48, Bruce Hill wrote:
> > On Tue, Oct 01, 2013 at 05:45:13PM +0200, Alan McKinnon wrote:
> >>
> >> Then you can all use mine with the greatest of pleasure:
> >>
> >> SYNC="rsync://ftp.is.co.za/gentoo-portage"
> >>
> >>
> >> I have the NetOps team BEGGING me weekly to try and generate more
> >> traffic out of our network going international. The in-out ratio on the
> >> peering links is seriously screwed and they badly want something to even
> >> it out a bit :-)
> > 
> > Just for giggles I tried it too, from the US. Slower than turtles mating. ;)
> > 
> 
> 
> Give the poor thing a break - it's not dedicated to Gentoo :-)
> 
> There's about 11TB of mirrors on that host = full copies of every major
> distro, most of the popular middle teir ones and full copies of all 3 BSDs
> 
> The general public keep it busy - I blame them :-)

I didn't mean anything disrespectful (notice my winky), just that from the US
it's rather slow. But it's still listed, commented, in make.conf.

Thanks, Alan!
-- 
Happy Penguin Computers               >')
126 Fenco Drive                       ( \
Tupelo, MS 38801                       ^^
support@happypenguincomputers.com
662-269-2706 662-205-6424
http://happypenguincomputers.com/

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

Don't top-post: http://en.wikipedia.org/wiki/Top_post#Top-posting

Re: [gentoo-user] emerge --sync issue on only one comp on LAN

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 1221 bytes --]

On Tuesday 01 Oct 2013 15:20:06 Bruce Hill wrote:
> There are 3 (or more) computers which sync (sometimes daily) on my LAN at
> work: server, router, and workstation. server has issues almost all the
> time getting a rsync server (for lack of better way to state it). All
> three comps have the exact same SYNC in make.conf:
> 
> mingdao@server ~ $ grep SYNC /etc/make.conf
> SYNC="rsync://rsync.us.gentoo.org/gentoo-portage"
[snip ...]

> Can anyone see something obvious, or tell me where to begin checking or
> something to change?

I assume that this problem occurs even when trying to sync the server alone, 
rather than concurrently with the other two machines.  If not, then the rsync 
mirror may be blocking you due to repeated attempts from the same source IP 
address.


Some things to check:

Have you looked at the router's logs to see what happens to the packets?

When this fault occurs, have you checked that the server can still resolve 
URLs and there is no DNS problem?

Finally, as has already been mentioned, you could set up a LAN rsync mirror 
for your machines, but this does not answer why the particular machine is 
having connection problems.

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 490 bytes --]