[gentoo-mirrors] Incredibly slow rsync from masterdistfiles.gentoo.org

[gentoo-mirrors] Incredibly slow rsync from masterdistfiles.gentoo.org

[Martin Eisenhardt]

Hi everyone,

I am (i.e. gentoo.mneisen.org is) currenty experiencing a seriously  
slow rsync from masterdistfiles.gentoo.org. The maximum bandwidth I  
get is about 11 k/s (ca. 90 kbit).

The problem started around 0900 UTC.

Connections to other sites are fine.

Can anyone reproduce this, or is this a problem on my end (but then  
again, connection speed to other sites *are* fine).


Regards
Martin

Re: [gentoo-mirrors] Incredibly slow rsync from masterdistfiles.gentoo.org

[Manuel Tiago Pereira]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Greetings,

This is what happened today:

> Started update at Wed Oct 28 08:00:01 WET 2009
> 2009/10/28 08:03:10 [29046] rsync: failed to connect to masterdistfiles.gentoo.org: Connection timed out (110)
> 2009/10/28 08:03:10 [29046] rsync error: error in socket IO (code 10) at clientserver.c(124) [receiver=3.0.5]
> End: Wed Oct 28 08:03:10 WET 2009

Best regards,

Martin Eisenhardt wrote:
> Hi everyone,
> 
> I am (i.e. gentoo.mneisen.org is) currenty experiencing a seriously slow
> rsync from masterdistfiles.gentoo.org. The maximum bandwidth I get is
> about 11 k/s (ca. 90 kbit).
> 
> The problem started around 0900 UTC.
> 
> Connections to other sites are fine.
> 
> Can anyone reproduce this, or is this a problem on my end (but then
> again, connection speed to other sites *are* fine).
> 
> 
> Regards
> Martin
> 


- --
Manuel Tiago Correia Casimiro da Silva Pereira
tiago.pereira@rnl.ist.utl.pt

Administração de Sistemas da Rede das Novas Licenciaturas
Instituto Superior Técnico

web: https://www.rnl.ist.utl.pt
email: rnl@rnl.ist.utl.pt
telefone: +351 218 41 77 71
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkroMtIACgkQEg3Bh8i7j0JhcwCfcHo/dZXM61X0BLVcSntPH4m2
vnMAnijg6iyYYFwAQF389tiE9fY3gIOY
=Ev4r
-----END PGP SIGNATURE-----

Re: [gentoo-mirrors] Incredibly slow rsync from masterdistfiles.gentoo.org

[Xavier Parizet]

Hi,

This is what happened too today:

> Started update at Wed Oct 28 12:00:01 CET 2009
> rsync: connection unexpectedly closed (0 bytes received so far) [receiver]
> rsync error: error in rsync protocol data stream (code 12) at io.c(600)
[receiver=3.0.5]
> Sync failed with error 12, retrying for the 1nth time...
> rsync: read error: Connection reset by peer (104)
> rsync error: error in rsync protocol data stream (code 12) at io.c(759)
[receiver=3.0.5]
> Sync failed with error 12, retrying for the 2nth time...
> Sync successfull at 2nth time !
> End: Wed Oct 28 12:41:24 CET 2009

Best regards,

Xavier

Manuel Tiago Pereira a écrit :
> Greetings,
> 
> This is what happened today:
> 
>> Started update at Wed Oct 28 08:00:01 WET 2009
>> 2009/10/28 08:03:10 [29046] rsync: failed to connect to masterdistfiles.gentoo.org: Connection timed out (110)
>> 2009/10/28 08:03:10 [29046] rsync error: error in socket IO (code 10) at clientserver.c(124) [receiver=3.0.5]
>> End: Wed Oct 28 08:03:10 WET 2009
> 
> Best regards,
> 
> Martin Eisenhardt wrote:
>> Hi everyone,
> 
>> I am (i.e. gentoo.mneisen.org is) currenty experiencing a seriously slow
>> rsync from masterdistfiles.gentoo.org. The maximum bandwidth I get is
>> about 11 k/s (ca. 90 kbit).
> 
>> The problem started around 0900 UTC.
> 
>> Connections to other sites are fine.
> 
>> Can anyone reproduce this, or is this a problem on my end (but then
>> again, connection speed to other sites *are* fine).
> 
> 
>> Regards
>> Martin
> 
> 
> 

Re: [gentoo-mirrors] Incredibly slow rsync from masterdistfiles.gentoo.org

[Jeremy Olexa]

On Wed, 28 Oct 2009 12:44:20 +0100, Martin Eisenhardt
<martin.eisenhardt@mneisen.org> wrote:
> Hi everyone,
> 
> I am (i.e. gentoo.mneisen.org is) currenty experiencing a seriously  
> slow rsync from masterdistfiles.gentoo.org. The maximum bandwidth I  
> get is about 11 k/s (ca. 90 kbit).
> 
> The problem started around 0900 UTC.
> 
> Connections to other sites are fine.
> 
> Can anyone reproduce this, or is this a problem on my end (but then  
> again, connection speed to other sites *are* fine).
> 
> 
> Regards
> Martin

Martin (and others),
Thanks for reporting. I am following up with the issue now. I will report
back later today.

-Jeremy

Re: [gentoo-mirrors] Incredibly slow rsync from masterdistfiles.gentoo.org

[Jeremy Olexa]

On Wed, 28 Oct 2009 14:38:43 +0000, Jeremy Olexa <darkside@gentoo.org>
wrote:
> On Wed, 28 Oct 2009 12:44:20 +0100, Martin Eisenhardt
> <martin.eisenhardt@mneisen.org> wrote:
>> Hi everyone,
>> 
>> I am (i.e. gentoo.mneisen.org is) currenty experiencing a seriously  
>> slow rsync from masterdistfiles.gentoo.org. The maximum bandwidth I  
>> get is about 11 k/s (ca. 90 kbit).
>> 
>> The problem started around 0900 UTC.
>> 
>> Connections to other sites are fine.
>> 
>> Can anyone reproduce this, or is this a problem on my end (but then  
>> again, connection speed to other sites *are* fine).
>> 
>> 
>> Regards
>> Martin
> 
> Martin (and others),
> Thanks for reporting. I am following up with the issue now. I will
report
> back later today.
> 
> -Jeremy

All -
I am able to pull 700K/s on my home ISP connection in the USA. If you are
still
having slow speeds then please provide a traceroute from your host that
you
are trying to sync from.

To those of you that are having connection timeouts. If it is still
happening then we need to see what IP you are getting from 'host
masterdistfiles.gentoo.org' and what your IP address is that is timing
out.

If you previously had issues and they are resolved now, please say what
your issue was and if it is resolved now.

Thanks for your help,
Jeremy

Re: [gentoo-mirrors] Incredibly slow rsync from masterdistfiles.gentoo.org

[Martin Eisenhardt]

Hi Jeremy, hi list,

On 28.10.2009, at 18:00, Jeremy Olexa wrote:
> If you are still having slow speeds then please provide a traceroute  
> from your host that
> you are trying to sync from.

Yes, I do still have problems. The traceroute from gentoo.mneisen.org  
to masterdistfiles.gentoo.org is as follows:

root@node3:~# traceroute masterdistfiles.gentoo.org
traceroute to rsync.osuosl.org (140.211.166.134), 64 hops max, 52 byte  
packets
  1  static.1.51.40.188.clients.your-server.de (188.40.51.1)  1 ms  0  
ms  0 ms
  2  hos-tr4.juniper2.rz10.hetzner.de (213.239.227.225)  0 ms hos- 
tr3.juniper2.rz10.hetzner.de (213.239.227.193)  0 ms hos- 
tr1.juniper1.rz10.hetzner.de (213.239.227.129)  0 ms
  3  hos-bb2.juniper2.s04.hetzner.de (213.239.240.138)  3 ms  3 ms  3 ms
  4  lambdanet-gw.hetzner.de (213.239.242.213)  3 ms  3 ms  3 ms
  5  FRA-3-eth000.de.lambdanet.net (217.71.96.73)  6 ms  6 ms  6 ms
  6  FRA-1-eth000.de.lambdanet.net (217.71.96.69)  6 ms  6 ms  6 ms
  7  xe-5-1-0.edge6.Frankfurt1.Level3.net (195.16.161.5)  6 ms  6 ms   
6 ms
  8  vlan89.csw3.Frankfurt1.Level3.net (4.68.23.190)  7 ms  11 ms  
vlan79.csw2.Frankfurt1.Level3.net (4.68.23.126)  18 ms
  9  ae-72-72.ebr2.Frankfurt1.Level3.net (4.69.140.21)  7 ms  
ae-82-82.ebr2.Frankfurt1.Level3.net (4.69.140.25)  7 ms  
ae-62-62.ebr2.Frankfurt1.Level3.net (4.69.140.17)  7 ms
10  ae-43-43.ebr2.Washington1.Level3.net (4.69.137.58)  96 ms  
ae-41-41.ebr2.Washington1.Level3.net (4.69.137.50)  97 ms  
ae-43-43.ebr2.Washington1.Level3.net (4.69.137.58)  96 ms
11  ae-2-2.ebr2.Chicago2.Level3.net (4.69.132.69)  115 ms  114 ms  114  
ms
12  ae-1-100.ebr1.Chicago2.Level3.net (4.69.132.113)  115 ms  115 ms   
114 ms
13  ae-3.ebr2.Denver1.Level3.net (4.69.132.61)  143 ms  138 ms  144 ms
14  ae-2.ebr2.Seattle1.Level3.net (4.69.132.53)  170 ms  173 ms  164 ms
15  ae-21-52.car1.Seattle1.Level3.net (4.68.105.34)  162 ms  164 ms   
163 ms
16  eugnor1wce1-univ-of-oregon.wcg.net (64.200.134.198)  188 ms  188  
ms  186 ms
17  corv-car1-gw.nero.net (207.98.64.18)  188 ms (TOS=32!)  188 ms   
188 ms
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
31  * * *
32  * * *
33  * * *
34  * * *
35  * * *
36  * * *
37  * * *
38  * * *
39  * * *
40  * * *
41  * * *
42  * * *
43  * * *
44  * * *
45  * * *
46  * * *
47  * * *
48  * * *
49  * * *
50  * * *
51  * * *
52  * * *
53  * * *
54  * * *
55  * * *
56  * * *
57  * * *
58  * * *
59  * * *
60  * * *
61  * * *
62  * * *
63  * * *
64  * * *


>
> To those of you that are having connection timeouts. If it is still
> happening then we need to see what IP you are getting from 'host
> masterdistfiles.gentoo.org' and what your IP address is that is timing
> out.

No timeouts on my site.

> If you previously had issues and they are resolved now, please say  
> what
> your issue was and if it is resolved now.

Sadly, my issue is still unresolved. If there is anything I can do to  
help you find the error and fix this problem, please let me know.


Best regards
Martin

Re: [gentoo-mirrors] Incredibly slow rsync from masterdistfiles.gentoo.org

[Martin Eisenhardt]

Hi Jeremy, hi list,

I just ran my update script manually to see what's happening. I get  
this:

distfiles/09f7962b8b44.tar.gz
      7906417 100%   36.60MB/s    0:00:00 (xfer#1, to-check=40510/40518)
distfiles/2.6.31-zen5.patch.lzma
      2336309 100%    7.20kB/s    0:05:17 (xfer#2, to-check=40476/40518)
distfiles/74891-systemloadviewer.tar.gz
        17385 100%   16.58MB/s    0:00:00 (xfer#3, to-check=40417/40518)
distfiles/99737-fancytasks-0.9.8.tar.bz2
       105003 100%    8.24kB/s    0:00:12 (xfer#4, to-check=40403/40518)
distfiles/FlickrNet2.2-Src-48055.zip
       125574 100%  326.15kB/s    0:00:00 (xfer#5, to-check=38911/40518)
distfiles/INIT.2008-07-25.tgz
       349502 100%  822.44kB/s    0:00:00 (xfer#6, to-check=38367/40518)
distfiles/Jubler-src-4.1.tar.bz2
      1761921 100%    2.23MB/s    0:00:00 (xfer#7, to-check=38129/40518)
distfiles/MUMmer3.22.tar.gz
      3156338 100%    7.36kB/s    0:06:58 (xfer#8, to-check=37778/40518)
distfiles/NVIDIA-FreeBSD-x86-173.14.17.tar.gz
     16206185 100%   11.18MB/s    0:00:01 (xfer#9, to-check=37493/40518)
distfiles/NVIDIA-FreeBSD-x86-177.82.tar.gz
     16708972 100%    7.46MB/s    0:00:02 (xfer#10, to- 
check=37490/40518)
distfiles/NVIDIA-FreeBSD-x86-180.22.tar.gz
     17867440 100%   13.14MB/s    0:00:01 (xfer#11, to- 
check=37489/40518)
distfiles/NVIDIA-FreeBSD-x86-180.35.tar.gz
     18575403 100%    7.56MB/s    0:00:02 (xfer#12, to- 
check=37487/40518)
distfiles/NVIDIA-FreeBSD-x86-180.44.tar.gz
     18591139 100%    9.83MB/s    0:00:01 (xfer#13, to- 
check=37486/40518)
distfiles/NVIDIA-Linux-x86-173.14.17-pkg0.run
     11881672 100%    8.29MB/s    0:00:01 (xfer#14, to- 
check=37465/40518)
distfiles/NVIDIA-Linux-x86-177.82-pkg0.run
     12747296 100%    3.83MB/s    0:00:03 (xfer#15, to- 
check=37462/40518)
distfiles/NVIDIA-Linux-x86-180.22-pkg0.run
     13205766 100%   10.13MB/s    0:00:01 (xfer#16, to- 
check=37461/40518)
distfiles/NVIDIA-Linux-x86-180.35-pkg0.run
     13550574 100%    6.37MB/s    0:00:02 (xfer#17, to- 
check=37459/40518)
distfiles/NVIDIA-Linux-x86-180.44-pkg0.run
     13565325 100%    8.36MB/s    0:00:01 (xfer#18, to- 
check=37458/40518)
distfiles/NVIDIA-Linux-x86_64-173.14.17-pkg2.run
     17153760 100%    1.83MB/s    0:00:08 (xfer#19, to- 
check=37434/40518)
distfiles/NVIDIA-Linux-x86_64-177.82-pkg2.run
     19416201 100%    6.58MB/s    0:00:02 (xfer#20, to- 
check=37431/40518)
distfiles/NVIDIA-Linux-x86_64-180.22-pkg2.run
     20754738 100%   12.43MB/s    0:00:01 (xfer#21, to- 
check=37430/40518)
distfiles/NVIDIA-Linux-x86_64-180.35-pkg2.run
     21457935 100%    9.95MB/s    0:00:02 (xfer#22, to- 
check=37428/40518)
distfiles/NVIDIA-Linux-x86_64-180.44-pkg2.run
     21472494 100%    9.81MB/s    0:00:02 (xfer#23, to- 
check=37427/40518)
distfiles/NativeThread-20080330.tar.bz2
         2707 100%    2.34kB/s    0:00:01 (xfer#24, to- 
check=37399/40518)
distfiles/OUI.list.20091029.gz
       131958 100%   19.86kB/s    0:00:06 (xfer#25, to- 
check=36906/40518)
distfiles/PyQt-x11-gpl-4.6.1.tar.gz
       786432  11%    9.43kB/s    0:10:57


So, some files are transferred at high speeds, some other ones are  
still incredibly slow.


HTH.


Best regards
Martin

Re: [gentoo-mirrors] Incredibly slow rsync from masterdistfiles.gentoo.org

[Jeremy Olexa]

On Thu, 29 Oct 2009 10:36:26 +0100, Martin Eisenhardt
<martin.eisenhardt@mneisen.org> wrote:
> Hi Jeremy, hi list,
> 
> On 28.10.2009, at 18:00, Jeremy Olexa wrote:
>> If you are still having slow speeds then please provide a traceroute  
>> from your host that
>> you are trying to sync from.
>
> Sadly, my issue is still unresolved. If there is anything I can do to  
> help you find the error and fix this problem, please let me know.

Martin,
Support from OSU (our source for masterdistfiles) said that there was some
problems in the past week with their DNS view. This may or may not be
related but they say that since you are getting the 140. address (which is
routed for people on I2) then there should not be any speed issues. Here,
they recommend to ask your upstream provider if they are having routing
issues to masterdistfiles or any other limitations. Additionally,
recommended to post the output of "mtr masterdistfiles.gentoo.org" as that
might provide some better clues.

Xavier,
You were seeing timeouts and since have told me that they went away. OSU
says this is from that mix up earlier. Nearly 80% of the traffic was routed
incorrectly. So they say.

I hope this problem resolves itself here.
Thanks,
-Jeremy

Re: [gentoo-mirrors] Incredibly slow rsync from masterdistfiles.gentoo.org

[Carlos Carvalho]

Jeremy Olexa (darkside@gentoo.org) wrote on 30 October 2009 19:34:
 >On Thu, 29 Oct 2009 10:36:26 +0100, Martin Eisenhardt
 ><martin.eisenhardt@mneisen.org> wrote:
 >> Hi Jeremy, hi list,
 >> 
 >> On 28.10.2009, at 18:00, Jeremy Olexa wrote:
 >>> If you are still having slow speeds then please provide a traceroute  
 >>> from your host that
 >>> you are trying to sync from.
 >>
 >> Sadly, my issue is still unresolved. If there is anything I can do to  
 >> help you find the error and fix this problem, please let me know.
 >
 >Martin,
 >Support from OSU (our source for masterdistfiles) said that there was some
 >problems in the past week with their DNS view. This may or may not be
 >related but they say that since you are getting the 140. address (which is
 >routed for people on I2) then there should not be any speed issues. Here,
 >they recommend to ask your upstream provider if they are having routing
 >issues to masterdistfiles or any other limitations. Additionally,
 >recommended to post the output of "mtr masterdistfiles.gentoo.org" as that
 >might provide some better clues.

Martin said that some files went fast and others slowly, so it doesn't
seem to be a dns problem.

Also, this week (on Tuesday), the same problems were seen in the
connection from osu to Europe. They delayed the rsync of the new
release of mozilla, to the point that the release was postponed by
several hours.

Re: [gentoo-mirrors] Incredibly slow rsync from masterdistfiles.gentoo.org

[Martin Eisenhardt]

Hi Jeremy,

On 30.10.2009, at 20:34, Jeremy Olexa wrote:
> Additionally,
> recommended to post the output of "mtr masterdistfiles.gentoo.org"  
> as that
> might provide some better clues.

output from mtr is as follows:

                                                  My traceroute  [v0.75]
node3  
(0.0.0.0 
)                                                                                Sat 
  Oct 31 22:40:26 2009
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                                Packets 
                Pings
   
Host 
                                                                         Loss 
%   Snt   Last   Avg  Best  Wrst StDev
  1. static.1.51.40.188.clients.your- 
server.de                                 0.0%  3313    0.4   0.4    
0.3  16.9   0.4
  2. hos- 
tr3.juniper2.rz10.hetzner.de                                           
0.0%  3313   24.2   0.4   0.1  51.6   6.9
  3. hos- 
bb2.juniper2.s04.hetzner.de                                            
0.0%  3313    2.8   2.8   2.8  66.2   4.9
  4. lambdanet- 
gw.hetzner.de                                                   0.0%   
3312    2.9   2.8   2.7  24.3   1.0
  5. FRA-3- 
eth110.de.lambdanet.net                                              
0.0%  3312    6.1   8.5   6.0 101.9  10.8
  6. FRA-1- 
eth000.de.lambdanet.net                                              
0.0%  3312    6.3   6.2   6.2  33.3   1.5
  7.  
xe 
-5 
-1-0.edge6.Frankfurt1.Level3.net                                       
0.0%  3312   63.8   8.5   6.3 108.5  10.7
  8.  
vlan79 
.csw2.Frankfurt1.Level3.net                                          
0.0%  3312    7.7  10.5   6.4  33.1   4.4
  9.  
ae 
-72 
-72.ebr2.Frankfurt1.Level3.net                                        
0.0%  3312    6.9   6.5   6.4  19.6   0.3
10.  
ae 
-42 
-42.ebr2.Washington1.Level3.net                                       
0.0%  3312   96.1  96.0  95.9 121.8   0.7
11.  
ae 
-2 
-2.ebr2.Chicago2.Level3.net                                            
0.0%  3312  113.8 113.6 113.5 136.2   0.9
12.  
ae 
-1 
-100.ebr1.Chicago2.Level3.net                                          
0.0%  3312  115.0 114.5 114.4 135.3   1.0
13.  
ae 
-3 
.ebr2.Denver1.Level3.net                                               
0.0%  3312  139.4 140.6 137.2 166.3   4.1
14.  
ae 
-2 
.ebr2.Seattle1.Level3.net                                              
0.0%  3312  169.3 168.5 164.1 183.6   4.4
15.  
ae 
-21 
-52.car1.Seattle1.Level3.net                                          
0.0%  3312  163.4 171.8 162.6 430.0  34.0
16. eugnor1wce1-univ-of- 
oregon.wcg.net                                        0.0%  3312   
188.1 187.9 187.5 384.1  11.2
17. corv-car1- 
gw.nero.net                                                     0.0%   
3312  188.0 188.1 187.7 381.7   9.2
18. ftp- 
osl.osuosl.org                                                         
0.0%  3312  188.5 187.9 187.7 213.1   0.7



Speed is still somewhat slow and has not increased noticeably. I will  
continue to check whether there are any problems on my side, though.

Regards
Martin