[gentoo-user] Weird problem with emerge sync

[gentoo-user] Weird problem with emerge sync

[Holly Bostick]

I've been having this problem for a couple months now, but today I'm
annoyed enough to ask if there's something I can do about it.

Normally, I run esync as a cron job in the middle of the night (the
standard 3AM), and normally it works fine (though I had a few days where
the error I'm detailing below also occurred, but that seems to have
cleared up). I get the mail and it has information as it should.

It even went fine this afternoon, as my computer was off at 3AM, due to
severe thunderstorms, so the cron job ran at 1:20 PM when I turned it
back on.

However the second cron job was for revdep-rebuild -p, which suggested
that I'd want to rebuild hal and dbus (not expected, but not
unreasonable based on my activities of the last couple days), and when
I tried to do that, I got the dreaded

!!! Security Violation: A file exists that is not in the manifest.

error.

Which was bizarre, but no need to freak out; I know that that such an
error will likely be fixed with a sync. So I tried to esync with my
usual mirror pool, rsync.europe.gentoo.org, but, as
seems to happen if I sync in the afternoon (the previous sync not
withstanding), it failed. The sync started, and progressed quite far,
but then--

(errors from running emerge sync in a terminal, so I could see the
output, having no patience to read /var/log/emerge-sync.log for errors):

io timeout after 90 seconds - exiting
rsync error: timeout in data send/receive (code 30) at io.c(109)

rsync: connection unexpectedly closed (2970632 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(189)

|>>> retry ...

receiving file list ...
124787 files to consider
io timeout after 90 seconds - exiting
rsync error: timeout in data send/receive (code 30) at io.c(109)
rsync: connection unexpectedly closed (3049937 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(189)

All three retries, using mirrors from the Europe pool in Germany,
Finland and The Netherlands failed to complete with these errors.

So, as I always do, I changed my rsync mirror in /etc/make.conf to
rsync.us.gentoo.org, and it synced perfectly on the first try, as it
always does when I encounter this issue (and being
out-of-sync *was*, of course, the problem with revdep-rebuild, which is
now running its oneshots with no complaints).

What is going wrong here? It can't be my implementation of rsync (latest
stable), or the US pool would fail as well. It could be the servers in
the European pool, but... all of them?

Or am I simply using the wrong SYNC= in /etc/make.conf? My default is
rsync.europe.gentoo.org, and I don't see any documentation that
indicates that that has changed or become deprecated or invalid, but
maybe it has. I don't particularly want to switch permanently to the US
pool, as that just seems to make more vectors of instability for
everybody, and is not really the point of having continental mirror
pools anyway.

Anybody got a clue as to what's happening and what, if anything, I can
do to fix it?

Holly
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Weird problem with emerge sync

[Tony Davison]

On Saturday 10 September 2005 15:24, Holly Bostick wrote:
<snip>
>
> Or am I simply using the wrong SYNC= in /etc/make.conf? My default is
> rsync.europe.gentoo.org, and I don't see any documentation that
> indicates that that has changed or become deprecated or invalid, but
> maybe it has. I don't particularly want to switch permanently to the
> US pool, as that just seems to make more vectors of instability for
> everybody, and is not really the point of having continental mirror
> pools anyway.
>
> Anybody got a clue as to what's happening and what, if anything, I
> can do to fix it?

Holly,
I just ran emerge --sync and it went OK using the mirror at Staler.net.
The only difference is that my default is rsync.uk.gentoo.org but I 
don't think the mirror pool is much different with the possible 
exception of Telehouse which doesn't appear much anyway.

Other than suggesting that the force is weak in The Nederlands today 
can't be of much help I'm afraid.
-- 
Tony Davison
tony.davison2@ntlworld.com
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Weird problem with emerge sync

[Paul Varner]

> What is going wrong here? It can't be my implementation of rsync (latest
> stable), or the US pool would fail as well. It could be the servers in
> the European pool, but... all of them?
> 
> Or am I simply using the wrong SYNC= in /etc/make.conf? My default is
> rsync.europe.gentoo.org, and I don't see any documentation that
> indicates that that has changed or become deprecated or invalid, but
> maybe it has. I don't particularly want to switch permanently to the US
> pool, as that just seems to make more vectors of instability for
> everybody, and is not really the point of having continental mirror
> pools anyway.
> 
> Anybody got a clue as to what's happening and what, if anything, I can
> do to fix it?

You didn't state which version of rsync, but the versions below 2.6.4 do
have timeout issues.  See the following bugs:

http://bugs.gentoo.org/show_bug.cgi?id=83254
https://bugzilla.samba.org/show_bug.cgi?id=2783

Regards,
Paul
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Weird problem with emerge sync

[Holly Bostick]

Paul Varner schreef:
>> What is going wrong here? It can't be my implementation of rsync
>> (latest stable), or the US pool would fail as well. It could be the
>> servers in the European pool, but... all of them?
>> 
>> Or am I simply using the wrong SYNC= in /etc/make.conf? My default
>> is rsync.europe.gentoo.org, and I don't see any documentation that 
>> indicates that that has changed or become deprecated or invalid,
>> but maybe it has. I don't particularly want to switch permanently
>> to the US pool, as that just seems to make more vectors of
>> instability for everybody, and is not really the point of having
>> continental mirror pools anyway.
>> 
>> Anybody got a clue as to what's happening and what, if anything, I
>> can do to fix it?
> 
> 
> You didn't state which version of rsync, but the versions below 2.6.4
> do have timeout issues.  See the following bugs:
> 
> http://bugs.gentoo.org/show_bug.cgi?id=83254 
> https://bugzilla.samba.org/show_bug.cgi?id=2783
> 

Thanks for the tip, Paul-- that could definitely be the issue:

net-misc/rsync
     Available versions:  2.6.0-r5 2.6.0-r6 ~2.6.3-r1 ~2.6.4 ~2.6.5 ~2.6.6
     Installed:           2.6.0-r6

For the time being, I've changed my sync mirror pool to the national one
(rather than the continental one), which seems like it's OK (but it's
early days yet, and I haven't tested extensively).

Is it worth going unstable with rsync for this? meaning, will using an
unstable rsync cause me any "real" problems, even as it (hopefully)
solves this (relatively minor) annoyance?

Thanks for the links; I'll look at the bugs later.

Holly
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Weird problem with emerge sync

[Paul Varner]

On Sun, 2005-09-11 at 16:49 +0200, Holly Bostick wrote:
> Is it worth going unstable with rsync for this? meaning, will using an
> unstable rsync cause me any "real" problems, even as it (hopefully)
> solves this (relatively minor) annoyance?

I have not seen any stability issues with 2.6.4 or higher both with
Gentoo and a completely unrelated project at work that uses rsync. The
project at work did have significant issues with the timeout until we
upgraded rsync to 2.6.4.

> Thanks for the links; I'll look at the bugs later.

Quick summary, when the load on the server side is high, rsync will
erroneously timeout.

Regards,
Paul
-- 
gentoo-user@gentoo.org mailing list