[gentoo-dev] Excessive rsync time after git migration

[gentoo-dev] Excessive rsync time after git migration

[Joshua Kinard]

This might already be covered in one of the other e-mail threads, but I've been
super-busy as of late and just recently ran 'emerge --sync' on my main dev box
for the first time after the git migration.  I just synced my main dev box
again, ~10 hours after the last sync, but it looks like the 'Manifest' files
for *every* package in the tree are getting downloaded with each sync.

I compared two Manifest files from a given package that shouldn't have changed
in the last 10 hours (sys-kernel/mips-sources), and there is nothing different
timestamp wise or content wise between then.  So I'm not sure what's causing
'emerge --sync' to re-fetch the whole file.

Has anyone else noticed this and/or have a workaround?  It's causing some
excessive disk thrashing, and what used to be a few seconds to sync is now
taking 5-10mins or more (depending on which machine I'm syncing).

-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
6144R/F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943

"The past tempts us, the present confuses us, the future frightens us.  And our
lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

Re: [gentoo-dev] Excessive rsync time after git migration

[Dale]

Joshua Kinard wrote:
> This might already be covered in one of the other e-mail threads, but I've been
> super-busy as of late and just recently ran 'emerge --sync' on my main dev box
> for the first time after the git migration.  I just synced my main dev box
> again, ~10 hours after the last sync, but it looks like the 'Manifest' files
> for *every* package in the tree are getting downloaded with each sync.
>
> I compared two Manifest files from a given package that shouldn't have changed
> in the last 10 hours (sys-kernel/mips-sources), and there is nothing different
> timestamp wise or content wise between then.  So I'm not sure what's causing
> 'emerge --sync' to re-fetch the whole file.
>
> Has anyone else noticed this and/or have a workaround?  It's causing some
> excessive disk thrashing, and what used to be a few seconds to sync is now
> taking 5-10mins or more (depending on which machine I'm syncing).
>


I have synced a few times since the change and each time, it takes a lot
longer to sync.  You are not the only one that has noticed this. 

No clue as to why or how to fix tho.

Dale

:-)  :-)

[gentoo-dev] Re: Excessive rsync time after git migration

[Holger Hoffstätte]

On Sat, 15 Aug 2015 15:29:21 -0400, Joshua Kinard wrote:

> This might already be covered in one of the other e-mail threads, but I've been
> super-busy as of late and just recently ran 'emerge --sync' on my main dev box
> for the first time after the git migration.  I just synced my main dev box
> again, ~10 hours after the last sync, but it looks like the 'Manifest' files
> for *every* package in the tree are getting downloaded with each sync.

https://bugs.gentoo.org/show_bug.cgi?id=557192

-h

Re: [gentoo-dev] Re: Excessive rsync time after git migration

[Philip Webb]

150815 Holger Hoffstätte wrote:
> On Sat, 15 Aug 2015 15:29:21 -0400, Joshua Kinard wrote:
>> I've been super-busy as of late and just recently ran 'emerge --sync'
>> on my main dev box for the first time after the git migration.
>> I just synced my main dev box again, ~10 hours after the last sync,
>> but it looks like the 'Manifest' files for *every* package in the tree
>> are getting downloaded with each sync.
> https://bugs.gentoo.org/show_bug.cgi?id=557192

I've just done my weekly 'eix-sync', which took  c 2 min :
it seems smoother than in CVS days & lack the lengthy 'metadata' list.
The problem seems to have been solved : thanks to whoever fixed it.

-- 
========================,,============================================
SUPPORT     ___________//___,   Philip Webb
ELECTRIC   /] [] [] [] [] []|   Cities Centre, University of Toronto
TRANSIT    `-O----------O---'   purslowatchassdotutorontodotca

Re: [gentoo-dev] Re: Excessive rsync time after git migration

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 947 bytes --]

Dnia 2015-08-22, o godz. 22:17:05
Philip Webb <purslow@ca.inter.net> napisał(a):

> 150815 Holger Hoffstätte wrote:
> > On Sat, 15 Aug 2015 15:29:21 -0400, Joshua Kinard wrote:
> >> I've been super-busy as of late and just recently ran 'emerge --sync'
> >> on my main dev box for the first time after the git migration.
> >> I just synced my main dev box again, ~10 hours after the last sync,
> >> but it looks like the 'Manifest' files for *every* package in the tree
> >> are getting downloaded with each sync.
> > https://bugs.gentoo.org/show_bug.cgi?id=557192
> 
> I've just done my weekly 'eix-sync', which took  c 2 min :
> it seems smoother than in CVS days & lack the lengthy 'metadata' list.
> The problem seems to have been solved : thanks to whoever fixed it.

No, it's not. And '2 minutes' is actually 'horribly long'. But I'll try
to fix it.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Re: Excessive rsync time after git migration

[Gregory Woodbury]

I am still getting the Manifest files completely reloaded when I sync
portage. (emerge --sync)

-- 
G.Wolfe Woodbury
redwolfe@gmail.com

Re: [gentoo-dev] Re: Excessive rsync time after git migration

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 444 bytes --]

Dnia 2015-08-23, o godz. 15:48:48
Gregory Woodbury <redwolfe@gmail.com> napisał(a):

> I am still getting the Manifest files completely reloaded when I sync
> portage. (emerge --sync)

What was the last time you synced? I applied a fix around 8:30 UTC.
Afterwards all Manifests were regenerated, so you should have one full
fetch and normal behavior afterwards.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] Re: Excessive rsync time after git migration

[Zac Medico]

On 08/23/2015 12:48 PM, Gregory Woodbury wrote:
> I am still getting the Manifest files completely reloaded when I sync
> portage. (emerge --sync)
> 

If it's truly fixed, you'll still have to sync all the Manifests at
least once before you'll have the new Manifests with stable mtimes [1].

[1] https://bugs.gentoo.org/show_bug.cgi?id=557192#c26
-- 
Thanks,
Zac

Re: [gentoo-dev] Re: Excessive rsync time after git migration

[Gregory Woodbury]

I'll sync soon and see.  Thanks.

-- 
G.Wolfe Woodbury
redwolfe@gmail.com