[gentoo-user] Switched from rsync to git, now emerge --sync fails

[gentoo-user] Switched from rsync to git, now emerge --sync fails

[Grant Edwards]

Yesterday I switched my gentoo repo from rsync to git, and the initial
--sync with an empty directory did a git clone successfully.

Today, when I try to sync, it always fails:

  $ sudo emerge --sync >>> Syncing repository 'gentoo' into '/var/db/repos/gentoo'...
  /usr/bin/git fetch origin
  error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504 fatal: the remote end hung up unexpectedly
  !!! git fetch error in /var/db/repos/gentoo

  Action: sync for repo: gentoo, returned code = 128

The SSL connection to anongit.gentoo.org happens, the handshake goes
OK, a few frames of data go back and forth, then the nothing happens
for 60s and the client end sends a FIN.

After the initial sync which does a git clone, how do you do
subsequent sync operations?

--
Grant

Re: [gentoo-user] Switched from rsync to git, now emerge --sync fails

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 1033 bytes --]

On Wed, 13 Oct 2021 18:25:12 -0000 (UTC), Grant Edwards wrote:

> Yesterday I switched my gentoo repo from rsync to git, and the initial
> --sync with an empty directory did a git clone successfully.
> 
> Today, when I try to sync, it always fails:
> 
>   $ sudo emerge --sync >>> Syncing repository 'gentoo' into
> '/var/db/repos/gentoo'... /usr/bin/git fetch origin
>   error: RPC failed; HTTP 504 curl 22 The requested URL returned error:
> 504 fatal: the remote end hung up unexpectedly !!! git fetch error in
> /var/db/repos/gentoo
> 
>   Action: sync for repo: gentoo, returned code = 128
> 
> The SSL connection to anongit.gentoo.org happens, the handshake goes
> OK, a few frames of data go back and forth, then the nothing happens
> for 60s and the client end sends a FIN.
> 
> After the initial sync which does a git clone, how do you do
> subsequent sync operations?

emerge --sync works here. What do you have in repos.conf?


-- 
Neil Bothwick

There's too much blood in my caffeine system.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[gentoo-user] Re: Switched from rsync to git, now emerge --sync fails

[Grant Edwards]

On 2021-10-13, Neil Bothwick <neil@digimed.co.uk> wrote:
> On Wed, 13 Oct 2021 18:25:12 -0000 (UTC), Grant Edwards wrote:
>
>> Yesterday I switched my gentoo repo from rsync to git, and the initial
>> --sync with an empty directory did a git clone successfully.
>> 
>> Today, when I try to sync, it always fails:
>> 
>>   $ sudo emerge --sync >>> Syncing repository 'gentoo' into
>> '/var/db/repos/gentoo'... /usr/bin/git fetch origin
>>   error: RPC failed; HTTP 504 curl 22 The requested URL returned error:
>> 504 fatal: the remote end hung up unexpectedly
[...]

>> After the initial sync which does a git clone, how do you do
>> subsequent sync operations?
>
> emerge --sync works here. What do you have in repos.conf?

I didn't have a sync-depth setting. Setting that to 1 fixed the
problem. It was apparently timing out because it takes too long for
the server to respond to a fetch with an unlimited depth.  Seems like
maybe it ought to default to 1? Is there some reason it should default
to doing unlimited depth fetch operations?

--
Grant

Re: [gentoo-user] Re: Switched from rsync to git, now emerge --sync fails

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 560 bytes --]

On Wed, 13 Oct 2021 18:50:10 -0000 (UTC), Grant Edwards wrote:

> > emerge --sync works here. What do you have in repos.conf?  
> 
> I didn't have a sync-depth setting. Setting that to 1 fixed the
> problem. It was apparently timing out because it takes too long for
> the server to respond to a fetch with an unlimited depth.  Seems like
> maybe it ought to default to 1? Is there some reason it should default
> to doing unlimited depth fetch operations?

I don't have that either.


-- 
Neil Bothwick

IRQs? We don't need no stinking IRQs!

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Re: Switched from rsync to git, now emerge --sync fails

[Rich Freeman]

On Wed, Oct 13, 2021 at 2:50 PM Grant Edwards <grant.b.edwards@gmail.com> wrote:
>
> Is there some reason it should default
> to doing unlimited depth fetch operations?
>

If all you want is a repo, no reason to set the depth higher.  If you
want to see the history then you'll want it all.

However, once you have an initial sync, I don't think it should go
back and fetch all the history unless you explicitly ask git to do so.
I don't see why this would cause issues after the initial sync.  If
you were fetching all the history, it would be the FIRST sync that
caused all the issues.  Well, unless portage is going out and trying
to pull it all in (and if so I'd think it would have done it from the
start).

Once you have the full repo then subsequent syncs should be very fast
and don't use much CPU server-side.  The git client sends the remote
server its current head, and then the server walks back from its head
until it finds yours, which will only be a short distance if you've
synced recently.  Then it is only the new objects in-between that have
to be sent.  The whole thing is de-duplicated and copy-on-write just
due to its data structure.

I'm suspecting some sort of server-side issue - maybe an intermittent
one.  Either that or portage is really trying to pull in that history
after the initial sync.

Another option is to do a pull from the github mirror.  That same repo
is hosted on both gentoo's server and github, and they're identical
(the content hash tells you as much), so you should be able to do a
pull from either seamlessly.  The signatures/etc are applied to both
as well.  Some don't care for github not being FOSS but if you're just
using it as a mirror I'd argue it is no different than if one of the
gazillion distfile mirrors happened to run IIS or have a firmware that
wasn't coreboot.  It is just another mirror.

-- 
Rich

[gentoo-user] Re: Switched from rsync to git, now emerge --sync fails

[Grant Edwards]

On 2021-10-13, Rich Freeman <rich0@gentoo.org> wrote:
> On Wed, Oct 13, 2021 at 2:50 PM Grant Edwards <grant.b.edwards@gmail.com> wrote:
>>
>> Is there some reason it should default
>> to doing unlimited depth fetch operations?
>>
>
> If all you want is a repo, no reason to set the depth higher.

Then a default of 1 seems like the obvious "right" answer.
Unfortunately, it defaults to "unlimited" according to
https://wiki.gentoo.org/wiki//etc/portage/repos.conf

  sync-depth

    Specifies sync depth to use for DVCS repositories. If set to 0,
    the depth is unlimited. Defaults to 0.

> If you want to see the history then you'll want it all.

Apparently, that's the default. Without any sync-depth setting the
fetch was stalling and then timing out after 60s. With sync-depth=1,
the fetch completed in about 1s.

--
Grant

Re: [gentoo-user] Re: Switched from rsync to git, now emerge --sync fails

[Peter Humphrey]

On Thursday, 14 October 2021 03:50:59 BST Grant Edwards wrote:
> On 2021-10-13, Rich Freeman <rich0@gentoo.org> wrote:
> > On Wed, Oct 13, 2021 at 2:50 PM Grant Edwards 
<grant.b.edwards@gmail.com> wrote:
> >> Is there some reason it should default
> >> to doing unlimited depth fetch operations?
> > 
> > If all you want is a repo, no reason to set the depth higher.
> 
> Then a default of 1 seems like the obvious "right" answer.
> Unfortunately, it defaults to "unlimited" according to
> https://wiki.gentoo.org/wiki//etc/portage/repos.conf
> 
>   sync-depth
> 
>     Specifies sync depth to use for DVCS repositories. If set to 0,
>     the depth is unlimited. Defaults to 0.
> 
> > If you want to see the history then you'll want it all.
> 
> Apparently, that's the default. Without any sync-depth setting the
> fetch was stalling and then timing out after 60s. With sync-depth=1,
> the fetch completed in about 1s.

I have it set to 8 here, for no reason I can remember. Is there a disadvantage 
in that?

-- 
Regards,
Peter.

Re: [gentoo-user] Re: Switched from rsync to git, now emerge --sync fails

[Miles Malone]

Look I mean in any other context for a git repository sync-depth=0
would be the obvious choice, why have a version control system if
you're limiting the version control.  It was added to gentoo so that
developers could work on their own branches of the git repo whilst
still having it actually work with portage.  And then the next logical
application is making your own /usr/local/portage not suck.  Using git
as the main sync instead of rsync is something that's come about way
later, and really is fairly niche, compared to what the git capability
in portage was originally designed for.

So the default being for git to act like git, instead of git to act
like an alternative to rsync, makes perfect sense in that context.

On Thu, 14 Oct 2021 at 18:22, Peter Humphrey <peter@prh.myzen.co.uk> wrote:
>
> On Thursday, 14 October 2021 03:50:59 BST Grant Edwards wrote:
> > On 2021-10-13, Rich Freeman <rich0@gentoo.org> wrote:
> > > On Wed, Oct 13, 2021 at 2:50 PM Grant Edwards
> <grant.b.edwards@gmail.com> wrote:
> > >> Is there some reason it should default
> > >> to doing unlimited depth fetch operations?
> > >
> > > If all you want is a repo, no reason to set the depth higher.
> >
> > Then a default of 1 seems like the obvious "right" answer.
> > Unfortunately, it defaults to "unlimited" according to
> > https://wiki.gentoo.org/wiki//etc/portage/repos.conf
> >
> >   sync-depth
> >
> >     Specifies sync depth to use for DVCS repositories. If set to 0,
> >     the depth is unlimited. Defaults to 0.
> >
> > > If you want to see the history then you'll want it all.
> >
> > Apparently, that's the default. Without any sync-depth setting the
> > fetch was stalling and then timing out after 60s. With sync-depth=1,
> > the fetch completed in about 1s.
>
> I have it set to 8 here, for no reason I can remember. Is there a disadvantage
> in that?
>
> --
> Regards,
> Peter.
>
>
>
>

Re: [gentoo-user] Re: Switched from rsync to git, now emerge --sync fails

[Wol]

On 14/10/2021 09:28, Miles Malone wrote:
> So the default being for git to act like git, instead of git to act
> like an alternative to rsync, makes perfect sense in that context.

And once it's sync'd the first time, I think sync depth is ignored. It 
normally doesn't make much difference anyway, because git stores the 
current version in full, and then just diffs going backwards.

The difference between a sync=1 and sync=0 isn't that much for linux I 
gather - maybe doubles the size?

For a normal user who just wants to USE portage, a sync of 1 makes 
sense. They can then always go back as far as when they first set it up, 
anyway.

Cheers,
Wol