* [gentoo-dev] git://anongit.gentoo.org is extremely slow
@ 2015-04-22 14:11 Nikos Chantziaras
2015-04-23 3:07 ` [gentoo-dev] " Duncan
2015-04-23 20:01 ` [gentoo-dev] " Robin H. Johnson
0 siblings, 2 replies; 10+ messages in thread
From: Nikos Chantziaras @ 2015-04-22 14:11 UTC (permalink / raw
To: gentoo-dev
Adding overlays with layman is now extremely slow after the overlays
were moved to anongit.gentoo.org. Cloning anything beginning with
"git://anongit.gentoo.org/" is capped at 15KB/s, sometimes 10KB/s.
This is unworkably slow.
Is this intentional?
^ permalink raw reply [flat|nested] 10+ messages in thread
* [gentoo-dev] Re: git://anongit.gentoo.org is extremely slow
2015-04-22 14:11 [gentoo-dev] git://anongit.gentoo.org is extremely slow Nikos Chantziaras
@ 2015-04-23 3:07 ` Duncan
2015-04-23 3:20 ` Kent Fredric
2015-04-23 20:01 ` [gentoo-dev] " Robin H. Johnson
1 sibling, 1 reply; 10+ messages in thread
From: Duncan @ 2015-04-23 3:07 UTC (permalink / raw
To: gentoo-dev
Nikos Chantziaras posted on Wed, 22 Apr 2015 17:11:13 +0300 as excerpted:
> Adding overlays with layman is now extremely slow after the overlays
> were moved to anongit.gentoo.org. Cloning anything beginning with
> "git://anongit.gentoo.org/" is capped at 15KB/s, sometimes 10KB/s.
>
> This is unworkably slow.
>
> Is this intentional?
Perhaps the slashdot effect of everybody almost at once needing to delete
and (re)add their overlays in layman, thus forcing a new clone, in
ordered to kill the sync errors due to the previously configured host
disappearing?
I had that happen a couple days ago and once I figured out that the old
host shutdowns was why I was getting the errors, the delete/re-add
process went reasonably smoothly, but that might have been before the big
rush.
Or maybe it's simply suffering an attack DDoS? Infra?
FWIW, once cloned, updates seem to be reasonably normal/fast, at least
for the two overlays (kde and x11) I have subscribed. Maybe they're
limiting cloning speed due to the above, figuring people will get thru it
eventually, to keep normal updates reasonable?
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [gentoo-dev] Re: git://anongit.gentoo.org is extremely slow
2015-04-23 3:07 ` [gentoo-dev] " Duncan
@ 2015-04-23 3:20 ` Kent Fredric
2015-04-23 3:38 ` Brian Dolbec
0 siblings, 1 reply; 10+ messages in thread
From: Kent Fredric @ 2015-04-23 3:20 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 679 bytes --]
On 23 April 2015 at 15:07, Duncan <1i5t5.duncan@cox.net> wrote:
> Perhaps the slashdot effect of everybody almost at once needing to delete
> and (re)add their overlays in layman, thus forcing a new clone,
>
Eh? That seems incredibly wrong for git.
git remote set-url master <newurl>
Would handle the git side.
You can even tweak git to rewrite all urls for you, like:
[url "git://anongit.gentoo.org"]
insteadOf = git://git.overlays.gentoo.org
Admittedly it may have been hard for layman to be intelligent about this,
but requiring everyone to remove/add simply for a URI change strikes me as
rather odd here.
--
Kent
*KENTNL* - https://metacpan.org/author/KENTNL
[-- Attachment #2: Type: text/html, Size: 1716 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [gentoo-dev] Re: git://anongit.gentoo.org is extremely slow
2015-04-23 3:20 ` Kent Fredric
@ 2015-04-23 3:38 ` Brian Dolbec
2015-04-23 9:02 ` Duncan
0 siblings, 1 reply; 10+ messages in thread
From: Brian Dolbec @ 2015-04-23 3:38 UTC (permalink / raw
To: gentoo-dev
On Thu, 23 Apr 2015 15:20:15 +1200
Kent Fredric <kentfredric@gmail.com> wrote:
> On 23 April 2015 at 15:07, Duncan <1i5t5.duncan@cox.net> wrote:
>
> > Perhaps the slashdot effect of everybody almost at once needing to
> > delete and (re)add their overlays in layman, thus forcing a new
> > clone,
> >
>
> Eh? That seems incredibly wrong for git.
>
> git remote set-url master <newurl>
>
> Would handle the git side.
>
> You can even tweak git to rewrite all urls for you, like:
>
> [url "git://anongit.gentoo.org"]
> insteadOf = git://git.overlays.gentoo.org
>
> Admittedly it may have been hard for layman to be intelligent about
> this, but requiring everyone to remove/add simply for a URI change
> strikes me as rather odd here.
>
>
No, layman does try to update the url first, failing that it
deletes/re-adds it.
But I think what happened for some, was, the urls had not yet been
updated or updated correctly in repositories.xml. That resulted in
failures to update to a new valid url.
--
Brian Dolbec <dolsen>
^ permalink raw reply [flat|nested] 10+ messages in thread
* [gentoo-dev] Re: git://anongit.gentoo.org is extremely slow
2015-04-23 3:38 ` Brian Dolbec
@ 2015-04-23 9:02 ` Duncan
2015-04-25 5:32 ` Kent Fredric
0 siblings, 1 reply; 10+ messages in thread
From: Duncan @ 2015-04-23 9:02 UTC (permalink / raw
To: gentoo-dev
Brian Dolbec posted on Wed, 22 Apr 2015 20:38:15 -0700 as excerpted:
> On Thu, 23 Apr 2015 15:20:15 +1200 Kent Fredric <kentfredric@gmail.com>
> wrote:
>
>> On 23 April 2015 at 15:07, Duncan <1i5t5.duncan@cox.net> wrote:
>>
>> > Perhaps the slashdot effect of everybody almost at once needing to
>> > delete and (re)add their overlays in layman, thus forcing a new
>> > clone,
>> >
>> >
>> Eh? That seems incredibly wrong for git.
>>
>> git remote set-url master <newurl>
>>
>> Would handle the git side.
>>
>> You can even tweak git to rewrite all urls for you, like:
>>
>> [url "git://anongit.gentoo.org"]
>> insteadOf = git://git.overlays.gentoo.org
>>
>> Admittedly it may have been hard for layman to be intelligent about
>> this, but requiring everyone to remove/add simply for a URI change
>> strikes me as rather odd here.
>
> No, layman does try to update the url first, failing that it
> deletes/re-adds it.
>
> But I think what happened for some, was, the urls had not yet been
> updated or updated correctly in repositories.xml. That resulted in
> failures to update to a new valid url.
What I know is I was getting a sync-error (IIRC 1) for the overlays, and
that deleting/re-adding the overlays via layman, fixed it.
If repositories.xml was not yet updated, then how would deleting/re-
adding have worked?
The overlays were the x11 and kde overlays, both git, both set in
repos.conf to sync-type = laymansync, with the syncing done via script
that calls emaint sync --repo (so syncs can be backgrounded and done in
parallel).
Meanwhile, from the bug reports it looks like this switch did trigger
reports for a number of latent bugs in layman, with patches already in
the live-git version, and I believe lessons were learned about
coordinating repo updates when a major host changes as well, so
hopefully, if there's a next time it'll go much smoother than this one
did. =:^)
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [gentoo-dev] git://anongit.gentoo.org is extremely slow
2015-04-22 14:11 [gentoo-dev] git://anongit.gentoo.org is extremely slow Nikos Chantziaras
2015-04-23 3:07 ` [gentoo-dev] " Duncan
@ 2015-04-23 20:01 ` Robin H. Johnson
2015-04-24 14:38 ` [gentoo-dev] " Nikos Chantziaras
1 sibling, 1 reply; 10+ messages in thread
From: Robin H. Johnson @ 2015-04-23 20:01 UTC (permalink / raw
To: gentoo-dev
On Wed, Apr 22, 2015 at 05:11:13PM +0300, Nikos Chantziaras wrote:
> Adding overlays with layman is now extremely slow after the overlays
> were moved to anongit.gentoo.org. Cloning anything beginning with
> "git://anongit.gentoo.org/" is capped at 15KB/s, sometimes 10KB/s.
>
> This is unworkably slow.
>
> Is this intentional?
Where are you located?
I'm wondering if you're suffering connectivity impacts of the location
change.
Both boxes have enough hardware:
anongit (manakin): Xeon E5-2620, 32GB RAM, 512GB SSD (RAID1), US-East
git (oystercatcher): Xeon E5-1650, 64GB RAM, 240GB SSD (RAID1), Germany
We'll be bringing a EU location for anongit online as well soon, mostly
as the US-East doesn't have IPv6, and we can't put a tunnel into there.
--
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
^ permalink raw reply [flat|nested] 10+ messages in thread
* [gentoo-dev] Re: git://anongit.gentoo.org is extremely slow
2015-04-23 20:01 ` [gentoo-dev] " Robin H. Johnson
@ 2015-04-24 14:38 ` Nikos Chantziaras
2015-04-24 15:04 ` Vadim A. Misbakh-Soloviov
0 siblings, 1 reply; 10+ messages in thread
From: Nikos Chantziaras @ 2015-04-24 14:38 UTC (permalink / raw
To: gentoo-dev
On 23/04/15 23:01, Robin H. Johnson wrote:
> On Wed, Apr 22, 2015 at 05:11:13PM +0300, Nikos Chantziaras wrote:
>> Adding overlays with layman is now extremely slow after the overlays
>> were moved to anongit.gentoo.org. Cloning anything beginning with
>> "git://anongit.gentoo.org/" is capped at 15KB/s, sometimes 10KB/s.
>>
>> This is unworkably slow.
>>
>> Is this intentional?
> Where are you located?
>
> I'm wondering if you're suffering connectivity impacts of the location
> change.
>
> Both boxes have enough hardware:
> anongit (manakin): Xeon E5-2620, 32GB RAM, 512GB SSD (RAID1), US-East
> git (oystercatcher): Xeon E5-1650, 64GB RAM, 240GB SSD (RAID1), Germany
>
> We'll be bringing a EU location for anongit online as well soon, mostly
> as the US-East doesn't have IPv6, and we can't put a tunnel into there.
The problem seems to have been fixed now.
It was not a location problem, because only git:// seemed to be affected:
git clone git://... <- 15KB/s
git clone https://... <- 2MB/s
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [gentoo-dev] Re: git://anongit.gentoo.org is extremely slow
2015-04-24 14:38 ` [gentoo-dev] " Nikos Chantziaras
@ 2015-04-24 15:04 ` Vadim A. Misbakh-Soloviov
0 siblings, 0 replies; 10+ messages in thread
From: Vadim A. Misbakh-Soloviov @ 2015-04-24 15:04 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 395 bytes --]
> It was not a location problem, because only git:// seemed to be affected:
>
> git clone git://... <- 15KB/s
> git clone https://... <- 2MB/s
Looks like shaping on ISP side (i.e. it doesn't hear about Network
Neutrality).
// Although, of course, it can be result server overloading and only infra-
team can investigate and tell real reason ;)
--
Best regards,
mva
[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [gentoo-dev] Re: git://anongit.gentoo.org is extremely slow
2015-04-23 9:02 ` Duncan
@ 2015-04-25 5:32 ` Kent Fredric
2015-04-25 13:58 ` Mike Gilbert
0 siblings, 1 reply; 10+ messages in thread
From: Kent Fredric @ 2015-04-25 5:32 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 656 bytes --]
On 23 April 2015 at 21:02, Duncan <1i5t5.duncan@cox.net> wrote:
> with patches already in
> the live-git version, and I believe lessons were learned about
> coordinating repo updates when a major host changes as well, so
> hopefully, if there's a next time it'll go much smoother than this one
> did. =:^)
>
Fixed in the live git version they said.
>>> Emerging (1 of 1) app-portage/layman-9999::gentoo
>>> Unpacking source...
Cloning into bare repository '/usr/portage/distfiles/egit-src/layman.git'...
fatal: Unable to look up git.overlays.gentoo.org (port 9418) (Name or
service not known)
--
Kent
*KENTNL* - https://metacpan.org/author/KENTNL
[-- Attachment #2: Type: text/html, Size: 1488 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [gentoo-dev] Re: git://anongit.gentoo.org is extremely slow
2015-04-25 5:32 ` Kent Fredric
@ 2015-04-25 13:58 ` Mike Gilbert
0 siblings, 0 replies; 10+ messages in thread
From: Mike Gilbert @ 2015-04-25 13:58 UTC (permalink / raw
To: Gentoo Dev
On Sat, Apr 25, 2015 at 1:32 AM, Kent Fredric <kentfredric@gmail.com> wrote:
>
> On 23 April 2015 at 21:02, Duncan <1i5t5.duncan@cox.net> wrote:
>>
>> with patches already in
>> the live-git version, and I believe lessons were learned about
>> coordinating repo updates when a major host changes as well, so
>> hopefully, if there's a next time it'll go much smoother than this one
>> did. =:^)
>
>
> Fixed in the live git version they said.
>
>
>>>> Emerging (1 of 1) app-portage/layman-9999::gentoo
>>>> Unpacking source...
> Cloning into bare repository '/usr/portage/distfiles/egit-src/layman.git'...
> fatal: Unable to look up git.overlays.gentoo.org (port 9418) (Name or
> service not known)
Fixed in CVS. ^_^
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2015-04-25 13:58 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-04-22 14:11 [gentoo-dev] git://anongit.gentoo.org is extremely slow Nikos Chantziaras
2015-04-23 3:07 ` [gentoo-dev] " Duncan
2015-04-23 3:20 ` Kent Fredric
2015-04-23 3:38 ` Brian Dolbec
2015-04-23 9:02 ` Duncan
2015-04-25 5:32 ` Kent Fredric
2015-04-25 13:58 ` Mike Gilbert
2015-04-23 20:01 ` [gentoo-dev] " Robin H. Johnson
2015-04-24 14:38 ` [gentoo-dev] " Nikos Chantziaras
2015-04-24 15:04 ` Vadim A. Misbakh-Soloviov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox