[gentoo-council] DVCS update

[gentoo-council] DVCS update

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 1648 bytes --]

Hi all,

Here's a quick update on the git-migration experiment. We've pretty much 
settled on doing one large repository for the entire tree for a couple 
of reasons:

- It's realistic to complete in the short term: It doesn't require a 
level of rearchitecture that will delay it forever. The alternative of 
one repo per package requires changes to many of our tools, and 
enhancements to git of varying scopes.

- Many of the biggest problems with it have been solved. For example, 
this week I found a tool that should retain history when merging a 
package in an overlay to the main tree, even if there is no common 
ancestry.

- The remaining problems are not blockers. For example, there is no way 
to check out just part of a tree. Disk space isn't so hard to find now 
that this is a requirement.

- Robin has commented that he doesn't know what kind of server-side 
resources are required. I've talked to a number of git admins 
(kernel.org, fedora, freedesktop.org, gnome), and they have all said the 
hardware requirements for git are negligible. They don't have any 
resource on their servers that's being used up. It's gitweb that is 
somewhat resource-intensive.


The next thing I can think of that needs to happen is putting together a 
concrete plan for how a migration would work and making sure all 
necessary tools support it, on all sides: developer, user, and 
infrastructure. We've got portage support now, but I don't have any idea 
what needs to change on the backend infra servers.

-- 
Thanks,
Donnie

Donnie Berkholz
Developer, Gentoo Linux
Blog: http://dberkholz.wordpress.com

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-council] DVCS update

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1184 bytes --]

On Tue, Feb 10, 2009 at 01:51:34PM -0800, Donnie Berkholz wrote:
> - Robin has commented that he doesn't know what kind of server-side 
> resources are required. I've talked to a number of git admins 
> (kernel.org, fedora, freedesktop.org, gnome), and they have all said the 
> hardware requirements for git are negligible. They don't have any 
> resource on their servers that's being used up. It's gitweb that is 
> somewhat resource-intensive.
A specific comment on this (I know I've been away lately, I'm busy
getting back to things now that I've moved house). Git devours RAM.
During pull actions, if the pull triggers a build of a custom pack on
the server, it can snowball to kill the box with IO load due to swap and
later OOM.

The git-upload-pack that runs on the server side when you do a 'git
clone ...' at the moment for the gentoo-x86.git repo exceeds 1GB of RAM
after 10 minutes of running. The box only has 2GB of RAM, so it hits
swap quickly, and things go downhill very fast from there.

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 329 bytes --]

Re: [gentoo-council] DVCS update

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 1593 bytes --]

On 14:08 Tue 10 Feb     , Robin H. Johnson wrote:
> On Tue, Feb 10, 2009 at 01:51:34PM -0800, Donnie Berkholz wrote:
> > - Robin has commented that he doesn't know what kind of server-side 
> > resources are required. I've talked to a number of git admins 
> > (kernel.org, fedora, freedesktop.org, gnome), and they have all said the 
> > hardware requirements for git are negligible. They don't have any 
> > resource on their servers that's being used up. It's gitweb that is 
> > somewhat resource-intensive.
>
> A specific comment on this (I know I've been away lately, I'm busy
> getting back to things now that I've moved house). Git devours RAM.
> During pull actions, if the pull triggers a build of a custom pack on
> the server, it can snowball to kill the box with IO load due to swap and
> later OOM.
> 
> The git-upload-pack that runs on the server side when you do a 'git
> clone ...' at the moment for the gentoo-x86.git repo exceeds 1GB of RAM
> after 10 minutes of running. The box only has 2GB of RAM, so it hits
> swap quickly, and things go downhill very fast from there.

Thanks, Robin! To be very specific about one of the above, the Gnome box 
is a dual-CPU with 8G RAM on some kind of RAID. Owen Taylor (otaylor on 
IRC) informed me that it handled git with no problem at insignicant 
load.

OTOH, I don't know of anyone with a repo more than 1/2 the size of ours 
and maybe 1/4 the object count. My local clone of the kernel is ~600M.

-- 
Thanks,
Donnie

Donnie Berkholz
Developer, Gentoo Linux
Blog: http://dberkholz.wordpress.com

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-council] DVCS update

[Luca Barbato]

Donnie Berkholz wrote:
> - Robin has commented that he doesn't know what kind of server-side 
> resources are required. I've talked to a number of git admins 
> (kernel.org, fedora, freedesktop.org, gnome), and they have all said the 
> hardware requirements for git are negligible. They don't have any 
> resource on their servers that's being used up. It's gitweb that is 
> somewhat resource-intensive.

cgit could be a valid alternative.


lu

-- 

Luca Barbato
Gentoo Council Member
Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

Re: [gentoo-council] DVCS update

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 636 bytes --]

On 14:30 Wed 11 Feb     , Luca Barbato wrote:
> Donnie Berkholz wrote:
>> - Robin has commented that he doesn't know what kind of server-side  
>> resources are required. I've talked to a number of git admins  
>> (kernel.org, fedora, freedesktop.org, gnome), and they have all said 
>> the hardware requirements for git are negligible. They don't have any  
>> resource on their servers that's being used up. It's gitweb that is  
>> somewhat resource-intensive.
>
> cgit could be a valid alternative.

Yes, I agree.

-- 
Thanks,
Donnie

Donnie Berkholz
Developer, Gentoo Linux
Blog: http://dberkholz.wordpress.com

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]