[gentoo-dev] [RFC] Some sync control

[gentoo-dev] [RFC] Some sync control

[Piotr Jaroszyński]

Hello,

After yesterday epkgmove fun I thought that it would be nice to have some 
control on when our cvstree is synced with mirrors. My first very basic idea 
is just to put a block_sync file in the tree when smth big is going on, like 
new kde version stabilization or big pkg move. Inside the file we could have 
smth like "user - why - until time&date".

What do you think?

-- 
Best Regards,
Piotr Jaroszyński

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] [RFC] Some sync control

[Simon Stelling]

Piotr Jaroszyński wrote:
> What do you think?

I think it would be much nicer to have a VCS with support for atomic 
commits.

-- 
Kind Regards,

Simon Stelling
Gentoo/AMD64 Developer
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] [RFC] Some sync control

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 2495 bytes --]

On Thu, Jan 11, 2007 at 10:29:58PM +0100, Piotr Jaroszy??ski wrote:
> After yesterday epkgmove fun I thought that it would be nice to have some 
> control on when our cvstree is synced with mirrors. My first very basic idea 
> is just to put a block_sync file in the tree when smth big is going on, like 
> new kde version stabilization or big pkg move. Inside the file we could have 
> smth like "user - why - until time&date".
If you'd asked the infra folks, in the past we have just blocked it when
people asked us to.

However I do really like the idea of moving the control towards
developers, and I'll take the first step of strongly defining the format
of the file.

0. File location is located at profiles/block_sync
1. Lines starting with a # are comments.
2. Each entry must be separated by one or more lines of whitespace.
3. Each entry based on RFC[2]822 format, with continuation lines if
   needed. This makes parsing very easy. (See people complaining about
   trying to parse comments in package.mask lately).
4. The following headers must be included: Entry-Date, Start-Date,
   Signed-Off-By, Reason.
5. All dates must be in UTC and conform to RFC2822. 
   (Use 'date -R -u' output)
6. Start-Date specifies when the block starts. Entry-Date is when the
   entry was created in the file.
7. Signed-Off-By must be the contact person for the block. More than one
   Signed-Off-By line may be present.
8. The following headers are optional: Completion-Date, References
9. References should include links or message-ID numbers to publicly
   visible reasons as to why the block is in place.
10. A Completion-Date may be present, but is a guideline only. No
    automatic action shall be taken based on this line.
11. Other headers may exist, and should not interfere with the file.

Example entry:
==============
Entry-Date: Thu, 11 Jan 2007 21:45:23 +0000
Start-Date: Thu, 11 Jan 2007 22:00:00 +0000
Completion-Date: Thu, 11 Jan 2007 23:00:00 +0000
Signed-off-By: Robin H. Johnson <robbat2@gentoo.org>
References: http://article.gmane.org/gmane.linux.gentoo.devel/45310
Reason: The following is an example of a block for the block_sync file.
 This is a continuation line. Notice the leading space. It is still part
 of the Reason header.
# The above is the example block
==============

-- 
Robin Hugh Johnson
Gentoo Linux Developer
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 240 bytes --]

Re: [gentoo-dev] [RFC] Some sync control

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 834 bytes --]

On Thu, Jan 11, 2007 at 11:49:43PM +0200, Simon Stelling wrote:
> Piotr Jaroszy??ski wrote:
> >What do you think?
> I think it would be much nicer to have a VCS with support for atomic 
> commits.
I agre, from multiple points of view including those of 1. developer
who has broken stuff with epkgmove, 2. developer who has broken things
moving by hand 3. CVS administrator, 4. general infra. 

However, for several reasons this is not yet feasible, and furthermore
purely atomic commits will not negate the need for the block_sync
functionality. Tree breakage has been noticed in the past, and infra has
been asked to block the rsync process until the breakage was fixed.

-- 
Robin Hugh Johnson
Gentoo Linux Developer
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 240 bytes --]

Re: [gentoo-dev] [RFC] Some sync control

[Seemant Kulleen]

On Thu, 2007-01-11 at 14:08 -0800, Robin H. Johnson wrote:

> However, for several reasons this is not yet feasible, and furthermore

Just for the sake of completeness can you outline those reasons?

Thanks,

-- 
Seemant Kulleen
Developer, Gentoo Linux

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] [RFC] Some sync control

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1984 bytes --]

On Thu, Jan 11, 2007 at 05:51:43PM -0500, Seemant Kulleen wrote:
> On Thu, 2007-01-11 at 14:08 -0800, Robin H. Johnson wrote:
> > However, for several reasons this is not yet feasible, and furthermore
> Just for the sake of completeness can you outline those reasons?
I'm not saying it won't happen ever, just not yet.

1. The work-in-progress for planning out the migration to the new
CVS/SVN box at GNi. The new hardware will be needed regardless of which
VCS we will need.

2. See the results (and as-yet unpublished GLEP) of Antarus's Summer of
Code research into VCS migrations. I'll include his summary verbatim
here:
   "If Gentoo is to switch now I could only recommend SVN.  GIT needs
   just those few extra features to be a viable canidate.  I think if
   there are volunteers to make GIT work for Gentoo than that would be
   best.  GIT is a better all around tool in terms of technical merits
   for most development tasks."
The upstream GIT developers have started work on some of these changes,
as they need them (see the recent scaling problems with gitweb on
kernel.org). GIT has improved in the 6 months since Antarus's SoC work
was done.

3. In regards to any possible migration, the point of infra is that it
is not acceptable to migration to $X now, and then $Y 6 months down the
road. Reasons behind this are concerns over loss of data in repeated
migrations, and having to revisit various scripts and tools more than
otherwise necessary.

My personal view (not infra) on it, is that I'm mostly negative about
changing VCS at all - I would prefer not to change, because the status
quo works very well as it is. If a change is going to be made, it should
be taken as a chance to resolve as many different issues at one time as
possible, and for that reason I favour GIT over SVN.

-- 
Robin Hugh Johnson
Gentoo Linux Developer
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 240 bytes --]

Re: [gentoo-dev] [RFC] Some sync control

[Alec Warner]

Seemant Kulleen wrote:
> On Thu, 2007-01-11 at 14:08 -0800, Robin H. Johnson wrote:
> 
>> However, for several reasons this is not yet feasible, and furthermore
> 
> Just for the sake of completeness can you outline those reasons?
> 
> Thanks,
> 

The status quo is a harsh mistress.  'There is no compelling reason to
switch systems'.  If we were starting anew?  SVN in a heartbeat.

But we have 6 years of cvs experience, and it works...fairly well as you
know.  The primary reasons for svn are better local support for stuff
(read diffs), atomic commits, and file moves.  Branching..I don't think
is really useful for gentoo-x86 as it doesn't fit the style of how we do
things.

GIT is a good alternative, but has massive changes of it's own,
particularly I think in workflow.  Workflow is important, and it's
beneficial to make the workflow changes at the same time as the backend
is changed.

-Alec Warner
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] [RFC] Some sync control

[Seemant Kulleen]

> GIT is a good alternative, but has massive changes of it's own,
> particularly I think in workflow.  Workflow is important, and it's
> beneficial to make the workflow changes at the same time as the backend
> is changed.


Alec,

Can you speak to some of these workflow changes?

I only have experience with cvs and svn, not git or bzr or any of the
other stuff that's out there, so I honestly do not know what their
models and workflows are.  So from my own uneducated stance, svn
wouldn't be a workflow issue at all for most things, no?

Thanks,
-- 
Seemant Kulleen
Developer, Gentoo Linux

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] [RFC] Some sync control

[Luca Barbato]

Seemant Kulleen wrote:
>> GIT is a good alternative, but has massive changes of it's own,
>> particularly I think in workflow.  Workflow is important, and it's
>> beneficial to make the workflow changes at the same time as the backend
>> is changed.
> 
> 
> Alec,
> 
> Can you speak to some of these workflow changes?
> 

In cvs and svn you have people working on their local copy and then 
commit things on the remote server.

In distributed systems everybody has his local history (full?) and you 
can push or pull for every repo (that are peers in this case) as you 
wish, so you could implement different workflows since you have many 
more degrees of liberty.

See http://www.kernel.org/pub/software/scm/git/docs/tutorial.html and 
the other docs to have some examples of workflows.

lu

-- 

Luca Barbato

Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] [RFC] Some sync control

[Petteri Räty]

[-- Attachment #1: Type: text/plain, Size: 531 bytes --]

Piotr Jaroszyński kirjoitti:
> Hello,
> 
> After yesterday epkgmove fun I thought that it would be nice to have some 
> control on when our cvstree is synced with mirrors. My first very basic idea 
> is just to put a block_sync file in the tree when smth big is going on, like 
> new kde version stabilization or big pkg move. Inside the file we could have 
> smth like "user - why - until time&date".
> 
> What do you think?
> 

Been wanting / talking about something like this for a while :)

Regards,
Petteri


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

[gentoo-dev] Re: [RFC] Some sync control

[Steve Long]

Robin H. Johnson wrote:
> My personal view (not infra) on it, is that I'm mostly negative about
> changing VCS at all - I would prefer not to change, because the status
> quo works very well as it is. If a change is going to be made, it should
> be taken as a chance to resolve as many different issues at one time as
> possible, and for that reason I favour GIT over SVN.
> 
<noob alert> I'm looking for a distributed SCM atm, and have come down to
git, bzr, svn or arch. (darcs looks nice but adds haskell dependency.) I'd
really like to know which one gentoo-devs prefer and why (without starting
a flame- if this is OT then np.) I'm leaning to git simply because it's
used for the kernel, which seems like a project that would really stretch a
VCS.

-- 
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: [RFC] Some sync control

[Markus Ullmann]

Steve Long schrieb:
> <noob alert> I'm looking for a distributed SCM atm, and have come down to
> git, bzr, svn or arch.

svn is centralized ;)

> I'm leaning to git simply because it's used for the kernel, which seems
 > like a project that would really stretch a VCS.

Well the kernel is quite large but doesn't use that many different 
things, so it heavily depends on what you do.

 > Robin H. Johnson wrote:
 > My personal view (not infra) on it, is that I'm mostly negative about
 > changing VCS at all - I would prefer not to change, because the status
 > quo works very well as it is.

Well it works, no question on that. But there's still room for 
enhancements ;)

 > If a change is going to be made, it should be taken as a chance to
 > resolve as many different issues at one time as possible, and for
 > that reason I favour GIT over SVN.

I've talked to a friend of mine recently. He's a FreeBSD dev and he said 
they tried git for their ports tree (which is basically the same what 
we're talking about) and it was more or less a big pain for multiple 
reasons.
He said he'd personally take svn after that experience.

Jokey

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: [RFC] Some sync control

[Greg KH]

On Wed, Jan 17, 2007 at 01:53:12AM +0100, Markus Ullmann wrote:
> I've talked to a friend of mine recently. He's a FreeBSD dev and he said 
> they tried git for their ports tree (which is basically the same what 
> we're talking about) and it was more or less a big pain for multiple 
> reasons.
> He said he'd personally take svn after that experience.

What was the reasons he cited?

thanks,

greg k-h
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: [RFC] Some sync control

[Donnie Berkholz]

Greg KH wrote:
> On Wed, Jan 17, 2007 at 01:53:12AM +0100, Markus Ullmann wrote:
>> I've talked to a friend of mine recently. He's a FreeBSD dev and he said 
>> they tried git for their ports tree (which is basically the same what 
>> we're talking about) and it was more or less a big pain for multiple 
>> reasons.
>> He said he'd personally take svn after that experience.
> 
> What was the reasons he cited?

Given that ports is pretty similar to our gentoo-x86, I'd guess about 
the same ones mentioned at 
http://dev.gentoo.org/~antarus/projects/gleps/glep-0666.txt -- I quote 
from there:

I think migration for many would be frustrating and detailed guides
for doing things in GIT would be the norm for quite some time.  GIT
also has some other issues:

1. Git currently requires you to check out the whole repository.
    This includes *all of the history*.

2. Git cannot update portions of the repository, it can only update
    the entire thing.

3. Due to git's choice of packing format (which does save a lot of
    space), the operations are quite CPU intensive.  Either the GIT
    server gets overwhelmed by the raw number of clients using it or
    the slower clients (arm, mips, sparc, hppa...basically anything not
    x86, ppc, ppc64, amd64) get screwed by the raw amount of CPU and
    RAM necessary to unpack a checkout from these packs.

4. git-daemon (and git over ssh) both are very stupid when it comes
    to generating packs for transfer, since often two or three
    fetches can be going on but the packs are not shared between
    fetches.  This only makes the already shakey server performance
    even worse, as the same packs are generated N times instead of
    once.

If GIT gets repository slicing (ability to check out and update
slices of the repository) as well as history slicing (only take the
last six months of history, for example) I think it would be a
better canidate.  The conversion to GIT from CVS was also lengthy
(approximately two weeks) althought many projects attempted a switch
this summer and tools have improved in speed.

Note: Both history and repository slicing are in the works for GIT,
but there is no date of completion for them.

-- 
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: Re: [RFC] Some sync control

[Steve Long]

Thanks for the info; git does seem to have issues, going by the `GLEP'. bzr
looks nice, but I don't know enough about it. From what i've read on the ml
and the website it will reach 1.0 in ~March, but has issues eg with
cross-platform development and diff/ commit.

svn seems the most mature of the next-gen tools.


-- 
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: [RFC] Some sync control

[Markus Ullmann]

Donnie Berkholz schrieb:
> Greg KH wrote:
>> What was the reasons he cited?
> 
> Given that ports is pretty similar to our gentoo-x86, I'd guess about 
> the same ones mentioned at 
> http://dev.gentoo.org/~antarus/projects/gleps/glep-0666.txt -- I quote 
> from there:
> 
> 1. Git currently requires you to check out the whole repository.
>    This includes *all of the history*.
> 2. Git cannot update portions of the repository, it can only update
>    the entire thing.

This was one of the big reasons. They (and we maybe as well) have people 
there with 56k/64k dialup connections. Checking out the whole thing 
would take ages.

Second thing was that absolutely none of the scripts would be able to 
handle it and they would have to be rewritten from ground up whereas 
most of them would work with svn if you just change the binary path (or 
symlink it even)

 > The conversion to GIT from CVS was also lengthy
 > (approximately two weeks) althought many projects attempted a switch
 > this summer and tools have improved in speed.

This one was the third. At the time they tried, the conversion could not 
be suspended, so cvs would have to be taken offline for a really long time.

And the last thing was the idea about distribution. There is one 
"centrally" maintained tree and people commit to it all day. So the 
chance of getting conflicts in pushes if one is on tour for three days 
would be very likely and so the distributed part of the VCs wouldn't be 
helpful.

Jokey

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: [RFC] Some sync control

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 2583 bytes --]

On Wed, Jan 17, 2007 at 09:03:59AM +0100, Markus Ullmann wrote:
> >1. Git currently requires you to check out the whole repository.
> >   This includes *all of the history*.
> >2. Git cannot update portions of the repository, it can only update
> >   the entire thing.
> 
> This was one of the big reasons. They (and we maybe as well) have people 
> there with 56k/64k dialup connections. Checking out the whole thing 
> would take ages.
See lower down in the GLEP where it states that upstream are working on
it, and such features would be completed sooner is Gentoo added some
manpower. I do however personally expect them to be ready by mid-2007
already.

> Second thing was that absolutely none of the scripts would be able to 
> handle it and they would have to be rewritten from ground up whereas 
> most of them would work with svn if you just change the binary path (or 
> symlink it even)
I disagree with this statement. There are several mapping scripts that
provide interfaces for the old CVS commands as close as possible
(exceedingly close actually).

> > The conversion to GIT from CVS was also lengthy
> > (approximately two weeks) althought many projects attempted a switch
> > this summer and tools have improved in speed.
> This one was the third. At the time they tried, the conversion could not 
> be suspended, so cvs would have to be taken offline for a really long time.
Upstream has moved beyond this point. If we were to convert to GIT right
now, it is intelligent enough to be able to start the conversion with a
snapshot, and then add the changes between the snapshot being taken, and
the final point after the initial conversion is complete.

> And the last thing was the idea about distribution. There is one 
> "centrally" maintained tree and people commit to it all day. So the 
> chance of getting conflicts in pushes if one is on tour for three days 
> would be very likely and so the distributed part of the VCs wouldn't be 
> helpful.
I refute this statement. You are no more or less likely to get conflicts
than with CVS (ignoring that fact that GIT has smarter merge
algorithms). If you do a CVS checkout, go away for 3 days, and then try
to commit, CVS will require you to update and resolve checkouts before
accepting your commit. GIT is no different, except that you can at least
have multiple revisions of your changes locally while working on them.

-- 
Robin Hugh Johnson
Gentoo Linux Developer
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 240 bytes --]

Re: [gentoo-dev] Re: [RFC] Some sync control

[Donnie Berkholz]

Markus Ullmann wrote:
> And the last thing was the idea about distribution. There is one 
> "centrally" maintained tree and people commit to it all day. So the 
> chance of getting conflicts in pushes if one is on tour for three days 
> would be very likely and so the distributed part of the VCs wouldn't be 
> helpful.

The other points were valid, but if it works anything like Gentoo, I 
think this is BS. Sure, everyone commits to the same tree, but not to 
the same lines of the same file. Unless all they do over in BSD-land is 
global seds all day long, I don't see this scenario.

Thanks,
Donnie
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: [RFC] Some sync control

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1114 bytes --]

On Wed, 17 Jan 2007 00:30:39 -0800 Donnie Berkholz
<dberkholz@gentoo.org> wrote:
| Markus Ullmann wrote:
| > And the last thing was the idea about distribution. There is one 
| > "centrally" maintained tree and people commit to it all day. So the 
| > chance of getting conflicts in pushes if one is on tour for three
| > days would be very likely and so the distributed part of the VCs
| > wouldn't be helpful.
| 
| The other points were valid, but if it works anything like Gentoo, I 
| think this is BS. Sure, everyone commits to the same tree, but not to 
| the same lines of the same file. Unless all they do over in BSD-land
| is global seds all day long, I don't see this scenario.

You mean like when eight or so archs keyword something for a security
bug within a few hours of each other? Or when three or four archs go
stable with a new KDE or Gnome release on the same day?

-- 
Ciaran McCreesh
Mail                                : ciaranm at ciaranm.org
Web                                 : http://ciaranm.org/
Paludis, the secure package manager : http://paludis.pioto.org/


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Re: [RFC] Some sync control

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1044 bytes --]

On Wed, Jan 17, 2007 at 08:38:34AM +0000, Ciaran McCreesh wrote:
> | The other points were valid, but if it works anything like Gentoo, I 
> | think this is BS. Sure, everyone commits to the same tree, but not to 
> | the same lines of the same file. Unless all they do over in BSD-land
> | is global seds all day long, I don't see this scenario.
> You mean like when eight or so archs keyword something for a security
> bug within a few hours of each other? Or when three or four archs go
> stable with a new KDE or Gnome release on the same day?
You get conflicts with CVS already in that case, it's not going to
increase the number of conflicts in any way.

As a different note, if we were so inclined GIT would actually allow us
to specify that the KEYWORDS line is safe to perform additive merges on
always - I wouldn't trust such a behavior myself, but the option is
there.

-- 
Robin Hugh Johnson
Gentoo Linux Developer
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 240 bytes --]

Re: [gentoo-dev] Re: [RFC] Some sync control

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1194 bytes --]

On Wed, 17 Jan 2007 01:06:31 -0800 "Robin H. Johnson"
<robbat2@gentoo.org> wrote:
| On Wed, Jan 17, 2007 at 08:38:34AM +0000, Ciaran McCreesh wrote:
| > | The other points were valid, but if it works anything like
| > | Gentoo, I think this is BS. Sure, everyone commits to the same
| > | tree, but not to the same lines of the same file. Unless all they
| > | do over in BSD-land is global seds all day long, I don't see this
| > | scenario.
| > You mean like when eight or so archs keyword something for a
| > security bug within a few hours of each other? Or when three or
| > four archs go stable with a new KDE or Gnome release on the same
| > day?
|
| You get conflicts with CVS already in that case, it's not going to
| increase the number of conflicts in any way.

Except that with CVS, you just update that one directory, which isn't
particularly painful even for all the arch people who live in backwards
countries with wet string internet connections.

-- 
Ciaran McCreesh
Mail                                : ciaranm at ciaranm.org
Web                                 : http://ciaranm.org/
Paludis, the secure package manager : http://paludis.pioto.org/


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Re: [RFC] Some sync control

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1401 bytes --]

On Wed, Jan 17, 2007 at 09:14:41AM +0000, Ciaran McCreesh wrote:
> | You get conflicts with CVS already in that case, it's not going to
> | increase the number of conflicts in any way.
> Except that with CVS, you just update that one directory, which isn't
> particularly painful even for all the arch people who live in backwards
> countries with wet string internet connections.

Please see my posting that I made before Donnie's (message id
<20070117083014.GC23219@curie-int.orbis-terrarum.net>), in which I
state:
> > See lower down in the GLEP where it states that upstream are working on
> > it, and such features would be completed sooner is Gentoo added some
> > manpower. I do however personally expect them to be ready by mid-2007
> > already.

I fully agree that right now GIT is not suitable as you cannot do partial
checkouts in time or directory dimensions. But it really is coming in the
future.

After the initial checkout (which sucks on wet-string+cans as well), GIT
actually uses less bandwidth than CVS, because it doesn't need to send
entire files back to the server to get diffs. It just uses rsync (where
available) to pull over the new files (the actual revision data files
never change once committed).

-- 
Robin Hugh Johnson
Gentoo Linux Developer
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 240 bytes --]

[gentoo-dev] Re: [RFC] Some sync control

[Holger Hoffstaette]

On Tue, 16 Jan 2007 16:12:21 +0000, Steve Long wrote:

> Robin H. Johnson wrote:
>> My personal view (not infra) on it, is that I'm mostly negative about
>> changing VCS at all - I would prefer not to change, because the status
>> quo works very well as it is. If a change is going to be made, it should
>> be taken as a chance to resolve as many different issues at one time as
>> possible, and for that reason I favour GIT over SVN.
>> 
> <noob alert> I'm looking for a distributed SCM atm, and have come down to
> git, bzr, svn or arch. (darcs looks nice but adds haskell dependency.)

As others have said, svn is centralized, and the working models of
distributed and centralized systems differ greatly. Process is a very
important factor for such a decision.
That being said, mercurial (http://www.selenic.com/mercurial/wiki/) was
not yet mentioned but might be a good choice because just like portage it
is written in python (with C core for performance).

There's a Google TechTalk video about it:
http://video.google.com/videoplay?docid=-7724296011317502612

-h


-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: [RFC] Some sync control

[Greg KH]

On Tue, Jan 16, 2007 at 05:36:34PM -0800, Donnie Berkholz wrote:
> The conversion to GIT from CVS was also lengthy (approximately two
> weeks) althought many projects attempted a switch this summer and
> tools have improved in speed.

Yes, the speed has increased a _lot_ now.  In fact yesterday someone
tweaked the tools even more and has reduced the time it takes to import
the entirety of the glibc cvs tree into git to a mere 45 minutes (with
all branches).  So this isn't really an issue anymore either.

> Note: Both history and repository slicing are in the works for GIT,
> but there is no date of completion for them.

Is there ever a "completion date" for opensource projects :)

thanks,

greg k-h
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: [RFC] Some sync control

[Rémi Cardona]

Steve Long wrote:
> Robin H. Johnson wrote:
>> My personal view (not infra) on it, is that I'm mostly negative about
>> changing VCS at all - I would prefer not to change, because the status
>> quo works very well as it is. If a change is going to be made, it should
>> be taken as a chance to resolve as many different issues at one time as
>> possible, and for that reason I favour GIT over SVN.
>>
> <noob alert> I'm looking for a distributed SCM atm, and have come down to
> git, bzr, svn or arch. (darcs looks nice but adds haskell dependency.) I'd
> really like to know which one gentoo-devs prefer and why (without starting
> a flame- if this is OT then np.) I'm leaning to git simply because it's
> used for the kernel, which seems like a project that would really stretch a
> VCS.
> 

As others have pointed out, SVN isn't distributed but it's very stable
and very user friendly (much more than git imho, but YMMV).

KDE migrated to svn about a year ago, and Gnome folks did it too a few
weeks ago. One thing that might bite us when doing any migration (svn,
git, whatever) is the CVS data.

Because atomic commits don't exist in CVS, the scripts rely on
commit/modification dates to recreate atomic commits in svn/git.
Unfortunately, in some not-so-rare cases, it can definitely mess things
up, and Gnome folks took about 6 months to get rid of these issues.

In any way, I think it'd be best to contact admins from
Gnome/KDE/FreeDesktop/kernel/... to see how to handle issues on the
server side, eg :

- git uses very little space but a lot of CPU
- svn uses a lot of disk space
- <add your favorite statement here>

Are those statements true for portage? Do they actually matter for us?

*Conclusion* We'd need to try a migration of snapshots to see how much
load it would be to migrate gentoo-x86 from CVS onto something else.

Cheers,

Rémi

PS, my daddy's SCM can beat the crap out your daddy's SCM anytime! ;)

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: [RFC] Some sync control

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1029 bytes --]

On Thu, Jan 18, 2007 at 12:09:30AM +0100, R?mi Cardona wrote:
> Because atomic commits don't exist in CVS, the scripts rely on
> commit/modification dates to recreate atomic commits in svn/git.
> Unfortunately, in some not-so-rare cases, it can definitely mess things
> up, and Gnome folks took about 6 months to get rid of these issues.
You should use dates AND usernames.

> Are those statements true for portage? Do they actually matter for us?
> 
> *Conclusion* We'd need to try a migration of snapshots to see how much
> load it would be to migrate gentoo-x86 from CVS onto something else.
Could you please actually look at the rest of the links in this thread?
Antarus did exactly this for his Summer-of-Code work. The original plan
was testing migrations to SVN, GIT and Mercurial. I believe that
Mercurial flunked very early, and I do not know the details.

-- 
Robin Hugh Johnson
Gentoo Linux Developer
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 240 bytes --]

Re: [gentoo-dev] [RFC] Some sync control

[Georgi Georgiev]

Quoting "Robin H. Johnson" <robbat2@gentoo.org>:
> 2. See the results (and as-yet unpublished GLEP) of Antarus's Summer of
> Code research into VCS migrations.

Where can I see these results?

The Gentoo SOC page only has a link to Planet Gentoo, Planet Gentoo  
has a link to a dead blog (http://scriptkitty.com/blog) and the RSS  
feed has exactly one post by antarus.

The Google SOC page does not have a link for antarus' project...


----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.


-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] [RFC] Some sync control

[Dan Meltzer]

see http://dev.gentoo.org/~antarus/projects/soc/glep-0052.txt and
http://article.gmane.org/gmane.linux.gentoo.devel/42044/match=

On 1/17/07, Georgi Georgiev <chutz@gg3.net> wrote:
> Quoting "Robin H. Johnson" <robbat2@gentoo.org>:
> > 2. See the results (and as-yet unpublished GLEP) of Antarus's Summer of
> > Code research into VCS migrations.
>
> Where can I see these results?
>
> The Gentoo SOC page only has a link to Planet Gentoo, Planet Gentoo
> has a link to a dead blog (http://scriptkitty.com/blog) and the RSS
> feed has exactly one post by antarus.
>
> The Google SOC page does not have a link for antarus' project...
>
>
> ----------------------------------------------------------------
> This message was sent using IMP, the Internet Messaging Program.
>
>
> --
> gentoo-dev@gentoo.org mailing list
>
>
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: [RFC] Some sync control

[sanchan]

Markus Ullmann wrote:
> This was one of the big reasons. They (and we maybe as well) have people
> there with 56k/64k dialup connections. Checking out the whole thing
> would take ages.

I can confirm we have people with 56k dial up :-) Checking out portage every day
for a developer on 56k takes already a lot of time using cvs. If I had to check
out the whole history every day I'd never become a gentoo developer.

> And the last thing was the idea about distribution. There is one
> "centrally" maintained tree and people commit to it all day. So the
> chance of getting conflicts in pushes if one is on tour for three days
> would be very likely and so the distributed part of the VCs wouldn't be
> helpful.

I agree.
-- 
Sandro (sanchan)
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: [RFC] Some sync control

[Greg KH]

On Thu, Jan 18, 2007 at 07:52:23PM +0100, sanchan wrote:
> Markus Ullmann wrote:
> > This was one of the big reasons. They (and we maybe as well) have people
> > there with 56k/64k dialup connections. Checking out the whole thing
> > would take ages.
> 
> I can confirm we have people with 56k dial up :-) Checking out portage every day
> for a developer on 56k takes already a lot of time using cvs. If I had to check
> out the whole history every day I'd never become a gentoo developer.

Once you have the original tree, syncing with git for the update would
be very small and workable with a 56k dialup.  It's just the original
sync would be a pain, but again, probably no more than the original cvs
checkout was.

thanks,

greg k-h
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] [RFC] Some sync control

[Thomas Rösner]

Dan Meltzer wrote:
> see http://dev.gentoo.org/~antarus/projects/soc/glep-0052.txt and
> http://article.gmane.org/gmane.linux.gentoo.devel/42044/match=

...in which mercurial is only mentioned in passing. Apart from Gentoo 
choosing a new VCS I'd really be interested in how the tests went with 
Hg (if there were any).

Regards,
    T.
-- 
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: [RFC] Some sync control

[Steve Long]

Thanks for all the comments about the different SCM systems.

I'm a bit confused about all the portage tree stuff. Since a couple of us
were discussing a QA db on this list, I've been working on a script to pull
the info from the /usr/portage/ hierarchy. There's just under 25,000
ebuilds, which are maintained by about 100 devs (not sure of exact number,
taken from a forum post.) I guess what I'm asking is why this isn't just a
database. You have a live tree, what's wrong with a live db?

Please note, I'm not talking about applications like portage or pkgcore,
just the ebuild text files, which I understand have one maintainer?


-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: [RFC] Some sync control

[Jan Kundrát]

Rémi Cardona wrote:
> - svn uses a lot of disk space

Could you please elaborate? Are you referring to the checkout size 
(which is about twice the actual size because SVN stores two copies of a 
file in the checkout to be able to perform diffs against latest revision 
without contacting the server) or something else?

Cheers,
-jkt

-- 
cd /local/pub && more beer > /dev/mouth
-- 
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: [RFC] Some sync control

[Steve Long]

i appreciate that source control is needed to maintain files over a period
of time and to roll back changes. does that happen with ebuilds?

-- 
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: [RFC] Some sync control

[Steve Long]

Side point: i am now aware that there is a better way to do this (pkgcore
cache/template.py and sql_template.py) thanks to ferringb.

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: [RFC] Some sync control

[Alec Warner]

Jan Kundrát wrote:
> Rémi Cardona wrote:
>> - svn uses a lot of disk space
> 
> Could you please elaborate? Are you referring to the checkout size
> (which is about twice the actual size because SVN stores two copies of a
> file in the checkout to be able to perform diffs against latest revision
> without contacting the server) or something else?
> 
> Cheers,
> -jkt
> 

SVN is a big beast on the server.

http://www.gentoo.org/proj/en/infrastructure/cvs-migration.xml
-- 
gentoo-dev@gentoo.org mailing list