[gentoo-portage-dev] VCS used for development of portage

[gentoo-portage-dev] VCS used for development of portage

[Sebastian Pipping]

Hello!


Is moving portage development over to Git planned anytime soon?
Anything keeping you from the move?

Anything I can do to speed it up?



Sebastian

Re: [gentoo-portage-dev] VCS used for development of portage

[Zac Medico]

On 02/26/2010 07:18 PM, Sebastian Pipping wrote:
> Hello!
> 
> 
> Is moving portage development over to Git planned anytime soon?

Yes, we've been discussing it on this bug:

  http://bugs.gentoo.org/show_bug.cgi?id=196025

> Anything keeping you from the move?

Well, the repository layout is somewhat non-trivial, but I think we
have a plan that will work. See discussion on bug 196025.

> Anything I can do to speed it up?

Robbin would know better than me.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] VCS used for development of portage

[Sebastian Pipping]

Hello!


I've been playing with Git conversions of the portage repository.
The current "demo" conversion from anon SVN is up here:

  http://git.goodpoint.de/?p=portage.git;a=summary

NOTE: Do not use it for development, yet - it's a demo!


At the end you can find the script I made for the conversion.

Notes on the script:
- An author map "portage-author-map.txt" is required, get from [1]
  (Note to zmedico/robbat2: fixed version, do not use the older one)
- If updated, the latest version of this script is up at [2]
- It works with anon SVN and rsync.  For the final run
  - use developer SVN instead
  - be sure to remove --rewrite-root !
- The final repository will be about 22MB in size
- Main conversion takes about two hours on my machine

Next we need to decide on
- a final location (git.overlays.gentoo.org/proj/portage?)
- when we convert (and freeze SVN)
- who runs the final conversion (zmedico, roobat2, me?)



Sebastian

[1] http://www.hartwork.org/public/portage-author-map.txt
[2] http://www.hartwork.org/public/portage-svn-to-git.sh


=================================================================
#!/usr/bin/env bash
starttime=$(date +'%Y-%m-%d--%H-%M-%S')
output_dir="portage-git-repo--${starttime}"

# Open logged subshell
(

# Print executed bash commands
set -x

# Rip/sync anon SVN using rsync
rsync -r rsync://anonvcs.gentoo.org/vcs-public-svnroot/portage/ \
    portage-anon-svn-repo-dump/ || exit 1

# Init git-svn repo, note double use of --trunk
[ -d "${output_dir}" ] && exit 1
git svn init file://${PWD}/portage-anon-svn-repo-dump/ \
    --rewrite-root=svn://anonsvn.gentoo.org/portage/ \
    --trunk=main/trunk --tags=main/tags --branches=main/branches \
    "${output_dir}"
cd "${output_dir}" || exit 1

# Convert commits
git config svn.authorsfile ../portage-author-map.txt
time git svn fetch || exit 1

# Make real Git tags from remotes/tags/* (two special cases)
for branch_tag in $(git branch -r | grep 'tags/' \
    | fgrep -v 'tags/portage-2.1_pre5'); do
  tag=$(sed 's|^tags/||' <<<"${branch_tag}")
  git tag "v${tag}" "remotes/${branch_tag}" || exit 1
done
git tag 'trunk@1888' 'remotes/trunk@1888' || exit 1

# Make local branches from remotes/* (excluding tags and trunk)
for branch in $(git branch -r | grep -v 'tags\|trunk'); do
  git checkout -b "${branch}" "remotes/${branch}" || exit 1
done
git checkout master || exit 1

# Reduce size of repository
dotgitsize() { du --human --total .git | tail -n 1; }
dotgitsize
git gc --aggressive
dotgitsize

# Wipe all traces of SVN
git config --remove-section 'svn-remote.svn'
git config --remove-section 'svn'
rm -R .git/svn
rm -R .git/logs/refs/remotes
for file in .git/info/refs .git/packed-refs ; do
  sed -e '/remotes\//d' -i "${file}"
done

# Hide executed bash commands
set +x


cat <<INFO

DONE.


NEXT STEPS:
  # cd "${output_dir}"

  Verify that branches and tags
  1. make sense
  2. are unambiguous
  3. are free of SVN
    # git branch -a
    # git show-branch --list
    # git tag -l

  Push full repository
    # git remote add \${remote_name} \${push_url}
    # git push --mirror \${remote_name}
INFO

) |& tee conversion--${starttime}.log
#================================================================

Re: [gentoo-portage-dev] VCS used for development of portage

[Zac Medico]

On 03/02/2010 03:22 PM, Sebastian Pipping wrote:
> Hello!
> 
> 
> I've been playing with Git conversions of the portage repository.
> The current "demo" conversion from anon SVN is up here:
> 
>   http://git.goodpoint.de/?p=portage.git;a=summary
> 
> NOTE: Do not use it for development, yet - it's a demo!

It looks very nice to me. I noticed that it preserved continuity
when branches got moved around, so the history seems like it will be
fully intact. Great job!
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] VCS used for development of portage

[Sebastian Pipping]

On 03/05/10 04:58, Zac Medico wrote:
>>   http://git.goodpoint.de/?p=portage.git;a=summary
>>
>> NOTE: Do not use it for development, yet - it's a demo!
> 
> It looks very nice to me. I noticed that it preserved continuity
> when branches got moved around, so the history seems like it will be
> fully intact. Great job!

Still, maybe we should not jump on this version yet:
- with svn2git we could split it to several repositories easily
  (see [1] for status you on related experiments)
- neither svn2git nor git-svn seem to support proper conversion of
  changes to svn:ignore

Summer of code could help about the latter:
http://en.gentoo-wiki.com/wiki/Google_Summer_of_Code_2010_ideas#Add_support_for_svn:ignore_to_svn2git

But pushing the conversion further into the future could also be a
trade-off reducing efficiency and the number of contributions.

I don't feel like proposing anything on that matter at the moment.  With
that said: what do you and Robin think?



Sebastian


[1] http://bugs.gentoo.org/show_bug.cgi?id=196025#c41

Re: [gentoo-portage-dev] VCS used for development of portage

[Robin H. Johnson]

On Fri, Mar 05, 2010 at 04:33:14PM +0100, Sebastian Pipping wrote:
> I don't feel like proposing anything on that matter at the moment.  With
> that said: what do you and Robin think?
Here's a related question.

Did the previous CVS -> SVN question generate the svn:ignore files from
.cvsignore, or simply discard them?

In either case, I'm starting to wonder if the change is just trivial
enough to get done in svn2git or git-svn directly. I think other
properties are already there.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-portage-dev] VCS used for development of portage

[Robert Buchholz]

[-- Attachment #1: Type: text/plain, Size: 741 bytes --]

On Saturday 06 March 2010, Robin H. Johnson wrote:
> In either case, I'm starting to wonder if the change is just trivial
> enough to get done in svn2git or git-svn directly. I think other
> properties are already there.

The git-svn man page states that a transformation is not trivial. I 
don't know if in general, svn:ignore is a subset of gitignore. 
Apprarantly, git-svnimport can handle this transformation, but it will 
probably come with other problems.

In my opinion, we should just ignore this small history loss and after 
the git conversion, commit a .gitignore file with the latest ignore 
rules to the active branches. But that is just my opinion, and for the 
actual portage developers to decide.



Robert

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-portage-dev] VCS used for development of portage

[Zac Medico]

On 03/07/2010 01:49 AM, Robert Buchholz wrote:
> On Saturday 06 March 2010, Robin H. Johnson wrote:
>> In either case, I'm starting to wonder if the change is just trivial
>> enough to get done in svn2git or git-svn directly. I think other
>> properties are already there.
> 
> The git-svn man page states that a transformation is not trivial. I 
> don't know if in general, svn:ignore is a subset of gitignore. 
> Apprarantly, git-svnimport can handle this transformation, but it will 
> probably come with other problems.
> 
> In my opinion, we should just ignore this small history loss and after 
> the git conversion, commit a .gitignore file with the latest ignore 
> rules to the active branches. But that is just my opinion, and for the 
> actual portage developers to decide.

I think that's reasonable. Mainly, the files that need to be ignored
are *.py[co] and doc/*.html.
-- 
Thanks,
Zac