[gentoo-dev] Moving packages to dev-vcs

[gentoo-dev] Moving packages to dev-vcs

[Sebastian Pipping]

Hello!


So now that we have a new category "dev-vcs" we need to move suitable
stuff over there.  Moving packages is complex and error prone:
This mail tries to guide you through and summarize the process, please
read on.

HINT:  Please keep CVS' radius of operation small to reduce risks.


0. Prepare
==========
- Check if your package fits:
    Is it a "utility focused on version control"?

- Ensure that you have permission  (if you are not the main maintainer)

- Sync the whole tree including profiles
  (as you'll be inspecting reverse dependencies)


1. Copy
=======
- Duplicate any traces of dev-util/${PN} in profiles/ to dev-vcs/${PN}
  (fgrep -Rw "dev-util/${PN}" profiles/)

- Copy complete package dev-util/${PN} to dev-vcs/${PN}
  (watch out CVS directories)


2. Switch
=========
- Update eclasses
    fgrep -w "dev-util/${PN}" eclass/*.eclass

- Update reverse dependencies
    http://tinderbox.dev.gentoo.org/misc/dindex/dev-util/${PN}
    http://tinderbox.dev.gentoo.org/misc/rindex/dev-util/${PN}
                                        ^^^
    fgrep -w "dev-util/${PN}" */*/*.ebuild

- Append "move dev-util/${PN} dev-vcs/${PN}" line to
  profiles/updates/1Q-2010
  (must come last, described in [1])


3. Remove
=========
- Remove package dev-util/${PN} from CVS (as in [2])
    cd dev-util
    cvs rm -Rf ${PN}
    cvs ci -m "dev-util/${PN}: Remove (renamed to dev-vcs/${PN})" ${PN}
                                                                  ^^^^^
- Remove any traces of dev-util/${PN} in profiles/


4. Notify
=========
- Report back problems with this process

- Mail fellow maintainers of dev-util/${PN} about the move

- If ${PN} is a big one (Subversion, Git, you know the list)
  - Update documentation  (now or open a bug at least, please)
  - Drop sping@g.o. a mail to update references in Layman
  - (Notify users and developers through Planet Gentoo?)


5. Final words
==============
- Help for filling "profiles/updates" documentation is wanted,
  it's currently _all empty_:
  http://devmanual.gentoo.org/profiles/updates/index.html

- Feel free to CC yourself to dev-vcs bug #56967 [3]



Sebastian


[1]
http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?part=2&chap=5#doc_chap6
[2]
http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?part=2&chap=5#doc_chap7
[3] https://bugs.gentoo.org/show_bug.cgi?id=56967

Re: [gentoo-dev] Moving packages to dev-vcs

[Ulrich Mueller]

>>>>> On Thu, 04 Mar 2010, Sebastian Pipping wrote:

> 2. Switch
> =========

> - Update reverse dependencies
>     http://tinderbox.dev.gentoo.org/misc/dindex/dev-util/${PN}
>     http://tinderbox.dev.gentoo.org/misc/rindex/dev-util/${PN}
>                                         ^^^
>     fgrep -w "dev-util/${PN}" */*/*.ebuild

Also check metadata.xml which may contain package references.
For example (from app-portage/layman/metadata.xml):

  <flag name='git'>Support <pkg>dev-util/git</pkg> based overlays</flag>

Ulrich

Re: [gentoo-dev] Moving packages to dev-vcs

[Markos Chandras]

[-- Attachment #1: Type: Text/Plain, Size: 1376 bytes --]

On Thursday 04 March 2010 23:08:06 Sebastian Pipping wrote:
> Hello!
> 
> 
> So now that we have a new category "dev-vcs" we need to move suitable
> stuff over there.  Moving packages is complex and error prone:
> This mail tries to guide you through and summarize the process, please
> read on.
> 
> HINT:  Please keep CVS' radius of operation small to reduce risks.
> 
> 
> 0. Prepare
> ==========
> - Check if your package fits:
>     Is it a "utility focused on version control"?
> 
> - Ensure that you have permission  (if you are not the main maintainer)
> 
> - Sync the whole tree including profiles
>   (as you'll be inspecting reverse dependencies)
> 
> 
> 1. Copy
> =======
> - Duplicate any traces of dev-util/${PN} in profiles/ to dev-vcs/${PN}
>   (fgrep -Rw "dev-util/${PN}" profiles/)
> 
> - Copy complete package dev-util/${PN} to dev-vcs/${PN}
>   (watch out CVS directories)
> 
> 
> 2. Switch
> =========
> - Update eclasses
>     fgrep -w "dev-util/${PN}" eclass/*.eclass
> 
> - Update reverse dependencies
>     http://tinderbox.dev.gentoo.org/misc/dindex/dev-util/${PN}
>     http://tinderbox.dev.gentoo.org/misc/rindex/dev-util/${PN}
This might require too much time so the tree will be broken for a while I 
guess

-- 
Markos Chandras (hwoarang)
Gentoo Linux Developer
Web: http://hwoarang.silverarrow.org

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Moving packages to dev-vcs

[Robin H. Johnson]

On Thu, Mar 04, 2010 at 10:08:06PM +0100, Sebastian Pipping wrote:
> So now that we have a new category "dev-vcs" we need to move suitable
> stuff over there.  Moving packages is complex and error prone:
> This mail tries to guide you through and summarize the process, please
> read on.
This contains a critical bug...

"cvs add" and the matching commit aren't mentioned anywhere...

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-dev] Moving packages to dev-vcs

[Dirkjan Ochtman]

On Thu, Mar 4, 2010 at 22:38, Robin H. Johnson <robbat2@gentoo.org> wrote:
> This contains a critical bug...
>
> "cvs add" and the matching commit aren't mentioned anywhere...

Well, it *is* a summary.

Thanks for the guide, that'll be useful.

Cheers,

Dirkjan

Re: [gentoo-dev] Moving packages to dev-vcs

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 3205 bytes --]

On Thu, Mar 04, 2010 at 10:08:06PM +0100, Sebastian Pipping wrote:
> 1. Copy
> =======
> - Duplicate any traces of dev-util/${PN} in profiles/ to dev-vcs/${PN}
>   (fgrep -Rw "dev-util/${PN}" profiles/)
> 
> - Copy complete package dev-util/${PN} to dev-vcs/${PN}
>   (watch out CVS directories)
> 
> 
> 2. Switch
> =========
> - Update eclasses
>     fgrep -w "dev-util/${PN}" eclass/*.eclass
> 
> - Update reverse dependencies
>     http://tinderbox.dev.gentoo.org/misc/dindex/dev-util/${PN}
>     http://tinderbox.dev.gentoo.org/misc/rindex/dev-util/${PN}
>                                         ^^^
>     fgrep -w "dev-util/${PN}" */*/*.ebuild

I'd suggest 

pquery --revdep dev-util/${PN} --raw --repo $PORTDIR

instead- throw in a '--attr inherited' if you want to look for 
eclasses that are implicated in addition, but it's a safer bet then 
relying on fgrep (since it's literal metadata searching) and doesn't 
have the potential for staleness tinderbox does.  Speed wise it'll be 
slower then grep, but not hugely so- reasonably up to date cache, it's 
<9s on my hardware so... there's no reason to not do it properly.


> - Append "move dev-util/${PN} dev-vcs/${PN}" line to
>   profiles/updates/1Q-2010
>   (must come last, described in [1])
> 
> 
> 3. Remove
> =========
> - Remove package dev-util/${PN} from CVS (as in [2])
>     cd dev-util
>     cvs rm -Rf ${PN}
>     cvs ci -m "dev-util/${PN}: Remove (renamed to dev-vcs/${PN})" ${PN}


Realistically, the dev *really* should be doing visibility scans 
after their efforts- yes bones does them too, but it's better to do 
the scan after to ensure any breakage is caught quickly.  Specifically 
via pkgcore-checks, I'd suggest running

pcheck -r $PORTDIR -c visibility '*'

prior to the move, and a dump of that commands output after the cvs rm 
invocation above.  Presuming everything was updated properly, the 
line count should be the same- output will vary slightly (previous 
complaints about dev-util/bzr being unavailable become dev-vcs/bzr 
unavailable specifically).

At least for my hardware, it's a 3.5 minute scan- so there really 
isn't a reason to *not* do this check (if the mips profiles were 
cleaned up/out prior it's more like 2.5 minutes on a sidenote)...


If folks are aware of alternate tools for doing these sort of checks 
in a timely fashion, feel free to pipe up also- I'm just suggesting 
the tools I know.


Random sidenote, anyone looked at using an alternate vcs to do the 
work, then proxy it back?  Specifically thinking of workflow like svk 
(or in this case hg cvs, 
https://wiki.mozilla.org/Using_Mercurial_locally_with_CVS ).  The 
reason I ask is that via building the work up outside of cvs, then 
proxying the add/remove/modifications back into it, it should be 
possible to minimize the window of cvs breakage down to bare minimum 
while still getting the same level of QA validation for the changes.

That's assuming hg/cvs behaves sanely of course, and doing a checkout 
(tip as it were) from cvs into hg is fast enough to be viable.  Just a 
thought- djc, you got any comments on this usage of hg?

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Moving packages to dev-vcs

[Sebastian Pipping]

On 03/04/10 22:38, Robin H. Johnson wrote:
> This contains a critical bug...
> 
> "cvs add" and the matching commit aren't mentioned anywhere...

That's a valid complaint, yes.

I left it out knowingly, maybe not for the better.



Sebastian

[gentoo-dev] Re: Moving packages to dev-vcs

[Ryan Hill]

[-- Attachment #1: Type: text/plain, Size: 625 bytes --]

On Thu, 04 Mar 2010 22:08:06 +0100
Sebastian Pipping <sping@gentoo.org> wrote:

> So now that we have a new category "dev-vcs" we need to move suitable
> stuff over there.  Moving packages is complex and error prone:
> This mail tries to guide you through and summarize the process, please
> read on.
> 
> [stuff]

Also search through our documentation and fix any package references.


-- 
fonts,                                            by design, by neglect
gcc-porting,                              for a fact or just for effect
wxwidgets @ gentoo     EFFD 380E 047A 4B51 D2BD C64F 8AA8 8346 F9A4 0662

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Moving packages to dev-vcs

[Serkan Kaba]

I'm hitting a repoman failure
--------
repoman: dev-vcs is not an official category.  Skipping QA checks in
this directory.
Please ensure that you add dev-vcs to
/home/firari/Desktop/çalışma/gentoo/gentoo-x86/profiles/categories
if it is a new category.
-----
After hitting it for the first time I updated the whole profiles dir and
still failing

Thanks in advance.
-- 
Sincerely,
Serkan KABA
Gentoo Developer

Re: [gentoo-dev] Moving packages to dev-vcs

[Petteri Räty]

[-- Attachment #1: Type: text/plain, Size: 500 bytes --]

On 03/04/2010 11:28 PM, Markos Chandras wrote:
> On Thursday 04 March 2010 23:08:06 Sebastian Pipping wrote:

>>
>> - Update reverse dependencies
>>     http://tinderbox.dev.gentoo.org/misc/dindex/dev-util/${PN}
>>     http://tinderbox.dev.gentoo.org/misc/rindex/dev-util/${PN}
> This might require too much time so the tree will be broken for a while I 
> guess
> 

When doing something like this, you can ask infra to suspend CVS to
rsync for you for a while.

Regards,
Petteri



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

[gentoo-dev] Re: Moving packages to dev-vcs

[Christian Faulhammer]

[-- Attachment #1: Type: text/plain, Size: 651 bytes --]

Hi,

Serkan Kaba <serkan@gentoo.org>:
> I'm hitting a repoman failure
> --------
> repoman: dev-vcs is not an official category.  Skipping QA checks in
> this directory.
> Please ensure that you add dev-vcs to
> /home/firari/Desktop/çalışma/gentoo/gentoo-x86/profiles/categories
> if it is a new category.
> -----
> After hitting it for the first time I updated the whole profiles dir
> and still failing

 It is in profiles/categories, so everything should be fine.

V-Li

-- 
Christian Faulhammer, Gentoo Lisp project
<URL:http://www.gentoo.org/proj/en/lisp/>, #gentoo-lisp on FreeNode

<URL:http://gentoo.faulhammer.org/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Moving packages to dev-vcs

[Serkan Kaba]

repoman was referencing the main tree for that (or maybe the overlays
additionally) and the issue was fixed when I syced.

2010/3/5 Christian Faulhammer <fauli@gentoo.org>:
> Hi,
>
> Serkan Kaba <serkan@gentoo.org>:
>> I'm hitting a repoman failure
>> --------
>> repoman: dev-vcs is not an official category.  Skipping QA checks in
>> this directory.
>> Please ensure that you add dev-vcs to
>> /home/firari/Desktop/çalışma/gentoo/gentoo-x86/profiles/categories
>> if it is a new category.
>> -----
>> After hitting it for the first time I updated the whole profiles dir
>> and still failing
>
>  It is in profiles/categories, so everything should be fine.
>
> V-Li
>
> --
> Christian Faulhammer, Gentoo Lisp project
> <URL:http://www.gentoo.org/proj/en/lisp/>, #gentoo-lisp on FreeNode
>
> <URL:http://gentoo.faulhammer.org/>
>

Re: [gentoo-dev] Moving packages to dev-vcs

[Jeroen Roovers]

On Thu, 04 Mar 2010 22:08:06 +0100
Sebastian Pipping <sping@gentoo.org> wrote:

> 4. Notify
> =========
> - Report back problems with this process
> 
> - Mail fellow maintainers of dev-util/${PN} about the move
> 
> - If ${PN} is a big one (Subversion, Git, you know the list)
>   - Update documentation  (now or open a bug at least, please)
>   - Drop sping@g.o. a mail to update references in Layman
>   - (Notify users and developers through Planet Gentoo?)

This step should probably include correcting all open bug
reports' Summaries to point to the new category, so that CAT/PN can be
found using the "simple" search interface.


Regards,
     jer

Re: [gentoo-dev] Moving packages to dev-vcs

[Sebastian Pipping]

On 03/05/10 18:10, Jeroen Roovers wrote:
>> 4. Notify
>> =========
[..]
> 
> This step should probably include correcting all open bug
> reports' Summaries to point to the new category, so that CAT/PN can be
> found using the "simple" search interface.

Good catch.  Thanks for fixing those on monotone (and others I guess).



Sebastian

Re: [gentoo-dev] Moving packages to dev-vcs

[Sebastian Pipping]

On 03/04/10 23:11, Brian Harring wrote:
> Random sidenote, anyone looked at using an alternate vcs to do the 
> work, then proxy it back?  Specifically thinking of workflow like svk 
> (or in this case hg cvs, 
> https://wiki.mozilla.org/Using_Mercurial_locally_with_CVS ).  The 
> reason I ask is that via building the work up outside of cvs, then 
> proxying the add/remove/modifications back into it, it should be 
> possible to minimize the window of cvs breakage down to bare minimum 
> while still getting the same level of QA validation for the changes.

Assuming the person using such a tool

 - is fluent in using such tool
   (to at least compensate extra abstraction. with plain CVS you
   at least know for sure what's happening)

 - is manually doing extra commits for manifest fixes
   (like "repoman commit" would do for him otherwise)



Sebastian

Re: [gentoo-dev] Moving packages to dev-vcs

[Petteri Räty]

[-- Attachment #1: Type: text/plain, Size: 632 bytes --]

On 03/05/2010 07:18 PM, Sebastian Pipping wrote:
> On 03/05/10 18:10, Jeroen Roovers wrote:
>>> 4. Notify
>>> =========
> [..]
>>
>> This step should probably include correcting all open bug
>> reports' Summaries to point to the new category, so that CAT/PN can be
>> found using the "simple" search interface.
> 
> Good catch.  Thanks for fixing those on monotone (and others I guess).
> 
> 
> 
> Sebastian
> 

After the move is done could you please come up with a list of all the
things you needed to take into account and then work with me for example
to get it included in devmanual.

Regards,
Petteri


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: [gentoo-dev] Moving packages to dev-vcs

[Sebastian Pipping]

On 03/06/10 08:08, Petteri Räty wrote:
> After the move is done could you please come up with a list of all the
> things you needed to take into account and then work with me for example
> to get it included in devmanual.

Good idea.

I opened a bug for it so we don't forget about it.
https://bugs.gentoo.org/show_bug.cgi?id=308151



Sebastian

Re: [gentoo-dev] Moving packages to dev-vcs

[Fabian Groffen]

On 04-03-2010 22:08:06 +0100, Sebastian Pipping wrote:
> 3. Remove
> =========
> - Remove package dev-util/${PN} from CVS (as in [2])
>     cd dev-util
>     cvs rm -Rf ${PN}
>     cvs ci -m "dev-util/${PN}: Remove (renamed to dev-vcs/${PN})" ${PN}
>                                                                   ^^^^^
> - Remove any traces of dev-util/${PN} in profiles/

Please take masks for packages you move into account for /all/ profiles.


-- 
Fabian Groffen
Gentoo on a different level

Re: [gentoo-dev] Moving packages to dev-vcs

[Sebastian Pipping]

I was curious to see how far we have come so far:


Still to be moved
=================
dev-util/aegis
dev-util/cola
dev-util/colorsvn
dev-util/cvs
dev-util/cvs2cl
dev-util/cvs2svn
dev-util/cvsd
dev-util/cvsgraph
dev-util/cvsps
dev-util/cvsq
dev-util/cvsutils
dev-util/easygit
dev-util/fossil
dev-util/git
dev-util/gitosis
dev-util/gitosis-gentoo
dev-util/hgsvn
dev-util/qgit
dev-util/rapidsvn
dev-util/subversion
dev-util/svk
dev-util/svn2cl
dev-util/tig
dev-util/tkcvs


Already in place
================
dev-vcs/archway
dev-vcs/bzr
dev-vcs/bzr-explorer
dev-vcs/bzr-gtk
dev-vcs/bzr-rewrite
dev-vcs/bzr-svn
dev-vcs/bzrtools
dev-vcs/colorcvs
dev-vcs/cssc
dev-vcs/cvsspam
dev-vcs/darcs
dev-vcs/giggle
dev-vcs/gitg
dev-vcs/git-sh
dev-vcs/guilt
dev-vcs/hg-git
dev-vcs/hgsubversion
dev-vcs/hgview
dev-vcs/kdesvn
dev-vcs/mcvs
dev-vcs/mercurial
dev-vcs/monotone
dev-vcs/mr
dev-vcs/qbzr
dev-vcs/qct
dev-vcs/qsvn
dev-vcs/rcs
dev-vcs/statcvs
dev-vcs/statsvn
dev-vcs/stgit
dev-vcs/svn2git
dev-vcs/svnmailer

Re: [gentoo-dev] Moving packages to dev-vcs

[Robin H. Johnson]

On Mon, Mar 15, 2010 at 08:50:16PM +0100, Sebastian Pipping wrote:
> dev-util/cvs
> dev-util/cvs2cl
> dev-util/cvs2svn
> dev-util/cvsd
> dev-util/cvsgraph
> dev-util/cvsps
> dev-util/cvsq
> dev-util/cvsutils
> dev-util/git
> dev-util/gitosis
> dev-util/gitosis-gentoo
For myself as the Git maintainer, and the main cvs-utils guy, feel free
to move all of the above. Just shoot me a reminder after you do, because
part of the infra cfengine stuff will start spewing errors at me (it
doesn't handle package moves on it's own, so I need to update it).

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-dev] Moving packages to dev-vcs

[Sebastian Pipping]

On 03/04/10 22:38, Robin H. Johnson wrote:
> This contains a critical bug...
> 
> "cvs add" and the matching commit aren't mentioned anywhere...

I've been moving Git today, properly hopefully.
This is what I've been doing on the shell basically:


cd gentoo-x86

# Copy package to new location
cd dev-vcs
  # Add empty dirs to please CVS
  for dir in $(find ../dev-util/git/ -name CVS -prune -or -type d
-printf 'git/%P\n'); do
    mkdir ${dir}
    cvs add ${dir}
  done
  cd git
    rsync -v --recursive --archive --cvs-exclude ../../dev-util/git/ .
    find . -name CVS -prune -or -type f -print0 | xargs --null cvs add
    echangelog 'New package (copied from dev-util/git)'
    fgrep -Rw dev-util/git . | fgrep -v Header  # Check for self-references
    repoman manifest
    repoman ci -m 'dev-vcs/git: New package (copied from dev-util/git)'
  cd ..
cd ..

# Extend profiles
cd profiles
  # Duplicate refs to old package
  # ...

  # Add move entry
  echo 'move dev-util/git dev-vcs/git' >> updates/1Q-2010
  echangelog 'Move dev-util/git to dev-vcs/git'
  cvs ci -m 'Move dev-util/git to dev-vcs/git' ChangeLog updates/1Q-2010
cd ..

# Fix eclasses
nano -w eclass/git.eclass
cvs commit -m 'Move from dev-util/git to dev-vcs/git' eclass/git.eclass

# Fix packages depending on the old package
# For each package:
cd category/package
  fgrep -Rwl dev-util/git . | xargs -n 1 sed
's|\bdev-util/git\b|dev-vcs/git|' -i
  echangelog 'Propagate move of dev-util/git to dev-vcs/git'
  repoman manifest
  cvs diff -u | colordiff | less -r
  repoman ci -m "$(basename $(dirname $PWD))/$(basename $PWD): Propagate
move of dev-util/git to dev-vcs/git"
cd ..

# Check if we missed anything
fgrep -Rwl dev-util/git . | fgrep -v ChangeLog | fgrep -v Repository

# Remove old package
cd dev-util
  cvs rm -Rf git
  cvs ci -m "dev-util/git: Remove (renamed to dev-vcs/git)" git
cd ..

# Clean-up profiles
cd profiles
  # Remove refs to old package
  # ...
cd ..


# EOF