[gentoo-user] Attic (cvs) -> ???(git)

[gentoo-user] Attic (cvs) -> ???(git)

[James]

Howdy,


So using wget to fetch {package/files} from the gentoo attic was/is a reliable
exercise to build things removed from the tree, into one's
/usr/local/portage tree. It still works, but I'm guessing there is a now a
"github_way" to do this.  A Fully automated script? I could not find a
gentoo wiki page to such effect, nor any other suggested pathway, but surely
it exists? I guessing the gentoo's cvs_attic is deprecated now and thus
subject to removal in the near future?


I'd would be excited to see a specific example, using git on on of  the 
recently removed packages "app-misc/multimon" [1], which was just removed 
from the portage tree, unless someone wants to use another example. That
package is about radio packets. 


Furthermore, I intend to rescue quite a few old C codes, so bear that in
mind should anyone's pedantic latencies give them urges for organizational
guidelines in addition to raw syntax tidbits.


James

[1] http://www.arrl.org/digital-data-modes

Re: [gentoo-user] Attic (cvs) -> ???(git)

[Rich Freeman]

On Mon, Feb 22, 2016 at 2:49 PM, James <wireless@tampabay.rr.com> wrote:
>
> So using wget to fetch {package/files} from the gentoo attic was/is a reliable
> exercise to build things removed from the tree, into one's
> /usr/local/portage tree. It still works, but I'm guessing there is a now a
> "github_way" to do this.

cvs was file-oriented, and git is not, so I'll warn you up-front that
finding deleted files is a bit of pita.

> A Fully automated script? I could not find a
> gentoo wiki page to such effect, nor any other suggested pathway, but surely
> it exists? I guessing the gentoo's cvs_attic is deprecated now and thus
> subject to removal in the near future?

I don't think anybody has plans to get rid of the old cvs, but nothing
new goes into it, so it is frozen in time as of the migration date.

> I'd would be excited to see a specific example, using git on on of  the
> recently removed packages "app-misc/multimon" [1], which was just removed
> from the portage tree, unless someone wants to use another example. That
> package is about radio packets.

The approach to take depends on whether you want to find EVERYTHING
that has ever been deleted, or a specific thing, but ultimately it
comes down to looking at the full log.  Lots of good stuff can be
found here:
https://stackoverflow.com/questions/953481/find-and-restore-a-deleted-file-in-a-git-repository

The way I'd do it is run "git log --diff-filter=D --summary" and
search for multimon.  That gives you the commit ID it was removed in.
Then you want to checkout the commit before it.

In this case doing that search will yield:
commit 760e17fcbac1b8c04a96ab08306dbcc644131dfb
Author: Pacho Ramos <pacho@gentoo.org>
Date:   Sat Feb 20 12:49:31 2016

    Remove masked for removal packages

...
 delete mode 100644 app-misc/multimon/Manifest
 delete mode 100644 app-misc/multimon/files/multimon-1.0-flags.patch
 delete mode 100644 app-misc/multimon/files/multimon-1.0-includes.patch
 delete mode 100644 app-misc/multimon/files/multimon-1.0-prll.patch
 delete mode 100644 app-misc/multimon/metadata.xml
 delete mode 100644 app-misc/multimon/multimon-1.0-r2.ebuild
 delete mode 100644 app-misc/multimon/multimon-1.0-r3.ebuild
...

Then what you want to do is checkout the previous commit:
git checkout "760e17fcbac1b8c04a96ab08306dbcc644131dfb^1"

Now you're looking at the repo containing the last known state of
those files, so you can just copy them or cat them from the directory
tree:
cat app-misc/multimon/multimon-1.0-r3.ebuild

You could build all that into a script.  If I were doing anything too
crazy with all this I'd probably use the python git module.  Then
you'll get all your query results in collections and such instead of
having to parse all that output.  If you do want to parse you can
control the output of git log.

I will say that deleted files are one of those things that isn't as
pretty in git.  It isn't like a file with a deleted state flag that
you can search by - they're identified by their presence in one commit
and absence in the next.  In fact, to identify them I'd think that
git-log has to basically has to diff every tree for every commit
historically.  That isn't as bad as it sounds as each directory is
shared with the previous commit COW-style - so if only one
subdirectory contains changes only that directory needs to be walked
to find what those differences are, and so on.  The structure of our
repository leads to a relatively well-balanced tree with fairly few
levels, which is a good case for git.  When I did the git validation I
had to walk all of it and doing it smartly in parallel you can get it
done remarkably quickly even in python (considering we have 2M
commits, which is 2M*<num-files-in-portage> files you could have to
diff in the brute force approach).

-- 
Rich

[gentoo-user] Re: Attic (cvs) -> ???(git)

[James]

Rich Freeman <rich0 <at> gentoo.org> writes:


> The way I'd do it is run "git log --diff-filter=D --summary" and
> search for multimon.  That gives you the commit ID it was removed in.
> Then you want to checkout the commit before it.

This seems very reasonable and systematic. I'm not trying for git_voodo,
just a logical way to restore from archives the last/latest version
of a given package on a system. Sure, Later on I'll create my own github
and push up the package and any new /files/ (patches) that are needed
as I maintain them. Rote methodologoy is what I think I want and is
needed as a general pathway for anyone that may have need of an old
package.

> 
> In this case doing that search will yield:
> commit 760e17fcbac1b8c04a96ab08306dbcc644131dfb
> Author: Pacho Ramos <pacho <at> gentoo.org>
> Date:   Sat Feb 20 12:49:31 2016
> 
>     Remove masked for removal packages
> 
> ...
>  delete mode 100644 app-misc/multimon/Manifest
>  delete mode 100644 app-misc/multimon/files/multimon-1.0-flags.patch
>  delete mode 100644 app-misc/multimon/files/multimon-1.0-includes.patch
>  delete mode 100644 app-misc/multimon/files/multimon-1.0-prll.patch
>  delete mode 100644 app-misc/multimon/metadata.xml
>  delete mode 100644 app-misc/multimon/multimon-1.0-r2.ebuild
>  delete mode 100644 app-misc/multimon/multimon-1.0-r3.ebuild
> ...
> 
> Then what you want to do is checkout the previous commit:
> git checkout "760e17fcbac1b8c04a96ab08306dbcc644131dfb^1"
> 
> Now you're looking at the repo containing the last known state of
> those files, so you can just copy them or cat them from the directory
> tree:
> cat app-misc/multimon/multimon-1.0-r3.ebuild
> 
> You could build all that into a script. 


Exactly my plan. Once I do a few of these manually, script it up. 

> If I were doing anything too
> crazy with all this I'd probably use the python git module. 


dev-python/git-python ???   Any others or related docs/howtos/examples?




> Then you'll get all your query results in collections and such instead of
> having to parse all that output.  If you do want to parse you can
> control the output of git log.

> I will say that deleted files are one of those things that isn't as
> pretty in git.   

Yep, from what I've read. Going back in time to find minor patches or
extraneous sources can be an adventure, with any system. Git pitfalls will
have some fun with me, for a while.....


> It isn't like a file with a deleted state flag that
> you can search by - they're identified by their presence in one commit
> and absence in the next.  In fact, to identify them I'd think that
> git-log has to basically has to diff every tree for every commit
> historically.  That isn't as bad as it sounds as each directory is
> shared with the previous commit COW-style - so if only one
> subdirectory contains changes only that directory needs to be walked
> to find what those differences are, and so on.  The structure of our
> repository leads to a relatively well-balanced tree with fairly few
> levels, which is a good case for git.  When I did the git validation I
> had to walk all of it and doing it smartly in parallel you can get it
> done remarkably quickly even in python (considering we have 2M
> commits, which is 2M*<num-files-in-portage> files you could have to
> diff in the brute force approach).


Since I'll be doing this rather frequently (I use the cvs attic
extensively), I'll post up some (ugly) python  code for suggestions.

Ultimately, this will be fun to on a gentoo cluster using ipython?


Thanks for the input!
James

Re: [gentoo-user] Re: Attic (cvs) -> ???(git)

[Rich Freeman]

On Mon, Feb 22, 2016 at 4:49 PM, James <wireless@tampabay.rr.com> wrote:
> Rich Freeman <rich0 <at> gentoo.org> writes:
>
>> If I were doing anything too
>> crazy with all this I'd probably use the python git module.
>
> dev-python/git-python ???   Any others or related docs/howtos/examples?
>

I used pygit2, but there are a few different implenentations and
plenty of docs online in general.

Here is an example program that runs through a history and dumps a
list of commits and their metadata in csv format:
https://github.com/rich0/gitvalidate/blob/master/gitdump/parsetrees.py

There are some other scripts that retrieve blobs and manipulate them
in the same directory.  This was part of the validation of the git
migration, which uses a map-reduce algorithm to diff every single
commit in a git history and identify all file revisions (which creates
a cvs-like per-file history which can then be compared with results
obtained from parsing a cvs repository for the same information).  The
only single-threaded step in the process is walking the list of
commits - all the diffs can be highly paralleled.

I doubt you need anything quite so fancy.  As you can see from the
script pulling metadata out of commits and walking through parents is
pretty easy.

My example doesn't account for merge commits.  There weren't any in
the cvs->git migration.  Obviously walking commits with merges will
get a lot messier.

-- 
Rich

[gentoo-user] Re: Attic (cvs) -> ???(git)

[James]

Rich Freeman <rich0 <at> gentoo.org> writes:


> I used pygit2, but there are a few different implenentations and
> plenty of docs online in general.


Just emerged this one. thx.


> Here is an example program that runs through a history and dumps a
> list of commits and their metadata in csv format:
> https://github.com/rich0/gitvalidate/blob/master/gitdump/parsetrees.py

very nice. I'll have to noodle around a bit with this script and
see what I came up with.



> There are some other scripts that retrieve blobs and manipulate them
> in the same directory.  This was part of the validation of the git
> migration, which uses a map-reduce algorithm to diff every single
> commit in a git history and identify all file revisions (which creates
> a cvs-like per-file history which can then be compared with results
> obtained from parsing a cvs repository for the same information).  The
> only single-threaded step in the process is walking the list of
> commits - all the diffs can be highly paralleled.

There use to be a central repo for many of the common gentoo admin scripts,
any idea where it is now that things have move to github?
(https://code.google.com/archive/p/genscripts/source)

I think there were others. The reason I like those, particularly for
new learning like python, is you get to see good and robust scripting
styles from the gentoo devs. Really helps when learning something new,
even if you have to figure out why things are written in a certain way.


> I doubt you need anything quite so fancy.  As you can see from the
> script pulling metadata out of commits and walking through parents is
> pretty easy.

ipython is some thing I want to learn by experimentation.


> My example doesn't account for merge commits.  There weren't any in
> the cvs->git migration.  Obviously walking commits with merges will
> get a lot messier.

I do not think I need 'merge commits' for one offs, that is pulling
old codes into a development system.  Later on, when I get more aggressive,
pulling old codes to run on gentoo cluster, I might pester the list seeking
more advice.


Thanks for the help, and encouragement,
James

[gentoo-user] Re: Attic (cvs) -> ???(git)

[walt]

On Mon, 22 Feb 2016 19:49:22 +0000 (UTC)
James <wireless@tampabay.rr.com> wrote:

> So using wget to fetch {package/files} from the gentoo attic was/is a
> reliable exercise to build things removed from the tree, into one's
> /usr/local/portage tree. It still works

Hi James.  I need a version of net-libs/gnutls from before the switch
to git.  Could I trouble you for an example of how you use wget?  So
far my googling hasn't even revealed the URL of the attic :-/

Thanks for any hints.

Re: [gentoo-user] Re: Attic (cvs) -> ???(git)

[Mike Gilbert]

On Wed, Feb 24, 2016 at 9:21 PM, walt <w41ter@gmail.com> wrote:
> On Mon, 22 Feb 2016 19:49:22 +0000 (UTC)
> James <wireless@tampabay.rr.com> wrote:
>
>> So using wget to fetch {package/files} from the gentoo attic was/is a
>> reliable exercise to build things removed from the tree, into one's
>> /usr/local/portage tree. It still works
>
> Hi James.  I need a version of net-libs/gnutls from before the switch
> to git.  Could I trouble you for an example of how you use wget?  So
> far my googling hasn't even revealed the URL of the attic :-/


https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/?hideattic=0

[gentoo-user] Re: Attic (cvs) -> ???(git)

[James]

walt <w41ter <at> gmail.com> writes:


> > So using wget to fetch {package/files} from the gentoo attic was/is a
> > reliable exercise to build things removed from the tree, into one's
> > /usr/local/portage tree. It still works

> Hi James.  I need a version of net-libs/gnutls from before the switch
> to git.  Could I trouble you for an example of how you use wget?  So
> far my googling hasn't even revealed the URL of the attic :-/

Sure,


I do it file by file; here is one of the 'files' (patches) I pulled down for
'showconsole' now also deprecated:

wget
http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/app-admin/showconsole/files/1.07-no-TIOCGDEV.patch


> Thanks for any hints.


If I know a package is going to be removed, I just emerge it and then copy
everything to /usr/local/portage/<category>/  prior to removal.

Hit me up with any other questions......


hth,
James

[gentoo-user] Re: Attic (cvs) -> ???(git)

[walt]

On Thu, 25 Feb 2016 13:07:34 +0000 (UTC)
James <wireless@tampabay.rr.com> wrote:

> walt <w41ter <at> gmail.com> writes:

> >  Could I trouble you for an example of how you use wget?

> Sure,
> 
> I do it file by file; here is one of the 'files' (patches) I pulled
> down for 'showconsole' now also deprecated:
> 
> wget
> http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/app-admin/showconsole/files/1.07-no-TIOCGDEV.patch

> If I know a package is going to be removed, I just emerge it and then
> copy everything to /usr/local/portage/<category>/  prior to removal.
> 
> Hit me up with any other questions......

Have you noticed that you can find lots of stuff with 'apropos' that
doesn't actually have a 'man' page?  Here's an example:

# apropos gnutls_x509_crt_export
gnutls_x509_crt_export (3)  - API function
gnutls_x509_crt_export2 (3)  - API function

# man gnutls_x509_crt_export
No manual entry for gnutls_x509_crt_export

Thanks to you and Mike for your examples :)

[gentoo-user] Re: Attic (cvs) -> ???(git)

[James]

walt <w41ter <at> gmail.com> writes:


> Have you noticed that you can find lots of stuff with 'apropos' that
> doesn't actually have a 'man' page?  Here's an example:


# equery belongs apropos
 * Searching for apropos ... 
app-shells/bash-completion-2.1_p20141224-r1
(/usr/share/bash-completion/completions/apropos -> man)
sys-apps/man-db-2.7.5 (/usr/bin/apropos -> whatis)

> # apropos gnutls_x509_crt_export
> gnutls_x509_crt_export (3)  - API function
> gnutls_x509_crt_export2 (3)  - API function

> # man gnutls_x509_crt_export
> No manual entry for gnutls_x509_crt_export

# whatis apropos
apropos (1)          - search the manual page names and descriptions

# whatis whatis
whatis (1)           - display one-line manual page descriptions


Looks like an alias to 'whatis'? 'whereis' still one of my favorite
little tools.

# whereis apropos
apropos: /usr/bin/apropos /usr/share/man/man1/apropos.1.bz2



> Thanks to you and Mike for your examples :)
You've help me far more that I have ever help you:: but, in any case
you are most welcome.


James

[gentoo-user] Re: Attic (cvs) -> ???(git)

[»Q«]

On Fri, 26 Feb 2016 16:43:05 +0000 (UTC)
James <wireless@tampabay.rr.com> wrote:

[ about aprospos ]
> Looks like an alias to 'whatis'? 'whereis' still one of my favorite
> little tools.

It queries the whatis database differently depending on how it's called.

  Think of the whatis database as columnar and containing two columns.
  The left column contains the program name (the command used to invoke
  the program) and the right side contains the first line of the
  manual's program synopsis. apropos searches both columns using the
  keyword as a regular expression to find all occurrences of the
  keyword. These occurrences may be embedded in the command word or the
  words of the synopsis. For example, apropos cat returns lines
  containing the word catalog, category, duplicate, application, etc.
  whatis, on the other hand, searches only the left hand column, which
  contains only the program name. This feature is helpful if you know
  the name of a command, but not its function.

That's from <http://www.linuxjournal.com/article/1329>, written 20
years ago, but AFAIK these things haven't changed.