[gentoo-dev] RFC: put new additions along with removals in GWN

[gentoo-dev] RFC: put new additions along with removals in GWN

[Yuri Vasilevski]

Hello,

After seeing "Upcoming package removals", for couple of weeks now, in
GWN I'm beginning to think that I would like to see also a list of new
packages added to portage next to the list of packages to be removed.
This will give a better view on the actual state of portage tree, as
well as, will help people (like me) to find potentially useful programs
that we didn't knew existed (as I'll certainly will check the packages
that sound interesting to me from a quick look on that list).

I think it'll be quite easy to implement by doing some CVS magic, or
better yet a diff of the two portage snapshots corresponding to the
days of this and the previous GWNs.

If you think this is a good idea, I'd be glad to write some scripts for
this.

Best,
Yuri.
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC: put new additions along with removals in GWN

[Diego 'Flameeyes' Pettenò]

[-- Attachment #1: Type: text/plain, Size: 464 bytes --]

On Wednesday 25 October 2006 21:56, Yuri Vasilevski wrote:
> After seeing "Upcoming package removals", for couple of weeks now, in
> GWN I'm beginning to think that I would like to see also a list of new
> packages added to portage next to the list of packages to be removed.
http://packages.gentoo.org/

-- 
Diego "Flameeyes" Pettenò - http://farragut.flameeyes.is-a-geek.org/
Gentoo/Alt lead, Gentoo/FreeBSD, Video, Sound, ALSA, PAM, KDE, CJK, Ruby ...

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC: put new additions along with removals in GWN

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 688 bytes --]

On Wed, 2006-10-25 at 14:56 -0500, Yuri Vasilevski wrote:
> If you think this is a good idea, I'd be glad to write some scripts for
> this.

If you write all the code, I'll run it, but I'm not taking my time to
track down all of the additions.  The current removals is done by hand
by some volunteers for the GWN.  There's nothing automatic about it.
Someone else already suggested this to us, but I explained to him that
unless someone had some way to automate it that it simply wasn't
feasible to do by hand.

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Alpha/AMD64/x86 Architecture Teams
Games Developer/Council Member/Foundation Trustee
Gentoo Foundation

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC: put new additions along with removals in GWN

[Caleb Cushing]

reporting additions of new programs aren't feasible? or are you
referring to version updates and package bumps and such....
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC: put new additions along with removals in GWN

[Chris Gianelloni]

[-- Attachment #1: Type: text/plain, Size: 699 bytes --]

On Wed, 2006-10-25 at 16:25 -0400, Caleb Cushing wrote:
> reporting additions of new programs aren't feasible? or are you
> referring to version updates and package bumps and such....

None of it is feasible if I'm left to do it by hand.  I have much better
things to do (like actually add new packages) than try to keep track of
what everyone else is doing.  The GWN takes a significant amount of time
as it is to put together.  Any process that isn't automated is pretty
much out of the question due to time constraints.

-- 
Chris Gianelloni
Release Engineering Strategic Lead
Alpha/AMD64/x86 Architecture Teams
Games Developer/Council Member/Foundation Trustee
Gentoo Foundation

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC: put new additions along with removals in GWN

[Alec Warner]

Caleb Cushing wrote:
> reporting additions of new programs aren't feasible? or are you
> referring to version updates and package bumps and such....

Reporting removals will be done by treecleaners once I have it implemented.

Reporting additions may require some cvs foo on lark; such as new 
directories in  ${PORTDIR}/$CAT/

Someone needs to implement the foo; however.
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC: put new additions along with removals in GWN

[Paul de Vrieze]

[-- Attachment #1: Type: text/plain, Size: 1136 bytes --]

On Thursday 26 October 2006 00:57, Alec Warner wrote:
> Caleb Cushing wrote:
> > reporting additions of new programs aren't feasible? or are you
> > referring to version updates and package bumps and such....
>
> Reporting removals will be done by treecleaners once I have it implemented.
>
> Reporting additions may require some cvs foo on lark; such as new
> directories in  ${PORTDIR}/$CAT/
>
> Someone needs to implement the foo; however.

(Not on cvs, but on a normal tree, but maybe works on cvs. There is a sanity 
check by checking that a dir contains at least 1 ebuild)

===== start =====
OLDPKGS=/var/cache/gwn/oldpkgs
NEWPKGS=`mktemp -t`
find /usr/portage -mindepth 2 -maxdepth 2 |while read in
do
  EB=`echo $in/*.ebuild`
  if [ -n "$EB" ]; then
    echo $in
  fi
done >$NEWPKGS
#These are new packages
cat $OLDPKGS $OLDPKGS $NEWPKGS|sort |uniq -u

#Old packages can be retrieved almost similarly:
#cat $OLDPKGS $NEWPKGS $NEWPKGS|sort |uniq -u

mv $NEWPKGS $OLDPKGS
===== end =====

Paul

-- 
Paul de Vrieze
Gentoo Developer
Mail: pauldv@gentoo.org
Homepage: http://www.devrieze.net

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] RFC: put new additions along with removals in GWN

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 564 bytes --]

On Sat, 28 Oct 2006 20:42:14 +0200 Paul de Vrieze <pauldv@gentoo.org>
wrote:
| > Someone needs to implement the foo; however.
| 
| (Not on cvs, but on a normal tree, but maybe works on cvs. There is a
| sanity check by checking that a dir contains at least 1 ebuild)

If the foo were that simple, someone would have done it long ago. It
needs to take into account updates/.

-- 
Ciaran McCreesh
Mail                : ciaranm at ciaranm.org
Web                 : http://ciaranm.org/
as-needed is broken : http://ciaranm.org/show_post.pl?post_id=13


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] RFC: put new additions along with removals in GWN

[Alec Warner]

Paul de Vrieze wrote:
> On Thursday 26 October 2006 00:57, Alec Warner wrote:
>> Caleb Cushing wrote:
>>> reporting additions of new programs aren't feasible? or are you
>>> referring to version updates and package bumps and such....
>> Reporting removals will be done by treecleaners once I have it implemented.
>>
>> Reporting additions may require some cvs foo on lark; such as new
>> directories in  ${PORTDIR}/$CAT/
>>
>> Someone needs to implement the foo; however.
> 
> (Not on cvs, but on a normal tree, but maybe works on cvs. There is a sanity 
> check by checking that a dir contains at least 1 ebuild)
> 
> ===== start =====
> OLDPKGS=/var/cache/gwn/oldpkgs
> NEWPKGS=`mktemp -t`
> find /usr/portage -mindepth 2 -maxdepth 2 |while read in
> do
>   EB=`echo $in/*.ebuild`
>   if [ -n "$EB" ]; then
>     echo $in
>   fi
> done >$NEWPKGS
> #These are new packages
> cat $OLDPKGS $OLDPKGS $NEWPKGS|sort |uniq -u
> 
> #Old packages can be retrieved almost similarly:
> #cat $OLDPKGS $NEWPKGS $NEWPKGS|sort |uniq -u
> 
> mv $NEWPKGS $OLDPKGS
> ===== end =====
> 
> Paul
> 

I've got some foo that can run on lark and figure out new and removed 
packages; as well as do it given date ranges.  It uses the cvs history 
file.  My foo basically looks at new and removed Changelogs in 
gentoo-x86.  It excludes profiles/ (there are Changelogs in there). 
I'll post it a bit later; I have a few bits of it to fix.

It doesn't cover updates/.  I don't think that corner case is easily 
covered.
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] RFC: put new additions along with removals in GWN

[Yuri Vasilevski]

[-- Attachment #1: Type: text/plain, Size: 784 bytes --]

Hi,

On Sat, 28 Oct 2006 19:58:09 +0100
Ciaran McCreesh <ciaranm@ciaranm.org> wrote:

> On Sat, 28 Oct 2006 20:42:14 +0200 Paul de Vrieze <pauldv@gentoo.org>
> wrote:
> | > Someone needs to implement the foo; however.
> | 
> | (Not on cvs, but on a normal tree, but maybe works on cvs. There is
> a | sanity check by checking that a dir contains at least 1 ebuild)
> 
> If the foo were that simple, someone would have done it long ago. It
> needs to take into account updates/.

I have sent my script directly to wolf31o2 some days ago thinking that
the exact technical details can considered spam for the majority
subscribed on the list, but for completeness I'm attaching in in this
e-mail.

It works comparing two different portage snapshots that it can also
download.

Best,
Yuri.

[-- Attachment #2: gen_newpkg_table.sh --]
[-- Type: application/x-shellscript, Size: 4363 bytes --]

Re: [gentoo-dev] RFC: put new additions along with removals in GWN

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 948 bytes --]

Yuri Vasilevski wrote:
> Hi,
> 
> On Sat, 28 Oct 2006 19:58:09 +0100
> Ciaran McCreesh <ciaranm@ciaranm.org> wrote:
> 
>> On Sat, 28 Oct 2006 20:42:14 +0200 Paul de Vrieze <pauldv@gentoo.org>
>> wrote:
>> | > Someone needs to implement the foo; however.
>> | 
>> | (Not on cvs, but on a normal tree, but maybe works on cvs. There is
>> a | sanity check by checking that a dir contains at least 1 ebuild)
>>
>> If the foo were that simple, someone would have done it long ago. It
>> needs to take into account updates/.
> 
> I have sent my script directly to wolf31o2 some days ago thinking that
> the exact technical details can considered spam for the majority
> subscribed on the list, but for completeness I'm attaching in in this
> e-mail.
> 
> It works comparing two different portage snapshots that it can also
> download.
> 
> Best,
> Yuri.

I was planning to have this cron'd on lark (cvs.gentoo.org) and run 
against the cvs history file.

[-- Attachment #2: find-cvs-adds-and-removals.py --]
[-- Type: text/x-python, Size: 3133 bytes --]

#!/usr/bin/env python
# Authored by Alec Warner <antarus@gentoo.org>
# Released under the GPL Version 2
# Copyright Gentoo Foundation 2006

# Changelog: Initial release 2006/10/27

doc = """
# Purpose: This script analyzes the cvs history file in an attempt to locate package
# additions and removals.  It takes 3 arguments; two of which are optional.  It needs
# the path to the history file to read.  If a start_date is not provided it will read
# the entire file and match any addition/removal.  If you provide a start date it will
# only match thins that are after that start_date.  If you provide an end date you can
# find matches over date ranges.  If an end date is not provided it defaults to now()
"""

import sys, os, re, time, datetime

new_package = re.compile("^A(.*)\|.*gentoo-x86\/(.*)\/(.*)\|.*\|ChangeLog$")
removed_package = re.compile("^R(.*)\|.*gentoo-x86\/(.*)\/(.*)\|.*\|ChangeLog$")

class record(object):
	def __init__(self, who, date, cp):
		"""
		    Who is a string
		    date is a unix timestamp
		    cp is a category/package
		"""
		self.who = who
		self.date = datetime.datetime.fromtimestamp( date ) 
		self.package = cp

	def __str__( self ):
		return "Package %s was removed by %s on %s" % (self.package, self.who, self.date)

def main():
	if (len(sys.argv) < 2):
		usage()
		sys.exit(1)

	args = sys.argv[1:]
	history_file = args[0]

	if len(args) >= 2: 
		start_date = args[1]
		start_date = time.strptime( start_date, "%d/%m/%Y")
		start_date = time.mktime( start_date )
	else:
		start_date = 0 # no real start date then.

	if len(args) >= 3:
		end_date = args[2]
		end_date = time.strptime( end_date, "%d/%m/%Y")
		end_date = time.mktime( end_date )
	else:
		end_date = time.time()

	try:
		lines = open( history_file ).readlines()
	except (IOError, OSError), e:
		print "Failed to open History file!"
		raise e

	removals = []
	adds = []
	for line in lines:
		match = new_package.match( line )
		if match:
			t = match.groups()[0]
			split = t.split("|")
			t = split[0]
			who = split[1]
			try:
				t = int(t, 16)
			except e:
				print "Failed to convert hex timestamp to integer"
				raise e

			if t < end_date and t > start_date:
				rec = record( who, t, match.groups()[1] + "/" + match.groups()[2] )
				adds.append( rec )
				continue
			else:
				continue # date out of range
		match = removed_package.match( line )

		if match:
			t = match.groups()[0]
			split = t.split("|")
			t = split[0]
			who = split[1]
			try:
				t = int(t, 16)
			except e:
				print "Failed to convert hex timestamp to integer"
				raise e
			if t < end_date and t > start_date:
				rec = record( who, t, match.groups()[1] + "/" + match.groups()[2] )
				removals.append( rec )
				continue
			else:
				continue # date out of range
	print "Removed Packages:"
	for pkg in removals:
		print pkg

	print "Added Packages:"
	for pkg in adds:
		print pkg
	print
	print "Done."

def usage():
	print sys.argv[0] + " <history file> <start date> [end date]"
	print "start date should be in dd/mm/yyyy format"
	print "end date is not required and defaults to 'now'"
	print doc

if __name__ == "__main__":
	main()

Re: [gentoo-dev] RFC: put new additions along with removals in GWN

[Marius Mauch]

On Sat, 28 Oct 2006 17:09:10 -0400
Alec Warner <antarus@gentoo.org> wrote:

> It doesn't cover updates/.  I don't think that corner case is easily 
> covered.

Ehm, if you have a list of CP entries that were added/removed and
access to the update files it shouldn't be all that complicated.

Just made up pseudocode:

for x in $raw_added_cps; do
	grep "^move [^ ]* $x\$" $current_update_file ||
$real_added_cps="$real_added_cps $x"
done

Hardest part is probably to select $current_update_file correctly.

Marius

-- 
Public Key at http://www.genone.de/info/gpg-key.pub

In the beginning, there was nothing. And God said, 'Let there be
Light.' And there was still nothing, but you could see a bit better.
-- 
gentoo-dev@gentoo.org mailing list