[gentoo-portage-dev] [PATCH 0 of 3] repoman changelog patches

[gentoo-portage-dev] [PATCH 0 of 3] repoman changelog patches

[Fabian Groffen]

The following three patches change repoman's echangelog feature
slightly:

1) default for updating the changelog is retrieved from layout.conf
2) skel.ChangeLog is used for constructing the header of new ChangeLogs
3) modified files are checked for copyright years to be updated

Please review before I push them.

[gentoo-portage-dev] [PATCH 1 of 3] repoman: get default to update changelog from layout.conf

[Fabian Groffen]

Updating the ChangeLog file may be desirable in more repos than just the
one named 'gentoo', like e.g. the Prefix one.  Hence, make this default
configurable though metadata/layout.conf.

This commit must go accompanied by a commit to
gentoo-x86/metadata/layout.conf that adds the following bit:

# Bug #337853 - gentoo's council says to enable
# --echangelog by default for the "gentoo" repo
update-changelog = true

diff --git a/bin/repoman b/bin/repoman
--- a/bin/repoman
+++ b/bin/repoman
@@ -644,10 +644,7 @@
 		print(prefix + line)
 	sys.exit(1)
 
-if options.echangelog is None and \
-	repo_config.name == "gentoo":
-	# Bug #337853 - gentoo's council says to enable
-	# --echangelog by default for the "gentoo" repo
+if options.echangelog is None and repo_config.update_changelog:
 	options.echangelog = 'y'
 
 if vcs is None:
diff --git a/pym/portage/repository/config.py b/pym/portage/repository/config.py
--- a/pym/portage/repository/config.py
+++ b/pym/portage/repository/config.py
@@ -48,7 +48,7 @@
 		'eclass_overrides', 'eclass_locations', 'format', 'location',
 		'main_repo', 'manifest_hashes', 'masters', 'missing_repo_name',
 		'name', 'priority', 'sign_manifest', 'sync', 'thin_manifest',
-		'user_location')
+		'update_changelog', 'user_location')
 
 	def __init__(self, name, repo_opts):
 		"""Build a RepoConfig with options in repo_opts
@@ -126,6 +126,7 @@
 		self.create_manifest = True
 		self.disable_manifest = False
 		self.manifest_hashes = None
+		self.update_changelog = False
 		self.cache_format = None
 
 	def get_pregenerated_cache(self, auxdbkeys, readonly=True, force=False):
@@ -431,6 +432,9 @@
 						DeprecationWarning)
 			repo.manifest_hashes = manifest_hashes
 
+			if layout_data.get('update-changelog', '').lower() == 'true':
+				repo.update_changelog = True
+
 		#Take aliases into account.
 		new_prepos = {}
 		for repo_name, repo in prepos.items():

[gentoo-portage-dev] [PATCH 2 of 3] repoman: get ChangeLog header from skel.ChangeLog

[Fabian Groffen]

Use skel.ChangeLog from the repo to create the header of a new ChangeLog
file.  Else, we just retain the original header of the ChangeLog.  When
no skel.ChangeLog file exists, and this is a new ChangeLog, no header is
used.

diff --git a/bin/repoman b/bin/repoman
--- a/bin/repoman
+++ b/bin/repoman
@@ -2450,7 +2450,7 @@
 			new_changelog = utilities.UpdateChangeLog(checkdir_relative, \
 				catdir, pkgdir, \
 				clnew, clremoved, clchanged, \
-				changelog_msg, options.pretend)
+				changelog_msg, options.pretend, repodir)
 			if new_changelog is None:
 				writemsg_level("!!! Updating the ChangeLog failed\n", \
 					level=logging.ERROR, noiselevel=-1)
diff --git a/pym/repoman/utilities.py b/pym/repoman/utilities.py
--- a/pym/repoman/utilities.py
+++ b/pym/repoman/utilities.py
@@ -523,7 +523,8 @@
 
 	return outvcs
 
-def UpdateChangeLog(pkgdir, category, package, new, removed, changed, msg, pretend):
+def UpdateChangeLog(pkgdir, category, package, new, removed, changed, \
+		msg, pretend, repodir):
 	""" Write an entry to an existing ChangeLog, or create a new one. """
 
 	# figure out who to write as
@@ -550,7 +551,6 @@
 	cl_path = os.path.join(pkgdir, 'ChangeLog')
 	clold_lines = []
 	clnew_lines = []
-	old_header_lines = []
 	header_lines = []
 
 	try:
@@ -559,41 +559,50 @@
 			mode='r', encoding=_encodings['repo.content'], errors='replace')
 	except EnvironmentError:
 		clold_file = None
+	
+	clskel_file = None
+	if clold_file is None:
+		# we will only need the ChangeLog skeleton if there is no
+		# ChangeLog yet
+		try:
+			clskel_path = os.path.join(repodir, 'skel.ChangeLog')
+			clskel_file = io.open(_unicode_encode(clskel_path,
+				encoding=_encodings['fs'], errors='strict'),
+				mode='r', encoding=_encodings['repo.content'],
+				errors='replace')
+		except EnvironmentError:
+			pass
 
 	f, clnew_path = mkstemp()
 
-	# create an empty ChangeLog.new with correct header first
+	# construct correct header first
 	try:
-		f = io.open(f, mode='w', encoding=_encodings['repo.content'],
-			errors='backslashreplace')
-
-		if clold_file is None:
-			header_lines.append(_unicode_decode('# ChangeLog for %s/%s\n' %
-				(category, package)))
-			year = time.strftime('%Y')
-			header_lines.append(_unicode_decode('# Copyright 1999-'
-				'%s Gentoo Foundation; Distributed under the GPL v2\n' % year))
-			header_lines.append(_unicode_decode('# $Header: $\n'))
-			header_lines.append(_unicode_decode('\n'))
-		else:
+		if clold_file is not None:
+			# retain header from old ChangeLog
 			for line in clold_file:
-				line_strip =  line.strip()
-				if line_strip and line[:1] != "#":
-					clold_lines.append(line)
+				line_strip = line.strip()
+				clold_lines.append(line)
+				if line_strip[:1] != '#':
 					break
-				header_lines.append(line)
+				if clskel_file is None:
+					clnew_lines.append(line)
 				if not line_strip:
 					break
-
-			# update the copyright year
-			old_header_lines = header_lines[:]
-			if len(header_lines) >= 2:
-				header_lines[1] = re.sub(r'^(# Copyright \d\d\d\d)-\d\d\d\d ',
-					r'\1-%s ' % time.strftime('%Y'), header_lines[1])
+		elif clskel_file is not None:
+			# read skel.ChangeLog up to first empty line
+			for line in clskel_file:
+				line_strip = line.strip()
+				if not line_strip:
+					break
+				line = line.replace('<CATEGORY>', category)
+				line = line.replace('<PACKAGE_NAME>', package)
+				line = re.sub(r'^(# Copyright \d\d\d\d)-\d\d\d\d ',
+					r'\1-%s ' % time.strftime('%Y'), line)
+				clnew_lines.append(line)
+			clnew_lines.append(_unicode_decode('\n'))
+			clskel_file.close()
 
 		# write new ChangeLog entry
-		clnew_lines.extend(header_lines)
-		date = time.strftime('%d %b %Y')
 		newebuild = False
 		for fn in new:
 			if not fn.endswith('.ebuild'):
@@ -632,20 +641,18 @@
 			clnew_lines.append(_unicode_decode('%s\n' % line))
 		clnew_lines.append(_unicode_decode('\n'))
 
+		f = io.open(f, mode='w', encoding=_encodings['repo.content'],
+			errors='backslashreplace')
+
 		for line in clnew_lines:
 			f.write(line)
 
 		# append stuff from old ChangeLog
 		if clold_file is not None:
-			# If the old ChangeLog didn't have a header, then
 			# clold_lines may contain a saved non-header line
 			# that we want to write first.
-			for line in clold_lines:
-				f.write(line)
-
-			# Now prepend old_header_lines to clold_lines, for use
-			# in the unified_diff call below.
-			clold_lines = old_header_lines + clold_lines
+			if clold_lines[-1].strip():
+				f.write(clold_lines[-1])
 
 			for line in clold_file:
 				f.write(line)
@@ -655,7 +662,7 @@
 		# show diff (do we want to keep on doing this, or only when
 		# pretend?)
 		for line in difflib.unified_diff(clold_lines, clnew_lines,
-			fromfile=cl_path, tofile=cl_path + '.new', n=0):
+			fromfile=cl_path, tofile=cl_path, n=0):
 			util.writemsg_stdout(line, noiselevel=-1)
 		util.writemsg_stdout("\n", noiselevel=-1)
 
@@ -663,7 +670,7 @@
 			# remove what we've done
 			os.remove(clnew_path)
 		else:
-			# rename ChangeLog.new to ChangeLog, and set permissions
+			# rename to ChangeLog, and set permissions
 			try:
 				clold_stat = os.stat(cl_path)
 			except OSError:

Re: [gentoo-portage-dev] [PATCH 2 of 3] repoman: get ChangeLog header from skel.ChangeLog

[Zac Medico]

On 10/19/2011 12:55 PM, Fabian Groffen wrote:
> +			if clold_lines[-1].strip():
> +				f.write(clold_lines[-1])

If the old ChangeLog happens to be an empty file, then clold_lines[-1]
will raise IndexError. So, this is safer:

   if clold_lines and clold_lines[-1].strip():

-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH 2 of 3] repoman: get ChangeLog header from skel.ChangeLog

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 490 bytes --]

On 19-10-2011 14:58:39 -0700, Zac Medico wrote:
> On 10/19/2011 12:55 PM, Fabian Groffen wrote:
> > +			if clold_lines[-1].strip():
> > +				f.write(clold_lines[-1])
> 
> If the old ChangeLog happens to be an empty file, then clold_lines[-1]
> will raise IndexError. So, this is safer:
> 
>    if clold_lines and clold_lines[-1].strip():

I applied your fix, and moved it from the other patch in this one, where
it belongs.


-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[gentoo-portage-dev] [PATCH 3 of 3] repoman: update copyright on modified files

[Fabian Groffen]

To retain the behaviour of echangelog, update the copyrights on modified
files (mostly ebuilds) when necessary.  We also use this to update the
ChangeLog's copyright.

diff --git a/pym/repoman/utilities.py b/pym/repoman/utilities.py
--- a/pym/repoman/utilities.py
+++ b/pym/repoman/utilities.py
@@ -523,9 +523,84 @@
 
 	return outvcs
 
+def update_copyrights(pkgdir, files, year, pretend):
+	"""
+	Check list of files for Copyright statements, and update them to
+	year.  The patterns used for replacing copyrights are taken from
+	echangelog.  Only the first lines of each file that start with a
+	hash ('#') are considered, until a line is found that doesn't start
+	with a hash.
+	patches and diffs (identified by .patch and .diff) are excluded
+	"""
+
+	for fn in files:
+		if fn.endswith('.diff') or fn.endswith('.patch'):
+			continue
+
+		fn_path = os.path.join(pkgdir, fn)
+		try:
+			fn_hdl = io.open(_unicode_encode(fn_path,
+				encoding=_encodings['fs'], errors='strict'),
+				mode='r', encoding=_encodings['repo.content'], errors='replace')
+		except EnvironmentError:
+			continue
+
+		orig_header = []
+		new_header = []
+
+		for line in fn_hdl:
+			line_strip = line.strip()
+			orig_header.append(line)
+			if not line_strip or line_strip[:1] != '#':
+				new_header.append(line)
+				break
+
+			# these two regexes are taken from
+			# echangelog update_copyright()
+			line = re.sub(r'^(# Copyright \d+) ',
+				r'\1-%s ' % year, line)
+			line = re.sub(r'^(# Copyright) \d\d\d\d-\d\d\d\d',
+				r'\1 1999-%s' % year, line)
+			new_header.append(line)
+
+		difflines = 0
+		for line in difflib.unified_diff(orig_header, new_header,
+				fromfile=fn_path, tofile=fn_path, n=0):
+			util.writemsg_stdout(line, noiselevel=-1)
+			difflines += 1
+		util.writemsg_stdout("\n", noiselevel=-1)
+
+		# unified diff has three lines to start with
+		if difflines > 3 and not pretend:
+			# write new file with changed header
+			f, fnnew_path = mkstemp()
+			f = io.open(f, mode='w', encoding=_encodings['repo.content'],
+				errors='backslashreplace')
+			for line in new_header:
+				f.write(line);
+			for line in fn_hdl:
+				f.write(line)
+			f.close()
+			try:
+				fn_stat = os.stat(fn_path)
+			except OSError:
+				fn_stat = None
+
+			shutil.move(fnnew_path, fn_path)
+
+			if fn_stat is None:
+				util.apply_permissions(fn_path, mode=0o644)
+			else:
+				util.apply_stat_permissions(fn_path, fn_stat)
+		fn_hdl.close()
+
 def UpdateChangeLog(pkgdir, category, package, new, removed, changed, \
 		msg, pretend, repodir):
-	""" Write an entry to an existing ChangeLog, or create a new one. """
+	"""
+	Write an entry to an existing ChangeLog, or create a new one.
+	Updates copyright year on changed files, and updates the header of
+	ChangeLog with the contents of skel.ChangeLog.
+	"""
 
 	# figure out who to write as
 	if 'GENTOO_COMMITTER_NAME' in os.environ and \
@@ -548,6 +623,12 @@
 		logging.critical(err)
 		return None
 
+	year = time.strftime('%Y')
+	date = time.strftime('%d %b %Y')
+
+	# check modified files and the ChangeLog for copyright updates
+	update_copyrights(pkgdir, new + changed + ['ChangeLog'], year, pretend)
+
 	cl_path = os.path.join(pkgdir, 'ChangeLog')
 	clold_lines = []
 	clnew_lines = []
@@ -578,7 +659,8 @@
 	# construct correct header first
 	try:
 		if clold_file is not None:
-			# retain header from old ChangeLog
+			# retain header from old ChangeLog, its copyright has
+			# already been updated by update_copyrights
 			for line in clold_file:
 				line_strip = line.strip()
 				clold_lines.append(line)
@@ -597,7 +679,7 @@
 				line = line.replace('<CATEGORY>', category)
 				line = line.replace('<PACKAGE_NAME>', package)
 				line = re.sub(r'^(# Copyright \d\d\d\d)-\d\d\d\d ',
-					r'\1-%s ' % time.strftime('%Y'), line)
+					r'\1-%s ' % year, line)
 				clnew_lines.append(line)
 			clnew_lines.append(_unicode_decode('\n'))
 			clskel_file.close()
@@ -651,7 +733,7 @@
 		if clold_file is not None:
 			# clold_lines may contain a saved non-header line
 			# that we want to write first.
-			if clold_lines[-1].strip():
+			if len(clold_lines) > 0 and clold_lines[-1].strip():
 				f.write(clold_lines[-1])
 
 			for line in clold_file:

Re: [gentoo-portage-dev] [PATCH 3 of 3] repoman: update copyright on modified files

[Zac Medico]

On 10/19/2011 12:55 PM, Fabian Groffen wrote:
> +def update_copyrights(pkgdir, files, year, pretend):

I think update_copyrights() could just as well operate on a single file,
and take a single file argument in place of the pkgdir and files
arguments. That would simplify the interface a tiny bit.

> @@ -651,7 +733,7 @@
>  		if clold_file is not None:
>  			# clold_lines may contain a saved non-header line
>  			# that we want to write first.
> -			if clold_lines[-1].strip():
> +			if len(clold_lines) > 0 and clold_lines[-1].strip():
>  				f.write(clold_lines[-1])

Good, this fixes a possible IndexError that I complained about in the
previous patch.
-- 
Thanks,
Zac

[gentoo-portage-dev] [PATCH] repoman: update copyright on modified files

[Fabian Groffen]

To retain the behaviour of echangelog, update the copyrights on modified
files (mostly ebuilds) when necessary.  Also update the ChangeLog's
copyright.

diff --git a/pym/repoman/utilities.py b/pym/repoman/utilities.py
--- a/pym/repoman/utilities.py
+++ b/pym/repoman/utilities.py
@@ -523,9 +523,77 @@
 
 	return outvcs
 
+def update_copyright(fn_path, year, pretend):
+	"""
+	Check file for a Copyright statement, and update its year.  The
+	patterns used for replacing copyrights are taken from echangelog.
+	Only the first lines of each file that start with a hash ('#') are
+	considered, until a line is found that doesn't start with a hash.
+	"""
+
+	try:
+		fn_hdl = io.open(_unicode_encode(fn_path,
+			encoding=_encodings['fs'], errors='strict'),
+			mode='r', encoding=_encodings['repo.content'], errors='replace')
+	except EnvironmentError:
+		return
+
+	orig_header = []
+	new_header = []
+
+	for line in fn_hdl:
+		line_strip = line.strip()
+		orig_header.append(line)
+		if not line_strip or line_strip[:1] != '#':
+			new_header.append(line)
+			break
+
+		# these two regexes are taken from
+		# echangelog update_copyright()
+		line = re.sub(r'^(# Copyright \d+) ',
+			r'\1-%s ' % year, line)
+		line = re.sub(r'^(# Copyright) \d\d\d\d-\d\d\d\d',
+			r'\1 1999-%s' % year, line)
+		new_header.append(line)
+
+	difflines = 0
+	for line in difflib.unified_diff(orig_header, new_header,
+			fromfile=fn_path, tofile=fn_path, n=0):
+		util.writemsg_stdout(line, noiselevel=-1)
+		difflines += 1
+	util.writemsg_stdout("\n", noiselevel=-1)
+
+	# unified diff has three lines to start with
+	if difflines > 3 and not pretend:
+		# write new file with changed header
+		f, fnnew_path = mkstemp()
+		f = io.open(f, mode='w', encoding=_encodings['repo.content'],
+			errors='backslashreplace')
+		for line in new_header:
+			f.write(line);
+		for line in fn_hdl:
+			f.write(line)
+		f.close()
+		try:
+			fn_stat = os.stat(fn_path)
+		except OSError:
+			fn_stat = None
+
+		shutil.move(fnnew_path, fn_path)
+
+		if fn_stat is None:
+			util.apply_permissions(fn_path, mode=0o644)
+		else:
+			util.apply_stat_permissions(fn_path, fn_stat)
+	fn_hdl.close()
+
 def UpdateChangeLog(pkgdir, category, package, new, removed, changed, \
 		msg, pretend, repodir):
-	""" Write an entry to an existing ChangeLog, or create a new one. """
+	"""
+	Write an entry to an existing ChangeLog, or create a new one.
+	Updates copyright year on changed files, and updates the header of
+	ChangeLog with the contents of skel.ChangeLog.
+	"""
 
 	# figure out who to write as
 	if 'GENTOO_COMMITTER_NAME' in os.environ and \
@@ -548,6 +616,16 @@
 		logging.critical(err)
 		return None
 
+	year = time.strftime('%Y')
+	date = time.strftime('%d %b %Y')
+
+	# check modified files and the ChangeLog for copyright updates
+	# patches and diffs (identified by .patch and .diff) are excluded
+	for fn in new + changed:
+		if fn.endswith('.diff') or fn.endswith('.patch'):
+			continue
+		update_copyright(os.path.join(pkgdir, fn), year, pretend)
+
 	cl_path = os.path.join(pkgdir, 'ChangeLog')
 	clold_lines = []
 	clnew_lines = []
@@ -584,8 +662,9 @@
 				clold_lines.append(line)
 				if line_strip[:1] != '#':
 					break
-				if clskel_file is None:
-					clnew_lines.append(line)
+				line = re.sub(r'^(# Copyright) \d\d\d\d-\d\d\d\d',
+					r'\1 1999-%s' % year, line)
+				clnew_lines.append(line)
 				if not line_strip:
 					break
 		elif clskel_file is not None:
@@ -597,7 +676,7 @@
 				line = line.replace('<CATEGORY>', category)
 				line = line.replace('<PACKAGE_NAME>', package)
 				line = re.sub(r'^(# Copyright \d\d\d\d)-\d\d\d\d ',
-					r'\1-%s ' % time.strftime('%Y'), line)
+					r'\1-%s ' % year, line)
 				clnew_lines.append(line)
 			clnew_lines.append(_unicode_decode('\n'))
 			clskel_file.close()

Re: [gentoo-portage-dev] [PATCH] repoman: update copyright on modified files

[Zac Medico]

On 10/20/2011 11:23 AM, Fabian Groffen wrote:
> To retain the behaviour of echangelog, update the copyrights on modified
> files (mostly ebuilds) when necessary.  Also update the ChangeLog's
> copyright.
> 
> diff --git a/pym/repoman/utilities.py b/pym/repoman/utilities.py
> --- a/pym/repoman/utilities.py
> +++ b/pym/repoman/utilities.py
> @@ -523,9 +523,77 @@
>  
>  	return outvcs
>  
> +def update_copyright(fn_path, year, pretend):
> +	"""
> +	Check file for a Copyright statement, and update its year.  The
> +	patterns used for replacing copyrights are taken from echangelog.
> +	Only the first lines of each file that start with a hash ('#') are
> +	considered, until a line is found that doesn't start with a hash.
> +	"""

Thanks, I like that much better. :)
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] repoman: update copyright on modified files

[Mike Gilbert]

On Thu, Oct 20, 2011 at 2:23 PM, Fabian Groffen <grobian@gentoo.org> wrote:
> To retain the behaviour of echangelog, update the copyrights on modified
> files (mostly ebuilds) when necessary.  Also update the ChangeLog's
> copyright.
>
> diff --git a/pym/repoman/utilities.py b/pym/repoman/utilities.py
> --- a/pym/repoman/utilities.py
> +++ b/pym/repoman/utilities.py
> @@ -548,6 +616,16 @@
>                logging.critical(err)
>                return None
>
> +       year = time.strftime('%Y')
> +       date = time.strftime('%d %b %Y')
> +

echangelog calls time.strftime('%d %b %Y', gmtime) so that UTC is always used.

Re: [gentoo-portage-dev] [PATCH] repoman: update copyright on modified files

[Zac Medico]

On 10/20/2011 11:55 AM, Mike Gilbert wrote:
> On Thu, Oct 20, 2011 at 2:23 PM, Fabian Groffen <grobian@gentoo.org> wrote:
>> To retain the behaviour of echangelog, update the copyrights on modified
>> files (mostly ebuilds) when necessary.  Also update the ChangeLog's
>> copyright.
>>
>> diff --git a/pym/repoman/utilities.py b/pym/repoman/utilities.py
>> --- a/pym/repoman/utilities.py
>> +++ b/pym/repoman/utilities.py
>> @@ -548,6 +616,16 @@
>>                logging.critical(err)
>>                return None
>>
>> +       year = time.strftime('%Y')
>> +       date = time.strftime('%d %b %Y')
>> +
> 
> echangelog calls time.strftime('%d %b %Y', gmtime) so that UTC is always used.
> 

Thanks, the existing code is fixed in git now:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=0f261405f63cd09639728da78e70a254cd3c5320
-- 
Thanks,
Zac