[gentoo-portage-dev] [PATCH] GenChangeLogs: parallelize remaining git calls, scale linearly (bug 565540)

[gentoo-portage-dev] [PATCH] GenChangeLogs: parallelize remaining git calls, scale linearly (bug 565540)

[Zac Medico]

Move all git calls to the subprocesses, so performance scales linearly
with --jobs.

X-Gentoo-Bug: 565540
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=565540
---
 bin/egencache | 33 ++++++++++++++++++---------------
 1 file changed, 18 insertions(+), 15 deletions(-)

diff --git a/bin/egencache b/bin/egencache
index b44ad11..1cc2f3d 100755
--- a/bin/egencache
+++ b/bin/egencache
@@ -781,6 +781,23 @@ class GenChangeLogs(object):
 				encoding=_encodings['stdio'], errors='strict')
 
 	def generate_changelog(self, cp):
+
+		os.chdir(os.path.join(self._repo_path, cp))
+		# Determine whether ChangeLog is up-to-date by comparing
+		# the newest commit timestamp with the ChangeLog timestamp.
+		lmod = self.grab(['git', self._work_tree, 'log', '--format=%ct', '-1', '.'])
+		if not lmod:
+			# This cp has not been added to the repo.
+			return
+
+		try:
+			cmod = os.stat('ChangeLog').st_mtime
+		except OSError:
+			cmod = 0
+
+		if float(cmod) >= float(lmod):
+			return
+
 		try:
 			output = io.open(self._changelog_output,
 				mode='w', encoding=_encodings['repo.content'],
@@ -913,21 +930,7 @@ class GenChangeLogs(object):
 			return
 
 		for cp in self._portdb.cp_all():
-			os.chdir(os.path.join(self._repo_path, cp))
-			# Determine whether ChangeLog is up-to-date by comparing
-			# the newest commit timestamp with the ChangeLog timestamp.
-			lmod = self.grab(['git', self._work_tree, 'log', '--format=%ct', '-1', '.'])
-			if not lmod:
-				# This cp has not been added to the repo.
-				continue
-
-			try:
-				cmod = os.stat('ChangeLog').st_mtime
-			except OSError:
-				cmod = 0
-
-			if float(cmod) < float(lmod):
-				yield AsyncFunction(target=self.generate_changelog, args=[cp])
+			yield AsyncFunction(target=self.generate_changelog, args=[cp])
 
 	def run(self):
 		return run_main_scheduler(
-- 
2.4.9

Re: [gentoo-portage-dev] [PATCH] GenChangeLogs: parallelize remaining git calls, scale linearly (bug 565540)

[Alexander Berntsen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

It's uh probably OK? If someone else that actually know enough about
what's going on in this area of the code wants to chime in, then by
all means, go ahead.

- -- 
Alexander
bernalex@gentoo.org
https://secure.plaimi.net/~alexander
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCgAGBQJWRde/AAoJENQqWdRUGk8BOFAP/jrMMUE5oehCPi9JKV+bGxm7
qkVpA5lYcaRoRvmOh3XDZAwMaX6egACgvXDXkVXZOBQFrPhj5mV85H4ChUCr76VW
09HrDuDpsw6byTSKXzlpBTEAcc5T6/EGEdN1XzcWj6ZJoqy6Y1APQkauiO1g4Biz
6EPaMnkOcJd6PXlYZQnoANmDIJxB0YK6flObvIsLKFgHrikPqtbXl/o9AFLoLHnm
oEHkuGHJOam6vsf54h6mH/1JAiJl9BCheyhgdeELfv5VsI1EhzhcMVtn1oOzhmvF
e9byPJySiHt59rnRKk4A0eYrD+KQoeh5AH6fdeT5JzoEKHfQJvmQcXMNumhgLv7N
sgLZO/EhomjOdIX/9wq4oU7kPaDXdZ/8iyF3YsybDVSbVq+zTIHJcGTC5LIORmao
3N+fAh+G0VSF07nL3N0IxRVUmWDhtGd48YWqm8Fswi6gy/PRnHdULS0EtfZ7I1Ig
SnKm0TvqN2yx7Jk1tq0fn0jprEqQTiEdM0hPSug2QgXIf2zSi8tTfRzdlBYWco1w
XgSFeYQx2NobfrMgqiHKQFelt21VXzUH9MiZS/4RARJ/2dpg6jvbaEZ2dz1eFT/i
kiaZ+HaGikrRMI7Qt9pbt2oQrWOsqVGCMxGcRGCRLcQxeMyDx/jQRvAQC1fgcBAU
zpPvdyfHjkGmZKM3CWrx
=xmdN
-----END PGP SIGNATURE-----