From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 920381384B4 for ; Fri, 13 Nov 2015 17:52:36 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 6D46621C064; Fri, 13 Nov 2015 17:52:32 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 185D521C064 for ; Fri, 13 Nov 2015 17:52:32 +0000 (UTC) Received: from oystercatcher.gentoo.org (oystercatcher.gentoo.org [148.251.78.52]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTPS id 2D54C34074E for ; Fri, 13 Nov 2015 17:52:31 +0000 (UTC) Received: from localhost.localdomain (localhost [127.0.0.1]) by oystercatcher.gentoo.org (Postfix) with ESMTP id 3200FA4F for ; Fri, 13 Nov 2015 17:52:27 +0000 (UTC) From: "Zac Medico" To: gentoo-commits@lists.gentoo.org Content-Transfer-Encoding: 8bit Content-type: text/plain; charset=UTF-8 Reply-To: gentoo-dev@lists.gentoo.org, "Zac Medico" Message-ID: <1447437121.1ac5e7f9b92b33f76cdb1e6e6f7e5bba42f2b496.zmedico@gentoo> Subject: [gentoo-commits] proj/portage:master commit in: bin/ X-VCS-Repository: proj/portage X-VCS-Files: bin/egencache X-VCS-Directories: bin/ X-VCS-Committer: zmedico X-VCS-Committer-Name: Zac Medico X-VCS-Revision: 1ac5e7f9b92b33f76cdb1e6e6f7e5bba42f2b496 X-VCS-Branch: master Date: Fri, 13 Nov 2015 17:52:27 +0000 (UTC) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-commits@lists.gentoo.org X-Archives-Salt: 3dc024e3-c040-4a3c-b9bb-2b77bb8ec25e X-Archives-Hash: 8a63e3f78c9bff31c39f3b7d1b6829b8 commit: 1ac5e7f9b92b33f76cdb1e6e6f7e5bba42f2b496 Author: Zac Medico gentoo org> AuthorDate: Fri Nov 13 07:18:38 2015 +0000 Commit: Zac Medico gentoo org> CommitDate: Fri Nov 13 17:52:01 2015 +0000 URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=1ac5e7f9 GenChangeLogs: parallelize remaining git calls, scale linearly (bug 565540) Move all git calls to the subprocesses, so performance scales linearly with --jobs. X-Gentoo-Bug: 565540 X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=565540 Acked-by: Alexander Berntsen gentoo.org> bin/egencache | 33 ++++++++++++++++++--------------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/bin/egencache b/bin/egencache index b44ad11..1cc2f3d 100755 --- a/bin/egencache +++ b/bin/egencache @@ -781,6 +781,23 @@ class GenChangeLogs(object): encoding=_encodings['stdio'], errors='strict') def generate_changelog(self, cp): + + os.chdir(os.path.join(self._repo_path, cp)) + # Determine whether ChangeLog is up-to-date by comparing + # the newest commit timestamp with the ChangeLog timestamp. + lmod = self.grab(['git', self._work_tree, 'log', '--format=%ct', '-1', '.']) + if not lmod: + # This cp has not been added to the repo. + return + + try: + cmod = os.stat('ChangeLog').st_mtime + except OSError: + cmod = 0 + + if float(cmod) >= float(lmod): + return + try: output = io.open(self._changelog_output, mode='w', encoding=_encodings['repo.content'], @@ -913,21 +930,7 @@ class GenChangeLogs(object): return for cp in self._portdb.cp_all(): - os.chdir(os.path.join(self._repo_path, cp)) - # Determine whether ChangeLog is up-to-date by comparing - # the newest commit timestamp with the ChangeLog timestamp. - lmod = self.grab(['git', self._work_tree, 'log', '--format=%ct', '-1', '.']) - if not lmod: - # This cp has not been added to the repo. - continue - - try: - cmod = os.stat('ChangeLog').st_mtime - except OSError: - cmod = 0 - - if float(cmod) < float(lmod): - yield AsyncFunction(target=self.generate_changelog, args=[cp]) + yield AsyncFunction(target=self.generate_changelog, args=[cp]) def run(self): return run_main_scheduler(