From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 38F00138334 for ; Sun, 20 Oct 2019 09:26:05 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 74826E089F; Sun, 20 Oct 2019 09:26:04 +0000 (UTC) Received: from smtp.gentoo.org (woodpecker.gentoo.org [IPv6:2001:470:ea4a:1:5054:ff:fec7:86e4]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 4BD63E089F for ; Sun, 20 Oct 2019 09:26:04 +0000 (UTC) Received: from oystercatcher.gentoo.org (unknown [IPv6:2a01:4f8:202:4333:225:90ff:fed9:fc84]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTPS id E134B34C09B for ; Sun, 20 Oct 2019 09:26:02 +0000 (UTC) Received: from localhost.localdomain (localhost [IPv6:::1]) by oystercatcher.gentoo.org (Postfix) with ESMTP id C5CC676A for ; Sun, 20 Oct 2019 09:26:00 +0000 (UTC) From: "Michał Górny" To: gentoo-commits@lists.gentoo.org Content-Transfer-Encoding: 8bit Content-type: text/plain; charset=UTF-8 Reply-To: gentoo-dev@lists.gentoo.org, "Michał Górny" Message-ID: <1571563510.9115e1f6ba35cdcd85f2292dc293f0696caa8f12.mgorny@gentoo> Subject: [gentoo-commits] proj/portage:master commit in: lib/portage/package/ebuild/ X-VCS-Repository: proj/portage X-VCS-Files: lib/portage/package/ebuild/fetch.py X-VCS-Directories: lib/portage/package/ebuild/ X-VCS-Committer: mgorny X-VCS-Committer-Name: Michał Górny X-VCS-Revision: 9115e1f6ba35cdcd85f2292dc293f0696caa8f12 X-VCS-Branch: master Date: Sun, 20 Oct 2019 09:26:00 +0000 (UTC) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-commits@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply X-Archives-Salt: 8e7b7bb9-104e-41be-9826-1049039d4cac X-Archives-Hash: 4bd47b526e438e65cf0247ae65a6f146 commit: 9115e1f6ba35cdcd85f2292dc293f0696caa8f12 Author: Michał Górny gentoo org> AuthorDate: Fri Oct 18 07:23:02 2019 +0000 Commit: Michał Górny gentoo org> CommitDate: Sun Oct 20 09:25:10 2019 +0000 URL: https://gitweb.gentoo.org/proj/portage.git/commit/?id=9115e1f6 fetch: Use distfile fetching method to get layout.conf Rewrite the layout.conf getter to reuse the standard fetch() method rather than using urlopen(). While at it, fix negative cache elision to apply to memory cache as well (and not get written to disk if next mirror was fine). Most importantly, this ensures that we respect FETCHCOMMAND while fetching layout.conf, and so layout.conf is fetched the same way normal distfiles are. With some uncommon configurations, the previous disjoint logic might have resulted in one of the fetches failing while the other succeeded. This also adds some nice verbosity. If mirror connection takes a while, the user sees that rather than having Portage wait silently. Bug: https://bugs.gentoo.org/697566 Reviewed-by: Zac Medico gentoo.org> Signed-off-by: Michał Górny gentoo.org> lib/portage/package/ebuild/fetch.py | 33 ++++++++++++++++----------------- 1 file changed, 16 insertions(+), 17 deletions(-) diff --git a/lib/portage/package/ebuild/fetch.py b/lib/portage/package/ebuild/fetch.py index 05de12740..cedf12b19 100644 --- a/lib/portage/package/ebuild/fetch.py +++ b/lib/portage/package/ebuild/fetch.py @@ -382,7 +382,7 @@ class MirrorLayoutConfig(object): return ret -def get_mirror_url(mirror_url, filename, cache_path=None): +def get_mirror_url(mirror_url, filename, mysettings, cache_path=None): """ Get correct fetch URL for a given file, accounting for mirror layout configuration. @@ -408,23 +408,22 @@ def get_mirror_url(mirror_url, filename, cache_path=None): if ts >= time.time() - 86400: mirror_conf.deserialize(data) else: + tmpfile = '.layout.conf.%s' % urlparse(mirror_url).hostname try: - f = urlopen(mirror_url + '/distfiles/layout.conf') - try: - data = io.StringIO(f.read().decode('utf8')) - finally: - f.close() - - mirror_conf.read_from_file(data) + if fetch({tmpfile: (mirror_url + '/distfiles/layout.conf',)}, + mysettings, force=1, try_mirrors=0): + tmpfile = os.path.join(mysettings['DISTDIR'], tmpfile) + mirror_conf.read_from_file(tmpfile) + else: + raise IOError() except (ConfigParserError, IOError, UnicodeDecodeError): - # Do not cache negative results. - cache_path = None - - cache[mirror_url] = (time.time(), mirror_conf.serialize()) - if cache_path is not None: - f = atomic_ofstream(cache_path, 'w') - json.dump(cache, f) - f.close() + pass + else: + cache[mirror_url] = (time.time(), mirror_conf.serialize()) + if cache_path is not None: + f = atomic_ofstream(cache_path, 'w') + json.dump(cache, f) + f.close() return (mirror_url + "/distfiles/" + mirror_conf.get_best_supported_layout().get_path(filename)) @@ -656,7 +655,7 @@ def fetch(myuris, mysettings, listonly=0, fetchonly=0, mirror_cache = None for l in locations: filedict[myfile].append(functools.partial( - get_mirror_url, l, myfile, mirror_cache)) + get_mirror_url, l, myfile, mysettings, mirror_cache)) if myuri is None: continue if myuri[:9]=="mirror://":