From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 02D7B15808B for ; Fri, 16 Feb 2024 20:39:47 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 158A2E29D7; Fri, 16 Feb 2024 20:39:46 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [IPv6:2001:470:ea4a:1:5054:ff:fec7:86e4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id E9920E29D7 for ; Fri, 16 Feb 2024 20:39:45 +0000 (UTC) Received: from oystercatcher.gentoo.org (oystercatcher.gentoo.org [148.251.78.52]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTPS id 19221341214 for ; Fri, 16 Feb 2024 20:39:45 +0000 (UTC) Received: from localhost.localdomain (localhost [IPv6:::1]) by oystercatcher.gentoo.org (Postfix) with ESMTP id 4FCA214B7 for ; Fri, 16 Feb 2024 20:39:43 +0000 (UTC) From: "Sam James" To: gentoo-commits@lists.gentoo.org Content-Transfer-Encoding: 8bit Content-type: text/plain; charset=UTF-8 Reply-To: gentoo-dev@lists.gentoo.org, "Sam James" Message-ID: <1708114369.6aef2c5968d66e91dc083820db489d85697f3587.sam@gentoo> Subject: [gentoo-commits] proj/gentoolkit:master commit in: pym/gentoolkit/ X-VCS-Repository: proj/gentoolkit X-VCS-Files: pym/gentoolkit/dependencies.py X-VCS-Directories: pym/gentoolkit/ X-VCS-Committer: sam X-VCS-Committer-Name: Sam James X-VCS-Revision: 6aef2c5968d66e91dc083820db489d85697f3587 X-VCS-Branch: master Date: Fri, 16 Feb 2024 20:39:43 +0000 (UTC) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-commits@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply X-Archives-Salt: e852ed44-2679-483b-b27b-c1f90383bd71 X-Archives-Hash: bacf7acb357c0bd1357b58ee9f071742 commit: 6aef2c5968d66e91dc083820db489d85697f3587 Author: John Turner gmail com> AuthorDate: Tue Feb 13 20:45:51 2024 +0000 Commit: Sam James gentoo org> CommitDate: Fri Feb 16 20:12:49 2024 +0000 URL: https://gitweb.gentoo.org/proj/gentoolkit.git/commit/?id=6aef2c59 dependencies.py: replace hand rolled depcache with functools.cache functools.cache caches the output of functions "automatically" without requiring any manual management of a cache value. When used on class methods, the cache is associated with each class instance and only lives as long as the instance does. The Dependencies.graph_reverse_depends method used a dict to cache the output from pkg.get_all_depends. The get_all_depends method involves calling portage's aux_get and parsing the DEPEND string that is returned by it. This dict has been removed and replaced with functools.cache. The graph_reverse_depends method did not cache the output of the "raw=True" get_all_depends calls. This "raw" output is the literal string value for the pkgdeps *DEPEND variables as returned by aux_get. Searching this for a category/package sub-string allows quickly ruling out non-matching pkgdeps, which allows skipping parsing the DEPEND string into a list of Atoms. Using functools cache the method that fetches the raw DEPEND string massively improves performance for graph_reverse_depends when searching for indirect reverse dependencies. "equery depends --indirect emacs" runtime is ~2s with the raw value being cached, and ~60s without. Searching for only direct reverse dependencies does not utilize the cache and does not see any chance in runtime for me. Signed-off-by: John Turner gmail.com> Closes: https://github.com/gentoo/gentoolkit/pull/44 Signed-off-by: Sam James gentoo.org> pym/gentoolkit/dependencies.py | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/pym/gentoolkit/dependencies.py b/pym/gentoolkit/dependencies.py index 4564d8c..3bbc757 100644 --- a/pym/gentoolkit/dependencies.py +++ b/pym/gentoolkit/dependencies.py @@ -12,6 +12,7 @@ __all__ = ("Dependencies",) # ======= import itertools +from functools import cache from enum import StrEnum from typing import List, Dict @@ -102,9 +103,11 @@ class Dependencies(Query): except portage.exception.InvalidPackageName as err: raise errors.GentoolkitInvalidCPV(err) + @cache def get_raw_depends(self) -> str: return self._get_depend([depkind for depkind in DependencyKind], raw=True) + @cache def get_depends(self) -> Dict[DependencyKind, List[Atom]]: depends = dict() for depkind in DependencyKind: @@ -189,7 +192,6 @@ class Dependencies(Query): printer_fn=None, # The rest of these are only used internally: depth=0, - depcache=None, seen=None, result=None, ): @@ -233,8 +235,6 @@ class Dependencies(Query): ) raise errors.GentoolkitFatalError(err % (self.__class__.__name__,)) - if depcache is None: - depcache = dict() if seen is None: seen = set() if result is None: @@ -250,12 +250,8 @@ class Dependencies(Query): # us the work of instantiating a whole Atom() for *every* # dependency of *every* package in pkgset. continue - try: - all_depends = depcache[pkgdep] - except KeyError: - all_depends = pkgdep.get_all_depends() - depcache[pkgdep] = all_depends + all_depends = pkgdep.get_all_depends() dep_is_displayed = False for dep in all_depends: # TODO: Add ability to determine if dep is enabled by USE flag. @@ -284,7 +280,6 @@ class Dependencies(Query): only_direct=only_direct, printer_fn=printer_fn, depth=depth + 1, - depcache=depcache, seen=seen, result=result, )