From: Chun-Yu Shei <cshei@google.com>
To: gentoo-portage-dev@lists.gentoo.org
Cc: Chun-Yu Shei <cshei@google.com>
Subject: [gentoo-portage-dev] [PATCH 2/3] Add caching to use_reduce function
Date: Fri, 26 Jun 2020 23:34:14 -0700 [thread overview]
Message-ID: <20200627063415.936177-3-cshei@google.com> (raw)
In-Reply-To: <20200627063415.936177-1-cshei@google.com>
This function is called extremely frequently with similar arguments, so
this optimization reduces "emerge -uDvpU --with-bdeps=y @world" runtime from
43.5 -> 34.5s -- a 25.8% speedup.
---
lib/portage/dep/__init__.py | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)
diff --git a/lib/portage/dep/__init__.py b/lib/portage/dep/__init__.py
index 72988357a..df296dd81 100644
--- a/lib/portage/dep/__init__.py
+++ b/lib/portage/dep/__init__.py
@@ -404,6 +404,8 @@ def paren_enclose(mylist, unevaluated_atom=False, opconvert=False):
mystrparts.append(x)
return " ".join(mystrparts)
+_use_reduce_cache = {}
+
def use_reduce(depstr, uselist=(), masklist=(), matchall=False, excludeall=(), is_src_uri=False, \
eapi=None, opconvert=False, flat=False, is_valid_flag=None, token_class=None, matchnone=False,
subset=None):
@@ -440,6 +442,27 @@ def use_reduce(depstr, uselist=(), masklist=(), matchall=False, excludeall=(), i
@rtype: List
@return: The use reduced depend array
"""
+ uselist_key = None
+ masklist_key = None
+ excludeall_key = None
+ subset_key = None
+ if uselist is not None:
+ uselist_key = tuple(uselist)
+ if masklist is not None:
+ masklist_key = tuple(masklist)
+ if excludeall is not None:
+ excludeall_key = tuple(excludeall)
+ if subset is not None:
+ subset_key = tuple(subset)
+ cache_key = (depstr, uselist_key, masklist_key, matchall, excludeall_key, \
+ is_src_uri, eapi, opconvert, flat, is_valid_flag, token_class, \
+ matchnone, subset_key)
+
+ cache_entry = _use_reduce_cache.get(cache_key)
+ if cache_entry is not None:
+ # The list returned by this function may be modified, so return a copy.
+ return cache_entry[:]
+
if isinstance(depstr, list):
if portage._internal_caller:
warnings.warn(_("Passing paren_reduced dep arrays to %s is deprecated. " + \
@@ -767,6 +790,9 @@ def use_reduce(depstr, uselist=(), masklist=(), matchall=False, excludeall=(), i
raise InvalidDependString(
_("Missing file name at end of string"))
+ # The list returned by this function may be modified, so store a copy.
+ _use_reduce_cache[cache_key] = stack[0][:]
+
return stack[0]
def dep_opconvert(deplist):
--
2.27.0.212.ge8ba1cc988-goog
next prev parent reply other threads:[~2020-06-27 6:34 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-06-27 6:34 [gentoo-portage-dev] Add caching to a few commonly used functions Chun-Yu Shei
2020-06-27 6:34 ` [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function Chun-Yu Shei
2020-06-27 11:33 ` Michał Górny
2020-06-29 1:58 ` Sid Spry
2020-07-06 15:26 ` Francesco Riosa
[not found] ` <cdb0d821-67c1-edb6-2cbc-f26eaa0d3d70@veremit.xyz>
2020-07-06 16:10 ` Francesco Riosa
2020-07-06 17:30 ` Chun-Yu Shei
2020-07-06 18:03 ` Zac Medico
2020-07-07 3:41 ` Zac Medico
2020-07-09 7:03 ` Chun-Yu Shei
2020-07-09 7:03 ` [gentoo-portage-dev] [PATCH] Add caching to use_reduce, vercmp, and catpkgsplit Chun-Yu Shei
2020-07-12 21:46 ` Zac Medico
2020-07-13 6:30 ` [gentoo-portage-dev] [PATCH] Add caching to use_reduce, Chun-Yu Shei
2020-07-13 6:30 ` [gentoo-portage-dev] [PATCH] Add caching to use_reduce, vercmp, and catpkgsplit Chun-Yu Shei
2020-07-13 17:28 ` Zac Medico
2020-07-13 18:54 ` Ulrich Mueller
2020-07-13 19:04 ` Chun-Yu Shei
2020-07-13 19:24 ` Ulrich Mueller
2020-07-09 21:04 ` [gentoo-portage-dev] [PATCH 1/3] Add caching to catpkgsplit function Alec Warner
2020-07-09 21:06 ` Chun-Yu Shei
2020-07-09 21:13 ` Alec Warner
2020-06-27 6:34 ` Chun-Yu Shei [this message]
2020-06-27 6:34 ` [gentoo-portage-dev] [PATCH 3/3] Add partial caching to match_from_list Chun-Yu Shei
2020-06-27 7:35 ` [gentoo-portage-dev] Add caching to a few commonly used functions Fabian Groffen
2020-06-27 7:43 ` Chun-Yu Shei
2020-06-27 8:31 ` Kent Fredric
2020-06-28 3:00 ` Zac Medico
2020-06-28 3:12 ` Michał Górny
2020-06-28 3:42 ` Zac Medico
2020-06-28 5:30 ` Michał Górny
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200627063415.936177-3-cshei@google.com \
--to=cshei@google.com \
--cc=gentoo-portage-dev@lists.gentoo.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox