[gentoo-portage-dev] [PATCH] VdbMetadataDelta.applyDelta: remove replaced versions (bug 547532)

[gentoo-portage-dev] [PATCH] VdbMetadataDelta.applyDelta: remove replaced versions (bug 547532)

[Zac Medico]

Since commit d800d224ab38c0f524d3fe858ebe201cbfa903c1, emerge --search
could randomly report incorrect results for the installed version
due to the replaced version of a given slot remaining in the cache.

Fixes: d800d224ab38 ("Log changes between vdb_metadata.pickle updates")
X-Gentoo-Bug: 547532
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=547532
---
 pym/_emerge/search.py                  | 10 ++++++++--
 pym/portage/dbapi/_VdbMetadataDelta.py | 22 +++++++++++++++++++++-
 2 files changed, 29 insertions(+), 3 deletions(-)

diff --git a/pym/_emerge/search.py b/pym/_emerge/search.py
index e7f6f44..5a8143c 100644
--- a/pym/_emerge/search.py
+++ b/pym/_emerge/search.py
@@ -1,4 +1,4 @@
-# Copyright 1999-2014 Gentoo Foundation
+# Copyright 1999-2015 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
 from __future__ import unicode_literals
@@ -442,7 +442,13 @@ class search(object):
 	def getInstallationStatus(self,package):
 		installed_package = self._vardb.match(package)
 		if installed_package:
-			installed_package = installed_package[-1]
+			try:
+				self._vardb.match_unordered
+			except AttributeError:
+				installed_package = installed_package[-1]
+			else:
+				installed_package = portage.best(installed_package)
+
 		else:
 			installed_package = ""
 		result = ""
diff --git a/pym/portage/dbapi/_VdbMetadataDelta.py b/pym/portage/dbapi/_VdbMetadataDelta.py
index 3e3ff18..2dbb07a 100644
--- a/pym/portage/dbapi/_VdbMetadataDelta.py
+++ b/pym/portage/dbapi/_VdbMetadataDelta.py
@@ -1,4 +1,4 @@
-# Copyright 2014 Gentoo Foundation
+# Copyright 2014-2015 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
 import errno
@@ -8,6 +8,7 @@ import os
 
 from portage import _encodings
 from portage.util import atomic_ofstream
+from portage.versions import cpv_getkey
 
 class VdbMetadataDelta(object):
 
@@ -138,10 +139,12 @@ class VdbMetadataDelta(object):
 
 	def applyDelta(self, data):
 		packages = self._vardb._aux_cache["packages"]
+		added_slots = {}
 		for delta in data["deltas"]:
 			cpv = delta["package"] + "-" + delta["version"]
 			event = delta["event"]
 			if event == "add":
+				added_slots[cpv] = delta
 				# Use aux_get to populate the cache
 				# for this cpv.
 				if cpv not in packages:
@@ -151,3 +154,20 @@ class VdbMetadataDelta(object):
 						pass
 			elif event == "remove":
 				packages.pop(cpv, None)
+
+		# Remove replaced versions from updated slots
+		for cached_cpv, (mtime, metadata) in list(packages.items()):
+			if cached_cpv in added_slots:
+				continue
+			replaced = False
+			for cpv, delta in added_slots.items():
+				if (cached_cpv.startswith(delta["package"]) and
+					metadata.get("SLOT") == delta["slot"] and
+					cpv_getkey(cached_cpv) == delta["package"]):
+					replaced = True
+					break
+			if replaced:
+				del packages[cached_cpv]
+				del added_slots[cpv]
+				if not added_slots:
+					break
-- 
2.3.5

Re: [gentoo-portage-dev] [PATCH] VdbMetadataDelta.applyDelta: remove replaced versions (bug 547532)

[Alexander Berntsen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 24/04/15 18:32, Zac Medico wrote:
> -			installed_package = installed_package[-1]
> +			try:
> +				self._vardb.match_unordered
> +			except AttributeError:
> +				installed_package = installed_package[-1]
> +			else:
> +				installed_package = portage.best(installed_package)
> +
This probably makes sense, but I don't understand how this helps to
alleviate the problem. Would you mind explaining it?

The rest of the patch looks OK, even if I dislike the strong imperative
nature of the replaced versions removal.
- -- 
Alexander
bernalex@gentoo.org
https://secure.plaimi.net/~alexander
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCgAGBQJVSx0VAAoJENQqWdRUGk8B3/AQAJnM8XhSS3ps1mDIIZZ656bF
cNlnkzzBnX7OkyhjFrSISSkov23I81+yb/3nsiR7/iT4N8TdzxvkX2nPOKWysVxE
K8FsF56KQny6vFEqtztRO715Nm92/V62zP0L1jDaG7snml4SFnc/+SWZRXimJsUr
wlgYXDvpaEpc6dh1U+nA3kbqcyiic4EYIWrd+LXY9WV0wPMJrjXNHw30gYEwoPpM
JonoJEjEyen8Y7KEQoIbBrMht/S4dCX+J2mnOgne4fod4oFWeNfwdgZ1QZ6HrcyA
WDR2akmX/4azcOIAXkm83GXY4VaCuYgKWqCBAuXcxALoEQBP9diCn0XcVuoTSAwU
z5YvDvuKvgx3uC1kLWLEUU7bwfbJ7MKHDKjal3rLaR9rYEt9jt6e4na/gpl8wiIN
ECILO64baHEnWz8mbKpq6E9e9Wte8mE87ACRpg/38P31EinTKCt1y/F9/eS/frjF
C2OwUoOyfsZyPlu4fLdF7bZbC0HvPYt4e+obrA06bqaY2ubSyS1qA1Xz41TBgELI
IeHcq+t+i5lXSpJTH0mMjtWyj7GhuB5ZxBIXAoly/p07y37FW4hz5sGGUryfEOVr
+HPV3iUAjUeCfaE9JzzJyWIJYzgKyYCBbi9V1eV4VpEjWppBkxzJgjNPF1v4xyh/
6nDTC0txOZMzWq6avA4Q
=WW8m
-----END PGP SIGNATURE-----

Re: [gentoo-portage-dev] [PATCH] VdbMetadataDelta.applyDelta: remove replaced versions (bug 547532)

[Alexander Berntsen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

I see this is in master now anyway. I still wouldn't mind an
explanation though, just for my own understanding's sake.
- -- 
Alexander
bernalex@gentoo.org
https://secure.plaimi.net/~alexander
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCgAGBQJVSygKAAoJENQqWdRUGk8BTMsP/RwpTe6fYrC/HiGNMhF9lXcd
Pc1C1bsDpdt5AYWCw5A/5QyR1btlr0emL615nJpu53bflOHWExadVmpAwFut2/oH
F5PCK88EdYYkDDJLnLySD894e+xFF4uvgMr1fhACZSNK2JVcsSpKKKzPpdSZ/fH0
+Peuj5uYGHUzo8bYz1ky4H3XZwFbYUjoFuu/pf85ojCGjG57BA1bEi64hqhgYT/F
ctX/o6w9V/Z42L1ook3TxtP+QQxcYRvdP6OL0INFxrgkGSfwda4wH1c7F3vI4uv6
ZNcZQbfIzc49On2Sz9ML2q5KHS68SBYbUiLrfchsizNyn2tGop1tz3WHkRfgsZSN
lR+S8viYrdSbJQ/keFcDNeVBIBXNeYk+R+GyvdvJ6whiuW8Rjd9Lnxyxf9FBHHRT
/7yNltkuIzPMOoh+3pJG3uF6RDAc/UVm/bLRAimlO+634vZEZgJSMpejmjCT7TfK
tBqoqTC1qA/JpiN0antgHo11i4H98ZQBBormXU+J4CXnrAG8/12RffqvGi++jXlx
1hhYadafjgB5+32OehHUIyB3fVfl+GGnl4yY7gyRjylgRfyLYkRA9FzvjTB9GC9k
KAPJ3blkUINmsV0GPdMp3XCLU7KnhlXO2AH0EKJcqPmExHVCo+twhF0bGz1MBdu3
z5dqmbOvr9M3NbCY5FlG
=Gu7y
-----END PGP SIGNATURE-----

Re: [gentoo-portage-dev] [PATCH] VdbMetadataDelta.applyDelta: remove replaced versions (bug 547532)

[Zac Medico]

On 05/07/2015 01:06 AM, Alexander Berntsen wrote:
> On 24/04/15 18:32, Zac Medico wrote:
>> -			installed_package = installed_package[-1]
>> +			try:
>> +				self._vardb.match_unordered
>> +			except AttributeError:
>> +				installed_package = installed_package[-1]
>> +			else:
>> +				installed_package = portage.best(installed_package)
>> +
> This probably makes sense, but I don't understand how this helps to
> alleviate the problem. Would you mind explaining it?

This hunk just fixes to ensure that it shows the highest version in the
case were there are multiple slot installed. Otherwise, since
IndexedVardb returns unordered match results, it could randomly select a
lower version from another slot.

> The rest of the patch looks OK, even if I dislike the strong imperative
> nature of the replaced versions removal.

I'm open to suggestions if you can think of a nicer way to express it.
It certainly be nicer if the data from vdb_metadata.pickle had the
packages indexed by ${CATEGORY}/${PN}, so that the sequential search
would not be necessary. We should do that later if we decide to modify
the vdb_metadata.pickle format in a backward-incompatible way.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] VdbMetadataDelta.applyDelta: remove replaced versions (bug 547532)

[Zac Medico]

On 05/07/2015 11:43 AM, Zac Medico wrote:
> It certainly be nicer if the data from vdb_metadata.pickle had the
> packages indexed by ${CATEGORY}/${PN}, so that the sequential search
> would not be necessary. We should do that later if we decide to modify
> the vdb_metadata.pickle format in a backward-incompatible way.
d
Actually, we can add as many additional indexes as we want to the
pickle, and older portage would still be able the read the existing
structure if we keep it compatible. It's not terribly important to
maintain this sort of backward compatibility though. The worst case is
that the old version of portage has to re-read /var/db/pkg and generate
a new pickle.
-- 
Thanks,
Zac