[gentoo-portage-dev] equery: deprecate --category filtering in belongs

[gentoo-portage-dev] equery: deprecate --category filtering in belongs

[Douglas Anderson]

Hi, does anyone use --category filtering in equery belongs? I want to
get rid of it, or at least deprecate it. My reasoning:

* We use 'equery belongs' when don't know to what package a file
belongs. Even if we have a suspicion, most users would have to look up
the category of the package before typing it in.
* Even if you happen to know the exact category of the package that
installed the file (why are you using belongs?), typing
--category=app-portage takes more time than is saved by filtering by
category (about 5 seconds more by my unscientific test).

Even in a script setting, I see no use for this. The time saved is minuscule:

$ time equery belongs /usr/bin/equery --category app-portage
[ Searching for file(s) /usr/bin/equery in app-portage... ]
app-portage/gentoolkit-0.2.4.2-r1 (/usr/bin/equery)

real    0m4.002s
user    0m3.680s
sys     0m0.076s
$ time equery belongs /usr/bin/equery
[ Searching for file(s) /usr/bin/equery in *... ]
app-portage/gentoolkit-0.2.4.2-r1 (/usr/bin/equery)

real    0m4.205s
user    0m3.738s
sys     0m0.102s


* Lastly, it's confusing. belongs takes a filename as input, but you
can filter by category? All other modules that take can filter by
category take a pkgspec.

-Doug

Re: [gentoo-portage-dev] equery: deprecate --category filtering in belongs

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 1965 bytes --]

On Sun, Feb 08, 2009 at 02:07:08PM +0900, Douglas Anderson wrote:
> Hi, does anyone use --category filtering in equery belongs? I want to
> get rid of it, or at least deprecate it. My reasoning:
> 
> * We use 'equery belongs' when don't know to what package a file
> belongs. Even if we have a suspicion, most users would have to look up
> the category of the package before typing it in.
> * Even if you happen to know the exact category of the package that
> installed the file (why are you using belongs?), typing
> --category=app-portage takes more time than is saved by filtering by
> category (about 5 seconds more by my unscientific test).
> 
> Even in a script setting, I see no use for this. The time saved is minuscule:
> 
> $ time equery belongs /usr/bin/equery --category app-portage
> [ Searching for file(s) /usr/bin/equery in app-portage... ]
> app-portage/gentoolkit-0.2.4.2-r1 (/usr/bin/equery)
> 
> real    0m4.002s
> user    0m3.680s
> sys     0m0.076s
> $ time equery belongs /usr/bin/equery
> [ Searching for file(s) /usr/bin/equery in *... ]
> app-portage/gentoolkit-0.2.4.2-r1 (/usr/bin/equery)
> 
> real    0m4.205s
> user    0m3.738s
> sys     0m0.102s

You should do profiling, and *verify* category actually works; it 
should be able to make a world of difference.  Case in point via 
pkgcore-

ferringb@beast ~ $ time pquery --owns /usr/bin/equery
app-portage/gentoolkit-0.2.4.2-r1

real    0m1.162s
user    0m1.142s
sys     0m0.020s
ferringb@beast ~ $ time pquery --owns /usr/bin/equery 'app-portage/*'
app-portage/gentoolkit-0.2.4.2-r1

real    0m0.168s
user    0m0.156s
sys     0m0.011s


Yes it's not equery, but I *know* pkgcores implementation works thus 
using it to point out the speed difference if implemented correctly.

It would probably help if the category support was enabled in the 
code, also (quick check of equery shows the cat filtering is 
disabled).

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-portage-dev] equery: deprecate --category filtering in belongs

[Brian Harring]

[-- Attachment #1.1: Type: text/plain, Size: 1709 bytes --]

patch attached against 0.2.4.2-r1; rough stats follow;

full cold cache

[ Searching for file(s) /usr/bin/equery in *... ]
app-portage/gentoolkit-0.2.4.2-r1 (/usr/bin/equery)

real    0m10.320s
user    0m0.733s
sys     0m0.162s

[ Searching for file(s) /usr/bin/equery in app-portage... ]
app-portage/gentoolkit-0.2.4.2-r1 (/usr/bin/equery)

real    0m8.512s
user    0m0.315s
sys     0m0.124s

That particular cold cache is a *full* cold cache; not the best test 
imo since most users have at least some chunks of portage 
configuration/python cached.


Cold cache, with equery --help primer to warm the cache; CONTENTS 
(what belongs operates on) is still out of the cache however making 
this a bit more likely use scenario.

[ Searching for file(s) /usr/bin/equery in *... ]
app-portage/gentoolkit-0.2.4.2-r1 (/usr/bin/equery)

real    0m2.335s
user    0m0.670s
sys     0m0.050s

[ Searching for file(s) /usr/bin/equery in app-portage... ]
app-portage/gentoolkit-0.2.4.2-r1 (/usr/bin/equery)

real    0m0.391s
user    0m0.248s
sys     0m0.046s

Pretty heavy difference, no?


hotcache:
[ Searching for file(s) /usr/bin/equery in *... ]
app-portage/gentoolkit-0.2.4.2-r1 (/usr/bin/equery)

real    0m0.710s
user    0m0.661s
sys     0m0.047s

[ Searching for file(s) /usr/bin/equery in app-portage... ]
app-portage/gentoolkit-0.2.4.2-r1 (/usr/bin/equery)

real    0m0.291s
user    0m0.237s
sys     0m0.053s


Mind you this isn't multiple runs, so the numbers are rough 
approximations- that said they're fairly representative.

Strongly suggest y'all keep category support (although I'll keep on 
using pquery instead ;).

Cheers,
~harring

[-- Attachment #1.2: equery-enable-categories.patch --]
[-- Type: text/plain, Size: 1082 bytes --]

--- /usr/bin/equery	2009-02-07 13:16:51.000000000 -0800
+++ /home/ferringb/equery	2009-02-07 14:15:52.000000000 -0800
@@ -345,7 +345,7 @@
 				need_help = 1
 				break
 			elif x in ["-c", "--category"]:
-				opts["category"] = args[i+1]
+				opts["category"] = args[i+1].rstrip("/")
 				skip = 1
 			elif x in ["-e", "--earlyout"]:
 				opts["earlyOut"] = 1
@@ -383,16 +383,14 @@
 			die(2, "The query '" + pp.regexpquery(q) + "' does not appear to be a valid regular expression")
 
 		# Pick out only selected categories
-		cat = opts["category"]
-		filter_fn = None
-		if cat != "*":
-			filter_fn = lambda x: x.find(cat+"/")==0
 
+		cat = opts["category"]
 		if not Config["piping"] and Config["verbosityLevel"] >= 3:
 			print_info(3, "[ Searching for file(s) " + pp.regexpquery(",".join(query)) + " in " + pp.cpv(cat) + "... ]")
 		
 		matches = portage.db["/"]["vartree"].dbapi.cpv_all()
-		#matches = gentoolkit.find_all_installed_packages(filter_fn)
+		if cat != "*":
+			matches = (x for x in matches if x.startswith(cat))
 
 		found = 0
 

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-portage-dev] equery: deprecate --category filtering in belongs

[Douglas Anderson]

On Sun, Feb 8, 2009 at 3:30 PM, Brian Harring <ferringb@gmail.com> wrote:
> patch attached against 0.2.4.2-r1; rough stats follow;
[snip]
> Mind you this isn't multiple runs, so the numbers are rough
> approximations- that said they're fairly representative.
>
> Strongly suggest y'all keep category support (although I'll keep on
> using pquery instead ;).
>
> Cheers,
> ~harring
>

Heh, OK I was looking at my local version of equery where category
filtering IS enabled while I was running tests with old equery, which
doesn't have it enabled. Thanks for pointing out my stupid mistake,
Brian. :)

We can in fact even make it go faster by filtering only cat/pkg before
it goes through cp_list:

- 		matches = portage.db["/"]["vartree"].dbapi.cpv_all()
+		matches = gentoolkit.find_all_installed_packages(lambda x: x.startswith(cat))

But still... why? When is it useful? Clearly it has never been used up
until now.

So I'm still in favor of just dumping it. Seems like complete
superfluous fluff to me. If anyone would have any real-world use for
this, I'll clean it up and keep it in. Otherwise I really don't think
anyone will miss it (especially not Brian :) )

-Doug

P.S. I'm strictly talking about belongs here, category filtering in
all other modules (where it's actually useful and makes sense) is not
only safe but has been upgraded to take multiple categories, so you
can now do:

$ ./equery.py l --category=app-portage,sys-apps port
[ Searching for port in app-portage, sys-apps ... ]
 * installed packages
[I--] [  ] app-portage/portage-utils-0.1.29 (0)
[I--] [ ~] sys-apps/portage-2.2_rc23 (0)

Re: [gentoo-portage-dev] equery: deprecate --category filtering in belongs

[Michael A. Smith]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

If you do drop it, Douglas, I think you should keep it around as a "Warning:
Deprecated" feature for a while before dropping it outright.

That way the users can give feedback via bugzie and their scripts don't break.

- -Michael

Douglas Anderson wrote:
> On Sun, Feb 8, 2009 at 3:30 PM, Brian Harring <ferringb@gmail.com> wrote:
>> patch attached against 0.2.4.2-r1; rough stats follow;
> [snip]
>> Mind you this isn't multiple runs, so the numbers are rough
>> approximations- that said they're fairly representative.
>>
>> Strongly suggest y'all keep category support (although I'll keep on
>> using pquery instead ;).
>>
>> Cheers,
>> ~harring
>>
> 
> Heh, OK I was looking at my local version of equery where category
> filtering IS enabled while I was running tests with old equery, which
> doesn't have it enabled. Thanks for pointing out my stupid mistake,
> Brian. :)
> 
> We can in fact even make it go faster by filtering only cat/pkg before
> it goes through cp_list:
> 
> - 		matches = portage.db["/"]["vartree"].dbapi.cpv_all()
> +		matches = gentoolkit.find_all_installed_packages(lambda x: x.startswith(cat))
> 
> But still... why? When is it useful? Clearly it has never been used up
> until now.
> 
> So I'm still in favor of just dumping it. Seems like complete
> superfluous fluff to me. If anyone would have any real-world use for
> this, I'll clean it up and keep it in. Otherwise I really don't think
> anyone will miss it (especially not Brian :) )
> 
> -Doug
> 
> P.S. I'm strictly talking about belongs here, category filtering in
> all other modules (where it's actually useful and makes sense) is not
> only safe but has been upgraded to take multiple categories, so you
> can now do:
> 
> $ ./equery.py l --category=app-portage,sys-apps port
> [ Searching for port in app-portage, sys-apps ... ]
>  * installed packages
> [I--] [  ] app-portage/portage-utils-0.1.29 (0)
> [I--] [ ~] sys-apps/portage-2.2_rc23 (0)
> 
> 

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iJwEAQECAAYFAkmPBDAACgkQwhlUDnr1of6K6AP/frkkntorPjwHHseoOEK8XzTW
PcvYXafeuJBlSBcna+RPHkvGJdZe6TBJrohXsDx+eTRNi74rdwjIvcNS0LIOx3oV
Ttx/M/onA6nk3iw8qOD5AZDUwTnYkY1eLdsR29hJnoGM2CYUoEWYO9TDQ0DiFdeE
k273HDefsSKxZP43Y/Q=
=KsaQ
-----END PGP SIGNATURE-----

Re: [gentoo-portage-dev] equery: deprecate --category filtering in belongs

[Douglas Anderson]

On Mon, Feb 9, 2009 at 1:11 AM, Michael A. Smith <michael@smith-li.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> If you do drop it, Douglas, I think you should keep it around as a "Warning:
> Deprecated" feature for a while before dropping it outright.
>
> That way the users can give feedback via bugzie and their scripts don't break.

Of course, but I think since it's never actually been turned on in the
first place, I'll make it a "not implemented" warning:

$ ./equery.py b -c app-portage /usr/bin/equery
!!! Module option -c, --category not implemented

[ Searching for /usr/bin/equery ... ]
app-portage/gentoolkit-0.2.4.2-r1 (/usr/bin/equery)

-Doug

BTW, for anyone interested, I've uploaded a summary of changes in the
refactored equery. Nothing is set in stone and if you have any
comments or criticisms whatsoever, let me know.

http://code.google.com/p/genscripts/source/browse/equery/trunk/equery/CHANGELOG