[gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Anthony G. Basile]

Hi everyone,

I've written a GLEP which outlines a standard for what information 
should be stored by any package management systems (PMS) in /var/db/pkg 
(VDB) and mandates some API for exporting it to other tools [1].  The 
need to do so was discussed in the Council during the 20130910 meeting 
[2].  During that meeting, the council focused on NEEDED.ELF.2 which is 
recorded by portage, but not paludis. Linkage information is generated 
during package builds, is expensive to recalculate and is needed by 
other packages like revdep-pax for PaX marking ELF objects.

I've aimed to make the GLEP open to how each PMS team wants to implement 
this without being too vague.  We should also consider carefully the 
list of items we want cached although we could always update this list 
later.

Feedback welcome.

Ref
[1] https://wiki.gentoo.org/wiki/GLEP:64
[2] http://www.gentoo.org/proj/en/council/meeting-logs/20130910-summary.txt

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 718 bytes --]

>>>>> On Sat, 30 Aug 2014, Anthony G Basile wrote:

> [...] package management systems (PMS) [...]

Apart from its other more common uses [1,2], "PMS" in a Gentoo context
stands for the "Package Manager Specification" [3] and the project
associated with it [4]. So please do not use the same abbreviation
for the package manager, as this will lead to confusion. Especially,
we must often distinguish between the specification and its
implementation in package managers, and using the same term for both
will not help with this.

Ulrich


[1] https://en.wikipedia.org/wiki/PMS
[2] https://www.youtube.com/watch?v=lF4PygY-k7Y
[3] http://dev.gentoo.org/~ulm/pms/head/pms.html
[4] https://wiki.gentoo.org/wiki/Project:PMS

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Anthony G. Basile]

On 08/30/14 17:05, Ulrich Mueller wrote:
>>>>>> On Sat, 30 Aug 2014, Anthony G Basile wrote:
>> [...] package management systems (PMS) [...]
> Apart from its other more common uses [1,2], "PMS" in a Gentoo context
> stands for the "Package Manager Specification" [3] and the project
> associated with it [4]. So please do not use the same abbreviation
> for the package manager, as this will lead to confusion. Especially,
> we must often distinguish between the specification and its
> implementation in package managers, and using the same term for both
> will not help with this.
>
> Ulrich
>
>
> [1] https://en.wikipedia.org/wiki/PMS
> [2] https://www.youtube.com/watch?v=lF4PygY-k7Y
> [3] http://dev.gentoo.org/~ulm/pms/head/pms.html
> [4] https://wiki.gentoo.org/wiki/Project:PMS

Thanks for that clarification!  There are other changes I want to make, 
but because I can't directly edit the wiki page, I'll wait for more 
input before sending a delta creffett's way.

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2288 bytes --]

Dnia 2014-08-30, o godz. 16:02:51
"Anthony G. Basile" <blueness@gentoo.org> napisał(a):

> I've written a GLEP which outlines a standard for what information 
> should be stored by any package management systems (PMS) in /var/db/pkg 
> (VDB) and mandates some API for exporting it to other tools [1].

I have trouble understanding the goal. As far as I can see, the idea
here is that every PM stores all information in any format, and exports
a Python API that has any synopsis and gives access to it... in any
format.

Wouldn't it be better to at least agree on some API for the metadata
exports? That will spare us the necessity of wrapping them all
in a common package or in every tool itself. Maybe it could even bring
some degree of interoperability between package managers.

As for the spec itself:

1. You're missing some of the metadata variables (RESTRICT,
PROPERTIES...). Wouldn't it be better to make one point worded like
'ebuild metadata as listed in PMS 13.2 Cache File Format'?

2. For *DEPEND, REQUIRED_USE (another one you missed) PMs store
dependency trees with USE conditionals evaluated. You may want to
explicitly note that.

3. I would use a copy of ebuild environment variables at the time of
completing the build (leaving last src_* phase?) -- IOW,
environment.bz2.

4. BUILD_TIME is not defined anywhere, so you may want to replace that
with verbose explanation of what is to be stored. For the remaining
metadata, you may want to reference PMS (the specification).

5. Please do not recommend Python modules since it discriminates
package managers written in C flavors. Instead, I suggest a plain CLI
API that gives best portability possibly. Alike:

  $ query-installed metadata sys-apps/coreutils-8.23 RDEPEND SLOT
  ...
  0

  $ query-installed file sys-apps/coreutils-8.23 /usr/bin/timeout SONAME
  ...
  
It should also have batch interface for querying multiple packages
quickly -- passing requests via stdin:

  $ query-installed batch
  [>] metadata sys-apps/coreutils-8.23 RDEPEND SLOT
  [<] ...
  [<] 0
  [>] file sys-apps/coreutils-8.23 /usr/bin/timeout SONAME
  [<] ...

6. Your Portage snippet uses outdated API. The modern one is to use
create_trees().

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 949 bytes --]

Re: [gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1059 bytes --]

On Sat, 30 Aug 2014 16:02:51 -0400
"Anthony G. Basile" <blueness@gentoo.org> wrote:
> I've written a GLEP which outlines a standard for what information 
> should be stored by any package management systems (PMS)
> in /var/db/pkg (VDB) and mandates some API for exporting it to other
> tools [1].

The problem isn't what the information is. It's the format.

>  The need to do so was discussed in the Council during the
> 20130910 meeting [2].  During that meeting, the council focused on
> NEEDED.ELF.2 which is recorded by portage, but not paludis. Linkage
> information is generated during package builds, is expensive to
> recalculate and is needed by other packages like revdep-pax for PaX
> marking ELF objects.

This isn't relevant to ebuilds.

> I've aimed to make the GLEP open to how each PMS team wants to
> implement this without being too vague.  We should also consider
> carefully the list of items we want cached although we could always
> update this list later.

This isn't a specification...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Anthony G. Basile]

On 08/31/14 03:26, Michał Górny wrote:
> Dnia 2014-08-30, o godz. 16:02:51
> "Anthony G. Basile" <blueness@gentoo.org> napisał(a):
>
>> I've written a GLEP which outlines a standard for what information
>> should be stored by any package management systems (PMS) in /var/db/pkg
>> (VDB) and mandates some API for exporting it to other tools [1].
> I have trouble understanding the goal. As far as I can see, the idea
> here is that every PM stores all information in any format, and exports
> a Python API that has any synopsis and gives access to it... in any
> format.
>
Not exactly.  The point is to *standardize* what is meant by "all 
information" so that all package managers export the same minimum set of 
information.  The most important being NEEDED.ELF.2 which is portage's 
VDB but not paludis.  Also, it can be in any format and exported in any 
way so long as it is well documented.  The goal is to have tools other 
than PM's make use of this information.  The example that began this is 
revdep-pax which uses NEEDED.ELF.2 to trace out linking so as to migrate 
PaX flags between ELF objects.

> Wouldn't it be better to at least agree on some API for the metadata
> exports? That will spare us the necessity of wrapping them all
> in a common package or in every tool itself. Maybe it could even bring
> some degree of interoperability between package managers.
Yes, I agree completely, but I didn't feel I was mandated to do so, 
which is what leaves me unsatisfied with what I wrote and reflected in 
your first paragraph.  However, I didn't want to bind the hands of the 
people writing the PM's either.  I wanted to make their task as easy as 
possible.  So long as all PM's export the same *minimum* set, developers 
writing non-PM tools which depend on VDB information can know what to 
expect and how to get at it.  Since NEEDED.ELF.2 is the critical for the 
reasons stated in the GLEP, and its not cached by some PM's, it was the 
subject of focus during the Council Meeting.

Having said that, I very much would like to say what that API looks 
like.  What you have below works for me.

>
> As for the spec itself:
>
> 1. You're missing some of the metadata variables (RESTRICT,
> PROPERTIES...). Wouldn't it be better to make one point worded like
> 'ebuild metadata as listed in PMS 13.2 Cache File Format'?

The point of the GLEP is to define a *minimum* set of exported metadata 
information, so I thought about what that minimum set should be.  I 
should have cleaned stuff up more before handing the draft over to 
creffett but I mistakingly assumed I could continue editing the wiki page.

Anyhow, now is the time for people to suggest what that minimum set 
should be.  I'll look at PMS 13.2 Cache File Format.

>
> 2. For *DEPEND, REQUIRED_USE (another one you missed) PMs store
> dependency trees with USE conditionals evaluated. You may want to
> explicitly note that.

Noted.

>
> 3. I would use a copy of ebuild environment variables at the time of
> completing the build (leaving last src_* phase?) -- IOW,
> environment.bz2.

Noted.

>
> 4. BUILD_TIME is not defined anywhere, so you may want to replace that
> with verbose explanation of what is to be stored. For the remaining
> metadata, you may want to reference PMS (the specification).

Noted.

> 5. Please do not recommend Python modules since it discriminates
> package managers written in C flavors.

What's wrong with C-python bindings?  Or anything-anything bindings.  
I'm not discriminating against flavors of PM's.  A PM can export this 
information via a C library, as long as the API is documented.  Its up 
to the developers of utilities consuming this information to "hook" in.

Having said that, I like what you have below and I'd be happy to work on 
it.  I can already see how to proceed with portage.

> Instead, I suggest a plain CLI
> API that gives best portability possibly. Alike:
>
>    $ query-installed metadata sys-apps/coreutils-8.23 RDEPEND SLOT
>    ...
>    0
>
>    $ query-installed file sys-apps/coreutils-8.23 /usr/bin/timeout SONAME
>    ...
>    
> It should also have batch interface for querying multiple packages
> quickly -- passing requests via stdin:
>
>    $ query-installed batch
>    [>] metadata sys-apps/coreutils-8.23 RDEPEND SLOT
>    [<] ...
>    [<] 0
>    [>] file sys-apps/coreutils-8.23 /usr/bin/timeout SONAME
>    [<] ...
>
> 6. Your Portage snippet uses outdated API. The modern one is to use
> create_trees().

You mean the |vardb = portage.db[portage.root]['vartree'].dbapi stuff?  
We can talk about this later.|



-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 815 bytes --]

On Sun, 31 Aug 2014 08:38:00 -0400
"Anthony G. Basile" <blueness@gentoo.org> wrote:
> Not exactly.  The point is to *standardize* what is meant by "all 
> information" so that all package managers export the same minimum set
> of information.  The most important being NEEDED.ELF.2 which is
> portage's VDB but not paludis.  Also, it can be in any format and
> exported in any way so long as it is well documented.  The goal is to
> have tools other than PM's make use of this information.  The example
> that began this is revdep-pax which uses NEEDED.ELF.2 to trace out
> linking so as to migrate PaX flags between ELF objects.

VDB is completely non-standard, undocumented, and hard to read
correctly. There's no point in having information if you aren't allowed
to use it.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Anthony G. Basile]

On 08/31/14 08:26, Ciaran McCreesh wrote:
> On Sat, 30 Aug 2014 16:02:51 -0400
> "Anthony G. Basile" <blueness@gentoo.org> wrote:
>> I've written a GLEP which outlines a standard for what information
>> should be stored by any package management systems (PMS)
>> in /var/db/pkg (VDB) and mandates some API for exporting it to other
>> tools [1].
> The problem isn't what the information is. It's the format.

That's not the point, the point is specifying what information should be 
cached and exported.  NEEDED.ELF.2 is cached by portage but not 
paludis.  Developers writing tools that make use of a PM's VDB should 
know what they can expect from VDB and how to get at it. You can use any 
format you like (plain ascii or some db format) as long as its 
exportable to other tools.

>
>>   The need to do so was discussed in the Council during the
>> 20130910 meeting [2].  During that meeting, the council focused on
>> NEEDED.ELF.2 which is recorded by portage, but not paludis. Linkage
>> information is generated during package builds, is expensive to
>> recalculate and is needed by other packages like revdep-pax for PaX
>> marking ELF objects.
> This isn't relevant to ebuilds.

The point is that developers writing tools that make use of a PM's VDB 
should now what's there.  A quote from the council minutes: "The council 
discussed if the contents of the VDB should be specified for 
interoperability between utilities, ..."

>
>> I've aimed to make the GLEP open to how each PMS team wants to
>> implement this without being too vague.  We should also consider
>> carefully the list of items we want cached although we could always
>> update this list later.
> This isn't a specification...
>
Correct.  How you implement this is up to you to make your life as easy 
as possible.  What is specified in the GLEP is what information should 
be cached and that a clearly documented API be produced.

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 462 bytes --]

On Sun, 31 Aug 2014 08:43:48 -0400
"Anthony G. Basile" <blueness@gentoo.org> wrote:
> What is specified in the GLEP is what information
> should be cached and that a clearly documented API be produced.

You are specifying the colour of the flowers on the shelf inside
the bikeshed before we've established whether we need to store bikes,
and if we do, whether we need a shed for them, and if we do, whether it
will have a shelf.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Anthony G. Basile]

On 08/31/14 08:43, Ciaran McCreesh wrote:
> On Sun, 31 Aug 2014 08:38:00 -0400
> "Anthony G. Basile" <blueness@gentoo.org> wrote:
>> Not exactly.  The point is to *standardize* what is meant by "all
>> information" so that all package managers export the same minimum set
>> of information.  The most important being NEEDED.ELF.2 which is
>> portage's VDB but not paludis.  Also, it can be in any format and
>> exported in any way so long as it is well documented.  The goal is to
>> have tools other than PM's make use of this information.  The example
>> that began this is revdep-pax which uses NEEDED.ELF.2 to trace out
>> linking so as to migrate PaX flags between ELF objects.
> VDB is completely non-standard, undocumented, and hard to read
> correctly. There's no point in having information if you aren't allowed
> to use it.
>

VDB as exported by portage is readable and useful to non-PM tools. I'd 
give you a link to git.gentoo.org/proj/elfix as a concrete example, but 
the site is still down.  This usefulness suggests that it should be 
standardize and documented so that VDB information can be read correctly 
by non-PM tools.

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Brian Dolbec]

On Sun, 31 Aug 2014 10:56:21 -0400
"Anthony G. Basile" <blueness@gentoo.org> wrote:

> On 08/31/14 08:43, Ciaran McCreesh wrote:
> > On Sun, 31 Aug 2014 08:38:00 -0400
> > "Anthony G. Basile" <blueness@gentoo.org> wrote:
> >> Not exactly.  The point is to *standardize* what is meant by "all
> >> information" so that all package managers export the same minimum
> >> set of information.  The most important being NEEDED.ELF.2 which is
> >> portage's VDB but not paludis.  Also, it can be in any format and
> >> exported in any way so long as it is well documented.  The goal is
> >> to have tools other than PM's make use of this information.  The
> >> example that began this is revdep-pax which uses NEEDED.ELF.2 to
> >> trace out linking so as to migrate PaX flags between ELF objects.
> > VDB is completely non-standard, undocumented, and hard to read
> > correctly. There's no point in having information if you aren't
> > allowed to use it.
> >
> 
> VDB as exported by portage is readable and useful to non-PM tools.
> I'd give you a link to git.gentoo.org/proj/elfix as a concrete
> example, but the site is still down.  This usefulness suggests that
> it should be standardize and documented so that VDB information can
> be read correctly by non-PM tools.
> 

+1

The new python based revdep-rebuild also uses scanelf and subsequently
the NEEDED.ELF.2 files.  And there are other tools that make use of vdb
info.

It is past due to have a minimum set of defined info to be recorded in
the vdb.

-- 
Brian Dolbec <dolsen>

Re: [gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 621 bytes --]

On Sun, 31 Aug 2014 10:56:21 -0400
"Anthony G. Basile" <blueness@gentoo.org> wrote:
> I'd give you a link to git.gentoo.org/proj/elfix as a concrete
> example, but the site is still down.

Are you emulating all the workarounds for reading previously-written
invalid data in there? Because if not, you're reading what you want VDB
to contain, not what it actually does...

Remember, VDB's format isn't specified anywhere, so if you claim you
can read it, you must be able to read whatever it contains, and you
can't claim that (for example) rogue 'stat' entries in CONTENTS are a
bug.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Anthony G. Basile]

On 08/31/14 08:46, Ciaran McCreesh wrote:
> On Sun, 31 Aug 2014 08:43:48 -0400
> "Anthony G. Basile" <blueness@gentoo.org> wrote:
>> What is specified in the GLEP is what information
>> should be cached and that a clearly documented API be produced.
> You are specifying the colour of the flowers on the shelf inside
> the bikeshed before we've established whether we need to store bikes,
> and if we do, whether we need a shed for them, and if we do, whether it
> will have a shelf.
>

If I follow your analogy you're suggesting there's no need to export 
VDB.  However, there is a need to follow linking both forwards and 
backwards.  This is generated by portage and cached in NEEDED.ELF.2.  On 
a PaX enabled kernel, we often have to migrate the PaX flags from 
libraries to *all* ELF executables that link against them --- this is 
similar to revdep-rebuild.py but for migrating PaX flags.  This 
information takes ~5 minutes to be generated on a typical desktop 
system.  However, it is just a *regeneration* of information already 
available in NEEDED.ELF.2.  Using the latter from VDB takes only seconds 
by comparison.  It is counterproductive not to make this information 
available.

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 437 bytes --]

On Sun, 31 Aug 2014 08:02:25 -0700
Brian Dolbec <dolsen@gentoo.org> wrote:
> It is past due to have a minimum set of defined info to be recorded in
> the vdb.

VDB is entirely unspecified, and it can and does change arbitrarily.
Worrying about whether or not one particular obscure bit of data is in
there is irrelevant, when you haven't established whether VDB
necessarily exists or is readable at all.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Anthony G. Basile]

On 08/31/14 11:02, Ciaran McCreesh wrote:
> On Sun, 31 Aug 2014 10:56:21 -0400
> "Anthony G. Basile" <blueness@gentoo.org> wrote:
>> I'd give you a link to git.gentoo.org/proj/elfix as a concrete
>> example, but the site is still down.
> Are you emulating all the workarounds for reading previously-written
> invalid data in there? Because if not, you're reading what you want VDB
> to contain, not what it actually does...
>
> Remember, VDB's format isn't specified anywhere, so if you claim you
> can read it, you must be able to read whatever it contains, and you
> can't claim that (for example) rogue 'stat' entries in CONTENTS are a
> bug.
>
I'm reading portage's code.  I'm not looking at paludis.  The council is 
proposing a standardization of this information so that we don't have 
the sitatuation where we're "reading what you want VDB to contain, not 
what it actually does"

I do not understand why you oppose the standardization of VDB?

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1217 bytes --]

On Sun, 31 Aug 2014 11:06:13 -0400
"Anthony G. Basile" <blueness@gentoo.org> wrote:
> On 08/31/14 08:46, Ciaran McCreesh wrote:
> > On Sun, 31 Aug 2014 08:43:48 -0400
> > "Anthony G. Basile" <blueness@gentoo.org> wrote:
> >> What is specified in the GLEP is what information
> >> should be cached and that a clearly documented API be produced.
> > You are specifying the colour of the flowers on the shelf inside
> > the bikeshed before we've established whether we need to store
> > bikes, and if we do, whether we need a shed for them, and if we do,
> > whether it will have a shelf.
> 
> If I follow your analogy you're suggesting there's no need to export 
> VDB.

No, I am saying that *if* you think there is a need to make this kind
of information available, then you should think carefully about the
design of a feature which allows this. What you should not do is go on
about NEEDED.ELF.2 at this stage, because that's a minor technicality.

There are plenty of good reasons that VDB is not specified in PMS. You
are ignoring them all and jumping straight to the minor triviality you
think you care about, whilst not doing anything to address the actual
issue.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1219 bytes --]

On Sun, 31 Aug 2014 11:08:27 -0400
"Anthony G. Basile" <blueness@gentoo.org> wrote:
> On 08/31/14 11:02, Ciaran McCreesh wrote:
> > On Sun, 31 Aug 2014 10:56:21 -0400
> > "Anthony G. Basile" <blueness@gentoo.org> wrote:
> >> I'd give you a link to git.gentoo.org/proj/elfix as a concrete
> >> example, but the site is still down.
> > Are you emulating all the workarounds for reading previously-written
> > invalid data in there? Because if not, you're reading what you want
> > VDB to contain, not what it actually does...
> >
> > Remember, VDB's format isn't specified anywhere, so if you claim you
> > can read it, you must be able to read whatever it contains, and you
> > can't claim that (for example) rogue 'stat' entries in CONTENTS are
> > a bug.
> >
> I'm reading portage's code.

Which version? Note that Portage can't read the VDB entries generated
by certain other Portage versions.

> I do not understand why you oppose the standardization of VDB?

If you would like to standardise VDB, I suggest you start by doing a
decent job of solving that problem, and not just jumping in and yelling
about how important it is that some particular file is in there.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Anthony G. Basile]

On 08/31/14 11:13, Ciaran McCreesh wrote:
> On Sun, 31 Aug 2014 11:08:27 -0400
> "Anthony G. Basile" <blueness@gentoo.org> wrote:
>> On 08/31/14 11:02, Ciaran McCreesh wrote:
>>> On Sun, 31 Aug 2014 10:56:21 -0400
>>> "Anthony G. Basile" <blueness@gentoo.org> wrote:
>>>> I'd give you a link to git.gentoo.org/proj/elfix as a concrete
>>>> example, but the site is still down.
>>> Are you emulating all the workarounds for reading previously-written
>>> invalid data in there? Because if not, you're reading what you want
>>> VDB to contain, not what it actually does...
>>>
>>> Remember, VDB's format isn't specified anywhere, so if you claim you
>>> can read it, you must be able to read whatever it contains, and you
>>> can't claim that (for example) rogue 'stat' entries in CONTENTS are
>>> a bug.
>>>
>> I'm reading portage's code.
> Which version? Note that Portage can't read the VDB entries generated
> by certain other Portage versions.'

Then you version the VDB cache and you write your API as an abstraction 
layer to take care of that.

>
>> I do not understand why you oppose the standardization of VDB?
> If you would like to standardise VDB, I suggest you start by doing a
> decent job of solving that problem, and not just jumping in and yelling
> about how important it is that some particular file is in there.
>
I will take this into account when I write the next version of the GLEP.

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Rich Freeman]

On Sun, Aug 31, 2014 at 11:08 AM, Anthony G. Basile <blueness@gentoo.org> wrote:
>
> I do not understand why you oppose the standardization of VDB?
>

I think it would make sense to take a step back.

What is it that you're actually after here?  You want to be able to
determine information about packages that are installed.  So, just
determine what information is required and define an API for providing
this information.  I'd focus on software/script-friendliness first and
foremost - then people can write as many wrappers around it as they
like.

How the package manager determines this information is not the
problem.  If the package manager wants to run ldd against all the
binaries installed by the package and then recompile every ebuild in
the tree to see if there is a match, that is its own problem.

Just focus on the interface, and trust those implementing the package
manager to do it in a competent fashion.  If they don't then users
probably won't use the package manager if they care about those
features.  Or maybe users care more about a few kilobytes of indexes
and would prefer that the package manager not store information which
it could re-derive, and that is perfectly fine as well.

And yes, I realize we're talking about how to accomplish something
before we've decided whether to accomplish it.  I think that for any
reasonable decision to be made about the latter it doesn't hurt to
have at least some sense of what the former is going to look like,
though not necessarily the detail of a full specification.  It
wouldn't hurt to point out use cases too.  This is a GLEP, so nobody
has to do anything unless the Council approves it...

--
Rich

Re: [gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Anthony G. Basile]

On 08/31/14 11:53, Rich Freeman wrote:
> On Sun, Aug 31, 2014 at 11:08 AM, Anthony G. Basile <blueness@gentoo.org> wrote:
>> I do not understand why you oppose the standardization of VDB?
>>
> I think it would make sense to take a step back.
>
> What is it that you're actually after here?  You want to be able to
> determine information about packages that are installed.  So, just
> determine what information is required and define an API for providing
> this information.  I'd focus on software/script-friendliness first and
> foremost - then people can write as many wrappers around it as they
> like.

I can agree with this except that's not how the Council originally 
phrased it.  We were specifically speaking of VDB.  Anyhow, I can change 
the language so it emphasizes the desired information suggesting that it 
may be obtained from VDB or otherwise as the PM wishes.

>
> How the package manager determines this information is not the
> problem.  If the package manager wants to run ldd against all the
> binaries installed by the package and then recompile every ebuild in
> the tree to see if there is a match, that is its own problem.

This is kinda silly.  The point is that all PM's generate certain 
information at build time and *should* cache it for later use.  How they 
wish to cache it is up to them as long as they export it somehow.  But 
if you get rid of the caching, then you drop one of the requirements 
we're seeking from the PM in this GLEP --- efficient lookup of 
information which is expensive to regenerate.

Moving past NEEDED.ELF.2, take a look at gentoolkit.  It also make use 
of portage's VDB in "class KeywordAnalyser: def __init__(...)" in 
enalyze/lib.py.  So asking a very simple question like `list all files 
that belong to sys-apps/coreutils` draws from VDB cache.  The equivalent 
of what you're suggesting is that the PM be allowed to recompile the 
package each time to answer that question.  That's silly.

How the caching is done is up to the PM, but what information needs to 
be cached to avoid expensive regeneration is not.  It should be 
specified in the GLEP.

>
> Just focus on the interface, and trust those implementing the package
> manager to do it in a competent fashion.  If they don't then users
> probably won't use the package manager if they care about those
> features.  Or maybe users care more about a few kilobytes of indexes
> and would prefer that the package manager not store information which
> it could re-derive, and that is perfectly fine as well.
>
> And yes, I realize we're talking about how to accomplish something
> before we've decided whether to accomplish it.  I think that for any
> reasonable decision to be made about the latter it doesn't hurt to
> have at least some sense of what the former is going to look like,
> though not necessarily the detail of a full specification.  It
> wouldn't hurt to point out use cases too.  This is a GLEP, so nobody
> has to do anything unless the Council approves it...
>
> --
> Rich
>



-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 403 bytes --]

On Sun, 31 Aug 2014 12:28:00 -0400
"Anthony G. Basile" <blueness@gentoo.org> wrote:
> This is kinda silly.  The point is that all PM's generate certain 
> information at build time and *should* cache it for later use.

The historical view of VDB being some kind of "cache" rather than a "DB"
is one of the reasons we're having to tidy up the dynamic dependencies
mess...

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Anthony G. Basile]

On 08/31/14 12:30, Ciaran McCreesh wrote:
> On Sun, 31 Aug 2014 12:28:00 -0400
> "Anthony G. Basile" <blueness@gentoo.org> wrote:
>> This is kinda silly.  The point is that all PM's generate certain
>> information at build time and *should* cache it for later use.
> The historical view of VDB being some kind of "cache" rather than a "DB"
> is one of the reasons we're having to tidy up the dynamic dependencies
> mess...
>

Indeed, some of VDB might not be something we wish to mandate as 
"cache".  But some of it should be, like the examples I gave.  This is 
our opportunity to think this out carefully and only export what is 
meaningfully cacheable information.  So when I submit version 2, let's 
keep that in mind with respect to any of the items I say should be "cached".

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 989 bytes --]

On 30-08-2014 16:02:51 -0400, Anthony G. Basile wrote:
> ...  During that meeting, the council focused on NEEDED.ELF.2 which is 
> recorded by portage, but not paludis. Linkage information is generated 
> during package builds, is expensive to recalculate and is needed by 
> other packages like revdep-pax for PaX marking ELF objects.

Portage is able to calculate dependency information for
@preserved-rebuild based on the information stored in multiple files,
(NEEDED,) NEEDED.ELF.2, NEEDED.MACHO.3, NEEDED.XCOFF, NEEDED.PECOFF, etc.
For this purpose, it would be helpful if a generalised interface would
be exported, instead of the raw files (and their storage format
version).  Portage, for instance, internally has a function that queries
for consumers of a specific library (file), abstracting away parts of
the shared library format internals, that otherwise would have to be
implemented by all tools.

Fabian


-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.

[Anthony G. Basile]

On 09/01/14 02:24, Fabian Groffen wrote:
> On 30-08-2014 16:02:51 -0400, Anthony G. Basile wrote:
>> ...  During that meeting, the council focused on NEEDED.ELF.2 which is
>> recorded by portage, but not paludis. Linkage information is generated
>> during package builds, is expensive to recalculate and is needed by
>> other packages like revdep-pax for PaX marking ELF objects.
> Portage is able to calculate dependency information for
> @preserved-rebuild based on the information stored in multiple files,
> (NEEDED,) NEEDED.ELF.2, NEEDED.MACHO.3, NEEDED.XCOFF, NEEDED.PECOFF, etc.
> For this purpose, it would be helpful if a generalised interface would
> be exported, instead of the raw files (and their storage format
> version).  Portage, for instance, internally has a function that queries
> for consumers of a specific library (file), abstracting away parts of
> the shared library format internals, that otherwise would have to be
> implemented by all tools.
>
> Fabian
>
>

Here are some packages that raw-read VDB, other than portage itself:

* sys-apps/elfix (python)
* app-portage/gentoolkit (python)
* app-portage/portage-utils (C)
* app-portage/eix (C++)

There are probably more, but I am familiar with their internals. 
Generally, these packages use VDB information for 1) lists of files and 
2) linking information.  Again, caching this information (because it is 
expensive to re-calculate) and exporting it via a general interface is 
useful.

Fabian thanks for pointing out the other NEEDED.* files.  I wondered how 
portage did this for alternative exe formats.

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

[gentoo-dev] RFC: GLEP 64: Export Package Manager Cached Information. (Was: RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.)

[Anthony G. Basile]

Hi everyone,

I've incorporated suggestions from the previous round of discussions in 
the GLEP.  In particular, note that I also changed the title to avoid 
referring to VDB as the means for caching.  Each package manager can 
decide how to cache the information internally.

The working copy is at https://wiki.gentoo.org/wiki/User:Blueness/GLEP64

The essential point I'm trying to make is: 1) there is information 
generated when a PM builds+installs a package.  2) some of this 
information is expensive to regenerate.  3) there are utilites that 
would like to make use of this information.  4) to spare these utilites 
the expense of regenerating this information, all package managers 
should cache this information and then export it via a common API.

Thanks in advance for your time!

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] RFC: GLEP 64: Export Package Manager Cached Information. (Was: RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 1574 bytes --]

On Sat, 06 Sep 2014 09:03:13 -0400
"Anthony G. Basile" <blueness@gentoo.org> wrote:
> Note that, as with the Metadata Cache, these variable should be stored
> with all the conditionals evaluated. 

Paludis doesn't do this (and historically, Portage didn't either). We
store USE etc. This is useful because it allows us to detect when
people have been mucking around with DEPEND and the like.

> A list of all files belonging to the package, along with a designation
> of the file type (regular, directory, symlink, pipe, etc), MD5SUM or
> other checksum, and mtime time. 

Packages aren't allowed to install pipes.

> A list of all executable or shared objects for each package and the
> corresponding linking information, including full path to the object,
> its architecture and ABI, SONAME, RPATH and any NEEDED objects they
> link against, as reported by `readelf` on ELF systems, or similar
> tools for other executable formats. Currently this information is
> being cached by Portage in NEEDED.ELF.2, NEEDED.MACHO.3, NEEDED.XCOFF,
> NEEDED.PECOFF, etc. 

This is utterly arbitrary, and introduces a dependency on particular
non-standard package whose behaviour we don't control. Not everything
is C and ELF, and we shouldn't be encouraging "solutions" that make
the assumption that they are...

You've also not discussed how this interacts with Portage's
package.provided misfeature.

Finally, you don't have any way of using this information, since you
don't have a way of knowing what packages are installed.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] RFC: GLEP 64: Export Package Manager Cached Information. (Was: RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.)

[Anthony G. Basile]

On 09/06/14 09:10, Ciaran McCreesh wrote:
> On Sat, 06 Sep 2014 09:03:13 -0400
> "Anthony G. Basile" <blueness@gentoo.org> wrote:
>> Note that, as with the Metadata Cache, these variable should be stored
>> with all the conditionals evaluated.
> Paludis doesn't do this (and historically, Portage didn't either). We
> store USE etc. This is useful because it allows us to detect when
> people have been mucking around with DEPEND and the like.

This was a suggestion from mgorny and I trust his opinion on the 
matter.  It does make sense to have metadata catche which is 
conditionally evaluated be exported.  If PM's also want to the 
conditionals in there, there is nothing in the GLEP which prevents it.

>
>> A list of all files belonging to the package, along with a designation
>> of the file type (regular, directory, symlink, pipe, etc), MD5SUM or
>> other checksum, and mtime time.
> Packages aren't allowed to install pipes.

Okay I will remove pipes.  Do you have a reference btw about what types 
of files can be installed.

>
>> A list of all executable or shared objects for each package and the
>> corresponding linking information, including full path to the object,
>> its architecture and ABI, SONAME, RPATH and any NEEDED objects they
>> link against, as reported by `readelf` on ELF systems, or similar
>> tools for other executable formats. Currently this information is
>> being cached by Portage in NEEDED.ELF.2, NEEDED.MACHO.3, NEEDED.XCOFF,
>> NEEDED.PECOFF, etc.
> This is utterly arbitrary, and introduces a dependency on particular
> non-standard package whose behaviour we don't control. Not everything
> is C and ELF, and we shouldn't be encouraging "solutions" that make
> the assumption that they are...

1) This cannot be utterly arbitrary because there are utilities and 
portions of portages code which does make use of precisely this information.

2) With the exception of some embedded systems where everything is 
statically linked, all modern systems have dynamic linking.  And all 
dynamic linking has information in its objects which associates the 
executables with the library they link against.  This is the essential 
information to be stored.

3) You are correct that the bias there is towards ELF.  So I will add 
language indicating similar information for the other executable formats.

4) Of course not everything is C.  (BTW the better example to make your 
case, is FFLAGS for fortran which I omitted --- I will add language here 
too).  The point is to record dynamical linking information where it is 
generated.  Such information is obtained at build time, is expensive to 
regenerate, is useful at a later time for both the PM and other 
utilities, and so should be cached.  We would record this even for an 
executable generated from fortran source, say, that links against some 
libraries.

5) For packages which don't have any dynamical linking, we just leave 
those things blank.  ie.  asking for the SONAME of some file 
/usr/share/pkgconfig/ returns null with some error.

6) Without a standard here, we have utilites which make use of this 
cached information in the tree which only work with portage but not 
paludis.  This problem can be solved by removing those utilities, which 
is undesireable, by standardizing what needs to be exported from the PM, 
or by living with the status quo which is having useful packages in the 
tree which don't work with paludis.

>
> You've also not discussed how this interacts with Portage's
> package.provided misfeature.
>
> Finally, you don't have any way of using this information, since you
> don't have a way of knowing what packages are installed.
>

I don't understand your reasoning behind these points, can you please 
explain.

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] RFC: GLEP 64: Export Package Manager Cached Information. (Was: RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.)

[hasufell]

Anthony G. Basile:
>>
>>> A list of all files belonging to the package, along with a designation
>>> of the file type (regular, directory, symlink, pipe, etc), MD5SUM or
>>> other checksum, and mtime time.
>> Packages aren't allowed to install pipes.
> 
> Okay I will remove pipes.  Do you have a reference btw about what types
> of files can be installed.
> 

http://dev.gentoo.org/~ulm/pms/head/pms.html#x1-15800012.6

> 12.6 Other Files
> 
> Ebuilds must not attempt to install any other type of file (FIFOs, device nodes etc). 

The explicit list of allowed ones is in the chapters above.

Re: [gentoo-dev] RFC: GLEP 64: Export Package Manager Cached Information. (Was: RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 2491 bytes --]

On Sat, 06 Sep 2014 09:51:58 -0400
"Anthony G. Basile" <blueness@gentoo.org> wrote:
> > Paludis doesn't do this (and historically, Portage didn't either).
> > We store USE etc. This is useful because it allows us to detect when
> > people have been mucking around with DEPEND and the like.
> 
> This was a suggestion from mgorny and I trust his opinion on the 
> matter.  It does make sense to have metadata catche which is 
> conditionally evaluated be exported.

Well what are you planning to use it for? Are you really suggesting
that people are going to implement EAPI-aware, compliant dependency
parsers that can't figure out conditionals?

> 1) This cannot be utterly arbitrary because there are utilities and 
> portions of portages code which does make use of precisely this
> information.

What Portage does internally is up to Portage. What we don't want is to
encourage external utilities that are utterly inflexible and that make
strange assumptions.

> 2) With the exception of some embedded systems where everything is 
> statically linked, all modern systems have dynamic linking.  And all 
> dynamic linking has information in its objects which associates the 
> executables with the library they link against.  This is the
> essential information to be stored.

Which isn't what you're asking for... You're asking for it in a
particular format.

> 6) Without a standard here, we have utilites which make use of this 
> cached information in the tree which only work with portage but not 
> paludis.  This problem can be solved by removing those utilities,
> which is undesireable, by standardizing what needs to be exported
> from the PM, or by living with the status quo which is having useful
> packages in the tree which don't work with paludis.

The solution is to replace those utilities with something that works in
proper generality.

> > You've also not discussed how this interacts with Portage's
> > package.provided misfeature.
> >
> > Finally, you don't have any way of using this information, since you
> > don't have a way of knowing what packages are installed.
> 
> I don't understand your reasoning behind these points, can you please 
> explain.

Well you can't ask for information about packages unless you know
what's installed, and you haven't asked for a general API for that.

And when you do ask, is a package that's "provided" installed, and if
so, what's its metadata?

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] RFC: GLEP 64: Export Package Manager Cached Information. (Was: RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.)

[Anthony G. Basile]

On 09/06/14 10:04, Ciaran McCreesh wrote:
> On Sat, 06 Sep 2014 09:51:58 -0400
> "Anthony G. Basile" <blueness@gentoo.org> wrote:
>>> Paludis doesn't do this (and historically, Portage didn't either).
>>> We store USE etc. This is useful because it allows us to detect when
>>> people have been mucking around with DEPEND and the like.
>> This was a suggestion from mgorny and I trust his opinion on the
>> matter.  It does make sense to have metadata catche which is
>> conditionally evaluated be exported.
> Well what are you planning to use it for? Are you really suggesting
> that people are going to implement EAPI-aware, compliant dependency
> parsers that can't figure out conditionals?

No, but the resolution of the conditionals may change over time and it 
would be nice to have the information as it was at the time of the 
build.  However, this point is minor.  I'm certain something can be 
worked out among the PMS people as to what is best here.  After all, the 
request did come from the portage camp.

>
>> 1) This cannot be utterly arbitrary because there are utilities and
>> portions of portages code which does make use of precisely this
>> information.
> What Portage does internally is up to Portage. What we don't want is to
> encourage external utilities that are utterly inflexible and that make
> strange assumptions.

Neither do we want to hamper what developers might want to do.  The 
authors of sys-apps/elfix, app-portage/gentoolkit, 
app-portage/portage-utils, app-portage/eix are not "utterly inflexible" 
utilities makeing "strange assumptions".  They are useful utilities as 
anyone following this thread would agree.  Their assumption would go 
from being non-standard to standard with the acceptance of this GLEP.  
They would then operate with both portage and paludis.

>
>> 2) With the exception of some embedded systems where everything is
>> statically linked, all modern systems have dynamic linking.  And all
>> dynamic linking has information in its objects which associates the
>> executables with the library they link against.  This is the
>> essential information to be stored.
> Which isn't what you're asking for... You're asking for it in a
> particular format.

I will incorporate better language which goes to the heart of what's 
needed and makes the ELF quantities an example rather than the demanded 
format.  In other words, I will remove the ELF bias.  Since dynamic 
linking information is generated at build time, is expensive to 
regenerate and is useful to other utilities, it falls under the scope of 
the GLEP.  Of course we wish to extend this to any executable format.  
We focus on ELF because of its familiarity, but not at the exclusion of 
others.

>
>> 6) Without a standard here, we have utilites which make use of this
>> cached information in the tree which only work with portage but not
>> paludis.  This problem can be solved by removing those utilities,
>> which is undesireable, by standardizing what needs to be exported
>> from the PM, or by living with the status quo which is having useful
>> packages in the tree which don't work with paludis.
> The solution is to replace those utilities with something that works in
> proper generality.

You can do this, but it would be terribly inefficient to regenerate 
expensive information already generated by PM.  For example, `equery` 
from gentoolkit allows the user to gather all sorts of information about 
installed packages, eg. "What package does this file belongs to?"  A 
very natural question.  Without this information cached in portage's 
VDB, how would equery do this?  It would  have to rebuild package by 
package until it finds one that gives the file we're looking for?!  This 
is absurd.  Rather gentoolkit opens up /var/db/pkg and read the 
information out of there.

For gentoolkit to work "in proper generality" we would need PM's to 
export this information by some common API, so there is a generality.  
That's what GLEP 64 is about.  You point argues in favor of the GLEP.


>
>>> You've also not discussed how this interacts with Portage's
>>> package.provided misfeature.
>>>
>>> Finally, you don't have any way of using this information, since you
>>> don't have a way of knowing what packages are installed.
>> I don't understand your reasoning behind these points, can you please
>> explain.
> Well you can't ask for information about packages unless you know
> what's installed, and you haven't asked for a general API for that.

Ah, okay.  That should be added explicitly since its only implied right 
now.  Thanks.

>
> And when you do ask, is a package that's "provided" installed, and if
> so, what's its metadata?
>

When the package is installed, that data should have been cached.

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] RFC: GLEP 64: Export Package Manager Cached Information. (Was: RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.)

[hasufell]

Anthony G. Basile:
>>
>> And when you do ask, is a package that's "provided" installed, and if
>> so, what's its metadata?
>>
> 
> When the package is installed, that data should have been cached.
> 

Afaik there is nothing "cached" if you put stuff in package.provided.
It's a terrible hack, unless I missed something.

Re: [gentoo-dev] RFC: GLEP 64: Export Package Manager Cached Information. (Was: RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.)

[Anthony G. Basile]

On 09/06/14 12:12, hasufell wrote:
> Anthony G. Basile:
>>> And when you do ask, is a package that's "provided" installed, and if
>>> so, what's its metadata?
>>>
>> When the package is installed, that data should have been cached.
>>
> Afaik there is nothing "cached" if you put stuff in package.provided.
> It's a terrible hack, unless I missed something.
>

I wasn't sure what Ciaran was talking about there.  If its hacky, then 
we certainly don't want to standardize it in the GLEP.

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] RFC: GLEP 64: Export Package Manager Cached Information. (Was: RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 3044 bytes --]

On Sat, 06 Sep 2014 12:07:10 -0400
"Anthony G. Basile" <blueness@gentoo.org> wrote:
> No, but the resolution of the conditionals may change over time and
> it would be nice to have the information as it was at the time of the 
> build. 

You have USE available...

> Neither do we want to hamper what developers might want to do.  The 
> authors of sys-apps/elfix, app-portage/gentoolkit, 
> app-portage/portage-utils, app-portage/eix are not "utterly
> inflexible" utilities makeing "strange assumptions".  They are useful
> utilities as anyone following this thread would agree. Their
> assumption would go from being non-standard to standard with the
> acceptance of this GLEP. They would then operate with both portage
> and paludis.

Well eix is buggy, PMS-violating and doesn't support EAPIs properly,
and the utilities in gentoolkit and portage-utils are better implemented
natively in a package manager. I don't know what elfix does, but if
it's anything like the other three, I'd rather reimplement it in a PMS
compliant manner for Paludis than to provide flaky external APIs that
encourage people to write broken code...

> You can do this, but it would be terribly inefficient to regenerate 
> expensive information already generated by PM.  For example, `equery` 
> from gentoolkit allows the user to gather all sorts of information
> about installed packages, eg. "What package does this file belongs
> to?"  A very natural question.  Without this information cached in
> portage's VDB, how would equery do this?  It would  have to rebuild
> package by package until it finds one that gives the file we're
> looking for?!  This is absurd.  Rather gentoolkit opens
> up /var/db/pkg and read the information out of there.

It would use a package-manager provided interface to do it, which would
remove the need to implement direct VDB reading externally. This is
good, because reading VDB *correctly* is difficult, and we don't want
lots of third party tools doing it badly. For example, with Paludis,
you would do 'cave print-owners' to get this information, and your code
works correctly even if VDB switches formats and even if it isn't
located at /var/db/pkg.

> For gentoolkit to work "in proper generality" we would need PM's to 
> export this information by some common API, so there is a
> generality. That's what GLEP 64 is about.  You point argues in favor
> of the GLEP.

Well gentoolkit is Portage-specific, since we've reimplemented all the
useful stuff properly in Paludis... To reimplement gentoolkit in proper
generality, externally, you'd need an awful lot more than what you're
asking for.

> > And when you do ask, is a package that's "provided" installed, and
> > if so, what's its metadata?
> 
> When the package is installed, that data should have been cached.

But package.provided packages *aren't* installed. They are merely
treated as if they were installed, without actually being installed, so
that data isn't available.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] RFC: GLEP 64: Export Package Manager Cached Information. (Was: RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.)

[hasufell]

Anthony G. Basile:
> On 09/06/14 12:12, hasufell wrote:
>> Anthony G. Basile:
>>>> And when you do ask, is a package that's "provided" installed, and if
>>>> so, what's its metadata?
>>>>
>>> When the package is installed, that data should have been cached.
>>>
>> Afaik there is nothing "cached" if you put stuff in package.provided.
>> It's a terrible hack, unless I missed something.
>>
> 
> I wasn't sure what Ciaran was talking about there.  If its hacky, then
> we certainly don't want to standardize it in the GLEP.
> 

Well, you have to to define what tools can expect from
provided/installed packages.

That means either say "you cannot expect anything, because there might
or might not be metadata" or say "you can expect metadata for any
provided/installed package" in which case package.provided feature has
to be removed from portage.

Re: [gentoo-dev] RFC: GLEP 64: Export Package Manager Cached Information. (Was: RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.)

[Anthony G. Basile]

On 09/06/14 12:18, Ciaran McCreesh wrote:
> Well eix is buggy, PMS-violating and doesn't support EAPIs properly,
> and the utilities in gentoolkit and portage-utils are better implemented
> natively in a package manager. I don't know what elfix does, but if
> it's anything like the other three, I'd rather reimplement it in a PMS
> compliant manner for Paludis than to provide flaky external APIs that
> encourage people to write broken code...

elfix contains revdep-pax which does what revdep-rebuild does, except 
that it migrates PaX flags from executables to libraries and vice versa.

There exists a category of tools that can make use of PM's cached 
information which should not be part of PM.  elfix is one such example.  
Let me give you another: 
https://bugs.gentoo.org/show_bug.cgi?id=506276#c42.  That bug is about 
removing SYMLINK_LIB=yes.  To do so properly and migrate systems that 
use symlinks for /libXX, vapier wrote a migration script which at one 
point "walk[s] the vdb looking for files that installed into /lib32 and 
/lib."

These sorts of utilites pop up over and over again.  You cannot put 
every utility that needs package information into a package manager.  An 
API for interoperability between PM's and other tools is meaningful.  
Refusing to do so leads to the sort of comment you see in 
https://bugs.gentoo.org/show_bug.cgi?id=506276#c43.

Ironically, the very standards I seek in GLEP 64 would benefit the 
paludis world most.

>
>> When the package is installed, that data should have been cached.
> But package.provided packages *aren't* installed. They are merely
> treated as if they were installed, without actually being installed, so
> that data isn't available.
>

Right.  hasufell's email made it clear.  This is pretty hacky so we 
don't want to go there.

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] RFC: GLEP 64: Export Package Manager Cached Information. (Was: RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 3394 bytes --]

On Sat, 06 Sep 2014 13:00:03 -0400
"Anthony G. Basile" <blueness@gentoo.org> wrote:
> On 09/06/14 12:18, Ciaran McCreesh wrote:
> > Well eix is buggy, PMS-violating and doesn't support EAPIs properly,
> > and the utilities in gentoolkit and portage-utils are better
> > implemented natively in a package manager. I don't know what elfix
> > does, but if it's anything like the other three, I'd rather
> > reimplement it in a PMS compliant manner for Paludis than to
> > provide flaky external APIs that encourage people to write broken
> > code...
> 
> elfix contains revdep-pax which does what revdep-rebuild does, except 
> that it migrates PaX flags from executables to libraries and vice
> versa.

There's a reason we reimplemented revdep-rebuild: doing it *properly*
requires passing special information to the dependency resolver telling
it not to reuse certain packages for breaking circular dependencies.

> These sorts of utilites pop up over and over again.

A lot of them are because Portage doesn't provide a decent API for
users...

> You cannot put  every utility that needs package information into a
> package manager.

No, just the ones that do anything fiddly, and that certainly includes
eix and most of gentoolkit.

> An API for interoperability between PM's and other
> tools is meaningful. Refusing to do so leads to the sort of comment
> you see in https://bugs.gentoo.org/show_bug.cgi?id=506276#c43.
> 
> Ironically, the very standards I seek in GLEP 64 would benefit the 
> paludis world most.

I'm not disagreeing with an API for interoperability, in principle. I
just think the way you're going about it is all wrong, and we're all
better not having an API at all than having a bad one that we'll have to
support for all eternity. Here's a quick lesson, starting with bad
design:

    for x in /var/db/pkg/*/* ; do
        cat $x/DESCRIPTION
        # etc

But what if VDB isn't there?

    for x in $(pmtool get_vdb_path)/*/* ; do
        cat $x/DESCRIPTION

But */* makes some strong assumptions about the filesystem layout. In
particular, it's going to be wrong if we ever do multi-slot. So:

    for x in $(pmtool get_vdb_package_directories) ; do
        cat $x/DESCRIPTION

But this is still very fragile. What if we switch to sqlite?

    for x in $(pmtool get_unique_ids_for_vdb_packages) ; do
        pmtool get_metadata_variable $x DESCRIPTION

But hang on... We don't really want VDB. We want installed packages.
(This is important, and it's not just a naming thing.)

    for x in $(pmtool get_unique_ids_for_installed_packages) ; do
        pmtool get_metadata_variable $x DESCRIPTION

What if descriptions move to metadata.xml?

    for x in $(pmtool get_unique_ids_for_installed_packages) ; do
        pmtool get_description $x

And finally, what if EAPI 7 changes things? It's probably best to error
out.

    export PMTOOL_BARF_ON_UNKNOWN_EAPIS="1 2 3 4 5 6"
    for x in $(pmtool get_unique_ids_for_installed_packages) ; do
        pmtool get_description $x

Now this is still fragile and makes all kinds of assumptions, and we
could argue eternally about the naming, but it's a heck of a lot better
than what we started off with. In particular, it's *very* high level,
and relies upon the package mangler for everything involving parsing.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] RFC: GLEP 64: Export Package Manager Cached Information. (Was: RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.)

[Anthony G. Basile]

On 09/06/14 13:09, Ciaran McCreesh wrote:
> I'm not disagreeing with an API for interoperability, in principle. I
> just think the way you're going about it is all wrong, and we're all
> better not having an API at all than having a bad one that we'll have to
> support for all eternity.

You and the portage team can get together to discuss how you wish to 
design this API.  The GLEP is clear on this point:

"It is not the purpose of this GLEP to specify the details of a common 
API for exporting the above information. Even less so is it our purpose 
to delineate the implemenatation details for each PM. However, a common 
API for exporting the above information should be developed and 
specified by the PM teams and include in future PMS documentation."

It is incorrect to say "I'm going about it all wrong" when I'm not doing 
any designing.  The GLEP specifies the information to be exported, 
without implementaton details --- it does give mgorny's suggested API 
only as a suggestion.  I've removed all references to VDB as the caching 
mechanism.  Each PM can choose whatever mechanism it wants as long as it 
caches and exports the information specified in the GLEP.  Once the 
teams decide, they document their common API in PMS.


> Here's a quick lesson, starting with bad
> design:
  ....
> Now this is still fragile and makes all kinds of assumptions, and we
> could argue eternally about the naming, but it's a heck of a lot better
> than what we started off with. In particular, it's *very* high level,
> and relies upon the package mangler for everything involving parsing.
>

I trust you and the portage team to do this right.  I've simply pointed 
out instances of utilities which do not belong in a PM, and which can 
make use of information generated by a PM at build time, but which is 
expensive to recalculate later.  This argues for caching this 
information and exporting it via a common API so tools can work as well 
with paludis as with portage as with xyz from the future.

At this point I've gotten enough feedback for the next re-write. 
Basically I'm going to change the emphasis on ELF and make it clearer 
that we're looking for whatever dynamic linking information is 
appropriate for the executable format in question and relegate ELF to an 
example.

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] RFC: GLEP 64: Export Package Manager Cached Information. (Was: RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.)

[Ciaran McCreesh]

[-- Attachment #1: Type: text/plain, Size: 804 bytes --]

On Sat, 06 Sep 2014 14:10:50 -0400
"Anthony G. Basile" <blueness@gentoo.org> wrote:
> It is incorrect to say "I'm going about it all wrong" when I'm not
> doing any designing.

Not doing any designing *is* going about it all wrong... Getting this
right is hard work. Someone needs to do it.

> At this point I've gotten enough feedback for the next re-write. 
> Basically I'm going to change the emphasis on ELF and make it clearer 
> that we're looking for whatever dynamic linking information is 
> appropriate for the executable format in question and relegate ELF to
> an example.

That's all very well, but it's completely useless until someone designs
the whole thing. We're back to specifying the colours of the shelves
before we've designed the bikeshed.

-- 
Ciaran McCreesh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-dev] RFC: GLEP 64: Export Package Manager Cached Information. (Was: RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.)

[Anthony G. Basile]

On 09/06/14 14:15, Ciaran McCreesh wrote:
> On Sat, 06 Sep 2014 14:10:50 -0400
> "Anthony G. Basile" <blueness@gentoo.org> wrote:
>> It is incorrect to say "I'm going about it all wrong" when I'm not
>> doing any designing.
> Not doing any designing *is* going about it all wrong... Getting this
> right is hard work. Someone needs to do it.

I plan to help the portage team with this.  In fact, I really like 
mgorny's query-installed|| thingy.  I am in the process of familiarizing 
myself with paludis and can help there too.  Brace yourself for patches!

>
>> At this point I've gotten enough feedback for the next re-write.
>> Basically I'm going to change the emphasis on ELF and make it clearer
>> that we're looking for whatever dynamic linking information is
>> appropriate for the executable format in question and relegate ELF to
>> an example.
> That's all very well, but it's completely useless until someone designs
> the whole thing. We're back to specifying the colours of the shelves
> before we've designed the bikeshed.
>

This is the beginings of design.  We can talk about something which does 
not exist and then make it exist.

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] RFC: GLEP 64: Export Package Manager Cached Information. (Was: RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.)

[Patrick Lauer]

On Saturday 06 September 2014 16:22:46 hasufell wrote:
> Anthony G. Basile:
> > On 09/06/14 12:12, hasufell wrote:
> >> Anthony G. Basile:
> >>>> And when you do ask, is a package that's "provided" installed, and if
> >>>> so, what's its metadata?
> >>> 
> >>> When the package is installed, that data should have been cached.
> >> 
> >> Afaik there is nothing "cached" if you put stuff in package.provided.
> >> It's a terrible hack, unless I missed something.
> > 
> > I wasn't sure what Ciaran was talking about there.  If its hacky, then
> > we certainly don't want to standardize it in the GLEP.
> 
> Well, you have to to define what tools can expect from
> provided/installed packages.
> 
> That means either say "you cannot expect anything, because there might
> or might not be metadata" or say "you can expect metadata for any
> provided/installed package" in which case package.provided feature has
> to be removed from portage.

"Provided" means "not managed by the package manager" and thus returning 
"empty metadata" for queries is perfectly fine.

I don't see why this feature would need to be removed ...

Re: [gentoo-dev] RFC: GLEP 64: Export Package Manager Cached Information. (Was: RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.)

[Anthony G. Basile]

On 09/07/14 22:36, Patrick Lauer wrote:
> On Saturday 06 September 2014 16:22:46 hasufell wrote:
>> Anthony G. Basile:
>>> On 09/06/14 12:12, hasufell wrote:
>>>> Anthony G. Basile:
>>>>>> And when you do ask, is a package that's "provided" installed, and if
>>>>>> so, what's its metadata?
>>>>> When the package is installed, that data should have been cached.
>>>> Afaik there is nothing "cached" if you put stuff in package.provided.
>>>> It's a terrible hack, unless I missed something.
>>> I wasn't sure what Ciaran was talking about there.  If its hacky, then
>>> we certainly don't want to standardize it in the GLEP.
>> Well, you have to to define what tools can expect from
>> provided/installed packages.
>>
>> That means either say "you cannot expect anything, because there might
>> or might not be metadata" or say "you can expect metadata for any
>> provided/installed package" in which case package.provided feature has
>> to be removed from portage.
> "Provided" means "not managed by the package manager" and thus returning
> "empty metadata" for queries is perfectly fine.
>
> I don't see why this feature would need to be removed ...
>

I will explicitly mention "package.provided" in the GLEP as not 
returning anything for metadata.

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-dev] RFC: GLEP 64: Export Package Manager Cached Information. (Was: RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.)

[hasufell]

Patrick Lauer:
> On Saturday 06 September 2014 16:22:46 hasufell wrote:
>> Anthony G. Basile:
>>> On 09/06/14 12:12, hasufell wrote:
>>>> Anthony G. Basile:
>>>>>> And when you do ask, is a package that's "provided" installed, and if
>>>>>> so, what's its metadata?
>>>>>
>>>>> When the package is installed, that data should have been cached.
>>>>
>>>> Afaik there is nothing "cached" if you put stuff in package.provided.
>>>> It's a terrible hack, unless I missed something.
>>>
>>> I wasn't sure what Ciaran was talking about there.  If its hacky, then
>>> we certainly don't want to standardize it in the GLEP.
>>
>> Well, you have to to define what tools can expect from
>> provided/installed packages.
>>
>> That means either say "you cannot expect anything, because there might
>> or might not be metadata" or say "you can expect metadata for any
>> provided/installed package" in which case package.provided feature has
>> to be removed from portage.
> 
> "Provided" means "not managed by the package manager" and thus returning 
> "empty metadata" for queries is perfectly fine.
> 
> I don't see why this feature would need to be removed ...
> 

You just rephrased "you cannot expect anything, because there might or
might not be metadata".

Re: [gentoo-dev] RFC: GLEP 64: Export Package Manager Cached Information. (Was: RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.)

[hasufell]

Anthony G. Basile:
> On 09/07/14 22:36, Patrick Lauer wrote:
>> On Saturday 06 September 2014 16:22:46 hasufell wrote:
>>> Anthony G. Basile:
>>>> On 09/06/14 12:12, hasufell wrote:
>>>>> Anthony G. Basile:
>>>>>>> And when you do ask, is a package that's "provided" installed,
>>>>>>> and if
>>>>>>> so, what's its metadata?
>>>>>> When the package is installed, that data should have been cached.
>>>>> Afaik there is nothing "cached" if you put stuff in package.provided.
>>>>> It's a terrible hack, unless I missed something.
>>>> I wasn't sure what Ciaran was talking about there.  If its hacky, then
>>>> we certainly don't want to standardize it in the GLEP.
>>> Well, you have to to define what tools can expect from
>>> provided/installed packages.
>>>
>>> That means either say "you cannot expect anything, because there might
>>> or might not be metadata" or say "you can expect metadata for any
>>> provided/installed package" in which case package.provided feature has
>>> to be removed from portage.
>> "Provided" means "not managed by the package manager" and thus returning
>> "empty metadata" for queries is perfectly fine.
>>
>> I don't see why this feature would need to be removed ...
>>
> 
> I will explicitly mention "package.provided" in the GLEP as not
> returning anything for metadata.
> 

Please don't. This is portage internal stuff and shouldn't be in the spec.

The important part is whether API users can always expect non-empty
valid metadata or not.

Is there any good argument for allowing empty metadata except to support
a broken portage implementation?

Re: [gentoo-dev] RFC: GLEP 64: Export Package Manager Cached Information. (Was: RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.)

[Patrick Lauer]

> >> That means either say "you cannot expect anything, because there might
> >> or might not be metadata" or say "you can expect metadata for any
> >> provided/installed package" in which case package.provided feature has
> >> to be removed from portage.
> > 
> > "Provided" means "not managed by the package manager" and thus returning
> > "empty metadata" for queries is perfectly fine.
> > 
> > I don't see why this feature would need to be removed ...
> 
> You just rephrased "you cannot expect anything, because there might or
> might not be metadata".

So you're saying that if I remove metadata it is removed.

Our tautology circle is true! Thanks for contributing these awesome insights.

Re: [gentoo-dev] RFC: GLEP 64: Export Package Manager Cached Information. (Was: RFC: GLEP 64: Standardize contents of VDB and establish and API for exporting this information.)

[hasufell]

Patrick Lauer:
> 
>>>> That means either say "you cannot expect anything, because there might
>>>> or might not be metadata" or say "you can expect metadata for any
>>>> provided/installed package" in which case package.provided feature has
>>>> to be removed from portage.
>>>
>>> "Provided" means "not managed by the package manager" and thus returning
>>> "empty metadata" for queries is perfectly fine.
>>>
>>> I don't see why this feature would need to be removed ...
>>
>> You just rephrased "you cannot expect anything, because there might or
>> might not be metadata".
> 
> So you're saying that if I remove metadata it is removed.
> 
> Our tautology circle is true! Thanks for contributing these awesome insights.
> 

I'm saying that you didn't make an actual argument why it is better to
allow non-existing metadata for installed packages and have every API
user handle this exception.

This will go horribly wrong if we start designing an API around portage
hacks, assuming that the VDB might just be broken, inconsistent or
"maybe valid". Is this about a portage API or a generic PM API?

If it's about the former, then I'd rather see eix and friends broken for
other PMs instead of spreading portage hacks over them (FYI: paludis has
the "provided" case implemented without breaking VDB).