[gentoo-dev] [RFC] Addition of a new field <remote-id type="debian"> to metadata.xml

[gentoo-dev] [RFC] Addition of a new field <remote-id type="debian"> to metadata.xml

[Jonas Stein]

Hello,

this RFC is about the addition of a new field
<remote-id type="debian">
to metadata.xml

The field should contain a list of the equivalent debian package names,
or a defined string for NONE or UNSET.
A feature request ticket was prepared here:
https://bugs.gentoo.org/show_bug.cgi?id=619132


1. Motivation
=============
Gentoo users and users of debian based distributions profit from linked
packages:

* can be used to embed screenshots from https://screenshots.debian.net
in https://packages.gentoo.org/

* when writing/reading documentation which is written for debian like
Linux distributions

* scripts can automatically compare latest versions, critical bugs, from
debian-like distributions and alert the interested maintainer or prepare
statistics. A greasemonkey script can show a link in bugzilla and the
maintainer can see open/solved bugs and the solutions on other
distributions.

* package names hardly differ between *buntu, mint, and debian so we
cover a broad spectrum

* Users who switch from debian like systems to gentoo, or administrate
gentoo and debian like systems at the same time can find the
corresponding packages quickly


2. Specification
================
A space separated list of the corresponding debian packages should be
written in the field
<remote-id type="debian"> </remote-id>

It should be NONE, if debian has no corresponding package.
UNSET or no field, if the creator of the ebuild did not set the field (yet).

example:
app-arch/tar/metadata.xml
<remote-id type="debian">tar</remote-id>

app-office/libreoffice-bin/metadata.xml
<remote-id type="debian">libreoffice libreoffice-base libreoffice-base
libreoffice-dev libreoffice-dmaths libreoffice-draw
libreoffice-evolution libreoffice-impress</remote-id>


3. Backwards compatibility
==========================
Fully compatible, no negative side effects to expect.


4. Comments
===========

What do you think?
Should we exclude translation packages, or just add them all?

-- 
Best regards,
Jonas Stein

Re: [gentoo-dev] [RFC] Addition of a new field <remote-id type="debian"> to metadata.xml

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 707 bytes --]

On Thu, 1 Jun 2017 23:18:22 +0200
Jonas Stein <jstein@gentoo.org> wrote:

> A space separated list of the corresponding debian packages should be
> written in the field
> <remote-id type="debian"> </remote-id>

Why space separated?

Its already legal to specify the field multiple times, and it should
work better that way for consistency with things that can already parse
XML.

That way there's no need to put an additional parser inside our XML
extraction.

<remote-id type="debian">libfoo</remote-id>
<remote-id type="debian">libfoo-debug</remote-id>

No?

It also means general purpose XML formatting tools can keep it tidy,
_and_ sorted, without having to reinvent new tools.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] [RFC] Addition of a new field <remote-id type="debian"> to metadata.xml

[Daniel Campbell]

[-- Attachment #1.1: Type: text/plain, Size: 1282 bytes --]

On 06/01/2017 06:09 PM, Kent Fredric wrote:
> On Thu, 1 Jun 2017 23:18:22 +0200
> Jonas Stein <jstein@gentoo.org> wrote:
> 
>> A space separated list of the corresponding debian packages should be
>> written in the field
>> <remote-id type="debian"> </remote-id>
> 
> Why space separated?
> 
> Its already legal to specify the field multiple times, and it should
> work better that way for consistency with things that can already parse
> XML.
> 
> That way there's no need to put an additional parser inside our XML
> extraction.
> 
> <remote-id type="debian">libfoo</remote-id>
> <remote-id type="debian">libfoo-debug</remote-id>
> 
> No?
> 
> It also means general purpose XML formatting tools can keep it tidy,
> _and_ sorted, without having to reinvent new tools.
> 
+1. Otherwise sounds good. But if we do this for Debian, will there be
movement to add in package names for rpm-based distros? Arch? BSD?
Slackware? Where do we draw the line?

Will developers be expected to treat this like a mandated element? If
not, which team will have authority to touch package metadata to make
this change?
-- 
Daniel Campbell - Gentoo Developer
OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net
fpr: AE03 9064 AE00 053C 270C  1DE4 6F7A 9091 1EA0 55D6


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] [RFC] Addition of a new field <remote-id type="debian"> to metadata.xml

[Jonas Stein]

>> A space separated list of the corresponding debian packages should be
>> written in the field
>> <remote-id type="debian"> </remote-id>
> Why space separated?
> Its already legal to specify the field multiple times, and it should
> work better that way for consistency with things that can already parse
> XML.
> 
> That way there's no need to put an additional parser inside our XML
> extraction.
> 
> <remote-id type="debian">libfoo</remote-id>
> <remote-id type="debian">libfoo-debug</remote-id>
> 
> No?
> 
> It also means general purpose XML formatting tools can keep it tidy,
> _and_ sorted, without having to reinvent new tools.

That is a very good point.
We should note it as you suggested, because it fits better to xml.
Thank you.
Best,

-- 
Jonas Stein

Re: [gentoo-dev] [RFC] Addition of a new field <remote-id type="debian"> to metadata.xml

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1589 bytes --]

On Thu, 1 Jun 2017 18:36:24 -0700
Daniel Campbell <zlg@gentoo.org> wrote:

> +1. Otherwise sounds good. But if we do this for Debian, will there be
> movement to add in package names for rpm-based distros? Arch? BSD?
> Slackware? Where do we draw the line?

I'd say "as need be". Here we have a few extra benefits from using a
debian identifier that aren't strictly related to "package mapping".

Its also easier for third party services to use our use of debian
identifiers to produce the other mappings for us where known ( that is,
non-gentoo entities can maintain a mapping of debian-to-foo, and we can
trivially hook into that by using the debian-id as the identifer )

> Will developers be expected to treat this like a mandated element?

I'd imagine not, given not everything in debian exists in gentoo, or
vice versa.

Similarly, I don't think there are any mandates that the other values
of remote-id be populated, only that its *encouraged* because that data
provides utility to an end user.

> If
> not, which team will have authority to touch package metadata to make
> this change?

I'd suggest it should stay within the controls of the package
maintainer for starters, where individual maintainers can provision it
as they feel fit, and we can review our stance on this later if we want
to make it a tree wide consistent thing.

Partly because individual maintainers are more likely to understand
correctly how equivalent their dists are to the referenced debian one,
and be more equipped to decide whether to include/exclude a given ref.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] [RFC] Addition of a new field <remote-id type="debian"> to metadata.xml

[Vadim A. Misbakh-Soloviov]

> <remote-id type="debian">libfoo-debug</remote-id>

Shouldn't we mention "debug" USE-flag in this context somehow?

Re: [gentoo-dev] [RFC] Addition of a new field <remote-id type="debian"> to metadata.xml

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1237 bytes --]

On Fri, 02 Jun 2017 14:07:44 +0700
"Vadim A. Misbakh-Soloviov" <gentoo@mva.name> wrote:

> Shouldn't we mention "debug" USE-flag in this context somehow?

Not sure it should. Even though one package may be the logical equivalent
of a handful of debian packages, doesn't mean there's going to be a useful
USE <-> package mapping for such things.

Maybe debug seems more obvious, but, well, with Gentoo, you don't actually *need*
a USE flag for that:

You control that via CFLAGS and the FEATURES="splitdebug" options.

USE="debug" *changes the compile* and introduces source-code level enhancements,
that can change the resulting binary parts.

Whereas the debian equivalents are just the equivalents of our splitdebugs rolled
up so you can debug stack traces with gdb, if I remember correctly.

eg: https://packages.debian.org/jessie/amd64/vlc-dbg/filelist 

In hindsight, maybe -dbg mappings might not make sense, and should be discouraged.

Otherwise you're gonna need to add a descriptive field of some kind to the <remote-id> data,
and at that point, it gets really hairy.

I have ideas how you get around that, but it moves away from <remote-id> entirely, and
that's really just over-complicating it.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] [RFC] Addition of a new field <remote-id type="debian"> to metadata.xml

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1489 bytes --]

On sob, 2017-06-03 at 02:38 +1200, Kent Fredric wrote:
> On Fri, 02 Jun 2017 14:07:44 +0700
> "Vadim A. Misbakh-Soloviov" <gentoo@mva.name> wrote:
> 
> > Shouldn't we mention "debug" USE-flag in this context somehow?
> 
> Not sure it should. Even though one package may be the logical equivalent
> of a handful of debian packages, doesn't mean there's going to be a useful
> USE <-> package mapping for such things.
> 
> Maybe debug seems more obvious, but, well, with Gentoo, you don't actually *need*
> a USE flag for that:
> 
> You control that via CFLAGS and the FEATURES="splitdebug" options.
> 
> USE="debug" *changes the compile* and introduces source-code level enhancements,
> that can change the resulting binary parts.
> 
> Whereas the debian equivalents are just the equivalents of our splitdebugs rolled
> up so you can debug stack traces with gdb, if I remember correctly.
> 
> eg: https://packages.debian.org/jessie/amd64/vlc-dbg/filelist 
> 
> In hindsight, maybe -dbg mappings might not make sense, and should be discouraged.
> 
> Otherwise you're gonna need to add a descriptive field of some kind to the <remote-id> data,
> and at that point, it gets really hairy.
> 
> I have ideas how you get around that, but it moves away from <remote-id> entirely, and
> that's really just over-complicating it.

...so if a Gentoo package is split into 40 packages in Debian, are you
going to list all of them?

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 988 bytes --]

Re: [gentoo-dev] [RFC] Addition of a new field <remote-id type="debian"> to metadata.xml

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 533 bytes --]

On Fri, 02 Jun 2017 16:51:25 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> ...so if a Gentoo package is split into 40 packages in Debian, are you
> going to list all of them?

If it would be useful to do so, maybe.

But its a text file, people are capable of making judgements about
adding 3.2k of text, I hope. ( worst-case, 40 lines of 80 characters each )

If the text at that size has a use, then, why not?

As long as that 40-package example is an exceptional case, where it was deemed
useful, not the norm.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] [RFC] Addition of a new field <remote-id type="debian"> to metadata.xml

[Daniel Campbell]

[-- Attachment #1.1: Type: text/plain, Size: 1914 bytes --]

On 06/01/2017 11:59 PM, Kent Fredric wrote:
> On Thu, 1 Jun 2017 18:36:24 -0700
> Daniel Campbell <zlg@gentoo.org> wrote:
> 
>> +1. Otherwise sounds good. But if we do this for Debian, will there be
>> movement to add in package names for rpm-based distros? Arch? BSD?
>> Slackware? Where do we draw the line?
> 
> I'd say "as need be". Here we have a few extra benefits from using a
> debian identifier that aren't strictly related to "package mapping".
> 
> Its also easier for third party services to use our use of debian
> identifiers to produce the other mappings for us where known ( that is,
> non-gentoo entities can maintain a mapping of debian-to-foo, and we can
> trivially hook into that by using the debian-id as the identifer )
> 
>> Will developers be expected to treat this like a mandated element?
> 
> I'd imagine not, given not everything in debian exists in gentoo, or
> vice versa.
> 
> Similarly, I don't think there are any mandates that the other values
> of remote-id be populated, only that its *encouraged* because that data
> provides utility to an end user.
> 
>> If
>> not, which team will have authority to touch package metadata to make
>> this change?
> 
> I'd suggest it should stay within the controls of the package
> maintainer for starters, where individual maintainers can provision it
> as they feel fit, and we can review our stance on this later if we want
> to make it a tree wide consistent thing.
> 
> Partly because individual maintainers are more likely to understand
> correctly how equivalent their dists are to the referenced debian one,
> and be more equipped to decide whether to include/exclude a given ref.
> 

That sounds very reasonable to me. Thanks for clarifying.

-- 
Daniel Campbell - Gentoo Developer
OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net
fpr: AE03 9064 AE00 053C 270C  1DE4 6F7A 9091 1EA0 55D6


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] [RFC] Addition of a new field <remote-id type="debian"> to metadata.xml

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1006 bytes --]

On sob, 2017-06-03 at 03:22 +1200, Kent Fredric wrote:
> On Fri, 02 Jun 2017 16:51:25 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
> 
> > ...so if a Gentoo package is split into 40 packages in Debian, are you
> > going to list all of them?
> 
> If it would be useful to do so, maybe.
> 
> But its a text file, people are capable of making judgements about
> adding 3.2k of text, I hope. ( worst-case, 40 lines of 80 characters each )
> 
> If the text at that size has a use, then, why not?
> 
> As long as that 40-package example is an exceptional case, where it was deemed
> useful, not the norm.

It's not that exceptional in binary distributions where it's normal to
split a single source into a few dozen packages (libraries, headers,
tools, plugins...):

https://packages.debian.org/source/sid/systemd

and that's a small one. I guess we could avoid this if you restricted
those remotes to the source package used to build them all.

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 988 bytes --]

Re: [gentoo-dev] [RFC] Addition of a new field <remote-id type="debian"> to metadata.xml

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 875 bytes --]

On Sat, 03 Jun 2017 09:58:28 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> and that's a small one. I guess we could avoid this if you restricted
> those remotes to the source package used to build them all.

I think in the event they're a form of conventional 

  foo
  foo-dev
  foo-dbg

etc, under the knowledge that those things can't possibly map to other
gentoo packages, we should codify only the first of those, and then have
it placed on the iteroperating code to make logical inferences from
that.

"foo-dev" in a search query would map to "foo" if no "foo-dev" existed.

But yeah, lots of complexity there.

That's why I'd just say those facets shouldn't really be mapped
explicitly.

The general pattern being:

 "If a debian id can be conjugated from another debian id by guessing
with a generic algorithm, only index the former"

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] [RFC] Addition of a new field <remote-id type="debian"> to metadata.xml

[Andrey Utkin]

On Sat, Jun 03, 2017 at 08:19:32PM +1200, Kent Fredric wrote:
> On Sat, 03 Jun 2017 09:58:28 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
> 
> > and that's a small one. I guess we could avoid this if you restricted
> > those remotes to the source package used to build them all.
> 
> I think in the event they're a form of conventional 
> 
>   foo
>   foo-dev
>   foo-dbg
> 
> etc, under the knowledge that those things can't possibly map to other
> gentoo packages, we should codify only the first of those, and then have
> it placed on the iteroperating code to make logical inferences from
> that.
> 
> "foo-dev" in a search query would map to "foo" if no "foo-dev" existed.
> 
> But yeah, lots of complexity there.
> 
> That's why I'd just say those facets shouldn't really be mapped
> explicitly.
> 
> The general pattern being:
> 
>  "If a debian id can be conjugated from another debian id by guessing
> with a generic algorithm, only index the former"

It is common for debian package names contain version numbers, besides
other ugliness:

https://packages.debian.org/search?keywords=libavcodec&searchon=names&suite=stable&section=all

You have searched for packages that names contain libavcodec in suite(s)
stable, all sections, and all architectures. Found 4 matching packages.
Package libavcodec-dev
Package libavcodec-extra
Package libavcodec-extra-56
Package libavcodec56

Obviously numbered package name libavcodec56 can be an attribute of
exact ebuild, but not of a Gentoo package.

With this ugliness of Debian packages naming, I'm afraid the proposed
change won't take off.

Re: [gentoo-dev] [RFC] Addition of a new field <remote-id type="debian"> to metadata.xml

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1048 bytes --]

On Sun, 4 Jun 2017 13:56:52 +0100
Andrey Utkin <andrey_utkin@fastmail.com> wrote:

> You have searched for packages that names contain libavcodec in
> suite(s) stable, all sections, and all architectures. Found 4
> matching packages. Package libavcodec-dev
> Package libavcodec-extra
> Package libavcodec-extra-56
> Package libavcodec56
> 
> Obviously numbered package name libavcodec56 can be an attribute of
> exact ebuild, but not of a Gentoo package.

Sure, but these sorts of things is why it wouldn't be mandatory to have
any of these, and it wouldn't be mandatory to have a complete mapping.

It would only be provisioned in as far as the maintainer of that
package considered it useful, at their discretion.

And even if you included several dozen, what are the downsides other
than "ugly"?

If the *utility* they provide outweighs the "ugly" factor, and there
are no *technical* downsides of this "ugliness", then surely, the
utility wins.

An index that is incomplete here is still better than no index at all.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] [RFC] Addition of a new field <remote-id type="debian"> to metadata.xml

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 1692 bytes --]

On Sat, Jun 3, 2017 at 3:58 AM, Michał Górny <mgorny@gentoo.org> wrote:

> On sob, 2017-06-03 at 03:22 +1200, Kent Fredric wrote:
> > On Fri, 02 Jun 2017 16:51:25 +0200
> > Michał Górny <mgorny@gentoo.org> wrote:
> >
> > > ...so if a Gentoo package is split into 40 packages in Debian, are you
> > > going to list all of them?
> >
> > If it would be useful to do so, maybe.
> >
> > But its a text file, people are capable of making judgements about
> > adding 3.2k of text, I hope. ( worst-case, 40 lines of 80 characters
> each )
> >
> > If the text at that size has a use, then, why not?
> >
> > As long as that 40-package example is an exceptional case, where it was
> deemed
> > useful, not the norm.
>
> It's not that exceptional in binary distributions where it's normal to
> split a single source into a few dozen packages (libraries, headers,
> tools, plugins...):
>
> https://packages.debian.org/source/sid/systemd
>
> and that's a small one. I guess we could avoid this if you restricted
> those remotes to the source package used to build them all.
>

I would argue that specifying source package remotes is what should be done
for every package. UI layers can use the source package remote to look up
metadata for the source package and get a list of the binary packages it
produces when built; maintaining this list inside of Gentoo seems ill
advised. (I could perhaps be swayed by a script where you populate the
source package remote and a tool fills in the binary packages as sourced
from debian sid or whatever.)

Do we really need to store and distribute this data though?

-A


>
> --
> Best regards,
> Michał Górny
>

[-- Attachment #2: Type: text/html, Size: 2517 bytes --]

Re: [gentoo-dev] [RFC] Addition of a new field <remote-id type="debian"> to metadata.xml

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1144 bytes --]

On Mon, 5 Jun 2017 11:40:15 -0400
Alec Warner <antarus@gentoo.org> wrote:

> Do we really need to store and distribute this data though?

Aggregating this kind of data by cross-referencing multiple providers
and then trying discovery against debians equivalents of that, while
workable, would be likely fragile and unpredictable.

If you can determine this list yourself from the existing data, I'd
suggest we start with that, and then see how painful it is to do on the
fly.

I'd expect the round-trip times to resolve such questions would be
little expensive, so you'd want a sort of job that does this
periodically and creates an index outside of the repository, and then
services can query *that* for things like "Can haz an screenshots?" 

And then, it could be reasonable to then simplify this proposal such
that the typical usage of introducing additional data would be to
exclusively augment data that can't be discovered by other means.

This would solve *both* the problems of "hard to automate" and "too much
cruft in the repo", while keeping the hand-edited metadata close to the
package definitions.




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] [RFC] Addition of a new field <remote-id type="debian"> to metadata.xml

[Sebastian Pipping]

On 01.06.2017 23:18, Jonas Stein wrote:
> 2. Specification
> ================
> A space separated list of the corresponding debian packages should be
> written in the field
> <remote-id type="debian"> </remote-id>
> 
> It should be NONE, if debian has no corresponding package.
> UNSET or no field, if the creator of the ebuild did not set the field (yet).

Please pick NONE or require absence eventually, but not multiple
options.  Else we're asking for inconsistent data from the beginning.


> example:
> app-arch/tar/metadata.xml
> <remote-id type="debian">tar</remote-id>
> 
> app-office/libreoffice-bin/metadata.xml
> <remote-id type="debian">libreoffice libreoffice-base libreoffice-base
> libreoffice-dev libreoffice-dmaths libreoffice-draw
> libreoffice-evolution libreoffice-impress</remote-id>

Since the difference between source and binary packages has already been
brought up, please adjust "<remote-id type="debian">" some way to
indicating if the text content is a source or a binary package (even if
we don't end up supporting both) to be 100% clear.  Otherwise people
will mix it up, and may not even notice.

Best



Sebastian