[gentoo-dev] RFC: upstream/watch in metadata.xml

[gentoo-dev] RFC: upstream/watch in metadata.xml

[Corentin Chary]

As some may know, I'm working on euscan, and currently euscan only use
CPV and SRC_URI to find new upstream versions.
This works well if upstream url and version scheme is sane or if
upstream has an API for that (rubygem, pypi, pecl, pear), but it's far
from optimal.

Debian use a specific file for that: debian/watch and it looks like
that (for media-plugins/vdr-softdevice):

opts=downloadurlmangle=s/prdownload/download/ \
   http://developer.berlios.de/project/showfiles.php?group_id=2051 \
   http://prdownload.berlios.de/softdevice/vdr-softdevice-(.+).tgz

opts specify some options to mangle the final url, and then there is a
list of url to scan. man uscan for more informations.

Currently, if you run euscan on this package, it doesn't work at all:
http://euscan.iksaif.net/package/media-plugins/vdr-softdevice/
1/ it's hosted on gentoo mirrors, and scanning them takes too long
because all files are in the same directory
2/ the url doesn't contain the version

So, to help euscan (and other tools) for some package, I think we
could introduce some hints in metadata.xml. This would extend the
existing "upstream" element:

<upstream>
<version-scan downloadurlmangle="s/prdownload/download/">http://developer.berlios.de/project/showfiles.php?group_id=2051
\
   http://prdownload.berlios.de/softdevice/vdr-softdevice-(.+).tgz</version-scan>
</upstream>

The format is not defined yet, but it would probably look like
debian/watch, that would allow to write a script to import (valid)
debian/watch files into associated metadata.xml when needed.

One other thing, metadata.xml already contain a remote-id tag, which
would be very great to help euscan do its job, but a lot of package
are lacking it:
- Should we patch repoman to scan SRC_URI and issue a warning when it
looks like an URI that match a well known remote-id
- Should we write a script to update metadata.xml ? It would be easy
for rubygem, pypi and pear packages.

Any comment ? Objections ? Ideas ?

Thanks,

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] RFC: upstream/watch in metadata.xml

[Dirkjan Ochtman]

On Mon, Feb 13, 2012 at 10:33, Corentin Chary <corentin.chary@gmail.com> wrote:
> One other thing, metadata.xml already contain a remote-id tag, which
> would be very great to help euscan do its job, but a lot of package
> are lacking it:
> - Should we patch repoman to scan SRC_URI and issue a warning when it
> looks like an URI that match a well known remote-id
> - Should we write a script to update metadata.xml ? It would be easy
> for rubygem, pypi and pear packages.
>
> Any comment ? Objections ? Ideas ?

I like the idea for keeping the data somewhere for known-insane cases,
and metadata.xml sounds like it might be fine. But I don't think we
should add anything for the likes of PyPI, if we can easily derive
that we should look on PyPI some other way (i.e. for python, many
packages list a PyPI page in their HOMEPAGE).

Cheers,

Dirkjan

Re: [gentoo-dev] RFC: upstream/watch in metadata.xml

[Corentin Chary]

On Mon, Feb 13, 2012 at 10:50 AM, Dirkjan Ochtman <djc@gentoo.org> wrote:
> On Mon, Feb 13, 2012 at 10:33, Corentin Chary <corentin.chary@gmail.com> wrote:
>> One other thing, metadata.xml already contain a remote-id tag, which
>> would be very great to help euscan do its job, but a lot of package
>> are lacking it:
>> - Should we patch repoman to scan SRC_URI and issue a warning when it
>> looks like an URI that match a well known remote-id
>> - Should we write a script to update metadata.xml ? It would be easy
>> for rubygem, pypi and pear packages.
>>
>> Any comment ? Objections ? Ideas ?
>
> I like the idea for keeping the data somewhere for known-insane cases,
> and metadata.xml sounds like it might be fine. But I don't think we
> should add anything for the likes of PyPI, if we can easily derive
> that we should look on PyPI some other way (i.e. for python, many
> packages list a PyPI page in their HOMEPAGE).

For pypi (and some others), looking at SRC_URI is enought: it starts
with mirror://pypi/.
Still for those <upstream><remote-id> *must* be set because the
package name is not always exactly the same as in gentoo. Currently
euscan tries to guess it, but it is not always accurate.
Most of the time, if remote-id is set, we don't need "version-scan"
because upstream provides a stable API to list versions.


-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] RFC: upstream/watch in metadata.xml

[Robin H. Johnson]

On Mon, Feb 13, 2012 at 10:33:11AM +0100, Corentin Chary wrote:
> Currently, if you run euscan on this package, it doesn't work at all:
> http://euscan.iksaif.net/package/media-plugins/vdr-softdevice/
> 1/ it's hosted on gentoo mirrors, and scanning them takes too long
> because all files are in the same directory
I've been wondering if it would help to have a pregenerated index go out
to the mirrors from our master box, would that be useful for you?

> - Should we write a script to update metadata.xml ? It would be easy
> for rubygem, pypi and pear packages.
CPAN should be easy as well.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-dev] RFC: upstream/watch in metadata.xml

[Corentin Chary]

On Mon, Feb 13, 2012 at 10:52 AM, Robin H. Johnson <robbat2@gentoo.org> wrote:
> On Mon, Feb 13, 2012 at 10:33:11AM +0100, Corentin Chary wrote:
>> Currently, if you run euscan on this package, it doesn't work at all:
>> http://euscan.iksaif.net/package/media-plugins/vdr-softdevice/
>> 1/ it's hosted on gentoo mirrors, and scanning them takes too long
>> because all files are in the same directory
> I've been wondering if it would help to have a pregenerated index go out
> to the mirrors from our master box, would that be useful for you?

Would be better, but the index would still be pretty big (and
currently euscan doesn't cache anything, maybe it should).


-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] RFC: upstream/watch in metadata.xml

[Corentin Chary]

On Mon, Feb 13, 2012 at 10:59 AM, Corentin Chary
<corentin.chary@gmail.com> wrote:
> On Mon, Feb 13, 2012 at 10:52 AM, Robin H. Johnson <robbat2@gentoo.org> wrote:
>> On Mon, Feb 13, 2012 at 10:33:11AM +0100, Corentin Chary wrote:
>>> Currently, if you run euscan on this package, it doesn't work at all:
>>> http://euscan.iksaif.net/package/media-plugins/vdr-softdevice/
>>> 1/ it's hosted on gentoo mirrors, and scanning them takes too long
>>> because all files are in the same directory
>> I've been wondering if it would help to have a pregenerated index go out
>> to the mirrors from our master box, would that be useful for you?
>
> Would be better, but the index would still be pretty big (and
> currently euscan doesn't cache anything, maybe it should).
>

Note that even with a HTTP cache, scanning it would take a lot of CPU
if it is too big :/.

-- 
Corentin Chary
http://xf.iksaif.net