[gentoo-dev] euscan proof of concept (like debian's uscan)

[gentoo-dev] euscan proof of concept (like debian's uscan)

[Corentin Chary]

Hi,

I recently started working on a small gentoo utility named "euscan"
(for Ebuild Upstream Scan)
For those who don't know debian's uscan, it allows to scan upstream
for new versions. It's used by packages.qa.debian.org (example:
http://packages.qa.debian.org/p/php-net-ipv4.html ).
It's available at: http://xf.iksaif.net/bordel/euscan

Currently, it uses two heuristics to find new versions, both based on
SRC_URI and PV:
- Directory scanning: scan directories to find files with newer version
- Brute Force: generate new possible versions, and try to download files

Note that it also works when only a part of the version is available
in the url.

I think that it would be great to have these informations on
http://packages.gentoo.org/ and/or unofficial
http://gentoo-portage.com/ website.

We could also add the ability to browse packages by maintainer to help
them see if they have any outdated package.

I started this mostly to see if it was possible, and I don't know if
i'll have the time to continue to work on that project, but I think
gentoo really needs an automated way to detect outdated packages.
This could also be a 2011 GSOC project (finishing euscan, and adding a
web based interface to browse the results).

Examples:

$ ./euscan PEAR-Validate-0.8.3
Package: dev-php/PEAR-Validate-0.8.3
Herd: php
Maintainer: none
Location: /usr/portage/dev-php/PEAR-Validate
 * Scanning: http://pear.php.net/get/Validate-${PV}.tgz
 * Scanning: http://pear.php.net/get
 * Generating version from 0.8.3
 * Brute forcing: http://pear.php.net/get/Validate-${PV}.tgz
 * Trying: http://pear.php.net/get/Validate-0.8.4.tgz ...               [ ok ]
 * Trying: http://pear.php.net/get/Validate-0.8.5.tgz ...               [ !! ]
 * Trying: http://pear.php.net/get/Validate-0.9.0.tgz ...               [ !! ]
 * Trying: http://pear.php.net/get/Validate-0.10.0.tgz ...              [ !! ]
 * Trying: http://pear.php.net/get/Validate-0.8.6.tgz ...               [ !! ]
New Upstream Version: 0.8.4 http://pear.php.net/get/Validate-0.8.4.tgz

$ ./euscan icu4j-4.4.2
Package: dev-java/icu4j-4.4.2
Herd: java
Maintainer: none
Location: /usr/portage/dev-java/icu4j
 * Scanning: http://download.icu-project.org/files/icu4j/${PV}/icu4j-4_4_2-src.jar
 * Scanning: http://download.icu-project.org/files/icu4j
New Upstream Version: 4.5.1 http://download.icu-project.org/files/icu4j/4.5.1/
New Upstream Version: 4.5.2 http://download.icu-project.org/files/icu4j/4.5.2/
New Upstream Version: 4.6 http://download.icu-project.org/files/icu4j/4.6/
New Upstream Version: 4.6rc2 http://download.icu-project.org/files/icu4j/4.6rc2/
New Upstream Version: 4.7.1 http://download.icu-project.org/files/icu4j/4.7.1/
New Upstream Version: 4.6.1 http://download.icu-project.org/files/icu4j/4.6.1/

$ ./euscan IceE-1.3.0-r1
Package: dev-cpp/IceE-1.3.0-r1
Herd: no-herd
Maintainer: maintainer-needed@gentoo.org
Description: The Internet Communications Engine (Ice) is a modern
object-oriented middleware with support for C++, .NET, Java, Python,
Ruby, and PHP
Location: /usr/portage/dev-cpp/IceE
 * Scanning: http://www.zeroc.com/download/IceE/${0}.${1}//IceE-${PV}-linux.tar.gz
 * Scanning: http://www.zeroc.com/download/IceE
 * Generating version from 1.3.0
 * Brute forcing:
http://www.zeroc.com/download/IceE/${0}.${1}//IceE-${PV}-linux.tar.gz
 * Trying: http://www.zeroc.com/download/IceE/1.3//IceE-1.3.1-linux.tar.gz
...      [ !! ]
 * Trying: http://www.zeroc.com/download/IceE/1.3//IceE-1.3.2-linux.tar.gz
...      [ !! ]
 * Trying: http://www.zeroc.com/download/IceE/1.4//IceE-1.4.0-linux.tar.gz
...      [ !! ]
 * Trying: http://www.zeroc.com/download/IceE/1.5//IceE-1.5.0-linux.tar.gz
...      [ !! ]

Thanks,
-- 
Corentin Chary
http://xf.iksaif.net

[gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Christian Faulhammer]

[-- Attachment #1: Type: text/plain, Size: 592 bytes --]

Hi,

Corentin Chary <corentin.chary@gmail.com>:
> I started this mostly to see if it was possible, and I don't know if
> i'll have the time to continue to work on that project, but I think
> gentoo really needs an automated way to detect outdated packages.
> This could also be a 2011 GSOC project (finishing euscan, and adding a
> web based interface to browse the results).

 Have you applied for a place in GSoC?

V-Li

-- 
Christian Faulhammer, Gentoo Lisp project
<URL:http://www.gentoo.org/proj/en/lisp/>, #gentoo-lisp on FreeNode

<URL:http://gentoo.faulhammer.org/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

On Sat, Mar 26, 2011 at 5:35 PM, Christian Faulhammer <fauli@gentoo.org> wrote:
> Hi,
>
> Corentin Chary <corentin.chary@gmail.com>:
>> I started this mostly to see if it was possible, and I don't know if
>> i'll have the time to continue to work on that project, but I think
>> gentoo really needs an automated way to detect outdated packages.
>> This could also be a 2011 GSOC project (finishing euscan, and adding a
>> web based interface to browse the results).
>
>  Have you applied for a place in GSoC?
>
> V-Li

I can't apply to GSoC, I was suggesting that another student could work on that.

-- 
Corentin Chary
http://xf.iksaif.net

[gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

Hi again,

I found the time to work a little more on euscan:

- Git tree: http://git.iksaif.net/?p=euscan.git
- Demo web interface: http://xf.iksaif.net/dev/.euscan/web/

To feed the web interface, I created a quick&dirty euscan-update. It
currently depends on eix-0.22.5 and gentoolkit-0.3.0.

I launched a full portage scan yesterday, and as you can see on web
interface, it seems to work. You can browse by category, herd, and
maintainer.
The scan is not finished yet (currently scanning net-misc) and keep in
mind that euscan doesn't support mirror:// urls (although, it would be
easy to support some of them).

But there is still a lot to do to make something useful with that, and
I said before, I don't have a lot of time allocated to that project,
so any help would be great.

Quick TODO and ideas:
euscan:
- clean euscan and add options parsing capabilities, with tons of options
- support mirror:// (when possible)
- fix known bugs (see BUGS)

euscan-update:
- clean
- optimize and parallelize
- remove gentoolkit dependency, handle overlays, handle cumulative scans

web interface:
- display all kind of informations (metadatas, keywords, etc...)
- create charts and statistics
- mark/report false positives
- display euscan-update status
- display last euscan output message
- show when euscan can/cannot guess upstream version for a given package

misc:
- automatic bug report
- automatic email report for maintainer/herds

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Hans de Graaff]

[-- Attachment #1: Type: text/plain, Size: 752 bytes --]

On Sun, 2011-04-03 at 17:20 +0000, Corentin Chary wrote:

> misc:
> - automatic bug report
> - automatic email report for maintainer/herds

I'm not sure if this makes sense. For example, it gets dev-lang/ruby
wrong, thinking that our old patch files are missing versions. Also, for
exiftool I only add production releases, so there will almost always be
newer upstream intermediate releases. My point here is that there are
often additional considerations in considering new releases, so fully
automating this is going to be hard. Perhaps a weekly/monthly opt-in
mail to the herd/maintainer with an overview would be useful?

The website is nice to have as another source of scanning for updates,
bookmarked.

Kind regards,

Hans


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

On Sun, Apr 10, 2011 at 8:43 AM, Hans de Graaff <graaff@gentoo.org> wrote:
> On Sun, 2011-04-03 at 17:20 +0000, Corentin Chary wrote:
>
>> misc:
>> - automatic bug report
>> - automatic email report for maintainer/herds
>
> I'm not sure if this makes sense. For example, it gets dev-lang/ruby
> wrong, thinking that our old patch files are missing versions. Also, for
> exiftool I only add production releases, so there will almost always be
> newer upstream intermediate releases. My point here is that there are
> often additional considerations in considering new releases, so fully
> automating this is going to be hard. Perhaps a weekly/monthly opt-in
> mail to the herd/maintainer with an overview would be useful?

Better, what about a per-herd/per-maintainer rss feed ?

> The website is nice to have as another source of scanning for updates,
> bookmarked.

The current site won't be updated, I'm working on a new django-based
website that should be up and running next week.
The url will probably be euscan.iksaif.net, but I'll post it here.

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Matthew Summers]

On Wed, Apr 13, 2011 at 1:47 AM, Corentin Chary
<corentin.chary@gmail.com> wrote:
> On Sun, Apr 10, 2011 at 8:43 AM, Hans de Graaff <graaff@gentoo.org> wrote:
>> On Sun, 2011-04-03 at 17:20 +0000, Corentin Chary wrote:
>>
>>> misc:
>>> - automatic bug report
>>> - automatic email report for maintainer/herds
>>
>> I'm not sure if this makes sense. For example, it gets dev-lang/ruby
>> wrong, thinking that our old patch files are missing versions. Also, for
>> exiftool I only add production releases, so there will almost always be
>> newer upstream intermediate releases. My point here is that there are
>> often additional considerations in considering new releases, so fully
>> automating this is going to be hard. Perhaps a weekly/monthly opt-in
>> mail to the herd/maintainer with an overview would be useful?
>
> Better, what about a per-herd/per-maintainer rss feed ?
>
>> The website is nice to have as another source of scanning for updates,
>> bookmarked.
>
> The current site won't be updated, I'm working on a new django-based
> website that should be up and running next week.
> The url will probably be euscan.iksaif.net, but I'll post it here.
>
> --
> Corentin Chary
> http://xf.iksaif.net
>
>

Hi Corentin,

Do you have a public repo for your django code/site available? I would
really enjoy taking a look at what you have here, sounds cool.

Thanks,
Matt
-- 
Matthew W. Summers

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

> Hi Corentin,
>
> Do you have a public repo for your django code/site available? I would
> really enjoy taking a look at what you have here, sounds cool.
>
> Thanks,
> Matt

Hi,
Yes, my git repo is here: http://git.iksaif.net/?p=euscan.git;a=summary
Current dependencies are: eix-0.22.5, gentoolkit-0.3.0 and portage of course.

I didn't took the time to write any kind of documentation, but here is
what you'll find in the repo:
- euscan standalone utility
- euscanwww

Currently euscanwww only have basic templates, models and view. But
you can alreay run these commands to fill the database:

$ emerge --sync
$ eix-update
$ python manage.py scan-portage --all
$ python manage.py scan-metadata --all

I'll implement scan-upstream soon.

The old index.php is still available in earlier revisions.

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

On Wed, Apr 13, 2011 at 4:58 PM, Corentin Chary
<corentin.chary@gmail.com> wrote:
>> Hi Corentin,
>>
>> Do you have a public repo for your django code/site available? I would
>> really enjoy taking a look at what you have here, sounds cool.
>>
>> Thanks,
>> Matt

Just pushed scan-upstream command (with --parallel support !).
Now, it's time to do a full scan and implement the views.
I staying on #gentoo-unregistered (iksaif) if someone want to get
involved in euscan :).

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

New website is up and running at

http://euscan.iksaif.net/

The git tree is still at http://git.iksaif.net/?p=euscan.git;a=summary

TODO:
- Make some charts to see how it's going
- Finish the "scan my world" feature
- Add a way to subsribe to herds/maintainer/packages in order to
receive weekly/monthly reports

I'll gladly accept any patch ! :)

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Marijn]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Corentin,

On 04/16/11 08:29, Corentin Chary wrote:
> New website is up and running at
> 
> http://euscan.iksaif.net/
> 
> The git tree is still at http://git.iksaif.net/?p=euscan.git;a=summary
> 
> TODO:
> - Make some charts to see how it's going
> - Finish the "scan my world" feature
> - Add a way to subsribe to herds/maintainer/packages in order to
> receive weekly/monthly reports
> 
> I'll gladly accept any patch ! :)

Nice work!

Marijn
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk2r9ckACgkQp/VmCx0OL2zW3ACfUiw+r/FIkAoUxFyBuD/73mYl
Fl4An2ngYC06EXM5vYq6vkiam6FRtBQQ
=69Nh
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

On Mon, Apr 18, 2011 at 10:26 AM, Marijn <hkBst@gentoo.org> wrote:
> Nice work!

Thanks !

Fyi: I ran a new scan this morning. It only took 3 hours !

GNU's Parallel is really a great tool:
`python manage.py list-packages | gparallel --eta --progress --jobs
400% euscan | python manage.py scan-upstream --feed > /dev/null`

I'll try to implement charts as soon as I've got enought data.
-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 1057 bytes --]

On 06:29 Sat 16 Apr     , Corentin Chary wrote:
> New website is up and running at
> 
> http://euscan.iksaif.net/
> 
> The git tree is still at http://git.iksaif.net/?p=euscan.git;a=summary
> 
> TODO:
> - Make some charts to see how it's going
> - Finish the "scan my world" feature
> - Add a way to subsribe to herds/maintainer/packages in order to
> receive weekly/monthly reports
> 
> I'll gladly accept any patch ! :)

This is cool! I just looked through the x11-team packages and it seems 
very useful already. I have a few suggestions if you have time. =)

- Some teams have "official" overlays. Supporting those as an additional 
  source of ebuilds would be pretty nice.

- Another useful thing would be a way to supply CPV tokens for any 
  "unstable" upstream series that we'll never add to the tree.

- This looks like a problem (a versioned tarball named cairo-5c):
  http://euscan.iksaif.net/package/x11-libs/cairo/


-- 
Thanks,
Donnie

Donnie Berkholz
Sr. Developer, Gentoo Linux
Blog: http://dberkholz.com

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

On Mon, Apr 18, 2011 at 3:32 PM, Donnie Berkholz <dberkholz@gentoo.org> wrote:
> On 06:29 Sat 16 Apr     , Corentin Chary wrote:
>> New website is up and running at
>>
>> http://euscan.iksaif.net/
>>
>> The git tree is still at http://git.iksaif.net/?p=euscan.git;a=summary
>>
>> TODO:
>> - Make some charts to see how it's going
>> - Finish the "scan my world" feature
>> - Add a way to subsribe to herds/maintainer/packages in order to
>> receive weekly/monthly reports
>>
>> I'll gladly accept any patch ! :)
>
> This is cool! I just looked through the x11-team packages and it seems
> very useful already. I have a few suggestions if you have time. =)
>
> - Some teams have "official" overlays. Supporting those as an additional
>  source of ebuilds would be pretty nice.

I'll just checkout these trees on my machine (and only enable them for euscan).
euscan use portage, gentoolkit and eix internaly, so overlays are not
a problem :).

Where could I get a list of "official" overlays that should be added ?

> - Another useful thing would be a way to supply CPV tokens for any
>  "unstable" upstream series that we'll never add to the tree.

There is already some kind of blacklist in euscan. Could you provide
examples so I can implement that ?

> - This looks like a problem (a versioned tarball named cairo-5c):
>  http://euscan.iksaif.net/package/x11-libs/cairo/

Hum ... yes, I'll try to use better regex :).

Thanks !
-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Daniel Pielmeier]

[-- Attachment #1: Type: text/plain, Size: 931 bytes --]

Corentin Chary schrieb am 18.04.2011 11:05:
>
> Fyi: I ran a new scan this morning. It only took 3 hours !
> 
> GNU's Parallel is really a great tool:
> `python manage.py list-packages | gparallel --eta --progress --jobs
> 400% euscan | python manage.py scan-upstream --feed > /dev/null`
> 
> I'll try to implement charts as soon as I've got enought data.

I really like it. Keep up the good work. It gives a warm and fuzzy
feeling if there are no unpackaged releases for the own packages :)

There is one issue I recognized. How do you get the links for the
unpackaged upstream versions and do you verify if the listed tarballs
exist? I am asking because for app-editors/bluefish [1] there are eleven
unpackaged versions listed but only sources for the three 2.0.3 releases
exist and there are no releases for 2.1.x and 2.2.x.

http://euscan.iksaif.net/package/app-editors/bluefish/

-- 
Daniel Pielmeier


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

On Mon, Apr 18, 2011 at 1:48 PM, Corentin Chary
<corentin.chary@gmail.com> wrote:
> On Mon, Apr 18, 2011 at 3:32 PM, Donnie Berkholz <dberkholz@gentoo.org> wrote:
>> On 06:29 Sat 16 Apr     , Corentin Chary wrote:
>>> New website is up and running at
>>>
>>> http://euscan.iksaif.net/
>>>
>>> The git tree is still at http://git.iksaif.net/?p=euscan.git;a=summary
>>>
>>> TODO:
>>> - Make some charts to see how it's going
>>> - Finish the "scan my world" feature
>>> - Add a way to subsribe to herds/maintainer/packages in order to
>>> receive weekly/monthly reports
>>>
>>> I'll gladly accept any patch ! :)
>>
>> This is cool! I just looked through the x11-team packages and it seems
>> very useful already. I have a few suggestions if you have time. =)
>>
>> - Some teams have "official" overlays. Supporting those as an additional
>>  source of ebuilds would be pretty nice.
>
> I'll just checkout these trees on my machine (and only enable them for euscan).
> euscan use portage, gentoolkit and eix internaly, so overlays are not
> a problem :).
>
> Where could I get a list of "official" overlays that should be added ?
>
>> - Another useful thing would be a way to supply CPV tokens for any
>>  "unstable" upstream series that we'll never add to the tree.
>
> There is already some kind of blacklist in euscan. Could you provide
> examples so I can implement that ?
>
>> - This looks like a problem (a versioned tarball named cairo-5c):
>>  http://euscan.iksaif.net/package/x11-libs/cairo/
>
> Hum ... yes, I'll try to use better regex :).

Fixed in http://git.iksaif.net/?p=euscan.git;a=commitdiff;h=a7a15c0ac72178bf2f3a73f94a70113ed9856e5c;hp=e5278e0e0f8cf617dff15f84540d0d6604081e82



-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

On Mon, Apr 18, 2011 at 5:22 PM, Daniel Pielmeier <billie@gentoo.org> wrote:
> Corentin Chary schrieb am 18.04.2011 11:05:
>>
>> Fyi: I ran a new scan this morning. It only took 3 hours !
>>
>> GNU's Parallel is really a great tool:
>> `python manage.py list-packages | gparallel --eta --progress --jobs
>> 400% euscan | python manage.py scan-upstream --feed > /dev/null`
>>
>> I'll try to implement charts as soon as I've got enought data.
>
> I really like it. Keep up the good work. It gives a warm and fuzzy
> feeling if there are no unpackaged releases for the own packages :)
>
> There is one issue I recognized. How do you get the links for the
> unpackaged upstream versions and do you verify if the listed tarballs
> exist? I am asking because for app-editors/bluefish [1] there are eleven
> unpackaged versions listed but only sources for the three 2.0.3 releases
> exist and there are no releases for 2.1.x and 2.2.x.
>
> http://euscan.iksaif.net/package/app-editors/bluefish/

For bluefish, it's because when you try to download
http://www.bennewitz.com/bluefish/stable/source/bluefish-2.2.3.tar.bz2
it redirects you to an earlier version.

Fixed here: http://git.iksaif.net/?p=euscan.git;a=commitdiff;h=d74b2c505619306ab1ce3d4af3f66084b4faeed9;hp=a7a15c0ac72178bf2f3a73f94a70113ed9856e5c

Thanks !

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 921 bytes --]

On 15:48 Mon 18 Apr     , Corentin Chary wrote:
> On Mon, Apr 18, 2011 at 3:32 PM, Donnie Berkholz <dberkholz@gentoo.org> wrote:
> > - Some teams have "official" overlays. Supporting those as an additional
> >  source of ebuilds would be pretty nice.
> 
> I'll just checkout these trees on my machine (and only enable them for euscan).
> euscan use portage, gentoolkit and eix internaly, so overlays are not
> a problem :).
> 
> Where could I get a list of "official" overlays that should be added ?

It will be a little tricky to get this perfect automatically. I would 
start with anything in the layman repository file [1] that has <repo ... 
status="official"> and <owner type="project">.


1. http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo/xml/htdocs/proj/en/overlays/repositories.xml?view=log

-- 
Thanks,
Donnie

Donnie Berkholz
Sr. Developer, Gentoo Linux
Blog: http://dberkholz.com

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 698 bytes --]

On 15:48 Mon 18 Apr     , Corentin Chary wrote:
> On Mon, Apr 18, 2011 at 3:32 PM, Donnie Berkholz <dberkholz@gentoo.org> wrote:
> > - Another useful thing would be a way to supply CPV tokens for any
> >  "unstable" upstream series that we'll never add to the tree.
> 
> There is already some kind of blacklist in euscan. Could you provide
> examples so I can implement that ?

Sorry, forgot about this bit. I'm just talking about standard package 
tokens, the same kind used in package.mask or wherever:

=x11-drivers/xf86-video-intel-2.14.90*
>=x11-base/xorg-server-1.10.900

etc.

-- 
Thanks,
Donnie

Donnie Berkholz
Sr. Developer, Gentoo Linux
Blog: http://dberkholz.com

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Jorge Manuel B. S. Vicetto]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi.

On 16-04-2011 06:29, Corentin Chary wrote:
> New website is up and running at
> 
> http://euscan.iksaif.net/

This site is providing interesting data and to me reinforces how much I
need to work on desktop-effects ebuilds.
However, I've found a few "glitches" in the checking of new versions:

http://euscan.iksaif.net/package/media-sound/amarok/

The list is missing amarok-2.4.0.90. You need to look both on stable and
unstable trees to find all versions.
ftp://ftp.kde.org/pub/kde/unstable/amarok/2.4.0.90/src

http://euscan.iksaif.net/package/dev-db/mariadb/

You're missing quite a few versions of mariadb. They can be found in the
official mysql overlay (
http://git.overlays.gentoo.org/gitweb/?p=proj/mysql.git;a=tree;f=dev-db/mariadb
)
http://downloads.askmonty.org/mariadb/5.1/
http://downloads.askmonty.org/mariadb/5.2/

http://euscan.iksaif.net/package/dev-db/mysql/

You're missing many of the new releases of mysql. Some can be found in
the official mysql overlay (
http://git.overlays.gentoo.org/gitweb/?p=proj/mysql.git;a=tree;f=dev-db/mysql
)
http://downloads.mysql.com/archives/

Keep up the great work.

- -- 
Regards,

Jorge Vicetto (jmbsvicetto) - jmbsvicetto at gentoo dot org
Gentoo- forums / Userrel / Devrel / KDE / Elections / RelEng
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJNrKhbAAoJEC8ZTXQF1qEPAGcQANTpChBIAcmEiifx1rrIumqy
6h6iBO97vInSqVWrdwGF/ZQ2FzA4QV01xxQyUbAFRTVcQPVnMXTDnX+oOZFipNAZ
Um/dE3oP4tTwlhYnt/iqXWG5Xf/jBX0r4IQ+JInxC4vSm7p2cSrIQz77yx4P8zUz
ysFEWicrq2TdqfWeV1Sp72fdfWafDmSBBEg6SBaZXstTIOVehUqVgGhi75SnE7tp
KCkHWLeFZsMMVyiQpTsX6P67JQVWtgBwO7YiE2FnE7BfJWdoqj2akWijO6gPWjOb
VOUAOSIXqNj/SCdUsnskR6jdJxNB1XI6S25m26zXkkTt10IQGNAfLkYl2dWJtvRR
C4m4eeHUQkWoq0ZYXXvMJENNkFPpWOg5d36hRycdwtPwsSY8whMF2ajXYZn+oqNj
49Nx3UzTyxI2eKAv5+I2ll5s7ZgXgR2owxDb7l2I6hFJmt6dJE5sEsWfThFW/U+v
f9EVoIgZlsbMEzYWHcMUR5Gkv5Tktl+UNhXczIgu9Bga6slrrxIoT3T0OrYz+YPK
A3mPgTP9RpJq86jhVlguNGAnEcW5TIHo2y/xeNNWF1G2PsHlw+8Ix65nQI5I8dm7
OPftwL9ouKq6V/nz9FZrlAP2S1iEcX6uXNHeuUILl7XoC9xo6t0VoJ2tzQAAmRmE
HQ2LbqLWAxsbWOd8bJi0
=YAbh
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

On Mon, Apr 18, 2011 at 7:45 PM, Donnie Berkholz <dberkholz@gentoo.org> wrote:
> On 15:48 Mon 18 Apr     , Corentin Chary wrote:
>> On Mon, Apr 18, 2011 at 3:32 PM, Donnie Berkholz <dberkholz@gentoo.org> wrote:
>> > - Another useful thing would be a way to supply CPV tokens for any
>> >  "unstable" upstream series that we'll never add to the tree.
>>
>> There is already some kind of blacklist in euscan. Could you provide
>> examples so I can implement that ?
>
> Sorry, forgot about this bit. I'm just talking about standard package
> tokens, the same kind used in package.mask or wherever:
>
> =x11-drivers/xf86-video-intel-2.14.90*
>>=x11-base/xorg-server-1.10.900
>
I'll try to implement that,
Thanks for the suggestion.

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

> The list is missing amarok-2.4.0.90. You need to look both on stable and
> unstable trees to find all versions.
> ftp://ftp.kde.org/pub/kde/unstable/amarok/2.4.0.90/src

Hum, it's hard to guess the "unstable" url from the "stable" one
automatically. But maybe a custom "site handler" for ftp.kde.org would
do it.

> http://euscan.iksaif.net/package/dev-db/mariadb/
>
> You're missing quite a few versions of mariadb. They can be found in the
> official mysql overlay (
> http://git.overlays.gentoo.org/gitweb/?p=proj/mysql.git;a=tree;f=dev-db/mariadb
> )

I'll add some "officials" overlay soon.

> http://downloads.askmonty.org/mariadb/5.1/
> http://downloads.askmonty.org/mariadb/5.2/

The 5.2 is hard to find, because all mirrors redirect to
"http://downloads.askmonty.org/mariadb/" when trying to list the top
directory. And I can't guess the version from this page.

Brute force would be able to find it, if 5.2.0 archive was still available.

> http://euscan.iksaif.net/package/dev-db/mysql/
>
> You're missing many of the new releases of mysql.

MySQL case may be easier to deal with than MariaDB;

I added these to my TODO. I'll try to find a generic way, but I don't
know if there is really one.

Thanks !
-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Dirkjan Ochtman]

On Wed, Apr 13, 2011 at 08:47, Corentin Chary <corentin.chary@gmail.com> wrote:
> Better, what about a per-herd/per-maintainer rss feed ?

This is what I'd most want, would be very cool to have.

Perhaps it would be possible for some classes of packages to fine-tune
the algorithm, e.g. check PyPI for Python things, PECL for PHP things,
etc.

Cheers,

Dirkjan

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

On Tue, Apr 19, 2011 at 9:29 AM, Dirkjan Ochtman <djc@gentoo.org> wrote:
> On Wed, Apr 13, 2011 at 08:47, Corentin Chary <corentin.chary@gmail.com> wrote:
>> Better, what about a per-herd/per-maintainer rss feed ?
>
> This is what I'd most want, would be very cool to have.
>
> Perhaps it would be possible for some classes of packages to fine-tune
> the algorithm, e.g. check PyPI for Python things, PECL for PHP things,
> etc.
>

Thanks,
Added to TODO, but if you got some time, feel free to give it a try :).

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Tomáš Chvátal]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dne 16.4.2011 08:29, Corentin Chary napsal(a):
> New website is up and running at
> 
> http://euscan.iksaif.net/
> 
> The git tree is still at http://git.iksaif.net/?p=euscan.git;a=summary
> 
> TODO:
> - Make some charts to see how it's going
> - Finish the "scan my world" feature
> - Add a way to subsribe to herds/maintainer/packages in order to
> receive weekly/monthly reports
> 
> I'll gladly accept any patch ! :)
> 
Hello found another issue,

http://euscan.iksaif.net/package/media-video/kmplayer/

0.11.2c IS newer than 0.11.2 :)

Cheers

Tom
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk2t6AgACgkQHB6c3gNBRYeF3ACfReU6gKMMgX3VsuTB43QLGTcl
XAQAn2zPs9zmKEyTcgwKDinKurmuZQsr
=bktf
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

2011/4/19 Tomáš Chvátal <scarabeus@gentoo.org>:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Dne 16.4.2011 08:29, Corentin Chary napsal(a):
>> New website is up and running at
>>
>> http://euscan.iksaif.net/
>>
>> The git tree is still at http://git.iksaif.net/?p=euscan.git;a=summary
>>
>> TODO:
>> - Make some charts to see how it's going
>> - Finish the "scan my world" feature
>> - Add a way to subsribe to herds/maintainer/packages in order to
>> receive weekly/monthly reports
>>
>> I'll gladly accept any patch ! :)
>>
> Hello found another issue,
>
> http://euscan.iksaif.net/package/media-video/kmplayer/
>
> 0.11.2c IS newer than 0.11.2 :)
>
> Cheers

Should be fixed this evening. I'll now use portage.vercmp() by
default, and fallback to pkg_resources.
Thanks for the report.

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

I updated http://euscan.iksaif.net yesterday.

I also added some overlays, and I now use local portage portage trees
instead of system trees (a script to do that is available in
euscanwww/script).

New features:
- Handle overlays correctly (special column for overlays)
- Add some charts (small bar charts in table, pies at the bottom)
- Less false positives with euscan

I'll try to add "all time" line charts this week.

Thanks,
-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Tomáš Chvátal]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dne 27.4.2011 10:04, Corentin Chary napsal(a):
> I updated http://euscan.iksaif.net yesterday.
> 
> I also added some overlays, and I now use local portage portage trees
> instead of system trees (a script to do that is available in
> euscanwww/script).
> 
> New features:
> - Handle overlays correctly (special column for overlays)
> - Add some charts (small bar charts in table, pies at the bottom)
> - Less false positives with euscan
> 
> I'll try to add "all time" line charts this week.
> 
> Thanks,
Found another issue, package stays there even if it was removed from the
portage tree (kde-misc/plasmaboard is a good example).

Cheers

Tom
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk24WiAACgkQHB6c3gNBRYeUEwCgwGaQUFSJQ3WyEYqNrsuE3XQE
I7IAnR/5srqiiin56N/sK2mO+uMuk4ew
=FzoM
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

2011/4/27 Tomáš Chvátal <scarabeus@gentoo.org>:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Dne 27.4.2011 10:04, Corentin Chary napsal(a):
>> I updated http://euscan.iksaif.net yesterday.
>>
>> I also added some overlays, and I now use local portage portage trees
>> instead of system trees (a script to do that is available in
>> euscanwww/script).
>>
>> New features:
>> - Handle overlays correctly (special column for overlays)
>> - Add some charts (small bar charts in table, pies at the bottom)
>> - Less false positives with euscan
>>
>> I'll try to add "all time" line charts this week.
>>
>> Thanks,
> Found another issue, package stays there even if it was removed from the
> portage tree (kde-misc/plasmaboard is a good example).

Good catch, I'll add a --purge option to scan-portage.


-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

On Thu, Apr 28, 2011 at 2:34 PM, Corentin Chary
<corentin.chary@gmail.com> wrote:
> 2011/4/27 Tomáš Chvátal <scarabeus@gentoo.org>:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Dne 27.4.2011 10:04, Corentin Chary napsal(a):
>>> I updated http://euscan.iksaif.net yesterday.
>>>
>>> I also added some overlays, and I now use local portage portage trees
>>> instead of system trees (a script to do that is available in
>>> euscanwww/script).
>>>
>>> New features:
>>> - Handle overlays correctly (special column for overlays)
>>> - Add some charts (small bar charts in table, pies at the bottom)
>>> - Less false positives with euscan
>>>
>>> I'll try to add "all time" line charts this week.
>>>
>>> Thanks,
>> Found another issue, package stays there even if it was removed from the
>> portage tree (kde-misc/plasmaboard is a good example).
>
> Good catch, I'll add a --purge option to scan-portage.

Done, http://git.iksaif.net/?p=euscan.git;a=commitdiff;h=120ae425af82061fee4e0d7f18104595c551a144;hp=244e7d64fdc6ddeb31ad54dfcc415c50672bdebb


-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

New dynamic charts:

- http://euscan.iksaif.net/maintainers/2/charts/versions-weekly.png
- http://euscan.iksaif.net/categories/app-accessibility/charts/packages-weekly.png
- http://euscan.iksaif.net/categories/app-accessibility/charts/packages-weekly-small.png

Feel free to use them in a plasma-applet, or whatever you want !

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Patrick Nagel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Corentin,

On 2011-05-05 23:54, Corentin Chary wrote:
> New dynamic charts:
> 
> - http://euscan.iksaif.net/maintainers/2/charts/versions-weekly.png
> - http://euscan.iksaif.net/categories/app-accessibility/charts/packages-weekly.png
> - http://euscan.iksaif.net/categories/app-accessibility/charts/packages-weekly-small.png
> 
> Feel free to use them in a plasma-applet, or whatever you want !

I just wanted to thank you for your work on euscan, it's a great tool.

Patrick.

- -- 
Key ID: 0x86E346D4            http://patrick-nagel.net/key.asc
Fingerprint: 7745 E1BE FA8B FBAD 76AB 2BFC C981 E686 86E3 46D4
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk3DWToACgkQyYHmhobjRtRe9gCgivcJc0Ljyzh+TaYsmo6Pbguv
CsAAoL3ekUpjdMyHtwVWyV8Jsis+CEBL
=61hV
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

Hi,

some news about euscan (still available at http://euscan.iksaif.net)

- New design (yay !)
- Atom feeds available for each herd/category/maintainer/package
(http://euscan.iksaif.net/maintainers/59/feed/)
- Specific handlers for PyPi, RubyGems, pecl and PEAR packages (check
http://git.iksaif.net/?p=euscan.git;a=tree;f=pym/euscan/handlers;h=9a995dfcebe6beecce71851abb84a875cf6e5979;hb=HEAD
).

Now, maybe we should find a way to integrate that with the GSoC
statistic project and with http://packages.gentoo.org/ (like done at
http://packages.qa.debian.org/p/php-net-ipv4.html ). A quick way would
be to host euscan on a gentoo server, and add some webservices to
publish the data in json or xml, then packages.gentoo.org and others
could parse that and display it.

Thanks,

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Alec Warner]

On Wed, Aug 31, 2011 at 6:41 AM, Corentin Chary
<corentin.chary@gmail.com> wrote:
> Hi,
>
> some news about euscan (still available at http://euscan.iksaif.net)
>
> - New design (yay !)
> - Atom feeds available for each herd/category/maintainer/package
> (http://euscan.iksaif.net/maintainers/59/feed/)
> - Specific handlers for PyPi, RubyGems, pecl and PEAR packages (check
> http://git.iksaif.net/?p=euscan.git;a=tree;f=pym/euscan/handlers;h=9a995dfcebe6beecce71851abb84a875cf6e5979;hb=HEAD
> ).
>
> Now, maybe we should find a way to integrate that with the GSoC
> statistic project and with http://packages.gentoo.org/ (like done at
> http://packages.qa.debian.org/p/php-net-ipv4.html ). A quick way would
> be to host euscan on a gentoo server, and add some webservices to
> publish the data in json or xml, then packages.gentoo.org and others
> could parse that and display it.

The Gentoostats app supports fetching data with json (just put 'json'
in the HTTP_ACCEPT header in your GET.

-A

>
> Thanks,
>
> --
> Corentin Chary
> http://xf.iksaif.net
>
>

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Alex Legler]

[-- Attachment #1: Type: text/plain, Size: 290 bytes --]

On Wednesday 31 August 2011 15:41:51 Corentin Chary wrote:
> Hi,
> 
> some news about euscan (still available at http://euscan.iksaif.net)
> 
> - New design (yay !)

Glad you like it. Be sure to credit where you got it from, though.

-- 
Alex Legler <a3li@gentoo.org>
Gentoo Security / Ruby

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

On Thu, Sep 1, 2011 at 9:23 AM, Alex Legler <a3li@gentoo.org> wrote:
> On Wednesday 31 August 2011 15:41:51 Corentin Chary wrote:
>> Hi,
>>
>> some news about euscan (still available at http://euscan.iksaif.net)
>>
>> - New design (yay !)
>
> Glad you like it. Be sure to credit where you got it from, though.

Sorry, that was done in the dev version, but I forgot to push it
(http://euscan.iksaif.net/about/).
Thanks,

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Tomáš Chvátal]

Dne 1.9.2011 09:44, Corentin Chary napsal(a):
> On Thu, Sep 1, 2011 at 9:23 AM, Alex Legler<a3li@gentoo.org>  wrote:
>> On Wednesday 31 August 2011 15:41:51 Corentin Chary wrote:
>>> Hi,
>>>
>>> some news about euscan (still available at http://euscan.iksaif.net)
>>>
>>> - New design (yay !)
>>
>> Glad you like it. Be sure to credit where you got it from, though.
>
> Sorry, that was done in the dev version, but I forgot to push it
> (http://euscan.iksaif.net/about/).
> Thanks,
>
So now we just need to put this onto gentoo infrastructure and make you 
dev :P

Btw I have feature request, could it remember the sorting method i set?
(so I don't have to click and reorder it every time i refresh)

Cheers

Tom

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

> Btw I have feature request, could it remember the sorting method i set?
> (so I don't have to click and reorder it every time i refresh)

 Per-page or globally ?

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Tomáš Chvátal]

Dne 1.9.2011 09:55, Corentin Chary napsal(a):
>> Btw I have feature request, could it remember the sorting method i set?
>> (so I don't have to click and reorder it every time i refresh)
>
>   Per-page or globally ?
>
I would say globaly i smore sane here

Tom

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 567 bytes --]

On Wed, 31 Aug 2011 15:41:51 +0200
Corentin Chary <corentin.chary@gmail.com> wrote:

> - Specific handlers for PyPi, RubyGems, pecl and PEAR packages (check
> http://git.iksaif.net/?p=euscan.git;a=tree;f=pym/euscan/handlers;h=9a995dfcebe6beecce71851abb84a875cf6e5979;hb=HEAD
> ).

AFAICS that specific handlers are required to grab a list of versions.
Is it or will it be possible to add a kind of semi-handlers which would
just grab a list of all URIs (e.g. on github project) and let euscan
match them with SRC_URI?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

On Thu, Sep 1, 2011 at 10:31 AM, Michał Górny <mgorny@gentoo.org> wrote:
> On Wed, 31 Aug 2011 15:41:51 +0200
> Corentin Chary <corentin.chary@gmail.com> wrote:
>
>> - Specific handlers for PyPi, RubyGems, pecl and PEAR packages (check
>> http://git.iksaif.net/?p=euscan.git;a=tree;f=pym/euscan/handlers;h=9a995dfcebe6beecce71851abb84a875cf6e5979;hb=HEAD
>> ).
>
> AFAICS that specific handlers are required to grab a list of versions.
> Is it or will it be possible to add a kind of semi-handlers which would
> just grab a list of all URIs (e.g. on github project) and let euscan
> match them with SRC_URI?

Yep, it's possible, you can do some specific stuff and import
functions from euscan.helpers for euscan.handlers.generic to do a
"generic" scan, then return a correct list of version.

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

2011/9/1 Tomáš Chvátal <scarabeus@gentoo.org>:
> Dne 1.9.2011 09:55, Corentin Chary napsal(a):
>>>
>>> Btw I have feature request, could it remember the sorting method i set?
>>> (so I don't have to click and reorder it every time i refresh)
>>
>>  Per-page or globally ?
>>
> I would say globaly i smore sane here

I did that per-page, as it was only one  line to add, I'll try to do
it globally later.

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

On Wed, Aug 31, 2011 at 3:41 PM, Corentin Chary
<corentin.chary@gmail.com> wrote:
> Hi,
>
> some news about euscan (still available at http://euscan.iksaif.net)
>
> - New design (yay !)
> - Atom feeds available for each herd/category/maintainer/package
> (http://euscan.iksaif.net/maintainers/59/feed/)
> - Specific handlers for PyPi, RubyGems, pecl and PEAR packages (check
> http://git.iksaif.net/?p=euscan.git;a=tree;f=pym/euscan/handlers;h=9a995dfcebe6beecce71851abb84a875cf6e5979;hb=HEAD
> ).

A quick example of what a custom site handler can bring (rubygems
here): http://euscan.iksaif.net/categories/dev-ruby/charts/versions-weekly.png
:)

> Now, maybe we should find a way to integrate that with the GSoC
> statistic project and with http://packages.gentoo.org/ (like done at
> http://packages.qa.debian.org/p/php-net-ipv4.html ). A quick way would
> be to host euscan on a gentoo server, and add some webservices to
> publish the data in json or xml, then packages.gentoo.org and others
> could parse that and display it.
>
> Thanks,
>
> --
> Corentin Chary
> http://xf.iksaif.net
>



-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Matt Turner]

On Sun, Apr 3, 2011 at 1:20 PM, Corentin Chary <corentin.chary@gmail.com> wrote:
> Hi again,

Found a little problem: it's not finding a newer version of
wireless-regdb, which uses a date-based versioning system. If euscan
tried to view/parse the directory index where the distfiles are
located, it would find this new file.

http://euscan.iksaif.net/package/net-wireless/wireless-regdb/

Matt

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

On Tue, Sep 6, 2011 at 8:20 PM, Matt Turner <mattst88@gentoo.org> wrote:
> On Sun, Apr 3, 2011 at 1:20 PM, Corentin Chary <corentin.chary@gmail.com> wrote:
>> Hi again,
>
> Found a little problem: it's not finding a newer version of
> wireless-regdb, which uses a date-based versioning system. If euscan
> tried to view/parse the directory index where the distfiles are
> located, it would find this new file.
>
> http://euscan.iksaif.net/package/net-wireless/wireless-regdb/

Hi,
If you try to run euscan on that, you'll get:

>  * Url doesn't seems to depend on version: 20101124 not found in http://wireless.kernel.org/download/wireless-regdb/wireless-regdb-2010.11.24.tar.bz2

What's happenning here is that the version can be found in the url, so
even a directory scan won't find anything because it won't be able to
match files.
It would work correctly if the version was 2010.11.24 or the file was
named wireless-regdb-20101124.tar.bz2.

Unfortunatly, making that work would probably introduce a lot of false
positive in other cases.

Thanks,

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1438 bytes --]

On Wed, 7 Sep 2011 09:32:30 +0200
Corentin Chary <corentin.chary@gmail.com> wrote:

> On Tue, Sep 6, 2011 at 8:20 PM, Matt Turner <mattst88@gentoo.org>
> wrote:
> > On Sun, Apr 3, 2011 at 1:20 PM, Corentin Chary
> > <corentin.chary@gmail.com> wrote:
> >> Hi again,
> >
> > Found a little problem: it's not finding a newer version of
> > wireless-regdb, which uses a date-based versioning system. If euscan
> > tried to view/parse the directory index where the distfiles are
> > located, it would find this new file.
> >
> > http://euscan.iksaif.net/package/net-wireless/wireless-regdb/
> 
> Hi,
> If you try to run euscan on that, you'll get:
> 
> >  * Url doesn't seems to depend on version: 20101124 not found in
> > http://wireless.kernel.org/download/wireless-regdb/wireless-regdb-2010.11.24.tar.bz2
> 
> What's happenning here is that the version can be found in the url, so
> even a directory scan won't find anything because it won't be able to
> match files.
> It would work correctly if the version was 2010.11.24 or the file was
> named wireless-regdb-20101124.tar.bz2.
> 
> Unfortunatly, making that work would probably introduce a lot of false
> positive in other cases.

To be honest, you should be possible to preset PN, PV and other ebuild
variables and run the ebuild afterwards to see what SRC_URI it gives.
But that's a significant amount of work.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] euscan proof of concept (like debian's uscan)

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 1030 bytes --]

On 3/21/11 1:24 AM, Corentin Chary wrote:
> I recently started working on a small gentoo utility named "euscan"
> (for Ebuild Upstream Scan)
> For those who don't know debian's uscan, it allows to scan upstream
> for new versions. It's used by packages.qa.debian.org (example:
> http://packages.qa.debian.org/p/php-net-ipv4.html ).
> It's available at: http://xf.iksaif.net/bordel/euscan
> I think that it would be great to have these informations on
> http://packages.gentoo.org/ and/or unofficial
> http://gentoo-portage.com/ website.

Okay, I think this is pretty cool and we should find it a new home in
the Gentoo infrastructure.

I was thinking about http://qa-reports.gentoo.org/ with the repo at
http://git.overlays.gentoo.org/gitweb/?p=proj/qa-scripts.git;a=summary

I can act as a proxy committer and reviewer for that code. Could you
break it up into some smaller parts (preferably backend first) and send
to me for review (if you're interested)?

How long does it take to generate the reports?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: [gentoo-dev] euscan proof of concept (like debian's uscan)

[Dirkjan Ochtman]

On Mon, Sep 19, 2011 at 00:27, "Paweł Hajdan, Jr."
<phajdan.jr@gentoo.org> wrote:
> Okay, I think this is pretty cool and we should find it a new home in
> the Gentoo infrastructure.
>
> I was thinking about http://qa-reports.gentoo.org/ with the repo at
> http://git.overlays.gentoo.org/gitweb/?p=proj/qa-scripts.git;a=summary
>
> I can act as a proxy committer and reviewer for that code. Could you
> break it up into some smaller parts (preferably backend first) and send
> to me for review (if you're interested)?
>
> How long does it take to generate the reports?

+1 I think it would be good to run this on Gentoo infra, and I
wouldn't mind helping out.

Bikeshedding: not sure "reports" is the best name for this, as reports
implies something more static? Also not sure how much it has to do
with QA.

How much of it constitutes the backend, in your opinion? It seems
there are two parts, right now:

1. euscan script, to find new versions for a single package
2. the django www app, including storage for the version data

IMO it would be nice to have a somewhat generic REST-style service
exposing the data, and build a simple UI on top of that. In
particular, I have different ideas about what the UI should look like,
so it would be nice if different people could experiment (and/or
integrate in other services like znurt.org).

Cheers,

Dirkjan

Re: [gentoo-dev] euscan proof of concept (like debian's uscan)

[Corentin Chary]

On Mon, Sep 19, 2011 at 9:35 AM, Dirkjan Ochtman <djc@gentoo.org> wrote:
> On Mon, Sep 19, 2011 at 00:27, "Paweł Hajdan, Jr."
> <phajdan.jr@gentoo.org> wrote:
>> Okay, I think this is pretty cool and we should find it a new home in
>> the Gentoo infrastructure.
>>
>> I was thinking about http://qa-reports.gentoo.org/ with the repo at
>> http://git.overlays.gentoo.org/gitweb/?p=proj/qa-scripts.git;a=summary
>>
>> I can act as a proxy committer and reviewer for that code. Could you
>> break it up into some smaller parts (preferably backend first) and send
>> to me for review (if you're interested)?
>>
>> How long does it take to generate the reports?
>
> +1 I think it would be good to run this on Gentoo infra, and I
> wouldn't mind helping out.
>
> Bikeshedding: not sure "reports" is the best name for this, as reports
> implies something more static?

Here is how it works, each week I launch this script on lt server.
I've got ~30 trees installed with layman. The server is an AMD X2
4600+ with 4GB of RAM and two 80GB HD in raid1 using ext4. My network
bandwidth is 20Mbps down 1Mbps up.

#!/bin/sh

## Setup some vars to use local portage tree
export PATH=${HOME}/euscan/bin:${PATH}
export PYTHONPATH=${HOME}/euscan/pym:${PYTHONPATH}
export PORTAGE_CONFIGROOT=${HOME}/local
export ROOT=${HOME}/local
export EIX_CACHEFILE=${HOME}/local/var/cache/eix

## Go to euscanwww dir
cd ${HOME}/euscan/euscanwww/

## Update local trees
## Bottleneck: disk and network bandwidth
## Time: less than 30mn
emerge --sync --root=${ROOT} --config-root=${PORTAGE_CONFIGROOT}
ROOT="/" layman -S --config=${ROOT}/etc/layman/layman.cfg

## Also update eix database, because we use eix internaly
## Bottleneck: disk and cpu
##Time: 30mn ~ 1h
eix-update

## Scan portage (packages, versions)
## Bottleneck: disk and cpu
## Time: < 15mn
## Note: this script uses eix to get a list of packages and versions
python manage.py scan-portage --all --purge-versions --purge-packages

## Scan metadata (herds, maintainers, homepages, ...)
## Bottleneck: disk
## Time: 1h ~ 1h30
## Note: this script uses gentoolkit to fetch metadata
python manage.py scan-metadata --all --progress

## Scan uptsream packages
## Bottleneck: disk, network bandwidth and latency, cpu
## Time: up to 6h
## Note: euscan is called on each package. euscan has a slow startup
caused by gentoolkit/portage.
##          gparallel is used here to limit the load caused by euscan,
and to launch up to 16 euscan instances at a time on this machine
##          this part is the longest, but scale very well
eix --only-names -x | gparallel --load 4 --jobs 800% euscan >>
${HOME}/logs/euscan-upstream.log
python manage.py scan-upstream --feed --purge-versions <
${HOME}/logs/euscan-upstream.log

## Update counters (6)
## Time: some minutes
## Bottleneck: cpu
## Note: this script could probably be implemented faster using raw SQL queries
python manage.py update-counters


> Also not sure how much it has to do
> with QA.
> How much of it constitutes the backend, in your opinion? It seems
> there are two parts, right now:
>
> 1. euscan script, to find new versions for a single package
> 2. the django www app, including storage for the version data

Yes, exactly. Here is how the tree is structured currently:

euscan script

bin/ -- contains the euscan python "binary"
pym/ -- contains most of the code used by the euscan script
pym/euscan/handlers -- contains specific site handlers (rubygems,
pypi, pecl, pear, ..)

euscanwww django app

euscanwww/ -- contains all the stuff for the django application, all
the django application needs is a working portage tree and euscan
available in the $PATH

> IMO it would be nice to have a somewhat generic REST-style service
> exposing the data, and build a simple UI on top of that. In
> particular, I have different ideas about what the UI should look like,
> so it would be nice if different people could experiment (and/or
> integrate in other services like znurt.org).

I already added some very dummy json formating (note that it also
exposes internal key id, which is bad, but I just wanted to
experiment).
All you need is to append "/json" to an url. For example:

- http://euscan.iksaif.net/maintainers/4/json
- http://euscan.iksaif.net/package/app-accessibility/brltty/json

This could be a lot better, we just need to define an API and the
implementation will be easy.


A first step would be to make an ebuild for euscan, and another for
euscanwww so that anyone can easilly install it and play with it.
Feel free to ping me on irc, I'm on #gentoo-sunrise, my nickname is "iksaif".

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] euscan proof of concept (like debian's uscan)

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 574 bytes --]

On Mon, 19 Sep 2011 10:39:11 +0200
Corentin Chary <corentin.chary@gmail.com> wrote:

> ## Also update eix database, because we use eix internaly
> ## Bottleneck: disk and cpu
> ##Time: 30mn ~ 1h
> eix-update

Using egencache to keep caches for overlays will make eix updates much
faster.

Here's my code for it (it uses overlays in /usr/portage/local):

cd /usr/portage/local && \
for O in */; do
	echo "${O}"
	egencache --jobs=8 --update --update-use-local-desc --rsync \
		--repo=$(cat ${O}profiles/repo_name)
done

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] euscan proof of concept (like debian's uscan)

[Corentin Chary]

On Mon, Sep 19, 2011 at 10:53 AM, Michał Górny <mgorny@gentoo.org> wrote:
> On Mon, 19 Sep 2011 10:39:11 +0200
> Corentin Chary <corentin.chary@gmail.com> wrote:
>
>> ## Also update eix database, because we use eix internaly
>> ## Bottleneck: disk and cpu
>> ##Time: 30mn ~ 1h
>> eix-update
>
> Using egencache to keep caches for overlays will make eix updates much
> faster.
>
> Here's my code for it (it uses overlays in /usr/portage/local):
>
> cd /usr/portage/local && \
> for O in */; do
>        echo "${O}"
>        egencache --jobs=8 --update --update-use-local-desc --rsync \
>                --repo=$(cat ${O}profiles/repo_name)
> done

Thanks, I'll try that.


-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Hans de Graaff]

[-- Attachment #1: Type: text/plain, Size: 1297 bytes --]

On Wed, 2011-08-31 at 15:41 +0200, Corentin Chary wrote:
> Hi,
> 
> some news about euscan (still available at http://euscan.iksaif.net)
> 
> - New design (yay !)
> - Atom feeds available for each herd/category/maintainer/package
> (http://euscan.iksaif.net/maintainers/59/feed/)
> - Specific handlers for PyPi, RubyGems, pecl and PEAR packages (check
> http://git.iksaif.net/?p=euscan.git;a=tree;f=pym/euscan/handlers;h=9a995dfcebe6beecce71851abb84a875cf6e5979;hb=HEAD
> ).
> 
> Now, maybe we should find a way to integrate that with the GSoC
> statistic project and with http://packages.gentoo.org/ (like done at
> http://packages.qa.debian.org/p/php-net-ipv4.html ). A quick way would
> be to host euscan on a gentoo server, and add some webservices to
> publish the data in json or xml, then packages.gentoo.org and others
> could parse that and display it.

One integration that might also be useful is what we track on the ruby
wiki currently: https://overlays.gentoo.org/proj/ruby/wiki/PendingBumps
which is a collection of quick notes on problems encountered while
bumping packages. The ability to attach notes to packages and to
acknowledge version bumps (and e.g. make them yellow instead of red)
could help with larger lists for herds.

Kind regards,

Hans

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

On Mon, Sep 19, 2011 at 9:23 PM, Hans de Graaff <graaff@gentoo.org> wrote:
> On Wed, 2011-08-31 at 15:41 +0200, Corentin Chary wrote:
>> Hi,
>>
>> some news about euscan (still available at http://euscan.iksaif.net)
>>
>> - New design (yay !)
>> - Atom feeds available for each herd/category/maintainer/package
>> (http://euscan.iksaif.net/maintainers/59/feed/)
>> - Specific handlers for PyPi, RubyGems, pecl and PEAR packages (check
>> http://git.iksaif.net/?p=euscan.git;a=tree;f=pym/euscan/handlers;h=9a995dfcebe6beecce71851abb84a875cf6e5979;hb=HEAD
>> ).
>>
>> Now, maybe we should find a way to integrate that with the GSoC
>> statistic project and with http://packages.gentoo.org/ (like done at
>> http://packages.qa.debian.org/p/php-net-ipv4.html ). A quick way would
>> be to host euscan on a gentoo server, and add some webservices to
>> publish the data in json or xml, then packages.gentoo.org and others
>> could parse that and display it.
>
> One integration that might also be useful is what we track on the ruby
> wiki currently: https://overlays.gentoo.org/proj/ruby/wiki/PendingBumps
> which is a collection of quick notes on problems encountered while
> bumping packages. The ability to attach notes to packages and to
> acknowledge version bumps (and e.g. make them yellow instead of red)
> could help with larger lists for herds.
>
> Kind regards,
>
> Hans
>

Yep,
You can already use that page: http://euscan.iksaif.net/herds/ruby/
And ruby packages are well tracked by euscan thanks to the rubygem
site handler, see how it helped to find new upstream versions while
http://euscan.iksaif.net/herds/ruby/charts/versions-monthly.png

Could someone write ebuilds for euscan and euscanwww ? It should not
take a lot of time, but my ebuilds skills are probably not good
enought to do that.

Thanks,

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 396 bytes --]

On 10:00 Tue 20 Sep     , Corentin Chary wrote:
> Could someone write ebuilds for euscan and euscanwww ? It should not 
> take a lot of time, but my ebuilds skills are probably not good 
> enought to do that.

Sounds like good practice for when you become a Gentoo dev. =)

-- 
Thanks,
Donnie

Donnie Berkholz
Council Member / Sr. Developer
Gentoo Linux
Blog: http://dberkholz.com

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

On Tue, Sep 20, 2011 at 9:04 PM, Donnie Berkholz <dberkholz@gentoo.org> wrote:
> On 10:00 Tue 20 Sep     , Corentin Chary wrote:
>> Could someone write ebuilds for euscan and euscanwww ? It should not
>> take a lot of time, but my ebuilds skills are probably not good
>> enought to do that.
>
> Sounds like good practice for when you become a Gentoo dev. =)

Hehe :). Maybe someday, but honestly I don't have any time right now
to fill the quizz seriously.

Anyway, here is an ebuild for euscan:
http://git.iksaif.net/?p=portage.git;a=blob;f=app-portage/euscan/euscan-9999.ebuild;h=760f3ad025a009e3c336b1e7b08b31b72b6fec1f;hb=45b37c438f3cc20325d03ada666834b9db53d7b1
for those who want to test.
Writing one for euscanwww is probably less interesting since anyway
you'll have to configure most of it by hand (database, cron, etc..)
and it's clearly not something more than 2-3 people will install.

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Markos Chandras]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 09/21/11 11:57, Corentin Chary wrote:
> On Tue, Sep 20, 2011 at 9:04 PM, Donnie Berkholz
> <dberkholz@gentoo.org> wrote:
>> On 10:00 Tue 20 Sep     , Corentin Chary wrote:
>>> Could someone write ebuilds for euscan and euscanwww ? It
>>> should not take a lot of time, but my ebuilds skills are
>>> probably not good enought to do that.
>> 
>> Sounds like good practice for when you become a Gentoo dev. =)
> 
> Hehe :). Maybe someday, but honestly I don't have any time right
> now to fill the quizz seriously.
> 
> Anyway, here is an ebuild for euscan: 
> http://git.iksaif.net/?p=portage.git;a=blob;f=app-portage/euscan/euscan-9999.ebuild;h=760f3ad025a009e3c336b1e7b08b31b72b6fec1f;hb=45b37c438f3cc20325d03ada666834b9db53d7b1
>
> 
for those who want to test.
> Writing one for euscanwww is probably less interesting since
> anyway you'll have to configure most of it by hand (database, cron,
> etc..) and it's clearly not something more than 2-3 people will
> install.
> 
Hi,

Open a bug, attach your ebuilds ( are there any releases besides the
git code? ) and I will take it from there

- -- 
Regards,
Markos Chandras / Gentoo Linux Developer / Key ID: B4AFF2C2
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)

iQIcBAEBCgAGBQJOecNIAAoJEPqDWhW0r/LC6DYP/jDR0fB94keAOKKLjLtsf/7K
/ViCI7dePybD7LIDA/q/Wz8KwB+DAM15hO+i6wEvGTUqTXqAexmYRA4GKqUuA/1a
pLYm80PDvumzT8m2Q42z2+9MJfpwIezT3LreFce1WbYB/kWLhXPr0cTiGY94nVpU
kWOF2vR+zNBtwcYZilPXdan4dvcnRtL/xx9FHuFO8HwonCmgYee/42vV4ywxnM7b
1Fs7YTk2CcN1BAIU3cPm5aREoDQ1z0b+9gPpf5q1ONNmQmqyimw7b6sTLBQy0zfR
1YA2FBLfsEZseV5/BmwsLVQWjOOLkpCixakwVjyS9j/GSOX+5Biss4JC+fXhsom9
feoF45Ugk+DMWNsMeINAjvTIRmUSuESFh9xQKtjlMJSOem/MkLJJJ1JMqE+LdbV4
KEn1cxo+qwYVHnK7yRpJiC6s125aX05APClusd8Hbhb7tF8iEqw90BuE5Iz8ruDf
DtwfyY5klYDhuLMXe5I2oWed6trOi+hm3ppIBXm0q+D+HmTOG0R6h3v75Iopvykm
tGc77jzFsP4/aVcx0XWELWitR4BLG75+bcYpV2myibUf2L4rcwsl/c6hoaLUpVpY
1VeBWhpbIuKe5Ccai5R/DeRPqgVjny40gbn7oAjWeGJjAxD0x/BIj6lzRZT4cvOq
vQV/heWskEHZopj0gPnm
=km3U
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

On Wed, Sep 21, 2011 at 12:58 PM, Markos Chandras <hwoarang@gentoo.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> On 09/21/11 11:57, Corentin Chary wrote:
>> On Tue, Sep 20, 2011 at 9:04 PM, Donnie Berkholz
>> <dberkholz@gentoo.org> wrote:
>>> On 10:00 Tue 20 Sep     , Corentin Chary wrote:
>>>> Could someone write ebuilds for euscan and euscanwww ? It
>>>> should not take a lot of time, but my ebuilds skills are
>>>> probably not good enought to do that.
>>>
>>> Sounds like good practice for when you become a Gentoo dev. =)
>>
>> Hehe :). Maybe someday, but honestly I don't have any time right
>> now to fill the quizz seriously.
>>
>> Anyway, here is an ebuild for euscan:
>> http://git.iksaif.net/?p=portage.git;a=blob;f=app-portage/euscan/euscan-9999.ebuild;h=760f3ad025a009e3c336b1e7b08b31b72b6fec1f;hb=45b37c438f3cc20325d03ada666834b9db53d7b1
>>
>>
> for those who want to test.
>> Writing one for euscanwww is probably less interesting since
>> anyway you'll have to configure most of it by hand (database, cron,
>> etc..) and it's clearly not something more than 2-3 people will
>> install.
>>
> Hi,
>
> Open a bug, attach your ebuilds ( are there any releases besides the
> git code? ) and I will take it from there

Here it is: https://bugs.gentoo.org/show_bug.cgi?id=383937
There are no "released" version, and the project have no homepage
(except http://euscan.iksaif.net).
 Thanks,

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] euscan proof of concept (like debian's uscan)

[Corentin Chary]

On Mon, Sep 19, 2011 at 10:53 AM, Michał Górny <mgorny@gentoo.org> wrote:
> On Mon, 19 Sep 2011 10:39:11 +0200
> Corentin Chary <corentin.chary@gmail.com> wrote:
>
>> ## Also update eix database, because we use eix internaly
>> ## Bottleneck: disk and cpu
>> ##Time: 30mn ~ 1h
>> eix-update
>
> Using egencache to keep caches for overlays will make eix updates much
> faster.
>
> Here's my code for it (it uses overlays in /usr/portage/local):
>
> cd /usr/portage/local && \
> for O in */; do
>        echo "${O}"
>        egencache --jobs=8 --update --update-use-local-desc --rsync \
>                --repo=$(cat ${O}profiles/repo_name)
> done
>

Just done that, and indeed, it's much faster.

Also I switched to PostgreSQL, and since that avoids a lot of deadlock
situations when making queries in parallel, I'm now able to do that:

eix --only-names -x | gparallel --eta --load 8 --jobs 400%
--max-args=64 python manage.py scan-metadata

And now it takes less than 30mn instead of more than 1h, and more
importantly, it scales.

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

>> Open a bug, attach your ebuilds ( are there any releases besides the
>> git code? ) and I will take it from there
>
> Here it is: https://bugs.gentoo.org/show_bug.cgi?id=383937
> There are no "released" version, and the project have no homepage
> (except http://euscan.iksaif.net).

euscan ebuild is now in portage,
Thanks Markos !

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Hans de Graaff]

[-- Attachment #1: Type: text/plain, Size: 1269 bytes --]

On Tue, 2011-09-20 at 10:00 +0200, Corentin Chary wrote:
> On Mon, Sep 19, 2011 at 9:23 PM, Hans de Graaff <graaff@gentoo.org> wrote:

> > One integration that might also be useful is what we track on the ruby
> > wiki currently: https://overlays.gentoo.org/proj/ruby/wiki/PendingBumps
> > which is a collection of quick notes on problems encountered while
> > bumping packages. The ability to attach notes to packages and to
> > acknowledge version bumps (and e.g. make them yellow instead of red)
> > could help with larger lists for herds.

> You can already use that page: http://euscan.iksaif.net/herds/ruby/

I think you missed my point. This page is more like what we have at
http://moving-innovations.com/~graaff/outdated-gems/ which is an
automated list based on what is in the tree and upstream. Your page is a
good replacement for that so we may just use that in the future.

What our wiki page does it keep *notes* on these bumps, so all of that
relies on manual input. (e.g. "we tried to bump this but it needs a new
dependency that we don't have yet.") The euscan page doesn't support
that yet. Arguably I should open bugs for each of these bumps, but that
seems like a lot of overhead for some quick notes.

Kind regards,

Hans

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

On Mon, Sep 26, 2011 at 8:42 PM, Hans de Graaff <graaff@gentoo.org> wrote:
> On Tue, 2011-09-20 at 10:00 +0200, Corentin Chary wrote:
>> On Mon, Sep 19, 2011 at 9:23 PM, Hans de Graaff <graaff@gentoo.org> wrote:
>
>> > One integration that might also be useful is what we track on the ruby
>> > wiki currently: https://overlays.gentoo.org/proj/ruby/wiki/PendingBumps
>> > which is a collection of quick notes on problems encountered while
>> > bumping packages. The ability to attach notes to packages and to
>> > acknowledge version bumps (and e.g. make them yellow instead of red)
>> > could help with larger lists for herds.
>
>> You can already use that page: http://euscan.iksaif.net/herds/ruby/
>
> I think you missed my point. This page is more like what we have at
> http://moving-innovations.com/~graaff/outdated-gems/ which is an
> automated list based on what is in the tree and upstream. Your page is a
> good replacement for that so we may just use that in the future.
>
> What our wiki page does it keep *notes* on these bumps, so all of that
> relies on manual input. (e.g. "we tried to bump this but it needs a new
> dependency that we don't have yet.") The euscan page doesn't support
> that yet. Arguably I should open bugs for each of these bumps, but that
> seems like a lot of overhead for some quick notes.

Right, having a stable API and a special euscan tag on the wiki could
solve that.

-- 
Corentin Chary
http://xf.iksaif.net

Re: [gentoo-dev] Re: euscan proof of concept (like debian's uscan)

[Corentin Chary]

> Right, having a stable API and a special euscan tag on the wiki could
> solve that.


Speeking of API, I found some time to play with django-piston this
week, and here is the result:

- http://euscan.iksaif.net/about/api

Some examples:
- http://euscan.iksaif.net/api/1.0/packages/by-maintainer/1.json
- http://euscan.iksaif.net/api/1.0/packages/by-herd/openoffice.json
- http://euscan.iksaif.net/api/1.0/packages/by-category/app-office.xml
- http://euscan.iksaif.net/api/1.0/package/app-office/libreoffice-bin.json


-- 
Corentin Chary
http://xf.iksaif.net