Re: [gentoo-dev] euscan proof of concept (like debian's uscan)

[Corentin Chary <corentin.chary@gmail.com>]

On Mon, Sep 19, 2011 at 9:35 AM, Dirkjan Ochtman <djc@gentoo.org> wrote:
> On Mon, Sep 19, 2011 at 00:27, "Paweł Hajdan, Jr."
> <phajdan.jr@gentoo.org> wrote:
>> Okay, I think this is pretty cool and we should find it a new home in
>> the Gentoo infrastructure.
>>
>> I was thinking about http://qa-reports.gentoo.org/ with the repo at
>> http://git.overlays.gentoo.org/gitweb/?p=proj/qa-scripts.git;a=summary
>>
>> I can act as a proxy committer and reviewer for that code. Could you
>> break it up into some smaller parts (preferably backend first) and send
>> to me for review (if you're interested)?
>>
>> How long does it take to generate the reports?
>
> +1 I think it would be good to run this on Gentoo infra, and I
> wouldn't mind helping out.
>
> Bikeshedding: not sure "reports" is the best name for this, as reports
> implies something more static?

Here is how it works, each week I launch this script on lt server.
I've got ~30 trees installed with layman. The server is an AMD X2
4600+ with 4GB of RAM and two 80GB HD in raid1 using ext4. My network
bandwidth is 20Mbps down 1Mbps up.

#!/bin/sh

## Setup some vars to use local portage tree
export PATH=${HOME}/euscan/bin:${PATH}
export PYTHONPATH=${HOME}/euscan/pym:${PYTHONPATH}
export PORTAGE_CONFIGROOT=${HOME}/local
export ROOT=${HOME}/local
export EIX_CACHEFILE=${HOME}/local/var/cache/eix

## Go to euscanwww dir
cd ${HOME}/euscan/euscanwww/

## Update local trees
## Bottleneck: disk and network bandwidth
## Time: less than 30mn
emerge --sync --root=${ROOT} --config-root=${PORTAGE_CONFIGROOT}
ROOT="/" layman -S --config=${ROOT}/etc/layman/layman.cfg

## Also update eix database, because we use eix internaly
## Bottleneck: disk and cpu
##Time: 30mn ~ 1h
eix-update

## Scan portage (packages, versions)
## Bottleneck: disk and cpu
## Time: < 15mn
## Note: this script uses eix to get a list of packages and versions
python manage.py scan-portage --all --purge-versions --purge-packages

## Scan metadata (herds, maintainers, homepages, ...)
## Bottleneck: disk
## Time: 1h ~ 1h30
## Note: this script uses gentoolkit to fetch metadata
python manage.py scan-metadata --all --progress

## Scan uptsream packages
## Bottleneck: disk, network bandwidth and latency, cpu
## Time: up to 6h
## Note: euscan is called on each package. euscan has a slow startup
caused by gentoolkit/portage.
##          gparallel is used here to limit the load caused by euscan,
and to launch up to 16 euscan instances at a time on this machine
##          this part is the longest, but scale very well
eix --only-names -x | gparallel --load 4 --jobs 800% euscan >>
${HOME}/logs/euscan-upstream.log
python manage.py scan-upstream --feed --purge-versions <
${HOME}/logs/euscan-upstream.log

## Update counters (6)
## Time: some minutes
## Bottleneck: cpu
## Note: this script could probably be implemented faster using raw SQL queries
python manage.py update-counters


> Also not sure how much it has to do
> with QA.
> How much of it constitutes the backend, in your opinion? It seems
> there are two parts, right now:
>
> 1. euscan script, to find new versions for a single package
> 2. the django www app, including storage for the version data

Yes, exactly. Here is how the tree is structured currently:

euscan script

bin/ -- contains the euscan python "binary"
pym/ -- contains most of the code used by the euscan script
pym/euscan/handlers -- contains specific site handlers (rubygems,
pypi, pecl, pear, ..)

euscanwww django app

euscanwww/ -- contains all the stuff for the django application, all
the django application needs is a working portage tree and euscan
available in the $PATH

> IMO it would be nice to have a somewhat generic REST-style service
> exposing the data, and build a simple UI on top of that. In
> particular, I have different ideas about what the UI should look like,
> so it would be nice if different people could experiment (and/or
> integrate in other services like znurt.org).

I already added some very dummy json formating (note that it also
exposes internal key id, which is bad, but I just wanted to
experiment).
All you need is to append "/json" to an url. For example:

- http://euscan.iksaif.net/maintainers/4/json
- http://euscan.iksaif.net/package/app-accessibility/brltty/json

This could be a lot better, we just need to define an API and the
implementation will be easy.


A first step would be to make an ebuild for euscan, and another for
euscanwww so that anyone can easilly install it and play with it.
Feel free to ping me on irc, I'm on #gentoo-sunrise, my nickname is "iksaif".

-- 
Corentin Chary
http://xf.iksaif.net