[gentoo-dev-announce] Gentoo repository mirror & CI project update: first public results

[gentoo-dev-announce] Gentoo repository mirror & CI project update: first public results

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 4604 bytes --]

Hello,

I've reduced my work hours this week and used the extra time to finally
finish setting up some scripts for the repository mirror & CI project.
They're all now running in cron on the machine offered by Todd Goodman
for the task, and uploading the results to github.


Long story short, what we have now is:

1. basic QA checks on repos (repo_name, masters, global-scope 'die'
calls -- like ebuilds using EAPIs removed from eclass), uploaded to
[1]. The results include a summary page for all repositories
and a detailed log for each repository (full of cruft at the moment).

2. Git mirrors of repositories supported by the scripts and passing
basic QA checks, with metadata cache pregenerated. Hosted at [2].

3. Non-Travis runs of pkgcheck on gentoo. Now with plain HTML results
(no AJAX!), distinction between warnings and errors (and colors that
make sense) and quick links to look through all of them. Starts at [3].

4. Also, automated runs of gpyutils. Not that it's very related but
since I had cron and gentoo repo handy, why not? Result files linked at
the sub-project page [4].

The current TODO includes:

A. Moving the result pages to Gentoo hosting. This requires Infra
action per bugs [5,6].

B. Enabling the script to automatically file bugs for new basic QA
violations (repo_name, masters). Requires Infra action per bug [7].

And before I get into details, the cron schedule for people willing to
look at the results:

1-2. repo check is run every hour, at HH:17. The full run usually takes
around 5 minutes but major eclass changes can prolong it.

3. gentoo pkgcheck is run every 30 minutes, at HH:04 and HH:34. It
takes 7-8 minutes.

4. gpyutils are run every 4 hours, at HH:47 UTC. It takes around 15
minutes.

Please note that github has some website caching, so the results get
updated few minutes later.


Now, a few more boring details for people interested.

The main CI scripts are run on all repositories listed
in repositories.xml (in api.gentoo.org). Layman is used to parse
the scripts; the syncing is done using pkgcore (if possible) or
portage.

Since pkgcore has partial and broken repos.conf support, all initial
syncing is done using portage sync modules. This limits supported
VCS-es to git and svn. Portage supposedly supports rsync too but
the module works only with gentoo repository, so I unconditionally
assume it's unsupported.

So if your repository is marked as unsupported, you can either:

a) switch to git,

b) fix pkgcore to handle repos.conf properly (assuming it supports
the vcs),

c) add/fix portage sync module for the vcs.

If a repository is considered supported, the script runs a few basic
checks on it, and then regenerates the metadata cache. The checks
include:

- checking for set and correct repo_name (must be equal to the repo
  name on the list),

- checking for set and correct masters= (must only list repositories
  available via the list),

- checking for other metadata violations that make pkgcore unhappy ;),

- checking if any failures happen during metadata cache regen (like
  'die' calls in global scope).

This generates .txt logs for each repository and a summary.json file
containing copy of repositories.xml metadata along with check results.
The logs and summary are formatted as HTML, and uploaded.

In the future, at this point another script would compare the new
summary with bug database, and report new issues or close fixed ones.
Assuming people won't block it because of 'potential bug spam'...

Afterwards, another Python script updates GitHub repository list. New
repositories without major QA issues are added to the mirror, while
removed or unsupported repositories are removed. Repositories that
start having QA issues afterwards are not removed from the list to avoid
extra noise.

For each mirrored repository, the local checkout (with generated
metadata cache) is rsynced into git, and the update is pushed
as a single commit.

And I guess that would be all the gory details. I'm planning to publish
the scripts as soon as I have time to clean them up a bit.


[1]:http://gentoo.github.io/repo-qa-check-results/
[2]:https://github.com/gentoo-mirror
[3]:http://gentoo.github.io/gentoo-qa-results/global.html
[4]:https://wiki.gentoo.org/wiki/Project:GPyUtils
[5]:https://bugs.gentoo.org/show_bug.cgi?id=552520
[6]:https://bugs.gentoo.org/show_bug.cgi?id=552524
[7]:https://bugs.gentoo.org/show_bug.cgi?id=552518

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 949 bytes --]