[gentoo-soc] Distfile patching support - Weekly report #3

[gentoo-soc] Distfile patching support - Weekly report #3

[Rafael Martins]

Hi all

Quick summary: Improve the performance of the Gentoo Linux mirrors by
reducing the overall bandwidth load, allowing people to fetch binary
patches from the mirrors, instead of the full source tarballs, when
updating some package. This project is partially based on GLEP 25.

The project is a bit ahead of schedule.

== Progress ==

- Finished the integration of DeltaDB with existing code. A sample
DeltaDB for sys-kernel/gentoo-source is available [1]
- Finished the implementation of the delta reconstructor script. The
script (distpatcher.py) gets a CPV, creates a list of all the
distfiles needed by that version, check which ones are already
available in $DISTDIR, walk the delta db in the reverse order, and
reconstruct the needed distfiles, if possible. It also handle the
checksums and is able to say if the checksum of the reconstructed
distfile compressed is the same of the original distfile compressed,
saving it in a diferent place, if needed, to be handled separately by
Portage (to be implemented in a next task).
- Generated a simple stats class. distdiffer.py produced 45 deltas for
sys-kernel/gentoo-sources, then I created a plot [2] with the
percentage of savings for each delta, if people download the delta
instead of the full destination distfile. The deltas are sorted by
percentage of savings, to make the visualization easy. This is just a
visualization bonus, there's no really useful info about which
distfiles are saving more download size. You need to check the DeltaDB
file [1] if want such info.

== Next steps ==

- Create the script to generate deltas for the full gentoo-x86 tree.
- Improve error handling for existing scripts.
- Rewrite some bits of existing code.
- Try a full tree run, if possible.

The project homepage now lives on the Gentoo infrastructure project
page: http://www.gentoo.org/proj/en/infrastructure/distpatch/

[1] http://paste.pocoo.org/raw/404807/
[2] http://i.imgur.com/nKqpM.png

That's all for now.

Thanks!

-- 
Rafael Goncalves Martins
http://rafaelmartins.eng.br/