[gentoo-soc] Benchmarking suite - Mid-term report

[gentoo-soc] Benchmarking suite - Mid-term report

[Andrea Arteaga]

With this email I present the status of my project just before the
mid-term evaluation.

Project description
=====================

The project aims to develop a simple yet powerful automated system of
benchmarking for numerical libraries. The Gentoo software system
provides many implementations of widely used standards such as BLAS,
CBLAS, LAPACK, ScaLAPACK and some other numerical library such as
FFTW, MKL. The developed tools will aid the system maintener to choose
the best suited implementation with respect to the machine hardware
and to test the samer implementation or different ones with different
compilers, compiler versions and compile flags.


Technical report
==================

In order to avoid a too long email, the report is published on my blog
[1]. Please feel free to post comments there or to email here.


Non-technical considerations
==============================

The Google Summer of Code programme is the much interesting and
exciting work that I could imagine. The contact with the community
through mailing lists, forums, IRC channels, surveys, is very
rewarding and helpful. I am really happy with this. Independently of
the result of my project, I am sure I will continue work within the
Gentoo community.

Alas, I had a problem some weeks ago and I was away for about 10 days
without having the possibility to code anything or contact my mentor.
I am doing my best to fill the gap, working more houres than stated at
the beginning of the programme.

Regarding the schedule, I wanted to provide until the mid-term
evaluation the following:
* BLAS, CBLAS, LAPACK benchmarks
* As much stability and usability as possible
* An ebuild for simple installing the software
* An initial work on the reports feature

All this has been completed. I ran as much tests as possible. They
show that the script is quite stable. But I would need more tests with
different hardware, so *some help from the community is really
welcome*.
Besides these milestones, the following have also been reached:

* BLAS_ACCURACY tests
* FFTW benchmarks
* Complete logging features
* Fancy and complete reports, including the logs, summary figures and
individual plots

This means that I'm ahead of schedule.

The following are the milestones for the next weeks:
* Develop distributed memory benchmarks: PBLAS, ScaLAPACK
* Improve the FFTW benchmarks, including 2-dimensional (maybe 3-D,
too) benchmarks and distributed memory ones
* Inlude support for MKL
* Better logging support (a few tasks are not still logged)
* Better ebuild for the Gentoo installation. Include this into the
Scientific Gentoo Project repository
* Much documentation. This includes:
** In-script docuentation (e.g. handling -h and --help switches)
** Man-page
** Web pages

For the benchmarks, I plan to spend 1 week for the distributed memory
ones. One week for FFTW and MKL. Another week for the better logging
support and the ebuild (this will require re design the git
repository, too). Some more time for the documentation (say 1.5
weeks). This amount to 4.5 weeks, which is the time before the final
evaluation.


Acknowledgements
==================

I have to say thanks to my mentor Sébastien who is very kind and
helped a lot with tests, suggestions, comments. We have had daily
communications and this was important for me.

The Gentoo community, and in particular the persons within the
gentoo-soc group and the scientific project were helpful. I asked
sometimes for help on the forums, through the mailing-lists and on the
chat channels, and I almost always received responsive help.

Finally, thanks to Google for encouraging the open-source software
development with such a programme.


Best regards
Andrea Arteaga

[1] http://andyspiros.wordpress.com/2011/07/13/mid-term-report/