[gentoo-science] Benchmark suite (SoC project)

[gentoo-science] Benchmark suite (SoC project)

[Andrea Arteaga]

Hello, scientific Gentoo folks!
My name is Andrea Arteaga (andyspiros) and I'm doing a Google Summer
of Code project on "Automated benchmark suite for numerical libraries
in Gentoo" [1], with Sébastien Fabbro as mentor. As you can read, the
project is about constructing a benchmarking suite for common
libraries like BLAS, CBLAS, LAPACK,... Which implementation of BLAS is
faster on your computer? Does ATLAS work better when compiled with
gcc-4.5.2 or with some version of icc? Should you compile gsl with
"-O3" or "-O2 -fexpensive-optimizations -fomit-frame-pointer
-foptimize-register-move" as flags in order to reach the maximal
performance? My script (is supposed to) anserws to similar questions
(and many more).

At this moment, a basic benchmarking suite -- based on BTL -- is ready
for BLAS, CBLAS and LAPACK implementations. Did I say "ready"? This is
wrong. It still needs to be *tested*. I'm sure it is full of bugs, and
I need that somebody finds them. If you are interested on helping me,
please read on.

The script is easy to run, but takes much computational time. The
prerequisite for the tests are:

* you have a Gentoo system with the bicatali overlay installed [2]
* no old blas,cblas,lapack libraries are installed, i.e. you make use
of the new "alternatives" eselect module
* you can spend some minutes of your personal time and some houres of
your machine's computational time

Now, follow the how-to on my blog [3] (the blog post is in english of
course, but the blog is in italian and outdated -- I'm sorry for
that).
Alas, only packages that have installed dependencies can be tested.
So, for now, don't expect mkl to work very good (the results could be
meaningless). Anyway, packages like *-reference, eigen, gsl, atlas,
openblas, gotoblas2 should work well.

If you run the script and find some error, bugs, problems, please send
an email to me with the full report (input file, command line
arguments and output). If you run it and don't find any problem,
please send me an email, too! If you find that the project is
interesting, but you wish some more features, please send me an email
too. I don't have any bugzilla or similar system for issues, so I rely
on emails.

Unfortunately, I will be away for some days, so I can't answer to
messages until my return; but I would appreciate very much finding the
email box full of your emails, so don't hesitate!

Best regards.
Andrea Arteaga


[1] http://www.google-melange.com/gsoc/proposal/review/google/gsoc2011/spiros/1
[2] http://git.overlays.gentoo.org/gitweb/?p=dev/bicatali.git
[3] http://andyspiros.wordpress.com/2011/06/09/gentoo-automated-bechmarks/

Re: [gentoo-science] Benchmark suite (SoC project)

[Andrea Arteaga]

For the interested gentooers: the script is now much more stable and
usable. You can test:
* blas implementations
* cblas implementations
* lapack implementations

for speed, and blas implementation for accuracy (use the module blas_accuracy).

I'm adding now support for benchmarking FFTW. This won't be a
comparative benchmark (e.g. FFTW vs. MKL), but a tool for discover the
best compilation environment for compiling FFTW (compiler, compiler
version, compiler flags).

If you are interested on FFTW, even if my project is uninteresting for
you, *PLEASE*, donate 30 seconds of your time and participate to the
4-questions survey on FFTW [1].

Best regards.
Andea Arteaga

[1] https://spreadsheets.google.com/spreadsheet/viewform?formkey=dGlKbjBfLW5uaGt4QnQxNFJYUVp1QlE6MQ