* Re: [gentoo-soc] Automated Numerical Libraries Benchmark suite
2011-04-06 16:10 [gentoo-soc] Automated Numerical Libraries Benchmark suite Óscar Nájera
@ 2011-04-07 17:34 ` Sébastien Fabbro
0 siblings, 0 replies; 2+ messages in thread
From: Sébastien Fabbro @ 2011-04-07 17:34 UTC (permalink / raw
To: gentoo-soc
On Wed, Apr 6, 2011 at 09:10, Óscar Nájera <najera.oscar@gmail.com> wrote:
> Hi,
> I'm interested in participating in th eGoogle Summer of code this year, I
> applied last year for the "Shape up linear algebra project", but maybe it
> was to ambitious for being eligible, because now it's contents are separated
> in two different projects: Support for Fortran modules/libraries with
> multiple compilers, and Automated numerical libraries benchmark suite.
>
> I'm interested as mentioned before in the later one, could I get some
> additional advise for developing the application. I read that about the
> requirement of actually writing a patch for the application, could I get
> some advise too for that, are there some easy to fix bugs I could find.
Dear Oscar,
Thanks for considering applying again this year. I do not have access
to your last year proposal anymore, but anyway here is what I am
thinking for this year. You are obviously welcome to change anything
to your own idea if you think it is worth it.
Numerical libraries in Gentoo. The concerns are mostly linear algebra
and fft libraries.
The idea is for the user to switch between implementations of given
official library API, and to provide an easy way to test correctness
and speed for each of the testing implementation. We already have a
framework [1], it has room for much improvement such as:
o) allow the user to switch libraries, not just root
o) make eselect blas/cblas/lapack robust to unpackaging
o) possibly allow compiler versioning
Some work is going on in my overlay [2] and look for the huge amount
of bugs on bugs.gentoo.org concerning blas, lapack, atlas, mkl,
eselect, etc...
Ultimately, we want to allow the user to test correctness and speed by
running benchmark suites, so that we could tune compilers, compiler
flags and library implementations in an automated fashion. This is
would be new in Gentoo. The idea is vague and very much up to the
student. One simple possibility is to use the standard LAPACK test suite and HPL
benchmarks. Then run the tests for every combination of fortran
compilers, fortran flags, linear algebra libraries and select the most
correct one, the fastest one, the most memory efficient one, etc...
It could be done generically to do other kind of Gentoo benchmarks, and
this is also good publicity for Gentoo.
So the plan could be roughly:
1) develop robust and portable suite of tests for both for
matrix/vector operations, solving algorithms, may be FFT routines if
there is time. The tests included in the lapack tar ball from netlib
and the HPL are very good starts. We also could test things for
parallel and for FFTs, where you would have to look whether standard
tests exist (this is part of your research for the proposal) and code
them if not. If you know some C++, you can take a look at the Eigen
[3] library where they have developed a decent framework to benchmark
linear algebra packages. Then you would have to create ebuilds for
your test benchmark tools.
2) automate the benchmarking for Gentoo based machines. You would have
to write a few scripts that run the tests for each blas or lapack
eselect implementation installed on the system and produce useful
outputs/plots for the user to decide on which to select. For FFT it
would have to be different because there is no "standard" API, but in
portage we have only few FFT implementations (fftw, acml, mkl of what
I know).
3) a user defined eselect module. I am working with Ulrich Mueller
(ulm) to produce an easy mechanism to add new eselect modules. It
would be nice to work with us to try to improve it even better, adding
the possibility for a user to eselect a different module than the
sysadmin does. Justin Bronder (jsbronder) had worked out something for
MPI called empi [4], you could take a look and think how we could do
it.
It is probably more important to focus on 1) and 2), although 3) would
be extremely useful for other parts of Gentoo and requires no
numerical skills but operating system ones.
As an example of a result of the project, this is how a user could
test blas from netlib vs. blas from ACML using your project
blas-benchmarks:
$ sudo emerge blas-reference acml blas-benchmarks
$ blas-benchmarks
study results...
$ eselect blas set <best implementation>
To compare sets of compiler and flags, this is a bit trickier, you
might have to think of a way to make a slot and an eselect module each
time blas is installed with a different set of compiler and use flags.
This is not clear why a user would need to keep all blas modules once
an efficient one has been chosen (may be for cross-compiling). You
could also just store the results in your blas-benchmarks, and
re-install a blas with different flags, re-run the benchmarks and
compare with the stored result. Finally you could also do something in
the line of what has been done for GNAT [5], which would require to
modify all the blas and lapack ebuilds but with a good eclass would
not be a big deal. Anyway it is up to you to imagine a solution if
needed.
Note all of this is a very open design, if you cook up something else,
for your proposal all the best! Think GPU could add some nice
features! Find us on irc at #gentoo-science, write to the gentoo-soc
mailing list or just email me for more.
Also I could not find any Gentoo contribution from you. So if you do
apply, make sure you read all the guidelines. We require the students
to submit a patch/fix a bug.
This is quite an exciting and challenging project. I hope this long
email will help you.
[1] http://www.gentoo.org/proj/en/science/blas-lapack.xml
[2] http://git.overlays.gentoo.org/gitweb/?p=dev/bicatali.git
[3] http://eigen.tuxfamily.org/index.php?title=Benchmark
[4] http://dev.gentoo.org/~jsbronder/empi.xml
[5] http://www.gentoo.org/proj/en/prog_lang/ada/dev_reference.xml
Best,
--
Sebastien
^ permalink raw reply [flat|nested] 2+ messages in thread