[gentoo-soc] Benchmark suite project

[gentoo-soc] Benchmark suite project

[Andrea Arteaga]

Hello all.
My name is Andrea Arteaga and I'm working on the project "Automated
benchmark suite for numerical libraries in Gentoo" for this year's
Summer of Code.

Very shortly: many implementations for numerical library standards
such as BLAS, CBLAS, LAPACK, exist and are supported by Gentoo. My
project aims to construct a benchmark suite which can detect the best
suited one for the user's system and needs by performing a comparative
benchmark analysis. This will also take into account packages emerged
with special CFLAGS, FFLAGS, compilers, compiler versions...

The benchmarks will be written in C or C++, re-using existing code
when possible; the main part will consist of Python scripts. The
project design is already defined and will be the following:

* The main script will perform common tasks and will drive the user
through the different steps
* A special portage interface will use the portage APIs [I will need
soime help here, see later] in order to provide a simple way for
emerging and removing packages with special flags or environment
* An interface-dependant module will tell the portage API which
packages to emerge (e.g.: install blas-reference-20070226 using
gcc-4.5.2), use them for the benchmarks and interpret the result. At
the end it will also be responsible for a report.
* An interface-dependant benchmarks program which is coupled with the
module and will test the BLAS, CBLAS, LAPACK implementations.

The program will do operations like "CFLAGS="-O3" emerge -B
cblas-reference-20030223-r4" and similars, with some environments
changes and building binary packages in special directories. I see
that different threads already treat the portage APIs. I would like to
ask which API is the best suited for this type of operations and where
to find documentation for them (apart from pydoc).

I still don't have a git repository, which I asked Robin to set up a
few days ago, but the actual work is really starting.

--
Best
Andrea Arteaga

Re: [gentoo-soc] Benchmark suite project

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 3896 bytes --]

On Mon, 2011-05-23 at 21:21 +0200, Andrea Arteaga wrote:
> Hello all.
> My name is Andrea Arteaga and I'm working on the project "Automated
> benchmark suite for numerical libraries in Gentoo" for this year's
> Summer of Code.
> 
> Very shortly: many implementations for numerical library standards
> such as BLAS, CBLAS, LAPACK, exist and are supported by Gentoo. My
> project aims to construct a benchmark suite which can detect the best
> suited one for the user's system and needs by performing a comparative
> benchmark analysis. This will also take into account packages emerged
> with special CFLAGS, FFLAGS, compilers, compiler versions...
> 
> The benchmarks will be written in C or C++, re-using existing code
> when possible; the main part will consist of Python scripts. The
> project design is already defined and will be the following:
> 
> * The main script will perform common tasks and will drive the user
> through the different steps
> * A special portage interface will use the portage APIs [I will need
> soime help here, see later] in order to provide a simple way for
> emerging and removing packages with special flags or environment
> * An interface-dependant module will tell the portage API which
> packages to emerge (e.g.: install blas-reference-20070226 using
> gcc-4.5.2), use them for the benchmarks and interpret the result. At
> the end it will also be responsible for a report.
> * An interface-dependant benchmarks program which is coupled with the
> module and will test the BLAS, CBLAS, LAPACK implementations.
> 
> The program will do operations like "CFLAGS="-O3" emerge -B
> cblas-reference-20030223-r4" and similars, with some environments
> changes and building binary packages in special directories. I see
> that different threads already treat the portage APIs. I would like to
> ask which API is the best suited for this type of operations and where
> to find documentation for them (apart from pydoc).
> 
> I still don't have a git repository, which I asked Robin to set up a
> few days ago, but the actual work is really starting.
> 
> --
> Best
> Andrea Arteaga
> 

Andrea, use the public_api branch of the portage git repo as much as
possible.  If there is additional functionality you need, then ask.  I
will gladly try to get it for you if possible.  I am just now adding
emerge capability to the api, so you should be able to use it later in
the project for emerging.  I would however set up your code to be able
to run emerge from a subprocess.  You should find the most common
information gathering functions in the api now.  They should be fairly
straightforward to use.  Keep in mind that the api is likely to change
somewhat yet as it is used more and improved.

start by creating a portage.api.settings.PortageSettings instance

Hopefully it will be merged into regular portage releases by the end of
this years gsoc.

As for custom CFLAGS and other
variables, /etc/portage/env/$CAT/$pkg/files are used for individual pkg
overrides.  As I've never used them I can't give you more detail about
it or where to look.

If you need to write to those files, I would like to establish some code
in the api to standardize it and make it available for use.  In the
meantime you can look if the one I use in porthole is suitable.  I run
it as root from a terminal call or import it and use it directly the
importing code has the right privileges.

http://porthole.git.sourceforge.net/git/gitweb.cgi?p=porthole/porthole;a=blob;f=porthole/backends/set_config.py;h=8e78126377f8fc79fc51f9d17499df48a2e0270e;hb=HEAD

And if you need to keep an internal db of all those:

http://porthole.git.sourceforge.net/git/gitweb.cgi?p=porthole/porthole;a=blob;f=porthole/db/user_configs.py;h=7f1c61459f26c6856f80d02cbdfb81ac3c04d7e2;hb=HEAD

-- 
Brian Dolbec <brian.dolbec@gmail.com>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-soc] Benchmark suite project

[Andrea Arteaga]

Dear Brian
Thank you for your email. Good to know that somebody is actively
working on this!
Actually what I need is the ability for the user (not root) to create
binary packages somewhere he can write (PKGDIR=somedir emerge -B
somepkg). It would also be useful to also retrieve the installable
packages matching a given pattern (e.g. eigen or dev-cpp/eigen or
>=dev-cpp/eigen-3.0.0).

I'm starting my project using the command-line interface to emerge and
equery. When I feel your public_api are ready anough for my usege, I
will switch to it, which won't be a big deal (I hope).

--
Cheers
Andrea Arteaga


2011/5/23 Brian Dolbec <brian.dolbec@gmail.com>:
> Andrea, use the public_api branch of the portage git repo as much as
> possible.  If there is additional functionality you need, then ask.  I
> will gladly try to get it for you if possible.  I am just now adding
> emerge capability to the api, so you should be able to use it later in
> the project for emerging.  I would however set up your code to be able
> to run emerge from a subprocess.  You should find the most common
> information gathering functions in the api now.  They should be fairly
> straightforward to use.  Keep in mind that the api is likely to change
> somewhat yet as it is used more and improved.
>
> start by creating a portage.api.settings.PortageSettings instance
>
> Hopefully it will be merged into regular portage releases by the end of
> this years gsoc.
>
> As for custom CFLAGS and other
> variables, /etc/portage/env/$CAT/$pkg/files are used for individual pkg
> overrides.  As I've never used them I can't give you more detail about
> it or where to look.
>
> If you need to write to those files, I would like to establish some code
> in the api to standardize it and make it available for use.  In the
> meantime you can look if the one I use in porthole is suitable.  I run
> it as root from a terminal call or import it and use it directly the
> importing code has the right privileges.
>
> http://porthole.git.sourceforge.net/git/gitweb.cgi?p=porthole/porthole;a=blob;f=porthole/backends/set_config.py;h=8e78126377f8fc79fc51f9d17499df48a2e0270e;hb=HEAD
>
> And if you need to keep an internal db of all those:
>
> http://porthole.git.sourceforge.net/git/gitweb.cgi?p=porthole/porthole;a=blob;f=porthole/db/user_configs.py;h=7f1c61459f26c6856f80d02cbdfb81ac3c04d7e2;hb=HEAD
>
> --
> Brian Dolbec <brian.dolbec@gmail.com>
>

Re: [gentoo-soc] Benchmark suite project

[Brian Dolbec]

[-- Attachment #1: Type: text/plain, Size: 3037 bytes --]

On Sun, 2011-05-29 at 17:48 +0200, Andrea Arteaga wrote:
> Dear Brian
> Thank you for your email. Good to know that somebody is actively
> working on this!
> Actually what I need is the ability for the user (not root) to create
> binary packages somewhere he can write (PKGDIR=somedir emerge -B
> somepkg). It would also be useful to also retrieve the installable
> packages matching a given pattern (e.g. eigen or dev-cpp/eigen or
> >=dev-cpp/eigen-3.0.0).
> 
> I'm starting my project using the command-line interface to emerge and
> equery. 

The Dependencies.graph_depends()  fix for the api return value is in the
-0.3.0.4 release.  You should use the gentoolkit python api's instead of
parsing terminal output for equery calls.  It is faster and more
dependable than parsing terminal output which has a habit of changing.

> When I feel your public_api are ready anough for my usege, I
> will switch to it, which won't be a big deal (I hope).
> --
> Cheers
> Andrea Arteaga
> 

No problem.  I'll let you know when it is ready.

> 
> 2011/5/23 Brian Dolbec <brian.dolbec@gmail.com>:
> > Andrea, use the public_api branch of the portage git repo as much as
> > possible.  If there is additional functionality you need, then ask.  I
> > will gladly try to get it for you if possible.  I am just now adding
> > emerge capability to the api, so you should be able to use it later in
> > the project for emerging.  I would however set up your code to be able
> > to run emerge from a subprocess.  You should find the most common
> > information gathering functions in the api now.  They should be fairly
> > straightforward to use.  Keep in mind that the api is likely to change
> > somewhat yet as it is used more and improved.
> >
> > start by creating a portage.api.settings.PortageSettings instance
> >
> > Hopefully it will be merged into regular portage releases by the end of
> > this years gsoc.
> >
> > As for custom CFLAGS and other
> > variables, /etc/portage/env/$CAT/$pkg/files are used for individual pkg
> > overrides.  As I've never used them I can't give you more detail about
> > it or where to look.
> >
> > If you need to write to those files, I would like to establish some code
> > in the api to standardize it and make it available for use.  In the
> > meantime you can look if the one I use in porthole is suitable.  I run
> > it as root from a terminal call or import it and use it directly the
> > importing code has the right privileges.
> >
> > http://porthole.git.sourceforge.net/git/gitweb.cgi?p=porthole/porthole;a=blob;f=porthole/backends/set_config.py;h=8e78126377f8fc79fc51f9d17499df48a2e0270e;hb=HEAD
> >
> > And if you need to keep an internal db of all those:
> >
> > http://porthole.git.sourceforge.net/git/gitweb.cgi?p=porthole/porthole;a=blob;f=porthole/db/user_configs.py;h=7f1c61459f26c6856f80d02cbdfb81ac3c04d7e2;hb=HEAD
> >
> > --
> > Brian Dolbec <brian.dolbec@gmail.com>
> >
> 


-- 
Brian Dolbec <brian.dolbec@gmail.com>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]