[gentoo-cluster] Installing and using multiple MPI implementations at the same time.

[gentoo-cluster] Installing and using multiple MPI implementations at the same time.

[Justin Bronder]

[-- Attachment #1: Type: text/plain, Size: 2887 bytes --]

Well bug 44132 [1] has been a real thorn in my side for some time.  The
problem being that all mpi implementations currently in portage block each
other.  This is a real pain for any serious administrators, makes testing
new/other implementations too time-consuming, and in my opinion has probably
slowed any progress related to getting cluster applications into portage or
the overlays.

All that being said, I set out to do something about it.  Donnie Berkholz was
nice enough to help me out early on and ended up suggesting that I look
at crossdev.  I shamelessly ripped off the aforementioned script and tuned
it for mpi implementations and packages.  It should be noted that all the
changes I've implemented do [should, hopefully will] not effect the way the
packages behave now, rather it just adds another option.  The idea is as
follows.

sys-cluster/empi:  Does the same stuff that crossdev does.  You create a new
implementation root by specifying a name and mpi implementation package to
build it with. [2]  Empi adds these to an overlay under a new category with
the name you gave.  The ebuilds inherit mpi.eclass which handles pushing all
the files to /usr/lib/mpi/<name>, and providing files for eselect-mpi.

Later, you can add packages by specifying the category name and packages to
install.  Empi grabs the ones that inherit mpi (including deps), adds them to
the overlay and emerges.  The deps stuff hasn't been tested all that much.

app-admin/eselect-mpi:  Used to setup a user's environment to call a named
implementation install.  Basically does this by dropping a bash-sourceable
file in ${HOME}/.env.d/mpi.  The not-so-fun part is that the user then has to
remember to source that file.  Also used a fair amount by mpi.eclass.

I threw some quick notes showing commands and output up in my devspace. [3]
If anyone is at all interested, please try this out and let me know what you
think.  Even if you don't try it out but have ideas/comments/rants, I'm all ears.

Current overlay has the following ready to be tried out.
eselect-mpi/eselect-mpi-0.0.1
sys-cluster/empi-0.1
sys-cluster/hpl-1.0-r3
sys-cluster/lam-mpi-7.1.4-r1
sys-cluster/mpi-examples-1.0
sys-cluster/openmpi-1.2.5-r1

A couple of final words, hpl and mpi-examples currently wouldn't work without
empi, mainly because I'm lazy :)  Also I still haven't figured out a good way
to handle do{man,doc,www} etcetera, ideas are welcome.  There's still a fair
amount of work to be done, but I wanted to see if anyone had any feedback
regarding the general concept first before pushing on.

You can pull the overlay from rsync://cold-front.ath.cx/mpi (for now...)

Thanks,


[1] http://bugs.gentoo.org/show_bug.cgi?id=44132
[2] Currently openmpi-1.2.5-r1 or lam-mpi-7.1.4-r1
[3] http://dev.gentoo.org/~jsbronder/README.empi.txt

-- 
Justin Bronder

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-cluster] Installing and using multiple MPI implementations at the same time.

[Donnie Berkholz]

On 22:19 Thu 07 Feb     , Justin Bronder wrote:
> Well bug 44132 [1] has been a real thorn in my side for some time.  The
> problem being that all mpi implementations currently in portage block each
> other.  This is a real pain for any serious administrators, makes testing
> new/other implementations too time-consuming, and in my opinion has probably
> slowed any progress related to getting cluster applications into portage or
> the overlays.
> 
> All that being said, I set out to do something about it.  Donnie Berkholz was
> nice enough to help me out early on and ended up suggesting that I look
> at crossdev.  I shamelessly ripped off the aforementioned script and tuned
> it for mpi implementations and packages.  It should be noted that all the
> changes I've implemented do [should, hopefully will] not effect the way the
> packages behave now, rather it just adds another option.  The idea is as
> follows.

Cool, thanks for doing this much-needed work! I'm happy to hear this 
approach worked. If there really is much common code, it might be worth 
breaking it out into a library of some sort.

> sys-cluster/empi:  Does the same stuff that crossdev does.  You create a new
> implementation root by specifying a name and mpi implementation package to
> build it with. [2]  Empi adds these to an overlay under a new category with
> the name you gave.  The ebuilds inherit mpi.eclass which handles pushing all
> the files to /usr/lib/mpi/<name>, and providing files for eselect-mpi.

lib == $(get_libdir) ?

> A couple of final words, hpl and mpi-examples currently wouldn't work without
> empi, mainly because I'm lazy :)  Also I still haven't figured out a good way
> to handle do{man,doc,www} etcetera, ideas are welcome. 

Do the same thing as gcc. Install to a path under 
/usr/libdir/mpi/..../{man,doc} and use environment variables (MANPATH 
etc) and/or symlinks.

> There's still a fair amount of work to be done, but I wanted to see if 
> anyone had any feedback regarding the general concept first before 
> pushing on.
> 
> You can pull the overlay from rsync://cold-front.ath.cx/mpi (for now...)

Is this in layman? The file's at 
gentoo/xml/htdocs/proj/en/overlays/layman-global.txt.

Thanks,
Donnie
-- 
gentoo-cluster@lists.gentoo.org mailing list

[gentoo-cluster] Re: Installing and using multiple MPI implementations at the same time.

[Justin Bronder]

[-- Attachment #1: Type: text/plain, Size: 1979 bytes --]

On 09/02/08 22:56 -0800, Donnie Berkholz wrote:
...
> > sys-cluster/empi:  Does the same stuff that crossdev does.  You create a new
> > implementation root by specifying a name and mpi implementation package to
> > build it with. [2]  Empi adds these to an overlay under a new category with
> > the name you gave.  The ebuilds inherit mpi.eclass which handles pushing all
> > the files to /usr/lib/mpi/<name>, and providing files for eselect-mpi.
> 
> lib == $(get_libdir) ?
>

Yup, it's actually grabbed during the addition of the implementation to
eselect-mpi.

> > A couple of final words, hpl and mpi-examples currently wouldn't work without
> > empi, mainly because I'm lazy :)  Also I still haven't figured out a good way
> > to handle do{man,doc,www} etcetera, ideas are welcome. 
> 
> Do the same thing as gcc. Install to a path under 
> /usr/libdir/mpi/..../{man,doc} and use environment variables (MANPATH 
> etc) and/or symlinks.
> 
That's the general idea, however it's not as simple as just setting *into
unless I mess with $D first.  The plan being to have something like mpi_doX
which would handle setting the correct install path and restore global
variables (like $D) on exit.  Not sure if this is the best method, comments
appreciated :)

> > There's still a fair amount of work to be done, but I wanted to see if 
> > anyone had any feedback regarding the general concept first before 
> > pushing on.
> > 
> > You can pull the overlay from rsync://cold-front.ath.cx/mpi (for now...)
> 
> Is this in layman? The file's at 
> gentoo/xml/htdocs/proj/en/overlays/layman-global.txt.
> 

Not yet, it's being hosted off my home machine which doesn't always have the
best connection.  If a consensus can be reached that this is a worthy
solution, I presume I can talk to the overlay/infra guys then, or maybe even
the sci team who already has an overlay with some MPI applications in it.


Thanks,

-- 
Justin Bronder

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-cluster] Installing and using multiple MPI implementations at the same time.

[Alexander Piavka]

  Hi Justin,

I've started playing with your empi implementation.

Some problems & suggestions:

1)'eselect mpi set ...' does not check for existance of ~/.env.d dir
and fails if one does not exists.

It creates ~/.env.d/mpi which looks like this:
----------------------
PATH="/usr/lib64/mpi/mpi-openmpi/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.1.2:/opt/blackdown-jdk-1.4.2.03/bin:/opt/blackdown-jdk-1.4.2.03/jre/bin"
MANPATH="/usr/lib64/mpi/mpi-openmpi/usr/share/man:/etc/java-config-2/current-system-vm/man:/usr/local/share/man:/usr/share/man:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.18/man:/usr/share/gcc-data/x86_64-pc-linux-gnu/4.1.2/man:/opt/blackdown-jdk-1.4.2.03/man:/etc/java-config/system-vm/man/"
LD_LIBRARY_PATH="/usr/lib64/mpi/mpi-openmpi/usr/lib64:"
ESELECT_MPI_IMP="mpi-openmpi"
export LD_LIBRARY_PATH
export PATH
export MANPATH
export ESELECT_MPI_IMP
----------------------

while the following would be better:
----------------------
PATH="/usr/lib64/mpi/mpi-openmpi/usr/bin:${PATH}"
MANPATH="/usr/lib64/mpi/mpi-openmpi/usr/share/man:${MANPATH}"
LD_LIBRARY_PATH="/usr/lib64/mpi/mpi-openmpi/usr/lib64:${LD_LIBRARY_PATH}"
ESELECT_MPI_IMP="mpi-openmpi"
export LD_LIBRARY_PATH
export PATH
export MANPATH
export ESELECT_MPI_IMP
----------------------

maybe even
----------------------
if [ "X${PATH}" != "X" ]; then
 	export PATH="/usr/lib64/mpi/mpi-openmpi/usr/bin:${PATH}"
else
 	export PATH="/usr/lib64/mpi/mpi-openmpi/usr/bin"
fi
if [ "X${MANPATH}" != "X" ]; then
 	export MANPATH="/usr/lib64/mpi/mpi-openmpi/usr/share/man:${MANPATH}"
else
 	export MANPATH="/usr/lib64/mpi/mpi-openmpi/usr/share/man"
fi
if [ "X${LD_LIBRARY_PATH}" != "X" ]; then
 	export LD_LIBRARY_PATH="/usr/lib64/mpi/mpi-openmpi/usr/lib64:${LD_LIBRARY_PATH}"
else
 	export LD_LIBRARY_PATH="/usr/lib64/mpi/mpi-openmpi/usr/lib64"
fi
export ESELECT_MPI_IMP
----------------------

  Also, probably . besides /etc/env.d/mpi/mpi-openmpi the /etc/env.d/XXmpi
file should also be created with the default empi profile then 'eselect mpi set <mpi-implementation>'
is run.

2)another problem is a failure to install binpkg 
of openmpi on another identical systems, the error is

  *
  * ERROR: mpi-openmpi/openmpi-1.2.5-r1 failed.
  * Call stack:
  *                 ebuild.sh, line 1717:  Called dyn_setup
  *                 ebuild.sh, line  768:  Called qa_call 'pkg_setup'
  *                 ebuild.sh, line   44:  Called pkg_setup
  *   openmpi-1.2.5-r1.ebuild, line   23:  Called mpi_pkg_setup
  *                mpi.eclass, line  306:  Called die
  * The specific snippet of code:
  *              [[ -f "${FILESDIR}"/${MPI_ESELECT_FILE} ]] \
  *                   || die "MPI_ESELECT_FILE is not defined/found. ${MPI_ESELECT_FILE}"
  *  The die message:
  *   MPI_ESELECT_FILE is not defined/found. eselect.mpi.openmpi
  *
  * If you need support, post the topmost build error, and the call stack if relevant.
  * A complete build log is located at '/var/tmp/portage/mpi-openmpi/openmpi-1.2.5-r1/temp/build.log'.
  *

I thinks this is due to MPI_ESELECT_FILE being defined in pkg_setup() of 
openmpi ebuild and not in top of ebuild (will check if this would help later)

3) If i have PORTDIR_OVERLAY="/usr/local/overlays/csbgu /tmp/empi"
empi --create --implementation mpi-openmpi =sys-cluster/openmpi-1.2.5-r1
  would create mpi-openmpi category tree under /usr/local/overlays/csbgu/mpi-openmpi
since it's first overlay in PORTDIR_OVERLAY, it would be nice if it could 
ALWAYS be created under the empi overlay i.e /tmp/empi/mpi-openmpi
Of couse i can put the empi overlay first in PORTDIR_OVERLAY instead
but i want to avoid manual tweaking as much as possible.
With all mpi-implementation residing in the same overlay tree as empi
it would much more convienient , for me, to auto distribute single overlay 
among cluster hosts and avoid possible need for commands
like 'empi --create --implementation mpi-openmpi ...'

  Thanks
  Alex

On Thu, 7 Feb 2008, Justin Bronder wrote:

> Well bug 44132 [1] has been a real thorn in my side for some time.  The
> problem being that all mpi implementations currently in portage block each
> other.  This is a real pain for any serious administrators, makes testing
> new/other implementations too time-consuming, and in my opinion has probably
> slowed any progress related to getting cluster applications into portage or
> the overlays.
>
> All that being said, I set out to do something about it.  Donnie Berkholz was
> nice enough to help me out early on and ended up suggesting that I look
> at crossdev.  I shamelessly ripped off the aforementioned script and tuned
> it for mpi implementations and packages.  It should be noted that all the
> changes I've implemented do [should, hopefully will] not effect the way the
> packages behave now, rather it just adds another option.  The idea is as
> follows.
>
> sys-cluster/empi:  Does the same stuff that crossdev does.  You create a new
> implementation root by specifying a name and mpi implementation package to
> build it with. [2]  Empi adds these to an overlay under a new category with
> the name you gave.  The ebuilds inherit mpi.eclass which handles pushing all
> the files to /usr/lib/mpi/<name>, and providing files for eselect-mpi.
>
> Later, you can add packages by specifying the category name and packages to
> install.  Empi grabs the ones that inherit mpi (including deps), adds them to
> the overlay and emerges.  The deps stuff hasn't been tested all that much.
>
> app-admin/eselect-mpi:  Used to setup a user's environment to call a named
> implementation install.  Basically does this by dropping a bash-sourceable
> file in ${HOME}/.env.d/mpi.  The not-so-fun part is that the user then has to
> remember to source that file.  Also used a fair amount by mpi.eclass.
>
> I threw some quick notes showing commands and output up in my devspace. [3]
> If anyone is at all interested, please try this out and let me know what you
> think.  Even if you don't try it out but have ideas/comments/rants, I'm all ears.
>
> Current overlay has the following ready to be tried out.
> eselect-mpi/eselect-mpi-0.0.1
> sys-cluster/empi-0.1
> sys-cluster/hpl-1.0-r3
> sys-cluster/lam-mpi-7.1.4-r1
> sys-cluster/mpi-examples-1.0
> sys-cluster/openmpi-1.2.5-r1
>
> A couple of final words, hpl and mpi-examples currently wouldn't work without
> empi, mainly because I'm lazy :)  Also I still haven't figured out a good way
> to handle do{man,doc,www} etcetera, ideas are welcome.  There's still a fair
> amount of work to be done, but I wanted to see if anyone had any feedback
> regarding the general concept first before pushing on.
>
> You can pull the overlay from rsync://cold-front.ath.cx/mpi (for now...)
>
> Thanks,
>
>
> [1] http://bugs.gentoo.org/show_bug.cgi?id=44132
> [2] Currently openmpi-1.2.5-r1 or lam-mpi-7.1.4-r1
> [3] http://dev.gentoo.org/~jsbronder/README.empi.txt
>
> -- 
> Justin Bronder
>

-- 
gentoo-cluster@lists.gentoo.org mailing list

[gentoo-cluster] Re: Installing and using multiple MPI implementations at the same time.

[Justin Bronder]

[-- Attachment #1: Type: text/plain, Size: 5933 bytes --]

On 10/03/08 18:31 +0200, Alexander Piavka wrote:
>
>  Hi Justin,
>
> I've started playing with your empi implementation.
>
> Some problems & suggestions:
>
> 1)'eselect mpi set ...' does not check for existance of ~/.env.d dir
> and fails if one does not exists.

Fixed in eselect-mpi-0.0.2

>
> It creates ~/.env.d/mpi which looks like this:
> ----------------------
> PATH="/usr/lib64/mpi/mpi-openmpi/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.1.2:/opt/blackdown-jdk-1.4.2.03/bin:/opt/blackdown-jdk-1.4.2.03/jre/bin"
> MANPATH="/usr/lib64/mpi/mpi-openmpi/usr/share/man:/etc/java-config-2/current-system-vm/man:/usr/local/share/man:/usr/share/man:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.18/man:/usr/share/gcc-data/x86_64-pc-linux-gnu/4.1.2/man:/opt/blackdown-jdk-1.4.2.03/man:/etc/java-config/system-vm/man/"
> LD_LIBRARY_PATH="/usr/lib64/mpi/mpi-openmpi/usr/lib64:"
> ESELECT_MPI_IMP="mpi-openmpi"
> export LD_LIBRARY_PATH
> export PATH
> export MANPATH
> export ESELECT_MPI_IMP
> ----------------------
>
> while the following would be better:
> ----------------------
> PATH="/usr/lib64/mpi/mpi-openmpi/usr/bin:${PATH}"
> MANPATH="/usr/lib64/mpi/mpi-openmpi/usr/share/man:${MANPATH}"
> LD_LIBRARY_PATH="/usr/lib64/mpi/mpi-openmpi/usr/lib64:${LD_LIBRARY_PATH}"
> ESELECT_MPI_IMP="mpi-openmpi"
> export LD_LIBRARY_PATH
> export PATH
> export MANPATH
> export ESELECT_MPI_IMP
> ----------------------
>
> maybe even
> ----------------------
> if [ "X${PATH}" != "X" ]; then
> 	export PATH="/usr/lib64/mpi/mpi-openmpi/usr/bin:${PATH}"
> else
> 	export PATH="/usr/lib64/mpi/mpi-openmpi/usr/bin"
> fi
> if [ "X${MANPATH}" != "X" ]; then
> 	export MANPATH="/usr/lib64/mpi/mpi-openmpi/usr/share/man:${MANPATH}"
> else
> 	export MANPATH="/usr/lib64/mpi/mpi-openmpi/usr/share/man"
> fi
> if [ "X${LD_LIBRARY_PATH}" != "X" ]; then
> 	export 
> LD_LIBRARY_PATH="/usr/lib64/mpi/mpi-openmpi/usr/lib64:${LD_LIBRARY_PATH}"
> else
> 	export LD_LIBRARY_PATH="/usr/lib64/mpi/mpi-openmpi/usr/lib64"
> fi
> export ESELECT_MPI_IMP
> ----------------------

Yeah, you're probably right.  However, I need a way to deal with cleaning out
the environment when the user calls the unset action, or changes from one
implementation to the other.  Using what you have above, if the user then
called 'eselect mpi set mpi-lam' and sourced ~/.env.d/mpi, they would first
have the correct paths for mpi-openmpi followed by the ones for mpi-lam in
their environment.  See below for why this scares me.

>
>  Also, probably . besides /etc/env.d/mpi/mpi-openmpi the /etc/env.d/XXmpi
> file should also be created with the default empi profile then 'eselect mpi 
> set <mpi-implementation>'
> is run.

I'm willing to be told why I'm wrong, but I left out the above for what I
believe is a good reason.  If you set say, openmpi to be your default
implementation on the system level, then a user eselects lam-mpi, the user
will still have mpif90 in their path.  This is a big deal because lam-mpi
does not provide bindings for f90, hence the user could quite quickly become
confused as to why their code using f90 and c is in shambles when they try to
compile.

The above can still happen if openmpi is emerged normally.  I have no clue
how to deal with that yet either.

If we keep the ugly ~/.env.d/mpi file, along with the environment stripping
ability, there is no reason that a global mpi profile couldn't be used.  What
do you think?

>
> 2)another problem is a failure to install binpkg of openmpi on another 
> identical systems, the error is
>
>  *
>  * ERROR: mpi-openmpi/openmpi-1.2.5-r1 failed.
>  * Call stack:
>  *                 ebuild.sh, line 1717:  Called dyn_setup
>  *                 ebuild.sh, line  768:  Called qa_call 'pkg_setup'
>  *                 ebuild.sh, line   44:  Called pkg_setup
>  *   openmpi-1.2.5-r1.ebuild, line   23:  Called mpi_pkg_setup
>  *                mpi.eclass, line  306:  Called die
>  * The specific snippet of code:
>  *              [[ -f "${FILESDIR}"/${MPI_ESELECT_FILE} ]] \
>  *                   || die "MPI_ESELECT_FILE is not defined/found. 
> ${MPI_ESELECT_FILE}"
>  *  The die message:
>  *   MPI_ESELECT_FILE is not defined/found. eselect.mpi.openmpi
>  *
>  * If you need support, post the topmost build error, and the call stack if 
> relevant.
>  * A complete build log is located at 
> '/var/tmp/portage/mpi-openmpi/openmpi-1.2.5-r1/temp/build.log'.
>  *
>
> I thinks this is due to MPI_ESELECT_FILE being defined in pkg_setup() of 
> openmpi ebuild and not in top of ebuild (will check if this would help 
> later)

Foolish mistake on my part.  MPI_ESELECT_FILE can be defined in pkg_setup as
that always gets called (I believe).  However I can't check that file there
as emerging binpkgs doesn't give access to FILESDIR.  I've committed a fix to
the overlay.

>
> 3) If i have PORTDIR_OVERLAY="/usr/local/overlays/csbgu /tmp/empi"
> empi --create --implementation mpi-openmpi =sys-cluster/openmpi-1.2.5-r1
>  would create mpi-openmpi category tree under 
> /usr/local/overlays/csbgu/mpi-openmpi
> since it's first overlay in PORTDIR_OVERLAY, it would be nice if it could 
> ALWAYS be created under the empi overlay i.e /tmp/empi/mpi-openmpi
> Of couse i can put the empi overlay first in PORTDIR_OVERLAY instead
> but i want to avoid manual tweaking as much as possible.
> With all mpi-implementation residing in the same overlay tree as empi
> it would much more convienient , for me, to auto distribute single overlay 
> among cluster hosts and avoid possible need for commands
> like 'empi --create --implementation mpi-openmpi ...'

Also fixed, added the --overlaydir to the command line arguments.

Thanks for trying this out, it makes me feel useful :)

-- 
Justin Bronder

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]