[gentoo-science] Empi, should it be in the science overlay?

[gentoo-science] Empi, should it be in the science overlay?

[Justin Bronder]

[-- Attachment #1: Type: text/plain, Size: 1318 bytes --]

I've been spending the majority of my Gentoo-related time working on a
solution to bug 44132 [1], basically, trying to find a way to gracefully
handle multiple installs of various MPI implementations at the same time in
Gentoo.  Theres more information about the solution in my devspace [2], but
a quick summary is that there is a new package (empi) that is much like
crossdev, a new eselect module for empi, and a new eclass that handles both
mpi implementations and packages depending on mpi.

So, I think I have pushed this work far enough along for it to actually be
somewhat officially offered.  My question then, is where should this be
located?  There are several mpi packages in the science overlay already, so
should I push this work to there, or would it be more appropriate to make a
new overlay specifically for hp-cluster?

Future work related to this project will be getting all mpi implementations
and dependant packages converted in the same overlay before bringing it up on
-dev for discussion about inclusion into the main tree.

I have no real preference either way, but the science team does already have
an overlay :)  Let me know what you think.

[1] https://bugs.gentoo.org/show_bug.cgi?id=44132
[2] http://dev.gentoo.org/~jsbronder/README.empi.txt

-- 
Justin Bronder

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-science] Empi, should it be in the science overlay?

[Alexey Shvetsov]

Thats Cool !

I'll test it on my cluster =)
it uses infiniband and mvpich2 but some people want to use ordinary
eth as interconnect so it will be very useful =)

2008/2/26, Justin Bronder <jsbronder@gentoo.org>:
> I've been spending the majority of my Gentoo-related time working on a
>  solution to bug 44132 [1], basically, trying to find a way to gracefully
>  handle multiple installs of various MPI implementations at the same time in
>  Gentoo.  Theres more information about the solution in my devspace [2], but
>  a quick summary is that there is a new package (empi) that is much like
>  crossdev, a new eselect module for empi, and a new eclass that handles both
>  mpi implementations and packages depending on mpi.
>
>  So, I think I have pushed this work far enough along for it to actually be
>  somewhat officially offered.  My question then, is where should this be
>  located?  There are several mpi packages in the science overlay already, so
>  should I push this work to there, or would it be more appropriate to make a
>  new overlay specifically for hp-cluster?
>
>  Future work related to this project will be getting all mpi implementations
>  and dependant packages converted in the same overlay before bringing it up on
>  -dev for discussion about inclusion into the main tree.
>
>  I have no real preference either way, but the science team does already have
>  an overlay :)  Let me know what you think.
>
>  [1] https://bugs.gentoo.org/show_bug.cgi?id=44132
>  [2] http://dev.gentoo.org/~jsbronder/README.empi.txt
>
>  --
>
> Justin Bronder
>
>


-- 
Gentoo GNU/Linux 2.6.23 Dual Xeon

Mail to
      alexxyum@gmail.com
      alexxy@gentoo.ru
-- 
gentoo-science@lists.gentoo.org mailing list

Re: [gentoo-science] Empi, should it be in the science overlay?

[Donnie Berkholz]

On 10:45 Tue 26 Feb     , Justin Bronder wrote:
> I've been spending the majority of my Gentoo-related time working on a
> solution to bug 44132 [1], basically, trying to find a way to gracefully
> handle multiple installs of various MPI implementations at the same time in
> Gentoo.  Theres more information about the solution in my devspace [2], but
> a quick summary is that there is a new package (empi) that is much like
> crossdev, a new eselect module for empi, and a new eclass that handles both
> mpi implementations and packages depending on mpi.

Is it enough like crossdev enough to share code, with perhaps a bit of 
abstraction? Maintaining the same thing twice is rarely a good idea...

> So, I think I have pushed this work far enough along for it to actually be
> somewhat officially offered.  My question then, is where should this be
> located?  There are several mpi packages in the science overlay already, so
> should I push this work to there, or would it be more appropriate to make a
> new overlay specifically for hp-cluster?
> 
> Future work related to this project will be getting all mpi implementations
> and dependant packages converted in the same overlay before bringing it up on
> -dev for discussion about inclusion into the main tree.
> 
> I have no real preference either way, but the science team does already have
> an overlay :)  Let me know what you think.

Seems like people already committing cluster stuff to the sci overlay 
could help; maybe they'll port packages, fix bugs, etc. With a new 
overlay, we'd have to start from scratch, and I don't really see the 
point.

Thanks,
Donnie
-- 
gentoo-science@lists.gentoo.org mailing list

[gentoo-science] Re: Empi, should it be in the science overlay?

[Justin Bronder]

[-- Attachment #1: Type: text/plain, Size: 2315 bytes --]

On 26/02/08 09:44 -0800, Donnie Berkholz wrote:
> On 10:45 Tue 26 Feb     , Justin Bronder wrote:
> > I've been spending the majority of my Gentoo-related time working on a
> > solution to bug 44132 [1], basically, trying to find a way to gracefully
> > handle multiple installs of various MPI implementations at the same time in
> > Gentoo.  Theres more information about the solution in my devspace [2], but
> > a quick summary is that there is a new package (empi) that is much like
> > crossdev, a new eselect module for empi, and a new eclass that handles both
> > mpi implementations and packages depending on mpi.
> 
> Is it enough like crossdev enough to share code, with perhaps a bit of 
> abstraction? Maintaining the same thing twice is rarely a good idea...

They are similar in that they both use the same finagling with portage to
install things to different locations, but it pretty much ends there.  So
far as sharing code, I can see maybe the symlink, portdir and /etc/portage
stuff that might be shared.  Given that crossdev is ~650 lines and empi is
half that though, I'm of the opinion that it's not worth the effort.  The
majority of the work in empi is reading command line arguments and testing to
make sure preconditions are met.

> 
> > So, I think I have pushed this work far enough along for it to actually be
> > somewhat officially offered.  My question then, is where should this be
> > located?  There are several mpi packages in the science overlay already, so
> > should I push this work to there, or would it be more appropriate to make a
> > new overlay specifically for hp-cluster?
> > 
> > Future work related to this project will be getting all mpi implementations
> > and dependant packages converted in the same overlay before bringing it up on
> > -dev for discussion about inclusion into the main tree.
> > 
> > I have no real preference either way, but the science team does already have
> > an overlay :)  Let me know what you think.
> 
> Seems like people already committing cluster stuff to the sci overlay 
> could help; maybe they'll port packages, fix bugs, etc. With a new 
> overlay, we'd have to start from scratch, and I don't really see the 
> point.

Pretty much sums up why I'm posting here :)

-- 
Justin Bronder

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-science] Re: Empi, should it be in the science overlay?

[Alexey Shvetsov]

I can rewrite ebuild for mpich2 and mvapich2 to use them with empi and
eselect mpi

2008/2/26, Justin Bronder <jsbronder@gentoo.org>:
> On 26/02/08 09:44 -0800, Donnie Berkholz wrote:
>  > On 10:45 Tue 26 Feb     , Justin Bronder wrote:
>  > > I've been spending the majority of my Gentoo-related time working on a
>  > > solution to bug 44132 [1], basically, trying to find a way to gracefully
>  > > handle multiple installs of various MPI implementations at the same time in
>  > > Gentoo.  Theres more information about the solution in my devspace [2], but
>  > > a quick summary is that there is a new package (empi) that is much like
>  > > crossdev, a new eselect module for empi, and a new eclass that handles both
>  > > mpi implementations and packages depending on mpi.
>  >
>  > Is it enough like crossdev enough to share code, with perhaps a bit of
>  > abstraction? Maintaining the same thing twice is rarely a good idea...
>
>
> They are similar in that they both use the same finagling with portage to
>  install things to different locations, but it pretty much ends there.  So
>  far as sharing code, I can see maybe the symlink, portdir and /etc/portage
>  stuff that might be shared.  Given that crossdev is ~650 lines and empi is
>  half that though, I'm of the opinion that it's not worth the effort.  The
>  majority of the work in empi is reading command line arguments and testing to
>  make sure preconditions are met.
>
>
>  >
>  > > So, I think I have pushed this work far enough along for it to actually be
>  > > somewhat officially offered.  My question then, is where should this be
>  > > located?  There are several mpi packages in the science overlay already, so
>  > > should I push this work to there, or would it be more appropriate to make a
>  > > new overlay specifically for hp-cluster?
>  > >
>  > > Future work related to this project will be getting all mpi implementations
>  > > and dependant packages converted in the same overlay before bringing it up on
>  > > -dev for discussion about inclusion into the main tree.
>  > >
>  > > I have no real preference either way, but the science team does already have
>  > > an overlay :)  Let me know what you think.
>  >
>  > Seems like people already committing cluster stuff to the sci overlay
>  > could help; maybe they'll port packages, fix bugs, etc. With a new
>  > overlay, we'd have to start from scratch, and I don't really see the
>  > point.
>
>
> Pretty much sums up why I'm posting here :)
>
>  --
>
> Justin Bronder
>
>


-- 
Gentoo GNU/Linux 2.6.23 Dual Xeon

Mail to
      alexxyum@gmail.com
      alexxy@gentoo.ru
-- 
gentoo-science@lists.gentoo.org mailing list

[gentoo-science] Re: Empi, should it be in the science overlay?

[Justin Bronder]

[-- Attachment #1: Type: text/plain, Size: 1481 bytes --]

On 26/02/08 10:45 -0500, Justin Bronder wrote:
> I've been spending the majority of my Gentoo-related time working on a
> solution to bug 44132 [1], basically, trying to find a way to gracefully
> handle multiple installs of various MPI implementations at the same time in
> Gentoo.  Theres more information about the solution in my devspace [2], but
> a quick summary is that there is a new package (empi) that is much like
> crossdev, a new eselect module for empi, and a new eclass that handles both
> mpi implementations and packages depending on mpi.
> 
> So, I think I have pushed this work far enough along for it to actually be
> somewhat officially offered.  My question then, is where should this be
> located?  There are several mpi packages in the science overlay already, so
> should I push this work to there, or would it be more appropriate to make a
> new overlay specifically for hp-cluster?
> 
> Future work related to this project will be getting all mpi implementations
> and dependant packages converted in the same overlay before bringing it up on
> -dev for discussion about inclusion into the main tree.
> 
> I have no real preference either way, but the science team does already have
> an overlay :)  Let me know what you think.
> 
> [1] https://bugs.gentoo.org/show_bug.cgi?id=44132
> [2] http://dev.gentoo.org/~jsbronder/README.empi.txt
> 

So I've heard no complaints, I'll be committing this tonight.

-- 
Justin Bronder

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-science] Re: Empi, should it be in the science overlay?

[Radice David]

Hi,
I am playing with it right now, but I had to patch it this way to make
it work with my /etc/portage/package.* directories:

--- empi.old	2008-05-14 19:35:00.000000000 +0200
+++ empi.new	2008-05-14 19:39:30.000000000 +0200
@@ -106,7 +106,7 @@
         fi
 
         if ! grep "[>=<]*${imp}/${atom[1]}" ${gfiles} &>/dev/null; then
-            grep "[>=<]*${atom[0]}/${atom[1]}" ${gfiles} 2>/dev/null \
+            grep -h "[>=<]*${atom[0]}/${atom[1]}" ${gfiles} 2>/dev/null \
                 | sed "s,${atom[0]},${imp}," \
             | while read line; do
                echo "${line}" >> ${f}

otherwise grep will produce a wrong output because it will prefix the
file name before the found expression and something like this is put
inside the keyword file:

/etc/portage/package.keywords/base:mpi-test/openmpi ~x86

Thanks,
David.

On Tue, 2008-02-26 at 10:45 -0500, Justin Bronder wrote:
> I've been spending the majority of my Gentoo-related time working on a
> solution to bug 44132 [1], basically, trying to find a way to gracefully
> handle multiple installs of various MPI implementations at the same time in
> Gentoo.  Theres more information about the solution in my devspace [2], but
> a quick summary is that there is a new package (empi) that is much like
> crossdev, a new eselect module for empi, and a new eclass that handles both
> mpi implementations and packages depending on mpi.
> 
> So, I think I have pushed this work far enough along for it to actually be
> somewhat officially offered.  My question then, is where should this be
> located?  There are several mpi packages in the science overlay already, so
> should I push this work to there, or would it be more appropriate to make a
> new overlay specifically for hp-cluster?
> 
> Future work related to this project will be getting all mpi implementations
> and dependant packages converted in the same overlay before bringing it up on
> -dev for discussion about inclusion into the main tree.
> 
> I have no real preference either way, but the science team does already have
> an overlay :)  Let me know what you think.
> 
> [1] https://bugs.gentoo.org/show_bug.cgi?id=44132
> [2] http://dev.gentoo.org/~jsbronder/README.empi.txt
> 

-- 
gentoo-science@lists.gentoo.org mailing list

[gentoo-science] Re: Empi, should it be in the science overlay?

[Justin Bronder]

[-- Attachment #1: Type: text/plain, Size: 1380 bytes --]

On 14/05/08 20:54 +0200, Radice David wrote:
> Hi,
> I am playing with it right now, but I had to patch it this way to make
> it work with my /etc/portage/package.* directories:
> 
> --- empi.old	2008-05-14 19:35:00.000000000 +0200
> +++ empi.new	2008-05-14 19:39:30.000000000 +0200
> @@ -106,7 +106,7 @@
>          fi
>  
>          if ! grep "[>=<]*${imp}/${atom[1]}" ${gfiles} &>/dev/null; then
> -            grep "[>=<]*${atom[0]}/${atom[1]}" ${gfiles} 2>/dev/null \
> +            grep -h "[>=<]*${atom[0]}/${atom[1]}" ${gfiles} 2>/dev/null \
>                  | sed "s,${atom[0]},${imp}," \
>              | while read line; do
>                 echo "${line}" >> ${f}
> 
> otherwise grep will produce a wrong output because it will prefix the
> file name before the found expression and something like this is put
> inside the keyword file:
> 
> /etc/portage/package.keywords/base:mpi-test/openmpi ~x86

Should be fixed in 0.4, thanks for letting me know.  I failed to test
directories in portage_confdir, but I have some now :)

I think I saw you ask about documentation on b.g.o as well.  As long as I'm
writing this email, I may as well state that I'm working on some guide-xml
stuff.  By 'working on' I mean I started it and have every intention of
finishing in the next week or so.

Thanks for trying this out!

-- 
Justin Bronder

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]