[gentoo-science] lapack transition

[gentoo-science] lapack transition

[Peter Bienstman]

[-- Attachment #1: Type: text/plain, Size: 600 bytes --]

Hi,

I've been looking through the tree to see which packages use lapack/blas, and 
which of them are prepared for the new infrastructure. There are actually far 
less packages than I anticipated:

Still depends on old infrastructure:

   sys-cluster/hpl

Already uses virtual/blas or virtual/lapack in unstable branch:

   dev-lang/R
   sci-geosciences/grass
   sci-misc/camfr
   sci-chemistry/mpqc
   sci-mathematics/octave
   sci-misc/xfoil

Uses own code, but could benefit from transition:

   dev-python/numeric (see http://bugs.gentoo.org/show_bug.cgi?id=81520)

Peter

[-- Attachment #2: Type: application/pgp-signature, Size: 307 bytes --]

Re: [gentoo-science] lapack transition

[Darren Dale]

On Sunday 21 August 2005 7:43 am, Peter Bienstman wrote:
> Hi,
>
> I've been looking through the tree to see which packages use lapack/blas,
> and which of them are prepared for the new infrastructure. There are
> actually far less packages than I anticipated:
>
> Still depends on old infrastructure:
>
>    sys-cluster/hpl
>
> Already uses virtual/blas or virtual/lapack in unstable branch:
>
>    dev-lang/R
>    sci-geosciences/grass
>    sci-misc/camfr
>    sci-chemistry/mpqc
>    sci-mathematics/octave
>    sci-misc/xfoil
>
> Uses own code, but could benefit from transition:
>
>    dev-python/numeric (see http://bugs.gentoo.org/show_bug.cgi?id=81520)

I use numeric and scipy, and have been trying for a long time to improve the 
ebuilds to use blas/lapack/atlas. I submitted an ebuild for numeric six 
months ago (http://bugs.gentoo.org/show_bug.cgi?id=81520), but it has 
languished in buzilla because it is not clear how to deal with ATLAS (my 
atlas USE flag was rejected). How should ATLAS dependencies be handled in the 
new infrastructure?

-- 
Darren S. Dale

Bard Hall
Department of Materials Science and Engineering
Cornell University
Ithaca, NY. 14850

dd55@cornell.edu
http://people.ccmr.cornell.edu/~dd55/
-- 
gentoo-science@gentoo.org mailing list

Re: [gentoo-science] lapack transition

[Peter Bienstman]

[-- Attachment #1: Type: text/plain, Size: 685 bytes --]

On Sunday 21 August 2005 15:22, Darren Dale wrote:

> I use numeric and scipy, and have been trying for a long time to improve
> the ebuilds to use blas/lapack/atlas. I submitted an ebuild for numeric six
> months ago (http://bugs.gentoo.org/show_bug.cgi?id=81520), but it has
> languished in buzilla because it is not clear how to deal with ATLAS (my
> atlas USE flag was rejected). How should ATLAS dependencies be handled in
> the new infrastructure?

In theory it's as simple as depending on virtual/blas and virtual/lapack. The 
user can then switch between different lapack and blas implementations at 
runtime with blas-config and lapack-config.

Cheers,

Peter

[-- Attachment #2: Type: application/pgp-signature, Size: 307 bytes --]

Re: [gentoo-science] lapack transition

[Darren Dale]

On Sunday 21 August 2005 10:25 am, Peter Bienstman wrote:
> On Sunday 21 August 2005 15:22, Darren Dale wrote:
> > I use numeric and scipy, and have been trying for a long time to improve
> > the ebuilds to use blas/lapack/atlas. I submitted an ebuild for numeric
> > six months ago (http://bugs.gentoo.org/show_bug.cgi?id=81520), but it has
> > languished in buzilla because it is not clear how to deal with ATLAS (my
> > atlas USE flag was rejected). How should ATLAS dependencies be handled in
> > the new infrastructure?
>
> In theory it's as simple as depending on virtual/blas and virtual/lapack.
> The user can then switch between different lapack and blas implementations
> at runtime with blas-config and lapack-config.

It is not that simple. If numeric and scipy are to make use of atlas, atlas 
has to be in their external library list at compile time. If atlas is in the 
list, but not installed on the system, compilation will fail.

-- 
Darren S. Dale

Bard Hall
Department of Materials Science and Engineering
Cornell University
Ithaca, NY. 14850

dd55@cornell.edu
http://people.ccmr.cornell.edu/~dd55/
-- 
gentoo-science@gentoo.org mailing list

Re: [gentoo-science] lapack transition

[Peter Bienstman]

[-- Attachment #1: Type: text/plain, Size: 1388 bytes --]

On Sunday 21 August 2005 16:33, Darren Dale wrote:
> On Sunday 21 August 2005 10:25 am, Peter Bienstman wrote:
> > On Sunday 21 August 2005 15:22, Darren Dale wrote:
> > > I use numeric and scipy, and have been trying for a long time to
> > > improve the ebuilds to use blas/lapack/atlas. I submitted an ebuild for
> > > numeric six months ago (http://bugs.gentoo.org/show_bug.cgi?id=81520),
> > > but it has languished in buzilla because it is not clear how to deal
> > > with ATLAS (my atlas USE flag was rejected). How should ATLAS
> > > dependencies be handled in the new infrastructure?
> >
> > In theory it's as simple as depending on virtual/blas and virtual/lapack.
> > The user can then switch between different lapack and blas
> > implementations at runtime with blas-config and lapack-config.
>
> It is not that simple. If numeric and scipy are to make use of atlas, atlas
> has to be in their external library list at compile time. If atlas is in
> the list, but not installed on the system, compilation will fail.

That's why I said 'in theory' ;-)

I'm currently working on a patch which just ignores any explicit atlas stuff 
at build time of SciPy. The idea is to build a library which simply links 
to /usr/lib/liblapack.so. SciPy doesn't need to know this is a symlink 
to /usr/lib/lapack/atlas/liblapack.so or any other implementation.

Peter

[-- Attachment #2: Type: application/pgp-signature, Size: 307 bytes --]

Re: [gentoo-science] lapack transition

[Darren Dale]

On Sunday 21 August 2005 10:56 am, Peter Bienstman wrote:
> On Sunday 21 August 2005 16:33, Darren Dale wrote:
> > On Sunday 21 August 2005 10:25 am, Peter Bienstman wrote:
> > > On Sunday 21 August 2005 15:22, Darren Dale wrote:
> > > > I use numeric and scipy, and have been trying for a long time to
> > > > improve the ebuilds to use blas/lapack/atlas. I submitted an ebuild
> > > > for numeric six months ago
> > > > (http://bugs.gentoo.org/show_bug.cgi?id=81520), but it has languished
> > > > in buzilla because it is not clear how to deal with ATLAS (my atlas
> > > > USE flag was rejected). How should ATLAS dependencies be handled in
> > > > the new infrastructure?
> > >
> > > In theory it's as simple as depending on virtual/blas and
> > > virtual/lapack. The user can then switch between different lapack and
> > > blas
> > > implementations at runtime with blas-config and lapack-config.
> >
> > It is not that simple. If numeric and scipy are to make use of atlas,
> > atlas has to be in their external library list at compile time. If atlas
> > is in the list, but not installed on the system, compilation will fail.
>
> That's why I said 'in theory' ;-)
>
> I'm currently working on a patch which just ignores any explicit atlas
> stuff at build time of SciPy. The idea is to build a library which simply
> links to /usr/lib/liblapack.so. SciPy doesn't need to know this is a
> symlink to /usr/lib/lapack/atlas/liblapack.so or any other implementation.

If you install lapack-atlas and blas-atlas, and link scipy to the blas and 
lapack libraries but not the atlas libraries, do you still get the enhanced 
performance from atlas? (This isn't a rhetorical question, I'm still 
learning.)

If so, great, if not, is there any point to providing the lapack-atlas and 
blas-atlas ebuilds?

On a related topic, I noticed that sci-libs/atlas is still being developed, 
isn't it deprecated? This ebuild installs its libraries in /usr/lib, instead 
of somewhere unique and providing symlinks in /usr/lib.

Darren
-- 
gentoo-science@gentoo.org mailing list

Re: [gentoo-science] lapack transition

[Peter Bienstman]

[-- Attachment #1: Type: text/plain, Size: 914 bytes --]

On Sunday 21 August 2005 17:31, Darren Dale wrote:

> If you install lapack-atlas and blas-atlas, and link scipy to the blas and
> lapack libraries but not the atlas libraries, do you still get the enhanced
> performance from atlas? (This isn't a rhetorical question, I'm still
> learning.)

Yes, that's the whole idea. The names 'lapack' and 'blas' are just symlinks to 
whatever library implements them, like the reference libraries, or ATLAS, or 
perhaps at a later stage MKL.

> On a related topic, I noticed that sci-libs/atlas is still being developed,
> isn't it deprecated? This ebuild installs its libraries in /usr/lib,
> instead of somewhere unique and providing symlinks in /usr/lib.

In the stable tree sci-libs/atlas is the only atlas package available. 
lapack-atlas is currently still masked, but the idea is to perform the 
transition soon and phase out sci-libs/atlas.

Peter

[-- Attachment #2: Type: application/pgp-signature, Size: 307 bytes --]

Re: [gentoo-science] lapack transition

[Darren Dale]

On Sunday 21 August 2005 11:41 am, Peter Bienstman wrote:
> On Sunday 21 August 2005 17:31, Darren Dale wrote:
> > If you install lapack-atlas and blas-atlas, and link scipy to the blas
> > and lapack libraries but not the atlas libraries, do you still get the
> > enhanced performance from atlas? (This isn't a rhetorical question, I'm
> > still learning.)
>
> Yes, that's the whole idea. The names 'lapack' and 'blas' are just symlinks
> to whatever library implements them, like the reference libraries, or
> ATLAS, or perhaps at a later stage MKL.

I think that clarifies things, but I wonder why Numeric and SciPy suggest 
including atlas in the library list if it isnt necessary.

Darren
-- 
gentoo-science@gentoo.org mailing list

Re: [gentoo-science] lapack transition

[Peter Bienstman]

[-- Attachment #1: Type: text/plain, Size: 778 bytes --]

On Sunday 21 August 2005 17:53, Darren Dale wrote:

> I think that clarifies things, but I wonder why Numeric and SciPy suggest
> including atlas in the library list if it isnt necessary.

The scipy and numeric docs don't know anything about the upcoming new Gentoo 
scheme, where liblapack can be a front for libatlas or any other library that 
implements lapack.

That's actually the beauty of the new system: even if you have a program which 
in its configure scripts does not know anything about alternate lapack 
implementations, its still possible to use it with ATLAS. 

But if you're building SciPy on a non-Gentoo system (or on the the current 
Gentoo, i.e. before the switch), then it's obviously necessary to include the 
correct atlas info.

Peter

[-- Attachment #2: Type: application/pgp-signature, Size: 307 bytes --]

Re: [gentoo-science] lapack transition

[M. Edward (Ed) Borasky]

I just returned to this list -- what is the "new infrastructure" we are 
"preparing for"?

BTW, I am also on the Atlas mailing list. There was quite a bit of 
activity over the weekend regarding GCC 4. It seems the default Fortran 
in GCC 4 is Fortran 95, which has instilled great trepidation in some of 
the "old-timers" on the Atlas list. In any event, should I wish to 
experiment with GCC 4, how do I go about doing so without creaming my 
system? Do I need a whole new machine/partition?

The good news is that Gentoo is *way* better than Debian or Fedora in 
its behavior re Atlas. Atlas is *designed* to tune itself to your 
machine, and the Debian packages, for example, are *way* out of date and 
compiled seperately for each architecture. I don't think I'll ever pry 
Dirk Eddelbuettel away from Debian, but there's a lot of hope for the 
rest of the scientific computing community, at least those who want a 
working Atlas out of the box. :)

Other notes: R. Clint Whaley, the head of the Atlas project, mentioned 
that he was interested in making shared libraries of Atlas. I told him 
that the Gentoo package already did that for BLAS-Atlas and LAPACK-Atlas. :)

Could we get a "testing/unstable" Atlas in Portage? Right now, they are 
at 3.7.10, and I only see a 3.7.10 for blas-atlas, not for atlas itself 
or lapack-atlas. I think the x86-64 users will want 3.7.10 across the 
board, and might also want to be able to compile selected code with GCC 4.

Peter Bienstman wrote:

>Hi,
>
>I've been looking through the tree to see which packages use lapack/blas, and 
>which of them are prepared for the new infrastructure. There are actually far 
>less packages than I anticipated:
>
>Still depends on old infrastructure:
>
>   sys-cluster/hpl
>
>Already uses virtual/blas or virtual/lapack in unstable branch:
>
>   dev-lang/R
>   sci-geosciences/grass
>   sci-misc/camfr
>   sci-chemistry/mpqc
>   sci-mathematics/octave
>   sci-misc/xfoil
>
>Uses own code, but could benefit from transition:
>
>   dev-python/numeric (see http://bugs.gentoo.org/show_bug.cgi?id=81520)
>
>Peter
>  
>
-- 
gentoo-science@gentoo.org mailing list

Re: [gentoo-science] lapack transition

[Peter Bienstman]

[-- Attachment #1: Type: text/plain, Size: 622 bytes --]

On Sunday 21 August 2005 17:33, M. Edward (Ed) Borasky wrote:
> I just returned to this list -- what is the "new infrastructure" we are
> "preparing for"?

The ability to switch between different lapack implementations (reference, 
ATLAS, later perhaps MKL) at run time.

> Could we get a "testing/unstable" Atlas in Portage? Right now, they are
> at 3.7.10, and I only see a 3.7.10 for blas-atlas, not for atlas itself
> or lapack-atlas. I think the x86-64 users will want 3.7.10 across the
> board, and might also want to be able to compile selected code with GCC 4.

That's also on the TODO list.

Peter

[-- Attachment #2: Type: application/pgp-signature, Size: 307 bytes --]

Re: [gentoo-science] lapack transition

[M. Edward (Ed) Borasky]

I just got an email from Clint Whaley on the Atlas mailing list. He's 
sent out 3.7.11 with bugfixes. Given the re-org, should I post a request 
for 3.7.11 on bugzilla? When is the re-org going to happen?

Peter Bienstman wrote:

>On Sunday 21 August 2005 17:33, M. Edward (Ed) Borasky wrote:
>  
>
>>I just returned to this list -- what is the "new infrastructure" we are
>>"preparing for"?
>>    
>>
>
>The ability to switch between different lapack implementations (reference, 
>ATLAS, later perhaps MKL) at run time.
>
>  
>
>>Could we get a "testing/unstable" Atlas in Portage? Right now, they are
>>at 3.7.10, and I only see a 3.7.10 for blas-atlas, not for atlas itself
>>or lapack-atlas. I think the x86-64 users will want 3.7.10 across the
>>board, and might also want to be able to compile selected code with GCC 4.
>>    
>>
>
>That's also on the TODO list.
>
>Peter
>  
>
-- 
gentoo-science@gentoo.org mailing list

Re: [gentoo-science] lapack transition

[Peter Bienstman]

[-- Attachment #1: Type: text/plain, Size: 397 bytes --]

On Sunday 21 August 2005 18:00, M. Edward (Ed) Borasky wrote:
> I just got an email from Clint Whaley on the Atlas mailing list. He's
> sent out 3.7.11 with bugfixes. Given the re-org, should I post a request
> for 3.7.11 on bugzilla? 

We'll concentrate on the reorg first, and then we''l see about a new ATLAS 
version.

All depends on the amount of time developers have, obviously ;-)

[-- Attachment #2: Type: application/pgp-signature, Size: 307 bytes --]

Re: [gentoo-science] lapack transition

[Donnie Berkholz]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

M. Edward (Ed) Borasky wrote:
| I just returned to this list -- what is the "new infrastructure" we are
| "preparing for"?
|
| BTW, I am also on the Atlas mailing list. There was quite a bit of
| activity over the weekend regarding GCC 4. It seems the default Fortran
| in GCC 4 is Fortran 95, which has instilled great trepidation in some of
| the "old-timers" on the Atlas list. In any event, should I wish to
| experiment with GCC 4, how do I go about doing so without creaming my
| system? Do I need a whole new machine/partition?

I thought f95 was not only the default, but the only choice, and f77 was
entirely gone.

gcc is slotted so you can have many versions installed at the same time
and switch with gcc-config. Just try unmasking in package.unmask and go.

Thanks,
Donnie
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFDCOQXXVaO67S1rtsRApNTAJ4qXjkRX39R0w2yki/7vvQ1vUPxTACferj/
XYnAgYAAy6IPYG3HsoijofE=
=KgvT
-----END PGP SIGNATURE-----
-- 
gentoo-science@gentoo.org mailing list

Re: [gentoo-science] lapack transition

[M. Edward (Ed) Borasky]

Donnie Berkholz wrote:

> I thought f95 was not only the default, but the only choice, and f77 was
> entirely gone.

Maybe ... it wasn't entirely clear to me from the discussion, which was 
relative to Atlas and (shudder) Fedora Core 4, where GCC 4 is the default.

>
> gcc is slotted so you can have many versions installed at the same time
> and switch with gcc-config. Just try unmasking in package.unmask and go.

In the case of Atlas, Clint Whaley tried a couple of tests with GCC 4 
and discovered on some pretty pure and simple scientific code, GCC 4 was 
quite a bit slower than 3.4 on an x86-32. He says, "stick with 3.4 for 
Atlas until 4 gets better". I don't have access to an AMD 64-bit machine 
and I don't have (legitimate, anyhow) access to an Intel x86-64 machine. 
So for now, anyhow, I can wait until all 10,000+ packages in Portage 
compile and test out with GCC 4 <ducking>
-- 
gentoo-science@gentoo.org mailing list