[gentoo-science] sci-libs/scipy-0.9.0-r1 undefined symbol: clapack_sgesv

[gentoo-science] sci-libs/scipy-0.9.0-r1 undefined symbol: clapack_sgesv

[Marc Vinyals]

[-- Attachment #1: Type: text/plain, Size: 924 bytes --]

Hi,

While updating sage and python to 4.7 and 2.7, I hit a variation of an
old bug (see http://bugs.gentoo.org/195619): scipy compiles with
lapack-reference but is unusable. Running in python

import scipy.optimize

fails with

[...]
    from scipy.linalg import clapack
ImportError: /usr/lib64/python2.7/site-packages/scipy/linalg/clapack.so:
undefined symbol: clapack_sgesv

I tried rebuilding numpy and scipy with lapack provided (eselect) by
lapack-reference, blas provided by blas-reference, cblas provided by
gsl and with/without clapack with the same result.

Then I tried rebuilding numpy and scipy with lapack, blas and cblas
provided (eselect) by atlas and scipy worked flawlessly.

Should scipy depend on a specific lapack provider instead of a virtual?
Enable it at compile time? Is it an unfortunate combination what fails?

I can provide build logs if you need them.

Best,
Marc.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-science] sci-libs/scipy-0.9.0-r1 undefined symbol: clapack_sgesv

[Francois Bissey]

> Hi,
> 
> While updating sage and python to 4.7 and 2.7, I hit a variation of an
> old bug (see http://bugs.gentoo.org/195619): scipy compiles with
> lapack-reference but is unusable. Running in python
> 
> import scipy.optimize
> 
> fails with
> 
> [...]
>     from scipy.linalg import clapack
> ImportError: /usr/lib64/python2.7/site-packages/scipy/linalg/clapack.so:
> undefined symbol: clapack_sgesv
> 
> I tried rebuilding numpy and scipy with lapack provided (eselect) by
> lapack-reference, blas provided by blas-reference, cblas provided by
> gsl and with/without clapack with the same result.
> 
> Then I tried rebuilding numpy and scipy with lapack, blas and cblas
> provided (eselect) by atlas and scipy worked flawlessly.
> 
> Should scipy depend on a specific lapack provider instead of a virtual?
> Enable it at compile time? Is it an unfortunate combination what fails?
> 
> I can provide build logs if you need them.
> 
Hi Marc,

Just one thing that I can think of on the top of my head. We recently made
scipy depend on arpack and superlu. They also have dependencies on blas/lapack
if you didn't rebuild them as well they may cause trouble. 

I had a similar problem recently but eselecting everything properly worked for
me. A last piece of info you could provide: are you using the main tree 
implementation or bicatali's overlay implementation?

Francois

Re: [gentoo-science] sci-libs/scipy-0.9.0-r1 undefined symbol: clapack_sgesv

[Marc Vinyals]

[-- Attachment #1: Type: text/plain, Size: 1867 bytes --]

EL Wed, 1 Jun 2011 06:57:03 +1200
Francois Bissey <fbissey@slingshot.co.nz> escrigué:

> > Hi,
> > 
> > While updating sage and python to 4.7 and 2.7, I hit a variation of
> > an old bug (see http://bugs.gentoo.org/195619): scipy compiles with
> > lapack-reference but is unusable. Running in python
> > 
> > import scipy.optimize
> > 
> > fails with
> > 
> > [...]
> >     from scipy.linalg import clapack
> > ImportError: /usr/lib64/python2.7/site-packages/scipy/linalg/clapack.so:
> > undefined symbol: clapack_sgesv
> > 
> > I tried rebuilding numpy and scipy with lapack provided (eselect) by
> > lapack-reference, blas provided by blas-reference, cblas provided by
> > gsl and with/without clapack with the same result.
> > 
> > Then I tried rebuilding numpy and scipy with lapack, blas and cblas
> > provided (eselect) by atlas and scipy worked flawlessly.
> > 
> > Should scipy depend on a specific lapack provider instead of a
> > virtual? Enable it at compile time? Is it an unfortunate
> > combination what fails?
> > 
> > I can provide build logs if you need them.
> > 
> Hi Marc,
> 
> Just one thing that I can think of on the top of my head. We recently
> made scipy depend on arpack and superlu. They also have dependencies
> on blas/lapack if you didn't rebuild them as well they may cause
> trouble. 
> 
> I had a similar problem recently but eselecting everything properly
> worked for me. A last piece of info you could provide: are you using
> the main tree implementation or bicatali's overlay implementation?
> 
> Francois
> 

Hi Francois,

I tried rebuilding arpack and superlu without luck. Since you mentioned
that eselect fixed your problems, I also tried installing
cblas-reference and using it instead of gsl, also without luck.

I'm using the main tree implementation.

Best,
Marc.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-science] sci-libs/scipy-0.9.0-r1 undefined symbol: clapack_sgesv

[Francois Bissey]

> EL Wed, 1 Jun 2011 06:57:03 +1200
> 
> Francois Bissey <fbissey@slingshot.co.nz> escrigué:
> > > Hi,
> > > 
> > > While updating sage and python to 4.7 and 2.7, I hit a variation of
> > > an old bug (see http://bugs.gentoo.org/195619): scipy compiles with
> > > lapack-reference but is unusable. Running in python
> > > 
> > > import scipy.optimize
> > > 
> > > fails with
> > > 
> > > [...]
> > > 
> > >     from scipy.linalg import clapack
> > > 
> > > ImportError:
> > > /usr/lib64/python2.7/site-packages/scipy/linalg/clapack.so: undefined
> > > symbol: clapack_sgesv
> > > 
> > > I tried rebuilding numpy and scipy with lapack provided (eselect) by
> > > lapack-reference, blas provided by blas-reference, cblas provided by
> > > gsl and with/without clapack with the same result.
> > > 
> > > Then I tried rebuilding numpy and scipy with lapack, blas and cblas
> > > provided (eselect) by atlas and scipy worked flawlessly.
> > > 
> > > Should scipy depend on a specific lapack provider instead of a
> > > virtual? Enable it at compile time? Is it an unfortunate
> > > combination what fails?
> > > 
> > > I can provide build logs if you need them.
> > 
> > Hi Marc,
> > 
> > Just one thing that I can think of on the top of my head. We recently
> > made scipy depend on arpack and superlu. They also have dependencies
> > on blas/lapack if you didn't rebuild them as well they may cause
> > trouble.
> > 
> > I had a similar problem recently but eselecting everything properly
> > worked for me. A last piece of info you could provide: are you using
> > the main tree implementation or bicatali's overlay implementation?
> > 
> > Francois
> 
> Hi Francois,
> 
> I tried rebuilding arpack and superlu without luck. Since you mentioned
> that eselect fixed your problems, I also tried installing
> cblas-reference and using it instead of gsl, also without luck.
> 
> I'm using the main tree implementation.
> 
Hi Marc,

I have been using bicatali's overlay for a bit and I recompiled all the 
cblas/lapack stack from sage a few times with different implementations
in the last few days. The only time i had an issue was when the value
eselected was not what I wanted by accident.

I am curious to see your build log for numpy and scipy. Also what does 
ldd /usr/lib64/python2.7/site-packages/scipy/linalg/clapack.so
reports?

Francois

Re: [gentoo-science] sci-libs/scipy-0.9.0-r1 undefined symbol: clapack_sgesv

[Marc Vinyals]

[-- Attachment #1.1: Type: text/plain, Size: 2885 bytes --]

EL Thu, 2 Jun 2011 07:22:18 +1200
Francois Bissey <fbissey@slingshot.co.nz> escrigué:

> > EL Wed, 1 Jun 2011 06:57:03 +1200
> > 
> > Francois Bissey <fbissey@slingshot.co.nz> escrigué:
> > > > Hi,
> > > > 
> > > > While updating sage and python to 4.7 and 2.7, I hit a
> > > > variation of an old bug (see http://bugs.gentoo.org/195619):
> > > > scipy compiles with lapack-reference but is unusable. Running
> > > > in python
> > > > 
> > > > import scipy.optimize
> > > > 
> > > > fails with
> > > > 
> > > > [...]
> > > > 
> > > >     from scipy.linalg import clapack
> > > > 
> > > > ImportError:
> > > > /usr/lib64/python2.7/site-packages/scipy/linalg/clapack.so:
> > > > undefined symbol: clapack_sgesv
> > > > 
> > > > I tried rebuilding numpy and scipy with lapack provided
> > > > (eselect) by lapack-reference, blas provided by blas-reference,
> > > > cblas provided by gsl and with/without clapack with the same
> > > > result.
> > > > 
> > > > Then I tried rebuilding numpy and scipy with lapack, blas and
> > > > cblas provided (eselect) by atlas and scipy worked flawlessly.
> > > > 
> > > > Should scipy depend on a specific lapack provider instead of a
> > > > virtual? Enable it at compile time? Is it an unfortunate
> > > > combination what fails?
> > > > 
> > > > I can provide build logs if you need them.
> > > 
> > > Hi Marc,
> > > 
> > > Just one thing that I can think of on the top of my head. We
> > > recently made scipy depend on arpack and superlu. They also have
> > > dependencies on blas/lapack if you didn't rebuild them as well
> > > they may cause trouble.
> > > 
> > > I had a similar problem recently but eselecting everything
> > > properly worked for me. A last piece of info you could provide:
> > > are you using the main tree implementation or bicatali's overlay
> > > implementation?
> > > 
> > > Francois
> > 
> > Hi Francois,
> > 
> > I tried rebuilding arpack and superlu without luck. Since you
> > mentioned that eselect fixed your problems, I also tried installing
> > cblas-reference and using it instead of gsl, also without luck.
> > 
> > I'm using the main tree implementation.
> > 
> Hi Marc,
> 
> I have been using bicatali's overlay for a bit and I recompiled all
> the cblas/lapack stack from sage a few times with different
> implementations in the last few days. The only time i had an issue
> was when the value eselected was not what I wanted by accident.
> 
> I am curious to see your build log for numpy and scipy. Also what
> does ldd /usr/lib64/python2.7/site-packages/scipy/linalg/clapack.so
> reports?
> 
> Francois
> 

Here are my logs. You'll find some warnings in the configuration
stage, I hope they'll mean something to you. Also, there's no trace of
libclapack.so in
ldd /usr/lib64/python2.7/site-packages/scipy/linalg/clapack.so.

[-- Attachment #1.2: dev-python:numpy-1.6.0:20110602-095859.log.bz2 --]
[-- Type: application/x-bzip, Size: 15878 bytes --]

[-- Attachment #1.3: sci-libs:scipy-0.9.0-r1:20110602-100122.log.bz2 --]
[-- Type: application/x-bzip, Size: 52433 bytes --]

[-- Attachment #1.4: ldd-clapack --]
[-- Type: application/octet-stream, Size: 448 bytes --]

	linux-vdso.so.1 =>  (0x00007fffd3716000)
	libpython2.7.so.1.0 => /usr/lib64/libpython2.7.so.1.0 (0x00007f574b7b6000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f574b42a000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f574b20c000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f574b008000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00007f574ae05000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f574ab81000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f574bdd1000)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-science] sci-libs/scipy-0.9.0-r1 undefined symbol: clapack_sgesv

[Francois Bissey]

> Here are my logs. You'll find some warnings in the configuration
> stage, I hope they'll mean something to you. Also, there's no trace of
> libclapack.so in
> ldd /usr/lib64/python2.7/site-packages/scipy/linalg/clapack.so.

The output of ldd /usr/lib64/python2.7/site-packages/scipy/linalg/clapack.so
is very suspicious with Atlas it should look like this:
ldd /usr/lib/python2.7/site-packages/scipy/linalg/clapack.so
        linux-gate.so.1 =>  (0xb7890000)
        liblapack.so.0 => /usr/lib/liblapack.so.0 (0xb732a000)
        libpython2.7.so.1.0 => /usr/lib/libpython2.7.so.1.0 (0xb71bb000)
        libc.so.6 => /lib/libc.so.6 (0xb7035000)
        libblas.so.0 => /usr/lib/blas/atlas/libblas.so.0 (0xb7014000)
        libcblas.so.0 => /usr/lib/blas/atlas/libcblas.so.0 (0xb6ff2000)
        libatlas.so.0 => /usr/lib/libatlas.so.0 (0xb6933000)
        libgfortran.so.3 => /usr/lib/gcc/i686-pc-linux-
gnu/4.5.2/libgfortran.so.3 (0xb6871000)
        libm.so.6 => /lib/libm.so.6 (0xb684b000)
        libgcc_s.so.1 => /usr/lib/gcc/i686-pc-linux-gnu/4.5.2/libgcc_s.so.1 
(0xb682d000)
        libpthread.so.0 => /lib/libpthread.so.0 (0xb6813000)
        libdl.so.2 => /lib/libdl.so.2 (0xb680f000)
        libutil.so.1 => /lib/libutil.so.1 (0xb680b000)
        /lib/ld-linux.so.2 (0xb7891000)

So nothing from blas/cblas/lapack was linked into your scipy which is very 
worrying. It may be that there is a problem with your current blas&co setup.
what's the output of
ls -la /usr/lib64/libcblas.so

Francois

Re: [gentoo-science] sci-libs/scipy-0.9.0-r1 undefined symbol: clapack_sgesv

[Marc Vinyals]

[-- Attachment #1: Type: text/plain, Size: 2041 bytes --]

EL Sat, 4 Jun 2011 12:15:18 +1200
Francois Bissey <fbissey@slingshot.co.nz> escrigué:

> > Here are my logs. You'll find some warnings in the configuration
> > stage, I hope they'll mean something to you. Also, there's no trace
> > of libclapack.so in
> > ldd /usr/lib64/python2.7/site-packages/scipy/linalg/clapack.so.
> 
> The output of
> ldd /usr/lib64/python2.7/site-packages/scipy/linalg/clapack.so is
> very suspicious with Atlas it should look like this:
> ldd /usr/lib/python2.7/site-packages/scipy/linalg/clapack.so
> linux-gate.so.1 =>  (0xb7890000) liblapack.so.0
> => /usr/lib/liblapack.so.0 (0xb732a000) libpython2.7.so.1.0
> => /usr/lib/libpython2.7.so.1.0 (0xb71bb000) libc.so.6
> => /lib/libc.so.6 (0xb7035000) libblas.so.0
> => /usr/lib/blas/atlas/libblas.so.0 (0xb7014000) libcblas.so.0
> => /usr/lib/blas/atlas/libcblas.so.0 (0xb6ff2000) libatlas.so.0
> => /usr/lib/libatlas.so.0 (0xb6933000) libgfortran.so.3
> => /usr/lib/gcc/i686-pc-linux- gnu/4.5.2/libgfortran.so.3 (0xb6871000)
>         libm.so.6 => /lib/libm.so.6 (0xb684b000)
>         libgcc_s.so.1
> => /usr/lib/gcc/i686-pc-linux-gnu/4.5.2/libgcc_s.so.1 (0xb682d000)
>         libpthread.so.0 => /lib/libpthread.so.0 (0xb6813000)
>         libdl.so.2 => /lib/libdl.so.2 (0xb680f000)
>         libutil.so.1 => /lib/libutil.so.1 (0xb680b000)
>         /lib/ld-linux.so.2 (0xb7891000)
> 
> So nothing from blas/cblas/lapack was linked into your scipy which is
> very worrying. It may be that there is a problem with your current
> blas&co setup. what's the output of
> ls -la /usr/lib64/libcblas.so
> 
> Francois
> 

ls -la /usr/lib64/libcblas.so
lrwxrwxrwx 1 root root 26  2 jun 13:44 /usr/lib64/libcblas.so ->
blas/reference/libcblas.so

ls -la /usr/lib64/libblas.so
lrwxrwxrwx 1 root root 25  2 jun 13:43 /usr/lib64/libblas.so ->
blas/reference/libblas.so

ls -la /usr/lib64/liblapack.so
lrwxrwxrwx 1 root root 29  2 jun 13:47 /usr/lib64/liblapack.so ->
lapack/reference/liblapack.so

It looks reasonable to me.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-science] sci-libs/scipy-0.9.0-r1 undefined symbol: clapack_sgesv

[Francois Bissey]

> EL Sat, 4 Jun 2011 12:15:18 +1200
> 
> ls -la /usr/lib64/libcblas.so
> lrwxrwxrwx 1 root root 26  2 jun 13:44 /usr/lib64/libcblas.so ->
> blas/reference/libcblas.so
> 
> ls -la /usr/lib64/libblas.so
> lrwxrwxrwx 1 root root 25  2 jun 13:43 /usr/lib64/libblas.so ->
> blas/reference/libblas.so
> 
> ls -la /usr/lib64/liblapack.so
> lrwxrwxrwx 1 root root 29  2 jun 13:47 /usr/lib64/liblapack.so ->
> lapack/reference/liblapack.so
> 
> It looks reasonable to me.
It does. Yet your scipy doesn't link to anything blas related according
to the output of ldd that you sent. 
I note you are using numpy-1.6.0 I don't if that's linked to that but sage
will not work properly with it: 
http://trac.sagemath.org/sage_trac/ticket/11334
Can you try again with numpy-1.5.1 at least.
And then what are the outputs of:
ldd /usr/lib64/python2.7/site-packages/numpy/core/_dotblas.so
ldd /usr/lib64/python2.7/site-packages/numpy/linalg/lapack_lite.so

Francois

Re: [gentoo-science] sci-libs/scipy-0.9.0-r1 undefined symbol: clapack_sgesv

[Marc Vinyals]

[-- Attachment #1: Type: text/plain, Size: 759 bytes --]

EL Tue, 7 Jun 2011 07:33:27 +1200
Francois Bissey <fbissey@slingshot.co.nz> escrigué:

> I note you are using numpy-1.6.0 I don't if that's linked to that but
> sage will not work properly with it: 
> http://trac.sagemath.org/sage_trac/ticket/11334
> Can you try again with numpy-1.5.1 at least.

I am a bit surprised, but downgrading numpy actually worked. Thanks a
lot for your help, Francois! Sage-4.7 should depend on ~numpy-1.5.1
then, since it already depends on >=numpy-1.5.1.

> And then what are the outputs of:
> ldd /usr/lib64/python2.7/site-packages/numpy/core/_dotblas.so
> ldd /usr/lib64/python2.7/site-packages/numpy/linalg/lapack_lite.so

They link to lapack, blas and cblas, and they didn't change when I
downgraded numpy.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-science] sci-libs/scipy-0.9.0-r1 undefined symbol: clapack_sgesv

[Francois Bissey]

> EL Tue, 7 Jun 2011 07:33:27 +1200
> 
> Francois Bissey <fbissey@slingshot.co.nz> escrigué:
> > I note you are using numpy-1.6.0 I don't if that's linked to that but
> > sage will not work properly with it:
> > http://trac.sagemath.org/sage_trac/ticket/11334
> > Can you try again with numpy-1.5.1 at least.
> 
> I am a bit surprised, but downgrading numpy actually worked. Thanks a
> lot for your help, Francois! Sage-4.7 should depend on ~numpy-1.5.1
> then, since it already depends on >=numpy-1.5.1.
> 
> > And then what are the outputs of:
> > ldd /usr/lib64/python2.7/site-packages/numpy/core/_dotblas.so
> > ldd /usr/lib64/python2.7/site-packages/numpy/linalg/lapack_lite.so
> 
> They link to lapack, blas and cblas, and they didn't change when I
> downgraded numpy.

I fixed the numpy dependencies in the sage tree. It is still curious that scipy
didn't apparently build properly when using numpy-1.6.0. Someone should
investigate that.

Francois

Re: [gentoo-science] sci-libs/scipy-0.9.0-r1 undefined symbol: clapack_sgesv

[Steven Trogdon]

[-- Attachment #1: Type: text/plain, Size: 1298 bytes --]

On Wed, 8 Jun 2011 08:07:10 +1200
Francois Bissey <fbissey@slingshot.co.nz> wrote:

> > EL Tue, 7 Jun 2011 07:33:27 +1200
> > 
> > Francois Bissey <fbissey@slingshot.co.nz> escrigué:
> > > I note you are using numpy-1.6.0 I don't if that's linked to that but
> > > sage will not work properly with it:
> > > http://trac.sagemath.org/sage_trac/ticket/11334
> > > Can you try again with numpy-1.5.1 at least.
> > 
> > I am a bit surprised, but downgrading numpy actually worked. Thanks a
> > lot for your help, Francois! Sage-4.7 should depend on ~numpy-1.5.1
> > then, since it already depends on >=numpy-1.5.1.
> > 
> > > And then what are the outputs of:
> > > ldd /usr/lib64/python2.7/site-packages/numpy/core/_dotblas.so
> > > ldd /usr/lib64/python2.7/site-packages/numpy/linalg/lapack_lite.so
> > 
> > They link to lapack, blas and cblas, and they didn't change when I
> > downgraded numpy.
> 
> I fixed the numpy dependencies in the sage tree. It is still curious that
> scipy didn't apparently build properly when using numpy-1.6.0. Someone should
> investigate that.
> 
> Francois
> 

It appears that this issue may arise somewhere in the distutils that's shipped
with numpy-1.6.0. We'll see. See bug

http://bugs.gentoo.org/show_bug.cgi?id=371099

Steve

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]