[gentoo-science] is numpy using the old lapack interface?

[gentoo-science] is numpy using the old lapack interface?

[Thomas Kahle]

[-- Attachment #1: Type: text/plain, Size: 1256 bytes --]

Hi,

since the recent move to -lrefblas and friends numpy-1.5.1
(a.k.a. sage-numpy) does not link anymore.  Probably -lblas is hardcoded
somehwere and should be replaced by $(pkg-config --libs blas)?

Thanks,
Thomas

[...]
x86_64-pc-linux-gnu-gcc: numpy/core/blasdot/_dotblas.c
/usr/bin/gfortran -Wall -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu -shared build-3.2/temp.linux-x86_64-3.2/numpy/core/blasdot/_dotblas.o -L/usr/lib64 -L/usr/lib64 -Lbuild-3.2/temp.linux-x86_64-3.2 -lblas -lpython3.2 -lgfortran -o build-3.2/lib.linux-x86_64-3.2/numpy/core/_dotblas.cpython-32.so
/usr/lib/gcc/x86_64-pc-linux-gnu/4.5.4/../../../../x86_64-pc-linux-gnu/bin/ld: cannot find -lblas
collect2: ld returned 1 exit status
/usr/lib/gcc/x86_64-pc-linux-gnu/4.5.4/../../../../x86_64-pc-linux-gnu/bin/ld: cannot find -lblas
collect2: ld returned 1 exit status
error: Command "/usr/bin/gfortran -Wall -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu -shared build-3.2/temp.linux-x86_64-3.2/numpy/core/blasdot/_dotblas.o -L/usr/lib64 -L/usr/lib64 -Lbuild-3.2/temp.linux-x86_64-3.2 -lblas -lpython3.2 -lgfortran -o build-3.2/lib.linux-x86_64-3.2/numpy/core/_dotblas.cpython-32.so" failed with exit status 1



-- 
Thomas Kahle
http://dev.gentoo.org/~tomka/

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-science] is numpy using the old lapack interface?

[Francois Bissey]

On 30/10/12 06:38, Thomas Kahle wrote:
> Hi,
> 
> since the recent move to -lrefblas and friends numpy-1.5.1
> (a.k.a. sage-numpy) does not link anymore.  Probably -lblas is hardcoded
> somehwere and should be replaced by $(pkg-config --libs blas)?
> 
> Thanks,
> Thomas
> 
> [...]
> x86_64-pc-linux-gnu-gcc: numpy/core/blasdot/_dotblas.c
> /usr/bin/gfortran -Wall -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu -shared build-3.2/temp.linux-x86_64-3.2/numpy/core/blasdot/_dotblas.o -L/usr/lib64 -L/usr/lib64 -Lbuild-3.2/temp.linux-x86_64-3.2 -lblas -lpython3.2 -lgfortran -o build-3.2/lib.linux-x86_64-3.2/numpy/core/_dotblas.cpython-32.so
> /usr/lib/gcc/x86_64-pc-linux-gnu/4.5.4/../../../../x86_64-pc-linux-gnu/bin/ld: cannot find -lblas
> collect2: ld returned 1 exit status
> /usr/lib/gcc/x86_64-pc-linux-gnu/4.5.4/../../../../x86_64-pc-linux-gnu/bin/ld: cannot find -lblas
> collect2: ld returned 1 exit status
> error: Command "/usr/bin/gfortran -Wall -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu -shared build-3.2/temp.linux-x86_64-3.2/numpy/core/blasdot/_dotblas.o -L/usr/lib64 -L/usr/lib64 -Lbuild-3.2/temp.linux-x86_64-3.2 -lblas -lpython3.2 -lgfortran -o build-3.2/lib.linux-x86_64-3.2/numpy/core/_dotblas.cpython-32.so" failed with exit status 1
> 

It is already supposed to use pkg-config. My numpy 1.5.1 is linked
against openblas with python-2.7.3. However I don't appear to have
dotblas at all in numpy with python 3.2 so there may be an issue
when compiling against python 3.2 - I will have to check my build logs.

Did it compile against python 2.7 normally?

Francois

Re: [gentoo-science] is numpy using the old lapack interface?

[Thomas Kahle]

[-- Attachment #1: Type: text/plain, Size: 487 bytes --]

On 07:27 Tue 30 Oct 2012, Francois Bissey wrote:
[..]
> It is already supposed to use pkg-config. My numpy 1.5.1 is linked
> against openblas with python-2.7.3. However I don't appear to have
> dotblas at all in numpy with python 3.2 so there may be an issue
> when compiling against python 3.2 - I will have to check my build logs.
> 
> Did it compile against python 2.7 normally?

Yes, no problem there.

Cheers,
Thomas

-- 
Thomas Kahle
http://dev.gentoo.org/~tomka/

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-science] is numpy using the old lapack interface?

[Thomas Kahle]

[-- Attachment #1: Type: text/plain, Size: 1234 bytes --]

Hi again,

this thing is somewhat mysterious.

On 07:27 Tue 30 Oct 2012, Francois Bissey wrote:
> > x86_64-pc-linux-gnu-gcc: numpy/core/blasdot/_dotblas.c
> > /usr/bin/gfortran -Wall -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu -shared build-3.2/temp.linux-x86_64-3.2/numpy/core/blasdot/_dotblas.o -L/usr/lib64 -L/usr/lib64 -Lbuild-3.2/temp.linux-x86_64-3.2 -lblas -lpython3.2 -lgfortran -o build-3.2/lib.linux-x86_64-3.2/numpy/core/_dotblas.cpython-32.so
> > /usr/lib/gcc/x86_64-pc-linux-gnu/4.5.4/../../../../x86_64-pc-linux-gnu/bin/ld: cannot find -lblas
> > collect2: ld returned 1 exit status


What happens here is that some setup.py calls gfortran with -lblas
instead of what is in site.cfg.  I tried various grep's across the
source tree, but I don't understand how this happens :(

Any ideas?



> It is already supposed to use pkg-config. My numpy 1.5.1 is linked
> against openblas with python-2.7.3. However I don't appear to have
> dotblas at all in numpy with python 3.2 so there may be an issue
> when compiling against python 3.2 - I will have to check my build logs.
> 
> Did it compile against python 2.7 normally?
> 
> Francois
> 
> 

-- 
Thomas Kahle
http://dev.gentoo.org/~tomka/

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-science] is numpy using the old lapack interface?

[Thomas Kahle]

[-- Attachment #1: Type: text/plain, Size: 1961 bytes --]

One more thing, This is fixed in 1.6 tree of numpy, where the 
call to gfortran uses what is in the site config.
Will sage move to a higher version of numpy at some point?



On 07:27 Tue 30 Oct 2012, Francois Bissey wrote:
> On 30/10/12 06:38, Thomas Kahle wrote:
> > Hi,
> > 
> > since the recent move to -lrefblas and friends numpy-1.5.1
> > (a.k.a. sage-numpy) does not link anymore.  Probably -lblas is hardcoded
> > somehwere and should be replaced by $(pkg-config --libs blas)?
> > 
> > Thanks,
> > Thomas
> > 
> > [...]
> > x86_64-pc-linux-gnu-gcc: numpy/core/blasdot/_dotblas.c
> > /usr/bin/gfortran -Wall -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu -shared build-3.2/temp.linux-x86_64-3.2/numpy/core/blasdot/_dotblas.o -L/usr/lib64 -L/usr/lib64 -Lbuild-3.2/temp.linux-x86_64-3.2 -lblas -lpython3.2 -lgfortran -o build-3.2/lib.linux-x86_64-3.2/numpy/core/_dotblas.cpython-32.so
> > /usr/lib/gcc/x86_64-pc-linux-gnu/4.5.4/../../../../x86_64-pc-linux-gnu/bin/ld: cannot find -lblas
> > collect2: ld returned 1 exit status
> > /usr/lib/gcc/x86_64-pc-linux-gnu/4.5.4/../../../../x86_64-pc-linux-gnu/bin/ld: cannot find -lblas
> > collect2: ld returned 1 exit status
> > error: Command "/usr/bin/gfortran -Wall -Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu -shared build-3.2/temp.linux-x86_64-3.2/numpy/core/blasdot/_dotblas.o -L/usr/lib64 -L/usr/lib64 -Lbuild-3.2/temp.linux-x86_64-3.2 -lblas -lpython3.2 -lgfortran -o build-3.2/lib.linux-x86_64-3.2/numpy/core/_dotblas.cpython-32.so" failed with exit status 1
> > 
> 
> It is already supposed to use pkg-config. My numpy 1.5.1 is linked
> against openblas with python-2.7.3. However I don't appear to have
> dotblas at all in numpy with python 3.2 so there may be an issue
> when compiling against python 3.2 - I will have to check my build logs.
> 
> Did it compile against python 2.7 normally?
> 
> Francois
> 
> 

-- 
Thomas Kahle
http://dev.gentoo.org/~tomka/

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-science] is numpy using the old lapack interface?

[fbissey]

Quoting Thomas Kahle <tomka@gentoo.org>:

> One more thing, This is fixed in 1.6 tree of numpy, where the
> call to gfortran uses what is in the site config.
> Will sage move to a higher version of numpy at some point?
>
>
Yes! The problem that prevent us to upgrade numpy is fixed in numpy-1.7. The
release cannot come quickly enough for us.

For numpy 1.5 would by any chance gfortran be called with -fexternal-blas at
some point? On my system with python 3.2 lapack was not properly detected,
that's why dotblas hasn't been built. Numpy 1.5.1 with python 3.2 used an
internal implementation of the lapack functions it uses.

This probably explains why scipy doesn't compile with numpy 1.5 and 
python 3.2.

Francois

Re: [gentoo-science] is numpy using the old lapack interface?

[Thomas Kahle]

[-- Attachment #1: Type: text/plain, Size: 1159 bytes --]

On 09:42 Thu 01 Nov 2012, fbissey@slingshot.co.nz wrote:
> Quoting Thomas Kahle <tomka@gentoo.org>:
> 
> > One more thing, This is fixed in 1.6 tree of numpy, where the
> > call to gfortran uses what is in the site config.
> > Will sage move to a higher version of numpy at some point?
> >
> >
> Yes! The problem that prevent us to upgrade numpy is fixed in numpy-1.7. The
> release cannot come quickly enough for us.

OK.  Any idea when this will happen?

> For numpy 1.5 would by any chance gfortran be called with -fexternal-blas at
> some point? 

Can't find '-fexternal' in build.log.

> On my system with python 3.2 lapack was not properly detected,
> that's why dotblas hasn't been built. Numpy 1.5.1 with python 3.2 used an
> internal implementation of the lapack functions it uses.
> 
> This probably explains why scipy doesn't compile with numpy 1.5 and 
> python 3.2.

And this where all this started for me ... 

Anyway, the solution of my broken numpy,scipy is to wait for 1.7 or
rollback to a reference-lapack that installs itself as liblapack?

Thanks,
Thomas

-- 
Thomas Kahle
http://dev.gentoo.org/~tomka/

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-science] is numpy using the old lapack interface?

[fbissey]

Quoting Thomas Kahle <tomka@gentoo.org>:

> On 09:42 Thu 01 Nov 2012, fbissey@slingshot.co.nz wrote:
>> Quoting Thomas Kahle <tomka@gentoo.org>:
>>
>> > One more thing, This is fixed in 1.6 tree of numpy, where the
>> > call to gfortran uses what is in the site config.
>> > Will sage move to a higher version of numpy at some point?
>> >
>> >
>> Yes! The problem that prevent us to upgrade numpy is fixed in numpy-1.7. The
>> release cannot come quickly enough for us.
>
> OK.  Any idea when this will happen?
>
>> For numpy 1.5 would by any chance gfortran be called with -fexternal-blas at
>> some point?
>
> Can't find '-fexternal' in build.log.
>
>> On my system with python 3.2 lapack was not properly detected,
>> that's why dotblas hasn't been built. Numpy 1.5.1 with python 3.2 used an
>> internal implementation of the lapack functions it uses.
>>
>> This probably explains why scipy doesn't compile with numpy 1.5 and 
>> python 3.2.
>
> And this where all this started for me ...
>
> Anyway, the solution of my broken numpy,scipy is to wait for 1.7 or
> rollback to a reference-lapack that installs itself as liblapack?
>

If you don't use numpy with python 3.x you could simply create a file
/etc/portage/env/dev-python/numpy with USE_PYTHON="2.7" in it.
That would skip the offending problem quite nicely without locking you
in a particular blas implementation.

Francois

Re: [gentoo-science] is numpy using the old lapack interface?

[Thomas Kahle]

[-- Attachment #1: Type: text/plain, Size: 1861 bytes --]

On 12:06 Fri 02 Nov 2012, fbissey@slingshot.co.nz wrote:
> Quoting Thomas Kahle <tomka@gentoo.org>:
> 
> > On 09:42 Thu 01 Nov 2012, fbissey@slingshot.co.nz wrote:
> >> Quoting Thomas Kahle <tomka@gentoo.org>:
> >>
> >> > One more thing, This is fixed in 1.6 tree of numpy, where the
> >> > call to gfortran uses what is in the site config.
> >> > Will sage move to a higher version of numpy at some point?
> >> >
> >> >
> >> Yes! The problem that prevent us to upgrade numpy is fixed in numpy-1.7. The
> >> release cannot come quickly enough for us.
> >
> > OK.  Any idea when this will happen?
> >
> >> For numpy 1.5 would by any chance gfortran be called with -fexternal-blas at
> >> some point?
> >
> > Can't find '-fexternal' in build.log.
> >
> >> On my system with python 3.2 lapack was not properly detected,
> >> that's why dotblas hasn't been built. Numpy 1.5.1 with python 3.2 used an
> >> internal implementation of the lapack functions it uses.
> >>
> >> This probably explains why scipy doesn't compile with numpy 1.5 and 
> >> python 3.2.
> >
> > And this where all this started for me ...
> >
> > Anyway, the solution of my broken numpy,scipy is to wait for 1.7 or
> > rollback to a reference-lapack that installs itself as liblapack?
> >
> 
> If you don't use numpy with python 3.x you could simply create a file
> /etc/portage/env/dev-python/numpy with USE_PYTHON="2.7" in it.
> That would skip the offending problem quite nicely without locking you
> in a particular blas implementation.

Do you have a reference for your method?  It does not work for me.
Instead in /etc/portage/env/ I have a file nopy3k.conf which contains
just USE_PYTHON="2.7".  Then in /etc/portage/package.env I configure

dev-python/numpy nopy3k.conf

Cheers,
Thomas


-- 
Thomas Kahle
http://dev.gentoo.org/~tomka/

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-science] is numpy using the old lapack interface?

[Francois Bissey]

On 01/12/12 15:27, Thomas Kahle wrote:
> On 12:06 Fri 02 Nov 2012, fbissey@slingshot.co.nz wrote:
>> Quoting Thomas Kahle <tomka@gentoo.org>:
>>
>>> On 09:42 Thu 01 Nov 2012, fbissey@slingshot.co.nz wrote:
>>>> Quoting Thomas Kahle <tomka@gentoo.org>:
>>>>
>>>>> One more thing, This is fixed in 1.6 tree of numpy, where the
>>>>> call to gfortran uses what is in the site config.
>>>>> Will sage move to a higher version of numpy at some point?
>>>>>
>>>>>
>>>> Yes! The problem that prevent us to upgrade numpy is fixed in numpy-1.7. The
>>>> release cannot come quickly enough for us.
>>>
>>> OK.  Any idea when this will happen?
>>>
>>>> For numpy 1.5 would by any chance gfortran be called with -fexternal-blas at
>>>> some point?
>>>
>>> Can't find '-fexternal' in build.log.
>>>
>>>> On my system with python 3.2 lapack was not properly detected,
>>>> that's why dotblas hasn't been built. Numpy 1.5.1 with python 3.2 used an
>>>> internal implementation of the lapack functions it uses.
>>>>
>>>> This probably explains why scipy doesn't compile with numpy 1.5 and 
>>>> python 3.2.
>>>
>>> And this where all this started for me ...
>>>
>>> Anyway, the solution of my broken numpy,scipy is to wait for 1.7 or
>>> rollback to a reference-lapack that installs itself as liblapack?
>>>
>>
>> If you don't use numpy with python 3.x you could simply create a file
>> /etc/portage/env/dev-python/numpy with USE_PYTHON="2.7" in it.
>> That would skip the offending problem quite nicely without locking you
>> in a particular blas implementation.
> 
> Do you have a reference for your method?  It does not work for me.
> Instead in /etc/portage/env/ I have a file nopy3k.conf which contains
> just USE_PYTHON="2.7".  Then in /etc/portage/package.env I configure
> 
> dev-python/numpy nopy3k.conf
> 

I suppose that your way is newer, I have been using mine for a *number*
of years. From "man portage":
/etc/portage/env/
    In this directory additional package-specific bashrc files can   be
created.  Note that if  package-specific  environment  variable
settings  are  all  that's needed, then /etc/portage/package.env should
be used instead of the bashrc approach that is described here. Also note
that special variables such as FEATURES and  INSTALL_MASK  will
not produce the intended results if they are set in bashrc, and
therefore /etc/portage/package.env should be used instead.

    Portage will source all of these bashrc files after
/etc/portage/bashrc in the following order:
              1. /etc/portage/env/${CATEGORY}/${PN}
              2. /etc/portage/env/${CATEGORY}/${PN}:${SLOT}
              3. /etc/portage/env/${CATEGORY}/${P}
              4. /etc/portage/env/${CATEGORY}/${PF}

So technically it should work. It has a defect that probably makes the
package.env approach extremely preferable as I found out. Variables set
with this method (/etc/portage/env/${CATEGORY}/${PN}) will override
variables you pass on the shell. I found that the hard way.

Of course with the new python eclass some adjustment will be needed.


Francois