[gentoo-science] On BLAS and LAPACK int64 ABI

[gentoo-science] On BLAS and LAPACK int64 ABI

[Benda Xu]

Hi,

This discussion emerged from https://github.com/gentoo/sci/pull/876.  I
am taking it here for better future reference.

cdluminate:
> It would be problematic to provide int64 ABI/API through libblas.so or
> liblapack.so since that would lead to a mess with the int32
> ABI/API. Adding the int64 USE flags to existing packages will make it
> hard to write ebuilds for users who want e.g. both int32 and int64
> openblas at the same time. My personal recommendation is to provide a
> new set of packages, say eselect-{blas64,lapack64} + sci-libs/lapack64,
> which are clearly isolated with the int32 ones. These new packages
> should change SONAMEs into say libblas64.so and liblapack64.so. This is
> the same as what I'm doing for Debian's 64bit-indexing BLAS/LAPACK
> packages.

heroxbd:
> We can make int64 as a use flag and by enabling it we install 2 sets of
> libraries: libblas.so and libblas64.so. What do you think?

kiwifb:
> I know why you want that @heroxbd - you want to avoid ebuild
> duplication. On the face of it it is less ebuilds to maintain in
> exchange of more complex ebuilds. However I think it would still be
> wise to have at least separate virtuals - that means dependencies on
> int32 and int64 are completely separate and allow a different set of
> implementations without impinging on each other.

> Whether to go for doubling the ebuilds or making them more complex is
> up to the maintainer. And we have to deal with the headers in any
> case.

cdluminate:
> I agree with @kiwifb . At least we need
> virtual/{blas64,cblas64,lapack64,lapacke64}. For the implementations
> we could introduce two new USE flags: int32 and int64, where each of
> them could trigger a separate full build, and int32 is enabled by
> default. Same for the eselect packages.

I agree with int32 and int64 USE flags. but int32 and int64 is too
simple and misleading, how about index_int32 and index_int64?  They are
named after python_targets_python3_7.

Virtual/blas could do the same: virtual/blas[index_int32,index_int64].

Yours,
Benda

Re: [gentoo-science] On BLAS and LAPACK int64 ABI

[Mo Zhou]

Hi Gentoo Science team,

In the sci overlay there is a long existing feature "int64" for the
BLAS/LAPACK packages, with which the type of array index in the
API/ABI will be changed into int64_t so that BLAS/LAPACK libraries
could be able to handle super large arrays. The reason of existence
of such feature stems from a Fortran compiler feature which could
change the default INTEGER size (e.g. 32bit, 64bit).

Recently the BLAS/LAPACK runtime switching mechanism has been merged
into ::gentoo, but you may find no counterpart to the "int64" feature
existing in ::sci. According to previous discussion, we could indeed
add a new alternative group called, for example, BLAS64 and LAPACK64.
However, it must be pointed out that, even if the "BLAS64" thing is
well supported by commonly used optimized implementations such as
BLIS, OpenBLAS, and MKL, the 64-bit-indexing feature is not well
supported by the reference CBLAS. Even worse, there is no common
standard for such "BLAS64" and "LAPACK64" libraries in terms of
ABI and SONAME. That means every move on the "BLAS64/LAPACK64"
runtime switching mechanism will be highly experimental and
may easily be broken by the authorities who define standards.

Before I started the GSoC project for Gentoo I had already started
working on adding BLAS64/LAPACK64 alternatives. We Debian science
team thought it's fine to experiment even if there is no standard
at all. Exactly due to this reason, the 64-bit-index feature wasn't
put on the original GSoC schedule. I thought we could first
experiment it on Debian.

However, now we have to face the problem again because some packages
such as Julia require the 64-bit-index ABI. (ah, yes, I was aware of
this since I also maintain Julia for Debian). Combining the fact that
there is no standard (even de-facto standard), and the demand of
some packages such as Julia, Benda and I reached an agreement that
we could simply add 64-bit-index support for OpenBLAS and postpone
the runtime switching feature for BLAS64/LAPACK64.

Julia calls BLAS64 and LAPACK64 through FFI. Theoretically Julia
could use any BLAS/LAPACK implementations but actually Julia's
LinearAlgebra standard library is not quite ready for such a
switching feature. Hence Debian's Julia package is actually
specifically linked against libopenblas.so (32-bit) and it is
irrelevant to the runtime switching feature. Afterall there is
no problem by forcing Julia users to use OpenBLAS.

In a word, what Benda and I plan to do are:

* add index-64bit USE flag to openblas, which compiles an extra
  libopenblas64.so for Julia's use.
* postpone the plan for BLAS64/LAPACK64 runtime switching, and
  experiment on Debian first.

That said, if anyone believes the BLAS64/LAPACK64 feature is sort
of urgent need, we can discuss further and revise the plan again.

Thanks.

See also:
[1] the 64-bit-indexing feature has been postponed by
    reference lapack upstream:
https://github.com/Reference-LAPACK/lapack/issues/184


On 2019-07-11 13:59, Benda Xu wrote:
> Hi,
> 
> This discussion emerged from https://github.com/gentoo/sci/pull/876.  I
> am taking it here for better future reference.
> 
> cdluminate:
>> It would be problematic to provide int64 ABI/API through libblas.so or
>> liblapack.so since that would lead to a mess with the int32
>> ABI/API. Adding the int64 USE flags to existing packages will make it
>> hard to write ebuilds for users who want e.g. both int32 and int64
>> openblas at the same time. My personal recommendation is to provide a
>> new set of packages, say eselect-{blas64,lapack64} + sci-libs/lapack64,
>> which are clearly isolated with the int32 ones. These new packages
>> should change SONAMEs into say libblas64.so and liblapack64.so. This is
>> the same as what I'm doing for Debian's 64bit-indexing BLAS/LAPACK
>> packages.
> 
> heroxbd:
>> We can make int64 as a use flag and by enabling it we install 2 sets of
>> libraries: libblas.so and libblas64.so. What do you think?
> 
> kiwifb:
>> I know why you want that @heroxbd - you want to avoid ebuild
>> duplication. On the face of it it is less ebuilds to maintain in
>> exchange of more complex ebuilds. However I think it would still be
>> wise to have at least separate virtuals - that means dependencies on
>> int32 and int64 are completely separate and allow a different set of
>> implementations without impinging on each other.
> 
>> Whether to go for doubling the ebuilds or making them more complex is
>> up to the maintainer. And we have to deal with the headers in any
>> case.
> 
> cdluminate:
>> I agree with @kiwifb . At least we need
>> virtual/{blas64,cblas64,lapack64,lapacke64}. For the implementations
>> we could introduce two new USE flags: int32 and int64, where each of
>> them could trigger a separate full build, and int32 is enabled by
>> default. Same for the eselect packages.
> 
> I agree with int32 and int64 USE flags. but int32 and int64 is too
> simple and misleading, how about index_int32 and index_int64?  They are
> named after python_targets_python3_7.
> 
> Virtual/blas could do the same: virtual/blas[index_int32,index_int64].
> 
> Yours,
> Benda

Re: [gentoo-science] On BLAS and LAPACK int64 ABI

[François Bissey]

From memory, as far as I know only octave can only be configured to 
use int64 (or at least it could). 
As you can imagine supporting int64 requires specific support
on the part of the application using the library. There has been
a few entries in bugzilla about people not understand why R broke
after linking it to int64 blas/lapack in the early days.

Apart from Julia and a few commercial code there are a few custom
scientific applications here and there using int64. Very few widely
distributed packages.

In that light, I think the proposal is quite sensible. People
who want to use int64 are likely to use either MKL or openblas,
may be BLIS [especially handy for C++ but then you probably don’t
care too much about the BLAS compatibility layer].

François

> On 17/07/2019, at 10:39 PM, Mo Zhou <lumin@debian.org> wrote:
> 
> Hi Gentoo Science team,
> 
> In the sci overlay there is a long existing feature "int64" for the
> BLAS/LAPACK packages, with which the type of array index in the
> API/ABI will be changed into int64_t so that BLAS/LAPACK libraries
> could be able to handle super large arrays. The reason of existence
> of such feature stems from a Fortran compiler feature which could
> change the default INTEGER size (e.g. 32bit, 64bit).
> 
> Recently the BLAS/LAPACK runtime switching mechanism has been merged
> into ::gentoo, but you may find no counterpart to the "int64" feature
> existing in ::sci. According to previous discussion, we could indeed
> add a new alternative group called, for example, BLAS64 and LAPACK64.
> However, it must be pointed out that, even if the "BLAS64" thing is
> well supported by commonly used optimized implementations such as
> BLIS, OpenBLAS, and MKL, the 64-bit-indexing feature is not well
> supported by the reference CBLAS. Even worse, there is no common
> standard for such "BLAS64" and "LAPACK64" libraries in terms of
> ABI and SONAME. That means every move on the "BLAS64/LAPACK64"
> runtime switching mechanism will be highly experimental and
> may easily be broken by the authorities who define standards.
> 
> Before I started the GSoC project for Gentoo I had already started
> working on adding BLAS64/LAPACK64 alternatives. We Debian science
> team thought it's fine to experiment even if there is no standard
> at all. Exactly due to this reason, the 64-bit-index feature wasn't
> put on the original GSoC schedule. I thought we could first
> experiment it on Debian.
> 
> However, now we have to face the problem again because some packages
> such as Julia require the 64-bit-index ABI. (ah, yes, I was aware of
> this since I also maintain Julia for Debian). Combining the fact that
> there is no standard (even de-facto standard), and the demand of
> some packages such as Julia, Benda and I reached an agreement that
> we could simply add 64-bit-index support for OpenBLAS and postpone
> the runtime switching feature for BLAS64/LAPACK64.
> 
> Julia calls BLAS64 and LAPACK64 through FFI. Theoretically Julia
> could use any BLAS/LAPACK implementations but actually Julia's
> LinearAlgebra standard library is not quite ready for such a
> switching feature. Hence Debian's Julia package is actually
> specifically linked against libopenblas.so (32-bit) and it is
> irrelevant to the runtime switching feature. Afterall there is
> no problem by forcing Julia users to use OpenBLAS.
> 
> In a word, what Benda and I plan to do are:
> 
> * add index-64bit USE flag to openblas, which compiles an extra
>  libopenblas64.so for Julia's use.
> * postpone the plan for BLAS64/LAPACK64 runtime switching, and
>  experiment on Debian first.
> 
> That said, if anyone believes the BLAS64/LAPACK64 feature is sort
> of urgent need, we can discuss further and revise the plan again.
> 
> Thanks.
> 
> See also:
> [1] the 64-bit-indexing feature has been postponed by
>    reference lapack upstream:
> https://github.com/Reference-LAPACK/lapack/issues/184
> 
> 
> On 2019-07-11 13:59, Benda Xu wrote:
>> Hi,
>> 
>> This discussion emerged from https://github.com/gentoo/sci/pull/876.  I
>> am taking it here for better future reference.
>> 
>> cdluminate:
>>> It would be problematic to provide int64 ABI/API through libblas.so or
>>> liblapack.so since that would lead to a mess with the int32
>>> ABI/API. Adding the int64 USE flags to existing packages will make it
>>> hard to write ebuilds for users who want e.g. both int32 and int64
>>> openblas at the same time. My personal recommendation is to provide a
>>> new set of packages, say eselect-{blas64,lapack64} + sci-libs/lapack64,
>>> which are clearly isolated with the int32 ones. These new packages
>>> should change SONAMEs into say libblas64.so and liblapack64.so. This is
>>> the same as what I'm doing for Debian's 64bit-indexing BLAS/LAPACK
>>> packages.
>> 
>> heroxbd:
>>> We can make int64 as a use flag and by enabling it we install 2 sets of
>>> libraries: libblas.so and libblas64.so. What do you think?
>> 
>> kiwifb:
>>> I know why you want that @heroxbd - you want to avoid ebuild
>>> duplication. On the face of it it is less ebuilds to maintain in
>>> exchange of more complex ebuilds. However I think it would still be
>>> wise to have at least separate virtuals - that means dependencies on
>>> int32 and int64 are completely separate and allow a different set of
>>> implementations without impinging on each other.
>> 
>>> Whether to go for doubling the ebuilds or making them more complex is
>>> up to the maintainer. And we have to deal with the headers in any
>>> case.
>> 
>> cdluminate:
>>> I agree with @kiwifb . At least we need
>>> virtual/{blas64,cblas64,lapack64,lapacke64}. For the implementations
>>> we could introduce two new USE flags: int32 and int64, where each of
>>> them could trigger a separate full build, and int32 is enabled by
>>> default. Same for the eselect packages.
>> 
>> I agree with int32 and int64 USE flags. but int32 and int64 is too
>> simple and misleading, how about index_int32 and index_int64?  They are
>> named after python_targets_python3_7.
>> 
>> Virtual/blas could do the same: virtual/blas[index_int32,index_int64].
>> 
>> Yours,
>> Benda