[gentoo-science] BLAS and LAPACK dependecy resolution

[gentoo-science] BLAS and LAPACK dependecy resolution

[Markus Oehme]

Hi everybody,

every time I touch anything realted to blas and lapack it blows up in my
face. I just did a fresh install on my computer and I'm getting all sorts of
'cannot find blas' and 'cannot find lapack' (hence currently no sage for me
since half it dependencies don't build). Is there some documentation as to
how to sort this mess out?

Even more fun: I'm currently unable to install eselect::science (which is as
I understand it necessary to flee the above mentioned hell), since
eselect-bashcomp requires >=eselect-1.3.8.

I'm even annoyed enough that I think I could devote some time to helping to
make an end to this snafu -- if somebody points out, how. (My first guess
is, that it would reduce the clusterfuck potential if the science overlay
and main tree machinery [eselect::science I'm looking at you], could be
synced again).


        Markus

-- 
Aoccdrnig to a threoy, it deosn't mttaer in waht oredr the ltteers in a wrod
are, the olny iprmoatnt tihng is taht the frist and lsat ltteer are in the
rghit pclae. The rset can be a taotl mses and you can sitll raed it in msot
csaes. Tihs is bcuseae the huamn mnid deos not raed ervey lteter by istlef,
but the wrod as a wlohe. And I awlyas thought slpeling was ipmorantt.

Re: [gentoo-science] BLAS and LAPACK dependecy resolution

[Horea Christian]

I would like to publicly endorse this. I also had plenty of issues with 
blas and lapack - thankfully they all got resolved as soon as I pulled 
eselect::science in as a dep for my ebuilds. I am still afraid that 
those ebuilds will break and crumble and I won't be able to do a thing 
about it as I could not get any information on how the blas/lapack 
problems arise and how come eselect solves them. Sorry for being unable 
to provide you with a solution, markus.

Cheers.

On Do 12 Sep 2013 00:16:54 CEST, Markus Oehme wrote:
> Hi everybody,
>
> every time I touch anything realted to blas and lapack it blows up in my
> face. I just did a fresh install on my computer and I'm getting all sorts of
> 'cannot find blas' and 'cannot find lapack' (hence currently no sage for me
> since half it dependencies don't build). Is there some documentation as to
> how to sort this mess out?
>
> Even more fun: I'm currently unable to install eselect::science (which is as
> I understand it necessary to flee the above mentioned hell), since
> eselect-bashcomp requires >=eselect-1.3.8.
>
> I'm even annoyed enough that I think I could devote some time to helping to
> make an end to this snafu -- if somebody points out, how. (My first guess
> is, that it would reduce the clusterfuck potential if the science overlay
> and main tree machinery [eselect::science I'm looking at you], could be
> synced again).
>
>
>          Markus
>

Re: [gentoo-science] BLAS and LAPACK dependecy resolution

[François Bissey]

On 2013-09-12 10:16, Markus Oehme wrote:
> Hi everybody,
> 
> every time I touch anything realted to blas and lapack it blows up in 
> my
> face. I just did a fresh install on my computer and I'm getting all 
> sorts of
> 'cannot find blas' and 'cannot find lapack' (hence currently no sage 
> for me
> since half it dependencies don't build). Is there some documentation as 
> to
> how to sort this mess out?
> 
> Even more fun: I'm currently unable to install eselect::science (which 
> is as
> I understand it necessary to flee the above mentioned hell), since
> eselect-bashcomp requires >=eselect-1.3.8.
> 
> I'm even annoyed enough that I think I could devote some time to 
> helping to
> make an end to this snafu -- if somebody points out, how. (My first 
> guess
> is, that it would reduce the clusterfuck potential if the science 
> overlay
> and main tree machinery [eselect::science I'm looking at you], could be
> synced again).
> 
My first step to solve problems is to mask
app-admin/eselect::gentoo
Second we should maintain and distribute a package.keywords list
for the blas/lapack stack, we could alternative distribute a 
package.mask
for the main tree blas/lapack stack.
Bear in mind that we cannot mask the main tree blas/lapack stack from
the overlay's profile. Such a package.mask only affects the overlay.
So in any case you will need to explicitly copy or link to a provided
file (whether it is a mask or a keyword file).

Francois

Re: [gentoo-science] BLAS and LAPACK dependecy resolution

[François Bissey]

[-- Attachment #1: Type: text/plain, Size: 247 bytes --]

The file in attachment is to be put under /etc/portage/package.mask
It will block any elements of the blas/lapack stock forcing you to use 
and keyword
elements from the science overlay instead. It may not be complete but I 
think it is.

Francois

[-- Attachment #2: blas.mask --]
[-- Type: text/plain, Size: 381 bytes --]

app-admin/eselect::gentoo
app-admin/eselect-blas
app-admin/eselect-cblas
app-admin/eselect-lapack
virtual/blas::gentoo
virtual/cblas::gentoo
virtual/lapack::gentoo
sci-libs/gsl::gentoo
app-doc/blas-docs::gentoo
app-doc/lapack-docs::gentoo
sci-libs/blas-reference::gentoo
sci-libs/cblas-reference::gentoo
sci-libs/lapack-reference::gentoo
sci-libs/mkl::gentoo
sci-libs/acml::gentoo

Re: [gentoo-science] BLAS and LAPACK dependecy resolution

[Markus Oehme]

Hi,

I did the following, which seems to have done the trick:
1. add the package.mask entries
2. merge eselect::science
3. remerge all libraries that come from the science overlay (everything from
   'eix -I --in-overlay science')
4. remove /usr/lib/libblas.so (which was kept by preserve-libs)
   that is actually do 'rm /usr/lib/libblas.so'
5. remerge lapack-reference (which provided /usr/lib/libblas.so)
6. remerge umfpack (which depended on /usr/lib/libblas.so)

In between 3. and 4. I tried 'emerge @preserved-rebuild' which did not
improve the situation (i.e. the preserved /usr/lib/libblas.so was still used
by the newly built packages).

Step 6 is probably collateral damage which is hard to formalize.

> [2 blas.mask <text/plain (base64)>]

The entry >=app-shells/bash-completion-2.1 should probably be added, since
the newer versions require >=eselect-1.3.8 which is only available in the
main tree.

Once the mask file is available via overlay it would probably be nice to
have the above steps documented. I would volunteer to write a short piece
for the wiki (but afaik I'm not allowed to edit
https://wiki.gentoo.org/wiki/Project:Science/Overlay so somebody else would
have to add it).


        Markus

-- 
Aoccdrnig to a threoy, it deosn't mttaer in waht oredr the ltteers in a wrod
are, the olny iprmoatnt tihng is taht the frist and lsat ltteer are in the
rghit pclae. The rset can be a taotl mses and you can sitll raed it in msot
csaes. Tihs is bcuseae the huamn mnid deos not raed ervey lteter by istlef,
but the wrod as a wlohe. And I awlyas thought slpeling was ipmorantt.

Re: [gentoo-science] BLAS and LAPACK dependecy resolution

[Francois Bissey]

On 12/09/13 21:19, Markus Oehme wrote:
> Hi,
> 
> I did the following, which seems to have done the trick:
> 1. add the package.mask entries
> 2. merge eselect::science
> 3. remerge all libraries that come from the science overlay (everything from
>    'eix -I --in-overlay science')
> 4. remove /usr/lib/libblas.so (which was kept by preserve-libs)
>    that is actually do 'rm /usr/lib/libblas.so'
> 5. remerge lapack-reference (which provided /usr/lib/libblas.so)
> 6. remerge umfpack (which depended on /usr/lib/libblas.so)
> 
> In between 3. and 4. I tried 'emerge @preserved-rebuild' which did not
> improve the situation (i.e. the preserved /usr/lib/libblas.so was still used
> by the newly built packages).
> 
> Step 6 is probably collateral damage which is hard to formalize.
> 
>> [2 blas.mask <text/plain (base64)>]
> 
> The entry >=app-shells/bash-completion-2.1 should probably be added, since
> the newer versions require >=eselect-1.3.8 which is only available in the
> main tree.
> 

That is incorrect. You should sync the science overlay again as there is
a eselect-1.3.8-r100 in overlay.


> Once the mask file is available via overlay it would probably be nice to
> have the above steps documented. I would volunteer to write a short piece
> for the wiki (but afaik I'm not allowed to edit
> https://wiki.gentoo.org/wiki/Project:Science/Overlay so somebody else would
> have to add it).
> 
> 

Yes that would be nice. We know about cleaning up the links but it is
not documented anywhere and yes it leads to interesting problems.
A clean up script automating the process would be useful.

Francois

Re: [gentoo-science] BLAS and LAPACK dependecy resolution

[Markus Oehme]

Hi,

I tried my hand at a wiki blurb which can be found at [1]. I changed the
procedure to disable preserved-libs which will probably give the most
reproducible results. Comments welcome.


                      Markus

[1] https://wiki.gentoo.org/wiki/User_talk:Houseofsuns

-- 
Aoccdrnig to a threoy, it deosn't mttaer in waht oredr the ltteers in a wrod
are, the olny iprmoatnt tihng is taht the frist and lsat ltteer are in the
rghit pclae. The rset can be a taotl mses and you can sitll raed it in msot
csaes. Tihs is bcuseae the huamn mnid deos not raed ervey lteter by istlef,
but the wrod as a wlohe. And I awlyas thought slpeling was ipmorantt.

Re: [gentoo-science] BLAS and LAPACK dependecy resolution

[François Bissey]

On 2013-09-13 00:52, Markus Oehme wrote:
> Hi,
> 
> I tried my hand at a wiki blurb which can be found at [1]. I changed 
> the
> procedure to disable preserved-libs which will probably give the most
> reproducible results. Comments welcome.
> 
> 
>                       Markus
> 
> [1] https://wiki.gentoo.org/wiki/User_talk:Houseofsuns

I made a couple of edits.

Francois

Re: [gentoo-science] BLAS and LAPACK dependecy resolution

[Markus Oehme]

Hi,

At Thu, 12 Sep 2013 11:19:11 +0200,
Markus wrote:
> 4. remove /usr/lib/libblas.so (which was kept by preserve-libs)
>    that is actually do 'rm /usr/lib/libblas.so'


I see something really strange: repeatedly merging lapack-reference causes
it to bounce between two states. Where in one state there are three
additional files installed by the package:
/usr/lib/debug/usr/lib64/libblas.so.debug
/usr/lib64/libblas.so
/usr/lib64/pkgconfig/blas.pc
I tried it a larger number of times and the package alternates predictably
between the two states. Any hints on how this can happen? Also it seems that
in the state where the files are not there other packages have difficulties
finding BLAS -- so the woes do not seem to be over yet. *sigh*


             Markus

-- 
Aoccdrnig to a threoy, it deosn't mttaer in waht oredr the ltteers in a wrod
are, the olny iprmoatnt tihng is taht the frist and lsat ltteer are in the
rghit pclae. The rset can be a taotl mses and you can sitll raed it in msot
csaes. Tihs is bcuseae the huamn mnid deos not raed ervey lteter by istlef,
but the wrod as a wlohe. And I awlyas thought slpeling was ipmorantt.

Re: [gentoo-science] BLAS and LAPACK dependecy resolution

[Francois Bissey]

On 14/09/13 20:14, Markus Oehme wrote:
> Hi,
> 
> At Thu, 12 Sep 2013 11:19:11 +0200,
> Markus wrote:
>> 4. remove /usr/lib/libblas.so (which was kept by preserve-libs)
>>    that is actually do 'rm /usr/lib/libblas.so'
> 
> 
> I see something really strange: repeatedly merging lapack-reference causes
> it to bounce between two states. Where in one state there are three
> additional files installed by the package:
> /usr/lib/debug/usr/lib64/libblas.so.debug
> /usr/lib64/libblas.so
> /usr/lib64/pkgconfig/blas.pc
> I tried it a larger number of times and the package alternates predictably
> between the two states. Any hints on how this can happen? Also it seems that
> in the state where the files are not there other packages have difficulties
> finding BLAS -- so the woes do not seem to be over yet. *sigh*
> 
> 
lapack-reference includes blas-reference. It looks to me that what
happens when you get these extra files is the following:
1) There is no blas properly eselected (or is broken)
2) because of (1) lapack-reference fails to find a blas at configure
time and therefore builds its own.
3) lapack-reference install libblas.so and related files.
At this stage in the cycle you merge lapack-reference again
1) while there is no blas properly eselected at configure stage the
previously installed libblas.so is found.
2) lapack-reference uses it to build libreflapack.so.
3) when lapack-reference is merged it doesn't include a libblas.so
and portage removes it when cleaning files from the previous merge.

Repeat....

So the solution is: properly eselect a blas and make sure it is a valid
and sane configuration.

I know this is annoying but after each time you merge a
blas/cblas/lapack and related friends which use altenatives, you need to
check
what is eselected.

Usually something valid is eselected but if you have several
implementations at the same time it tends to reset to the first one
in the list after each merge.

Francois

Re: [gentoo-science] BLAS and LAPACK dependecy resolution

[Markus Oehme]

Hi,

At Sat, 14 Sep 2013 21:33:35 +1200,
Francois Bissey wrote:
> 
> So the solution is: properly eselect a blas and make sure it is a valid
> and sane configuration.

thanks, that worked. I modified the wiki piece to prevent this accident.


             Markus

-- 
Aoccdrnig to a threoy, it deosn't mttaer in waht oredr the ltteers in a wrod
are, the olny iprmoatnt tihng is taht the frist and lsat ltteer are in the
rghit pclae. The rset can be a taotl mses and you can sitll raed it in msot
csaes. Tihs is bcuseae the huamn mnid deos not raed ervey lteter by istlef,
but the wrod as a wlohe. And I awlyas thought slpeling was ipmorantt.

Wiki stuff (was: [gentoo-science] BLAS and LAPACK dependecy resolution)

[Markus Oehme]

Hi,

everything seems to work fine, so it's probably indicated to move the blurb
to its proper location. Anyone up for this?


       Markus

-- 
Aoccdrnig to a threoy, it deosn't mttaer in waht oredr the ltteers in a wrod
are, the olny iprmoatnt tihng is taht the frist and lsat ltteer are in the
rghit pclae. The rset can be a taotl mses and you can sitll raed it in msot
csaes. Tihs is bcuseae the huamn mnid deos not raed ervey lteter by istlef,
but the wrod as a wlohe. And I awlyas thought slpeling was ipmorantt.

Re: [gentoo-science] BLAS and LAPACK dependecy resolution

[Markus Oehme]

Hi,

I wrote some paragraphs on how to migrate to science overlay as a result of
this thread. Does anybody care to move this somewhere near the
Project:Science/Overlay page? (Since I'm unable to edit that page.)


                                Markus

[1] https://wiki.gentoo.org/wiki/User_talk:Houseofsuns

-- 
Aoccdrnig to a threoy, it deosn't mttaer in waht oredr the ltteers in a wrod
are, the olny iprmoatnt tihng is taht the frist and lsat ltteer are in the
rghit pclae. The rset can be a taotl mses and you can sitll raed it in msot
csaes. Tihs is bcuseae the huamn mnid deos not raed ervey lteter by istlef,
but the wrod as a wlohe. And I awlyas thought slpeling was ipmorantt.