[gentoo-dev] ceph's static-libs

[gentoo-dev] ceph's static-libs

[Alessandro Barbieri]

[-- Attachment #1: Type: text/plain, Size: 546 bytes --]

I was trying to remove static-libs from hwloc and I noticed that the last
bump of ceph is requiring hwloc:=[static-libs?]
And I notices it needs also alot of other dependencies with [static-libs?]
Is there a *valid* reason for having ceph[static-libs] around in the first
place?

For more context on static-libs see:
https://projects.gentoo.org/qa/policy-guide/installed-files.html?highlight=static#pg0302
https://flameeyes.blog/2011/08/29/useless-flag-static-libs/
https://archives.gentoo.org/gentoo-dev/message/2dada80c2b9c85b0e83e6328428bf8ab

[-- Attachment #2: Type: text/html, Size: 1003 bytes --]

Re: [gentoo-dev] ceph's static-libs

[James Le Cuirot]

[-- Attachment #1: Type: text/plain, Size: 1613 bytes --]

On Sat, 4 Apr 2020 08:12:09 +0200
Alessandro Barbieri <lssndrbarbieri@gmail.com> wrote:

> I was trying to remove static-libs from hwloc and I noticed that the last
> bump of ceph is requiring hwloc:=[static-libs?]
> And I notices it needs also alot of other dependencies with [static-libs?]
> Is there a *valid* reason for having ceph[static-libs] around in the first
> place?
> 
> For more context on static-libs see:
> https://projects.gentoo.org/qa/policy-guide/installed-files.html?highlight=static#pg0302
> https://flameeyes.blog/2011/08/29/useless-flag-static-libs/
> https://archives.gentoo.org/gentoo-dev/message/2dada80c2b9c85b0e83e6328428bf8ab

I do like to have the option for static-libs where it's not too much
trouble. It's obviously not a mainline use case but I have needed it on
occasions.

I think these dependencies are wrong though and I've seen the same
thing in other packages. You don't need the dependent static libs
when building other static libs. For example. I have webp[-static-libs]
installed and I can build leptonica[static-libs,webp] just fine. They
are only needed when linking executable binaries and for that, you'll
typically have a static USE flag rather than static-libs. QEMU is a good
example with its static and static-user USE flags. You could force a
package to build static or partially static binaries through toolchain
flags but then it's down to the user to ensure that all the dependent
static libs are in place.

If the above paragraph is wrong, someone please correct me. :)

-- 
James Le Cuirot (chewi)
Gentoo Linux Developer

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] ceph's static-libs

[James Le Cuirot]

[-- Attachment #1: Type: text/plain, Size: 2380 bytes --]

On Sat, 4 Apr 2020 10:43:26 +0100
James Le Cuirot <chewi@gentoo.org> wrote:

> On Sat, 4 Apr 2020 08:12:09 +0200
> Alessandro Barbieri <lssndrbarbieri@gmail.com> wrote:
> 
> > I was trying to remove static-libs from hwloc and I noticed that the last
> > bump of ceph is requiring hwloc:=[static-libs?]
> > And I notices it needs also alot of other dependencies with [static-libs?]
> > Is there a *valid* reason for having ceph[static-libs] around in the first
> > place?
> > 
> > For more context on static-libs see:
> > https://projects.gentoo.org/qa/policy-guide/installed-files.html?highlight=static#pg0302
> > https://flameeyes.blog/2011/08/29/useless-flag-static-libs/
> > https://archives.gentoo.org/gentoo-dev/message/2dada80c2b9c85b0e83e6328428bf8ab  
> 
> I do like to have the option for static-libs where it's not too much
> trouble. It's obviously not a mainline use case but I have needed it on
> occasions.
> 
> I think these dependencies are wrong though and I've seen the same
> thing in other packages. You don't need the dependent static libs
> when building other static libs. For example. I have webp[-static-libs]
> installed and I can build leptonica[static-libs,webp] just fine. They
> are only needed when linking executable binaries and for that, you'll
> typically have a static USE flag rather than static-libs. QEMU is a good
> example with its static and static-user USE flags. You could force a
> package to build static or partially static binaries through toolchain
> flags but then it's down to the user to ensure that all the dependent
> static libs are in place.
> 
> If the above paragraph is wrong, someone please correct me. :)

Damn, I realised just as I hit send that there's a caveat here and
that's sub-dependencies. If you're building a partially static binary
then I think you're okay. A fully static binary obviously needs all its
dependencies to be static and that includes any sub-dependencies. If
an executable statically links libwebp.a with JPEG support but you
don't have libjpeg.a then you'll end up with undefined references. That
probably explains why the ceph dependencies are as they are but the
resulting headache makes me think it's not worth the hassle. This is a
power user case and I'd leave the users to figure it out.

-- 
James Le Cuirot (chewi)
Gentoo Linux Developer

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] ceph's static-libs

[Peter Stuge]

James Le Cuirot wrote:
> Damn, I realised just as I hit send that there's a caveat here and
> that's sub-dependencies. If you're building a partially static binary
> then I think you're okay. A fully static binary obviously needs all its
> dependencies to be static and that includes any sub-dependencies.

Note that there isn't really a way to express "partially static" to the
toolchain when building a binary.

If you link a binary -static then that is always "fully static". No .so
will satisfy any -l options.

The only way a "partially static" binary gets created is when linking
*without* -static but some -l libraries only exist as static libraries,
or if a library/object archive is specified with full .a filename,
without using -l.

And "partially static" also only means that some dependencies were
included into the binary, but unlike "fully static" the binary is
not runnable without ld.so and a fitting libc.


> If an executable statically links libwebp.a with JPEG support but you
> don't have libjpeg.a then you'll end up with undefined references.

It's a bit more complicated..

Assume: ( note: without -static )

gcc -o binary source.c /usr/lib/libwebp.a

If libwebp requires libjpeg, and libwebp was not built to include
libjpeg.a then the above will not build. So we try:

gcc -o binary source.c /usr/lib/libwebp.a /usr/lib/libjpeg.a

If both .a exist this builds, but binary is still *not* linked statically,
at a minimum, ld.so and libc are still required to run it.

If we do:

gcc -o binary source.c /usr/lib/libwebp.a -ljpeg

Then the compiler will pick any libjpeg that is available, preferably
.so but if that can't be found then it will also look for .a. binary is
still *not* static.


Finally consider:

gcc -static -o binary source.c -lwebp -ljpeg
gcc -static -o binary source.c /usr/lib/libwebp.a /usr/lib/libjpeg.a

The above two are equivalent, but only thanks to -static. This fails
if either library .a is not found. When this succeeds, we have a static
binary, which runs anywhere regardless of ld.so and libc.


pkg-config .pc files for libraries are a very convenient solution to
the problem of sub-dependencies. In the above case, libwebp.pc would
perhaps have -lwebp in Libs and libjpeg in Requires.private (or maybe
-ljpeg in Libs.private).


> That probably explains why the ceph dependencies are as they are

I think USE=static-libs is nice to have, and ideally (IMHO) it would
be a global USE flag, respected by every package that installs a
library. The flag says nothing about consumers, it only promises
availability of the .a files, which I think is nice.

One technical reason for a consumer to DEPEND on libotherpackage[static-libs]
would be if an upstream Makefile has /usr/lib/libother.a instead of -lother,
arguably it would make more sense to apply a patch to the Makefile then.

Another technical reason would be that libotherpackage doesn't adhere to
common sense/best practice for library ABI/API compatibility, and make
the static library *different* from the shared object. If libotherpackage
maintainer heroines have made it possible to have one SLOT installed as
static library and another, incompatible, SLOT installed as shared object
and the consumer maintainer might know that only the static variant works.
This one is very hard to do anything about about in Gentoo, but it is also
a very construed case. The closest I've seen to this is giflib, which
changed API and ABI without bumping SONAME.


//Peter

Re: [gentoo-dev] ceph's static-libs

[James Le Cuirot]

[-- Attachment #1: Type: text/plain, Size: 1945 bytes --]

On Sun, 5 Apr 2020 17:20:07 +0000
Peter Stuge <peter@stuge.se> wrote:

> James Le Cuirot wrote:
> > Damn, I realised just as I hit send that there's a caveat here and
> > that's sub-dependencies. If you're building a partially static binary
> > then I think you're okay. A fully static binary obviously needs all its
> > dependencies to be static and that includes any sub-dependencies.  
> 
> Note that there isn't really a way to express "partially static" to the
> toolchain when building a binary.
> 
> If you link a binary -static then that is always "fully static". No .so
> will satisfy any -l options.
> 
> The only way a "partially static" binary gets created is when linking
> *without* -static but some -l libraries only exist as static libraries,
> or if a library/object archive is specified with full .a filename,
> without using -l.
> 
> And "partially static" also only means that some dependencies were
> included into the binary, but unlike "fully static" the binary is
> not runnable without ld.so and a fitting libc.

Yeah, I am aware and was glossing over the details slightly but thanks
for clarifying.

You can force static when linking specific libraries with -Wl,-static
and then undo this with -Wl,-dynamic. I don't know whether it's
feasible to do this with flags passed to Portage though, haven't tried.
Another approach might be to use INSTALL_MASK to filter out the share
libraries you don't want but that may have issues too.

> > That probably explains why the ceph dependencies are as they are  
> 
> I think USE=static-libs is nice to have, and ideally (IMHO) it would
> be a global USE flag, respected by every package that installs a
> library. The flag says nothing about consumers, it only promises
> availability of the .a files, which I think is nice.

Agreed, I think this is a reasonable position to take.

-- 
James Le Cuirot (chewi)
Gentoo Linux Developer

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]