[gentoo-dev] doubtful about libjpeg-turbo vs. libjpeg binary compatibility

[gentoo-dev] doubtful about libjpeg-turbo vs. libjpeg binary compatibility

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 2630 bytes --]

While dealing with <https://bugs.gentoo.org/show_bug.cgi?id=393471> I
started discussing with developers working on libjpeg-turbo support in
WebKit, and I learned that despite
<http://archives.gentoo.org/gentoo-dev/msg_e67bedf25dd178ec09a325a1220724e6.xml>
libjpeg-turbo is not necessarily binary compatible with libjpeg, and
even with different versions of itself.

Here's why. See
<http://libjpeg-turbo.svn.sourceforge.net/viewvc/libjpeg-turbo/trunk/libjpeg.txt?revision=299&view=markup>
and search for "as a shared library". I'll paste the relevant quote here
anyway:

> While you can build the JPEG library as a shared library if the whim strikes
> you, we don't really recommend it.  The trouble with shared libraries is that
> at some point you'll probably try to substitute a new version of the library
> without recompiling the calling applications.  That generally doesn't work
> because the parameter struct declarations usually change with each new
> version.  In other words, the library's API is *not* guaranteed binary
> compatible across versions; we only try to ensure source-code compatibility.

The particular problem with www-client/chromium is caused by
<http://trac.webkit.org/changeset/101286/trunk/Source/WebCore/platform/image-decoders/jpeg/JPEGImageDecoder.cpp>
which sort of "hardcodes" in the compiled binary whether it was compiled
against libjpeg-turbo with swizzle support (whatever that is) or not.

The real problem here is that with above "no guarantee" of binary
compatibility such a solution may be considered legitimate, especially
that for e.g. Google Chrome the bundled copy of libjpeg-turbo is always
used.

What do you guys think we should do with this on the Gentoo side? At
this moment I've just reverted the mentioned change in
www-client/chromium ebuild, but it's not feasible to keep a local patch
too long (it needs rebasing too often).

I was thinking about changing SONAMES, which would trigger rebuilds make
things work, but then don't we rely on specific libjpeg SONAMES for
binary-only software to work? Or does such binary-only software just use
libjpeg-6b?

Are there some other solutions we could apply on the Gentoo side? The
main point here is that Chromium upstream considers
<http://trac.webkit.org/changeset/101286/trunk/Source/WebCore/platform/image-decoders/jpeg/JPEGImageDecoder.cpp>
a legitimate change, and based on the referenced
<http://libjpeg-turbo.svn.sourceforge.net/viewvc/libjpeg-turbo/trunk/libjpeg.txt?revision=299&view=markup>
disclaimer about no guarantee of binary compatibility, I think it makes
sense.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: [gentoo-dev] doubtful about libjpeg-turbo vs. libjpeg binary compatibility

[Samuli Suominen]

On 01/19/2012 11:19 AM, "Paweł Hajdan, Jr." wrote:
> While dealing with<https://bugs.gentoo.org/show_bug.cgi?id=393471>  I
> started discussing with developers working on libjpeg-turbo support in
> WebKit, and I learned that despite
> <http://archives.gentoo.org/gentoo-dev/msg_e67bedf25dd178ec09a325a1220724e6.xml>
> libjpeg-turbo is not necessarily binary compatible with libjpeg, and
> even with different versions of itself.
>
> Here's why. See
> <http://libjpeg-turbo.svn.sourceforge.net/viewvc/libjpeg-turbo/trunk/libjpeg.txt?revision=299&view=markup>
> and search for "as a shared library". I'll paste the relevant quote here
> anyway:
>
>> While you can build the JPEG library as a shared library if the whim strikes
>> you, we don't really recommend it.  The trouble with shared libraries is that
>> at some point you'll probably try to substitute a new version of the library
>> without recompiling the calling applications.  That generally doesn't work
>> because the parameter struct declarations usually change with each new
>> version.  In other words, the library's API is *not* guaranteed binary
>> compatible across versions; we only try to ensure source-code compatibility.
>
> The particular problem with www-client/chromium is caused by
> <http://trac.webkit.org/changeset/101286/trunk/Source/WebCore/platform/image-decoders/jpeg/JPEGImageDecoder.cpp>
> which sort of "hardcodes" in the compiled binary whether it was compiled
> against libjpeg-turbo with swizzle support (whatever that is) or not.
>
> The real problem here is that with above "no guarantee" of binary
> compatibility such a solution may be considered legitimate, especially
> that for e.g. Google Chrome the bundled copy of libjpeg-turbo is always
> used.
>
> What do you guys think we should do with this on the Gentoo side?At

always use system libraries and i'm in process of dropping keywording 
from media-libs/jpeg wrt[1] since we have no need for source slot of it

[1] http://bugs.gentoo.org/398909

both providers will be left in the virtual/jpeg, but only libjpeg-turbo 
will be keyworded (and thus supported), eliminating rest of the issues 
raised here

- Samuli

> this moment I've just reverted the mentioned change in
> www-client/chromium ebuild, but it's not feasible to keep a local patch
> too long (it needs rebasing too often).
>
> I was thinking about changing SONAMES, which would trigger rebuilds make
> things work, but then don't we rely on specific libjpeg SONAMES for
> binary-only software to work? Or does such binary-only software just use
> libjpeg-6b?
>
> Are there some other solutions we could apply on the Gentoo side? The
> main point here is that Chromium upstream considers
> <http://trac.webkit.org/changeset/101286/trunk/Source/WebCore/platform/image-decoders/jpeg/JPEGImageDecoder.cpp>
> a legitimate change, and based on the referenced
> <http://libjpeg-turbo.svn.sourceforge.net/viewvc/libjpeg-turbo/trunk/libjpeg.txt?revision=299&view=markup>
> disclaimer about no guarantee of binary compatibility, I think it makes
> sense.
>

Re: [gentoo-dev] doubtful about libjpeg-turbo vs. libjpeg binary compatibility

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2965 bytes --]

On Thu, 19 Jan 2012 10:19:27 +0100
""Paweł Hajdan, Jr."" <phajdan.jr@gentoo.org> wrote:

> While dealing with <https://bugs.gentoo.org/show_bug.cgi?id=393471> I
> started discussing with developers working on libjpeg-turbo support in
> WebKit, and I learned that despite
> <http://archives.gentoo.org/gentoo-dev/msg_e67bedf25dd178ec09a325a1220724e6.xml>
> libjpeg-turbo is not necessarily binary compatible with libjpeg, and
> even with different versions of itself.
> 
> Here's why. See
> <http://libjpeg-turbo.svn.sourceforge.net/viewvc/libjpeg-turbo/trunk/libjpeg.txt?revision=299&view=markup>
> and search for "as a shared library". I'll paste the relevant quote
> here anyway:
> 
> > While you can build the JPEG library as a shared library if the
> > whim strikes you, we don't really recommend it.  The trouble with
> > shared libraries is that at some point you'll probably try to
> > substitute a new version of the library without recompiling the
> > calling applications.  That generally doesn't work because the
> > parameter struct declarations usually change with each new
> > version.  In other words, the library's API is *not* guaranteed
> > binary compatible across versions; we only try to ensure
> > source-code compatibility.
> 
> The particular problem with www-client/chromium is caused by
> <http://trac.webkit.org/changeset/101286/trunk/Source/WebCore/platform/image-decoders/jpeg/JPEGImageDecoder.cpp>
> which sort of "hardcodes" in the compiled binary whether it was
> compiled against libjpeg-turbo with swizzle support (whatever that
> is) or not.
> 
> The real problem here is that with above "no guarantee" of binary
> compatibility such a solution may be considered legitimate, especially
> that for e.g. Google Chrome the bundled copy of libjpeg-turbo is
> always used.
> 
> What do you guys think we should do with this on the Gentoo side? At
> this moment I've just reverted the mentioned change in
> www-client/chromium ebuild, but it's not feasible to keep a local
> patch too long (it needs rebasing too often).
> 
> I was thinking about changing SONAMES, which would trigger rebuilds
> make things work, but then don't we rely on specific libjpeg SONAMES
> for binary-only software to work? Or does such binary-only software
> just use libjpeg-6b?

Hmm, does this mean the ABI differs on runtime compilation options?
SONAME should be changed with new versions, not options. If upstream
does allow things like that, that's bad. If chromium uses that, it's
even worse.

I'd go with the simplest solution which is either enabling
the particular configure option unconditionally or disabling it.
If possible -- synced with SONAME change. But it should be upstream
SONAME change (considering they do change SONAMEs when releasing
binary-incompatible versions); we don't really want to go the Debian
way, keeping our own SONAME cycles.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] doubtful about libjpeg-turbo vs. libjpeg binary compatibility

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 1505 bytes --]

On 1/19/12 10:45 AM, Michał Górny wrote:
> Hmm, does this mean the ABI differs on runtime compilation options?

No, at least I'm not aware of such a thing.

I'd sum it up as "libjpeg-turbo is not binary-compatible with libjpeg
and also with a different version of itself, and is not supposed to be".

> SONAME should be changed with new versions, not options. If upstream 
> does allow things like that, that's bad.

I think upstream keeps the SONAME at "libjpeg.so.8" . And as indicated
by <https://bugs.gentoo.org/show_bug.cgi?id=393471#c3> and the next
comment, libjpeg-turbo-1.1.1 and libjpeg-1.1.90 shouldn't have the same
SONAME (they're not binary compatible wrt JCS extensions Chromium uses).

> If chromium uses that, it's even worse.

That's what I initially thought, but the "as a shared library" paragraph
I quoted from
<http://libjpeg-turbo.svn.sourceforge.net/viewvc/libjpeg-turbo/trunk/libjpeg.txt?revision=299&view=markup>
strongly suggests it's legitimate to do what they've done.

> But it should be upstream SONAME change (considering they do change
> SONAMEs when releasing binary-incompatible versions); we don't really
> want to go the Debian way, keeping our own SONAME cycles.

I think OpenBSD also has its own SONAME cycles, which is not necessarily
too bad (except for binary-only software). In Gentoo, we also have our
own SONAME for V8 JavaScript engine (upstream only provides
infrastructure for setting SONAME, i.e. a build system switch).


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: [gentoo-dev] doubtful about libjpeg-turbo vs. libjpeg binary compatibility

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 2658 bytes --]

On Thursday 19 January 2012 04:35:43 Samuli Suominen wrote:
> On 01/19/2012 11:19 AM, "Paweł Hajdan, Jr." wrote:
> > While dealing with<https://bugs.gentoo.org/show_bug.cgi?id=393471>  I
> > started discussing with developers working on libjpeg-turbo support in
> > WebKit, and I learned that despite
> > <http://archives.gentoo.org/gentoo-dev/msg_e67bedf25dd178ec09a325a1220724
> > e6.xml> libjpeg-turbo is not necessarily binary compatible with libjpeg,
> > and even with different versions of itself.
> > 
> > Here's why. See
> > <http://libjpeg-turbo.svn.sourceforge.net/viewvc/libjpeg-turbo/trunk/libj
> > peg.txt?revision=299&view=markup> and search for "as a shared library".
> > I'll paste the relevant quote here
> > 
> > anyway:
> >> While you can build the JPEG library as a shared library if the whim
> >> strikes you, we don't really recommend it.  The trouble with shared
> >> libraries is that at some point you'll probably try to substitute a new
> >> version of the library without recompiling the calling applications. 
> >> That generally doesn't work because the parameter struct declarations
> >> usually change with each new version.  In other words, the library's
> >> API is *not* guaranteed binary compatible across versions; we only try
> >> to ensure source-code compatibility.
> > 
> > The particular problem with www-client/chromium is caused by
> > <http://trac.webkit.org/changeset/101286/trunk/Source/WebCore/platform/im
> > age-decoders/jpeg/JPEGImageDecoder.cpp> which sort of "hardcodes" in the
> > compiled binary whether it was compiled against libjpeg-turbo with
> > swizzle support (whatever that is) or not.
> > 
> > The real problem here is that with above "no guarantee" of binary
> > compatibility such a solution may be considered legitimate, especially
> > that for e.g. Google Chrome the bundled copy of libjpeg-turbo is always
> > used.
> > 
> > What do you guys think we should do with this on the Gentoo side?At
> 
> always use system libraries

that doesn't help.  the libjpeg turbo peeps themselves have said they don't 
guarantee compatibility across their own versions.

> and i'm in process of dropping keywording
> from media-libs/jpeg wrt[1] since we have no need for source slot of it

err, no.  libjpeg-turbo doesn't provide the older SONAME's like jpeg does.  
those SLOT's aren't going anywhere (SLOT!=0).

history has shown that the canonical version stays around while the 
derivatives come and go.  so until the seemingly braindead ABI policies of 
libjpeg-turbo can get sorted out, i don't think we can drop KEYWORDS on SLOT=0 
jpeg.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] doubtful about libjpeg-turbo vs. libjpeg binary compatibility

[Samuli Suominen]

On 01/19/2012 06:56 PM, Mike Frysinger wrote:
> On Thursday 19 January 2012 04:35:43 Samuli Suominen wrote:
>> On 01/19/2012 11:19 AM, "Paweł Hajdan, Jr." wrote:
>>> While dealing with<https://bugs.gentoo.org/show_bug.cgi?id=393471>   I
>>> started discussing with developers working on libjpeg-turbo support in
>>> WebKit, and I learned that despite
>>> <http://archives.gentoo.org/gentoo-dev/msg_e67bedf25dd178ec09a325a1220724
>>> e6.xml>  libjpeg-turbo is not necessarily binary compatible with libjpeg,
>>> and even with different versions of itself.
>>>
>>> Here's why. See
>>> <http://libjpeg-turbo.svn.sourceforge.net/viewvc/libjpeg-turbo/trunk/libj
>>> peg.txt?revision=299&view=markup>  and search for "as a shared library".
>>> I'll paste the relevant quote here
>>>
>>> anyway:
>>>> While you can build the JPEG library as a shared library if the whim
>>>> strikes you, we don't really recommend it.  The trouble with shared
>>>> libraries is that at some point you'll probably try to substitute a new
>>>> version of the library without recompiling the calling applications.
>>>> That generally doesn't work because the parameter struct declarations
>>>> usually change with each new version.  In other words, the library's
>>>> API is *not* guaranteed binary compatible across versions; we only try
>>>> to ensure source-code compatibility.
>>>
>>> The particular problem with www-client/chromium is caused by
>>> <http://trac.webkit.org/changeset/101286/trunk/Source/WebCore/platform/im
>>> age-decoders/jpeg/JPEGImageDecoder.cpp>  which sort of "hardcodes" in the
>>> compiled binary whether it was compiled against libjpeg-turbo with
>>> swizzle support (whatever that is) or not.
>>>
>>> The real problem here is that with above "no guarantee" of binary
>>> compatibility such a solution may be considered legitimate, especially
>>> that for e.g. Google Chrome the bundled copy of libjpeg-turbo is always
>>> used.
>>>
>>> What do you guys think we should do with this on the Gentoo side?At
>>
>> always use system libraries
>
> that doesn't help.  the libjpeg turbo peeps themselves have said they don't
> guarantee compatibility across their own versions.

it's forward compatible, which is all we should care about

>> and i'm in process of dropping keywording
>> from media-libs/jpeg wrt[1] since we have no need for source slot of it
>
> err, no.  libjpeg-turbo doesn't provide the older SONAME's like jpeg does.
> those SLOT's aren't going anywhere (SLOT!=0).

i said "source slot" which was supposed to mean SLOT="0" alone, the 
SLOT="62" will still have keywording, SLOT="7" is not used anything in 
tree but can still have the keywording (unintresting to this topic)

> history has shown that the canonical version stays around while the
> derivatives come and go.

and the ones before never had this level of people behind it, and 
projects where people get paid to get it working, like fedora

> so until the seemingly braindead ABI policies of
> libjpeg-turbo can get sorted out, i don't think we can drop KEYWORDS on SLOT=0
> jpeg.

the only thing that's really broken is building against libjpeg-turbo, 
and then switching to ijg's jpeg without rebuilding the package

and downgrading of libjpeg-turbo

both of which are not worth of supporting

Re: [gentoo-dev] doubtful about libjpeg-turbo vs. libjpeg binary compatibility

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 1335 bytes --]

On 1/19/12 6:02 PM, Samuli Suominen wrote:
> On 01/19/2012 06:56 PM, Mike Frysinger wrote:
>> that doesn't help.  the libjpeg turbo peeps themselves have said they
>> don't
>> guarantee compatibility across their own versions.
> 
> it's forward compatible, which is all we should care about

Just a note: that'd require me to DEPEND on a recent enough version of
libjpeg-turbo in the www-client/chromium ebuild, which would mean either:

a) changing the virtual/jpeg dependency to >=libjpeg-turbo-...

b) adding a versioned virtual/jpeg to require a recent enough
libjpeg-turbo (but then what about libjpeg versions - it can easily
become complicated)

c) similar to a) but also adding || ( >=libjpeg-turbo-... libjpeg )

With proper SONAMEs in libjpeg-turbo that would be a non-issue (bump the
SONAME on incompatible change, revdep-rebuild does the rest).

> the only thing that's really broken is building against libjpeg-turbo,
> and then switching to ijg's jpeg without rebuilding the package

I'm fine with not supporting that, provided keywords for libjpeg are
dropped (except the 62 slot iirc).

> and downgrading of libjpeg-turbo

I think this one should "just work", or at least not allow obvious
mistakes. See my a) b) c) notes above in this e-mail for possible
solutions and ideal SONAME.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]

Re: [gentoo-dev] doubtful about libjpeg-turbo vs. libjpeg binary compatibility

[Samuli Suominen]

On 01/19/2012 07:16 PM, "Paweł Hajdan, Jr." wrote:
> On 1/19/12 6:02 PM, Samuli Suominen wrote:
>> On 01/19/2012 06:56 PM, Mike Frysinger wrote:
>>> that doesn't help.  the libjpeg turbo peeps themselves have said they
>>> don't
>>> guarantee compatibility across their own versions.
>>
>> it's forward compatible, which is all we should care about
>
> Just a note: that'd require me to DEPEND on a recent enough version of
> libjpeg-turbo in the www-client/chromium ebuild, which would mean either:
>
> a) changing the virtual/jpeg dependency to>=libjpeg-turbo-...

will be done soon as 1.2.0 is released and stabilized, i'd like to skip 
1.1.90

>> and downgrading of libjpeg-turbo
>
> I think this one should "just work", or at least not allow obvious
> mistakes. See my a) b) c) notes above in this e-mail for possible
> solutions and ideal SONAME.
>

a) is fine, preventing any downgrades.   a fatal check, like glibc and 
qt4 has to prevent downgrading is an option too, but a bit overkill imho

Re: [gentoo-dev] doubtful about libjpeg-turbo vs. libjpeg binary compatibility

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 547 bytes --]

On 1/19/12 6:42 PM, Samuli Suominen wrote:
>> a) changing the virtual/jpeg dependency to>=libjpeg-turbo-...
> 
> will be done soon as 1.2.0 is released and stabilized, i'd like to skip
> 1.1.90

Sounds good to me.

> a) is fine, preventing any downgrades.   a fatal check, like glibc and
> qt4 has to prevent downgrading is an option too, but a bit overkill imho

Agreed, in my opinion such a check would be a bad idea. People might
have reasons to temporarily downgrade libjpeg-turbo.

An elog notice could be worthwhile though.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 203 bytes --]