[gentoo-user] openssl upgrade may miss some needed rebuilds

[gentoo-user] openssl upgrade may miss some needed rebuilds

[walt]

Today's upgrade of openssl to 1.0.2g-r1 may cause some necessary
rebuilds to fail due to missing symbol errors.

Example:  libcurl was broken and caused the rebuilds of virtualbox and
git to fail until I forced a rebuild of curl.  Any installed package
that is actually linked against openssl will be affected by this,
notably curl or wget, which may prevent portage from fetching source
files.

I suggest using quickpkg to back up openssl before the upgrade in case
you need to recover urgently in the middle of the update.

[gentoo-user] Re: openssl upgrade may miss some needed rebuilds

[Nikos Chantziaras]

On 02/03/16 16:41, walt wrote:
> Today's upgrade of openssl to 1.0.2g-r1 may cause some necessary
> rebuilds to fail due to missing symbol errors.
>
> Example:  libcurl was broken and caused the rebuilds of virtualbox and
> git to fail until I forced a rebuild of curl.  Any installed package
> that is actually linked against openssl will be affected by this,
> notably curl or wget, which may prevent portage from fetching source
> files.

Does that mean that the library name is the same and the "preserve-libs" 
FEATURE doesn't kick in in this case?

Re: [gentoo-user] Re: openssl upgrade may miss some needed rebuilds

[Todd Goodman]

* Nikos Chantziaras <realnc@gmail.com> [160302 10:16]:
> On 02/03/16 16:41, walt wrote:
> > Today's upgrade of openssl to 1.0.2g-r1 may cause some necessary
> > rebuilds to fail due to missing symbol errors.
> >
> > Example:  libcurl was broken and caused the rebuilds of virtualbox and
> > git to fail until I forced a rebuild of curl.  Any installed package
> > that is actually linked against openssl will be affected by this,
> > notably curl or wget, which may prevent portage from fetching source
> > files.
> 
> Does that mean that the library name is the same and the "preserve-libs" 
> FEATURE doesn't kick in in this case?

It's not working for me either and I've had to manually rebuild curl and
w3m.

Todd

Re: [gentoo-user] Re: openssl upgrade may miss some needed rebuilds

[Rich Freeman]

On Wed, Mar 2, 2016 at 10:15 AM, Nikos Chantziaras <realnc@gmail.com> wrote:
> On 02/03/16 16:41, walt wrote:
>>
>> Today's upgrade of openssl to 1.0.2g-r1 may cause some necessary
>> rebuilds to fail due to missing symbol errors.
>>
>> Example:  libcurl was broken and caused the rebuilds of virtualbox and
>> git to fail until I forced a rebuild of curl.  Any installed package
>> that is actually linked against openssl will be affected by this,
>> notably curl or wget, which may prevent portage from fetching source
>> files.
>
> Does that mean that the library name is the same and the "preserve-libs"
> FEATURE doesn't kick in in this case?
>

https://forums.gentoo.org/viewtopic-p-7886940.html
https://bugs.gentoo.org/show_bug.cgi?id=576128

They changed ABI without changing SONAME, which is an absolutely
braid-dead thing for upstream to do, because it causes exactly this
kind of breakage.

revdep-rebuild is incapable of detecting this kind of breakage.  Your
linking will appear intact, but things will crash because the ABI
changed.

Everybody should be on the lookout for this update and carefully
follow the forum post instructions to get through it.

-- 
Rich

Re: [gentoo-user] Re: openssl upgrade may miss some needed rebuilds

[Alan McKinnon]

On 02/03/2016 17:49, Rich Freeman wrote:
> https://forums.gentoo.org/viewtopic-p-7886940.html
> https://bugs.gentoo.org/show_bug.cgi?id=576128
> 
> They changed ABI without changing SONAME, which is an absolutely
> braid-dead thing for upstream to do, because it causes exactly this
> kind of breakage.


brain dead is being kind to folks with non-functioning brains...

I'm now seriously considering the libressl folks might have a point.

-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] Re: openssl upgrade may miss some needed rebuilds

[Rich Freeman]

On Wed, Mar 2, 2016 at 10:54 AM, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> On 02/03/2016 17:49, Rich Freeman wrote:
>> https://forums.gentoo.org/viewtopic-p-7886940.html
>> https://bugs.gentoo.org/show_bug.cgi?id=576128
>>
>> They changed ABI without changing SONAME, which is an absolutely
>> braid-dead thing for upstream to do, because it causes exactly this
>> kind of breakage.
>
> brain dead is being kind to folks with non-functioning brains...
>
> I'm now seriously considering the libressl folks might have a point.
>

You mean the project that forked openssl, changed the APIs and ABIs,
and also kept the same SONAMEs for "compatibility?"  We can see how
well that worked with libav...

-- 
Rich

[gentoo-user] Re: openssl upgrade may miss some needed rebuilds

[James]

Rich Freeman <rich0 <at> gentoo.org> writes:


> >> Today's upgrade of openssl to 1.0.2g-r1 may cause some necessary
> >> rebuilds to fail due to missing symbol errors.

> https://forums.gentoo.org/viewtopic-p-7886940.html
> https://bugs.gentoo.org/show_bug.cgi?id=576128

> They changed ABI without changing SONAME, which is an absolutely
> braid-dead thing for upstream to do, because it causes exactly this
> kind of breakage.

Hmmmm. I've been working on my ebuild and end-o-mentoring quizes:: so in
that vein, should not the gentoo dev have bumped the gentoo rev numbers, or
did I miss-read the gentoo docs?




> Everybody should be on the lookout for this update and carefully
> follow the forum post instructions to get through it.


Again, in light of the dev-quizes, should not the package maintainer have
posted a news item prior/simultaneously to the new package release?


Not trying to stir things up, just scratching many itches here on the
dev-quizes. Surely we are all human(oid) and thus forginving of our
comrades....even to the point of encouragement?



quizfully,
James

Re: [gentoo-user] Re: openssl upgrade may miss some needed rebuilds

[Rich Freeman]

On Wed, Mar 2, 2016 at 11:06 AM, James <wireless@tampabay.rr.com> wrote:
> Rich Freeman <rich0 <at> gentoo.org> writes:
>
>> They changed ABI without changing SONAME, which is an absolutely
>> braid-dead thing for upstream to do, because it causes exactly this
>> kind of breakage.
>
> Hmmmm. I've been working on my ebuild and end-o-mentoring quizes:: so in
> that vein, should not the gentoo dev have bumped the gentoo rev numbers, or
> did I miss-read the gentoo docs?
>

So, first, this isn't really the forum to critique what the devs did,
and I haven't spoken to them so I can't vouch for what their knowledge
was at the time.

Revbumping wouldn't help, and I'm pretty sure they did revbump it.
The real issue was upstream, and I'd have to think about whether
trying to fix it with a Gentoo patch would make things better or worse
(it would make Gentoo different from everybody else, causing havoc if
you had a proprietary binary you wanted to run and so on).

Upstream really dropped the ball on this.  When I'm updating packages
I certainly don't carefully review all their ABIs and SONAMEs.
Without some kind of automatic QA tool it would be a pretty big
undertaking.  I might go see if there is such a tool though, maybe
that might be a good outcome if such a tool exists.

>
>> Everybody should be on the lookout for this update and carefully
>> follow the forum post instructions to get through it.
>
> Again, in light of the dev-quizes, should not the package maintainer have
> posted a news item prior/simultaneously to the new package release?

Sure, if they had known about it.  However, it sounds like they may
have been as surprised as anybody else.  I'd really like to see one
right away though.

The way openssl handles their ABIs really makes me think that libressl
may not be the lesser evil.  Sloppy SONAME handling causes all kinds
of issues though and seeing it in high-profile projects like these is
pretty concerning.

>
> Not trying to stir things up, just scratching many itches here on the
> dev-quizes. Surely we are all human(oid) and thus forginving of our
> comrades....even to the point of encouragement?
>

Of course.  To err is human.  To stabilize errs carries the death penalty.  :)

(I'm sure somebody will file that away for the next stable package I break.)

-- 
Rich

Re: [gentoo-user] Re: openssl upgrade may miss some needed rebuilds

[Jeremi Piotrowski]

On Wed, Mar 2, 2016 at 6:54 PM, Rich Freeman <rich0@gentoo.org> wrote:
> Upstream really dropped the ball on this.  When I'm updating packages
> I certainly don't carefully review all their ABIs and SONAMEs.
> Without some kind of automatic QA tool it would be a pretty big
> undertaking.  I might go see if there is such a tool though, maybe
> that might be a good outcome if such a tool exists.
>

I recall reading about such a tool:
http://ispras.linuxbase.org/index.php/ABI_compliance_checker
I haven't tried it out, but I would be curious to see whether it would have
caught this case.

[gentoo-user] Re: openssl upgrade may miss some needed rebuilds

[James]

Rich Freeman <rich0 <at> gentoo.org> writes:


> >> They changed ABI without changing SONAME, which is an absolutely
> >> braid-dead thing for upstream to do, because it causes exactly this
> >> kind of breakage.
> >
> > Hmmmm. I've been working on my ebuild and end-o-mentoring quizes:: so in
> > that vein, should not the gentoo dev have bumped the gentoo rev  
> > numbers, or did I miss-read the gentoo docs?
> >
> 
> So, first, this isn't really the forum to critique what the devs did,
> and I haven't spoken to them so I can't vouch for what their knowledge
> was at the time.

Excuse me, but I did not criticize anyone. I *appreciate* what the devs do;
in fact so much, I've started down that path myself. As one who has put
together dozens of ebuilds, but few published, I greatly appreciated their
work and the opportunity to learn from all mistakes, mine and the devs.
Besides, I'm not a dev, so what forum would be more appropriate to question
and learn about ebuilds and booboos? So please appreciated that thge focus
of my questions, *are to learn* with a robust discussion, as I do intend to
seek dev_status one day. Are 'users' discouraged from breaking down
package/ebuild issues in this forum? If so, which forum can I ask questions,
even the dumb ones?


> Revbumping wouldn't help, and I'm pretty sure they did revbump it.
> The real issue was upstream, and I'd have to think about whether
> trying to fix it with a Gentoo patch would make things better or worse
> (it would make Gentoo different from everybody else, causing havoc if
> you had a proprietary binary you wanted to run and so on).

One of the dev-quiz questions is about how long to leave a package in
testing, with 30 days being the minimum, unless there is critical need,
or have I not correctly understood the docs and devmanual? Again, I have no
idea how long this package was in 'testing' but, this does sound like an
excellent opportunity for fledgling devs to learn a bit deeper?  My
intentions are only based on the good for this distro, but, close
examination, at least for me, is highly warranted. 


So what commands do I run (git style) to see the history of the relevant
build/release dates for openssl? The changelog seems incomplete....


> Upstream really dropped the ball on this.  When I'm updating packages
> I certainly don't carefully review all their ABIs and SONAMEs.
> Without some kind of automatic QA tool it would be a pretty big
> undertaking.  I might go see if there is such a tool though, maybe
> that might be a good outcome if such a tool exists.

> >> Everybody should be on the lookout for this update and carefully
> >> follow the forum post instructions to get through it.  Again, in 
> >> light of the dev-quizes, should not the package maintainer have
> >> posted a news item prior/simultaneously to the new package release?

> Sure, if they had known about it.  However, it sounds like they may
> have been as surprised as anybody else.  I'd really like to see one
> right away though.


Thanks!  Good answer and now I'll have to go an edited/update my dev quiz
responses to indicate that a late news items, for something critical or that
touches so many packages, is warranted. Excellent, concrete example. One of
the things I have been working on, is supplying more details examples to the
devmanual current editor, just like this one, to reinforce the key
principles of the devmanual. I think some kind of footnotes to lots of
practical examples, is *exactly what the dev manual is missing* imho.


> The way openssl handles their ABIs really makes me think that libressl
> may not be the lesser evil.  Sloppy SONAME handling causes all kinds
> of issues though and seeing it in high-profile projects like these is
> pretty concerning.

Good to know. In fact gentoo supports such a wide variety of libs so all of
this information, in a practical example, is very valuable imho.


> > Not trying to stir things up, just scratching many itches here on the
> > dev-quizes. Surely we are all human(oid) and thus forgiving of our
> > comrades....even to the point of encouragement?

> Of course.  To err is human.  To stabilize errs carries the death  
> penalty.  :)  (I'm sure somebody will file that away for the next 
> stable package I  break.)

Easy on being so critical, either for others or yourself. I've been hacking
on ebuilds for almost a year now, and there is good reason quite a few
of mine are still not published....... Besides this is excellent evidence
for CI (Jenkins + Gerrit) ?   Are you not a proponent of CI for Gentoo?
That's a common and ordinary usage for clusters these days.....


I do appreciate the information and candor!


be at peace,
James

Re: [gentoo-user] Re: openssl upgrade may miss some needed rebuilds

[Rich Freeman]

On Wed, Mar 2, 2016 at 2:11 PM, James <wireless@tampabay.rr.com> wrote:
> Rich Freeman <rich0 <at> gentoo.org> writes:
>
> Excuse me, but I did not criticize anyone.

I know.  It was really meant to temper my remarks, since email is easy
to misconstrue.  It wasn't really directed at you, and you did get at
your intent at the end of your previous post.

>
>> Revbumping wouldn't help, and I'm pretty sure they did revbump it.
>> The real issue was upstream, and I'd have to think about whether
>> trying to fix it with a Gentoo patch would make things better or worse
>> (it would make Gentoo different from everybody else, causing havoc if
>> you had a proprietary binary you wanted to run and so on).
>
> One of the dev-quiz questions is about how long to leave a package in
> testing, with 30 days being the minimum, unless there is critical need,
> or have I not correctly understood the docs and devmanual? Again, I have no
> idea how long this package was in 'testing' but, this does sound like an
> excellent opportunity for fledgling devs to learn a bit deeper?

So far this package is only in testing.  Nobody would have run into
this issue if they weren't running ~arch.  While disruptions this
large are undesirable even in ~arch, the reality is that you're much
more likely to run into them since you are the guinea pigs.

This is actually a security issue as well, so there is going to be a
rush to get it stabilized somehow.  I'm not entirely sure how yet.
Security issues are exempt from the 30 day rule, and we don't always
backport them.

>
> So what commands do I run (git style) to see the history of the relevant
> build/release dates for openssl? The changelog seems incomplete....

Are you talking about upstream, or within Gentoo?

Within gentoo online you can just browse:
https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-libs/openssl

Hit log next to any file you're interested in, or go up a directory
and hit log next to the openssl directory itself to see everything
including file deletions/etc.

Or with git you can run:
git clone git://anongit.gentoo.org/repo/gentoo.git
cd gentoo/dev-libs/openssl
git log .

>
>> The way openssl handles their ABIs really makes me think that libressl
>> may not be the lesser evil.  Sloppy SONAME handling causes all kinds
>> of issues though and seeing it in high-profile projects like these is
>> pretty concerning.
>
> Good to know. In fact gentoo supports such a wide variety of libs so all of
> this information, in a practical example, is very valuable imho.

There are pros and cons to it, but I wouldn't be here if I didn't
think that letting the users pick the winner between openssl/libressl
wasn't a good thing.  Initially I was pushing back on adding libressl
to the tree a bit just to see if we could come up with a better way to
do it in light of the mess we ran into with libav.  In the end we
couldn't come up with anything so it moved forward.

> Easy on being so critical, either for others or yourself.

I was just joking with that, hence the point about somebody bringing
it up when I inevitably make a mistake.


> Besides this is excellent evidence
> for CI (Jenkins + Gerrit) ?   Are you not a proponent of CI for Gentoo?

I'm definitely a proponent.  It can be a bit problematic resource-wise
and with latency.  However, I should really get into the habit of
trying to do commits via pull-requests that hit our CI system.

-- 
Rich

Re: [gentoo-user] Re: openssl upgrade may miss some needed rebuilds

[Adam Carter]

[-- Attachment #1: Type: text/plain, Size: 392 bytes --]

FYI for anyone concerned about this latest issue "DROWN" - its only a
problem if SSLv2 is enabled. SSLv2 has been broken for a long time, so
should be disabled. However, if it is exposed then an attacker can retrieve
the private key, and in doing so will be able to also decrypt secure TLS
1.2+ sessions to any server using that private key.

https://www.openssl.org/news/secadv/20160301.txt

[-- Attachment #2: Type: text/html, Size: 501 bytes --]

[gentoo-user] Re: openssl upgrade may miss some needed rebuilds

[»Q«]

On Wed, 2 Mar 2016 10:49:59 -0500
Rich Freeman <rich0@gentoo.org> wrote:

> https://forums.gentoo.org/viewtopic-p-7886940.html
> https://bugs.gentoo.org/show_bug.cgi?id=576128

I had wget with USE="gnutls" already, so I took the plunge yesterday
and followed PolynomialC's instructions at the first link above.

When I used

revdep-rebuild.sh -i -L "libssl\.so.*" -- --exclude=openssl --keep-going

the only package that failed to rebuild was
www-client/w3mmee-0.3.2_p24-r7, and that failure is due to
<https://bugs.gentoo.org/show_bug.cgi?id=556928>, nothing to do with
openssl.

Re: [gentoo-user] Re: openssl upgrade may miss some needed rebuilds

[Håkon Alstadheim]

Den 02. mars 2016 19:19, skrev »Q«:
> On Wed, 2 Mar 2016 10:49:59 -0500
> Rich Freeman <rich0@gentoo.org> wrote:
>
>> https://forums.gentoo.org/viewtopic-p-7886940.html
>> https://bugs.gentoo.org/show_bug.cgi?id=576128
> I had wget with USE="gnutls" already, so I took the plunge yesterday
> and followed PolynomialC's instructions at the first link above.
>
> When I used
>
> revdep-rebuild.sh -i -L "libssl\.so.*" -- --exclude=openssl --keep-going
>
> the only package that failed to rebuild was
> www-client/w3mmee-0.3.2_p24-r7, and that failure is due to
> <https://bugs.gentoo.org/show_bug.cgi?id=556928>, nothing to do with
> openssl.
>
>
>
Would "revdep-rebuild.sh -i -L "libssl\.so.*" -- -f" before emerging, be
sufficient ? I.e. that should obviate the need for compiling wget with
gnutls ?

Re: [gentoo-user] Re: openssl upgrade may miss some needed rebuilds

[Rich Freeman]

On Thu, Mar 3, 2016 at 3:15 AM, Håkon Alstadheim
<hakon@alstadheim.priv.no> wrote:
> Would "revdep-rebuild.sh -i -L "libssl\.so.*" -- -f" before emerging, be
> sufficient ? I.e. that should obviate the need for compiling wget with
> gnutls ?
>

No, and no.  The problem is the ABI is silently changing.  Rebuilding
everything before updating openssl will just recompile everything with
the old ABI, and then when it silently changes everything will still
break.  The purpose of building wget with gnutls is to make it no
longer use openssl, so then it doesn't break at all.

-- 
Rich

Re: [gentoo-user] Re: openssl upgrade may miss some needed rebuilds

[Håkon Alstadheim]

On 03. mars 2016 12:26, Rich Freeman wrote:
> On Thu, Mar 3, 2016 at 3:15 AM, Håkon Alstadheim
> <hakon@alstadheim.priv.no> wrote:
>> Would "revdep-rebuild.sh -i -L "libssl\.so.*" -- -f" before emerging, be
>> sufficient ? I.e. that should obviate the need for compiling wget with
>> gnutls ?
>>
> No, and no.  The problem is the ABI is silently changing.  Rebuilding
> everything 
Not rebuilding, "-f" is supposed to mean "fetchonly" , i.e. all wget
would get run first, and then a second run of emerge would do the actual
compilation with the distfiles already on local disk, That is if I
understadnd "-f" correctly, and given enough disk-space and no sneaky
purging distfiles between runs.

> before updating openssl will just recompile everything with
> the old ABI, and then when it silently changes everything will still
> break.  The purpose of building wget with gnutls is to make it no
> longer use openssl, so then it doesn't break at all.
>

Re: [gentoo-user] Re: openssl upgrade may miss some needed rebuilds

[Rich Freeman]

On Thu, Mar 3, 2016 at 8:05 AM, Håkon Alstadheim
<hakon@alstadheim.priv.no> wrote:
> On 03. mars 2016 12:26, Rich Freeman wrote:
>> On Thu, Mar 3, 2016 at 3:15 AM, Håkon Alstadheim
>> <hakon@alstadheim.priv.no> wrote:
>>> Would "revdep-rebuild.sh -i -L "libssl\.so.*" -- -f" before emerging, be
>>> sufficient ? I.e. that should obviate the need for compiling wget with
>>> gnutls ?
>>>
>> No, and no.  The problem is the ABI is silently changing.  Rebuilding
>> everything
> Not rebuilding, "-f" is supposed to mean "fetchonly" , i.e. all wget
> would get run first, and then a second run of emerge would do the actual
> compilation with the distfiles already on local disk, That is if I
> understadnd "-f" correctly, and given enough disk-space and no sneaky
> purging distfiles between runs.
>

Ah, missed that.  Yes, this would likely eliminate the need to rebuild
wget.  It would still require you to rebuild everything else
afterwards.  And, if that fails there is another workaround for
fetching in the forum.

-- 
Rich