[gentoo-dev] manifest-hashes changing to 'BLAKE2B SHA512' on 2017-11-21

[gentoo-dev] manifest-hashes changing to 'BLAKE2B SHA512' on 2017-11-21

[Michał Górny]

Hi, everyone.

The Council has approved the manifest-hashes switch on 2017-11-12
meeting [1]. The transition will occur to the initial plan, with small
changes. The updated plan is included at the end of this mail.

According to this plan, BLAKE2B will be enabled on 2017-11-21. This
means that starting at this time, all new and updated DIST entries will
use BLAKE2B+SHA512. Old DIST entries will still use the current hash set
until updated.

The developers are required to upgrade to a package manager supporting
this hash. That is:

a. Portage 2.3.5 when using py3.6+,

b. Portage 2.3.13 + pyblake2 installed manually,

c. Portage 2.3.13-r1 that includes the pyblake2 dep.

Modern (and old) Portage will refuse to update Manifests if it does not
support the necessary hashes. However, Portage versions between 2.3.5
and 2.3.13 inclusively will create Manifests missing BLAKE2B hash rather
than failing when no hash provider is present. Those Manifests will be
rejected by the git hook.

Users will not be affected noticeably as the SHA512 hash continues being
used for compatibility.


That said, I'd like to request developers not to start proactively
converting all old Manifest entries to the new set immediately,
and instead give some time for things to settle down.



The updated plan
================

Already done:

- revbumped Portage with pyblake2 dep and started stabilizing it,

- added git update hook to reject invalid Manifest entries.

2017-11-21 (T+7d):

- manifest-hashes = BLAKE2B SHA512

2018-02-14 (T+3m):

- manifest-required-hashes = BLAKE2B

2018-05-14 (T+6m):

- last rite fetch-restricted packages that do not use BLAKE2B.

The final removal of SHA512 will be decided by the Council separately.


-- 
Best regards,
Michał Górny

Re: [gentoo-dev] manifest-hashes changing to 'BLAKE2B SHA512' on 2017-11-21

[R0b0t1]

[-- Attachment #1: Type: text/plain, Size: 2121 bytes --]

On Wednesday, November 15, 2017, Michał Górny <mgorny@gentoo.org> wrote:
> Hi, everyone.
>
> The Council has approved the manifest-hashes switch on 2017-11-12
> meeting [1]. The transition will occur to the initial plan, with small
> changes. The updated plan is included at the end of this mail.
>
> According to this plan, BLAKE2B will be enabled on 2017-11-21. This
> means that starting at this time, all new and updated DIST entries will
> use BLAKE2B+SHA512. Old DIST entries will still use the current hash set
> until updated.
>
> The developers are required to upgrade to a package manager supporting
> this hash. That is:
>
> a. Portage 2.3.5 when using py3.6+,
>
> b. Portage 2.3.13 + pyblake2 installed manually,
>
> c. Portage 2.3.13-r1 that includes the pyblake2 dep.
>
> Modern (and old) Portage will refuse to update Manifests if it does not
> support the necessary hashes. However, Portage versions between 2.3.5
> and 2.3.13 inclusively will create Manifests missing BLAKE2B hash rather
> than failing when no hash provider is present. Those Manifests will be
> rejected by the git hook.
>
> Users will not be affected noticeably as the SHA512 hash continues being
> used for compatibility.
>
>
> That said, I'd like to request developers not to start proactively
> converting all old Manifest entries to the new set immediately,
> and instead give some time for things to settle down.
>
>
>
> The updated plan
> ================
>
> Already done:
>
> - revbumped Portage with pyblake2 dep and started stabilizing it,
>
> - added git update hook to reject invalid Manifest entries.
>
> 2017-11-21 (T+7d):
>
> - manifest-hashes = BLAKE2B SHA512
>
> 2018-02-14 (T+3m):
>
> - manifest-required-hashes = BLAKE2B
>
> 2018-05-14 (T+6m):
>
> - last rite fetch-restricted packages that do not use BLAKE2B.
>
> The final removal of SHA512 will be decided by the Council separately.
>

Does the existence of a decision mean I would need to contact the trustees
if I feel the changes have not been adequately justified?

Respectfully,
    R0b0t1

[-- Attachment #2: Type: text/html, Size: 2477 bytes --]

Re: [gentoo-dev] manifest-hashes changing to 'BLAKE2B SHA512' on 2017-11-21

[NP-Hardass]

[-- Attachment #1.1: Type: text/plain, Size: 2643 bytes --]

On 11/15/2017 12:47 PM, R0b0t1 wrote:
> On Wednesday, November 15, 2017, Michał Górny <mgorny@gentoo.org
> <mailto:mgorny@gentoo.org>> wrote:
>> Hi, everyone.
>>
>> The Council has approved the manifest-hashes switch on 2017-11-12
>> meeting [1]. The transition will occur to the initial plan, with small
>> changes. The updated plan is included at the end of this mail.
>>
>> According to this plan, BLAKE2B will be enabled on 2017-11-21. This
>> means that starting at this time, all new and updated DIST entries will
>> use BLAKE2B+SHA512. Old DIST entries will still use the current hash set
>> until updated.
>>
>> The developers are required to upgrade to a package manager supporting
>> this hash. That is:
>>
>> a. Portage 2.3.5 when using py3.6+,
>>
>> b. Portage 2.3.13 + pyblake2 installed manually,
>>
>> c. Portage 2.3.13-r1 that includes the pyblake2 dep.
>>
>> Modern (and old) Portage will refuse to update Manifests if it does not
>> support the necessary hashes. However, Portage versions between 2.3.5
>> and 2.3.13 inclusively will create Manifests missing BLAKE2B hash rather
>> than failing when no hash provider is present. Those Manifests will be
>> rejected by the git hook.
>>
>> Users will not be affected noticeably as the SHA512 hash continues being
>> used for compatibility.
>>
>>
>> That said, I'd like to request developers not to start proactively
>> converting all old Manifest entries to the new set immediately,
>> and instead give some time for things to settle down.
>>
>>
>>
>> The updated plan
>> ================
>>
>> Already done:
>>
>> - revbumped Portage with pyblake2 dep and started stabilizing it,
>>
>> - added git update hook to reject invalid Manifest entries.
>>
>> 2017-11-21 (T+7d):
>>
>> - manifest-hashes = BLAKE2B SHA512
>>
>> 2018-02-14 (T+3m):
>>
>> - manifest-required-hashes = BLAKE2B
>>
>> 2018-05-14 (T+6m):
>>
>> - last rite fetch-restricted packages that do not use BLAKE2B.
>>
>> The final removal of SHA512 will be decided by the Council separately.
>>
> 
> Does the existence of a decision mean I would need to contact the
> trustees if I feel the changes have not been adequately justified?
> 
> Respectfully,
>     R0b0t1

No, if you think there is an issue with the Council decision, you should
speak with the Council.  Moreover... The Council is responsible for
technical decisions within Gentoo.  Unless it violates the Social
Contract, I cannot see how the Trustees should be involved here.  They
have empowered the Council to make technical decisions as they see fit.

-- 
NP-Hardass


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] manifest-hashes changing to 'BLAKE2B SHA512' on 2017-11-21

[William L. Thomson Jr.]

[-- Attachment #1: Type: text/plain, Size: 1784 bytes --]

On Wed, 15 Nov 2017 14:21:59 -0500
NP-Hardass <NP-Hardass@gentoo.org> wrote:
>
> 
> No, if you think there is an issue with the Council decision, you
> should speak with the Council.  Moreover... The Council is
> responsible for technical decisions within Gentoo.  Unless it
> violates the Social Contract, I cannot see how the Trustees should be
> involved here.  They have empowered the Council to make technical
> decisions as they see fit.

There is nothing empowering the Council from the Trustees. Legally the
Trustees could have final say. The Council is not a legal body nor
formal in any legal filing. They have no say when it comes down to it
from a legal perspective.

However everything is structured such that the Council does have final
say over all technical matters. That is something the community
did, and adheres to. There is nothing, to my recollection, in Foundation
By Laws or other that would connect the two. The council has no power
over Trustees, and Trustees do have legal power over all of Gentoo.

The Trustees just abstain from technical matters, as that is the
structure the community has accepted. I am not aware of anything else
with such structure.

Nothing legally gives Council final say technically. That is just how
things are, and likely will always remain as such. Which is some what
strange as Trustees can be responsible for the actions of others.
Despite having no say in such matters till it becomes a legal issue.

IMHO I rather see a structure where Council is more like CTO. Still top
for technical, but falls under Trustees for oversight, final say, etc.
A normal structure like exists in most any business globally. They
should work together more as one vs two bodies.

-- 
William L. Thomson Jr.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] manifest-hashes changing to 'BLAKE2B SHA512' on 2017-11-21

[Rich Freeman]

On Wed, Nov 15, 2017 at 3:21 PM, William L. Thomson Jr.
<wlt-ml@o-sinc.com> wrote:
>
> The council has no power
> over Trustees, and Trustees do have legal power over all of Gentoo.

Sure, just keep in mind that legally Gentoo is basically nothing but a
name and a logo.

The Trustees could ask the community to stop using the name and logo,
but in practice that would be about all they could do, in a
hypothetical world where the Trustees and the Community were in some
kind of conflict (which is unlikely in practice).

The actual distro is largely community-run.

Gentoo isn't like Redhat.  It isn't a legal entity that happens to
have a community.  It is more of a community that happens to have a
legal entity.  Since the Foundation doesn't actually have any
employees it is actually fairly limited in its ability to make things
happen in the real world.  With its funding it is actually fairly
limited in making things happen legally as well.  Its main function at
this time (IMO) is to protect the name so that somebody else who has
more money doesn't tell us to stop using it, and to pay for moderate
expenses.

> Which is some what strange as Trustees can be responsible for the actions of others. Despite having no say in such matters till it becomes a legal issue.

As long as the Trustees aren't violating any laws or blatantly
ignoring their violation I doubt they'd have much personal liability
to outsiders.  The Foundation has more liability, but really the main
risk there is losing whatever donations are sitting around unspent and
maybe having somebody tell us we can't call ourselves Gentoo.  The
practical independence of the community from the Foundation works both
ways.  Having to change our name would be inconvenient, but it is
still an unlikely outcome as we do try to be reasonably careful.

-- 
Rich

Re: [gentoo-dev] manifest-hashes changing to 'BLAKE2B SHA512' on 2017-11-21

[William L. Thomson Jr.]

[-- Attachment #1: Type: text/plain, Size: 1401 bytes --]

On Wed, 15 Nov 2017 16:15:18 -0500
Rich Freeman <rich0@gentoo.org> wrote:

> On Wed, Nov 15, 2017 at 3:21 PM, William L. Thomson Jr.
> <wlt-ml@o-sinc.com> wrote:
> >
> > The council has no power
> > over Trustees, and Trustees do have legal power over all of
> > Gentoo.  
> 
> Sure, just keep in mind that legally Gentoo is basically nothing but a
> name and a logo.

That is what others have made it to be. It was never intended to be
such by the person who created the foundation in the first place.
Paying for those legal filings and attorney fees out of pocket! The
only time any proper IRS filings was done for Gentoo....
I have showed in -project what it was supposed to be.
https://archives.gentoo.org/gentoo-project/message/3ac5418dd061fc53f4b8d55a99773f4c

Like FreeBSD Foundation, a real organization paying for development
via grants and other things...
https://www.freebsdfoundation.org/
https://www.freebsdfoundation.org/what-we-do/grants/

https://www.freebsdfoundation.org/wp-content/uploads/2015/12/2017-Q2-Profit-Loss.pdf

Presently working on raising $1.25 Million USD for next year.
Look at the donors on their home page. Intel, Netapp, Versign, Netflix
even Microsoft...

Gentoo is missing out big time! But it is the communities choice. If
Gentoo is just a name and logo, that is pretty sad....
Limited vision....

-- 
William L. Thomson Jr.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] manifest-hashes changing to 'BLAKE2B SHA512' on 2017-11-21

[William L. Thomson Jr.]

[-- Attachment #1: Type: text/plain, Size: 774 bytes --]

On Wed, 15 Nov 2017 17:10:11 -0500
"William L. Thomson Jr." <wlt-ml@o-sinc.com> wrote:
>
> Like FreeBSD Foundation, a real organization paying for development
> via grants and other things...
> https://www.freebsdfoundation.org/
> https://www.freebsdfoundation.org/what-we-do/grants/
> 
> https://www.freebsdfoundation.org/wp-content/uploads/2015/12/2017-Q2-Profit-Loss.pdf

For the record, FreeBSD was not like that long ago when Gentoo/we had
booths near them at Linux World Expo. Most of that has happened since.

" A budget of $80,000 was allocated for 2008 to fund multiple
development projects." 
https://www.freebsd.org/news/2008/index.html

https://www.freebsdfoundation.org/wp-content/uploads/2015/12/Budget2011.pdf

-- 
William L. Thomson Jr.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] manifest-hashes changing to 'BLAKE2B SHA512' on 2017-11-21

[William L. Thomson Jr.]

[-- Attachment #1: Type: text/plain, Size: 864 bytes --]

On Wed, 15 Nov 2017 11:47:41 -0600
R0b0t1 <r030t1@gmail.com> wrote:
.> 
> Does the existence of a decision mean I would need to contact the
> trustees if I feel the changes have not been adequately justified?

They have no power here. Consider the foundation as second head of a 2
headed snake. Used to be 3 when infra was more powerful. Trustees maybe
could if there was a different structure. But the foundations role is
pretty limited and intentionally crippled.

Unless you can provide a compelling legal case that would fall under
the legal protection duties of the trustees. But in general Trustees
cannot dictate to council, its the other way around. Council dictates
to Trustees. I doubt it will ever change. I tried long ago.
 
Council has final say on all technical matters unless it involves
legalities.

-- 
William L. Thomson Jr.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] manifest-hashes changing to 'BLAKE2B SHA512' on 2017-11-21

[Nils Freydank]

[-- Attachment #1: Type: text/plain, Size: 718 bytes --]

Am Mittwoch, 15. November 2017, 17:28:44 CET schrieb Michał Górny:
> Hi, everyone.
> 
> The Council has approved the manifest-hashes switch on 2017-11-12
> meeting [1]. The transition will occur to the initial plan, with small
> changes. The updated plan is included at the end of this mail.
> [...]
Just for general convenience:
It appears you forgot the actual link [1], and I assume it should be:
https://wiki.gentoo.org/wiki/Project:Council/Meeting_logs
resp. taken from this site
https://projects.gentoo.org/council/meeting-logs/20171112.txt

Signatures for the textfile logs are linked on the wiki page aswell.
-- 
GPG fingerprint: '766B 8122 1342 6912 3401 492A 8B54 D7A3 FF3C DB17'
Holgersson

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] manifest-hashes changing to 'BLAKE2B SHA512' on 2017-11-21

[Michał Górny]

W dniu śro, 15.11.2017 o godzinie 17∶28 +0100, użytkownik Michał Górny
napisał:
> Hi, everyone.
> 
> The Council has approved the manifest-hashes switch on 2017-11-12
> meeting [1]. The transition will occur to the initial plan, with small
> changes. The updated plan is included at the end of this mail.

Missing reference spotted by Nils Freydank (thanks!):

[1]:https://projects.gentoo.org/council/meeting-logs/20171112-summary.txt

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] manifest-hashes changing to 'BLAKE2B SHA512' on 2017-11-21

[Michał Górny]

W dniu śro, 15.11.2017 o godzinie 17∶28 +0100, użytkownik Michał Górny
napisał:
> Hi, everyone.
> 
> The Council has approved the manifest-hashes switch on 2017-11-12
> meeting [1]. The transition will occur to the initial plan, with small
> changes. The updated plan is included at the end of this mail.
> 
> According to this plan, BLAKE2B will be enabled on 2017-11-21. This
> means that starting at this time, all new and updated DIST entries will
> use BLAKE2B+SHA512. Old DIST entries will still use the current hash set
> until updated.
> 
> The developers are required to upgrade to a package manager supporting
> this hash. That is:
> 
> a. Portage 2.3.5 when using py3.6+,
> 
> b. Portage 2.3.13 + pyblake2 installed manually,
> 
> c. Portage 2.3.13-r1 that includes the pyblake2 dep.
> 
> Modern (and old) Portage will refuse to update Manifests if it does not
> support the necessary hashes. However, Portage versions between 2.3.5
> and 2.3.13 inclusively will create Manifests missing BLAKE2B hash rather
> than failing when no hash provider is present. Those Manifests will be
> rejected by the git hook.
> 
> Users will not be affected noticeably as the SHA512 hash continues being
> used for compatibility.
> 
> 
> That said, I'd like to request developers not to start proactively
> converting all old Manifest entries to the new set immediately,
> and instead give some time for things to settle down.
> 
> 
> 
> The updated plan
> ================
> 
> Already done:
> 
> - revbumped Portage with pyblake2 dep and started stabilizing it,
> 
> - added git update hook to reject invalid Manifest entries.
> 
> 2017-11-21 (T+7d):
> 
> - manifest-hashes = BLAKE2B SHA512

FYI, this is now online. Please ping me if you have any issues.

> 
> 2018-02-14 (T+3m):
> 
> - manifest-required-hashes = BLAKE2B
> 
> 2018-05-14 (T+6m):
> 
> - last rite fetch-restricted packages that do not use BLAKE2B.
> 
> The final removal of SHA512 will be decided by the Council separately.
> 
> 

-- 
Best regards,
Michał Górny

[gentoo-dev] Manifest2 hashes, take n+1-th

[Michał Górny]

Hi, everyone.

The previous discussion on Manifest2 hashes pretty much died away
pending fixes to Portage. Since Portage was fixed a while ago, and we
can now safely switch, I'd like to reboot the discussion before
submitting the item for the next Council meeting.

Considering all arguments made so far, I'd like to propose changing:

  manifest-hashes = SHA256 SHA512 WHIRLPOOL

to:

  manifest-hashes = SHA512 SHA3_512

In other words, removing SHA256 and WHIRLPOOL, and adding SHA3_512.


Rationale
---------

1. The main argument for using multiple hashes is to prevent the (very
unlikely) possibility that if a weakness is discovered in one of
the hashes, the other would still hold. This is given by using two
algorithms; more than two do not increase security significantly, while
they do increase performance cost.

2. For the above to hold, the hashes should be diverse. SHA256
and SHA512 are the same algorithm, so a weakness discovered in either
would probably apply to both -- keeping both does not make sense at all.
Furthermore, both SHA2 and WHIRLPOOL use the same construct (MD), so
a weakness in the construct would apply to both.

3. Keeping one of the three old hashes is necessary for compatibility
reasons. Furthermore, the current versions of Portage consider SHA512
obligatory, so we can't remove it without redesigning Portage first
(though I think this applies only to developer installs, i.e. those
creating Manifests).

4. The new hashes that are stronger and commonly available are
SHA3/Keccak (using sponges) and BLAKE2 (HAIFA). Both are diverse from
our current algorithms, so either is a good candidate. The choice of
Keccak is purely arbitrary (because it's the winner?).

All the above considered, I think it's most reasonable to use two hashes
with diverse constructs. SHA512 needs to be one of them, for
compatibility reasons. The other could be either SHA3_512 or BLAKE2B,
as a strong, future-proof hash. SHA3 is probably a better choice because
it's going to have more support as the official recommendation.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Michał Górny]

W dniu czw, 19.10.2017 o godzinie 21∶08 +0200, użytkownik Michał Górny
napisał:
> 
> 4. The new hashes that are stronger and commonly available are
> SHA3/Keccak (using sponges) and BLAKE2 (HAIFA). Both are diverse from
> our current algorithms, so either is a good candidate. The choice of
> Keccak is purely arbitrary (because it's the winner?).
> 

Actually, a small correction here: we support more implementations
of SHA3 than BLAKE2, so the first one is less problematic for us.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Francesco Riosa]

[-- Attachment #1: Type: text/plain, Size: 1157 bytes --]

2017-10-19 23:00 GMT+02:00 Michał Górny <mgorny@gentoo.org>:

> W dniu czw, 19.10.2017 o godzinie 21∶08 +0200, użytkownik Michał Górny
> napisał:
> >
> > 4. The new hashes that are stronger and commonly available are
> > SHA3/Keccak (using sponges) and BLAKE2 (HAIFA). Both are diverse from
> > our current algorithms, so either is a good candidate. The choice of
> > Keccak is purely arbitrary (because it's the winner?).
> >
>
> Actually, a small correction here: we support more implementations
> of SHA3 than BLAKE2, so the first one is less problematic for us.
>

Not researched in depth but:
B2sum provided by coreutils is quite satisfacting*, it's pretty fast, while
sha-3 is deemed to be slower than sha-2, maybe this could be weighted while
choosing the algorithm wanted.

Both seem to take advantage of modern multicore CPUs but sha-3 does 11.7
[cpb]#0 (see #1) while an email seen on the internet say blake2 can reach 1
[cpb] (see #2)

#0 cpb = cpu cycles per byte
#1 https://en.wikipedia.org/wiki/SHA-3#Speed
#2 http://www.metzdowd.com/pipermail/cryptography/2016-May/029297.html
* (in my limited experience)

[-- Attachment #2: Type: text/html, Size: 1787 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Michał Górny]

W dniu pią, 20.10.2017 o godzinie 00∶20 +0200, użytkownik Francesco
Riosa napisał:
> 2017-10-19 23:00 GMT+02:00 Michał Górny <mgorny@gentoo.org>:
> 
> > W dniu czw, 19.10.2017 o godzinie 21∶08 +0200, użytkownik Michał Górny
> > napisał:
> > > 
> > > 4. The new hashes that are stronger and commonly available are
> > > SHA3/Keccak (using sponges) and BLAKE2 (HAIFA). Both are diverse from
> > > our current algorithms, so either is a good candidate. The choice of
> > > Keccak is purely arbitrary (because it's the winner?).
> > > 
> > 
> > Actually, a small correction here: we support more implementations
> > of SHA3 than BLAKE2, so the first one is less problematic for us.
> > 
> 
> Not researched in depth but:
> B2sum provided by coreutils is quite satisfacting*, it's pretty fast, while
> sha-3 is deemed to be slower than sha-2, maybe this could be weighted while
> choosing the algorithm wanted.
> 
> Both seem to take advantage of modern multicore CPUs but sha-3 does 11.7
> [cpb]#0 (see #1) while an email seen on the internet say blake2 can reach 1
> [cpb] (see #2)
> 
> #0 cpb = cpu cycles per byte
> #1 https://en.wikipedia.org/wiki/SHA-3#Speed
> #2 http://www.metzdowd.com/pipermail/cryptography/2016-May/029297.html
> * (in my limited experience)

I've taken a closer look at BLAKE2 possibilities, and it seems that it's
going to be our choice after all. I'm adding dev-python/pyblake2
as a fallback implementation now.

Curious enough, after disabling the 'optimized' SSE2 assembly, the plain
register implementation is 2.5 times faster than the SSE2 implementation
that is used by default, and two times faster than the built-in SHA2
implementation in Python.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[R0b0t1]

Hello,

I missed some messages in the time I wrote my reply. This also touches
on some of the points in Mr. Górny's other message about time.

On Fri, Oct 20, 2017 at 6:38 PM, Michał Górny <mgorny@gentoo.org> wrote:
> W dniu pią, 20.10.2017 o godzinie 00∶20 +0200, użytkownik Francesco
> Riosa napisał:
>> 2017-10-19 23:00 GMT+02:00 Michał Górny <mgorny@gentoo.org>:
>>
>> > W dniu czw, 19.10.2017 o godzinie 21∶08 +0200, użytkownik Michał Górny
>> > napisał:
>> > >
>> > > 4. The new hashes that are stronger and commonly available are
>> > > SHA3/Keccak (using sponges) and BLAKE2 (HAIFA). Both are diverse from
>> > > our current algorithms, so either is a good candidate. The choice of
>> > > Keccak is purely arbitrary (because it's the winner?).
>> > >
>> >
>> > Actually, a small correction here: we support more implementations
>> > of SHA3 than BLAKE2, so the first one is less problematic for us.
>> >
>>
>> Not researched in depth but:
>> B2sum provided by coreutils is quite satisfacting*, it's pretty fast, while
>> sha-3 is deemed to be slower than sha-2, maybe this could be weighted while
>> choosing the algorithm wanted.
>>
>> Both seem to take advantage of modern multicore CPUs but sha-3 does 11.7
>> [cpb]#0 (see #1) while an email seen on the internet say blake2 can reach 1
>> [cpb] (see #2)
>>
>> #0 cpb = cpu cycles per byte
>> #1 https://en.wikipedia.org/wiki/SHA-3#Speed
>> #2 http://www.metzdowd.com/pipermail/cryptography/2016-May/029297.html
>> * (in my limited experience)
>
> I've taken a closer look at BLAKE2 possibilities, and it seems that it's
> going to be our choice after all. I'm adding dev-python/pyblake2
> as a fallback implementation now.
>
> Curious enough, after disabling the 'optimized' SSE2 assembly, the plain
> register implementation is 2.5 times faster than the SSE2 implementation
> that is used by default, and two times faster than the built-in SHA2
> implementation in Python.
>

It is likely that the register implementation makes better use of the
data and instruction cache and processor instruction pipeline. For a
similar reason, functions with larger block sizes tend to run more
slowly over the same amount of data than their counterparts with
smaller block sizes.

If speed truly is crucial then it may be a better idea to use one very
strong hash function and two weaker but faster hash functions. This is
why I recommended RIPEMD160. If BLAKE2B is used, it may be possible to
switch SHA512 for SHA256.

It seems important to me to use three hash functions.

Again, though, I think it needs to be pointed out that on slower
machines the hash time is on the order of tens of seconds. This should
be negligible compared to the build time.

Cheers,
     R0b0t1

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Hanno Böck]

On Thu, 19 Oct 2017 21:08:40 +0200
Michał Górny <mgorny@gentoo.org> wrote:

>   manifest-hashes = SHA512 SHA3_512

Counterproposal: Just use SHA512.

There isn't any evidence that any SHA2-based hash algorithm is going to
be broken any time soon. If that changes there will very likely be
decades of warning before a break becomes practical.

Having just one hash is simpler and using a well supported one like
SHA512 may make things easier than using something that's still not
very widely supported.

-- 
Hanno Böck
https://hboeck.de/

mail/jabber: hanno@hboeck.de
GPG: FE73757FA60E4E21B937579FA5880072BBB51E42

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Gordon Pettey]

[-- Attachment #1: Type: text/plain, Size: 933 bytes --]

On Thu, Oct 19, 2017 at 5:32 PM, Hanno Böck <hanno@gentoo.org> wrote:

> On Thu, 19 Oct 2017 21:08:40 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
>
> >   manifest-hashes = SHA512 SHA3_512
>
> Counterproposal: Just use SHA512.
>
> There isn't any evidence that any SHA2-based hash algorithm is going to
> be broken any time soon. If that changes there will very likely be
> decades of warning before a break becomes practical.
>
> Having just one hash is simpler and using a well supported one like
> SHA512 may make things easier than using something that's still not
> very widely supported.


Yet having more than one lets you match make sure nobody hijacked your
manifest file when an attack vector is inevitably discovered for the old
new algorithm (whether SHA2, SHA3, or BLAKE2), because you'll be able to
confirm the file is the same one that matched the old checksum in addition
to the new one.

[-- Attachment #2: Type: text/html, Size: 1324 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Dirkjan Ochtman]

[-- Attachment #1: Type: text/plain, Size: 1703 bytes --]

On Fri, Oct 20, 2017 at 12:49 AM, Gordon Pettey <petteyg359@gmail.com>
wrote:

> On Thu, Oct 19, 2017 at 5:32 PM, Hanno Böck <hanno@gentoo.org> wrote:
>
>> On Thu, 19 Oct 2017 21:08:40 +0200
>> Michał Górny <mgorny@gentoo.org> wrote:
>>
>> >   manifest-hashes = SHA512 SHA3_512
>>
>> Counterproposal: Just use SHA512.
>>
>> There isn't any evidence that any SHA2-based hash algorithm is going to
>> be broken any time soon. If that changes there will very likely be
>> decades of warning before a break becomes practical.
>>
>> Having just one hash is simpler and using a well supported one like
>> SHA512 may make things easier than using something that's still not
>> very widely supported.
>
>
> Yet having more than one lets you match make sure nobody hijacked your
> manifest file when an attack vector is inevitably discovered for the old
> new algorithm (whether SHA2, SHA3, or BLAKE2), because you'll be able to
> confirm the file is the same one that matched the old checksum in addition
> to the new one.
>

As Hanno was saying, we'll have decades of warning before a break becomes
practical, so I don't think this is a real concern.

I think the problem of having this discussion on gentoo-dev this way is
that people with vastly different levels of security/crypto expertise are
discussing different options without much regard for the level of expertise
(and maybe even unaware of others' relevant expertise).

I support Hanno's suggestion of doing just SHA512, but would be interested
in hearing opinions from others who have apparent security/crypto
experience. Maybe the Security project can weigh the suggestions as well?

Cheers,

Dirkjan

[-- Attachment #2: Type: text/html, Size: 2690 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 990 bytes --]

>>>>> On Fri, 20 Oct 2017, Dirkjan Ochtman wrote:

> As Hanno was saying, we'll have decades of warning before a break
> becomes practical, so I don't think this is a real concern.

How can we be sure of that? I guess the same reasoning was applied
when MD5 and SHA1 hashes were used.

> I think the problem of having this discussion on gentoo-dev this way
> is that people with vastly different levels of security/crypto
> expertise are discussing different options without much regard for
> the level of expertise (and maybe even unaware of others' relevant
> expertise).

> I support Hanno's suggestion of doing just SHA512, but would be
> interested in hearing opinions from others who have apparent
> security/crypto experience. Maybe the Security project can weigh the
> suggestions as well?

Don't put all eggs in one basket. Having at least one additional hash
(and from a different family) doesn't cost us much and provides an
upgrade path when it should become necessary.

Ulrich

[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Dirkjan Ochtman]

[-- Attachment #1: Type: text/plain, Size: 695 bytes --]

On Fri, Oct 20, 2017 at 11:23 AM, Ulrich Mueller <ulm@gentoo.org> wrote:

> >>>>> On Fri, 20 Oct 2017, Dirkjan Ochtman wrote:
>
> > As Hanno was saying, we'll have decades of warning before a break
> > becomes practical, so I don't think this is a real concern.
>
> How can we be sure of that? I guess the same reasoning was applied
> when MD5 and SHA1 hashes were used.
>

Yeah, and it actually did happen that way. Typically before preimage
attacks (which are what we really care about here, as far as I understand
it) happen there are several other types of attacks that will happen first,
and that will provide advance warning about the level of security provided
by SHA2.

Cheers,

Dirkjan

[-- Attachment #2: Type: text/html, Size: 1148 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Hanno Böck]

[-- Attachment #1: Type: text/plain, Size: 1957 bytes --]

On Fri, 20 Oct 2017 11:23:06 +0200
Ulrich Mueller <ulm@gentoo.org> wrote:

> >>>>> On Fri, 20 Oct 2017, Dirkjan Ochtman wrote:  
> 
> > As Hanno was saying, we'll have decades of warning before a break
> > becomes practical, so I don't think this is a real concern.  
> 
> How can we be sure of that? I guess the same reasoning was applied
> when MD5 and SHA1 hashes were used.

MD5 warning 1996:
ftp://ftp.iks-jena.de/mitarb/lutz/crypt/hash/dobbertin.ps

MD5 broken 2005:
http://merlot.usc.edu/csac-f06/papers/Wang05a.pdf

SHA1 warning 2005:
https://people.csail.mit.edu/yiqun/SHA1AttackProceedingVersion.pdf

SHA1 broken 2017:
https://shattered.io/


It's reasonable to assume that modern hash functions will have a far
longer warning period. For two reasons:
* their safety margin is much higher to begin with, particularly if
  you choose something like SHA512 (256 bit collission resistance). It
  was more or less always clear that MD5 (64 bit) and SHA1 (80 bit) are
  in risky terrain even without any cryptographic breakthrough.
* hash function research in 2017 is lightyears ahead of hash function
  research in the 90s and early 2000s. One major outcome of the
  research after the big hash breakdown in 2005 was that SHA-2 is much
  safer than people previously thought.


I don' have a very strong opinion on this. Having two hash functions
probably won't harm. Though I tend to prefer the simplest solutions if
it's secure. And all my cryptographic knowledge tells me that "What if
sha512 is broken?" isn't a realistic problem to be concerned about.


I do feel it's a bit ironic that we have these lengthy discussions
about hash functions while at the same time they provide little
security to begin with, because they aren't transmitted over a secure
channel and not signed...

-- 
Hanno Böck
https://hboeck.de/

mail/jabber: hanno@hboeck.de
GPG: FE73757FA60E4E21B937579FA5880072BBB51E42

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Kristian Fiskerstrand]

[-- Attachment #1.1: Type: text/plain, Size: 1071 bytes --]

On 10/20/2017 11:10 AM, Dirkjan Ochtman wrote:
> 
> I support Hanno's suggestion of doing just SHA512, but would be
> interested in hearing opinions from others who have apparent
> security/crypto experience. Maybe the Security project can weigh the
> suggestions as well?
> 

The whole discussion is moot so long as we don't have OpenPGP signed
gentoo repository in rsync.

SHA2-512 is generally quicker than sha256 on 64 bit architectures, but
considerably slower for some architectures. Introducing a non-optimized
keccak on top of it will have a significant negative performance impact
for these arches without much security gain.

if we still want two separate hashes, the choice of sha2 and sha3
compination is a good one given they are based on separate constructs.

But IMHO we should start where things matter and complete an
implementation for OpenPGP signatures of MetaManifests in Portage.

-- 
Kristian Fiskerstrand
OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Rich Freeman]

On Fri, Oct 20, 2017 at 6:04 AM, Kristian Fiskerstrand <k_f@gentoo.org> wrote:
> On 10/20/2017 11:10 AM, Dirkjan Ochtman wrote:
>>
>> I support Hanno's suggestion of doing just SHA512, but would be
>> interested in hearing opinions from others who have apparent
>> security/crypto experience. Maybe the Security project can weigh the
>> suggestions as well?
>>
>
> The whole discussion is moot so long as we don't have OpenPGP signed
> gentoo repository in rsync.
>
> SHA2-512 is generally quicker than sha256 on 64 bit architectures, but
> considerably slower for some architectures. Introducing a non-optimized
> keccak on top of it will have a significant negative performance impact
> for these arches without much security gain.
>
> if we still want two separate hashes, the choice of sha2 and sha3
> compination is a good one given they are based on separate constructs.
>
> But IMHO we should start where things matter and complete an
> implementation for OpenPGP signatures of MetaManifests in Portage.
>

Nobody is stopping you from implementing this.

However, saying that we can't improve any other aspects of the
security of Gentoo until somebody improves this one aspect (that
people have been talking about for a decade) seems like a false
dependency.  I do get that a chain is as strong as its weakest link,
but there are attacks that an unsigned hash protects us against.  If
somebody has the ability to modify the upstream binary but not the
ability to modify our hashes, then the hash protects us even if it is
unsigned.  Sure, it would be better still if it were signed, but that
is no reason to make improvements all the same.

IMO having two hash functions makes sense.

Sure, the last two times a hash has failed have had a decade of
warning.  However, there is no theoretical reason why this has to be
the case.  At best you can make hand-waving arguments.

I wouldn't look at md5/sha1 and take from them the lesson that we can
relax because if a hash function fails we have plenty of warning, or
that we don't have to worry because we use more bits today.

Rather I would look at md5/sha1 and take from them this lesson: They
were never intended to be defeated in less than brute force time, and
yet both were.  Adding more bits makes it take longer to brute-force,
but that provides no assurance against an attack that runs faster than
brute force.  No such attack should exist, but the same was true of
md5/sha1.

Sure, some hash failures are worse than others, and we're lucky in
that a less severe failure was discovered first, but there is no
theoretical reason that failures have to be discovered in some
particular order.

If it were a matter of adding support for multiple hash functions to
portage that would be one thing.  However, all our tools already
support this, so why not continue to take advantage of this in the
most efficient manner possible?  Why give up a layer of security that
costs us little but a bit of CPU.

-- 
Rich

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[R0b0t1]

On Fri, Oct 20, 2017 at 8:04 AM, Kristian Fiskerstrand <k_f@gentoo.org> wrote:
> On 10/20/2017 11:10 AM, Dirkjan Ochtman wrote:
>>
>> I support Hanno's suggestion of doing just SHA512, but would be
>> interested in hearing opinions from others who have apparent
>> security/crypto experience. Maybe the Security project can weigh the
>> suggestions as well?
>>
>
> The whole discussion is moot so long as we don't have OpenPGP signed
> gentoo repository in rsync.
>
> SHA2-512 is generally quicker than sha256 on 64 bit architectures, but
> considerably slower for some architectures. Introducing a non-optimized
> keccak on top of it will have a significant negative performance impact
> for these arches without much security gain.
>
> if we still want two separate hashes, the choice of sha2 and sha3
> compination is a good one given they are based on separate constructs.
>
> But IMHO we should start where things matter and complete an
> implementation for OpenPGP signatures of MetaManifests in Portage.
>

This is why I use webrsync-gpg. Git commits are supposed to be
GPG-signed, so that may be suitable for your purposes.

Cheers,
     R0b0t1.

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Anton Molyboha]

[-- Attachment #1: Type: text/plain, Size: 1248 bytes --]

On Thu, Oct 19, 2017 at 6:49 PM, Gordon Pettey <petteyg359@gmail.com> wrote:

> On Thu, Oct 19, 2017 at 5:32 PM, Hanno Böck <hanno@gentoo.org> wrote:
>
>> On Thu, 19 Oct 2017 21:08:40 +0200
>> Michał Górny <mgorny@gentoo.org> wrote:
>>
>> >   manifest-hashes = SHA512 SHA3_512
>>
>> Counterproposal: Just use SHA512.
>>
>> There isn't any evidence that any SHA2-based hash algorithm is going to
>> be broken any time soon. If that changes there will very likely be
>> decades of warning before a break becomes practical.
>>
>> Having just one hash is simpler and using a well supported one like
>> SHA512 may make things easier than using something that's still not
>> very widely supported.
>
>
> Yet having more than one lets you match make sure nobody hijacked your
> manifest file when an attack vector is inevitably discovered for the old
> new algorithm (whether SHA2, SHA3, or BLAKE2), because you'll be able to
> confirm the file is the same one that matched the old checksum in addition
> to the new one.
>

Would it make sense then to support several hashes but let the user
optionally turn off the verification of some of them, depending on the
user's security vs performance requirements?

-- 
Anton

[-- Attachment #2: Type: text/html, Size: 2005 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Gordon Pettey]

[-- Attachment #1: Type: text/plain, Size: 1869 bytes --]

On Fri, Oct 20, 2017 at 5:42 PM, Anton Molyboha <anton.stay.connected@gmail.
com> wrote:

> On Thu, Oct 19, 2017 at 6:49 PM, Gordon Pettey <petteyg359@gmail.com>
> wrote:
>
>> On Thu, Oct 19, 2017 at 5:32 PM, Hanno Böck <hanno@gentoo.org> wrote:
>>
>>> On Thu, 19 Oct 2017 21:08:40 +0200
>>> Michał Górny <mgorny@gentoo.org> wrote:
>>>
>>> >   manifest-hashes = SHA512 SHA3_512
>>>
>>> Counterproposal: Just use SHA512.
>>>
>>> There isn't any evidence that any SHA2-based hash algorithm is going to
>>> be broken any time soon. If that changes there will very likely be
>>> decades of warning before a break becomes practical.
>>>
>>> Having just one hash is simpler and using a well supported one like
>>> SHA512 may make things easier than using something that's still not
>>> very widely supported.
>>
>>
>> Yet having more than one lets you match make sure nobody hijacked your
>> manifest file when an attack vector is inevitably discovered for the old
>> new algorithm (whether SHA2, SHA3, or BLAKE2), because you'll be able to
>> confirm the file is the same one that matched the old checksum in addition
>> to the new one.
>>
>
> Would it make sense then to support several hashes but let the user
> optionally turn off the verification of some of them, depending on the
> user's security vs performance requirements?
>
I would strongly question whether anybody is actually running emerge (or
whatever command that would be using the manifests) on systems that don't
have the CPU power to check a few hashes. If the CPU is really that weak,
there are likely much more important issues to deal with than what
combination of hashing algorithms manifests use. Things like "I should be
using pre-built system images because my CPU is orders of magnitude to even
do dependency tree calculation in less than a decade"...

[-- Attachment #2: Type: text/html, Size: 2845 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Michał Górny]

W dniu pią, 20.10.2017 o godzinie 18∶42 -0400, użytkownik Anton Molyboha
napisał:
> On Thu, Oct 19, 2017 at 6:49 PM, Gordon Pettey <petteyg359@gmail.com> wrote:
> 
> > On Thu, Oct 19, 2017 at 5:32 PM, Hanno Böck <hanno@gentoo.org> wrote:
> > 
> > > On Thu, 19 Oct 2017 21:08:40 +0200
> > > Michał Górny <mgorny@gentoo.org> wrote:
> > > 
> > > >   manifest-hashes = SHA512 SHA3_512
> > > 
> > > Counterproposal: Just use SHA512.
> > > 
> > > There isn't any evidence that any SHA2-based hash algorithm is going to
> > > be broken any time soon. If that changes there will very likely be
> > > decades of warning before a break becomes practical.
> > > 
> > > Having just one hash is simpler and using a well supported one like
> > > SHA512 may make things easier than using something that's still not
> > > very widely supported.
> > 
> > 
> > Yet having more than one lets you match make sure nobody hijacked your
> > manifest file when an attack vector is inevitably discovered for the old
> > new algorithm (whether SHA2, SHA3, or BLAKE2), because you'll be able to
> > confirm the file is the same one that matched the old checksum in addition
> > to the new one.
> > 
> 
> Would it make sense then to support several hashes but let the user
> optionally turn off the verification of some of them, depending on the
> user's security vs performance requirements?
> 

I won't block anyone from implementing such an option but I won't spend
my time on it either. However, if you believe verifying two checksums
could be a problem, then I have serious doubts if you hardware is
capable of building anything.

-- 
Best regards,
Michał Górny

[gentoo-dev] Re: Manifest2 hashes, take n+1-th

[Duncan]

Michał Górny posted on Sat, 21 Oct 2017 01:39:55 +0200 as excerpted:

> W dniu pią, 20.10.2017 o godzinie 18∶42 -0400, użytkownik Anton Molyboha
> napisał:

>> Would it make sense then to support several hashes but let the user
>> optionally turn off the verification of some of them, depending on the
>> user's security vs performance requirements?
>> 
> I won't block anyone from implementing such an option but I won't spend
> my time on it either. However, if you believe verifying two checksums
> could be a problem, then I have serious doubts if you hardware is
> capable of building anything.

When does this verification happen?

If it's during --sync or --pretend/ask, as I believe it is based on when 
I get errors if I edit and forget to manifest/digest, then arguably time 
matters rather more than it does if it's only after the user has OKed the 
merge and it's doing the build.

Because the time before the PM tells me what it's going to do and asks my 
OK before doing it is time I'm generally actually waiting for it (tho I'm 
normally doing something else while waiting, but I /am/ waiting) to 
decide whether I want to go ahead, or perhaps I need to change something 
first, while the actual build time after I've OKed it, doesn't matter so 
much, because I'm not actually waiting on it, I'm doing other things, 
which can actually include turning in for the night or going to work, 
with the intent being that it'll be done when I get back to it.

So the hash verification time really does matter, even if it's minutes 
compared to hours of actual build time, because that's time I'm actively 
waiting for it, vs. letting it do its thing in the background, with much 
less concern about how long (within reason) it might take.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Michael Orlitzky]

On 10/19/2017 06:32 PM, Hanno Böck wrote:
> 
> Counterproposal: Just use SHA512.
> 
> There isn't any evidence that any SHA2-based hash algorithm is going to
> be broken any time soon. If that changes there will very likely be
> decades of warning before a break becomes practical.
> 

Every WiFi network on the planet essentially became Starbucks overnight
on Sunday->Monday, so in my opinion we shouldn't bet against immediate
and catastrophic failure of anything, no matter how well-tested.

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Kristian Fiskerstrand]

[-- Attachment #1.1: Type: text/plain, Size: 443 bytes --]

On 10/20/2017 03:05 PM, Michael Orlitzky wrote:
> Every WiFi network on the planet essentially became Starbucks overnight
> on Sunday->Monday, so in my opinion we shouldn't bet against immediate
> and catastrophic failure of anything, no matter how well-tested.

Post Hoc ergo Propter Hoc

-- 
Kristian Fiskerstrand
OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Paweł Hajdan, Jr.]

[-- Attachment #1.1: Type: text/plain, Size: 642 bytes --]

On 19/10/2017 21:08, Michał Górny wrote:
> Considering all arguments made so far, I'd like to propose changing:
>   manifest-hashes = SHA256 SHA512 WHIRLPOOL
> to:
>   manifest-hashes = SHA512 SHA3_512

+1, fine for me

> 1. The main argument for using multiple hashes is to prevent the (very
> unlikely) possibility that if a weakness is discovered in one of
> the hashes, the other would still hold. This is given by using two
> algorithms; more than two do not increase security significantly, while
> they do increase performance cost.

Curious, do we have any measurements/estimates of the performance cost?

Paweł


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Michał Górny]

W dniu pią, 20.10.2017 o godzinie 17∶42 +0200, użytkownik Paweł Hajdan,
Jr. napisał:
> On 19/10/2017 21:08, Michał Górny wrote:
> > Considering all arguments made so far, I'd like to propose changing:
> >   manifest-hashes = SHA256 SHA512 WHIRLPOOL
> > to:
> >   manifest-hashes = SHA512 SHA3_512
> 
> +1, fine for me
> 
> > 1. The main argument for using multiple hashes is to prevent the (very
> > unlikely) possibility that if a weakness is discovered in one of
> > the hashes, the other would still hold. This is given by using two
> > algorithms; more than two do not increase security significantly, while
> > they do increase performance cost.
> 
> Curious, do we have any measurements/estimates of the performance cost?
> 

Not specific but I think it's pretty simple, assuming we don't get any
multithread-friendly algorithms.

With a single thread serial processing of all hashes, it's just sum of
times involved in every hash, i.e. Th = T1 + T2 + T3 + ... You'd have to
get some numbers to get something smarter out of it.

If we assume we can do N threads, then cost of N algorithms is equal to
the slowest of them all. Which implies that having N algorithms is
fastest on systems capable of at least N threads.

Taking a random comparison [1], it seems that SHA3/512 is 3-5 times
slower than SHA2/512. If we take that as conclusive, the relative times
would be:

a. single hash:

 SHA512 - 1
 SHA3_512 - 3-5

b. both hashes:

 serial - 4-6
 parallel - 3-5

[1]:http://wireilla.com/papers/ijcis/V3N3/3313ijcis01.pdf

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Paweł Hajdan, Jr.]

[-- Attachment #1.1: Type: text/plain, Size: 1042 bytes --]

On 20/10/2017 18:15, Michał Górny wrote:
> W dniu pią, 20.10.2017 o godzinie 17∶42 +0200, użytkownik Paweł Hajdan,
> Jr. napisał:
>> Curious, do we have any measurements/estimates of the performance cost?
> 
> With a single thread serial processing of all hashes, it's just sum of
> times involved in every hash, i.e. Th = T1 + T2 + T3 + ... You'd have to
> get some numbers to get something smarter out of it.
> 
> If we assume we can do N threads, then cost of N algorithms is equal to
> the slowest of them all. Which implies that having N algorithms is
> fastest on systems capable of at least N threads.
> 
> Taking a random comparison [1], it seems that SHA3/512 is 3-5 times
> slower than SHA2/512.
How large part of dependency calculation / other portage's operation is
this though?

My point is, did profiling turn out hash computation as bottleneck, or
is this more speculative?

I'm still in favor of modernizing the hashes, just somewhat skeptical
when performance is being mentioned.

Paweł


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Michał Górny]

W dniu sob, 21.10.2017 o godzinie 10∶01 +0200, użytkownik Paweł Hajdan,
Jr. napisał:
> On 20/10/2017 18:15, Michał Górny wrote:
> > W dniu pią, 20.10.2017 o godzinie 17∶42 +0200, użytkownik Paweł Hajdan,
> > Jr. napisał:
> > > Curious, do we have any measurements/estimates of the performance cost?
> > 
> > With a single thread serial processing of all hashes, it's just sum of
> > times involved in every hash, i.e. Th = T1 + T2 + T3 + ... You'd have to
> > get some numbers to get something smarter out of it.
> > 
> > If we assume we can do N threads, then cost of N algorithms is equal to
> > the slowest of them all. Which implies that having N algorithms is
> > fastest on systems capable of at least N threads.
> > 
> > Taking a random comparison [1], it seems that SHA3/512 is 3-5 times
> > slower than SHA2/512.
> 
> How large part of dependency calculation / other portage's operation is
> this though?
> 
> My point is, did profiling turn out hash computation as bottleneck, or
> is this more speculative?

Purely speculative.

> I'm still in favor of modernizing the hashes, just somewhat skeptical
> when performance is being mentioned.
> 

FWICS BLAKE2 can be even 2.5 times faster than SHA2, so we'll probably
go with that. In this case, the performance impact will be negligible --
in fact, it should be faster than the current set of three hashes.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[R0b0t1]

Hello,

On Thu, Oct 19, 2017 at 2:08 PM, Michał Górny <mgorny@gentoo.org> wrote:
> Hi, everyone.
>
> The previous discussion on Manifest2 hashes pretty much died away
> pending fixes to Portage. Since Portage was fixed a while ago, and we
> can now safely switch, I'd like to reboot the discussion before
> submitting the item for the next Council meeting.
>
> Considering all arguments made so far, I'd like to propose changing:
>
>   manifest-hashes = SHA256 SHA512 WHIRLPOOL
>
> to:
>
>   manifest-hashes = SHA512 SHA3_512
>
> In other words, removing SHA256 and WHIRLPOOL, and adding SHA3_512.
>

In general I do not mind updating the algorithms used, but I do feel
it is important to keep at least three present. Without at least three
(or a larger odd number) it is not possible to break a tie.

That may ultimately be beside the point, as any invalid hashes should
result in the user contacting the developers or doing something else,
but it is hard to know.

>
> Rationale
> ---------
>
> 1. The main argument for using multiple hashes is to prevent the (very
> unlikely) possibility that if a weakness is discovered in one of
> the hashes, the other would still hold. This is given by using two
> algorithms; more than two do not increase security significantly, while
> they do increase performance cost.
>
> 2. For the above to hold, the hashes should be diverse. SHA256
> and SHA512 are the same algorithm, so a weakness discovered in either
> would probably apply to both -- keeping both does not make sense at all.
> Furthermore, both SHA2 and WHIRLPOOL use the same construct (MD), so
> a weakness in the construct would apply to both.
>

Algorithms of different block sizes can easily have different
cryptographic properties.

Notable is https://en.wikipedia.org/wiki/Advanced_Encryption_Standard,
see box. More relevant is https://en.wikipedia.org/wiki/SHA-2.

In general only the computational complexity is affected, but in the
case of AES some attacks work for the larger key lengths that do not
work for smaller ones. There is one other case I am having problems
remembering where a longer keylength led to a massively easier to
attack function.

For SHA3_512, I am worried that some of the novel properties of the
function that do not currently lead to attacks may in the future lead
to attacks.

> 3. Keeping one of the three old hashes is necessary for compatibility
> reasons. Furthermore, the current versions of Portage consider SHA512
> obligatory, so we can't remove it without redesigning Portage first
> (though I think this applies only to developer installs, i.e. those
> creating Manifests).
>
> 4. The new hashes that are stronger and commonly available are
> SHA3/Keccak (using sponges) and BLAKE2 (HAIFA). Both are diverse from
> our current algorithms, so either is a good candidate. The choice of
> Keccak is purely arbitrary (because it's the winner?).
>
> All the above considered, I think it's most reasonable to use two hashes
> with diverse constructs. SHA512 needs to be one of them, for
> compatibility reasons. The other could be either SHA3_512 or BLAKE2B,
> as a strong, future-proof hash. SHA3 is probably a better choice because
> it's going to have more support as the official recommendation.
>

I would like to present my suggestions:

SHA512, (RIPEMD160 | WHIRLPOOL | BLAKE2B), (SHA3_512 | BLAKE2B);

or more definitively:

SHA512, RIPEMD160, BLAKE2B.

My understanding is that whirlpool is the least secure of the 512 bit
hashes, but it would still be a serious undertaking to generate a
collision. If there is any cause to replace it RIPEMD160 is likely a
good candidate. It is similar in operation to the SHA1/2 functions but
was produced by the academic community and is quite dissimilar from
the SHA1/2 family. The "definitive" suggestions provide a wide sample
of time, size, and implementation.

SHA3_512 is novel and may be resistant to attack in ways the others
are not, but it is my opinion that BLAKE2B's author is better known
and respected.

I could also see the combination SHA512, BLAKE2B, SHA3_512 being
desirable, as that selection represents 3 very diverse algorithms.

If the suggestion of RIPEMD160 seems odd, realize that some
distribution platforms still use MD5 along with other functions. It is
fast to compute and with another hash function quite credible. If the
time taken to generate the hashes is a legitimate concern then using a
older, simpler, and heavily optimized function may be better for one
of the choices.

Respectfully,
     R0b0t1

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 985 bytes --]

On Fri, Oct 20, 2017 at 05:21:47PM -0500, R0b0t1 wrote:
> I would like to present my suggestions:
> 
> SHA512, (RIPEMD160 | WHIRLPOOL | BLAKE2B), (SHA3_512 | BLAKE2B);
> 
> or more definitively:
> 
> SHA512, RIPEMD160, BLAKE2B.
Please do NOT reintroduce RIPEMD160. It was one of the older Portage
hashes prior to implementation of GLEP059, and was removed because it
was shown to fall to parts of the same attacks at MD4/MD5 by Wang's
paper in 2004.

Wang, X. et al. (2004). "Collisions for Hash Functions MD4, MD5,
HAVAL-128 and RIPEMD", rump session, CRYPTO 2004, Cryptology ePrint
Archive, Report 2004/199, first version (August 16, 2004), second
version (August 17, 2004). Available online from:
http://eprint.iacr.org/2004/199.pdf

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 1113 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[R0b0t1]

On Sat, Oct 21, 2017 at 11:26 AM, Robin H. Johnson <robbat2@gentoo.org> wrote:
> On Fri, Oct 20, 2017 at 05:21:47PM -0500, R0b0t1 wrote:
>> I would like to present my suggestions:
>>
>> SHA512, (RIPEMD160 | WHIRLPOOL | BLAKE2B), (SHA3_512 | BLAKE2B);
>>
>> or more definitively:
>>
>> SHA512, RIPEMD160, BLAKE2B.
> Please do NOT reintroduce RIPEMD160. It was one of the older Portage
> hashes prior to implementation of GLEP059, and was removed because it
> was shown to fall to parts of the same attacks at MD4/MD5 by Wang's
> paper in 2004.
>
> Wang, X. et al. (2004). "Collisions for Hash Functions MD4, MD5,
> HAVAL-128 and RIPEMD", rump session, CRYPTO 2004, Cryptology ePrint
> Archive, Report 2004/199, first version (August 16, 2004), second
> version (August 17, 2004). Available online from:
> http://eprint.iacr.org/2004/199.pdf
>

That is precisely why I didn't suggest it be used on its own (see note
about extant use of MD5), and why I gave alternatives. If it is
desired that the hashes be computed quickly then weaker hashes will
need to be used. One usually can't have both security and speed.

Can anyone defend the transition to two hashes, or is it just based on
speculation?

People are discussing collision resistance, but no one here appears to
be trained in cryptography. The only reasonable solution in that case
is not to rely on the particular mostly unknowable merits of an
algorithm, but the hardness of a successful collision of multiple
functions at the same time.

*If* collision resistance is important, and *if* no one here can
evaluate any of the algorithms intensively by themselves, then *why*
are two hashes going to be used instead of three? That is making the
system much weaker than it was.

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[R0b0t1]

On Sat, Oct 21, 2017 at 12:12 PM, R0b0t1 <r030t1@gmail.com> wrote:
> On Sat, Oct 21, 2017 at 11:26 AM, Robin H. Johnson <robbat2@gentoo.org> wrote:
>> On Fri, Oct 20, 2017 at 05:21:47PM -0500, R0b0t1 wrote:
>>> I would like to present my suggestions:
>>>
>>> SHA512, (RIPEMD160 | WHIRLPOOL | BLAKE2B), (SHA3_512 | BLAKE2B);
>>>
>>> or more definitively:
>>>
>>> SHA512, RIPEMD160, BLAKE2B.
>> Please do NOT reintroduce RIPEMD160. It was one of the older Portage
>> hashes prior to implementation of GLEP059, and was removed because it
>> was shown to fall to parts of the same attacks at MD4/MD5 by Wang's
>> paper in 2004.
>>
>> Wang, X. et al. (2004). "Collisions for Hash Functions MD4, MD5,
>> HAVAL-128 and RIPEMD", rump session, CRYPTO 2004, Cryptology ePrint
>> Archive, Report 2004/199, first version (August 16, 2004), second
>> version (August 17, 2004). Available online from:
>> http://eprint.iacr.org/2004/199.pdf
>>
>

Also important is that the existence of a constructed collision is not
necessarily an indication that the function is weak for real data.


> Can anyone defend the transition to two hashes, or is it just based on
> speculation?
>

This thread in particular is the worst case of bikeshedding I have
seen on gentoo-dev. No one here is well equipped to evaluate the
cryptographic primitives being discussed[1] but there are still many
strong opinions and unwarranted suggestions.

Respectfully,
     R0b0t1


[1]: In fairness perhaps no one is, as the cryptography of one
particular function takes very intensive study. Most published
algorithms are never studied intently until they are adopted. Still,
people should be justifying any suggestion by referencing real data or
tested deficiencies. Not guessing.

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Hanno Böck]

On Sat, 21 Oct 2017 12:12:44 -0500
R0b0t1 <r030t1@gmail.com> wrote:

> That is precisely why I didn't suggest it be used on its own (see note
> about extant use of MD5), and why I gave alternatives. If it is
> desired that the hashes be computed quickly then weaker hashes will
> need to be used. One usually can't have both security and speed.

You can have that. Blake2 is faster than any broken legacy hash.
And ripemd isn't particularly fast

> People are discussing collision resistance, but no one here appears to
> be trained in cryptography.

For the record, I'd claim I am.

-- 
Hanno Böck
https://hboeck.de/

mail/jabber: hanno@hboeck.de
GPG: FE73757FA60E4E21B937579FA5880072BBB51E42

[gentoo-dev] Re: Manifest2 hashes, take n+1-th

[Duncan]

Hanno Böck posted on Sat, 21 Oct 2017 19:50:11 +0200 as excerpted:

> On Sat, 21 Oct 2017 12:12:44 -0500 R0b0t1 <r030t1@gmail.com> wrote:
> 
>> People are discussing collision resistance, but no one here appears to
>> be trained in cryptography.
> 
> For the record, I'd claim I am.

... And with a number of vuln discoveries to your credit, it's safe to 
say it's not just paper certs for you, too. =:^)

(And FWIW I'd point to Robin H Johnson/robbat2 as someone I know has 
authority in this area as well.  There may be others.  FTR I'm not one of 
them, tho as any good admin I try to follow the security news especially 
where it touches machines I administer, so I'm following this thread with 
particular interest.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Manifest2 hashes, take n+1-th

[R0b0t1]

On Sat, Oct 21, 2017 at 3:11 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> Hanno Böck posted on Sat, 21 Oct 2017 19:50:11 +0200 as excerpted:
>
>> On Sat, 21 Oct 2017 12:12:44 -0500 R0b0t1 <r030t1@gmail.com> wrote:
>>
>>> People are discussing collision resistance, but no one here appears to
>>> be trained in cryptography.
>>
>> For the record, I'd claim I am.

On what basis? I performed a search on your name, and found at least
one person who was belligerently calling you a liar who wastes
people's time. The others results seemed to have no relation to
cryptography and were about technology journalism.

>
> ... And with a number of vuln discoveries to your credit, it's safe to
> say it's not just paper certs for you, too. =:^)
>

Of what nature are these vulnerabilities? There is a vast gulf between
discussing cryptography with a mathematical basis and finding code
which improperly implements cryptography. Or, as it seems based on my
searches, simply finding bugs or logical errors in programs.

> (And FWIW I'd point to Robin H Johnson/robbat2 as someone I know has
> authority in this area as well.  There may be others.  FTR I'm not one of
> them, tho as any good admin I try to follow the security news especially
> where it touches machines I administer, so I'm following this thread with
> particular interest.)
>

On what basis? As above, programming and cryptography have very little
in common, besides the fact computers can be programmed to perform
cryptography operations.


These posts are concerning because it looks like someone became stir
crazy and invented a problem to solve. The changes proposed to date
have remained poorly justified, and no one has addressed the concern
that multiple hashes *is* actually more secure.

If it was deemed necessary at one point, what justification was used?
I.e. https://en.wikipedia.org/wiki/Wikipedia:Chesterton's_fence.

Respectfully,
     R0b0t1

Re: [gentoo-dev] Re: Manifest2 hashes, take n+1-th

[Jonas Stein]

[-- Attachment #1.1: Type: text/plain, Size: 336 bytes --]

Hi "R0b0t1",

>>> For the record, I'd claim I am.

The question
> On what basis? 
is ok, but

> I performed a search on your name, and found at least
> one person who was belligerently calling you a liar [..]

does not fit here. Please keep this dirt away from our mailinglist and
stay professional.

Thank you,
Jonas


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[R0b0t1]

[-- Attachment #1: Type: text/plain, Size: 872 bytes --]

Hello,

On Wednesday, November 8, 2017, Jonas Stein <jstein@gentoo.org> wrote:
> Hi "R0b0t1",
>
>>>> For the record, I'd claim I am.
>
> The question
>> On what basis?
> is ok, but
>
>> I performed a search on your name, and found at least
>> one person who was belligerently calling you a liar [..]
>
> does not fit here. Please keep this dirt away from our mailinglist and
> stay professional.
>

Actions have consequences. I may not have found the same Mr. Bock, in which
case he can certainly say something to that effect. Otherwise I see no
reason why his actions off-list related to security are not precisely what
should be discussed.

He is the one who failed to provide any support for his statement. I
researched him myself and that is the first thing I found. I did not
immediately link it as I felt it may be unsuitable for the list.

Respectfully,
    R0b0t1

[-- Attachment #2: Type: text/html, Size: 1064 bytes --]

[gentoo-dev] Manifest2 hashes: validation of single hash per MANIFESTx_REQUIRED_HASH

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 4173 bytes --]

Replying to your original question here, to repeat the answer I emphasised
before, along with significantly more detail in the history of Portage hashes
(pulled from my notes back to GLEP57 and some minor updates).

On Wed, Nov 08, 2017 at 12:57:49PM -0600, R0b0t1 wrote:
> These posts are concerning because it looks like someone became stir
> crazy and invented a problem to solve. The changes proposed to date
> have remained poorly justified, and no one has addressed the concern
> that multiple hashes *is* actually more secure.
> 
> If it was deemed necessary at one point, what justification was used?
> I.e. https://en.wikipedia.org/wiki/Wikipedia:Chesterton's_fence.
On Wed, Nov 15, 2017 at 11:47:41AM -0600, R0b0t1 wrote:
> Does the existence of a decision mean I would need to contact the trustees
> if I feel the changes have not been adequately justified?

In GLEP59, I referenced a paper by Joux [J04], in which it was shown that a
concatenation of multiple hashes is NOT much more secure against collisions
than the strongest of the individual hashes.

That was cited as original logic in GLEP59 for the removal of SHA256 (that
removal was never implemented). WHIRLPOOL & SHA512 were kept out of an
abundance of caution at the time, mostly to implementation bugs in hashes (as I
have referenced in the related threads since).

Your logic regarding removing something you think I don't understand is wrong
(Chesterton's Fence): 

If you dig in the history of Portage, you will see that it's always been valid,
to have just a SINGLE hash for each file in a Manifest.  Required hashes has
NEVER contained more than one hash.

If multiple hashes are present, then Portage will validate all of them, but a
potential attacker can still modify the Manifest and have only a single hash
listed.  Exactly which hash MUST be present has changed over time. 

Manifest1 is very old, and was stored in $CAT/$PN/files/digest-$P
Manifest2 is the current $CAT/$PN/Manifest (and soon in more locations per MetaManifest).

1999/xx/xx: Portage starts with Manifest1 format, MD5-only (CVS)
2004/08/25: Portage gets SHA1 support in Manifest1, but is problematic, SHA1 generation manual only.
https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-src/portage/pym/portage_checksum.py?revision=1.1&view=markup
https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-src/portage/pym/portage.py?r1=1.485&r2=1.486
2005/12/19: Portage Manifest1 supports MD5,SHA1,SHA256,RMD160, but still requires only a single hash present. Generates MD5+SHA256+RMD160.
https://gitweb.gentoo.org/proj/portage.git/commit/?id=cd3e3775966a9f58aebb91f58cbdb5903faad3de
2006/03/24: Manifest2 introduced.
https://gitweb.gentoo.org/proj/portage.git/commit/?id=f993747ca501e8a70d6f6174711149a172cfc3c2
2007/01/20: MANIFEST2_REQUIRED_HASH introduced, SHA1, it must be present & pass
https://gitweb.gentoo.org/proj/portage.git/commit/?id=e768571187d1655fbb558c23d61fa2983e48e411
2007/12/18: MANIFEST1_REQUIRED_HASH introduced, MD5, it must be present & pass
https://gitweb.gentoo.org/proj/portage.git/commit/?id=d9b10deaa03ce174d5ccc3b59c477549ad87e884
2008/02/28: Manifest1 support dropped.
https://gitweb.gentoo.org/proj/portage.git/commit/?id=66940e1f2f0549ee8f01dad59016e168105e193d
2011/10/02: GLEP59 implemented, MANIFEST2_REQUIRED_HASH changes to SHA256
https://gitweb.gentoo.org/proj/portage.git/commit/?id=c8cd3a985cc529299411d7343a11004b7d1330ef
2017/06/15: MANIFEST2_REQUIRED_HASH changes to SHA512
https://gitweb.gentoo.org/proj/portage.git/commit/?id=e6abcc0b7cbdca481862a5c7cca946c01c471ffb

[J04] Joux, Antoie. (2004). "Multicollisions in Iterated Hash Functions - Application to Cascaded Constructions;" 
Proceedings of CRYPTO 2004, Franklin, M. (Ed); Lecture Notes in Computer Science 3152, pp. 306-316. 
Available online from: http://web.cecs.pdx.edu/~teshrim/spring06/papers/general-attacks/multi-joux.pdf

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 1113 bytes --]

Re: [gentoo-dev] Manifest2 hashes: validation of single hash per MANIFESTx_REQUIRED_HASH

[R0b0t1]

Hello friends!

On Wed, Nov 15, 2017 at 3:02 PM, Robin H. Johnson <robbat2@gentoo.org> wrote:
> Replying to your original question here, to repeat the answer I emphasised
> before, along with significantly more detail in the history of Portage hashes
> (pulled from my notes back to GLEP57 and some minor updates).
>
> On Wed, Nov 08, 2017 at 12:57:49PM -0600, R0b0t1 wrote:
>> These posts are concerning because it looks like someone became stir
>> crazy and invented a problem to solve. The changes proposed to date
>> have remained poorly justified, and no one has addressed the concern
>> that multiple hashes *is* actually more secure.
>>
>> If it was deemed necessary at one point, what justification was used?
>> I.e. https://en.wikipedia.org/wiki/Wikipedia:Chesterton's_fence.
> On Wed, Nov 15, 2017 at 11:47:41AM -0600, R0b0t1 wrote:
>> Does the existence of a decision mean I would need to contact the trustees
>> if I feel the changes have not been adequately justified?
>
> In GLEP59, I referenced a paper by Joux [J04], in which it was shown that a
> concatenation of multiple hashes is NOT much more secure against collisions
> than the strongest of the individual hashes.
>
> That was cited as original logic in GLEP59 for the removal of SHA256 (that
> removal was never implemented). WHIRLPOOL & SHA512 were kept out of an
> abundance of caution at the time, mostly to implementation bugs in hashes (as I
> have referenced in the related threads since).
>
> Your logic regarding removing something you think I don't understand is wrong
> (Chesterton's Fence):
>
> If you dig in the history of Portage, you will see that it's always been valid,
> to have just a SINGLE hash for each file in a Manifest.  Required hashes has
> NEVER contained more than one hash.
>
> If multiple hashes are present, then Portage will validate all of them, but a
> potential attacker can still modify the Manifest and have only a single hash
> listed.  Exactly which hash MUST be present has changed over time.
>
> Manifest1 is very old, and was stored in $CAT/$PN/files/digest-$P
> Manifest2 is the current $CAT/$PN/Manifest (and soon in more locations per MetaManifest).
>
> 1999/xx/xx: Portage starts with Manifest1 format, MD5-only (CVS)
> 2004/08/25: Portage gets SHA1 support in Manifest1, but is problematic, SHA1 generation manual only.
> https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-src/portage/pym/portage_checksum.py?revision=1.1&view=markup
> https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-src/portage/pym/portage.py?r1=1.485&r2=1.486
> 2005/12/19: Portage Manifest1 supports MD5,SHA1,SHA256,RMD160, but still requires only a single hash present. Generates MD5+SHA256+RMD160.
> https://gitweb.gentoo.org/proj/portage.git/commit/?id=cd3e3775966a9f58aebb91f58cbdb5903faad3de
> 2006/03/24: Manifest2 introduced.
> https://gitweb.gentoo.org/proj/portage.git/commit/?id=f993747ca501e8a70d6f6174711149a172cfc3c2
> 2007/01/20: MANIFEST2_REQUIRED_HASH introduced, SHA1, it must be present & pass
> https://gitweb.gentoo.org/proj/portage.git/commit/?id=e768571187d1655fbb558c23d61fa2983e48e411
> 2007/12/18: MANIFEST1_REQUIRED_HASH introduced, MD5, it must be present & pass
> https://gitweb.gentoo.org/proj/portage.git/commit/?id=d9b10deaa03ce174d5ccc3b59c477549ad87e884
> 2008/02/28: Manifest1 support dropped.
> https://gitweb.gentoo.org/proj/portage.git/commit/?id=66940e1f2f0549ee8f01dad59016e168105e193d
> 2011/10/02: GLEP59 implemented, MANIFEST2_REQUIRED_HASH changes to SHA256
> https://gitweb.gentoo.org/proj/portage.git/commit/?id=c8cd3a985cc529299411d7343a11004b7d1330ef
> 2017/06/15: MANIFEST2_REQUIRED_HASH changes to SHA512
> https://gitweb.gentoo.org/proj/portage.git/commit/?id=e6abcc0b7cbdca481862a5c7cca946c01c471ffb
>
> [J04] Joux, Antoie. (2004). "Multicollisions in Iterated Hash Functions - Application to Cascaded Constructions;"
> Proceedings of CRYPTO 2004, Franklin, M. (Ed); Lecture Notes in Computer Science 3152, pp. 306-316.
> Available online from: http://web.cecs.pdx.edu/~teshrim/spring06/papers/general-attacks/multi-joux.pdf
>

This is the information I was looking for, thank you. I feel that the
matter has been adequately explained. I apologize for missing your
response.

The paper gives a counter intuitive result, so I suspect I will have
to spend more time with it.

Cheers,
     R0b0t1

Re: [gentoo-dev] Manifest2 hashes: validation of single hash per MANIFESTx_REQUIRED_HASH

[R0b0t1]

On Mon, Nov 20, 2017 at 9:00 PM, R0b0t1 <r030t1@gmail.com> wrote:
> Hello friends!
>
> On Wed, Nov 15, 2017 at 3:02 PM, Robin H. Johnson <robbat2@gentoo.org> wrote:
>> Replying to your original question here, to repeat the answer I emphasised
>> before, along with significantly more detail in the history of Portage hashes
>> (pulled from my notes back to GLEP57 and some minor updates).
>>
>> On Wed, Nov 08, 2017 at 12:57:49PM -0600, R0b0t1 wrote:
>>> These posts are concerning because it looks like someone became stir
>>> crazy and invented a problem to solve. The changes proposed to date
>>> have remained poorly justified, and no one has addressed the concern
>>> that multiple hashes *is* actually more secure.
>>>
>>> If it was deemed necessary at one point, what justification was used?
>>> I.e. https://en.wikipedia.org/wiki/Wikipedia:Chesterton's_fence.
>> On Wed, Nov 15, 2017 at 11:47:41AM -0600, R0b0t1 wrote:
>>> Does the existence of a decision mean I would need to contact the trustees
>>> if I feel the changes have not been adequately justified?
>>
>> In GLEP59, I referenced a paper by Joux [J04], in which it was shown that a
>> concatenation of multiple hashes is NOT much more secure against collisions
>> than the strongest of the individual hashes.
>>
>> That was cited as original logic in GLEP59 for the removal of SHA256 (that
>> removal was never implemented). WHIRLPOOL & SHA512 were kept out of an
>> abundance of caution at the time, mostly to implementation bugs in hashes (as I
>> have referenced in the related threads since).
>>
>> Your logic regarding removing something you think I don't understand is wrong
>> (Chesterton's Fence):
>>
>> If you dig in the history of Portage, you will see that it's always been valid,
>> to have just a SINGLE hash for each file in a Manifest.  Required hashes has
>> NEVER contained more than one hash.
>>
>> If multiple hashes are present, then Portage will validate all of them, but a
>> potential attacker can still modify the Manifest and have only a single hash
>> listed.  Exactly which hash MUST be present has changed over time.
>>
>> Manifest1 is very old, and was stored in $CAT/$PN/files/digest-$P
>> Manifest2 is the current $CAT/$PN/Manifest (and soon in more locations per MetaManifest).
>>
>> 1999/xx/xx: Portage starts with Manifest1 format, MD5-only (CVS)
>> 2004/08/25: Portage gets SHA1 support in Manifest1, but is problematic, SHA1 generation manual only.
>> https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-src/portage/pym/portage_checksum.py?revision=1.1&view=markup
>> https://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-src/portage/pym/portage.py?r1=1.485&r2=1.486
>> 2005/12/19: Portage Manifest1 supports MD5,SHA1,SHA256,RMD160, but still requires only a single hash present. Generates MD5+SHA256+RMD160.
>> https://gitweb.gentoo.org/proj/portage.git/commit/?id=cd3e3775966a9f58aebb91f58cbdb5903faad3de
>> 2006/03/24: Manifest2 introduced.
>> https://gitweb.gentoo.org/proj/portage.git/commit/?id=f993747ca501e8a70d6f6174711149a172cfc3c2
>> 2007/01/20: MANIFEST2_REQUIRED_HASH introduced, SHA1, it must be present & pass
>> https://gitweb.gentoo.org/proj/portage.git/commit/?id=e768571187d1655fbb558c23d61fa2983e48e411
>> 2007/12/18: MANIFEST1_REQUIRED_HASH introduced, MD5, it must be present & pass
>> https://gitweb.gentoo.org/proj/portage.git/commit/?id=d9b10deaa03ce174d5ccc3b59c477549ad87e884
>> 2008/02/28: Manifest1 support dropped.
>> https://gitweb.gentoo.org/proj/portage.git/commit/?id=66940e1f2f0549ee8f01dad59016e168105e193d
>> 2011/10/02: GLEP59 implemented, MANIFEST2_REQUIRED_HASH changes to SHA256
>> https://gitweb.gentoo.org/proj/portage.git/commit/?id=c8cd3a985cc529299411d7343a11004b7d1330ef
>> 2017/06/15: MANIFEST2_REQUIRED_HASH changes to SHA512
>> https://gitweb.gentoo.org/proj/portage.git/commit/?id=e6abcc0b7cbdca481862a5c7cca946c01c471ffb
>>
>> [J04] Joux, Antoie. (2004). "Multicollisions in Iterated Hash Functions - Application to Cascaded Constructions;"
>> Proceedings of CRYPTO 2004, Franklin, M. (Ed); Lecture Notes in Computer Science 3152, pp. 306-316.
>> Available online from: http://web.cecs.pdx.edu/~teshrim/spring06/papers/general-attacks/multi-joux.pdf
>>
>
> This is the information I was looking for, thank you. I feel that the
> matter has been adequately explained. I apologize for missing your
> response.
>
> The paper gives a counter intuitive result, so I suspect I will have
> to spend more time with it.
>

I appreciate the thought that robbat2 gave to his response, but I
would like to clarify that it is beyond and above what I expected.

What I wanted to avoid was something I encountered on the GCC mailing
list: When I asked why GCJ was removed, I was told that it was hard to
maintain. When I asked for an example of past maintenance issues (and
made it clear I had no interest in disputing whether the issues were
valid or not) I received no reply from the maintainer except his
original answer, leaving me to wonder whether GCJ was actually hard to
maintain.

I have seen similar exchanges associated with other projects.

Cheers,
    R0b0t1

Re: [gentoo-dev] Manifest2 hashes: validation of single hash per MANIFESTx_REQUIRED_HASH

[Matt Turner]

On Mon, Nov 20, 2017 at 7:15 PM, R0b0t1 <r030t1@gmail.com> wrote:
> What I wanted to avoid was something I encountered on the GCC mailing
> list: When I asked why GCJ was removed, I was told that it was hard to
> maintain. When I asked for an example of past maintenance issues (and
> made it clear I had no interest in disputing whether the issues were
> valid or not) I received no reply from the maintainer except his
> original answer, leaving me to wonder whether GCJ was actually hard to
> maintain.
>
> I have seen similar exchanges associated with other projects.

When you have no standing in the project, there's little incentive to
justify one's actions and decisions to you.

Re: [gentoo-dev] Manifest2 hashes: validation of single hash per MANIFESTx_REQUIRED_HASH

[R0b0t1]

On Mon, Nov 20, 2017 at 10:19 PM, Matt Turner <mattst88@gentoo.org> wrote:
> On Mon, Nov 20, 2017 at 7:15 PM, R0b0t1 <r030t1@gmail.com> wrote:
>> What I wanted to avoid was something I encountered on the GCC mailing
>> list: When I asked why GCJ was removed, I was told that it was hard to
>> maintain. When I asked for an example of past maintenance issues (and
>> made it clear I had no interest in disputing whether the issues were
>> valid or not) I received no reply from the maintainer except his
>> original answer, leaving me to wonder whether GCJ was actually hard to
>> maintain.
>>
>> I have seen similar exchanges associated with other projects.
>
> When you have no standing in the project, there's little incentive to
> justify one's actions and decisions to you.
>

No, but common courtesy would be to provide a short answer. If what I
have requested is more work than the individual wants to undertake
they would be free to say so (which is why I was confused, I only
expected a sentence or two, and this is why I felt I should explain
robbat2's answer was better than I expected). At the same time, I aim
to only ask questions that I feel the other person would have already
considered. In this case, if the decision is the right one, then a
coherent explanation of why the actions being taken are being taken
should already exist in some form.

If they don't, then why is action being taken?


There is another comment of mine on this list where I asked a
developer why a package was being dropped. I had no intention of
disputing their decision, but I felt the given reason was too vague to
be useful. Their response was maybe two sentences and added something
like "upstream is being difficult." That seemed fine.

Cheers,
     R0b0t1

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[R0b0t1]

My apologies, I forgot to address something:

On Sat, Oct 21, 2017 at 12:50 PM, Hanno Böck <hanno@gentoo.org> wrote:
> On Sat, 21 Oct 2017 12:12:44 -0500
> R0b0t1 <r030t1@gmail.com> wrote:
>
>> That is precisely why I didn't suggest it be used on its own (see note
>> about extant use of MD5), and why I gave alternatives. If it is
>> desired that the hashes be computed quickly then weaker hashes will
>> need to be used. One usually can't have both security and speed.
>
> You can have that. Blake2 is faster than any broken legacy hash.
> And ripemd isn't particularly fast
>

Fair enough, but it is new and may have security problems related to
its operation that have not been found. This is hard to reason about,
but I would note that many cryptographic standards are fairly
conservative for similar reasons.

Ease of computation reduces security.

Respectfully,
     R0b0t1

Re: [gentoo-dev] Manifest2 hashes, take n+1-th: 3 hashes for the tie-breaker case

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 2151 bytes --]

On Fri, Oct 20, 2017 at 05:21:47PM -0500, R0b0t1 wrote:
> In general I do not mind updating the algorithms used, but I do feel
> it is important to keep at least three present. Without at least three
> (or a larger odd number) it is not possible to break a tie.
> 
> That may ultimately be beside the point, as any invalid hashes should
> result in the user contacting the developers or doing something else,
> but it is hard to know.
I'm dropping the rest of your email about about exactly which hashes
we're bike-shedding, to focus on the number of hashes.

I agree with your opinion to have three hashes present, and I've give a
solid rationale with historical references.

The major reason to have 3 hashes, is as a tie-breaker, to detect if
there was a bug in the hash somehow (implementation, compiler/assembler,
interpreter), and not the distfile. This also strongly suggests that 3
hashes should have different construction.

It's come up enough times in Gentoo history already. Here's 3 of the
instances that came to mind and I could link up with bugs easily. I also
recall an instance where the entire SHA2 family was bitten by a buggy
arch-specific (mips? arm?) GCC patch, but I can't the bug for it.

2006: https://bugs.gentoo.org/121182
pycrypto RMD160 on ia64 (and many other 64bit arches)
(it also had a big cleanup for the tree as a result: https://bugs.gentoo.org/121124)

2009: https://bugs.gentoo.org/255131
app-crypt/mhash-0.9.9 segfaults with NULL digest in whirlpool/snefru
(portage uses python-mhash bindings)

2012: https://bugs.gentoo.org/406407
sys-apps/portage-2.1.10.49: internal version of whirlpool algorithm generates wrong hash

Since we're going to much newer hashes, I think there is a non-zero
chance we WILL hit errors in the hashes again, and it would be wise to
cover the bases.

This ends up probably looking like: SHA512, BLAKE2B, SHA3_512

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 1113 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th: 3 hashes for the tie-breaker case

[Michał Górny]

Dnia 23 października 2017 10:16:38 CEST, "Robin H. Johnson" <robbat2@gentoo.org> napisał(a):
>On Fri, Oct 20, 2017 at 05:21:47PM -0500, R0b0t1 wrote:
>> In general I do not mind updating the algorithms used, but I do feel
>> it is important to keep at least three present. Without at least
>three
>> (or a larger odd number) it is not possible to break a tie.
>> 
>> That may ultimately be beside the point, as any invalid hashes should
>> result in the user contacting the developers or doing something else,
>> but it is hard to know.
>I'm dropping the rest of your email about about exactly which hashes
>we're bike-shedding, to focus on the number of hashes.
>
>I agree with your opinion to have three hashes present, and I've give a
>solid rationale with historical references.
>
>The major reason to have 3 hashes, is as a tie-breaker, to detect if
>there was a bug in the hash somehow (implementation,
>compiler/assembler,
>interpreter), and not the distfile. This also strongly suggests that 3
>hashes should have different construction.

1. How are you planning to distinguish a successful attack against two hashes from a bug in one of them?

2. Even if you do, what's the value of knowing that?

>
>It's come up enough times in Gentoo history already. Here's 3 of the
>instances that came to mind and I could link up with bugs easily. I
>also
>recall an instance where the entire SHA2 family was bitten by a buggy
>arch-specific (mips? arm?) GCC patch, but I can't the bug for it.
>
>2006: https://bugs.gentoo.org/121182
>pycrypto RMD160 on ia64 (and many other 64bit arches)
>(it also had a big cleanup for the tree as a result:
>https://bugs.gentoo.org/121124)
>
>2009: https://bugs.gentoo.org/255131
>app-crypt/mhash-0.9.9 segfaults with NULL digest in whirlpool/snefru
>(portage uses python-mhash bindings)

How is this one relevant? AFAICS it did not cause wrong result.

>
>2012: https://bugs.gentoo.org/406407
>sys-apps/portage-2.1.10.49: internal version of whirlpool algorithm
>generates wrong hash
>
>Since we're going to much newer hashes, I think there is a non-zero
>chance we WILL hit errors in the hashes again, and it would be wise to
>cover the bases.
>
>This ends up probably looking like: SHA512, BLAKE2B, SHA3_512


-- 
Best regards,
Michał Górny (by phone)

Re: [gentoo-dev] Manifest2 hashes, take n+1-th: 3 hashes for the tie-breaker case

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 3869 bytes --]

On Mon, Oct 23, 2017 at 01:33:15PM +0200, Michał Górny wrote:
> Dnia 23 października 2017 10:16:38 CEST, "Robin H. Johnson" <robbat2@gentoo.org> napisał(a):
> >On Fri, Oct 20, 2017 at 05:21:47PM -0500, R0b0t1 wrote:
> >> In general I do not mind updating the algorithms used, but I do feel
> >> it is important to keep at least three present. Without at least
> >three
> >> (or a larger odd number) it is not possible to break a tie.
> >> 
> >> That may ultimately be beside the point, as any invalid hashes should
> >> result in the user contacting the developers or doing something else,
> >> but it is hard to know.
> >I'm dropping the rest of your email about about exactly which hashes
> >we're bike-shedding, to focus on the number of hashes.
> >
> >I agree with your opinion to have three hashes present, and I've give a
> >solid rationale with historical references.
> >
> >The major reason to have 3 hashes, is as a tie-breaker, to detect if
> >there was a bug in the hash somehow (implementation,
> >compiler/assembler,
> >interpreter), and not the distfile. This also strongly suggests that 3
> >hashes should have different construction.
> 
> 1. How are you planning to distinguish a successful attack against two hashes from a bug in one of them?
> 
> 2. Even if you do, what's the value of knowing that?
[BOBO06] is relevant research here, I cited it in the work that went into
GLEP59, the last time we updated the hashes. The less-technical explanation of it is:
"If you can express the output of H1(x)H2(x) in LESS bits than the combined
output size of H1,H2, then the attacks get a little bit easier"

Some important pieces from it:
[J04] "showed that the concatenation of two Merkle-Damgard functions is not
much more secure than the individual functions.", but this holds ONLY if
the hash functions chosen are of the same construction (MD).
Choosing hashes with different constructions (Merkle-Damgard, HAIFA,
Sponge) is important, and given a black box environment, 

The original mail reached the same approximate decision, just to look
for diverse hashes, but decided that 2 was enough.

Q: What are the odds of a simultaneous successful attack against two hashes? 
A: IDK, but if the hash functions are truly independent, it must be provably
   lower than the odds of an attack against a single hash.

Q: What's the big difference between a bug and a successful attack in a hash?
A: Bugs are more likely initially, and attacks come later.

All of that said, is there really a significant long-term gain in
multiple hashes? (setting aside the short-term advantage in a transition
period for changing hashes)

> >2009: https://bugs.gentoo.org/255131
> >app-crypt/mhash-0.9.9 segfaults with NULL digest in whirlpool/snefru
> >(portage uses python-mhash bindings)
> How is this one relevant? AFAICS it did not cause wrong result.
It output inconsistent garbage for the hash in at least one case that I
recall.

[BOBO06] Boneh, D. and Boyen, X. (2006). 
"On the Impossibility of Efficiently Combining Collision Resistant Hash Functions"; 
Proceedings of CRYPTO 2006, Dwork, C. (Ed.); 
Lecture Notes in Computer Science 4117, pp. 570-583. 
Available online from: http://crypto.stanford.edu/~dabo/abstracts/hashing.html

[J04] Joux A. (2004).
"Multicollisions in Iterated Hash Functions. Application to Cascaded Constructions". 
In: Franklin M. (eds) Advances in Cryptology – CRYPTO 2004. CRYPTO 2004. 
Lecture Notes in Computer Science, vol 3152. Springer, Berlin, Heidelberg
https://www.iacr.org/archive/crypto2004/31520306/multicollisions.pdf

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 1113 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th: 3 hashes for the tie-breaker case

[Michał Górny]

W dniu pon, 23.10.2017 o godzinie 21∶00 +0000, użytkownik Robin H.
Johnson napisał:
> On Mon, Oct 23, 2017 at 01:33:15PM +0200, Michał Górny wrote:
> > Dnia 23 października 2017 10:16:38 CEST, "Robin H. Johnson" <robbat2@gentoo.org> napisał(a):
> > > On Fri, Oct 20, 2017 at 05:21:47PM -0500, R0b0t1 wrote:
> > > > In general I do not mind updating the algorithms used, but I do feel
> > > > it is important to keep at least three present. Without at least
> > > 
> > > three
> > > > (or a larger odd number) it is not possible to break a tie.
> > > > 
> > > > That may ultimately be beside the point, as any invalid hashes should
> > > > result in the user contacting the developers or doing something else,
> > > > but it is hard to know.
> > > 
> > > I'm dropping the rest of your email about about exactly which hashes
> > > we're bike-shedding, to focus on the number of hashes.
> > > 
> > > I agree with your opinion to have three hashes present, and I've give a
> > > solid rationale with historical references.
> > > 
> > > The major reason to have 3 hashes, is as a tie-breaker, to detect if
> > > there was a bug in the hash somehow (implementation,
> > > compiler/assembler,
> > > interpreter), and not the distfile. This also strongly suggests that 3
> > > hashes should have different construction.
> > 
> > 1. How are you planning to distinguish a successful attack against two hashes from a bug in one of them?
> > 
> > 2. Even if you do, what's the value of knowing that?
> 
> [BOBO06] is relevant research here, I cited it in the work that went into
> GLEP59, the last time we updated the hashes. The less-technical explanation of it is:
> "If you can express the output of H1(x)H2(x) in LESS bits than the combined
> output size of H1,H2, then the attacks get a little bit easier"
> 
> Some important pieces from it:
> [J04] "showed that the concatenation of two Merkle-Damgard functions is not
> much more secure than the individual functions.", but this holds ONLY if
> the hash functions chosen are of the same construction (MD).
> Choosing hashes with different constructions (Merkle-Damgard, HAIFA,
> Sponge) is important, and given a black box environment, 
> 
> The original mail reached the same approximate decision, just to look
> for diverse hashes, but decided that 2 was enough.
> 
> Q: What are the odds of a simultaneous successful attack against two hashes? 
> A: IDK, but if the hash functions are truly independent, it must be provably
>    lower than the odds of an attack against a single hash.

We're talking about really huge (→∞) numbers here. It's not a 'random'
attack against one hash. It's an attack that allows to sneak a malicious
code with unchanged file size (since we store that too), and no apparent
side effects (what's the point of spending up that much resources
if the user is going to notice?).

> Q: What's the big difference between a bug and a successful attack in a hash?
> A: Bugs are more likely initially, and attacks come later.

Sounds like an entirely abstract point in time ;-).

> All of that said, is there really a significant long-term gain in
> multiple hashes? (setting aside the short-term advantage in a transition
> period for changing hashes)

No, and that's my point. One hash is perfectly fine.

Two hashes are useful for transition purposes. If we take two fast
hashes (e.g. proposed SHA512 + BLAKE2B which have similar speed),
we can use 2 threads to prevent the speed loss (except for old single-
core machines).

Three hashes don't give any noticeable advantage. If we want a diverse
construct, we take SHA3. SHA3 is slower than SHA2 + BLAKE2 combined, so
even with 3 threaded computation it's going to be slower.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] Manifest2 hashes, take n+1-th: 3 hashes for the tie-breaker case

[Michał Górny]

W dniu wto, 24.10.2017 o godzinie 06∶04 +0200, użytkownik Michał Górny
napisał:
> W dniu pon, 23.10.2017 o godzinie 21∶00 +0000, użytkownik Robin H.
> Johnson napisał:
> > On Mon, Oct 23, 2017 at 01:33:15PM +0200, Michał Górny wrote:
> > > Dnia 23 października 2017 10:16:38 CEST, "Robin H. Johnson" <robbat2@gentoo.org> napisał(a):
> > > > On Fri, Oct 20, 2017 at 05:21:47PM -0500, R0b0t1 wrote:
> > > > > In general I do not mind updating the algorithms used, but I do feel
> > > > > it is important to keep at least three present. Without at least
> > > > 
> > > > three
> > > > > (or a larger odd number) it is not possible to break a tie.
> > > > > 
> > > > > That may ultimately be beside the point, as any invalid hashes should
> > > > > result in the user contacting the developers or doing something else,
> > > > > but it is hard to know.
> > > > 
> > > > I'm dropping the rest of your email about about exactly which hashes
> > > > we're bike-shedding, to focus on the number of hashes.
> > > > 
> > > > I agree with your opinion to have three hashes present, and I've give a
> > > > solid rationale with historical references.
> > > > 
> > > > The major reason to have 3 hashes, is as a tie-breaker, to detect if
> > > > there was a bug in the hash somehow (implementation,
> > > > compiler/assembler,
> > > > interpreter), and not the distfile. This also strongly suggests that 3
> > > > hashes should have different construction.
> > > 
> > > 1. How are you planning to distinguish a successful attack against two hashes from a bug in one of them?
> > > 
> > > 2. Even if you do, what's the value of knowing that?
> > 
> > [BOBO06] is relevant research here, I cited it in the work that went into
> > GLEP59, the last time we updated the hashes. The less-technical explanation of it is:
> > "If you can express the output of H1(x)H2(x) in LESS bits than the combined
> > output size of H1,H2, then the attacks get a little bit easier"
> > 
> > Some important pieces from it:
> > [J04] "showed that the concatenation of two Merkle-Damgard functions is not
> > much more secure than the individual functions.", but this holds ONLY if
> > the hash functions chosen are of the same construction (MD).
> > Choosing hashes with different constructions (Merkle-Damgard, HAIFA,
> > Sponge) is important, and given a black box environment, 
> > 
> > The original mail reached the same approximate decision, just to look
> > for diverse hashes, but decided that 2 was enough.
> > 
> > Q: What are the odds of a simultaneous successful attack against two hashes? 
> > A: IDK, but if the hash functions are truly independent, it must be provably
> >    lower than the odds of an attack against a single hash.
> 
> We're talking about really huge (→∞) numbers here. It's not a 'random'
> attack against one hash. It's an attack that allows to sneak a malicious
> code with unchanged file size (since we store that too), and no apparent
> side effects (what's the point of spending up that much resources
> if the user is going to notice?).
> 
> > Q: What's the big difference between a bug and a successful attack in a hash?
> > A: Bugs are more likely initially, and attacks come later.
> 
> Sounds like an entirely abstract point in time ;-).
> 
> > All of that said, is there really a significant long-term gain in
> > multiple hashes? (setting aside the short-term advantage in a transition
> > period for changing hashes)
> 
> No, and that's my point. One hash is perfectly fine.
> 
> Two hashes are useful for transition purposes. If we take two fast
> hashes (e.g. proposed SHA512 + BLAKE2B which have similar speed),
> we can use 2 threads to prevent the speed loss (except for old single-
> core machines).
> 
> Three hashes don't give any noticeable advantage. If we want a diverse
> construct, we take SHA3. SHA3 is slower than SHA2 + BLAKE2 combined, so
> even with 3 threaded computation it's going to be slower.

Oh, and most notably, the speed loss will be mostly visible to users.
An attacker would have to compute the additional hashes only
if the fastest hash already matched, i.e. rarely. Users will have to
compute them all the time.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] Manifest2 hashes, take n+1-th: 3 hashes for the tie-breaker case

[Paweł Hajdan, Jr.]

[-- Attachment #1.1: Type: text/plain, Size: 897 bytes --]

On 24/10/2017 06:11, Michał Górny wrote:
> W dniu wto, 24.10.2017 o godzinie 06∶04 +0200, użytkownik Michał Górny
> napisał:
>> Three hashes don't give any noticeable advantage. If we want a diverse
>> construct, we take SHA3. SHA3 is slower than SHA2 + BLAKE2 combined, so
>> even with 3 threaded computation it's going to be slower.
> 
> Oh, and most notably, the speed loss will be mostly visible to users.
> An attacker would have to compute the additional hashes only
> if the fastest hash already matched, i.e. rarely. Users will have to
> compute them all the time.

I'm surprised to see bikeshedding about this, where the performance
argument was shown to be speculative.

Consider clarifying what's the goal of this thread.

It seemed like a relatively obvious cleanup / modernizing the set of
hash functions, and I'd still be supportive of that.

Paweł


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th: 3 hashes for the tie-breaker case

[Rich Freeman]

On Tue, Oct 24, 2017 at 4:21 AM, Paweł Hajdan, Jr.
<phajdan.jr@gentoo.org> wrote:
> On 24/10/2017 06:11, Michał Górny wrote:
>> W dniu wto, 24.10.2017 o godzinie 06∶04 +0200, użytkownik Michał Górny
>> napisał:
>>> Three hashes don't give any noticeable advantage. If we want a diverse
>>> construct, we take SHA3. SHA3 is slower than SHA2 + BLAKE2 combined, so
>>> even with 3 threaded computation it's going to be slower.
>>
>> Oh, and most notably, the speed loss will be mostly visible to users.
>> An attacker would have to compute the additional hashes only
>> if the fastest hash already matched, i.e. rarely. Users will have to
>> compute them all the time.
>
> I'm surprised to see bikeshedding about this, where the performance
> argument was shown to be speculative.
>
> Consider clarifying what's the goal of this thread.
>
> It seemed like a relatively obvious cleanup / modernizing the set of
> hash functions, and I'd still be supportive of that.
>

++

IMO nothing really new has come up for the most part.  People disagree
on a few points, as is inevitable.  The purpose of mailing lists isn't
to keep reiterating the same points until there is unanimous
agreement.  Best to just move on.

-- 
Rich

Re: [gentoo-dev] Manifest2 hashes, take n+1-th: 3 hashes for the tie-breaker case

[Chí-Thanh Christopher Nguyễn]

Michał Górny schrieb:
> Oh, and most notably, the speed loss will be mostly visible to users.
> An attacker would have to compute the additional hashes only
> if the fastest hash already matched, i.e. rarely. Users will have to
> compute them all the time.

That is currently the case with portage, but not an inevitable consequence of
having 3 hash functions in the Manifest. Portage could be made to check only
one or two of them (even by default), giving the tie-breaking ability to
those who need it, and speeding up things for those who don't.


Best regards,
Chí-Thanh Christopher Nguyễn

Re: [gentoo-dev] Manifest2 hashes, take n+1-th: 3 hashes for the tie-breaker case

[Michał Górny]

W dniu wto, 24.10.2017 o godzinie 13∶56 +0200, użytkownik Chí-Thanh
Christopher Nguyễn napisał:
> Michał Górny schrieb:
> > Oh, and most notably, the speed loss will be mostly visible to users.
> > An attacker would have to compute the additional hashes only
> > if the fastest hash already matched, i.e. rarely. Users will have to
> > compute them all the time.
> 
> That is currently the case with portage, but not an inevitable consequence of
> having 3 hash functions in the Manifest. Portage could be made to check only
> one or two of them (even by default), giving the tie-breaking ability to
> those who need it, and speeding up things for those who don't.

No, it can't. The specification (GLEP 59) requires it to check all hashes.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] Manifest2 hashes, take n+1-th: 3 hashes for the tie-breaker case

[Allan Wegan]

[-- Attachment #1.1: Type: text/plain, Size: 1170 bytes --]

>> That is currently the case with portage, but not an inevitable
>> consequence of having 3 hash functions in the Manifest. Portage could
>> be made to check only one or two of them (even by default), giving
>> the tie-breaking ability to those who need it, and speeding up things
>> for those who don't.
> No, it can't. The specification (GLEP 59) requires it to check all
> hashes.

Of course it can: The code of the specification just has to be changed
before changing the code of portage. The question is not whether it is
possible to make portage skip hash verification - but whether it is a
good idea to make it do that...

I would not mind as long as the default is to always check all the
hashes and the option to disable it is properly named (like
"--disable-hash-verification" or something similar) and documented.



-- 
Allan Wegan
<http://www.allanwegan.de/>
Jabber: allanwegan@ffnord.net
 OTR-Fingerprint: E4DCAA40 4859428E B3912896 F2498604 8CAA126F
Jabber: allanwegan@jabber.ccc.de
 OTR-Fingerprint: A1AAA1B9 C067F988 4A424D33 98343469 29164587
ICQ: 209459114
 OTR-Fingerprint: 71DE5B5E 67D6D758 A93BF1CE 7DA06625 205AC6EC


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th: one hash to decide them all

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 2855 bytes --]

On Tue, Oct 24, 2017 at 11:33:39PM +0200, Allan Wegan wrote:
> >> That is currently the case with portage, but not an inevitable
> >> consequence of having 3 hash functions in the Manifest. Portage could
> >> be made to check only one or two of them (even by default), giving
> >> the tie-breaking ability to those who need it, and speeding up things
> >> for those who don't.
> > No, it can't. The specification (GLEP 59) requires it to check all
> > hashes.
> 
> Of course it can: The code of the specification just has to be changed
> before changing the code of portage. The question is not whether it is
> possible to make portage skip hash verification - but whether it is a
> good idea to make it do that...
> 
> I would not mind as long as the default is to always check all the
> hashes and the option to disable it is properly named (like
> "--disable-hash-verification" or something similar) and documented.
At that point, and this is a serious proposal:
The package manager shall decide which hashes to check, but is required
to check at least one hash. The choice may be 'fastest', 'most secure',
or any local factor.

For local values of 'most secure' or 'fastest'.

I wrote GLEP59, and specified at the time that all hashes should be
checked, based on prior experience with hash implementation problems.

Skipping them entirely is a bad idea, but only checking one of them is a
reasonable suggestion.

I retract my prior suggestion that there should be 3 hashes, as I
realize a key statements in GLEP59 that were NEVER implemented:
>> - Removal of depreciated checksums shall happen after no less than 18
>>   months or one major Portage version cycle, whichever is greater.
>> ...
>> After the majority of Portage installations include SHA512 support:
>> - Remove SHA256.

This GLEP to update the GLEP59 specification should state the following:
- The package manager or verification tool is required to verify at
  least one hash, preferably the strongest supported hash.
- Multiple hashes may be present for transitional periods.
- SHA3 or BLAKE2B shall be added to the official Manifest2 hashes.

For implementation:
- Generation of WHIRLPOOL and SHA256 shall be disabled in the next
  Portage minor release (as soon as possible)
- Generation of the next choice of hash, SHA3 or BLAKE2B shall be
  enabled in an upcoming minor Portage release (soon)
- 18 months after the next GLEP is approved, SHA512 shall be dropped
  (put the date into the Portage code so it happens automatically this
  time, unlike SHA256 that should have been removed in 2010!).

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 1113 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th: one hash to decide them all

[Hanno Böck]

[-- Attachment #1: Type: text/plain, Size: 2271 bytes --]

Hi,

On Wed, 25 Oct 2017 02:40:58 +0000
"Robin H. Johnson" <robbat2@gentoo.org> wrote:

> At that point, and this is a serious proposal:
> The package manager shall decide which hashes to check, but is
> required to check at least one hash. The choice may be 'fastest',
> 'most secure', or any local factor.

Sorry to contribute again to the bikeshedding, but I'd really like to
get one thought across here:
Good security includes reducing complexity. Tough (as evident by this
thread) it's a thought many people find hard to accept.

I don't think this is most important in this discussion, but I feel
it's something people should keep in mind also for other decisions to
be made.

This thread is going into a completely different direction and I find
that worriesome. We have two non-problems ("what if secure hash X gets
broken?" and "what if it's too slow? I haven't benchmarked, but what if
it's too slow??") and people proposing increasingly complex solutions.

If you do what you propose my worries aren't that any hash gets broken
or that it's too slow. It's that some bug will chime in where in some
situation no hash gets checked whatsoever.

Having more than one hash is already unneeded complexity. Nobody does
that. TLS signatures use one hash. GPG signatures uses one hash. Signal
uses one hash. I'm not aware of any credible cryptographic product that
feels the need to have multiple hashes concatenated. (The only real
example I'm aware of is old TLS versions who chose to concat two
insecure hashes - MD5+sha1 - which obviously wasn't the smartest idea
either, but one can credibly say they didn't know better back then.)

Having a situation where one can either check one hash or multiple and
add configurability around that is another step of adding unneeded
complexity.


Also one more comment about the issue with potentially buggy Hash
implementations: I feel this is a software testing problem rather than
anything that should influence our package manager format or be tested
at runtime. Add a self-test of hash functions with a large batch of
test vectors that you can easily run.

-- 
Hanno Böck
https://hboeck.de/

mail/jabber: hanno@hboeck.de
GPG: FE73757FA60E4E21B937579FA5880072BBB51E42

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th: one hash to decide them all

[Paweł Hajdan, Jr.]

[-- Attachment #1.1: Type: text/plain, Size: 883 bytes --]

On 25/10/2017 14:32, Hanno Böck wrote:
> Good security includes reducing complexity. Tough (as evident by this
> thread) it's a thought many people find hard to accept.
>
> This thread is going into a completely different direction and I find
> that worriesome. We have two non-problems ("what if secure hash X gets
> broken?" and "what if it's too slow? I haven't benchmarked, but what if
> it's too slow??") and people proposing increasingly complex solutions.
> 
> If you do what you propose my worries aren't that any hash gets broken
> or that it's too slow. It's that some bug will chime in where in some
> situation no hash gets checked whatsoever.

+1

I consider the multiple hashes we have a part of providing smooth
migration path (keeping around hashes supported by older portage
versions). Other than that, yeah, watch out for complexity.

Paweł


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th: one hash to decide them all

[R0b0t1]

On Tue, Oct 24, 2017 at 9:40 PM, Robin H. Johnson <robbat2@gentoo.org> wrote:
> On Tue, Oct 24, 2017 at 11:33:39PM +0200, Allan Wegan wrote:
>> >> That is currently the case with portage, but not an inevitable
>> >> consequence of having 3 hash functions in the Manifest. Portage could
>> >> be made to check only one or two of them (even by default), giving
>> >> the tie-breaking ability to those who need it, and speeding up things
>> >> for those who don't.
>> > No, it can't. The specification (GLEP 59) requires it to check all
>> > hashes.
>>
>> Of course it can: The code of the specification just has to be changed
>> before changing the code of portage. The question is not whether it is
>> possible to make portage skip hash verification - but whether it is a
>> good idea to make it do that...
>>
>> I would not mind as long as the default is to always check all the
>> hashes and the option to disable it is properly named (like
>> "--disable-hash-verification" or something similar) and documented.
> At that point, and this is a serious proposal:
> The package manager shall decide which hashes to check, but is required
> to check at least one hash. The choice may be 'fastest', 'most secure',
> or any local factor.
>
> For local values of 'most secure' or 'fastest'.
>
> I wrote GLEP59, and specified at the time that all hashes should be
> checked, based on prior experience with hash implementation problems.
>
> Skipping them entirely is a bad idea, but only checking one of them is a
> reasonable suggestion.
>

The talk on gentoo-dev related to the security of hash functions still
has me confused as to why anyone would want to remove the
alternatives. Arguably this gives the setup most of its security.

I acknowledge Mr. Böck's observation that most programs only use one
hash. When I first encountered portage, I thought the use of multiple
hashes was a very novel and security conscious design choice. The cost
is negligible compared to the difficulty it adds to generating a
collision.

Of course, the difficulty of generating a collision that results in
usable code is still very high, but there was an interesting paper
that described how it could be easier than most people think. I am not
sure how to find it again. It was based on the principle that the
solution space for collisions that were valid was actually far denser
than the solution space of all collisions. One of the results of this
is that finding usefully malicious collisions is probably easier than
finding a collision in general, although there may be more non-useful
collisions overall.

> I retract my prior suggestion that there should be 3 hashes, as I
> realize a key statements in GLEP59 that were NEVER implemented:
>>> - Removal of depreciated checksums shall happen after no less than 18
>>>   months or one major Portage version cycle, whichever is greater.
>>> ...
>>> After the majority of Portage installations include SHA512 support:
>>> - Remove SHA256.
>
> This GLEP to update the GLEP59 specification should state the following:
> - The package manager or verification tool is required to verify at
>   least one hash, preferably the strongest supported hash.
> - Multiple hashes may be present for transitional periods.
> - SHA3 or BLAKE2B shall be added to the official Manifest2 hashes.
>


I'm still kind of confused as to why these changes are being made. Can
they be justified?

> For implementation:
> - Generation of WHIRLPOOL and SHA256 shall be disabled in the next
>   Portage minor release (as soon as possible)
> - Generation of the next choice of hash, SHA3 or BLAKE2B shall be
>   enabled in an upcoming minor Portage release (soon)
> - 18 months after the next GLEP is approved, SHA512 shall be dropped
>   (put the date into the Portage code so it happens automatically this
>   time, unlike SHA256 that should have been removed in 2010!).
>

This makes sense, but I would hope deprecation can be justified in a useful way.

Cheers,
     R0b0t1

Re: [gentoo-dev] Manifest2 hashes, take n+1-th: 3 hashes for the tie-breaker case

[Joshua Kinard]

On 10/24/2017 00:11, Michał Górny wrote:
> W dniu wto, 24.10.2017 o godzinie 06∶04 +0200, użytkownik Michał Górny
> napisał:

[snip]

>>> [BOBO06] is relevant research here, I cited it in the work that went into
>>> GLEP59, the last time we updated the hashes. The less-technical explanation of it is:
>>> "If you can express the output of H1(x)H2(x) in LESS bits than the combined
>>> output size of H1,H2, then the attacks get a little bit easier"
>>>
>>> Some important pieces from it:
>>> [J04] "showed that the concatenation of two Merkle-Damgard functions is not
>>> much more secure than the individual functions.", but this holds ONLY if
>>> the hash functions chosen are of the same construction (MD).
>>> Choosing hashes with different constructions (Merkle-Damgard, HAIFA,
>>> Sponge) is important, and given a black box environment, 
>>>
>>> The original mail reached the same approximate decision, just to look
>>> for diverse hashes, but decided that 2 was enough.
>>>
>>> Q: What are the odds of a simultaneous successful attack against two hashes? 
>>> A: IDK, but if the hash functions are truly independent, it must be provably
>>>    lower than the odds of an attack against a single hash.
>>
>> We're talking about really huge (→∞) numbers here. It's not a 'random'
>> attack against one hash. It's an attack that allows to sneak a malicious
>> code with unchanged file size (since we store that too), and no apparent
>> side effects (what's the point of spending up that much resources
>> if the user is going to notice?).
>>
>>> Q: What's the big difference between a bug and a successful attack in a hash?
>>> A: Bugs are more likely initially, and attacks come later.
>>
>> Sounds like an entirely abstract point in time ;-).
>>
>>> All of that said, is there really a significant long-term gain in
>>> multiple hashes? (setting aside the short-term advantage in a transition
>>> period for changing hashes)
>>
>> No, and that's my point. One hash is perfectly fine.
>>
>> Two hashes are useful for transition purposes. If we take two fast
>> hashes (e.g. proposed SHA512 + BLAKE2B which have similar speed),
>> we can use 2 threads to prevent the speed loss (except for old single-
>> core machines).

Minor clarification, old single core //and// uni-processor.  Some older
machines have multiple physical CPUs that are single-core.  Threading should be
okay on these, as long as the thread count stays under NR_CPUS.

I also have a really old single-CPU system, MIPS (obviously).  Not the fastest
on the block compared to the other equipment I've got, but does anyone know of
any simple timing scripts/programs available that can benchmark some of these
proposed digest hashes?  If they turn out to be reasonably quick on my old
machine, I doubt then that speed will be too much of an issue.

Also, for whatever hashes we ultimately go with, what are considerations for
the userland support for them on non-glibc systems?  E.g., are they provided by
third-party libraries or do they need implementations in
uclibc/uclibc-ng/musl/*?  And what about the Alt/BSD side of things?  I assume
FreeBSD implements this already, but worth verifying with all of the
combinations of arches/libc's/kernels and whatnot.  I mean, there still might
be a lonely m68k install out there...

-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
6144R/F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943

"The past tempts us, the present confuses us, the future frightens us.  And our
lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

Re: [gentoo-dev] Manifest2 hashes, take n+1-th: 3 hashes for the tie-breaker case

[Gordon Pettey]

[-- Attachment #1: Type: text/plain, Size: 922 bytes --]

On Sun, Nov 12, 2017 at 8:22 PM, Joshua Kinard <kumba@gentoo.org> wrote:

> Minor clarification, old single core //and// uni-processor.  Some older
> machines have multiple physical CPUs that are single-core.  Threading
> should be
> okay on these, as long as the thread count stays under NR_CPUS.
>
> I also have a really old single-CPU system, MIPS (obviously).  Not the
> fastest
> on the block compared to the other equipment I've got, but does anyone
> know of
> any simple timing scripts/programs available that can benchmark some of
> these
> proposed digest hashes?  If they turn out to be reasonably quick on my old
> machine, I doubt then that speed will be too much of an issue.
>

Even on your "old single-CPU MIPS" system, what percentage of time is
spent verifying manifest hashes compared to actually building/installing?
The whole "slow and/or multiple hashes will cause problems" argument
seems specious.

[-- Attachment #2: Type: text/html, Size: 1369 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th: 3 hashes for the tie-breaker case

[Joshua Kinard]

On 11/12/2017 22:48, Gordon Pettey wrote:
> On Sun, Nov 12, 2017 at 8:22 PM, Joshua Kinard <kumba@gentoo.org> wrote:
> 
>> Minor clarification, old single core //and// uni-processor.  Some older
>> machines have multiple physical CPUs that are single-core.  Threading
>> should be
>> okay on these, as long as the thread count stays under NR_CPUS.
>>
>> I also have a really old single-CPU system, MIPS (obviously).  Not the
>> fastest
>> on the block compared to the other equipment I've got, but does anyone
>> know of
>> any simple timing scripts/programs available that can benchmark some of
>> these
>> proposed digest hashes?  If they turn out to be reasonably quick on my old
>> machine, I doubt then that speed will be too much of an issue.
>>
> 
> Even on your "old single-CPU MIPS" system, what percentage of time is
> spent verifying manifest hashes compared to actually building/installing?
> The whole "slow and/or multiple hashes will cause problems" argument
> seems specious.

It appears that you have misread my inquiry quite severely.  I am not terribly
interested in the whole argument of "multiple hashes" or "slow hashes".  It's
just my curiosity wanting to know if there's a reliable way to benchmark, on
the command line, several common/known digest hashing algorithms.  For my MIPS
systems, the type of CPU and system architecture is very different between each
machine.  Being able to compare between each of them might have uses down the road.

But also having a rough idea of how the actual hashes perform might also be of
use to the discussion.  Only to enlighten, not to push a particular side.

-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
6144R/F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943

"The past tempts us, the present confuses us, the future frightens us.  And our
lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

Re: [gentoo-dev] Manifest2 hashes, take n+1-th: 3 hashes for the tie-breaker case

[Michał Górny]

W dniu nie, 12.11.2017 o godzinie 21∶22 -0500, użytkownik Joshua Kinard
napisał:
> On 10/24/2017 00:11, Michał Górny wrote:
> > W dniu wto, 24.10.2017 o godzinie 06∶04 +0200, użytkownik Michał Górny
> > napisał:
> 
> [snip]
> 
> > > > [BOBO06] is relevant research here, I cited it in the work that went into
> > > > GLEP59, the last time we updated the hashes. The less-technical explanation of it is:
> > > > "If you can express the output of H1(x)H2(x) in LESS bits than the combined
> > > > output size of H1,H2, then the attacks get a little bit easier"
> > > > 
> > > > Some important pieces from it:
> > > > [J04] "showed that the concatenation of two Merkle-Damgard functions is not
> > > > much more secure than the individual functions.", but this holds ONLY if
> > > > the hash functions chosen are of the same construction (MD).
> > > > Choosing hashes with different constructions (Merkle-Damgard, HAIFA,
> > > > Sponge) is important, and given a black box environment, 
> > > > 
> > > > The original mail reached the same approximate decision, just to look
> > > > for diverse hashes, but decided that 2 was enough.
> > > > 
> > > > Q: What are the odds of a simultaneous successful attack against two hashes? 
> > > > A: IDK, but if the hash functions are truly independent, it must be provably
> > > >    lower than the odds of an attack against a single hash.
> > > 
> > > We're talking about really huge (→∞) numbers here. It's not a 'random'
> > > attack against one hash. It's an attack that allows to sneak a malicious
> > > code with unchanged file size (since we store that too), and no apparent
> > > side effects (what's the point of spending up that much resources
> > > if the user is going to notice?).
> > > 
> > > > Q: What's the big difference between a bug and a successful attack in a hash?
> > > > A: Bugs are more likely initially, and attacks come later.
> > > 
> > > Sounds like an entirely abstract point in time ;-).
> > > 
> > > > All of that said, is there really a significant long-term gain in
> > > > multiple hashes? (setting aside the short-term advantage in a transition
> > > > period for changing hashes)
> > > 
> > > No, and that's my point. One hash is perfectly fine.
> > > 
> > > Two hashes are useful for transition purposes. If we take two fast
> > > hashes (e.g. proposed SHA512 + BLAKE2B which have similar speed),
> > > we can use 2 threads to prevent the speed loss (except for old single-
> > > core machines).
> 
> Minor clarification, old single core //and// uni-processor.  Some older
> machines have multiple physical CPUs that are single-core.  Threading should be
> okay on these, as long as the thread count stays under NR_CPUS.
> 
> I also have a really old single-CPU system, MIPS (obviously).  Not the fastest
> on the block compared to the other equipment I've got, but does anyone know of
> any simple timing scripts/programs available that can benchmark some of these
> proposed digest hashes?  If they turn out to be reasonably quick on my old
> machine, I doubt then that speed will be too much of an issue.

You could play with utils/benchmark.py inside gemato [1]. Note that it's
not very precise though but should give a rough measurement. Also note
that it is suited for one big file while we mostly deal with a lot of
small files and that changes things a bit.

[1]:https://github.com/mgorny/gemato

> Also, for whatever hashes we ultimately go with, what are considerations for
> the userland support for them on non-glibc systems?  E.g., are they provided by
> third-party libraries or do they need implementations in
> uclibc/uclibc-ng/musl/*?  And what about the Alt/BSD side of things?  I assume
> FreeBSD implements this already, but worth verifying with all of the
> combinations of arches/libc's/kernels and whatnot.  I mean, there still might
> be a lonely m68k install out there...

We've selected the hashes that are guaranteed to be included in CPython
3.6+. For older versions of Python, we are using the Python extension
based on the reference implementation (just like the code in CPython)
pyblake2.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Jason A. Donenfeld]

[-- Attachment #1: Type: text/plain, Size: 2431 bytes --]

Blake2 is in coreutils already, provides an excellent security margin, and
is considerably faster than both sha2 and sha3.

On Oct 19, 2017 21:09, "Michał Górny" <mgorny@gentoo.org> wrote:

> Hi, everyone.
>
> The previous discussion on Manifest2 hashes pretty much died away
> pending fixes to Portage. Since Portage was fixed a while ago, and we
> can now safely switch, I'd like to reboot the discussion before
> submitting the item for the next Council meeting.
>
> Considering all arguments made so far, I'd like to propose changing:
>
>   manifest-hashes = SHA256 SHA512 WHIRLPOOL
>
> to:
>
>   manifest-hashes = SHA512 SHA3_512
>
> In other words, removing SHA256 and WHIRLPOOL, and adding SHA3_512.
>
>
> Rationale
> ---------
>
> 1. The main argument for using multiple hashes is to prevent the (very
> unlikely) possibility that if a weakness is discovered in one of
> the hashes, the other would still hold. This is given by using two
> algorithms; more than two do not increase security significantly, while
> they do increase performance cost.
>
> 2. For the above to hold, the hashes should be diverse. SHA256
> and SHA512 are the same algorithm, so a weakness discovered in either
> would probably apply to both -- keeping both does not make sense at all.
> Furthermore, both SHA2 and WHIRLPOOL use the same construct (MD), so
> a weakness in the construct would apply to both.
>
> 3. Keeping one of the three old hashes is necessary for compatibility
> reasons. Furthermore, the current versions of Portage consider SHA512
> obligatory, so we can't remove it without redesigning Portage first
> (though I think this applies only to developer installs, i.e. those
> creating Manifests).
>
> 4. The new hashes that are stronger and commonly available are
> SHA3/Keccak (using sponges) and BLAKE2 (HAIFA). Both are diverse from
> our current algorithms, so either is a good candidate. The choice of
> Keccak is purely arbitrary (because it's the winner?).
>
> All the above considered, I think it's most reasonable to use two hashes
> with diverse constructs. SHA512 needs to be one of them, for
> compatibility reasons. The other could be either SHA3_512 or BLAKE2B,
> as a strong, future-proof hash. SHA3 is probably a better choice because
> it's going to have more support as the official recommendation.
>
> --
> Best regards,
> Michał Górny
>
>
>

[-- Attachment #2: Type: text/html, Size: 2898 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Michał Górny]

W dniu sob, 21.10.2017 o godzinie 04∶01 +0200, użytkownik Jason A.
Donenfeld napisał:
> Blake2 is in coreutils already, provides an excellent security margin, and
> is considerably faster than both sha2 and sha3.
> 

Yes, we've already switched the proposal to BLAKE2B. Although it is only
faster if you use a good implementation. The silly implementation
in Python is slower than SHA2.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Chí-Thanh Christopher Nguyễn]

Michał Górny schrieb:
> to:
> 
>   manifest-hashes = SHA512 SHA3_512

+1

Just wondering about the performance argument on weak systems:
Does Portage absolutely have to check all of the hashes or can it be
configured by the user to check only a subset of them?


Best regards,
Chí-Thanh Christopher Nguyễn

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Michał Górny]

W dniu sob, 21.10.2017 o godzinie 04∶08 +0200, użytkownik Chí-Thanh
Christopher Nguyễn napisał:
> Michał Górny schrieb:
> > to:
> > 
> >   manifest-hashes = SHA512 SHA3_512
> 
> +1
> 
> Just wondering about the performance argument on weak systems:
> Does Portage absolutely have to check all of the hashes or can it be
> configured by the user to check only a subset of them?
> 

Yes, it is required to check all hashes. As said on the other thread,
I'm not blocking anyone from adding another option but it's not going
tobe on by default.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Michał Górny]

Hi,

So here's my proposed plan, after considering all the replies.


Immediately after accepting
---------------------------

a. Revbump Portage to add pyblake2 dep (to ensure BLAKE2 is supported
on py<3.6) and request stabilizing this version.

b. Create a git update hook that rejects Manifest entries that contain
SHA512 only, to prevent a bug in current versions of Portage, that
causes it to skip BLAKE2 when no implementation is installed instead
of complaining [optional].


Now, let T = day when the new version is stable on amd64.


T + 7 days
----------

Set:

  manifest-hashes = BLAKE2B SHA512
  manifest-required-hashes = SHA512

New Manifest entries will use the new hashes but Portage will keep the
old hash set whenever it would need to refetch old distfiles.



T + 3 months
------------

Set:

  manifest-required-hashes = BLAKE2B

Portage will now request updating hashes for all files, including
old distfiles. We will start proactively updating Manifests here,
and file bugs for fetch-restricted packages.


T + 6 months
------------

All Manifests should use the new hashes by this time. The remaining
fetch-restricted packages should be last-rited.


T + 36 months
-------------

Set:

  manifest-hashes = BLAKE2B

Remove SHA512 from all Manifests.


-- 
Best regards,
Michał Górny

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 896 bytes --]

+1 overall, just one timeline clarification.

On Mon, Nov 06, 2017 at 05:58:21PM +0100, Michał Górny wrote:
> T + 7 days
> ----------
> Set:
>   manifest-hashes = BLAKE2B SHA512
>   manifest-required-hashes = SHA512
> 
> New Manifest entries will use the new hashes but Portage will keep the
> old hash set whenever it would need to refetch old distfiles.
Query:
Do we need to wait for it to be stable before making this change?
Shouldn't old stable versions of Portage continue to verify SHA512 fine?
Mostly I think devs need to be using a new enough Portage that can
generate the BLAKE2B entries, but it shouldn't impact user Portage
versions.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer
E-Mail   : robbat2@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 1113 bytes --]

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Mike Gilbert]

On Mon, Nov 6, 2017 at 2:13 PM, Robin H. Johnson <robbat2@gentoo.org> wrote:
> +1 overall, just one timeline clarification.
>
> On Mon, Nov 06, 2017 at 05:58:21PM +0100, Michał Górny wrote:
>> T + 7 days
>> ----------
>> Set:
>>   manifest-hashes = BLAKE2B SHA512
>>   manifest-required-hashes = SHA512
>>
>> New Manifest entries will use the new hashes but Portage will keep the
>> old hash set whenever it would need to refetch old distfiles.
> Query:
> Do we need to wait for it to be stable before making this change?
> Shouldn't old stable versions of Portage continue to verify SHA512 fine?
> Mostly I think devs need to be using a new enough Portage that can
> generate the BLAKE2B entries, but it shouldn't impact user Portage
> versions.

Quite a few devs use stable versions of Portage and Repoman when committing.

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Michał Górny]

W dniu pon, 06.11.2017 o godzinie 19∶13 +0000, użytkownik Robin H.
Johnson napisał:
> +1 overall, just one timeline clarification.
> 
> On Mon, Nov 06, 2017 at 05:58:21PM +0100, Michał Górny wrote:
> > T + 7 days
> > ----------
> > Set:
> >   manifest-hashes = BLAKE2B SHA512
> >   manifest-required-hashes = SHA512
> > 
> > New Manifest entries will use the new hashes but Portage will keep the
> > old hash set whenever it would need to refetch old distfiles.
> 
> Query:
> Do we need to wait for it to be stable before making this change?
> Shouldn't old stable versions of Portage continue to verify SHA512 fine?
> Mostly I think devs need to be using a new enough Portage that can
> generate the BLAKE2B entries, but it shouldn't impact user Portage
> versions.
> 

Devs are who I'm worried about. Those 7 days should give them enough
time to upgrade their stable Portage.

-- 
Best regards,
Michał Górny