[gentoo-dev] Updating all Manifest to contain SHA256 SHA512 WHIRLPOOL

[gentoo-dev] Updating all Manifest to contain SHA256 SHA512 WHIRLPOOL

[Justin (jlec)]

[-- Attachment #1: Type: text/plain, Size: 823 bytes --]

Hello,

there are quite a number of Manifest still not containing one or more of the
three hashes. I would like to update them as far as we can download the sources.

Procedure would be:
1. Download package
2. verify current hashes match
3. Calculate new
4. commit

Following question need to be answered first:

Does anybody have any general objections, remarks or ideas on that?

2.
Any suggestion how to do this? repoman has a manifest-check function but that is
not functioning (bug filed). Any other tool around? Perhaps using pkgcheck?

Ugly hack would be, tampering the downloaded sources and run repoman manifest
which would redownload the tarball and check again.

4.
What do you think is the best commit mode? PKG based, Cat based or repo based?


Thanks for your comments,
Justin



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 951 bytes --]

Re: [gentoo-dev] Updating all Manifest to contain SHA256 SHA512 WHIRLPOOL

[hasufell]

On 09/18/2015 10:58 AM, Justin (jlec) wrote:
> 
> 4.
> What do you think is the best commit mode? PKG based, Cat based or repo based?
> 

Repo based, don't bother with hundreds of commit messages. It's all
about the same problem.

OpenPGP verification of source files (was: Re: [gentoo-dev] Updating all Manifest to contain SHA256 SHA512 WHIRLPOOL)

[Kristian Fiskerstrand]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 09/18/2015 10:58 AM, Justin (jlec) wrote:
> Hello,
> 
> there are quite a number of Manifest still not containing one or
> more of the three hashes. I would like to update them as far as we
> can download the sources.
> 
> Procedure would be: 1. Download package 2. verify current hashes
> match 3. Calculate new 4. commit
> 
> Following question need to be answered first:
> 
> Does anybody have any general objections, remarks or ideas on
> that?

As long as the current hashes are verified for the download I'm fine
with this, but I'd like to take the opportunity to bring up a general
note with regards to manifest generation and OpenPGP verification of
source files.

Now that we're hopefully getting closer to a fully signed OpenPGP
Gentoo Tree, it is also important that package maintainers pay
attention to OpenPGP signatures when generating the initial manifest
files e.g. on a version bump. This also brings up some interesting key
management issues with regards to ensuring that the package is signed
with the correct key. Of course, where the maintainer has met the
developer and cross-signed the keys, this part is relatively easy, as
the key will have full validity or can be easily verified by one hope
distance.

Where this becomes more difficult is of course where no direct
certification has been made, leading into more probabilistic
approaches to determining key validity. I would expect maintainers of
a package following the mailing lists giving a high expectation of the
key being correct, and as such keeping a local copy of the keys used
for distribution with a local signature (lsign in GnuPG's edit-key
interface) marking this key as valid.

We currently don't (well, I don't at least) store information about
the file verification in the git commit messages, and I'm not sure if
this is something that would be valuable exceeding the cost of the
added message and finding a format to do so. But given that we're
talking about the manifests, I do sincerely hope package maintainers
have a well thought out setup for key management locally and in fact
verify the OpenPGP signatures vs known good keys, and that appropriate
measures are being taken in the case of non-maintainer commits that
doesn't reduce the level of security.

- -- 
Kristian Fiskerstrand
Public PGP key 0xE3EDFAE3 at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJV+9Z8AAoJECULev7WN52FgucH/jN6bwIe/AJuv6y2VkVC7gT2
pdtZY4hEv2TlVJUcGKgMfk5BWD2vm0vBdOCTwyPMgNXf+fnXv70507RmReecRiyB
ouVgacu1nQYMCG2urvuQckXPdGfycbgk0ESe+XcKbRnOmmJ2a4ZVKENXk0TbA38Y
hJ/c2boxpJiVZHF6JSPwfXBrC0j6GpRsLnce/vKybH0uDye4/7Q1Hw9R76KQDATd
DB+hcAsQfonj7rDy4FoKviuiSiZmbHam0yCQGiBaR2fqQOc+erSJ29Hy+MLkdCCa
Zy36sUv299u71J/9LYXuQBpeULV0XQ82ERz1VuJ6SV4YPYRtroqoKmnasA77Prw=
=bV4C
-----END PGP SIGNATURE-----

Re: OpenPGP verification of source files (was: Re: [gentoo-dev] Updating all Manifest to contain SHA256 SHA512 WHIRLPOOL)

[Rich Freeman]

On Fri, Sep 18, 2015 at 5:16 AM, Kristian Fiskerstrand <k_f@gentoo.org> wrote:
> I do sincerely hope package maintainers
> have a well thought out setup for key management locally and in fact
> verify the OpenPGP signatures vs known good keys, and that appropriate
> measures are being taken in the case of non-maintainer commits that
> doesn't reduce the level of security.

I'd be utterly shocked if even 30% of maintainers are checking
upstream gpg keys when doing new releases.  I'm sure it happens
sometimes.

I'd suggest adding it to the DCO when we actually have a DCO, though
that doesn't actually ensure that anybody follows it.  And the wording
would have to be careful since not all upstreams even sign their
releases at all, and if they do many/most maintainers probably haven't
personally verified the keys.  I certainly haven't met the upstream
developers of any of the packages I maintain in-person - I haven't
even met another Gentoo dev in-person.

-- 
Rich

Re: [gentoo-dev] Updating all Manifest to contain SHA256 SHA512 WHIRLPOOL

[Robin H. Johnson]

On Fri, Sep 18, 2015 at 10:58:22AM +0200, Justin (jlec) wrote:
> Hello,
> 
> there are quite a number of Manifest still not containing one or more of the
> three hashes. I would like to update them as far as we can download the sources.
540 of 17841 Manifest files have the problem, about 3%.
Quick way to find them:
find -name Manifest |xargs egrep -v '^DIST .* [0-9]+ (SHA256|SHA512|WHIRLPOOL)' |cut -d: -f1|uniq

> Procedure would be:
> 1. Download package
> 2. verify current hashes match
> 3. Calculate new
> 4. commit
> 
> Following question need to be answered first:
> 
> Does anybody have any general objections, remarks or ideas on that?
Good luck finding some of them; there are fetch-restricted packages in
that set.

> 2.
> Any suggestion how to do this? repoman has a manifest-check function but that is
> not functioning (bug filed). Any other tool around? Perhaps using pkgcheck?
> 
> Ugly hack would be, tampering the downloaded sources and run repoman manifest
> which would redownload the tarball and check again.
I tested, and this worked:
ebuild ... fetch
ebuild ... manifest
tested with dev-scheme/hop/hop-2.0.1.ebuild.

The distfile was downloaded, verified, and the Manifest got updated
correctly.

> 4.
> What do you think is the best commit mode? PKG based, Cat based or repo based?
repo-based means less commits, but I don't know if the portage changelog
generation will handle it correctly.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-dev] Updating all Manifest to contain SHA256 SHA512 WHIRLPOOL

[Justin Lecher (jlec)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 20/09/15 19:41, Robin H. Johnson wrote:
> On Fri, Sep 18, 2015 at 10:58:22AM +0200, Justin (jlec) wrote:
>> Hello,
>> 
>> there are quite a number of Manifest still not containing one or
>> more of the three hashes. I would like to update them as far as
>> we can download the sources.
> 540 of 17841 Manifest files have the problem, about 3%. Quick way
> to find them: find -name Manifest |xargs egrep -v '^DIST .* [0-9]+
> (SHA256|SHA512|WHIRLPOOL)' |cut -d: -f1|uniq
> 
>> Procedure would be: 1. Download package 2. verify current hashes
>> match 3. Calculate new 4. commit
>> 
>> Following question need to be answered first:
>> 
>> Does anybody have any general objections, remarks or ideas on
>> that?
> Good luck finding some of them; there are fetch-restricted packages
> in that set.
> 

yeah, I know. I will try what I can get and will put some information
up on the remaining packages. When we have an overview, we can act.

>> 2. Any suggestion how to do this? repoman has a manifest-check
>> function but that is not functioning (bug filed). Any other tool
>> around? Perhaps using pkgcheck?
>> 
>> Ugly hack would be, tampering the downloaded sources and run
>> repoman manifest which would redownload the tarball and check
>> again.
> I tested, and this worked: ebuild ... fetch ebuild ... manifest 
> tested with dev-scheme/hop/hop-2.0.1.ebuild.
> 
> The distfile was downloaded, verified, and the Manifest got
> updated correctly.

thanks for confirming. I will do an extra check that the present
sha256 sum didn't change.


thanks for the comment,

Justin
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0

iQJ8BAEBCgBmBQJV/vY1XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ0QUU0N0I4NzFERUI0MTJFN0EyODE0NUFF
OTQwMkE3OUIwMzUyOUEyAAoJEOlAKnmwNSmiZb8QAKhAGKo1gKZxILFyhU+Az5l3
VsJeJF6F0ZnclgUQSDKaqY06VxTjsG1W9SUDx0VYrqfJcSkIODZ55RjCio22f+tP
6DA5z6sTwcBOcRIunWjjWD4rHrSgHj/HodzShFnTdSt8R6hzFumZWRaFXfN1DUj8
nibH3n1o6flAm8eV7ZxbypZ5Y1QOtFqxJc6fuHLeeSsJwu1iph/OnNagdx3ZCU29
RsiKxIjjWDLtew2BWCSSNTD7ORLNmogTis5xRAuQXevvwZ+wtXGhUNQmb9tlWkHW
O0ndCVtH3+yiEPn+s+2TRbnYE0KE2Ce4vnGsmq1zuf0zzwXhKmgPNBLVihHU83nF
M0bWPvqM7bdIGKZmdhfG4HOY0TE/9HZ7SHgr7u3OTgy+cqvouDoQf3XROKO9cgRQ
iVMsNwjWJe4wZs5LAfgq0ancNZKHN0iorp0nDbE/cvkMUf5MkB/v3fz/C7XnYcFg
54f7uluO2NQDv6852iGE4usYECQByHdZ+eU0ZPyPw/elrGbhaaBN+h4SBoVd6G4d
vgpUTBCO1pVSZV2Rwx2AeDEin0An9r4+s2wHi0IXI/p0yQbLQ+rHCQpOpK72HrdH
5O8dVr7zNiI96Qd5iN04Q0gNyTrmaSAevsNdhnLxouwlclKhtRY7tirlcEZQ7t4V
Tlv5pykl8ITH9g5IS1yf
=dwyV
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Updating all Manifest to contain SHA256 SHA512 WHIRLPOOL

[Tim Harder]

[-- Attachment #1: Type: text/plain, Size: 738 bytes --]

On 2015-09-18 04:58, Justin (jlec) wrote:
> 2.
> Any suggestion how to do this? repoman has a manifest-check function but that is
> not functioning (bug filed). Any other tool around? Perhaps using pkgcheck?

With regards to pkgcheck, run the following in a configured gentoo repo
to generate a list of missing checksums in Manifest files:

pkgcheck -c repo_metadata

or a variation where the current working directory doesn't matter:

pkgcheck -c repo_metadata -r gentoo '*'

Note that this will probably output a bunch of metadata exceptions as
well since the tree has a bunch of ebuilds with unknown mirrors in it
(mostly berlios and bitbucket) and running the above disables the check
for them so they show up as errors instead.

Tim

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [gentoo-dev] Updating all Manifest to contain SHA256 SHA512 WHIRLPOOL

[Justin Lecher (jlec)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi there,

I did a first fix run.

https://github.com/jlec/gentoo/commit/0df86dcca0aa981fa7bdba633653697e2b
40781c

Although my script checks whether the size and SHA256 changed, but
better you could also take a look.

Thanks,
Justin
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0

iQJ8BAEBCgBmBQJWAaq2XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ0QUU0N0I4NzFERUI0MTJFN0EyODE0NUFF
OTQwMkE3OUIwMzUyOUEyAAoJEOlAKnmwNSmiu4oQALR2JdGsAYF9UoTefwH6pic6
LZQWlkiMrvAY8SvsE6g9gW9IDzSzuZtbi4EgN0nkvYWskN9KlyUyMOQH/snbpAsd
eiijG98pFNqEa4nBJBDuBUjPPjZ8M30i6/dsHMEW3HCgi73Va1Rr/6U5l+eO111b
W578EhpAqcIhRenr9ssbUEA0EdTKsZYdbtpZDGrrXy9+Zw0ZOwAXqQyeylML1kJN
usr08KHyqi9VMom19G1EeENO9XTKU9hHoCkHjSnfsl/Zi3KJT7VQXhBLIEcxuX8m
WeGQ+tGA0Ih1G1zNfepUle3eCV1gaXu6h+m+wUDk2enMCJPIFtDUbI/gDOvf3Gny
uZhl9MHFUcYuGiroAXHiZUEGvvhjDMXwciUavvsR/813L+uqt+rgiDA7YMNmVjeV
dACcaxH76zHweHQh5IKm4V63a2cuwLvpq6WVviiwUPNrI32B4C8WAM79SfbQDILL
Il4QJvTYJKLfgcz2RCsnfBIT5jdvrmJLD6yozGpHP8/EaBtiAhvtAkT5H7JTv8yv
/9QR5rMoEQczeJGJVV+nPvSePZs1WxMAP5RraElL/Jdu7nnZf++WD/H9YmnvXZfi
TJqOyETEJ+tisyx5z/EXaBAS0hZVl7Hswu1gItZoiDEF0FdOgvJU0tJA/rOSftmB
Nv4KyknRPPm8pwcAn4z8
=WapW
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Updating all Manifest to contain SHA256 SHA512 WHIRLPOOL

[Tim Harder]

[-- Attachment #1: Type: text/plain, Size: 435 bytes --]

On 2015-09-22 15:23, Justin Lecher (jlec) wrote:
> https://github.com/jlec/gentoo/commit/0df86dcca0aa981fa7bdba633653697e2b
> 40781c

> Although my script checks whether the size and SHA256 changed, but
> better you could also take a look.

You could open a pullreq against the gentoo github repo and I think
pkgcheck will be automatically run against it. Then look if there are
any remaining MissingChksum reports in the output.

Tim

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [gentoo-dev] Updating all Manifest to contain SHA256 SHA512 WHIRLPOOL

[Justin (jlec)]

[-- Attachment #1: Type: text/plain, Size: 1009 bytes --]

On 22/09/15 22:16, Tim Harder wrote:
> On 2015-09-22 15:23, Justin Lecher (jlec) wrote:
>> https://github.com/jlec/gentoo/commit/0df86dcca0aa981fa7bdba633653697e2b
>> 40781c
> 
>> Although my script checks whether the size and SHA256 changed, but
>> better you could also take a look.
> 
> You could open a pullreq against the gentoo github repo and I think
> pkgcheck will be automatically run against it. Then look if there are
> any remaining MissingChksum reports in the output.
> 
> Tim
> 

Hi Tim,

I checked it using the pkgcheck you posted in your earlier mail. There are still
some [1] packages missing all digests. Mostly they are fetch restricted, but
some game ebuilds with mirror restrictions exhibit also problems during fetch.

Perhaps individual maintainers can look into the missing hashes and update where
ever they have either access or already downloaded the sources and update the
hashes.

Justin

1)
http://dev.gentoo.org/~jlec/paste/manifests-150922.html


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 951 bytes --]