[gentoo-dev] Need GitHub snapshot hash verification failure samples

[gentoo-dev] Need GitHub snapshot hash verification failure samples

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 510 bytes --]

Hi, everyone.

I've seen multiple bugs related to hash verification failures for GitHub
snapshots lately. However, none of the maintainers have been so far able
to provide me with a sample of the old and new snapshot for comparison,
so we still have no clue what's happening exactly.

if you see your package failing or get a report for it, then *please*
save the original tarball before replacing it with the new one and send
me both for comparison. Thank you.

-- 
Best regards,
Michał Górny

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 988 bytes --]

Re: [gentoo-dev] Need GitHub snapshot hash verification failure samples

[Vadim A. Misbakh-Soloviov]

By the way, that is "known issue" on github (I already discussed that with 
their support even few years ago). The answer was kinda "well, it can happen 
time to time, since we can upgrade software like tar and/or git on some or all 
of our servers and we never declared tarballs checksums similarity".

So, even if somebody will trigger that once more, I doub't github support will 
answer something different this time.

Re: [gentoo-dev] Need GitHub snapshot hash verification failure samples

[Sergei Trofimovich]

[-- Attachment #1: Type: text/plain, Size: 6451 bytes --]

On Wed, 05 Jul 2017 21:48:12 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> Hi, everyone.
> 
> I've seen multiple bugs related to hash verification failures for GitHub
> snapshots lately. However, none of the maintainers have been so far able
> to provide me with a sample of the old and new snapshot for comparison,
> so we still have no clue what's happening exactly.
> 
> if you see your package failing or get a report for it, then *please*
> save the original tarball before replacing it with the new one and send
> me both for comparison. Thank you.

Sounds easy to verify.
1. grab all the github tarballs (should be a better way to do it with proper USE expansiion):
    $ egrep -R 'SRC_URI.*github.com' metadata/ | grep -o '[^/ ]*$' | sort -u > github_distfiles.list
2. grab all manifest files that look like defining these files and remove them locally:
    $ git grep -l -F -f ./github_distfiles.list | grep -F /Manifest | xargs rm -v
3. Refetch distfiles from internets:
    $ mkdir /tmp/fresh
    $ GENTOO_MIRRORS= DISTDIR=/tmp/fresh repoman manifest

As a result each 'git diff' report is your potential candidate.
You have new file in /tmp/fresh/<file>
and old one on http://distfiles.gentoo.org/distfiles/<file>

A few samples:
--- a/app-admin/qtpass/Manifest
+++ b/app-admin/qtpass/Manifest
@@ -1,4 +1,4 @@
-DIST qtpass-1.0.5.tar.gz 636461 SHA256 0c07bd1eb9e5336c0225f891e5b9a9df103f218619cf7ec6311edf654e8db281
-DIST qtpass-1.1.0.tar.gz 671525 SHA256 60b458062f54184057e55dbd9c93958a8bf845244ffd70b9cb31bf58697f0dc6
+DIST qtpass-1.0.5.tar.gz 636457 SHA256 b9f1c1ecf4afbe716915792ff692e7114568de5bd8c47750d5c8404aa28699e7
+DIST qtpass-1.1.0.tar.gz 671537 SHA256 f2fff7922902c4c118e04164c078ca80e9a28221320b4253d3117d885e8417b6

diffoscope reports case change only in root dir name:

$ diffoscope old/qtpass-1.1.0.tar.gz new/qtpass-1.1.0.tar.gz
│   │ @@ -1,83 +1,83 @@
│   │ -drwxrwxr-x   0 root         (0) root         (0)        0 2016-01-25 09:58:18.000000 qtpass-1.1.0/
│   │ +drwxrwxr-x   0 root         (0) root         (0)        0 2016-01-25 09:58:18.000000 QtPass-1.1.0/
...

I guess somebody decided to rename github repo slightly.

Both files are at:

http://dev.gentoo.org/~slyfox/unstable_tarballs/old/qtpass-1.1.0.tar.gz
http://dev.gentoo.org/~slyfox/unstable_tarballs/new/qtpass-1.1.0.tar.gz

--- a/app-crypt/acme/Manifest
+++ b/app-crypt/acme/Manifest
@@ -1,3 +1,3 @@
 DIST certbot-0.14.1.tar.gz 851705 SHA256 7992fced742649e7b7668e4db7685de12248a4ffba66810cb336e9b6412e3567
 DIST certbot-0.15.0.tar.gz 942788 SHA256 87d306b1c013b472b8f548b38ccc476c125816435bb3b99e932fed09ac777296
-DIST letsencrypt-0.1.0.tar.gz 524821 SHA256 1c1ac7b41e5e0fc0e41a7ef159ac9147a4aafff54453d57b519eb05bf52ade14
+DIST letsencrypt-0.1.0.tar.gz 524854 SHA256 3ba1add217fc1665ad1d3c4812c0de60590f406cb83d6514332898ab60b26f62

$ diffoscope old/letsencrypt-0.1.0.tar.gz new/letsencrypt-0.1.0.tar.gz
│   │ @@ -1,579 +1,579 @@
│   │ -drwxrwxr-x   0 root         (0) root         (0)        0 2015-12-02 23:55:43.000000 letsencrypt-0.1.0/
│   │ +drwxrwxr-x   0 root         (0) root         (0)        0 2015-12-02 23:55:43.000000 certbot-0.1.0/

Same thing.


http://dev.gentoo.org/~slyfox/unstable_tarballs/old/letsencrypt-0.1.0.tar.gz
http://dev.gentoo.org/~slyfox/unstable_tarballs/new/letsencrypt-0.1.0.tar.gz

Zip file!

--- a/app-crypt/etcd-ca/Manifest
+++ b/app-crypt/etcd-ca/Manifest
@@ -1,2 +1,2 @@
-DIST etcd-ca-0_p20140903.zip 1178338 SHA256 5da9f7afad6dd373d96c5d36dd30e9f43cfc8fc2359bbf2d0c6a864fff139f81
+DIST etcd-ca-0_p20140903.zip 1178338 SHA256 7ef6b7f34324bd4b48b369990a7eb70e30809240f3c3d97b7d56d021af3f43f3

$ diffoscope old/etcd-ca-0_p20140903.zip new/etcd-ca-0_p20140903.zip
│  drwx---     0.0 fat        0 bx stor 14-Sep-03 21:30 etcd-ca-812f3626796be16d9db052720ce9c54f5a40bb26/
│ --rw----     0.0 fat       24 bx stor 14-Sep-03 21:30 etcd-ca-812f3626796be16d9db052720ce9c54f5a40bb26/.gitconfig
│ --rw----     0.0 fat     3924 bx defN 14-Sep-03 21:30 etcd-ca-812f3626796be16d9db052720ce9c54f5a40bb26/CONTRIBUTING.md
│ +-rw----     0.0 fat       24 tx stor 14-Sep-03 21:30 etcd-ca-812f3626796be16d9db052720ce9c54f5a40bb26/.gitconfig
│ +-rw----     0.0 fat     3924 tx defN 14-Sep-03 21:30 etcd-ca-812f3626796be16d9db052720ce9c54f5a40bb26/CONTRIBUTING.md

Here contents didn't change but zip compressor decided to pick different file type (bx/tx is binary/text).

http://dev.gentoo.org/~slyfox/unstable_tarballs/old/etcd-ca-0_p20140903.zip
http://dev.gentoo.org/~slyfox/unstable_tarballs/new/etcd-ca-0_p20140903.zip

--- a/app-emacs/lua-mode/Manifest
+++ b/app-emacs/lua-mode/Manifest
@@ -1 +1 @@
-DIST lua-mode-20130419.tar.gz 26236 SHA256 75c1696421983fbb58946ea649d2917f0deefc8b4f1dbc16b819e0cd603e396a
+DIST lua-mode-20130419.tar.gz 26242 SHA256 7a5e1a21e53aeab6e7cad8c616f6b026fd32f414bc6a32371e04d4e7424800c7

This one is different. Tag expansion changed (on GitHub's side?):

$ diffoscope old/lua-mode-20130419.tar.gz new/lua-mode-20130419.tar.gz  | lv

│   ├── lua-mode-rel-20130419/lua-mode.el
│   │ @@ -31,15 +31,15 @@
│   │  ;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
│   │  ;; MA 02110-1301, USA.
│   │  
│   │  ;; Keywords: languages, processes, tools
│   │  
│   │  ;; This field is expanded to commit SHA, date & associated heads/tags during
│   │  ;; archive creation.
│   │ -;; Revision: 040bc8f (Fri, 19 Apr 2013 11:27:32 +0400  (rel-20130419))
│   │ +;; Revision: 040bc8f (Fri, 19 Apr 2013 11:27:32 +0400  (tag: rel-20130419))
│   │  ;;
│   │  
│   │  ;;; Commentary:
│   │  
│   │  ;; Thanks to d87 <github.com/d87> for an idea of highlighting lua
│   │  ;; builtins/numbers
│   │  
│   ╵

http://dev.gentoo.org/~slyfox/unstable_tarballs/old/lua-mode-20130419.tar.gz
http://dev.gentoo.org/~slyfox/unstable_tarballs/new/lua-mode-20130419.tar.gz

--- a/app-emulation/docker/Manifest
+++ b/app-emulation/docker/Manifest
@@ -1,3 +1,3 @@
-DIST docker-17.03.1.tar.gz 7773296 SHA256 a8f1eefadf3966885ad0579facfc2017cca7dd3a0b20d086dfd798168716cb83
+DIST docker-17.03.1.tar.gz 7773988 SHA256 411e32ee388ad6d99479b97a3937c851bd84dacf4267be9d5501665e468e148e

$ diffoscope old/docker-17.03.1.tar.gz new/docker-17.03.1.tar.gz


-- 

  Sergei

[-- Attachment #2: Цифровая подпись OpenPGP --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Need GitHub snapshot hash verification failure samples

[William L. Thomson Jr.]

[-- Attachment #1: Type: text/plain, Size: 192 bytes --]

For anyone interested in such, I opened a feature request bug for
allowing use of sets in profile packages. 

https://bugs.gentoo.org/show_bug.cgi?id=624300

-- 
William L. Thomson Jr.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]