[gentoo-user] package download verification

[gentoo-user] package download verification

[James]

So,

Since (forever) I have manually checked the .Digest and such using
openssl or gpg, not unlike what is in the gentoo handbook.

This is retarded, and I'm too old to do that now, so I went shopping
for some script/tool/code to do it for me. I sure that a sinlple
script with diff would be sufficient to compare the download hash
against the one openssl generates... In fact, I do not know
why the integrity check is not fully integrated into ftp. rsync.
or whatever the download tool is?

If futher suspicion warrants, I can always perform a manual spot check,
but some integrated integrity should be part of the download process?



But why not just use a simple script:

<scriptname> package.just.downloaded package.just.downloaded.DIGESTS

and have it return:

<ok or match or corrupted>

After all this is intuitively obviously, when I burn a cd/dvd
and is an integrated option.

???

So I found this python script  "verify.py"

https://bbs.archlinux.org/viewtopic.php?id=83839


Sure there is a slicker, newer, better  scheme?
Pardon my (lazy) ignorance here..... 


James

Re: [gentoo-user] package download verification

[Alan McKinnon]

On 07/05/2014 07:23, James wrote:
> So,
> 
> Since (forever) I have manually checked the .Digest and such using
> openssl or gpg, not unlike what is in the gentoo handbook.
> 
> This is retarded, and I'm too old to do that now, so I went shopping
> for some script/tool/code to do it for me. I sure that a sinlple
> script with diff would be sufficient to compare the download hash
> against the one openssl generates... In fact, I do not know
> why the integrity check is not fully integrated into ftp. rsync.
> or whatever the download tool is?
> 
> If futher suspicion warrants, I can always perform a manual spot check,
> but some integrated integrity should be part of the download process?
> 
> 
> 
> But why not just use a simple script:
> 
> <scriptname> package.just.downloaded package.just.downloaded.DIGESTS
> 
> and have it return:
> 
> <ok or match or corrupted>
> 
> After all this is intuitively obviously, when I burn a cd/dvd
> and is an integrated option.
> 
> ???
> 
> So I found this python script  "verify.py"
> 
> https://bbs.archlinux.org/viewtopic.php?id=83839
> 
> 
> Sure there is a slicker, newer, better  scheme?
> Pardon my (lazy) ignorance here..... 
> 
> 
> James
> 
> 
> 
> 


Perhaps I'm just old and retarded myself, but portage already does what
you want. I edited the hashes in the Manifest file for something in my
local overlay and tried to emerge it. Here's what I got:

# emerge -1 mysql-refman
Calculating dependencies... done!

>>> Verifying ebuild manifests

>>> Emerging (1 of 1) app-doc/mysql-refman-5.5::alan
!!! Previously fetched file:
'/var/distfiles/refman-5.5-en.html-chapter.tar.gz'
!!! Reason: Failed on SHA256 verification
!!! Got:
2eb9f21b4bc88b89a05e28b8a25ec221d36677ee13f2733c1dd1d0d28e81ad0d
!!! Expected:
2eb9f21b4bc88b89a05e28b8a25ec221d36677ee13f2733c1dd1d0d28e81ad0e
Refetching... File renamed to
'/var/distfiles/refman-5.5-en.html-chapter.tar.gz._checksum_failure_.1s4y_D'


In this case I had the download files already in distfiles, portage
however applies the same check if it has to download things first.


-- 
Alan McKinnon
alan.mckinnon@gmail.com

Re: [gentoo-user] package download verification

[Michael Orlitzky]

On 05/07/2014 08:59 AM, Alan McKinnon wrote:
> 
>>>> Verifying ebuild manifests
> 
>>>> Emerging (1 of 1) app-doc/mysql-refman-5.5::alan
> !!! Previously fetched file:
> '/var/distfiles/refman-5.5-en.html-chapter.tar.gz'
> !!! Reason: Failed on SHA256 verification
> !!! Got:
> 2eb9f21b4bc88b89a05e28b8a25ec221d36677ee13f2733c1dd1d0d28e81ad0d
> !!! Expected:
> 2eb9f21b4bc88b89a05e28b8a25ec221d36677ee13f2733c1dd1d0d28e81ad0e
> Refetching... File renamed to
> '/var/distfiles/refman-5.5-en.html-chapter.tar.gz._checksum_failure_.1s4y_D'

This relies on two things. First, that the maintainer got the right
tarball and actually verified the upstream signature (one can hope).
Second, that the manifest you got wasn't modified by an evil mirror.

It's possible for maintainers to sign the manifest with their GPG keys,
but not required at the moment. Once signed manifests are ubiquitous,
we'll be able to automatically verify the signatures... somehow.

There are other problems though. Like the fact that the eclasses are
unsigned, and can do whatever they want to an ebuild. There are GLEPs
for some of this stuff, and 63 was just finalized, but I'm not sure
about the state of the rest of them.

[gentoo-user] Re: package download verification

[James]

Alan McKinnon <alan.mckinnon <at> gmail.com> writes:


> > This is retarded, and I'm too old to do that now, so I went shopping
> > for some script/tool/code to do it for me. In fact, I do not know
> > why the integrity check is not fully integrated into ftp. rsync.
> > or whatever the download tool is?


> Perhaps I'm just old and retarded myself, but portage already does what
> you want. 


Well certainly portage is one common methods to download, and we can explore
that thread. But, I was thinking more general. Late last night (too late) I
decided to download 'lilblue'. I poked around on several gentoo mirrors and
could not find it. So with my google hat on, I found it on a non typical
(mirror) server. The download was slow (300K) so naturally, I became
suspicious. I checked manually, but it was late and I was tired.....


The download was kicked off from the web browser (seamonkey). Now that I
think about it, there are a myriad of ways to download sources. What I was
suggesting (inquiring?) is that a command line tool could be readily
developed (if it did not already exist) to simple check any download
against the published data (keys/hashes/etc) depending on what is in the
local dir where the download lands (is stored). It could be used with
protage files too. 
But why not just use a simple script:

<scriptname> package.just.downloaded package.just.downloaded.DIGESTS



But then I got to questioning the integrity of both the downloaded sources
and the digest originating on the same server........ Probably not a good
idea either? So the digest should come from elsewhere? Maybe pull the digest
from a certificated (pontificated?) (gentoo controlled) server and not
somebody's (low priority managed) public server. Or maybe a master list of
digests (hashes) could be included on every (hardened) gentoo box?



It seems *everything is hacked* now. Certainly the NSA has fessed up to that
as have others.  Sure it may be just "good business" but the brightest minds
now days are mostly focused on security comprimises, particularly offensive
strategies, imho. So it seems to me, there is probably a "fly in the
ointment" common to what everyone is doing on a semi regular basis. To me
this sort of (justified/unjustified) paranoia should be incorporated into
the entire "hardened" effort at gentoo, imho, if not on a wider basis.


So please continue the "protage" thread discussion, but also a wider thread
concerning other source downloads. Afterall, *if" you can inject* into
sources, which are then compiled, who checks under the under_garments? If
you read about "The rat" the most secure implementation had/has tainted it's
very soul. [1]



curiously,
James

[1]
http://arstechnica.com/information-technology/2014/04/openssl-code-beyond-repair-claims-creator-of-libressl-fork/

Re: [gentoo-user] Re: package download verification

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 1155 bytes --]

On Wednesday 07 May 2014 15:12:53 James wrote:

> So please continue the "protage" thread discussion, but also a wider thread
> concerning other source downloads. Afterall, *if" you can inject* into
> sources, which are then compiled, who checks under the under_garments? 

Ha!  You need to go a few clicks back, or should I say under?  What if the 
hash algo itself is borked and collisions are becoming accepted?  What if the 
RNG you use on your PC is either backdoored by Intel (if hardware generated), 
or it has such a low entropy that it is trivial to crack its algorithmic 
derivatives.

I was quite surprised to see that the random pool available on a laptop I was 
working on at the time, was exceedingly lower than the 4096 max entropy.

Try this to see yours:  cat /proc/sys/kernel/random/entropy_avail

I now run sys-apps/haveged in the background, at least when I am generating 
ssl/gpg/ssh keys.


> [1]
> http://arstechnica.com/information-technology/2014/04/openssl-code-beyond-r
> epair-claims-creator-of-libressl-fork/

Useful to know someone is cleansing the code.  Thanks for sharing!

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-user] Re: package download verification

[Alan McKinnon]

On 07/05/2014 16:12, James wrote:
> Alan McKinnon <alan.mckinnon <at> gmail.com> writes:
> 
> 
>>> This is retarded, and I'm too old to do that now, so I went shopping
>>> for some script/tool/code to do it for me. In fact, I do not know
>>> why the integrity check is not fully integrated into ftp. rsync.
>>> or whatever the download tool is?
> 
> 
>> Perhaps I'm just old and retarded myself, but portage already does what
>> you want. 
> 
> 
> Well certainly portage is one common methods to download, and we can explore
> that thread. But, I was thinking more general. Late last night (too late) I
> decided to download 'lilblue'. I poked around on several gentoo mirrors and
> could not find it. So with my google hat on, I found it on a non typical
> (mirror) server. The download was slow (300K) so naturally, I became
> suspicious. I checked manually, but it was late and I was tired.....
> 
> 
> The download was kicked off from the web browser (seamonkey). Now that I
> think about it, there are a myriad of ways to download sources. What I was
> suggesting (inquiring?) is that a command line tool could be readily
> developed (if it did not already exist) to simple check any download
> against the published data (keys/hashes/etc) depending on what is in the
> local dir where the download lands (is stored). It could be used with
> protage files too. 
> But why not just use a simple script:
> 
> <scriptname> package.just.downloaded package.just.downloaded.DIGESTS
> 
> 
> 
> But then I got to questioning the integrity of both the downloaded sources
> and the digest originating on the same server........ Probably not a good
> idea either? So the digest should come from elsewhere? Maybe pull the digest
> from a certificated (pontificated?) (gentoo controlled) server and not
> somebody's (low priority managed) public server. Or maybe a master list of
> digests (hashes) could be included on every (hardened) gentoo box?
> 
> 
> 
> It seems *everything is hacked* now. Certainly the NSA has fessed up to that
> as have others.  Sure it may be just "good business" but the brightest minds
> now days are mostly focused on security comprimises, particularly offensive
> strategies, imho. So it seems to me, there is probably a "fly in the
> ointment" common to what everyone is doing on a semi regular basis. To me
> this sort of (justified/unjustified) paranoia should be incorporated into
> the entire "hardened" effort at gentoo, imho, if not on a wider basis.
> 
> 
> So please continue the "protage" thread discussion, but also a wider thread
> concerning other source downloads. Afterall, *if" you can inject* into
> sources, which are then compiled, who checks under the under_garments? If
> you read about "The rat" the most secure implementation had/has tainted it's
> very soul. [1]
> 
> 
> 
> curiously,
> James
> 
> [1]
> http://arstechnica.com/information-technology/2014/04/openssl-code-beyond-repair-claims-creator-of-libressl-fork/

Thanks, now I understand better the question you are asking.

I don't think it can be solved at all in the general case, for two reasons.

One, the internet and it's core protocols are inherently not worthy of
trust. There just isn't any way to prove that traffic is what it claims
to be and no crypto verification built into the core of it. You either
trust the traffic or you don't, but there's nothing inherent in the
traffic to help you decide. So, all the download protocols have security
checking bolted on afterwards by individual apps. These apps may or may
not be compatible with each other and may or may not do their checks
similarly from one protocol to the next. Somebody would have to garner
enough support so that all the major projects doing file and data
transfers agree on some way to implement crypto checks. Good luck with
that :-) if they do agree on something, we have the second problem.

Internet downloads have an inherent problem - you download an unknown
bunch of bits from somewhere and can't fully trust the result. You can
check hashes against the downloaded file, but you have to get them from
somewhere. And the method to get them is the same as getting the data
file itself - a bunch of bits from somewhere and you can't trust it. How
can you download trusted hash data from a source where you don't trust
the regular downloads? Can't work; two no trusts don't make a one trust.

And who's global hash store of all known hashes of all known
downloadables would you trust anyway? The NSAs? :-)

Best you can do is make something for the specific case. The Gentoo tree
and distfiles can be GPG signed and if you agree to trust Gentoo's keys
then you are good to go and it can be automated (which is the easy bit
btw).

For the general case/ I can't see that work at all. I trust Gentoo with
Gentoo, but I don't see myself ever trusting $ARB_3RD_PARTY with $EVERYTHING

-- 
Alan McKinnon
alan.mckinnon@gmail.com

[gentoo-user] Re: package download verification

[James]

Mick <michaelkintzios <at> gmail.com> writes:

> What if the 
> RNG you use on your PC is either backdoored by Intel (if hardware  
> generated),  or it has such a low entropy that it is trivial to 
> crack its algorithmic derivatives.

Rest easy here. ALL commercial hardware is "backdoor" at the silicon layer, 
not only by US interests, but various others, with extreme amounts of 
financial resources. That is a whole other topic. If you want to fix that,
you'd better plan on building up, from a FPGA or such. [1]


I usually do not work about such powerful forces as they usually "police"
their own. If you are part of an anarchy, terrorist to looking to supplant
those folks, then you have to worry about them. I'm more concerned with the
petty criminals, interlopers, and script_kiddies who destroy things for fun.
So hardwware comprises, although fully acknowledged, are of little concern
to me, as they are closely managed by folks with a very limited scope of
usage. Furthermore, the way they propage their (digitally undetecable, low
bandwidwth) information pretty much makes them immune from exploitation by
the pecker_heads (hacking commmunity without access to billions of dollars
nor Rf signal intercept resources). It's pretty much the domain of a few
dozen "nation states".



> I was quite surprised to see that the random pool available on a 
> laptop I was working on at the time, was exceedingly lower than 
> the 4096 max entropy.
> Try this to see yours:  cat /proc/sys/kernel/random/entropy_avail
> 
> I now run sys-apps/haveged in the background, at least when I am 
> generating ssl/gpg/ssh keys.

Interesting [2]

Do you have a formal document/wiki that explains it's usage in some detail?
Some further discussion on it's usage and verification would be interested.
Maybe "haveged " should have it's own page on the gentoo wiki?
Do tell more on this. my FX-8350 came back with:
entropy_avail 2188. It seems low and I would think that it is fixable
in the kernel sources?

Do tell me more on entropy, or anyone else that can delineate this
entropy further......?

> > [1]
> > http://arstechnica.com/information-technology/2014/04/openssl-
> > code-beyond-repair-claims-creator-of-libressl-fork/
> 
> Useful to know someone is cleansing the code.  Thanks for sharing!


The Rat is a very interesting humanoid. He has worked both sides of the
fence and is a brilliant coder; idolized my some (many?) young pups...... [3]



[1] http://opencores.org/

[2] http://www.issihosts.com/haveged/history.html

[3] http://www.theos.com/deraadt/

[gentoo-user] Re: package download verification

[James]

Alan McKinnon <alan.mckinnon <at> gmail.com> writes:

> > But why not just use a simple script:

> > <scriptname> package.just.downloaded package.just.downloaded.DIGESTS


Right now, I perform manual inspections, which are essential only if deemed
essential, proned to (visual inspection) mistakes  and time consuming. It
there is  (which I'm unaware of) scripts, programs, gui-interfaces and such
that greatly simplify this manual spot checking random approach?



> >
http://arstechnica.com/information-technology/2014/04/openssl-code-beyond-repair-claims-creator-of-libressl-fork/
> 
> Thanks, now I understand better the question you are asking.


Ok, cleaning up of this tool (openssl code) is but one part of the work that
 needs to be done. The Rat is well qualified to clean up this code.


> I don't think it can be solved at all in the general case, 
> for two reasons.

ems fight'n words......


> One, the internet and it's core protocols are inherently not worthy of
> trust. There just isn't any way to prove that traffic is what it claims
> to be and no crypto verification built into the core of it. You either
> trust the traffic or you don't, but there's nothing inherent in the
> traffic to help you decide. So, all the download protocols have security
> checking bolted on afterwards by individual apps. These apps may or may
> not be compatible with each other and may or may not do their checks
> similarly from one protocol to the next. Somebody would have to garner
> enough support so that all the major projects doing file and data
> transfers agree on some way to implement crypto checks. Good luck with
> that  if they do agree on something, we have the second problem.
> 
> Internet downloads have an inherent problem - you download an unknown
> bunch of bits from somewhere and can't fully trust the result. You can
> check hashes against the downloaded file, but you have to get them from
> somewhere. And the method to get them is the same as getting the data
> file itself - a bunch of bits from somewhere and you can't trust it. How
> can you download trusted hash data from a source where you don't trust
> the regular downloads? Can't work; two no trusts don't make a one trust.
> 
> And who's global hash store of all known hashes of all known
> downloadables would you trust anyway? The NSAs? 
> 
> Best you can do is make something for the specific case. The Gentoo tree
> and distfiles can be GPG signed and if you agree to trust Gentoo's keys
> then you are good to go and it can be automated (which is the easy bit
> btw).
> 
> For the general case/ I can't see that work at all. I trust Gentoo with
> Gentoo, but I don't see myself ever trusting $ARB_3RD_PARTY 
> with $EVERYTHING


Your comments are well received and I do not even disagree with your points.
I think you need to relax, grab your favorite beverage, recline and put
on your "deep thinking hat". Perhaps a foot massage from your least 
productive  underling would set your mind at ease?


So, let us assume you are correct in everything you have stated. But,
try on this idea and shoot away. Note in this context, I use the terms
code=package=sofware=download, so as to focus on the 10,000 foot view
of the idea, not the minutia. 


Premiss: 
Any individual code/software/package/download can be hacked as can it's
keys/hashes, regardless of where they are located.  But, it would be very
difficult for an interloper, to inject into  such codes  at a thousand 
differnet locations without detection. Note, at each repository, hashes
can be regenerated and had better match the hashes of the the orignation
site(s).

Proposal: 
So rather than a static singular  check-point of where you code check, why
not develop checking tools that check the integrity of any given piece of
code, from many multiple locations? (Fault tolerance via redundancy, if
you like).


Possible solution:
1) Source archives usually contain revision histories and sync those up with
revision releases. So mantain a master list of hashes/keys on their sources
in the form of a histogram. So a code  periodically updated n(10) times
would have n(10) hashes with  n(10) timestamps as the basis of the
histogram. Think of a digial (camera) histogram. [1] This would develop a
histogram of changes in the hashes for a given code/package not only at the
sourcecode reporsitory, but also at those institutional repositories who
generate their own hashes/keys and link them to release date-time-stamps;
had better have convergence with the development sources. 

Now we would not only have the hashes, which can be manually checked
anywhere anytime, but a historm image check, based on the historical dates
where the code is known to have changed. Every code changes does not have to
be included, only significant, period releases.  Code could be check by a
bit by bit number by number approach, as well as a single image that is a
compilation of those bits into the form of a histogram. [2]

The archive sites (common download repositories) should be able to check
histograms each time a code they offer is changed. Nothing would stop
capable users from using these sorts of tools too. Please observe: these
"histograms", particularly if well distributed across the net, would greatly
enhance forensic and integrity ensurance efforts.


2) The individual could mantain a master list of hashes/keys on their
(gentoo) system(s). Yes they would have to be periodically updated, but an
archival database approach, complete with timestamps when a particular codes
hask/key changes, would be logged, per package. This could probably be a
compliment to  portage.


3) For every (non-gentoo users also) individual, a distributed checking tool
could be develop to simulataneously check against dozens or hundreds of
hashes from random sites against their copy of the hash/key. It'd be pretty
hard to hack many of those sources in a coordinated fashion.


For the paranoid usb stick(s) could be used  to house the hashes for
transient usage/updates. The pathelogically paranoic could download, drop
the their ether connection, insert the usb(s) and perform hash, code and
system checks.


Nothing in this scenario would stop tainted code from the original
development team. But wait, holy_oscars_batman, the fact that those
(trusted) codes are developed in an open fashion where other folks can audit
the sources, historically and concurrently, should drasitcally reduce
nefarious codes, as we currently have evidence to support.


So, what a torrent_style tool that uses a distributed  hashes/keys to check
code integrity; is possible?

Surely the code histogram idea is possible?


James


[1] http://digital-photography-school.com/understanding-histograms/

[2] Not the proper forum here to refine this part, but, Z and fourier
transforms make quick, easy work for this sort of quick image parsing.

Re: [gentoo-user] Re: package download verification

[Alan McKinnon]

On 08/05/2014 21:13, James wrote:
> So, what a torrent_style tool that uses a distributed  hashes/keys to check
> code integrity; is possible?

In one word: git

> Surely the code histogram idea is possible?

Again, git.

An aspect of the git design spec is to try deal with the kind of things
you are pondering here. It seems a valid approach - if many people out
there clone and make copies of the code then work on it, and if a bad
hat injects some weirdness, there are enough eyes to hopefully catch it.

Now that I think of it, it's an elegant solution:

Avoid the problems of a single master store but not having one.

-- 
Alan McKinnon
alan.mckinnon@gmail.com

[gentoo-user] Re: package download verification

[James]

Alan McKinnon <alan.mckinnon <at> gmail.com> writes:


> On 08/05/2014 21:13, James wrote:
> > So, what a torrent_style tool that uses a distributed  hashes/keys 
> > to check code integrity; is possible?
> In one word: git
> > Surely the code histogram idea is possible?
> Again, git.


Wow, this is freaky. I've never read up on git's diff-stat.
There are numerous advanced mathematical methodologies that
can be applied to histograms, some of which run blazingly
fast on GPU and customer gate arrays. I wonder where I can 
read up on "diffstat" and see what mathematics they are applied
or are contemplating to use? Google reveals nothing yet
and I'm too lazy (right now) to read the code.

"Copyright © 1996-2013,2014 by Thomas E. Dickey
diffstat reads the output of diff and displays a histogram of the
insertions, deletions, and modifications per-file. It is useful for
reviewing large, complex patch files. "


this is really cool, I shoot one from the hip and it is a well
established tool? scary, but, math is math..... 

Do you know (of) this guy? is he easy to approach or territorial?
I never heard of this guy.....

> An aspect of the git design spec is to try deal with the kind of things
> you are pondering here. It seems a valid approach - if many people out
> there clone and make copies of the code then work on it, and if a bad
> hat injects some weirdness, there are enough eyes to hopefully catch it.
> Now that I think of it, it's an elegant solution:
> Avoid the problems of a single master store but not having one.


Beyond elegant. Some very cool mathematics can be applied to (histogram)
verious images and image manipultation as a sort of "out of band" security
scan..... There are also some very cool bit manipulations deep in
H.264 that are similarly applicable.  In fact "motion detection" 
is a first cousin to these transfroms (z, laplace, fft) which are often
embedded in hardware for system checks). DOD......

Well, I'm off to go play basketball and the the Tampabay Devil rays
(hopefully) pound those cleveland indians.........

I'm jazzed. I usually lucky when hips shots do not richochet around and 
hit me in the rear!.............

Since I'm not driving, I going to have a BIG FAT MARGARITA, and give you
a salute at the BB game.... You've made my week!  




thx!
James