[gentoo-dev] binary packages, crazy idea

[gentoo-dev] binary packages, crazy idea

[J Robert Ray]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi, I have an idea I've been thinking about, I'd like to share it and
see anyone thinks there is any merit to it.

The goal is to preserve gentoo's customizability but gain the
convenience of instantly available binary packages.

The problem with binary packages is that they come in one flavor and
would ignore your local USE variable settings.  Plus, creating separate
binary packages for every combination of USE variables would be an
enormous, never ending task.

My idea is to automate the process of creating binary packages, and turn
the installed base of gentoo users into a giant compile farm.

When a user goes to emerge a package, first portage would analyze the
ebuild to determine what USE variables are used by the package.  Then it
would build an ordered list of the user's USE variables, limited to only
those that are used by this particular package, but indicating if they
are enabled or not.

To this list of USE variables would be added the gcc version, CHOST,
CFLAGS, CXXFLAGS, perhaps portage version, plus what ever else that
makes up the uniqueness of the user's configuration.  I'm taking a cue
from ccache for this.

Portage could then take this unique code and check a public server to
see if a binary package exists matching this combination.  If so, the
binary package would be fetched and installed.  If not, the package
would be compiled locally and then uploaded to the public server.

To prevent from malicious tampering, some kind of system would need to
be in place to verify that a package hasn't been trojaned or something.
~ The server could wait until it received a certain number of copies of a
specific package from unique sources and compare them to see if they are
all the same before deciding a package is safe.

Security would be the largest stumbling block of this system.  I think
the most likely system to be safe would be a PGP-like web of trust.
Each gentoo installation would generate a set of keys that it would use
to sign packages it uploads.  Small groups of gentoo users who have the
same setup could exchange keys.   As the server receives binary packages
with the same checksum, it accumulates the keys from the signature.
Portage would only then download a package if it has been signed by a
key that the user trusts.  Of course, if no such key is available, it
would build the package locally.  If the local binary package matches
what is available on the server, this user would sign the binary package
available on the server.  Optionally, a user to configure his system to
accept a binary package after it has been signed by a certain number of
keys.

I believe the ideal storage facility for such a system is with something
like freenet.  It is distributed, non-permanent storage where more
popular files are longer-lived than infrequently requested files.
Assuming the vast majority of gentoo users are using the same default
settings for most things, the popular packages would survive
indefinately on freenet as long as there are people installing them.

A system that took advantage of freenet could upload the binary package
to freenet with its md5sum as the key, and submit the package details
with key to a central server.  Portage would when check with the central
server for a key, fetch the package out of freenet with the key, and
then install the binary package.

That's basically it.  Sure, it's a far-fetched idea but I'd to hear what
people think about it.

- - Robert

- --
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xBB929E54
Key fingerprint = BEA9 490C D2B9 AD83 E88B  3148 3136 34E4 BB92 9E54
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQE9b3XqMTY05LuSnlQRAvMWAJwOqQQsfZiJOO9G8TpcD1EOLq9yTACeI8af
KGbprUFS/QwGRLwygQ0lA44=
=FxhM
-----END PGP SIGNATURE-----

Re: [gentoo-dev] binary packages, crazy idea

[Thomas T. Veldhouse]

> Security would be the largest stumbling block of this system.

Actually, I think a signed package would suffice.  MD5 checksum perhaps.
But, I think the biggest stumbling block is not security, but rather disk
space and bandwidth.  All of these combinations would just become an
enormous archive.

Tom Veldhouse

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi, I have an idea I've been thinking about, I'd like to share it and
> see anyone thinks there is any merit to it.
>
> The goal is to preserve gentoo's customizability but gain the
> convenience of instantly available binary packages.
>
> The problem with binary packages is that they come in one flavor and
> would ignore your local USE variable settings.  Plus, creating separate
> binary packages for every combination of USE variables would be an
> enormous, never ending task.
>
> My idea is to automate the process of creating binary packages, and turn
> the installed base of gentoo users into a giant compile farm.
>
> When a user goes to emerge a package, first portage would analyze the
> ebuild to determine what USE variables are used by the package.  Then it
> would build an ordered list of the user's USE variables, limited to only
> those that are used by this particular package, but indicating if they
> are enabled or not.
>
> To this list of USE variables would be added the gcc version, CHOST,
> CFLAGS, CXXFLAGS, perhaps portage version, plus what ever else that
> makes up the uniqueness of the user's configuration.  I'm taking a cue
> from ccache for this.
>
> Portage could then take this unique code and check a public server to
> see if a binary package exists matching this combination.  If so, the
> binary package would be fetched and installed.  If not, the package
> would be compiled locally and then uploaded to the public server.
>
> To prevent from malicious tampering, some kind of system would need to
> be in place to verify that a package hasn't been trojaned or something.
> ~ The server could wait until it received a certain number of copies of a
> specific package from unique sources and compare them to see if they are
> all the same before deciding a package is safe.
>
> Security would be the largest stumbling block of this system.  I think
> the most likely system to be safe would be a PGP-like web of trust.
> Each gentoo installation would generate a set of keys that it would use
> to sign packages it uploads.  Small groups of gentoo users who have the
> same setup could exchange keys.   As the server receives binary packages
> with the same checksum, it accumulates the keys from the signature.
> Portage would only then download a package if it has been signed by a
> key that the user trusts.  Of course, if no such key is available, it
> would build the package locally.  If the local binary package matches
> what is available on the server, this user would sign the binary package
> available on the server.  Optionally, a user to configure his system to
> accept a binary package after it has been signed by a certain number of
> keys.
>
> I believe the ideal storage facility for such a system is with something
> like freenet.  It is distributed, non-permanent storage where more
> popular files are longer-lived than infrequently requested files.
> Assuming the vast majority of gentoo users are using the same default
> settings for most things, the popular packages would survive
> indefinately on freenet as long as there are people installing them.
>
> A system that took advantage of freenet could upload the binary package
> to freenet with its md5sum as the key, and submit the package details
> with key to a central server.  Portage would when check with the central
> server for a key, fetch the package out of freenet with the key, and
> then install the binary package.
>
> That's basically it.  Sure, it's a far-fetched idea but I'd to hear what
> people think about it.
>
> - - Robert
>
> - --
> http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xBB929E54
> Key fingerprint = BEA9 490C D2B9 AD83 E88B  3148 3136 34E4 BB92 9E54
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.0.7 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iD8DBQE9b3XqMTY05LuSnlQRAvMWAJwOqQQsfZiJOO9G8TpcD1EOLq9yTACeI8af
> KGbprUFS/QwGRLwygQ0lA44=
> =FxhM
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> gentoo-dev mailing list
> gentoo-dev@gentoo.org
> http://lists.gentoo.org/mailman/listinfo/gentoo-dev
>

Re: [gentoo-dev] binary packages, crazy idea

[J Robert Ray]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thomas T. Veldhouse wrote:
|>Security would be the largest stumbling block of this system.
|
|
| Actually, I think a signed package would suffice.  MD5 checksum perhaps.
| But, I think the biggest stumbling block is not security, but rather disk
| space and bandwidth.  All of these combinations would just become an
| enormous archive.

That's why I proposed freenet, it is a distributed datastore with
p2p-like qualities.  A file you put into it is mirrored around to other
freenet servers.  If a file isn't requested for a long time, and space
is needed for newer files, it is purged.  When you fetch a file from
freenet it can come from any number of servers that have the file.

It's worth checking out if you haven't played with it.

- - Robert


- --
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xBB929E54
Key fingerprint = BEA9 490C D2B9 AD83 E88B  3148 3136 34E4 BB92 9E54
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQE9b3uvMTY05LuSnlQRAheEAJ0fCA68UCPdHojPt3pCFUbaO8fw3ACfX4Ov
Fclt0/rCBLDSBn1I2ZIoeHs=
=emel
-----END PGP SIGNATURE-----

Re: [gentoo-dev] binary packages, crazy idea

[Paul de Vrieze]

On Friday 30 August 2002 16:05, J Robert Ray wrote:
> Thomas T. Veldhouse wrote:
> |>Security would be the largest stumbling block of this system.
> |
> | Actually, I think a signed package would suffice.  MD5 checksum perhaps.
> | But, I think the biggest stumbling block is not security, but rather disk
> | space and bandwidth.  All of these combinations would just become an
> | enormous archive.
>
> That's why I proposed freenet, it is a distributed datastore with
> p2p-like qualities.  A file you put into it is mirrored around to other
> freenet servers.  If a file isn't requested for a long time, and space
> is needed for newer files, it is purged.  When you fetch a file from
> freenet it can come from any number of servers that have the file.
>
> It's worth checking out if you haven't played with it.
>

We've had a very very lengthy discussion about this topic about a month ago. 
Maybe you check out the maillist archives before continuing the discussion.

Paul

-- 
Paul de Vrieze
Junior Researcher
Mail: pauldv@cs.kun.nl
Homepage: http://www.devrieze.net

Re: [gentoo-dev] binary packages, crazy idea

[Paul de Vrieze]

On Friday 30 August 2002 17:04, you wrote:
> Paul de Vrieze wrote:
> | We've had a very very lengthy discussion about this topic about a
>
> month ago.
>
> | Maybe you check out the maillist archives before continuing the
>
> discussion.
>
> Any chance you could point me to a message?  I did a scan of the past
> few months of gentoo-dev and didn't notice anything related.  There
> weren't many threads that I would consider "very very lengthy."
>

It is the thread called Peer-to-Peer which starts with this message:
http://lists.gentoo.org/pipermail/gentoo-dev/2002-July/013653.html

Paul

-- 
Paul de Vrieze
Junior Researcher
Mail: pauldv@cs.kun.nl
Homepage: http://www.devrieze.net