[gentoo-dev] Peer-to-Peer?

[gentoo-dev] Peer-to-Peer?

[Yannick Koehler]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


I read about bittorrent recently.  I was wondering if the mirror and tools 
used by gentoo (fetch/wget) support such system.  If so, wouldn't be great 
that people like me who could participate in providing packages could install 
bittorrent or another file sharing peer-to-peer tools and help in spreading 
gentoo ;-)

I see two kind of packages distribution, the source one 
(/usr/portage/distfiles/*) and the compiled package.  Someone could setup on 
the mirror a binary version of gentoo compile with specific use/compile flags 
(preferrably the default one set inside the various gentoo config files) and 
then people that like that configuration could just install those package 
instead of re-compiling, they could still recompile the package that they 
want with their own option but for may speed up a lot upgrade of package for 
end-user satisfied with the pre-compile cflags/use of the mirror.  

There could also be more than one mirror with different flags/use.  It would 
even be nice that the mirror only reflect content of package held on end-user 
system for the peer-to-peer system to kick in and offer them to other users.

- -- 

Yannick Koehler
 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE9NsIQfuKOJNEyL1URAijgAJ4wbPpcwcvXr+BgU24OWMV4UyvNSgCeMeLq
OMaHi86uiQF2c9XO2GkOduQ=
=wlCx
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Peer-to-Peer?

[Michael Cummings]

then we could rename all the extensions to rpm and...

sorry, not trying to be callous, but one flaw I would see is that there
is more to the flags than just the hardware flag. I have three machines
running gentoo, each of them with hardware in various states of
degredation. What you suggest would also require, in addition to a
package per hardware config, is one per possible config line (USE
variables can differ from user to user depending on their needs - i have
a box with a USE of -X -java -qt -gnome -kde -gtk just to insure that
nothing got put on that might have a dependancy on those), not to
mention dependancies (F begot G which begot H which begot I). For
instance, as time has progressed I have noticed that emerge -pu world
displays some packages marked as N, which means that new dependancies
have arisen since I fist installed the package in question.

But then, that would just be my meager $.02 worth, which these days
won't even cover taxes on a soda can. Cool idea though for using
distributed package sources...would require that every peer keep up to
date tarballs though, to account for security fixes and patches, might
be a little bandwith intensive...but i ramble.

On Thu, Jul 18, 2002 at 09:26:38AM -0400, Yannick Koehler wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 
> I read about bittorrent recently.  I was wondering if the mirror and tools 
> used by gentoo (fetch/wget) support such system.  If so, wouldn't be great 
> that people like me who could participate in providing packages could install 
> bittorrent or another file sharing peer-to-peer tools and help in spreading 
> gentoo ;-)
> 
> I see two kind of packages distribution, the source one 
> (/usr/portage/distfiles/*) and the compiled package.  Someone could setup on 
> the mirror a binary version of gentoo compile with specific use/compile flags 
> (preferrably the default one set inside the various gentoo config files) and 
> then people that like that configuration could just install those package 
> instead of re-compiling, they could still recompile the package that they 
> want with their own option but for may speed up a lot upgrade of package for 
> end-user satisfied with the pre-compile cflags/use of the mirror.  
> 
> There could also be more than one mirror with different flags/use.  It would 
> even be nice that the mirror only reflect content of package held on end-user 
> system for the peer-to-peer system to kick in and offer them to other users.
> 
> - -- 
> 
> Yannick Koehler
>  
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.0.6 (GNU/Linux)
> Comment: For info see http://www.gnupg.org
> 
> iD8DBQE9NsIQfuKOJNEyL1URAijgAJ4wbPpcwcvXr+BgU24OWMV4UyvNSgCeMeLq
> OMaHi86uiQF2c9XO2GkOduQ=
> =wlCx
> -----END PGP SIGNATURE-----
> 
> _______________________________________________
> gentoo-dev mailing list
> gentoo-dev@gentoo.org
> http://lists.gentoo.org/mailman/listinfo/gentoo-dev

Re: [gentoo-dev] Peer-to-Peer?

[Michael Cummings]

I just wanted to repeat, I meant absolutely no offense or disrespect
with this posting, turn the flame guns off, yada yada yada. I think the
"technology"/technique posted about is interesting, could be useful,
just IMHO binaries for the gentoo packages goes against why something
like gentoo is so great...

On Thu, Jul 18, 2002 at 09:39:20AM -0400, Michael Cummings wrote:
> then we could rename all the extensions to rpm and...
> 
> sorry, not trying to be callous, but one flaw I would see is that there
> is more to the flags than just the hardware flag. I have three machines
> running gentoo, each of them with hardware in various states of
> degredation. What you suggest would also require, in addition to a
> package per hardware config, is one per possible config line (USE
> variables can differ from user to user depending on their needs - i have
> a box with a USE of -X -java -qt -gnome -kde -gtk just to insure that
> nothing got put on that might have a dependancy on those), not to
> mention dependancies (F begot G which begot H which begot I). For
> instance, as time has progressed I have noticed that emerge -pu world
> displays some packages marked as N, which means that new dependancies
> have arisen since I fist installed the package in question.
> 
> But then, that would just be my meager $.02 worth, which these days
> won't even cover taxes on a soda can. Cool idea though for using
> distributed package sources...would require that every peer keep up to
> date tarballs though, to account for security fixes and patches, might
> be a little bandwith intensive...but i ramble.
> 
> On Thu, Jul 18, 2002 at 09:26:38AM -0400, Yannick Koehler wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> > 
> > 
> > I read about bittorrent recently.  I was wondering if the mirror and tools 
> > used by gentoo (fetch/wget) support such system.  If so, wouldn't be great 
> > that people like me who could participate in providing packages could install 
> > bittorrent or another file sharing peer-to-peer tools and help in spreading 
> > gentoo ;-)
> > 
> > I see two kind of packages distribution, the source one 
> > (/usr/portage/distfiles/*) and the compiled package.  Someone could setup on 
> > the mirror a binary version of gentoo compile with specific use/compile flags 
> > (preferrably the default one set inside the various gentoo config files) and 
> > then people that like that configuration could just install those package 
> > instead of re-compiling, they could still recompile the package that they 
> > want with their own option but for may speed up a lot upgrade of package for 
> > end-user satisfied with the pre-compile cflags/use of the mirror.  
> > 
> > There could also be more than one mirror with different flags/use.  It would 
> > even be nice that the mirror only reflect content of package held on end-user 
> > system for the peer-to-peer system to kick in and offer them to other users.
> > 
> > - -- 
> > 
> > Yannick Koehler
> >  
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.0.6 (GNU/Linux)
> > Comment: For info see http://www.gnupg.org
> > 
> > iD8DBQE9NsIQfuKOJNEyL1URAijgAJ4wbPpcwcvXr+BgU24OWMV4UyvNSgCeMeLq
> > OMaHi86uiQF2c9XO2GkOduQ=
> > =wlCx
> > -----END PGP SIGNATURE-----
> > 
> > _______________________________________________
> > gentoo-dev mailing list
> > gentoo-dev@gentoo.org
> > http://lists.gentoo.org/mailman/listinfo/gentoo-dev
> _______________________________________________
> gentoo-dev mailing list
> gentoo-dev@gentoo.org
> http://lists.gentoo.org/mailman/listinfo/gentoo-dev

Re: [gentoo-dev] Peer-to-Peer?

[Jean-Michel Smith]

On Thursday 18 July 2002 09:36 am, Michael Cummings wrote:
> I just wanted to repeat, I meant absolutely no offense or disrespect
> with this posting, turn the flame guns off, yada yada yada. I think the
> "technology"/technique posted about is interesting, could be useful,
> just IMHO binaries for the gentoo packages goes against why something
> like gentoo is so great...

While I have absolutely no interest in precompiled binaries for Gentoo, I do 
think that using some kind of p2p approach, like FreeNet for example, as a 
way of distributing tarballs and perhaps even portage trees, would be very 
cool.  I've had 'emerge sync's fail on more than one occasion because the 
round robin rsync server I connect to happens to have its connections maxed 
out.

One of the real strengths of FreeNet is that the more popular something 
becomes, the more available it becomes, rather than the opposite as is the 
case with more traditional client-server designs (which http and ftp 
essentially are).

Jean. 

Re: [gentoo-dev] Peer-to-Peer?

[Yannick Koehler]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On July 18, 2002 10:57 am, Jean-Michel Smith wrote:
> On Thursday 18 July 2002 09:36 am, Michael Cummings wrote:
> > I just wanted to repeat, I meant absolutely no offense or disrespect
> > with this posting, turn the flame guns off, yada yada yada. I think the
> > "technology"/technique posted about is interesting, could be useful,
> > just IMHO binaries for the gentoo packages goes against why something
> > like gentoo is so great...
>
> While I have absolutely no interest in precompiled binaries for Gentoo, I
> do think that using some kind of p2p approach, like FreeNet for example, as
> a way of distributing tarballs and perhaps even portage trees, would be
> very cool.  I've had 'emerge sync's fail on more than one occasion because
> the round robin rsync server I connect to happens to have its connections
> maxed out.
>
> One of the real strengths of FreeNet is that the more popular something
> becomes, the more available it becomes, rather than the opposite as is the
> case with more traditional client-server designs (which http and ftp
> essentially are).

Hmm, interesting, is there others?    

- -- 

Yannick Koehler
 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE9Ntd0fuKOJNEyL1URAg52AJ0dqkUYxXGQzVssKRfrA3cEKUY/DwCeJwLO
++wDwKwbw3KzKR4yiG49lXo=
=n5ES
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Peer-to-Peer?

[Yannick Koehler]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On July 18, 2002 09:39 am, Michael Cummings wrote:
> then we could rename all the extensions to rpm and...
>
> sorry, not trying to be callous, but one flaw I would see is that there
> is more to the flags than just the hardware flag. I have three machines
> running gentoo, each of them with hardware in various states of
> degredation. What you suggest would also require, in addition to a
> package per hardware config, is one per possible config line (USE
> variables can differ from user to user depending on their needs - i have
> a box with a USE of -X -java -qt -gnome -kde -gtk just to insure that
> nothing got put on that might have a dependancy on those), not to
> mention dependancies (F begot G which begot H which begot I). For
> instance, as time has progressed I have noticed that emerge -pu world
> displays some packages marked as N, which means that new dependancies
> have arisen since I fist installed the package in question.

Well the idea was the following, If I build a package for my computer and 
could make other benefit from the fact that I've done so and someone maybe 
doing the exact same thing, so instead of having it re-compile the same 
things in the same way he could just pick up mine.  If you do not have common 
system then you don't benefit from it but you don't lose anything.  It is an 
addition not a removal.

That's why I was putting talk about a peer-to-peer system.  It would be nice 
that in some way, if want to emerge a build that has been emerged already on 
another system using the same configuration then you could at your choice 
decide not to re-do it but take the one that has been done.

It does imply trust, security issues and all of this, but that is also true 
whenever you compile source code that you didn't investigate yourself anyway 
even thought there's a digest file, that file may have been created or 
modified on the mirror to make you download malicious source code.

- -- 

Yannick Koehler
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE9NtRPfuKOJNEyL1URAriGAJ46+BN+prnnKwl+jeHwa06IohAB3ACgnFzv
0Gqq6R2QwgIMQERhxWa7sI0=
=G4Id
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Peer-to-Peer?

[Paul de Vrieze]

On Thursday 18 July 2002 16:44, Yannick Koehler wrote:
> On July 18, 2002 09:39 am, Michael Cummings wrote:
> Well the idea was the following, If I build a package for my computer and
> could make other benefit from the fact that I've done so and someone maybe
> doing the exact same thing, so instead of having it re-compile the same
> things in the same way he could just pick up mine.  If you do not have
> common system then you don't benefit from it but you don't lose anything. 
> It is an addition not a removal.
>
> That's why I was putting talk about a peer-to-peer system.  It would be
> nice that in some way, if want to emerge a build that has been emerged
> already on another system using the same configuration then you could at
> your choice decide not to re-do it but take the one that has been done.
>
> It does imply trust, security issues and all of this, but that is also true
> whenever you compile source code that you didn't investigate yourself
> anyway even thought there's a digest file, that file may have been created
> or modified on the mirror to make you download malicious source code.

Appart from the trust issue, the major problem is comparing the two systems. 
It is unlikely they are exactly the same. But even if they are, it is a hell 
of a job finding out. The only way such a distribution for binaries works is 
with a binary only distribution like,..... (you know who).

Paul

-- 
Paul de Vrieze
Junior Researcher
Mail: pauldv@cs.kun.nl
Homepage: http://www.devrieze.net

Re: [gentoo-dev] Peer-to-Peer?

[Yannick Koehler]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> Appart from the trust issue, the major problem is comparing the two
> systems. It is unlikely they are exactly the same. But even if they are, it
> is a hell of a job finding out. The only way such a distribution for
> binaries works is with a binary only distribution like,..... (you know
> who).

Portage keeps certain file which help on this such as 
/var/db/pkg/<cat>/<name>/CFLAGS and USE.  Maybe they could be named on the 
ftp server to

	<name>.specification
	<name>.tar.gz

So that emerge download the .specification, validate the similarity and then 
propose or take it from the distribution system.  Basically the same as 
ccache ;-) 

- -- 

Yannick Koehler
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE9NxX4fuKOJNEyL1URAshWAKCi7C5zv8OrDXPiYPNlb3ZU/RQxZACeJC2A
ChJ/UMw4ywuVbCvrhd12BCU=
=fO57
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Peer-to-Peer?

[Marko Mikulicic]

Yannick Koehler wrote:
 > -----BEGIN PGP SIGNED MESSAGE-----
 > Hash: SHA1
 >
 >
 >>Appart from the trust issue, the major problem is comparing the two
 >>systems. It is unlikely they are exactly the same. But even if they 
are, it
 >>is a hell of a job finding out. The only way such a distribution for
 >>binaries works is with a binary only distribution like,..... (you know
 >>who).
 >
 >
 > Portage keeps certain file which help on this such as
 > /var/db/pkg/<cat>/<name>/CFLAGS and USE.  Maybe they could be named 
on the
 > ftp server to
 >
 > 	<name>.specification
 > 	<name>.tar.gz
 >
 > So that emerge download the .specification, validate the similarity 
and then
 > propose or take it from the distribution system.  Basically the same as
 > ccache ;-)
 >

I like the idea. I was thinking of something similar.
I think it's possible to hash the use flags used to build
the package and compare it to the package to be downloaded.

However I doubt of the practical usefulness of a global peer-to-peer
solution. I have an 128bps upstream bandwidth and everyone going to
copy compiled kde-3.0 from me would compile it faster on PIII500 (~).
Since the userbase not so big as *pster and the number of combinations
of use flags is big, it is not very likely to find a package provided by
a fast host.

I think it can be useful in a lan where, for whatever reasion, the
machines doesn't share the same use flag configuration. If a package
desn't use the "mysql" use-flag then it doesn't depend of having it or not.

Marko

Re: [gentoo-dev] Peer-to-Peer?

[Nils Decker]

Marko Mikulicic <marko@seul.org> wrote:
> Yannick Koehler wrote:
> and then
>  > propose or take it from the distribution system.  Basically the
>  > same as ccache ;-)
>  >
> 
> I like the idea. I was thinking of something similar.
> I think it's possible to hash the use flags used to build
> the package and compare it to the package to be downloaded.

I see another problem with this. There is no way to make the packages trusted.
In the portage tree, every downloaded file is checked against a MD5 hash.
This means, I have to trust the person who build the port. This is not
a big problem to me, because those people are "near" to the gentoo core,
and everybody can check the MD5s against the official downloads of the packet.

I can't do this sort of check agains precompiled binaries, because every binary
would have a different MD5. The only way to check would to compile the package
myself with the same flags, thus defeating the purpose.
Using those binary packages means to trust every user of gentoo, that he
doesn't put trojans or whatever on my system.

My 0,02 EUR
  Nils

-- 
The primary purpose of the DATA statement is to give names to
constants; instead of referring to pi as 3.141592653589793 at every
appearance, the variable PI can be given that value with a DATA
statement and used instead of the longer form of the constant.  This
also simplifies modifying the program, should the value of pi change.

		-- FORTRAN manual for Xerox Computers
Nils Decker <ndecker@gmx.de>

Re: [gentoo-dev] Peer-to-Peer?

[Yannick Koehler]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On July 19, 2002 05:20 am, Nils Decker wrote:
> >  > propose or take it from the distribution system.  Basically the
> >  > same as ccache ;-)
> >
> > I like the idea. I was thinking of something similar.
> > I think it's possible to hash the use flags used to build
> > the package and compare it to the package to be downloaded.
>
> I see another problem with this. There is no way to make the packages
> trusted. In the portage tree, every downloaded file is checked against a
> MD5 hash. This means, I have to trust the person who build the port. This
> is not a big problem to me, because those people are "near" to the gentoo
> core, and everybody can check the MD5s against the official downloads of
> the packet.
>
> I can't do this sort of check agains precompiled binaries, because every
> binary would have a different MD5. The only way to check would to compile
> the package myself with the same flags, thus defeating the purpose.
> Using those binary packages means to trust every user of gentoo, that he
> doesn't put trojans or whatever on my system.

The MD5 hash verification is only providing proof that the file you've 
transferred between the distribution server and your PC was the same intended 
by the server on which you did your rsync of the digest files.

You actually implicitely trust that whoever put the digest files inside the 
rsync server used "correct" sources tarballs, you could verify that but the 
process is kind of lenghty as it would be for a binary check too.

And now because there may be multiple rsync server, that trust is getting less 
and less meaningful.

To fix that, one would have to actually use the same PGP signature of the 
package as the one provided on the original distribution site from the 
original author.

- -- 

Yannick Koehler
 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE9OA5ofuKOJNEyL1URAgjIAJ9uevL5x70xa9gpTZsckyivZzAcRQCdEVry
YpQYX7E3DVoJtRlhTXQyqpg=
=djr/
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Peer-to-Peer?

[Jean-Michel Smith]

On Friday 19 July 2002 04:20 am, Nils Decker wrote:
> Marko Mikulicic <marko@seul.org> wrote:
> > Yannick Koehler wrote:
> > and then
> >
> >  > propose or take it from the distribution system.  Basically the
> >  > same as ccache ;-)
> >
> > I like the idea. I was thinking of something similar.
> > I think it's possible to hash the use flags used to build
> > the package and compare it to the package to be downloaded.
>
> I see another problem with this. There is no way to make the packages
> trusted. In the portage tree, every downloaded file is checked against a
> MD5 hash. This means, I have to trust the person who build the port. This
> is not a big problem to me, because those people are "near" to the gentoo
> core, and everybody can check the MD5s against the official downloads of
> the packet.

Yeah, we need a keyright of GPG public keys for gentoo developers, and a GPG 
signature for each ebuild (which in turn already contains an MD5 sum for all 
the source URLs in the digest file).

They keyring would have to be (a) bought with a CD ordered directly from 
gentoo, (b) downloaded from the gentoo website (not perfectly secure, but 
"good enough" for most people) or (c) obtained in person (credit card CDRs 
anyone) from Gentoo representatives at free software/linux conferences.

Then we could pull ebuilds of the P2P network, check the signatures against a 
trusted keyring and verify that the ebuild is bona fide, then pull the 
tarball in off the same P2P network, and emerge as usual (emerge already 
checks the MD5 sum, the important part is making sure the ebuild itself is 
trustworthy).

There are good performance reasons to consider this approach in addition to 
the current method of distribution, but there are also good geo-political 
reasons for doing this: distribution of legally Free Software (as opposed to 
warez, pr0n, and infringing mp3s).  When Hollywood tries to shut down FreeNet 
we could point to it as an infrastructure that is used for the widespread 
dissemination of GNU/Linux (or at least Gentoo), and whatever infringement is 
going on is as secondary as it is for other protocols like FTP and HTTP.

The performance boost though is IMHO reason enough to at least consider the 
idea (though the idea of precompiled binary packages is utterly uninteresting 
to me, the ability to get source tarballs and ebuilds more readilly, without 
having the 'emerge sync' fail because a site is maxed out is compelling).

My $0.02 (what is that, 0.01 Euro these days?)

Jean.