[gentoo-dev] torrent support in portage

[gentoo-dev] torrent support in portage

[Caleb Shay]

I've been giving a lot of thought about how to deal with things like the
mirror rush we had when kde-3.1 was released, and we got downgraded to
second-class citizens on ibiblio :), or releases of new ISOs.  I think
I've come up with a workable solution, easy to implement in portage, but
it would require an infrastructure addition.  We could add bittorrent
(http://bitconjurer.org/BitTorrent/) support for larger packages.  I say
larger packages, because bittorrent doesn't work well for small files. 
Here is what I envision it entailing.

A bittorrent server (this is the infrastructure part), that maintains
torrents of the larger packages (I define larger as 20MB+), such as
XFree.

Additional FEATURE:
bittorrent

Additional var in ebuilds:
TORRENT_URI=http://torrent.gentoo.org/${A}.torrent

(though, seeing as how we already have the file name, and therefore, the
torrent file name, it could just as easily be HAS_TORRENT=1, or maybe we
just maintain a database of which packages have torrents available that
portage checks)

What happens now is, if the user has the bittorrent FEATURE enabled, AND
has bittorrent installed (much like what is required to use the ccache
feature), and the package has a torrent available, we can then use
btdownloadheadless.py or btdownloadcurses.py to download the file (ie,
btdownloadheadl.py --url http://torrent.gentoo.org/${A}.torrent --saveas
/usr/portage/distfiles/${A}).  This takes load off of the download
mirrors, and helps everybody achieve maximum bandwidth while
downloading.

Any thoughts?

Cheers,

Caleb Shay
caleb@webninja.com




--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] torrent support in portage

[James H. Cloos Jr.]

Note that bittorrent does not work very well for dialup users.  The
overhead ends up being twice as large or more than the payload.  It
also ends up blocking most other traffic.  

It generally helps for those with more bandwidth, of course, but
something that does not open a couple of hundred sockets works
better over straws.

It will work well as a supplement to the staus quo,
but not so well as a replacement.

-JimC


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] torrent support in portage

[Caleb Shay]

Noted.  I don't have dial-up, so I wasn't aware of this.  However, since
(in my little pipe-dream), it would be a FEATURE option, if you aren't
on dial-up, don't enable it.  It would still have a good side-effect for
dial-up users.  If all of use with fat pipes are downloading from each
other, we aren't sucking up all the mirror's bandwidth, and dial-up
people can actually get through via the standard SRC_URI.

Caleb

On Mon, 2003-04-14 at 20:42, James H. Cloos Jr. wrote:
> Note that bittorrent does not work very well for dialup users.  The
> overhead ends up being twice as large or more than the payload.  It
> also ends up blocking most other traffic.  
> 
> It generally helps for those with more bandwidth, of course, but
> something that does not open a couple of hundred sockets works
> better over straws.
> 
> It will work well as a supplement to the staus quo,
> but not so well as a replacement.
> 
> -JimC
> 
> 
> --
> gentoo-dev@gentoo.org mailing list


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] torrent support in portage

[Robin H.Johnson]

[-- Attachment #1: Type: text/plain, Size: 2102 bytes --]

On Mon, Apr 14, 2003 at 04:14:23PM -0400, Caleb Shay wrote:
> What happens now is, if the user has the bittorrent FEATURE enabled, AND
> has bittorrent installed (much like what is required to use the ccache
> feature), and the package has a torrent available, we can then use
> btdownloadheadless.py or btdownloadcurses.py to download the file (ie,
> btdownloadheadl.py --url http://torrent.gentoo.org/${A}.torrent --saveas
> /usr/portage/distfiles/${A}).  This takes load off of the download
> mirrors, and helps everybody achieve maximum bandwidth while
> downloading.
Two questions here:
How much bandwidth is required for each bittorrent tracker? Nowhere I
have I seen this information.

Secondly for people to be able to offer the package to the BT network,
that means they would have it on their hard drive, and be running an
active program to provide it 24x7. So far what I have seen of BT is that
it takes one program instance per tracked file. In that case things are
going to get very big, very fast. Additionally I certianly will not be
letting people eat the bandwidth that I pay for on my servers at work.
Some residential cable and DSL ISPs are also very anal about bandwidth
usage so that could cause serious issues there (a friends ISP limits him
to 1GiB/day).

Your idea would be wonderful if bandwidth were free, or at least
significently cheaper than it is now for much of the world.

> A bittorrent server (this is the infrastructure part), that maintains
> torrents of the larger packages (I define larger as 20MB+), such as
> XFree.
Looking at the master distfiles directory, there are 343 files larger
than 10MiB. Only 140 files larger than 20MiB. There are 9316 files in
total, spanning 18Gb. The 343 files take up 9.4Gb of space. So they are
definetely out of proportion. So having a lower limit would be better.

-- 
Robin Hugh Johnson
E-Mail     : robbat2@orbis-terrarum.net
Home Page  : http://www.orbis-terrarum.net/?l=people.robbat2
ICQ#       : 30269588 or 41961639
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: [gentoo-dev] torrent support in portage

[James H. Cloos Jr.]

>>>>> "Caleb" == Caleb Shay <caleb@webninja.com> writes:

JimC> It will work well as a supplement to the staus quo,
JimC> but not so well as a replacement.

Caleb> However, ... it would be a FEATURE option, ...

Good to know you agree.

Caleb> If all of use with fat pipes are downloading from each other,
Caleb> we aren't sucking up all the mirror's bandwidth,

Yes, that is a benefit for all of us.

In addition to its use by leaves, it may also be beneficial for the
mirror sites themselves to use bt to grab the larger distfiles.

-JimC


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] torrent support in portage

[Ragnar Hojland Espinosa]

On Mon, Apr 14, 2003 at 08:42:34PM -0400, James H. Cloos Jr. wrote:
> Note that bittorrent does not work very well for dialup users.  The
> overhead ends up being twice as large or more than the payload.  It
> also ends up blocking most other traffic.  
> 
> It generally helps for those with more bandwidth, of course, but
> something that does not open a couple of hundred sockets works
> better over straws.

A couple of hundred sockets may be too much for reliable download even for
some broadband users.  For example, here Telefonica installs a Zyxel 643
ADSL router, and its NAT table has a limit of 256 simultaneous entries with
a default 180 seconds timeout (yah, it sucks)
-- 
Ragnar Hojland - Project Manager
Linalco "Especialistas Linux y en Software Libre"
http://www.linalco.com Tel: +34-91-5970074 Fax: +34-91-5970083

--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] torrent support in portage

[Brian Friday]

[-- Attachment #1: Type: text/plain, Size: 888 bytes --]

Just a additional thought/warning with regard to bittorrent.

The biggest problem associated with new bittorrent users (IMHO) is 
that they swamp their connection (unless they are on a T-1+) if 
a reasonable "--max_upload_rate" is not set. This leads to the 
endless cycle of frustrated users who hollar and frustrated
helpers who prematurely go bald.

Perhaps a good way of minimizing this problem would be to add an 
additional variable in make.conf called "TORRENT_MAX_UPLOAD=" 
which would accept a number and have a simple blurb that explains 
why setting this to zero(unlimited) might be a bad thing.

Not sure about anyone else but in my own testing, the sweet spot
for me on my 384 down / 128 up DSL connection is about 7K as a max
upload rate. At that speed, I can still interact with the outside
world with little to no speed loss and still serve as much as possible.

--
Brian

[-- Attachment #2: Type: text/html, Size: 2410 bytes --]

Re: [gentoo-dev] torrent support in portage

[James H. Cloos Jr.]

>>>>> "Brian" == Brian Friday <bfriday@lasierra.edu> writes:

Brian> The biggest problem associated with new bittorrent users (IMHO)
Brian> is that they swamp their connection (unless they are on a T-1+)
Brian> if a reasonable "--max_upload_rate" is not set.

Yes, that is part of the problem when connected via straws.  But
a dialup is entirely swamped even with --max_upload_rate 1.  

Now, if it can be limited to just, say, 32 or 64 sockets, with a
reasonably good guarantee that no more nodes will try to connect
while that limit is reached, I'd suspect it would work well even
on such low bandwidth connections.

-JimC


--
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] torrent support in portage

[Caleb Shay]

On Mon, 2003-04-14 at 21:38, Robin H.Johnson wrote:

> Two questions here:
> How much bandwidth is required for each bittorrent tracker? Nowhere I
> have I seen this information.

I'm not sure, but since it is only tracking who is up/downloading which
file, I wouldn't imagine it would be very much.

> 
> Secondly for people to be able to offer the package to the BT network,
> that means they would have it on their hard drive, and be running an
> active program to provide it 24x7.

There must be at least 1 person with the full file at all times, this is
generally (but does not have to be), the tracker server.

>  So far what I have seen of BT is that
> it takes one program instance per tracked file. In that case things are
> going to get very big, very fast.

Yes, it does.  I'm not sure of the overhead of each instance, but I've
come across sites with literally hundreds of files available.


>  Additionally I certianly will not be
> letting people eat the bandwidth that I pay for on my servers at work.

I'm not asking you to.  Set your upload cap to 2K and your simultaneous
uploads to 1 for all I care.  You only need upload for as long as you
are downloading.  Or, don't use it at all.

> Some residential cable and DSL ISPs are also very anal about bandwidth
> usage so that could cause serious issues there (a friends ISP limits him
> to 1GiB/day).

Then he probably wouldn't want to use it.  My ISP doesn't limit my
usage, so I would use it.

> 
> Your idea would be wonderful if bandwidth were free, or at least
> significently cheaper than it is now for much of the world.

But, in some parts of the world, it is pretty cheap, for those of us
that HAVE cheap bandwidth, we could be using each others and let those
of you with pay-by-the-minute access, small pipes, or bandwidth usage
limits make better use of the mirrors.  Everybody is happy :) 

> 
> > A bittorrent server (this is the infrastructure part), that maintains
> > torrents of the larger packages (I define larger as 20MB+), such as
> > XFree.
> Looking at the master distfiles directory, there are 343 files larger
> than 10MiB. Only 140 files larger than 20MiB. There are 9316 files in
> total, spanning 18Gb. The 343 files take up 9.4Gb of space. So they are
> definetely out of proportion. So having a lower limit would be better.

I chose 20MiB as an arbitrary number as "something that doesn't download
instantaneously on my cable modem", lowering it to 10MiB would still
work.


Cheers,

Caleb


--
gentoo-dev@gentoo.org mailing list