[gentoo-mirrors] Gentoo Linux 2007.0 preparations

[gentoo-mirrors] Gentoo Linux 2007.0 preparations

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1479 bytes --]

Salutations to all mirror administrators.

jforman has been busy with real life, so some of the other infra folk
are stepping in to help him out with getting this going.

As with previous releases, don't distribute this information beyond this
list quite yet, just to avoid hassle.

Would all mirrors that carry the experimental tree please see my
follow-up email about BitTorrent HTTP-seeding?

The overall release size has shrunk by almost 2Gb since 2006.1.

Full file listing for releases and experimental is attached.

Rough size breakdown:

240M	releases/alpha
1.3G	releases/amd64
928M	releases/hppa
259M	releases/ia64
386M	releases/mips
3.3G	releases/ppc
34M		releases/snapshots
1.4G	releases/sparc
1.7G	releases/x86

releases-2006.1 = 11891568kb
releases-2007.0 =  9780540kb

629M	experimental/ia64
699M	experimental/ppc
376M	experimental/s390
180M	experimental/sh

experimental-2006.1 = 1828120kb
experimental-2007.0 = 1927008kb

In the identical fashion to jforman's previous release work, we're going
to pre-stage our mirrors, starting with the smallest architectures and
ending with the big ones, followed by checking up on things, and then
flipping the bit later.

As always, many thanks for your ongoing support of Gentoo, all of us
really appreciate it.

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Council Member
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 321 bytes --]

[gentoo-mirrors] Gentoo Linux 2007.0 and BitTorrent

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 3310 bytes --]

On Thu, May 03, 2007 at 04:34:19PM -0700, Robin H. Johnson wrote:
> Would all mirrors that carry the experimental tree please see my
> follow-up email about BitTorrent HTTP-seeding?

Hi folks,

So now stuff on BitTorrent for the 2007.0 release.

There's two portions to this email, firstly stuff distributed by
BitTorrent only, and then everything else.

We have 94 torrents in total, 2 of which are only media that is only
being distributed by BitTorrent.

We needs seeds for all of it, but the bittorrent-only stuff more than
the rest. I have one sponsor offering 50Mbit * 2 weeks for seeding
everything so far.

In terms of basic structure, there is a .torrent for every .tar.bz2 or
.iso in releases or experimental, containing the core file, plus it's
DIGESTS, signature, and CONTENTS files.

BitTorrent-only:
3.7G	livedvd-amd64-installer-2007.0/
3.8G	livedvd-i686-installer-2007.0/

If you're interested in direct BitTorrent seeding (esp. for the above
two, even if you aren't seeding anything else), please give me a shout
so that I can arrange to get them to you.

Now for the experimental part...

To help out long-term seeding of the majority of torrents (those that
aren't bittorrent-only), I'd like to take advantage of the newish
HTTP-seeding support of BitTorrent [1]. 

HTTP-seeding kicks in at a MUCH lower priority than regular BitTorrent
peering, basically helping clients that cannot contact any other source
for missing pieces. It effectively converts the BitTorrent client to a
slightly-smart HTTP fetcher that then shares the blocks back over
BitTorrent to other clients.

For all experimental/ mirrors, that means we're adding a directory tree
under experimental/ that has directories containing only symlinks. Your
rsync options will need to have the various symlink-transfer options,
which I think should be in place already.

Here's an example of one.
experimental/torrent-symlinks/stage3-x86-2007.0:
lrwxrwxrwx stage3-x86-2007.0.tar.bz2 -> ../../releases/x86/2007.0/stages/stage3-x86-2007.0.tar.bz2
lrwxrwxrwx stage3-x86-2007.0.tar.bz2.CONTENTS -> ../../releases/x86/2007.0/stages/stage3-x86-2007.0.tar.bz2.CONTENTS
lrwxrwxrwx stage3-x86-2007.0.tar.bz2.DIGESTS -> ../../releases/x86/2007.0/stages/stage3-x86-2007.0.tar.bz2.DIGESTS
lrwxrwxrwx stage3-x86-2007.0.tar.bz2.asc -> ../../releases/x86/2007.0/stages/stage3-x86-2007.0.tar.bz2.asc

This in itself doesn't make the BitTorrent clients contact you, so even
if you aren't participating, it doesn't cause harm to have it.

For part two of the HTTP-seeding (eg making the BT clients actually use
the mirrors), I'd like to know which of the experimental mirrors are
interested in being used for HTTP seeding. 

Your webservers will need to support HTTP Range requests for this to
work. The BitTorrent clients will send range requests for 32K..256K
blocks (powers of two only).

I'll be placing the base potion of your URLs into the .torrent files, as
"$BASEURL/experimental/torrent-symlinks/"

[1] The actual specification for the HTTP-seeding is here:
    http://www.getright.com/seedtorrent.html

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Council Member
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 321 bytes --]

Re: [gentoo-mirrors] Gentoo Linux 2007.0 and BitTorrent

[Martin Eisenhardt]

[-- Attachment #1: Type: text/plain, Size: 1257 bytes --]

Hello Robin,
hello list,

On Friday 04 May 2007 03:44:06 Robin H. Johnson wrote:
> BitTorrent-only:
> 3.7G  livedvd-amd64-installer-2007.0/
> 3.8G  livedvd-i686-installer-2007.0/
>
> If you're interested in direct BitTorrent seeding (esp. for the above
> two, even if you aren't seeding anything else), please give me a shout
> so that I can arrange to get them to you.

Shout. :-D

I am interested in seeding the new DVD images, although not at the 50 Mbps you 
mentioned  in your posting. But a few Mbps might help as well, I guess ...

> For part two of the HTTP-seeding (eg making the BT clients actually use
> the mirrors), I'd like to know which of the experimental mirrors are
> interested in being used for HTTP seeding.
>
> Your webservers will need to support HTTP Range requests for this to
> work. The BitTorrent clients will send range requests for 32K..256K
> blocks (powers of two only).

Assuming that a vanilla Apache 2.0 supports range requests (which it does, 
AFAIK), I would like to volunteer for that as well.

Just contact me with the details about how to get the files and what 
information you need.

Kind regards
Martin Eisenhardt
---
GnuPG key: 3BBC EBB2 4A36 D2AD E636 556F 30D9 0EE4 C7F1 C1CF

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-mirrors] Gentoo Linux 2007.0 and BitTorrent

[Georgi Georgiev]

[-- Attachment #1: Type: text/plain, Size: 1137 bytes --]

maillog: 03/05/2007-18:44:06(-0700): Robin H. Johnson types
...
> For part two of the HTTP-seeding (eg making the BT clients actually use
> the mirrors), I'd like to know which of the experimental mirrors are
> interested in being used for HTTP seeding. 
> 
> Your webservers will need to support HTTP Range requests for this to
> work. The BitTorrent clients will send range requests for 32K..256K
> blocks (powers of two only).
...

How is HTTP BT seeding different than ordinary mirroring (assuming the
server supports Range)? As I don't need to do anything on the server
side (that's my understanding for the moment at least), is there a
particular reason why I should care? We (mirrors) are still going to
serve that same content that we have already agreed to serve.

If this was just out of courtesy, then sure, thanks for asking,
apologies for the above, and feel free to add gg3.net to those torrent
files.

-- 
(*   Georgi Georgiev   (* "Go to Heaven for the climate, Hell for    (*
*)    chutz@gg3.net    *) the company." -- Mark Twain                *)
(* http://www.gg3.net/ (*                                            (*

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-mirrors] Gentoo Linux 2007.0 and BitTorrent

[Francois Petillon]

Robin H. Johnson wrote:
> BitTorrent-only:
> 3.7G	livedvd-amd64-installer-2007.0/
> 3.8G	livedvd-i686-installer-2007.0/

After a small discussion with Robin on IRC, I would like to know if I am 
the only one who is willing to host these two ISOs on my mirror (FTP 
only) or if other people would appreciate to host them too...

François
-- 
gentoo-mirrors@gentoo.org mailing list

Re: [gentoo-mirrors] Gentoo Linux 2007.0 and BitTorrent

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1800 bytes --]

On Fri, May 04, 2007 at 06:57:21PM +0900, Georgi Georgiev wrote:
> > Your webservers will need to support HTTP Range requests for this to
> > work. The BitTorrent clients will send range requests for 32K..256K
> > blocks (powers of two only).
> How is HTTP BT seeding different than ordinary mirroring (assuming the
> server supports Range)? As I don't need to do anything on the server
> side (that's my understanding for the moment at least), is there a
> particular reason why I should care? We (mirrors) are still going to
> serve that same content that we have already agreed to serve.
I asked it mainly as courtesy of the mirrors, but also because I was
concerned about any cases where it might cause trouble for the mirrors:

- In the best case, if your-choice-httpd supports both KeepAlive and
  Range, then the BitTorrent client will pick a point N, and just
  sequentially ask for block N, N+1, N+2, ... on that single HTTP
  connection - this takes advantage of disk read-ahead on the server
  side.
- If your httpd does not support range, there's going to be a separate
  connection for each block, which might accidentally trigger some abuse
  detection systems. A client with 1Mbit could cause 3 connections per
  second to your mirror.
- If your httpd does not support Range, the BitTorrent client would
  fail-out.
- From a discussion that was asked of me on #gentoo-mirrors, if your
  mirror server used some non-default IO readahead (eg posix_fadvise),
  there's also the possibility that it would increase your disk
  thrashing since it asks for lots of small chunks instead of big ones.

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Council Member
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 321 bytes --]

RE: [gentoo-mirrors] Gentoo Linux 2007.0 and BitTorrent

[Greg Cornell]

Hi Robin,

I'm happy to serve up the content via whatever protocols Gentoo needs.
However, I haven't set up bittorrent before so maybe a short howto on
settting up bittorrent (and even checking for the range support in apache)
would be helpful.

Greg

> -----Original Message-----
> From: Robin H. Johnson [mailto:robbat2@gentoo.org]
> Sent: Thursday, May 03, 2007 6:44 PM
> To: gentoo-mirrors@lists.gentoo.org
> Cc: wolf31o2@gentoo.org
> Subject: [gentoo-mirrors] Gentoo Linux 2007.0 and BitTorrent
> 
> On Thu, May 03, 2007 at 04:34:19PM -0700, Robin H. Johnson wrote:
> > Would all mirrors that carry the experimental tree please see my
> > follow-up email about BitTorrent HTTP-seeding?
> 
> Hi folks,
> 
> So now stuff on BitTorrent for the 2007.0 release.
> 
> There's two portions to this email, firstly stuff distributed by
> BitTorrent only, and then everything else.
> 
> We have 94 torrents in total, 2 of which are only media that is only
> being distributed by BitTorrent.
> 
> We needs seeds for all of it, but the bittorrent-only stuff more than
> the rest. I have one sponsor offering 50Mbit * 2 weeks for seeding
> everything so far.
> 
> In terms of basic structure, there is a .torrent for every .tar.bz2 or
> .iso in releases or experimental, containing the core file, plus it's
> DIGESTS, signature, and CONTENTS files.
> 
> BitTorrent-only:
> 3.7G	livedvd-amd64-installer-2007.0/
> 3.8G	livedvd-i686-installer-2007.0/
> 
> If you're interested in direct BitTorrent seeding (esp. for the above
> two, even if you aren't seeding anything else), please give me a shout
> so that I can arrange to get them to you.
> 
> Now for the experimental part...
> 
> To help out long-term seeding of the majority of torrents (those that
> aren't bittorrent-only), I'd like to take advantage of the newish HTTP-
> seeding support of BitTorrent [1].
> 
> HTTP-seeding kicks in at a MUCH lower priority than regular BitTorrent
> peering, basically helping clients that cannot contact any other source
> for missing pieces. It effectively converts the BitTorrent client to a
> slightly-smart HTTP fetcher that then shares the blocks back over
> BitTorrent to other clients.
> 
> For all experimental/ mirrors, that means we're adding a directory tree
> under experimental/ that has directories containing only symlinks. Your
> rsync options will need to have the various symlink-transfer options,
> which I think should be in place already.
> 
> 
> Here's an example of one.
> experimental/torrent-symlinks/stage3-x86-2007.0:
> lrwxrwxrwx stage3-x86-2007.0.tar.bz2 ->
> ../../releases/x86/2007.0/stages/stage3-x86-2007.0.tar.bz2
> lrwxrwxrwx stage3-x86-2007.0.tar.bz2.CONTENTS ->
> ../../releases/x86/2007.0/stages/stage3-x86-2007.0.tar.bz2.CONTENTS
> lrwxrwxrwx stage3-x86-2007.0.tar.bz2.DIGESTS ->
> ../../releases/x86/2007.0/stages/stage3-x86-2007.0.tar.bz2.DIGESTS
> lrwxrwxrwx stage3-x86-2007.0.tar.bz2.asc ->
> ../../releases/x86/2007.0/stages/stage3-x86-2007.0.tar.bz2.asc
> 
> This in itself doesn't make the BitTorrent clients contact you, so even
> if you aren't participating, it doesn't cause harm to have it.
> 
> For part two of the HTTP-seeding (eg making the BT clients actually use
> the mirrors), I'd like to know which of the experimental mirrors are
> interested in being used for HTTP seeding.
> 
> Your webservers will need to support HTTP Range requests for this to
> work. The BitTorrent clients will send range requests for 32K..256K
> blocks (powers of two only).
> 
> I'll be placing the base potion of your URLs into the .torrent files,
> as "$BASEURL/experimental/torrent-symlinks/"
> 
> [1] The actual specification for the HTTP-seeding is here:
>     http://www.getright.com/seedtorrent.html
> 
> --
> Robin Hugh Johnson
> Gentoo Linux Developer & Council Member
> E-Mail     : robbat2@gentoo.org
> GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

-- 
gentoo-mirrors@gentoo.org mailing list

Re: [gentoo-mirrors] Gentoo Linux 2007.0 and BitTorrent

[Vladimir Brik]

I would be interested to in participating in both direct and HTTP
seeding using gentoo.chem.wisc.edu. The server is on a shared 100Mbps
connection, with about 80% of bandwidth being typically available. I
could also seed bittorrent-only files from a machine on a different
network for a limited time (no HTTP seeding for that one though) if
need be.

Vlad



On 5/3/07, Robin H. Johnson <robbat2@gentoo.org> wrote:
> On Thu, May 03, 2007 at 04:34:19PM -0700, Robin H. Johnson wrote:
> > Would all mirrors that carry the experimental tree please see my
> > follow-up email about BitTorrent HTTP-seeding?
>
> Hi folks,
>
> So now stuff on BitTorrent for the 2007.0 release.
>
> There's two portions to this email, firstly stuff distributed by
> BitTorrent only, and then everything else.
>
> We have 94 torrents in total, 2 of which are only media that is only
> being distributed by BitTorrent.
>
> We needs seeds for all of it, but the bittorrent-only stuff more than
> the rest. I have one sponsor offering 50Mbit * 2 weeks for seeding
> everything so far.
>
> In terms of basic structure, there is a .torrent for every .tar.bz2 or
> .iso in releases or experimental, containing the core file, plus it's
> DIGESTS, signature, and CONTENTS files.
>
> BitTorrent-only:
> 3.7G    livedvd-amd64-installer-2007.0/
> 3.8G    livedvd-i686-installer-2007.0/
>
> If you're interested in direct BitTorrent seeding (esp. for the above
> two, even if you aren't seeding anything else), please give me a shout
> so that I can arrange to get them to you.
>
> Now for the experimental part...
>
> To help out long-term seeding of the majority of torrents (those that
> aren't bittorrent-only), I'd like to take advantage of the newish
> HTTP-seeding support of BitTorrent [1].
>
> HTTP-seeding kicks in at a MUCH lower priority than regular BitTorrent
> peering, basically helping clients that cannot contact any other source
> for missing pieces. It effectively converts the BitTorrent client to a
> slightly-smart HTTP fetcher that then shares the blocks back over
> BitTorrent to other clients.
>
> For all experimental/ mirrors, that means we're adding a directory tree
> under experimental/ that has directories containing only symlinks. Your
> rsync options will need to have the various symlink-transfer options,
> which I think should be in place already.
>
> Here's an example of one.
> experimental/torrent-symlinks/stage3-x86-2007.0:
> lrwxrwxrwx stage3-x86-2007.0.tar.bz2 -> ../../releases/x86/2007.0/stages/stage3-x86-2007.0.tar.bz2
> lrwxrwxrwx stage3-x86-2007.0.tar.bz2.CONTENTS -> ../../releases/x86/2007.0/stages/stage3-x86-2007.0.tar.bz2.CONTENTS
> lrwxrwxrwx stage3-x86-2007.0.tar.bz2.DIGESTS -> ../../releases/x86/2007.0/stages/stage3-x86-2007.0.tar.bz2.DIGESTS
> lrwxrwxrwx stage3-x86-2007.0.tar.bz2.asc -> ../../releases/x86/2007.0/stages/stage3-x86-2007.0.tar.bz2.asc
>
> This in itself doesn't make the BitTorrent clients contact you, so even
> if you aren't participating, it doesn't cause harm to have it.
>
> For part two of the HTTP-seeding (eg making the BT clients actually use
> the mirrors), I'd like to know which of the experimental mirrors are
> interested in being used for HTTP seeding.
>
> Your webservers will need to support HTTP Range requests for this to
> work. The BitTorrent clients will send range requests for 32K..256K
> blocks (powers of two only).
>
> I'll be placing the base potion of your URLs into the .torrent files, as
> "$BASEURL/experimental/torrent-symlinks/"
>
> [1] The actual specification for the HTTP-seeding is here:
>     http://www.getright.com/seedtorrent.html
>
> --
> Robin Hugh Johnson
> Gentoo Linux Developer & Council Member
> E-Mail     : robbat2@gentoo.org
> GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85
>
>
-- 
gentoo-mirrors@gentoo.org mailing list

Re: [gentoo-mirrors] Gentoo Linux 2007.0 and BitTorrent

[Georgi Georgiev]

[-- Attachment #1: Type: text/plain, Size: 523 bytes --]

maillog: 04/05/2007-18:57:21(+0900): Георги Георгиев types
>
> If this was just out of courtesy, then sure, thanks for asking,
> apologies for the above, and feel free to add gg3.net to those torrent
> files.

Oops, scratch the above. I just realized that gg3.net is not serving
releases.

-- 
()   Georgi Georgiev   () Blow it out your ear.                      ()
()    chutz@gg3.net    ()                                            ()
() http://www.gg3.net/ ()                                            ()

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-mirrors] Gentoo Linux 2007.0 and BitTorrent

[Philip M. White]

[-- Attachment #1: Type: text/plain, Size: 1226 bytes --]

On Thu, May 03, 2007 at 06:44:06PM -0700, Robin H. Johnson wrote:
> So now stuff on BitTorrent for the 2007.0 release.
> 
> There's two portions to this email, firstly stuff distributed by
> BitTorrent only, and then everything else.
> 
> We have 94 torrents in total, 2 of which are only media that is only
> being distributed by BitTorrent.
> 
> We needs seeds for all of it, but the bittorrent-only stuff more than
> the rest. I have one sponsor offering 50Mbit * 2 weeks for seeding
> everything so far.
> 
> In terms of basic structure, there is a .torrent for every .tar.bz2 or
> .iso in releases or experimental, containing the core file, plus it's
> DIGESTS, signature, and CONTENTS files.
> 
> BitTorrent-only:
> 3.7G	livedvd-amd64-installer-2007.0/
> 3.8G	livedvd-i686-installer-2007.0/
> 
> If you're interested in direct BitTorrent seeding (esp. for the above
> two, even if you aren't seeding anything else), please give me a shout
> so that I can arrange to get them to you.

I've recently received permission from the network administration of my
mirror to run a BitTorrent client.  Where can I get these torrents
(other than downloading them off the web one-by-one)?

-- 
Philip

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-mirrors] Gentoo Linux 2007.0 and BitTorrent

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 607 bytes --]

On Tue, Jul 03, 2007 at 08:30:33PM -0500, Philip M. White wrote:
> I've recently received permission from the network administration of my
> mirror to run a BitTorrent client.  Where can I get these torrents
> (other than downloading them off the web one-by-one)?
http://torrents.gentoo.org/torrents/gentoo-torrents-2007.0.tar.gz
and then this one separately:
http://torrents.gentoo.org/torrents/install-x86-minimal-2007.0-r1.torrent

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Council Member
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 321 bytes --]