[gentoo-dev] Using emerge-webrsync to simplify the handbook

[gentoo-dev] Using emerge-webrsync to simplify the handbook

[Richard Yao]

[-- Attachment #1: Type: text/plain, Size: 162 bytes --]

We could slightly simplify the handbook installation procedure if we
told people to use emerge-webrsync to fetch the initial snapshot. What
do people think?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Maxim Kammerer]

On Wed, Nov 28, 2012 at 3:54 PM, Richard Yao <ryao@gentoo.org> wrote:
> We could slightly simplify the handbook installation procedure if we
> told people to use emerge-webrsync to fetch the initial snapshot.

Using emerge-webrsync also makes the installation process more robust,
since it only requires HTTP access (whereas many firewalls restrict
RSYNC). Besides, emerge-webrsync can check PGP signatures, so I think
that it should be the primary recommended portage tree synchronization
method.

-- 
Maxim Kammerer
Liberté Linux: http://dee.su/liberte

Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Richard Yao]

[-- Attachment #1: Type: text/plain, Size: 905 bytes --]

On 11/28/2012 09:17 AM, Maxim Kammerer wrote:
> On Wed, Nov 28, 2012 at 3:54 PM, Richard Yao <ryao@gentoo.org> wrote:
>> We could slightly simplify the handbook installation procedure if we
>> told people to use emerge-webrsync to fetch the initial snapshot.
> 
> Using emerge-webrsync also makes the installation process more robust,
> since it only requires HTTP access (whereas many firewalls restrict
> RSYNC). Besides, emerge-webrsync can check PGP signatures, so I think
> that it should be the primary recommended portage tree synchronization
> method.
> 

The only downside of which I am aware is increased network traffic.
However, we could redesign emerge-webrsync to take advantage of GNU
Tar's incremental archive functionality.

That would permit us to mirror compressed diffs in addition to regular
portage snapshots. Doing that well could reduce bandwidth requirements.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Matthew Thode]

[-- Attachment #1: Type: text/plain, Size: 1050 bytes --]

On 11/28/2012 09:05 AM, Richard Yao wrote:
> On 11/28/2012 09:17 AM, Maxim Kammerer wrote:
>> On Wed, Nov 28, 2012 at 3:54 PM, Richard Yao <ryao@gentoo.org> wrote:
>>> We could slightly simplify the handbook installation procedure if we
>>> told people to use emerge-webrsync to fetch the initial snapshot.
>>
>> Using emerge-webrsync also makes the installation process more robust,
>> since it only requires HTTP access (whereas many firewalls restrict
>> RSYNC). Besides, emerge-webrsync can check PGP signatures, so I think
>> that it should be the primary recommended portage tree synchronization
>> method.
>>
> 
> The only downside of which I am aware is increased network traffic.
> However, we could redesign emerge-webrsync to take advantage of GNU
> Tar's incremental archive functionality.
> 
> That would permit us to mirror compressed diffs in addition to regular
> portage snapshots. Doing that well could reduce bandwidth requirements.
> 
weekly fulls and daily diffs?

-- 
-- Matthew Thode (prometheanfire)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1376 bytes --]

On Wed, 28 Nov 2012 10:05:55 -0500
Richard Yao <ryao@gentoo.org> wrote:

> On 11/28/2012 09:17 AM, Maxim Kammerer wrote:
> > On Wed, Nov 28, 2012 at 3:54 PM, Richard Yao <ryao@gentoo.org> wrote:
> >> We could slightly simplify the handbook installation procedure if we
> >> told people to use emerge-webrsync to fetch the initial snapshot.
> > 
> > Using emerge-webrsync also makes the installation process more robust,
> > since it only requires HTTP access (whereas many firewalls restrict
> > RSYNC). Besides, emerge-webrsync can check PGP signatures, so I think
> > that it should be the primary recommended portage tree synchronization
> > method.
> > 
> 
> The only downside of which I am aware is increased network traffic.
> However, we could redesign emerge-webrsync to take advantage of GNU
> Tar's incremental archive functionality.
> 
> That would permit us to mirror compressed diffs in addition to regular
> portage snapshots. Doing that well could reduce bandwidth requirements.

There's emerge-delta-webrsync but it's mostly hand-work to reconstruct
the webrsync tarball. Therefore, it is very slow and not worth
the effort when syncing often.

However, I'm not aware of gnu tar's incremental archive. If it's much
faster than the above, then it should probably replace
emerge-delta-webrsync.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Markos Chandras]

On 28 November 2012 13:54, Richard Yao <ryao@gentoo.org> wrote:
> We could slightly simplify the handbook installation procedure if we
> told people to use emerge-webrsync to fetch the initial snapshot. What
> do people think?
>

Seems a good improvement to me.

-- 
Regards,
Markos Chandras / Gentoo Linux Developer / Key ID: B4AFF2C2

Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Sven Vermeulen]

[-- Attachment #1: Type: text/plain, Size: 389 bytes --]

On Nov 29, 2012 10:24 AM, "Markos Chandras" <hwoarang@gentoo.org> wrote:
> > We could slightly simplify the handbook installation procedure if we
> > told people to use emerge-webrsync to fetch the initial snapshot. What
> > do people think?
> >
>
> Seems a good improvement to me.

I'm ok with it as well. I'll draft up something. I suspect emerge-webrsync
is available on all platforms?

[-- Attachment #2: Type: text/html, Size: 551 bytes --]

Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Richard Yao]

[-- Attachment #1: Type: text/plain, Size: 497 bytes --]

On 11/30/2012 06:46 AM, Sven Vermeulen wrote:
> On Nov 29, 2012 10:24 AM, "Markos Chandras" <hwoarang@gentoo.org> wrote:
>>> We could slightly simplify the handbook installation procedure if we
>>> told people to use emerge-webrsync to fetch the initial snapshot. What
>>> do people think?
>>>
>>
>> Seems a good improvement to me.
> 
> I'm ok with it as well. I'll draft up something. I suspect emerge-webrsync
> is available on all platforms?
> 

It is part of sys-apps/portage.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Richard Yao]

[-- Attachment #1: Type: text/plain, Size: 1632 bytes --]

On 11/28/2012 11:08 AM, Matthew Thode wrote:
> On 11/28/2012 09:05 AM, Richard Yao wrote:
>> On 11/28/2012 09:17 AM, Maxim Kammerer wrote:
>>> On Wed, Nov 28, 2012 at 3:54 PM, Richard Yao <ryao@gentoo.org> wrote:
>>>> We could slightly simplify the handbook installation procedure if we
>>>> told people to use emerge-webrsync to fetch the initial snapshot.
>>>
>>> Using emerge-webrsync also makes the installation process more robust,
>>> since it only requires HTTP access (whereas many firewalls restrict
>>> RSYNC). Besides, emerge-webrsync can check PGP signatures, so I think
>>> that it should be the primary recommended portage tree synchronization
>>> method.
>>>
>>
>> The only downside of which I am aware is increased network traffic.
>> However, we could redesign emerge-webrsync to take advantage of GNU
>> Tar's incremental archive functionality.
>>
>> That would permit us to mirror compressed diffs in addition to regular
>> portage snapshots. Doing that well could reduce bandwidth requirements.
>>
> weekly fulls and daily diffs?
> 

Determining what is right here probably requires calculus, but this
scheme does not seem like a bad choice to me. My main concern is that
maintaining weekly full snapshots would require too much space for the
mirrors. It might be better go monthly, with diffs on the following
intervals:

1 week
1 day
30 minutes

Doing that would eliminate the benefit of rsync entirely, with the
caveat that we now need to mirror a ton of diffs. This would make it
easy for us to provide the ability to obtain historical snapshots, which
would be nice.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Michael Mol]

On Fri, Nov 30, 2012 at 10:57 AM, Richard Yao <ryao@gentoo.org> wrote:
> On 11/28/2012 11:08 AM, Matthew Thode wrote:
>> On 11/28/2012 09:05 AM, Richard Yao wrote:
>>> On 11/28/2012 09:17 AM, Maxim Kammerer wrote:
>>>> On Wed, Nov 28, 2012 at 3:54 PM, Richard Yao <ryao@gentoo.org> wrote:
>>>>> We could slightly simplify the handbook installation procedure if we
>>>>> told people to use emerge-webrsync to fetch the initial snapshot.
>>>>
>>>> Using emerge-webrsync also makes the installation process more robust,
>>>> since it only requires HTTP access (whereas many firewalls restrict
>>>> RSYNC). Besides, emerge-webrsync can check PGP signatures, so I think
>>>> that it should be the primary recommended portage tree synchronization
>>>> method.
>>>>
>>>
>>> The only downside of which I am aware is increased network traffic.
>>> However, we could redesign emerge-webrsync to take advantage of GNU
>>> Tar's incremental archive functionality.
>>>
>>> That would permit us to mirror compressed diffs in addition to regular
>>> portage snapshots. Doing that well could reduce bandwidth requirements.
>>>
>> weekly fulls and daily diffs?
>>
>
> Determining what is right here probably requires calculus, but this
> scheme does not seem like a bad choice to me. My main concern is that
> maintaining weekly full snapshots would require too much space for the
> mirrors. It might be better go monthly, with diffs on the following
> intervals:
>
> 1 week
> 1 day
> 30 minutes
>
> Doing that would eliminate the benefit of rsync entirely, with the
> caveat that we now need to mirror a ton of diffs. This would make it
> easy for us to provide the ability to obtain historical snapshots, which
> would be nice.

Worth noting that all this moves us nicely in the direction of
allowing HTTP proxies to cache data, reducing load on mirrors. And
moves us in the direction of implementing mirrors themselves as a
network of caching proxies.

--
:wq

Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 30/11/12 11:15 AM, Michael Mol wrote:
> On Fri, Nov 30, 2012 at 10:57 AM, Richard Yao <ryao@gentoo.org>
> wrote:
>> On 11/28/2012 11:08 AM, Matthew Thode wrote:
>>> On 11/28/2012 09:05 AM, Richard Yao wrote:
>>>> On 11/28/2012 09:17 AM, Maxim Kammerer wrote:
>>>>> On Wed, Nov 28, 2012 at 3:54 PM, Richard Yao
>>>>> <ryao@gentoo.org> wrote:
>>>>>> We could slightly simplify the handbook installation
>>>>>> procedure if we told people to use emerge-webrsync to
>>>>>> fetch the initial snapshot.
>>>>> 
>>>>> Using emerge-webrsync also makes the installation process
>>>>> more robust, since it only requires HTTP access (whereas
>>>>> many firewalls restrict RSYNC). Besides, emerge-webrsync
>>>>> can check PGP signatures, so I think that it should be the
>>>>> primary recommended portage tree synchronization method.
>>>>> 
>>>> 
>>>> The only downside of which I am aware is increased network
>>>> traffic. However, we could redesign emerge-webrsync to take
>>>> advantage of GNU Tar's incremental archive functionality.
>>>> 
>>>> That would permit us to mirror compressed diffs in addition
>>>> to regular portage snapshots. Doing that well could reduce
>>>> bandwidth requirements.
>>>> 
>>> weekly fulls and daily diffs?
>>> 
>> 
>> Determining what is right here probably requires calculus, but
>> this scheme does not seem like a bad choice to me. My main
>> concern is that maintaining weekly full snapshots would require
>> too much space for the mirrors. It might be better go monthly,
>> with diffs on the following intervals:
>> 
>> 1 week 1 day 30 minutes
>> 
>> Doing that would eliminate the benefit of rsync entirely, with
>> the caveat that we now need to mirror a ton of diffs. This would
>> make it easy for us to provide the ability to obtain historical
>> snapshots, which would be nice.
> 
> Worth noting that all this moves us nicely in the direction of 
> allowing HTTP proxies to cache data, reducing load on mirrors. And 
> moves us in the direction of implementing mirrors themselves as a 
> network of caching proxies.
> 

Idea makes sense, I wonder if implementation would be better served by
leveraging the fact that we already produce daily full snapshots:

1 - continue to provide the daily snapshots we do now
2 - provide two weeks (more?) of daily diffs, such that a daily
snapshot from up to two weeks ago can be updated to present day
3 - provide hourly or 30-minute update diffs to get latest changes.

If the tree is more than two weeks old, emerge-webrsync would just
grab the latest daily plus the hourly diffs.

If the tree is less than two weeks old, grab the daily diffs and
hourly diffs.  The local copy of the tree itself would need to be
rolled back to the best-available daily diff before these diff updates
could be applied; this may mean that a local cache of the latest
full-day snapshot needs to be kept and/or generated.  Also if said
cache doesn't exist, then the whole full-day snapshot would be grabbed.

The advantage to this would be significantly fewer distfiles, although
the logic in emerge-webrsync would possibly be more complex.

Regarding rolling back the local tree to a known-good state, I think
that would be required regardless of the method as any local changes
made to the tree by users would need to be discarded, right?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlC45f0ACgkQ2ugaI38ACPChKgD9GOBptQ9jJ1/eYyq1NEl5Oq1E
dVy9UOab80bG5FZB9LwBAKwsifnT+iE3n/4d/ljnuT2qCnbtXNYr7yBjF/VcEpkq
=y9eB
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Maxim Kammerer]

On Fri, Nov 30, 2012 at 5:57 PM, Richard Yao <ryao@gentoo.org> wrote:
> My main concern is that
> maintaining weekly full snapshots would require too much space for the
> mirrors.

Mirrors already keep last 8 full daily snapshots, in 2 formats each, e.g.:
http://mirrors.kernel.org/gentoo/snapshots/

-- 
Maxim Kammerer
Liberté Linux: http://dee.su/liberte

Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Alec Warner]

On Fri, Nov 30, 2012 at 8:59 AM, Ian Stakenvicius <axs@gentoo.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 30/11/12 11:15 AM, Michael Mol wrote:
>> On Fri, Nov 30, 2012 at 10:57 AM, Richard Yao <ryao@gentoo.org>
>> wrote:
>>> On 11/28/2012 11:08 AM, Matthew Thode wrote:
>>>> On 11/28/2012 09:05 AM, Richard Yao wrote:
>>>>> On 11/28/2012 09:17 AM, Maxim Kammerer wrote:
>>>>>> On Wed, Nov 28, 2012 at 3:54 PM, Richard Yao
>>>>>> <ryao@gentoo.org> wrote:
>>>>>>> We could slightly simplify the handbook installation
>>>>>>> procedure if we told people to use emerge-webrsync to
>>>>>>> fetch the initial snapshot.
>>>>>>
>>>>>> Using emerge-webrsync also makes the installation process
>>>>>> more robust, since it only requires HTTP access (whereas
>>>>>> many firewalls restrict RSYNC). Besides, emerge-webrsync
>>>>>> can check PGP signatures, so I think that it should be the
>>>>>> primary recommended portage tree synchronization method.
>>>>>>
>>>>>
>>>>> The only downside of which I am aware is increased network
>>>>> traffic. However, we could redesign emerge-webrsync to take
>>>>> advantage of GNU Tar's incremental archive functionality.
>>>>>
>>>>> That would permit us to mirror compressed diffs in addition
>>>>> to regular portage snapshots. Doing that well could reduce
>>>>> bandwidth requirements.
>>>>>
>>>> weekly fulls and daily diffs?
>>>>
>>>
>>> Determining what is right here probably requires calculus, but
>>> this scheme does not seem like a bad choice to me. My main
>>> concern is that maintaining weekly full snapshots would require
>>> too much space for the mirrors. It might be better go monthly,
>>> with diffs on the following intervals:
>>>
>>> 1 week 1 day 30 minutes
>>>
>>> Doing that would eliminate the benefit of rsync entirely, with
>>> the caveat that we now need to mirror a ton of diffs. This would
>>> make it easy for us to provide the ability to obtain historical
>>> snapshots, which would be nice.
>>
>> Worth noting that all this moves us nicely in the direction of
>> allowing HTTP proxies to cache data, reducing load on mirrors. And
>> moves us in the direction of implementing mirrors themselves as a
>> network of caching proxies.
>>
>
> Idea makes sense, I wonder if implementation would be better served by
> leveraging the fact that we already produce daily full snapshots:
>
> 1 - continue to provide the daily snapshots we do now
> 2 - provide two weeks (more?) of daily diffs, such that a daily
> snapshot from up to two weeks ago can be updated to present day
> 3 - provide hourly or 30-minute update diffs to get latest changes.
>
> If the tree is more than two weeks old, emerge-webrsync would just
> grab the latest daily plus the hourly diffs.
>
> If the tree is less than two weeks old, grab the daily diffs and
> hourly diffs.  The local copy of the tree itself would need to be
> rolled back to the best-available daily diff before these diff updates
> could be applied; this may mean that a local cache of the latest
> full-day snapshot needs to be kept and/or generated.  Also if said
> cache doesn't exist, then the whole full-day snapshot would be grabbed.
>
> The advantage to this would be significantly fewer distfiles, although
> the logic in emerge-webrsync would possibly be more complex.
>
> Regarding rolling back the local tree to a known-good state, I think
> that would be required regardless of the method as any local changes
> made to the tree by users would need to be discarded, right?

How about we not change the docs until someone eagerly implements all
the stuff you just said?

Note that from an infra POV our existing system works fairly well and
requires no day-to-day tinkering.
I'm always happy to consider new options, but work needs to put in to
make it feasible. I'm sure if we switched to http with zsync or
something, we could make it feasible. I want to see a prototype, etc.

-A

>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.19 (GNU/Linux)
>
> iF4EAREIAAYFAlC45f0ACgkQ2ugaI38ACPChKgD9GOBptQ9jJ1/eYyq1NEl5Oq1E
> dVy9UOab80bG5FZB9LwBAKwsifnT+iE3n/4d/ljnuT2qCnbtXNYr7yBjF/VcEpkq
> =y9eB
> -----END PGP SIGNATURE-----
>

Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Zac Medico]

On 11/28/2012 09:50 AM, Michał Górny wrote:
> On Wed, 28 Nov 2012 10:05:55 -0500
> Richard Yao <ryao@gentoo.org> wrote:
> 
>> On 11/28/2012 09:17 AM, Maxim Kammerer wrote:
>>> On Wed, Nov 28, 2012 at 3:54 PM, Richard Yao <ryao@gentoo.org> wrote:
>>>> We could slightly simplify the handbook installation procedure if we
>>>> told people to use emerge-webrsync to fetch the initial snapshot.
>>>
>>> Using emerge-webrsync also makes the installation process more robust,
>>> since it only requires HTTP access (whereas many firewalls restrict
>>> RSYNC). Besides, emerge-webrsync can check PGP signatures, so I think
>>> that it should be the primary recommended portage tree synchronization
>>> method.
>>>
>>
>> The only downside of which I am aware is increased network traffic.
>> However, we could redesign emerge-webrsync to take advantage of GNU
>> Tar's incremental archive functionality.
>>
>> That would permit us to mirror compressed diffs in addition to regular
>> portage snapshots. Doing that well could reduce bandwidth requirements.
> 
> There's emerge-delta-webrsync but it's mostly hand-work to reconstruct
> the webrsync tarball. Therefore, it is very slow and not worth
> the effort when syncing often.

At least because I maintain emerge-delta-webrsync, I use it regularly as
my sync method. Latest versions of emerge-delta-webrsync use a temp
directory inside $PORTAGE_TMPDIR/portage, on which I have a tmpfs
filesystem mounted. With tmpfs, performance does not seem so bad (using
a sandy bridge core i5 here).

> However, I'm not aware of gnu tar's incremental archive. If it's much
> faster than the above, then it should probably replace
> emerge-delta-webrsync.

If it has benefits over the current diffball approach used by
emerge-delta-webrsync, then it seems like a good idea. It would be nice
to integrate it directly into emerge-webrsync, and eventually deprecate
emerge-delta-webrsync.
-- 
Thanks,
Zac

Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Ian Stakenvicius]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 30/11/12 12:30 PM, Alec Warner wrote:
>> How about we not change the docs until someone eagerly implements
>> all the stuff you just said?
> 

Well, using emerge-webrsync for grabbing the initial snapshot during
an installation still makes sense regardless of whether or not we do
any of the above, iirc that was the original documentation request
change wasn't it?


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iF4EAREIAAYFAlC49Y4ACgkQ2ugaI38ACPDETgD/ZzleSQxyWCrgBYc8vg5Wvviz
QVpnQa7CRF33qVYvYkkBAKDUx6bB2LbPmGQgFm22su72i32tKfUqiKOgEs6tgboa
=CNSD
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Alec Warner]

On Fri, Nov 30, 2012 at 10:06 AM, Ian Stakenvicius <axs@gentoo.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> On 30/11/12 12:30 PM, Alec Warner wrote:
>>> How about we not change the docs until someone eagerly implements
>>> all the stuff you just said?
>>
>
> Well, using emerge-webrsync for grabbing the initial snapshot during
> an installation still makes sense regardless of whether or not we do
> any of the above, iirc that was the original documentation request
> change wasn't it?

Zac can probably comment on where it fetches from. I can say with some
certainty that we have enough rsync capacity, I can't say the same for
HTTP based services.

-A

>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.19 (GNU/Linux)
>
> iF4EAREIAAYFAlC49Y4ACgkQ2ugaI38ACPDETgD/ZzleSQxyWCrgBYc8vg5Wvviz
> QVpnQa7CRF33qVYvYkkBAKDUx6bB2LbPmGQgFm22su72i32tKfUqiKOgEs6tgboa
> =CNSD
> -----END PGP SIGNATURE-----
>

Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Zac Medico]

[-- Attachment #1: Type: text/plain, Size: 883 bytes --]

On 11/30/2012 09:55 PM, Alec Warner wrote:
> On Fri, Nov 30, 2012 at 10:06 AM, Ian Stakenvicius <axs@gentoo.org> wrote:
> On 30/11/12 12:30 PM, Alec Warner wrote:
>>>>> How about we not change the docs until someone eagerly implements
>>>>> all the stuff you just said?
>>>>
> 
> Well, using emerge-webrsync for grabbing the initial snapshot during
> an installation still makes sense regardless of whether or not we do
> any of the above, iirc that was the original documentation request
> change wasn't it?
> 
>> Zac can probably comment on where it fetches from. I can say with some
>> certainty that we have enough rsync capacity, I can't say the same for
>> HTTP based services.

I just did a quick test on 107 http mirrors returned from mirrorselect
--list-only, and found that 102 returned a mirror://snapshots/index.html
containing the string "portage-2012".
-- 
Thanks,
Zac

[-- Attachment #2: Add-mirrorselect-list-only-option.patch --]
[-- Type: text/x-patch, Size: 1576 bytes --]

From a4126418bf102f70249092cab9fa294be5b27635 Mon Sep 17 00:00:00 2001
From: Zac Medico <zmedico@gentoo.org>
Date: Fri, 30 Nov 2012 23:28:30 -0800
Subject: [PATCH] Add mirrorselect --list-only option

---
 mirrorselect/main.py | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/mirrorselect/main.py b/mirrorselect/main.py
index 55c1cb0..02f15ae 100755
--- a/mirrorselect/main.py
+++ b/mirrorselect/main.py
@@ -260,6 +260,9 @@ class MirrorSelect(object):
 
 		group = parser.add_option_group("Other options")
 		group.add_option(
+			"--list-only", action="store_true", default=False,
+			help="Output the list of available candidate host urls and then quit.")
+		group.add_option(
 			"-o", "--output", action="store_true", default=False,
 			help="Output Only Mode, this is especially useful "
 			"when being used during installation, to redirect "
@@ -319,7 +322,7 @@ class MirrorSelect(object):
 				'You do not appear to have netselect on your system. '
 				'You must use the -D flag')
 
-		if (os.getuid() != 0) and not options.output:
+		if (os.getuid() != 0) and not (options.output or options.list_only):
 			self.output.print_err('Must be root to write to %s!\n' % config_path)
 
 		if args:
@@ -393,6 +396,10 @@ class MirrorSelect(object):
 		fsmirrors = self.get_filesystem_mirrors(config_path, options.rsync)
 
 		hosts = self.get_available_hosts(options)
+		if options.list_only:
+			sys.stdout.write("".join("%s\n" % url[0] for url in hosts))
+			sys.stdout.flush()
+			return
 
 		urls = self.select_urls(hosts, options)
 
-- 
1.8.0


[-- Attachment #3: snapshots_http_urls.txt --]
[-- Type: text/plain, Size: 4849 bytes --]

http://archive.mmu.edu.my/gentoo/snapshots/
http://cesium.di.uminho.pt/pub/gentoo/snapshots/
http://chi-10g-1-mirror.fastsoft.net/pub/linux/gentoo/gentoo-distfiles/snapshots/
http://darkstar.ist.utl.pt/gentoo/snapshots/
http://de-mirror.org/gentoo/snapshots/
http://distfiles.gentoo.bg/snapshots/
http://files.gentoo.gr/snapshots/
http://ftp.cc.uoc.gr/mirrors/linux/gentoo/snapshots/
http://ftp.daum.net/gentoo/snapshots/
http://ftp.dei.uc.pt/pub/linux/gentoo/snapshots/
http://ftp.df.lth.se/pub/gentoo/snapshots/
http://ftp.fi.muni.cz/pub/linux/gentoo/snapshots/
http://ftp.gentoo.bg/snapshots/
http://ftp.halifax.rwth-aachen.de/gentoo/snapshots/
http://ftp.heanet.ie/pub/gentoo/snapshots/
http://ftp.iij.ad.jp/pub/linux/gentoo/snapshots/
http://ftp.jaist.ac.jp/pub/Linux/Gentoo/snapshots/
http://ftp.kaist.ac.kr/pub/gentoo/snapshots/
http://ftp.klid.dk/ftp/gentoo/snapshots/
http://ftp.lecl.net/pub/gentoo/snapshots/
http://ftp.linux.org.tr/gentoo/snapshots/
http://ftp.ntua.gr/pub/linux/gentoo/snapshots/
http://ftp.rhnet.is/pub/gentoo/snapshots/
http://ftp.rnl.ist.utl.pt/pub/gentoo/gentoo-distfiles/snapshots/
http://ftp.romnet.org/gentoo/snapshots/
http://ftp.snt.utwente.nl/pub/os/linux/gentoo/snapshots/
http://ftp.spline.inf.fu-berlin.de/mirrors/gentoo/snapshots/
http://ftp-stud.hs-esslingen.de/pub/Mirrors/gentoo/snapshots/
http://ftp.swin.edu.au/gentoo/snapshots/
http://ftp.twaren.net/Linux/Gentoo/snapshots/
http://ftp.ucsb.edu/pub/mirrors/linux/gentoo/snapshots/
http://ftp.uni-erlangen.de/pub/mirrors/gentoo/snapshots/
http://ftp.vectranet.pl/gentoo/snapshots/
http://gd.tuwien.ac.at/opsys/linux/gentoo/snapshots/
http://gentoo.aditsu.net:8000/snapshots/
http://gentoo.arcticnetwork.ca/snapshots/
http://gentoo.bloodhost.ru/snapshots/
http://gentoo.c3sl.ufpr.br/snapshots/
http://gentoo.channelx.biz/snapshots/
http://gentoo.cites.uiuc.edu/pub/gentoo/snapshots/
http://gentoo.cs.nctu.edu.tw/gentoo/snapshots/
http://gentoo.cs.uni.edu/snapshots/
http://gentoo-euetib.upc.es/mirror/gentoo/snapshots/
http://gentoo.gg3.net/snapshots/
http://gentoo.gossamerhost.com/snapshots/
http://gentoo.inf.elte.hu/snapshots/
http://gentoo.inode.at/snapshots/
http://gentoo.in.th/snapshots/
http://gentoo.iteam.net.ua/snapshots/
http://gentoo.kems.net/snapshots/
http://gentoo.kiev.ua/ftp/snapshots/
http://gentoo.lagis.at/snapshots/
http://gentoo.llarian.net/snapshots/
http://gentoo.localhost.net.ar/snapshots/
http://gentoo.mirror.dkm.cz/pub/gentoo/snapshots/
http://gentoo.mirror.pw.edu.pl/snapshots/
http://gentoo.mirrors.easynews.com/linux/gentoo/snapshots/
http://gentoo.mirrors.hoobly.com/snapshots/
http://gentoo.mirrors.pair.com/snapshots/
http://gentoo.mirrors.tds.net/gentoo/snapshots/
http://gentoo.mirrors.tera-byte.com/snapshots/
http://gentoo.mirror.web4u.cz/snapshots/
http://gentoo.mneisen.org/snapshots/
http://gentoo.modulix.net/gentoo/snapshots/
http://gentoo.netnitco.net/snapshots/
http://gentoo.osuosl.org/snapshots/
http://gentoo.po.opole.pl/snapshots/
http://gentoo.prz.rzeszow.pl/snapshots/
http://gentoo.supp.name/snapshots/
http://gentoo.tiscali.nl/snapshots/
http://gentoo.tups.lv/source/snapshots/
http://gentoo.wheel.sk/snapshots/
http://linux.rz.ruhr-uni-bochum.de/download/gentoo-mirror/snapshots/
http://lug.mtu.edu/gentoo/snapshots/
http://mirror2.corbina.ru/gentoo-distfiles/snapshots/
http://mirror.bytemark.co.uk/gentoo/snapshots/
http://mirror.csclub.uwaterloo.ca/gentoo-distfiles/snapshots/
http://mirror.datapipe.net/gentoo/snapshots/
http://mirror.iawnet.sandia.gov/gentoo/snapshots/
http://mirror.isoc.org.il/pub/gentoo/snapshots/
http://mirror.leaseweb.com/gentoo/snapshots/
http://mirror.lug.udel.edu/pub/gentoo/snapshots/
http://mirror.mcs.anl.gov/pub/gentoo/snapshots/
http://mirror.mdfnet.se/gentoo/snapshots/
http://mirror.neolabs.kz/gentoo/pub/snapshots/
http://mirror.netcologne.de/gentoo/snapshots/
http://mirror.opteamax.de/gentoo/snapshots/
http://mirror.ovh.net/gentoo-distfiles/snapshots/
http://mirror.qubenet.net/mirror/gentoo/snapshots/
http://mirrors.163.com/gentoo/snapshots/
http://mirrors.linuxant.fr/distfiles.gentoo.org/snapshots/
http://mirrors.rit.edu/gentoo/snapshots/
http://mirrors.sohu.com/gentoo/snapshots/
http://mirrors.stuhome.net/gentoo/snapshots/
http://mirrors.telepoint.bg/gentoo/snapshots/
http://mirror.switch.ch/ftp/mirror/gentoo/snapshots/
http://mirrors.xmu.edu.cn/gentoo/snapshots/
http://mirrors.xservers.ro/gentoo/snapshots/
http://mirror.the-best-hosting.net/snapshots/
http://mirror.usu.edu/mirrors/gentoo/snapshots/
http://mirror.yandex.ru/gentoo-distfiles/snapshots/
http://portage.org.ua/snapshots/
http://trumpetti.atm.tut.fi/gentoo/snapshots/
http://tux.rainside.sk/gentoo/snapshots/
http://www.gtlib.gatech.edu/pub/gentoo/snapshots/
http://www.las.ic.unicamp.br/pub/gentoo/snapshots/
http://www.mirrorservice.org/sites/www.ibiblio.org/gentoo/snapshots/

Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Richard Yao]

[-- Attachment #1: Type: text/plain, Size: 475 bytes --]

On 11/30/2012 01:06 PM, Ian Stakenvicius wrote:
> On 30/11/12 12:30 PM, Alec Warner wrote:
>>> How about we not change the docs until someone eagerly implements
>>> all the stuff you just said?
> 
> 
> Well, using emerge-webrsync for grabbing the initial snapshot during
> an installation still makes sense regardless of whether or not we do
> any of the above, iirc that was the original documentation request
> change wasn't it?
> 
> 
> 

That is correct.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Richard Yao]

[-- Attachment #1: Type: text/plain, Size: 1028 bytes --]

On 11/30/2012 12:30 PM, Alec Warner wrote:
> How about we not change the docs until someone eagerly implements all
> the stuff you just said?
> 
> Note that from an infra POV our existing system works fairly well and
> requires no day-to-day tinkering.
> I'm always happy to consider new options, but work needs to put in to
> make it feasible. I'm sure if we switched to http with zsync or
> something, we could make it feasible. I want to see a prototype, etc.
> 
> -A

The initial suggestion was to use emerge-webrsync to fetch the initial
portage snapshot, which requires no code changes.

A suggestion was made in response was that we should take this a step
further by using emerge-webrsync for everything. I feel that the code
needs improvements before it can formally replace `emerge --sync` as our
recommended way of updating the tree. Others seem to agree.

That does not impact our ability to use emerge-webrsync to fetch the
initial snapshot in the handbook, which is my original suggestion.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Robin H. Johnson]

On Fri, Nov 30, 2012 at 09:35:07AM -0800, Zac Medico wrote:
> > However, I'm not aware of gnu tar's incremental archive. If it's much
> > faster than the above, then it should probably replace
> > emerge-delta-webrsync.
> If it has benefits over the current diffball approach used by
> emerge-delta-webrsync, then it seems like a good idea. It would be nice
> to integrate it directly into emerge-webrsync, and eventually deprecate
> emerge-delta-webrsync.
I went and did a rough comparison of Tar incrementals vs the existing
deltas.

TL;DR:
======
- Existing deltas are 8-9x better than other options.
- We should consider retaining monthly snapshots, plus all the deltas.

Results:
========
1.
Using bzip2 -9 compression:
- Existing deltas are 9x smaller than tar-incremental.
- Existing deltas are 8x smaller than rsync-batch.

2.
If you just want to save bandwidth, the average full snapshot,
compressed w/ BZIP2, is 55M. The average delta is 269k.
55M/269k = ~209.
Ergo it is LESS bandwidth to download ~180 deltas and apply those than
it is to download the full snapshot (assuming upstream side of the
transaction accounts for ~30 snapshots worth of overhead).

Notes:
======
1.
Extracting tar incrementals, you must be VERY careful to perform
operations in the correct order, otherwise removed files will not
actually be deleted.

2.
When the Git repo goes live, we should tag at the point we take the
daily snapshot, and use this to also consider git bundles.

Numbers:
========

Baseline tarball:
57919736 portage-20121123.0.tar.bz2

Tar incrementals, daily:
 2554334 portage-20121123-20121124.1.tar.bz2
 2045216 portage-20121124-20121125.1.tar.bz2
 1936313 portage-20121125-20121126.1.tar.bz2
 2355342 portage-20121126-20121127.1.tar.bz2
 2063612 portage-20121127-20121128.1.tar.bz2
 2582600 portage-20121128-20121129.1.tar.bz2
 2720135 portage-20121129-20121130.1.tar.bz2

Rsync incrementals, daily:
 2224311 portage-20121123-20121124.rsync-batch.bz2
 1869241 portage-20121124-20121125.rsync-batch.bz2
 1802648 portage-20121125-20121126.rsync-batch.bz2
 1936937 portage-20121126-20121127.rsync-batch.bz2
 1868771 portage-20121127-20121128.rsync-batch.bz2
 2240386 portage-20121128-20121129.rsync-batch.bz2
 2028207 portage-20121129-20121130.rsync-batch.bz2

Existing deltas, daily:
 252400 snapshot-20121123-20121124.patch.bz2
 267094 snapshot-20121124-20121125.patch.bz2
 161136 snapshot-20121125-20121126.patch.bz2
 225349 snapshot-20121126-20121127.patch.bz2
 245804 snapshot-20121127-20121128.patch.bz2
 232549 snapshot-20121128-20121129.patch.bz2
 332835 snapshot-20121129-20121130.patch.bz2

Rsync incrementals, from baseline:
 2224311 portage-20121123-20121124.rsync-batch.bz2
 2536620 portage-20121123-20121125.rsync-batch.bz2
 2700715 portage-20121123-20121126.rsync-batch.bz2
 2986403 portage-20121123-20121127.rsync-batch.bz2
 3258723 portage-20121123-20121128.rsync-batch.bz2
 3824015 portage-20121123-20121129.rsync-batch.bz2
 4232674 portage-20121123-20121130.rsync-batch.bz2

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Sven Vermeulen]

On Fri, Nov 30, 2012 at 10:44:31AM -0500, Richard Yao wrote:
> On 11/30/2012 06:46 AM, Sven Vermeulen wrote:
> > On Nov 29, 2012 10:24 AM, "Markos Chandras" <hwoarang@gentoo.org> wrote:
> > > > We could slightly simplify the handbook installation procedure if we
> > > > told people to use emerge-webrsync to fetch the initial snapshot. What
> > > > do people think?
> > >
> > > Seems a good improvement to me.
> > 
> > I'm ok with it as well. I'll draft up something. I suspect emerge-webrsync
> > is available on all platforms?
> 
> It is part of sys-apps/portage.

Okay, I've updated the instructions.

First, I removed the references to the stage3 snapshots on the "Universal
CD" as I don't think we have a (recent enough) Universal CD, and we would
recommend the stage3 files from our mirrors anyway.

Second, the Portage tree snapshots are now installed through emerge-webrsync
(which means the entire section on downloading the tarballs, checking
integrity, extracting is now a single paragraph).

Finally, the section on updating the Portage tree (using emerge --sync) is
now marked as optional, with the remark that if you're behind a firewall you
can safely ignore this section as the user will already have a quite
up-to-date tree installed.

I don't know if we should remove the section altogether (about emerge
--sync) or not. It is a small step and users *will* create bug reports about
it if they don't notice it in the documentation anymore. Marking it optional
seems to be a good approach here.

Wkr,
	Sven Vermeulen

PS Commit was made a few minutes ago, so please give it an hour before it
shows up on the site.

[gentoo-dev] Handbook updates Was: Using emerge-webrsync to simplify the handbook

[Duncan]

Sven Vermeulen posted on Sat, 08 Dec 2012 20:41:42 +0000 as excerpted:

> On Fri, Nov 30, 2012 at 10:44:31AM -0500, Richard Yao wrote:
>> On 11/30/2012 06:46 AM, Sven Vermeulen wrote:
>> > On Nov 29, 2012 10:24 AM, "Markos Chandras" <hwoarang@gentoo.org>
>> > wrote:
>> > > > We could slightly simplify the handbook installation procedure if
>> > > > we told people to use emerge-webrsync to fetch the initial
>> > > > snapshot. What do people think?

> Okay, I've updated the instructions.

> Second, the Portage tree snapshots are now installed through
> emerge-webrsync (which means the entire section on downloading the
> tarballs, checking integrity, extracting is now a single paragraph).

Thanks, swift.  =:^)

> Finally, the section on updating the Portage tree (using emerge --sync)
> is now marked as optional, with the remark that if you're behind a
> firewall you can safely ignore this section as the user will already
> have a quite up-to-date tree installed.
> 
> I don't know if we should remove the section altogether (about emerge
> --sync) or not. It is a small step and users *will* create bug reports
> about it if they don't notice it in the documentation anymore. Marking
> it optional seems to be a good approach here.

++ on optional.

IMHO, one of the finer points of a good handbook-based install is that it 
"sets the user up for success" by encouraging correct routine maintenance 
practices in the future, stepping them thru the process once during the 
initial install.  Unfortunately, there's /way/ too many users that read 
only the handbook's install section (and perhaps Working with Gentoo) and 
never read the rest, so instilling at least some minimal basics of good 
routine practices there is important.

Thus, even if the webrsync process grabbed them a reasonably fresh (24 
hours) initial tree, keeping the now optional section on updating the 
tree to the very latest is a good idea, since even if it's optional, 
it'll step the new user thru the process at least once, establishing a 
good idea of that routine basic as a good practice for future updates.

...

For the same reason (but likely more controversially), I'd actually argue 
that the install section should (optionally?) step thru installing 
gentoolkit and running revdep-rebuild and emerge --depclean (with an 
appropriate --ask/--pretend first) as the (near) final steps of the 
initial install, as well, even if gentoolkit is the only thing in @world 
when they do so, and it's technically unnecessary as there should be 
nothing to rebuild or depclean.  For depclean especially, establishing 
the practice right at the beginning will help to ensure that @world 
always contains all desired "leaf" packages, and that as a result, users 
will never accumulate a huge backlog of "uncovered" packages subject to 
getting stale due to lack of @world updating, and that an initial depclean 
run when they DO get around to it won't "accidentally" uninstall half 
their system, because it never got in @world to begin with.  (Of course, 
a successful emerge run now includes a helpful depclean reminder anyway, 
but stepping the user thru the process once during the initial guided 
install certainly can't hurt.)

A quick review of the installation section in the manual suggests either 
covering this as part of chapter 11, Finalizing your Gentoo Installation, 
or instead, inserting it as chapter 11, bumping the current 11 and 12 to 
12 and 13.  I'd suggest the latter, with a title for the new chapter 11 
of something like "Your first emerge --update @world".  It could 
encourage readers to look at the working with gentoo and working with 
portage sections for more, but would be an initial step-thru of a basic 
emerge --update @world and related maintenance (allowing a condensing of 
the corresponding chapters in the other sections), for those who never 
read past install in the handbook.  Additionally, with such a title it'd 
be easier for users to refer back to (and see the encouragement to read 
the other sections again, when they maybe have more time to do so) for 
their second update, etc, if they needed to.


One more suggestion while I'm at it.  I remember all those many years ago 
when I first installed gentoo, even after reading and absorbing the 
handbook well enough to be pointing people to specific chapters and 
subchapters in answer to specific questions on the user list, I was still 
somewhat confused about the exact nature of the world list, and what 
packages should be listed there vs. those that shouldn't.  Unless I've 
overlooked it, to this day there's still not a real good explanation of 
the distinction between "leaf packages" and dependencies, and why "leaf 
packages" belong in @world but dependencies don't.  Something along the 
lines of:

"""""

If you want to use and care about that specific package, put it in the 
world list with emerge <package>, without --oneshot.  If you don't really 
care about that specific package and it's simply pulled in by something 
else that you DO care about, but you happen to be specifically emerging 
it anyway, use emerge --oneshot <package>, so it doesn't end up in the 
world list and emerge --depclean can unmerge it in the future if the 
packages pulling it in no longer need it.

"""""

(Actually, here, all my default emerge stub-scripts use --oneshot by 
default.  I use a special "2" (not oneshot) version when I WANT something 
added to @world, which for me is a sort of package testing purgatory 
between "I for sure want to keep it and it's in a set listed in 
world_sets accordingly" and "I'm not sure I really want to keep it yet 
and thus want to be reminded about it the next time I update and then run 
depclean --ask to cleanup.")

As a result of my confusion, I ended up with more packages in the world 
list than I really needed/wanted, and somewhere along the line, I had to 
use some tool (I guess it'd be emaint --clean world, now) to weed out the 
unnecessary cruft in my world list.

(Later, with kde4 and portage 2.2 with sets support, I actually went thru 
and broke my world file into about a dozen individual sets by functional 
category, starting with the kde sets which I sync to those in the kde 
overlay with every 6-month kde4-minor bump, but also including sets such 
as net-admin with traceroute, etc, and net-user with firefox, etc.  This 
was actually as a result of sorting out my existing world list to copy it 
to my netbook, but I've kept everything in sets since then, and I've run 
with an empty world list for about three years now, since everything is 
in sets as enumerated in my world_sets file.  That practice keeps the 
sets that replace my world file REAL clean, since everything listed is in 
its own functional category and each set is small enough it's immediately 
clear upon cat-ing the set whether every package listed in the set is 
something I actually care about or not.  (For the kde sets I keep synced 
with the ones in the overlay, I comment out the libs and apps I don't 
need or care about, but keep them in the set, commented, for easier 
syncing with the overlay versions at the next six-month bump.)  I'm 
really looking forward to the day when proper sets support appears in 
stable portage, and sets can eliminate the unsorted world list mess for 
others just as they have for me.  Unfortunately, that looks to be 
something of a "bluesky" enhancement bug for stable or even ~arch, but at 
least 2.2's there to be unmasked for people who like me have real, 
practical use for sets.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Rich Freeman]

On Sat, Dec 8, 2012 at 3:41 PM, Sven Vermeulen <swift@gentoo.org> wrote:
> Second, the Portage tree snapshots are now installed through emerge-webrsync
> (which means the entire section on downloading the tarballs, checking
> integrity, extracting is now a single paragraph).

Uh, does emerge-webrsync have some kind of automagical detection of
the fact that you're building a chroot?  I would think that it would
sync /usr/portage, and not /mnt/gentoo/usr/portage.  Perhaps you
should move that down into the chroot section, and mkdir /usr/portage
if that is needed?

Rich

Re: [gentoo-dev] Using emerge-webrsync to simplify the handbook

[Sven Vermeulen]

On Sat, Dec 08, 2012 at 08:10:32PM -0500, Rich Freeman wrote:
> Uh, does emerge-webrsync have some kind of automagical detection of
> the fact that you're building a chroot?  I would think that it would
> sync /usr/portage, and not /mnt/gentoo/usr/portage.  Perhaps you
> should move that down into the chroot section, and mkdir /usr/portage
> if that is needed?

Crap. Indeed, section moved towards the place where we optionally recommend
"emerge --sync", and put in an mkdir /usr/portage.

Wkr,
	Sven Vermeulen