[gentoo-dev] rsync mirrorstats page (generation and parsing)

[gentoo-dev] rsync mirrorstats page (generation and parsing)

[Sebastian Pipping]

Hello!


When collecting information on the SYNC variable for my Summer of Code
gentoo stats project I'd like to check if the URL in SYNC is publically
known or some private/secret rsync mirror.  The page behind

  http://mirrorstats.gentoo.org/rsync/

seems to show that publicly known list.
As parsing the current page format is not trivial I'd like to know

 - if there is code parsing that page anywhere already

 - if the script generating that page could be extended
   to produce an XML equivalent of the provided information.

Integration of such a parser into app-portage/mirrorselect could be
interesting, too.



Sebastian

[gentoo-dev] mirrorselect: request for extraction of class MirrorParser

[Sebastian Pipping]

Hello!


app-portage/mirrorselect is a single file Python program.
It contains a class MirrorParser that parses mirrors.xml from the Gentoo
website.  I would like to use that code (unmodified) for my GSoC
project.

  My request is to extract an extra file for that class from
  mirrorselect so the Gentoo part of smolt can depend on mirrorselect
  in the near future instead of shipping a "dependency fork" forever.

mirrorselect's author seems to be Colin Kingsley (tercel@g.o.) whom
I cannot find in the list of current gentoo devs.

Please let me know if you can help/guide/.. with this.



Sebastian

Re: [gentoo-dev] rsync mirrorstats page (generation and parsing)

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1941 bytes --]

On Sun, Jul 05, 2009 at 02:44:07AM +0200, Sebastian Pipping wrote:
> When collecting information on the SYNC variable for my Summer of Code
> gentoo stats project I'd like to check if the URL in SYNC is publically
> known or some private/secret rsync mirror.  The page behind
>   http://mirrorstats.gentoo.org/rsync/
Mirrorstats is known to be out of date, because somebody needs to sit
down and integrate it with the datasources, so manual updates aren't
needed. Even better, would be hooking it into bouncer2 for the sentry
output.

It needs somebody to update it and hook at into the SOURCE of this data:
http://www.gentoo.org/main/en/mirrors3.xml

But wait, you say, that page is distfiles mirrors? Mirror-admin have a
common data source, non-published as it contains private contact details
for each administrator. From that data source, mirrors3 and rsync
mirrors gets updated.

mirrors.xml - old page, only used by mirrorselect now, manually updated.
mirrors3.xml - new page, generated from internal dataset.
mirrors2.xml - not a real page (See
http://www.gentoo.org/main/en/mirrors2.xml?passthru=1 and the magic
<mirrorlist> element.

Relatedly, the original author of mirrorselect retired from Gentoo
several years ago. The tools-portage team maintain it now, so you should
co-operate with them. It would be nice if they implemented the mirrors3
usage too, I think mirror-admin asked them more than a year ago, but I
can't find the bug.

In the meantime, for your original question:
> is the URL in SYNC public or private
Simply check by matching against gentoo.org$ in the hostname part of the
field.

P.S. Please report empty SYNC variables too ;-). These turn up when
users/devs have their tree coming from a VCS instead of rsync.

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 330 bytes --]

Re: [gentoo-dev] rsync mirrorstats page (generation and parsing)

[Sebastian Pipping]

Robin H. Johnson wrote:
> On Sun, Jul 05, 2009 at 02:44:07AM +0200, Sebastian Pipping wrote:
>> When collecting information on the SYNC variable for my Summer of Code
>> gentoo stats project I'd like to check if the URL in SYNC is publically
>> known or some private/secret rsync mirror.  The page behind
>>   http://mirrorstats.gentoo.org/rsync/
> Mirrorstats is known to be out of date, because somebody needs to sit
> down and integrate it with the datasources, so manual updates aren't
> needed. Even better, would be hooking it into bouncer2 for the sentry
> output.

What are these datasources?

What kind of integration are you thinking of?


> It needs somebody to update it and hook at into the SOURCE of this data:
> http://www.gentoo.org/main/en/mirrors3.xml
> 
> But wait, you say, that page is distfiles mirrors? Mirror-admin have a
> common data source, non-published as it contains private contact details
> for each administrator. From that data source, mirrors3 and rsync
> mirrors gets updated.

I see.


> mirrors.xml - old page, only used by mirrorselect now, manually updated.
> mirrors3.xml - new page, generated from internal dataset.
> mirrors2.xml - not a real page (See
> http://www.gentoo.org/main/en/mirrors2.xml?passthru=1 and the magic
> <mirrorlist> element.

Compared to

   [..]/mirrors.xml?passthru=1

it seems to me that on mirror3

   [..]/mirrors3.xml?passthru=1

passthru= is working in the opposite direction:

  1 turns style sheets on
  0 turns them off (default)

The one for mirrors3 makes less sense to me.
Is this inconsistency intended?


> Relatedly, the original author of mirrorselect retired from Gentoo
> several years ago. The tools-portage team maintain it now, so you should
> co-operate with them. It would be nice if they implemented the mirrors3
> usage too, I think mirror-admin asked them more than a year ago, but I
> can't find the bug.

I agree that would be a good idea and another reason to touch
mirrorselect.  Does it have a source repo somewhere?, not seen any.


> In the meantime, for your original question:
>> is the URL in SYNC public or private
> Simply check by matching against gentoo.org$ in the hostname part of the
> field.

Good idea, now implemented:

http://git.goodpoint.de/?p=smolt-gentoo.git;a=commitdiff;h=aeb14433e7c29a6045fb702775a3455ebb61aa1d


> P.S. Please report empty SYNC variables too ;-). These turn up when
> users/devs have their tree coming from a VCS instead of rsync.

Good point.  Now also implemented, same commit as above.



Sebastian

Re: [gentoo-dev] rsync mirrorstats page (generation and parsing)

[Alec Warner]

On Sat, Jul 4, 2009 at 6:48 PM, Robin H. Johnson<robbat2@gentoo.org> wrote:
> On Sun, Jul 05, 2009 at 02:44:07AM +0200, Sebastian Pipping wrote:
>> When collecting information on the SYNC variable for my Summer of Code
>> gentoo stats project I'd like to check if the URL in SYNC is publically
>> known or some private/secret rsync mirror.  The page behind
>>   http://mirrorstats.gentoo.org/rsync/
> Mirrorstats is known to be out of date, because somebody needs to sit
> down and integrate it with the datasources, so manual updates aren't
> needed. Even better, would be hooking it into bouncer2 for the sentry
> output.
>
> It needs somebody to update it and hook at into the SOURCE of this data:
> http://www.gentoo.org/main/en/mirrors3.xml
>
> But wait, you say, that page is distfiles mirrors? Mirror-admin have a
> common data source, non-published as it contains private contact details
> for each administrator. From that data source, mirrors3 and rsync
> mirrors gets updated.
>
> mirrors.xml - old page, only used by mirrorselect now, manually updated.
> mirrors3.xml - new page, generated from internal dataset.
> mirrors2.xml - not a real page (See
> http://www.gentoo.org/main/en/mirrors2.xml?passthru=1 and the magic
> <mirrorlist> element.
>
> Relatedly, the original author of mirrorselect retired from Gentoo
> several years ago. The tools-portage team maintain it now, so you should
> co-operate with them. It would be nice if they implemented the mirrors3
> usage too, I think mirror-admin asked them more than a year ago, but I
> can't find the bug.

+cc tools-portage

shit, I think I was the last one to touch that thing ;p

Where is mirrorselect hiding these days, a private git repo?

-A

>
> In the meantime, for your original question:
>> is the URL in SYNC public or private
> Simply check by matching against gentoo.org$ in the hostname part of the
> field.
>
> P.S. Please report empty SYNC variables too ;-). These turn up when
> users/devs have their tree coming from a VCS instead of rsync.
>
> --
> Robin Hugh Johnson
> Gentoo Linux Developer & Infra Guy
> E-Mail     : robbat2@gentoo.org
> GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85
>

Re: [gentoo-dev] rsync mirrorstats page (generation and parsing)

[Zac Medico]

Alec Warner wrote:
> Where is mirrorselect hiding these days, a private git repo?

Yeah, here's the history since I started maintaining it:

http://dev.gentoo.org/~zmedico/projects/mirrorselect.git/
-- 
Thanks,
Zac

Re: [gentoo-dev] rsync mirrorstats page (generation and parsing)

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 567 bytes --]

On Mon, Jul 06, 2009 at 12:28:59AM -0700, Zac Medico wrote:
> Alec Warner wrote:
> > Where is mirrorselect hiding these days, a private git repo?
> Yeah, here's the history since I started maintaining it:
> http://dev.gentoo.org/~zmedico/projects/mirrorselect.git/
I'll try to suck that down soon and build up a larger history with old
tarballs, and then push it somewhere useful.

Thanks for it.

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 330 bytes --]

Re: [gentoo-dev] rsync mirrorstats page (generation and parsing)

[Sebastian Pipping]

Robin H. Johnson wrote:
> I'll try to suck that down soon and build up a larger history with old
> tarballs, and then push it somewhere useful.

To re-build mirrorselect's complete history we'd need the original
tarballs for each line starting with "[ ] " below.

Please let us now if you have some of these files on some harddisc of
yours.  Here's the list

[F] 0.1
[F] 0.1-r1
[F] 0.1-r2
[F] 0.2
[F] 0.2-r1
[F] 0.3
[ ] 0.4
[ ] 0.5
[ ] 0.6
[ ] 0.7
[ ] 0.7-r1
[ ] 0.8
[ ] 0.81
[ ] 0.82
[R] 0.82-r1
[R] 0.82-r2
[R] 0.82-r3
[R] 0.83
[R] 0.84
[ ] 0.85
[ ] 0.86
[R] 0.87
[R] 0.89
[ ] 1.0
[ ] 1.0.1
[ ] 1.0.2
[ ] 1.0.3
[ ] 1.0.4_rc2
[ ] 1.1
[ ] 1.1.1
[ ] 1.1.2
[ ] 1.1.3
[ ] 1.1.4
[ ] 1.1.5
[R] 1.1.6
[R] 1.1.7
[R] 1.2
[R] 1.3
[R] 1.4
[R] 1.4.1
[R] 1.4.2


F = files dir in
http://sources.gentoo.org/viewcvs.py/gentoo-x86/app-admin/mirrorselect/?hideattic=0
R = http://dev.gentoo.org/~robbat2/mirrorselect-archive/



Sebastian

Re: [gentoo-dev] rsync mirrorstats page (generation and parsing)

[Sebastian Pipping]

Zac Medico wrote:
> Yeah, here's the history since I started maintaining it:
> 
> http://dev.gentoo.org/~zmedico/projects/mirrorselect.git/

I've been adding mirror3.xml support to the above today.

Repo over here:
http://git.goodpoint.de/?p=mirrorselect.git;a=summary



Sebastian

Re: [gentoo-dev] rsync mirrorstats page (generation and parsing)

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 767 bytes --]

On Wed, Jul 08, 2009 at 12:09:21AM +0200, Sebastian Pipping wrote:
> Zac Medico wrote:
> > Yeah, here's the history since I started maintaining it:
> > http://dev.gentoo.org/~zmedico/projects/mirrorselect.git/
> I've been adding mirror3.xml support to the above today.
> Repo over here:
> http://git.goodpoint.de/?p=mirrorselect.git;a=summary
Nice, thanks for the start.

The mirrors3 format was designed to explicitly make it clear:
- Geographic structure: region -> country -> mirror
- Separate protocols
- IPv4, IPv6: mirrors may have either or both.
- Easy separation of partial mirrors.

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 330 bytes --]

Re: [gentoo-dev] rsync mirrorstats page (generation and parsing)

[Sebastian Pipping]

Hello!


So what's needed to get a new mirrorselect release out?



Sebastian

Re: [gentoo-dev] rsync mirrorstats page (generation and parsing)

[Zac Medico]

Sebastian Pipping wrote:
> Hello!
> 
> 
> So what's needed to get a new mirrorselect release out?

Are all of your changes here?

git://git.goodpoint.de/mirrorselect.git

Now we just need to create an ebuild to install it, and put it in
the tree. You can file a bug for that and assign it to tools-portage.

> 
> 
> 
> Sebastian


-- 
Thanks,
Zac

Re: [gentoo-dev] rsync mirrorstats page (generation and parsing)

[Sebastian Pipping]

Zac Medico wrote:
>> So what's needed to get a new mirrorselect release out?
> 
> Are all of your changes here?
> 
> git://git.goodpoint.de/mirrorselect.git

Yes.


> Now we just need to create an ebuild to install it, and put it in
> the tree. You can file a bug for that and assign it to tools-portage.

Done.

https://bugs.gentoo.org/show_bug.cgi?id=278351



Sebastian

Re: [gentoo-dev] mirrorselect: request for extraction of class MirrorParser

[Zac Medico]

Sebastian Pipping wrote:
> Hello!
> 
> 
> app-portage/mirrorselect is a single file Python program.
> It contains a class MirrorParser that parses mirrors.xml from the Gentoo
> website.  I would like to use that code (unmodified) for my GSoC
> project.
> 
>   My request is to extract an extra file for that class from
>   mirrorselect so the Gentoo part of smolt can depend on mirrorselect
>   in the near future instead of shipping a "dependency fork" forever.
> 
> mirrorselect's author seems to be Colin Kingsley (tercel@g.o.) whom
> I cannot find in the list of current gentoo devs.
> 
> Please let me know if you can help/guide/.. with this.
> 
> 
> 
> Sebastian

It's only about 30 lines of code, so I wouldn't worry too much about
this "dependency fork" thing. Maybe if it was larger then I would.
So, I suggest that you simply copy it and include a comment about
its origin.

-- 
Thanks,
Zac