[gentoo-dev] mirroring distfiles/patches on patches.gentoo.org

[gentoo-dev] mirroring distfiles/patches on patches.gentoo.org

[Stefan Schweizer]

To my fellow Gentoo Devlopers and Users,

The Gentoo Infrastructure team proposed this idea in june 2006
 * Reduce mirror time for Gentoo specific patches/tarballs
 * Offer an official location for Gentoo specific patches/tarballs
   (Instead of using dev.g.o, this would be the official place)
 * Offer a distributed (3+ servers) mirror rotation for this

This would be the "official" location for those distfiles when infra makes
it available. I am asking them for status reports on this you will maybe
see some here :)

But for the meantime I am using http://gentooexperimental.org/~genstef/dist
for this purpose. The reason is that I can remove and add stuff there
myself and can keep control of what gets deleted. I can use my favourite
scripts to create distfiles and keep them ordered. For example I am using
this for firefox-2.0. This workaround I initially had to use when I
discovered a bug in the mirroring script that was annoying me regularly.
The mirroring script does ignore all RESTRICT=mirror SRC_URIs even those at
mirror://gentoo/.
"Bug 121332 ppp patch missing from distfiles"
"Bug 100260 The File foo2zjs-20050319.tar.gz missed on Gentoo-Mirrors"
Those get uploaded and look fine but after 6 months they are suddenly being
removed by the script.

Initially of course I needed a solution of that bug badly and because the
mirrors kept dropping it I have just uploaded it to dev.gentoo.org in lack
of any other proper place to put it. But using dev.gentoo.org is deprecated
by our infra because the server may not be able to cope with the traffic.
Fortunately later I was able to get an account on Patrick Lauers
development server http://gentooexperimental.org/~genstef/dist to upload my
distfiles there.

So this is my current practice and I am eagerly waiting for infra to allow
me to use an "official" service instead of my bandaid. Unfortunately some
people are really afraid of what I am doing and want to harass me into
using the mirror://gentoo that I do not want to use. This mail is dedicated
to explain the issue to those people. 

Please, dont argue such discussions that only give you personal satisfaction
of being correct or an excuse to annoy other people. Useless yelling at
each other because of a minority where opinions differ is misplaced in a
project that is driven by volunteers. That is also why I have stopped and
written this mail.

But I can see that you are a bit frustrated because you did not get your way
through. So here is a guide of what you can do. I will gladly stop using
workarounds when I am allowed to have an equally bugfree and fast workflow
as currently. What you can do:

1) write a patch for portage to allow granular mirror restrictions for the
SRC_URI and work with infra on improving the mirroring script to not ignore
mirror://gentoo in mirror-restricted ebuilds. Also for this solution the
time for a distfile in /space/distfiles-local to hit the first mirror
should be equal to the time for ebuilds to get into the rsync rotation so
that problems for users who are too early.

2) get infra to provide patches.gentoo.org as a permanent solution that I
have asked for since march 2005 and it sometimes even looked close to
getting it on bug 85098.

Thanks for understanding
Stefan Schweizer

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] mirroring distfiles/patches on patches.gentoo.org

[Alec Warner]

Stefan Schweizer wrote:
> To my fellow Gentoo Devlopers and Users,
> 
> The Gentoo Infrastructure team proposed this idea in june 2006
>  * Reduce mirror time for Gentoo specific patches/tarballs
>  * Offer an official location for Gentoo specific patches/tarballs
>    (Instead of using dev.g.o, this would be the official place)
>  * Offer a distributed (3+ servers) mirror rotation for this
> 
> This would be the "official" location for those distfiles when infra makes
> it available. I am asking them for status reports on this you will maybe
> see some here :)
> 
> But for the meantime I am using http://gentooexperimental.org/~genstef/dist
> for this purpose. The reason is that I can remove and add stuff there
> myself and can keep control of what gets deleted. I can use my favourite
> scripts to create distfiles and keep them ordered. For example I am using
> this for firefox-2.0. This workaround I initially had to use when I
> discovered a bug in the mirroring script that was annoying me regularly.
> The mirroring script does ignore all RESTRICT=mirror SRC_URIs even those at
> mirror://gentoo/.
> "Bug 121332 ppp patch missing from distfiles"
> "Bug 100260 The File foo2zjs-20050319.tar.gz missed on Gentoo-Mirrors"
> Those get uploaded and look fine but after 6 months they are suddenly being
> removed by the script.
> 

There is a distfiles whitelist[1] for this exact purpose...isn't there?

[1] 
http://www.gentoo.org/proj/en/infrastructure/mirrors/overview-distfile.xml
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] mirroring distfiles/patches on patches.gentoo.org

[Grant Goodyear]

[-- Attachment #1: Type: text/plain, Size: 436 bytes --]

Stefan Schweizer wrote: [Mon Oct 23 2006, 04:35:50PM CDT]
> To my fellow Gentoo Devlopers and Users,
[lots snipped]

Truly, the sarcasm doesn't help.  It takes a potentially reasonable
grievance and reduces it to just the author appearing to be a jerk.

-g2boojum-
-- 
Grant Goodyear	
Gentoo Developer
g2boojum@gentoo.org
http://www.gentoo.org/~g2boojum
GPG Fingerprint: D706 9802 1663 DEF5 81B0  9573 A6DC 7152 E0F6 5B76

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

[gentoo-dev] Re: mirroring distfiles/patches on patches.gentoo.org

[Stefan Schweizer]

Alec Warner wrote:
>> Those get uploaded and look fine but after 6 months they are suddenly
>> being removed by the script.
>> 
> 
> There is a distfiles whitelist[1] for this exact purpose...isn't there?
> 
> [1]
> http://www.gentoo.org/proj/en/infrastructure/mirrors/overview-distfile.xml

Thanks for the heads up. I was not aware of this when I was looking for a
solution badly back then. Must be a rather recent development. Looks easy
and straightforward - even deleting is possible. Good job, guys :)

Still the developer needs to know about it and think of it in the moment of
adding mirror restrictions to an ebuild with mirror://gentoo sources. A
risk that is not going to be taken in any of my ebuilds.

I hope the people that prefer mirror://gentoo know about the issue and the
solution :)

Thanks,
Stefan

-- 
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: mirroring distfiles/patches on patches.gentoo.org

[Stefan Schweizer]

Grant Goodyear wrote:
> Truly, the sarcasm doesn't help.  It takes a potentially reasonable
> grievance and reduces it to just the author appearing to be a jerk.

Sorry, I have no intention to cause any ill fate. Is meant explanatory.

- Stefan

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] mirroring distfiles/patches on patches.gentoo.org

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 1040 bytes --]

On Mon, Oct 23, 2006 at 11:35:50PM +0200, Stefan Schweizer wrote:
> But I can see that you are a bit frustrated because you did not get your way
> through. So here is a guide of what you can do. I will gladly stop using
> workarounds when I am allowed to have an equally bugfree and fast workflow
> as currently. What you can do:
> 
> 1) write a patch for portage to allow granular mirror restrictions for the
> SRC_URI and work with infra on improving the mirroring script to not ignore
> mirror://gentoo in mirror-restricted ebuilds.

Crack is bad for you, mm'kay...

That particular change was *already* added, despite my objections; bug 
number I don't have on hand, but...

acroread-7.0.8 (restrict=mirror)
wget http://gentoo.osuosl.org/distfiles/libcups.so-i386.bz2

doom3-cdoom-1 (restrict=mirror)
wget http://gentoo.osuosl.org/distfiles/doom3-cdoom.png

Spose worth noting portage also handles it fine, and that if it 
*didn't*, the person demanding changes usually needs to supply the 
patch...

~harring

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] Re: mirroring distfiles/patches on patches.gentoo.org

[Alec Warner]

Stefan Schweizer wrote:
> Alec Warner wrote:
>>> Those get uploaded and look fine but after 6 months they are suddenly
>>> being removed by the script.
>>>
>> There is a distfiles whitelist[1] for this exact purpose...isn't there?
>>
>> [1]
>> http://www.gentoo.org/proj/en/infrastructure/mirrors/overview-distfile.xml
> 
> Thanks for the heads up. I was not aware of this when I was looking for a
> solution badly back then. Must be a rather recent development. Looks easy
> and straightforward - even deleting is possible. Good job, guys :)
> 
> Still the developer needs to know about it and think of it in the moment of
> adding mirror restrictions to an ebuild with mirror://gentoo sources. A
> risk that is not going to be taken in any of my ebuilds.
> 

I agree it could use a bit more...Preaching.  I think the innards of 
really how our mirroring system works is lost on some people.  They know 
the abstraction; "it's automated and does X,Y,Z" but they don't really 
know how it all works.

-Alec Warner
antarus@gentoo.org
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: mirroring distfiles/patches on patches.gentoo.org

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 684 bytes --]

On Monday 23 October 2006 18:51, Stefan Schweizer wrote:
> Thanks for the heads up. I was not aware of this when I was looking for a
> solution badly back then. Must be a rather recent development.

i guess that depends on your definition of "recent" ... it's been in place 
since June/July

> Still the developer needs to know about it and think of it in the moment of
> adding mirror restrictions to an ebuild with mirror://gentoo sources.

*shrug* the document itself could be linked from the gentoo handbook, but it 
isnt like it's a secret ... it was announced/discussed on -core or -dev

> A risk that is not going to be taken in any of my ebuilds.

lame
-mike

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: [gentoo-dev] Re: mirroring distfiles/patches on patches.gentoo.org

[Brian Harring]

[-- Attachment #1: Type: text/plain, Size: 2328 bytes --]

On Tue, Oct 24, 2006 at 12:51:59AM +0200, Stefan Schweizer wrote:
> Alec Warner wrote:
> >> Those get uploaded and look fine but after 6 months they are suddenly
> >> being removed by the script.
> >> 
> > 
> > There is a distfiles whitelist[1] for this exact purpose...isn't there?
> > 
> > [1]
> > http://www.gentoo.org/proj/en/infrastructure/mirrors/overview-distfile.xml
> 
> Thanks for the heads up. I was not aware of this when I was looking for a
> solution badly back then. Must be a rather recent development. Looks easy
> and straightforward - even deleting is possible. Good job, guys :)

I added whitelist over 18 months ago... not a new feature, although 
infra finally updated the documentation to finally be accurate :)

And correcting vapier in this email, mirror-dist has been running the 
show since around 04/05 - 05/05; not june/july of this year...


> Still the developer needs to know about it and think of it in the moment of
> adding mirror restrictions to an ebuild with mirror://gentoo sources. A
> risk that is not going to be taken in any of my ebuilds.

Whitelist protects files that have a long term (> 2 weeks) refcount of 
0; freebsd stages being the usual example [1].  mirror restricted 
ebuilds that have mirror://gentoo/ (and just mirror://gentoo/) have a 
refcount of 1, thus they do *not* need to be whitelisted.

In general, ebuild maintainers have no valid reason to be screwing 
with whitelists unless they're doing stages, or (odd case) need to 
ensure a file is on mirrors for test purposes, mirrorselect in this 
case.

Thats a polite way of saying the impolite "stick random crap in the 
whitelist, and I'll make a dev cut off your balls" :)

Whitelist totally bypasses all measures to keep the mirror image at a 
sane size; it's intended strictly for material that is *not* ref'd by 
an ebuild, and has a *valid* reason to be in the mirror tier.

Short version: if you're not a stage monkey, you shouldn't be touching 
whitelists.


> I hope the people that prefer mirror://gentoo know about the issue and the
> solution :)

Think you need to lay out other reasons... thus far you seem to be 
forgetting about a change *you* forced through.

~harring

[1] http://dev.gentoo.org/~zmedico/infra/distfiles/failure.xml#doc_chap6

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-dev] mirroring distfiles/patches on patches.gentoo.org

[Marius Mauch]

On Mon, 23 Oct 2006 23:35:50 +0200
Stefan Schweizer <genstef@gentoo.org> wrote:

[snip]

I don't quite get the point of your mail ...
Don't even know if it's a proposal, a complaint, a rant, a RFC, ...

Marius
-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] mirroring distfiles/patches on patches.gentoo.org

[Francesco Riosa]

Stefan Schweizer ha scritto:
> To my fellow Gentoo Devlopers and Users,
> 
> The Gentoo Infrastructure team proposed this idea in june 2006
>  * Reduce mirror time for Gentoo specific patches/tarballs
>  * Offer an official location for Gentoo specific patches/tarballs
>    (Instead of using dev.g.o, this would be the official place)
>  * Offer a distributed (3+ servers) mirror rotation for this
> 
> This would be the "official" location for those distfiles when infra makes
> it available. I am asking them for status reports on this you will maybe
> see some here :)
> 
> But for the meantime I am using http://gentooexperimental.org/~genstef/dist
> for this purpose. The reason is that I can remove and add stuff there
> myself and can keep control of what gets deleted. I can use my favourite
> scripts to create distfiles and keep them ordered. For example I am using
> this for firefox-2.0. This workaround I initially had to use when I
> discovered a bug in the mirroring script that was annoying me regularly.
> The mirroring script does ignore all RESTRICT=mirror SRC_URIs even those at
> mirror://gentoo/.
> "Bug 121332 ppp patch missing from distfiles"
> "Bug 100260 The File foo2zjs-20050319.tar.gz missed on Gentoo-Mirrors"
> Those get uploaded and look fine but after 6 months they are suddenly being
> removed by the script.
> 
> Initially of course I needed a solution of that bug badly and because the
> mirrors kept dropping it I have just uploaded it to dev.gentoo.org in lack
> of any other proper place to put it. But using dev.gentoo.org is deprecated
> by our infra because the server may not be able to cope with the traffic.
> Fortunately later I was able to get an account on Patrick Lauers
> development server http://gentooexperimental.org/~genstef/dist to upload my
> distfiles there.
> 
> So this is my current practice and I am eagerly waiting for infra to allow
> me to use an "official" service instead of my bandaid. Unfortunately some
> people are really afraid of what I am doing and want to harass me into
> using the mirror://gentoo that I do not want to use. This mail is dedicated
> to explain the issue to those people. 
> 
> Please, dont argue such discussions that only give you personal satisfaction
> of being correct or an excuse to annoy other people. Useless yelling at
> each other because of a minority where opinions differ is misplaced in a
> project that is driven by volunteers. That is also why I have stopped and
> written this mail.
> 
> But I can see that you are a bit frustrated because you did not get your way
> through. So here is a guide of what you can do. I will gladly stop using
> workarounds when I am allowed to have an equally bugfree and fast workflow
> as currently. What you can do:
> 
> 1) write a patch for portage to allow granular mirror restrictions for the
> SRC_URI and work with infra on improving the mirroring script to not ignore
> mirror://gentoo in mirror-restricted ebuilds. Also for this solution the
> time for a distfile in /space/distfiles-local to hit the first mirror
> should be equal to the time for ebuilds to get into the rsync rotation so
> that problems for users who are too early.
> 
> 2) get infra to provide patches.gentoo.org as a permanent solution that I
> have asked for since march 2005 and it sometimes even looked close to
> getting it on bug 85098.
> 
2) ++ , has I told you before genstef  I like very much this second one.

Take it also on a mirror point of view, the number of files in distfiles
is already huge, this has been discussed in the past.
To put all our patches.tbz on mirror would increment the number of files
considerably and there will be people whining or stopping mirror for this.
With reiserfs 3 becoming obsolete and reiser 4 with an uncertain future
this may become un-acceptable.

Also I'm offering again, some space and bandwidth in germany
, the box is already controlled by two devs (one being me)

regards,
Francesco Riosa
-- 
gentoo-dev@gentoo.org mailing list

[gentoo-dev] Re: mirroring distfiles/patches on patches.gentoo.org

[Markus Ullmann]

Francesco Riosa schrieb:
> Also I'm offering again, some space and bandwidth in germany

Maybe we could also set up the box from bug #108379 (as mirror?) for
patches. It has plenty of bandwith sitting around and not doing that
much currently ;)

Greetz
Jokey

-- 
gentoo-dev@gentoo.org mailing list

Re: [gentoo-dev] Re: mirroring distfiles/patches on patches.gentoo.org

[Diego 'Flameeyes' Pettenò]

[-- Attachment #1: Type: text/plain, Size: 457 bytes --]

On Tuesday 24 October 2006 02:08, Brian Harring wrote:
> freebsd stages being the usual example [1].
s/stages/source tarballs/

The stages are in another section of the mirror, the source tarballs were 
there when we were not in the main tree. Now the same tecnique is used by 
openbsd/netbsd.

-- 
Diego "Flameeyes" Pettenò - http://farragut.flameeyes.is-a-geek.org/
Gentoo/Alt lead, Gentoo/FreeBSD, Video, Sound, ALSA, PAM, KDE, CJK, Ruby ...

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]