[gentoo-dev] On hosting self-produced distfiles

[gentoo-dev] On hosting self-produced distfiles

[Diego Elio Pettenò]

[-- Attachment #1: Type: text/plain, Size: 1525 bytes --]

Hi all,

I just wanted to write here a clarification regarding self-produced
distfiles, such as patchset tarballs, SCM snapshots and the like. Some
people seem under the impression that the correct way to host these is
to use mirror://gentoo/ and copy them on /space/distfiles-local on
dev.g.o. Please don't do this.

If you produced the file yourself, and it doesn't matter if the file is
reproducible (unless it is reproducible to sha512 identity), please use
the public_html directory in your dev.gentoo.org home to host these.
This makes sure that the file won't be deleted from all its sources if
the ebuild is removed (or more likely replaced) from tree. Ask the Emacs
team how "easy" has been to recover gentoo-syntax files before.

Yes, we all know that the history with the Infra team has been against
this idea, but until there is a proper replacement for this handling,
the Gentoo sources archive, we really shouldn't be putting the data in
non-permanent locations, the team should, nowadays, be on the same page
as me on this.

Pushing files that are still available somewhere else, but cannot be
directly fetched for whatever reason is still to be bone
through /space/distfiles-local.

*PLEASE NOTE:* This is to be considered QA policy, so we're going to ask
soon to enforce this. This requirement, though, _will_ be superseded as
soon as Infra provides us with a proper archive for this kind of files.

Thank you,
-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] On hosting self-produced distfiles

[Chí-Thanh Christopher Nguyễn]

Diego Elio Pettenò schrieb:
> I just wanted to write here a clarification regarding self-produced
> distfiles, such as patchset tarballs, SCM snapshots and the like. Some
> people seem under the impression that the correct way to host these is
> to use mirror://gentoo/ and copy them on /space/distfiles-local on
> dev.g.o. Please don't do this.
>
> If you produced the file yourself, and it doesn't matter if the file is
> reproducible (unless it is reproducible to sha512 identity), please use
> the public_html directory in your dev.gentoo.org home to host these.
>    

http://devmanual.gentoo.org/general-concepts/mirrors/index.html claims 
that dev.gentoo.org is not acceptable for hosting main-tree items, and 
they must be moved to mirror://gentoo before release.

Maybe putting a clarification there too would help in avoiding confusion 
regarding this issue.


Best regards,
Chí-Thanh Christopher Nguyễn

Re: [gentoo-dev] On hosting self-produced distfiles

[Rich Freeman]

On Wed, Jan 19, 2011 at 7:50 PM, Diego Elio Pettenò <flameeyes@gmail.com> wrote:
> *PLEASE NOTE:* This is to be considered QA policy, so we're going to ask
> soon to enforce this.

Forward going?  Or, should we go ahead and start retroactively
updating ebuilds that have mirror:// in them right now?  Presumably
without a revbump over something like this...

Rich

Re: [gentoo-dev] On hosting self-produced distfiles

[Theo Chatzimichos]

[-- Attachment #1: Type: Text/Plain, Size: 689 bytes --]

On Thursday 20 January 2011 02:50:35 Diego Elio Pettenò wrote:
> *PLEASE NOTE:* This is to be considered QA policy, so we're going to ask
> soon to enforce this. This requirement, though, _will_ be superseded as
> soon as Infra provides us with a proper archive for this kind of files.

I usually create snapshots for kde apps (or kde app deps) till the actual 
release arises. Why should I keep them in my homedir, since I do want those 
tarballs to be deleted after the ebuild's removal? I'm sorry, but I object to 
that enforcement, me (the maintainer) knows better on what to do with his 
custom tarballs.
-- 
Theo Chatzimichos (tampakrap)
Gentoo KDE/Qt, Planet, Overlays

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] On hosting self-produced distfiles

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 542 bytes --]

On Wednesday, January 19, 2011 19:50:35 Diego Elio Pettenò wrote:
> *PLEASE NOTE:* This is to be considered QA policy, so we're going to ask
> soon to enforce this. This requirement, though, _will_ be superseded as
> soon as Infra provides us with a proper archive for this kind of files.

you really need to start off discussions as "here is the problem i perceive 
and here is a solution i think will address it".  shooting off e-mails from on 
high as an official edict without room for discussion is no way to run a team.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] On hosting self-produced distfiles

[Diego Elio Pettenò]

[-- Attachment #1: Type: text/plain, Size: 1091 bytes --]

Il giorno mer, 19/01/2011 alle 21.44 -0500, Mike Frysinger ha scritto:
> 
> you really need to start off discussions as "here is the problem i
> perceive 
> and here is a solution i think will address it".  shooting off e-mails
> from on 
> high as an official edict without room for discussion is no way to run
> a team. 

There is nothing new or revolutionary in what I said — are you trying to
challenge the need for traceability of distfiles?

Sure we should find a better solution than dev.gentoo.org, but Robin's
already working on that. This is a transitional solution that allows us
to have stable source URIs, and encodes what most of us (including me
and *you* by the way) have been doing already.

Do you really think I should have "discussed" with "a team" about this?
More than asking Robin as part of infra if it's okay (with his answer
being "yeah, that's fine | i do it too", literally)?

I'm not going to force my hand here with stuff that is up to debate, but
seriously, *this*?

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[gentoo-dev] Re: On hosting self-produced distfiles

[Diego Elio Pettenò]

[-- Attachment #1: Type: text/plain, Size: 410 bytes --]

Il giorno gio, 20/01/2011 alle 02.05 +0100, Chí-Thanh Christopher Nguyễn
ha scritto:
> 
> http://devmanual.gentoo.org/general-concepts/mirrors/index.html claims 
> that dev.gentoo.org is not acceptable for hosting main-tree items,
> and 
> they must be moved to mirror://gentoo before release. 

Yes I know and I'll update ASAP.

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[gentoo-dev] Re: On hosting self-produced distfiles

[Diego Elio Pettenò]

[-- Attachment #1: Type: text/plain, Size: 804 bytes --]

Il giorno gio, 20/01/2011 alle 03.17 +0200, Theo Chatzimichos ha
scritto:
> 
> 
> I usually create snapshots for kde apps (or kde app deps) till the
> actual 
> release arises. Why should I keep them in my homedir, since I do want
> those 
> tarballs to be deleted after the ebuild's removal? I'm sorry, but I
> object to 
> that enforcement, me (the maintainer) knows better on what to do with
> his 
> custom tarballs. 

We need a way to track the files for things that are provided in Gentoo,
by license if they are used for binary packages as well.

If you keep the prerelease in overlay, do as you wish, but for main tree
please use stable URLs please. Especially if the ebuild ever gets to
stable keywording.

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[gentoo-dev] Re: On hosting self-produced distfiles

[Diego Elio Pettenò]

[-- Attachment #1: Type: text/plain, Size: 603 bytes --]

Il giorno mer, 19/01/2011 alle 20.07 -0500, Rich Freeman ha scritto:
> Forward going?  Or, should we go ahead and start retroactively
> updating ebuilds that have mirror:// in them right now?  Presumably
> without a revbump over something like this...
> 

I wouldn't mind if it was done retroactively, but I'm not going to ask
right now for all the ebuilds in tree right now to be converted. If you
do happen to pass through a bunch of old ebuilds and edit them anyway
please do update them to use long-term-reachable URLs.

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] On hosting self-produced distfiles

[Matt Turner]

On Thu, Jan 20, 2011 at 2:50 AM, Diego Elio Pettenò <flameeyes@gmail.com> wrote:
> Il giorno mer, 19/01/2011 alle 21.44 -0500, Mike Frysinger ha scritto:
>>
>> you really need to start off discussions as "here is the problem i
>> perceive
>> and here is a solution i think will address it".  shooting off e-mails
>> from on
>> high as an official edict without room for discussion is no way to run
>> a team.
>
> There is nothing new or revolutionary in what I said — are you trying to
> challenge the need for traceability of distfiles?

Good grief. No.

He seems to be simply stating that you'd do better to say "we have a
problem with $this and I think it'd be fixed with $that" instead of
"This is to be considered QA policy, so we're going to ask soon to
enforce this."

Not hosting distfiles from dev.g.o is a reasonable thing that I think
everyone understands and agrees with, but "this will be official
policy" jargon is annoying and doesn't really serve any purpose.

Matt

Re: [gentoo-dev] On hosting self-produced distfiles

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1265 bytes --]

On Wednesday, January 19, 2011 21:56:52 Matt Turner wrote:
> On Thu, Jan 20, 2011 at 2:50 AM, Diego Elio Pettenò wrote:
> > Il giorno mer, 19/01/2011 alle 21.44 -0500, Mike Frysinger ha scritto:
> >> you really need to start off discussions as "here is the problem i
> >> perceive
> >> and here is a solution i think will address it".  shooting off e-mails
> >> from on
> >> high as an official edict without room for discussion is no way to run
> >> a team.
> > 
> > There is nothing new or revolutionary in what I said — are you trying to
> > challenge the need for traceability of distfiles?
> 
> Good grief. No.
> 
> He seems to be simply stating that you'd do better to say "we have a
> problem with $this and I think it'd be fixed with $that" instead of
> "This is to be considered QA policy, so we're going to ask soon to
> enforce this."

pretty much.  no point in getting your panties in a bunch.

here's a better idea: figure out something with the infra team.  we havent 
hosted files on dev.g.o because we've felt the distfiles tree to be 
sufficient.  since there seems to be more need now, let's find out what infra 
can do to help out.

again, declaring policy ahead of talking to anyone else is not the way to go.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[gentoo-dev] Re: On hosting self-produced distfiles

[Diego Elio Pettenò]

Il giorno mer, 19/01/2011 alle 22.31 -0500, Mike Frysinger ha scritto:
>  we havent 
> hosted files on dev.g.o because we've felt the distfiles tree to be 
> sufficient.  since there seems to be more need now, let's find out
> what infra 
> can do to help out.

I'm pretty sure you have, for pax-utils and portage-utils, didn't you?


> here's a better idea: figure out something with the infra team.
> [snip]
> again, declaring policy ahead of talking to anyone else is not the way
> to go.

Actually, we meant to move to a stable archive of distfiles for years,
and Robin has been working on it for months already.

The policy, that as chithanh pointed out needs to be updated, is just
going to cause more grief by the time you want to pick up the old files
for whatever reason. Ulrich (ulm) and Christian (fauli) knows how much a
pain it becomes.
 
-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

Re: [gentoo-dev] Re: On hosting self-produced distfiles

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 665 bytes --]

On Thu, Jan 20, 2011 at 04:57:59AM +0100, Diego Elio Pettenò wrote:
> > here's a better idea: figure out something with the infra team.
> > [snip]
> > again, declaring policy ahead of talking to anyone else is not the way
> > to go.
> Actually, we meant to move to a stable archive of distfiles for years,
> and Robin has been working on it for months already.
Here's the previous proposal, with all comments merged into it 
http://bugs.gentoo.org/show_bug.cgi?id=176186#c15

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 330 bytes --]

Re: [gentoo-dev] On hosting self-produced distfiles

["Paweł Hajdan, Jr."]

[-- Attachment #1: Type: text/plain, Size: 616 bytes --]

On 1/20/11 1:50 AM, Diego Elio Pettenò wrote:
> If you produced the file yourself, and it doesn't matter if the file is
> reproducible (unless it is reproducible to sha512 identity), please use
> the public_html directory in your dev.gentoo.org home to host these.
> This makes sure that the file won't be deleted from all its sources if
> the ebuild is removed (or more likely replaced) from tree. Ask the Emacs
> team how "easy" has been to recover gentoo-syntax files before.

Storing distfiles in public_html is not a perfect solution either. If
the developer retires, what do we do with the files?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

[gentoo-dev] Re: On hosting self-produced distfiles

[Ulrich Mueller]

>>>>> On Thu, 20 Jan 2011, Diego Elio Pettenò wrote:

> The policy, that as chithanh pointed out needs to be updated, is
> just going to cause more grief by the time you want to pick up the
> old files for whatever reason. Ulrich (ulm) and Christian (fauli)
> knows how much a pain it becomes.

<irony>
Oh, that was easy: We found some forgotten distfile mirrors that
hadn't been updated for several years. And for the files then still
missing, we sent a plea for help to -dev (or was it planet?) and had
them mailed to us by users.
</irony>

Thanks again to our users who had helped us at the time.

Ulrich

Re: [gentoo-dev] On hosting self-produced distfiles

[Peter Volkov]

В Чтв, 20/01/2011 в 03:50 +0100, Diego Elio Pettenò пишет:
> Do you really think I should have "discussed" with "a team" about this?
> More than asking Robin as part of infra if it's okay (with his answer
> being "yeah, that's fine | i do it too", literally)?

This was already mentioned in this thread, but still. AFAIR (betelgeuse
knows better) average developer's age is less then three years and thus,
many files hosted in home directory became unavailable after developer
retires. Thus the solution you propose does not address the problem we
have and it's not a nice to enforce it in the policy (without some
additional policy to keep developer's home directory on server for three
years and anyway this needs discussion with infra first).

-- 
Peter.

[gentoo-dev] Re: On hosting self-produced distfiles

[Christian Faulhammer]

[-- Attachment #1: Type: text/plain, Size: 371 bytes --]

Hi,

Ulrich Mueller <ulm@gentoo.org>:
> missing, we sent a plea for help to -dev (or was it planet?) and had

 http://www.faulhammer.org/archiv-mainmenu-31/35-gentoo/293-we-are-looking-for-distfiles

V-Li

-- 
Christian Faulhammer, Gentoo Lisp project
<URL:http://www.gentoo.org/proj/en/lisp/>, #gentoo-lisp on FreeNode

<URL:http://gentoo.faulhammer.org/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: On hosting self-produced distfiles

[Diego Elio Pettenò]

[-- Attachment #1: Type: text/plain, Size: 648 bytes --]

Il giorno gio, 20/01/2011 alle 07.23 +0100, "Paweł Hajdan, Jr." ha
scritto:
> 
> Storing distfiles in public_html is not a perfect solution either. If
> the developer retires, what do we do with the files? 

That's why (and I answer to Peter here as well), it is an ad interim
solution. As I said and repeated Robin is working on the final one but
until then we still prefer this method.

Besides, developers' home is usually archived already on retirement so
we can recover the files from there without having to beg users to send
them to us, as Ulrich testified.

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] On hosting self-produced distfiles

[Anthony G. Basile]

On 01/20/2011 01:23 AM, "Paweł Hajdan, Jr." wrote:
> On 1/20/11 1:50 AM, Diego Elio Pettenò wrote:
>> If you produced the file yourself, and it doesn't matter if the file is
>> reproducible (unless it is reproducible to sha512 identity), please use
>> the public_html directory in your dev.gentoo.org home to host these.
>> This makes sure that the file won't be deleted from all its sources if
>> the ebuild is removed (or more likely replaced) from tree. Ask the Emacs
>> team how "easy" has been to recover gentoo-syntax files before.
> Storing distfiles in public_html is not a perfect solution either. If
> the developer retires, what do we do with the files?
>

There is another problem:

   grep mirror /usr/portage/eclass/* | sed -e 's/:.*$//' | sort | uniq

shows 39 eclasses which refer to mirror://

-- 
Anthony G. Basile, Ph.D.
Gentoo Developer

Re: [gentoo-dev] On hosting self-produced distfiles

[Anthony G. Basile]

On 01/20/2011 01:34 PM, Anthony G. Basile wrote:
> On 01/20/2011 01:23 AM, "Paweł Hajdan, Jr." wrote:
>> On 1/20/11 1:50 AM, Diego Elio Pettenò wrote:
>>> If you produced the file yourself, and it doesn't matter if the file is
>>> reproducible (unless it is reproducible to sha512 identity), please use
>>> the public_html directory in your dev.gentoo.org home to host these.
>>> This makes sure that the file won't be deleted from all its sources if
>>> the ebuild is removed (or more likely replaced) from tree. Ask the Emacs
>>> team how "easy" has been to recover gentoo-syntax files before.
>> Storing distfiles in public_html is not a perfect solution either. If
>> the developer retires, what do we do with the files?
>>
> There is another problem:
>
>    grep mirror /usr/portage/eclass/* | sed -e 's/:.*$//' | sort | uniq
>
> shows 39 eclasses which refer to mirror://
>
Sorry darkside pointed out that there are other mirrors,

  grep "mirror://gentoo" /usr/portage/eclass/* | sed -e 's/:.*$//' |
sort | uniq

shows 16 eclasses which would have to be changed.

-- 
Anthony G. Basile, Ph.D.
Gentoo Developer

Re: [gentoo-dev] On hosting self-produced distfiles

[Matti Bickel]

[-- Attachment #1: Type: text/plain, Size: 1133 bytes --]

On 01/20/2011 01:50 AM, Diego Elio Pettenò wrote:
> I just wanted to write here a clarification regarding self-produced
> distfiles, such as patchset tarballs, SCM snapshots and the like. Some
> people seem under the impression that the correct way to host these is
> to use mirror://gentoo/ and copy them on /space/distfiles-local on
> dev.g.o. Please don't do this.

As one of those under the impression that mirror://gentoo was the right
choice: why is it bad?

From Fauli's post I gather the emacs team wanted to support ancient
versions and have all files necessary to install an ebuild whatsoever
it's age.
In my case, that would mean installing php-4.0, for example. Why on
earth should I support something like that? If I'm PHP upstream, okay,
maybe allow others to see the evolution of the language. But the
evolution of the php patchset? Sounds not that necessary to me.

So, I'm not opposed to your idea. If ya want to archive your stuff
forever, by all means do it. I just see no point in forcing this on all
devs. So, care to explain or give me pointer on why this is necessary?

Thanks, Matti


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

[gentoo-dev] Re: On hosting self-produced distfiles

[Diego Elio Pettenò]

Il giorno gio, 20/01/2011 alle 13.34 -0500, Anthony G. Basile ha
scritto:
> 
> shows 39 eclasses which refer to mirror:// 

That's not much of a problem, mirror://kde/ mirror://debian/ and the
like are fine, most of the time.

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

[gentoo-dev] Re: On hosting self-produced distfiles

[Diego Elio Pettenò]

Il giorno gio, 20/01/2011 alle 19.41 +0100, Matti Bickel ha scritto:
> 
> So, I'm not opposed to your idea. If ya want to archive your stuff
> forever, by all means do it. I just see no point in forcing this on
> all
> devs. So, care to explain or give me pointer on why this is necessary?
> 

License compliance when distributing binaries; distributions built upon
Gentoo that might use old and set-in-stone Portage trees; security
issues that might be reported and needs to be investigated, ...

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

Re: [gentoo-dev] Re: On hosting self-produced distfiles

[Matti Bickel]

[-- Attachment #1: Type: text/plain, Size: 895 bytes --]

First, thanks for pointing this out.

On 01/20/2011 07:51 PM, Diego Elio Pettenò wrote:
> License compliance when distributing binaries;

Not sure what you mean: if someone quickpkg's php and needs all the
source? Well, they already downloaded them. Better keep them around,
since it's *your* binary, not mine.

> distributions built upon Gentoo that might use old and set-in-stone
> Portage trees;

Same thing, as already pointed out in another message. I see the point
in making it easier for them. That's okay. So what you're saying is
we're upstream too and upstream's should provide their historical stuff.

> security issues that might be reported and needs to be
> investigated, ...

If you're reporting a security issue in a ebuild that's no longer in
tree (in php's case, chances are it got removed b/c of security :p), the
bug wouldn't be investigated, right?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-dev] Re: On hosting self-produced distfiles

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1069 bytes --]

On Wednesday, January 19, 2011 22:57:59 Diego Elio Pettenò wrote:
> Il giorno mer, 19/01/2011 alle 22.31 -0500, Mike Frysinger ha scritto:
> >  we havent
> > 
> > hosted files on dev.g.o because we've felt the distfiles tree to be
> > sufficient.  since there seems to be more need now, let's find out
> > what infra
> > can do to help out.
> 
> I'm pretty sure you have, for pax-utils and portage-utils, didn't you?

my personal choice to keep an archive of packages i build does not mean a hard 
policy must be forced upon everyone.

> > here's a better idea: figure out something with the infra team.
> > [snip]
> > again, declaring policy ahead of talking to anyone else is not the way
> > to go.
> 
> Actually, we meant to move to a stable archive of distfiles for years,
> and Robin has been working on it for months already.

then you should have mentioned this in your original e-mail rather than 
painting infra as angry unhelpful tools.  the changes you propose are merely a 
stop gap measure until infra finishes their work.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Re: On hosting self-produced distfiles

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 918 bytes --]

On Thursday, January 20, 2011 13:51:23 Diego Elio Pettenò wrote:
> Il giorno gio, 20/01/2011 alle 19.41 +0100, Matti Bickel ha scritto:
> > So, I'm not opposed to your idea. If ya want to archive your stuff
> > forever, by all means do it. I just see no point in forcing this on
> > all
> > devs. So, care to explain or give me pointer on why this is necessary?
> 
> License compliance when distributing binaries

how is this relevant ?  if there are issues with distributing binaries, then 
were they get distributed isnt important.

> distributions built upon Gentoo that might use old and set-in-stone Portage
> trees

last i heard, the FSF's interpretation of the GPL was that this is the problem 
of said derivatives.  i'm not saying we shouldnt help out where possible, just 
that we shouldnt be bending over backwards for them.  they want to stay 
behind, then that is their problem.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[gentoo-dev] Re: Re: On hosting self-produced distfiles

[Diego Elio Pettenò]

Il giorno gio, 20/01/2011 alle 20.27 +0100, Matti Bickel ha scritto:
> Not sure what you mean: if someone quickpkg's php and needs all the
> source? Well, they already downloaded them. Better keep them around,
> since it's *your* binary, not mine.

We do distribute part of our packages as binaries already so we have to
be compliant with their licenses to begin with. Better doing it with a
single sweep than trying to come up with abstruse case-by-case points,
no?

> Same thing, as already pointed out in another message. I see the point
> in making it easier for them. That's okay. So what you're saying is
> we're upstream too and upstream's should provide their historical stuff.

This is but _one_ reason, and just another thing to trickle down. I
don't care if "FSF says it's their problem"; what is it costing us,
really? The cost is minimal (as we need the archive anyway), and the
gain is there for many people.

Arguing against this is just getting to the point of arguing because
somebody is doing what nobody did for a long time: taking decisions.

> If you're reporting a security issue in a ebuild that's no longer in
> tree (in php's case, chances are it got removed b/c of security :p), the
> bug wouldn't be investigated, right?

There are cases and cases there; in the case of _custom_ tarballs, I'd
expect us to investigate if the security issues was found on our version
and not in the upstream-provided one for instance.

Once again, please tell me: what does it change to you? If anybody
should complain about this request is Infra. And Infra in the person of
Robin is okay with this policy as it was planned anyway.

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

[gentoo-dev] Re: Re: On hosting self-produced distfiles

[Diego Elio Pettenò]

Il giorno gio, 20/01/2011 alle 14.28 -0500, Mike Frysinger ha scritto:
> 
> then you should have mentioned this in your original e-mail rather
> than 
> painting infra as angry unhelpful tools.  the changes you propose are
> merely a 
> stop gap measure until infra finishes their work. 

You of all people to sound surprised about this really doesn't look
right. The fact that such an archive has been in the work is something
know since I became a dev, six years ago; the bug was open in 2007.

Also "angry unhelpful tools"? You're putting words in my mail that
aren't there. I said infra _used_ to be against that; it was obvious
that knowing that I wouldn't just push the issue against their will.

If you were trying to pick a fight for the sake of it, I'd suggest you
find something else to do.

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

Re: [gentoo-dev] Re: On hosting self-produced distfiles

[Jeroen Roovers]

[-- Attachment #1: Type: text/plain, Size: 1352 bytes --]

On Thu, 20 Jan 2011 14:25:23 +0100
Diego Elio Pettenò <flameeyes@gmail.com> wrote:

> Il giorno gio, 20/01/2011 alle 07.23 +0100, "Paweł Hajdan, Jr." ha
> scritto:
> > 
> > Storing distfiles in public_html is not a perfect solution either.
> > If the developer retires, what do we do with the files? 
> 
> That's why (and I answer to Peter here as well), it is an ad interim
> solution. As I said and repeated Robin is working on the final one but
> until then we still prefer this method.

It isn't exactly a solution and the interim has lasted for years now.

What legitimate use does mirror://gentoo retain when we do have a
solution? Ultimate patch attached.

The way I see it, losing important files because you didn't
store copies privately or publicly is not a problem our distfiles
mirrors should solve.

Having SRC_URI files taking up space indefinitely because of a special
SRC_URI value is not the solution either, say when we would create a
static-distfiles.gentoo.org for that purpose.

What if an upstream dies? Lots of packages have broken SRC_URIs because
of that, and yet as long as the ebuilds are in the tree, the files are
nicely mirrored for us.

So, as asked here and there in this thread, what exact problem are we
solving? Hopefully not merely the loss of some files in the past?


     jer

[-- Attachment #2: mirror-gentoo-deprecation.patch --]
[-- Type: text/x-patch, Size: 2806 bytes --]

Index: thirdpartymirrors
===================================================================
RCS file: /var/cvsroot/gentoo-x86/profiles/thirdpartymirrors,v
retrieving revision 1.269
diff -u -B -r1.269 thirdpartymirrors
--- thirdpartymirrors	19 Jan 2011 06:03:50 -0000	1.269
+++ thirdpartymirrors	20 Jan 2011 19:48:14 -0000
@@ -13,7 +13,6 @@
 filefront	http://ftp.games.skynet.be/pub/www.filesnetwork.com ftp://ftp.games.skynet.be/pub/www.filesnetwork.com
 flightgear	ftp://ftp.de.flightgear.org/pub/fgfs http://mirrors.ibiblio.org/pub/mirrors/flightgear/ftp ftp://mirrors.ibiblio.org/pub/mirrors/flightgear/ftp ftp://ftp.kingmont.com/flightsims/flightgear ftp://ftp.ihg.uni-duisburg.de/Mirrors/ftp.flightgear.org
 freebsd		ftp://ftp.FreeBSD.org/pub/FreeBSD/ ftp://ftp2.FreeBSD.org/pub/FreeBSD/ ftp://ftp.ar.FreeBSD.org/pub/FreeBSD/ ftp://ftp.au.FreeBSD.org/pub/FreeBSD/ ftp://ftp2.au.FreeBSD.org/pub/FreeBSD/ ftp://ftp.at.FreeBSD.org/pub/FreeBSD/ ftp://ftp2.at.FreeBSD.org/pub/FreeBSD/ ftp://ftp.br.FreeBSD.org/pub/FreeBSD/ ftp://ftp2.br.FreeBSD.org/pub/FreeBSD/ ftp://ftp.ca.FreeBSD.org/pub/FreeBSD/ ftp://ftp2.ca.FreeBSD.org/ ftp://ftp.cn.FreeBSD.org/pub/FreeBSD/ ftp://ftp.cz.FreeBSD.org/pub/FreeBSD/ ftp://ftp.dk.FreeBSD.org/pub/FreeBSD/ ftp://ftp2.dk.FreeBSD.org/pub/FreeBSD/ ftp://ftp.ee.FreeBSD.org/pub/FreeBSD/ ftp://ftp.fr.FreeBSD.org/pub/FreeBSD/ ftp://ftp2.fr.FreeBSD.org/pub/FreeBSD/ ftp://ftp.de.FreeBSD.org/pub/FreeBSD/ ftp://ftp2.de.FreeBSD.org/pub/FreeBSD/ ftp://ftp.gr.FreeBSD.org/pub/FreeBSD/ ftp://ftp2.gr.FreeBSD.org/pub/FreeBSD/ ftp://ftp.hk.FreeBSD.org/pub/FreeBSD/ ftp://ftp.is.FreeBSD.org/pub/FreeBSD/ ftp://ftp.id.FreeBSD.org/pub/FreeBSD/ ftp://ftp.ie.FreeBSD.org/pub/FreeBSD/ !
 ftp://ftp2.ie.FreeBSD.org/pub/FreeBSD/ ftp://ftp.it.FreeBSD.org/pub/FreeBSD/ ftp://ftp.jp.FreeBSD.org/pub/FreeBSD/ ftp://ftp2.jp.FreeBSD.org/pub/FreeBSD/ ftp://ftp2.kr.FreeBSD.org/pub/FreeBSD/ ftp://ftp1.us.FreeBSD.org/pub/FreeBSD/ ftp://ftp2.us.FreeBSD.org/pub/FreeBSD/
-gentoo		http://gentoo.osuosl.org/distfiles http://mirrors.kernel.org/gentoo/distfiles http://ftp.halifax.rwth-aachen.de/gentoo http://gentoo-distfiles.mirrors.tds.net/distfiles http://gentoo.ussg.indiana.edu/distfiles
 ggz		http://ftp.belnet.be/packages/ggzgamingzone/ggz http://mirrors.dotsrc.org/ggzgamingzone/ggz http://mirrors.ibiblio.org/pub/mirrors/ggzgamingzone/ggz ftp://ftp.belnet.be/packages/ggzgamingzone/ggz ftp://mirrors.dotsrc.org/mirrors/ggzgamingzone/ggz
 gimp		ftp://ftp.gimp.org/pub/gimp http://ftp.gwdg.de/pub/misc/grafik/gimp/gimp ftp://ftp.cs.umn.edu/pub/gimp
 gmt		ftp://mirror.geosci.usyd.edu.au/pub/gmt/ ftp://ftp.soest.hawaii.edu/gmt/ ftp://ftp.soest.hawaii.edu/gmt/ ftp://ibis.grdl.noaa.gov/pub/gmt/ ftp://ftp.iris.washington.edu/pub/gmt/ ftp://ftp.iag.usp.br/pub/gmt/ ftp://ftp.geologi.uio.no/pub/gmt/

[gentoo-dev] Re: Re: On hosting self-produced distfiles

[Diego Elio Pettenò]

Il giorno gio, 20/01/2011 alle 21.02 +0100, Jeroen Roovers ha scritto:
> It isn't exactly a solution and the interim has lasted for years now.

No, for years now we had policies going one way ("don't use dev.g.o")
because they were written at one point by one person, and practices for
most of us going the other (using dev.gentoo.org), as the original
reason not to use it is no longer relevant.

Now we're re-joining policy and practice.

> What legitimate use does mirror://gentoo retain when we do have a
> solution? Ultimate patch attached.

Yes and? We're going to have a distinct mirror://gentoo-projects/ (just
to be on the safe side for overlays mainly) to fetch the distfiles for
the custom packages.

> The way I see it, losing important files because you didn't
> store copies privately or publicly is not a problem our distfiles
> mirrors should solve. 

For Gentoo-produced distfiles, it is nothing new that we have to have
long term access available. We've been meaning to for years as you said.
I'm positive that the issue went to the council once already.

Let's be clear here: Infra is the same page as this; this _is_ going
through. This was being worked on for months and months, and people
start complain now because... they are being asked for all of us to
follow a single policy rather than case-by-case whether to delete
distfiles or not?

There isn't _more_ work to be done with the exception of using a script
that signs the files rather than simply scp'ing them over, so it's not a
matter of "you're asking us to do more work in the future" as much as
"you're asking us to follow a procedure". Well, duh!

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

Re: [gentoo-dev] Re: Re: On hosting self-produced distfiles

[Matti Bickel]

[-- Attachment #1: Type: text/plain, Size: 1699 bytes --]

On 01/20/2011 08:42 PM, Diego Elio Pettenò wrote:
> We do distribute part of our packages as binaries already so we have to
> be compliant with their licenses to begin with. Better doing it with a
> single sweep than trying to come up with abstruse case-by-case points,
> no?

No. Licenses are not a valid argument to me. I'd accept that if we're
Debian and pushing 100% of *our* stuff as binary. What we do 90% of the
time is distributing text - ebuilds.

> Arguing against this is just getting to the point of arguing because
> somebody is doing what nobody did for a long time: taking decisions.

Yes, and I'm not going to stop you. Frankly, I don't care enough where
my tarballs end up. I just was curious about the reasons, as I see no
compelling point in *forcing* this.

>> If you're reporting a security issue in a ebuild that's no longer in
>> tree (in php's case, chances are it got removed b/c of security :p), the
>> bug wouldn't be investigated, right?
> 
> There are cases and cases there; in the case of _custom_ tarballs, I'd
> expect us to investigate if the security issues was found on our version
> and not in the upstream-provided one for instance.

Take php-5.3.2: I don't care if you found a security issue in my tarball
or in php's tarball. I'll have a look to determine if the bug's still in
the newest version. If it is, I'll rename the bug. If it is not, it
doesn't matter to me.

> Once again, please tell me: what does it change to you? If anybody
> should complain about this request is Infra.

What it changes for me? The target argument of my scp command. Which is
so small that I don't care (see above for why I still replied).


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-dev] Re: Re: On hosting self-produced distfiles

[Jeroen Roovers]

On Thu, 20 Jan 2011 21:23:49 +0100
Diego Elio Pettenò <flameeyes@gmail.com> wrote:

> Now we're re-joining policy and practice.

By forcing use of dev.gentoo.org for self-hosted SRC_URI files?

> > What legitimate use does mirror://gentoo retain when we do have a
> > solution? Ultimate patch attached.

> Yes and? We're going to have a distinct mirror://gentoo-projects/
> (just to be on the safe side for overlays mainly) to fetch the
> distfiles for the custom packages.

By forcing use of mirror://gentoo-projects for self-hosted SRC_URI
files?

I'm lost now. Do you/infra/$GENTOO_DEITY plan to first force everyone to
use dev.g.o and then move to mirror://gentoo-projects soon after?

Why don't we just change the meaning of mirror://gentoo and be done
with it, without the interim being extended? We could even have the
mirror sync scripts interpret mirror://gentoo in a new way. All we then
need to do on the QA side is to somehow enforce the way mirror://gentoo
is used. :)


    jer

[gentoo-dev] Re: Re: Re: On hosting self-produced distfiles

[Diego Elio Pettenò]

Il giorno gio, 20/01/2011 alle 21.35 +0100, Matti Bickel ha scritto:
> On 01/20/2011 08:42 PM, Diego Elio Pettenò wrote:
> No. Licenses are not a valid argument to me. I'd accept that if we're
> Debian and pushing 100% of *our* stuff as binary. What we do 90% of the
> time is distributing text - ebuilds.

So I'm not asking _you_ to waste 90% of your time discussing and
auditing licenses. We have a team for that.

At the same time I'm not going to ask the developers to all evaluate
case by case whether they should or shouldn't keep their stuff
available. I'm telling them to put it there rather than in another
place; what that will change shouldn't really be a problem.

> I just was curious about the reasons, as I see no
> compelling point in *forcing* this.

The reason to *force* this is two fold: we need a policy so that we stop
the fact that everybody does as he pleases and this is replacing a
_different_ forcing that we _used_ to have, and which I'm not surprised
you didn't hear about, that told developers to use mirror://gentoo/.

Which is unfortunately troublesome *as Ulrich and Christian already
shown*.

And since in Gentoo we cannot simply scratch rules, as otherwise people
will keep referencing them forever and ever, a new rule replaces the old
rule: you use dev.gentoo.org rather than mirror://gentoo/.

> Take php-5.3.2: I don't care if you found a security issue in my tarball
> or in php's tarball. I'll have a look to determine if the bug's still in
> the newest version. If it is, I'll rename the bug. If it is not, it
> doesn't matter to me.

You might not care. I would, and it's not just a matter of being the
current QA lead in charge, but rather a question of professionalism. If
I would find you or another developer to have introduced a backdoor in a
custom tarball, I'm going to have said developer booted, quickly.

Again, you might not care, we have other teams that do care. Since I'm
not asking you to do a 180° jump changing your habits, can we please
just agree that you don't see the point but you'll follow the request
anyway?

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

[gentoo-dev] Re: Re: Re: On hosting self-produced distfiles

[Diego Elio Pettenò]

Il giorno gio, 20/01/2011 alle 21.38 +0100, Jeroen Roovers ha scritto:
> 
> 
> I'm lost now. Do you/infra/$GENTOO_DEITY plan to first force everyone
> to
> use dev.g.o and then move to mirror://gentoo-projects soon after? 

"soon after"?

Sorry I don't foresee the next move to happen within six months.

And once we know which files need to be migrated, I can do the migration
myself, or infra can do that.

Let's be clear, I'm not asking for anybody to spend the next couple of
days to convert all the packages out there for this very reason.

*But* I'm asking for the _future_ ebuilds to have it moved in a place
where we can pick it up to move it when possible.

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

Re: [gentoo-dev] Re: Re: Re: On hosting self-produced distfiles

[Jeroen Roovers]

On Thu, 20 Jan 2011 21:58:43 +0100
Diego Elio Pettenò <flameeyes@gmail.com> wrote:

> Let's be clear, I'm not asking for anybody to spend the next couple of
> days to convert all the packages out there for this very reason.

That wasn't entirely clear.

> *But* I'm asking for the _future_ ebuilds to have it moved in a place
> where we can pick it up to move it when possible.

That wasn't clear either.

I'd be happier if you responded my proposal of changing the meaning of
mirror://gentoo - the mirror sync scripts could take care of the entire
thing by forever storing files described that way. That way, no
ebuilds/eclasses using mirror://gentoo would need changing, and we
could conveniently move away from using dev.gentoo.org for availability
of old self-hosted SRC_URI files.

(This is probably why several people have requested you propose this
change to the developer community instead of wearing your QA hat and
pretending to lay down the law - I see QA as the executor of common
sense, technically sensible, commonly approved approaches to problems,
i.e. generally approved policy, not in the role of policy maker. If
we've been doing things against policy for years, then maybe we should
change the policy, or at least discuss it without the pretence of a
developer hierarchy.)


     jer

[gentoo-dev] Re: Re: Re: Re: On hosting self-produced distfiles

[Diego Elio Pettenò]

Il giorno gio, 20/01/2011 alle 22.14 +0100, Jeroen Roovers ha scritto:
> 
> I'd be happier if you responded my proposal of changing the meaning of
> mirror://gentoo - the mirror sync scripts could take care of the
> entire
> thing by forever storing files described that way. That way, no
> ebuilds/eclasses using mirror://gentoo would need changing, and we
> could conveniently move away from using dev.gentoo.org for
> availability
> of old self-hosted SRC_URI files. 

http://bugs.gentoo.org/show_bug.cgi?id=176186

I'VE NOT DREAMT ABOUT THIS LAST NIGHT. This is why there was no
"discussion": it was all planned already, from a long time ago.

You want to change the plan? Talk with Robin/infra. In the mean time,
though, this is what future ebuilds/eclasses will have to look right
until further notice.

Notice that I hope will soon come with a final solution without the
current drawbacks (which are still a bit less troublesome than those we
had before).

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

Re: [gentoo-dev] On hosting self-produced distfiles

[Alexis Ballier]

On Wednesday, January 19, 2011 09:50:35 pm Diego Elio Pettenò wrote:
> Hi all,
> 
> I just wanted to write here a clarification regarding self-produced
> distfiles, such as patchset tarballs, SCM snapshots and the like. Some
> people seem under the impression that the correct way to host these is
> to use mirror://gentoo/ and copy them on /space/distfiles-local on
> dev.g.o. Please don't do this.
> 
> If you produced the file yourself, and it doesn't matter if the file is
> reproducible (unless it is reproducible to sha512 identity), please use
> the public_html directory in your dev.gentoo.org home to host these.
> This makes sure that the file won't be deleted from all its sources if
> the ebuild is removed (or more likely replaced) from tree. Ask the Emacs
> team how "easy" has been to recover gentoo-syntax files before.


Solution b): update the distfiles-local mirroring script to store a copy where 
it wont be deleted...
The interim solution is the current one with the files being deleted when not 
used in the tree.

I'm sorry but your solution doesn't really seem well thought. I'm used and 
like to use $P and alikes, sometimes with versionator, in SRC_URI for not 
having to modify this part when bumping a package. Your solution more or less 
annihilates this in the case it's bumped by different developers. IIRC at least 
one of {devmanual, policy, quizzes} mandates to have such scalable SRC_URI's.

A.

Re: [gentoo-dev] On hosting self-produced distfiles

[Diego Elio Pettenò]

Il giorno gio, 20/01/2011 alle 18.42 -0300, Alexis Ballier ha scritto:
> Solution b): update the distfiles-local mirroring script to store a copy where 
> it wont be deleted...
> The interim solution is the current one with the files being deleted when not 
> used in the tree.

Won't happen, infra is already spending time on the _final_ solution.

> Your solution more or less 
> annihilates this in the case it's bumped by different developers.

Wrong. Check app-emulation/libvirt for a two-developers package, or
dev-lang/ruby for shared team packages.

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

Re: [gentoo-dev] Re: Re: Re: On hosting self-produced distfiles

[Matti Bickel]

[-- Attachment #1: Type: text/plain, Size: 1092 bytes --]

On 01/20/2011 09:46 PM, Diego Elio Pettenò wrote:
> So I'm not asking _you_ to waste 90% of your time discussing and
> auditing licenses. We have a team for that.

We do? Cool, I didn't know about that. Which team is it?

> At the same time I'm not going to ask the developers to all evaluate
> case by case whether they should or shouldn't keep their stuff
> available. I'm telling them to put it there rather than in another
> place; what that will change shouldn't really be a problem.

I reiterate: I just wanted to understand what the bloody point is. I saw
your logic for "let's have permanent space", not "we need everybody to
use permanent space". But "Because I say so" 's valid enough for me.

Dunno, but I'd try to make stuff so cool and easy to use that people
*want* it and forget about the rest. But that's just me.

> Again, you might not care, we have other teams that do care. Since I'm
> not asking you to do a 180° jump changing your habits, can we please
> just agree that you don't see the point but you'll follow the request
> anyway?

Sure.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-dev] On hosting self-produced distfiles

[Alexis Ballier]

On Thursday, January 20, 2011 06:45:19 pm Diego Elio Pettenò wrote:
> > Your solution more or less
> > annihilates this in the case it's bumped by different developers.
> 
> Wrong. Check app-emulation/libvirt for a two-developers package, or
> dev-lang/ruby for shared team packages.

This either disallows non team members to do anything or force them to change 
SRC_URI. With every dev having write access to the tree this doesn't sound 
like a good solution to me.

Another problem: I'm currently keeping two releases of texlive in my home dir 
on pecker. Each of which is about 1G. Had I kept the 4 releases we have had, 
I'm almost sure infra would have come and asked me why I'm using so much 
space. According to 'df', /home is 124G big on pecker, meaning I'm currently 
using more space than what I should be allowed.

A.

Re: [gentoo-dev] Re: Re: On hosting self-produced distfiles

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1531 bytes --]

On Thursday, January 20, 2011 14:47:24 Diego Elio Pettenò wrote:
> Il giorno gio, 20/01/2011 alle 14.28 -0500, Mike Frysinger ha scritto:
> > then you should have mentioned this in your original e-mail rather
> > than
> > painting infra as angry unhelpful tools.  the changes you propose are
> > merely a
> > stop gap measure until infra finishes their work.
> 
> You of all people to sound surprised about this really doesn't look
> right. The fact that such an archive has been in the work is something
> know since I became a dev, six years ago; the bug was open in 2007.

just because a bug has been opened for a long time doesnt mean i'm aware of 
it.  ive never heard of this project before.

> Also "angry unhelpful tools"? You're putting words in my mail that
> aren't there. I said infra _used_ to be against that; it was obvious
> that knowing that I wouldn't just push the issue against their will.

no, you didnt.  you said "the Infra team has been against this idea".  you 
didnt say anywhere that infra had changed their mind, and this statement made 
it sound like basically "screw infra, this is what QA says, and if infra 
doesnt like it they can figure something out on their own".

> If you were trying to pick a fight for the sake of it, I'd suggest you
> find something else to do.

ah yes, i'm sure that's what i'm doing.  i love picking fights with you 
because you're awesome and i am not.  and/or i have no fscking idea what 
you're talking about.  hmm, one of those.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Re: Re: On hosting self-produced distfiles

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 2007 bytes --]

On Thursday, January 20, 2011 14:42:03 Diego Elio Pettenò wrote:
> Il giorno gio, 20/01/2011 alle 20.27 +0100, Matti Bickel ha scritto:
> > Not sure what you mean: if someone quickpkg's php and needs all the
> > source? Well, they already downloaded them. Better keep them around,
> > since it's *your* binary, not mine.
> 
> We do distribute part of our packages as binaries already so we have to
> be compliant with their licenses to begin with. Better doing it with a
> single sweep than trying to come up with abstruse case-by-case points,
> no?

my understanding is that releng already has a process in place so that when 
they do a binary release, they tag all the versions so that our mirrors retain 
the source archives.

> > Same thing, as already pointed out in another message. I see the point
> > in making it easier for them. That's okay. So what you're saying is
> > we're upstream too and upstream's should provide their historical stuff.
> 
> This is but _one_ reason, and just another thing to trickle down. I
> don't care if "FSF says it's their problem"; what is it costing us,
> really? The cost is minimal (as we need the archive anyway), and the
> gain is there for many people.

if we needed the archive, then this bullet point wouldnt have been relevant.  
but if we dont need the archive, then keeping it around for some unknown 
derivative distro out there doesnt make sense.  how do you pick which archives 
to keep ?  all of them ?  for how long ?  if you cant come up with a clear 
expiration process, then dont bother.

and yes, there is real cost to keeping around archives we dont need.  i cant 
imagine the people providing mirrors for our project for free are going to say 
"sure, balloon the lists of files we have to mirror all you want".

> Arguing against this is just getting to the point of arguing because
> somebody is doing what nobody did for a long time: taking decisions.

semantically speaking, you make decisions
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Re: Re: Re: Re: On hosting self-produced distfiles

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1175 bytes --]

On Thursday, January 20, 2011 16:21:33 Diego Elio Pettenò wrote:
> Il giorno gio, 20/01/2011 alle 22.14 +0100, Jeroen Roovers ha scritto:
> > I'd be happier if you responded my proposal of changing the meaning of
> > mirror://gentoo - the mirror sync scripts could take care of the
> > entire
> > thing by forever storing files described that way. That way, no
> > ebuilds/eclasses using mirror://gentoo would need changing, and we
> > could conveniently move away from using dev.gentoo.org for
> > availability
> > of old self-hosted SRC_URI files.
> 
> http://bugs.gentoo.org/show_bug.cgi?id=176186
> 
> I'VE NOT DREAMT ABOUT THIS LAST NIGHT. This is why there was no
> "discussion": it was all planned already, from a long time ago.

first off, drop the caps crap.  second, while *you* might be aware of a long 
history, you provided absolutely none in your first e-mail.  thus it 
completely looked like only 1 person (you) was making a decision which had not 
been discussed with anyone else.  and having recently been placed into QA 
lead, you decided to use that position to force everyone else to agree.  
again, without any discussion.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Re: Re: On hosting self-produced distfiles

[Dale]

Mike Frysinger wrote:
> On Thursday, January 20, 2011 14:47:24 Diego Elio Pettenò wrote:
>    
> << SNIP >>
>> Also "angry unhelpful tools"? You're putting words in my mail that
>> aren't there. I said infra _used_ to be against that; it was obvious
>> that knowing that I wouldn't just push the issue against their will.
>>      
> no, you didnt.  you said "the Infra team has been against this idea".  you
> didnt say anywhere that infra had changed their mind, and this statement made
> it sound like basically "screw infra, this is what QA says, and if infra
> doesnt like it they can figure something out on their own".
> <<SNIP>>-mike
>    

Actually, yea he did.  In your quote of him, he said "has been'.  Maybe 
you misread it but that means they had a different view or opinion in 
the past but that has changed.

Dale

:-)  :-)

Re: [gentoo-dev] Re: Re: On hosting self-produced distfiles

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1372 bytes --]

On Friday, January 21, 2011 02:53:57 Dale wrote:
> Mike Frysinger wrote:
> > On Thursday, January 20, 2011 14:47:24 Diego Elio Pettenò wrote:
> >> Also "angry unhelpful tools"? You're putting words in my mail that
> >> aren't there. I said infra _used_ to be against that; it was obvious
> >> that knowing that I wouldn't just push the issue against their will.
> > 
> > no, you didnt.  you said "the Infra team has been against this idea". 
> > you didnt say anywhere that infra had changed their mind, and this
> > statement made it sound like basically "screw infra, this is what QA
> > says, and if infra doesnt like it they can figure something out on their
> > own".
> 
> Actually, yea he did.  In your quote of him, he said "has been'.  Maybe
> you misread it but that means they had a different view or opinion in
> the past but that has changed.

no, not really.  "has been" is the present perfect continuous tense which 
means he is describing something that has continued up to or through $now.  
without explicitly stating that infra has changed their minds, my assessment 
above stands.

yes, this is a nuance that might be hard for non-native (and probably many 
native) english speakers to pick up, but that's why it's even more important 
for people to provide more supplementary details so that they arent 
misconstrued.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Re: Re: On hosting self-produced distfiles

[Luca Barbato]

On 01/21/2011 04:17 PM, Mike Frysinger wrote:
>> Actually, yea he did.  In your quote of him, he said "has been'.  Maybe
>> you misread it but that means they had a different view or opinion in
>> the past but that has changed.
> 
> no, not really.  "has been" is the present perfect continuous tense which 
> means he is describing something that has continued up to or through $now.  
> without explicitly stating that infra has changed their minds, my assessment 
> above stands.

> yes, this is a nuance that might be hard for non-native (and probably many 
> native) english speakers to pick up, but that's why it's even more important 
> for people to provide more supplementary details so that they arent 
> misconstrued.

I looked it up since I recall and used it to state

1. An action that has just stopped or recently stopped

while you interpreted it as

2. An action continuing up to now

Apparently both are right, the latter usually needs a temporal reference
like "since $time", "for $time".

"Yes, we all know that the history with the Infra team has been against
this idea, but until there is a proper replacement for this handling,
the Gentoo sources archive, we really shouldn't be putting the data in
non-permanent locations, the team should, nowadays, be on the same page
as me on this."

I read it as: before infra was against, now they should be on the same
page -> thus agree.

my 2 eurocents

lu

-- 

Luca Barbato
Gentoo/linux
http://dev.gentoo.org/~lu_zero

Re: [gentoo-dev] Re: Re: On hosting self-produced distfiles

[Rich Freeman]

On Fri, Jan 21, 2011 at 11:00 AM, Luca Barbato <lu_zero@gentoo.org> wrote:
> "Yes, we all know that the history with the Infra team has been against
> this idea, but until there is a proper replacement for this handling,
> the Gentoo sources archive, we really shouldn't be putting the data in
> non-permanent locations, the team should, nowadays, be on the same page
> as me on this."
>
> I read it as: before infra was against, now they should be on the same
> page -> thus agree.
>
> my 2 eurocents

I read it the same way - they used to be against it, but now are.  The
word "history" implied to me that this was something that was
discussed in the past, but perhaps not recently.  The sense I got out
of it is that once upon a time this was discussed and we went one way,
and now for whatever reason we're thinking differently.  Probably
in-between nobody thought about it much at all.

That's my 2 US cents (probably worth a ha'penny these days, minus any
collector status) as a native speaker of American all my life.  I
won't insult my friends across the pond by calling it "English."  :)

In any case, we're not really debating English here.  Some of the
replies in the thread did come across to me as a bit
personal/degrading/etc.  Let's just all try to get along, and also
let's also try to understand that not everybody here speaks the
Queen's English as well as I don't.

Re: [gentoo-dev] Re: Re: On hosting self-produced distfiles

[George Shapovalov]

>[bunch of linguistic stuff skipped..]
Huh?
Is it just me or is there anybody else who thinks that this went way beyound 
ridiculous? Especially considering that Diego himself is not a native English 
speaker..
Anyway, after having been all over the world (including 7 years in California) 
and having been in contact with many people I tend to *not* look at precise 
word/phrase meaning any more. Remember - oral/written, actually any kind of 
communication in any language is a *noisy channel*. Lets get back on topic? 
(OTOH it seems to me the original topic has also been bitten to death 
already).

George

Re: [gentoo-dev] Re: On hosting self-produced distfiles

[Roy Bamford]

[-- Attachment #1: Type: text/plain, Size: 1072 bytes --]

On 2011.01.20 02:55, Diego Elio Pettenò wrote:
> Il giorno mer, 19/01/2011 alle 20.07 -0500, Rich Freeman ha scritto:
> > Forward going?  Or, should we go ahead and start retroactively
> > updating ebuilds that have mirror:// in them right now?  Presumably
> > without a revbump over something like this...
> > 
> 
> I wouldn't mind if it was done retroactively, but I'm not going to 
> ask
> right now for all the ebuilds in tree right now to be converted. If
> you
> do happen to pass through a bunch of old ebuilds and edit them anyway
> please do update them to use long-term-reachable URLs.
> 
> -- 
> Diego Elio Pettenò — Flameeyes
> http://blog.flameeyes.eu/
> 

Diego,

A few questions ...

How does this work when a developer leaves and his account is removed.  
This will break the links to ~/public_html.

What about our GPL obligations to provide sources?
That carries on even after an ex developers ~/public_html has been 
deleted.

-- 
Regards,

Roy Bamford
(Neddyseagoon) a member of
gentoo-ops
forum-mods
trustees

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] On hosting self-produced distfiles

[Jeremy Olexa]

 On Thu, 20 Jan 2011 01:50:35 +0100, Diego Elio Pettenò wrote:
> <snip>
> Pushing files that are still available somewhere else, but cannot be
> directly fetched for whatever reason is still to be bone
> through /space/distfiles-local.

 This, above, is an important point for the below comment.
 
> *PLEASE NOTE:* This is to be considered QA policy, so we're going to 
> ask
> soon to enforce this. This requirement, though, _will_ be superseded 
> as
> soon as Infra provides us with a proper archive for this kind of 
> files.

 I wonder how the QA team is going to enforce this? I can't think of any 
 heuristics to distinguish between mirror://gentoo/<self-produced 
 distfile> and mirror://gentoo/<upstream distfile not self-produced> - 
 One way this can happen is dead upstream but useful application. Yes, we 
 have GENTOO_MIRRORS but I've seen it often enough that GENTOO_MIRRORS 
 value is invalid (user error) and SRC_URI is invalid which leads to a 
 non-ideal user experience - at which point $dev normally changes SRC_URI 
 to mirror://gentoo/.

 Care to touch on that topic? Note: An acceptable answer to me would be 
 "update documentation and use common sense" because I don't think there 
 is any other option. At risk of being too wordy, I'll leave it at that.

 Thanks,
 Jeremy

[gentoo-dev] Re: On hosting self-produced distfiles

[Duncan]

Roy Bamford posted on Fri, 21 Jan 2011 20:50:53 +0000 as excerpted:

> On 2011.01.20 02:55, Diego Elio Pettenò wrote:
>> Il giorno mer, 19/01/2011 alle 20.07 -0500, Rich Freeman ha scritto:
>> > Forward going?  Or, should we go ahead and start retroactively
>> > updating ebuilds that have mirror:// in them right now?  Presumably
>> > without a revbump over something like this...
>> > 
>> > 
>> I wouldn't mind if it was done retroactively, but I'm not going to ask
>> right now for all the ebuilds in tree right now to be converted. If you
>> do happen to pass through a bunch of old ebuilds and edit them anyway
>> please do update them to use long-term-reachable URLs.
>> 
> How does this work when a developer leaves and his account is removed.
> This will break the links to ~/public_html.
> 
> What about our GPL obligations to provide sources?
> That carries on even after an ex developers ~/public_html has been
> deleted.

<My understanding, IANAL, etc.>

Thankfully, Gentoo doesn't have to worry about that in most cases, because 
we don't provide binaries in most cases, and that's what triggers it.

Where we do provide binaries, I believe we're fine as long as we've 
offered sources along with (and in the same manner as, CD for CD, ISO for 
ISO, link for link) the binaries as that fulfills our obligation.  As long 
as the sources remain available with the binaries and the binaries are 
removed either first or at the same time as the sources, we're covered.  
If someone didn't choose to grab the sources, that's their problem.

Only if we don't offer sources at the same time and in the same manner, 
but instead accompany the binaries with an offer for sources, does the 
(AFAIK) 3-year delay kick in and we have to worry about providing sources 
so long after the binaries are long outdated and for our part, forgotten.

IOW, it pays to ensure that whenever we distribute binaries, we make 
sources available at the same time and in the same manner, as by doing so 
we avoid the three year clause entirely. =:^)  Are we always ensuring that 
source availability in every act of (L)GPLed binary distribution?  That's 
the BIG question, as it avoids at least the legal obligation (tho 
retention can be useful for practical reasons) of worrying about what 
happens to the sources after we've stopped distributing the binaries (and 
developers have left, etc) entirely.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

[gentoo-dev] Re: On hosting self-produced distfiles

[Christian Faulhammer]

[-- Attachment #1: Type: text/plain, Size: 845 bytes --]

Hi,

Mike Frysinger <vapier@gentoo.org>:
> first off, drop the caps crap.  second, while *you* might be aware of
> a long history, you provided absolutely none in your first e-mail.
> thus it completely looked like only 1 person (you) was making a
> decision which had not been discussed with anyone else.  and having
> recently been placed into QA lead, you decided to use that position
> to force everyone else to agree. again, without any discussion.

 So now that all misunderstandings are cleared out and everybody agrees
that there was a failure in communication (both on sending and
receiving end), we can go on committing awesome stuff to the tree.
Thanks.

V-Li

-- 
Christian Faulhammer, Gentoo Lisp project
<URL:http://www.gentoo.org/proj/en/lisp/>, #gentoo-lisp on FreeNode

<URL:http://gentoo.faulhammer.org/>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] On hosting self-produced distfiles

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1121 bytes --]

Hi Diego,

I need a clarification on something.

On Thu, Jan 20, 2011 at 01:50:35AM +0100, Diego Elio Petten? wrote:
> If you produced the file yourself, and it doesn't matter if the file is
> reproducible (unless it is reproducible to sha512 identity), please use
> the public_html directory in your dev.gentoo.org home to host these.
> This makes sure that the file won't be deleted from all its sources if
> the ebuild is removed (or more likely replaced) from tree. Ask the Emacs
> team how "easy" has been to recover gentoo-syntax files before.

What is your feeling about projects that use tags in their source
repositories and have a way to build the tarball directly from the
repository?

e.g. for openrc, if you have a clone of the repository you can do:

git checkout [tag]
make dist

and you will have the same tarball that we released.

What about things hosted on github? I am upstream for some code hosted
there and they prefer that you do not upload source tarballs, but use
tags in your repository which will automatically be propegated to your
downloads page.

Your input would be appreciated.

William


[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

[gentoo-dev] Re: On hosting self-produced distfiles

[Diego Elio Pettenò]

Il giorno sab, 22/01/2011 alle 10.20 -0600, William Hubbs ha scritto:
> What is your feeling about projects that use tags in their source
> repositories and have a way to build the tarball directly from the
> repository?

I have written that already: can you provide a SHA512 identity of the
file? "The same tarball" is not really sufficient.

In general, though, for Gentoo-produced distfiles the archive _will_
have to host the files, even if Git provides an "easy" way to deal with
that.

> What about things hosted on github? I am upstream for some code hosted
> there and they prefer that you do not upload source tarballs, but use
> tags in your repository which will automatically be propegated to your
> downloads page.

GitHub _used_ to be broken and provide non-stable downloads, but they
have since fixed that and the download of a tag will always have the
same SHA512 digest, so it's just fine.

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

Re: [gentoo-dev] Re: On hosting self-produced distfiles

[Enrico Weigelt]

* Diego Elio Pettenò <flameeyes@gmail.com> schrieb:

> GitHub _used_ to be broken and provide non-stable downloads, but they
> have since fixed that and the download of a tag will always have the
> same SHA512 digest, so it's just fine.

That's great. How exactly did they do that ?

IIRC one of the problems is/was that both tar and gz might not
be strictly deterministic (same input content might produce
differing tarball output) - am I wrong here ?


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [gentoo-dev] On hosting self-produced distfiles

[Jesus Rivero (Neurogeek)]

Hi all,

On Wed, Jan 19, 2011 at 7:50 PM, Diego Elio Pettenò <flameeyes@gmail.com> wrote:
> Hi all,
>
> I just wanted to write here a clarification regarding self-produced
> distfiles, such as patchset tarballs, SCM snapshots and the like. Some
> people seem under the impression that the correct way to host these is
> to use mirror://gentoo/ and copy them on /space/distfiles-local on
> dev.g.o. Please don't do this.
>
> If you produced the file yourself, and it doesn't matter if the file is
> reproducible (unless it is reproducible to sha512 identity), please use
> the public_html directory in your dev.gentoo.org home to host these.
> This makes sure that the file won't be deleted from all its sources if
> the ebuild is removed (or more likely replaced) from tree. Ask the Emacs
> team how "easy" has been to recover gentoo-syntax files before.

So, after all that, I'm not really clear if we are enforcing this, or not.
A have a patchset. I put it in my devspace as suggested here. Is this
right or should I still use the mirror://gentoo as the devmanual
suggests?

Thanks,

-- 
Jesus Rivero (Neurogeek)
Gentoo Developer