[gentoo-dev] Proposal: ban mirror://gentoo/ from ebuilds

[gentoo-dev] Proposal: ban mirror://gentoo/ from ebuilds

[Diego Elio Pettenò]

[-- Attachment #1: Type: text/plain, Size: 1283 bytes --]

Hello everybody,

I have already said this before, but it looks like nobody cared. We have
a problem for what concerns Gentoo-generated distfiles.

This includes custom snapshots, custom packages, patches, patchsets, and
so on so forth. While it was infra that (back when I joined at least)
asked not to use dev.gentoo.org for hosting said fails and rather prefer
to use mirror://gentoo/, they already stated that it's not a problem to
do so until we have a proper system in place (system that has been
considered and worked on for quite a bit already and yet is not
available).

Unfortunately, as long as the mirror://gentoo/ option is still
maintained, we'll end up with situations like today's gnuconfig that
couldn't be fetched, causing all ~arch users to see the same failure,
because the distfile wasn't uploaded to the staging area. Of course the
same could happen with a stable SRC_URI, but then it would fail after
hitting that, rather than going through half the gentoo mirrors trying
to find a file that is not there.

So, anybody has reasons beside laziness, or concern for infra's disk
usage (that argument is allowed to come only from infra members!), to
not go this route?

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Proposal: ban mirror://gentoo/ from ebuilds

[Patrick Lauer]

On 08/18/11 10:50, Diego Elio Pettenò wrote:
> Hello everybody,
> 
> I have already said this before, but it looks like nobody cared. We have
> a problem for what concerns Gentoo-generated distfiles.

People being quiet doesn't imply they don't care - just that it gets
really frustrating to repeat the same thing every week ;)

> This includes custom snapshots, custom packages, patches, patchsets, and
> so on so forth. While it was infra that (back when I joined at least)
> asked not to use dev.gentoo.org for hosting said fails and rather prefer
> to use mirror://gentoo/, they already stated that it's not a problem to
> do so until we have a proper system in place (system that has been
> considered and worked on for quite a bit already and yet is not
> available).

So - what needs to be done to finally make it happen?

> Unfortunately, as long as the mirror://gentoo/ option is still
> maintained, we'll end up with situations like today's gnuconfig that
> couldn't be fetched, 

That's not the only one - if you want to recreate a year-old system
you'll have trouble finding those patches because the ebuilds were
removed and the distfiles removed from the mirrors

> causing all ~arch users to see the same failure,
> because the distfile wasn't uploaded to the staging area. Of course the
> same could happen with a stable SRC_URI, but then it would fail after
> hitting that, rather than going through half the gentoo mirrors trying
> to find a file that is not there.
> 
> So, anybody has reasons beside laziness, or concern for infra's disk
> usage (that argument is allowed to come only from infra members!), to
> not go this route?

I'll even donate two 2TB disks to kill the disk space argument ;)
Hardware is cheap, the maintenance and planning is most likely more of a
bottleneck - but then there's enough smart people around if anyone needs
help.

Re: [gentoo-dev] Proposal: ban mirror://gentoo/ from ebuilds

[Ulrich Mueller]

>>>>> On Thu, 18 Aug 2011, Patrick Lauer wrote:

> On 08/18/11 10:50, Diego Elio Pettenò wrote:
>> This includes custom snapshots, custom packages, patches,
>> patchsets, and so on so forth. While it was infra that (back when I
>> joined at least) asked not to use dev.gentoo.org for hosting said
>> fails and rather prefer to use mirror://gentoo/, they already
>> stated that it's not a problem to do so until we have a proper
>> system in place (system that has been considered and worked on for
>> quite a bit already and yet is not available).

> So - what needs to be done to finally make it happen?

It's outlined in bug 176186.
<https://bugs.gentoo.org/show_bug.cgi?id=176186>

Ulrich

Re: [gentoo-dev] Proposal: ban mirror://gentoo/ from ebuilds

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 2039 bytes --]

Diego Elio Pettenò schrieb:
> Hello everybody,
> 
> I have already said this before, but it looks like nobody cared. We have
> a problem for what concerns Gentoo-generated distfiles.
> 
> This includes custom snapshots, custom packages, patches, patchsets, and
> so on so forth. While it was infra that (back when I joined at least)
> asked not to use dev.gentoo.org for hosting said fails and rather prefer
> to use mirror://gentoo/, they already stated that it's not a problem to
> do so until we have a proper system in place (system that has been
> considered and worked on for quite a bit already and yet is not
> available).
> 
> Unfortunately, as long as the mirror://gentoo/ option is still
> maintained, we'll end up with situations like today's gnuconfig that
> couldn't be fetched, causing all ~arch users to see the same failure,
> because the distfile wasn't uploaded to the staging area. Of course the
> same could happen with a stable SRC_URI, but then it would fail after
> hitting that, rather than going through half the gentoo mirrors trying
> to find a file that is not there.
> 
> So, anybody has reasons beside laziness, or concern for infra's disk
> usage (that argument is allowed to come only from infra members!), to
> not go this route?
> 

I see no improvement from this proposal. If a tarball is not uploaded, the users will always fail to
get it, independent from the place, where it should be going to. And even if you put the
dev.gentoo.org address in SRC_URI, portage will first check the mirrors before it checks SRC_URI, so
this would in the end result in even 1 more download try without success.

The argument about dropped tarballs, once the ebuilds gets removed might weight a bit more, but you
cannot depend on other upstream keeping their tarballs around forever, so i see no requirement for
us preserving only specific tarballs (those created by our devs), while upstream tarballs could
already be gone and are not preserved, once the ebuild for it is gone.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 380 bytes --]

Re: [gentoo-dev] Proposal: ban mirror://gentoo/ from ebuilds

[Anthony G. Basile]

On 08/18/2011 05:15 AM, Thomas Sachau wrote:
> Diego Elio Pettenò schrieb:
>> Hello everybody,
>>
>> I have already said this before, but it looks like nobody cared. We have
>> a problem for what concerns Gentoo-generated distfiles.
>>
>> This includes custom snapshots, custom packages, patches, patchsets, and
>> so on so forth. While it was infra that (back when I joined at least)
>> asked not to use dev.gentoo.org for hosting said fails and rather prefer
>> to use mirror://gentoo/, they already stated that it's not a problem to
>> do so until we have a proper system in place (system that has been
>> considered and worked on for quite a bit already and yet is not
>> available).
>>
>> Unfortunately, as long as the mirror://gentoo/ option is still
>> maintained, we'll end up with situations like today's gnuconfig that
>> couldn't be fetched, causing all ~arch users to see the same failure,
>> because the distfile wasn't uploaded to the staging area. Of course the
>> same could happen with a stable SRC_URI, but then it would fail after
>> hitting that, rather than going through half the gentoo mirrors trying
>> to find a file that is not there.
>>
>> So, anybody has reasons beside laziness, or concern for infra's disk
>> usage (that argument is allowed to come only from infra members!), to
>> not go this route?
>>
> 
> I see no improvement from this proposal. If a tarball is not uploaded, the users will always fail to
> get it, independent from the place, where it should be going to. And even if you put the
> dev.gentoo.org address in SRC_URI, portage will first check the mirrors before it checks SRC_URI, so
> this would in the end result in even 1 more download try without success.
> 
> The argument about dropped tarballs, once the ebuilds gets removed might weight a bit more, but you
> cannot depend on other upstream keeping their tarballs around forever, so i see no requirement for
> us preserving only specific tarballs (those created by our devs), while upstream tarballs could
> already be gone and are not preserved, once the ebuild for it is gone.
> 

What alternative are you proposing to mirror://gentoo/ if upstream
doesn't provide a tarball, eg with large patchsets the maintainer
constructs?  Anticipating your answer might be "keep them in your dev
space", then what would be the deprecation policy for distfiles that are
no longer used by ebuilds?  If foresee a tension between keeping all the
distfiles vs disk space, although Patrick's point of cheap hardware is
well taken.


-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 8040 5A4D 8709 21B1 1A88  33CE 979C AF40 D045 5535
GnuPG ID  : D0455535

[gentoo-dev] Re: Proposal: ban mirror://gentoo/ from ebuilds

[Diego Elio Pettenò]

[-- Attachment #1: Type: text/plain, Size: 797 bytes --]

Il giorno gio, 18/08/2011 alle 05.46 -0400, Anthony G. Basile ha
scritto:
> 
> What alternative are you proposing to mirror://gentoo/ if upstream
> doesn't provide a tarball, eg with large patchsets the maintainer
> constructs?  Anticipating your answer might be "keep them in your dev
> space", then what would be the deprecation policy for distfiles that
> are
> no longer used by ebuilds?  If foresee a tension between keeping all
> the
> distfiles vs disk space, although Patrick's point of cheap hardware is
> well taken. 

Keep it in your dev space. As I said, the resources argument is only
available for infra to complain about, and since last I knew from them
was that it was not a problem right now...

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[gentoo-dev] Re: Proposal: ban mirror://gentoo/ from ebuilds

[Diego Elio Pettenò]

[-- Attachment #1: Type: text/plain, Size: 1094 bytes --]

Il giorno gio, 18/08/2011 alle 11.15 +0200, Thomas Sachau ha scritto:
> 
> The argument about dropped tarballs, once the ebuilds gets removed
> might weight a bit more, but you
> cannot depend on other upstream keeping their tarballs around forever,
> so i see no requirement for
> us preserving only specific tarballs (those created by our devs),
> while upstream tarballs could
> already be gone and are not preserved, once the ebuild for it is gone.

Perfect is the enemy of good.

_Most_ upstream projects keep tarballs available of all historic
releases; okay maybe not _all_ projects, but most, especially those
using services such as SourceForge, Google Code, RubyForge,
Rubygems, ...

For _our_ projects, snapshots, or packages, let's try to apply a sane
policy. And that starts by not arguing that we shouldn't be doing so
because others might not be doing so themselves.

Heck, let's try to look up for the projects doing things right, not
justify our failures with those doing things wrong.

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: Proposal: ban mirror://gentoo/ from ebuilds

[Anthony G. Basile]

On 08/18/2011 05:53 AM, Diego Elio Pettenò wrote:
> Il giorno gio, 18/08/2011 alle 05.46 -0400, Anthony G. Basile ha
> scritto:
>>
>> What alternative are you proposing to mirror://gentoo/ if upstream
>> doesn't provide a tarball, eg with large patchsets the maintainer
>> constructs?  Anticipating your answer might be "keep them in your dev
>> space", then what would be the deprecation policy for distfiles that
>> are
>> no longer used by ebuilds?  If foresee a tension between keeping all
>> the
>> distfiles vs disk space, although Patrick's point of cheap hardware is
>> well taken. 
> 
> Keep it in your dev space. As I said, the resources argument is only
> available for infra to complain about, and since last I knew from them
> was that it was not a problem right now...
> 

Understood that infra gets to complain, but that still doesn't tell me
what the deprecation policy is.  Keep all my large patchsets
indefinitely? Or remove them when an ebuild no longer needs them?  Or 1
year after an ebuild no longer needs them?

-- 
Anthony G. Basile, Ph. D.
Chair of Information Technology
D'Youville College
Buffalo, NY 14201
(716) 829-8197

Re: [gentoo-dev] Re: Proposal: ban mirror://gentoo/ from ebuilds

[Chí-Thanh Christopher Nguyễn]

Diego Elio Pettenò schrieb:
> Keep it in your dev space. As I said, the resources argument is only
> available for infra to complain about, and since last I knew from them
> was that it was not a problem right now...

I think robbat2 complained on IRC recently to devs which had their home
directories full of some stuff.

Regarding the problem of the developer neglecting (happened to me too,
ahem) to upload distfiles, maybe repoman could check for this?


Best regards,
Chí-Thanh Christopher Nguyễn

[gentoo-dev] Re: Re: Proposal: ban mirror://gentoo/ from ebuilds

[Diego Elio Pettenò]

[-- Attachment #1: Type: text/plain, Size: 636 bytes --]

Il giorno gio, 18/08/2011 alle 12.09 +0200, Chí-Thanh Christopher Nguyễn
ha scritto:
> 
> Regarding the problem of the developer neglecting (happened to me too,
> ahem) to upload distfiles, maybe repoman could check for this? 

https://bugs.gentoo.org/show_bug.cgi?id=315243

I know I'm not as useful by just reporting wishlist features for
repoman, but I'm afraid my Python skills are nowhere near what I'd be
needing, and either I report the tinderbox stuff or I try hacking
something within repoman... the former is probably more useful right
now.

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: Proposal: ban mirror://gentoo/ from ebuilds

[Thomas Sachau]

[-- Attachment #1: Type: text/plain, Size: 853 bytes --]

Chí-Thanh Christopher Nguyễn schrieb:
> Diego Elio Pettenò schrieb:
>> Keep it in your dev space. As I said, the resources argument is only
>> available for infra to complain about, and since last I knew from them
>> was that it was not a problem right now...
> 
> I think robbat2 complained on IRC recently to devs which had their home
> directories full of some stuff.

He did and this means, that while it would be ok to have patchsets or smaller tarballs in your
homedir, this is no longer ok for big tarballs or many bigger tarballs (e.g. proper tarball for
upstream sources or many tarballs due to many upstream releases), so homedir on dev.gentoo.org
cannot be the only answer.

This means, that the space for dev-created tarballs should be ready before we can deprecate direct
uploads to our mirrors via mirror://gentoo.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 380 bytes --]

Re: [gentoo-dev] Proposal: ban mirror://gentoo/ from ebuilds

[Jorge Manuel B. S. Vicetto]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 18-08-2011 08:50, Diego Elio Pettenò wrote:


Diego,

I understand the concern, but instead of banning the use of
mirror://gentoo, why not make it mandatory to have a dev.gentoo.org link
in SRC_URI when there's a mirror://gentoo link?

- -- 
Regards,

Jorge Vicetto (jmbsvicetto) - jmbsvicetto at gentoo dot org
Gentoo- forums / Userrel / Devrel / KDE / Elections / RelEng
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJOTP1+AAoJEC8ZTXQF1qEP/EQQAKMv5JQsj6dH9ry2+Pukf6oh
PqXJkcJJBe2b3vIHRdaj50qiXvpZDYSsRX4U5juCG9nwlYSx0UyZkBVZl9IQHmpB
8XRXZAsjtku100mejuOm8RffINXWvy1pefvN1Qdofnu1US/l8SXSZCQf9VikRvLI
HEcrCAHETSjm4ZbuChc9o2ZH2qbkeRZCctZvmVSOWcTRMdkBoNAwYTDnCYzly36m
OfMLWCwpNBhIT3z4mK7BEMYx8kGAr7xM5tzBnBXaLu3XJ6Q2iX/yFsrdxJLoaxnx
OxMk3DxG/8zdD41u1UPh2oSDaYIfDl1JAGv2etvNPmatCv3RqL4vKRp8J1buZmJw
/IK8C7mzj0HU7sOaIkIAZDP0fi/9QG0RPYSh6bi2mVF+mQ3lOWLdXdzDtT9X7aUW
zvBgVUvmNLgn52jvGK0fzaqtfQ9rL+RYuM46ziTPg2eJOK7adrlAHcyE7RVIl4ks
z07qV4EqzFWmDXriRo8C556XUkCUO16X0QQFKmm6TrM58PEqjP1mviCxMKj6RTYI
4pkNKyFnIDI7z2i6NaCVwi+V3OvB1BthiMMVt0TgM8zjV/99sZw7LVpX3WIaiFdR
yUz/f8OPTPmZfQNtXz2MGmoX6CNSOIUyH6uo6GPIY/dtsh/jLiOlOOUQB9FWpsq9
5UFiY6VYzkDd1wkMA1hb
=KKq4
-----END PGP SIGNATURE-----

Re: [gentoo-dev] Proposal: ban mirror://gentoo/ from ebuilds

[Jorge Manuel B. S. Vicetto]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 18-08-2011 08:50, Diego Elio Pettenò wrote:
> Hello everybody,
> 
> I have already said this before, but it looks like nobody cared. We 
> have a problem for what concerns Gentoo-generated distfiles.
> 
> This includes custom snapshots, custom packages, patches, patchsets, 
> and so on so forth. While it was infra that (back when I joined at 
> least) asked not to use dev.gentoo.org for hosting said fails and 
> rather prefer to use mirror://gentoo/, they already stated that it's 
> not a problem to do so until we have a proper system in place
> (system that has been considered and worked on for quite a bit
> already and yet is not available).
> 
> Unfortunately, as long as the mirror://gentoo/ option is still 
> maintained, we'll end up with situations like today's gnuconfig that
>  couldn't be fetched, causing all ~arch users to see the same 
> failure, because the distfile wasn't uploaded to the staging area.
> Of course the same could happen with a stable SRC_URI, but then it
> would fail after hitting that, rather than going through half the
> gentoo mirrors trying to find a file that is not there.
> 
> So, anybody has reasons beside laziness, or concern for infra's disk
>  usage (that argument is allowed to come only from infra members!), 
> to not go this route?

Diego,

I understand the concern, but instead of banning the use of
mirror://gentoo, why not make it mandatory to have a dev.gentoo.org link
in SRC_URI when there's a mirror://gentoo link?

- -- 
Regards,

Jorge Vicetto (jmbsvicetto) - jmbsvicetto at gentoo dot org
Gentoo- forums / Userrel / Devrel / KDE / Elections / RelEng
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJOTP2OAAoJEC8ZTXQF1qEPWnoP/jpHJAonBiWm7p6xp1AjuAaE
t2vkFYFpxQRJ0IOYSaXh1Qrynhzc7ky5sT3G6/sLZPARfbCKinjWEXqGJMyvEsHU
tkk4cRdTeoi1iCa8tDtMVgXCEVofhwYJZQvbm+rVRzwuGgks4kNxsVOlAcmPZ3iU
tj9mfgFqKmhuNbFHe/fcPTkEFmCfyMFQdbt0btdk9PKOZK4MK39lEd6kifkNFRKp
Xtzud3uttud+5bOJIQ4er2OthNQhQGEA2SUnS8M5TL0qZyQDfHNrxZGMwbCu/qqz
vsldI6H6d0mCpuBDJ0WeFTcwwgSRCrM2+WiHsseJCUQODcSNkAikSvtaSNyx1LZg
PDQt05m4cX4bQ1I2obwJUvvss7LrDUH9P0JRE+4SHQsPAPwUMNMl3wfPg+PTXXF+
EI1QVEySaMsbJE3czQ7x8stt7Ogzh54eswwKn7Z40yjByX95gH+Oy75ouZmNV5AX
FEnnvpOCyOorc+K1ye+q26rwYHOndtuI/geGRHaF4HDpY+DKyzkq5Pj4v+m7/njP
V1QzYbMu+CYeBlnpnvpQ690Kxayn9e2+5n6DDzoJw1SjBMfJcZ9RnqXJvi0+6uJ3
fuH/FR9iczxambDEgafACKLGXfaNUuIfDeeXK3Xf1qXp2JYfDNmR68HWN8a0hjBm
SDk6u/zlsSpq6oYuUgAg
=96T7
-----END PGP SIGNATURE-----

[gentoo-dev] Re: Proposal: ban mirror://gentoo/ from ebuilds

[Diego Elio Pettenò]

[-- Attachment #1: Type: text/plain, Size: 711 bytes --]

Il giorno gio, 18/08/2011 alle 11.54 +0000, Jorge Manuel B. S. Vicetto
ha scritto:
> 
> I understand the concern, but instead of banning the use of
> mirror://gentoo, why not make it mandatory to have a dev.gentoo.org
> link
> in SRC_URI when there's a mirror://gentoo link? 

Because they are functionally equivalent.

If you do have a SRC_URI pointing to dev.g.o, then portage _will_ look
for the file in the gentoo mirrors by default unless you explicitly
RESTRICT it.

Having two SRC_URI entries is not really going to make a difference, and
it's easier to catch: you just remove the gentoo entry from
thirdpartymirrors.

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

[gentoo-dev] Re: Proposal: ban mirror://gentoo/ from ebuilds

[Ulrich Mueller]

>>>>> On Thu, 18 Aug 2011, Diego Elio Pettenò wrote:

>> I understand the concern, but instead of banning the use of
>> mirror://gentoo, why not make it mandatory to have a dev.gentoo.org
>> link
>> in SRC_URI when there's a mirror://gentoo link? 

> Because they are functionally equivalent.

They are not, in fact.

> If you do have a SRC_URI pointing to dev.g.o, then portage _will_
> look for the file in the gentoo mirrors by default unless you
> explicitly RESTRICT it.

And if you have an SRC_URI pointing to mirror://gentoo/ then Portage
will look for the file on Gentoo mirrors, regardless of any fetch or
mirror restriction. This can be useful, e.g., if you need the mirror
restriction only for one file in SRC_URI.

Ulrich

[gentoo-dev] Re: Proposal: ban mirror://gentoo/ from ebuilds

[Diego Elio Pettenò]

[-- Attachment #1: Type: text/plain, Size: 759 bytes --]

Il giorno gio, 18/08/2011 alle 16.11 +0200, Ulrich Mueller ha scritto:
> 
> And if you have an SRC_URI pointing to mirror://gentoo/ then Portage
> will look for the file on Gentoo mirrors, regardless of any fetch or
> mirror restriction. This can be useful, e.g., if you need the mirror
> restriction only for one file in SRC_URI. 

Right, I was avoiding that point, because as far as I can tell, it's not
what Jorge was thinking of.

SRC_URI="http://dev.gentoo.org/~flameeyes/${PN}/${P}.tar.xz"

and

SRC_URI="mirror://gentoo/${P}.tar.xz
http://dev.gentoo.org/~flameeyes/${PN}/${P}.tar.xz"

are functionally equivalent though, as long as there is no mirror
restriction.

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Proposal: ban mirror://gentoo/ from ebuilds

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 527 bytes --]

On Thursday, August 18, 2011 04:50:23 Diego Elio Pettenò wrote:
> Unfortunately, as long as the mirror://gentoo/ option is still
> maintained, we'll end up with situations like today's gnuconfig that
> couldn't be fetched, causing all ~arch users to see the same failure,
> because the distfile wasn't uploaded to the staging area.

you're understanding of this particular bug is completely off.  i forgot to 
upload the pkg -- the only place it existed was on my desktop.  it had nothing 
to do with SRC_URI.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[gentoo-dev] Re: Proposal: ban mirror://gentoo/ from ebuilds

[Diego Elio Pettenò]

[-- Attachment #1: Type: text/plain, Size: 623 bytes --]

Il giorno gio, 18/08/2011 alle 10.54 -0400, Mike Frysinger ha scritto:
> 
> you're understanding of this particular bug is completely off.  i
> forgot to 
> upload the pkg -- the only place it existed was on my desktop.  it had
> nothing 
> to do with SRC_URI. 

I understand the problem pretty well, thank you very much.

But I, and others besides me I'd be ready to bet, didn't act on it much
sooner because the first reaction to a similar fetch failure on
mirror://gentoo/ is "oh the mirrors haven't synced, it'll be okay by
next hour".

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: Proposal: ban mirror://gentoo/ from ebuilds

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 557 bytes --]

On Thu, 18 Aug 2011 12:09:39 +0200
Chí-Thanh Christopher Nguyễn <chithanh@gentoo.org> wrote:

> Diego Elio Pettenò schrieb:
> > Keep it in your dev space. As I said, the resources argument is only
> > available for infra to complain about, and since last I knew from
> > them was that it was not a problem right now...
> 
> I think robbat2 complained on IRC recently to devs which had their
> home directories full of some stuff.

Wasn't that more because of space supposedly wasted for no good reason?

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Re: Proposal: ban mirror://gentoo/ from ebuilds

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1377 bytes --]

On Thu, 18 Aug 2011 11:57:01 +0200
Diego Elio Pettenò <flameeyes@gentoo.org> wrote:

> Il giorno gio, 18/08/2011 alle 11.15 +0200, Thomas Sachau ha scritto:
> > 
> > The argument about dropped tarballs, once the ebuilds gets removed
> > might weight a bit more, but you
> > cannot depend on other upstream keeping their tarballs around
> > forever, so i see no requirement for
> > us preserving only specific tarballs (those created by our devs),
> > while upstream tarballs could
> > already be gone and are not preserved, once the ebuild for it is
> > gone.
> 
> _Most_ upstream projects keep tarballs available of all historic
> releases; okay maybe not _all_ projects, but most, especially those
> using services such as SourceForge, Google Code, RubyForge,
> Rubygems, ...
> 
> For _our_ projects, snapshots, or packages, let's try to apply a sane
> policy. And that starts by not arguing that we shouldn't be doing so
> because others might not be doing so themselves.

I'd say do it like that:

1) for our projects -- keep all historical releases (be a good
	upstream),

2) for snapshots -- keep them as long as they're used (considering that
	a snapshot could be recreated at any point),

3) for patches -- it would be great if we could keep them as long as
	upstream keeps the relevant tarballs.

-- 
Best regards,
Michał Górny

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 316 bytes --]

Re: [gentoo-dev] Re: Proposal: ban mirror://gentoo/ from ebuilds

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 551 bytes --]

On Thursday, August 18, 2011 11:02:55 Diego Elio Pettenò wrote:
> Il giorno gio, 18/08/2011 alle 10.54 -0400, Mike Frysinger ha scritto:
> > you're understanding of this particular bug is completely off.  i
> > forgot to
> > upload the pkg -- the only place it existed was on my desktop.  it had
> > nothing
> > to do with SRC_URI.
> 
> I understand the problem pretty well, thank you very much.

the context you snipped from my reply implied differently.  ~arch failing to 
fetch the file had nothing to do with mirror://gentoo/.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[gentoo-dev] Re: Re: Proposal: ban mirror://gentoo/ from ebuilds

[Diego Elio Pettenò]

[-- Attachment #1: Type: text/plain, Size: 376 bytes --]

Il giorno gio, 18/08/2011 alle 11.20 -0400, Mike Frysinger ha scritto:
> 
> the context you snipped from my reply implied differently.  ~arch
> failing to 
> fetch the file had nothing to do with mirror://gentoo/. 

~arch failing to fetch for more than a couple of hours, though, had to
do with it.

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: Re: Proposal: ban mirror://gentoo/ from ebuilds

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 575 bytes --]

On Thursday, August 18, 2011 11:34:49 Diego Elio Pettenò wrote:
> Il giorno gio, 18/08/2011 alle 11.20 -0400, Mike Frysinger ha scritto:
> > the context you snipped from my reply implied differently.  ~arch
> > failing to
> > fetch the file had nothing to do with mirror://gentoo/.
> 
> ~arch failing to fetch for more than a couple of hours, though, had to
> do with it.

i think you've inverted this statement from what you meant.  ~arch failing to 
fetch for more than a couple of hours was because i went to sleep.  SRC_URI 
wouldnt have woken me up.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[gentoo-dev] Re: Re: Re: Proposal: ban mirror://gentoo/ from ebuilds

[Diego Elio Pettenò]

[-- Attachment #1: Type: text/plain, Size: 699 bytes --]

Il giorno gio, 18/08/2011 alle 11.55 -0400, Mike Frysinger ha scritto:
> 
> i think you've inverted this statement from what you meant.  ~arch
> failing to 
> fetch for more than a couple of hours was because i went to sleep.
> SRC_URI 
> wouldnt have woken me up. 

Not you, but it would have masked much earlier, by me or someone else.

Seriously, can you stop making this about you and look at the big
picture?

Ulrich did provide a good reason not to simply ban mirror://gentoo/ —
now can you make up any good reason why you shouldn't be uploading this
to _your devspace_ rather than the distfiles directly?

-- 
Diego Elio Pettenò — Flameeyes
http://blog.flameeyes.eu/

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [gentoo-dev] Re: Proposal: ban mirror://gentoo/ from ebuilds

[Roy Bamford]

[-- Attachment #1: Type: text/plain, Size: 814 bytes --]

On 2011.08.18 10:59, Anthony G. Basile wrote:
[snip]
 
> Understood that infra gets to complain, but that still doesn't tell 
> me
> what the deprecation policy is.  Keep all my large patchsets
> indefinitely? Or remove them when an ebuild no longer needs them?  Or
> 1
> year after an ebuild no longer needs them?
> 
> -- 
> Anthony G. Basile, Ph. D.
> Chair of Information Technology
> D'Youville College
> Buffalo, NY 14201
> (716) 829-8197
> 
> 
Just as long as we can provide the patch sets for a period of at least 
three years, in case someone asks.  Thats a GPL requirement.

Thats not to say the files need to be online but Gentoo needs to be 
able to provide them on request.

-- 
Regards,

Roy Bamford
(Neddyseagoon) a member of
elections
gentoo-ops
forum-mods
trustees

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Re: Re: Proposal: ban mirror://gentoo/ from ebuilds

[Mike Frysinger]

[-- Attachment #1: Type: Text/Plain, Size: 1123 bytes --]

On Thursday, August 18, 2011 12:13:44 Diego Elio Pettenò wrote:
> Il giorno gio, 18/08/2011 alle 11.55 -0400, Mike Frysinger ha scritto:
> > i think you've inverted this statement from what you meant.  ~arch
> > failing to
> > fetch for more than a couple of hours was because i went to sleep.
> > SRC_URI
> > wouldnt have woken me up.
> 
> Not you, but it would have masked much earlier, by me or someone else.
> 
> Seriously, can you stop making this about you and look at the big
> picture?

i corrected your wrong statements wrt gnuconfig.  i dont know why you're 
getting upset over nothing.

as for "the big picture", that would involve an infra solution, not devspace

> Ulrich did provide a good reason not to simply ban mirror://gentoo/ —
> now can you make up any good reason why you shouldn't be uploading this
> to _your devspace_ rather than the distfiles directly?

all of my archives have been mirrored in my personal devspace.  if you're 
implying that i dont do that, you're mistaken.  i am however not a fan of 
listing dev.g.o in SRC_URI for the same reasons infra is.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [gentoo-dev] Re: Proposal: ban mirror://gentoo/ from ebuilds

[Robin H. Johnson]

On Thu, Aug 18, 2011 at 05:08:43PM +0200, Michał Górny wrote:
> On Thu, 18 Aug 2011 12:09:39 +0200
> Chí-Thanh Christopher Nguyễn <chithanh@gentoo.org> wrote:
> 
> > Diego Elio Pettenò schrieb:
> > > Keep it in your dev space. As I said, the resources argument is only
> > > available for infra to complain about, and since last I knew from
> > > them was that it was not a problem right now...
> > 
> > home directories full of some stuff.
> 
> 
Yes, to clarify, I said distfiles were fine. It was other junk that
needed cleanup, somebody had a pile of old stages that he was comparing
a tmpdir, somebody else had temp KDE distfiles, that sort of cleanup.
-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[gentoo-dev] Re: Proposal: ban mirror://gentoo/ from ebuilds

[Duncan]

Roy Bamford posted on Thu, 18 Aug 2011 17:41:15 +0100 as excerpted:

> On 2011.08.18 10:59, Anthony G. Basile wrote:
> [snip]
>  
>> Understood that infra gets to complain, but that still doesn't tell me
>> what the deprecation policy is.  Keep all my large patchsets
>> indefinitely? Or remove them when an ebuild no longer needs them?  Or 1
>> year after an ebuild no longer needs them?
>> 
> Just as long as we can provide the patch sets for a period of at least
> three years, in case someone asks.  Thats a GPL requirement.
> 
> Thats not to say the files need to be online but Gentoo needs to be able
> to provide them on request.

That's a very good point and "legal matters, and potentially it matters a 
lot!", but AFAIK it's not exactly correct (and "legal matters, and 
potentially it matters a lot!").

I am not a lawyer, etc, but as I understand it...

1) Generally, the GPL source-provision rules *ONLY* apply if you're 
shipping binaries in the first place, not for sources-only, which fulfill 
the source-provision by definition.

Since in *MOST* cases, Gentoo is shipping source, not binaries, where 
that is the case, we don't need to worry about the GPL source provision 
rules.

However, we DO ship SOME binaries, both in the packages images and in the 
stages and live-images.  Here, we DO need to worry about the GPL's source-
provision rules.  The below point applies to this case.

2) For those shipping binaries, the GPLv2 (I'm not particularly familiar 
with the GPLv3 in this regard so can't say, for it) offers two 
independent ways to fill the source-provision requirements.

2a) The provider can make sources available at the same time/place and by 
the same method as the binaries.  There's some precise definitional 
detail to same time/place and method requirements designed to ensure that 
if a user finds the binaries, they at least should be aware of the 
availability of the sources, but the point is, AS LONG AS the sources are 
made available similarly, that fulfills the sources-provision 
requirement, NO THREE YEAR RULE APPLIES!

2b) If the provider chooses NOT to make sources available at the same 
time/place and by the same method, AN ALTERNATIVE is to be able to 
provide them on demand FOR THREE YEARS AFTER THE BINARY IS NO LONGER 
PROVIDED.  *THIS* is where the three-year rule comes in.  It ONLY applies 
if the binary provider chose not to go with the same time/place/method 
alternative.  An additional requirement here is that sources must be 
available to ALL (not just customers/those-who-originally-downloaded-the-
binaries) who make the request, but again, it only applies if the same 
time/place/method alternative wasn't chosen.

It's worth noting that this discussion has come up before on this list 
and is archived.  At that time, it was noted that Gentoo was still making 
available downloads of the 1.4 and earlier install media, "for historical 
interest", and that because we were still shipping them (and because they 
contained GPLv2 content), that obligated Gentoo to providing exact 
sources (including but not limited to the original tarballs, and all 
patches, plus scripts, etc, as necessary) until three years after Gentoo 
quit making them available.

IIRC, the (first) decision then, since providing those sources was going 
to be very difficult if anyone DID ask, was to take down those "for 
historical interest" images ASAP, so at least the clock would be ticking 
on that three-year requirement.

I know I didn't follow up to ensure it was done.  I hope someone did.  
FWIW, the legal responsibility would (AFAIK) fall on the foundation, so 
it would presumably be their job to ensure this was done, and from that 
point, that previous media were taken down in a timely manner so as not 
to get Gentoo in that situation once again.

The second decision then (again, IIRC, but it's in the archives if anyone 
feels the need to look it up), and I've seen it made elsewhere as well 
(Gentoo's not the first to conclude it's by *FAR* the least hassle), was 
to have and enforce a Gentoo policy that we ALWAYS complied with the same 
time/place/method option, so we never had to worry about the three-year-
clock on the by-request option at all.

Again, I believe it's the Foundation's responsibility here, so they're 
the ones that should be ensuring this is enforced.  FWIW, while I do have 
an interest in software freedom, thus my reasonably close following of 
the discussion at the time (and the degree to which I understand the 
distinction between the two sources-provision options in the first 
place), my personal legal butt isn't in the sling here, so I've felt no 
need to personally follow-up and ensure this policy is being followed.

Thus my point in the context of this thread.  As long as Gentoo is 
continuing to follow the policy decided then, that of always ensuring 
that sources are made available at the same time and place, and by the 
same method, as the binaries we are shipping, we've done our duty and 
shouldn't need to worry about digging up sources, at least to fulfill the 
LEGAL GPLv2 requirements, at all.

As for GPLv3, while as I said I've not researched it to the same degree, 
I /believe/ that the same ideas apply, except that if anything, they've 
stressed the same time/place/method option even more, since it far better 
fits the Internet world of today, at least relative to the somewhat 
quaint implications of physical media requested and sent by snail mail, 
that the by-request-for-three-years option sort of has.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: Proposal: ban mirror://gentoo/ from ebuilds

[Roy Bamford]

[-- Attachment #1: Type: text/plain, Size: 2264 bytes --]

On 2011.08.19 07:50, Duncan wrote:
> Roy Bamford posted on Thu, 18 Aug 2011 17:41:15 +0100 as excerpted:
> 
[snip] 
> > Just as long as we can provide the patch sets for a period of at
> least
> > three years, in case someone asks.  Thats a GPL requirement.
> > 
> > Thats not to say the files need to be online but Gentoo needs to be
> able
> > to provide them on request.
> 
[snip]
> I know I didn't follow up to ensure it was done.  I hope someone did. 

> FWIW, the legal responsibility would (AFAIK) fall on the foundation,
> so it would presumably be their job to ensure this was done, and 
> from that  point, that previous media were taken down in a timely 
> manner so as not to get Gentoo in that situation once again.

[snip]
> -- 
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
> 
That's a very interesting point.

As far as I am aware, under the current organisation there is no legal 
connection between the Gentoo Foundation Inc., and the Gentoo 
distribution.

The Foundation is a legally constituted NFP, registered in New Mexico.  
Gentoo the distro, has no legal body status at all, anywhere in the 
world.

You could make an argument that the Foundation 'owns' Gentoo as the 
Gentoo produced codebase is copyright the Gentoo Foundation Inc. 
However ownership is irrelevant. It does not confer control.
Control rests with the council, who have no legal ties to the 
Foundation.

Think of it as in this example.  A member of the English aristocracy 
owns a Rolls Royce.  It is kept in a garage at chauffeurs cottage so 
the chauffeur is the registered keeper of the car.

One day, the car driven by the chauffeur is involved in a accident and
someone is killed.  Who is prosecuted for the death, the owner or the 
chauffeur?
Well, the owner was clearly not in control (he wasn't driving). 

The total separation of 'ownership' and 'control' of Gentoo is 
fundamentally broken for the same reasons.  Enough of it here, its an 
active thread on  -project just now.
  
-- 
Regards,

Roy Bamford
(Neddyseagoon) a member of
elections
gentoo-ops
forum-mods
trustees

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Proposal: ban mirror://gentoo/ from ebuilds

[Rich Freeman]

On Fri, Aug 19, 2011 at 12:52 PM, Roy Bamford <neddyseagoon@gentoo.org> wrote:
> As far as I am aware, under the current organisation there is no legal
> connection between the Gentoo Foundation Inc., and the Gentoo
> distribution.

True, but that doesn't mean that the Foundation might not be found
liable for things done using Foundation property, like servers, domain
names, and trademarks - especially if the Foundation has prior
knowledge of these actions.  I'm not sure the car example was
accurate, since in this case the distribution is being facilitated
using Foundation property and clearly at this point the Foundation has
knowledge that it is occurring.

Regardless of who can get sued, we should comply with the GPL
regardless.  To that end I agree that we should refrain from shipping
historical binaries unless we also tarball portage and the distfiles
from the same period of time.  I definitely agree with his suggestion
that we should always make both available online simultaneously to
avoid the 3-year rule.

I'm sure that if the FSF were concerned they'd ask us nicely before
suing anybody - especially since we are an FOSS effort ourselves that
tries to comply with the GPL/etc.  So, the risk is low, but that isn't
a reason not to comply.

Does anybody have a strong objection to removing historical binary
distributions of Gentoo on any Gentoo-controlled/linked servers, or
general disagreement with Duncan's proposal?  I don't really want to
get sidetracked into a debate about the boundaries between
Council/Trustees/Foundation/Distro unless there is an actual
disagreement on the matter at hand.

Rich

Re: [gentoo-dev] Re: Proposal: ban mirror://gentoo/ from ebuilds

[Roy Bamford]

[-- Attachment #1: Type: text/plain, Size: 1281 bytes --]

On 2011.08.20 23:20, Rich Freeman wrote:
> On Fri, Aug 19, 2011 at 12:52 PM, Roy Bamford
> <neddyseagoon@gentoo.org> wrote:
> > As far as I am aware, under the current organisation there is no
> legal
> > connection between the Gentoo Foundation Inc., and the Gentoo
> > distribution.
> 
[snip]
> 
> Regardless of who can get sued, we should comply with the GPL
> regardless.  To that end I agree that we should refrain from shipping
> historical binaries unless we also tarball portage and the distfiles
> from the same period of time.  I definitely agree with his suggestion
> that we should always make both available online simultaneously to
> avoid the 3-year rule.
> 
[snip]
> 
> Rich
> 
> 

Rich,

There are two separate issue here and they have separate timescales for 
being addressed.  

We need to comply with the GPL now and always, so we should do as 
Duncan suggests.  Take down old binaries now and post sources with new 
binaries. e.g. the liveDVDs.

Sorting out our internal structure is a separate issue to address in 
slower time but it needs to be done before something goes wrong because 
there just won't be time then.

-- 
Regards,

Roy Bamford
(Neddyseagoon) a member of
elections
gentoo-ops
forum-mods
trustees

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Proposal: ban mirror://gentoo/ from ebuilds

[Donnie Berkholz]

[-- Attachment #1: Type: text/plain, Size: 524 bytes --]

On 17:41 Thu 18 Aug     , Roy Bamford wrote:
> Just as long as we can provide the patch sets for a period of at least 
> three years, in case someone asks.  Thats a GPL requirement.

Just to be clear, it's only a requirement if you're going with the 
"written offer to provide source" clause rather than the "providing 
source at the same time as binary" clause (i.e., 3b rather than 3a in 
GPL-2).

-- 
Thanks,
Donnie

Donnie Berkholz
Council Member / Sr. Developer
Gentoo Linux
Blog: http://dberkholz.com

[-- Attachment #2: Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-dev] Re: Proposal: ban mirror://gentoo/ from ebuilds

[Piotr Jaroszyński]

On 18 August 2011 11:53, Diego Elio Pettenò <flameeyes@gentoo.org> wrote:
> Il giorno gio, 18/08/2011 alle 05.46 -0400, Anthony G. Basile ha
> scritto:
>>
>> What alternative are you proposing to mirror://gentoo/ if upstream
>> doesn't provide a tarball, eg with large patchsets the maintainer
>> constructs?  Anticipating your answer might be "keep them in your dev
>> space", then what would be the deprecation policy for distfiles that
>> are
>> no longer used by ebuilds?  If foresee a tension between keeping all
>> the
>> distfiles vs disk space, although Patrick's point of cheap hardware is
>> well taken.
>
> Keep it in your dev space. As I said, the resources argument is only
> available for infra to complain about, and since last I knew from them
> was that it was not a problem right now...

And what if the dev retires/is kicked out?

-- 
Best Regards
Piotr Jaroszyński