[gentoo-dev] Pushing to distfiles?

[gentoo-dev] Pushing to distfiles?

[Joshua Kinard]

Forgive me for being a dunce, but what is the current procedure to push
files to distfiles for distribution to the mirrors?  The devmanual is blank
on this topic, and GLEP75 just talks about the motivations behind the change
away from the flat directory format.  I am not easily finding anything that
details how to get new distfiles onto the mirrors.

Thanks,

-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
rsa6144/5C63F4E3F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943

"The past tempts us, the present confuses us, the future frightens us.  And
our lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

Re: [gentoo-dev] Pushing to distfiles?

[Rich Freeman]

On Wed, Nov 11, 2020 at 7:23 PM Joshua Kinard <kumba@gentoo.org> wrote:
>
> Forgive me for being a dunce, but what is the current procedure to push
> files to distfiles for distribution to the mirrors?  The devmanual is blank
> on this topic, and GLEP75 just talks about the motivations behind the change
> away from the flat directory format.  I am not easily finding anything that
> details how to get new distfiles onto the mirrors.
>

I thought that the whole mirror:// thing was discouraged.  For
anything else you just stick SRC_URI in the ebuild and the mirrors
should fetch it when they see it in the repo.

I just host stuff like that on my dev webspace, or better yet on
github or something else that will auto-tarball stuff.

-- 
Rich

Re: [gentoo-dev] Pushing to distfiles?

[Georgy Yakovlev]

[-- Attachment #1: Type: text/plain, Size: 1016 bytes --]

On 11.11.2020 19:23, Joshua Kinard wrote:
> Forgive me for being a dunce, but what is the current procedure to push
> files to distfiles for distribution to the mirrors?  The devmanual is blank
> on this topic, and GLEP75 just talks about the motivations behind the change
> away from the flat directory format.  I am not easily finding anything that
> details how to get new distfiles onto the mirrors.
Hi,

Just put to devspace.
for example I keep distfiles in ~/public_html/distfiles

This is preferred of having file on mirrors only.
And it will be automatically mirrored from devspace, so no extra steps
required.
> 
> Thanks,
> 
> -- 
> Joshua Kinard
> Gentoo/MIPS
> kumba@gentoo.org
> rsa6144/5C63F4E3F5C6C943 2015-04-27
> 177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943
> 
> "The past tempts us, the present confuses us, the future frightens us.  And
> our lives slip away, moment by moment, lost in that vast, terrible in-between."
> 
> --Emperor Turhan, Centauri Republic
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 902 bytes --]

Re: [gentoo-dev] Pushing to distfiles?

[Joshua Kinard]

On 11/11/2020 19:38, Rich Freeman wrote:
> On Wed, Nov 11, 2020 at 7:23 PM Joshua Kinard <kumba@gentoo.org> wrote:
>>
>> Forgive me for being a dunce, but what is the current procedure to push
>> files to distfiles for distribution to the mirrors?  The devmanual is blank
>> on this topic, and GLEP75 just talks about the motivations behind the change
>> away from the flat directory format.  I am not easily finding anything that
>> details how to get new distfiles onto the mirrors.
>>
> 
> I thought that the whole mirror:// thing was discouraged.  For
> anything else you just stick SRC_URI in the ebuild and the mirrors
> should fetch it when they see it in the repo.
> 
> I just host stuff like that on my dev webspace, or better yet on
> github or something else that will auto-tarball stuff.
> 

And I found it.  It is in the devmanual after all, just under different
wording.  I was searching for "distfiles", when I should have been looking
at the "Mirrors" section.  Looks like we do have some behind-the-scenes
magic that copies SRC_URI files from devspace straight to the mirrors.  Just
way too used to the /space/distfiles-local thing.

RESOLVED::PEBKAC

-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
rsa6144/5C63F4E3F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943

"The past tempts us, the present confuses us, the future frightens us.  And
our lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

Re: [gentoo-dev] Pushing to distfiles?

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1222 bytes --]

On Wed, 11 Nov 2020 19:38:35 -0500
Rich Freeman <rich0@gentoo.org> wrote:

> I thought that the whole mirror:// thing was discouraged.  For
> anything else you just stick SRC_URI in the ebuild and the mirrors
> should fetch it when they see it in the repo.
> 
> I just host stuff like that on my dev webspace, or better yet on
> github or something else that will auto-tarball stuff.

In my last foray into building things from historical portage
snapshots, the use of "push stuff directly to mirror:// and then only
have mirror:// in SRC_URI" pattern was *the* leading cause of "oh noes,
patches and distfiles no longer exist".

Because naturally, once there is no longer an ebuild that requires the
distfile, it can be reaped from the mirror, and this leaves no fallback
fetch location.

I found myself reviling this so much I feel it apt that this pattern
should be more than discouraged, it should be banned outright and all
ebuilds that *only* have mirror://gentoo/ as a source for any file
should be shot repeatedly till it doesn't do that anymore.

The resilience of Gentoo-authored distfiles should be *as good* as
random 3rd parties we currently source from, not *exponentially worse*


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Pushing to distfiles?

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 738 bytes --]

On Wed, 11 Nov 2020 19:38:35 -0500
Rich Freeman <rich0@gentoo.org> wrote:

> I just host stuff like that on my dev webspace, or better yet on
> github or something else that will auto-tarball stuff.

Oh, yeah, and don't rely on github auto-tarball stuff.

History has demonstrated github sometimes "forgets" their cached copies
of those tarballs, and then later when requested, it will regenerate
them fresh ... but with different SHAsums.

If you're gonna use github for tarballs, roll that tarball yourself,
and attach it to a "release", manually and explicitly, and then use the
URL to the release asset.

Only then can you be sure: 

a) Of what the tarball actually contains
b) Of what the tarballs SHAsum will be

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Pushing to distfiles?

[James Le Cuirot]

[-- Attachment #1: Type: text/plain, Size: 1235 bytes --]

On Sun, 15 Nov 2020 11:03:29 +1300
Kent Fredric <kentnl@gentoo.org> wrote:

> On Wed, 11 Nov 2020 19:38:35 -0500
> Rich Freeman <rich0@gentoo.org> wrote:
> 
> > I just host stuff like that on my dev webspace, or better yet on
> > github or something else that will auto-tarball stuff.  
> 
> Oh, yeah, and don't rely on github auto-tarball stuff.
> 
> History has demonstrated github sometimes "forgets" their cached copies
> of those tarballs, and then later when requested, it will regenerate
> them fresh ... but with different SHAsums.
> 
> If you're gonna use github for tarballs, roll that tarball yourself,
> and attach it to a "release", manually and explicitly, and then use the
> URL to the release asset.
> 
> Only then can you be sure: 
> 
> a) Of what the tarball actually contains
> b) Of what the tarballs SHAsum will be

I'm not claiming this has never actually happened but I use these
GitHub tarballs *a lot* and I don't recall ever seeing it. Does anyone
know for sure that it's happened in, say, the last 3 years?  It's a lot
of extra work for a problem that may no longer exist or is so rare that
it's just not worth the effort.

-- 
James Le Cuirot (chewi)
Gentoo Linux Developer

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] Pushing to distfiles?

[Thomas Deutschmann]

On 2020-11-14 23:12, James Le Cuirot wrote:
> I'm not claiming this has never actually happened but I use these
> GitHub tarballs*a lot*  and I don't recall ever seeing it. Does anyone
> know for sure that it's happened in, say, the last 3 years?  It's a lot
> of extra work for a problem that may no longer exist or is so rare that
> it's just not worth the effort.

I am aware of three incidents:

In 2011 but I cannot find details at the moment.

In 2013, a bugfix in git 
(https://github.com/git/git/commit/22f0dcd9634a818a0c83f23ea1a48f2d620c0546) 
caused such a change.

In second half of 2017, GitHub was rolling out a GZIP update across 
their fleet which caused such a change. It mostly hit rarely downloaded 
packages which were cleaned from CDN so tarball had to be re-generated.

You can use GitHub Actions for example to automate this or include it in 
existing release workflows. But yes, you have to get upstream's 
attention to implement this.

And it's not just GitHub, don't forget about GitLab and those 
self-hosted GitLab instances which often don't support to upload 
arbitrary assets...


-- 
Regards,
Thomas Deutschmann / Gentoo Linux Developer
C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5