[gentoo-user] Ebuild: How to deal with external repositories properly (best practise)?

[gentoo-user] Ebuild: How to deal with external repositories properly (best practise)?

[Ramon Fischer]

Hello list,

I have set up my first ebuild repository[1] with one package 
"x11-misc/drm_master_util" to run "X" without root privileges. I am 
still making it to work[2]. The ebuild file accesses an external git 
repository[3].

The thing I am concerned about, is, that I am pulling something from an 
external source, which I am installing on my system and giving it root 
privileges[4].

The only best practise I can think of, is, to fork the external 
repository, linking the ebuild to my fork and updating it on demand, so 
I have full control over it.

Would this be the way to do it?

[1] https://codeberg.org/keks24/gentoo-overlay
[2] https://wiki.gentoo.org/wiki/User:Keks24/drafts/Non_root_Xorg
[3] https://github.com/gch1p/drm_master_util.git
[4] https://wiki.gentoo.org/wiki/Non_root_Xorg#Handling_of_DRM_ioctls

Re: [gentoo-user] Ebuild: How to deal with external repositories properly (best practise)?

[tastytea]

On 2020-07-28 06:47+0200 Ramon Fischer <Ramon_Fischer@hotmail.de> wrote:

> […]
> The thing I am concerned about, is, that I am pulling something from
> an external source, which I am installing on my system and giving it
> root privileges[4].
> 
> The only best practise I can think of, is, to fork the external 
> repository, linking the ebuild to my fork and updating it on demand,
> so I have full control over it.
> 
> Would this be the way to do it?

You can mask all packages from a repository in
/etc/portage/package.mask/ with
    */*::repo-name
and unmask the packages you want in /etc/portage/package.unmask/ with
    x11-misc/drm_master_util::repo-name
or just the version you want with
    =x11-misc/drm_master_util-9999::repo-name
.

The maintainer of the repo could still replace the ebuild with a
malware installer.

Re: [gentoo-user] Ebuild: How to deal with external repositories properly (best practise)?

[Ramon Fischer]

Hello tastytea,

I am aware of this "workaround", thank you. :)

I guess, I was not precise enough:

The ebuild "drm_master_util-9999"[1] is hosted on my repository, but the 
ebuild file itself pulls in an external repository[2].

My question is: Is it a best practise to fork the external 
repository[2], to link my fork with "drm_master_util-9999"[1], so I have 
full control about updating the fork. Just to check, that the external 
source is not doing shenanigans?

-Ramon

[1] 
https://codeberg.org/keks24/gentoo-overlay/src/branch/master/x11-misc/drm_master_util/drm_master_util-9999.ebuild#L27
[2] https://github.com/gch1p/drm_master_util.git

On 28/07/2020 10:53, tastytea wrote:
> On 2020-07-28 06:47+0200 Ramon Fischer <Ramon_Fischer@hotmail.de> wrote:
>
>> […]
>> The thing I am concerned about, is, that I am pulling something from
>> an external source, which I am installing on my system and giving it
>> root privileges[4].
>>
>> The only best practise I can think of, is, to fork the external
>> repository, linking the ebuild to my fork and updating it on demand,
>> so I have full control over it.
>>
>> Would this be the way to do it?
> You can mask all packages from a repository in
> /etc/portage/package.mask/ with
>      */*::repo-name
> and unmask the packages you want in /etc/portage/package.unmask/ with
>      x11-misc/drm_master_util::repo-name
> or just the version you want with
>      =x11-misc/drm_master_util-9999::repo-name
> .
>
> The maintainer of the repo could still replace the ebuild with a
> malware installer.
>

Re: [gentoo-user] Ebuild: How to deal with external repositories properly (best practise)?

[tastytea]

On 2020-07-28 12:02+0200 Ramon Fischer <Ramon_Fischer@hotmail.de> wrote:

> Hello tastytea,
> 
> I am aware of this "workaround", thank you. :)
> 
> I guess, I was not precise enough:
> 
> The ebuild "drm_master_util-9999"[1] is hosted on my repository, but
> the ebuild file itself pulls in an external repository[2].

Sorry, I misread your first email.

> My question is: Is it a best practise to fork the external 
> repository[2], to link my fork with "drm_master_util-9999"[1], so I
> have full control about updating the fork. Just to check, that the
> external source is not doing shenanigans?

I would use either EGIT_COMMIT from git-r3.eclass¹ or download a
snapshot via SRC_URI².

¹ <https://devmanual.gentoo.org/eclass-reference/git-r3.eclass/index.html#lbAF>
² <https://github.com/gch1p/drm_master_util/archive/<COMMIT-HASH>.tar.gz>

Re: [gentoo-user] Ebuild: How to deal with external repositories properly (best practise)?

[Ramon Fischer]

Oh yeah, as Neil was pointing that out.

I will give it a try and report soon.

-Ramon

On 28/07/2020 14:52, tastytea wrote:
> On 2020-07-28 12:02+0200 Ramon Fischer <Ramon_Fischer@hotmail.de> wrote:
>
>> Hello tastytea,
>>
>> I am aware of this "workaround", thank you. :)
>>
>> I guess, I was not precise enough:
>>
>> The ebuild "drm_master_util-9999"[1] is hosted on my repository, but
>> the ebuild file itself pulls in an external repository[2].
> Sorry, I misread your first email.
>
>> My question is: Is it a best practise to fork the external
>> repository[2], to link my fork with "drm_master_util-9999"[1], so I
>> have full control about updating the fork. Just to check, that the
>> external source is not doing shenanigans?
> I would use either EGIT_COMMIT from git-r3.eclass¹ or download a
> snapshot via SRC_URI².
>
> ¹ <https://devmanual.gentoo.org/eclass-reference/git-r3.eclass/index.html#lbAF>
> ² <https://github.com/gch1p/drm_master_util/archive/<COMMIT-HASH>.tar.gz>
>

[SOLVED] Re: [gentoo-user] Ebuild: How to deal with external repositories properly (best practise)?

[Ramon Fischer]

I decided to use "EGIT_COMMIT" to let the ebuild pulling a certain commit.

Using the archive tarball is indeed interesting!

Thank you for your help!

-Ramon

On 28/07/2020 15:32, Ramon Fischer wrote:
> Oh yeah, as Neil was pointing that out.
>
> I will give it a try and report soon.
>
> -Ramon
>
> On 28/07/2020 14:52, tastytea wrote:
>> On 2020-07-28 12:02+0200 Ramon Fischer <Ramon_Fischer@hotmail.de> wrote:
>>
>>> Hello tastytea,
>>>
>>> I am aware of this "workaround", thank you. :)
>>>
>>> I guess, I was not precise enough:
>>>
>>> The ebuild "drm_master_util-9999"[1] is hosted on my repository, but
>>> the ebuild file itself pulls in an external repository[2].
>> Sorry, I misread your first email.
>>
>>> My question is: Is it a best practise to fork the external
>>> repository[2], to link my fork with "drm_master_util-9999"[1], so I
>>> have full control about updating the fork. Just to check, that the
>>> external source is not doing shenanigans?
>> I would use either EGIT_COMMIT from git-r3.eclass¹ or download a
>> snapshot via SRC_URI².
>>
>> ¹ 
>> <https://devmanual.gentoo.org/eclass-reference/git-r3.eclass/index.html#lbAF>
>> ² <https://github.com/gch1p/drm_master_util/archive/<COMMIT-HASH>.tar.gz>
>>
>
>

Re: [SOLVED] Re: [gentoo-user] Ebuild: How to deal with external repositories properly (best practise)?

[Alexey Mishustin]

вс, 2 авг. 2020 г. в 13:52, Ramon Fischer <Ramon_Fischer@hotmail.de>:
>
> I decided to use "EGIT_COMMIT" to let the ebuild pulling a certain commit.

And even that would not give the sense of security...

Just read in gentoo-dev [1]:
...unannounced serverside change by GitHub, which broke download of
tarballs by git-tree-hash, e.g. previously https://
api.github.com/repos/JuliaLang/MbedTLS.jl/tarball/
2d94286a9c2f52c63a16146bb86fd6cdfbf677c6 would give the tarball for
that tree- hash, while it now gives the tarball for master instead.

[1] - https://archives.gentoo.org/gentoo-dev/message/41d8c5457df392ed0309153651db5b3c

-- 
Best regards,
Alex

Re: [SOLVED] Re: [gentoo-user] Ebuild: How to deal with external repositories properly (best practise)?

[Rich Freeman]

On Tue, Aug 4, 2020 at 6:57 PM Alexey Mishustin <shumkar@shumkar.ru> wrote:
>
> вс, 2 авг. 2020 г. в 13:52, Ramon Fischer <Ramon_Fischer@hotmail.de>:
> >
> > I decided to use "EGIT_COMMIT" to let the ebuild pulling a certain commit.
>
> And even that would not give the sense of security...
>
> Just read in gentoo-dev [1]:
> ...unannounced serverside change by GitHub, which broke download of
> tarballs by git-tree-hash, e.g. previously https://
> api.github.com/repos/JuliaLang/MbedTLS.jl/tarball/
> 2d94286a9c2f52c63a16146bb86fd6cdfbf677c6 would give the tarball for
> that tree- hash, while it now gives the tarball for master instead.
>

I'm pretty sure EGIT_COMMIT will fetch by commit ID using git, not
download a hash-labeled tarball, so I don't think this issue would
impact you if that is how you're fetching things.

If you did use a hash tarball with SRC_URI and a conventional
download, then emerge would still refuse to use the tarball if it
failed the manifest hash check, so it wouldn't be installing anything
you didn't want.

Generally this isn't going to immediately break anything used by the
Gentoo repo since 99% of this stuff will be mirrored, and the mirrors
check hashes too.  So, when github breaks the download link the
mirrors will preserve their existing tarballs and refuse to replace
them with new ones that don't have a matching hash (I'm talking about
the actual hash of the file using multiple algorithms, not the hash in
the filename).  When you fetch from a mirror you'll still get the
correct version of the file.  If for some reason you can't reach any
mirrors then you would download the broken link from github and then
emerge would reject the file due to hash mismatch.

Still, unless github fixes this we'll probably have to fix a bunch of
links in the repositories - at least any based on hashes.  I'm not
sure if this impacts tags.  The SRC_URIs are still invalid and we
don't want to maintain that state as new mirrors won't be able to
retrieve the file, and we generally want a valid SRC_URI for
everything.  Devs can always just upload the tarball to any random
webserver and change the URI to point to it.  My guess though is that
everybody will want to give this a few days to see if github fixes
their links.

Really this could happen with any web hosting service - github is just
a really prominent one.  Back in the day if sourceforge suddenly went
down a whole bunch of SRC_URIs would have broken too.

-- 
Rich

Re: [SOLVED] Re: [gentoo-user] Ebuild: How to deal with external repositories properly (best practise)?

[tastytea]

On 2020-08-04 19:36-0400 Rich Freeman <rich0@gentoo.org> wrote:

> On Tue, Aug 4, 2020 at 6:57 PM Alexey Mishustin <shumkar@shumkar.ru>
> wrote:
> >
> > вс, 2 авг. 2020 г. в 13:52, Ramon Fischer
> > <Ramon_Fischer@hotmail.de>:  
> > >
> > > I decided to use "EGIT_COMMIT" to let the ebuild pulling a
> > > certain commit.  
> >
> > And even that would not give the sense of security...
> >
> > Just read in gentoo-dev [1]:
> > ...unannounced serverside change by GitHub, which broke download of
> > tarballs by git-tree-hash, e.g. previously https://
> > api.github.com/repos/JuliaLang/MbedTLS.jl/tarball/
> > 2d94286a9c2f52c63a16146bb86fd6cdfbf677c6 would give the tarball for
> > that tree- hash, while it now gives the tarball for master instead.
> >  

This seems to affect only api.github.com, packages in ::guru use
https://github.com/<REPO>/archive/<COMMIT>.tar.gz instead, which is not
affected (just checked with net-wireless/rtl8192eu-0_pre20200123).

> I'm pretty sure EGIT_COMMIT will fetch by commit ID using git, not
> download a hash-labeled tarball, so I don't think this issue would
> impact you if that is how you're fetching things.

Correct.

> […]
> Still, unless github fixes this we'll probably have to fix a bunch of
> links in the repositories - at least any based on hashes.  I'm not
> sure if this impacts tags.  The SRC_URIs are still invalid and we
> don't want to maintain that state as new mirrors won't be able to
> retrieve the file, and we generally want a valid SRC_URI for
> everything.  Devs can always just upload the tarball to any random
> webserver and change the URI to point to it.  My guess though is that
> everybody will want to give this a few days to see if github fixes
> their links.

A quick grep indicated that the only packages in ::gentoo using
api\.github\.com.*tarball are net-analyzer/tcpflow, dev-python/mypy,
dev-lang/julia and app-forensics/dfxml.

> Really this could happen with any web hosting service - github is just
> a really prominent one.  Back in the day if sourceforge suddenly went
> down a whole bunch of SRC_URIs would have broken too.
> 

Re: [SOLVED] Re: [gentoo-user] Ebuild: How to deal with external repositories properly (best practise)?

[Rich Freeman]

On Tue, Aug 4, 2020 at 7:51 PM tastytea <tastytea+gentoo@tastytea.de> wrote:
>
> This seems to affect only api.github.com, packages in ::guru use
> https://github.com/<REPO>/archive/<COMMIT>.tar.gz instead, which is not
> affected (just checked with net-wireless/rtl8192eu-0_pre20200123).

Ah, didn't notice that.  This is the more common approach for Gentoo
packages, if they use hashes at all.  Usually tags are preferred.

And if upstream actually has an official source tarball that is what
gets used.  The only reason anybody in Gentoo uses github links at all
is either because upstream uses it officially, or upstream doesn't
even bother to release source tarballs.

-- 
Rich

Re: [gentoo-user] Ebuild: How to deal with external repositories properly (best practise)?

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 815 bytes --]

On Tue, 28 Jul 2020 06:47:50 +0200, Ramon Fischer wrote:

> I have set up my first ebuild repository[1] with one package 
> "x11-misc/drm_master_util" to run "X" without root privileges. I am 
> still making it to work[2]. The ebuild file accesses an external git 
> repository[3].
> 
> The thing I am concerned about, is, that I am pulling something from an 
> external source, which I am installing on my system and giving it root 
> privileges[4].

Isn't that the case with any software you install? If your concern is
that you are pulling from a git repository, so the code could change at
any minute, you can use the git eclass to pull a specified branch or
commit, so you know what you are getting.


-- 
Neil Bothwick

Anything is possible if you don't know what
you are talking about.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]