[gentoo-dev] rfc: go 1.13 and go modules

[gentoo-dev] rfc: go 1.13 and go modules

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1790 bytes --]

All,

First off, everything I'm saying here is definitely open to discussion;
they are issues I have found with go 1.13.

with Go 1.13, go modules are the default way to build software written
in Go. If upstream has a go.mod file in their distribution, this
version of Go will assume that they are using modules unless modules are
disabled explicitly by setting the GO111MODULE environment variable to
off[1], so if your upstream does not have go.mod, it will work as
before.

This is not turned off globally because upstream is encouraging projects
to move to using modules [3] [4].

So far, here is what I've found if an upstream uses go modules.

dev-vcs/hub vendors the modules into their upstream repository using
"go mod vendor" and keeps go.mod and go.sum up to date. The build commands in
their makefiles use the "-mod vendor" switch as well. In this case,
the ebuild author doesn't have to do anything about the modules.

If the modules are not vendored (there is no vendor directory in the
upstream repository), the only option I've tried so far is to run "go
mod vendor" in the root of the upstream repository then tar up the
vendor directory and mirror it on our infrastructure. In that scenario,
you need to add "-mod vendor" to any "go build" commands upstream uses
and have a src_prepare function that moves the vendor directory into the
appropriate place. See www-apps/trickster (not trickster-bin). This also
seems to avoid the module mirror database.

There is another option I want to try which is adding "go mod vendor" to
src_unpack for go packages.

Thoughts?

Thanks,

William

[1] https://golang.org/cmd/go/#hdr-Module_support
[2] https://golang.org/doc/go1.13#modules
[3] https://blog.golang.org/migrating-to-go-modules
[4] https://blog.golang.org/using-go-modules

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] rfc: go 1.13 and go modules

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 518 bytes --]

On 9/9/19 10:34 AM, William Hubbs wrote:

> There is another option I want to try which is adding "go mod vendor" to
> src_unpack for go packages.

If you do that then it will violate FEATURES=network-sandbox (default)
unless you also do PROPERTIES+=" live".

We could add a separate PROPERTIES value for this, I've suggested it
before but both Michał Górny and Ulrich Mueller were against it:

https://archives.gentoo.org/gentoo-portage-dev/message/6d696661b29b6e2b8c82061e89e4718f
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] rfc: go 1.13 and go modules

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 789 bytes --]

On Mon, Sep 09, 2019 at 11:19:02AM -0700, Zac Medico wrote:
> On 9/9/19 10:34 AM, William Hubbs wrote:
> 
> > There is another option I want to try which is adding "go mod vendor" to
> > src_unpack for go packages.
> 
> If you do that then it will violate FEATURES=network-sandbox (default)
> unless you also do PROPERTIES+=" live".
> 
> We could add a separate PROPERTIES value for this, I've suggested it
> before but both Michał Górny and Ulrich Mueller were against it:
> 
> https://archives.gentoo.org/gentoo-portage-dev/message/6d696661b29b6e2b8c82061e89e4718f

If checksum verification is the concern, Go 1.13 also has this:

https://blog.golang.org/module-mirror-launch

Thoughts? Does this make the case for a property for these kinds of
ebuilds?

William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] rfc: go 1.13 and go modules

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1272 bytes --]

On Mon, 2019-09-09 at 13:41 -0500, William Hubbs wrote:
> On Mon, Sep 09, 2019 at 11:19:02AM -0700, Zac Medico wrote:
> > On 9/9/19 10:34 AM, William Hubbs wrote:
> > 
> > > There is another option I want to try which is adding "go mod vendor" to
> > > src_unpack for go packages.
> > 
> > If you do that then it will violate FEATURES=network-sandbox (default)
> > unless you also do PROPERTIES+=" live".
> > 
> > We could add a separate PROPERTIES value for this, I've suggested it
> > before but both Michał Górny and Ulrich Mueller were against it:
> > 
> > https://archives.gentoo.org/gentoo-portage-dev/message/6d696661b29b6e2b8c82061e89e4718f
> 
> If checksum verification is the concern, Go 1.13 also has this:
> 
> https://blog.golang.org/module-mirror-launch
> 
> Thoughts? Does this make the case for a property for these kinds of
> ebuilds?
> 

Checksum verification is only one of the problems.  The other one is
that it won't work if you don't have permanent Internet access or are
using strong firewalling.

Ebuilds are using a single fetching mechanisms so that people can adjust
it as necessary for their setup.  It's not fine to try to silently work
around that and access Internet.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] rfc: go 1.13 and go modules

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1775 bytes --]

On Mon, Sep 09, 2019 at 09:00:31PM +0200, Michał Górny wrote:
> On Mon, 2019-09-09 at 13:41 -0500, William Hubbs wrote:
> > On Mon, Sep 09, 2019 at 11:19:02AM -0700, Zac Medico wrote:
> > > On 9/9/19 10:34 AM, William Hubbs wrote:
> > > 
> > > > There is another option I want to try which is adding "go mod vendor" to
> > > > src_unpack for go packages.
> > > 
> > > If you do that then it will violate FEATURES=network-sandbox (default)
> > > unless you also do PROPERTIES+=" live".
> > > 
> > > We could add a separate PROPERTIES value for this, I've suggested it
> > > before but both Michał Górny and Ulrich Mueller were against it:
> > > 
> > > https://archives.gentoo.org/gentoo-portage-dev/message/6d696661b29b6e2b8c82061e89e4718f
> > 
> > If checksum verification is the concern, Go 1.13 also has this:
> > 
> > https://blog.golang.org/module-mirror-launch
> > 
> > Thoughts? Does this make the case for a property for these kinds of
> > ebuilds?
> > 
> 
> Checksum verification is only one of the problems.  The other one is
> that it won't work if you don't have permanent Internet access or are
> using strong firewalling.
>
> Ebuilds are using a single fetching mechanisms so that people can adjust
> it as necessary for their setup.  It's not fine to try to silently work
> around that and access Internet.

This argument is pretty irrelivent, because if you are using that strong
firewalling, you will block people from compiling go sourcewith external
dependencies whether or not they are using ebuilds.
Also, you will block emerge --sync since it uses rsync and that protocol
is more likely to be blocked than http.

This also applies to your comment about not having a permanent internet
connection.

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] rfc: go 1.13 and go modules

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 2688 bytes --]

On Mon, Sep 9, 2019 at 3:10 PM William Hubbs <williamh@gentoo.org> wrote:

> On Mon, Sep 09, 2019 at 09:00:31PM +0200, Michał Górny wrote:
> > On Mon, 2019-09-09 at 13:41 -0500, William Hubbs wrote:
> > > On Mon, Sep 09, 2019 at 11:19:02AM -0700, Zac Medico wrote:
> > > > On 9/9/19 10:34 AM, William Hubbs wrote:
> > > >
> > > > > There is another option I want to try which is adding "go mod
> vendor" to
> > > > > src_unpack for go packages.
> > > >
> > > > If you do that then it will violate FEATURES=network-sandbox
> (default)
> > > > unless you also do PROPERTIES+=" live".
> > > >
> > > > We could add a separate PROPERTIES value for this, I've suggested it
> > > > before but both Michał Górny and Ulrich Mueller were against it:
> > > >
> > > >
> https://archives.gentoo.org/gentoo-portage-dev/message/6d696661b29b6e2b8c82061e89e4718f
> > >
> > > If checksum verification is the concern, Go 1.13 also has this:
> > >
> > > https://blog.golang.org/module-mirror-launch
> > >
> > > Thoughts? Does this make the case for a property for these kinds of
> > > ebuilds?
> > >
> >
> > Checksum verification is only one of the problems.  The other one is
> > that it won't work if you don't have permanent Internet access or are
> > using strong firewalling.
> >
> > Ebuilds are using a single fetching mechanisms so that people can adjust
> > it as necessary for their setup.  It's not fine to try to silently work
> > around that and access Internet.
>
> This argument is pretty irrelivent, because if you are using that strong
> firewalling, you will block people from compiling go sourcewith external
> dependencies whether or not they are using ebuilds.
> Also, you will block emerge --sync since it uses rsync and that protocol
> is more likely to be blocked than http.
>
> This also applies to your comment about not having a permanent internet
> connection.
>

There are multiple ways to sync (e.g. websync, snapshots, rsync, git..) but
the idea is that you can do things like:

emerge --sync # download new tree
emerge --fetchonly # download my stuff

# Disconnect my computer
emerge foo # use downloaded stuff

You will note that there is no pkg_fetch function; the sources are all in
SRC_URI and the package-manager fetches all of them. So golang modules
would need to follow this in order to emulate this functionality or be
marked LIVE and not be able to be installed in this scheme.

I'm not quite grasping why we cannot parse the dependency syntax and build
the SRC_URIs from the dependency listing? Because its too hard? Because we
can't do it in bash? Some other reason?

-A


>
> William
>
>

[-- Attachment #2: Type: text/html, Size: 3832 bytes --]

Re: [gentoo-dev] rfc: go 1.13 and go modules

[Michael Orlitzky]

On 9/9/19 2:19 PM, Zac Medico wrote:
> On 9/9/19 10:34 AM, William Hubbs wrote:
> 
>> There is another option I want to try which is adding "go mod vendor" to
>> src_unpack for go packages.
> 
> If you do that then it will violate FEATURES=network-sandbox (default)
> unless you also do PROPERTIES+=" live".
> 
> We could add a separate PROPERTIES value for this, I've suggested it
> before but both Michał Górny and Ulrich Mueller were against it:
> 

This is the precisely what FEATURES=network-sandbox exists to prevent.

Bundling dependencies, hitting the network, shelling out to a
language-specific package manager, depending on specific upstream
commits, and sidestepping existing QA mechanisms to make this all
possible... it's bad idea bingo.

Re: [gentoo-dev] rfc: go 1.13 and go modules

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1050 bytes --]

On Mon, Sep 09, 2019 at 02:54:51PM -0400, Michael Orlitzky wrote:
> On 9/9/19 2:19 PM, Zac Medico wrote:
> > On 9/9/19 10:34 AM, William Hubbs wrote:
> > 
> >> There is another option I want to try which is adding "go mod vendor" to
> >> src_unpack for go packages.
> > 
> > If you do that then it will violate FEATURES=network-sandbox (default)
> > unless you also do PROPERTIES+=" live".
> > 
> > We could add a separate PROPERTIES value for this, I've suggested it
> > before but both Michał Górny and Ulrich Mueller were against it:
> > 
> 
> This is the precisely what FEATURES=network-sandbox exists to prevent.
> 
> Bundling dependencies, hitting the network, shelling out to a
> language-specific package manager, depending on specific upstream
> commits, and sidestepping existing QA mechanisms to make this all
> possible... it's bad idea bingo.

Michael,

Please read all of the info on go modules. In particular, I don't think
you read the info about the mirror database and checksum verification.

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] rfc: go 1.13 and go modules

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 429 bytes --]

>>>>> On Mon, 09 Sep 2019, William Hubbs wrote:

> There is another option I want to try which is adding "go mod vendor"
> to src_unpack for go packages.

That's pretty much a no-go for a regular Gentoo package, because it
would circumvent checksum verification, and would require network access
at build time.

In other words, it would make the ebuild a "live" ebuild, which means
that it could never have any keywords.

Ulrich

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [gentoo-dev] rfc: go 1.13 and go modules

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 473 bytes --]

On Mon, 9 Sep 2019 12:34:18 -0500
William Hubbs <williamh@gentoo.org> wrote:

> There is another option I want to try which is adding "go mod vendor" to
> src_unpack for go packages.

Is it infeasible to write a tool that you execute as a maintainer, that simulates
what "go mod vendor" would do, but instead emits a list of entries for SRC_URI,
and then have an eclass or something construct the vendor dir from those?

That's what is available for rust stuff.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] rfc: go 1.13 and go modules

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 907 bytes --]

On Tue, Sep 10, 2019 at 08:35:17AM +1200, Kent Fredric wrote:
> On Mon, 9 Sep 2019 12:34:18 -0500
> William Hubbs <williamh@gentoo.org> wrote:
> 
> > There is another option I want to try which is adding "go mod vendor" to
> > src_unpack for go packages.
> 
> Is it infeasible to write a tool that you execute as a maintainer, that simulates
> what "go mod vendor" would do, but instead emits a list of entries for SRC_URI,
> and then have an eclass or something construct the vendor dir from those?
> 
> That's what is available for rust stuff.

I'm not sure how feasible something like that is.

$ go list -m all

will list the dependencies of a module, but that doesn't look like it
can be translated into src_uri format.

You would basically have to parse go.mod exactly the way upstream does
it and come up with a way to download the correct versions of the
source.

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] rfc: go 1.13 and go modules

[Georgy Yakovlev]

[-- Attachment #1: Type: text/plain, Size: 1495 bytes --]

On Monday, September 9, 2019 2:46:16 PM PDT William Hubbs wrote:
> On Tue, Sep 10, 2019 at 08:35:17AM +1200, Kent Fredric wrote:
> > On Mon, 9 Sep 2019 12:34:18 -0500
> > 
> > William Hubbs <williamh@gentoo.org> wrote:
> > > There is another option I want to try which is adding "go mod vendor" to
> > > src_unpack for go packages.
> > 
> > Is it infeasible to write a tool that you execute as a maintainer, that
> > simulates what "go mod vendor" would do, but instead emits a list of
> > entries for SRC_URI, and then have an eclass or something construct the
> > vendor dir from those?
> > 
> > That's what is available for rust stuff.
> 
> I'm not sure how feasible something like that is.
> 
> $ go list -m all
> 
> will list the dependencies of a module, but that doesn't look like it
> can be translated into src_uri format.
> 
> You would basically have to parse go.mod exactly the way upstream does
> it and come up with a way to download the correct versions of the
> source.
> 
> William

check mail-client/aerc ebuild.
I use "go list -m all" and manually format EGO_VENDOR string which will be 
translated into SRC_URI by eclass.
tool is certainly possible and should be quite easy to implement.
some manual editing will still be needed if dealing with forked packages/
repos, but looks pretty straightforward.
This is very similar approach to cargo ebuilds and it supports offline 
installs, PM checksumming and does not require packaging every single go 
dependency as a package.


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] rfc: go 1.13 and go modules

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1952 bytes --]

On Mon, Sep 09, 2019 at 03:57:18PM -0700, Georgy Yakovlev wrote:
> On Monday, September 9, 2019 2:46:16 PM PDT William Hubbs wrote:
> > On Tue, Sep 10, 2019 at 08:35:17AM +1200, Kent Fredric wrote:
> > > On Mon, 9 Sep 2019 12:34:18 -0500
> > > 
> > > William Hubbs <williamh@gentoo.org> wrote:
> > > > There is another option I want to try which is adding "go mod vendor" to
> > > > src_unpack for go packages.
> > > 
> > > Is it infeasible to write a tool that you execute as a maintainer, that
> > > simulates what "go mod vendor" would do, but instead emits a list of
> > > entries for SRC_URI, and then have an eclass or something construct the
> > > vendor dir from those?
> > > 
> > > That's what is available for rust stuff.
> > 
> > I'm not sure how feasible something like that is.
> > 
> > $ go list -m all
> > 
> > will list the dependencies of a module, but that doesn't look like it
> > can be translated into src_uri format.
> > 
> > You would basically have to parse go.mod exactly the way upstream does
> > it and come up with a way to download the correct versions of the
> > source.
> > 
> > William
> 
> check mail-client/aerc ebuild.
> I use "go list -m all" and manually format EGO_VENDOR string which will be 
> translated into SRC_URI by eclass.
> tool is certainly possible and should be quite easy to implement.
> some manual editing will still be needed if dealing with forked packages/
> repos, but looks pretty straightforward.
> This is very similar approach to cargo ebuilds and it supports offline 
> installs, PM checksumming and does not require packaging every single go 
> dependency as a package.

Ok, I took a quick look at this.
We will need another eclass similar to the golang-vcs-snapshot eclass,
but it doesn't need to mess with GOPATH since GOPATH isn't used by go
modules.

I will also look at go list -m all and see what comes out of it.

Thanks,

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] rfc: go 1.13 and go modules

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 2198 bytes --]

On Mon, Sep 09, 2019 at 06:21:42PM -0500, William Hubbs wrote:
> On Mon, Sep 09, 2019 at 03:57:18PM -0700, Georgy Yakovlev wrote:
> > On Monday, September 9, 2019 2:46:16 PM PDT William Hubbs wrote:
> > > On Tue, Sep 10, 2019 at 08:35:17AM +1200, Kent Fredric wrote:
> > > > On Mon, 9 Sep 2019 12:34:18 -0500
> > > > 
> > > > William Hubbs <williamh@gentoo.org> wrote:
> > > > > There is another option I want to try which is adding "go mod vendor" to
> > > > > src_unpack for go packages.
> > > > 
> > > > Is it infeasible to write a tool that you execute as a maintainer, that
> > > > simulates what "go mod vendor" would do, but instead emits a list of
> > > > entries for SRC_URI, and then have an eclass or something construct the
> > > > vendor dir from those?
> > > > 
> > > > That's what is available for rust stuff.
> > > 
> > > I'm not sure how feasible something like that is.
> > > 
> > > $ go list -m all
> > > 
> > > will list the dependencies of a module, but that doesn't look like it
> > > can be translated into src_uri format.
> > > 
> > > You would basically have to parse go.mod exactly the way upstream does
> > > it and come up with a way to download the correct versions of the
> > > source.
> > > 
> > > William
> > 
> > check mail-client/aerc ebuild.
> > I use "go list -m all" and manually format EGO_VENDOR string which will be 
> > translated into SRC_URI by eclass.
> > tool is certainly possible and should be quite easy to implement.
> > some manual editing will still be needed if dealing with forked packages/
> > repos, but looks pretty straightforward.
> > This is very similar approach to cargo ebuilds and it supports offline 
> > installs, PM checksumming and does not require packaging every single go 
> > dependency as a package.
> 
> Ok, I took a quick look at this.
> We will need another eclass similar to the golang-vcs-snapshot eclass,
> but it doesn't need to mess with GOPATH since GOPATH isn't used by go
> modules.
> 
> I will also look at go list -m all and see what comes out of it.

It looks like we would also need a way to honor the GOPROXY environment
variable as well.

Thanks,

William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] rfc: go 1.13 and go modules

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 215 bytes --]

On Tue, 10 Sep 2019 13:31:01 -0500
William Hubbs <williamh@gentoo.org> wrote:

> It looks like we would also need a way to honor the GOPROXY environment
> variable as well.

Or ... mirror://goproxy/<stuff>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] rfc: go 1.13 and go modules

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 627 bytes --]

On Mon, 9 Sep 2019 16:46:16 -0500
William Hubbs <williamh@gentoo.org> wrote:

> will list the dependencies of a module, but that doesn't look like it
> can be translated into src_uri format.

If you use the mirror as mentioned in https://blog.golang.org/module-mirror-launch

Then you should be able to do it in a straight-forward way.

You could even encode the dependencies in a bash array like:

GO_MODULES=(
     Foo@v0.2.3
)

Which would be expanded by the ebuild during "use go" or whatever, to

  https://proxy.golang.org/${PN}/foo/@v/v0.2.3.zip -> ${PN}-foo-v0.2.3.zip

Or something like that.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]