[gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[William Hubbs]

Signed-off-by: William Hubbs <williamh@gentoo.org>
---
 eclass/go-module.eclass | 100 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 100 insertions(+)
 create mode 100644 eclass/go-module.eclass

diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
new file mode 100644
index 00000000000..501e334b23f
--- /dev/null
+++ b/eclass/go-module.eclass
@@ -0,0 +1,100 @@
+# Copyright 2019 gentoo authors
+# Distributed under the terms of the GNU General Public License v2
+
+# @ECLASS: go-module.eclass
+# @MAINTAINER:
+# William Hubbs <williamh@gentoo.org>
+# @SUPPORTED_EAPIS: 7
+# @BLURB: basic eclass for building software written in the go
+# programming language that uses go modules.
+# @DESCRIPTION:
+# This eclass provides some basic things needed by all software
+# written in the go programming language that uses go modules.
+#
+# You will know the software you are packaging uses modules because
+# it will have files named go.sum and go.mod in its top-level source
+# directory. If it does not have these files, use the golang-* eclasses.
+#
+# If the software you are packaging uses modules, the next question is
+# whether it has a directory named "vendor" at the top-level of the source tree.
+#
+# If it doesn't, you need to create a tarball of what would be in the
+# vendor directory and mirror it locally.
+# If foo-1.0 is the name of your project and you have the tarball for it
+# in your current directory,  this is done with the following commands:
+#
+# @CODE:
+#
+# tar -xf foo-1.0.tar.gz
+# cd foo-1.0
+# go mod vendor
+# cd ..
+# tar -acf foo-1.0-vendor.tar.gz foo-1.0/vendor
+#
+# @CODE:
+
+# If we uncomment src_prepare below, the last two lines in the above
+# code block are reduced to one:
+#
+# @CODE:
+#
+# tar -acf foo-1.0-vendor.tar.gz vendor
+#
+# @CODE:
+
+case ${EAPI:-0} in
+	7) ;;
+	*) die "${ECLASS} API in EAPI ${EAPI} not yet established."
+esac
+
+if [[ -z ${_GO_MODULE} ]]; then
+
+_GO_MODULE=1
+
+BDEPEND=">=dev-lang/go-1.12"
+
+# The following go flags should be used for all go builds.
+# -mod=vendor stopps downloading of dependencies from the internet.
+# -v prints the names of packages as they are compiled
+# -x prints commands as they are executed
+export GOFLAGS="-mod=vendor -v -x"
+
+# Do not complain about CFLAGS etc since go projects do not use them.
+QA_FLAGS_IGNORED='.*'
+
+# Go packages should not be stripped with strip(1).
+RESTRICT="strip"
+
+# EXPORT_FUNCTIONS src_prepare
+#
+# @FUNCTION: go-module_src_prepare
+# @DESCRIPTION:
+# Run a default src_prepare then move our provided vendor directory to
+# the appropriate spot if upstream doesn't provide a vendor directory.
+#
+# This is commented out because I want to see where the discussion on
+# the ml leads.
+# Commenting it out and following the above instructions means that you
+# are forced to manually re-tar the vendored dependencies for every
+# version bump.
+# Using the previous method, it would be possible to decide if you need
+# to do this by comparing the contents of go.mod in the previous and new
+# version.
+# go-module_src_prepare() {
+#	default
+#	# If upstream vendors the dependencies and we provide a vendor
+#	# tarball, generate a qa warning.
+#	[[ -d vendor ]] && [[ -d ../vendor ]] &&
+#		eqawarn "This package's upstream source includes a vendor
+#		eqawarn "directory and the maintainer provides a vendor tarball."
+#		eqawarn "Please report this on https://bugs.gentoo.org"
+#	# Use the upstream provided vendor directory if it exists.
+#	[[ -d vendor ]] && return
+#	# If we are not providing a mirror of a vendor directory we created
+#	# manually, return since there may be nothing to vendor.
+#	[[ ! -d ../vendor ]] && return
+#	# At this point, we know we are providing a vendor mirror.
+#	mv ../vendor . || die "Unable to move ../vendor directory"
+# }
+
+fi
-- 
2.21.0

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 121 bytes --]

I know about the typos on the eqawarns, they need to be a block if, so
I'll take care of that before I commit.

Wililam


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[gentoo-dev] [PATCH 0/1] introduce new eclass to handle go modules (round 3)

[William Hubbs]

All,

this is the third iteration of this eclass.
I have added a pkg_postinst function to warn about go being statically
linked. Also, the src_prepare function, which is still commented out,
has been adjusted to show how I could generate a qa warning if the
maintainer packs a vendor tarball and upstream vendors.

This could not happen if we use the method forcing the version into the
path stored in the vendor tarball.

Thanks,

William

William Hubbs (1):
  go-module.eclass: introduce new eclass to handle go modules

 eclass/go-module.eclass | 117 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 117 insertions(+)
 create mode 100644 eclass/go-module.eclass

-- 
2.21.0

[gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[William Hubbs]

Signed-off-by: William Hubbs <williamh@gentoo.org>
---
 eclass/go-module.eclass | 117 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 117 insertions(+)
 create mode 100644 eclass/go-module.eclass

diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
new file mode 100644
index 00000000000..7e16ec4e95c
--- /dev/null
+++ b/eclass/go-module.eclass
@@ -0,0 +1,117 @@
+# Copyright 2019 gentoo authors
+# Distributed under the terms of the GNU General Public License v2
+
+# @ECLASS: go-module.eclass
+# @MAINTAINER:
+# William Hubbs <williamh@gentoo.org>
+# @SUPPORTED_EAPIS: 7
+# @BLURB: basic eclass for building software written in the go
+# programming language that uses go modules.
+# @DESCRIPTION:
+# This eclass provides some basic things needed by all software
+# written in the go programming language that uses go modules.
+#
+# You will know the software you are packaging uses modules because
+# it will have files named go.sum and go.mod in its top-level source
+# directory. If it does not have these files, use the golang-* eclasses.
+#
+# If the software you are packaging uses modules, the next question is
+# whether it has a directory named "vendor" at the top-level of the source tree.
+#
+# If it doesn't, you need to create a tarball of what would be in the
+# vendor directory and mirror it locally.
+# If foo-1.0 is the name of your project and you have the tarball for it
+# in your current directory,  this is done with the following commands:
+#
+# @CODE:
+#
+# tar -xf foo-1.0.tar.gz
+# cd foo-1.0
+# go mod vendor
+# cd ..
+# tar -acf foo-1.0-vendor.tar.gz foo-1.0/vendor
+#
+# @CODE:
+
+# If we uncomment src_prepare below, the last two lines in the above
+# code block are reduced to one:
+#
+# @CODE:
+#
+# tar -acf foo-1.0-vendor.tar.gz vendor
+#
+# @CODE:
+
+case ${EAPI:-0} in
+	7) ;;
+	*) die "${ECLASS} API in EAPI ${EAPI} not yet established."
+esac
+
+if [[ -z ${_GO_MODULE} ]]; then
+
+_GO_MODULE=1
+
+BDEPEND=">=dev-lang/go-1.12"
+
+# The following go flags should be used for all go builds.
+# -mod=vendor stopps downloading of dependencies from the internet.
+# -v prints the names of packages as they are compiled
+# -x prints commands as they are executed
+export GOFLAGS="-mod=vendor -v -x"
+
+# Do not complain about CFLAGS etc since go projects do not use them.
+QA_FLAGS_IGNORED='.*'
+
+# Go packages should not be stripped with strip(1).
+RESTRICT="strip"
+
+# EXPORT_FUNCTIONS src_prepare pkg_postinst
+ EXPORT_FUNCTIONS pkg_postinst
+
+# @FUNCTION: go-module_src_prepare
+# @DESCRIPTION:
+# Run a default src_prepare then move our provided vendor directory to
+# the appropriate spot if upstream doesn't provide a vendor directory.
+#
+# This is commented out because I want to see where the discussion on
+# the ml leads.
+# Commenting it out and following the above instructions means that you
+# are forced to manually re-tar the vendored dependencies for every
+# version bump.
+# Using the previous method, it would be possible to decide if you need
+# to do this by comparing the contents of go.mod in the previous and new
+# version.
+# Also, note that we can generate a qa warning if a maintainer forgets
+# to drop the vendor tarball and upstream starts vendoring.
+# go-module_src_prepare() {
+#	default
+#	# If upstream vendors the dependencies and we provide a vendor
+#	# tarball, generate a qa warning.
+#	if [[ -d vendor ]] && [[ -d ../vendor ]] ; then
+#		eqawarn "This package's upstream source includes a vendor
+#		eqawarn "directory and the maintainer provides a vendor tarball."
+#		eqawarn "Please report this on https://bugs.gentoo.org"
+#	fi
+#	# Use the upstream provided vendor directory if it exists.
+#	[[ -d vendor ]] && return
+#	# If we are not providing a mirror of a vendor directory we created
+#	# manually, return since there may be nothing to vendor.
+#	[[ ! -d ../vendor ]] && return
+#	# At this point, we know we are providing a vendor mirror.
+#	mv ../vendor . || die "Unable to move ../vendor directory"
+# }
+
+# @FUNCTION: go-module_pkg_postinst
+# @DESCRIPTION:
+# Display a warning about security updates for Go programs.
+go-module_pkg_postinst() {
+	ewarn "${PN} is written in the Go programming language."
+	ewarn "Since this language is statically linked, security"
+	ewarn "updates will be handled in individual packages and will be"
+	ewarn "difficult for us to track as a distribution."
+	ewarn "For this reason, please update any go packages asap when new"
+	ewarn "versions enter the tree or go stable if you are running the"
+	ewarn "stable tree."
+}
+
+fi
-- 
2.21.0

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 474 bytes --]

All,

here is my thought about the src_prepare function in the eclass.
It is commented right now, but I want to uncomment it and not force ${P}
into the path in the vendor tarball if no one objects; I think it is
easier to do that because it would allow me to detect when upstream
vendors and the maintainer tries to provide a vendor tarball at the same
time as shown in the first if block in the function.

Let me know if you agree or disagree with this.

Thanks,

William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 570 bytes --]

On 9/16/19 7:17 AM, William Hubbs wrote:
> +BDEPEND=">=dev-lang/go-1.12"
> +
> +# The following go flags should be used for all go builds.
> +# -mod=vendor stopps downloading of dependencies from the internet.
> +# -v prints the names of packages as they are compiled
> +# -x prints commands as they are executed
> +export GOFLAGS="-mod=vendor -v -x"

My experience with g-1.12.x was that you have to export GO111MODULE=on
or else GOFLAGS="-mod=vendor" triggers an error like this:

build flag -mod=vendor only valid when using modules
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 1258 bytes --]

On 9/16/19 7:17 AM, William Hubbs wrote:
> +# You will know the software you are packaging uses modules because
> +# it will have files named go.sum and go.mod in its top-level source
> +# directory. If it does not have these files, use the golang-* eclasses.
> +#
> +# If the software you are packaging uses modules, the next question is
> +# whether it has a directory named "vendor" at the top-level of the source tree.
> +#
> +# If it doesn't, you need to create a tarball of what would be in the
> +# vendor directory and mirror it locally.
> +# If foo-1.0 is the name of your project and you have the tarball for it
> +# in your current directory,  this is done with the following commands:
> +#
> +# @CODE:
> +#
> +# tar -xf foo-1.0.tar.gz
> +# cd foo-1.0
> +# go mod vendor
> +# cd ..
> +# tar -acf foo-1.0-vendor.tar.gz foo-1.0/vendor

For packages that I maintain, I'd prefer to continue using EGO_VENDOR to
even with packages using go.mod. I hope that this go-module.class will
not preclude this sort of usage. For example, the latest go-tools ebuild
uses EGO_VENDOR together with GOFLAGS="-mod=vendor":

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=8cc6d401139526e2f9a6dbadbd31f0ff2387705f
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 5311 bytes --]

On Mon, 2019-09-16 at 09:17 -0500, William Hubbs wrote:
> Signed-off-by: William Hubbs <williamh@gentoo.org>
> ---
>  eclass/go-module.eclass | 117 ++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 117 insertions(+)
>  create mode 100644 eclass/go-module.eclass
> 
> diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
> new file mode 100644
> index 00000000000..7e16ec4e95c
> --- /dev/null
> +++ b/eclass/go-module.eclass
> @@ -0,0 +1,117 @@
> +# Copyright 2019 gentoo authors
> +# Distributed under the terms of the GNU General Public License v2
> +
> +# @ECLASS: go-module.eclass
> +# @MAINTAINER:
> +# William Hubbs <williamh@gentoo.org>
> +# @SUPPORTED_EAPIS: 7
> +# @BLURB: basic eclass for building software written in the go
> +# programming language that uses go modules.
> +# @DESCRIPTION:
> +# This eclass provides some basic things needed by all software
> +# written in the go programming language that uses go modules.
> +#
> +# You will know the software you are packaging uses modules because
> +# it will have files named go.sum and go.mod in its top-level source
> +# directory. If it does not have these files, use the golang-* eclasses.

Please add a big fat warning around here somewhere that people need to
look through LICENSE files in all vendored modules, and list them
in LICENSE.  They also need to watch out for license conflicts.

> +#
> +# If the software you are packaging uses modules, the next question is
> +# whether it has a directory named "vendor" at the top-level of the source tree.
> +#
> +# If it doesn't, you need to create a tarball of what would be in the
> +# vendor directory and mirror it locally.
> +# If foo-1.0 is the name of your project and you have the tarball for it
> +# in your current directory,  this is done with the following commands:
> +#
> +# @CODE:
> +#
> +# tar -xf foo-1.0.tar.gz
> +# cd foo-1.0
> +# go mod vendor
> +# cd ..
> +# tar -acf foo-1.0-vendor.tar.gz foo-1.0/vendor
> +#
> +# @CODE:
> +
> +# If we uncomment src_prepare below, the last two lines in the above
> +# code block are reduced to one:
> +#
> +# @CODE:
> +#
> +# tar -acf foo-1.0-vendor.tar.gz vendor
> +#
> +# @CODE:
> +
> +case ${EAPI:-0} in
> +	7) ;;
> +	*) die "${ECLASS} API in EAPI ${EAPI} not yet established."
> +esac
> +
> +if [[ -z ${_GO_MODULE} ]]; then
> +
> +_GO_MODULE=1
> +
> +BDEPEND=">=dev-lang/go-1.12"
> +
> +# The following go flags should be used for all go builds.
> +# -mod=vendor stopps downloading of dependencies from the internet.
> +# -v prints the names of packages as they are compiled
> +# -x prints commands as they are executed
> +export GOFLAGS="-mod=vendor -v -x"
> +
> +# Do not complain about CFLAGS etc since go projects do not use them.
> +QA_FLAGS_IGNORED='.*'
> +
> +# Go packages should not be stripped with strip(1).
> +RESTRICT="strip"
> +
> +# EXPORT_FUNCTIONS src_prepare pkg_postinst
> + EXPORT_FUNCTIONS pkg_postinst
> +
> +# @FUNCTION: go-module_src_prepare
> +# @DESCRIPTION:
> +# Run a default src_prepare then move our provided vendor directory to
> +# the appropriate spot if upstream doesn't provide a vendor directory.
> +#
> +# This is commented out because I want to see where the discussion on
> +# the ml leads.
> +# Commenting it out and following the above instructions means that you
> +# are forced to manually re-tar the vendored dependencies for every
> +# version bump.
> +# Using the previous method, it would be possible to decide if you need
> +# to do this by comparing the contents of go.mod in the previous and new
> +# version.
> +# Also, note that we can generate a qa warning if a maintainer forgets
> +# to drop the vendor tarball and upstream starts vendoring.
> +# go-module_src_prepare() {
> +#	default
> +#	# If upstream vendors the dependencies and we provide a vendor
> +#	# tarball, generate a qa warning.
> +#	if [[ -d vendor ]] && [[ -d ../vendor ]] ; then
> +#		eqawarn "This package's upstream source includes a vendor
> +#		eqawarn "directory and the maintainer provides a vendor tarball."
> +#		eqawarn "Please report this on https://bugs.gentoo.org"

Why aren't you making it fatal?

> +#	fi
> +#	# Use the upstream provided vendor directory if it exists.
> +#	[[ -d vendor ]] && return
> +#	# If we are not providing a mirror of a vendor directory we created
> +#	# manually, return since there may be nothing to vendor.
> +#	[[ ! -d ../vendor ]] && return
> +#	# At this point, we know we are providing a vendor mirror.
> +#	mv ../vendor . || die "Unable to move ../vendor directory"
> +# }
> +
> +# @FUNCTION: go-module_pkg_postinst
> +# @DESCRIPTION:
> +# Display a warning about security updates for Go programs.
> +go-module_pkg_postinst() {
> +	ewarn "${PN} is written in the Go programming language."
> +	ewarn "Since this language is statically linked, security"
> +	ewarn "updates will be handled in individual packages and will be"
> +	ewarn "difficult for us to track as a distribution."
> +	ewarn "For this reason, please update any go packages asap when new"
> +	ewarn "versions enter the tree or go stable if you are running the"
> +	ewarn "stable tree."
> +}
> +
> +fi

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 826 bytes --]

On Mon, Sep 16, 2019 at 10:48:14AM -0700, Zac Medico wrote:
> On 9/16/19 7:17 AM, William Hubbs wrote:
> > +BDEPEND=">=dev-lang/go-1.12"
> > +
> > +# The following go flags should be used for all go builds.
> > +# -mod=vendor stopps downloading of dependencies from the internet.
> > +# -v prints the names of packages as they are compiled
> > +# -x prints commands as they are executed
> > +export GOFLAGS="-mod=vendor -v -x"
> 
> My experience with g-1.12.x was that you have to export GO111MODULE=on
> or else GOFLAGS="-mod=vendor" triggers an error like this:
> 
> build flag -mod=vendor only valid when using modules

I thought that was only if ${S} was in GOPATH, but I'll take a look real
quick. If it is, would it be best to bump the BDEPEND or turn on the
environment setting?

Thanks,

William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1624 bytes --]

On Mon, Sep 16, 2019 at 11:01:38AM -0700, Zac Medico wrote:
> On 9/16/19 7:17 AM, William Hubbs wrote:
> > +# You will know the software you are packaging uses modules because
> > +# it will have files named go.sum and go.mod in its top-level source
> > +# directory. If it does not have these files, use the golang-* eclasses.
> > +#
> > +# If the software you are packaging uses modules, the next question is
> > +# whether it has a directory named "vendor" at the top-level of the source tree.
> > +#
> > +# If it doesn't, you need to create a tarball of what would be in the
> > +# vendor directory and mirror it locally.
> > +# If foo-1.0 is the name of your project and you have the tarball for it
> > +# in your current directory,  this is done with the following commands:
> > +#
> > +# @CODE:
> > +#
> > +# tar -xf foo-1.0.tar.gz
> > +# cd foo-1.0
> > +# go mod vendor
> > +# cd ..
> > +# tar -acf foo-1.0-vendor.tar.gz foo-1.0/vendor
> 
> For packages that I maintain, I'd prefer to continue using EGO_VENDOR to
> even with packages using go.mod. I hope that this go-module.class will
> not preclude this sort of usage. For example, the latest go-tools ebuild
> uses EGO_VENDOR together with GOFLAGS="-mod=vendor":
> 
> https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=8cc6d401139526e2f9a6dbadbd31f0ff2387705f

Can you elaborate on why you want to keep EGO_VENDOR?

The "go mod vendor" command above downloads all the correct versions
of the dependencies and puts them in the vendor directory, so I'm not
sure why you would need the EGO_VENDOR variable.

Thanks,

William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 4740 bytes --]

On Mon, Sep 16, 2019 at 08:05:50PM +0200, Michał Górny wrote:
> On Mon, 2019-09-16 at 09:17 -0500, William Hubbs wrote:
> > Signed-off-by: William Hubbs <williamh@gentoo.org>
> > ---
> >  eclass/go-module.eclass | 117 ++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 117 insertions(+)
> >  create mode 100644 eclass/go-module.eclass
> > 
> > diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
> > new file mode 100644
> > index 00000000000..7e16ec4e95c
> > --- /dev/null
> > +++ b/eclass/go-module.eclass
> > @@ -0,0 +1,117 @@
> > +# Copyright 2019 gentoo authors
> > +# Distributed under the terms of the GNU General Public License v2
> > +
> > +# @ECLASS: go-module.eclass
> > +# @MAINTAINER:
> > +# William Hubbs <williamh@gentoo.org>
> > +# @SUPPORTED_EAPIS: 7
> > +# @BLURB: basic eclass for building software written in the go
> > +# programming language that uses go modules.
> > +# @DESCRIPTION:
> > +# This eclass provides some basic things needed by all software
> > +# written in the go programming language that uses go modules.
> > +#
> > +# You will know the software you are packaging uses modules because
> > +# it will have files named go.sum and go.mod in its top-level source
> > +# directory. If it does not have these files, use the golang-* eclasses.
> 
> Please add a big fat warning around here somewhere that people need to
> look through LICENSE files in all vendored modules, and list them
> in LICENSE.  They also need to watch out for license conflicts.
> 
> > +#
> > +# If the software you are packaging uses modules, the next question is
> > +# whether it has a directory named "vendor" at the top-level of the source tree.
> > +#
> > +# If it doesn't, you need to create a tarball of what would be in the
> > +# vendor directory and mirror it locally.
> > +# If foo-1.0 is the name of your project and you have the tarball for it
> > +# in your current directory,  this is done with the following commands:
> > +#
> > +# @CODE:
> > +#
> > +# tar -xf foo-1.0.tar.gz
> > +# cd foo-1.0
> > +# go mod vendor
> > +# cd ..
> > +# tar -acf foo-1.0-vendor.tar.gz foo-1.0/vendor
> > +#
> > +# @CODE:
> > +
> > +# If we uncomment src_prepare below, the last two lines in the above
> > +# code block are reduced to one:
> > +#
> > +# @CODE:
> > +#
> > +# tar -acf foo-1.0-vendor.tar.gz vendor
> > +#
> > +# @CODE:
> > +
> > +case ${EAPI:-0} in
> > +	7) ;;
> > +	*) die "${ECLASS} API in EAPI ${EAPI} not yet established."
> > +esac
> > +
> > +if [[ -z ${_GO_MODULE} ]]; then
> > +
> > +_GO_MODULE=1
> > +
> > +BDEPEND=">=dev-lang/go-1.12"
> > +
> > +# The following go flags should be used for all go builds.
> > +# -mod=vendor stopps downloading of dependencies from the internet.
> > +# -v prints the names of packages as they are compiled
> > +# -x prints commands as they are executed
> > +export GOFLAGS="-mod=vendor -v -x"
> > +
> > +# Do not complain about CFLAGS etc since go projects do not use them.
> > +QA_FLAGS_IGNORED='.*'
> > +
> > +# Go packages should not be stripped with strip(1).
> > +RESTRICT="strip"
> > +
> > +# EXPORT_FUNCTIONS src_prepare pkg_postinst
> > + EXPORT_FUNCTIONS pkg_postinst
> > +
> > +# @FUNCTION: go-module_src_prepare
> > +# @DESCRIPTION:
> > +# Run a default src_prepare then move our provided vendor directory to
> > +# the appropriate spot if upstream doesn't provide a vendor directory.
> > +#
> > +# This is commented out because I want to see where the discussion on
> > +# the ml leads.
> > +# Commenting it out and following the above instructions means that you
> > +# are forced to manually re-tar the vendored dependencies for every
> > +# version bump.
> > +# Using the previous method, it would be possible to decide if you need
> > +# to do this by comparing the contents of go.mod in the previous and new
> > +# version.
> > +# Also, note that we can generate a qa warning if a maintainer forgets
> > +# to drop the vendor tarball and upstream starts vendoring.
> > +# go-module_src_prepare() {
> > +#	default
> > +#	# If upstream vendors the dependencies and we provide a vendor
> > +#	# tarball, generate a qa warning.
> > +#	if [[ -d vendor ]] && [[ -d ../vendor ]] ; then
> > +#		eqawarn "This package's upstream source includes a vendor
> > +#		eqawarn "directory and the maintainer provides a vendor tarball."
> > +#		eqawarn "Please report this on https://bugs.gentoo.org"
> 
> Why aren't you making it fatal?

I didn't make it fatal because it doesn't break the build. The build
will ignore the ../vendor directory from the tarball since it is not
under ${S}. Do you want it to be fatal?

Thanks,

William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 1106 bytes --]

On 9/16/19 11:35 AM, William Hubbs wrote:
> On Mon, Sep 16, 2019 at 11:01:38AM -0700, Zac Medico wrote:
>> For packages that I maintain, I'd prefer to continue using EGO_VENDOR to
>> even with packages using go.mod. I hope that this go-module.class will
>> not preclude this sort of usage. For example, the latest go-tools ebuild
>> uses EGO_VENDOR together with GOFLAGS="-mod=vendor":
>>
>> https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=8cc6d401139526e2f9a6dbadbd31f0ff2387705f
> 
> Can you elaborate on why you want to keep EGO_VENDOR?
> 
> The "go mod vendor" command above downloads all the correct versions
> of the dependencies and puts them in the vendor directory, so I'm not
> sure why you would need the EGO_VENDOR variable.

EGO_VENDOR eliminates to need to generate and host monolithic tarballs
containing vendored dependencies. It's more space-efficient in the sense
that each vendored dependency is stored in a separate tarball, so
multiple ebuilds can share the same tarball if the version of a
particular vendored dependency has not changed.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 5153 bytes --]

On Mon, 2019-09-16 at 13:46 -0500, William Hubbs wrote:
> On Mon, Sep 16, 2019 at 08:05:50PM +0200, Michał Górny wrote:
> > On Mon, 2019-09-16 at 09:17 -0500, William Hubbs wrote:
> > > Signed-off-by: William Hubbs <williamh@gentoo.org>
> > > ---
> > >  eclass/go-module.eclass | 117 ++++++++++++++++++++++++++++++++++++++++
> > >  1 file changed, 117 insertions(+)
> > >  create mode 100644 eclass/go-module.eclass
> > > 
> > > diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
> > > new file mode 100644
> > > index 00000000000..7e16ec4e95c
> > > --- /dev/null
> > > +++ b/eclass/go-module.eclass
> > > @@ -0,0 +1,117 @@
> > > +# Copyright 2019 gentoo authors
> > > +# Distributed under the terms of the GNU General Public License v2
> > > +
> > > +# @ECLASS: go-module.eclass
> > > +# @MAINTAINER:
> > > +# William Hubbs <williamh@gentoo.org>
> > > +# @SUPPORTED_EAPIS: 7
> > > +# @BLURB: basic eclass for building software written in the go
> > > +# programming language that uses go modules.
> > > +# @DESCRIPTION:
> > > +# This eclass provides some basic things needed by all software
> > > +# written in the go programming language that uses go modules.
> > > +#
> > > +# You will know the software you are packaging uses modules because
> > > +# it will have files named go.sum and go.mod in its top-level source
> > > +# directory. If it does not have these files, use the golang-* eclasses.
> > 
> > Please add a big fat warning around here somewhere that people need to
> > look through LICENSE files in all vendored modules, and list them
> > in LICENSE.  They also need to watch out for license conflicts.
> > 
> > > +#
> > > +# If the software you are packaging uses modules, the next question is
> > > +# whether it has a directory named "vendor" at the top-level of the source tree.
> > > +#
> > > +# If it doesn't, you need to create a tarball of what would be in the
> > > +# vendor directory and mirror it locally.
> > > +# If foo-1.0 is the name of your project and you have the tarball for it
> > > +# in your current directory,  this is done with the following commands:
> > > +#
> > > +# @CODE:
> > > +#
> > > +# tar -xf foo-1.0.tar.gz
> > > +# cd foo-1.0
> > > +# go mod vendor
> > > +# cd ..
> > > +# tar -acf foo-1.0-vendor.tar.gz foo-1.0/vendor
> > > +#
> > > +# @CODE:
> > > +
> > > +# If we uncomment src_prepare below, the last two lines in the above
> > > +# code block are reduced to one:
> > > +#
> > > +# @CODE:
> > > +#
> > > +# tar -acf foo-1.0-vendor.tar.gz vendor
> > > +#
> > > +# @CODE:
> > > +
> > > +case ${EAPI:-0} in
> > > +	7) ;;
> > > +	*) die "${ECLASS} API in EAPI ${EAPI} not yet established."
> > > +esac
> > > +
> > > +if [[ -z ${_GO_MODULE} ]]; then
> > > +
> > > +_GO_MODULE=1
> > > +
> > > +BDEPEND=">=dev-lang/go-1.12"
> > > +
> > > +# The following go flags should be used for all go builds.
> > > +# -mod=vendor stopps downloading of dependencies from the internet.
> > > +# -v prints the names of packages as they are compiled
> > > +# -x prints commands as they are executed
> > > +export GOFLAGS="-mod=vendor -v -x"
> > > +
> > > +# Do not complain about CFLAGS etc since go projects do not use them.
> > > +QA_FLAGS_IGNORED='.*'
> > > +
> > > +# Go packages should not be stripped with strip(1).
> > > +RESTRICT="strip"
> > > +
> > > +# EXPORT_FUNCTIONS src_prepare pkg_postinst
> > > + EXPORT_FUNCTIONS pkg_postinst
> > > +
> > > +# @FUNCTION: go-module_src_prepare
> > > +# @DESCRIPTION:
> > > +# Run a default src_prepare then move our provided vendor directory to
> > > +# the appropriate spot if upstream doesn't provide a vendor directory.
> > > +#
> > > +# This is commented out because I want to see where the discussion on
> > > +# the ml leads.
> > > +# Commenting it out and following the above instructions means that you
> > > +# are forced to manually re-tar the vendored dependencies for every
> > > +# version bump.
> > > +# Using the previous method, it would be possible to decide if you need
> > > +# to do this by comparing the contents of go.mod in the previous and new
> > > +# version.
> > > +# Also, note that we can generate a qa warning if a maintainer forgets
> > > +# to drop the vendor tarball and upstream starts vendoring.
> > > +# go-module_src_prepare() {
> > > +#	default
> > > +#	# If upstream vendors the dependencies and we provide a vendor
> > > +#	# tarball, generate a qa warning.
> > > +#	if [[ -d vendor ]] && [[ -d ../vendor ]] ; then
> > > +#		eqawarn "This package's upstream source includes a vendor
> > > +#		eqawarn "directory and the maintainer provides a vendor tarball."
> > > +#		eqawarn "Please report this on https://bugs.gentoo.org"
> > 
> > Why aren't you making it fatal?
> 
> I didn't make it fatal because it doesn't break the build. The build
> will ignore the ../vendor directory from the tarball since it is not
> under ${S}. Do you want it to be fatal?
> 

Yes, it's ebuild author's mistake and there is no point why such mistake
should be committed.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 1098 bytes --]

On 9/16/19 11:26 AM, William Hubbs wrote:
> On Mon, Sep 16, 2019 at 10:48:14AM -0700, Zac Medico wrote:
>> On 9/16/19 7:17 AM, William Hubbs wrote:
>>> +BDEPEND=">=dev-lang/go-1.12"
>>> +
>>> +# The following go flags should be used for all go builds.
>>> +# -mod=vendor stopps downloading of dependencies from the internet.
>>> +# -v prints the names of packages as they are compiled
>>> +# -x prints commands as they are executed
>>> +export GOFLAGS="-mod=vendor -v -x"
>>
>> My experience with g-1.12.x was that you have to export GO111MODULE=on
>> or else GOFLAGS="-mod=vendor" triggers an error like this:
>>
>> build flag -mod=vendor only valid when using modules
> 
> I thought that was only if ${S} was in GOPATH, but I'll take a look real
> quick. If it is, would it be best to bump the BDEPEND or turn on the
> environment setting?

I've tested again, and in fact it appears that GOPATH="${S}" triggers
this error, so I guess it would be sufficient to ensure that GOPATH is
unset. Maybe it's cleaner to explicitly export GO111MODULE=on though.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1700 bytes --]

On Mon, Sep 16, 2019 at 11:50:12AM -0700, Zac Medico wrote:
> On 9/16/19 11:35 AM, William Hubbs wrote:
> > On Mon, Sep 16, 2019 at 11:01:38AM -0700, Zac Medico wrote:
> >> For packages that I maintain, I'd prefer to continue using EGO_VENDOR to
> >> even with packages using go.mod. I hope that this go-module.class will
> >> not preclude this sort of usage. For example, the latest go-tools ebuild
> >> uses EGO_VENDOR together with GOFLAGS="-mod=vendor":
> >>
> >> https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=8cc6d401139526e2f9a6dbadbd31f0ff2387705f
> > 
> > Can you elaborate on why you want to keep EGO_VENDOR?
> > 
> > The "go mod vendor" command above downloads all the correct versions
> > of the dependencies and puts them in the vendor directory, so I'm not
> > sure why you would need the EGO_VENDOR variable.
> 
> EGO_VENDOR eliminates to need to generate and host monolithic tarballs
> containing vendored dependencies. It's more space-efficient in the sense
> that each vendored dependency is stored in a separate tarball, so
> multiple ebuilds can share the same tarball if the version of a
> particular vendored dependency has not changed.

I see what you are saying, but I haven't yet found a way to generate
these separate tarballs that I'm comfortable with. Also, thinking about
this, there will be many more tarballs on our mirrors if we store one
dependency in each tarball than if we generate vendor tarballs that
contain all dependencies for a package.

I would consider this an enhancement to the eclass if you  still feel
that we need it, but let me get the eclass into the tree first then we
can work on that.

Thanks,

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[William Hubbs]

Signed-off-by: William Hubbs <williamh@gentoo.org>
---
 eclass/go-module.eclass | 105 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 105 insertions(+)
 create mode 100644 eclass/go-module.eclass

diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
new file mode 100644
index 00000000000..810e66e1c5b
--- /dev/null
+++ b/eclass/go-module.eclass
@@ -0,0 +1,105 @@
+# Copyright 2019 gentoo authors
+# Distributed under the terms of the GNU General Public License v2
+
+# @ECLASS: go-module.eclass
+# @MAINTAINER:
+# William Hubbs <williamh@gentoo.org>
+# @SUPPORTED_EAPIS: 7
+# @BLURB: basic eclass for building software written in the go
+# programming language that uses go modules.
+# @DESCRIPTION:
+# This eclass provides some basic things needed by all software
+# written in the go programming language that uses go modules.
+#
+# You will know the software you are packaging uses modules because
+# it will have files named go.sum and go.mod in its top-level source
+# directory. If it does not have these files, use the golang-* eclasses.
+#
+# If the software you are packaging uses modules, the next question is
+# whether it has a directory named "vendor" at the top-level of the source tree.
+#
+# If it doesn't, you need to create a tarball of what would be in the
+# vendor directory and mirror it locally.
+# If foo-1.0 is the name of your project and you have the tarball for it
+# in your current directory,  this is done with the following commands:
+#
+# @CODE:
+#
+# tar -xf foo-1.0.tar.gz
+# cd foo-1.0
+# go mod vendor
+# tar -acf foo-1.0-vendor.tar.gz vendor
+#
+# @CODE:
+#
+# Regardless of whether  you create a vendor tarball, Since Go programs
+# are statically linked, it is important that the  ebuild's LICENSE=
+# setting includes the licenses of all statically linked
+# dependencies. So please make sure it is accurate.
+
+case ${EAPI:-0} in
+	7) ;;
+	*) die "${ECLASS} API in EAPI ${EAPI} not yet established."
+esac
+
+if [[ -z ${_GO_MODULE} ]]; then
+
+_GO_MODULE=1
+
+BDEPEND=">=dev-lang/go-1.12"
+
+# Force go to build in module mode.
+# In this mode the GOPATH environment variable is ignored.
+# this will become the default in the future.
+export GO111MODULE=on
+
+# The following go flags should be used for all builds.
+# -mod=vendor stopps downloading of dependencies from the internet.
+# -v prints the names of packages as they are compiled
+# -x prints commands as they are executed
+export GOFLAGS="-mod=vendor -v -x"
+
+# Do not complain about CFLAGS etc since go projects do not use them.
+QA_FLAGS_IGNORED='.*'
+
+# Go packages should not be stripped with strip(1).
+RESTRICT="strip"
+
+EXPORT_FUNCTIONS src_prepare pkg_postinst
+
+# @FUNCTION: go-module_src_prepare
+# @DESCRIPTION:
+# Run a default src_prepare then move our provided vendor directory to
+# the appropriate spot if upstream doesn't provide a vendor directory.
+# If upstream vendors and we provide a vendor tarball, this is a fatal
+# error.
+go-module_src_prepare() {
+	default
+	if [[ -d "${S}"/vendor ]] && [[ -d ../vendor ]] ; then
+		eerror "The upstream source for ${P} includes a vendor directory"
+		eerror "and a local vendor tarball is provided."
+		die "vendored dependencies are provided upstream"
+	fi
+	# Use the upstream provided vendor directory if it exists.
+	[[ -d "${S}"/vendor ]] && return
+	# If we are not providing a mirror of a vendor directory we created
+	# manually, return since there may be nothing to vendor.
+	[[ ! -d ../vendor ]] && return
+	# At this point, we know we are providing a vendor mirror.
+	mv ../vendor "${S}" || die "Unable to move ../vendor directory"
+}
+
+# @FUNCTION: go-module_pkg_postinst
+# @DESCRIPTION:
+# Display a warning about security updates for Go programs.
+go-module_pkg_postinst() {
+	ewarn "${PN} is written in the Go programming language."
+	ewarn "Since this language is statically linked, security"
+	ewarn "updates will be handled in individual packages and will be"
+	ewarn "difficult for us to track as a distribution."
+	ewarn "For this reason, please update any go packages asap when new"
+	ewarn "versions enter the tree or go stable if you are running the"
+	ewarn "stable tree."
+}
+
+fi
-- 
2.21.0

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2073 bytes --]

On Mon, 2019-09-16 at 17:00 -0500, William Hubbs wrote:
> On Mon, Sep 16, 2019 at 11:50:12AM -0700, Zac Medico wrote:
> > On 9/16/19 11:35 AM, William Hubbs wrote:
> > > On Mon, Sep 16, 2019 at 11:01:38AM -0700, Zac Medico wrote:
> > > > For packages that I maintain, I'd prefer to continue using EGO_VENDOR to
> > > > even with packages using go.mod. I hope that this go-module.class will
> > > > not preclude this sort of usage. For example, the latest go-tools ebuild
> > > > uses EGO_VENDOR together with GOFLAGS="-mod=vendor":
> > > > 
> > > > https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=8cc6d401139526e2f9a6dbadbd31f0ff2387705f
> > > 
> > > Can you elaborate on why you want to keep EGO_VENDOR?
> > > 
> > > The "go mod vendor" command above downloads all the correct versions
> > > of the dependencies and puts them in the vendor directory, so I'm not
> > > sure why you would need the EGO_VENDOR variable.
> > 
> > EGO_VENDOR eliminates to need to generate and host monolithic tarballs
> > containing vendored dependencies. It's more space-efficient in the sense
> > that each vendored dependency is stored in a separate tarball, so
> > multiple ebuilds can share the same tarball if the version of a
> > particular vendored dependency has not changed.
> 
> I see what you are saying, but I haven't yet found a way to generate
> these separate tarballs that I'm comfortable with. Also, thinking about
> this, there will be many more tarballs on our mirrors if we store one
> dependency in each tarball than if we generate vendor tarballs that
> contain all dependencies for a package.
> 
> I would consider this an enhancement to the eclass if you  still feel
> that we need it, but let me get the eclass into the tree first then we
> can work on that.
> 

That sounds like a bad idea.  If there are any potential enhancements
that can happen, I'd rather see them happen before there's a bunch of
ebuilds using the eclass in the wild, and potentially limiting possible
changes.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 2709 bytes --]

On Tue, Sep 17, 2019 at 07:36:07AM +0200, Michał Górny wrote:
> On Mon, 2019-09-16 at 17:00 -0500, William Hubbs wrote:
> > On Mon, Sep 16, 2019 at 11:50:12AM -0700, Zac Medico wrote:
> > > On 9/16/19 11:35 AM, William Hubbs wrote:
> > > > On Mon, Sep 16, 2019 at 11:01:38AM -0700, Zac Medico wrote:
> > > > > For packages that I maintain, I'd prefer to continue using EGO_VENDOR to
> > > > > even with packages using go.mod. I hope that this go-module.class will
> > > > > not preclude this sort of usage. For example, the latest go-tools ebuild
> > > > > uses EGO_VENDOR together with GOFLAGS="-mod=vendor":
> > > > > 
> > > > > https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=8cc6d401139526e2f9a6dbadbd31f0ff2387705f
> > > > 
> > > > Can you elaborate on why you want to keep EGO_VENDOR?
> > > > 
> > > > The "go mod vendor" command above downloads all the correct versions
> > > > of the dependencies and puts them in the vendor directory, so I'm not
> > > > sure why you would need the EGO_VENDOR variable.
> > > 
> > > EGO_VENDOR eliminates to need to generate and host monolithic tarballs
> > > containing vendored dependencies. It's more space-efficient in the sense
> > > that each vendored dependency is stored in a separate tarball, so
> > > multiple ebuilds can share the same tarball if the version of a
> > > particular vendored dependency has not changed.
> > 
> > I see what you are saying, but I haven't yet found a way to generate
> > these separate tarballs that I'm comfortable with. Also, thinking about
> > this, there will be many more tarballs on our mirrors if we store one
> > dependency in each tarball than if we generate vendor tarballs that
> > contain all dependencies for a package.
> > 
> > I would consider this an enhancement to the eclass if you  still feel
> > that we need it, but let me get the eclass into the tree first then we
> > can work on that.
> > 
> 
> That sounds like a bad idea.  If there are any potential enhancements
> that can happen, I'd rather see them happen before there's a bunch of
> ebuilds using the eclass in the wild, and potentially limiting possible
> changes.

Like I just said on IRC, it would have been better if you responded in
terms of discussing the enhancement itself.

The main blocker for me is that EGO_VENDOR is basically the same
information as go.mod, but it isn't quite the same format.
You can get close with "go list -m all", but EGO_VENDOR doesn't
automatically handle imports that start with things like golang.org/x or
gopkg.in; you have to manually fix those, and you would have to do that
every time. That seems to be a lot of work for little gain.

William



[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 3784 bytes --]

On 9/17/19 7:10 AM, William Hubbs wrote:
> On Tue, Sep 17, 2019 at 07:36:07AM +0200, Michał Górny wrote:
>> On Mon, 2019-09-16 at 17:00 -0500, William Hubbs wrote:
>>> On Mon, Sep 16, 2019 at 11:50:12AM -0700, Zac Medico wrote:
>>>> On 9/16/19 11:35 AM, William Hubbs wrote:
>>>>> On Mon, Sep 16, 2019 at 11:01:38AM -0700, Zac Medico wrote:
>>>>>> For packages that I maintain, I'd prefer to continue using EGO_VENDOR to
>>>>>> even with packages using go.mod. I hope that this go-module.class will
>>>>>> not preclude this sort of usage. For example, the latest go-tools ebuild
>>>>>> uses EGO_VENDOR together with GOFLAGS="-mod=vendor":
>>>>>>
>>>>>> https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=8cc6d401139526e2f9a6dbadbd31f0ff2387705f
>>>>>
>>>>> Can you elaborate on why you want to keep EGO_VENDOR?
>>>>>
>>>>> The "go mod vendor" command above downloads all the correct versions
>>>>> of the dependencies and puts them in the vendor directory, so I'm not
>>>>> sure why you would need the EGO_VENDOR variable.
>>>>
>>>> EGO_VENDOR eliminates to need to generate and host monolithic tarballs
>>>> containing vendored dependencies. It's more space-efficient in the sense
>>>> that each vendored dependency is stored in a separate tarball, so
>>>> multiple ebuilds can share the same tarball if the version of a
>>>> particular vendored dependency has not changed.
>>>
>>> I see what you are saying, but I haven't yet found a way to generate
>>> these separate tarballs that I'm comfortable with. Also, thinking about
>>> this, there will be many more tarballs on our mirrors if we store one
>>> dependency in each tarball than if we generate vendor tarballs that
>>> contain all dependencies for a package.
>>>
>>> I would consider this an enhancement to the eclass if you  still feel
>>> that we need it, but let me get the eclass into the tree first then we
>>> can work on that.
>>>
>>
>> That sounds like a bad idea.  If there are any potential enhancements
>> that can happen, I'd rather see them happen before there's a bunch of
>> ebuilds using the eclass in the wild, and potentially limiting possible
>> changes.
> 
> Like I just said on IRC, it would have been better if you responded in
> terms of discussing the enhancement itself.
> 
> The main blocker for me is that EGO_VENDOR is basically the same
> information as go.mod, but it isn't quite the same format.
> You can get close with "go list -m all", but EGO_VENDOR doesn't
> automatically handle imports that start with things like golang.org/x or
> gopkg.in; you have to manually fix those, and you would have to do that
> every time. That seems to be a lot of work for little gain.

It looks like it should not be too difficult to create a script that
will convert from go.mod to EGO_VENDOR format. For example, take this
go.mod file:

    https://github.com/golang/tools/blob/master/go.mod

It contains a line like this:

    golang.org/x/net v0.0.0-20190620200207-3b0461eec859

The part after the last hyphen corresponds to the commit hash which can
be used directly or expanded like this:

    $ curl -sS https://api.github.com/repos/golang/net/commits/3b0461eec859 | jq -r .sha
    3b0461eec859c4b73bb64fdc8285971fd33e3938

The github owner and repo names can resolved like this:

    $ curl -sS https://golang.org/x/net | grep go-source
    <meta name="go-source" content="golang.org/x/net https://github.com/golang/net/ https://github.com/golang/net/tree/master{/dir} https://github.com/golang/net/blob/master{/dir}/{file}#L{line}">

I've found that `go get` parses a similar meta element named "go-import"
here:

    https://github.com/golang/go/blob/master/src/cmd/go/internal/get/discovery.go
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[Michael Orlitzky]

On 9/16/19 10:17 AM, William Hubbs wrote:
> +
> +# @FUNCTION: go-module_pkg_postinst
> +# @DESCRIPTION:
> +# Display a warning about security updates for Go programs.
> +go-module_pkg_postinst() {
> +	ewarn "${PN} is written in the Go programming language."
> +	ewarn "Since this language is statically linked, security"
> +	ewarn "updates will be handled in individual packages and will be"
> +	ewarn "difficult for us to track as a distribution."
> +	ewarn "For this reason, please update any go packages asap when new"
> +	ewarn "versions enter the tree or go stable if you are running the"
> +	ewarn "stable tree."
> +}
> +
> +fi
> 

This word salad is 100% misinformation that gets tangled in itself
trying to apologize for what we're about to do:

  * Go is not a "statically linked language." There's gccgo, and as Alec
    pointed out, the official compiler has supported dynamic linking for
    years now.

  * Updating DOES NOT HELP AT ALL. That's the whole problem. You're
    trying to make it sound like we haven't thrown people under a bus,
    but saying "for this reason, please update..." is just misleading.

Here's what it should say:

  WARNING: due to a lack of manpower/interest, Go packages on Gentoo
  are statically linked. Contrary to our existing policies and what
  the website says, Go packages will never receive any security updates
  on Gentoo. Use at your own risk!

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 1874 bytes --]

On Wed, Sep 18, 2019 at 10:50 AM Michael Orlitzky <mjo@gentoo.org> wrote:

> On 9/16/19 10:17 AM, William Hubbs wrote:
> > +
> > +# @FUNCTION: go-module_pkg_postinst
> > +# @DESCRIPTION:
> > +# Display a warning about security updates for Go programs.
> > +go-module_pkg_postinst() {
> > +     ewarn "${PN} is written in the Go programming language."
> > +     ewarn "Since this language is statically linked, security"
> > +     ewarn "updates will be handled in individual packages and will be"
> > +     ewarn "difficult for us to track as a distribution."
> > +     ewarn "For this reason, please update any go packages asap when
> new"
> > +     ewarn "versions enter the tree or go stable if you are running the"
> > +     ewarn "stable tree."
> > +}
> > +
> > +fi
> >
>
> This word salad is 100% misinformation that gets tangled in itself
> trying to apologize for what we're about to do:
>
>   * Go is not a "statically linked language." There's gccgo, and as Alec
>     pointed out, the official compiler has supported dynamic linking for
>     years now.
>

I'm actually pretty fine with this wording, upstream has said not to
dynamically link in these use cases.


>
>   * Updating DOES NOT HELP AT ALL. That's the whole problem. You're
>     trying to make it sound like we haven't thrown people under a bus,
>     but saying "for this reason, please update..." is just misleading.
>
> Here's what it should say:
>
>   WARNING: due to a lack of manpower/interest, Go packages on Gentoo
>   are statically linked. Contrary to our existing policies and what
>   the website says, Go packages will never receive any security updates
>   on Gentoo. Use at your own risk!


So if the package *maintainer* bumps each package every time it, or a dep
has a security issue; then updating will work fine.
I'm skeptical go maintainers are volunteering for this though.

-A

[-- Attachment #2: Type: text/html, Size: 2722 bytes --]

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[Michael Orlitzky]

On 9/18/19 2:04 PM, Alec Warner wrote:
> 
> I'm actually pretty fine with this wording, upstream has said not to
> dynamically link in these use cases.
>  

Respectfully, the fact that you're OK with it doesn't make it not BS. It
reads like "there's no way we can fix this!" when really it means "we
don't feel like doing this properly!"

Upstreams suggest dumb stuff all the time. We fix it. That's, like, what
we do here.


> 
> So if the package *maintainer* bumps each package every time it, or a
> dep has a security issue; then updating will work fine.
> 

Simply not true. If there's a security problem in a dependency and if
you bump the packages that depend on it... nothing happens. Everyone
reinstalls the vulnerable dependency, because the vulnerable dependency
is bundled in every single one of those packages.

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 2547 bytes --]

On 9/18/19 11:04 AM, Alec Warner wrote:
> 
> 
> On Wed, Sep 18, 2019 at 10:50 AM Michael Orlitzky <mjo@gentoo.org
> <mailto:mjo@gentoo.org>> wrote:
> 
>     On 9/16/19 10:17 AM, William Hubbs wrote:
>     > +
>     > +# @FUNCTION: go-module_pkg_postinst
>     > +# @DESCRIPTION:
>     > +# Display a warning about security updates for Go programs.
>     > +go-module_pkg_postinst() {
>     > +     ewarn "${PN} is written in the Go programming language."
>     > +     ewarn "Since this language is statically linked, security"
>     > +     ewarn "updates will be handled in individual packages and
>     will be"
>     > +     ewarn "difficult for us to track as a distribution."
>     > +     ewarn "For this reason, please update any go packages asap
>     when new"
>     > +     ewarn "versions enter the tree or go stable if you are
>     running the"
>     > +     ewarn "stable tree."
>     > +}
>     > +
>     > +fi
>     >
> 
>     This word salad is 100% misinformation that gets tangled in itself
>     trying to apologize for what we're about to do:
> 
>       * Go is not a "statically linked language." There's gccgo, and as Alec
>         pointed out, the official compiler has supported dynamic linking for
>         years now.
> 
> 
> I'm actually pretty fine with this wording, upstream has said not to
> dynamically link in these use cases.
>  
> 
> 
>       * Updating DOES NOT HELP AT ALL. That's the whole problem. You're
>         trying to make it sound like we haven't thrown people under a bus,
>         but saying "for this reason, please update..." is just misleading.
> 
>     Here's what it should say:
> 
>       WARNING: due to a lack of manpower/interest, Go packages on Gentoo
>       are statically linked. Contrary to our existing policies and what
>       the website says, Go packages will never receive any security updates
>       on Gentoo. Use at your own risk!
> 
> 
> So if the package *maintainer* bumps each package every time it, or a
> dep has a security issue; then updating will work fine.
> I'm skeptical go maintainers are volunteering for this though.

There's a script here which helps to automate refresh of commit hashes
in EGO_VENDOR:

    https://github.com/hsoft/gentoo-ego-vendor-update

Just now I've used it to refresh vendored dependencies in
net-misc/drive:

    https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=3993b893d4788beaad945bc82df0f4efd91ce697
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 1878 bytes --]

On Wed, Sep 18, 2019 at 12:15 PM Michael Orlitzky <mjo@gentoo.org> wrote:

> On 9/18/19 2:04 PM, Alec Warner wrote:
> >
> > I'm actually pretty fine with this wording, upstream has said not to
> > dynamically link in these use cases.
> >
>
> Respectfully, the fact that you're OK with it doesn't make it not BS. It
> reads like "there's no way we can fix this!" when really it means "we
> don't feel like doing this properly!"
>
> Upstreams suggest dumb stuff all the time. We fix it. That's, like, what
> we do here.
>

> >
> > So if the package *maintainer* bumps each package every time it, or a
> > dep has a security issue; then updating will work fine.
> >
>
> Simply not true. If there's a security problem in a dependency and if
> you bump the packages that depend on it... nothing happens. Everyone
> reinstalls the vulnerable dependency, because the vulnerable dependency

is bundled in every single one of those packages.
>


I think the problem I have with this conversation is that I am discussing
things that are technically possible (e.g. we can in fact propagate
security fixes to all go packages, same as dynamically linked packages)
with things we do not think we will do.

If A deps on B and B has a sec vuln we can modify A's go.mod files to
depend on B-next (with security fixes), vendor that in, and bump A.

We don't do this, not because it's not possible, but because it's expensive
and people don't want to do it. The benefit of such a discussion is that
when we don't do this work, we can describe it to end users and say "hey
this is what it takes to run these packages securely, Gentoo has chosen not
to do it, but if you want to use these packages here is the work necessary."

I think that presents a better message than "Upstream is crap" or "these
packages are crap but we are forced to carry them for $reason so use them
at your own risk!".

-A

[-- Attachment #2: Type: text/html, Size: 2786 bytes --]

[gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[William Hubbs]

Signed-off-by: William Hubbs <williamh@gentoo.org>
---
 eclass/go-module.eclass | 161 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 161 insertions(+)
 create mode 100644 eclass/go-module.eclass

diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
new file mode 100644
index 00000000000..6f609e94542
--- /dev/null
+++ b/eclass/go-module.eclass
@@ -0,0 +1,161 @@
+# Copyright 2019 gentoo authors
+# Distributed under the terms of the GNU General Public License v2
+
+# @ECLASS: go-module.eclass
+# @MAINTAINER:
+# William Hubbs <williamh@gentoo.org>
+# @SUPPORTED_EAPIS: 7
+# @BLURB: basic eclass for building software written in the go
+# programming language that uses go modules.
+# @DESCRIPTION:
+# This eclass provides some basic things needed by all software
+# written in the go programming language that uses go modules.
+#
+# You will know the software you are packaging uses modules because
+# it will have files named go.sum and go.mod in its top-level source
+# directory. If it does not have these files, use the golang-* eclasses.
+#
+# This eclass provides a src_unpack function which unpacks the 
+# first tarball mentioned in SRC_URI to the usual location and unpacks
+# any modules mentioned in EGO_VENDOR to ${S}/vendor if upstream doesn't
+# vendor its own dependencies.
+#
+# If upstream vendors its dependencies, setting EGO_VENDOR is an error.
+#
+# Please note that this eclass currently handles only tarballs
+# (.tar.gz), but support for more formats may be added in the future.
+#
+# Since Go programs are statically linked, it is important that your ebuild's
+# LICENSE= setting includes the licenses of all statically linked
+# dependencies. So please make sure it is accurate.
+#
+# @EXAMPLE:
+#
+# @CODE
+# EGO_VENDOR=(
+#	"github.com/xenolf/lego 6cac0ea7d8b28c889f709ec7fa92e92b82f490dd"
+# "golang.org/x/crypto 453249f01cfeb54c3d549ddb75ff152ca243f9d8 github.com/golang/crypto"
+# )
+#
+# inherit go-module
+#
+# SRC_URI="https://github.com/example/${PN}/archive/v${PV}.tar.gz -> ${P}.tar.gz
+# ${EGO_VENDOR_URI}"
+# @CODE
+#
+# The above example will extract the tarball to ${S} and
+# extract the contents of EGO_VENDOR to ${S}/vendor.
+
+case ${EAPI:-0} in
+	7) ;;
+	*) die "${ECLASS} API in EAPI ${EAPI} not yet established."
+esac
+
+if [[ -z ${_GO_MODULE} ]]; then
+
+_GO_MODULE=1
+
+BDEPEND=">=dev-lang/go-1.12"
+
+# Force go to build in module mode.
+# In this mode the GOPATH environment variable is ignored.
+# this will become the default in the future.
+export GO111MODULE=on
+
+# The following go flags should be used for all builds.
+# -mod=vendor stopps downloading of dependencies from the internet.
+# -v prints the names of packages as they are compiled
+# -x prints commands as they are executed
+export GOFLAGS="-mod=vendor -v -x"
+
+# Do not complain about CFLAGS etc since go projects do not use them.
+QA_FLAGS_IGNORED='.*'
+
+# Go packages should not be stripped with strip(1).
+RESTRICT="strip"
+
+EXPORT_FUNCTIONS src_unpack pkg_postinst
+
+# @ECLASS-VARIABLE: EGO_VENDOR
+# @DESCRIPTION:
+# This variable contains a list of vendored packages.
+# The items of this array are strings that contain the
+# import path and the git commit hash for a vendored package.
+# If the import path does not start with github.com, the third argument
+# can be used to point to a github repository.
+
+declare -arg EGO_VENDOR
+
+_go-module_set_vendor_uri() {
+	EGO_VENDOR_URI=
+	local lib
+	for lib in "${EGO_VENDOR[@]}"; do
+		lib=(${lib})
+		if [[ -n ${lib[2]} ]]; then
+			EGO_VENDOR_URI+=" https://${lib[2]}/archive/${lib[1]}.tar.gz -> ${lib[2]//\//-}-${lib[1]}.tar.gz"
+		else
+			EGO_VENDOR_URI+=" https://${lib[0]}/archive/${lib[1]}.tar.gz -> ${lib[0]//\//-}-${lib[1]}.tar.gz"
+		fi
+	done
+	readonly EGO_VENDOR_URI
+}
+
+_go-module_set_vendor_uri
+unset -f _go-module_set_vendor_uri
+
+_go-module_dovendor() {
+	local VENDOR_PATH=$1 VENDORPN=$2 TARBALL=$3
+	rm -fr "${VENDOR_PATH}/${VENDORPN}" || die
+	mkdir -p "${VENDOR_PATH}/${VENDORPN}" || die
+	tar -C "${VENDOR_PATH}/${VENDORPN}" -x --strip-components 1\
+		-f "${DISTDIR}/${TARBALL}" || die
+}
+
+# @FUNCTION: go-module_src_unpack
+# @DESCRIPTION:
+# Extract the first archive from ${A} to ${S}, then extract
+# any sources mentioned in ${EGO_VENDOR} to ${S}/vendor.
+go-module_src_unpack() {
+	local lib vendor_path x
+	set -- ${A}
+	x="$1"
+	mkdir -p "${S}" || die
+	tar -C "${S}/" -x --strip-components 1 \
+		-f "${DISTDIR}/${x}" || die
+
+	if [[ -n "${EGO_VENDOR}" ]]; then
+		vendor_path="${S}/vendor"
+		if [[ -d "${vendor_path}" ]]; then
+			eerror "Upstream for ${P}.ebuild vendors dependencies."
+			die "Remove EGO_VENDOR from the ebuild."
+		fi
+		mkdir -p "${vendor_path}" || die
+		for lib in "${EGO_VENDOR[@]}"; do
+			lib=(${lib})
+			if [[ -n ${lib[2]} ]]; then
+				einfo "Vendoring ${lib[0]} ${lib[2]//\//-}-${lib[1]}.tar.gz"
+				_go-module_dovendor "${vendor_path}" ${lib[0]} \
+					${lib[2]//\//-}-${lib[1]}.tar.gz
+			else
+				einfo "Vendoring ${lib[0]} ${lib[0]//\//-}-${lib[1]}.tar.gz"
+				_go-module_dovendor "${vendor_path}" ${lib[0]} \
+				${lib[0]//\//-}-${lib[1]}.tar.gz
+			fi
+		done
+	fi
+}
+
+# @FUNCTION: go-module_pkg_postinst
+# @DESCRIPTION:
+# Display a warning about security updates for Go programs.
+go-module_pkg_postinst() {
+	ewarn "${PN} is written in the Go programming language."
+	ewarn "Since this language is statically linked, security"
+	ewarn "updates will be handled in individual packages and will be"
+	ewarn "difficult for us to track as a distribution."
+	ewarn "For this reason, please update any go packages asap when new"
+	ewarn "versions enter the tree or go stable if you are running the"
+	ewarn "stable tree."
+}
+
+fi
-- 
2.21.0

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 6498 bytes --]

On Wed, 2019-09-18 at 15:26 -0500, William Hubbs wrote:
> Signed-off-by: William Hubbs <williamh@gentoo.org>
> ---
>  eclass/go-module.eclass | 161 ++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 161 insertions(+)
>  create mode 100644 eclass/go-module.eclass
> 
> diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
> new file mode 100644
> index 00000000000..6f609e94542
> --- /dev/null
> +++ b/eclass/go-module.eclass
> @@ -0,0 +1,161 @@
> +# Copyright 2019 gentoo authors
> +# Distributed under the terms of the GNU General Public License v2
> +
> +# @ECLASS: go-module.eclass
> +# @MAINTAINER:
> +# William Hubbs <williamh@gentoo.org>
> +# @SUPPORTED_EAPIS: 7
> +# @BLURB: basic eclass for building software written in the go
> +# programming language that uses go modules.
> +# @DESCRIPTION:
> +# This eclass provides some basic things needed by all software
> +# written in the go programming language that uses go modules.
> +#
> +# You will know the software you are packaging uses modules because
> +# it will have files named go.sum and go.mod in its top-level source
> +# directory. If it does not have these files, use the golang-* eclasses.
> +#
> +# This eclass provides a src_unpack function which unpacks the 
> +# first tarball mentioned in SRC_URI to the usual location and unpacks
> +# any modules mentioned in EGO_VENDOR to ${S}/vendor if upstream doesn't
> +# vendor its own dependencies.
> +#
> +# If upstream vendors its dependencies, setting EGO_VENDOR is an error.
> +#
> +# Please note that this eclass currently handles only tarballs
> +# (.tar.gz), but support for more formats may be added in the future.
> +#
> +# Since Go programs are statically linked, it is important that your ebuild's
> +# LICENSE= setting includes the licenses of all statically linked
> +# dependencies. So please make sure it is accurate.
> +#
> +# @EXAMPLE:
> +#
> +# @CODE
> +# EGO_VENDOR=(
> +#	"github.com/xenolf/lego 6cac0ea7d8b28c889f709ec7fa92e92b82f490dd"
> +# "golang.org/x/crypto 453249f01cfeb54c3d549ddb75ff152ca243f9d8 github.com/golang/crypto"
> +# )
> +#
> +# inherit go-module
> +#
> +# SRC_URI="https://github.com/example/${PN}/archive/v${PV}.tar.gz -> ${P}.tar.gz
> +# ${EGO_VENDOR_URI}"
> +# @CODE
> +#
> +# The above example will extract the tarball to ${S} and
> +# extract the contents of EGO_VENDOR to ${S}/vendor.
> +
> +case ${EAPI:-0} in
> +	7) ;;
> +	*) die "${ECLASS} API in EAPI ${EAPI} not yet established."
> +esac
> +
> +if [[ -z ${_GO_MODULE} ]]; then
> +
> +_GO_MODULE=1
> +
> +BDEPEND=">=dev-lang/go-1.12"
> +
> +# Force go to build in module mode.
> +# In this mode the GOPATH environment variable is ignored.
> +# this will become the default in the future.
> +export GO111MODULE=on
> +
> +# The following go flags should be used for all builds.
> +# -mod=vendor stopps downloading of dependencies from the internet.
> +# -v prints the names of packages as they are compiled
> +# -x prints commands as they are executed
> +export GOFLAGS="-mod=vendor -v -x"
> +
> +# Do not complain about CFLAGS etc since go projects do not use them.
> +QA_FLAGS_IGNORED='.*'
> +
> +# Go packages should not be stripped with strip(1).
> +RESTRICT="strip"
> +
> +EXPORT_FUNCTIONS src_unpack pkg_postinst
> +
> +# @ECLASS-VARIABLE: EGO_VENDOR
> +# @DESCRIPTION:
> +# This variable contains a list of vendored packages.
> +# The items of this array are strings that contain the
> +# import path and the git commit hash for a vendored package.
> +# If the import path does not start with github.com, the third argument
> +# can be used to point to a github repository.
> +
> +declare -arg EGO_VENDOR
> +
> +_go-module_set_vendor_uri() {
> +	EGO_VENDOR_URI=
> +	local lib
> +	for lib in "${EGO_VENDOR[@]}"; do
> +		lib=(${lib})
> +		if [[ -n ${lib[2]} ]]; then
> +			EGO_VENDOR_URI+=" https://${lib[2]}/archive/${lib[1]}.tar.gz -> ${lib[2]//\//-}-${lib[1]}.tar.gz"
> +		else
> +			EGO_VENDOR_URI+=" https://${lib[0]}/archive/${lib[1]}.tar.gz -> ${lib[0]//\//-}-${lib[1]}.tar.gz"
> +		fi
> +	done
> +	readonly EGO_VENDOR_URI
> +}
> +
> +_go-module_set_vendor_uri
> +unset -f _go-module_set_vendor_uri
> +
> +_go-module_dovendor() {
> +	local VENDOR_PATH=$1 VENDORPN=$2 TARBALL=$3
> +	rm -fr "${VENDOR_PATH}/${VENDORPN}" || die
> +	mkdir -p "${VENDOR_PATH}/${VENDORPN}" || die
> +	tar -C "${VENDOR_PATH}/${VENDORPN}" -x --strip-components 1\
> +		-f "${DISTDIR}/${TARBALL}" || die
> +}
> +
> +# @FUNCTION: go-module_src_unpack
> +# @DESCRIPTION:
> +# Extract the first archive from ${A} to ${S}, then extract
> +# any sources mentioned in ${EGO_VENDOR} to ${S}/vendor.
> +go-module_src_unpack() {
> +	local lib vendor_path x
> +	set -- ${A}
> +	x="$1"
> +	mkdir -p "${S}" || die
> +	tar -C "${S}/" -x --strip-components 1 \
> +		-f "${DISTDIR}/${x}" || die
> +
> +	if [[ -n "${EGO_VENDOR}" ]]; then
> +		vendor_path="${S}/vendor"
> +		if [[ -d "${vendor_path}" ]]; then
> +			eerror "Upstream for ${P}.ebuild vendors dependencies."
> +			die "Remove EGO_VENDOR from the ebuild."
> +		fi
> +		mkdir -p "${vendor_path}" || die
> +		for lib in "${EGO_VENDOR[@]}"; do
> +			lib=(${lib})
> +			if [[ -n ${lib[2]} ]]; then
> +				einfo "Vendoring ${lib[0]} ${lib[2]//\//-}-${lib[1]}.tar.gz"
> +				_go-module_dovendor "${vendor_path}" ${lib[0]} \
> +					${lib[2]//\//-}-${lib[1]}.tar.gz
> +			else
> +				einfo "Vendoring ${lib[0]} ${lib[0]//\//-}-${lib[1]}.tar.gz"
> +				_go-module_dovendor "${vendor_path}" ${lib[0]} \
> +				${lib[0]//\//-}-${lib[1]}.tar.gz
> +			fi
> +		done
> +	fi
> +}
> +
> +# @FUNCTION: go-module_pkg_postinst
> +# @DESCRIPTION:
> +# Display a warning about security updates for Go programs.
> +go-module_pkg_postinst() {
> +	ewarn "${PN} is written in the Go programming language."
> +	ewarn "Since this language is statically linked, security"
> +	ewarn "updates will be handled in individual packages and will be"
> +	ewarn "difficult for us to track as a distribution."
> +	ewarn "For this reason, please update any go packages asap when new"
> +	ewarn "versions enter the tree or go stable if you are running the"
> +	ewarn "stable tree."

I don't really understand why you're adding it to *this* eclass.  Isn't
it true for all Go packages?  So I suppose golang-* eclasses are
affected as well.

> +}
> +
> +fi

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 2857 bytes --]

On Wed, Sep 18, 2019 at 12:28:29PM -0700, Zac Medico wrote:
> On 9/18/19 11:04 AM, Alec Warner wrote:
> > 
> > 
> > On Wed, Sep 18, 2019 at 10:50 AM Michael Orlitzky <mjo@gentoo.org
> > <mailto:mjo@gentoo.org>> wrote:
> > 
> >     On 9/16/19 10:17 AM, William Hubbs wrote:
> >     > +
> >     > +# @FUNCTION: go-module_pkg_postinst
> >     > +# @DESCRIPTION:
> >     > +# Display a warning about security updates for Go programs.
> >     > +go-module_pkg_postinst() {
> >     > +     ewarn "${PN} is written in the Go programming language."
> >     > +     ewarn "Since this language is statically linked, security"
> >     > +     ewarn "updates will be handled in individual packages and
> >     will be"
> >     > +     ewarn "difficult for us to track as a distribution."
> >     > +     ewarn "For this reason, please update any go packages asap
> >     when new"
> >     > +     ewarn "versions enter the tree or go stable if you are
> >     running the"
> >     > +     ewarn "stable tree."
> >     > +}
> >     > +
> >     > +fi
> >     >
> > 
> >     This word salad is 100% misinformation that gets tangled in itself
> >     trying to apologize for what we're about to do:
> > 
> >       * Go is not a "statically linked language." There's gccgo, and as Alec
> >         pointed out, the official compiler has supported dynamic linking for
> >         years now.
> > 
> > 
> > I'm actually pretty fine with this wording, upstream has said not to
> > dynamically link in these use cases.
> >  
> > 
> > 
> >       * Updating DOES NOT HELP AT ALL. That's the whole problem. You're
> >         trying to make it sound like we haven't thrown people under a bus,
> >         but saying "for this reason, please update..." is just misleading.
> > 
> >     Here's what it should say:
> > 
> >       WARNING: due to a lack of manpower/interest, Go packages on Gentoo
> >       are statically linked. Contrary to our existing policies and what
> >       the website says, Go packages will never receive any security updates
> >       on Gentoo. Use at your own risk!
> > 
> > 
> > So if the package *maintainer* bumps each package every time it, or a
> > dep has a security issue; then updating will work fine.
> > I'm skeptical go maintainers are volunteering for this though.
> 
> There's a script here which helps to automate refresh of commit hashes
> in EGO_VENDOR:
> 
>     https://github.com/hsoft/gentoo-ego-vendor-update
> 
> Just now I've used it to refresh vendored dependencies in
> net-misc/drive:
> 
>     https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=3993b893d4788beaad945bc82df0f4efd91ce697

I have seen that script, and it really doesn't work for modules. it
would need to parse go.mod and grab the dependencies based on the
information in that file.

William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 7224 bytes --]

On Wed, Sep 18, 2019 at 10:29:12PM +0200, Michał Górny wrote:
> On Wed, 2019-09-18 at 15:26 -0500, William Hubbs wrote:
> > Signed-off-by: William Hubbs <williamh@gentoo.org>
> > ---
> >  eclass/go-module.eclass | 161 ++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 161 insertions(+)
> >  create mode 100644 eclass/go-module.eclass
> > 
> > diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
> > new file mode 100644
> > index 00000000000..6f609e94542
> > --- /dev/null
> > +++ b/eclass/go-module.eclass
> > @@ -0,0 +1,161 @@
> > +# Copyright 2019 gentoo authors
> > +# Distributed under the terms of the GNU General Public License v2
> > +
> > +# @ECLASS: go-module.eclass
> > +# @MAINTAINER:
> > +# William Hubbs <williamh@gentoo.org>
> > +# @SUPPORTED_EAPIS: 7
> > +# @BLURB: basic eclass for building software written in the go
> > +# programming language that uses go modules.
> > +# @DESCRIPTION:
> > +# This eclass provides some basic things needed by all software
> > +# written in the go programming language that uses go modules.
> > +#
> > +# You will know the software you are packaging uses modules because
> > +# it will have files named go.sum and go.mod in its top-level source
> > +# directory. If it does not have these files, use the golang-* eclasses.
> > +#
> > +# This eclass provides a src_unpack function which unpacks the 
> > +# first tarball mentioned in SRC_URI to the usual location and unpacks
> > +# any modules mentioned in EGO_VENDOR to ${S}/vendor if upstream doesn't
> > +# vendor its own dependencies.
> > +#
> > +# If upstream vendors its dependencies, setting EGO_VENDOR is an error.
> > +#
> > +# Please note that this eclass currently handles only tarballs
> > +# (.tar.gz), but support for more formats may be added in the future.
> > +#
> > +# Since Go programs are statically linked, it is important that your ebuild's
> > +# LICENSE= setting includes the licenses of all statically linked
> > +# dependencies. So please make sure it is accurate.
> > +#
> > +# @EXAMPLE:
> > +#
> > +# @CODE
> > +# EGO_VENDOR=(
> > +#	"github.com/xenolf/lego 6cac0ea7d8b28c889f709ec7fa92e92b82f490dd"
> > +# "golang.org/x/crypto 453249f01cfeb54c3d549ddb75ff152ca243f9d8 github.com/golang/crypto"
> > +# )
> > +#
> > +# inherit go-module
> > +#
> > +# SRC_URI="https://github.com/example/${PN}/archive/v${PV}.tar.gz -> ${P}.tar.gz
> > +# ${EGO_VENDOR_URI}"
> > +# @CODE
> > +#
> > +# The above example will extract the tarball to ${S} and
> > +# extract the contents of EGO_VENDOR to ${S}/vendor.
> > +
> > +case ${EAPI:-0} in
> > +	7) ;;
> > +	*) die "${ECLASS} API in EAPI ${EAPI} not yet established."
> > +esac
> > +
> > +if [[ -z ${_GO_MODULE} ]]; then
> > +
> > +_GO_MODULE=1
> > +
> > +BDEPEND=">=dev-lang/go-1.12"
> > +
> > +# Force go to build in module mode.
> > +# In this mode the GOPATH environment variable is ignored.
> > +# this will become the default in the future.
> > +export GO111MODULE=on
> > +
> > +# The following go flags should be used for all builds.
> > +# -mod=vendor stopps downloading of dependencies from the internet.
> > +# -v prints the names of packages as they are compiled
> > +# -x prints commands as they are executed
> > +export GOFLAGS="-mod=vendor -v -x"
> > +
> > +# Do not complain about CFLAGS etc since go projects do not use them.
> > +QA_FLAGS_IGNORED='.*'
> > +
> > +# Go packages should not be stripped with strip(1).
> > +RESTRICT="strip"
> > +
> > +EXPORT_FUNCTIONS src_unpack pkg_postinst
> > +
> > +# @ECLASS-VARIABLE: EGO_VENDOR
> > +# @DESCRIPTION:
> > +# This variable contains a list of vendored packages.
> > +# The items of this array are strings that contain the
> > +# import path and the git commit hash for a vendored package.
> > +# If the import path does not start with github.com, the third argument
> > +# can be used to point to a github repository.
> > +
> > +declare -arg EGO_VENDOR
> > +
> > +_go-module_set_vendor_uri() {
> > +	EGO_VENDOR_URI=
> > +	local lib
> > +	for lib in "${EGO_VENDOR[@]}"; do
> > +		lib=(${lib})
> > +		if [[ -n ${lib[2]} ]]; then
> > +			EGO_VENDOR_URI+=" https://${lib[2]}/archive/${lib[1]}.tar.gz -> ${lib[2]//\//-}-${lib[1]}.tar.gz"
> > +		else
> > +			EGO_VENDOR_URI+=" https://${lib[0]}/archive/${lib[1]}.tar.gz -> ${lib[0]//\//-}-${lib[1]}.tar.gz"
> > +		fi
> > +	done
> > +	readonly EGO_VENDOR_URI
> > +}
> > +
> > +_go-module_set_vendor_uri
> > +unset -f _go-module_set_vendor_uri
> > +
> > +_go-module_dovendor() {
> > +	local VENDOR_PATH=$1 VENDORPN=$2 TARBALL=$3
> > +	rm -fr "${VENDOR_PATH}/${VENDORPN}" || die
> > +	mkdir -p "${VENDOR_PATH}/${VENDORPN}" || die
> > +	tar -C "${VENDOR_PATH}/${VENDORPN}" -x --strip-components 1\
> > +		-f "${DISTDIR}/${TARBALL}" || die
> > +}
> > +
> > +# @FUNCTION: go-module_src_unpack
> > +# @DESCRIPTION:
> > +# Extract the first archive from ${A} to ${S}, then extract
> > +# any sources mentioned in ${EGO_VENDOR} to ${S}/vendor.
> > +go-module_src_unpack() {
> > +	local lib vendor_path x
> > +	set -- ${A}
> > +	x="$1"
> > +	mkdir -p "${S}" || die
> > +	tar -C "${S}/" -x --strip-components 1 \
> > +		-f "${DISTDIR}/${x}" || die
> > +
> > +	if [[ -n "${EGO_VENDOR}" ]]; then
> > +		vendor_path="${S}/vendor"
> > +		if [[ -d "${vendor_path}" ]]; then
> > +			eerror "Upstream for ${P}.ebuild vendors dependencies."
> > +			die "Remove EGO_VENDOR from the ebuild."
> > +		fi
> > +		mkdir -p "${vendor_path}" || die
> > +		for lib in "${EGO_VENDOR[@]}"; do
> > +			lib=(${lib})
> > +			if [[ -n ${lib[2]} ]]; then
> > +				einfo "Vendoring ${lib[0]} ${lib[2]//\//-}-${lib[1]}.tar.gz"
> > +				_go-module_dovendor "${vendor_path}" ${lib[0]} \
> > +					${lib[2]//\//-}-${lib[1]}.tar.gz
> > +			else
> > +				einfo "Vendoring ${lib[0]} ${lib[0]//\//-}-${lib[1]}.tar.gz"
> > +				_go-module_dovendor "${vendor_path}" ${lib[0]} \
> > +				${lib[0]//\//-}-${lib[1]}.tar.gz
> > +			fi
> > +		done
> > +	fi
> > +}
> > +
> > +# @FUNCTION: go-module_pkg_postinst
> > +# @DESCRIPTION:
> > +# Display a warning about security updates for Go programs.
> > +go-module_pkg_postinst() {
> > +	ewarn "${PN} is written in the Go programming language."
> > +	ewarn "Since this language is statically linked, security"
> > +	ewarn "updates will be handled in individual packages and will be"
> > +	ewarn "difficult for us to track as a distribution."
> > +	ewarn "For this reason, please update any go packages asap when new"
> > +	ewarn "versions enter the tree or go stable if you are running the"
> > +	ewarn "stable tree."
> 
> I don't really understand why you're adding it to *this* eclass.  Isn't
> it true for all Go packages?  So I suppose golang-* eclasses are
> affected as well.

You are correct, they are affected. No one, including myself, caught
that during the review cycle for those eclasses.
I'm not sure of a way we can go back and fix that without breaking
compatibility.

The goal here is that the go upstream world will move to modules and
eventually we will be able to move off of and get rid of the golang-*
eclasses.

William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[Michael Orlitzky]

On 9/18/19 5:28 PM, William Hubbs wrote:
>>
>> I don't really understand why you're adding it to *this* eclass.  Isn't
>> it true for all Go packages?  So I suppose golang-* eclasses are
>> affected as well.
> 
> You are correct, they are affected. No one, including myself, caught
> that during the review cycle for those eclasses.

I've been bitching about this for at least four years:

https://archives.gentoo.org/gentoo-dev/message/2e89a3ccf9ceb4b61418d0ce30939f92

Re: [gentoo-dev] [PATCH 1/1] go-module.eclass: introduce new eclass to handle go modules

[Michael Orlitzky]

On 9/18/19 3:33 PM, Alec Warner wrote:
> 
> I think the problem I have with this conversation is that I am
> discussing things that are technically possible (e.g. we can in fact
> propagate security fixes to all go packages, same as dynamically linked
> packages) with things we do not think we will do.
> 
> If A deps on B and B has a sec vuln we can modify A's go.mod files to
> depend on B-next (with security fixes), vendor that in, and bump A.
> 

How does the Gentoo maintainer find out that there's a security
vulnerability in a dependency that was statically linked onto my system
when that dependency was specified in a text file using a commit hash in
a tarball in SRC_URI?

Without an answer to that question, even calling it "technically
possible" is disingenuous.


> We don't do this, not because it's not possible, but because it's
> expensive and people don't want to do it. The benefit of such a
> discussion is that when we don't do this work, we can describe it to end
> users and say "hey this is what it takes to run these packages securely,
> Gentoo has chosen not to do it, but if you want to use these packages
> here is the work necessary."

And the message in the patch says none of that. Instead, it tries to
shift the blame to upstream and lies to you about how to fix it (there
is no way to fix it).