[gentoo-dev] [PATCH 0/2] introduce new eclasses to handle go modules

[gentoo-dev] [PATCH 0/2] introduce new eclasses to handle go modules

[William Hubbs]

All,

after some further discussion, I decided to split the eclass.
The go-module eclass in this thread would be used if your upstream
project vendors its dependencies and the go-module-vendor eclass would
be used otherwise.

Thanks,

William

William Hubbs (2):
  go-module.eclass: new eclass for go modules
  go-module-vendor.eclass: new eclass for go modules that do not vendor

 eclass/go-module-vendor.eclass | 133 +++++++++++++++++++++++++++++++++
 eclass/go-module.eclass        |  79 ++++++++++++++++++++
 2 files changed, 212 insertions(+)
 create mode 100644 eclass/go-module-vendor.eclass
 create mode 100644 eclass/go-module.eclass

-- 
2.21.0

[gentoo-dev] [PATCH 1/2] go-module.eclass: new eclass for go modules

[William Hubbs]

This eclass includes the basic settings and a pkg_postinst function for
go modules.

Signed-off-by: William Hubbs <williamh@gentoo.org>
---
 eclass/go-module.eclass | 79 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 79 insertions(+)
 create mode 100644 eclass/go-module.eclass

diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
new file mode 100644
index 00000000000..0c2e072bd78
--- /dev/null
+++ b/eclass/go-module.eclass
@@ -0,0 +1,79 @@
+# Copyright 2019 gentoo authors
+# Distributed under the terms of the GNU General Public License v2
+
+# @ECLASS: go-module.eclass
+# @MAINTAINER:
+# William Hubbs <williamh@gentoo.org>
+# @SUPPORTED_EAPIS: 7
+# @BLURB: basic eclass for building software written in the go
+# programming language that uses go modules.
+# @DESCRIPTION:
+# This eclass provides some basic settings and a pkg_postinst function
+# needed by all software written in the go programming language that uses
+# go modules.
+#
+# You will know the software you are packaging uses modules because
+# it will have files named go.sum and go.mod in its top-level source
+# directory. If it does not have these files, use the golang-* eclasses.
+#
+# If it has these two files and a directory named vendor at the top
+# level, your ebuild should inherit this eclass since upstream is vendoring
+# their dependencies. If it does not have a vendor directory, your ebuild
+# should inherit the go-module-vendor eclass.
+#
+# Since Go programs are statically linked, it is important that your ebuild's
+# LICENSE= setting includes the licenses of all statically linked
+# dependencies. So please make sure it is accurate.
+#
+# @EXAMPLE:
+#
+# @CODE
+#
+# inherit go-module
+#
+# @CODE
+
+case ${EAPI:-0} in
+	7) ;;
+	*) die "${ECLASS} API in EAPI ${EAPI} not yet established."
+esac
+
+if [[ -z ${_GO_MODULE} ]]; then
+
+_GO_MODULE_=1
+
+BDEPEND=">=dev-lang/go-1.12"
+
+# Force go to build in module mode.
+# In this mode the GOPATH environment variable is ignored.
+# this will become the default in the future.
+export GO111MODULE=on
+
+# The following go flags should be used for all builds.
+# -mod=vendor stopps downloading of dependencies from the internet.
+# -v prints the names of packages as they are compiled
+# -x prints commands as they are executed
+export GOFLAGS="-mod=vendor -v -x"
+
+# Do not complain about CFLAGS etc since go projects do not use them.
+QA_FLAGS_IGNORED='.*'
+
+# Go packages should not be stripped with strip(1).
+RESTRICT="strip"
+
+EXPORT_FUNCTIONS pkg_postinst
+
+# @FUNCTION: go-module_pkg_postinst
+# @DESCRIPTION:
+# Display a warning about security updates for Go programs.
+go-module_pkg_postinst() {
+	ewarn "${PN} is written in the Go programming language."
+	ewarn "Since this language is statically linked, security"
+	ewarn "updates will be handled in individual packages and will be"
+	ewarn "difficult for us to track as a distribution."
+	ewarn "For this reason, please update any go packages asap when new"
+	ewarn "versions enter the tree or go stable if you are running the"
+	ewarn "stable tree."
+}
+
+fi
-- 
2.21.0

[gentoo-dev] [PATCH 2/2] go-module-vendor.eclass: new eclass for go modules that do not vendor

[William Hubbs]

This eclass adds a src_unpack function that supports the EGO_VENDOR
variable for vendoring modules.
---
 eclass/go-module-vendor.eclass | 133 +++++++++++++++++++++++++++++++++
 1 file changed, 133 insertions(+)
 create mode 100644 eclass/go-module-vendor.eclass

diff --git a/eclass/go-module-vendor.eclass b/eclass/go-module-vendor.eclass
new file mode 100644
index 00000000000..af9007df411
--- /dev/null
+++ b/eclass/go-module-vendor.eclass
@@ -0,0 +1,133 @@
+# Copyright 2019 gentoo authors
+# Distributed under the terms of the GNU General Public License v2
+
+# @ECLASS: go-module-vendor.eclass
+# @MAINTAINER:
+# William Hubbs <williamh@gentoo.org>
+# @SUPPORTED_EAPIS: 7
+# @BLURB: Eclass for building software written in the go
+# programming language that uses go modules and does not vendor.
+# @DESCRIPTION:
+# This eclass provides a src_unpack function which supports vendoring
+# dependencies for software written in the go programming language that
+# uses go modules.
+#
+# You will know the software you are packaging uses modules because
+# it will have files named go.sum and go.mod in its top-level source
+# directory. If it does not have these files, use the golang-* eclasses.
+#
+# If there is also a directory named vendor in the top level source directory
+# of your package, use the golang-module eclass instead of this one.
+#
+# This eclass provides a src_unpack function which unpacks the 
+# first tarball mentioned in SRC_URI to the usual location and unpacks
+# any modules mentioned in EGO_VENDOR to ${S}/vendor.
+#
+# Please note that this eclass currently handles only tarballs
+# (.tar.gz), but support for more formats may be added in the future.
+#
+# Since Go programs are statically linked, it is important that your ebuild's
+# LICENSE= setting includes the licenses of all statically linked
+# dependencies. So please make sure it is accurate.
+#
+# @EXAMPLE:
+#
+# @CODE
+# EGO_VENDOR=(
+#	"github.com/xenolf/lego 6cac0ea7d8b28c889f709ec7fa92e92b82f490dd"
+# "golang.org/x/crypto 453249f01cfeb54c3d549ddb75ff152ca243f9d8 github.com/golang/crypto"
+# )
+#
+# inherit go-module-vendor
+#
+# SRC_URI="https://github.com/example/${PN}/archive/v${PV}.tar.gz -> ${P}.tar.gz
+# ${EGO_VENDOR_URI}"
+# @CODE
+#
+# The above example will extract the tarball to ${S} and
+# extract the contents of EGO_VENDOR to ${S}/vendor.
+
+inherit go-module
+
+case ${EAPI:-0} in
+	7) ;;
+	*) die "${ECLASS} API in EAPI ${EAPI} not yet established."
+esac
+
+if [[ -z ${_GO_MODULE_VENDOR} ]]; then
+
+_GO_MODULE_VENDOR=1
+
+EXPORT_FUNCTIONS src_unpack
+
+# @ECLASS-VARIABLE: EGO_VENDOR
+# @DESCRIPTION:
+# This variable contains a list of vendored packages.
+# The items of this array are strings that contain the
+# import path and the git commit hash for a vendored package.
+# If the import path does not start with github.com, the third argument
+# can be used to point to a github repository.
+
+declare -arg EGO_VENDOR
+
+_go-module-vendor_set_vendor_uri() {
+	EGO_VENDOR_URI=
+	local lib
+	for lib in "${EGO_VENDOR[@]}"; do
+		lib=(${lib})
+		if [[ -n ${lib[2]} ]]; then
+			EGO_VENDOR_URI+=" https://${lib[2]}/archive/${lib[1]}.tar.gz -> ${lib[2]//\//-}-${lib[1]}.tar.gz"
+		else
+			EGO_VENDOR_URI+=" https://${lib[0]}/archive/${lib[1]}.tar.gz -> ${lib[0]//\//-}-${lib[1]}.tar.gz"
+		fi
+	done
+	readonly EGO_VENDOR_URI
+}
+
+_go-module-vendor_set_vendor_uri
+unset -f _go-module-vendor_set_vendor_uri
+
+_go-module-vendor_dovendor() {
+	local VENDOR_PATH=$1 VENDORPN=$2 TARBALL=$3
+	rm -fr "${VENDOR_PATH}/${VENDORPN}" || die
+	mkdir -p "${VENDOR_PATH}/${VENDORPN}" || die
+	tar -C "${VENDOR_PATH}/${VENDORPN}" -x --strip-components 1\
+		-f "${DISTDIR}/${TARBALL}" || die
+}
+
+# @FUNCTION: go-module-vendor_src_unpack
+# @DESCRIPTION:
+# Extract the first archive from ${A} to ${S}, then extract
+# any sources mentioned in ${EGO_VENDOR} to ${S}/vendor.
+go-module-vendor_src_unpack() {
+	local lib vendor_path x
+	set -- ${A}
+	x="$1"
+	mkdir -p "${S}" || die
+	tar -C "${S}/" -x --strip-components 1 \
+		-f "${DISTDIR}/${x}" || die
+
+	if [[ -d "${S}/vendor" ]]; then
+		eerror "Upstream for ${P}.ebuild vendors dependencies."
+		die "This ebuild should inherit go-module.eclass"
+	fi
+
+	if [[ -n "${EGO_VENDOR}" ]]; then
+		vendor_path="${S}/vendor"
+		mkdir -p "${vendor_path}" || die
+		for lib in "${EGO_VENDOR[@]}"; do
+			lib=(${lib})
+			if [[ -n ${lib[2]} ]]; then
+				einfo "Vendoring ${lib[0]} ${lib[2]//\//-}-${lib[1]}.tar.gz"
+				_go-module_dovendor "${vendor_path}" ${lib[0]} \
+					${lib[2]//\//-}-${lib[1]}.tar.gz
+			else
+				einfo "Vendoring ${lib[0]} ${lib[0]//\//-}-${lib[1]}.tar.gz"
+				_go-module_dovendor "${vendor_path}" ${lib[0]} \
+				${lib[0]//\//-}-${lib[1]}.tar.gz
+			fi
+		done
+	fi
+}
+
+fi
-- 
2.21.0

Re: [gentoo-dev] [PATCH 2/2] go-module-vendor.eclass: new eclass for go modules that do not vendor

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 6799 bytes --]

On Sat, 2019-09-21 at 17:10 -0500, William Hubbs wrote:
> This eclass adds a src_unpack function that supports the EGO_VENDOR
> variable for vendoring modules.
> ---
>  eclass/go-module-vendor.eclass | 133 +++++++++++++++++++++++++++++++++
>  1 file changed, 133 insertions(+)
>  create mode 100644 eclass/go-module-vendor.eclass
> 
> diff --git a/eclass/go-module-vendor.eclass b/eclass/go-module-vendor.eclass
> new file mode 100644
> index 00000000000..af9007df411
> --- /dev/null
> +++ b/eclass/go-module-vendor.eclass
> @@ -0,0 +1,133 @@
> +# Copyright 2019 gentoo authors
> +# Distributed under the terms of the GNU General Public License v2
> +
> +# @ECLASS: go-module-vendor.eclass
> +# @MAINTAINER:
> +# William Hubbs <williamh@gentoo.org>
> +# @SUPPORTED_EAPIS: 7
> +# @BLURB: Eclass for building software written in the go
> +# programming language that uses go modules and does not vendor.
> +# @DESCRIPTION:
> +# This eclass provides a src_unpack function which supports vendoring
> +# dependencies for software written in the go programming language that
> +# uses go modules.
> +#
> +# You will know the software you are packaging uses modules because
> +# it will have files named go.sum and go.mod in its top-level source
> +# directory. If it does not have these files, use the golang-* eclasses.
> +#
> +# If there is also a directory named vendor in the top level source directory
> +# of your package, use the golang-module eclass instead of this one.
> +#
> +# This eclass provides a src_unpack function which unpacks the 
> +# first tarball mentioned in SRC_URI to the usual location and unpacks
> +# any modules mentioned in EGO_VENDOR to ${S}/vendor.
> +#
> +# Please note that this eclass currently handles only tarballs
> +# (.tar.gz), but support for more formats may be added in the future.
> +#
> +# Since Go programs are statically linked, it is important that your ebuild's
> +# LICENSE= setting includes the licenses of all statically linked
> +# dependencies. So please make sure it is accurate.
> +#
> +# @EXAMPLE:
> +#
> +# @CODE
> +# EGO_VENDOR=(
> +#	"github.com/xenolf/lego 6cac0ea7d8b28c889f709ec7fa92e92b82f490dd"
> +# "golang.org/x/crypto 453249f01cfeb54c3d549ddb75ff152ca243f9d8 github.com/golang/crypto"
> +# )
> +#
> +# inherit go-module-vendor
> +#
> +# SRC_URI="https://github.com/example/${PN}/archive/v${PV}.tar.gz -> ${P}.tar.gz
> +# ${EGO_VENDOR_URI}"
> +# @CODE
> +#
> +# The above example will extract the tarball to ${S} and
> +# extract the contents of EGO_VENDOR to ${S}/vendor.
> +
> +inherit go-module
> +
> +case ${EAPI:-0} in
> +	7) ;;
> +	*) die "${ECLASS} API in EAPI ${EAPI} not yet established."
> +esac
> +
> +if [[ -z ${_GO_MODULE_VENDOR} ]]; then
> +
> +_GO_MODULE_VENDOR=1
> +
> +EXPORT_FUNCTIONS src_unpack
> +
> +# @ECLASS-VARIABLE: EGO_VENDOR

You want @REQUIRED here.

> +# @DESCRIPTION:
> +# This variable contains a list of vendored packages.
> +# The items of this array are strings that contain the
> +# import path and the git commit hash for a vendored package.
> +# If the import path does not start with github.com, the third argument
> +# can be used to point to a github repository.
> +
> +declare -arg EGO_VENDOR
> +
> +_go-module-vendor_set_vendor_uri() {

I think you ought to check whether EGO_VENDOR_URI is non-empty here,
and error out if it's empty.  This will be important in catching people
accidentally defining it after 'inherit'.

Another thing worth considering is to actually permit defining it
anywhere.  If you replaced EGO_VENDOR_URI with a function, user would
only need to define it prior to SRC_URI, e.g.:

  inherit go-module-vendor
  # ...

  EGO_VENDOR_URI=(...)
  SRC_URI="...
    $(ego_vendor_uri)"

But it's up to you.  I personally feel like very long lists prior to
inherit are inconvenient for the reader.

> +	EGO_VENDOR_URI=
> +	local lib
> +	for lib in "${EGO_VENDOR[@]}"; do
> +		lib=(${lib})
> +		if [[ -n ${lib[2]} ]]; then
> +			EGO_VENDOR_URI+=" https://${lib[2]}/archive/${lib[1]}.tar.gz -> ${lib[2]//\//-}-${lib[1]}.tar.gz"
> +		else
> +			EGO_VENDOR_URI+=" https://${lib[0]}/archive/${lib[1]}.tar.gz -> ${lib[0]//\//-}-${lib[1]}.tar.gz"
> +		fi
> +	done
> +	readonly EGO_VENDOR_URI
> +}
> +
> +_go-module-vendor_set_vendor_uri
> +unset -f _go-module-vendor_set_vendor_uri
> +
> +_go-module-vendor_dovendor() {
> +	local VENDOR_PATH=$1 VENDORPN=$2 TARBALL=$3

Common convention is to use lowercase names for local variables.

> +	rm -fr "${VENDOR_PATH}/${VENDORPN}" || die
> +	mkdir -p "${VENDOR_PATH}/${VENDORPN}" || die
> +	tar -C "${VENDOR_PATH}/${VENDORPN}" -x --strip-components 1\

Add space before `\`.

> +		-f "${DISTDIR}/${TARBALL}" || die
> +}
> +
> +# @FUNCTION: go-module-vendor_src_unpack
> +# @DESCRIPTION:
> +# Extract the first archive from ${A} to ${S}, then extract
> +# any sources mentioned in ${EGO_VENDOR} to ${S}/vendor.
> +go-module-vendor_src_unpack() {
> +	local lib vendor_path x
> +	set -- ${A}
> +	x="$1"
> +	mkdir -p "${S}" || die
> +	tar -C "${S}/" -x --strip-components 1 \
> +		-f "${DISTDIR}/${x}" || die

It's probably a bad idea to hardcode the 'first argument' logic.  My
suggestion would be to match all ${A} against ${EGO_VENDOR}, and unpack
those that aren't present in the latter.

> +
> +	if [[ -d "${S}/vendor" ]]; then
> +		eerror "Upstream for ${P}.ebuild vendors dependencies."
> +		die "This ebuild should inherit go-module.eclass"
> +	fi

I'm sorry but all things said, there might be a valid use case to permit
this after all -- if packager wants to upgrade vendored dependencies
e.g. to fix a security problem.

> +
> +	if [[ -n "${EGO_VENDOR}" ]]; then

Empty EGO_VENDOR should probably be an error here.

> +		vendor_path="${S}/vendor"
> +		mkdir -p "${vendor_path}" || die
> +		for lib in "${EGO_VENDOR[@]}"; do
> +			lib=(${lib})
> +			if [[ -n ${lib[2]} ]]; then

Sounds like you could simplify that by:

  [[ -z ${lib[2]} ]] && set -- "${lib[@]}" "${lib[0]}"

and then leaving only one branch of the 'if'.  You could also use
readable variable names instead of [0], [1], [2].  And then, you could
inline _go-module_dovendor since it has only one call site now and has
all nice variable names already.

> +				einfo "Vendoring ${lib[0]} ${lib[2]//\//-}-${lib[1]}.tar.gz"
> +				_go-module_dovendor "${vendor_path}" ${lib[0]} \
> +					${lib[2]//\//-}-${lib[1]}.tar.gz
> +			else
> +				einfo "Vendoring ${lib[0]} ${lib[0]//\//-}-${lib[1]}.tar.gz"
> +				_go-module_dovendor "${vendor_path}" ${lib[0]} \
> +				${lib[0]//\//-}-${lib[1]}.tar.gz
> +			fi
> +		done
> +	fi
> +}
> +
> +fi

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]