* [gentoo-dev] [PATCH 0/2] introduce new eclasses to handle go modules (round 2) @ 2019-09-23 23:36 William Hubbs 2019-09-23 23:36 ` [gentoo-dev] [PATCH 1/2] go-module.eclass: new eclass for go modules William Hubbs 2019-09-23 23:36 ` [gentoo-dev] [PATCH 2/2] go-module-vendor.eclass: new eclass for go modules that do not vendor William Hubbs 0 siblings, 2 replies; 5+ messages in thread From: William Hubbs @ 2019-09-23 23:36 UTC (permalink / raw To: gentoo-dev; +Cc: William Hubbs *** BLURB HERE *** All, This is my second iteration of the split eclasses I am working on for Go modules. There was no feedback on go-module.eclass from the last thread, so it hasn't been changed. Patch 2 encorporates the changes to go-module-vendor.eclass suggested in the last thread. Thanks, William William Hubbs (2): go-module.eclass: new eclass for go modules go-module-vendor.eclass: new eclass for go modules that do not vendor eclass/go-module-vendor.eclass | 124 +++++++++++++++++++++++++++++++++ eclass/go-module.eclass | 79 +++++++++++++++++++++ 2 files changed, 203 insertions(+) create mode 100644 eclass/go-module-vendor.eclass create mode 100644 eclass/go-module.eclass -- 2.21.0 ^ permalink raw reply [flat|nested] 5+ messages in thread
* [gentoo-dev] [PATCH 1/2] go-module.eclass: new eclass for go modules 2019-09-23 23:36 [gentoo-dev] [PATCH 0/2] introduce new eclasses to handle go modules (round 2) William Hubbs @ 2019-09-23 23:36 ` William Hubbs 2019-09-23 23:36 ` [gentoo-dev] [PATCH 2/2] go-module-vendor.eclass: new eclass for go modules that do not vendor William Hubbs 1 sibling, 0 replies; 5+ messages in thread From: William Hubbs @ 2019-09-23 23:36 UTC (permalink / raw To: gentoo-dev; +Cc: William Hubbs This eclass includes the basic settings and a pkg_postinst function for go modules. Signed-off-by: William Hubbs <williamh@gentoo.org> --- eclass/go-module.eclass | 79 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100644 eclass/go-module.eclass diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass new file mode 100644 index 00000000000..0c2e072bd78 --- /dev/null +++ b/eclass/go-module.eclass @@ -0,0 +1,79 @@ +# Copyright 2019 gentoo authors +# Distributed under the terms of the GNU General Public License v2 + +# @ECLASS: go-module.eclass +# @MAINTAINER: +# William Hubbs <williamh@gentoo.org> +# @SUPPORTED_EAPIS: 7 +# @BLURB: basic eclass for building software written in the go +# programming language that uses go modules. +# @DESCRIPTION: +# This eclass provides some basic settings and a pkg_postinst function +# needed by all software written in the go programming language that uses +# go modules. +# +# You will know the software you are packaging uses modules because +# it will have files named go.sum and go.mod in its top-level source +# directory. If it does not have these files, use the golang-* eclasses. +# +# If it has these two files and a directory named vendor at the top +# level, your ebuild should inherit this eclass since upstream is vendoring +# their dependencies. If it does not have a vendor directory, your ebuild +# should inherit the go-module-vendor eclass. +# +# Since Go programs are statically linked, it is important that your ebuild's +# LICENSE= setting includes the licenses of all statically linked +# dependencies. So please make sure it is accurate. +# +# @EXAMPLE: +# +# @CODE +# +# inherit go-module +# +# @CODE + +case ${EAPI:-0} in + 7) ;; + *) die "${ECLASS} API in EAPI ${EAPI} not yet established." +esac + +if [[ -z ${_GO_MODULE} ]]; then + +_GO_MODULE_=1 + +BDEPEND=">=dev-lang/go-1.12" + +# Force go to build in module mode. +# In this mode the GOPATH environment variable is ignored. +# this will become the default in the future. +export GO111MODULE=on + +# The following go flags should be used for all builds. +# -mod=vendor stopps downloading of dependencies from the internet. +# -v prints the names of packages as they are compiled +# -x prints commands as they are executed +export GOFLAGS="-mod=vendor -v -x" + +# Do not complain about CFLAGS etc since go projects do not use them. +QA_FLAGS_IGNORED='.*' + +# Go packages should not be stripped with strip(1). +RESTRICT="strip" + +EXPORT_FUNCTIONS pkg_postinst + +# @FUNCTION: go-module_pkg_postinst +# @DESCRIPTION: +# Display a warning about security updates for Go programs. +go-module_pkg_postinst() { + ewarn "${PN} is written in the Go programming language." + ewarn "Since this language is statically linked, security" + ewarn "updates will be handled in individual packages and will be" + ewarn "difficult for us to track as a distribution." + ewarn "For this reason, please update any go packages asap when new" + ewarn "versions enter the tree or go stable if you are running the" + ewarn "stable tree." +} + +fi -- 2.21.0 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [gentoo-dev] [PATCH 2/2] go-module-vendor.eclass: new eclass for go modules that do not vendor 2019-09-23 23:36 [gentoo-dev] [PATCH 0/2] introduce new eclasses to handle go modules (round 2) William Hubbs 2019-09-23 23:36 ` [gentoo-dev] [PATCH 1/2] go-module.eclass: new eclass for go modules William Hubbs @ 2019-09-23 23:36 ` William Hubbs 2019-09-24 0:30 ` William Hubbs 1 sibling, 1 reply; 5+ messages in thread From: William Hubbs @ 2019-09-23 23:36 UTC (permalink / raw To: gentoo-dev; +Cc: William Hubbs This eclass adds a src_unpack function that supports the EGO_VENDOR variable for vendoring modules. --- eclass/go-module-vendor.eclass | 124 +++++++++++++++++++++++++++++++++ 1 file changed, 124 insertions(+) create mode 100644 eclass/go-module-vendor.eclass diff --git a/eclass/go-module-vendor.eclass b/eclass/go-module-vendor.eclass new file mode 100644 index 00000000000..ceb362d89ba --- /dev/null +++ b/eclass/go-module-vendor.eclass @@ -0,0 +1,124 @@ +# Copyright 2019 gentoo authors +# Distributed under the terms of the GNU General Public License v2 + +# @ECLASS: go-module-vendor.eclass +# @MAINTAINER: +# William Hubbs <williamh@gentoo.org> +# @SUPPORTED_EAPIS: 7 +# @BLURB: Eclass for building software written in the go +# programming language that uses go modules and does not vendor. +# @DESCRIPTION: +# This eclass provides a src_unpack function which supports vendoring +# dependencies for software written in the go programming language that +# uses go modules. +# +# You will know the software you are packaging uses modules because +# it will have files named go.sum and go.mod in its top-level source +# directory. If it does not have these files, use the golang-* eclasses. +# +# If there is also a directory named vendor in the top level source directory +# of your package, use the golang-module eclass instead of this one. +# +# This eclass provides a src_unpack function which unpacks the +# first tarball mentioned in SRC_URI to ${S} and unpacks any modules +# mentioned in EGO_VENDOR to ${S}/vendor. +# +# Please note that this eclass currently handles only tarballs +# (.tar.gz), but support for more formats may be added in the future. +# +# Since Go programs are statically linked, it is important that your ebuild's +# LICENSE= setting includes the licenses of all statically linked +# dependencies. So please make sure it is accurate. +# +# @EXAMPLE: +# +# @CODE +# inherit go-module-vendor +# +# EGO_VENDOR=( +# "github.com/xenolf/lego 6cac0ea7d8b28c889f709ec7fa92e92b82f490dd" +# "golang.org/x/crypto 453249f01cfeb54c3d549ddb75ff152ca243f9d8 github.com/golang/crypto" +# ) +# +# SRC_URI="https://github.com/example/${PN}/archive/v${PV}.tar.gz -> ${P}.tar.gz +# $(go-module-vendor_get_vendor_uri)" +# @CODE + +inherit go-module + +case ${EAPI:-0} in + 7) ;; + *) die "${ECLASS} API in EAPI ${EAPI} not yet established." +esac + +if [[ -z ${_GO_MODULE_VENDOR} ]]; then + +_GO_MODULE_VENDOR=1 + +EXPORT_FUNCTIONS src_unpack + +# @ECLASS-VARIABLE: EGO_VENDOR +# @REQUIRED +# @DESCRIPTION: +# This variable contains a list of vendored packages. +# The items of this array are strings that contain the +# import path and the git commit hash for a vendored package. +# If the import path does not start with github.com, the third argument +# can be used to point to a github repository. + +# @FUNCTION: go-module-vendor_get_vendor_uri +# @DESCRIPTION: +# Convert the information in EGO_VENDOR to a format suitable for +# SRC_URI. +# A call to this function should be added to SRC_URI in your ebuild. +go-module-vendor_get_vendor_uri() { + local hash import line repo + for line in "${EGO_VENDOR[@]}"; do + read -r import hash repo _ <<< "${line}" + if [[ -n ${repo} ]]; then + echo "https://${repo}/archive/${hash}.tar.gz -> ${repo//\//-}-${hash}.tar.gz" + else + echo "https://${import}/archive/${hash}.tar.gz -> ${import//\//-}-${hash}.tar.gz" + fi + done +} + +# @FUNCTION: go-module-vendor_src_unpack +# @DESCRIPTION: +# Extract all archives in ${a} which are not nentioned in ${EGO_VENDOR} +# to their usual locations then extract all archives mentioned in +# ${EGO_VENDOR} to ${S}/vendor. +go-module-vendor_src_unpack() { + local f hash import line repo tarball vendor_uri + if [[ -z "${EGO_VENDOR}" ]]; then + die "EGO_VENDOR is not defined" + fi + + vendor_uri="$(go-module-vendor_get_vendor_uri)" + for f in $A; do + [[ $vendor_uri == *"$f"* ]] && continue + unpack $f + done + + if [[ -d "${S}/vendor" ]]; then + eerror "Upstream for ${P}.ebuild vendors dependencies." + die "This ebuild should inherit go-module.eclass" + fi + + mkdir -p "${S}/vendor" || die + for line in "${EGO_VENDOR[@]}"; do + read -r import hash repo _ <<< "${line}" + if [[ -n ${repo} ]]; then + tarball=${repo//\//-}-${hash}.tar.gz + else + tarball=${import//\//-}-${hash}.tar.gz + fi + einfo "Vendoring ${import} ${tarball}" + rm -fr "${S}/vendor/${import}" || die + mkdir -p "${S}/vendor/${import}" || die + tar -C "${S}/vendor/${import}" -x --strip-components 1\ + -f "${DISTDIR}/${tarball}" || die + done +} + +fi -- 2.21.0 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [gentoo-dev] [PATCH 2/2] go-module-vendor.eclass: new eclass for go modules that do not vendor 2019-09-23 23:36 ` [gentoo-dev] [PATCH 2/2] go-module-vendor.eclass: new eclass for go modules that do not vendor William Hubbs @ 2019-09-24 0:30 ` William Hubbs 2019-09-24 17:49 ` Zac Medico 0 siblings, 1 reply; 5+ messages in thread From: William Hubbs @ 2019-09-24 0:30 UTC (permalink / raw To: gentoo-dev; +Cc: mgorny [-- Attachment #1: Type: text/plain, Size: 5383 bytes --] On Mon, Sep 23, 2019 at 06:36:49PM -0500, William Hubbs wrote: > This eclass adds a src_unpack function that supports the EGO_VENDOR > variable for vendoring modules. > --- > eclass/go-module-vendor.eclass | 124 +++++++++++++++++++++++++++++++++ > 1 file changed, 124 insertions(+) > create mode 100644 eclass/go-module-vendor.eclass > > diff --git a/eclass/go-module-vendor.eclass b/eclass/go-module-vendor.eclass > new file mode 100644 > index 00000000000..ceb362d89ba > --- /dev/null > +++ b/eclass/go-module-vendor.eclass > @@ -0,0 +1,124 @@ > +# Copyright 2019 gentoo authors > +# Distributed under the terms of the GNU General Public License v2 > + > +# @ECLASS: go-module-vendor.eclass > +# @MAINTAINER: > +# William Hubbs <williamh@gentoo.org> > +# @SUPPORTED_EAPIS: 7 > +# @BLURB: Eclass for building software written in the go > +# programming language that uses go modules and does not vendor. > +# @DESCRIPTION: > +# This eclass provides a src_unpack function which supports vendoring > +# dependencies for software written in the go programming language that > +# uses go modules. > +# > +# You will know the software you are packaging uses modules because > +# it will have files named go.sum and go.mod in its top-level source > +# directory. If it does not have these files, use the golang-* eclasses. > +# > +# If there is also a directory named vendor in the top level source directory > +# of your package, use the golang-module eclass instead of this one. > +# > +# This eclass provides a src_unpack function which unpacks the > +# first tarball mentioned in SRC_URI to ${S} and unpacks any modules > +# mentioned in EGO_VENDOR to ${S}/vendor. > +# > +# Please note that this eclass currently handles only tarballs > +# (.tar.gz), but support for more formats may be added in the future. > +# > +# Since Go programs are statically linked, it is important that your ebuild's > +# LICENSE= setting includes the licenses of all statically linked > +# dependencies. So please make sure it is accurate. > +# > +# @EXAMPLE: > +# > +# @CODE > +# inherit go-module-vendor > +# > +# EGO_VENDOR=( > +# "github.com/xenolf/lego 6cac0ea7d8b28c889f709ec7fa92e92b82f490dd" > +# "golang.org/x/crypto 453249f01cfeb54c3d549ddb75ff152ca243f9d8 github.com/golang/crypto" > +# ) > +# > +# SRC_URI="https://github.com/example/${PN}/archive/v${PV}.tar.gz -> ${P}.tar.gz > +# $(go-module-vendor_get_vendor_uri)" > +# @CODE > + > +inherit go-module > + > +case ${EAPI:-0} in > + 7) ;; > + *) die "${ECLASS} API in EAPI ${EAPI} not yet established." > +esac > + > +if [[ -z ${_GO_MODULE_VENDOR} ]]; then > + > +_GO_MODULE_VENDOR=1 > + > +EXPORT_FUNCTIONS src_unpack > + > +# @ECLASS-VARIABLE: EGO_VENDOR > +# @REQUIRED > +# @DESCRIPTION: > +# This variable contains a list of vendored packages. > +# The items of this array are strings that contain the > +# import path and the git commit hash for a vendored package. > +# If the import path does not start with github.com, the third argument > +# can be used to point to a github repository. > + > +# @FUNCTION: go-module-vendor_get_vendor_uri > +# @DESCRIPTION: > +# Convert the information in EGO_VENDOR to a format suitable for > +# SRC_URI. > +# A call to this function should be added to SRC_URI in your ebuild. > +go-module-vendor_get_vendor_uri() { > + local hash import line repo > + for line in "${EGO_VENDOR[@]}"; do > + read -r import hash repo _ <<< "${line}" > + if [[ -n ${repo} ]]; then > + echo "https://${repo}/archive/${hash}.tar.gz -> ${repo//\//-}-${hash}.tar.gz" > + else > + echo "https://${import}/archive/${hash}.tar.gz -> ${import//\//-}-${hash}.tar.gz" > + fi > + done > +} > + > +# @FUNCTION: go-module-vendor_src_unpack > +# @DESCRIPTION: > +# Extract all archives in ${a} which are not nentioned in ${EGO_VENDOR} > +# to their usual locations then extract all archives mentioned in > +# ${EGO_VENDOR} to ${S}/vendor. > +go-module-vendor_src_unpack() { > + local f hash import line repo tarball vendor_uri > + if [[ -z "${EGO_VENDOR}" ]]; then > + die "EGO_VENDOR is not defined" > + fi > + > + vendor_uri="$(go-module-vendor_get_vendor_uri)" > + for f in $A; do > + [[ $vendor_uri == *"$f"* ]] && continue > + unpack $f > + done > + > + if [[ -d "${S}/vendor" ]]; then > + eerror "Upstream for ${P}.ebuild vendors dependencies." > + die "This ebuild should inherit go-module.eclass" > + fi All, I want to talk about the if block just above where I am writing. If the vendor directory exists after the sources are unpacked, the idea is that upstream is vendoring their dependencies and we probably don't want to mess with the contents of the vendor directory in that case. Mgorny, you suggested that there might be a valid use case for being able to insert our own dependencies even when upstream bundles them for security. Something like that is an easy enough change (deleting the if block), but I want to know more about whether this is a strong case for it. My thought is that if the issue is reported to upstream, they should do a new release after updating their vendored dependencies, so this is more an upstream thing. Thoughts? Is there a strong enough use case for messing with the bundled dependencies ourself? Thanks, William [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 195 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [gentoo-dev] [PATCH 2/2] go-module-vendor.eclass: new eclass for go modules that do not vendor 2019-09-24 0:30 ` William Hubbs @ 2019-09-24 17:49 ` Zac Medico 0 siblings, 0 replies; 5+ messages in thread From: Zac Medico @ 2019-09-24 17:49 UTC (permalink / raw To: gentoo-dev, mgorny [-- Attachment #1.1: Type: text/plain, Size: 1926 bytes --] On 9/23/19 5:30 PM, William Hubbs wrote: >> +# @FUNCTION: go-module-vendor_src_unpack >> +# @DESCRIPTION: >> +# Extract all archives in ${a} which are not nentioned in ${EGO_VENDOR} >> +# to their usual locations then extract all archives mentioned in >> +# ${EGO_VENDOR} to ${S}/vendor. >> +go-module-vendor_src_unpack() { >> + local f hash import line repo tarball vendor_uri >> + if [[ -z "${EGO_VENDOR}" ]]; then >> + die "EGO_VENDOR is not defined" >> + fi >> + >> + vendor_uri="$(go-module-vendor_get_vendor_uri)" >> + for f in $A; do >> + [[ $vendor_uri == *"$f"* ]] && continue >> + unpack $f >> + done >> + >> + if [[ -d "${S}/vendor" ]]; then >> + eerror "Upstream for ${P}.ebuild vendors dependencies." >> + die "This ebuild should inherit go-module.eclass" >> + fi > > All, > > I want to talk about the if block just above where I am writing. > > If the vendor directory exists after the sources are unpacked, the idea > is that upstream is vendoring their dependencies and we probably don't > want to mess with the contents of the vendor directory in that case. > > Mgorny, you suggested that there might be a valid use case for being > able to insert our own dependencies even when upstream bundles them for > security. Something like that is an easy enough change (deleting the if > block), but I want to know more about whether this is a strong case for > it. My thought is that if the issue is reported to upstream, they should > do a new release after updating their vendored dependencies, so this is > more an upstream thing. > > Thoughts? Is there a strong enough use case for messing with the bundled > dependencies ourself? If you feel like it would add unnecessary complexity, then it's probably fine to leave that case unsupported. The worst case is that ebuild maintainers will have to copy and modify the eclass function. -- Thanks, Zac [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 981 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-09-24 17:49 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2019-09-23 23:36 [gentoo-dev] [PATCH 0/2] introduce new eclasses to handle go modules (round 2) William Hubbs 2019-09-23 23:36 ` [gentoo-dev] [PATCH 1/2] go-module.eclass: new eclass for go modules William Hubbs 2019-09-23 23:36 ` [gentoo-dev] [PATCH 2/2] go-module-vendor.eclass: new eclass for go modules that do not vendor William Hubbs 2019-09-24 0:30 ` William Hubbs 2019-09-24 17:49 ` Zac Medico
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox