[gentoo-dev] [PATCH 0/3] add eclass to handle go modules

[gentoo-dev] [PATCH 0/3] add eclass to handle go modules

[William Hubbs]

This patch series adds an eclass to build software written in Go which
uses go modules.

Also, it converts a couple of packages I maintain to this system so that
you can see how ebuilds might look that use it.

William Hubbs (3):
  go-module.eclass: introduce new eclass to handle go modules
  app-misc/spire: migrate to go-module.eclass
  dev-vcs/hub: migrate to go-module.eclass

 app-misc/spire/spire-0.8.1.ebuild | 14 ++----
 dev-vcs/hub/hub-2.12.3.ebuild     |  6 +--
 eclass/go-module.eclass           | 76 +++++++++++++++++++++++++++++++
 3 files changed, 80 insertions(+), 16 deletions(-)
 create mode 100644 eclass/go-module.eclass

-- 
2.21.0

[gentoo-dev] [PATCH 1/3] go-module.eclass: introduce new eclass to handle go modules

[William Hubbs]

Copyright: Sony Interactive Entertainment Inc.
Signed-off-by: William Hubbs <williamh@gentoo.org>
---
 eclass/go-module.eclass | 76 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 76 insertions(+)
 create mode 100644 eclass/go-module.eclass

diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
new file mode 100644
index 00000000000..7009fcd3beb
--- /dev/null
+++ b/eclass/go-module.eclass
@@ -0,0 +1,76 @@
+# Copyright 1999-2015 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+# @ECLASS: go-module.eclass
+# @MAINTAINER:
+# William Hubbs <williamh@gentoo.org>
+# @SUPPORTED_EAPIS: 7
+# @BLURB: basic eclass for building software written in the go
+# programming language that uses go modules.
+# @DESCRIPTION:
+# This eclass provides a convenience src_prepare() phase and some basic
+# settings needed for all software written in the go programming
+# language that uses go modules.
+#
+# You will know the software you are packaging uses modules because
+# it will have files named go.sum and go.mod in its top-level source
+# directory. If it does not have these files, use the golang-* eclasses.
+#
+# If the software you are packaging uses modules, the next question is
+# whether it has a directory named "vendor" at the top-level of the source tree.
+#
+# If it doesn't, you need to create a tarball of what would be in the
+# vendor directory and mirror it locally. This is done with the
+# following commands if upstream is using a git repository:
+#
+# @CODE:
+#
+# $ cd /my/clone/of/upstream
+# $ git checkout <release>
+# $ go mod vendor
+# $ tar cvf project-version-vendor.tar.gz vendor
+#
+# @CODE:
+#
+# Other than this, all you need to do is inherit this eclass then
+# make sure  the exported src_prepare function is run.
+
+case ${EAPI:-0} in
+	7) ;;
+	*) die "${ECLASS} API in EAPI ${EAPI} not yet established."
+esac
+
+if [[ -z ${_GO_MODULE} ]]; then
+
+_GO_MODULE=1
+
+BDEPEND=">=dev-lang/go-1.12"
+
+# Do not download dependencies from the internet
+# make build output verbose by default
+export GOFLAGS="-mod=vendor -v -x"
+
+# Do not complain about CFLAGS etc since go projects do not use them.
+QA_FLAGS_IGNORED='.*'
+
+# Upstream does not support stripping go packages
+RESTRICT="strip"
+
+EXPORT_FUNCTIONS src_prepare
+
+# @FUNCTION: go-module_src_prepare
+# @DESCRIPTION:
+# Run a default src_prepare then move our provided vendor directory to
+# the appropriate spot if upstream doesn't provide a vendor directory.
+go-module_src_prepare() {
+	default
+	# Use the upstream provided vendor directory if it exists.
+	[[ -d vendor ]] && return
+	# If we are not providing a mirror of a vendor directory we created
+	# manually, return since there may be nothing to vendor.
+	[[ ! -d ../vendor ]] && return
+	# At this point, we know we are providing a vendor mirror.
+	mv ../vendor . || die "Unable to move ../vendor directory"
+}
+
+fi
-- 
2.21.0

[gentoo-dev] [PATCH 2/3] app-misc/spire: migrate to go-module.eclass

[William Hubbs]

Copyright: Sony Interactive Entertainment Inc.
Package-Manager: Portage-2.3.69, Repoman-2.3.16
Signed-off-by: William Hubbs <williamh@gentoo.org>
---
 app-misc/spire/spire-0.8.1.ebuild | 14 +++-----------
 1 file changed, 3 insertions(+), 11 deletions(-)

diff --git a/app-misc/spire/spire-0.8.1.ebuild b/app-misc/spire/spire-0.8.1.ebuild
index 552104b1bd8..5a72e6e1528 100644
--- a/app-misc/spire/spire-0.8.1.ebuild
+++ b/app-misc/spire/spire-0.8.1.ebuild
@@ -2,12 +2,12 @@
 # Distributed under the terms of the GNU General Public License v2
 
 EAPI=7
-VENDOR_URI="https://dev.gentoo.org/~williamh/dist/${P}-vendor.tar.gz"
+inherit go-module
 
 DESCRIPTION="the spiffe runtime environment"
 HOMEPAGE="https://github.com/spiffe/spire"
 SRC_URI="https://github.com/spiffe/spire/archive/${PV}.tar.gz -> ${P}.tar.gz
-	${VENDOR_URI}"
+	https://dev.gentoo.org/~williamh/dist/${P}-vendor.tar.gz"
 
 LICENSE="Apache-2.0"
 SLOT="0"
@@ -16,17 +16,9 @@ IUSE=""
 
 COMMON_DEPEND="acct-group/spire
 	acct-user/spire"
-DEPEND="${COMMON_DEPEND}
-	dev-lang/go"
+DEPEND="${COMMON_DEPEND}"
 RDEPEND="${COMMON_DEPEND}"
 
-RESTRICT="strip"
-
-src_prepare() {
-	default
-	mv ../vendor . || die "Unable to move ../vendor directory"
-}
-
 do_cmd() {
 	if [[ -z "$@" ]]; then
 		die "No arguments passed to do_cmd"
-- 
2.21.0

[gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[William Hubbs]

Copyright: Sony Interactive Entertainment Inc.
Package-Manager: Portage-2.3.69, Repoman-2.3.16
Signed-off-by: William Hubbs <williamh@gentoo.org>
---
 dev-vcs/hub/hub-2.12.3.ebuild | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/dev-vcs/hub/hub-2.12.3.ebuild b/dev-vcs/hub/hub-2.12.3.ebuild
index 5191e3388ce..652cf977598 100644
--- a/dev-vcs/hub/hub-2.12.3.ebuild
+++ b/dev-vcs/hub/hub-2.12.3.ebuild
@@ -3,7 +3,7 @@
 
 EAPI=7
 
-inherit bash-completion-r1
+inherit bash-completion-r1 go-module
 
 DESCRIPTION="Command-line wrapper for git that makes you better at GitHub"
 HOMEPAGE="https://github.com/github/hub"
@@ -13,12 +13,8 @@ LICENSE="MIT"
 SLOT="0"
 KEYWORDS="~amd64 ~x86 ~amd64-linux ~x86-linux"
 
-DEPEND=">=dev-lang/go-1.5.1:="
 RDEPEND=">=dev-vcs/git-1.7.3"
 
-QA_FLAGS_IGNORED=".*"
-RESTRICT="strip"
-
 src_compile() {
 	emake bin/hub man-pages
 }
-- 
2.21.0

Re: [gentoo-dev] [PATCH 1/3] go-module.eclass: introduce new eclass to handle go modules

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 3619 bytes --]

On Wed, 2019-09-11 at 12:21 -0500, William Hubbs wrote:
> Copyright: Sony Interactive Entertainment Inc.
> Signed-off-by: William Hubbs <williamh@gentoo.org>
> ---
>  eclass/go-module.eclass | 76 +++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 76 insertions(+)
>  create mode 100644 eclass/go-module.eclass
> 
> diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
> new file mode 100644
> index 00000000000..7009fcd3beb
> --- /dev/null
> +++ b/eclass/go-module.eclass
> @@ -0,0 +1,76 @@
> +# Copyright 1999-2015 Gentoo Foundation

You need to replace your calendar.  And copyright holder.

> +# Distributed under the terms of the GNU General Public License v2
> +
> +# @ECLASS: go-module.eclass

Any reason to change naming from golang-* to go-* now?

> +# @MAINTAINER:
> +# William Hubbs <williamh@gentoo.org>
> +# @SUPPORTED_EAPIS: 7
> +# @BLURB: basic eclass for building software written in the go
> +# programming language that uses go modules.
> +# @DESCRIPTION:
> +# This eclass provides a convenience src_prepare() phase and some basic
> +# settings needed for all software written in the go programming
> +# language that uses go modules.
> +#
> +# You will know the software you are packaging uses modules because
> +# it will have files named go.sum and go.mod in its top-level source
> +# directory. If it does not have these files, use the golang-* eclasses.
> +#
> +# If the software you are packaging uses modules, the next question is
> +# whether it has a directory named "vendor" at the top-level of the source tree.
> +#
> +# If it doesn't, you need to create a tarball of what would be in the
> +# vendor directory and mirror it locally. This is done with the
> +# following commands if upstream is using a git repository:
> +#
> +# @CODE:
> +#
> +# $ cd /my/clone/of/upstream
> +# $ git checkout <release>
> +# $ go mod vendor
> +# $ tar cvf project-version-vendor.tar.gz vendor
> +#
> +# @CODE:
> +#
> +# Other than this, all you need to do is inherit this eclass then
> +# make sure  the exported src_prepare function is run.
> +
> +case ${EAPI:-0} in
> +	7) ;;
> +	*) die "${ECLASS} API in EAPI ${EAPI} not yet established."
> +esac
> +
> +if [[ -z ${_GO_MODULE} ]]; then
> +
> +_GO_MODULE=1
> +
> +BDEPEND=">=dev-lang/go-1.12"
> +
> +# Do not download dependencies from the internet
> +# make build output verbose by default
> +export GOFLAGS="-mod=vendor -v -x"
> +
> +# Do not complain about CFLAGS etc since go projects do not use them.
> +QA_FLAGS_IGNORED='.*'
> +
> +# Upstream does not support stripping go packages
> +RESTRICT="strip"
> +
> +EXPORT_FUNCTIONS src_prepare

Don't you need to inherit some other eclass to make it build?

> +
> +# @FUNCTION: go-module_src_prepare
> +# @DESCRIPTION:
> +# Run a default src_prepare then move our provided vendor directory to
> +# the appropriate spot if upstream doesn't provide a vendor directory.
> +go-module_src_prepare() {
> +	default
> +	# Use the upstream provided vendor directory if it exists.
> +	[[ -d vendor ]] && return
> +	# If we are not providing a mirror of a vendor directory we created
> +	# manually, return since there may be nothing to vendor.
> +	[[ ! -d ../vendor ]] && return
> +	# At this point, we know we are providing a vendor mirror.
> +	mv ../vendor . || die "Unable to move ../vendor directory"

Wouldn't it be much simpler to create appropriate directory structure
in the tarball?  Then you wouldn't need a new eclass at all.

> +}
> +
> +fi

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Michael Orlitzky]

On 9/11/19 1:21 PM, William Hubbs wrote:
> +++ b/dev-vcs/hub/hub-2.12.3.ebuild
> ...
>  
> LICENSE="MIT"

This license is wrong, as it's pretty much guaranteed to be every time
you commit one of these packages. I find it pretty troubling that one
corporation is able to force this stuff through even though it's a
security and legal hazard for everyone else.

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 595 bytes --]

On Wed, Sep 11, 2019 at 01:39:32PM -0400, Michael Orlitzky wrote:
> On 9/11/19 1:21 PM, William Hubbs wrote:
> > +++ b/dev-vcs/hub/hub-2.12.3.ebuild
> > ...
> >  
> > LICENSE="MIT"
> 
> This license is wrong, as it's pretty much guaranteed to be every time
> you commit one of these packages. I find it pretty troubling that one
> corporation is able to force this stuff through even though it's a
> security and legal hazard for everyone else.

Sorry, That train already left the station with the golang-* eclasses
and there is nothing we can do about it.

Thanks,

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Michael Orlitzky]

On 9/11/19 1:47 PM, William Hubbs wrote:
> On Wed, Sep 11, 2019 at 01:39:32PM -0400, Michael Orlitzky wrote:
>> On 9/11/19 1:21 PM, William Hubbs wrote:
>>> +++ b/dev-vcs/hub/hub-2.12.3.ebuild
>>> ...
>>>  
>>> LICENSE="MIT"
>>
>> This license is wrong, as it's pretty much guaranteed to be every time
>> you commit one of these packages. I find it pretty troubling that one
>> corporation is able to force this stuff through even though it's a
>> security and legal hazard for everyone else.
> 
> Sorry, That train already left the station with the golang-* eclasses
> and there is nothing we can do about it.
> 

"We" could just stop.

Re: [gentoo-dev] [PATCH 1/3] go-module.eclass: introduce new eclass to handle go modules

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 4711 bytes --]

On Wed, Sep 11, 2019 at 07:38:17PM +0200, Michał Górny wrote:
> On Wed, 2019-09-11 at 12:21 -0500, William Hubbs wrote:
> > Copyright: Sony Interactive Entertainment Inc.
> > Signed-off-by: William Hubbs <williamh@gentoo.org>
> > ---
> >  eclass/go-module.eclass | 76 +++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 76 insertions(+)
> >  create mode 100644 eclass/go-module.eclass
> > 
> > diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
> > new file mode 100644
> > index 00000000000..7009fcd3beb
> > --- /dev/null
> > +++ b/eclass/go-module.eclass
> > @@ -0,0 +1,76 @@
> > +# Copyright 1999-2015 Gentoo Foundation
> 
> You need to replace your calendar.  And copyright holder.

Sure, I thought I ffixed that.

> > +# Distributed under the terms of the GNU General Public License v2
> > +
> > +# @ECLASS: go-module.eclass
> 
> Any reason to change naming from golang-* to go-* now?

Well, "lang" is sort of redundant, and there will be only one eclass, so
I thought I would make things a bit more simple.

> 
> > +# @MAINTAINER:
> > +# William Hubbs <williamh@gentoo.org>
> > +# @SUPPORTED_EAPIS: 7
> > +# @BLURB: basic eclass for building software written in the go
> > +# programming language that uses go modules.
> > +# @DESCRIPTION:
> > +# This eclass provides a convenience src_prepare() phase and some basic
> > +# settings needed for all software written in the go programming
> > +# language that uses go modules.
> > +#
> > +# You will know the software you are packaging uses modules because
> > +# it will have files named go.sum and go.mod in its top-level source
> > +# directory. If it does not have these files, use the golang-* eclasses.
> > +#
> > +# If the software you are packaging uses modules, the next question is
> > +# whether it has a directory named "vendor" at the top-level of the source tree.
> > +#
> > +# If it doesn't, you need to create a tarball of what would be in the
> > +# vendor directory and mirror it locally. This is done with the
> > +# following commands if upstream is using a git repository:
> > +#
> > +# @CODE:
> > +#
> > +# $ cd /my/clone/of/upstream
> > +# $ git checkout <release>
> > +# $ go mod vendor
> > +# $ tar cvf project-version-vendor.tar.gz vendor
> > +#
> > +# @CODE:
> > +#
> > +# Other than this, all you need to do is inherit this eclass then
> > +# make sure  the exported src_prepare function is run.
> > +
> > +case ${EAPI:-0} in
> > +	7) ;;
> > +	*) die "${ECLASS} API in EAPI ${EAPI} not yet established."
> > +esac
> > +
> > +if [[ -z ${_GO_MODULE} ]]; then
> > +
> > +_GO_MODULE=1
> > +
> > +BDEPEND=">=dev-lang/go-1.12"
> > +
> > +# Do not download dependencies from the internet
> > +# make build output verbose by default
> > +export GOFLAGS="-mod=vendor -v -x"
> > +
> > +# Do not complain about CFLAGS etc since go projects do not use them.
> > +QA_FLAGS_IGNORED='.*'
> > +
> > +# Upstream does not support stripping go packages
> > +RESTRICT="strip"
> > +
> > +EXPORT_FUNCTIONS src_prepare
> 
> Don't you need to inherit some other eclass to make it build?

The primary reason for all of the golang-* eclasses was the GOPATH
variable, which is not relevant when you are using modules.

I can look at adding a src_compile to this eclass, but I haven't thought
about what it would contain yet.
 
> > +
> > +# @FUNCTION: go-module_src_prepare
> > +# @DESCRIPTION:
> > +# Run a default src_prepare then move our provided vendor directory to
> > +# the appropriate spot if upstream doesn't provide a vendor directory.
> > +go-module_src_prepare() {
> > +	default
> > +	# Use the upstream provided vendor directory if it exists.
> > +	[[ -d vendor ]] && return
> > +	# If we are not providing a mirror of a vendor directory we created
> > +	# manually, return since there may be nothing to vendor.
> > +	[[ ! -d ../vendor ]] && return
> > +	# At this point, we know we are providing a vendor mirror.
> > +	mv ../vendor . || die "Unable to move ../vendor directory"
> 
> Wouldn't it be much simpler to create appropriate directory structure
> in the tarball?  Then you wouldn't need a new eclass at all.

You would definitely need an eclass (see the settings and dependencies).

Take a look at the differences in the spire and hub ebuilds in this
series. I'm not sure what you mean by adding the directory structure to
the tarball? I guess you could add something to the vendor tarball when
you create it.

What I tried to avoid was stomping on the vendor directory if it is
included upstream.

William
> > +}
> > +
> > +fi
> 
> -- 
> Best regards,
> Michał Górny
> 



[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] [PATCH 1/3] go-module.eclass: introduce new eclass to handle go modules

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 5291 bytes --]

On Wed, 2019-09-11 at 13:22 -0500, William Hubbs wrote:
> On Wed, Sep 11, 2019 at 07:38:17PM +0200, Michał Górny wrote:
> > On Wed, 2019-09-11 at 12:21 -0500, William Hubbs wrote:
> > > Copyright: Sony Interactive Entertainment Inc.
> > > Signed-off-by: William Hubbs <williamh@gentoo.org>
> > > ---
> > >  eclass/go-module.eclass | 76 +++++++++++++++++++++++++++++++++++++++++
> > >  1 file changed, 76 insertions(+)
> > >  create mode 100644 eclass/go-module.eclass
> > > 
> > > diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
> > > new file mode 100644
> > > index 00000000000..7009fcd3beb
> > > --- /dev/null
> > > +++ b/eclass/go-module.eclass
> > > @@ -0,0 +1,76 @@
> > > +# Copyright 1999-2015 Gentoo Foundation
> > 
> > You need to replace your calendar.  And copyright holder.
> 
> Sure, I thought I ffixed that.
> 
> > > +# Distributed under the terms of the GNU General Public License v2
> > > +
> > > +# @ECLASS: go-module.eclass
> > 
> > Any reason to change naming from golang-* to go-* now?
> 
> Well, "lang" is sort of redundant, and there will be only one eclass, so
> I thought I would make things a bit more simple.
> 
> > > +# @MAINTAINER:
> > > +# William Hubbs <williamh@gentoo.org>
> > > +# @SUPPORTED_EAPIS: 7
> > > +# @BLURB: basic eclass for building software written in the go
> > > +# programming language that uses go modules.
> > > +# @DESCRIPTION:
> > > +# This eclass provides a convenience src_prepare() phase and some basic
> > > +# settings needed for all software written in the go programming
> > > +# language that uses go modules.
> > > +#
> > > +# You will know the software you are packaging uses modules because
> > > +# it will have files named go.sum and go.mod in its top-level source
> > > +# directory. If it does not have these files, use the golang-* eclasses.
> > > +#
> > > +# If the software you are packaging uses modules, the next question is
> > > +# whether it has a directory named "vendor" at the top-level of the source tree.
> > > +#
> > > +# If it doesn't, you need to create a tarball of what would be in the
> > > +# vendor directory and mirror it locally. This is done with the
> > > +# following commands if upstream is using a git repository:
> > > +#
> > > +# @CODE:
> > > +#
> > > +# $ cd /my/clone/of/upstream
> > > +# $ git checkout <release>
> > > +# $ go mod vendor
> > > +# $ tar cvf project-version-vendor.tar.gz vendor
> > > +#
> > > +# @CODE:
> > > +#
> > > +# Other than this, all you need to do is inherit this eclass then
> > > +# make sure  the exported src_prepare function is run.
> > > +
> > > +case ${EAPI:-0} in
> > > +	7) ;;
> > > +	*) die "${ECLASS} API in EAPI ${EAPI} not yet established."
> > > +esac
> > > +
> > > +if [[ -z ${_GO_MODULE} ]]; then
> > > +
> > > +_GO_MODULE=1
> > > +
> > > +BDEPEND=">=dev-lang/go-1.12"
> > > +
> > > +# Do not download dependencies from the internet
> > > +# make build output verbose by default
> > > +export GOFLAGS="-mod=vendor -v -x"
> > > +
> > > +# Do not complain about CFLAGS etc since go projects do not use them.
> > > +QA_FLAGS_IGNORED='.*'
> > > +
> > > +# Upstream does not support stripping go packages
> > > +RESTRICT="strip"
> > > +
> > > +EXPORT_FUNCTIONS src_prepare
> > 
> > Don't you need to inherit some other eclass to make it build?
> 
> The primary reason for all of the golang-* eclasses was the GOPATH
> variable, which is not relevant when you are using modules.
> 
> I can look at adding a src_compile to this eclass, but I haven't thought
> about what it would contain yet.
>  
> > > +
> > > +# @FUNCTION: go-module_src_prepare
> > > +# @DESCRIPTION:
> > > +# Run a default src_prepare then move our provided vendor directory to
> > > +# the appropriate spot if upstream doesn't provide a vendor directory.
> > > +go-module_src_prepare() {
> > > +	default
> > > +	# Use the upstream provided vendor directory if it exists.
> > > +	[[ -d vendor ]] && return
> > > +	# If we are not providing a mirror of a vendor directory we created
> > > +	# manually, return since there may be nothing to vendor.
> > > +	[[ ! -d ../vendor ]] && return
> > > +	# At this point, we know we are providing a vendor mirror.
> > > +	mv ../vendor . || die "Unable to move ../vendor directory"
> > 
> > Wouldn't it be much simpler to create appropriate directory structure
> > in the tarball?  Then you wouldn't need a new eclass at all.
> 
> You would definitely need an eclass (see the settings and dependencies).
> 
> Take a look at the differences in the spire and hub ebuilds in this
> series. I'm not sure what you mean by adding the directory structure to
> the tarball? I guess you could add something to the vendor tarball when
> you create it.

I mean packing it as 'spire-1.2.3/vendor' or whatever the package
directory is, so that it extracts correctly instead of making a tarball
that needs to be moved afterwards.

> 
> What I tried to avoid was stomping on the vendor directory if it is
> included upstream.

You do that anyway by moving files.

> 
> William
> > > +}
> > > +
> > > +fi
> > 
> > -- 
> > Best regards,
> > Michał Górny
> > 
> 
> 

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 470 bytes --]

On Wed, 11 Sep 2019 12:47:20 -0500
William Hubbs <williamh@gentoo.org> wrote:

> Sorry, That train already left the station with the golang-* eclasses
> and there is nothing we can do about it.

Saying "this creates a legal problem" followed by "eh, nothing we can
do about it, carry on" really doesn't work in reality, as far as I'm
aware.

Usually if you find yourself rationalizing around a legal problem, you
end up getting bitten by said legal problem.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 967 bytes --]

On Thu, Sep 12, 2019 at 07:15:28AM +1200, Kent Fredric wrote:
> On Wed, 11 Sep 2019 12:47:20 -0500
> William Hubbs <williamh@gentoo.org> wrote:
> 
> > Sorry, That train already left the station with the golang-* eclasses
> > and there is nothing we can do about it.
> 
> Saying "this creates a legal problem" followed by "eh, nothing we can
> do about it, carry on" really doesn't work in reality, as far as I'm
> aware.
> 
> Usually if you find yourself rationalizing around a legal problem, you
> end up getting bitten by said legal problem.

After chatting a bit more about this on IRC, let me clarify.

This is an issue that is not related to the eclass, but an
example package (patch 3 isn't the eclass, it is just an example of how
to use it). There is no LICENSE setting in the eclass, so it doesn't
affect the eclass.

What I was advised is that we are all supposed to audit our packages and
make sure LICENSE= is correct.

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] [PATCH 1/3] go-module.eclass: introduce new eclass to handle go modules

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 5708 bytes --]

On Wed, Sep 11, 2019 at 08:31:16PM +0200, Michał Górny wrote:
> On Wed, 2019-09-11 at 13:22 -0500, William Hubbs wrote:
> > On Wed, Sep 11, 2019 at 07:38:17PM +0200, Michał Górny wrote:
> > > On Wed, 2019-09-11 at 12:21 -0500, William Hubbs wrote:
> > > > Copyright: Sony Interactive Entertainment Inc.
> > > > Signed-off-by: William Hubbs <williamh@gentoo.org>
> > > > ---
> > > >  eclass/go-module.eclass | 76 +++++++++++++++++++++++++++++++++++++++++
> > > >  1 file changed, 76 insertions(+)
> > > >  create mode 100644 eclass/go-module.eclass
> > > > 
> > > > diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
> > > > new file mode 100644
> > > > index 00000000000..7009fcd3beb
> > > > --- /dev/null
> > > > +++ b/eclass/go-module.eclass
> > > > @@ -0,0 +1,76 @@
> > > > +# Copyright 1999-2015 Gentoo Foundation
> > > 
> > > You need to replace your calendar.  And copyright holder.
> > 
> > Sure, I thought I ffixed that.
> > 
> > > > +# Distributed under the terms of the GNU General Public License v2
> > > > +
> > > > +# @ECLASS: go-module.eclass
> > > 
> > > Any reason to change naming from golang-* to go-* now?
> > 
> > Well, "lang" is sort of redundant, and there will be only one eclass, so
> > I thought I would make things a bit more simple.
> > 
> > > > +# @MAINTAINER:
> > > > +# William Hubbs <williamh@gentoo.org>
> > > > +# @SUPPORTED_EAPIS: 7
> > > > +# @BLURB: basic eclass for building software written in the go
> > > > +# programming language that uses go modules.
> > > > +# @DESCRIPTION:
> > > > +# This eclass provides a convenience src_prepare() phase and some basic
> > > > +# settings needed for all software written in the go programming
> > > > +# language that uses go modules.
> > > > +#
> > > > +# You will know the software you are packaging uses modules because
> > > > +# it will have files named go.sum and go.mod in its top-level source
> > > > +# directory. If it does not have these files, use the golang-* eclasses.
> > > > +#
> > > > +# If the software you are packaging uses modules, the next question is
> > > > +# whether it has a directory named "vendor" at the top-level of the source tree.
> > > > +#
> > > > +# If it doesn't, you need to create a tarball of what would be in the
> > > > +# vendor directory and mirror it locally. This is done with the
> > > > +# following commands if upstream is using a git repository:
> > > > +#
> > > > +# @CODE:
> > > > +#
> > > > +# $ cd /my/clone/of/upstream
> > > > +# $ git checkout <release>
> > > > +# $ go mod vendor
> > > > +# $ tar cvf project-version-vendor.tar.gz vendor
> > > > +#
> > > > +# @CODE:
> > > > +#
> > > > +# Other than this, all you need to do is inherit this eclass then
> > > > +# make sure  the exported src_prepare function is run.
> > > > +
> > > > +case ${EAPI:-0} in
> > > > +	7) ;;
> > > > +	*) die "${ECLASS} API in EAPI ${EAPI} not yet established."
> > > > +esac
> > > > +
> > > > +if [[ -z ${_GO_MODULE} ]]; then
> > > > +
> > > > +_GO_MODULE=1
> > > > +
> > > > +BDEPEND=">=dev-lang/go-1.12"
> > > > +
> > > > +# Do not download dependencies from the internet
> > > > +# make build output verbose by default
> > > > +export GOFLAGS="-mod=vendor -v -x"
> > > > +
> > > > +# Do not complain about CFLAGS etc since go projects do not use them.
> > > > +QA_FLAGS_IGNORED='.*'
> > > > +
> > > > +# Upstream does not support stripping go packages
> > > > +RESTRICT="strip"
> > > > +
> > > > +EXPORT_FUNCTIONS src_prepare
> > > 
> > > Don't you need to inherit some other eclass to make it build?
> > 
> > The primary reason for all of the golang-* eclasses was the GOPATH
> > variable, which is not relevant when you are using modules.
> > 
> > I can look at adding a src_compile to this eclass, but I haven't thought
> > about what it would contain yet.
> >  
> > > > +
> > > > +# @FUNCTION: go-module_src_prepare
> > > > +# @DESCRIPTION:
> > > > +# Run a default src_prepare then move our provided vendor directory to
> > > > +# the appropriate spot if upstream doesn't provide a vendor directory.
> > > > +go-module_src_prepare() {
> > > > +	default
> > > > +	# Use the upstream provided vendor directory if it exists.
> > > > +	[[ -d vendor ]] && return
> > > > +	# If we are not providing a mirror of a vendor directory we created
> > > > +	# manually, return since there may be nothing to vendor.
> > > > +	[[ ! -d ../vendor ]] && return
> > > > +	# At this point, we know we are providing a vendor mirror.
> > > > +	mv ../vendor . || die "Unable to move ../vendor directory"
> > > 
> > > Wouldn't it be much simpler to create appropriate directory structure
> > > in the tarball?  Then you wouldn't need a new eclass at all.
> > 
> > You would definitely need an eclass (see the settings and dependencies).
> > 
> > Take a look at the differences in the spire and hub ebuilds in this
> > series. I'm not sure what you mean by adding the directory structure to
> > the tarball? I guess you could add something to the vendor tarball when
> > you create it.
> 
> I mean packing it as 'spire-1.2.3/vendor' or whatever the package
> directory is, so that it extracts correctly instead of making a tarball
> that needs to be moved afterwards.

That would clobber the upstream provided vendor directory and that's
what I want to avoid with the first test in src_prepare.

> 
> > 
> > What I tried to avoid was stomping on the vendor directory if it is
> > included upstream.
> 
> You do that anyway by moving files.

See the first test in src_prepare. I go out of my way to not overwrite
the upstream vendor directory.

William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] [PATCH 1/3] go-module.eclass: introduce new eclass to handle go modules

[Michał Górny]

Dnia September 11, 2019 7:40:41 PM UTC, William Hubbs <williamh@gentoo.org> napisał(a):
>On Wed, Sep 11, 2019 at 08:31:16PM +0200, Michał Górny wrote:
>> On Wed, 2019-09-11 at 13:22 -0500, William Hubbs wrote:
>> > On Wed, Sep 11, 2019 at 07:38:17PM +0200, Michał Górny wrote:
>> > > On Wed, 2019-09-11 at 12:21 -0500, William Hubbs wrote:
>> > > > Copyright: Sony Interactive Entertainment Inc.
>> > > > Signed-off-by: William Hubbs <williamh@gentoo.org>
>> > > > ---
>> > > >  eclass/go-module.eclass | 76
>+++++++++++++++++++++++++++++++++++++++++
>> > > >  1 file changed, 76 insertions(+)
>> > > >  create mode 100644 eclass/go-module.eclass
>> > > > 
>> > > > diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
>> > > > new file mode 100644
>> > > > index 00000000000..7009fcd3beb
>> > > > --- /dev/null
>> > > > +++ b/eclass/go-module.eclass
>> > > > @@ -0,0 +1,76 @@
>> > > > +# Copyright 1999-2015 Gentoo Foundation
>> > > 
>> > > You need to replace your calendar.  And copyright holder.
>> > 
>> > Sure, I thought I ffixed that.
>> > 
>> > > > +# Distributed under the terms of the GNU General Public
>License v2
>> > > > +
>> > > > +# @ECLASS: go-module.eclass
>> > > 
>> > > Any reason to change naming from golang-* to go-* now?
>> > 
>> > Well, "lang" is sort of redundant, and there will be only one
>eclass, so
>> > I thought I would make things a bit more simple.
>> > 
>> > > > +# @MAINTAINER:
>> > > > +# William Hubbs <williamh@gentoo.org>
>> > > > +# @SUPPORTED_EAPIS: 7
>> > > > +# @BLURB: basic eclass for building software written in the go
>> > > > +# programming language that uses go modules.
>> > > > +# @DESCRIPTION:
>> > > > +# This eclass provides a convenience src_prepare() phase and
>some basic
>> > > > +# settings needed for all software written in the go
>programming
>> > > > +# language that uses go modules.
>> > > > +#
>> > > > +# You will know the software you are packaging uses modules
>because
>> > > > +# it will have files named go.sum and go.mod in its top-level
>source
>> > > > +# directory. If it does not have these files, use the golang-*
>eclasses.
>> > > > +#
>> > > > +# If the software you are packaging uses modules, the next
>question is
>> > > > +# whether it has a directory named "vendor" at the top-level
>of the source tree.
>> > > > +#
>> > > > +# If it doesn't, you need to create a tarball of what would be
>in the
>> > > > +# vendor directory and mirror it locally. This is done with
>the
>> > > > +# following commands if upstream is using a git repository:
>> > > > +#
>> > > > +# @CODE:
>> > > > +#
>> > > > +# $ cd /my/clone/of/upstream
>> > > > +# $ git checkout <release>
>> > > > +# $ go mod vendor
>> > > > +# $ tar cvf project-version-vendor.tar.gz vendor
>> > > > +#
>> > > > +# @CODE:
>> > > > +#
>> > > > +# Other than this, all you need to do is inherit this eclass
>then
>> > > > +# make sure  the exported src_prepare function is run.
>> > > > +
>> > > > +case ${EAPI:-0} in
>> > > > +	7) ;;
>> > > > +	*) die "${ECLASS} API in EAPI ${EAPI} not yet established."
>> > > > +esac
>> > > > +
>> > > > +if [[ -z ${_GO_MODULE} ]]; then
>> > > > +
>> > > > +_GO_MODULE=1
>> > > > +
>> > > > +BDEPEND=">=dev-lang/go-1.12"
>> > > > +
>> > > > +# Do not download dependencies from the internet
>> > > > +# make build output verbose by default
>> > > > +export GOFLAGS="-mod=vendor -v -x"
>> > > > +
>> > > > +# Do not complain about CFLAGS etc since go projects do not
>use them.
>> > > > +QA_FLAGS_IGNORED='.*'
>> > > > +
>> > > > +# Upstream does not support stripping go packages
>> > > > +RESTRICT="strip"
>> > > > +
>> > > > +EXPORT_FUNCTIONS src_prepare
>> > > 
>> > > Don't you need to inherit some other eclass to make it build?
>> > 
>> > The primary reason for all of the golang-* eclasses was the GOPATH
>> > variable, which is not relevant when you are using modules.
>> > 
>> > I can look at adding a src_compile to this eclass, but I haven't
>thought
>> > about what it would contain yet.
>> >  
>> > > > +
>> > > > +# @FUNCTION: go-module_src_prepare
>> > > > +# @DESCRIPTION:
>> > > > +# Run a default src_prepare then move our provided vendor
>directory to
>> > > > +# the appropriate spot if upstream doesn't provide a vendor
>directory.
>> > > > +go-module_src_prepare() {
>> > > > +	default
>> > > > +	# Use the upstream provided vendor directory if it exists.
>> > > > +	[[ -d vendor ]] && return
>> > > > +	# If we are not providing a mirror of a vendor directory we
>created
>> > > > +	# manually, return since there may be nothing to vendor.
>> > > > +	[[ ! -d ../vendor ]] && return
>> > > > +	# At this point, we know we are providing a vendor mirror.
>> > > > +	mv ../vendor . || die "Unable to move ../vendor directory"
>> > > 
>> > > Wouldn't it be much simpler to create appropriate directory
>structure
>> > > in the tarball?  Then you wouldn't need a new eclass at all.
>> > 
>> > You would definitely need an eclass (see the settings and
>dependencies).
>> > 
>> > Take a look at the differences in the spire and hub ebuilds in this
>> > series. I'm not sure what you mean by adding the directory
>structure to
>> > the tarball? I guess you could add something to the vendor tarball
>when
>> > you create it.
>> 
>> I mean packing it as 'spire-1.2.3/vendor' or whatever the package
>> directory is, so that it extracts correctly instead of making a
>tarball
>> that needs to be moved afterwards.
>
>That would clobber the upstream provided vendor directory and that's
>what I want to avoid with the first test in src_prepare.

If upstream already includes vendored modules, why would you create your own tarball in the first place?

>
>> 
>> > 
>> > What I tried to avoid was stomping on the vendor directory if it is
>> > included upstream.
>> 
>> You do that anyway by moving files.
>
>See the first test in src_prepare. I go out of my way to not overwrite
>the upstream vendor directory.
>
>William


--
Best regards, 
Michał Górny

Re: [gentoo-dev] [PATCH 1/3] go-module.eclass: introduce new eclass to handle go modules

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 7188 bytes --]

On Wed, Sep 11, 2019 at 07:47:04PM +0000, Michał Górny wrote:
> Dnia September 11, 2019 7:40:41 PM UTC, William Hubbs <williamh@gentoo.org> napisał(a):
> >On Wed, Sep 11, 2019 at 08:31:16PM +0200, Michał Górny wrote:
> >> On Wed, 2019-09-11 at 13:22 -0500, William Hubbs wrote:
> >> > On Wed, Sep 11, 2019 at 07:38:17PM +0200, Michał Górny wrote:
> >> > > On Wed, 2019-09-11 at 12:21 -0500, William Hubbs wrote:
> >> > > > Copyright: Sony Interactive Entertainment Inc.
> >> > > > Signed-off-by: William Hubbs <williamh@gentoo.org>
> >> > > > ---
> >> > > >  eclass/go-module.eclass | 76
> >+++++++++++++++++++++++++++++++++++++++++
> >> > > >  1 file changed, 76 insertions(+)
> >> > > >  create mode 100644 eclass/go-module.eclass
> >> > > > 
> >> > > > diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
> >> > > > new file mode 100644
> >> > > > index 00000000000..7009fcd3beb
> >> > > > --- /dev/null
> >> > > > +++ b/eclass/go-module.eclass
> >> > > > @@ -0,0 +1,76 @@
> >> > > > +# Copyright 1999-2015 Gentoo Foundation
> >> > > 
> >> > > You need to replace your calendar.  And copyright holder.
> >> > 
> >> > Sure, I thought I ffixed that.
> >> > 
> >> > > > +# Distributed under the terms of the GNU General Public
> >License v2
> >> > > > +
> >> > > > +# @ECLASS: go-module.eclass
> >> > > 
> >> > > Any reason to change naming from golang-* to go-* now?
> >> > 
> >> > Well, "lang" is sort of redundant, and there will be only one
> >eclass, so
> >> > I thought I would make things a bit more simple.
> >> > 
> >> > > > +# @MAINTAINER:
> >> > > > +# William Hubbs <williamh@gentoo.org>
> >> > > > +# @SUPPORTED_EAPIS: 7
> >> > > > +# @BLURB: basic eclass for building software written in the go
> >> > > > +# programming language that uses go modules.
> >> > > > +# @DESCRIPTION:
> >> > > > +# This eclass provides a convenience src_prepare() phase and
> >some basic
> >> > > > +# settings needed for all software written in the go
> >programming
> >> > > > +# language that uses go modules.
> >> > > > +#
> >> > > > +# You will know the software you are packaging uses modules
> >because
> >> > > > +# it will have files named go.sum and go.mod in its top-level
> >source
> >> > > > +# directory. If it does not have these files, use the golang-*
> >eclasses.
> >> > > > +#
> >> > > > +# If the software you are packaging uses modules, the next
> >question is
> >> > > > +# whether it has a directory named "vendor" at the top-level
> >of the source tree.
> >> > > > +#
> >> > > > +# If it doesn't, you need to create a tarball of what would be
> >in the
> >> > > > +# vendor directory and mirror it locally. This is done with
> >the
> >> > > > +# following commands if upstream is using a git repository:
> >> > > > +#
> >> > > > +# @CODE:
> >> > > > +#
> >> > > > +# $ cd /my/clone/of/upstream
> >> > > > +# $ git checkout <release>
> >> > > > +# $ go mod vendor
> >> > > > +# $ tar cvf project-version-vendor.tar.gz vendor
> >> > > > +#
> >> > > > +# @CODE:
> >> > > > +#
> >> > > > +# Other than this, all you need to do is inherit this eclass
> >then
> >> > > > +# make sure  the exported src_prepare function is run.
> >> > > > +
> >> > > > +case ${EAPI:-0} in
> >> > > > +	7) ;;
> >> > > > +	*) die "${ECLASS} API in EAPI ${EAPI} not yet established."
> >> > > > +esac
> >> > > > +
> >> > > > +if [[ -z ${_GO_MODULE} ]]; then
> >> > > > +
> >> > > > +_GO_MODULE=1
> >> > > > +
> >> > > > +BDEPEND=">=dev-lang/go-1.12"
> >> > > > +
> >> > > > +# Do not download dependencies from the internet
> >> > > > +# make build output verbose by default
> >> > > > +export GOFLAGS="-mod=vendor -v -x"
> >> > > > +
> >> > > > +# Do not complain about CFLAGS etc since go projects do not
> >use them.
> >> > > > +QA_FLAGS_IGNORED='.*'
> >> > > > +
> >> > > > +# Upstream does not support stripping go packages
> >> > > > +RESTRICT="strip"
> >> > > > +
> >> > > > +EXPORT_FUNCTIONS src_prepare
> >> > > 
> >> > > Don't you need to inherit some other eclass to make it build?
> >> > 
> >> > The primary reason for all of the golang-* eclasses was the GOPATH
> >> > variable, which is not relevant when you are using modules.
> >> > 
> >> > I can look at adding a src_compile to this eclass, but I haven't
> >thought
> >> > about what it would contain yet.
> >> >  
> >> > > > +
> >> > > > +# @FUNCTION: go-module_src_prepare
> >> > > > +# @DESCRIPTION:
> >> > > > +# Run a default src_prepare then move our provided vendor
> >directory to
> >> > > > +# the appropriate spot if upstream doesn't provide a vendor
> >directory.
> >> > > > +go-module_src_prepare() {
> >> > > > +	default
> >> > > > +	# Use the upstream provided vendor directory if it exists.
> >> > > > +	[[ -d vendor ]] && return
> >> > > > +	# If we are not providing a mirror of a vendor directory we
> >created
> >> > > > +	# manually, return since there may be nothing to vendor.
> >> > > > +	[[ ! -d ../vendor ]] && return
> >> > > > +	# At this point, we know we are providing a vendor mirror.
> >> > > > +	mv ../vendor . || die "Unable to move ../vendor directory"
> >> > > 
> >> > > Wouldn't it be much simpler to create appropriate directory
> >structure
> >> > > in the tarball?  Then you wouldn't need a new eclass at all.
> >> > 
> >> > You would definitely need an eclass (see the settings and
> >dependencies).
> >> > 
> >> > Take a look at the differences in the spire and hub ebuilds in this
> >> > series. I'm not sure what you mean by adding the directory
> >structure to
> >> > the tarball? I guess you could add something to the vendor tarball
> >when
> >> > you create it.
> >> 
> >> I mean packing it as 'spire-1.2.3/vendor' or whatever the package
> >> directory is, so that it extracts correctly instead of making a
> >tarball
> >> that needs to be moved afterwards.
> >
> >That would clobber the upstream provided vendor directory and that's
> >what I want to avoid with the first test in src_prepare.
> 
> If upstream already includes vendored modules, why would you create your own tarball in the first place?
 
You are right, and currently I quietly ignore your vendor tarball if upstream
vendors the dependencies also. I could change this to generate a warning
or die and force you to fix the ebuild, but that would not be possible
if I follow your suggestion because I would not be able to tell whether
the vendored dependencies came from us or upstream.

Also, another concern about your suggestion is the  --transform switch
that would have to be added to the  tar command people use to create the
vendor tarball, something like:

tar -acvf package-version-vendor.tar.gz --transform='s#^vendor#package-version-vendor#' vendor

You suggested that a maintainer could create a new tarball and build on
top of it. I guess you mean  don't use upstream's tarball if they don't
vendor and create my own tarball and add the vendor directory to it. I'm
against that option because  I don't feel that we should manually tinker
with upstream tarballs. That opens a pretty big can of worms imo.

William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] [PATCH 1/3] go-module.eclass: introduce new eclass to handle go modules

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 3461 bytes --]

On Wed, Sep 11, 2019 at 10:28 AM William Hubbs <williamh@gentoo.org> wrote:

> Copyright: Sony Interactive Entertainment Inc.
> Signed-off-by: William Hubbs <williamh@gentoo.org>
> ---
>  eclass/go-module.eclass | 76 +++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 76 insertions(+)
>  create mode 100644 eclass/go-module.eclass
>
> diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
> new file mode 100644
> index 00000000000..7009fcd3beb
> --- /dev/null
> +++ b/eclass/go-module.eclass
> @@ -0,0 +1,76 @@
> +# Copyright 1999-2015 Gentoo Foundation
> +# Distributed under the terms of the GNU General Public License v2
> +
> +# @ECLASS: go-module.eclass
> +# @MAINTAINER:
> +# William Hubbs <williamh@gentoo.org>
> +# @SUPPORTED_EAPIS: 7
> +# @BLURB: basic eclass for building software written in the go
> +# programming language that uses go modules.
> +# @DESCRIPTION:
> +# This eclass provides a convenience src_prepare() phase and some basic
> +# settings needed for all software written in the go programming
> +# language that uses go modules.
> +#
> +# You will know the software you are packaging uses modules because
> +# it will have files named go.sum and go.mod in its top-level source
> +# directory. If it does not have these files, use the golang-* eclasses.
> +#
> +# If the software you are packaging uses modules, the next question is
> +# whether it has a directory named "vendor" at the top-level of the
> source tree.
> +#
> +# If it doesn't, you need to create a tarball of what would be in the
> +# vendor directory and mirror it locally. This is done with the
> +# following commands if upstream is using a git repository:
> +#
> +# @CODE:
> +#
> +# $ cd /my/clone/of/upstream
> +# $ git checkout <release>
> +# $ go mod vendor
> +# $ tar cvf project-version-vendor.tar.gz vendor
> +#
> +# @CODE:
> +#
> +# Other than this, all you need to do is inherit this eclass then
> +# make sure  the exported src_prepare function is run.
> +
> +case ${EAPI:-0} in
> +       7) ;;
> +       *) die "${ECLASS} API in EAPI ${EAPI} not yet established."
> +esac
> +
> +if [[ -z ${_GO_MODULE} ]]; then
> +
> +_GO_MODULE=1
> +
> +BDEPEND=">=dev-lang/go-1.12"
> +
> +# Do not download dependencies from the internet
> +# make build output verbose by default
> +export GOFLAGS="-mod=vendor -v -x"
> +
> +# Do not complain about CFLAGS etc since go projects do not use them.
> +QA_FLAGS_IGNORED='.*'
> +
> +# Upstream does not support stripping go packages
> +RESTRICT="strip"
>

https://golang.org/cmd/link/ implies you can pass -s -w to the compiler to
reduce binary size.

Does that not work in portage by default, or does upstream just consider
that bad practice?

-A


> +
> +EXPORT_FUNCTIONS src_prepare
> +
> +# @FUNCTION: go-module_src_prepare
> +# @DESCRIPTION:
> +# Run a default src_prepare then move our provided vendor directory to
> +# the appropriate spot if upstream doesn't provide a vendor directory.
> +go-module_src_prepare() {
> +       default
> +       # Use the upstream provided vendor directory if it exists.
> +       [[ -d vendor ]] && return
> +       # If we are not providing a mirror of a vendor directory we created
> +       # manually, return since there may be nothing to vendor.
> +       [[ ! -d ../vendor ]] && return
> +       # At this point, we know we are providing a vendor mirror.
> +       mv ../vendor . || die "Unable to move ../vendor directory"
> +}
> +
> +fi
> --
> 2.21.0
>
>
>

[-- Attachment #2: Type: text/html, Size: 4584 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 526 bytes --]

On Wed, Sep 11, 2019 at 10:39 AM Michael Orlitzky <mjo@gentoo.org> wrote:

> On 9/11/19 1:21 PM, William Hubbs wrote:
> > +++ b/dev-vcs/hub/hub-2.12.3.ebuild
> > ...
> >
> > LICENSE="MIT"
>
> This license is wrong, as it's pretty much guaranteed to be every time
> you commit one of these packages. I find it pretty troubling that one
> corporation is able to force this stuff through even though it's a
> security and legal hazard for everyone else.
>

How is it wrong?

https://github.com/github/hub/blob/master/LICENSE

-A

[-- Attachment #2: Type: text/html, Size: 992 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 967 bytes --]

On Wed, Sep 11, 2019 at 04:34:27PM -0700, Alec Warner wrote:
> On Wed, Sep 11, 2019 at 10:39 AM Michael Orlitzky <mjo@gentoo.org> wrote:
> 
> > On 9/11/19 1:21 PM, William Hubbs wrote:
> > > +++ b/dev-vcs/hub/hub-2.12.3.ebuild
> > > ...
> > >
> > > LICENSE="MIT"
> >
> > This license is wrong, as it's pretty much guaranteed to be every time
> > you commit one of these packages. I find it pretty troubling that one
> > corporation is able to force this stuff through even though it's a
> > security and legal hazard for everyone else.
> >
> 
> How is it wrong?
> 
> https://github.com/github/hub/blob/master/LICENSE

The argument is that because of the vendoring, LICENSE= needs to list
all licenses for the vendored dependencies that are different from MIT
as well.

Personally I don't have a comment about this, but that's what is being
pushed for. I'll let you guys debate this but it isn't really relevant
to the eclass. ;-)

William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] [PATCH 1/3] go-module.eclass: introduce new eclass to handle go modules

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 3284 bytes --]

On Wed, Sep 11, 2019 at 04:31:00PM -0700, Alec Warner wrote:
> On Wed, Sep 11, 2019 at 10:28 AM William Hubbs <williamh@gentoo.org> wrote:
> 
> > Copyright: Sony Interactive Entertainment Inc.
> > Signed-off-by: William Hubbs <williamh@gentoo.org>
> > ---
> >  eclass/go-module.eclass | 76 +++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 76 insertions(+)
> >  create mode 100644 eclass/go-module.eclass
> >
> > diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
> > new file mode 100644
> > index 00000000000..7009fcd3beb
> > --- /dev/null
> > +++ b/eclass/go-module.eclass
> > @@ -0,0 +1,76 @@
> > +# Copyright 1999-2015 Gentoo Foundation
> > +# Distributed under the terms of the GNU General Public License v2
> > +
> > +# @ECLASS: go-module.eclass
> > +# @MAINTAINER:
> > +# William Hubbs <williamh@gentoo.org>
> > +# @SUPPORTED_EAPIS: 7
> > +# @BLURB: basic eclass for building software written in the go
> > +# programming language that uses go modules.
> > +# @DESCRIPTION:
> > +# This eclass provides a convenience src_prepare() phase and some basic
> > +# settings needed for all software written in the go programming
> > +# language that uses go modules.
> > +#
> > +# You will know the software you are packaging uses modules because
> > +# it will have files named go.sum and go.mod in its top-level source
> > +# directory. If it does not have these files, use the golang-* eclasses.
> > +#
> > +# If the software you are packaging uses modules, the next question is
> > +# whether it has a directory named "vendor" at the top-level of the
> > source tree.
> > +#
> > +# If it doesn't, you need to create a tarball of what would be in the
> > +# vendor directory and mirror it locally. This is done with the
> > +# following commands if upstream is using a git repository:
> > +#
> > +# @CODE:
> > +#
> > +# $ cd /my/clone/of/upstream
> > +# $ git checkout <release>
> > +# $ go mod vendor
> > +# $ tar cvf project-version-vendor.tar.gz vendor
> > +#
> > +# @CODE:
> > +#
> > +# Other than this, all you need to do is inherit this eclass then
> > +# make sure  the exported src_prepare function is run.
> > +
> > +case ${EAPI:-0} in
> > +       7) ;;
> > +       *) die "${ECLASS} API in EAPI ${EAPI} not yet established."
> > +esac
> > +
> > +if [[ -z ${_GO_MODULE} ]]; then
> > +
> > +_GO_MODULE=1
> > +
> > +BDEPEND=">=dev-lang/go-1.12"
> > +
> > +# Do not download dependencies from the internet
> > +# make build output verbose by default
> > +export GOFLAGS="-mod=vendor -v -x"
> > +
> > +# Do not complain about CFLAGS etc since go projects do not use them.
> > +QA_FLAGS_IGNORED='.*'
> > +
> > +# Upstream does not support stripping go packages
> > +RESTRICT="strip"
> >
> 
> https://golang.org/cmd/link/ implies you can pass -s -w to the compiler to
> reduce binary size.
> 
> Does that not work in portage by default, or does upstream just consider
> that bad practice?

I haven't tried it, but here are the definitions of -s and -w.

-s	Omit the symbol table and debug information.
-w	Omit the DWARF symbol table.

These look like Go's equivalent of stripping the binaries, and I have my
doubts as to whether we should force this.

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 1415 bytes --]

On Wed, Sep 11, 2019 at 4:48 PM William Hubbs <williamh@gentoo.org> wrote:

> On Wed, Sep 11, 2019 at 04:34:27PM -0700, Alec Warner wrote:
> > On Wed, Sep 11, 2019 at 10:39 AM Michael Orlitzky <mjo@gentoo.org>
> wrote:
> >
> > > On 9/11/19 1:21 PM, William Hubbs wrote:
> > > > +++ b/dev-vcs/hub/hub-2.12.3.ebuild
> > > > ...
> > > >
> > > > LICENSE="MIT"
> > >
> > > This license is wrong, as it's pretty much guaranteed to be every time
> > > you commit one of these packages. I find it pretty troubling that one
> > > corporation is able to force this stuff through even though it's a
> > > security and legal hazard for everyone else.
> > >
> >
> > How is it wrong?
> >
> > https://github.com/github/hub/blob/master/LICENSE
>
> The argument is that because of the vendoring, LICENSE= needs to list
> all licenses for the vendored dependencies that are different from MIT
> as well.
>

I see, I tend to believe that argument in that case.


>
> Personally I don't have a comment about this, but that's what is being
> pushed for. I'll let you guys debate this but it isn't really relevant
> to the eclass. ;-)
>

I think it's difficult to put instructions in the eclass like:

+# $ cd /my/clone/of/upstream
+# $ git checkout <release>
+# $ go mod vendor
+# $ tar cvf project-version-vendor.tar.gz vendor

And then not mention this fairly easy trap (it's so easy to fall into you
did it twice.)

-A


> William
>

[-- Attachment #2: Type: text/html, Size: 2550 bytes --]

Re: [gentoo-dev] [PATCH 1/3] go-module.eclass: introduce new eclass to handle go modules

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 3722 bytes --]

On Wed, Sep 11, 2019 at 5:05 PM William Hubbs <williamh@gentoo.org> wrote:

> On Wed, Sep 11, 2019 at 04:31:00PM -0700, Alec Warner wrote:
> > On Wed, Sep 11, 2019 at 10:28 AM William Hubbs <williamh@gentoo.org>
> wrote:
> >
> > > Copyright: Sony Interactive Entertainment Inc.
> > > Signed-off-by: William Hubbs <williamh@gentoo.org>
> > > ---
> > >  eclass/go-module.eclass | 76 +++++++++++++++++++++++++++++++++++++++++
> > >  1 file changed, 76 insertions(+)
> > >  create mode 100644 eclass/go-module.eclass
> > >
> > > diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
> > > new file mode 100644
> > > index 00000000000..7009fcd3beb
> > > --- /dev/null
> > > +++ b/eclass/go-module.eclass
> > > @@ -0,0 +1,76 @@
> > > +# Copyright 1999-2015 Gentoo Foundation
> > > +# Distributed under the terms of the GNU General Public License v2
> > > +
> > > +# @ECLASS: go-module.eclass
> > > +# @MAINTAINER:
> > > +# William Hubbs <williamh@gentoo.org>
> > > +# @SUPPORTED_EAPIS: 7
> > > +# @BLURB: basic eclass for building software written in the go
> > > +# programming language that uses go modules.
> > > +# @DESCRIPTION:
> > > +# This eclass provides a convenience src_prepare() phase and some
> basic
> > > +# settings needed for all software written in the go programming
> > > +# language that uses go modules.
> > > +#
> > > +# You will know the software you are packaging uses modules because
> > > +# it will have files named go.sum and go.mod in its top-level source
> > > +# directory. If it does not have these files, use the golang-*
> eclasses.
> > > +#
> > > +# If the software you are packaging uses modules, the next question is
> > > +# whether it has a directory named "vendor" at the top-level of the
> > > source tree.
> > > +#
> > > +# If it doesn't, you need to create a tarball of what would be in the
> > > +# vendor directory and mirror it locally. This is done with the
> > > +# following commands if upstream is using a git repository:
> > > +#
> > > +# @CODE:
> > > +#
> > > +# $ cd /my/clone/of/upstream
> > > +# $ git checkout <release>
> > > +# $ go mod vendor
> > > +# $ tar cvf project-version-vendor.tar.gz vendor
> > > +#
> > > +# @CODE:
> > > +#
> > > +# Other than this, all you need to do is inherit this eclass then
> > > +# make sure  the exported src_prepare function is run.
> > > +
> > > +case ${EAPI:-0} in
> > > +       7) ;;
> > > +       *) die "${ECLASS} API in EAPI ${EAPI} not yet established."
> > > +esac
> > > +
> > > +if [[ -z ${_GO_MODULE} ]]; then
> > > +
> > > +_GO_MODULE=1
> > > +
> > > +BDEPEND=">=dev-lang/go-1.12"
> > > +
> > > +# Do not download dependencies from the internet
> > > +# make build output verbose by default
> > > +export GOFLAGS="-mod=vendor -v -x"
> > > +
> > > +# Do not complain about CFLAGS etc since go projects do not use them.
> > > +QA_FLAGS_IGNORED='.*'
> > > +
> > > +# Upstream does not support stripping go packages
> > > +RESTRICT="strip"
> > >
> >
> > https://golang.org/cmd/link/ implies you can pass -s -w to the compiler
> to
> > reduce binary size.
> >
> > Does that not work in portage by default, or does upstream just consider
> > that bad practice?
>
> I haven't tried it, but here are the definitions of -s and -w.
>
> -s      Omit the symbol table and debug information.
> -w      Omit the DWARF symbol table.
>
> These look like Go's equivalent of stripping the binaries, and I have my
> doubts as to whether we should force this.
>

I don't care if you strip or not (I'm not even sure portage knows how to do
it for go binaries) but I'm fairly sure the reason isn't because "upstream
does not support stripping go binaries" because they clearly do...unless
upstream is portage here...?


>
> William
>
>

[-- Attachment #2: Type: text/html, Size: 5313 bytes --]

Re: [gentoo-dev] [PATCH 1/3] go-module.eclass: introduce new eclass to handle go modules

[Michał Górny]

Dnia September 11, 2019 11:11:15 PM UTC, William Hubbs <williamh@gentoo.org> napisał(a):
>On Wed, Sep 11, 2019 at 07:47:04PM +0000, Michał Górny wrote:
>> Dnia September 11, 2019 7:40:41 PM UTC, William Hubbs
><williamh@gentoo.org> napisał(a):
>> >On Wed, Sep 11, 2019 at 08:31:16PM +0200, Michał Górny wrote:
>> >> On Wed, 2019-09-11 at 13:22 -0500, William Hubbs wrote:
>> >> > On Wed, Sep 11, 2019 at 07:38:17PM +0200, Michał Górny wrote:
>> >> > > On Wed, 2019-09-11 at 12:21 -0500, William Hubbs wrote:
>> >> > > > Copyright: Sony Interactive Entertainment Inc.
>> >> > > > Signed-off-by: William Hubbs <williamh@gentoo.org>
>> >> > > > ---
>> >> > > >  eclass/go-module.eclass | 76
>> >+++++++++++++++++++++++++++++++++++++++++
>> >> > > >  1 file changed, 76 insertions(+)
>> >> > > >  create mode 100644 eclass/go-module.eclass
>> >> > > > 
>> >> > > > diff --git a/eclass/go-module.eclass
>b/eclass/go-module.eclass
>> >> > > > new file mode 100644
>> >> > > > index 00000000000..7009fcd3beb
>> >> > > > --- /dev/null
>> >> > > > +++ b/eclass/go-module.eclass
>> >> > > > @@ -0,0 +1,76 @@
>> >> > > > +# Copyright 1999-2015 Gentoo Foundation
>> >> > > 
>> >> > > You need to replace your calendar.  And copyright holder.
>> >> > 
>> >> > Sure, I thought I ffixed that.
>> >> > 
>> >> > > > +# Distributed under the terms of the GNU General Public
>> >License v2
>> >> > > > +
>> >> > > > +# @ECLASS: go-module.eclass
>> >> > > 
>> >> > > Any reason to change naming from golang-* to go-* now?
>> >> > 
>> >> > Well, "lang" is sort of redundant, and there will be only one
>> >eclass, so
>> >> > I thought I would make things a bit more simple.
>> >> > 
>> >> > > > +# @MAINTAINER:
>> >> > > > +# William Hubbs <williamh@gentoo.org>
>> >> > > > +# @SUPPORTED_EAPIS: 7
>> >> > > > +# @BLURB: basic eclass for building software written in the
>go
>> >> > > > +# programming language that uses go modules.
>> >> > > > +# @DESCRIPTION:
>> >> > > > +# This eclass provides a convenience src_prepare() phase
>and
>> >some basic
>> >> > > > +# settings needed for all software written in the go
>> >programming
>> >> > > > +# language that uses go modules.
>> >> > > > +#
>> >> > > > +# You will know the software you are packaging uses modules
>> >because
>> >> > > > +# it will have files named go.sum and go.mod in its
>top-level
>> >source
>> >> > > > +# directory. If it does not have these files, use the
>golang-*
>> >eclasses.
>> >> > > > +#
>> >> > > > +# If the software you are packaging uses modules, the next
>> >question is
>> >> > > > +# whether it has a directory named "vendor" at the
>top-level
>> >of the source tree.
>> >> > > > +#
>> >> > > > +# If it doesn't, you need to create a tarball of what would
>be
>> >in the
>> >> > > > +# vendor directory and mirror it locally. This is done with
>> >the
>> >> > > > +# following commands if upstream is using a git repository:
>> >> > > > +#
>> >> > > > +# @CODE:
>> >> > > > +#
>> >> > > > +# $ cd /my/clone/of/upstream
>> >> > > > +# $ git checkout <release>
>> >> > > > +# $ go mod vendor
>> >> > > > +# $ tar cvf project-version-vendor.tar.gz vendor
>> >> > > > +#
>> >> > > > +# @CODE:
>> >> > > > +#
>> >> > > > +# Other than this, all you need to do is inherit this
>eclass
>> >then
>> >> > > > +# make sure  the exported src_prepare function is run.
>> >> > > > +
>> >> > > > +case ${EAPI:-0} in
>> >> > > > +	7) ;;
>> >> > > > +	*) die "${ECLASS} API in EAPI ${EAPI} not yet
>established."
>> >> > > > +esac
>> >> > > > +
>> >> > > > +if [[ -z ${_GO_MODULE} ]]; then
>> >> > > > +
>> >> > > > +_GO_MODULE=1
>> >> > > > +
>> >> > > > +BDEPEND=">=dev-lang/go-1.12"
>> >> > > > +
>> >> > > > +# Do not download dependencies from the internet
>> >> > > > +# make build output verbose by default
>> >> > > > +export GOFLAGS="-mod=vendor -v -x"
>> >> > > > +
>> >> > > > +# Do not complain about CFLAGS etc since go projects do not
>> >use them.
>> >> > > > +QA_FLAGS_IGNORED='.*'
>> >> > > > +
>> >> > > > +# Upstream does not support stripping go packages
>> >> > > > +RESTRICT="strip"
>> >> > > > +
>> >> > > > +EXPORT_FUNCTIONS src_prepare
>> >> > > 
>> >> > > Don't you need to inherit some other eclass to make it build?
>> >> > 
>> >> > The primary reason for all of the golang-* eclasses was the
>GOPATH
>> >> > variable, which is not relevant when you are using modules.
>> >> > 
>> >> > I can look at adding a src_compile to this eclass, but I haven't
>> >thought
>> >> > about what it would contain yet.
>> >> >  
>> >> > > > +
>> >> > > > +# @FUNCTION: go-module_src_prepare
>> >> > > > +# @DESCRIPTION:
>> >> > > > +# Run a default src_prepare then move our provided vendor
>> >directory to
>> >> > > > +# the appropriate spot if upstream doesn't provide a vendor
>> >directory.
>> >> > > > +go-module_src_prepare() {
>> >> > > > +	default
>> >> > > > +	# Use the upstream provided vendor directory if it exists.
>> >> > > > +	[[ -d vendor ]] && return
>> >> > > > +	# If we are not providing a mirror of a vendor directory
>we
>> >created
>> >> > > > +	# manually, return since there may be nothing to vendor.
>> >> > > > +	[[ ! -d ../vendor ]] && return
>> >> > > > +	# At this point, we know we are providing a vendor mirror.
>> >> > > > +	mv ../vendor . || die "Unable to move ../vendor directory"
>> >> > > 
>> >> > > Wouldn't it be much simpler to create appropriate directory
>> >structure
>> >> > > in the tarball?  Then you wouldn't need a new eclass at all.
>> >> > 
>> >> > You would definitely need an eclass (see the settings and
>> >dependencies).
>> >> > 
>> >> > Take a look at the differences in the spire and hub ebuilds in
>this
>> >> > series. I'm not sure what you mean by adding the directory
>> >structure to
>> >> > the tarball? I guess you could add something to the vendor
>tarball
>> >when
>> >> > you create it.
>> >> 
>> >> I mean packing it as 'spire-1.2.3/vendor' or whatever the package
>> >> directory is, so that it extracts correctly instead of making a
>> >tarball
>> >> that needs to be moved afterwards.
>> >
>> >That would clobber the upstream provided vendor directory and that's
>> >what I want to avoid with the first test in src_prepare.
>> 
>> If upstream already includes vendored modules, why would you create
>your own tarball in the first place?
> 
>You are right, and currently I quietly ignore your vendor tarball if
>upstream
>vendors the dependencies also. I could change this to generate a
>warning
>or die and force you to fix the ebuild, but that would not be possible
>if I follow your suggestion because I would not be able to tell whether
>the vendored dependencies came from us or upstream.

Why would anyone create a vendor tarball if things work without it? That makes no sense. Also adding unused archives to SRC_URI is a QA violation.

>
>Also, another concern about your suggestion is the  --transform switch
>that would have to be added to the  tar command people use to create
>the
>vendor tarball, something like:
>
>tar -acvf package-version-vendor.tar.gz
>--transform='s#^vendor#package-version-vendor#' vendor
>
>You suggested that a maintainer could create a new tarball and build on
>top of it. I guess you mean  don't use upstream's tarball if they don't
>vendor and create my own tarball and add the vendor directory to it.
>I'm
>against that option because  I don't feel that we should manually
>tinker
>with upstream tarballs. That opens a pretty big can of worms imo.

No. I suggested that rather than adding another git clone and checking out a tag (which sooner or later would mean someone forgetting and using master instead), you could unpack the same archive you're going to use in the ebuild.
>
>William


--
Best regards, 
Michał Górny

Re: [gentoo-dev] [PATCH 1/3] go-module.eclass: introduce new eclass to handle go modules

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 4514 bytes --]

On Wed, Sep 11, 2019 at 05:28:22PM -0700, Alec Warner wrote:
> On Wed, Sep 11, 2019 at 5:05 PM William Hubbs <williamh@gentoo.org> wrote:
> 
> > On Wed, Sep 11, 2019 at 04:31:00PM -0700, Alec Warner wrote:
> > > On Wed, Sep 11, 2019 at 10:28 AM William Hubbs <williamh@gentoo.org>
> > wrote:
> > >
> > > > Copyright: Sony Interactive Entertainment Inc.
> > > > Signed-off-by: William Hubbs <williamh@gentoo.org>
> > > > ---
> > > >  eclass/go-module.eclass | 76 +++++++++++++++++++++++++++++++++++++++++
> > > >  1 file changed, 76 insertions(+)
> > > >  create mode 100644 eclass/go-module.eclass
> > > >
> > > > diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
> > > > new file mode 100644
> > > > index 00000000000..7009fcd3beb
> > > > --- /dev/null
> > > > +++ b/eclass/go-module.eclass
> > > > @@ -0,0 +1,76 @@
> > > > +# Copyright 1999-2015 Gentoo Foundation
> > > > +# Distributed under the terms of the GNU General Public License v2
> > > > +
> > > > +# @ECLASS: go-module.eclass
> > > > +# @MAINTAINER:
> > > > +# William Hubbs <williamh@gentoo.org>
> > > > +# @SUPPORTED_EAPIS: 7
> > > > +# @BLURB: basic eclass for building software written in the go
> > > > +# programming language that uses go modules.
> > > > +# @DESCRIPTION:
> > > > +# This eclass provides a convenience src_prepare() phase and some
> > basic
> > > > +# settings needed for all software written in the go programming
> > > > +# language that uses go modules.
> > > > +#
> > > > +# You will know the software you are packaging uses modules because
> > > > +# it will have files named go.sum and go.mod in its top-level source
> > > > +# directory. If it does not have these files, use the golang-*
> > eclasses.
> > > > +#
> > > > +# If the software you are packaging uses modules, the next question is
> > > > +# whether it has a directory named "vendor" at the top-level of the
> > > > source tree.
> > > > +#
> > > > +# If it doesn't, you need to create a tarball of what would be in the
> > > > +# vendor directory and mirror it locally. This is done with the
> > > > +# following commands if upstream is using a git repository:
> > > > +#
> > > > +# @CODE:
> > > > +#
> > > > +# $ cd /my/clone/of/upstream
> > > > +# $ git checkout <release>
> > > > +# $ go mod vendor
> > > > +# $ tar cvf project-version-vendor.tar.gz vendor
> > > > +#
> > > > +# @CODE:
> > > > +#
> > > > +# Other than this, all you need to do is inherit this eclass then
> > > > +# make sure  the exported src_prepare function is run.
> > > > +
> > > > +case ${EAPI:-0} in
> > > > +       7) ;;
> > > > +       *) die "${ECLASS} API in EAPI ${EAPI} not yet established."
> > > > +esac
> > > > +
> > > > +if [[ -z ${_GO_MODULE} ]]; then
> > > > +
> > > > +_GO_MODULE=1
> > > > +
> > > > +BDEPEND=">=dev-lang/go-1.12"
> > > > +
> > > > +# Do not download dependencies from the internet
> > > > +# make build output verbose by default
> > > > +export GOFLAGS="-mod=vendor -v -x"
> > > > +
> > > > +# Do not complain about CFLAGS etc since go projects do not use them.
> > > > +QA_FLAGS_IGNORED='.*'
> > > > +
> > > > +# Upstream does not support stripping go packages
> > > > +RESTRICT="strip"
> > > >
> > >
> > > https://golang.org/cmd/link/ implies you can pass -s -w to the compiler
> > to
> > > reduce binary size.
> > >
> > > Does that not work in portage by default, or does upstream just consider
> > > that bad practice?
> >
> > I haven't tried it, but here are the definitions of -s and -w.
> >
> > -s      Omit the symbol table and debug information.
> > -w      Omit the DWARF symbol table.
> >
> > These look like Go's equivalent of stripping the binaries, and I have my
> > doubts as to whether we should force this.
> >
> 
> I don't care if you strip or not (I'm not even sure portage knows how to do
> it for go binaries) but I'm fairly sure the reason isn't because "upstream
> does not support stripping go binaries" because they clearly do...unless
> upstream is portage here...?

Well, go binaries aren't supposed to be stripped using strip(1), which
is how portage strips binaries. You strip go binaries by passing -s and
-w to ldflags, so it has to be done during the build.

In order to do it right I would have to have a pms-compatible way to
check restrictions/features, and there isn't one that I'm aware of.

If there is a way to do this, I'm all ears, or I can update the comment
in the eclass.

William

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 1900 bytes --]

On Wed, Sep 11, 2019 at 05:05:50PM -0700, Alec Warner wrote:
> On Wed, Sep 11, 2019 at 4:48 PM William Hubbs <williamh@gentoo.org> wrote:
> 
> > On Wed, Sep 11, 2019 at 04:34:27PM -0700, Alec Warner wrote:
> > > On Wed, Sep 11, 2019 at 10:39 AM Michael Orlitzky <mjo@gentoo.org>
> > wrote:
> > >
> > > > On 9/11/19 1:21 PM, William Hubbs wrote:
> > > > > +++ b/dev-vcs/hub/hub-2.12.3.ebuild
> > > > > ...
> > > > >
> > > > > LICENSE="MIT"
> > > >
> > > > This license is wrong, as it's pretty much guaranteed to be every time
> > > > you commit one of these packages. I find it pretty troubling that one
> > > > corporation is able to force this stuff through even though it's a
> > > > security and legal hazard for everyone else.
> > > >
> > >
> > > How is it wrong?
> > >
> > > https://github.com/github/hub/blob/master/LICENSE
> >
> > The argument is that because of the vendoring, LICENSE= needs to list
> > all licenses for the vendored dependencies that are different from MIT
> > as well.
> >
> 
> I see, I tend to believe that argument in that case.
> 
> 
> >
> > Personally I don't have a comment about this, but that's what is being
> > pushed for. I'll let you guys debate this but it isn't really relevant
> > to the eclass. ;-)
> >
> 
> I think it's difficult to put instructions in the eclass like:
> 
> +# $ cd /my/clone/of/upstream
> +# $ git checkout <release>
> +# $ go mod vendor
> +# $ tar cvf project-version-vendor.tar.gz vendor
> 
> And then not mention this fairly easy trap (it's so easy to fall into you
> did it twice.)

In the case of hub, I didn't make a vendor tarball because upstream does
the vendoring, so I don't see how these two things are related.

In other words, the way I see this is a tree-wide issue. LICENSE= for
any package should list every license for every package it links to or
uses.

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Michael Orlitzky]

On 9/12/19 11:46 AM, William Hubbs wrote:
> 
> In other words, the way I see this is a tree-wide issue. LICENSE= for
> any package should list every license for every package it links to or
> uses.
>
There is no issue tree-wide, because no one else is trying to cut
corners and bundle every dependency of every package. All of our
dependencies are in separate packages, with separate LICENSE variables.
So when you install one package, the LICENSE variables of all its
dependencies pulled in by the package manager are indeed taken into
account. That's the whole point of a package manager -- it can do these
things for you if the developers do their jobs and package software
correctly.

Re: [gentoo-dev] [PATCH 1/3] go-module.eclass: introduce new eclass to handle go modules

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 8648 bytes --]

On Thu, Sep 12, 2019 at 05:39:42AM +0000, Michał Górny wrote:
> Dnia September 11, 2019 11:11:15 PM UTC, William Hubbs <williamh@gentoo.org> napisał(a):
> >On Wed, Sep 11, 2019 at 07:47:04PM +0000, Michał Górny wrote:
> >> Dnia September 11, 2019 7:40:41 PM UTC, William Hubbs
> ><williamh@gentoo.org> napisał(a):
> >> >On Wed, Sep 11, 2019 at 08:31:16PM +0200, Michał Górny wrote:
> >> >> On Wed, 2019-09-11 at 13:22 -0500, William Hubbs wrote:
> >> >> > On Wed, Sep 11, 2019 at 07:38:17PM +0200, Michał Górny wrote:
> >> >> > > On Wed, 2019-09-11 at 12:21 -0500, William Hubbs wrote:
> >> >> > > > Copyright: Sony Interactive Entertainment Inc.
> >> >> > > > Signed-off-by: William Hubbs <williamh@gentoo.org>
> >> >> > > > ---
> >> >> > > >  eclass/go-module.eclass | 76
> >> >+++++++++++++++++++++++++++++++++++++++++
> >> >> > > >  1 file changed, 76 insertions(+)
> >> >> > > >  create mode 100644 eclass/go-module.eclass
> >> >> > > > 
> >> >> > > > diff --git a/eclass/go-module.eclass
> >b/eclass/go-module.eclass
> >> >> > > > new file mode 100644
> >> >> > > > index 00000000000..7009fcd3beb
> >> >> > > > --- /dev/null
> >> >> > > > +++ b/eclass/go-module.eclass
> >> >> > > > @@ -0,0 +1,76 @@
> >> >> > > > +# Copyright 1999-2015 Gentoo Foundation
> >> >> > > 
> >> >> > > You need to replace your calendar.  And copyright holder.
> >> >> > 
> >> >> > Sure, I thought I ffixed that.
> >> >> > 
> >> >> > > > +# Distributed under the terms of the GNU General Public
> >> >License v2
> >> >> > > > +
> >> >> > > > +# @ECLASS: go-module.eclass
> >> >> > > 
> >> >> > > Any reason to change naming from golang-* to go-* now?
> >> >> > 
> >> >> > Well, "lang" is sort of redundant, and there will be only one
> >> >eclass, so
> >> >> > I thought I would make things a bit more simple.
> >> >> > 
> >> >> > > > +# @MAINTAINER:
> >> >> > > > +# William Hubbs <williamh@gentoo.org>
> >> >> > > > +# @SUPPORTED_EAPIS: 7
> >> >> > > > +# @BLURB: basic eclass for building software written in the
> >go
> >> >> > > > +# programming language that uses go modules.
> >> >> > > > +# @DESCRIPTION:
> >> >> > > > +# This eclass provides a convenience src_prepare() phase
> >and
> >> >some basic
> >> >> > > > +# settings needed for all software written in the go
> >> >programming
> >> >> > > > +# language that uses go modules.
> >> >> > > > +#
> >> >> > > > +# You will know the software you are packaging uses modules
> >> >because
> >> >> > > > +# it will have files named go.sum and go.mod in its
> >top-level
> >> >source
> >> >> > > > +# directory. If it does not have these files, use the
> >golang-*
> >> >eclasses.
> >> >> > > > +#
> >> >> > > > +# If the software you are packaging uses modules, the next
> >> >question is
> >> >> > > > +# whether it has a directory named "vendor" at the
> >top-level
> >> >of the source tree.
> >> >> > > > +#
> >> >> > > > +# If it doesn't, you need to create a tarball of what would
> >be
> >> >in the
> >> >> > > > +# vendor directory and mirror it locally. This is done with
> >> >the
> >> >> > > > +# following commands if upstream is using a git repository:
> >> >> > > > +#
> >> >> > > > +# @CODE:
> >> >> > > > +#
> >> >> > > > +# $ cd /my/clone/of/upstream
> >> >> > > > +# $ git checkout <release>
> >> >> > > > +# $ go mod vendor
> >> >> > > > +# $ tar cvf project-version-vendor.tar.gz vendor
> >> >> > > > +#
> >> >> > > > +# @CODE:
> >> >> > > > +#
> >> >> > > > +# Other than this, all you need to do is inherit this
> >eclass
> >> >then
> >> >> > > > +# make sure  the exported src_prepare function is run.
> >> >> > > > +
> >> >> > > > +case ${EAPI:-0} in
> >> >> > > > +	7) ;;
> >> >> > > > +	*) die "${ECLASS} API in EAPI ${EAPI} not yet
> >established."
> >> >> > > > +esac
> >> >> > > > +
> >> >> > > > +if [[ -z ${_GO_MODULE} ]]; then
> >> >> > > > +
> >> >> > > > +_GO_MODULE=1
> >> >> > > > +
> >> >> > > > +BDEPEND=">=dev-lang/go-1.12"
> >> >> > > > +
> >> >> > > > +# Do not download dependencies from the internet
> >> >> > > > +# make build output verbose by default
> >> >> > > > +export GOFLAGS="-mod=vendor -v -x"
> >> >> > > > +
> >> >> > > > +# Do not complain about CFLAGS etc since go projects do not
> >> >use them.
> >> >> > > > +QA_FLAGS_IGNORED='.*'
> >> >> > > > +
> >> >> > > > +# Upstream does not support stripping go packages
> >> >> > > > +RESTRICT="strip"
> >> >> > > > +
> >> >> > > > +EXPORT_FUNCTIONS src_prepare
> >> >> > > 
> >> >> > > Don't you need to inherit some other eclass to make it build?
> >> >> > 
> >> >> > The primary reason for all of the golang-* eclasses was the
> >GOPATH
> >> >> > variable, which is not relevant when you are using modules.
> >> >> > 
> >> >> > I can look at adding a src_compile to this eclass, but I haven't
> >> >thought
> >> >> > about what it would contain yet.
> >> >> >  
> >> >> > > > +
> >> >> > > > +# @FUNCTION: go-module_src_prepare
> >> >> > > > +# @DESCRIPTION:
> >> >> > > > +# Run a default src_prepare then move our provided vendor
> >> >directory to
> >> >> > > > +# the appropriate spot if upstream doesn't provide a vendor
> >> >directory.
> >> >> > > > +go-module_src_prepare() {
> >> >> > > > +	default
> >> >> > > > +	# Use the upstream provided vendor directory if it exists.
> >> >> > > > +	[[ -d vendor ]] && return
> >> >> > > > +	# If we are not providing a mirror of a vendor directory
> >we
> >> >created
> >> >> > > > +	# manually, return since there may be nothing to vendor.
> >> >> > > > +	[[ ! -d ../vendor ]] && return
> >> >> > > > +	# At this point, we know we are providing a vendor mirror.
> >> >> > > > +	mv ../vendor . || die "Unable to move ../vendor directory"
> >> >> > > 
> >> >> > > Wouldn't it be much simpler to create appropriate directory
> >> >structure
> >> >> > > in the tarball?  Then you wouldn't need a new eclass at all.
> >> >> > 
> >> >> > You would definitely need an eclass (see the settings and
> >> >dependencies).
> >> >> > 
> >> >> > Take a look at the differences in the spire and hub ebuilds in
> >this
> >> >> > series. I'm not sure what you mean by adding the directory
> >> >structure to
> >> >> > the tarball? I guess you could add something to the vendor
> >tarball
> >> >when
> >> >> > you create it.
> >> >> 
> >> >> I mean packing it as 'spire-1.2.3/vendor' or whatever the package
> >> >> directory is, so that it extracts correctly instead of making a
> >> >tarball
> >> >> that needs to be moved afterwards.
> >> >
> >> >That would clobber the upstream provided vendor directory and that's
> >> >what I want to avoid with the first test in src_prepare.
> >> 
> >> If upstream already includes vendored modules, why would you create
> >your own tarball in the first place?
> > 
> >You are right, and currently I quietly ignore your vendor tarball if
> >upstream
> >vendors the dependencies also. I could change this to generate a
> >warning
> >or die and force you to fix the ebuild, but that would not be possible
> >if I follow your suggestion because I would not be able to tell whether
> >the vendored dependencies came from us or upstream.
> 
> Why would anyone create a vendor tarball if things work without it? That makes no sense. Also adding unused archives to SRC_URI is a QA violation.

All the more reason to not have the vendor tarball overwrite the vendor
directory upstream. I will show you when I update the eclass.

> 
> >
> >Also, another concern about your suggestion is the  --transform switch
> >that would have to be added to the  tar command people use to create
> >the
> >vendor tarball, something like:
> >
> >tar -acvf package-version-vendor.tar.gz
> >--transform='s#^vendor#package-version-vendor#' vendor
> >
> >You suggested that a maintainer could create a new tarball and build on
> >top of it. I guess you mean  don't use upstream's tarball if they don't
> >vendor and create my own tarball and add the vendor directory to it.
> >I'm
> >against that option because  I don't feel that we should manually
> >tinker
> >with upstream tarballs. That opens a pretty big can of worms imo.
> 
> No. I suggested that rather than adding another git clone and checking out a tag (which sooner or later would mean someone forgetting and using master instead), you could unpack the same archive you're going to use in the ebuild.

Ok, I am really not following you, so let's talk about this in the
context of an example.

Look at app-misc/spire and tell me how you would do it differently.

William


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 3079 bytes --]

On Thu, Sep 12, 2019 at 9:14 AM Michael Orlitzky <mjo@gentoo.org> wrote:

> On 9/12/19 11:46 AM, William Hubbs wrote:
> >
> > In other words, the way I see this is a tree-wide issue. LICENSE= for
> > any package should list every license for every package it links to or
> > uses.
> >
> There is no issue tree-wide, because no one else is trying to cut
> corners and bundle every dependency of every package. All of our
> dependencies are in separate packages, with separate LICENSE variables.
> So when you install one package, the LICENSE variables of all its
> dependencies pulled in by the package manager are indeed taken into
> account. That's the whole point of a package manager -- it can do these
> things for you if the developers do their jobs and package software
> correctly.
>

I'm not really keen on this language ("cutting corners"). Newer languages
are not C, they don't act like C, build like C or do many other things like
C[0]. That presents engineering challenges to Gentoo; because the standard
way of doing stuff doesn't work. I nominally see two avenues here:

(1) go ebuilds with modules put the modules in DEPEND, every module is a
package, we have some kind of tooling to essentially handle the creation of
these packages (and by package I mean some kind of Gentoo compatible
package, including the tarballs, checksums, LICENSE, and so forth.) Then
during src_prepare() we essentially do the work necessary to place these
package sources in the right place (e.g. wherever the module system expects
them.)

(2) Is Williamh's current implementation, developers themselves run go mod,
get all the modules, package them up, and distribute them. The tooling for
this should handle LICENSE.

One immediate problem I see with (1) is that these modules often don't have
releases like traditional packages do, so you end up with a dependency on
(somemodulename, <SHA1HASH>). This then leads to a version explosion as
every go package (N) depends on a different SHA1HASH of module M; it ends
up being a bit of a hot mess. This may cause some complexities (it bloats
the depgraph for one) and so William's approach simplifies the problem.

In general I don't see bundling as a major problem. In the land of dynamic
binaries, it's a big advantage because you can upgrade libfoo and all
consumers of libfoo get the upgrade upon process restart. This isn't true
for most go programs which are statically linked; so you end up asking
yourself "why should I make a package for every go module?" One obvious
answer is that portage then tracks what packages are consuming a given
module and you can plausibly write a tool that does things like "moduleX
has a security update, please recompile all packages that DEPEND on
moduleX" which seems like a tool people would want.

[0] I feel like this is a common idea in Gentoo throughout. Anything new is
bad. Anything that violates norms is bad. Anything that violates the model
we have been using for 20 years is bad. I wish people were more open to
have a discussion without crapping on new ideas quite so thoroughly.

-A

[-- Attachment #2: Type: text/html, Size: 3631 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 2949 bytes --]

On Thu, Sep 12, 2019 at 8:46 AM William Hubbs <williamh@gentoo.org> wrote:

> On Wed, Sep 11, 2019 at 05:05:50PM -0700, Alec Warner wrote:
> > On Wed, Sep 11, 2019 at 4:48 PM William Hubbs <williamh@gentoo.org>
> wrote:
> >
> > > On Wed, Sep 11, 2019 at 04:34:27PM -0700, Alec Warner wrote:
> > > > On Wed, Sep 11, 2019 at 10:39 AM Michael Orlitzky <mjo@gentoo.org>
> > > wrote:
> > > >
> > > > > On 9/11/19 1:21 PM, William Hubbs wrote:
> > > > > > +++ b/dev-vcs/hub/hub-2.12.3.ebuild
> > > > > > ...
> > > > > >
> > > > > > LICENSE="MIT"
> > > > >
> > > > > This license is wrong, as it's pretty much guaranteed to be every
> time
> > > > > you commit one of these packages. I find it pretty troubling that
> one
> > > > > corporation is able to force this stuff through even though it's a
> > > > > security and legal hazard for everyone else.
> > > > >
> > > >
> > > > How is it wrong?
> > > >
> > > > https://github.com/github/hub/blob/master/LICENSE
> > >
> > > The argument is that because of the vendoring, LICENSE= needs to list
> > > all licenses for the vendored dependencies that are different from MIT
> > > as well.
> > >
> >
> > I see, I tend to believe that argument in that case.
> >
> >
> > >
> > > Personally I don't have a comment about this, but that's what is being
> > > pushed for. I'll let you guys debate this but it isn't really relevant
> > > to the eclass. ;-)
> > >
> >
> > I think it's difficult to put instructions in the eclass like:
> >
> > +# $ cd /my/clone/of/upstream
> > +# $ git checkout <release>
> > +# $ go mod vendor
> > +# $ tar cvf project-version-vendor.tar.gz vendor
> >
> > And then not mention this fairly easy trap (it's so easy to fall into you
> > did it twice.)
>
> In the case of hub, I didn't make a vendor tarball because upstream does
> the vendoring, so I don't see how these two things are related.
>
> In other words, the way I see this is a tree-wide issue. LICENSE= for
> any package should list every license for every package it links to or
> uses.
>

So for packages managed by portage this is true by recursion.

A -> B -> C
A_LICENSE: [GPL-2], B_LICENSE: [MIT], C_LICENSE: [BSD]

So to install A we have to install [A,B,C] and accept licenses [GPL-2, MIT,
BSD]. Presumably if ACCEPT_LICENSE was set to "-*" you would be forced to
actually accept each of these individually; but the default
is @OSI_APPROVED or similar, which contains many common OSS licenses.

If you bundle a bunch of stuff in package C and don't bother to set the
LICENSE variable, this is no longer true; I suspect this is why people are
complaining.
I don't think we are asking you to do extra work. Current practice is to
add dependencies as other packages (with their own LICENSE variable.) Your
scheme doesn't do this (saving you work), it bundles the dependencies
instead. This means to be equivalent to existing practice the LICENSE
should contain the licenses for the bundles as well.

-A


> William
>
>

[-- Attachment #2: Type: text/html, Size: 4284 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Michael Orlitzky]

On 9/12/19 12:42 PM, Alec Warner wrote:
> 
> In general I don't see bundling as a major problem. In the land of
> dynamic binaries, it's a big advantage because you can upgrade libfoo
> and all consumers of libfoo get the upgrade upon process restart. This
> isn't true for most go programs which are statically linked; so you end
> up asking yourself "why should I make a package for every go module?"
> One obvious answer is that portage then tracks what packages are
> consuming a given module and you can plausibly write a tool that does
> things like "moduleX has a security update, please recompile all
> packages that DEPEND on moduleX" which seems like a tool people would want.
> 

Subslots do this already. Portage does this already. We have this "tool
that people would want," but only if developers can be bothered to
package things.


> [0] I feel like this is a common idea in Gentoo throughout. Anything new
> is bad. Anything that violates norms is bad. Anything that violates the
> model we have been using for 20 years is bad. I wish people were more
> open to have a discussion without crapping on new ideas quite so thoroughly.

This is computer *science*. Some ideas are just wrong, and nothing of
value is gained by trying not to hurt the feelings of the flat-earthers.

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Mike Gilbert]

On Thu, Sep 12, 2019 at 12:52 PM Michael Orlitzky <mjo@gentoo.org> wrote:
>
> On 9/12/19 12:42 PM, Alec Warner wrote:
> >
> > In general I don't see bundling as a major problem. In the land of
> > dynamic binaries, it's a big advantage because you can upgrade libfoo
> > and all consumers of libfoo get the upgrade upon process restart. This
> > isn't true for most go programs which are statically linked; so you end
> > up asking yourself "why should I make a package for every go module?"
> > One obvious answer is that portage then tracks what packages are
> > consuming a given module and you can plausibly write a tool that does
> > things like "moduleX has a security update, please recompile all
> > packages that DEPEND on moduleX" which seems like a tool people would want.
> >
>
> Subslots do this already. Portage does this already. We have this "tool
> that people would want," but only if developers can be bothered to
> package things.

Portage only handles rebuilds for slot-operator deps in RDEPEND. It
ignores slot-operators in DEPEND.

Re: [gentoo-dev] [PATCH 1/3] go-module.eclass: introduce new eclass to handle go modules

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2685 bytes --]

On Thu, 2019-09-12 at 11:39 -0500, William Hubbs wrote:
> On Thu, Sep 12, 2019 at 05:39:42AM +0000, Michał Górny wrote:
> > Dnia September 11, 2019 11:11:15 PM UTC, William Hubbs <williamh@gentoo.org> napisał(a):
> > > You are right, and currently I quietly ignore your vendor tarball if
> > > upstream
> > > vendors the dependencies also. I could change this to generate a
> > > warning
> > > or die and force you to fix the ebuild, but that would not be possible
> > > if I follow your suggestion because I would not be able to tell whether
> > > the vendored dependencies came from us or upstream.
> > 
> > Why would anyone create a vendor tarball if things work without it? That makes no sense. Also adding unused archives to SRC_URI is a QA violation.
> 
> All the more reason to not have the vendor tarball overwrite the vendor
> directory upstream. I will show you when I update the eclass.

If there is a vendor directory, then you should not have custom vendor
tarball to override it in the first place.  If you have the tarball,
the eclass should explode and tell the developer he's doing silly things
and needs to stop, not silently pretend everything is fine and surprise
people by coincidentally choosing not to do anything.

> 
> > > Also, another concern about your suggestion is the  --transform switch
> > > that would have to be added to the  tar command people use to create
> > > the
> > > vendor tarball, something like:
> > > 
> > > tar -acvf package-version-vendor.tar.gz
> > > --transform='s#^vendor#package-version-vendor#' vendor
> > > 
> > > You suggested that a maintainer could create a new tarball and build on
> > > top of it. I guess you mean  don't use upstream's tarball if they don't
> > > vendor and create my own tarball and add the vendor directory to it.
> > > I'm
> > > against that option because  I don't feel that we should manually
> > > tinker
> > > with upstream tarballs. That opens a pretty big can of worms imo.
> > 
> > No. I suggested that rather than adding another git clone and checking out a tag (which sooner or later would mean someone forgetting and using master instead), you could unpack the same archive you're going to use in the ebuild.
> 
> Ok, I am really not following you, so let's talk about this in the
> context of an example.
> 
> Look at app-misc/spire and tell me how you would do it differently.
> 

ebuild spire-0.8.1.ebuild fetch
tar -xf ${DISTDIR}/spire-0.8.1.tar.gz
cd spire-0.8.1/
go mod vendor
cd ../
tar -cf spire-0.8.1-vendor.tar spire-0.8.1/vendor

Now you don't need special src_prepare() to unpack it.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Michael Orlitzky]

On 9/12/19 12:55 PM, Mike Gilbert wrote:
> 
> Portage only handles rebuilds for slot-operator deps in RDEPEND. It
> ignores slot-operators in DEPEND.
> 

Sure, but putting them in RDEPEND isn't a problem. It's not like the
statically-linked bundled dependencies actually go away with a depclean,
anyway.

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1220 bytes --]

On Thu, 2019-09-12 at 09:42 -0700, Alec Warner wrote:
> On Thu, Sep 12, 2019 at 9:14 AM Michael Orlitzky <mjo@gentoo.org> wrote:
> 
> > On 9/12/19 11:46 AM, William Hubbs wrote:
> > > In other words, the way I see this is a tree-wide issue. LICENSE= for
> > > any package should list every license for every package it links to or
> > > uses.
> > > 
> > There is no issue tree-wide, because no one else is trying to cut
> > corners and bundle every dependency of every package. All of our
> > dependencies are in separate packages, with separate LICENSE variables.
> > So when you install one package, the LICENSE variables of all its
> > dependencies pulled in by the package manager are indeed taken into
> > account. That's the whole point of a package manager -- it can do these
> > things for you if the developers do their jobs and package software
> > correctly.
> > 
> 
> I'm not really keen on this language ("cutting corners"). Newer languages
> are not C, they don't act like C, build like C or do many other things like
> C[0].

This doesn't mean they do anything better.  It just means they are
easier for 'write, deploy, sell, forget' model.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Mike Gilbert]

On Thu, Sep 12, 2019 at 1:05 PM Michael Orlitzky <mjo@gentoo.org> wrote:
>
> On 9/12/19 12:55 PM, Mike Gilbert wrote:
> >
> > Portage only handles rebuilds for slot-operator deps in RDEPEND. It
> > ignores slot-operators in DEPEND.
> >
>
> Sure, but putting them in RDEPEND isn't a problem. It's not like the
> statically-linked bundled dependencies actually go away with a depclean,
> anyway.

They do "go away" if you pass the right options to emerge, or if you
install it from a binpkg in the first place.

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 1630 bytes --]

On Thu, Sep 12, 2019 at 9:52 AM Michael Orlitzky <mjo@gentoo.org> wrote:

> On 9/12/19 12:42 PM, Alec Warner wrote:
> >
> > In general I don't see bundling as a major problem. In the land of
> > dynamic binaries, it's a big advantage because you can upgrade libfoo
> > and all consumers of libfoo get the upgrade upon process restart. This
> > isn't true for most go programs which are statically linked; so you end
> > up asking yourself "why should I make a package for every go module?"
> > One obvious answer is that portage then tracks what packages are
> > consuming a given module and you can plausibly write a tool that does
> > things like "moduleX has a security update, please recompile all
> > packages that DEPEND on moduleX" which seems like a tool people would
> want.
> >
>
> Subslots do this already. Portage does this already. We have this "tool
> that people would want," but only if developers can be bothered to
> package things.
>

Sure; and I listed this as an option. It's certainly not the only option.


>
>
> > [0] I feel like this is a common idea in Gentoo throughout. Anything new
> > is bad. Anything that violates norms is bad. Anything that violates the
> > model we have been using for 20 years is bad. I wish people were more
> > open to have a discussion without crapping on new ideas quite so
> thoroughly.
>
> This is computer *science*. Some ideas are just wrong, and nothing of
> value is gained by trying not to hurt the feelings of the flat-earthers.
>

Er, I'm fairly sure computer *science* has not conclusively proven that
dynamic binaries are somehow superior to static binaries.

-A

[-- Attachment #2: Type: text/html, Size: 2317 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 731 bytes --]

On Thu, 12 Sep 2019 12:52:31 -0400
Michael Orlitzky <mjo@gentoo.org> wrote:

> Subslots do this already. Portage does this already. We have this "tool
> that people would want," but only if developers can be bothered to
> package things.

For some things (go, rust), using dynamic linking for all dependencies,
and using subslots to regulate it... this may result in a reality where
all dependencies need a unique slot for every version, and every
version can be parallel installed with every other.

I don't think you want that, it has all the inflexibility of static binaries
and the only benefit it gives you is the visibility of the dependency
stack from portage.

Surely there are better ways to manage this.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] [PATCH 1/3] go-module.eclass: introduce new eclass to handle go modules

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 485 bytes --]

On Thu, 12 Sep 2019 19:03:02 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> ebuild spire-0.8.1.ebuild fetch
> tar -xf ${DISTDIR}/spire-0.8.1.tar.gz
> cd spire-0.8.1/
> go mod vendor
> cd ../
> tar -cf spire-0.8.1-vendor.tar spire-0.8.1/vendor
> 
> Now you don't need special src_prepare() to unpack it.

Of course, that means if spire-0.8.1 and spire-0.8.2 use identical
vendor dirs, you can no longer trivially bump the version on the dist
without a manual re-tar.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] [PATCH 1/3] go-module.eclass: introduce new eclass to handle go modules

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 655 bytes --]

On Wed, 11 Sep 2019 17:28:22 -0700
Alec Warner <antarus@gentoo.org> wrote:

> I don't care if you strip or not (I'm not even sure portage knows how to do
> it for go binaries) but I'm fairly sure the reason isn't because "upstream
> does not support stripping go binaries" because they clearly do...unless
> upstream is portage here...?

I know rust at least has some sort of magic in place where if you do
strip a binary, the ability for it to produce useful stack traces when
it crashes is reduced.

( In that, it can make use of debugging symbols without the aid of a
debugger )

I can imagine that could be a reason to not support it.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] [PATCH 1/3] go-module.eclass: introduce new eclass to handle go modules

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 1023 bytes --]

On Thu, Sep 12, 2019 at 1:20 PM Kent Fredric <kentnl@gentoo.org> wrote:

> On Wed, 11 Sep 2019 17:28:22 -0700
> Alec Warner <antarus@gentoo.org> wrote:
>
> > I don't care if you strip or not (I'm not even sure portage knows how to
> do
> > it for go binaries) but I'm fairly sure the reason isn't because
> "upstream
> > does not support stripping go binaries" because they clearly do...unless
> > upstream is portage here...?
>
> I know rust at least has some sort of magic in place where if you do
> strip a binary, the ability for it to produce useful stack traces when
> it crashes is reduced.


> ( In that, it can make use of debugging symbols without the aid of a
> debugger )
>
> I can imagine that could be a reason to not support it.
>

You definitely should not call 'strip' on a go binary. If you build with
the aforementioned linker flags you still get proper panic backtraces, but
also smaller binaries that you cannot load into gdb. Why 'strip' can't do
this but the go compiler can seems to be a bug ;)

-A

[-- Attachment #2: Type: text/html, Size: 1684 bytes --]

Re: [gentoo-dev] [PATCH 1/3] go-module.eclass: introduce new eclass to handle go modules

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 760 bytes --]

On Fri, 2019-09-13 at 08:16 +1200, Kent Fredric wrote:
> On Thu, 12 Sep 2019 19:03:02 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
> 
> > ebuild spire-0.8.1.ebuild fetch
> > tar -xf ${DISTDIR}/spire-0.8.1.tar.gz
> > cd spire-0.8.1/
> > go mod vendor
> > cd ../
> > tar -cf spire-0.8.1-vendor.tar spire-0.8.1/vendor
> > 
> > Now you don't need special src_prepare() to unpack it.
> 
> Of course, that means if spire-0.8.1 and spire-0.8.2 use identical
> vendor dirs, you can no longer trivially bump the version on the dist
> without a manual re-tar.

It also means you won't mistakenly re-tar with old vendor because you
forget to do the whole magic.  Or just re-tar with wrong tag checked
out.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Michael Orlitzky]

On 9/12/19 1:43 PM, Mike Gilbert wrote:
> 
> They do "go away" if you pass the right options to emerge, or if you
> install it from a binpkg in the first place.
> 

The dependencies are statically linked into the final executable forever
and receive no security updates. Portage doesn't even know they're
there. Depclean doesn't do what you think it does in that case. (I'm
sure you personally understand how this works, but a regular user has no
idea that we've installed 100MB of vulnerable code on his machine and
have just abandoned it there.)

Re: [gentoo-dev] [PATCH 1/3] go-module.eclass: introduce new eclass to handle go modules

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1451 bytes --]

On Thu, 2019-09-12 at 13:38 -0700, Alec Warner wrote:
> On Thu, Sep 12, 2019 at 1:20 PM Kent Fredric <kentnl@gentoo.org> wrote:
> 
> > On Wed, 11 Sep 2019 17:28:22 -0700
> > Alec Warner <antarus@gentoo.org> wrote:
> > 
> > > I don't care if you strip or not (I'm not even sure portage knows how to
> > do
> > > it for go binaries) but I'm fairly sure the reason isn't because
> > "upstream
> > > does not support stripping go binaries" because they clearly do...unless
> > > upstream is portage here...?
> > 
> > I know rust at least has some sort of magic in place where if you do
> > strip a binary, the ability for it to produce useful stack traces when
> > it crashes is reduced.
> > ( In that, it can make use of debugging symbols without the aid of a
> > debugger )
> > 
> > I can imagine that could be a reason to not support it.
> > 
> 
> You definitely should not call 'strip' on a go binary. If you build with
> the aforementioned linker flags you still get proper panic backtraces, but
> also smaller binaries that you cannot load into gdb. Why 'strip' can't do
> this but the go compiler can seems to be a bug ;)
> 

Since when it is a bug that when you strip debug info, you don't have
debug info?  I thought that's precisely what stripping debug info means
but maybe in the special Go world it is different, and debug info is
expected to remain after stripping it.

-- 
Best regards,
Michał Górny


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Mike Gilbert]

On Thu, Sep 12, 2019 at 5:11 PM Michael Orlitzky <mjo@gentoo.org> wrote:
>
> On 9/12/19 1:43 PM, Mike Gilbert wrote:
> >
> > They do "go away" if you pass the right options to emerge, or if you
> > install it from a binpkg in the first place.
> >
>
> The dependencies are statically linked into the final executable forever
> and receive no security updates. Portage doesn't even know they're
> there. Depclean doesn't do what you think it does in that case. (I'm
> sure you personally understand how this works, but a regular user has no
> idea that we've installed 100MB of vulnerable code on his machine and
> have just abandoned it there.)

Putting the dependencies in RDEPEND means users get stuck with yet
another copy of the code installed, in addition to the copy that is
statically linked into all reverse dependencies.

It's not a very good solution to the problem.

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Michael Orlitzky]

On 9/12/19 1:45 PM, Alec Warner wrote:
> 
> Er, I'm fairly sure computer *science* has not conclusively proven that
> dynamic binaries are somehow superior to static binaries.
> 

What are the benefits of static linking to the end user on Gentoo? The
comprehensive list is usually,

  * The application works independent of what libraries are installed
    on the system.

That however reduces to the empty list on Gentoo, because we have
dependency management (the right libraries are guaranteed be installed)
and subslot/revdep-rebuilds (the application gets rebuilt when they
change). The disadvantages, of course, remain:

  * They increase compile times.
  * They break LICENSE handling.
  * They break security updates.
  * They break all other (bug/feature) updates in dependencies, too.
  * They use more space on disk.
  * They use more memory at runtime.

What kind of math would convince you that an idea with all "cons" and no
"pros" is bad?

Re: [gentoo-dev] [PATCH 1/3] go-module.eclass: introduce new eclass to handle go modules

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 2121 bytes --]

On Thu, Sep 12, 2019 at 2:13 PM Michał Górny <mgorny@gentoo.org> wrote:

> On Thu, 2019-09-12 at 13:38 -0700, Alec Warner wrote:
> > On Thu, Sep 12, 2019 at 1:20 PM Kent Fredric <kentnl@gentoo.org> wrote:
> >
> > > On Wed, 11 Sep 2019 17:28:22 -0700
> > > Alec Warner <antarus@gentoo.org> wrote:
> > >
> > > > I don't care if you strip or not (I'm not even sure portage knows
> how to
> > > do
> > > > it for go binaries) but I'm fairly sure the reason isn't because
> > > "upstream
> > > > does not support stripping go binaries" because they clearly
> do...unless
> > > > upstream is portage here...?
> > >
> > > I know rust at least has some sort of magic in place where if you do
> > > strip a binary, the ability for it to produce useful stack traces when
> > > it crashes is reduced.
> > > ( In that, it can make use of debugging symbols without the aid of a
> > > debugger )
> > >
> > > I can imagine that could be a reason to not support it.
> > >
> >
> > You definitely should not call 'strip' on a go binary. If you build with
> > the aforementioned linker flags you still get proper panic backtraces,
> but
> > also smaller binaries that you cannot load into gdb. Why 'strip' can't do
> > this but the go compiler can seems to be a bug ;)
> >
>
> Since when it is a bug that when you strip debug info, you don't have
> debug info?  I thought that's precisely what stripping debug info means
> but maybe in the special Go world it is different, and debug info is
> expected to remain after stripping it.
>

So I have not checked (I should ask go-nuts about it) but it's possible
that:

strip <some_go_binary> breaks panic() tracebacks # This is generally bad.
go_compiler -w -s <some_binary> removes debug info, produces a smaller
binary, but has working panic() tracebacks.

In this case we would:
 - Prefer the latter over the former.
 - Ideally make it so that strip emulates -w -s, but keeps panic metadata
for go programs.

Not sure if upstream has rejected those patches, I can follow up.

-A



>
> --
> Best regards,
> Michał Górny
>
>

[-- Attachment #2: Type: text/html, Size: 3123 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Michael Orlitzky]

On 9/12/19 5:23 PM, Mike Gilbert wrote:
> 
> Putting the dependencies in RDEPEND means users get stuck with yet
> another copy of the code installed, in addition to the copy that is
> statically linked into all reverse dependencies.
> 
> It's not a very good solution to the problem.
> 

No argument there. The elegant solution to static linking is to not do
it. But I guess that's off the table? So now we're trying to find the
best not very good solution.

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 1481 bytes --]

On Thu, Sep 12, 2019 at 5:14 PM Michael Orlitzky <mjo@gentoo.org> wrote:

> On 9/12/19 5:23 PM, Mike Gilbert wrote:
> >
> > Putting the dependencies in RDEPEND means users get stuck with yet
> > another copy of the code installed, in addition to the copy that is
> > statically linked into all reverse dependencies.
> >
> > It's not a very good solution to the problem.
> >
>
> No argument there. The elegant solution to static linking is to not do
> it. But I guess that's off the table? So now we're trying to find the
> best not very good solution.
>
>
Well I think you end up with a weird tradeoff here.

Most people who build and package go-based packages use static compilation,
so we could in theory build dynamically linked executables, but then we
diverge from the upstream practice.
Maybe this is the right approach, but I think its a bunch of extra
engineering work (to make things build dynamically) and will be pretty
different from what upstream is expecting.

https://docs.google.com/document/d/1nr-TQHw_er6GOQRsF6T43GGhFDelrAP0NqSS_00RgZQ/edit
describes
the new execution modes; it even discusses this very topic!

"This mode is mainly intended to support distro builders.  They can
distribute Go packages or groups of packages as shared libraries, and can
thus update all Go programs by updating the shared libraries, without
requiring the programs to be relinked."

I wonder who pushed for this to Ian, and whether distros ended up using
this kind of target?

-A

[-- Attachment #2: Type: text/html, Size: 2715 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 2041 bytes --]

On Thu, Sep 12, 2019 at 6:56 PM Alec Warner <antarus@gentoo.org> wrote:

>
>
> On Thu, Sep 12, 2019 at 5:14 PM Michael Orlitzky <mjo@gentoo.org> wrote:
>
>> On 9/12/19 5:23 PM, Mike Gilbert wrote:
>> >
>> > Putting the dependencies in RDEPEND means users get stuck with yet
>> > another copy of the code installed, in addition to the copy that is
>> > statically linked into all reverse dependencies.
>> >
>> > It's not a very good solution to the problem.
>> >
>>
>> No argument there. The elegant solution to static linking is to not do
>> it. But I guess that's off the table? So now we're trying to find the
>> best not very good solution.
>>
>>
> Well I think you end up with a weird tradeoff here.
>
> Most people who build and package go-based packages use static
> compilation, so we could in theory build dynamically linked executables,
> but then we diverge from the upstream practice.
> Maybe this is the right approach, but I think its a bunch of extra
> engineering work (to make things build dynamically) and will be pretty
> different from what upstream is expecting.
>
>
> https://docs.google.com/document/d/1nr-TQHw_er6GOQRsF6T43GGhFDelrAP0NqSS_00RgZQ/edit describes
> the new execution modes; it even discusses this very topic!
>
> "This mode is mainly intended to support distro builders.  They can
> distribute Go packages or groups of packages as shared libraries, and can
> thus update all Go programs by updating the shared libraries, without
> requiring the programs to be relinked."
>

Another thing I neglected to point out is that even with these dynamic libs
you end up with toolchain problems. Once you upgrade you go toolchain (goN
and go-N+1) and build a new libfoo with go-N+1, it becomes incompatible
with binaries built with go-N. Other languages have this problem as well,
but its not great and it's a great recipe to hose a system while you are
rebuilding your go programs after an upgrade.


>
> I wonder who pushed for this to Ian, and whether distros ended up using
> this kind of target?
>
> -A
>
>
>

[-- Attachment #2: Type: text/html, Size: 3769 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Mike Gilbert]

On Thu, Sep 12, 2019 at 8:14 PM Michael Orlitzky <mjo@gentoo.org> wrote:
>
> On 9/12/19 5:23 PM, Mike Gilbert wrote:
> >
> > Putting the dependencies in RDEPEND means users get stuck with yet
> > another copy of the code installed, in addition to the copy that is
> > statically linked into all reverse dependencies.
> >
> > It's not a very good solution to the problem.
> >
>
> No argument there. The elegant solution to static linking is to not do
> it. But I guess that's off the table? So now we're trying to find the
> best not very good solution.

I'm really objecting to your suggestion that we abuse an existing
Portage/PMS feature to do something beyond its design. Adding
fictitious runtime dependencies is wrong, and seems like a very lazy
solution.

If you want to propose an extension to PMS to handle this situation,
that's something I can support.

Re: [gentoo-dev] [PATCH 1/3] go-module.eclass: introduce new eclass to handle go modules

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1410 bytes --]

On Thu, 12 Sep 2019 23:12:52 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> Since when it is a bug that when you strip debug info, you don't have
> debug info?  I thought that's precisely what stripping debug info means
> but maybe in the special Go world it is different, and debug info is
> expected to remain after stripping it.

I think the distinction may be because typically, people use external
tools to consume debug info, thus, people without said tools aren't
disadvantaged by stripping.

But when the program itself uses its own debug info as part of its
behaviour, it seems more reasonable that you might want to keep those.

For instance, if you build a rust binary "without debugging" and "with
optimization", you still actually get *some* debugging symbols, just
not all of them, and you still get useful backtraces that are vaguely
related when a panic happens.

But if you point strip at it, panic-backtraces become much less useful.

And given the nature of Rust and Go, where tripping a panic is pretty
much a "file a bug immediately" situation, it seems silly to have
programs having the capacity to print their own backtraces, but also
being useless for reporting bugs.

( I need to look into if rust does the right thing if you do
split-debugs, but I doubt it, I was under the impression bringing the
split debug info back into context is gdb magic )


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 998 bytes --]

On Thu, 12 Sep 2019 17:58:08 -0400
Michael Orlitzky <mjo@gentoo.org> wrote:

> What kind of math would convince you that an idea with all "cons" and no
> "pros" is bad?

Is "upstream tooling doesn't work without static compilation" or
"built packages tend to need exact version matching at runtime to work"
( which necessitates massive-scale multi-slotting, where every version
of every packaged "thing" has a co-existing slot ) a problem for you?

Having the same problem as static-linking in terms of disk use ( all
those different parallel versions ), but adding dependency hell to it
, and adding compile time overhead (even if those assets were no-op
virtuals, portage overhead is still pretty steep) isn't fun.

Not to mention reduced opportunities for whole-program optimization.

Yes, In general I'm against static linking, and I really dislike this
trend.

But when upstreams ecosystem is built around it as a core concept, its
really hard to buck that trend.




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Michael Orlitzky]

On 9/12/19 11:13 PM, Mike Gilbert wrote:
> 
> I'm really objecting to your suggestion that we abuse an existing
> Portage/PMS feature to do something beyond its design. Adding
> fictitious runtime dependencies is wrong, and seems like a very lazy
> solution.

Ok, you're right.


> If you want to propose an extension to PMS to handle this situation,
> that's something I can support.

Portage and the PMS are fine. This junk belongs in an overlay.

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Michael Orlitzky]

On 9/13/19 5:19 AM, Kent Fredric wrote:
> On Thu, 12 Sep 2019 17:58:08 -0400
> Michael Orlitzky <mjo@gentoo.org> wrote:
> 
>> What kind of math would convince you that an idea with all "cons" and no
>> "pros" is bad?
> 
> Is "upstream tooling doesn't work without static compilation" or
> "built packages tend to need exact version matching at runtime to work"
> ( which necessitates massive-scale multi-slotting, where every version
> of every packaged "thing" has a co-existing slot ) a problem for you?
> 

I see it as a problem, but not one that has to be my problem. I don't
see it as a foregone conclusion that we have to package every piece of
software -- no matter how bad -- and distribute it with the OS that I
use to do my banking.

These languages are badly implemented, and very little of value is
written in them. If their developers ever hit 2+ users, I'm sure they'll
realize that everyone else was right back in the 1970s, and fix the
design. But in the meantime, this stuff belongs in an overlay. Lowering
our standards until they match upstream's is antithetical to how a
distribution is supposed to improve my life.

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Michael Orlitzky]

On 9/12/19 1:45 PM, Alec Warner wrote:
> 
> Er, I'm fairly sure computer *science* has not conclusively proven that
> dynamic binaries are somehow superior to static binaries.
> 

Please don't make me work this hard ever again.

The principal of modularity in software design goes back to at least
1970. In Parnas's famous 1971 paper[0], he cites "Designing Systems
Programs" by Gauthier and Pont from 1970, and suggests that the idea
goes back even further, to at least 1967 (Balzer & Mealy). On the very
first page of Parnas' paper, he says

  The major advancement in the area of modular programming has been the
  development of coding techniques and assemblers which

    (1) allow one module to be written with little knowledge of the code
        in another module, and

    (2) allow modules to be reassembled and replaced without reassembly
        of the whole system.

The second item has a clear interpretation in terms of static/dynamic
linking. Parnas concludes that

  ...one begins with a list of difficult design decisions or design
  decisions which are likely to change. Each module is then designed to
  hide such a decision from the others.

If you statically link to a library, then none of its implementation
details are hidden from you -- you need to "reassemble" your program
whenever the library changes.

If we jump way forward to 1979, the SICP[1] is basically a thousand-page
treatise on abstraction. But not only at one level of computation:

  Well-designed computational systems, like well-designed automobiles or
  nuclear reactors, are designed in a modular manner, so that the parts
  can be constructed, replaced, and debugged separately.

"Replaced" here is of course what I want to draw your attention to. But
also on the fact that abstraction and modularity don't just apply at one
level of software design and engineering -- it's a fractal. At the
lowest levels, we abstract machine code to assembler, assembler to
low-level languages, low-level languages to high-level languages,
high-level languages to functions and procedures, functions and
procedures to libraries, libraries to services, and services to
distributed APIs. The same principles that apply to a collection of
functions (a program) also apply to a collection of programs (an
operating system). The rule "don't copy and paste code" applies to your
linker just as much as it applies to the first program you wrote in
CS101, and for the same reasons.

As you commented on IRC, the cost in terms of man-power is that
someone's workload jumps from O(n) to O(m*n), because you have to
duplicate everything you do for every statically-linked dependency. And
you can find as much theory as you like on software modularity in papers
from the 1970s and 1980s, but the benefits are not only theoretical.
There's a mountain of empirical data that supports the theory. Some
choice quotes [2][3]:

  Poorly placed dependencies, especially those that link otherwise
  independent modules, may result in a cascade of unwanted and hard-to-
  detect indirect interactions. Our results suggest that purposeful
  actions to reduce such "rogue dependencies" can be effective (the
  redesign of Mozilla reduced propagation cost by over 80%).

  Our results confirm the existence of a relationship between component
  modularity and design evolution that is both statistically significant
  and large in magnitude... Tightly-coupled components are... less
  adaptable  via the processes of exclusion or substitution; they are
  more likely to  experience "surprise" dependency additions unrelated
  to new functionality, implying that they demand greater maintenance
  efforts;  and they are harder to augment, in that the mix of new
  components is  more modular than the legacy design.

The only difference between static linking and copy/pasting chunks of
code around is in who pays the price. If the programmer copies & pastes
code into his own program, he will eventually have to deal with the
mess. On the other hand, it he forces his program to be statically
linked, then it is the end user who is harmed by his decision.

In any case, the theory says that modularity is superior, and the
empirical data confirm it. The fact that you won't find a paper saying
"dynamic linking is better than static linking" is somewhat beside the
point, and only muddies the water. Linking is just one specific instance
of a problem that was solved 40 years ago.


[0]
https://www.win.tue.nl/~wstomv/edu/2ip30/references/criteria_for_modularization.pdf

[1] https://web.mit.edu/alexmv/6.037/sicp.pdf

[2] https://pubsonline.informs.org/doi/10.1287/mnsc.1060.0552

[3] http://www.hbs.edu/research/pdf/08-038.pdf

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Patrick McLean]

On Fri, 13 Sep 2019 08:29:20 -0400
Michael Orlitzky <mjo@gentoo.org> wrote:

> On 9/13/19 5:19 AM, Kent Fredric wrote:
> > On Thu, 12 Sep 2019 17:58:08 -0400
> > Michael Orlitzky <mjo@gentoo.org> wrote:
> >   
> >> What kind of math would convince you that an idea with all "cons"
> >> and no "pros" is bad?  
> > 
> > Is "upstream tooling doesn't work without static compilation" or
> > "built packages tend to need exact version matching at runtime to
> > work" ( which necessitates massive-scale multi-slotting, where
> > every version of every packaged "thing" has a co-existing slot ) a
> > problem for you? 
> 
> I see it as a problem, but not one that has to be my problem. I don't
> see it as a foregone conclusion that we have to package every piece of
> software -- no matter how bad -- and distribute it with the OS that I
> use to do my banking.
> 
I don't think anyone here has suggested that any go packages are
installed in the stage3 tarballs, or included in profiles. Something's
presence in the tree does not mean that you are required to install it.
A package's presence in the tree really has little to zero effect on
any user that does not use the package. If you do not install the
package, it will have zero effect on your banking.

I also want to point out that the Gentoo packages for Firefox,
Chromium, and Webkit all have a _lot_ of bundled dependencies and
absolutely do static linking internally. If you are using a browser to
do your banking, you are almost certainly using static linking, even
without the presence of code written in golang.

> These languages are badly implemented, and very little of value is
> written in them. If their developers ever hit 2+ users, I'm sure
> they'll realize that everyone else was right back in the 1970s, and
> fix the design. But in the meantime, this stuff belongs in an
> overlay. Lowering our standards until they match upstream's is
> antithetical to how a distribution is supposed to improve my life.

Despite your (and my) objections to it's approach to linking, golang is
a very popular language these days with some very popular packages
written in it. Docker and Kubernetes immediately come to mind, but
there are many others. The argument "I don't use, and I dislike the
implementation language, so no one should use it" is not a very
compelling argument.

These are very popular packages, that users and developers absolutely
want to be available in Gentoo. Given this fact, and the fact that
there are Gentoo developers who want these packages enough that they
will maintain the packages, they absolutely do belong in the tree.

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Patrick McLean]

On Fri, 13 Sep 2019 12:50:48 -0400
Michael Orlitzky <mjo@gentoo.org> wrote:

> On 9/12/19 1:45 PM, Alec Warner wrote:
> > 
> > Er, I'm fairly sure computer *science* has not conclusively proven
> > that dynamic binaries are somehow superior to static binaries.
> >   
> 
> 
> If you statically link to a library, then none of its implementation
> details are hidden from you -- you need to "reassemble" your program
> whenever the library changes.
> 
> If we jump way forward to 1979, the SICP[1] is basically a
> thousand-page treatise on abstraction. But not only at one level of
> computation:
 
While I personally have opinions about static linking (I basically
completely agree with you that it's a dumb idea). That said, this has
nothing to do with this particular discussion, I suggest you take it up
with the golang upstream. I don't think anyone here is arguing that
static linking is a great idea and everyone should do it.

We are arguing that the golang community believes strongly in static
linking, and the entire ecosystem is designed around this idea, as such
trying to bodge dynamic linking on to the existing ecosystem is
probably more work than it is worth. In general, developers Gentoo
tries to stay close to upstream as much as possible, as this reduces
the maintenance burden.

The entire point of this thread is to try to find a sane way to support
the golang community's standards. I think the merits of static linking
are out of scope here.

>   Well-designed computational systems, like well-designed automobiles
> or nuclear reactors, are designed in a modular manner, so that the
> parts can be constructed, replaced, and debugged separately.
> 
> "Replaced" here is of course what I want to draw your attention to.
> But also on the fact that abstraction and modularity don't just apply
> at one level of software design and engineering -- it's a fractal. At
> the lowest levels, we abstract machine code to assembler, assembler to
> low-level languages, low-level languages to high-level languages,
> high-level languages to functions and procedures, functions and
> procedures to libraries, libraries to services, and services to
> distributed APIs. The same principles that apply to a collection of
> functions (a program) also apply to a collection of programs (an
> operating system). The rule "don't copy and paste code" applies to
> your linker just as much as it applies to the first program you wrote
> in CS101, and for the same reasons.
> 
> As you commented on IRC, the cost in terms of man-power is that
> someone's workload jumps from O(n) to O(m*n), because you have to
> duplicate everything you do for every statically-linked dependency.
> And you can find as much theory as you like on software modularity in
> papers from the 1970s and 1980s, but the benefits are not only
> theoretical. There's a mountain of empirical data that supports the
> theory. Some choice quotes [2][3]:
> 
>   Poorly placed dependencies, especially those that link otherwise
>   independent modules, may result in a cascade of unwanted and
> hard-to- detect indirect interactions. Our results suggest that
> purposeful actions to reduce such "rogue dependencies" can be
> effective (the redesign of Mozilla reduced propagation cost by over
> 80%).
> 
>   Our results confirm the existence of a relationship between
> component modularity and design evolution that is both statistically
> significant and large in magnitude... Tightly-coupled components
> are... less adaptable  via the processes of exclusion or
> substitution; they are more likely to  experience "surprise"
> dependency additions unrelated to new functionality, implying that
> they demand greater maintenance efforts;  and they are harder to
> augment, in that the mix of new components is  more modular than the
> legacy design.
> 
> The only difference between static linking and copy/pasting chunks of
> code around is in who pays the price. If the programmer copies &
> pastes code into his own program, he will eventually have to deal
> with the mess. On the other hand, it he forces his program to be
> statically linked, then it is the end user who is harmed by his
> decision.
> 
> In any case, the theory says that modularity is superior, and the
> empirical data confirm it. The fact that you won't find a paper saying
> "dynamic linking is better than static linking" is somewhat beside the
> point, and only muddies the water. Linking is just one specific
> instance of a problem that was solved 40 years ago.
> 
> 
> [0]
> https://www.win.tue.nl/~wstomv/edu/2ip30/references/criteria_for_modularization.pdf
> 
> [1] https://web.mit.edu/alexmv/6.037/sicp.pdf
> 
> [2] https://pubsonline.informs.org/doi/10.1287/mnsc.1060.0552
> 
> [3] http://www.hbs.edu/research/pdf/08-038.pdf
> 

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Michael Orlitzky]

(Replying to both messages at once.)


On 9/13/19 4:17 PM, Patrick McLean wrote:
>>
> I don't think anyone here has suggested that any go packages are
> installed in the stage3 tarballs, or included in profiles. Something's
> presence in the tree does not mean that you are required to install it.
> A package's presence in the tree really has little to zero effect on
> any user that does not use the package. If you do not install the
> package, it will have zero effect on your banking.

This is true only so far as they never become dependencies of anything
else. Do all new developers know that dev-go is an insecure ghetto? Do
our users? Or might someone accidentally install or depend upon
something in dev-go before learning that crucial bit of information?


> I also want to point out that the Gentoo packages for Firefox,
> Chromium, and Webkit all have a _lot_ of bundled dependencies and
> absolutely do static linking internally. If you are using a browser to
> do your banking, you are almost certainly using static linking, even
> without the presence of code written in golang.

Is this is a "two wrongs make a right" argument? I'm telling mom =P


> Despite your (and my) objections to it's approach to linking, golang is
> a very popular language these days with some very popular packages
> written in it.

No it's not. It's below Delphi and Object Pascal on TIOBE this month.
It's a trend that a tiny percentage of people jumped on because they
heard the name "Google" back when Google was cool.

The "people want this in Gentoo" argument I understand, but people don't
really have it "in Gentoo." They have a thin wrapper around the "go"
command. They don't get the Gentoo security guarantees, they don't get
the Gentoo license handling, they don't get the ease of management that
comes with a Gentoo @world update. They silently get something less than
they're expecting. We would be better off telling people to run "go
whatever" themselves, or by putting this stuff in an overlay where
expectations are clearly defined.


> While I personally have opinions about static linking (I basically
> completely agree with you that it's a dumb idea). That said, this has
> nothing to do with this particular discussion, I suggest you take it up
> with the golang upstream. I don't think anyone here is arguing that
> static linking is a great idea and everyone should do it.

We just have a philosophical difference here. I don't think we should
commit admittedly-dumb ideas to ::gentoo. These packages would work fine
in an overlay until such a time as someone is interested in doing things
correctly. They also work "fine" if you install them with "go" yourself:
Portage isn't doing much for you when everything is bundled, statically
linked, and has LICENSE set incorrectly.

I don't want to keep replying to these threads -- I've said everything
that I've got to say, and I'm boring myself, so I can only imagine how
you all feel. This will get pushed through anyway, because it always
does. It's just demoralizing constantly begging people not to make
things worse and being ignored.

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Patrick McLean]

On Fri, 13 Sep 2019 19:44:55 -0400
Michael Orlitzky <mjo@gentoo.org> wrote:

> (Replying to both messages at once.)
> 
> 
> On 9/13/19 4:17 PM, Patrick McLean wrote:
> >>  
> > I don't think anyone here has suggested that any go packages are
> > installed in the stage3 tarballs, or included in profiles.
> > Something's presence in the tree does not mean that you are
> > required to install it. A package's presence in the tree really has
> > little to zero effect on any user that does not use the package. If
> > you do not install the package, it will have zero effect on your
> > banking.  
> 
> This is true only so far as they never become dependencies of anything
> else. Do all new developers know that dev-go is an insecure ghetto? Do
> our users? Or might someone accidentally install or depend upon
> something in dev-go before learning that crucial bit of information?

A suggestion was made on IRC to have a pkg_postinst in the eclass that
warn about golang package dependencies not having the same level of
Gentoo security coverage that other packages in the tree have due to
static linking. I think this is a reasonable approach, and users and
developers will know. There is precedent for this, see
sys-kernel/vanilla-sources

> > I also want to point out that the Gentoo packages for Firefox,
> > Chromium, and Webkit all have a _lot_ of bundled dependencies and
> > absolutely do static linking internally. If you are using a browser
> > to do your banking, you are almost certainly using static linking,
> > even without the presence of code written in golang.  
> 
> Is this is a "two wrongs make a right" argument? I'm telling mom =P

I am pointing out that we can't ban all static linking in the tree,
many upstream packages won't work without it (or significant effort
that no one has the time or motivation for).

> > Despite your (and my) objections to it's approach to linking,
> > golang is a very popular language these days with some very popular
> > packages written in it.  
> 
> No it's not. It's below Delphi and Object Pascal on TIOBE this month.
> It's a trend that a tiny percentage of people jumped on because they
> heard the name "Google" back when Google was cool.

Random stats from a website are not really an indication of how much a
language is being used. There are plenty of very popular packages that
are written in golang.

> The "people want this in Gentoo" argument I understand, but people
> don't really have it "in Gentoo." They have a thin wrapper around the
> "go" command. They don't get the Gentoo security guarantees, they
> don't get the Gentoo license handling, they don't get the ease of
> management that comes with a Gentoo @world update. They silently get
> something less than they're expecting. We would be better off telling
> people to run "go whatever" themselves, or by putting this stuff in
> an overlay where expectations are clearly defined.

Users and Gentoo developers want Docker and Kubernetes (to name a
couple) in the main tree. These are written in golang. I don't think we
should ban packages because of the language they are written in.
Especially if there are developers who want to maintain them.

They do get the ease of management of @world in that if the upstream
package releases a new version, it will be pulled in via an @world
update. That is quite a large advantage to users, and is worth doing if
there are developers willing to maintain the packages in the tree.

> 
> > While I personally have opinions about static linking (I basically
> > completely agree with you that it's a dumb idea). That said, this
> > has nothing to do with this particular discussion, I suggest you
> > take it up with the golang upstream. I don't think anyone here is
> > arguing that static linking is a great idea and everyone should do
> > it.  
> 
> We just have a philosophical difference here. I don't think we should
> commit admittedly-dumb ideas to ::gentoo. These packages would work
> fine in an overlay until such a time as someone is interested in
> doing things correctly. They also work "fine" if you install them
> with "go" yourself: Portage isn't doing much for you when everything
> is bundled, statically linked, and has LICENSE set incorrectly.

When "doing things correctly" means basically forking the entire
ecosystem and maintaining all the forks internally, that is not
something that is ever going to happen. There is demand from users and
developers for golang packages.

It's the same reason why we don't unbundle everything in Firefox and
Chromium, it's simply too much work. It basically means maintaining our
own fork of the package. That also means security updates will take
significantly longer, as the fork will need to be rebased on the new
upstream version.

> I don't want to keep replying to these threads -- I've said everything
> that I've got to say, and I'm boring myself, so I can only imagine how
> you all feel. This will get pushed through anyway, because it always
> does. It's just demoralizing constantly begging people not to make
> things worse and being ignored.

Then don't, golang and packages written in it are going to stay in the
tree and new golang packages are going to be added. This entire
thread has been about how we are going to support a newer packaging
style upstream adopted.

I encourage you to package.mask dev-lang/go, and carefully inspect any
-bin packages you install to make sure you don't install anything
written in golang on your machine.

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Ulrich Mueller]

[-- Attachment #1: Type: text/plain, Size: 392 bytes --]

>>>>> On Fri, 13 Sep 2019, Patrick McLean wrote:

> Something's presence in the tree does not mean that you are required
> to install it. A package's presence in the tree really has little to
> zero effect on any user that does not use the package.

I wonder if that's always true. Distros adopting a package will make
it more likely that other software will gain a dependency on it.

Ulrich

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Alec Warner]

[-- Attachment #1: Type: text/plain, Size: 4091 bytes --]

On Fri, Sep 13, 2019 at 4:45 PM Michael Orlitzky <mjo@gentoo.org> wrote:

> (Replying to both messages at once.)
>
>
> On 9/13/19 4:17 PM, Patrick McLean wrote:
> >>
> > I don't think anyone here has suggested that any go packages are
> > installed in the stage3 tarballs, or included in profiles. Something's
> > presence in the tree does not mean that you are required to install it.
> > A package's presence in the tree really has little to zero effect on
> > any user that does not use the package. If you do not install the
> > package, it will have zero effect on your banking.
>
> This is true only so far as they never become dependencies of anything
> else. Do all new developers know that dev-go is an insecure ghetto? Do
> our users? Or might someone accidentally install or depend upon
> something in dev-go before learning that crucial bit of information?
>

>
> > I also want to point out that the Gentoo packages for Firefox,
> > Chromium, and Webkit all have a _lot_ of bundled dependencies and
> > absolutely do static linking internally. If you are using a browser to
> > do your banking, you are almost certainly using static linking, even
> > without the presence of code written in golang.
>
> Is this is a "two wrongs make a right" argument? I'm telling mom =P
>
>
> > Despite your (and my) objections to it's approach to linking, golang is
> > a very popular language these days with some very popular packages
> > written in it.
>
> No it's not. It's below Delphi and Object Pascal on TIOBE this month.
> It's a trend that a tiny percentage of people jumped on because they
> heard the name "Google" back when Google was cool.
>

> The "people want this in Gentoo" argument I understand, but people don't
> really have it "in Gentoo." They have a thin wrapper around the "go"
> command. They don't get the Gentoo security guarantees, they don't get
> the Gentoo license handling, they don't get the ease of management that
> comes with a Gentoo @world update. They silently get something less than
> they're expecting. We would be better off telling people to run "go
> whatever" themselves, or by putting this stuff in an overlay where
> expectations are clearly defined.
>

> > While I personally have opinions about static linking (I basically
> > completely agree with you that it's a dumb idea). That said, this has
> > nothing to do with this particular discussion, I suggest you take it up
> > with the golang upstream. I don't think anyone here is arguing that
> > static linking is a great idea and everyone should do it.
>
> We just have a philosophical difference here. I don't think we should
> commit admittedly-dumb ideas to ::gentoo. These packages would work fine
> in an overlay until such a time as someone is interested in doing things
> correctly. They also work "fine" if you install them with "go" yourself:
> Portage isn't doing much for you when everything is bundled, statically
> linked, and has LICENSE set incorrectly.
>

> I don't want to keep replying to these threads -- I've said everything
> that I've got to say, and I'm boring myself, so I can only imagine how
> you all feel. This will get pushed through anyway, because it always
> does. It's just demoralizing constantly begging people not to make
> things worse and being ignored.


So I think there are really two sorts of issues here.

 - We discuss things on the mailing list in order to learn more about the
problem space. William got feedback on his eclasses, we discovered dynamic
go binaries exist, but they are probably not a good idea to use, and I
think we have a solution to the LICENSE problem based on some tooling
william is also looking into.
 - There appears to be some expectation that consensus is required on the
ML; this has (IMHO) never been true. The 'decider' for what to do isn't the
mailing list (by GLEP, it's the council). So this idea that you can object
on the ML and stop a thing isn't really something I'd be counting on.
Sometimes you convince the OP, and sometimes you don't. I don't think you
need to walk away sad when the latter happens.

-A

[-- Attachment #2: Type: text/html, Size: 5425 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Michael Orlitzky]

On 9/14/19 1:06 PM, Alec Warner wrote:
> 
>  - There appears to be some expectation that consensus is required on
> the ML; this has (IMHO) never been true. The 'decider' for what to do
> isn't the mailing list (by GLEP, it's the council). So this idea that
> you can object on the ML and stop a thing isn't really something I'd be
> counting on. Sometimes you convince the OP, and sometimes you don't. I
> don't think you need to walk away sad when the latter happens.
> 

I'm not going to cry about it or anything. I'm trying to explain my
point of view. I regularly spend hours fixing little "quality of life"
issues in Gentoo. It's not fun, and I wouldn't do it if I didn't think
it was possible to make a difference.

But things like this give impression that nobody cares, and that any
time you spend trying to fix things is wasted: someone's going to be
adding new bugs faster than you can fix the old ones. It's like trying
to paint a mural that gets spray-painted over every night. Eventually
the artist is going to decide that the people who live there deserve to
look at the side of an abandoned building all day.

I've filed ~100 bugs for minor security issues, like root exploits in
config files, user-controlled binaries in /usr/bin, and race conditions
in init scripts. But who actually gives a fuck about a race condition in
an init script, when there are parts of the tree that get no security
updates at all? It takes YEARS to find, report, and fix a single one of
these issues. How long does it take to add a new Go package?

It starts to feel like a losing battle.

And I'm not throwing in the towel yet, but every time I essentially get
told "nobody cares," I agree with this nobody person more and more.

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 1125 bytes --]

On Fri, 13 Sep 2019 19:44:55 -0400
Michael Orlitzky <mjo@gentoo.org> wrote:

>  They silently get something less than
> they're expecting. We would be better off telling people to run "go
> whatever" themselves, or by putting this stuff in an overlay where
> expectations are clearly defined.

That suggestion actually decreases security.

Especially if the package in question is intended to be run as root.

At least with using portage, you can side-step the nonsense of "and
here's how you install this in /usr/bin .... curl url | sudo bash - "

And additionally, we get a sandbox and all the features of file
ownership tracking.

And if there is a complaint about the package misbehaving, a bug can be
filed in a common location, and a gentoo dev can actually fix the
problem, even if upstream have moved on to greener pastures. ( This is
the sad state of a lot of older perl stuff these days, they simply
don't work vanilla any more, and gentoo are putting the patches in to
keep it working )

So in summary, Portage does a lot more for the end user than "ensure
dynamic linking works".




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-dev] [PATCH 3/3] dev-vcs/hub: migrate to go-module.eclass

[Kent Fredric]

[-- Attachment #1: Type: text/plain, Size: 2232 bytes --]

On Fri, 13 Sep 2019 12:50:48 -0400
Michael Orlitzky <mjo@gentoo.org> wrote:

>  The rule "don't copy and paste code" applies to your
> linker just as much as it applies to the first program you wrote in
> CS101, and for the same reasons.

My experience has taught me to ignore the rule as written, and consider
it more a guideline aimed at novices.

Importantly, don't copy and paste code you don't understand.

However, if you re-use a given implementation, you can sometimes become
dependent on specifics in that implementation.

And upstream have an annoying tendency to change implementation details
in incompatible ways.

This breaks your code, sometimes obviously, sometimes subtly.

So the benefit you'd hoped to attain by modularity only remains true if
the modules in question are immutable for the problem you're using them
for.

Also, these external modules bring with them complexity you may not
want or need (and their own dependencies with the same issues), and
that's not free either.

Subsequently, when considering a new dependency, I have to ask
questions about whether I can implement a subset of the dependency
faithfully that suits my needs, and ask if I have the skills to be an
expert in this implementation.

A level of distrust in the people who author and maintain your
dependencies is something I consider healthy.

Does this map at all to static linking? You bet.

And that's part of why Go and Rust use it.  The fragility of
other-peoples code in the long view of time is high, because with
dynamic linking, you don't have any guarantees about integrity.

You have to cross your fingers and hope upstream so-named their
libraries right, hope upstream changed their API markers right, hope
upstream never accidentally made a breaking change without changing an
API marker, hope your code doesn't need a mountain of work to recompile
after the API marker changes, and hope upstream/gentoo maintainers for
your package can still be bothered to ship fixes for the API consumer.

That last paragraph sums up a large volume of the daily hell of being a
maintainer and a gentoo user.

Static linking is more a result of a sickness, not the sickness itself.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]