[gentoo-dev] [PATCH] go-module.eclass: add functions for use in custom src_unpack phase

[gentoo-dev] [PATCH] go-module.eclass: add functions for use in custom src_unpack phase

[William Hubbs]

If an ebuild uses EGO_SUM and needs to define a custom src_unpack phase,
these functions will make that easier.

go-module_setup_proxy is used to create a local file proxy of the
dependencies listed in EGO_SUM and go-module_filter_proxy is used to
create a new ${A} with the EGO_SUM_SRC_URI values removed.

Signed-off-by: William Hubbs <williamh@gentoo.org>
---
 eclass/go-module.eclass | 69 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 69 insertions(+)

diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
index c9a7ab12eaf..80e1f711215 100644
--- a/eclass/go-module.eclass
+++ b/eclass/go-module.eclass
@@ -236,6 +236,75 @@ go-module_set_globals() {
 	_GO_MODULE_SET_GLOBALS_CALLED=1
 }
 
+# @FUNCTION: go-module_filter_proxy
+# @DESCRIPTION:
+# If your ebuild redefines src_unpack and uses EGO_SUM, use the return
+# value of this function in place of ${A} in your src_unpack function.
+# It filters the EGO_SUM_SRC_URI values out of SRC_URI.
+go-module_filter_proxy() {
+	# shellcheck disable=SC2120
+	debug-print-function "${FUNCNAME}" "$@"
+
+	if [[ ! ${_GO_MODULE_SET_GLOBALS_CALLED} ]]; then
+		die "go-module_set_globals must be called in global scope"
+	fi
+
+	# Skip golang modules and add the rest of the entries to a local
+	# array.
+	local f
+	local -a src
+	for f in ${A}; do
+		if [[ -z ${_GOMODULE_GOSUM_REVERSE_MAP["${f}"]} ]]; then
+		src+=("${f}")
+		fi
+	done
+	echo "${src[@]}"
+}
+
+# @FUNCTION: go-module_setup_proxy
+# @DESCRIPTION:
+# If your ebuild redefines src_unpack and uses EGO_SUM you need to call
+# this function in src_unpack.
+# It sets up the go module proxy in the appropriate location and exports
+# the GOPROXY environment variable so that go calls will be able to
+# locate the proxy directory.
+go-module_setup_proxy() {
+	# shellcheck disable=SC2120
+	debug-print-function "${FUNCNAME}" "$@"
+
+	if [[ ! ${_GO_MODULE_SET_GLOBALS_CALLED} ]]; then
+		die "go-module_set_globals must be called in global scope"
+	fi
+
+	local goproxy_dir="${T}/go-proxy"
+	mkdir -p "${goproxy_dir}" || die
+
+	# For each Golang module distfile, look up where it's supposed to go and
+	# symlink it into place.
+	local f
+	local goproxy_mod_dir
+	for f in ${A}; do
+		goproxy_mod_path="${_GOMODULE_GOSUM_REVERSE_MAP["${f}"]}"
+		if [[ -n "${goproxy_mod_path}" ]]; then
+			debug-print-function "Populating go proxy for ${goproxy_mod_path}"
+			# Build symlink hierarchy
+			goproxy_mod_dir=$( dirname "${goproxy_dir}"/"${goproxy_mod_path}" )
+			mkdir -p "${goproxy_mod_dir}" || die
+			ln -sf "${DISTDIR}"/"${f}" "${goproxy_dir}/${goproxy_mod_path}" ||
+				die "Failed to ln"
+			local v=${goproxy_mod_path}
+			v="${v%.mod}"
+			v="${v%.zip}"
+			v="${v//*\/}"
+			_go-module_gosum_synthesize_files "${goproxy_mod_dir}" "${v}"
+		fi
+	done
+	export GOPROXY="file://${goproxy_dir}"
+
+	# Validate the gosum now
+	_go-module_src_unpack_verify_gosum
+}
+
 # @FUNCTION: go-module_src_unpack
 # @DESCRIPTION:
 # If EGO_SUM is set, unpack the base tarball(s) and set up the
-- 
2.26.3

Re: [gentoo-dev] [PATCH] go-module.eclass: add functions for use in custom src_unpack phase

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 318 bytes --]

Robin,

I would like your thoughts also. The idea is to allow something like
this. Maybe an example like this one should go in the top of the eclass
as well.

src_unpack() {
	local srcs
	go-module_setup_proxy
	srcs="$(go-module_filter_proxy)"
	# unpack or do what you need to do with everything in ${srcs}
}

William


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[gentoo-dev] Re: [PATCH] go-module.eclass: add functions for use in custom src_unpack phase

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 3910 bytes --]

On 5/19/21 12:48 PM, William Hubbs wrote:
> If an ebuild uses EGO_SUM and needs to define a custom src_unpack phase,
> these functions will make that easier.
> 
> go-module_setup_proxy is used to create a local file proxy of the
> dependencies listed in EGO_SUM and go-module_filter_proxy is used to
> create a new ${A} with the EGO_SUM_SRC_URI values removed.
> 
> Signed-off-by: William Hubbs <williamh@gentoo.org>
> ---
>  eclass/go-module.eclass | 69 +++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 69 insertions(+)
> 
> diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
> index c9a7ab12eaf..80e1f711215 100644
> --- a/eclass/go-module.eclass
> +++ b/eclass/go-module.eclass
> @@ -236,6 +236,75 @@ go-module_set_globals() {
>  	_GO_MODULE_SET_GLOBALS_CALLED=1
>  }
>  
> +# @FUNCTION: go-module_filter_proxy
> +# @DESCRIPTION:
> +# If your ebuild redefines src_unpack and uses EGO_SUM, use the return
> +# value of this function in place of ${A} in your src_unpack function.
> +# It filters the EGO_SUM_SRC_URI values out of SRC_URI.
> +go-module_filter_proxy() {
> +	# shellcheck disable=SC2120
> +	debug-print-function "${FUNCNAME}" "$@"
> +
> +	if [[ ! ${_GO_MODULE_SET_GLOBALS_CALLED} ]]; then
> +		die "go-module_set_globals must be called in global scope"
> +	fi
> +
> +	# Skip golang modules and add the rest of the entries to a local
> +	# array.
> +	local f
> +	local -a src
> +	for f in ${A}; do
> +		if [[ -z ${_GOMODULE_GOSUM_REVERSE_MAP["${f}"]} ]]; then
> +		src+=("${f}")
> +		fi
> +	done
> +	echo "${src[@]}"
> +}

While this go-module_filter_proxy might seem like a convenient way for
an ebuild to list distfiles which are not derived from EGO_SUM, I think
it's not the responsibility of go-module.eclass to provide this
information, therefore it's probably better to force the ebuild
developer to provide their own mechanism to track those files if needed,
rather than add a superfluous function to the eclass.

> +# @FUNCTION: go-module_setup_proxy
> +# @DESCRIPTION:
> +# If your ebuild redefines src_unpack and uses EGO_SUM you need to call
> +# this function in src_unpack.
> +# It sets up the go module proxy in the appropriate location and exports
> +# the GOPROXY environment variable so that go calls will be able to
> +# locate the proxy directory.
> +go-module_setup_proxy() {
> +	# shellcheck disable=SC2120
> +	debug-print-function "${FUNCNAME}" "$@"
> +
> +	if [[ ! ${_GO_MODULE_SET_GLOBALS_CALLED} ]]; then
> +		die "go-module_set_globals must be called in global scope"
> +	fi
> +
> +	local goproxy_dir="${T}/go-proxy"
> +	mkdir -p "${goproxy_dir}" || die
> +
> +	# For each Golang module distfile, look up where it's supposed to go and
> +	# symlink it into place.
> +	local f
> +	local goproxy_mod_dir
> +	for f in ${A}; do
> +		goproxy_mod_path="${_GOMODULE_GOSUM_REVERSE_MAP["${f}"]}"
> +		if [[ -n "${goproxy_mod_path}" ]]; then
> +			debug-print-function "Populating go proxy for ${goproxy_mod_path}"
> +			# Build symlink hierarchy
> +			goproxy_mod_dir=$( dirname "${goproxy_dir}"/"${goproxy_mod_path}" )
> +			mkdir -p "${goproxy_mod_dir}" || die
> +			ln -sf "${DISTDIR}"/"${f}" "${goproxy_dir}/${goproxy_mod_path}" ||
> +				die "Failed to ln"
> +			local v=${goproxy_mod_path}
> +			v="${v%.mod}"
> +			v="${v%.zip}"
> +			v="${v//*\/}"
> +			_go-module_gosum_synthesize_files "${goproxy_mod_dir}" "${v}"
> +		fi
> +	done
> +	export GOPROXY="file://${goproxy_dir}"
> +
> +	# Validate the gosum now
> +	_go-module_src_unpack_verify_gosum
> +}
> +
>  # @FUNCTION: go-module_src_unpack
>  # @DESCRIPTION:
>  # If EGO_SUM is set, unpack the base tarball(s) and set up the
> 

The go-module_setup_proxy function solves bug 790851 nicely, since
sys-cluster/k3s ebuilds can call that instead of go-module_src_unpack.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

[gentoo-dev] Re: [PATCH] go-module.eclass: add functions for use in custom src_unpack phase

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 2306 bytes --]

On 5/19/21 1:45 PM, Zac Medico wrote:
>> +# @FUNCTION: go-module_setup_proxy
>> +# @DESCRIPTION:
>> +# If your ebuild redefines src_unpack and uses EGO_SUM you need to call
>> +# this function in src_unpack.
>> +# It sets up the go module proxy in the appropriate location and exports
>> +# the GOPROXY environment variable so that go calls will be able to
>> +# locate the proxy directory.
>> +go-module_setup_proxy() {
>> +	# shellcheck disable=SC2120
>> +	debug-print-function "${FUNCNAME}" "$@"
>> +
>> +	if [[ ! ${_GO_MODULE_SET_GLOBALS_CALLED} ]]; then
>> +		die "go-module_set_globals must be called in global scope"
>> +	fi
>> +
>> +	local goproxy_dir="${T}/go-proxy"
>> +	mkdir -p "${goproxy_dir}" || die
>> +
>> +	# For each Golang module distfile, look up where it's supposed to go and
>> +	# symlink it into place.
>> +	local f
>> +	local goproxy_mod_dir
>> +	for f in ${A}; do
>> +		goproxy_mod_path="${_GOMODULE_GOSUM_REVERSE_MAP["${f}"]}"
>> +		if [[ -n "${goproxy_mod_path}" ]]; then
>> +			debug-print-function "Populating go proxy for ${goproxy_mod_path}"
>> +			# Build symlink hierarchy
>> +			goproxy_mod_dir=$( dirname "${goproxy_dir}"/"${goproxy_mod_path}" )
>> +			mkdir -p "${goproxy_mod_dir}" || die
>> +			ln -sf "${DISTDIR}"/"${f}" "${goproxy_dir}/${goproxy_mod_path}" ||
>> +				die "Failed to ln"
>> +			local v=${goproxy_mod_path}
>> +			v="${v%.mod}"
>> +			v="${v%.zip}"
>> +			v="${v//*\/}"
>> +			_go-module_gosum_synthesize_files "${goproxy_mod_dir}" "${v}"
>> +		fi
>> +	done
>> +	export GOPROXY="file://${goproxy_dir}"
>> +
>> +	# Validate the gosum now
>> +	_go-module_src_unpack_verify_gosum
>> +}
>> +
>>  # @FUNCTION: go-module_src_unpack
>>  # @DESCRIPTION:
>>  # If EGO_SUM is set, unpack the base tarball(s) and set up the
>>
> 
> The go-module_setup_proxy function solves bug 790851 nicely, since
> sys-cluster/k3s ebuilds can call that instead of go-module_src_unpack.

I do have one criticism of the go-module_setup_proxy, which is that it
relies on the side-effect of the GOPROXY export for its operation. We
can instead echo the GOPROXY value to stdout and force the caller to
export it themselves, and provide a convenience wrapper function which
works based on side-effects.
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] Re: [PATCH] go-module.eclass: add functions for use in custom src_unpack phase

[William Hubbs]

[-- Attachment #1: Type: text/plain, Size: 2602 bytes --]

On Wed, May 19, 2021 at 01:57:38PM -0700, Zac Medico wrote:
> On 5/19/21 1:45 PM, Zac Medico wrote:
> >> +# @FUNCTION: go-module_setup_proxy
> >> +# @DESCRIPTION:
> >> +# If your ebuild redefines src_unpack and uses EGO_SUM you need to call
> >> +# this function in src_unpack.
> >> +# It sets up the go module proxy in the appropriate location and exports
> >> +# the GOPROXY environment variable so that go calls will be able to
> >> +# locate the proxy directory.
> >> +go-module_setup_proxy() {
> >> +	# shellcheck disable=SC2120
> >> +	debug-print-function "${FUNCNAME}" "$@"
> >> +
> >> +	if [[ ! ${_GO_MODULE_SET_GLOBALS_CALLED} ]]; then
> >> +		die "go-module_set_globals must be called in global scope"
> >> +	fi
> >> +
> >> +	local goproxy_dir="${T}/go-proxy"
> >> +	mkdir -p "${goproxy_dir}" || die
> >> +
> >> +	# For each Golang module distfile, look up where it's supposed to go and
> >> +	# symlink it into place.
> >> +	local f
> >> +	local goproxy_mod_dir
> >> +	for f in ${A}; do
> >> +		goproxy_mod_path="${_GOMODULE_GOSUM_REVERSE_MAP["${f}"]}"
> >> +		if [[ -n "${goproxy_mod_path}" ]]; then
> >> +			debug-print-function "Populating go proxy for ${goproxy_mod_path}"
> >> +			# Build symlink hierarchy
> >> +			goproxy_mod_dir=$( dirname "${goproxy_dir}"/"${goproxy_mod_path}" )
> >> +			mkdir -p "${goproxy_mod_dir}" || die
> >> +			ln -sf "${DISTDIR}"/"${f}" "${goproxy_dir}/${goproxy_mod_path}" ||
> >> +				die "Failed to ln"
> >> +			local v=${goproxy_mod_path}
> >> +			v="${v%.mod}"
> >> +			v="${v%.zip}"
> >> +			v="${v//*\/}"
> >> +			_go-module_gosum_synthesize_files "${goproxy_mod_dir}" "${v}"
> >> +		fi
> >> +	done
> >> +	export GOPROXY="file://${goproxy_dir}"
> >> +
> >> +	# Validate the gosum now
> >> +	_go-module_src_unpack_verify_gosum
> >> +}
> >> +
> >>  # @FUNCTION: go-module_src_unpack
> >>  # @DESCRIPTION:
> >>  # If EGO_SUM is set, unpack the base tarball(s) and set up the
> >>
> > 
> > The go-module_setup_proxy function solves bug 790851 nicely, since
> > sys-cluster/k3s ebuilds can call that instead of go-module_src_unpack.
> 
> I do have one criticism of the go-module_setup_proxy, which is that it
> relies on the side-effect of the GOPROXY export for its operation. We
> can instead echo the GOPROXY value to stdout and force the caller to
> export it themselves, and provide a convenience wrapper function which
> works based on side-effects.

You really only need GOPROXY if you use EGO_SUM in your ebuild, so I
could probably export GOPROXY in go-module_set_globals.

William

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [gentoo-dev] Re: [PATCH] go-module.eclass: add functions for use in custom src_unpack phase

[Zac Medico]

[-- Attachment #1.1: Type: text/plain, Size: 2780 bytes --]

On 5/20/21 9:03 AM, William Hubbs wrote:
> On Wed, May 19, 2021 at 01:57:38PM -0700, Zac Medico wrote:
>> On 5/19/21 1:45 PM, Zac Medico wrote:
>>>> +# @FUNCTION: go-module_setup_proxy
>>>> +# @DESCRIPTION:
>>>> +# If your ebuild redefines src_unpack and uses EGO_SUM you need to call
>>>> +# this function in src_unpack.
>>>> +# It sets up the go module proxy in the appropriate location and exports
>>>> +# the GOPROXY environment variable so that go calls will be able to
>>>> +# locate the proxy directory.
>>>> +go-module_setup_proxy() {
>>>> +	# shellcheck disable=SC2120
>>>> +	debug-print-function "${FUNCNAME}" "$@"
>>>> +
>>>> +	if [[ ! ${_GO_MODULE_SET_GLOBALS_CALLED} ]]; then
>>>> +		die "go-module_set_globals must be called in global scope"
>>>> +	fi
>>>> +
>>>> +	local goproxy_dir="${T}/go-proxy"
>>>> +	mkdir -p "${goproxy_dir}" || die
>>>> +
>>>> +	# For each Golang module distfile, look up where it's supposed to go and
>>>> +	# symlink it into place.
>>>> +	local f
>>>> +	local goproxy_mod_dir
>>>> +	for f in ${A}; do
>>>> +		goproxy_mod_path="${_GOMODULE_GOSUM_REVERSE_MAP["${f}"]}"
>>>> +		if [[ -n "${goproxy_mod_path}" ]]; then
>>>> +			debug-print-function "Populating go proxy for ${goproxy_mod_path}"
>>>> +			# Build symlink hierarchy
>>>> +			goproxy_mod_dir=$( dirname "${goproxy_dir}"/"${goproxy_mod_path}" )
>>>> +			mkdir -p "${goproxy_mod_dir}" || die
>>>> +			ln -sf "${DISTDIR}"/"${f}" "${goproxy_dir}/${goproxy_mod_path}" ||
>>>> +				die "Failed to ln"
>>>> +			local v=${goproxy_mod_path}
>>>> +			v="${v%.mod}"
>>>> +			v="${v%.zip}"
>>>> +			v="${v//*\/}"
>>>> +			_go-module_gosum_synthesize_files "${goproxy_mod_dir}" "${v}"
>>>> +		fi
>>>> +	done
>>>> +	export GOPROXY="file://${goproxy_dir}"
>>>> +
>>>> +	# Validate the gosum now
>>>> +	_go-module_src_unpack_verify_gosum
>>>> +}
>>>> +
>>>>  # @FUNCTION: go-module_src_unpack
>>>>  # @DESCRIPTION:
>>>>  # If EGO_SUM is set, unpack the base tarball(s) and set up the
>>>>
>>>
>>> The go-module_setup_proxy function solves bug 790851 nicely, since
>>> sys-cluster/k3s ebuilds can call that instead of go-module_src_unpack.
>>
>> I do have one criticism of the go-module_setup_proxy, which is that it
>> relies on the side-effect of the GOPROXY export for its operation. We
>> can instead echo the GOPROXY value to stdout and force the caller to
>> export it themselves, and provide a convenience wrapper function which
>> works based on side-effects.
> 
> You really only need GOPROXY if you use EGO_SUM in your ebuild, so I
> could probably export GOPROXY in go-module_set_globals.
> 
> William
> 

Yes, go-module_set_globals is a nice place to consolidate all of the
exports. Great idea!
-- 
Thanks,
Zac


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 981 bytes --]