[gentoo-dev] [PATCH] cargo.eclass: use CARGO_CRATE_URIS if already available

[gentoo-dev] [PATCH] cargo.eclass: use CARGO_CRATE_URIS if already available

[Florian Schmaus]

With 59dbfb80f748 ("cargo.eclass: Add variable alternative to
$(cargo_crate_uris)") the _cargo_set_crate_uris function was
introduced. This function runs when the elcass is inherited and sets
CARGO_CRATE_URIS.

Ebuilds that use $(cargo_crate_uris) in SRC_URI will again invoke this
function, even though CARGO_CRATE_URIS is already set. Avoiding this
unnecessary computation reduces the ebuild source time of
app-shells/nushell-0.83.0 from 21 ms to 14 ms.

This is a significant reduction when compared to the variable-setting
alternative that the commit 59dbfb80f748 ("cargo.eclass: Add variable
alternative to $(cargo_crate_uris)") introduced. Using the
variable-setting approach would reduce the ebuild source time only by a
little bit more than one millisecond.

Using

pk pkg source --bench 10s '=app-shells/nushell-0.83.0'

as benchmark, yields

|                           | Cached CARGO_CRATE_URIS | Non-Cached CARGO_CRATE_URIS |
|---------------------------+-------------------------+-----------------------------|
| $(cargo_crate_uris)       | mean: 14.189ms          | mean: 21.445ms              |
| variable-setting approach | mean: 12.822ms          | mean: 12.852ms              |

full benchmark output

|                           | Cached CARGO_CRATE_URIS (this commit)                            | Non-Cached CARGO_CRATE_URIS                                      |
|---------------------------+------------------------------------------------------------------+------------------------------------------------------------------|
| $(cargo_crate_uris)       | mean: 14.189ms, min: 13.646ms, max: 15.103ms, σ = 149µs, N = 705 | mean: 21.445ms, min: 20.79ms,  max: 22.832ms, σ = 228µs, N = 467 |
| variable-setting approach | mean: 12.822ms, min: 12.41ms,  max: 13.909ms, σ = 165µs, N = 780 | mean: 12.852ms, min: 12.367ms, max: 15.437ms, σ = 227µs, N = 779 |

Signed-off-by: Florian Schmaus <flow@gentoo.org>
---
 eclass/cargo.eclass | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
index 70b6008d9cd8..5d6911801097 100644
--- a/eclass/cargo.eclass
+++ b/eclass/cargo.eclass
@@ -240,6 +240,14 @@ _cargo_set_crate_uris "${CRATES}"
 # Constructs a list of crates from its arguments.
 # If no arguments are provided, it uses the CRATES variable.
 cargo_crate_uris() {
+	# Use already existing value for CARGO_CRATE_URIS, computed by
+	# _cargo_set_crate_uris, when this function is invoked without
+	# arguments.
+	if [[ $# -eq 0 && -n "${CARGO_CRATE_URIS}" ]]; then
+		echo "${CARGO_CRATE_URIS}"
+		return
+	fi
+
 	local crates=${*-${CRATES}}
 	if [[ -z ${crates} ]]; then
 		eerror "CRATES variable is not defined and nothing passed as argument"
-- 
2.41.0

Re: [gentoo-dev] [PATCH] cargo.eclass: use CARGO_CRATE_URIS if already available

[Sam James]

Florian Schmaus <flow@gentoo.org> writes:

> With 59dbfb80f748 ("cargo.eclass: Add variable alternative to
> $(cargo_crate_uris)") the _cargo_set_crate_uris function was
> introduced. This function runs when the elcass is inherited and sets
> CARGO_CRATE_URIS.

(I would've CC'd mgorny given presumably you want his input as the
author.)

> Ebuilds that use $(cargo_crate_uris) in SRC_URI will again invoke this
> function, even though CARGO_CRATE_URIS is already set. Avoiding this
> unnecessary computation reduces the ebuild source time of
> app-shells/nushell-0.83.0 from 21 ms to 14 ms.
>
> This is a significant reduction when compared to the variable-setting
> alternative that the commit 59dbfb80f748 ("cargo.eclass: Add variable
> alternative to $(cargo_crate_uris)") introduced. Using the
> variable-setting approach would reduce the ebuild source time only by a
> little bit more than one millisecond.
>
> Using
>
> pk pkg source --bench 10s '=app-shells/nushell-0.83.0'
>
> as benchmark, yields
>
> |                           | Cached CARGO_CRATE_URIS | Non-Cached CARGO_CRATE_URIS |
> |---------------------------+-------------------------+-----------------------------|
> | $(cargo_crate_uris)       | mean: 14.189ms          | mean: 21.445ms              |
> | variable-setting approach | mean: 12.822ms          | mean: 12.852ms              |
>
> full benchmark output
>
> |                           | Cached CARGO_CRATE_URIS (this commit)                            | Non-Cached CARGO_CRATE_URIS                                      |
> |---------------------------+------------------------------------------------------------------+------------------------------------------------------------------|
> | $(cargo_crate_uris)       | mean: 14.189ms, min: 13.646ms, max: 15.103ms, σ = 149µs, N = 705 | mean: 21.445ms, min: 20.79ms,  max: 22.832ms, σ = 228µs, N = 467 |
> | variable-setting approach | mean: 12.822ms, min: 12.41ms,  max: 13.909ms, σ = 165µs, N = 780 | mean: 12.852ms, min: 12.367ms, max: 15.437ms, σ = 227µs, N = 779 |
>

Anyway, nice work, this seems reasonable, and it's consistent with the
kind of thing I've done for Ruby. But let's see what mgorny says as
well.

> Signed-off-by: Florian Schmaus <flow@gentoo.org>
> ---
>  eclass/cargo.eclass | 8 ++++++++
>  1 file changed, 8 insertions(+)
>
> diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
> index 70b6008d9cd8..5d6911801097 100644
> --- a/eclass/cargo.eclass
> +++ b/eclass/cargo.eclass
> @@ -240,6 +240,14 @@ _cargo_set_crate_uris "${CRATES}"
>  # Constructs a list of crates from its arguments.
>  # If no arguments are provided, it uses the CRATES variable.
>  cargo_crate_uris() {
> +	# Use already existing value for CARGO_CRATE_URIS, computed by
> +	# _cargo_set_crate_uris, when this function is invoked without
> +	# arguments.
> +	if [[ $# -eq 0 && -n "${CARGO_CRATE_URIS}" ]]; then
> +		echo "${CARGO_CRATE_URIS}"
> +		return
> +	fi
> +
>  	local crates=${*-${CRATES}}
>  	if [[ -z ${crates} ]]; then
>  		eerror "CRATES variable is not defined and nothing passed as argument"

Re: [gentoo-dev] [PATCH] cargo.eclass: use CARGO_CRATE_URIS if already available

[Michał Górny]

On Mon, 2023-07-31 at 12:41 +0200, Florian Schmaus wrote:
> With 59dbfb80f748 ("cargo.eclass: Add variable alternative to
> $(cargo_crate_uris)") the _cargo_set_crate_uris function was
> introduced. This function runs when the elcass is inherited and sets
> CARGO_CRATE_URIS.
> 
> Ebuilds that use $(cargo_crate_uris) in SRC_URI will again invoke this
> function, even though CARGO_CRATE_URIS is already set. Avoiding this
> unnecessary computation reduces the ebuild source time of
> app-shells/nushell-0.83.0 from 21 ms to 14 ms.
> 
> This is a significant reduction when compared to the variable-setting
> alternative that the commit 59dbfb80f748 ("cargo.eclass: Add variable
> alternative to $(cargo_crate_uris)") introduced. Using the
> variable-setting approach would reduce the ebuild source time only by a
> little bit more than one millisecond.
> 
> Using
> 
> pk pkg source --bench 10s '=app-shells/nushell-0.83.0'
> 
> as benchmark, yields
> 
> >                           | Cached CARGO_CRATE_URIS | Non-Cached CARGO_CRATE_URIS |
> > ---------------------------+-------------------------+-----------------------------|
> > $(cargo_crate_uris)       | mean: 14.189ms          | mean: 21.445ms              |
> > variable-setting approach | mean: 12.822ms          | mean: 12.852ms              |
> 
> full benchmark output
> 
> >                           | Cached CARGO_CRATE_URIS (this commit)                            | Non-Cached CARGO_CRATE_URIS                                      |
> > ---------------------------+------------------------------------------------------------------+------------------------------------------------------------------|
> > $(cargo_crate_uris)       | mean: 14.189ms, min: 13.646ms, max: 15.103ms, σ = 149µs, N = 705 | mean: 21.445ms, min: 20.79ms,  max: 22.832ms, σ = 228µs, N = 467 |
> > variable-setting approach | mean: 12.822ms, min: 12.41ms,  max: 13.909ms, σ = 165µs, N = 780 | mean: 12.852ms, min: 12.367ms, max: 15.437ms, σ = 227µs, N = 779 |
> 
> Signed-off-by: Florian Schmaus <flow@gentoo.org>
> ---
>  eclass/cargo.eclass | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
> index 70b6008d9cd8..5d6911801097 100644
> --- a/eclass/cargo.eclass
> +++ b/eclass/cargo.eclass
> @@ -240,6 +240,14 @@ _cargo_set_crate_uris "${CRATES}"
>  # Constructs a list of crates from its arguments.
>  # If no arguments are provided, it uses the CRATES variable.
>  cargo_crate_uris() {
> +	# Use already existing value for CARGO_CRATE_URIS, computed by
> +	# _cargo_set_crate_uris, when this function is invoked without
> +	# arguments.
> +	if [[ $# -eq 0 && -n "${CARGO_CRATE_URIS}" ]]; then
> +		echo "${CARGO_CRATE_URIS}"
> +		return
> +	fi
> +
>  	local crates=${*-${CRATES}}
>  	if [[ -z ${crates} ]]; then
>  		eerror "CRATES variable is not defined and nothing passed as argument"

This incorrectly assumes that the value of CRATES did not change which
isn't guaranteed anywhere.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] [PATCH] cargo.eclass: use CARGO_CRATE_URIS if already available

[Ionen Wolkens]

[-- Attachment #1: Type: text/plain, Size: 1243 bytes --]

On Mon, Jul 31, 2023 at 12:41:19PM +0200, Florian Schmaus wrote:
> With 59dbfb80f748 ("cargo.eclass: Add variable alternative to
> $(cargo_crate_uris)") the _cargo_set_crate_uris function was
> introduced. This function runs when the elcass is inherited and sets
> CARGO_CRATE_URIS.
> 
> Ebuilds that use $(cargo_crate_uris) in SRC_URI will again invoke this
> function, even though CARGO_CRATE_URIS is already set. Avoiding this
> unnecessary computation reduces the ebuild source time of
> app-shells/nushell-0.83.0 from 21 ms to 14 ms.

Note that pkgcheck (just) gained a check for this, ebuilds should
get fixed in time which would makes this change only an interim
thing and imo not all that worth it.

Aka:

app-shells/nushell$ pkgcheck scan
app-shells/nushell
  SuboptimalCratesSeparator: version 0.82.0: line: 9: using - as name-version separator in CRATES is suboptimal, use name@version instead
  SuboptimalCratesURICall: version 0.82.0: line: 601: calling '$(cargo_crate_uris)' is suboptimal, use '${CARGO_CRATE_URIS}' for global CRATES instead
  SuboptimalCratesURICall: version 0.83.0: line: 574: calling '$(cargo_crate_uris)' is suboptimal, use '${CARGO_CRATE_URIS}' for global CRATES instead

-- 
ionen

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] [PATCH] cargo.eclass: use CARGO_CRATE_URIS if already available

[Sam James]

Michał Górny <mgorny@gentoo.org> writes:

> On Mon, 2023-07-31 at 12:41 +0200, Florian Schmaus wrote:
>> With 59dbfb80f748 ("cargo.eclass: Add variable alternative to
>> $(cargo_crate_uris)") the _cargo_set_crate_uris function was
>> introduced. This function runs when the elcass is inherited and sets
>> CARGO_CRATE_URIS.
>> 
>> Ebuilds that use $(cargo_crate_uris) in SRC_URI will again invoke this
>> function, even though CARGO_CRATE_URIS is already set. Avoiding this
>> unnecessary computation reduces the ebuild source time of
>> app-shells/nushell-0.83.0 from 21 ms to 14 ms.
>> 
>> This is a significant reduction when compared to the variable-setting
>> alternative that the commit 59dbfb80f748 ("cargo.eclass: Add variable
>> alternative to $(cargo_crate_uris)") introduced. Using the
>> variable-setting approach would reduce the ebuild source time only by a
>> little bit more than one millisecond.
>> 
>> Using
>> 
>> pk pkg source --bench 10s '=app-shells/nushell-0.83.0'
>> 
>> as benchmark, yields
>> 
>> >                           | Cached CARGO_CRATE_URIS | Non-Cached CARGO_CRATE_URIS |
>> > ---------------------------+-------------------------+-----------------------------|
>> > $(cargo_crate_uris)       | mean: 14.189ms          | mean: 21.445ms              |
>> > variable-setting approach | mean: 12.822ms          | mean: 12.852ms              |
>> 
>> full benchmark output
>> 
>> >                           | Cached CARGO_CRATE_URIS (this commit)                            | Non-Cached CARGO_CRATE_URIS                                      |
>> > ---------------------------+------------------------------------------------------------------+------------------------------------------------------------------|
>> > $(cargo_crate_uris)       | mean: 14.189ms, min: 13.646ms, max: 15.103ms, σ = 149µs, N = 705 | mean: 21.445ms, min: 20.79ms,  max: 22.832ms, σ = 228µs, N = 467 |
>> > variable-setting approach | mean: 12.822ms, min: 12.41ms,  max: 13.909ms, σ = 165µs, N = 780 | mean: 12.852ms, min: 12.367ms, max: 15.437ms, σ = 227µs, N = 779 |
>> 
>> Signed-off-by: Florian Schmaus <flow@gentoo.org>
>> ---
>>  eclass/cargo.eclass | 8 ++++++++
>>  1 file changed, 8 insertions(+)
>> 
>> diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
>> index 70b6008d9cd8..5d6911801097 100644
>> --- a/eclass/cargo.eclass
>> +++ b/eclass/cargo.eclass
>> @@ -240,6 +240,14 @@ _cargo_set_crate_uris "${CRATES}"
>>  # Constructs a list of crates from its arguments.
>>  # If no arguments are provided, it uses the CRATES variable.
>>  cargo_crate_uris() {
>> +	# Use already existing value for CARGO_CRATE_URIS, computed by
>> +	# _cargo_set_crate_uris, when this function is invoked without
>> +	# arguments.
>> +	if [[ $# -eq 0 && -n "${CARGO_CRATE_URIS}" ]]; then
>> +		echo "${CARGO_CRATE_URIS}"
>> +		return
>> +	fi
>> +
>>  	local crates=${*-${CRATES}}
>>  	if [[ -z ${crates} ]]; then
>>  		eerror "CRATES variable is not defined and nothing passed as argument"
>
> This incorrectly assumes that the value of CRATES did not change which
> isn't guaranteed anywhere.

Ah, thanks! Now I remember - we discussed it before and rejected it for
this reason.