[gentoo-dev] cargo.eclass improvements

[gentoo-dev] cargo.eclass improvements

[Georgy Yakovlev]

Series of patches to cargo.eclass
most important chages are:
1. minimum rust/cargo version bump, which will propogate to all
consumers.
2. introduction of custom build profile (similar to what we do in Cmake)
 This will allow to consistently specify build options, yeat allows
 overrides from users or developers via envvars and/or args.

PR: https://github.com/gentoo/gentoo/pull/29510

PS: still WIP, but most changes are ready.
Another upcoming change is to allow running 'cargo update' to be able to
bump vulnerable/outdated pkgs listed in CRATES.

[gentoo-dev] [PATCH 1/7] cargo.eclass: bump minimum rust to 1.57.0

[Georgy Yakovlev]

and mark RUST_DEPEND as readonly

Signed-off-by: Georgy Yakovlev <gyakovlev@gentoo.org>
---
 eclass/cargo.eclass | 28 ++++++++++++++++------------
 1 file changed, 16 insertions(+), 12 deletions(-)

diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
index a92fe97ec502..d37293ada136 100644
--- a/eclass/cargo.eclass
+++ b/eclass/cargo.eclass
@@ -13,28 +13,32 @@
 if [[ -z ${_CARGO_ECLASS} ]]; then
 _CARGO_ECLASS=1
 
-# check and document RUST_DEPEND and options we need below in case conditions.
+# @ECLASS_VARIABLE: RUST_DEPEND
+# @INTERNAL
+# @DESCRIPTION:
+# Minimum rust/cargo version that should be used with this eclass.
+# Versions below that may not provide functionality we rely on.
+# This variable should not be overriden by ebuilds, just manually
+# add more recent version if package requires it.
 # https://github.com/rust-lang/cargo/blob/master/CHANGELOG.md
-RUST_DEPEND="virtual/rust"
+# 1.37 added 'cargo vendor' subcommand and net.offline config knob
+# 1.39 added --workspace
+# 1.46 added --target dir
+# 1.48 added term.progress config option
+# 1.51 added split-debuginfo profile option
+# 1.52 may need setting RUSTC_BOOTSTRAP envvar for some crates
+# 1.53 added cargo update --offline, can be used to update vulnerable crates from pre-fetched registry without editing toml
+# 1.57 added named profile support in stable
+readonly RUST_DEPEND=">=virtual/rust-1.57"
 
 case "${EAPI:-0}" in
 	0|1|2|3|4|5|6)
 		die "Unsupported EAPI=${EAPI:-0} (too old) for ${ECLASS}"
 		;;
 	7)
-		# 1.37 added 'cargo vendor' subcommand and net.offline config knob
-		RUST_DEPEND=">=virtual/rust-1.37.0"
 		;;
 
 	8)
-		# 1.39 added --workspace
-		# 1.46 added --target dir
-		# 1.48 added term.progress config option
-		# 1.51 added split-debuginfo profile option
-		# 1.52 may need setting RUSTC_BOOTSTRAP envvar for some crates
-		# 1.53 added cargo update --offline, can be used to update vulnerable crates from pre-fetched registry without editing toml
-		RUST_DEPEND=">=virtual/rust-1.53"
-
 		if [[ -z ${CRATES} && "${PV}" != *9999* ]]; then
 			eerror "undefined CRATES variable in non-live EAPI=8 ebuild"
 			die "CRATES variable not defined"
-- 
2.39.1

[gentoo-dev] [PATCH 2/7] cargo.eclass: pass --no-track to cargo install

[Georgy Yakovlev]

and drop file removal hack.
with --no-track those files are never created.

Signed-off-by: Georgy Yakovlev <gyakovlev@gentoo.org>
---
 eclass/cargo.eclass | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
index d37293ada136..1a8d665fdad2 100644
--- a/eclass/cargo.eclass
+++ b/eclass/cargo.eclass
@@ -516,6 +516,7 @@ cargo_src_install() {
 		die "FATAL: please call cargo_gen_config before using ${FUNCNAME}"
 
 	set -- cargo install $(has --path ${@} || echo --path ./) \
+		--no-track \
 		--root "${ED}/usr" \
 		${GIT_CRATES[@]:+--frozen} \
 		$(usex debug --debug "") \
@@ -523,9 +524,6 @@ cargo_src_install() {
 	einfo "${@}"
 	"${@}" || die "cargo install failed"
 
-	rm -f "${ED}/usr/.crates.toml" || die
-	rm -f "${ED}/usr/.crates2.json" || die
-
 	# it turned out to be non-standard dir, so get rid of it future EAPI
 	# and only run for EAPI=7
 	# https://bugs.gentoo.org/715890
-- 
2.39.1

[gentoo-dev] [PATCH 3/7] cargo.eclass: document undocumented variables, mark as readonly

[Georgy Yakovlev]

Signed-off-by: Georgy Yakovlev <gyakovlev@gentoo.org>
---
 eclass/cargo.eclass | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
index 1a8d665fdad2..9c624d607cdd 100644
--- a/eclass/cargo.eclass
+++ b/eclass/cargo.eclass
@@ -58,8 +58,15 @@ fi
 
 IUSE="${IUSE} debug"
 
-ECARGO_HOME="${WORKDIR}/cargo_home"
-ECARGO_VENDOR="${ECARGO_HOME}/gentoo"
+# @ECLASS_VARIABLE: ECARGO_HOME
+# @DESCRIPTION:
+# Directory for CARGO_HOME used by build process.
+readonly ECARGO_HOME="${WORKDIR}/cargo_home"
+
+# @ECLASS_VARIABLE: ECARGO_VENDOR
+# @DESCRIPTION:
+# Directory for 'cargo vendor' subcommand output.
+readonly ECARGO_VENDOR="${ECARGO_HOME}/gentoo"
 
 # @ECLASS_VARIABLE: CRATES
 # @DEFAULT_UNSET
-- 
2.39.1

[gentoo-dev] [PATCH 4/7] cargo.eclass: set progress.when = "never" in config

[Georgy Yakovlev]

to avoid possible log file pollution

Signed-off-by: Georgy Yakovlev <gyakovlev@gentoo.org>
---
 eclass/cargo.eclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
index 9c624d607cdd..0ab7ee0dc9b2 100644
--- a/eclass/cargo.eclass
+++ b/eclass/cargo.eclass
@@ -273,6 +273,7 @@ cargo_gen_config() {
 	[term]
 	verbose = true
 	$([[ "${NOCOLOR}" = true || "${NOCOLOR}" = yes ]] && echo "color = 'never'")
+	progress.when = "never"
 	$(_cargo_gen_git_config)
 	_EOF_
 
-- 
2.39.1

[gentoo-dev] [PATCH 5/7] cargo.eclass: use custom profile for all builds

[Georgy Yakovlev]

also move install path to config file, so it can be
overriden via command line arg if required.

Signed-off-by: Georgy Yakovlev <gyakovlev@gentoo.org>
---
 eclass/cargo.eclass | 33 ++++++++++++++++++++++++++-------
 1 file changed, 26 insertions(+), 7 deletions(-)

diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
index 0ab7ee0dc9b2..a7c7bffd3c0c 100644
--- a/eclass/cargo.eclass
+++ b/eclass/cargo.eclass
@@ -266,10 +266,26 @@ cargo_gen_config() {
 	[net]
 	offline = true
 
+	[profile.gentoo]
+	# https://doc.rust-lang.org/cargo/reference/profiles.html#custom-profiles
+	inherits = "release"
+
+	# emulate dev profile with USE=debug
+	# https://doc.rust-lang.org/cargo/reference/profiles.html#dev
+	debug = $(usex debug true false)
+	debug-assertions = $(usex debug true false)
+	overflow-checks = $(usex debug true false)
+	strip = "none"
+	$(usex debug 'opt-level = 0' '')
+	$(usex debug 'lto = false' '')
+
 	[build]
 	jobs = $(makeopts_jobs)
 	incremental = false
 
+	[install]
+	root = "${ED}/usr"
+
 	[term]
 	verbose = true
 	$([[ "${NOCOLOR}" = true || "${NOCOLOR}" = yes ]] && echo "color = 'never'")
@@ -506,7 +522,7 @@ cargo_src_compile() {
 
 	tc-export AR CC CXX PKG_CONFIG
 
-	set -- cargo build $(usex debug "" --release) ${ECARGO_ARGS[@]} "$@"
+	set -- cargo build --profile gentoo ${ECARGO_ARGS[@]} "$@"
 	einfo "${@}"
 	"${@}" || die "cargo build failed"
 }
@@ -524,14 +540,17 @@ cargo_src_install() {
 		die "FATAL: please call cargo_gen_config before using ${FUNCNAME}"
 
 	set -- cargo install $(has --path ${@} || echo --path ./) \
-		--no-track \
-		--root "${ED}/usr" \
-		${GIT_CRATES[@]:+--frozen} \
-		$(usex debug --debug "") \
-		${ECARGO_ARGS[@]} "$@"
+		--profile gentoo --no-track \
+		 ${GIT_CRATES[@]:+--frozen} ${ECARGO_ARGS[@]} "$@"
 	einfo "${@}"
 	"${@}" || die "cargo install failed"
 
+	# HACK: compat symlinks until old ebuilds migrate.
+	# create target/{debug,release} symlinks that some ebuilds rely on.
+	# This only affects ebuilds that pick extra generated files from target directory, that's rare.
+	ln -s gentoo "${S}"/target/debug || :
+	ln -s gentoo "${S}"/target/release || :
+
 	# it turned out to be non-standard dir, so get rid of it future EAPI
 	# and only run for EAPI=7
 	# https://bugs.gentoo.org/715890
@@ -553,7 +572,7 @@ cargo_src_test() {
 	[[ ${_CARGO_GEN_CONFIG_HAS_RUN} ]] || \
 		die "FATAL: please call cargo_gen_config before using ${FUNCNAME}"
 
-	set -- cargo test $(usex debug "" --release) ${ECARGO_ARGS[@]} "$@"
+	set -- cargo test --profile gentoo ${ECARGO_ARGS[@]} "$@"
 	einfo "${@}"
 	"${@}" || die "cargo test failed"
 }
-- 
2.39.1

[gentoo-dev] [PATCH 6/7] cargo.eclass: set codegen-units = 1

[Georgy Yakovlev]

This might increase build and lto times a bit,
but may result in faster and better optimized result.
It also honors resource limits properly.

Signed-off-by: Georgy Yakovlev <gyakovlev@gentoo.org>
---
 eclass/cargo.eclass | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
index a7c7bffd3c0c..00b8078f80ea 100644
--- a/eclass/cargo.eclass
+++ b/eclass/cargo.eclass
@@ -279,6 +279,11 @@ cargo_gen_config() {
 	$(usex debug 'opt-level = 0' '')
 	$(usex debug 'lto = false' '')
 
+	# https://doc.rust-lang.org/rustc/codegen-options/index.html#codegen-units
+	# We use single codegen unit for most optimized code and to honor -j from MAKEOPTS.
+	# Users can override via e.g. CARGO_PROFILE_gentoo_CODEGEN_UNITS="16" in make.conf.
+	codegen-units = 1
+
 	[build]
 	jobs = $(makeopts_jobs)
 	incremental = false
-- 
2.39.1

[gentoo-dev] [PATCH 7/7] cargo.eclass: filter out lto flags for C/CXX compilers

[Georgy Yakovlev]

we do it in src_compile to avoid excessive flag stripping in projects
using cargo.eclass just to fetch crates.

Closes: https://bugs.gentoo.org/893658
Signed-off-by: Georgy Yakovlev <gyakovlev@gentoo.org>
---
 eclass/cargo.eclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
index 00b8078f80ea..27ccee1295b3 100644
--- a/eclass/cargo.eclass
+++ b/eclass/cargo.eclass
@@ -49,7 +49,7 @@ case "${EAPI:-0}" in
 		;;
 esac
 
-inherit multiprocessing toolchain-funcs
+inherit flag-o-matic multiprocessing toolchain-funcs
 
 if [[ ! ${CARGO_OPTIONAL} ]]; then
 	BDEPEND="${RUST_DEPEND}"
@@ -525,6 +525,7 @@ cargo_src_compile() {
 	[[ ${_CARGO_GEN_CONFIG_HAS_RUN} ]] || \
 		die "FATAL: please call cargo_gen_config before using ${FUNCNAME}"
 
+	filter-lto
 	tc-export AR CC CXX PKG_CONFIG
 
 	set -- cargo build --profile gentoo ${ECARGO_ARGS[@]} "$@"
-- 
2.39.1

Re: [gentoo-dev] [PATCH 5/7] cargo.eclass: use custom profile for all builds

[Ionen Wolkens]

[-- Attachment #1: Type: text/plain, Size: 305 bytes --]

On Thu, Feb 09, 2023 at 12:48:45PM -0800, Georgy Yakovlev wrote:
> +	strip = "none"

strip was stabilized in rust-1.59, this will likely fail without
USE=nightly on older rusts and other patch is only setting >=1.57

Haven't checked other options, I just happened to remember this one.
-- 
ionen

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-dev] [PATCH 4/7] cargo.eclass: set progress.when = "never" in config

[Michał Górny]

On Thu, 2023-02-09 at 12:48 -0800, Georgy Yakovlev wrote:
> to avoid possible log file pollution
> 
> Signed-off-by: Georgy Yakovlev <gyakovlev@gentoo.org>
> ---
>  eclass/cargo.eclass | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
> index 9c624d607cdd..0ab7ee0dc9b2 100644
> --- a/eclass/cargo.eclass
> +++ b/eclass/cargo.eclass
> @@ -273,6 +273,7 @@ cargo_gen_config() {
>  	[term]
>  	verbose = true
>  	$([[ "${NOCOLOR}" = true || "${NOCOLOR}" = yes ]] && echo "color = 'never'")
> +	progress.when = "never"
>  	$(_cargo_gen_git_config)
>  	_EOF_
>  

What's that and why don't we want it?  I thought we generally preferred
more verbosity.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] [PATCH 4/7] cargo.eclass: set progress.when = "never" in config

[Georgy Yakovlev]

On Fri, 2023-02-10 at 07:03 +0100, Michał Górny wrote:
> On Thu, 2023-02-09 at 12:48 -0800, Georgy Yakovlev wrote:
> > to avoid possible log file pollution
> > 
> > Signed-off-by: Georgy Yakovlev <gyakovlev@gentoo.org>
> > ---
> >  eclass/cargo.eclass | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
> > index 9c624d607cdd..0ab7ee0dc9b2 100644
> > --- a/eclass/cargo.eclass
> > +++ b/eclass/cargo.eclass
> > @@ -273,6 +273,7 @@ cargo_gen_config() {
> >         [term]
> >         verbose = true
> >         $([[ "${NOCOLOR}" = true || "${NOCOLOR}" = yes ]] && echo
> > "color = 'never'")
> > +       progress.when = "never"
> >         $(_cargo_gen_git_config)
> >         _EOF_
> >  
> 
> What's that and why don't we want it?  I thought we generally
> preferred
> more verbosity.
> 
This knob controls progress bar that can pollute logs in rare cases,
not build verbosity. verbose is set just above.

Re: [gentoo-dev] [PATCH 5/7] cargo.eclass: use custom profile for all builds

[Georgy Yakovlev]

On Thu, 2023-02-09 at 16:35 -0500, Ionen Wolkens wrote:
> On Thu, Feb 09, 2023 at 12:48:45PM -0800, Georgy Yakovlev wrote:
> > +       strip = "none"
> 
> strip was stabilized in rust-1.59, this will likely fail without
> USE=nightly on older rusts and other patch is only setting >=1.57
> 
> Haven't checked other options, I just happened to remember this one.
might have missed one, thanks.
Will re-check and if it's the case - document it above minimum req line
and bump it.

Re: [gentoo-dev] [PATCH 4/7] cargo.eclass: set progress.when = "never" in config

[Michał Górny]

On Fri, 2023-02-10 at 13:13 -0800, Georgy Yakovlev wrote:
> On Fri, 2023-02-10 at 07:03 +0100, Michał Górny wrote:
> > On Thu, 2023-02-09 at 12:48 -0800, Georgy Yakovlev wrote:
> > > to avoid possible log file pollution
> > > 
> > > Signed-off-by: Georgy Yakovlev <gyakovlev@gentoo.org>
> > > ---
> > >  eclass/cargo.eclass | 1 +
> > >  1 file changed, 1 insertion(+)
> > > 
> > > diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass
> > > index 9c624d607cdd..0ab7ee0dc9b2 100644
> > > --- a/eclass/cargo.eclass
> > > +++ b/eclass/cargo.eclass
> > > @@ -273,6 +273,7 @@ cargo_gen_config() {
> > >         [term]
> > >         verbose = true
> > >         $([[ "${NOCOLOR}" = true || "${NOCOLOR}" = yes ]] && echo
> > > "color = 'never'")
> > > +       progress.when = "never"
> > >         $(_cargo_gen_git_config)
> > >         _EOF_
> > >  
> > 
> > What's that and why don't we want it?  I thought we generally
> > preferred
> > more verbosity.
> > 
> This knob controls progress bar that can pollute logs in rare cases,
> not build verbosity. verbose is set just above.
> 

Ah, ok.  I thought it was maybe "output something every N minutes when
nothing else visible is happening".

-- 
Best regards,
Michał Górny