From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 1D03F1581F3 for ; Mon, 25 Nov 2024 03:36:31 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 4A5F9E08E2; Mon, 25 Nov 2024 03:36:01 +0000 (UTC) Received: from smtp.gentoo.org (woodpecker.gentoo.org [140.211.166.183]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 0307FE08D1 for ; Mon, 25 Nov 2024 03:36:01 +0000 (UTC) From: kangie@gentoo.org To: gentoo-dev@lists.gentoo.org Cc: Matt Jolly Subject: [gentoo-dev] [PATCH 1/3] cargo.eclass: add trivial crate overrides Date: Mon, 25 Nov 2024 13:35:28 +1000 Message-ID: <20241125033530.115757-2-kangie@gentoo.org> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241125033530.115757-1-kangie@gentoo.org> References: <20241125033530.115757-1-kangie@gentoo.org> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Archives-Salt: d47540c9-98e7-488d-8f4c-76dfec656073 X-Archives-Hash: b538644d241e49e8ba73dbcebe033e2b From: Matt Jolly Updating vulnerable (or otherwise outdated) crates in Rust ebuilds is painful. Generally speaking, there are 5 options: - Run `cargo update` to fetch new versions from the web. This is obviously not suitable for use in Portage. - Patch the software via Portage to accept a non-vulnerable crate. This is a reasonable option when the package is not too complex but still requires significant developer effort and some familiarity with Cargo. In the case of complex patches this may not be feasible, or require the generation of a dependency tarball. - [patch] the source (repository) in Cargo.toml. This enables the targeting of specific crates, but does not allow the replacement of only a specific version in the depgraph. - [replace] a particular crate:version in the Cargo.toml. This enables the targeting of a particular version with an arbitrary path however the replacement crate must *have the same version* as the one being overridden. - `paths = [...]` overrides: pass an array of paths to directories that contain a Cargo.toml. Cargo will override any crate with the same package name arbitrarily, ignoring the lock file and versions; typically used for testing. Is applied via ${CARGO_HOME}/config.toml (i.e. globally) This commit: - Implements the `paths` overrides, which will work even when Cargo is configured to use a vendored directory. This is not a 'smart' replacement and care must be taken to ensure that all versions of the crate in use are compatible (`cargo tree` will help). - Provides a helper which runs `cargo --update --offline` against ${ECARGO_VENDOR} (where ${CRATES} are unpacked). This enables the replacement of vulnerable versions in ${CRATES}. It is up to the consumer to ensure that only the desired crates are being replaced and that package behaviour does not change. - Adds a new `CARGO_BOOTSTRAP` variable which enables packages to ignore the minimum version requirement of the eclass. This is only used for bootstrapping Rust; if it's being used in any non dev-lang/rust ebuilds be sure that you have a good reason. Resources: - https://doc.rust-lang.org/cargo/reference/overriding-dependencies.html - https://github.com/rust-lang/cargo/issues/3308 Signed-off-by: Matt Jolly --- eclass/cargo.eclass | 115 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 109 insertions(+), 6 deletions(-) diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass index 95ff317e1f21..a49ef818a351 100644 --- a/eclass/cargo.eclass +++ b/eclass/cargo.eclass @@ -7,6 +7,7 @@ # @AUTHOR: # Doug Goldstein # Georgy Yakovlev +# Matt Jolly # @SUPPORTED_EAPIS: 8 # @PROVIDES: rust # @BLURB: common functions and variables for cargo builds @@ -37,8 +38,10 @@ case ${EAPI} in if [[ -n ${RUST_MIN_VER} ]]; then # This is _very_ unlikely given that we leverage the rust eclass but just in case cargo requires a newer version # than the oldest in-tree in future. - if ver_test "${RUST_MIN_VER}" -lt "${_CARGO_ECLASS_RUST_MIN_VER}"; then - die "RUST_MIN_VERSION must be at least ${_CARGO_ECLASS_RUST_MIN_VER}" + if [[ -z ${CARGO_BOOTSTRAP} ]]; then + if ver_test "${RUST_MIN_VER}" -lt "${_CARGO_ECLASS_RUST_MIN_VER}"; then + die "RUST_MIN_VERSION must be at least ${_CARGO_ECLASS_RUST_MIN_VER}" + fi fi else RUST_MIN_VER="${_CARGO_ECLASS_RUST_MIN_VER}" @@ -46,6 +49,10 @@ case ${EAPI} in ;; esac +if [[ -n ${CRATE_PATHS_OVERRIDE} ]]; then + CRATES="${CRATES} ${CRATE_PATHS_OVERRIDE}" +fi + inherit flag-o-matic multiprocessing rust rust-toolchain toolchain-funcs IUSE="${IUSE} debug" @@ -76,6 +83,39 @@ ECARGO_VENDOR="${ECARGO_HOME}/gentoo" # SRC_URI="${CARGO_CRATE_URIS}" # @CODE +# @ECLASS_VARIABLE: CRATE_PATHS_OVERRIDE +# @DEFAULT_UNSET +# @PRE_INHERIT +# @DESCRIPTION: +# Bash string containing paths to crates that will be used to override +# dependencies. This is not "smart", _all crates_ which match the +# `Cargo.toml` in a path will be overridden, ignoring lockfiles, +# version constraints, etc. +# +# This should be used as a last resort where (e.g.) you are +# bootstrapping Rust and need to override a vendored crate +# with a newer version, and all versions in use are compatible. +# +# Crate names and versions must be separated by a `@`; +# multiple crates are separated by a space or newline. +# Crates in CRATE_PATHS_OVERRIDE are implicitly added to CRATES. +# +# Example: +# @CODE +# CRATES=" +# foo@1.2.3 +# " +# +# CRATE_PATHS_OVERRIDE=" +# openssl@0.10.35 +# openssl-sys@0.9.65 +# " +# +# inherit cargo +# ... +# SRC_URI="${CARGO_CRATE_URIS}" +# @CODE + # @ECLASS_VARIABLE: GIT_CRATES # @DEFAULT_UNSET # @PRE_INHERIT @@ -109,6 +149,13 @@ ECARGO_VENDOR="${ECARGO_HOME}/gentoo" # ) # @CODE +# @ECLASS_VARIABLE: CARGO_BOOTSTRAP +# @DEFAULT_UNSET +# @PRE_INHERIT +# @DESCRIPTION: +# Ignore `_CARGO_ECLASS_RUST_MIN_VER` checks. +# If you aren't bootstrapping Rust you probably don't need this. + # @ECLASS_VARIABLE: CARGO_OPTIONAL # @DEFAULT_UNSET # @PRE_INHERIT @@ -265,6 +312,26 @@ cargo_crate_uris() { echo "${CARGO_CRATE_URIS}" } +# @FUNCTION: _cargo_gen_override_paths_config +# @INTERNAL +# @DESCRIPTION: +# Generate the TOML content for overriding crates using the package manager. +# This is called from within cargo_gen_config to insert the appropriate snippet +# into the generated config.toml. Does not support git crates. +_cargo_gen_override_paths_config() { + if [[ ! ${#CRATE_PATHS_OVERRIDE[@]} -gt 0 ]]; then + return + fi + local content override path + content=( 'paths = [' ) + for override in ${CRATE_PATHS_OVERRIDE}; do + local path="${ECARGO_VENDOR}/${override//@/-}" + content+=( "'${path}'," ) + done + content+=( ']' ) + printf "%s\n" "${content[@]}" +} + # @FUNCTION: cargo_gen_config # @DESCRIPTION: # Generate the $CARGO_HOME/config.toml necessary to use our local registry and settings. @@ -281,6 +348,8 @@ cargo_gen_config() { mkdir -p "${ECARGO_HOME}" || die cat > "${ECARGO_HOME}/config.toml" <<- _EOF_ || die "Failed to create cargo config" + $(_cargo_gen_override_paths_config) + [source.gentoo] directory = "${ECARGO_VENDOR}" @@ -299,6 +368,7 @@ cargo_gen_config() { verbose = true $([[ "${NOCOLOR}" = true || "${NOCOLOR}" = yes ]] && echo "color = 'never'") $(_cargo_gen_git_config) + _EOF_ export CARGO_HOME="${ECARGO_HOME}" @@ -321,16 +391,16 @@ _cargo_gen_git_config() { if [[ ${git_crates_type} == "declare -A "* ]]; then local crate commit crate_uri crate_dir - local -A crate_patches + local -A CRATE_UPDATES for crate in "${!GIT_CRATES[@]}"; do IFS=';' read -r crate_uri commit crate_dir <<< "${GIT_CRATES[${crate}]}" : "${crate_dir:=${crate}-%commit%}" - crate_patches["${crate_uri}"]+="${crate} = { path = \"${WORKDIR}/${crate_dir//%commit%/${commit}}\" };;" + CRATE_UPDATES["${crate_uri}"]+="${crate} = { path = \"${WORKDIR}/${crate_dir//%commit%/${commit}}\" };;" done - for crate_uri in "${!crate_patches[@]}"; do - printf -- "[patch.'%s']\\n%s\n" "${crate_uri}" "${crate_patches["${crate_uri}"]//;;/$'\n'}" + for crate_uri in "${!CRATE_UPDATES[@]}"; do + printf -- "[patch.'%s']\\n%s\n" "${crate_uri}" "${CRATE_UPDATES["${crate_uri}"]//;;/$'\n'}" done elif [[ -n ${git_crates_type} ]]; then @@ -346,6 +416,39 @@ cargo_target_dir() { echo "${CARGO_TARGET_DIR:-target}/$(rust_abi)/$(usex debug debug release)" } +# @FUNCTION: cargo_update_crates +# @USAGE: +# @DESCRIPTION: +# Helper function to call `cargo update --offline` with the given Cargo.toml. +# This will update Cargo.{toml,lock}. This should provide a straightforward +# approach to updating vulnerable crates in a package. +# +# To use: replace any vulnerable crates in ${CRATES} with updated (and compatible) +# versions, then call `cargo_update_crates` in src_prepare. If Cargo.toml is not +# in the root of ${S}, pass the path to the Cargo.toml as the first argument. +# It is up to the ebuild to ensure that the updated crates are compatible with the +# package and that no unexpected breakage occurs. +cargo_update_crates () { + debug-print-function ${FUNCNAME} "$@" + + if [[ -z ${CARGO} ]]; then + die "CARGO is not set; was rust_pkg_setup run?" + fi + + local path="${S}/Cargo.toml" + if [[ $# -eq 1 ]]; then + path="${1}" + elif [[ $# -gt 1 ]]; then + die "Usage: cargo_update_crates [path_to_Cargo.toml]" + fi + [[ -f ${path} ]] || die "${path} does not exist" + + set -- "${CARGO}" update --offline --manifest-path ${path} + einfo "${@}" + # This is overkill (we're not using rustflags (etc) here) but it's safe. + cargo_env "${@}" || die "Failed to update crates" +} + # @FUNCTION: cargo_src_unpack # @DESCRIPTION: # Unpacks the package and the cargo registry. -- 2.47.0