From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 2B4D91581D3 for ; Fri, 17 May 2024 06:26:05 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 301C0E2A80; Fri, 17 May 2024 06:26:03 +0000 (UTC) Received: from smtp.gentoo.org (dev.gentoo.org [IPv6:2001:470:ea4a:1:5054:ff:fec7:86e4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 170AFE2A80 for ; Fri, 17 May 2024 06:26:03 +0000 (UTC) Received: from oystercatcher.gentoo.org (oystercatcher.gentoo.org [148.251.78.52]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTPS id 3F56933C84E for ; Fri, 17 May 2024 06:26:02 +0000 (UTC) Received: from localhost.localdomain (localhost [IPv6:::1]) by oystercatcher.gentoo.org (Postfix) with ESMTP id ECAC61AE6 for ; Fri, 17 May 2024 06:25:58 +0000 (UTC) From: "Michał Górny" To: gentoo-commits@lists.gentoo.org Content-Transfer-Encoding: 8bit Content-type: text/plain; charset=UTF-8 Reply-To: gentoo-dev@lists.gentoo.org, "Michał Górny" Message-ID: <1715927148.bf9a8b7af96bb7161956fc8276d9d07f021bda8a.mgorny@gentoo> Subject: [gentoo-commits] repo/gentoo:master commit in: eclass/ X-VCS-Repository: repo/gentoo X-VCS-Files: eclass/cargo.eclass X-VCS-Directories: eclass/ X-VCS-Committer: mgorny X-VCS-Committer-Name: Michał Górny X-VCS-Revision: bf9a8b7af96bb7161956fc8276d9d07f021bda8a X-VCS-Branch: master Date: Fri, 17 May 2024 06:25:58 +0000 (UTC) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-commits@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply X-Archives-Salt: 62e29e99-852d-4f6b-9a1b-6ac8a1fd4643 X-Archives-Hash: 91eb9e9313a480d28a0ff6ce7aabbc7b commit: bf9a8b7af96bb7161956fc8276d9d07f021bda8a Author: Michał Górny gentoo org> AuthorDate: Sat May 11 18:53:28 2024 +0000 Commit: Michał Górny gentoo org> CommitDate: Fri May 17 06:25:48 2024 +0000 URL: https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=bf9a8b7a cargo.eclass: Optimize crate unpacking Unpack crates in parallel using xargs to utilize multicore systems better. Perform checksumming via a single sha256sum invocation. For dev-python/watchfiles, this speeds up unpacking on my machine from 2.6 s to 0.75 s (warm cache). Signed-off-by: Michał Górny gentoo.org> Closes: https://github.com/gentoo/gentoo/pull/36645 Signed-off-by: Michał Górny gentoo.org> eclass/cargo.eclass | 56 +++++++++++++++++++++++++++++++---------------------- 1 file changed, 33 insertions(+), 23 deletions(-) diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass index 0f2da982f60c..a685cd99fb38 100644 --- a/eclass/cargo.eclass +++ b/eclass/cargo.eclass @@ -329,40 +329,50 @@ _cargo_gen_git_config() { cargo_src_unpack() { debug-print-function ${FUNCNAME} "$@" - mkdir -p "${ECARGO_VENDOR}" || die - mkdir -p "${S}" || die + mkdir -p "${ECARGO_VENDOR}" "${S}" || die local archive shasum pkg + local crates=() for archive in ${A}; do case "${archive}" in *.crate) - # when called by pkgdiff-mg, do not unpack crates - [[ ${PKGBUMPING} == ${PVR} ]] && continue - - ebegin "Loading ${archive} into Cargo registry" - tar -xf "${DISTDIR}"/${archive} -C "${ECARGO_VENDOR}/" || die - # generate sha256sum of the crate itself as cargo needs this - shasum=$(sha256sum "${DISTDIR}"/${archive} | cut -d ' ' -f 1) - pkg=$(basename ${archive} .crate) - cat <<- EOF > ${ECARGO_VENDOR}/${pkg}/.cargo-checksum.json - { - "package": "${shasum}", - "files": {} - } - EOF - # if this is our target package we need it in ${WORKDIR} too - # to make ${S} (and handle any revisions too) - if [[ ${P} == ${pkg}* ]]; then - tar -xf "${DISTDIR}"/${archive} -C "${WORKDIR}" || die - fi - eend $? + crates+=( "${archive}" ) ;; *) - unpack ${archive} + unpack "${archive}" ;; esac done + if [[ ${PKGBUMPING} != ${PVR} && ${crates[@]} ]]; then + pushd "${DISTDIR}" >/dev/null || die + + ebegin "Unpacking crates" + printf '%s\0' "${crates[@]}" | + xargs -0 -P "$(makeopts_jobs)" -n 1 -t -- \ + tar -x -C "${ECARGO_VENDOR}" -f + assert + eend $? + + while read -d '' -r shasum archive; do + pkg=${archive%.crate} + cat <<- EOF > ${ECARGO_VENDOR}/${pkg}/.cargo-checksum.json || die + { + "package": "${shasum}", + "files": {} + } + EOF + + # if this is our target package we need it in ${WORKDIR} too + # to make ${S} (and handle any revisions too) + if [[ ${P} == ${pkg}* ]]; then + tar -xf "${archive}" -C "${WORKDIR}" || die + fi + done < <(sha256sum -z "${crates[@]}" || die) + + popd >/dev/null || die + fi + cargo_gen_config }