From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id EBC9A158064 for ; Sun, 12 May 2024 02:44:51 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 077DEE29EB; Sun, 12 May 2024 02:44:48 +0000 (UTC) Received: from smtp.gentoo.org (woodpecker.gentoo.org [140.211.166.183]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id A79EAE29E5 for ; Sun, 12 May 2024 02:44:47 +0000 (UTC) From: Sam James To: =?utf-8?B?TWljaGHFgiBHw7Nybnk=?= Cc: gentoo-dev@lists.gentoo.org Subject: Re: [gentoo-dev] [PATCH] cargo.eclass: Optimize crate unpacking In-Reply-To: <20240512022603.48576-1-mgorny@gentoo.org> (=?utf-8?Q?=22Mich?= =?utf-8?Q?a=C5=82_G=C3=B3rny=22's?= message of "Sun, 12 May 2024 04:26:03 +0200") Organization: Gentoo References: <20240512022603.48576-1-mgorny@gentoo.org> Date: Sun, 12 May 2024 03:44:43 +0100 Message-ID: <871q67iw9w.fsf@gentoo.org> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Archives-Salt: 815c215a-acc7-4148-acbb-f5fa0fdf0c45 X-Archives-Hash: 2e1deb9cd198a78605951315b1b8dab6 Micha=C5=82 G=C3=B3rny writes: > Unpack crates in parallel using xargs to utilize multicore systems > better. Perform checksumming via a single sha256sum invocation. > > For dev-python/watchfiles, this speeds up unpacking on my machine > from 2.6 s to 0.75 s (warm cache). > > Signed-off-by: Micha=C5=82 G=C3=B3rny For completeness (acked on PR), lgtm. I assume tested by building all cargo inheritees (nearly said 'heirs' but.. lol)? Give a chance for others to look though. Also, thank you! > --- > eclass/cargo.eclass | 56 ++++++++++++++++++++++++++------------------- > 1 file changed, 33 insertions(+), 23 deletions(-) > > diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass > index 0f2da982f60c..5a16d3a30528 100644 > --- a/eclass/cargo.eclass > +++ b/eclass/cargo.eclass > @@ -329,40 +329,50 @@ _cargo_gen_git_config() { > cargo_src_unpack() { > debug-print-function ${FUNCNAME} "$@" >=20=20 > - mkdir -p "${ECARGO_VENDOR}" || die > - mkdir -p "${S}" || die > + mkdir -p "${ECARGO_VENDOR}" "${S}" || die >=20=20 > local archive shasum pkg > + local crates=3D() > for archive in ${A}; do > case "${archive}" in > *.crate) > - # when called by pkgdiff-mg, do not unpack crates > - [[ ${PKGBUMPING} =3D=3D ${PVR} ]] && continue > - > - ebegin "Loading ${archive} into Cargo registry" > - tar -xf "${DISTDIR}"/${archive} -C "${ECARGO_VENDOR}/" || die > - # generate sha256sum of the crate itself as cargo needs this > - shasum=3D$(sha256sum "${DISTDIR}"/${archive} | cut -d ' ' -f 1) > - pkg=3D$(basename ${archive} .crate) > - cat <<- EOF > ${ECARGO_VENDOR}/${pkg}/.cargo-checksum.json > - { > - "package": "${shasum}", > - "files": {} > - } > - EOF > - # if this is our target package we need it in ${WORKDIR} too > - # to make ${S} (and handle any revisions too) > - if [[ ${P} =3D=3D ${pkg}* ]]; then > - tar -xf "${DISTDIR}"/${archive} -C "${WORKDIR}" || die > - fi > - eend $? > + crates+=3D( "${archive}" ) > ;; > *) > - unpack ${archive} > + unpack "${archive}" > ;; > esac > done >=20=20 > + if [[ ${PKGBUMPING} !=3D ${PVR} ]]; then > + pushd "${DISTDIR}" >/dev/null || die > + > + ebegin "Unpacking crates" > + printf '%s\0' "${crates[@]}" | > + xargs -0 -P "$(makeopts_jobs)" -n 1 -- \ > + tar -x -C "${ECARGO_VENDOR}" -f > + assert > + eend $? > + > + while read -d '' -r shasum archive; do > + pkg=3D${archive%.crate} > + cat <<- EOF > ${ECARGO_VENDOR}/${pkg}/.cargo-checksum.json || die > + { > + "package": "${shasum}", > + "files": {} > + } > + EOF > + > + # if this is our target package we need it in ${WORKDIR} too > + # to make ${S} (and handle any revisions too) > + if [[ ${P} =3D=3D ${pkg}* ]]; then > + tar -xf "${archive}" -C "${WORKDIR}" || die > + fi > + done < <(sha256sum -z "${crates[@]}" || die) > + > + popd >/dev/null || die > + fi > + > cargo_gen_config > }