From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 27036158064 for ; Sun, 12 May 2024 18:21:37 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 1FEFDE2AAA; Sun, 12 May 2024 18:21:33 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [IPv6:2001:470:ea4a:1:5054:ff:fec7:86e4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id C9C09E2AA7 for ; Sun, 12 May 2024 18:21:32 +0000 (UTC) Message-ID: <849a540bd028ac0fed47c75c71ecf077f703c28d.camel@gentoo.org> Subject: Re: [gentoo-dev] [PATCH] cargo.eclass: Optimize crate unpacking From: =?UTF-8?Q?Micha=C5=82_G=C3=B3rny?= To: gentoo-dev@lists.gentoo.org Date: Sun, 12 May 2024 20:21:27 +0200 In-Reply-To: <6c25db91-4a22-48d5-b639-8009487fb7bf@gentoo.org> References: <20240512022603.48576-1-mgorny@gentoo.org> <6c25db91-4a22-48d5-b639-8009487fb7bf@gentoo.org> Organization: Gentoo Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-jzDWTGFWOjihDuyXWLOO" User-Agent: Evolution 3.52.1 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 X-Archives-Salt: 3295c00f-8c21-4a6e-9e0f-1055d77dfb29 X-Archives-Hash: 8ba53dd91a241f30b4fe0e48e96b237f --=-jzDWTGFWOjihDuyXWLOO Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sun, 2024-05-12 at 19:22 +0200, Florian Schmaus wrote: > On 12/05/2024 04.26, Micha=C5=82 G=C3=B3rny wrote: > > Unpack crates in parallel using xargs to utilize multicore systems > > better. Perform checksumming via a single sha256sum invocation. > >=20 > > For dev-python/watchfiles, this speeds up unpacking on my machine > > from 2.6 s to 0.75 s (warm cache). > >=20 > > Signed-off-by: Micha=C5=82 G=C3=B3rny > > --- > > eclass/cargo.eclass | 56 ++++++++++++++++++++++++++------------------= - > > 1 file changed, 33 insertions(+), 23 deletions(-) > >=20 > > diff --git a/eclass/cargo.eclass b/eclass/cargo.eclass > > index 0f2da982f60c..5a16d3a30528 100644 > > --- a/eclass/cargo.eclass > > +++ b/eclass/cargo.eclass > > @@ -329,40 +329,50 @@ _cargo_gen_git_config() { > > cargo_src_unpack() { > > debug-print-function ${FUNCNAME} "$@" > > =20 > > - mkdir -p "${ECARGO_VENDOR}" || die > > - mkdir -p "${S}" || die > > + mkdir -p "${ECARGO_VENDOR}" "${S}" || die > > =20 > > local archive shasum pkg > > + local crates=3D() > > for archive in ${A}; do > > case "${archive}" in > > *.crate) > > - # when called by pkgdiff-mg, do not unpack crates > > - [[ ${PKGBUMPING} =3D=3D ${PVR} ]] && continue > > - > > - ebegin "Loading ${archive} into Cargo registry" > > - tar -xf "${DISTDIR}"/${archive} -C "${ECARGO_VENDOR}/" || die > > - # generate sha256sum of the crate itself as cargo needs this > > - shasum=3D$(sha256sum "${DISTDIR}"/${archive} | cut -d ' ' -f 1) > > - pkg=3D$(basename ${archive} .crate) > > - cat <<- EOF > ${ECARGO_VENDOR}/${pkg}/.cargo-checksum.json > > - { > > - "package": "${shasum}", > > - "files": {} > > - } > > - EOF > > - # if this is our target package we need it in ${WORKDIR} too > > - # to make ${S} (and handle any revisions too) > > - if [[ ${P} =3D=3D ${pkg}* ]]; then > > - tar -xf "${DISTDIR}"/${archive} -C "${WORKDIR}" || die > > - fi > > - eend $? > > + crates+=3D( "${archive}" ) > > ;; > > *) > > - unpack ${archive} > > + unpack "${archive}" > > ;; > > esac > > done > > =20 > > + if [[ ${PKGBUMPING} !=3D ${PVR} ]]; then > > + pushd "${DISTDIR}" >/dev/null || die > > + > > + ebegin "Unpacking crates" > > + printf '%s\0' "${crates[@]}" | > > + xargs -0 -P "$(makeopts_jobs)" -n 1 -- \ >=20 > Consider using get_makeopts_jobs instead of makeopts_jobs, as it=20 > searches more variables for --jobs. Whose bright idea was to add a second similarly named function that does roughly the same thing but apparently differently? It can hardly get more confusing. --=20 Best regards, Micha=C5=82 G=C3=B3rny --=-jzDWTGFWOjihDuyXWLOO Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- iQFGBAABCgAwFiEEx2qEUJQJjSjMiybFY5ra4jKeJA4FAmZBCKcSHG1nb3JueUBn ZW50b28ub3JnAAoJEGOa2uIyniQOV9sIAL/g89ByXcB8Dt0+c3nV3gzGXBUyd6yy UobR+PWRv1qGM6qmdxOfBnkSMAjpbPk1jf3e0sO0q4a01EKcb3H/IOFQfeSwED2X +qO7oDcIzS7XRiIl7wEWX99qcE6+7N5QbLSI2Gle43oBGU7VjKGFghrK8Syy6REV PYS5lnEK4fvhvmz0nC97/58Xht8o1VocorclKcPqj7mUul3+jz8cMDcqoUMHGqJ/ Rkuw0dz+Ahk260aQWX7D6jEr7LUi+VumA3hg6yo+L3u+MJFMMkeD/rNTOOjzqW6N YrSS83exevM1qKPShbYLpHW6+Qef7LXB1+WVUef2aWcxNi85o3r6xRc= =7haw -----END PGP SIGNATURE----- --=-jzDWTGFWOjihDuyXWLOO--