From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 8002015817D for ; Fri, 21 Jun 2024 13:14:21 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id CB993E2A8B; Fri, 21 Jun 2024 13:14:19 +0000 (UTC) Received: from smtp.gentoo.org (mail.gentoo.org [IPv6:2001:470:ea4a:1:5054:ff:fec7:86e4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id AD3AEE2A8B for ; Fri, 21 Jun 2024 13:14:19 +0000 (UTC) Received: from oystercatcher.gentoo.org (oystercatcher.gentoo.org [148.251.78.52]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTPS id DD49A33C4EE for ; Fri, 21 Jun 2024 13:14:17 +0000 (UTC) Received: from localhost.localdomain (localhost [IPv6:::1]) by oystercatcher.gentoo.org (Postfix) with ESMTP id 021781D53 for ; Fri, 21 Jun 2024 13:14:15 +0000 (UTC) From: "Sam James" To: gentoo-commits@lists.gentoo.org Content-Transfer-Encoding: 8bit Content-type: text/plain; charset=UTF-8 Reply-To: gentoo-dev@lists.gentoo.org, "Sam James" Message-ID: <1718176002.eb788da3f1af8f506226986f12de6b2b04cdaba1.sam@gentoo> Subject: [gentoo-commits] proj/gentoo-functions:master commit in: / X-VCS-Repository: proj/gentoo-functions X-VCS-Files: functions.sh test-functions X-VCS-Directories: / X-VCS-Committer: sam X-VCS-Committer-Name: Sam James X-VCS-Revision: eb788da3f1af8f506226986f12de6b2b04cdaba1 X-VCS-Branch: master Date: Fri, 21 Jun 2024 13:14:15 +0000 (UTC) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-commits@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply X-Archives-Salt: bdeef1dc-69c6-44e9-8525-9ec6b2490dc6 X-Archives-Hash: 075e9b2cd5ca52cb439460dbecde07d2 commit: eb788da3f1af8f506226986f12de6b2b04cdaba1 Author: Kerin Millar plushkava net> AuthorDate: Sun Jun 2 03:42:00 2024 +0000 Commit: Sam James gentoo org> CommitDate: Wed Jun 12 07:06:42 2024 +0000 URL: https://gitweb.gentoo.org/proj/gentoo-functions.git/commit/?id=eb788da3 Add the parallel_run() function Here is an example of how to use it. $ parallel_run 0 sha256sum dir/* Simple commands of a greater level of sophistication can easily be composed with the aid of functions. $ sm3sum() { cksum -a sm3 "$@"; } $ parallel_run 0 sm3sum dir/* Signed-off-by: Kerin Millar plushkava.net> functions.sh | 44 ++++++++++++++++++++++++++++++++++++++++++++ test-functions | 21 ++++++++++++++++++++- 2 files changed, 64 insertions(+), 1 deletion(-) diff --git a/functions.sh b/functions.sh index 60a4b2e..e849541 100644 --- a/functions.sh +++ b/functions.sh @@ -495,6 +495,50 @@ oldest() _select_by_mtime -- "$@" } +# +# Executes a simple command in parallel. At least two parameters are expected. +# The first parameter shall be taken as the maximum number of jobs to run +# concurrently. If specified as less than or equal to 0, the number shall be +# determined by running the nproc function. The second parameter shall be taken +# as a command name. The remaining parameters shall be conveyed to the specified +# command, one at a time. Should at least one command fail, the return value +# shall be greater than 0. +# +parallel_run() +{ + local arg cmd i statedir w workers + + if [ "$#" -lt 3 ]; then + warn "parallel_run: too few arguments (got $#, expected at least 3)" + return 1 + elif ! is_int "$1"; then + _warn_for_args parallel_run "$1" + return 1 + elif [ "$1" -le 0 ] && ! workers=$(get_nprocs); then + return 1 + elif ! statedir=${TMPDIR:-/tmp}/parallel_run.$$.$(srandom); then + return 1 + fi + workers=${workers:-$1} cmd=$2 + shift 2 + w=0 + i=0 + ( + while [ "$(( w += 1 ))" -le "${workers}" ]; do + i=$w + while [ "$i" -le "$#" ]; do + eval "arg=\$${i}" + if ! "${cmd}" "${arg}"; then + mkdir -p -- "${statedir}" + fi + i=$(( i + workers )) + done & + done + wait + ) + ! rmdir -- "${statedir}" 2>/dev/null +} + # # Declare the vebegin, veerror, veindent, veinfo, veinfon, veoutdent and vewarn # functions. These differ from their non-v-prefixed counterparts in that they diff --git a/test-functions b/test-functions index c734141..ed3da42 100755 --- a/test-functions +++ b/test-functions @@ -556,6 +556,25 @@ test_get_nprocs() { iterate_tests 2 "$@" } +test_parallel_run() { + set -- \ + ge 1 N/A N/A \ + eq 0 / N/A \ + ge 1 /var/empty/nonexistent N/A \ + eq 0 / / \ + ge 1 / /var/empty/nonexistent \ + ge 1 /var/empty/nonexistent /var/empty/nonexistent \ + ge 1 /var/empty/nonexistent / + + callback() { + shift + test_description="parallel_run $(_print_args 0 ls "$@")" + parallel_run 0 ls "$@" >/dev/null 2>&1 + } + + iterate_tests 4 "$@" +} + iterate_tests() { slice_width=$1 shift @@ -621,7 +640,7 @@ test_newest || rc=1 test_trim || rc=1 test_hr || rc=1 test_whenceforth || rc=1 -test_get_nprocs || rc=1 +test_parallel_run || rc=1 cleanup_tmpdir