From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 97BE359CA5 for ; Sun, 20 Mar 2016 22:42:51 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 7C21E21C00A; Sun, 20 Mar 2016 22:42:49 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 0D96221C00A for ; Sun, 20 Mar 2016 22:42:48 +0000 (UTC) Received: from oystercatcher.gentoo.org (oystercatcher.gentoo.org [148.251.78.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.gentoo.org (Postfix) with ESMTPS id A915C34082B for ; Sun, 20 Mar 2016 22:42:47 +0000 (UTC) Received: from localhost.localdomain (localhost [127.0.0.1]) by oystercatcher.gentoo.org (Postfix) with ESMTP id E62A4850 for ; Sun, 20 Mar 2016 22:42:45 +0000 (UTC) From: "Mike Frysinger" To: gentoo-commits@lists.gentoo.org Content-Transfer-Encoding: 8bit Content-type: text/plain; charset=UTF-8 Reply-To: gentoo-dev@lists.gentoo.org, "Mike Frysinger" Message-ID: <1458510299.6325704c0812f0752b94e4e23e807860bae6f184.vapier@gentoo> Subject: [gentoo-commits] proj/releng:master commit in: scripts/ X-VCS-Repository: proj/releng X-VCS-Files: scripts/copy_buildsync.sh X-VCS-Directories: scripts/ X-VCS-Committer: vapier X-VCS-Committer-Name: Mike Frysinger X-VCS-Revision: 6325704c0812f0752b94e4e23e807860bae6f184 X-VCS-Branch: master Date: Sun, 20 Mar 2016 22:42:45 +0000 (UTC) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-commits@lists.gentoo.org X-Archives-Salt: 37379ba4-ee27-47c7-8b96-a530719c78f9 X-Archives-Hash: 43502f98b0f2dfe266242452ad13a6c7 commit: 6325704c0812f0752b94e4e23e807860bae6f184 Author: Mike Frysinger gentoo org> AuthorDate: Sun Mar 20 21:44:59 2016 +0000 Commit: Mike Frysinger gentoo org> CommitDate: Sun Mar 20 21:44:59 2016 +0000 URL: https://gitweb.gentoo.org/proj/releng.git/commit/?id=6325704c copy_buildsync.sh: convert input copying logic to a func Minor clean up ... should not be any real functional changes. scripts/copy_buildsync.sh | 68 ++++++++++++++++++++++++++++++----------------- 1 file changed, 43 insertions(+), 25 deletions(-) diff --git a/scripts/copy_buildsync.sh b/scripts/copy_buildsync.sh index b2e5f8a..927a1fd 100755 --- a/scripts/copy_buildsync.sh +++ b/scripts/copy_buildsync.sh @@ -1,5 +1,12 @@ #!/bin/bash +# Where artifacts are uploaded by builders. +INCOMING_BASE="/release/weekly/builds" +# Where artifacts are moved to so they can be uploaded to mirrors. +OUTGOING_BASE="/release/distfiles/weekly" +# Scratch space used when moving files from incoming to outgoing. +TMPDIR_BASE="/release/distfiles/tmp/buildsync/partial" + ARCHES=( alpha amd64 @@ -43,39 +50,50 @@ EOF exit ${1:-1} } +# Copy artifacts for an arch to the outgoing directory. +copy_arch_to_outgoing() { + local ARCH=$1 indir=$2 outdir=$3 tmpdir=$4 + local i t rc + + if [[ ! -d ${indir} ]]; then + # Nothing to do for this arch. + return + fi + + # Copying + for i in $(find ${indir} -type f | grep -- '-20[0123][0-9]\{5\}' | sed -e 's:^.*-\(20[^.]\+\).*$:\1:' | sort -ur); do + #echo "Doing $i" + t="${outdir}/${i}" + mkdir -p ${t} 2>/dev/null + rsync "${RSYNC_OPTS[@]}" --temp-dir=${tmpdir} --partial-dir=${tmpdir} ${indir}/ --filter "S *${i}*" --filter 'S **/' --filter 'H *' ${t} + rc=$? + if [ $rc -eq 0 ]; then + find ${indir} -type f -name "*${i}*" -print0 | xargs -0 --no-run-if-empty $DEBUGP rm $VERBOSEP -f + else + echo "Not deleting ${indir}/*${i}*, rsync failed!" 1>&2 + fail=1 + fi + done + find ${outdir} -mindepth 1 -type d \ + | egrep -v current \ + | sort -r \ + | tr '\n' '\0' \ + | xargs -0 --no-run-if-empty rmdir --ignore-fail-on-non-empty +} + process_arch() { local ARCH=$1 - rc=0 fail=0 - indir=/release/weekly/builds/${ARCH} - outdir=/release/distfiles/weekly/${ARCH} - tmpdir=/release/distfiles/tmp/buildsync/partial/${ARCH} + indir="${INCOMING_BASE}/${ARCH}" + outdir="${OUTGOING_BASE}/${ARCH}" + tmpdir="${TMPDIR_BASE}/${ARCH}" mkdir -p ${tmpdir} 2>/dev/null - # Copying - if [ -d "${indir}" ]; then - for i in $(find ${indir} -type f | grep -- '-20[0123][0-9]\{5\}' | sed -e 's:^.*-\(20[^.]\+\).*$:\1:' | sort -ur); do - #echo "Doing $i" - t="${outdir}/${i}" - mkdir -p ${t} 2>/dev/null - rsync "${RSYNC_OPTS[@]}" --temp-dir=${tmpdir} --partial-dir=${tmpdir} ${indir}/ --filter "S *${i}*" --filter 'S **/' --filter 'H *' ${t} - rc=$? - if [ $rc -eq 0 ]; then - find ${indir} -type f -name "*${i}*" -print0 | xargs -0 --no-run-if-empty $DEBUGP rm $VERBOSEP -f - else - echo "Not deleting ${indir}/*${i}*, rsync failed!" 1>&2 - fail=1 - fi - done - find ${outdir} -mindepth 1 -type d \ - | egrep -v current \ - | sort -r \ - | tr '\n' '\0' \ - |xargs -0 --no-run-if-empty rmdir --ignore-fail-on-non-empty - fi + # Sync incoming->outgoing first. + copy_arch_to_outgoing "${ARCH}" "${indir}" "${outdir}" "${tmpdir}" # ================================================================ # Build data for revealing latest: