From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id B632B1382C5 for ; Fri, 15 May 2020 16:32:34 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 63412E0928; Fri, 15 May 2020 16:32:31 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [IPv6:2001:470:ea4a:1:5054:ff:fec7:86e4]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 4EBC3E0928 for ; Fri, 15 May 2020 16:32:31 +0000 (UTC) Received: by mail-ed1-f45.google.com with SMTP id r7so2708664edo.11 for ; Fri, 15 May 2020 09:32:28 -0700 (PDT) X-Gm-Message-State: AOAM531jX6Uj57FtB31/sqiPqyInAQHRqMfR8dOw97RL7/tWzrqawHcO zTJllql4EBHSkxUtFIrCi6otMInvS6/In4dNgks= X-Google-Smtp-Source: ABdhPJzG2Am3pa66W6cPAoVXkjOXER9dOhDvXvHZvxwDnZtMeUj97dYiSTucVj+ZR6siBkwmrYCCUPQJvWWeH8lTOUA= X-Received: by 2002:aa7:dc49:: with SMTP id g9mr3653301edu.167.1589560346043; Fri, 15 May 2020 09:32:26 -0700 (PDT) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-releng@lists.gentoo.org Reply-to: gentoo-releng@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 References: <20200515130719.3138513-1-bkohler@gentoo.org> In-Reply-To: <20200515130719.3138513-1-bkohler@gentoo.org> From: Matt Turner Date: Fri, 15 May 2020 09:32:13 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [gentoo-releng] [PATCH 1/2] catalyst-auto-amd64*.conf: explicitly list files to be transferred To: gentoo-releng@lists.gentoo.org Content-Type: text/plain; charset="UTF-8" X-Archives-Salt: fa2364c3-02ef-4cc5-b50a-364b3a41cccf X-Archives-Hash: 674170f6b706d88ca75531a63ea1a4ad On Fri, May 15, 2020 at 6:07 AM Ben Kohler wrote: > > Previously when we were copying locally there was no (major) problem > with wildcarding a long list of files to be transferred after every > single spec finishes. Now that we are transferring to a remote host, > this results in a lot of wasted transfer time & bandwidth. > > I have adjusted the post_build function to only transfer the new files > known to be produced by each spec (which has files meant to be > published). Thanks Ben. Both patches look good to me. Thanks for working on this. Just a couple small comments inline. > Signed-off-by: Ben Kohler > --- > tools/catalyst-auto-amd64-experimental.conf | 87 ++++++++++++++------- Do we actually use this file in production? It looks like the main difference is TYPE="experimental" vs TYPE="auto" and I don't see TYPE used anywhere. Can we just remove this file? > tools/catalyst-auto-amd64.conf | 86 +++++++++++++------- > 2 files changed, 113 insertions(+), 60 deletions(-) > > diff --git a/tools/catalyst-auto-amd64-experimental.conf b/tools/catalyst-auto-amd64-experimental.conf > index e7b1ea79..2aa91a03 100644 > --- a/tools/catalyst-auto-amd64-experimental.conf > +++ b/tools/catalyst-auto-amd64-experimental.conf > @@ -26,6 +26,7 @@ SET_minimal_nomultilib_OPTIONAL_SPECS="stage4-nomultilib-minimal.spec" > > SET_x32_SPECS="stage1-x32.spec stage2-x32.spec stage3-x32.spec" > > + Stray newline, but I see that this is to reduce the differences between the experimental and non-experimental versions of the file. I might remove the newline from the other instead, but I don't feel strongly. > SET_hardened_multilib_SPECS="hardened/stage1.spec hardened/stage2.spec hardened/stage3.spec" > SET_hardened_multilib_OPTIONAL_SPECS="hardened/admincd-stage1.spec hardened/admincd-stage2.spec" > SET_minimal_hardened_multilib_OPTIONAL_SPECS="hardened/stage4-minimal.spec" > @@ -43,7 +44,7 @@ EXTENSIONS="[.tar.xz,.tar.bz2,.tar.gz,.tar,.sfs]" > update_symlinks() { > # Symlink the latest stages3 to build from > for d in ${BUILD_SRCDIR_BASE}/{default,hardened} ; do > - pushd $d >/dev/null > + pushd "${d}" >/dev/null || exit > for f in $(ls stage3*${EXTENSIONS} | grep -v latest | give_latest_from_dates ) ; do > # 20yymmddThhmmssZ > # 20yymmddhhmmss > @@ -57,11 +58,13 @@ update_symlinks() { > <<<"$f") > ln -svf "$f" "$of" > done > - popd >/dev/null > + popd >/dev/null || exit > done > } > > post_build() { > + local set=$1 spec=$2 > + > cmd=( > rsync This would be a good opportunity to create and use an upload() function like the other conf files have.