From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 8B01A138359 for ; Tue, 3 Nov 2020 18:20:11 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 75FD9E0A99; Tue, 3 Nov 2020 18:20:10 +0000 (UTC) Received: from smtp.gentoo.org (smtp.gentoo.org [140.211.166.183]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 0BA89E0A97; Tue, 3 Nov 2020 18:20:10 +0000 (UTC) Received: by mail-ej1-f49.google.com with SMTP id gn41so8294977ejc.4; Tue, 03 Nov 2020 10:20:06 -0800 (PST) X-Gm-Message-State: AOAM533/CcUR+W8NsrO1mdtSb0MdvhG+cTxfCtDgyUsIQfyTFZZez6ii UVPgESJU80ZXOWcSCMvZw8NmyaKjtG/6Yew2OiY= X-Google-Smtp-Source: ABdhPJzusqf761hG2v2nRUPDK10O1TN7fodD1x9ezqyE9TAFyJdjNICxwd7pBsxOZih15+q8YCMoksxv0aQRcBydsds= X-Received: by 2002:a17:906:4495:: with SMTP id y21mr1307997ejo.209.1604427603444; Tue, 03 Nov 2020 10:20:03 -0800 (PST) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-catalyst@lists.gentoo.org Reply-to: gentoo-catalyst@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 References: <20201103105456.GA3468@dysnomia.localdomain> In-Reply-To: <20201103105456.GA3468@dysnomia.localdomain> From: Matt Turner Date: Tue, 3 Nov 2020 13:19:51 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [gentoo-catalyst] catalyst changes for improving automation To: gentoo-catalyst@lists.gentoo.org Cc: gentoo-releng@lists.gentoo.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Archives-Salt: 877a88a5-e157-4a49-8506-02c99a5e400c X-Archives-Hash: 7ab8d10110e7111b69d4be5c99c3ade4 On Tue, Nov 3, 2020 at 5:56 AM Daniel Cordero wro= te: > > On Mon, Nov 02, 2020 at 10:44:07PM -0500, Matt Turner wrote: > > The catalyst-auto automation scripts live in a repo separate from > > catalyst. That increases the difficulty of changing catalyst's > > interface, and it doesn't seem to offer any advantages otherwise. > > (Keeping build specs in a separate repo allows them to be updated > > independent of catalyst and that is valuable). Additionally, since the > > primary way catalyst is used is via this automation, it makes sense to > > support this workflow in catalyst directly. > > > > What would be more heavily impacted are those users who may not already > have infra set up to do builds or just starting out using catalyst for > the first time and haven't written their own automation. > > I suggest prioritising the collection of up-to-date documentation, > especially regarding running catalyst manually, since it'll be > completely different to the literature that's currently out there. I'm a bit unsure what you mean. Do you suggest prioritizing documenting the current method of running catalyst before changing it? > > But to get there, there are some changes to catalyst that I think are > > improvements on their own and simplify the path to integrating > > automation capabilities directly into catalyst. That's what I'd like > > to discuss here. > > > > I'd like to: > > > > 1) Replace the custom .spec file format with TOML > > > > Fine. Aside from the extra quotes and commas, I'd be happy with any well > defined format that can handle strings and lists properly. > > > 2) Combine .spec file sequences (e.g., stage1 -> stage2 -> stage3 -> > > livecd-stage1 -> livecd-stage2) into a single file. I suggest naming > > this a ".build" file. This will also allow us to remove the redundant > > information that currently has to be specified in stage1.spec, > > stage2.spec, stage3.spec, like rel_type, version, profile, etc. It > > also means that we remove the nonsensical ability to change settings > > from one stage to the next that should not change (e.g., rel_type, > > version). > > > > How would a target that depends on a different rel_type work? Forks in > the dependency tree. > > > 3) Add ability to denote which stage builds produce artifacts we care > > about (and want to save and/or upload) and which are just temporary. > > If they're temporary (e.g., a stage1 build) we can delete the artifact > > after the build sequence has no further use of it, and we can skip > > compressing the result, etc. > > > > This feature should (haven't tested) already exist - it's just not > documented. > > compression_mode: rsync > options=3D['seedcache'] Hah! I was completely unaware of this. Thanks. > or don't call 'capture' and/or 'remove_chroot' in action_/finish_sequence= . > > > > > To that end, I'm starting by figuring out what I would like the new > > spec file format to look like. Below are some open questions and then > > a strawman new-style spec file. > > > > =E2=80=A2 The .spec files in releng.git are really templates that are n= ot > > directly usable without sed'ing @REPO_DIR@ and @TIMESTAMP@. It would > > be nice if they were directly usable as that would reduce confusion > > from users. > > =E2=80=A2 Can we make them directly usable? > > =E2=80=A2 Perhaps we can make catalyst handle the replacements direct= ly? > > =E2=80=A2 Calculating @TIMESTAMP@ is trivially doable=E2=80=94we do= it today (see below) > > Maybe a strftime() template, or even fstring-like tokens? > (e.g. "{year}-{month}-{day}") One goal I have is to make it more transparent what is actually in a particular stage tarball or ISO and along with that to make it easier to reproduce the result. Obviously we'll want to keep the ability to specify a particular version, as you describe, but I think for Gentoo releases we will want to continue using a timestamp that's unambiguously tied to the git SHA1 of gentoo.git as is possible. > > =E2=80=A2 We could configure @REPO_DIR@ in catalyst.conf and let ca= talyst > > do the replacement, or we could just make the field relative to some > > path specified in catalyst.conf? > > > > While nice to have, I don't agree with locking users into a particular > repository layout. Can you explain what you mean? I don't know how what I said would require a particular repository layout. Perhaps you're confused by the @REPO_DIR@ name? It is the path to the releng.git repository (containing the .specs and the /etc/portage/ files) on the build machine and is not in any way connected with the ebuild repositories. The name predates my involvement, so don't blame me :) > > =E2=80=A2 In the current automation scripts, we generate a value for > > @TIMESTAMP@ from the git HEAD used in creating the snapshot. > > =E2=80=A2 Would be nice to remove the dependence on the squashfs snap= shot > > generation=E2=80=94not difficult to do > > > > I have no comment on this. > > > =E2=80=A2 Can we generate and upload a .build file with replacements do= ne to > > make stage builds more easily reproducible? Seems easy. > > > > These can just be artifacts from the build. Yes, that's what I'm thinking too.