From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id A2EF51581F3 for ; Mon, 25 Nov 2024 17:15:38 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 0F077E07FA; Mon, 25 Nov 2024 17:15:33 +0000 (UTC) Received: from smtp.gentoo.org (woodpecker.gentoo.org [140.211.166.183]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 7D4E5E07EE for ; Mon, 25 Nov 2024 17:15:31 +0000 (UTC) Date: Mon, 25 Nov 2024 10:15:27 -0700 From: Tim Harder To: gentoo-dev@lists.gentoo.org Subject: [gentoo-dev] verifying commits via server-side git pre-receive hook Message-ID: Mail-Followup-To: gentoo-dev@lists.gentoo.org Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Disposition: inline X-Archives-Salt: 5d6bb5dc-7814-428c-ad7a-9c7d82005377 X-Archives-Hash: e1179b549bbc03e22b54dd62108d5253 Hi all, I've been working for a while on pkgcraft[1] and related tooling that could have the potential to run various CI checks server-side as a pre-receive git hook verifying commits. Previously, this wasn't at all possible due to portage/repoman and pkgcore/pkgcheck being too slow to feasibly run and give feedback within a second or two of interactively pushing. For more information on benchmarks, see the related sections in the discussions of pkgcraft's metadata generation[2] and pkgcruft[3] comparing them to similar functionality from portage and pkgcore. Also, note that a majority of commits only involve scanning a limited number of packages which is much faster than scanning the entire tree. However, even with all that making it semi-feasible for most commits (except perhaps those doing tree-wide changes or altering eclasses with extensive usage) much work has to be done on the infra and git-hook service side. First off, the server-side git repo is probably hosted in a bare format so the git hook would need to notify a scanning service that runs remotely on a relatively powerful machine. In terms of service design, my high-level thoughts are as follows: the scanning service tracks repo changes using some copy-on-write solution that supports snapshots such as lvm2 or btrfs on top of a fast, backing I/O solution (direct NVME or tmpfs). This would be leveraged to keep track of the most recent tree state with metadata layered in. As commits come in they would be put on a queue, from which the most recent would be pulled, checked out on top of the most recent snapshot, tested, and merged back into the most recent snapshot when scanning passes. Alternatively, Gentoo could officially move to a merge queue approach which would alleviate a lot of this complexity, but also bring up questions of how merge commits and signing would work among other issues. From all this, my main questions to the dev community are the following: - Do you see value in running a service that can reject commits due to issues like invalid metadata during `git push`? - Would the project be open to moving to a merge queue model? - Are there others who would be interested in helping with the development, testing, and maintenance of a git-hook service if that is the chosen path? Thanks, Tim [1]: https://github.com/pkgcraft [2]: https://pkgcraft.github.io/posts/metadata-cache-generation/ [3]: https://pkgcraft.github.io/posts/evolving-qa-tooling/