From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 96B7E13933E for ; Sat, 17 Jul 2021 04:20:48 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id E195DE0B4E; Sat, 17 Jul 2021 04:20:47 +0000 (UTC) Received: from smtp.gentoo.org (dev.gentoo.org [IPv6:2001:470:ea4a:1:5054:ff:fec7:86e4]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id B8F4BE0B4E for ; Sat, 17 Jul 2021 04:20:47 +0000 (UTC) Received: from [2a0c:b641:69c:e789:6d3:b0ff:fe40:5fd8] (port=56292 helo=robot) by muon with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94) (envelope-from ) id 1m4boh-0007Cx-Hl; Sat, 17 Jul 2021 04:20:43 +0000 From: Benda Xu To: "Yuan Liao (Leo)" Cc: gentoo-soc@lists.gentoo.org Subject: Re: [gentoo-soc] Week 5 Report for Big Data Infrastructure and H2O ebuilds Project References: Date: Sat, 17 Jul 2021 12:20:44 +0800 In-Reply-To: (Yuan Liao's message of "Sun, 11 Jul 2021 20:56:19 -0700") Message-ID: <87o8b1y3j7.fsf@robot.e.airelinux.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-soc@lists.gentoo.org Reply-to: gentoo-soc@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 Content-Type: text/plain X-Archives-Salt: 8166e848-9529-4c3c-9a3f-b55fdf8203ef X-Archives-Hash: 0c4651e5211ebc8bb46181935194b1e7 "Yuan Liao (Leo)" writes: > This week, I have moved on to the next part of my project: an ebuild > testing framework. For sophisticated ebuilds with many dependencies > and a lot of USE flags, a framework which facilitates automated ebuild > installation tests can be very useful. Such a testing framework can > also integrate with a CI service so the tests can be run periodically > and automatically, reminding developers of ebuild breakages early. > > Maintainers of ebuilds for Java packages -- especially those ebuilds > in the Spark overlay -- might want to test installing those ebuilds > periodically and with different USE flag configurations. To start > with, Java packages usually have multiple dependencies, and successful > installation of all those dependencies is necessary for successful > installation of such a Java package itself. Some packages in the > Spark overlay depend on ebuilds in ::gentoo, but chances are the > latter set of ebuilds get updated or removed, preventing those ebuilds > in the overlay from being merged properly. As a matter of fact, some > ebuilds in the Spark overlay are already non-installable due to > removal of some Java packages from ::gentoo. In this case, if there > is a set of ebuild installation tests that run every day or every > week, then problems like this can be discovered as soon as they > emerge. Besides dependencies, USE flags are another type of > parameters that can determine whether an ebuild can be installed > because a USE flag can affect the list of dependencies and/or how the > ebuild is compiled and installed. Many ebuilds in the Spark overlay > have a 'binary' USE flag (just like any ebuild created by > java-ebuilder) which installs a pre-built binary JAR from Maven > Central when enabled. Some of the ebuilds can be installed with > USE="binary" but not with USE="-binary" due to errors in Java source > compilation. The same ebuild should be tested with both of the USE > flag configurations if the goal is to support both binary and source > installation. > > My original plan with regards to ebuild testing is to use an existing > solution called ebuildtester [1], which is a program that can install > a specified package in a stage3 Docker container. After doing some > research on it, I found that it cannot satisfy some of my ebuild > testing requirements for the Kotlin ebuilds I had created in the > previous weeks at least. I would like to fully customize USE flag > settings just like how I could specify USE flag settings in > /etc/portage/package.use, but ebuildtester only supports a global USE > flag setting and USE configuration for the package being tested > itself, so the granularity is too coarse. My Kotlin ebuilds need to > be installed with multiple emerge commands, which is quite uncommon > for ebuilds to be honest, but with ebuildtester, it does not seem to > be possible to run more than one emerge command. However, I still > liked the idea of testing ebuild installation in a Docker container, > so based on ebuildtester's concept, I created my own tool that tests > ebuilds in a container: ebuild-commander [2]. This tool does not take > an atom to be installed as input; instead, it takes a list of commands > to execute in the container (hence the name "ebuild-commander"). It > also supports copying a specified directory's contents to the > container's /etc/portage, allowing full customization on Portage > settings in ebuild tests. ebuild-commander is like a rewrite of > ebuildtester strived to allow full control over the ebuild test > execution with a simpler interface. > > Automated and periodic ebuild testing with ebuild-commander has now > been set up for my fork of the Spark overlay [3]. I have only created > two test cases for my Kotlin ebuilds at this point, but more tests can > be added later quickly and easily. At this point, I am planning to > implement a mechanism to automatically compute a minimal set of > packages that need to be specified in arguments to emerge in order to > install all packages in the Spark overlay at least once. Because a > test case can take a very long time to run, the tests are configured > to run once a day only instead of on every new commit. > > That is all for this week. See you in the next weekly report! Glad to see ebuild-commander finally works on GitHub Actions. This lies down the foundation for long term stability of the overlay. Great job! Cheers, Benda