From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 41BD7158232 for ; Sun, 8 Dec 2024 04:54:08 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 52F72E122D; Sun, 8 Dec 2024 04:54:03 +0000 (UTC) Received: from smtp.gentoo.org (woodpecker.gentoo.org [140.211.166.183]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 5147AE120F for ; Sun, 8 Dec 2024 04:54:02 +0000 (UTC) From: Sam James To: =?utf-8?B?TWljaGHFgiBHw7Nybnk=?= Cc: gentoo-dev@lists.gentoo.org Subject: LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM) In-Reply-To: <87f27044c599b4168d27d79367fd4b47575502c9.camel@gentoo.org> (=?utf-8?Q?=22Micha=C5=82_G=C3=B3rny=22's?= message of "Sat, 07 Dec 2024 17:07:46 +0100") Organization: Gentoo References: <87f27044c599b4168d27d79367fd4b47575502c9.camel@gentoo.org> User-Agent: mu4e 1.12.7; emacs 31.0.50 Date: Sun, 08 Dec 2024 04:53:58 +0000 Message-ID: <87a5d6wza1.fsf@gentoo.org> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Archives-Salt: 16535b55-ad79-4038-92ca-00927fefe768 X-Archives-Hash: 592d6cfc4587e514f785a078092e67a9 Micha=C5=82 G=C3=B3rny writes: > Hello, > > Given that the number of LLVM packages is growing, and probably will > grow again (I'm introducing "offload" right now, expect at least MLIR > soon, there are open requests for flang, polly...), I'd like to propose > creating dedicated categories for these packages and moving them there. > > [...] > > WDYT? I fear this sort of assumes we won't switch to monobuild any time soon. I keep thinking [0] about how sustainable our current setup is: * Fedora moved away from it for >=3D18 [1]. * As we saw with offload, it broke a few times in just a week. So it's only Gentoo who cares about this configuration AFAIK. * It's not working great if we're already not able to easily package mlir, flang, or polly. Violet attempted working on a merge before [2]. The trade-offs I see are: * Increased disk space requirement for building LLVM I think that's a legitimate concern although perhaps not that big of a deal given it is, after all, a whole toolchain. GCC takes quite a bit to build too. * Build time Build time for mono LLVM would increase as we're building more components at least for some users. But the time added by building more components (see below) should be balanced out by ccache if doing development and also, importantly, not needing to force on all targets anymore (they keep growing). The cumulative time should be the same (although maybe a bit less given the targets change) given that most people need the same set of components because of mesa, firefox, or other things which need libclang. * Moving to an upstream supported-configuration We may have a bit of pain in moving to it and getting used to it, but we're no longer swimming upstream and being the only people caring about our choice of build configuration (and regularly having to explain and justify it to upstream). Upstream also recommend doing a bootstrap build. We don't and can't do that right now. Folks upstream state at every opportunity that they don't care about standalone builds, e.g. https://discourse.llvm.org/t/rfc-do-something-with-the-subproject-ta= rballs-in-the-release-page/75024/2. * Better support for LLVM as a system toolchain The current setup doesn't work well for people using LLVM as a system toolchain (because some of the components *must* be upgraded together), it doesn't work well for people who want to use mlir/flang/polly, and it doesn't work well for users on constrained hardware because we have to force on all targets. It also prohibits more optimisation, PGO, and bootstrapping it to test reliability. (This is why I'm not too sympathetic to claims that the monobuild is mostly for binary distributions, because we're actually *more* vulnerable to issues as a result of it being split when building from source if using the LLVM toolchain.) * Maintaining older LLVMs It's easier for older LLVMs to be either maintained by somebody else (for e.g. Rust's purposes) or in an overlay if it's a single package rather than many. This is also true for e.g. testing old snapshots or keeping binary packages to downgrade to once they got cleaned up. At the moment, I fear us choosing the non-recommended path gives us very little, and causes a bunch of problems in return. I don't particularly care much personally about using LLVM as the system toolchain and if that's the only reason to move to monobuild, I'd be far less worried about it (even if I think using it as a system toolchain is reasonable for someone to work on). But if it's getting in the way of packaging things like flang and it's 100% on us to make standalone work - and keep it working - is it really worth it? I will note that it seems like the breakage has reduced upstream in the last few cycles, although the offload case was pretty bad just in the last week. On the other hand, if you think the problems with mlir and friends are easily-solvable, then we may be fine as we are? IIRC, Violet ended up running into issues with getting the CMake to properly handle both shared and static libraries to install with the monobuild, so maybe LLVM's CMake is just broken in both directions and we just have to pick whichever one is least-worst for us (which might be the standalone builds as we do now)? [0] https://github.com/gentoo/gentoo/pull/32716#issuecomment-1713689410 [1] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject= .org/thread/23QT5ISGGOQUOMTOOLMUBV6MSBB2C2C6/#PDAXH7PZTA7SFSSKF65KIXYLPYHAI= 3SS [2] https://github.com/gentoo/gentoo/pull/32716 sam