From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id EA459158089 for ; Tue, 19 Sep 2023 09:37:18 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 680412BC039; Tue, 19 Sep 2023 09:37:14 +0000 (UTC) Received: from mout.web.de (mout.web.de [212.227.15.14]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id BD8592BC019 for ; Tue, 19 Sep 2023 09:37:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=s29768273; t=1695116227; x=1695721027; i=finkandreas@web.de; bh=i6qM6KQHWGN1KsVJXJNMuqk1L7D7BCsKjeuUSj1HGCg=; h=X-UI-Sender-Class:Date:From:To:Subject:In-Reply-To:References; b=put4NyaY42qITn7BBZwYI22lacdI04zL3ENFPXbro6qZ7RT01y4nztXHyvnagOWu/OMS7z6OkoK 6s8r41WsE6Thg7sLLMa6LDoLI0cLY/JapbR4pMYyM6cMOtr5OtNOlIEk7tCxmSfdh7cBoJCMkT/bR 686lgpOwSqbovHH2z7wHvdxYl9SVI8NkP/UUxiostp7+TpHSzpyLyOF3ypDqs0ogZfcW/pZzRpFla HtWPQKk6ify1kZE1NyAtYGVO3q2GHRFMHHDI7kYByivVheZEOQu29NybCrk2r0kUYCy+LeFq5e6vu 6Ej2w2f1f0rZiuo3lHInZk6cK7Uf6wh4Orrg== X-UI-Sender-Class: 814a7b36-bfc1-4dae-8640-3722d8ec6cd6 Received: from anfink-laptop ([46.14.9.177]) by smtp.web.de (mrweb006 [213.165.67.108]) with ESMTPSA (Nemesis) id 1Mx0Ix-1rbsXw0qbL-00y6BB for ; Tue, 19 Sep 2023 11:37:07 +0200 Date: Tue, 19 Sep 2023 11:37:00 +0200 From: Andreas Fink To: gentoo-user@lists.gentoo.org Subject: Re: [gentoo-user] Controlling emerges Message-ID: <20230919113700.2678c8b8@anfink-laptop> In-Reply-To: References: <4531844.LvFx2qVVIh@wstn> <7140559f-ca28-80c3-99ac-985a0be91230@iinet.net.au> <2709340.mvXUDI8C0e@wstn> X-Mailer: Claws Mail 4.1.1 (GTK 3.24.38; x86_64-pc-linux-gnu) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K1:67AI4hGSDunGoR1zI8ztaf12v7rNcQtFuY4z6voQUzANiXJd3fm OOHZ9UKLv/Si3Z4VCQL5uxq+wf0UYNj1lDDF182lhMY9gG1/24XeJvUVBbSb2gyWrUNzy6Q Zmh78oiNF1Y5QmVudsNfsSPWhjJlfZoTAjmCdkUlAa5aFqqC3hj1VR+wcb6Tw5Y2j4ZrLGm CaN8YibSYJX8mY6hrQ84Q== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:rtNoxfilGGo=;kiaEPvbIYOT/o0z5YjRJSw6Ji20 HDtBieXhNNz3Nrd0wPiGBRGv5VZLV+E4QhDdqd3SZyvgP5PPTd9jpIgaOv13EOVhmq/almGWf sDSrvDmekZ81NBQDhD16Ite3RSeqjdm20ISt+TSkbgqlnUF1L5GcTgpQN/RXNLf4Mxw1ORBrF nUaqx4DEn4aduR4oc55WCU++B5pdpC5s0CnhiVxJoom+I3v+IVHdGm/wtzPMWyQ1XT55q/EWI vOH7HWYTjEWeNdzkfBK+54myvnrh3F5lhQlJ6LBl8jyfMIWbHQ8EdKbMBDd1VN3WR/yHehLjP PO+zvMO8oqg6Tas3w/rptlkb+Czy4ugnhE0CitTCqVxmkI1xXziFdJcsWtGN6T8RbbGToXiZl L47Q2Daeg9bsdm25P6YFfQi3v7qc2e2snbehNdiU7DSGVSdpi9Etcb39OUxYJHTMBejV8GOOY 73Vp4s31eWOfBJjyfDp3fsT0P0JlYqsQ/VjzGTqbV20ic23vrQrxwTvtXCuzdPfvjppv9kyTy c3dxVx8LdWFhC/dZbO1UJh+QKP5Nmmt5mekn14SaSuyDOU70DTOWH7srx4ANFU4wtOvmH6uCf DAz5arXTEboS+uYsJSwaZMwvvo1pxvB3OIO79whwXiMO5mMC6Pt3cwqk2yit78zzuG3u2tgVw iMIfdpgFUOcyPZV/MHmv5X5qQHhmRspTWb3f4xvjtR6YKCPtfhmMwQu6IFKqiCOJa1mVxfM3J JSt/DcJQruyei1I8dI4AGQwM3RdmXtdGRdIVnGomJu01iyaTFonWSUgSwtx59URxh25lHBESi u3MwOwZ/J6GSH6v7Mjg42gyAhn4VjfueV1E6JH3lniCZ5AtxHgwSgWXbcxO41qLFaard68x2q 9y5LbQLGfdfXoKdj4IVlSQHnYmU17eIvNajdj4AZJF29326Sgl0NowKYZTuB40ge9MK3Vfqr5 SpHTjg== X-Archives-Salt: 3c27124d-ad73-4706-b239-46c3426132a0 X-Archives-Hash: f6f2b648d00e661b0efa9cc91abff216 On Tue, 19 Sep 2023 17:14:42 +0800 William Kenworthy wrote: > That is where you set per package compiler parameters by overriding > make.conf settings. > > BillK > > I would argue, that per package compiler parameters is not what is needed, because in the example of chromium 99% of the compile time can be done with -j16 on my machine, but at a very short time I would need to run with -j1, because I otherwise run out of memory otherwise. In short: I want to run with as many jobs as I have cores, as long as I do not run out of memory, and when I run out of memory I want to run with as little jobs as possible until the pressure on the memory is gone. Then I want to continue with as many jobs as possible. And this is not something that make / ninja provide. They have a concept of global number of jobs, which in this concept must be set to the maximum number that your RAM can take at the very short period in time where you have a high watermark on your RAM, but that number would be at 99% of the compilation time way too low. FWIW, I have a hacky solution that I use privately, but I never published it anywhere, because it could break some builds, and at the moment I'm not ready to support it. Basically it tries to run with as many jobs as the number of CPU cores at all times. It watches memory pressure in the background and kills build jobs as soon as a high watermark is reached. At this point, make would normally exit, because a build job failed. However my hacky solution overrides the exec-family of system calls, and if a job fails, it is being retried exclusively, i.e. no other build job is allowed to run at the same time as the failed job. It fails ultimately, when the second and exclusive run fails too. This way, if the job failed only because of lack of memory, it will be retried exclusively and succeeds. If it failed due to a programming error, it will fail also the second time, and then the error is forwarded to make.