From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 38249158012 for ; Wed, 22 Sep 2021 20:03:17 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 81009E0909; Wed, 22 Sep 2021 20:03:16 +0000 (UTC) Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) (using TLSv1.2 with cipher ECDHE-RSA-CHACHA20-POLY1305 (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 6DCF9E08CE for ; Wed, 22 Sep 2021 20:03:16 +0000 (UTC) Received: by mail-pg1-f177.google.com with SMTP id 17so3911770pgp.4 for ; Wed, 22 Sep 2021 13:03:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=VxrSIOoKtfIchy0aVVv3e7xY8pMYwg73XmKnaIh1hwU=; b=LwQEFOS+hs+jRDjSeNw+pbC8USVOXXqje3o6BL+5Dqx/tzoFfpzLQ6OKE8vXQmlOAQ +OIUbWXl/vJRNmXNh1UkTBpm48O/+DLHlZNAgK50VzprdeGyi4wFtLd/JAotv6YtSw0D SM0nfNXUF3pDXG2zDKE2qMk6yv0hPosIIN5BMn/d+9xhEUmjVrP+j9Jhe6VkiSWsliZU B1fff5ampTNRXEpInJ1c4y7KecYL9qfjFQIfSnI+Vv8ONAB1vGcjzq5nOXQP575swwzu QnK0oDJkq/FUa9pWdSHRdUPbpSL1vzq83qHhSKE1PkTxcYy471xM/mGh4qFBIrY6IM2U SMdw== X-Gm-Message-State: AOAM531Fuo7wN9jwrWf7D0Vtt0i+3QL1rAX7hZM16TiDOceukKkCLAC/ TKx8jOwTTGVM4+N7wuUO1YhVl3cQUu4ngo0+vzl973Hnj6ESeg== X-Google-Smtp-Source: ABdhPJwhKHDTIj7F6zruu3KcoHrw4XQUnHgcy3Y6zRUoFqdd7a69fPG64QQQ8RSW6lDm80df5y+rDcjv2DeE8q9fYSg= X-Received: by 2002:a62:6493:0:b0:43c:e252:3dfc with SMTP id y141-20020a626493000000b0043ce2523dfcmr806391pfb.60.1632340995001; Wed, 22 Sep 2021 13:03:15 -0700 (PDT) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-embedded@lists.gentoo.org Reply-to: gentoo-embedded@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 References: <15daab32-737d-621e-20b6-18c34294a37c@wildgooses.com> <1d2db9ca-7a12-ca9b-88c3-e20d45b7b647@wildgooses.com> In-Reply-To: <1d2db9ca-7a12-ca9b-88c3-e20d45b7b647@wildgooses.com> From: Michael Jones Date: Wed, 22 Sep 2021 15:03:03 -0500 Message-ID: Subject: Re: [gentoo-embedded] Mixed arch chroot to optimise building in qemu To: gentoo-embedded@lists.gentoo.org Cc: lists@wildgooses.com Content-Type: multipart/alternative; boundary="00000000000028e4f905cc9b0167" X-Archives-Salt: d920742d-d0e0-473f-81e4-2371208a8cf8 X-Archives-Hash: 67b30a9bb179657109d0fc824cb7c261 --00000000000028e4f905cc9b0167 Content-Type: text/plain; charset="UTF-8" On Wed, Sep 22, 2021, 14:54 Ed W wrote: > On 22/09/2021 20:26, Michael Jones wrote: > > > > On Wed, Sep 22, 2021 at 1:20 PM Ed W wrote: > >> Hi all, traffic seems to have dropped off here significantly, but here >> goes >> >> I am building a bunch of armv7a images on an AMD Ryzen9 machine (amd64). >> So to keep things simple I >> have just been doing the whole thing using qemu up until now, by which I >> mean I have an arm stage 3 >> somewhere, I chroot into it and then using userspace qemu binaries I just >> run my whole script to >> generate the target build from inside that chroot. This works but it's at >> least a 5x slowdown from >> native >> >> To optimise this I have tried >> >> - turning on the various compiler options for python (claimed to give a >> 30% improvement) + LTO/PGO. >> I don't notice any difference in the chroot - presume that the emulation >> overhead is dominant effect >> >> - tried compiling qemu with -O3 and LTO (claimed to be supported since >> 6.0). Doesn't give any >> noticeable different in performance of emerge >> >> - Added a static compiled amd64 /bin/bash to the chroot - now this does >> give a noticeable boost to >> compile and emerge speeds. (random benchmark went from 26s to 22s) >> >> >> So motivated by the last item I want to try and see how many native exes >> I can push into the chroot >> (since I'm running under usermode qemu! why not!). The obvious one is the >> compiler >> >> Now, I have a cross compiler built, but a) that's not static, so I would >> need to find a way to get >> native libc into the chroot, and b) I'm not clear how I would call it >> inside the chroot, could I >> just move a symlink to the other compiler into the path? How does it find >> things like libgcc*.so etc? >> >> Or perhaps this is easier than this? Can I just use some incantation in >> the same way that the >> crosscompiler must be working to build myself a straight gcc inside the >> chroot which is native arch >> and statically compiled? eg is it enough that assuming I can build gcc >> static, can I just do this >> from outside the chroot and overwrite the native: >> >> ROOT=$PWD emerge -1v --nodeps gcc >> >> >> It seems to me that this should work at least for the gcc binaries, etc. >> However, I'm completely >> ignorant of whether I want things like the linker plugin in native arch >> or target arch? What about >> the libgcc*.so files? (They don't actually exist in my cross compiler >> directories, but they are >> linked in as dependencies in some binaries in target and exist in the >> native compiler dir) >> >> Hacker news had someone do this recently and I believe meego used to do >> something similar, so really >> just trying to work out the details for this on gentoo. Any thoughts? >> >> Thanks >> >> Ed W >> > > > It's not clear to me if you're building gentoo images, or just building > some application. > > If you're building gentoo images, you might consider this project > https://github.com/GenPi64 , we'd love to work with you on the mixed arch > situation, since we suffer the same problem. > > > These are whole gentoo images. :-) > > So it's nothing special, but something like I drop into the arm chroot, > then there is a whole pile of something like: > > ROOT=/mnt/new_image emerge $stuff > > And at the end of all of that you have a shiny image to boot from (on an > imx based SOM as it happens). > > Nice thing about this approach is that I need to build the same system for > i386, amd64 and 32bit arm, and basically it means only running the same > build script in each individual chroot, so it's quite nice not needing to > fixup stuff for each platform. > > > There are arm64bit boxes you can rent from AWS and similar, but we see a > few build oddities on this which still need fixing and at least as near as > I can see they are still quite a bit slower than using an intel processor > in native mode. > > > I'm just about to (re) try using distcc, which basically achieves the > required end goal, so that I can measure performance. So something like run > up a side by side chroot using crossdev, then fire up distcc in there and > talk to it from your arm chroot. This gives less speedup than you would > like because it needs quite a lot of work on the arm qemu side and > serialising stuff, etc. Also linking etc is still on the arm side. > > I think the replacing of the bash binary with a native static binary is > giving a decent speedup. I'm about to try swapping in pypy to see how that > behaves. > > However, there is no doubt that getting the native cross compiler into the > chroot is the solution, but there are more than a few challenges here, such > as how to get it statically compiled and how to insert some or all of it > into the arm chroot. > > See here for inspiration and I guess also the meego stuff from history: > > https://news.ycombinator.com/item?id=28376447 > > > Thanks for any tips! > > Ed W > The genpi64 project does use distcc for building images when configured. Like I said, I think there'd be a big benefit to collaborating, but the image builder is usable as is for your purpose, if I understand it correctly. Its just missing the native binaries to speed things up. > --00000000000028e4f905cc9b0167 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Wed, Sep 22, 2021, 14:54 Ed W <lists@wildgooses.com> wrote:
=20 =20 =20
On 22/09/2021 20:26, Michael Jones wrote:
=20


On Wed, Sep 22, 2021 at 1:2= 0 PM Ed W <lists@wildgooses.com> wrote:
Hi all, traffic= seems to have dropped off here significantly, but here goes

I am building a bunch of armv7a images on an AMD Ryzen9 machine (amd64). So to keep things simple I
have just been doing the whole thing using qemu up until now, by which I mean I have an arm stage 3
somewhere, I chroot into it and then using userspace qemu binaries I just run my whole script to
generate the target build from inside that chroot. This works but it's at least a 5x slowdown from
native

To optimise this I have tried

- turning on the various compiler options for python (claimed to give a 30% improvement) + LTO/PGO.
I don't notice any difference in the chroot - presume that the emulation overhead is dominant effect

- tried compiling qemu with -O3 and LTO (claimed to be supported since 6.0). Doesn't give any
noticeable different in performance of emerge

- Added a static compiled amd64 /bin/bash to the chroot - now this does give a noticeable boost to
compile and emerge speeds. (random benchmark went from 26s to 22s)


So motivated by the last item I want to try and see how many native exes I can push into the chroot
(since I'm running under usermode qemu! why not!). The obvious one is the compiler

Now, I have a cross compiler built, but a) that's not static, so I would need to find a way to get
native libc into the chroot, and b) I'm not clear how I would call it inside the chroot, could I
just move a symlink to the other compiler into the path? How does it find things like libgcc*.so etc?

Or perhaps this is easier than this? Can I just use some incantation in the same way that the
crosscompiler must be working to build myself a straight gcc inside the chroot which is native arch
and statically compiled? eg is it enough that assuming I can build gcc static, can I just do this
from outside the chroot and overwrite the native:

=C2=A0=C2=A0=C2=A0 ROOT=3D$PWD emerge -1v --nodeps gcc


It seems to me that this should work at least for the gcc binaries, etc. However, I'm completely
ignorant of whether I want things like the linker plugin in native arch or target arch? What about
the libgcc*.so files? (They don't actually exist in my cros= s compiler directories, but they are
linked in as dependencies in some binaries in target and exist in the native compiler dir)

Hacker news had someone do this recently and I believe meego used to do something similar, so really
just trying to work out the details for this on gentoo. Any thoughts?

Thanks

Ed W


It's not clear to me if you're building gentoo image= s, or just building some application.

If you're building gentoo images, you might consider thi= s project https://github.com/GenPi64 , we'd love to work with you on the mixed arch situation, since we suffer the same problem.


These are whole gentoo images. :-)

So it's nothing special, but something like I drop into the arm chroot, then there is a whole pile of something like:

=C2=A0=C2=A0=C2=A0 ROOT=3D/mnt/new_image emerge $stuff

And at the end of all of that you have a shiny image to boot from (on an imx based SOM as it happens).=C2=A0

Nice thing about this approach is that I need to build the same system for i386, amd64 and 32bit arm, and basically it means only running the same build script in each individual chroot, so it's quite nice not needing to fixup stuff for each platform.


There are arm64bit boxes you can rent from AWS and similar, but we see a few build oddities on this which still need fixing and at least as near as I can see they are still quite a bit slower than using an intel processor in native mode.


I'm just about to (re) try using distcc, which basically achieve= s the required end goal, so that I can measure performance. So something like run up a side by side chroot using crossdev, then fire up distcc in there and talk to it from your arm chroot. This gives less speedup than you would like because it needs quite a lot of work on the arm qemu side and serialising stuff, etc. Also linking etc is still on the arm side.

I think the replacing of the bash binary with a native static binary is giving a decent speedup. I'm about to try swapping in pypy to see how that behaves.

However, there is no doubt that getting the native cross compiler into the chroot is the solution, but there are more than a few challenges here, such as how to get it statically compiled and how to insert some or all of it into the arm chroot.

See here for inspiration and I guess also the meego stuff from history:

=C2=A0=C2=A0=C2=A0 https://news.ycombinator.= com/item?id=3D28376447


Thanks for any tips!

Ed W



The genpi64 project does u= se distcc for building images when configured.

<= /div>
Like I said, I think there'd be a big benefit to= collaborating, but the image builder is usable as is for your purpose, if = I understand it correctly. Its just missing the native binaries to speed th= ings up.

--00000000000028e4f905cc9b0167--