[gentoo-commits] proj/crossdev:master commit in: /

["Sergei Trofimovich" <slyfox@gentoo.org>]

commit:     7ea765f7ff98a0f07ee1f21cc83d2b4db81c02b2
Author:     Sergei Trofimovich <slyfox <AT> gentoo <DOT> org>
AuthorDate: Thu Apr  5 22:15:26 2018 +0000
Commit:     Sergei Trofimovich <slyfox <AT> gentoo <DOT> org>
CommitDate: Thu Apr  5 22:15:26 2018 +0000
URL:        https://gitweb.gentoo.org/proj/crossdev.git/commit/?id=7ea765f7

README: basic description of what crossdev

Signed-off-by: Sergei Trofimovich <slyfox <AT> gentoo.org>

 README | 281 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 281 insertions(+)

diff --git a/README b/README
new file mode 100644
index 0000000..4924258
--- /dev/null
+++ b/README
@@ -0,0 +1,281 @@
+What is crossdev
+----------------
+
+crossdev is a cross-compiler environment generator for Gentoo.
+
+It is useful for various purposes:
+
+- build cross-compiler toolchain for an operating system
+- build cross-compiler toolchain for embedded targets (bare metal)
+- cross-compile whole Gentoo on a new (or existing) target
+- cross-compile your favourite tool for every target out there
+  just to make sure it still compiles and works. Countless bugs
+  were found fixed like that :)
+
+Crossdev nano HOWTO
+-------------------
+
+So you want to cross-compile a Gentoo package (say busybox):
+
+# crossdev -t s390x-unknown-linux-gnu
+# (optional) ARCH=s390 PORTAGE_CONFIGROOT=/usr/s390x-unknown-linux-gnu eselect profile set default/linux/s390/17.0/s390x
+# USE=static s390x-unknown-linux-gnu emerge -v1 busybox
+# file /usr/s390x-unknown-linux-gnu/bin/busybox
+/usr/s390x-unknown-linux-gnu/bin/busybox: ELF 64-bit MSB executable, IBM S/390, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, stripped
+
+Done!
+
+You can even use qemu-user to run this binary (or even chroot
+to /usr/s390x-unknown-linux-gnu!
+
+https://wiki.gentoo.org/wiki/Crossdev_qemu-static-user-chroot
+
+Supported platforms
+-------------------
+
+Cross-compilation is fairly well supported to linux targets.
+Windows is not too broken either.
+
+But prepare for rough corners. This doc will try to help you
+understand what crossdev does and does not do.
+
+A few examples of targets that worked today (produce running
+executables or kernels if applies):
+
+ aarch64-unknown-linux-gnu-7.3.0
+ alpha-unknown-linux-gnu-7.3.0
+ arm-none-eabi-7.3.0
+ armv5tel-softfloat-linux-gnueabi-7.3.0
+ armv7a-hardfloat-linux-gnueabi-7.3.0
+ armv7a-unknown-linux-gnueabi-7.3.0
+ avr-7.3.0
+ hppa-unknown-linux-gnu-7.3.0
+ hppa2.0-unknown-linux-gnu-7.3.0
+ hppa64-unknown-linux-gnu-7.3.0
+ i686-pc-gnu-7.3.0
+ i686-w64-mingw32-7.3.0
+ ia64-unknown-linux-gnu-7.3.0
+ m68k-unknown-linux-gnu-7.3.0
+ mips-unknown-linux-gnu-7.3.0
+ mips64-unknown-linux-gnu-7.3.0
+ mips64el-unknown-linux-gnu-7.3.0
+ mipsel-unknown-linux-gnu-7.3.0
+ mmix-7.3.0
+ nios2-unknown-linux-gnu-7.3.0
+ powerpc-unknown-linux-gnu-7.3.0
+ powerpc64-unknown-linux-gnu-7.3.0
+ powerpc64le-unknown-linux-gnu-7.3.0
+ s390-unknown-linux-gnu-7.3.0
+ s390x-unknown-linux-gnu-7.3.0
+ sh4-unknown-linux-gnu-7.3.0
+ sparc-unknown-linux-gnu-7.3.0
+ sparc64-unknown-linux-gnu-7.3.0
+ x86_64-HEAD-linux-gnu-7.3.0
+ x86_64-UNREG-linux-gnu-7.3.0
+ x86_64-pc-linux-gnu-7.3.0
+ x86_64-w64-mingw32-7.3.0
+
+A few more things are likely to Just Work
+and many more can be made work with a litle touch.
+
+How crossdev works (high-level overview)
+----------------------------------------
+
+crossdev is a tiny shell wrapper around emerge. Wrapper overrides
+a few variabled to aim emerge to another target.
+
+Crossdev leverages following features of portage (and ::gentoo ebulds):
+
+- ability to override ROOT=/usr/<target> to install cross-compiled
+  packages into a new root on a filesystem to avoid cluttering host.
+
+- ability to override PORTAGE_CONFIGROOT=/usr/<target> to untangle
+  from host's portage configuration.
+
+  Namely crossdev populates /usr/<target>/etc/portage/ with defaults
+  suitable for cross-compiling (ARCH, KERNEL, ELIBC variables and so on).
+  You can change all of it.
+
+- set CBUILD/CHOST/CTARGET variables accordingly to force build
+  system into cross-compiling mode. For autotools-based system
+  it means running the following configure script:
+    ./configure --build=${CBUILD} --host=${CHOST} --target=${CTARGET}
+
+If toolchains were simple programs crossdev would be a one-liner script:
+
+  ARCH=...    \
+  CBUILD=...  \
+  CHOST=...   \
+  CTARGET=... \
+  ROOT=...    \
+      emerge "$@"
+
+Unfortunately todays' toolchains have loops in their build-time dependencies:
+
+- cross-compiler itself normally needs a libc built for <target> because
+  libc defines various aspects of userland ABI and features provided.
+- and libc for <target> needs a cross-comiler because it's written in C
+
+That's where crossdev comes in useful. It unties this vicious compiler<->libc
+circle by carefully rinning the following emerge commands (assume s390x-linux
+example).
+
+Here is what crossdev actually does:
+
+1. create an overlay with new ebuilds (symlinks to existing ebuilds)
+2. build cross-binutils
+    $ emerge cross-s390x-unknown-linux-gnu/binutils
+3. Install minimal set of system headers (kernel and libc)
+   $ USE="headers-only" emerge cross-s390x-unknown-linux-gnu/linux-headers
+   $ USE="headers-only" emerge cross-s390x-unknown-linux-gnu/glibc
+4. Build minimal GCC without libc support (not able link final executables yet)
+   $ USE="-*" emerge cross-s390x-unknown-linux-gnu/gcc
+5. Build complete libc
+   $ USE="-headers-only" emerge cross-s390x-unknown-linux-gnu/linux-headers
+   $ USE="-headers-only" emerge cross-s390x-unknown-linux-gnu/glibc
+6. Build full GCC (able to link final binaries and can do c++)
+   $ USE="" emerge cross-s390x-unknown-linux-gnu/gcc
+
+Done!
+
+How crossdev works (more details)
+---------------------------------
+
+This section contains more details on what actually happens.
+Here we expand on each step briefly outlined in previous section:
+
+1. create an overlay with new ebuilds (symlinks to existing ebuilds)
+   <skipping actual commands>. After this step the outcomes are:
+
+   - overlay layout is formed:
+
+     $ ls -l cross-overlay/cross-s390x-unknown-linux-gnu
+     binutils -> /gentoo-ebuilds/gentoo/sys-devel/binutils
+     gcc -> /gentoo-ebuilds/gentoo/sys-devel/gcc
+     glibc -> /gentoo-ebuilds/gentoo/sys-libs/glibc
+     linux-headers -> /gentoo-ebuilds/gentoo/sys-kernel/linux-headers
+
+   - /usr/cross-s390x-unknown-linux-gnu (aka $SYSROOT) layout is set:
+
+     $ ls -l /usr/s390x-unknown-linux-gnu/etc/portage/
+     make.conf
+     make.profile -> /gentoo-ebuilds/gentoo/profiles/embedded
+     profile/
+
+     Here we override ARCH, LIBC, KERNEL, CBUILD, CHOST, CTARGET and a
+     few other variables.
+
+
+   - a few convenience wrappers are created:
+
+     /usr/bin/s390x-unknown-linux-gnu-emerge -> cross-emerge
+     /usr/bin/s390x-unknown-linux-gnu-pkg-config -> cross-pkg-config
+     /usr/bin/s390x-unknown-linux-gnu-fix-root -> cross-fix-root
+
+   This way we share ebuild code and still can install cross-compilers
+   independently. Each with it's owv version of libc.
+
+2. build cross-binutils
+   $ emerge cross-s390x-unknown-linux-gnu/binutils
+
+   This way we can install the same version of binutils aiming different
+   targets. As a result we get tools like:
+   s390x-unknown-linux-gnu-ar (static library archiver)
+   s390x-unknown-linux-gnu-as (assembler)
+   s390x-unknown-linux-gnu-ld (linker)
+   ... <many others>
+
+   Nothing special here.
+
+3. Install minimal set of system headers (kernel and libc)
+   $ USE="headers-only" emerge cross-s390x-unknown-linux-gnu/linux-headers
+   $ USE="headers-only" emerge cross-s390x-unknown-linux-gnu/glibc
+
+   As we don't have cross-compiler yet we just copy a bunch on
+   header files into
+       /usr/s390x-unknown-linux-gnu/usr/include
+   and setup symlinks like:
+       /usr/s390x-unknown-linux-gnu/sys-include -> usr/include
+   to make cross-gcc happy.
+
+   These symlinks are target-dependent. A few unusual examples:
+
+   - windows (mingw): /usr/x86_64-w64-mingw32/mingw -> usr
+   - hurd: /usr/i686-pc-gnu/include -> usr/include
+   - DOS (didn't try yet): /usr/i686-pc-gnu/dev/env/DJDIR/include -> ../../../usr/include
+
+   Side note: we could have omited symlink creation completely
+   and build gcc with parameter:
+     --with-native-system-header-dir=${EPREFIX}/usr/include
+   That way ${SYSROOT} would be even more like normal root.
+   Worth a try this one line :)
+
+4. Build minimal GCC without libc support (not able link final executables yet)
+   $ USE="-*" emerge cross-s390x-unknown-linux-gnu/gcc
+
+   Here gcc uses headers from step [3.] to find out what
+   target libc can do:
+
+   - POSIX support
+   - trigonometry functions
+   - threading
+   - vital constants
+
+   As a result we only get C code generator. No knowledge
+   of how to link executables or shared libraries as those
+   require bits of libc.
+
+   For tiniest targets (bare-metal) this can be a final step to get basic toolchain.
+
+5. Build complete libc
+   $ USE="-headers-only" emerge cross-s390x-unknown-linux-gnu/linux-headers
+   $ USE="-headers-only" emerge cross-s390x-unknown-linux-gnu/glibc
+
+   Here we rebuild full libc against system headers. As a result we get C startup
+   files and can link full programs!
+
+6. Build full GCC (able to link final binaries and can do c++)
+   $ USE="" emerge cross-s390x-unknown-linux-gnu/gcc
+
+   Here we get full c++ support, various default flags enabled (pie, sanitizers,
+   stack protectors and many other things).
+
+   This thing is ready for large-scale operations.
+
+Various notes (AKA dirty little tricks)
+---------------------------------------
+
+- config.site
+
+  Some ./configure scripts rely on runtime feature testing. We would
+  still like to enable things even in cross-environment.
+
+  crossdev installs /usr/share/config.site with a bunch of cache
+  variables preset for targets. It might be a nice place to drop
+  more things into. Or a source of all your cross-compilation
+  problems :)
+
+- eclass inheritance
+
+  To find out various things about target crossdev loads multilib.eclass
+  and tries to find out default ABI supported by the target.
+
+- crossdev is just a tiny shell script around emerge :)
+
+  It's full source code of comparable to the size of this README.
+
+- USE=headers-only
+
+  Many toolchain ebuilds (mostly libcs and kernel headers) are aware
+  of headers-only install specifically for crossdev and similar tools
+  to be able to build cross-toolchains.
+
+- How to test crossdev layout generation:
+
+  $ mkdir -p foo
+  $ PORTAGE_CONFIGROOT=$(pwd)/foo EPREFIX=$(pwd)/foo ./crossdev -t mmix -P -p
+
+  This needs some local patching. TODO: fix it to Just Work (perhaps with
+  additional --test options).
+
+Happy cross-compiling!