[gentoo-alt] [PREFIX] the road ahead for Darwin/macOS

[gentoo-alt] [PREFIX] the road ahead for Darwin/macOS

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 3033 bytes --]

To all interested in Gentoo Prefix on recent macOS.

In the past few releases of macOS, /usr/include has disappeared.  While
this previously was remedied by performing some commands, Apple has made
this more strict, and it appears the way forwards, is by no longer
relying on /usr/include.

To deal with this, the packages sys-kernel/xnu-headers and
sys-libs/darwin-libc-headers were added to @system for newer macOS
profiles.  While you may have seen these packages coming in, getting
some updates here and there, they may still not be perfect, yet I think
we're getting somewhere.  The purpose of these packages obviously is to
provide system headers and frameworks without relying on Xcode.

Next I've modified the bootstrap script to unpack a Prefix-built clang
compiler, and use it to bootstrap.  This, in addition to the headers,
allowed me to progress until stage2 where a compiler is built.  This
failed, for a number of reasons.  1) requirement of cmake, which is
horrible.  It requires way too much in terms of dependencies, so I
resorted to an unpacked Prefix-built version to provide it.  Can't use
upstream's binaries for cmake, because they insist on using Xcode SDKs.
Next, 2) clang fails to build halfway because it throws away CXXFLAGS
pointing to the headers provided.  I seem to remember that this is what
GCC build does as well, so not that surprising, but it basically renders
the approach useless.

This lead me to investigating how to tell the bootstrap compiler to keep
looking for the headers we provided.  Turns out the compiler is
hard-wired to add c++ headers (stdinc) to its search-path, and this is
necessary (at least on Darwin) because the math.h header appears twice,
and only one of them is compatible with c++ code.  Anyway.  Apple seems
to use -isysroot to point to an SDK that's selected/targetted.  It seems
to me, at this point we should follow that lead, and simply turn the
Prefix into such sysroot.  Experimenting with this option on our
Prefix-build clang, results in not exactly desirable behaviour.  Because
the offset prefix is added to the compiler's search paths, nothing ends
up right.
So I downloaded clang binary from llvm.org, and tried the same.  That
made more sense, and it seems the search path is correct there.

What I want to try now, is if I can set CC/CXX to "clang -isysroot
/path/to/prefix", and if that will get me through compiler building of
stage2 during bootstrap.

If that works, it seems the first step towards Xcode free bootstrapping
on macOS.  There will be some questions towards how we build our clang,
if we should step away from patching it and using isysroot instead.
Finally, how we're going to provide the binaries that we want to use
during bootstraps.  Ideally we can provide stage1/stage2 by simply
installing form binpkgs, for instance using q/portage-utils.

End of update, hopefully bootstrapping can be working again soon.

Thanks,
Fabian

-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [gentoo-alt] [PREFIX] the road ahead for Darwin/macOS

[Fabian Groffen]

[-- Attachment #1: Type: text/plain, Size: 4406 bytes --]

To whom it may concern.

After a few turbulent weeks, with lots of input and help from various
people, the macOS bootstraps appear to be functional again.

The recent default Python switch from 3.7 to 3.8 threw some mud on the
road, but I can confirm this is now sorted out/fixed for:
- powerpc-apple-darwin9
- x86_64-apple-darwin17

I have reasons to believe darwin20 (Big Sur) is also functioning, right
before the Python thing happened, darwin19 and 20 were bootstrapping
just fine.

Bootstraps use Python 3.8.6 straight from stage1 now, and the resulting
Prefix has just that single Python installed, so it's pretty much lean
and clean.  On macOS, by default a strategy of using FSF GCC (10.1.0 at
this time) is used, which means, no Clang, but efforts for trying to get
that running are underway (bug #758167).  There is no linker/cctools
being built, instead the Xcode ld/as etc. are used.  This functions
fairly well, and should provide a stable and functional enough Prefix
for most users.

Thanks to those who have been patient, and thanks Sam for jumping on the
Prefix train to bring it forwards.

Fabian


On 27-12-2019 11:45:46 +0100, Fabian Groffen wrote:
> To all interested in Gentoo Prefix on recent macOS.
> 
> In the past few releases of macOS, /usr/include has disappeared.  While
> this previously was remedied by performing some commands, Apple has made
> this more strict, and it appears the way forwards, is by no longer
> relying on /usr/include.
> 
> To deal with this, the packages sys-kernel/xnu-headers and
> sys-libs/darwin-libc-headers were added to @system for newer macOS
> profiles.  While you may have seen these packages coming in, getting
> some updates here and there, they may still not be perfect, yet I think
> we're getting somewhere.  The purpose of these packages obviously is to
> provide system headers and frameworks without relying on Xcode.
> 
> Next I've modified the bootstrap script to unpack a Prefix-built clang
> compiler, and use it to bootstrap.  This, in addition to the headers,
> allowed me to progress until stage2 where a compiler is built.  This
> failed, for a number of reasons.  1) requirement of cmake, which is
> horrible.  It requires way too much in terms of dependencies, so I
> resorted to an unpacked Prefix-built version to provide it.  Can't use
> upstream's binaries for cmake, because they insist on using Xcode SDKs.
> Next, 2) clang fails to build halfway because it throws away CXXFLAGS
> pointing to the headers provided.  I seem to remember that this is what
> GCC build does as well, so not that surprising, but it basically renders
> the approach useless.
> 
> This lead me to investigating how to tell the bootstrap compiler to keep
> looking for the headers we provided.  Turns out the compiler is
> hard-wired to add c++ headers (stdinc) to its search-path, and this is
> necessary (at least on Darwin) because the math.h header appears twice,
> and only one of them is compatible with c++ code.  Anyway.  Apple seems
> to use -isysroot to point to an SDK that's selected/targetted.  It seems
> to me, at this point we should follow that lead, and simply turn the
> Prefix into such sysroot.  Experimenting with this option on our
> Prefix-build clang, results in not exactly desirable behaviour.  Because
> the offset prefix is added to the compiler's search paths, nothing ends
> up right.
> So I downloaded clang binary from llvm.org, and tried the same.  That
> made more sense, and it seems the search path is correct there.
> 
> What I want to try now, is if I can set CC/CXX to "clang -isysroot
> /path/to/prefix", and if that will get me through compiler building of
> stage2 during bootstrap.
> 
> If that works, it seems the first step towards Xcode free bootstrapping
> on macOS.  There will be some questions towards how we build our clang,
> if we should step away from patching it and using isysroot instead.
> Finally, how we're going to provide the binaries that we want to use
> during bootstraps.  Ideally we can provide stage1/stage2 by simply
> installing form binpkgs, for instance using q/portage-utils.
> 
> End of update, hopefully bootstrapping can be working again soon.
> 
> Thanks,
> Fabian
> 
> -- 
> Fabian Groffen
> Gentoo on a different level



-- 
Fabian Groffen
Gentoo on a different level

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]