Re: [gentoo-pms] [PATCH] Correct the definition of ESYSROOT as EPREFIX isn't always applicable

[James Le Cuirot <chewi@gentoo.org>]

[-- Attachment #1: Type: text/plain, Size: 7592 bytes --]

On Sun, 02 Jun 2019 08:10:34 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> On Sat, 2019-06-01 at 23:29 +0100, James Le Cuirot wrote:
> > Unfortunately my conception of ESYSROOT was a little short-sighted. It
> > was previously defined as SYSROOT + EPREFIX but if SYSROOT does not
> > equal ROOT then EPREFIX does not make sense in this context. Consider
> > a more concrete example:
> >   
> 
> Once again, you are missing the point that e.g. pkg-config files will
> have EPREFIX hardcoded.

I had remembered that you objected to this but I had forgotten why. I
said at the time that I believed your concerns were misplaced and that
hasn't changed but I wanted to actually try it out to prove it to
myself as much as you.

If EPREFIX is set in the environment by the user, it only applies to
ROOT. If you're emerging a bunch of packages in a single run with some
going to / and some going to ROOT, Portage will adjust EPREFIX for each
destination and hardcode the correct prefix value into the .pc files
accordingly.

Up till now, we have said that you can set SYSROOT to either match / or
ROOT. The former is typically used for native while the latter is
typically used for cross. Whether you're building against / or ROOT,
the .pc files picked up from each will have the prefix hardcoded for
each. This is exactly what we want!

It was surprisingly hard to find a simple package that worked under
prefix and located a header via pkg-config that otherwise would not be
found without an -I flag. After a long search, the package I eventually
settled on was your very own x11-libs/libtinynotify, which uses
pkg-config to query dbus-1.

My setup is as follows:
BROOT=/home/chewi/prefix
ROOT=/home/chewi/myroot
EPREFIX=/myprefix

Therefore, Portage sets:
EROOT=/home/chewi/myroot/myprefix

I wanted to try this with both SYSROOT=/ and SYSROOT=${ROOT}. Without
my associated changes to Portage, the former actually blows up.

  Calculating dependencies... done!
  Traceback (most recent call last):
    File "/home/chewi/prefix/usr/lib/python-exec/python3.6/emerge", line 53, in <module>
      retval = emerge_main()
    File "/home/chewi/prefix/usr/lib64/python3.6/site-packages/_emerge/main.py", line 1289, in emerge_main
      return run_action(emerge_config)
    File "/home/chewi/prefix/usr/lib64/python3.6/site-packages/_emerge/actions.py", line 3325, in run_action
      retval = action_build(emerge_config, spinner=spinner)
    ...
    File "/home/chewi/prefix/usr/lib64/python3.6/site-packages/_emerge/depgraph.py", line 4722, in _select_atoms_highest_available
      pkgsettings = self._frozen_config.pkgsettings[root]
  KeyError: '/myprefix/'

This error precisely highlights the problem with PMS as it stands now.
If SYSROOT=/ and EPREFIX=/myprefix then SYSROOT + EPREFIX is
also /myprefix, which Portage hasn't been told about because it doesn't
even exist. It only exists under /home/chewi/myroot.

With my changes in place, things are resolved as follows. dbus is
installed in both locations because libtinynotify is only EAPI 6.

[ebuild  N     ] sys-apps/dbus-1.12.14 to /home/chewi/myroot/myprefix/ USE="-X -debug -doc -elogind (-selinux) -static-libs -systemd -test -user-session" 
[ebuild  N     ] sys-apps/dbus-1.12.14  USE="-X -debug -doc -elogind (-selinux) -static-libs -systemd -test -user-session" 
[ebuild  N     ] x11-libs/libtinynotify-0.2.1 to /home/chewi/myroot/myprefix/ USE="-debug -doc -static-libs" 

Updating libtinynotify to EAPI 7 while building with SYSROOT=${ROOT}
changes the output to this.

[ebuild  N     ] sys-apps/dbus-1.12.14 to /home/chewi/myroot/myprefix/ USE="-X -debug -doc -elogind (-selinux) -static-libs -systemd -test -user-session" 
[ebuild  N     ] x11-libs/libtinynotify-0.2.1 to /home/chewi/myroot/myprefix/ USE="-debug -doc -static-libs" 

In all cases, both packages build successfully but pkg-config adds
-I/home/chewi/prefix/usr/include/dbus-1.0 regardless of SYSROOT. This
is because native builds like this have no special SYSROOT handling for
pkg-config. This handling exists in crossdev's cross-pkg-config
for cross builds but even that currently has no prefix support
whatsoever. I therefore have some heavy modifications pending for
cross-pkg-config.

It is debatable whether we should apply SYSROOT handling to pkg-config
all the time. While we do technically allow SYSROOT!=/ for native
builds, this is so problematic in general that we probably shouldn't go
out of our way to support it. We don't need to do anything for the
SYSROOT=/ case because the default behaviour is what we want.

In my example, I'm not cross-compiling so I could force the use of
cross-pkg-config to make it work but to keep things simple, I have
written a short bashrc that does the equivalent. This file is placed
in /home/chewi/myroot/myprefix/etc/portage because we only want it to
apply when building packages for ROOT. We therefore need to set
PORTAGE_CONFIGROOT so that it actually gets used.

  if [[ -n ${SYSROOT%/} ]]; then
      myprefix=${EPREFIX%/}
  else
      myprefix=${PORTAGE_OVERRIDE_EPREFIX%/}
  fi

  unset PKG_CONFIG_PATH
  export PKG_CONFIG_SYSTEM_LIBRARY_PATH="${myprefix}/usr/lib64:${myprefix}/lib64"
  export PKG_CONFIG_SYSROOT_DIR="${SYSROOT}"
  export PKG_CONFIG_LIBDIR="${SYSROOT}${myprefix}/usr/lib64/pkgconfig:${SYSROOT}${myprefix}/usr/share/pkgconfig"

In cross-pkg-config, I check ESYSROOT first and determine the prefix by
chopping off SYSROOT. This allows us to support a distinct SYSROOT and
is necessary because we didn't define a variable for SYSROOT's prefix.

For EAPIs before 7, I fall back to something along the lines of the
above. Using PORTAGE_OVERRIDE_EPREFIX works but it's not ideal so I may
just bake the value into cross-pkg-config when crossdev is installed.

And the result? It works as expected. Regardless of whether
libtinynotify uses EAPI 6 or 7, the build output includes the following:

With SYSROOT=/ :

  libtool: compile:  x86_64-pc-linux-gnu-gcc -DHAVE_CONFIG_H -I. -I/home/chewi/prefix/usr/include/dbus-1.0 -I/home/chewi/prefix/usr/lib64/dbus-1.0/include -O2 -pipe -O2 -pipe -c lib/common.c  -fPIC -DPIC -o .libs/libtinynotify_la-common.o

With SYSROOT=${ROOT} :

  libtool: compile:  x86_64-pc-linux-gnu-gcc -DHAVE_CONFIG_H -I. -I/home/chewi/myroot/myprefix/usr/include/dbus-1.0 -I/home/chewi/myroot/myprefix/usr/lib64/dbus-1.0/include -O2 -pipe -O2 -pipe -c lib/common.c  -fPIC -DPIC -o .libs/libtinynotify_la-common.o

For completeness, I decided to try the distinct SYSROOT case as well
with SYSROOT=/home/chewi/mysysroot. The short bashrc above does not
support this but the revised cross-pkg-config does. You would typically
need the libc and kernel headers installed to this SYSROOT first but I
haven't defined CC="x86_64-pc-linux-gnu-gcc --sysroot=${ESYSROOT}" like
I normally do under cross-boss. Only pkg-config is being affected here,
which is sufficient for this demonstration. And it works!

  libtool: compile:  x86_64-pc-linux-gnu-gcc -DHAVE_CONFIG_H -I. -I/home/chewi/mysysroot/usr/include/dbus-1.0 -I/home/chewi/mysysroot/usr/lib64/dbus-1.0/include -O2 -pipe -O2 -pipe -c lib/common.c  -fPIC -DPIC -o .libs/libtinynotify_la-common.o

Distinct SYSROOTs are not something I expect to be used frequently but
they are initially needed when cross-building a brand new system into
an empty directory, which is exactly what cross-boss does.

I hope this makes sense now.

-- 
James Le Cuirot (chewi)
Gentoo Linux Developer

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]