[gentoo-alt] Prefix bootstrap ca-certificates build failure with Python portageq

[Stephen McCamant <mccamant@cs.umn.edu>]

[-- Attachment #1: message body text --]
[-- Type: text/plain, Size: 4193 bytes --]

I appreciate Fabian looking into the previous problem I reported and
suggesting a workaround, which sounds like it should probably work.
However I haven't been able to test it yet, because when I retry the
bootstrap, a new problem arises earlier in the bootstrap process, on
ca-certificates. It's not so clear to me in this case what the real
problem is, including that I haven't found what might have changed
recently to trigger it. So I'd be interested if anyone has any
insights.

As I understand it, ca-certificates's "src_compile" phase uses a
Python script certdata2pem.py in the upstream package in order to
convert the certificate information into the appropriate format to
install.  Because of this, it comes after python-3.5.5 in the
dependency order, and the ebuild checks to see that an appropriate
Python version is available. In the bootstrap I did on Monday, this
seemed to work correctly. But when I tried again yesterday and today,
there was a failure in the process of detecting the installed Python
version. It looks like the key error message is:

ERROR: This version of portageq only supports <eroot>s ending in
'/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018'. The provided <eroot>,
'/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/tmp', doesn't.
 * ERROR: app-misc/ca-certificates-20180409.3.37::gentoo failed
   (compile phase):
 *   has_version: unexpected portageq exit code: 64
 * 
 * Call stack:
 *          ebuild.sh, line  124:  Called src_compile
 *        environment, line 3035:  Called python_setup
 *        environment, line 2908:  Called _python_EPYTHON_supported 'python3.5'
 *        environment, line  338:  Called python_is_installed 'python3_5'
 *        environment, line 2762:  Called has_version '--host-root'
                                            'dev-lang/python:3.5'
 *   phase-helpers.sh, line  882:  Called die

I see the same error message from portageq if I manually rerun a
similar command:

% $PREFIX/tmp/usr/bin/portageq has_version \
     /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018 \
     dev-lang/python:3.5      
ERROR: This version of portageq only supports <eroot>s ending in
'/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/tmp'. The
provided <eroot>,
'/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018', doesn't.

If I add "/tmp" to the end of the eroot argument, the script returns 1
(not present). That seems correct, because there is indeed no
python3.5 in $PREFIX/tmp, only in $PREFIX. 

$PREFIX/tmp has python2.7, which seems like it would be sufficient if
I'm reading the PYTHON_COMPAT variable in the ca-certifcates ebuild
file correctly. If this is working like a cross-compile, maybe that's
the one we would want to use, if $PREFIX/tmp is the host and $PREFIX
is the target.

As I understand it, $PREFIX/tmp is the place where the special version
of the Portage tools installed in stage1 goes. This version of Portage
is still in use at this part of stage3, since it's not until later in
stage3 that Portage is rebuilt. But at this point we are building
packages that will be installed into the real Prefix environment, not
temporary bootstrapping versions. So on one level it makes sense that
the temporary portageq would complain when you ask about a the
presence of a package in a different root. However this seems like the
kind of difference that must be dealt with in lots of places to make
the bootstrapping approach work, so I guess the real mystery is why it
has recently started failing here. There's some fairly complex
machinery in the levels of the backtrace between python_setup and the
shell out to portageq that isn't very transparent to me, so maybe it
used to take a different code path that avoided this problem.

The general setting here, as before, is an x86_64 Ubuntu 16.04 host
system which has a bunch of packages of its own installed. I've
attached the "emerge --info" output and the tail of the stage3.log.

The version of ca-certificates recently increased from 3.36.1 to 3.37,
but that doesn't look to me like it's relevant because a bootstrap
yesterday failed with 3.36.1, and one today failed with 3.37.

Thanks,

 -- Stephen


[-- Attachment #2: emerge --info output --]
[-- Type: text/plain, Size: 4502 bytes --]

Portage 2.3.18-prefix (python 2.7.3-final-0, default/linux/amd64/17.0/no-multilib/prefix/kernel-3.2+, gcc-7.3.0, unavailable, 4.13.0-39-generic x86_64)
=================================================================
                         System Settings
=================================================================
System uname: Linux-4.13.0-39-generic-x86_64-with-debian-stretch-sid
KiB Mem:   197991096 total, 127774272 free
KiB Swap:    3905532 total,   3905532 free
Timestamp of repository gentoo: Wed, 09 May 2018 00:45:01 +0000
Head commit of repository gentoo: eaecf78a1fa14f381ced2ff3931eaeba16238a6c
sh bash 4.4_p19
ld GNU ld (GNU Binutils for Ubuntu) 2.26.1
app-shells/bash:      4.4_p19::gentoo
sys-devel/binutils:   2.30-r2::gentoo
sys-devel/gcc:        7.3.0-r1::gentoo
sys-devel/gcc-config: 1.9.1::gentoo
Repositories:

gentoo
    location: /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.prefix.bitzolder.nl/gentoo-portage-prefix
    priority: -1000
    sync-rsync-extra-opts: 

ACCEPT_KEYWORDS="amd64 ~amd64 ~amd64-linux"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc"
CONFIG_PROTECT_MASK="/etc/env.d /etc/gconf /etc/gentoo-release /etc/terminfo"
CXXFLAGS="-O2 -pipe -O2 -pipe"
DISTDIR="/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/tmp/usr/portage/distfiles"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles force-prefix merge-sync multilib-strict news parallel-fetch preserve-libs protect-owned sfperms strict unknown-features-warn unmerge-logs unmerge-orphans unprivileged"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j7"
PKGDIR="/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/tmp/usr/portage/packages"
PORTAGE_CONFIGROOT="/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/tmp/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/tmp/var/tmp"
USE="acl amd64 berkdb bzip2 cli crypt cxx dri fortran gdbm iconv ipv6 modules ncurses nls nptl openmp pcre prefix readline seccomp ssl tcpd unicode zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon plan sheets stage words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6 php7-0" POSTGRES_TARGETS="postgres9_5" PYTHON_SINGLE_TARGET="python3_5" PYTHON_TARGETS="python2_7 python3_5" RUBY_TARGETS="ruby22 ruby23" USERLAND="GNU" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

[-- Attachment #3: Tail of stage3.log --]
[-- Type: text/plain, Size: 9753 bytes --]

>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/http/__pycache__/client.cpython-35.opt-2.pyc
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/http/__pycache__/server.cpython-35.opt-2.pyc
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/http/__pycache__/cookies.cpython-35.opt-1.pyc
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/http/__pycache__/cookiejar.cpython-35.opt-1.pyc
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/http/__pycache__/cookiejar.cpython-35.pyc
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/http/__pycache__/server.cpython-35.pyc
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/http/__pycache__/__init__.cpython-35.pyc
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/http/__pycache__/client.cpython-35.opt-1.pyc
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/http/__pycache__/__init__.cpython-35.opt-2.pyc
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/http/__pycache__/cookies.cpython-35.opt-2.pyc
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/http/__pycache__/cookiejar.cpython-35.opt-2.pyc
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/http/__pycache__/__init__.cpython-35.opt-1.pyc
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/http/__pycache__/cookies.cpython-35.pyc
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/http/server.py
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/http/client.py
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/http/__init__.py
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/http/cookiejar.py
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/http/cookies.py
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/gettext.py
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/signal.py
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/sysconfig.py
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/py_compile.py
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/__phello__.foo.py
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/_collections_abc.py
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/python3.5/linecache.py
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/libpython3.5m.a
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/libpython3.5m.so.1.0
--- /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/pkgconfig/
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/pkgconfig/python-3.5.pc
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/pkgconfig/python-3.5m.pc -> python-3.5.pc
--- /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/etc/
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/etc/init.d/
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/etc/init.d/pydoc-3.5
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/etc/conf.d/
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/etc/conf.d/pydoc-3.5
>>> /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/lib64/libpython3.5m.so -> libpython3.5m.so.1.0
>>> dev-lang/python-3.5.5-r1 merged.
>>> Regenerating /etc/ld.so.cache...

>>> Emerging (30 of 37) app-misc/ca-certificates-20180409.3.37::gentoo
 * ca-certificates_20180409.tar.xz BLAKE2B SHA512 size ;-) ...           [ ok ]
 * nss-3.37.tar.gz BLAKE2B SHA512 size ;-) ...                           [ ok ]
 * You should run update-ca-certificates manually after etc-update
>>> Unpacking source...
>>> Unpacking ca-certificates_20180409.tar.xz to /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/var/tmp/portage/app-misc/ca-certificates-20180409.3.37/work
>>> Unpacking nss-3.37.tar.gz to /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/var/tmp/portage/app-misc/ca-certificates-20180409.3.37/work
>>> Source unpacked in /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/var/tmp/portage/app-misc/ca-certificates-20180409.3.37/work
>>> Preparing source in /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/var/tmp/portage/app-misc/ca-certificates-20180409.3.37/work ...
 * Applying ca-certificates-20150426-root.patch ...
 [ ok ]
>>> Source prepared.
>>> Configuring source in /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/var/tmp/portage/app-misc/ca-certificates-20180409.3.37/work ...
>>> Source configured.
>>> Compiling source in /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/var/tmp/portage/app-misc/ca-certificates-20180409.3.37/work ...
ERROR: This version of portageq only supports <eroot>s ending in '/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018'. The provided <eroot>, '/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/tmp', doesn't.
 * ERROR: app-misc/ca-certificates-20180409.3.37::gentoo failed (compile phase):
 *   has_version: unexpected portageq exit code: 64
 * 
 * Call stack:
 *          ebuild.sh, line  124:  Called src_compile
 *        environment, line 3035:  Called python_setup
 *        environment, line 2908:  Called _python_EPYTHON_supported 'python3.5'
 *        environment, line  338:  Called python_is_installed 'python3_5'
 *        environment, line 2762:  Called has_version '--host-root' 'dev-lang/python:3.5'
 *   phase-helpers.sh, line  882:  Called die
 * The specific snippet of code:
 *   				die "${FUNCNAME[0]}: unexpected portageq exit code: ${retval}"
 * 
 * If you need support, post the output of `emerge --info '=app-misc/ca-certificates-20180409.3.37::gentoo'`,
 * the complete build log and the output of `emerge -pqv '=app-misc/ca-certificates-20180409.3.37::gentoo'`.
 * The complete build log is located at '/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/var/tmp/portage/app-misc/ca-certificates-20180409.3.37/temp/build.log'.
 * The ebuild environment file is located at '/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/var/tmp/portage/app-misc/ca-certificates-20180409.3.37/temp/environment'.
 * Working directory: '/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/var/tmp/portage/app-misc/ca-certificates-20180409.3.37/work/image/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018'
 * S: '/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/var/tmp/portage/app-misc/ca-certificates-20180409.3.37/work'

>>> Failed to emerge app-misc/ca-certificates-20180409.3.37, Log file:

>>>  '/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/var/tmp/portage/app-misc/ca-certificates-20180409.3.37/temp/build.log'

 * Messages for package dev-lang/python-3.5.5-r1:

 * Failed to set PT_PAX markings -m python.
 * Failed to set XATTR_PAX markings -m python.
 * Failed to set PT_PAX markings -m /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/var/tmp/portage/dev-lang/python-3.5.5-r1/image/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/bin/python3.5m.
 * Failed to set XATTR_PAX markings -m /scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/var/tmp/portage/dev-lang/python-3.5.5-r1/image/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/usr/bin/python3.5m.

 * Messages for package app-misc/ca-certificates-20180409.3.37:

 * You should run update-ca-certificates manually after etc-update
 * ERROR: app-misc/ca-certificates-20180409.3.37::gentoo failed (compile phase):
 *   has_version: unexpected portageq exit code: 64
 * 
 * Call stack:
 *          ebuild.sh, line  124:  Called src_compile
 *        environment, line 3035:  Called python_setup
 *        environment, line 2908:  Called _python_EPYTHON_supported 'python3.5'
 *        environment, line  338:  Called python_is_installed 'python3_5'
 *        environment, line 2762:  Called has_version '--host-root' 'dev-lang/python:3.5'
 *   phase-helpers.sh, line  882:  Called die
 * The specific snippet of code:
 *   				die "${FUNCNAME[0]}: unexpected portageq exit code: ${retval}"
 * 
 * If you need support, post the output of `emerge --info '=app-misc/ca-certificates-20180409.3.37::gentoo'`,
 * the complete build log and the output of `emerge -pqv '=app-misc/ca-certificates-20180409.3.37::gentoo'`.
 * The complete build log is located at '/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/var/tmp/portage/app-misc/ca-certificates-20180409.3.37/temp/build.log'.
 * The ebuild environment file is located at '/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/var/tmp/portage/app-misc/ca-certificates-20180409.3.37/temp/environment'.
 * Working directory: '/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/var/tmp/portage/app-misc/ca-certificates-20180409.3.37/work/image/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018'
 * S: '/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/var/tmp/portage/app-misc/ca-certificates-20180409.3.37/work'

 * IMPORTANT: config file '/scratch/heathrow/mccamant-public/gentoo-prefix-64-may2018/etc/hosts' needs updating.
 * See the CONFIGURATION FILES and CONFIGURATION FILES UPDATE TOOLS
 * sections of the emerge man page to learn how to update config files.