[gentoo-releng] Uclibc chroots and catalyst

[gentoo-releng] Uclibc chroots and catalyst

[Joshua Kinard]

I am curious to know if anyone has had any odd issues with using Gentoo's
releng tool, catalyst, to build Uclibc-based stages before.

I was attempting to resurrect MIPS-III, big-endian stages once more using an
old ~2012 mips3 seed stage that Anthony Basile put together a while back.  I
started by manually updating the seed to be current (sans ncurses-6.x), and
that all worked out fine.  I then fed that seed stage into Catalyst to build a
new stage1, and that also worked fine.

But when I went to run the stage2 spec, it failed due to bash generating a
segfault via an attempted NULL pointer dereference.  So I manually tested by
attempting to chroot into the /tmp/stage1root directly, and that's also
segfaulting.  I can run other executables within the stage1root via chroot
directly, but it appears that anything linked to ncurses is triggering a segfault.

The original stages were built with gcc-5.3, so I fell back to gcc-4.8.5, but
the segfaults are still happening.  I forced the stage1root to upgrade to
ncurses-6.0, but that isn't working either.  I manually remerged bash by
setting ROOT="/tmp/stage1root", but it's linking against /lib/libncurses.so.5
from the seed root instead of /lib/libncurses.so.6 in the stage1root.

Not sure how to work around this.  Everything in the seed root works fine.  If
I manually merge select packages into a different $ROOT (uclibc, ncurses, bash,
readline, libiconv), then copy libgcc_s.so into $ROOT/lib manually, I can
execute bash there just fine.  So part of the problem may also be in how
catalyst's stage1 script is merging the packages into $ROOT.

Thoughts?

-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
6144R/F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943

"The past tempts us, the present confuses us, the future frightens us.  And our
lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic

Re: [gentoo-releng] Uclibc chroots and catalyst

[Anthony G. Basile]

On 12/14/15 1:56 AM, Joshua Kinard wrote:
> I am curious to know if anyone has had any odd issues with using Gentoo's
> releng tool, catalyst, to build Uclibc-based stages before.
>
> I was attempting to resurrect MIPS-III, big-endian stages once more using an
> old ~2012 mips3 seed stage that Anthony Basile put together a while back.  I
> started by manually updating the seed to be current (sans ncurses-6.x), and
> that all worked out fine.  I then fed that seed stage into Catalyst to build a
> new stage1, and that also worked fine.
>
> But when I went to run the stage2 spec, it failed due to bash generating a
> segfault via an attempted NULL pointer dereference.  So I manually tested by
> attempting to chroot into the /tmp/stage1root directly, and that's also
> segfaulting.  I can run other executables within the stage1root via chroot
> directly, but it appears that anything linked to ncurses is triggering a segfault.
>
> The original stages were built with gcc-5.3, so I fell back to gcc-4.8.5, but
> the segfaults are still happening.  I forced the stage1root to upgrade to
> ncurses-6.0, but that isn't working either.  I manually remerged bash by
> setting ROOT="/tmp/stage1root", but it's linking against /lib/libncurses.so.5
> from the seed root instead of /lib/libncurses.so.6 in the stage1root.
>
> Not sure how to work around this.  Everything in the seed root works fine.  If
> I manually merge select packages into a different $ROOT (uclibc, ncurses, bash,
> readline, libiconv), then copy libgcc_s.so into $ROOT/lib manually, I can
> execute bash there just fine.  So part of the problem may also be in how
> catalyst's stage1 script is merging the packages into $ROOT.
>
> Thoughts?
>

because mips is ~arch we are always building the cutting edge which is
going to get you into trouble.  without knowing what's cauing the seg
fault, its hard to proceed.  you can `strace chroot /path <cmd>` but
this may bog down in some heavy debugging.  i'd just start with the
original stage, keep updates back by masking in /etc/portage and
emerging that way.  take a look in the releng git repo I have some
scripts for automating stage1-2-3 with a customized /etc/portage
directory.  putting stuff in /etc/portage for our stage's is evil, but
its sometimes a necessary evil.  (vapier opened a bug about it.)

-- 
Anthony G. Basile, Ph.D.
Gentoo Linux Developer [Hardened]
E-Mail    : blueness@gentoo.org
GnuPG FP  : 1FED FAD9 D82C 52A5 3BAB  DC79 9384 FA6E F52D 4BBA
GnuPG ID  : F52D4BBA

Re: [gentoo-releng] Uclibc chroots and catalyst

[Joshua Kinard]

On 12/14/2015 02:25, Anthony G. Basile wrote:
> 
> because mips is ~arch we are always building the cutting edge which is
> going to get you into trouble.  without knowing what's cauing the seg
> fault, its hard to proceed.  you can `strace chroot /path <cmd>` but
> this may bog down in some heavy debugging.  i'd just start with the
> original stage, keep updates back by masking in /etc/portage and
> emerging that way.  take a look in the releng git repo I have some
> scripts for automating stage1-2-3 with a customized /etc/portage
> directory.  putting stuff in /etc/portage for our stage's is evil, but
> its sometimes a necessary evil.  (vapier opened a bug about it.)

I've already run strace, both from outside the alt $ROOT and from inside.  It
crashes two particular ioctl() calls, as far as I can tell, so that doesn't
tell me anything useful.  I also don't think it's tied to any particular
version of a package, because everything inside the seed stage works fine.  The
problem is only exposing itself when using 'emerge' to install to an alternate
$ROOT.

I did run into a hiccup where, when updating my seed root, something got merged
out-of-order and binutils/gcc both broke, so I had to do some temporary
kludgery to get them working again, then recompiled uclibc, and that problem
went away.

That said, I just did some experimenting and ran 'ldd' from inside the chroot
against bin/bash to see what libs bash is trying to load:

octane stage1root # chroot . usr/bin/ldd bin/bash
        libreadline.so.6 => /lib/libreadline.so.6 (0x77580000)
        libhistory.so.6 => /lib/libhistory.so.6 (0x77560000)
        libncurses.so.5 => /lib/libncurses.so.5 (0x774f0000)
        libiconv.so.2 => /lib/libiconv.so.2 (0x773e0000)
        libc.so.0 => /lib/libc.so.0 (0x77370000)
        ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x77630000)

Going by that, I moved the actual libs being pointed at to a temp folder and
copied the corresponding libs from my seed root in, ran "chroot . bin/bash",
and got a working shell.  So I know the problem *definitely* involves the way
the libraries are being linked to bash.

Next I swapped the libs back around again, and started replacing them
one-by-one, running "chroot . bin/bash" each time.  I got a working shell in
every instance *except* when I didn't swap libncurses.so.5.9 with the seed
$ROOT copy.  This correlates with other binaries linked again libncurses also
segfaulting.

So I think there's several potential issues here:

1. emerge isn't setting the compile environment up correctly so that
gcc/binutils are linking against libraries in the alt $ROOT instead of in the
seed $ROOT and only ncurses has issues with this.

2. uclibc's dynamic loader isn't as forgiving or flexible as glibc's in working
around whatever discrepancy exists in loading libncurses.so.5.9 (because all
things considered, the libncurses.so.5.9 in the seed $ROOT should be the
virtually same as in alt $ROOT).

3. There's a bug in the toolchain that only appears when under uclibc and using
an alt $ROOT, and only when compiling ncurses.

4. There's a bug in ncurses itself that only appears under uclibc (and/or alt
$ROOT).

5. I am using -mplt in my CFLAGS, and that might also be involved (going to
test this next).

6. Some other unidentified issue.


CC'ing toolchain in case they have any ideas.  I'll hold off on opening a bug
up until the actual problem is pinned down.


-- 
Joshua Kinard
Gentoo/MIPS
kumba@gentoo.org
6144R/F5C6C943 2015-04-27
177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943

"The past tempts us, the present confuses us, the future frightens us.  And our
lives slip away, moment by moment, lost in that vast, terrible in-between."

--Emperor Turhan, Centauri Republic