Re: [gentoo-mips] multilib problems on mips64 profiles

["Anthony G. Basile" <basile@opensource.dyc.edu>]

On 09/21/14 21:49, Joshua Kinard wrote:
> On 09/21/2014 21:11, Anthony G. Basile wrote:
>> On 09/21/14 18:53, Joshua Kinard wrote:
>>> On 09/21/2014 17:55, Anthony G. Basile wrote:
>>>> On 09/17/14 16:19, Anthony G. Basile wrote:
>>> [snip]
>>>>
>>>> This isn't going to work.  The problem is that the above structure doesn't
>>>> distinguish between mips/o32 and mips64/o32 (and equivalent el versions).  Now
>>>> mips64/o32 is funny, but possible: it would be o32 (a 32-bit abi) on 64 bit
>>>> arch/kernel, like ppc64-32ul.  The structure is possible at the level of
>>>> profile/arch, but it would mean a deep restructuring of
>>>> default/linux/mips/13.0.
>>>
>>> Err, I've been using this setup for the last 9+ years?  My Octane and O2 both
>>> boot mips64 kernels and run an o32-only userland.  It works fine with the
>>> current profile setup.  You just use a mips-unknown-linux-gnu CHOST.  I may not
>>> be able to fully utilize all of the CPU registers like I would under n32, but
>>> that's never been an issue for me.
>>>
>>> So there should be no need to differentiate at the userland level between
>>> mips/o32 and mips64/o32.  As long as the CHOST is set correctly, o32 will work
>>> fine under a mips64 kernel (caveat: make sure CONFIG_MIPS32_O32 is set in the
>>> kernel).
>>
>> I agree that there shouldn't be any userland difference, but I'm not 100%
>> sure.  Anyhow, let's go with that assumption for now.
>
> There isn't.  If there is a difference that can be detected (meaning, with a
> machine that can run in 32-bit or 64-bit mode with the same userland), then
> you're looking at a kernel problem and that would get handled via upstream
> channels.  You're actually at a big disadvantage running o32 under a 64-bit
> kernel, because of not being able to use all of the registers in the CPU, can
> only pass the first four function args via registers (8 in n32), etc.  o32 is
> for all intents and purposes, the MIPS-I and MIPS-II ISAs, which are core to
> the overall ISA set, and so all MIPS CPUs to date should (in theory, because I
> do not have access to mips64r* hardware) run o32 just fine.
>
> You can compile o32 with higher ISAs, i.e., mips3 and mips4, but it's always
> been debatable how much extra that helps you (Debian used to question our logic
> on this back in the day).  Can't use 64-bit features of 64-bit CPUs via that,
> but it is supposed to enable things like the square-root instructions available
> at the mips4 level.
>
> The only reason I'm still running o32 is I just haven't been motivated enough
> to do a full re-install of both systems.  Even if I switch to n32, I'll
> probably keep o32 chroots around just to make sure things don't break there.
> Might do this soon, though, as I just bought ten new hard drives for all of my
> MIPS systems.  Just need to get the right brackets for them (because they're
> 2.5" 10K SCA drives) so I can mount everything and do data migration.
>
>
>>>
>>> Correct me if wrong, but it seems the core problem here is that multilib
>>> inherits from the o32 base profile.  While I think the proper longterm fix is
>>> to have more discrete, modular/pluggable profile components (like OOP and
>>> multiple base classes), that's not going to happen in Portage for a long time.
>>
>> Not exactly.  The problem is that everything inherits from profiles/arch/mips
>> and currently that forces o32 with mgorny's multilib stuff.  We need to get
>> that out of the way for the other profiles that inherit it.
>>
>
> Agreed.  This worked fine in the past when o32 was our primary focus and
> n32/n64 were still experimental (and multilib didn't exist).  Nowadays, the
> base profile shouldn't prescribe a default ABI at all.  That needs to be
> handled under a subprofile somehow.
>
> The real problem is Portage's stacking profiles feature, while soving a lot of
> problems that the previous profile system had, wasn't thought up with all of
> MIPS' happyfun included.  Because endianness, ABI, ISA, kernel, libc, compiler,
> and even specific machine support can all be interchangeable in MIPS, this
> causes the stacking profile system to become a thing that would frighten
> Yog-Sothoth itself, if we were to fully implement all of the combinations
> possible under this system.
>
>
>>> So I think the solution is to split the profiles into "mips" and "mips64",
>>> i.e., 32/ and 64/ under the 'mips' base profile folder.  32/ contains
>>> everything needed to run pure o32-only under either a mips or mips64 kernel.
>>> I.e., mips-unknown-linux-gnu CHOST.  So on my Octane, /etc/portage/make.profile
>>> symlinks to /usr/portage/default/linux/mips/32/<VERSION>/
>>
>> That's one approach, but I'm trying to find the least intrusive. I do have the
>> problem fixed with the following:
>>
>>    [1]   default/linux/mips/13.0/o32
>>    [2]   default/linux/mips/13.0/n32
>>    [3]   default/linux/mips/13.0/n64
>>    [4]   default/linux/mips/13.0/multilib/o32
>>    [5]   default/linux/mips/13.0/multilib/n32
>>    [6]   default/linux/mips/13.0/multilib/n64
>>    [7]   default/linux/mips/13.0/mipsel/o32
>>    [8]   default/linux/mips/13.0/mipsel/n32
>>    [9]   default/linux/mips/13.0/mipsel/n64
>>    [10]  default/linux/mips/13.0/mipsel/multilib/o32
>>    [11]  default/linux/mips/13.0/mipsel/multilib/n32
>>    [12]  default/linux/mips/13.0/mipsel/multilib/n64
>>
>> and would like to push this through as a fix for now.  1 and 4 have
>> CHOST=mips-unknown-linux-gnu and mipsel-unknown-linux-gnu respectively.  2,3
>> and 8,9 have mips64-unknown-linux-gnu and mips64el-unknown-linux-gnu
>> respectively.  Only users of profiles 1 and 7 need to change their sym link one
>> level down to ../o32.  That was necessary to move the o32 stuff out of the way
>> of profiles/arch/mips.
>
> This sounds reasonable.  I've got the Octane tasked again on building glibc to
> hunt down the R10000 bug I've been stuck on for the past three months, but I
> can sync later and test this under your hardened overlay...maybe.  I just tried
> overlays on Gentoo/FreeBSD for nigoro's 9.3 profiles and gave up in utter
> frustration.
>
> I still think these need to be under a separate versioned directory structure,
> though.  14.0/ or 15.0/ or testing/ or pikachu/, whatever.  Then we mark 13.0
> as deprecated and allow time for users to migrate off before purging them out
> of the tree in a month or two.  It makes for a messy commit (thanks, CVS), but
> it avoids breaking people's systems out there who may not be on this mailing list.
>
> Also, we should get a news item added as well, specific to MIPS systems only.
>
>
>>>
>>> 64/ is where the fun is at.  I think the default ABI there should be n32 at the
>>> base, with a subprofile for multilib that itself contains subprofiles to handle
>>> the combinations of ABIs desired.  Still have to workout what CHOST you use for
>>> the base (GNU standard for n32 is mips64-unknown-linux-gnueabin32).  multilib
>>> would just use mips64-unknown-linux-gnu, and -mabi would be passed to gcc to
>>> pick the ABI to build for.
>>>
>>> Also, if we are going to do any kind of reorganizing of the profiles, let's not
>>> do it under 13.0.  Personally, I'd like to get away from datestamps as profile
>>> versions (13.0 refers to 2013) and either pick a fixed version number that we
>>> increment or come up with some other kind of versioning scheme.  Or just do
>>> away with it altogether and have something like a "stable" profile where things
>>> work and an "unstable" branch that only exists when we're doing insane changes
>>> to the profiles, which after testing, becomes "stable" (and current "stable"
>>> becomes "oldstable" or such).
>>>
>>> If we have to stick with datestamped versions, then use 15.0.  Cause it'll be
>>> 2015 by the time this goes active.
>>>
>>> rough draft of the top-level layout.  Doesn't care about chosen libc, ISA, or
>>> <VERSION>, but does reflect endianness.
>>>
>>> default/linux/mips
>>>      |
>>>      |-32/
>>>      |  |-be/
>>>      |  |-le/
>>>      |
>>>      |-64/
>>>      |  |-be/
>>>      |  |-le/
>>>      |  |
>>>      |  |-multilib/
>>>      |  |  |-be/
>>>      |  |  |-le/
>>>      |  |
>>>      |
>>>
>>
>> I like this, but the problem is disentangling the situation under arch/mips,
>> not under default/linux/mips.  Again, the inheritance problem stems from
>> everything under profiles/arch/mips inheriting from profiles/arch/mips.  So
>> even if you do implement the above, you still have to address the issue of how
>> the arch/mips stuff is stacking.
>
> I'll poke around arch/mips later and see if some of this can't be extruded
> there.  I don't know if arch/mips is for us exclusively, or if we share it with
> a few other project teams (other than -embedded -- do we have an OpenBSD MIPS
> team or such??).
>
>

I don't think anyone really "owns" any of the profiles exclusively 
because arch teams often have to mask things in other profiles.  We do 
own the default/linux/mips more than arch/mips because there is less 
reason for arch testers to modify the former compared to the latter [1]. 
However, the profile are fragile and I've had to ask people to not do 
stuff in hardened for example because of the cooky stacking problem.

I'll send out an email to all gentoo-dev@ about it, and prepare a news 
item for only mips users.  I do want to move on this because its 
blocking the upgrade of my lemotes yeeloong images.

One final point, there's also a set of hardened mips profiles which I 
haven't listed under profiles.desc.  I'm testing them too and they seem 
to be doing fine with the above structure.

Note:
[1] An example here is a package which builds on amd64 and mips, but 
only has some optimization in amd64 assembly which you can enable with a 
USE flag.  That flag needs to be masked on mips because there is no 
equivalent mips assembly.  I've seen this before.

-- 
Anthony G. Basile, Ph. D.
Chair of Information Technology
D'Youville College
Buffalo, NY 14201
(716) 829-8197