[gentoo-dev] How to structure our RISC-V support

[gentoo-dev] How to structure our RISC-V support

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 1789 bytes --]

Howdy. 

I'm sending this not only to the team members on the alias, but also 
to the whole dev list for discussion. 

So far I've been trying to support in Gentoo the full risc-v multilib 
directory structure and the ABI sets supported by glibc. According to 
specs this means for riscv64 

-mabi=rv64gc -march=lp64d 
  libdir = lib64/lp64d
  ("hardfloat")

-mabi=rv64imac -march=lp64
  libdir = lib64/lp64
  ("softfloat")

and theoretically similar for riscv32 (which just landed in glibc and 
is still broken in qemu-user).

However, this leads to several levels of pain (and I definitely dont 
have time to deal with it myself):

a) In many places the two-level libdir (e.g., /usr/lib64/lp64d) leads 
to difficulties in build systems. Right now Qt5 and CMake are still 
somewhat broken (in *Gentoo*).

b) Rust only supports rv64gc/lp64d. (Which is arguably what you should 
have for decent Linux support.)

I'm pretty sure these are not the only things, but they are somewhat 
symptomatic.

So, I would like to bring two proposals up for discussion.

1) We stop caring about anything except rv64gc/lp64d.
People can still bootstrap other stuff with crossdev etc, but the 
Gentoo tree and the riscv keyword reflect that things work with above 
-mabi and -march settings.

2) We drop the multilib paths and use "normal" lib64, with additional 
"safety symlinks" (/usr)/lib64/lp64d -> .
This is what SuSE and (I think) Fedora already does. The symlink 
should be there since "lib64" is NOT an official fallback coded into 
gcc/glibc/binutils; the only fallback present is "lib" ...

Opinions?

Cheers,
Andreas



-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] How to structure our RISC-V support

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 416 bytes --]

> 
> -mabi=rv64gc -march=lp64d
should be -march=rv64gc -mabi=lp64d

>   libdir = lib64/lp64d
>   ("hardfloat")
> 
> -mabi=rv64imac -march=lp64
should be -march=rv64imac -mabi=lp64

>   libdir = lib64/lp64
>   ("softfloat")
> 

(but that doesnt change the rest of the argument)

-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

[gentoo-dev] Re: How to structure our RISC-V support

[Palmer Dabbelt]

On Thu, 06 May 2021 13:01:23 PDT (-0700), dilfridge@gentoo.org wrote:
> Howdy. 
>
> I'm sending this not only to the team members on the alias, but also 
> to the whole dev list for discussion. 
>
> So far I've been trying to support in Gentoo the full risc-v multilib 
> directory structure and the ABI sets supported by glibc. According to 
> specs this means for riscv64 
>
> -mabi=rv64gc -march=lp64d 
>   libdir = lib64/lp64d
>   ("hardfloat")
>
> -mabi=rv64imac -march=lp64
>   libdir = lib64/lp64
>   ("softfloat")
>
> and theoretically similar for riscv32 (which just landed in glibc and 
> is still broken in qemu-user).
>
> However, this leads to several levels of pain (and I definitely dont 
> have time to deal with it myself):
>
> a) In many places the two-level libdir (e.g., /usr/lib64/lp64d) leads 
> to difficulties in build systems. Right now Qt5 and CMake are still 
> somewhat broken (in *Gentoo*).
>
> b) Rust only supports rv64gc/lp64d. (Which is arguably what you should 
> have for decent Linux support.)
>
> I'm pretty sure these are not the only things, but they are somewhat 
> symptomatic.
>
> So, I would like to bring two proposals up for discussion.
>
> 1) We stop caring about anything except rv64gc/lp64d.
> People can still bootstrap other stuff with crossdev etc, but the 
> Gentoo tree and the riscv keyword reflect that things work with above 
> -mabi and -march settings.

IMO dropping multilib is fine from a "working on real hardware" 
perspective, as that's all anyone is going to have for the forseeable 
future -- at least for the sort of hardware you'd want to run a 
desktop/server style Linux distro on (something like buildroot might be 
interested in other ISA/ABI combos).

That said, it seems like a lot of the distros are punting on multilib 
because of this path issue.  If that's really the worry then I wouldn't 
be opposed to just fixing it in GCC/glibc.  Ideally we'd be able to just 
add paths so we don't break the ABI, but if the ABI we came up with 
really hasn't been used by anyone then I wouldn't be opposed to breaking 
it if it means we can actually get multilib up and running -- we'd have 
to talk with a lot of people for that, though.

In the long run I think we will want multilib to work, as we're likely 
to end up with some ISA extensions that are useful enough to want to 
build general code for but not so useful that everyone has them -- B and 
V, for example.  We're a long way away from having good code gen for V 
(at least for arbitrary code, some special libraries might show up 
sooner), but B is a bit closer and might be amenable to having a system 
built with it.  That may be a bit less interesting for Gentoo, where 
we're building things locally (and there shouldn't be any -bin packages 
on RISC-V yet, we're not getting ports of proprietary stuff yet), but if 
we can be a forcing function to put together a multlib ABI that actually 
works then that's a win for everyone.

> 2) We drop the multilib paths and use "normal" lib64, with additional 
> "safety symlinks" (/usr)/lib64/lp64d -> .
> This is what SuSE and (I think) Fedora already does. The symlink 
> should be there since "lib64" is NOT an official fallback coded into 
> gcc/glibc/binutils; the only fallback present is "lib" ...

IMO we can just fix that one upstream.  If all the distros are going to 
do it anyway then it'd be better to just fix it, there's not a lot of 
sense in forcing everyone to patch these things.  IIUC that one would be 
an ABI-compatible addition, but again we'd need to talk about it in 
GCC/glibc land (and with some other distro folks).

Sorry this is such a mess!

Re: [gentoo-dev] How to structure our RISC-V support

[Michał Górny]

On Thu, 2021-05-06 at 22:01 +0200, Andreas K. Huettel wrote:
> Howdy. 
> 
> I'm sending this not only to the team members on the alias, but also 
> to the whole dev list for discussion. 
> 
> So far I've been trying to support in Gentoo the full risc-v multilib 
> directory structure and the ABI sets supported by glibc. According to 
> specs this means for riscv64 
> 
> -mabi=rv64gc -march=lp64d 
>   libdir = lib64/lp64d
>   ("hardfloat")
> 
> -mabi=rv64imac -march=lp64
>   libdir = lib64/lp64
>   ("softfloat")
> 
> and theoretically similar for riscv32 (which just landed in glibc and 
> is still broken in qemu-user).
> 
> However, this leads to several levels of pain (and I definitely dont 
> have time to deal with it myself):
> 
> a) In many places the two-level libdir (e.g., /usr/lib64/lp64d) leads 
> to difficulties in build systems. Right now Qt5 and CMake are still 
> somewhat broken (in *Gentoo*).
> 
> b) Rust only supports rv64gc/lp64d. (Which is arguably what you should 
> have for decent Linux support.)
> 
> I'm pretty sure these are not the only things, but they are somewhat 
> symptomatic.
> 
> So, I would like to bring two proposals up for discussion.
> 
> 1) We stop caring about anything except rv64gc/lp64d.
> People can still bootstrap other stuff with crossdev etc, but the 
> Gentoo tree and the riscv keyword reflect that things work with above 
> -mabi and -march settings.
> 
> 2) We drop the multilib paths and use "normal" lib64, with additional 
> "safety symlinks" (/usr)/lib64/lp64d -> .
> This is what SuSE and (I think) Fedora already does. The symlink 
> should be there since "lib64" is NOT an official fallback coded into 
> gcc/glibc/binutils; the only fallback present is "lib" ...
> 
> Opinions?
> 

Haven't I told you using two-level libdirs is stupid?  So yes, please do
that and let us be happy once again.

That said, where does lp64gc land?  Or isnon-multilib one-or-the-other
the goal?

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] How to structure our RISC-V support

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 810 bytes --]

> 
> Haven't I told you using two-level libdirs is stupid?  So yes,
> please do that and let us be happy once again.
> 
> That said, where does lp64gc land?  Or isnon-multilib
> one-or-the-other the goal?

It would be non-multilib one-or-the-other then for us.
The main relevant combination is rv64gc/lp64d, which is arguably what 
a linux machine "should have".

(I could also imagine to keep rv64imac/lp64 profile and stages (also 
using lib64), these would have to mask stuff like rust then though.)

(Unless Palmer et al come up with a fix for the libdirs on the 
upstream side of things. Already e.g. libdir=lib64-lp64d would be much 
easier to handle I suspect.)

-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] How to structure our RISC-V support

[Palmer Dabbelt]

On Thu, 06 May 2021 13:30:45 PDT (-0700), dilfridge@gentoo.org wrote:
>> 
>> Haven't I told you using two-level libdirs is stupid?  So yes,
>> please do that and let us be happy once again.
>> 
>> That said, where does lp64gc land?  Or isnon-multilib
>> one-or-the-other the goal?
>
> It would be non-multilib one-or-the-other then for us.
> The main relevant combination is rv64gc/lp64d, which is arguably what 
> a linux machine "should have".
>
> (I could also imagine to keep rv64imac/lp64 profile and stages (also 
> using lib64), these would have to mask stuff like rust then though.)
>
> (Unless Palmer et al come up with a fix for the libdirs on the 
> upstream side of things. Already e.g. libdir=lib64-lp64d would be much 
> easier to handle I suspect.)

TBH: I'm not really going to come up with something better beacuse I 
came up with the current (and likely broken) scheme and I still don't 
fully understand why.  So if you have suggestions as to something that 
would actually work that would be great, as otherwise I'm just going to 
come up with something broken again ;)

Is the constraint just "no sub-directories for libraries"?  IIRC we did 
that because someone else was already doing it and it seems to be less 
of a FHS break that adding a bunch of first-level directories.  I can 
totally buy I was just wrong, though.  If that's all we need then this 
shouldn't be that hard to fix upstream, of course it'll take a while to 
get everyone updated but at least we'll have a way to steer everyone.

Re: [gentoo-dev] How to structure our RISC-V support

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 1058 bytes --]

Am Donnerstag, 6. Mai 2021, 22:34:52 CEST schrieb Palmer Dabbelt:
>
> TBH: I'm not really going to come up with something better beacuse I
> came up with the current (and likely broken) scheme and I still
> don't fully understand why.  So if you have suggestions as to
> something that would actually work that would be great, as
> otherwise I'm just going to come up with something broken again ;)
> 

Heh. No worries. I actually liked the idea, just until the moment when 
I remembered that we had been campaigning for months to replace all 
absolute symlinks in Gentoo with relative ones. And suddenly there is 
only on riscv not enough "../"  in the link.

> Is the constraint just "no sub-directories for libraries"?  IIRC we
> did that because someone else was already doing it and it seems to
> be less of a FHS break that adding a bunch of first-level
> directories.  

I need to read up on FHS. Time...


-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] How to structure our RISC-V support

[Yixun Lan]

On 22:30 Thu 06 May     , Andreas K. Huettel wrote:
> > 
> > Haven't I told you using two-level libdirs is stupid?  So yes,
> > please do that and let us be happy once again.
> > 
> > That said, where does lp64gc land?  Or isnon-multilib
> > one-or-the-other the goal?
> 
> It would be non-multilib one-or-the-other then for us.
> The main relevant combination is rv64gc/lp64d, which is arguably what 
> a linux machine "should have".
> 
> (I could also imagine to keep rv64imac/lp64 profile and stages (also 
> using lib64), these would have to mask stuff like rust then though.)
> 
I'm fine with rust masked in lp64/other profile..
but in my opinion: it's really up to upstream should fix/support it

> (Unless Palmer et al come up with a fix for the libdirs on the 
> upstream side of things. Already e.g. libdir=lib64-lp64d would be much 
> easier to handle I suspect.)

using one level path (eg. lib64-lp64d) won't fix the problem,
the root cause is that we use a 'non-standard' lib path (QT5, Cmake issue),
not matter it's one level or two level path, see bug here [1]

[1] https://bugs.gentoo.org/781134
https://gitlab.kitware.com/cmake/cmake/-/issues/22138

-- 
Yixun Lan (dlan)
Gentoo Linux Developer
GPG Key ID AABEFD55

Re: [gentoo-dev] How to structure our RISC-V support

[Michał Górny]

On Fri, 2021-05-07 at 14:15 +0800, Yixun Lan wrote:
> On 22:30 Thu 06 May     , Andreas K. Huettel wrote:
> > > 
> > > Haven't I told you using two-level libdirs is stupid?  So yes,
> > > please do that and let us be happy once again.
> > > 
> > > That said, where does lp64gc land?  Or isnon-multilib
> > > one-or-the-other the goal?
> > 
> > It would be non-multilib one-or-the-other then for us.
> > The main relevant combination is rv64gc/lp64d, which is arguably what 
> > a linux machine "should have".
> > 
> > (I could also imagine to keep rv64imac/lp64 profile and stages (also 
> > using lib64), these would have to mask stuff like rust then though.)
> > 
> I'm fine with rust masked in lp64/other profile..
> but in my opinion: it's really up to upstream should fix/support it
> 
> > (Unless Palmer et al come up with a fix for the libdirs on the 
> > upstream side of things. Already e.g. libdir=lib64-lp64d would be much 
> > easier to handle I suspect.)
> 
> using one level path (eg. lib64-lp64d) won't fix the problem,
> the root cause is that we use a 'non-standard' lib path (QT5, Cmake issue),
> not matter it's one level or two level path, see bug here [1]
> 
> [1] https://bugs.gentoo.org/781134
> https://gitlab.kitware.com/cmake/cmake/-/issues/22138
> 

Maybe it doesn't matter for CMake but it does matter for us simpletons
who want '../' to work as its supposed to.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] How to structure our RISC-V support

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 809 bytes --]

> I'm fine with rust masked in lp64/other profile..
> but in my opinion: it's really up to upstream should fix/support it
> 
> > (Unless Palmer et al come up with a fix for the libdirs on the
> > upstream side of things. Already e.g. libdir=lib64-lp64d would be
> > much easier to handle I suspect.)
> 
> using one level path (eg. lib64-lp64d) won't fix the problem,
> the root cause is that we use a 'non-standard' lib path (QT5, Cmake
> issue), not matter it's one level or two level path, see bug here

I suspect this may need some more analysis.

But, if we transition in non-multilib cases to normal lib64, then we 
have all the time in the world to figure it out.

-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] How to structure our RISC-V support

[Yixun Lan]

On 22:01 Thu 06 May     , Andreas K. Huettel wrote:
> Howdy. 
> 
> I'm sending this not only to the team members on the alias, but also 
> to the whole dev list for discussion. 
> 
> So far I've been trying to support in Gentoo the full risc-v multilib 
> directory structure and the ABI sets supported by glibc. According to 
> specs this means for riscv64 
> 
> -mabi=rv64gc -march=lp64d 
>   libdir = lib64/lp64d
>   ("hardfloat")
> 
> -mabi=rv64imac -march=lp64
>   libdir = lib64/lp64
>   ("softfloat")
> 
> and theoretically similar for riscv32 (which just landed in glibc and 
> is still broken in qemu-user).
> 
> However, this leads to several levels of pain (and I definitely dont 
> have time to deal with it myself):
> 
> a) In many places the two-level libdir (e.g., /usr/lib64/lp64d) leads 
> to difficulties in build systems. Right now Qt5 and CMake are still 
> somewhat broken (in *Gentoo*).
> 
> b) Rust only supports rv64gc/lp64d. (Which is arguably what you should 
> have for decent Linux support.)
> 
> I'm pretty sure these are not the only things, but they are somewhat 
> symptomatic.
> 
> So, I would like to bring two proposals up for discussion.
> 
> 1) We stop caring about anything except rv64gc/lp64d.
> People can still bootstrap other stuff with crossdev etc, but the 
> Gentoo tree and the riscv keyword reflect that things work with above 
> -mabi and -march settings.
> 
fine by me, for current software/upstream state, it's probably the most practical way 
to only support lp64d, this will significantly ease our life ..
besides, it's relatively easy if people want to support more (lp64/lp32..) later

> 2) We drop the multilib paths and use "normal" lib64, with additional 
> "safety symlinks" (/usr)/lib64/lp64d -> .
> This is what SuSE and (I think) Fedora already does. The symlink 
> should be there since "lib64" is NOT an official fallback coded into 
> gcc/glibc/binutils; the only fallback present is "lib" ...
> 
can we use different scheme for non-multilib vs multilib?
1) non-multilibe: just use "normal" lib64, keep align with other ARCHs (amd64)?
2) multilib: just stick to current two level lib path

> -- 
> Andreas K. Hüttel
> dilfridge@gentoo.org
> Gentoo Linux developer
> (council, toolchain, base-system, perl, libreoffice)



-- 
Yixun Lan (dlan)
Gentoo Linux Developer
GPG Key ID AABEFD55

Re: [gentoo-dev] How to structure our RISC-V support

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 1725 bytes --]

> > 1) We stop caring about anything except rv64gc/lp64d.
> > People can still bootstrap other stuff with crossdev etc, but the
> > Gentoo tree and the riscv keyword reflect that things work with
> > above -mabi and -march settings.
> 
> fine by me, for current software/upstream state, it's probably the
> most practical way to only support lp64d, this will significantly
> ease our life .. besides, it's relatively easy if people want to
> support more (lp64/lp32..) later
>

++

> > 2) We drop the multilib paths and use "normal" lib64, with
> > additional "safety symlinks" (/usr)/lib64/lp64d -> .
> > This is what SuSE and (I think) Fedora already does. The symlink
> > should be there since "lib64" is NOT an official fallback coded
> > into gcc/glibc/binutils; the only fallback present is "lib" ...
> can we use different scheme for non-multilib vs multilib?
> 1) non-multilibe: just use "normal" lib64, keep align with other
> ARCHs (amd64)? 

++

> 2) multilib: just stick to current two level lib path

We can try that but it doesn't solve any of our problems (and we'd 
have to keep carrying the two-level dir patches).

(I've already decided some time ago that supporting real multilib 
stages is too much effort for insufficient gain and stopped publishing 
them. Until recently I still built them; since glibc-2.33 they broke 
and while I know how to fix things I haven't had time to do it yet.)

So if we keep the multilib scheme around, then IMHO only as internal 
workaround until we/upstream/... have figured out a better directory 
scheme.



-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 981 bytes --]

Re: [gentoo-dev] How to structure our RISC-V support -- Summary

[Andreas K. Huettel]

[-- Attachment #1: Type: text/plain, Size: 2036 bytes --]

So, here's what I took away from the thread. Please shout if you 
disagree.

1) Advertised riscv profiles will all be non-multilib and use /usr/
lib64 (or /usr/lib if we ever get around to riscv32). [A]

2) The standard for keywording and stabilization is rv64gc/lp64d. 
We keep stages for other variants [B] around if feasible, but on these 
important packages may be masked and unavailable [C].

3) We try to internally keep the multilib variant with the two-stage 
path going for now, as very low-priority thing. [D]

4) Medium term we discuss with the RISC-V, glibc, gcc people how 
multilib could be simplified, and then switch the multilib settings 
once this comes to a conclusion.

If there are no protests I'll start planning the path migration for 
1). (Maybe making a riscv-specific new profile version is best.)

Cheers,
Andreas


[A] Note that the actual specs use /usr/lib32/...
[B] Other ABI (lp64) or other ISA (riscv32...)
[C] See rust etc.
[D] Low priority means, it pretty much won't build every now and then, 
as, e.g., right now [E].
[E] Our monkeypatched glibc-2.32 rv32 support was experimental enough 
so it didnt survive the transition to official upstream glibc-2.33 
support, so the multilib stages will have to be re-bootstrapped.

> So, I would like to bring two proposals up for discussion.
> 
> 1) We stop caring about anything except rv64gc/lp64d.
> People can still bootstrap other stuff with crossdev etc, but the
> Gentoo tree and the riscv keyword reflect that things work with
> above -mabi and -march settings.
> 
> 2) We drop the multilib paths and use "normal" lib64, with
> additional "safety symlinks" (/usr)/lib64/lp64d -> .
> This is what SuSE and (I think) Fedora already does. The symlink
> should be there since "lib64" is NOT an official fallback coded into
> gcc/glibc/binutils; the only fallback present is "lib" ...

-- 
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, libreoffice)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 981 bytes --]