[gentoo-dev] [RFC] New USE_EXPAND: LLVM_TARGETS

[gentoo-dev] [RFC] New USE_EXPAND: LLVM_TARGETS

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1383 bytes --]

Hi,

I'd like to introduce a new USE_EXPAND for LLVM & clang. It'd be named
LLVM_TARGETS, and it's going to replace the current solution based on
USE=multitarget & VIDEO_CARDS=radeon.

In the old system, the following rules applied:

- host (implicitly figured out by LLVM) and BPF targets were always
  built,

- VIDEO_CARDS=radeon enabled additional R600 target,

- USE=multitarget enabled all targets.

In the new system, LLVM_TARGETS explicitly controls *all* targets
built. To avoid dependency hell, the host target is package.use.forced
in specific arch profiles. Additionally, the BPF target is on by
default.

The new system will be applied to 3.9.0 and 9999 ebuilds. VIDEO_CARDS
flag will be removed completely because of no revdeps. USE=multitarget
will be kept for compatibility until all revdeps are clean.

As far as revdeps are concerned:

- Packages that require only host target will not have to be changed at
  all (since host is still forced-on).

- Packages that used to require multitarget will eventually have to be
  changed to request the specific targets they need. They could switch
  to this when they stop supporting LLVM < 3.9.

The draft implementation is available in the following PR:
https://github.com/gentoo/gentoo/pull/2405

Any comments?

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 931 bytes --]

Re: [gentoo-dev] [RFC] New USE_EXPAND: LLVM_TARGETS

[Mart Raudsepp]

Ühel kenal päeval, P, 25.09.2016 kell 23:08, kirjutas Michał Górny:
> I'd like to introduce a new USE_EXPAND for LLVM & clang. It'd be
> named
> LLVM_TARGETS, and it's going to replace the current solution based on
> USE=multitarget & VIDEO_CARDS=radeon.
> 
> - VIDEO_CARDS=radeon enabled additional R600 target,

No. It enables AMDGPU target these days, which is for the modern stuff
and very much needed by them.
r600 stuff was in the llvm 3.3-3.6 era, which was used by old
experimental mesa[r600-llvm-compiler] as an alternative shader compiler
for r600 instead of builtin mesa stuff. This work has been ditched long
ago afaik.
Instead now VIDEO_CARDS=radeon is required on llvm for radeonsi and
later AMD GPUs for _ANY_ shader compiler support at all, plus other
things (from it adding AMDGPU to llvm targets in current ebuild).

> The new system will be applied to 3.9.0 and 9999 ebuilds. VIDEO_CARDS
> flag will be removed completely because of no revdeps.

People with radeonsi graphics set VIDEO_CARDS=radeon already, I'm a bit
reserved about having to force them to set some LLVM_TARGETS=radeon or
LLVM_TARGETS=amdgpu on top of that to satisfy some USE depends on
mesa[video_cards_radeon].

Re: [gentoo-dev] [RFC] New USE_EXPAND: LLVM_TARGETS

[Mart Raudsepp]

Ühel kenal päeval, E, 26.09.2016 kell 00:42, kirjutas Mart Raudsepp:
> > The new system will be applied to 3.9.0 and 9999 ebuilds.
> > VIDEO_CARDS
> > flag will be removed completely because of no revdeps.
> 
> People with radeonsi graphics set VIDEO_CARDS=radeon already, I'm a
> bit
> reserved about having to force them to set some LLVM_TARGETS=radeon
> or
> LLVM_TARGETS=amdgpu on top of that to satisfy some USE depends on
> mesa[video_cards_radeon].
> 

I meant they set VIDEO_CARDS="radeon radeonsi" already. Got that wrong
from being an actual r600 mesa/evergreen user still with "radeon r600"
set instead :(

Re: [gentoo-dev] [RFC] New USE_EXPAND: LLVM_TARGETS

[Matthias Maier]

[-- Attachment #1: Type: text/plain, Size: 1008 bytes --]


On Sun, Sep 25, 2016, at 16:08 CDT, Michał Górny <mgorny@gentoo.org> wrote:

> Hi,
>
> I'd like to introduce a new USE_EXPAND for LLVM & clang. It'd be named
> LLVM_TARGETS, and it's going to replace the current solution based on
> USE=multitarget & VIDEO_CARDS=radeon.
>
> In the old system, the following rules applied:
>
> - host (implicitly figured out by LLVM) and BPF targets were always
>   built,
>
> - VIDEO_CARDS=radeon enabled additional R600 target,
>
> - USE=multitarget enabled all targets.
>
> In the new system, LLVM_TARGETS explicitly controls *all* targets
> built. To avoid dependency hell, the host target is package.use.forced
> in specific arch profiles. Additionally, the BPF target is on by
> default.

+1

We also avoid a very subtle abuse of the VIDEO_CARDS use expand with
it. So far, I had to explain a surprising amount of times that
VIDEO_CARDS=radeon for llvm does *not* mean that llvm magically uses
the GPU for compiling.

Best,
Matthias


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 820 bytes --]

Re: [gentoo-dev] [RFC] New USE_EXPAND: LLVM_TARGETS

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1823 bytes --]

On Mon, 26 Sep 2016 00:42:11 +0300
Mart Raudsepp <leio@gentoo.org> wrote:

> Ühel kenal päeval, P, 25.09.2016 kell 23:08, kirjutas Michał Górny:
> > I'd like to introduce a new USE_EXPAND for LLVM & clang. It'd be
> > named
> > LLVM_TARGETS, and it's going to replace the current solution based on
> > USE=multitarget & VIDEO_CARDS=radeon.
> > 
> > - VIDEO_CARDS=radeon enabled additional R600 target,  
> 
> No. It enables AMDGPU target these days, which is for the modern stuff
> and very much needed by them.
> r600 stuff was in the llvm 3.3-3.6 era, which was used by old
> experimental mesa[r600-llvm-compiler] as an alternative shader compiler
> for r600 instead of builtin mesa stuff. This work has been ditched long
> ago afaik.
> Instead now VIDEO_CARDS=radeon is required on llvm for radeonsi and
> later AMD GPUs for _ANY_ shader compiler support at all, plus other
> things (from it adding AMDGPU to llvm targets in current ebuild).

Yes, yes, I am old :-P. You are right, it's AMDGPU these days.

> > The new system will be applied to 3.9.0 and 9999 ebuilds. VIDEO_CARDS
> > flag will be removed completely because of no revdeps.  
> 
> People with radeonsi graphics set VIDEO_CARDS=radeon already, I'm a bit
> reserved about having to force them to set some LLVM_TARGETS=radeon or
> LLVM_TARGETS=amdgpu on top of that to satisfy some USE depends on
> mesa[video_cards_radeon].

How about nvidia users who seem to require NVPTX for libclc these days?
Do they set VIDEO_CARDS='nvidia nv nouveau ...'? The problem is that
this abuse of VIDEO_CARDS is never going to be 100% clear to users.

I guess we can enable GPU targets in desktop profiles by default to
save most of our users from the issues.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 931 bytes --]

Re: [gentoo-dev] [RFC] New USE_EXPAND: LLVM_TARGETS

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 2108 bytes --]

On Mon, 26 Sep 2016 08:18:27 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> On Mon, 26 Sep 2016 00:42:11 +0300
> Mart Raudsepp <leio@gentoo.org> wrote:
> 
> > Ühel kenal päeval, P, 25.09.2016 kell 23:08, kirjutas Michał Górny:  
> > > I'd like to introduce a new USE_EXPAND for LLVM & clang. It'd be
> > > named
> > > LLVM_TARGETS, and it's going to replace the current solution based on
> > > USE=multitarget & VIDEO_CARDS=radeon.
> > > 
> > > - VIDEO_CARDS=radeon enabled additional R600 target,    
> > 
> > No. It enables AMDGPU target these days, which is for the modern stuff
> > and very much needed by them.
> > r600 stuff was in the llvm 3.3-3.6 era, which was used by old
> > experimental mesa[r600-llvm-compiler] as an alternative shader compiler
> > for r600 instead of builtin mesa stuff. This work has been ditched long
> > ago afaik.
> > Instead now VIDEO_CARDS=radeon is required on llvm for radeonsi and
> > later AMD GPUs for _ANY_ shader compiler support at all, plus other
> > things (from it adding AMDGPU to llvm targets in current ebuild).  
> 
> Yes, yes, I am old :-P. You are right, it's AMDGPU these days.
> 
> > > The new system will be applied to 3.9.0 and 9999 ebuilds. VIDEO_CARDS
> > > flag will be removed completely because of no revdeps.    
> > 
> > People with radeonsi graphics set VIDEO_CARDS=radeon already, I'm a bit
> > reserved about having to force them to set some LLVM_TARGETS=radeon or
> > LLVM_TARGETS=amdgpu on top of that to satisfy some USE depends on
> > mesa[video_cards_radeon].  
> 
> How about nvidia users who seem to require NVPTX for libclc these days?
> Do they set VIDEO_CARDS='nvidia nv nouveau ...'? The problem is that
> this abuse of VIDEO_CARDS is never going to be 100% clear to users.
> 
> I guess we can enable GPU targets in desktop profiles by default to
> save most of our users from the issues.

Hmm, I actually see we're enabling them in arch profiles. So I guess
matching enable there would fit there as well.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 931 bytes --]

Re: [gentoo-dev] [RFC] New USE_EXPAND: LLVM_TARGETS

[Raymond Jennings]

[-- Attachment #1: Type: text/plain, Size: 2432 bytes --]

On Sun, Sep 25, 2016 at 11:29 PM, Michał Górny <mgorny@gentoo.org> wrote:

> On Mon, 26 Sep 2016 08:18:27 +0200
> Michał Górny <mgorny@gentoo.org> wrote:
>
> > On Mon, 26 Sep 2016 00:42:11 +0300
> > Mart Raudsepp <leio@gentoo.org> wrote:
> >
> > > Ühel kenal päeval, P, 25.09.2016 kell 23:08, kirjutas Michał Górny:
> > > > I'd like to introduce a new USE_EXPAND for LLVM & clang. It'd be
> > > > named
> > > > LLVM_TARGETS, and it's going to replace the current solution based on
> > > > USE=multitarget & VIDEO_CARDS=radeon.
> > > >
> > > > - VIDEO_CARDS=radeon enabled additional R600 target,
> > >
> > > No. It enables AMDGPU target these days, which is for the modern stuff
> > > and very much needed by them.
> > > r600 stuff was in the llvm 3.3-3.6 era, which was used by old
> > > experimental mesa[r600-llvm-compiler] as an alternative shader compiler
> > > for r600 instead of builtin mesa stuff. This work has been ditched long
> > > ago afaik.
> > > Instead now VIDEO_CARDS=radeon is required on llvm for radeonsi and
> > > later AMD GPUs for _ANY_ shader compiler support at all, plus other
> > > things (from it adding AMDGPU to llvm targets in current ebuild).
> >
> > Yes, yes, I am old :-P. You are right, it's AMDGPU these days.
> >
> > > > The new system will be applied to 3.9.0 and 9999 ebuilds. VIDEO_CARDS
> > > > flag will be removed completely because of no revdeps.
> > >
> > > People with radeonsi graphics set VIDEO_CARDS=radeon already, I'm a bit
> > > reserved about having to force them to set some LLVM_TARGETS=radeon or
> > > LLVM_TARGETS=amdgpu on top of that to satisfy some USE depends on
> > > mesa[video_cards_radeon].
> >
> > How about nvidia users who seem to require NVPTX for libclc these days?
> > Do they set VIDEO_CARDS='nvidia nv nouveau ...'? The problem is that
> > this abuse of VIDEO_CARDS is never going to be 100% clear to users.
> >
> > I guess we can enable GPU targets in desktop profiles by default to
> > save most of our users from the issues.
>
> Hmm, I actually see we're enabling them in arch profiles. So I guess
> matching enable there would fit there as well.
>
> --
> Best regards,
> Michał Górny
> <http://dev.gentoo.org/~mgorny/>
>

This might be a stupid question, but I have a hunch I'm not the only user
curious about this.

Doesn't VIDEO_CARDS factor in on xorg-server's video driver selection?

[-- Attachment #2: Type: text/html, Size: 3405 bytes --]

Re: [gentoo-dev] [RFC] New USE_EXPAND: LLVM_TARGETS

[Andy Mender]

[-- Attachment #1: Type: text/plain, Size: 2772 bytes --]

It does. In effect it determines which xorg-drivers are to be merged. This,
and some other packages with options for specific graphic cards/adapters.

Best regards,
Andy Mender.

On 27 Sep 2016 22:52, "Raymond Jennings" <shentino@gmail.com> wrote:

> On Sun, Sep 25, 2016 at 11:29 PM, Michał Górny <mgorny@gentoo.org> wrote:
>
>> On Mon, 26 Sep 2016 08:18:27 +0200
>> Michał Górny <mgorny@gentoo.org> wrote:
>>
>> > On Mon, 26 Sep 2016 00:42:11 +0300
>> > Mart Raudsepp <leio@gentoo.org> wrote:
>> >
>> > > Ühel kenal päeval, P, 25.09.2016 kell 23:08, kirjutas Michał Górny:
>> > > > I'd like to introduce a new USE_EXPAND for LLVM & clang. It'd be
>> > > > named
>> > > > LLVM_TARGETS, and it's going to replace the current solution based
>> on
>> > > > USE=multitarget & VIDEO_CARDS=radeon.
>> > > >
>> > > > - VIDEO_CARDS=radeon enabled additional R600 target,
>> > >
>> > > No. It enables AMDGPU target these days, which is for the modern stuff
>> > > and very much needed by them.
>> > > r600 stuff was in the llvm 3.3-3.6 era, which was used by old
>> > > experimental mesa[r600-llvm-compiler] as an alternative shader
>> compiler
>> > > for r600 instead of builtin mesa stuff. This work has been ditched
>> long
>> > > ago afaik.
>> > > Instead now VIDEO_CARDS=radeon is required on llvm for radeonsi and
>> > > later AMD GPUs for _ANY_ shader compiler support at all, plus other
>> > > things (from it adding AMDGPU to llvm targets in current ebuild).
>> >
>> > Yes, yes, I am old :-P. You are right, it's AMDGPU these days.
>> >
>> > > > The new system will be applied to 3.9.0 and 9999 ebuilds.
>> VIDEO_CARDS
>> > > > flag will be removed completely because of no revdeps.
>> > >
>> > > People with radeonsi graphics set VIDEO_CARDS=radeon already, I'm a
>> bit
>> > > reserved about having to force them to set some LLVM_TARGETS=radeon or
>> > > LLVM_TARGETS=amdgpu on top of that to satisfy some USE depends on
>> > > mesa[video_cards_radeon].
>> >
>> > How about nvidia users who seem to require NVPTX for libclc these days?
>> > Do they set VIDEO_CARDS='nvidia nv nouveau ...'? The problem is that
>> > this abuse of VIDEO_CARDS is never going to be 100% clear to users.
>> >
>> > I guess we can enable GPU targets in desktop profiles by default to
>> > save most of our users from the issues.
>>
>> Hmm, I actually see we're enabling them in arch profiles. So I guess
>> matching enable there would fit there as well.
>>
>> --
>> Best regards,
>> Michał Górny
>> <http://dev.gentoo.org/~mgorny/>
>>
>
> This might be a stupid question, but I have a hunch I'm not the only user
> curious about this.
>
> Doesn't VIDEO_CARDS factor in on xorg-server's video driver selection?
>

[-- Attachment #2: Type: text/html, Size: 3933 bytes --]

Re: [gentoo-dev] [RFC] New USE_EXPAND: LLVM_TARGETS

[Marek Szuba]

[-- Attachment #1.1: Type: text/plain, Size: 358 bytes --]

On 2016-09-27 22:51, Raymond Jennings wrote:

> Doesn't VIDEO_CARDS factor in on xorg-server's video driver
> selection?
It does. Which is why with the way things are now, and which Michał's
proposal should improve, someone with an amdgpu-compatible card will
still have xf86-video-ati lying around - VIDEO_CARDS=radeon will pull it in.

-- 
MS


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-dev] [RFC] New USE_EXPAND: LLVM_TARGETS

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 738 bytes --]

On Wed, 28 Sep 2016 08:31:06 +0200
Marek Szuba <marecki@gentoo.org> wrote:

> On 2016-09-27 22:51, Raymond Jennings wrote:
> 
> > Doesn't VIDEO_CARDS factor in on xorg-server's video driver
> > selection?  
> It does. Which is why with the way things are now, and which Michał's
> proposal should improve, someone with an amdgpu-compatible card will
> still have xf86-video-ati lying around - VIDEO_CARDS=radeon will pull it in.

Also note there's VIDEO_CARDS=amdgpu for newer cards, to increase your
confusion. And the LLVM target is probably needed for
VIDEO_CARDS=radeonsi and newer, not by VIDEO_CARDS=radeon. Though it
may be needed for OpenCL.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 931 bytes --]

Re: [gentoo-dev] [RFC] New USE_EXPAND: LLVM_TARGETS

[Mart Raudsepp]

Ühel kenal päeval, K, 28.09.2016 kell 08:39, kirjutas Michał Górny:
> On Wed, 28 Sep 2016 08:31:06 +0200
> Marek Szuba <marecki@gentoo.org> wrote:
> 
> > 
> > On 2016-09-27 22:51, Raymond Jennings wrote:
> > 
> > > 
> > > Doesn't VIDEO_CARDS factor in on xorg-server's video driver
> > > selection?  
> > It does. Which is why with the way things are now, and which
> > Michał's
> > proposal should improve, someone with an amdgpu-compatible card
> > will
> > still have xf86-video-ati lying around - VIDEO_CARDS=radeon will
> > pull it in.
> 
> Also note there's VIDEO_CARDS=amdgpu for newer cards, to increase
> your
> confusion. And the LLVM target is probably needed for
> VIDEO_CARDS=radeonsi and newer, not by VIDEO_CARDS=radeon. Though it
> may be needed for OpenCL.

As discussed on IRC, I think the amdgpu target should be enabled via
video_cards_radeonsi, because the only consumer of that target that
pulls it in is currently mesa[video_cards_radeonsi], so users don't
need to go fiddling with yet another USE_EXPAND when they already set
VIDEO_CARDS="radeonsi" or similar anyways in their make.conf.

I don't quite understand the rest of the LLVM_TARGETS proposed. Seems
like all the rest of them would be use.masked and unmasked in specific
arch profiles?
It seems that once you remove AMDGPU from the equation by keeping it
behind VIDEO_CARDS, all the rest are CPU specific stuff, not GPU, so
less mixing things up in a way; though with completely unhelpful
llvm_targets.desc I don't know if any of the others might not be
something else (like BPF, Hexagon, Lanai, MSP430 and more)


And yes, the existing VIDEO_CARDS=radeon* stuff is rather confusing in
mixing up GL driver names, DRM backend names and more, but a separate
issue that I'm thinking on how to clean up and working with the rest of
the x11 team towards something. Meanwhile llvm[video_cards_radeonsi]
would make sure those that need it, already have it if they enable the
thing they need to get it from mesa globally, and won't need to fiddle
with other flags too.
And yes, it's currently llvm[video_cards_radeon] in old versions of
llvm, and that's not accurate anymore since a while imho, and should be
converted to video_cards_radeonsi as well.


Mart

Re: [gentoo-dev] [RFC] New USE_EXPAND: LLVM_TARGETS

[Michał Górny]

[-- Attachment #1: Type: text/plain, Size: 1556 bytes --]

On Sun, 25 Sep 2016 23:08:52 +0200
Michał Górny <mgorny@gentoo.org> wrote:

> Hi,
> 
> I'd like to introduce a new USE_EXPAND for LLVM & clang. It'd be named
> LLVM_TARGETS, and it's going to replace the current solution based on
> USE=multitarget & VIDEO_CARDS=radeon.
> 
> In the old system, the following rules applied:
> 
> - host (implicitly figured out by LLVM) and BPF targets were always
>   built,
> 
> - VIDEO_CARDS=radeon enabled additional R600 target,
> 
> - USE=multitarget enabled all targets.
> 
> In the new system, LLVM_TARGETS explicitly controls *all* targets
> built. To avoid dependency hell, the host target is package.use.forced
> in specific arch profiles. Additionally, the BPF target is on by
> default.
> 
> The new system will be applied to 3.9.0 and 9999 ebuilds. VIDEO_CARDS
> flag will be removed completely because of no revdeps. USE=multitarget
> will be kept for compatibility until all revdeps are clean.
> 
> As far as revdeps are concerned:
> 
> - Packages that require only host target will not have to be changed at
>   all (since host is still forced-on).
> 
> - Packages that used to require multitarget will eventually have to be
>   changed to request the specific targets they need. They could switch
>   to this when they stop supporting LLVM < 3.9.
> 
> The draft implementation is available in the following PR:
> https://github.com/gentoo/gentoo/pull/2405
> 
> Any comments?

Committed now.

-- 
Best regards,
Michał Górny
<http://dev.gentoo.org/~mgorny/>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 931 bytes --]