[gentoo-user] Issues with AMD_IOMMU

[gentoo-user] Issues with AMD_IOMMU

[Adam Carter]

[-- Attachment #1: Type: text/plain, Size: 371 bytes --]

Tried kernels 4.10.13 and 4.11, with
CONFIG_GART_IOMMU=y
CONFIG_IOMMU_HELPER=y
CONFIG_IOMMU_API=y
CONFIG_IOMMU_SUPPORT=y
CONFIG_IOMMU_IOVA=y
CONFIG_AMD_IOMMU=y
CONFIG_AMD_IOMMU_V2=m
Chipset is 990FX, and AFAICT the V2 is for the APU (bdver3 and 4 vintage).

When I enable the IOMMU in the BIOS I get stack traces. Is anyone using
AMD_IOMMU successfully? If so, any tips?

[-- Attachment #2: Type: text/html, Size: 460 bytes --]

Re: [gentoo-user] Issues with AMD_IOMMU

[Alan Grimes]

Adam Carter wrote:
> Tried kernels 4.10.13 and 4.11, with
> CONFIG_GART_IOMMU=y
> CONFIG_IOMMU_HELPER=y
> CONFIG_IOMMU_API=y
> CONFIG_IOMMU_SUPPORT=y
> CONFIG_IOMMU_IOVA=y
> CONFIG_AMD_IOMMU=y
> CONFIG_AMD_IOMMU_V2=m
> Chipset is 990FX, and AFAICT the V2 is for the APU (bdver3 and 4 vintage).
>
> When I enable the IOMMU in the BIOS I get stack traces. Is anyone
> using AMD_IOMMU successfully? If so, any tips?

On a Phenom II?    Yeah, I just retired mine a month ago. On that
system, the IOMMU hardware was kinda a legacy orphan thingy, I had to go
through all kinds of gyrations on the kernel command line in order to
initialize it correctly. I think I had it off in the bios, then enabled
it using a bunch of kernel commands.


-- 
Strange Game.
The only winning move is not to play. 

Powers are not rights.

Re: [gentoo-user] Issues with AMD_IOMMU

[Taiidan]

On 05/14/2017 01:31 AM, Adam Carter wrote:

> Tried kernels 4.10.13 and 4.11, with
> CONFIG_GART_IOMMU=y
> CONFIG_IOMMU_HELPER=y
> CONFIG_IOMMU_API=y
> CONFIG_IOMMU_SUPPORT=y
> CONFIG_IOMMU_IOVA=y
> CONFIG_AMD_IOMMU=y
> CONFIG_AMD_IOMMU_V2=m
> Chipset is 990FX, and AFAICT the V2 is for the APU (bdver3 and 4 vintage).
>
> When I enable the IOMMU in the BIOS I get stack traces. Is anyone using
> AMD_IOMMU successfully? If so, any tips?
>
Is this a gentoo kernel or one from kernel.org?

What are the exact errors you are getting? random? can you post?
On 05/14/2017 11:06 AM, Alan Grimes wrote:
> Adam Carter wrote:
>> Tried kernels 4.10.13 and 4.11, with
>> CONFIG_GART_IOMMU=y
>> CONFIG_IOMMU_HELPER=y
>> CONFIG_IOMMU_API=y
>> CONFIG_IOMMU_SUPPORT=y
>> CONFIG_IOMMU_IOVA=y
>> CONFIG_AMD_IOMMU=y
>> CONFIG_AMD_IOMMU_V2=m
>> Chipset is 990FX, and AFAICT the V2 is for the APU (bdver3 and 4 vintage).
>>
>> When I enable the IOMMU in the BIOS I get stack traces. Is anyone
>> using AMD_IOMMU successfully? If so, any tips?
> On a Phenom II?    Yeah, I just retired mine a month ago. On that
> system, the IOMMU hardware was kinda a legacy orphan thingy, I had to go
> through all kinds of gyrations on the kernel command line in order to
> initialize it correctly. I think I had it off in the bios, then enabled
> it using a bunch of kernel commands.
990FX would probably be an AMD FX CPU such as the 8350, it usually works 
fine and is enabled by default on most distros.

Re: [gentoo-user] Issues with AMD_IOMMU

[Corbin Bird]

On 05/14/2017 12:31 AM, Adam Carter wrote:
> Tried kernels 4.10.13 and 4.11, with
> CONFIG_GART_IOMMU=y
> CONFIG_IOMMU_HELPER=y
> CONFIG_IOMMU_API=y
> CONFIG_IOMMU_SUPPORT=y
> CONFIG_IOMMU_IOVA=y
> CONFIG_AMD_IOMMU=y
> CONFIG_AMD_IOMMU_V2=m
> Chipset is 990FX, and AFAICT the V2 is for the APU (bdver3 and 4 vintage).
> 
> When I enable the IOMMU in the BIOS I get stack traces. Is anyone using
> AMD_IOMMU successfully? If so, any tips?

-----

Background / System Info : 2 systems.
Chipset 990FX, FX-9590 ( -march=bdver2 / Fam15h )
Chipset 790FX, PhenomII 980 ( -march=amdfam10 / Fam10h )
Gentoo x86_64, multilib, Kernel 4.9.x
IOMMU enabled in UEFI, BIOS.

-----


General Info : ( if you don't know already ... )

The 990FX / 790FX Chipset doesn't have a GART / AGP Aperature or IOMMU
in it.
The CPU contains the original K8 IOMMU ( v1 ) engineered / converted
from a GART.

Standard output in '/var/log/dmesg'.

> [    0.000000] AGP: No AGP bridge found

> [    0.926911] PCI-DMA: Disabling AGP.
> [    0.927142] PCI-DMA: aperture base @ b4000000 size 65536 KB
> [    0.927263] PCI-DMA: using GART IOMMU.
> [    0.927376] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture

-----

Suggestion :

read the Kernel Parameter docs. The automatic setup may need tweaking.

You have to do a manual setup if you want, both, a working IOMMU and an
AGP Aperature. Firmware can throw a major monkey wrench into that type
of setup however.

> /usr/src/linux/Documentation/kernel-parameters.txt

This doc has the x86_64 IOMMU options listed ...
> /usr/src/linux/Documentation/x86/x86_64/boot-options.txt


Question :

APU?
I was under the impression that those CPU's used a different chipset,
not the 990FX.


Corbin

Re: [gentoo-user] Issues with AMD_IOMMU

[Adam Carter]

[-- Attachment #1: Type: text/plain, Size: 606 bytes --]

I've added iommu=pt to my kernel command line, and now have;

[    1.240242] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40

I was looking at IOMMU in the hope that my Win10 vmware guest would be more
stable. No change, however.


> APU?
> I was under the impression that those CPU's used a different chipset,
> not the 990FX.
>

Thanks for clearing up my confusion on the location of the IOMMU being the
CPU, not the chipset. FWIW i'm using an FX-8350.

The APU comment was with respect to CONFIG_AMD_IOMMU_V2. I believe FX CPU
just uses CONFIG_AMD_IOMMU - that's certainly working for me with the
FX-8350.

[-- Attachment #2: Type: text/html, Size: 989 bytes --]

Re: [gentoo-user] Issues with AMD_IOMMU

[Adam Carter]

[-- Attachment #1: Type: text/plain, Size: 440 bytes --]

On a Phenom II?    Yeah, I just retired mine a month ago. On that

> system, the IOMMU hardware was kinda a legacy orphan thingy, I had to go
> through all kinds of gyrations on the kernel command line in order to
> initialize it correctly. I think I had it off in the bios, then enabled
> it using a bunch of kernel commands.
>
> If it was off in the BIOS then i think that means you were just using
CONFIG_GART_IOMMU not CONFIG_AMD_IOMMU

[-- Attachment #2: Type: text/html, Size: 714 bytes --]

[gentoo-user] Re: Issues with AMD_IOMMU

[Ian Zimmerman]

On 2017-05-14 20:07, Corbin Bird wrote:

> Background / System Info : 2 systems.
> Chipset 990FX, FX-9590 ( -march=bdver2 / Fam15h )
> Chipset 790FX, PhenomII 980 ( -march=amdfam10 / Fam10h )
> Gentoo x86_64, multilib, Kernel 4.9.x
> IOMMU enabled in UEFI, BIOS.

> General Info : ( if you don't know already ... )
> 
> The 990FX / 790FX Chipset doesn't have a GART / AGP Aperature or IOMMU
> in it.  The CPU contains the original K8 IOMMU ( v1 ) engineered /
> converted from a GART.
> 
> Standard output in '/var/log/dmesg'.
> 
> > [    0.000000] AGP: No AGP bridge found
> > [    0.926911] PCI-DMA: Disabling AGP.
> > [    0.927142] PCI-DMA: aperture base @ b4000000 size 65536 KB
> > [    0.927263] PCI-DMA: using GART IOMMU.
> > [    0.927376] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture

I know we have been over this in an earlier thread "Another x11
problem", but I am still very confused.

What BIOS do you have, on the computer that is BIOS based?  I assume you
have one BIOS based and one EFI based, although you don't really say
that.  I am asking because the setup screen of my BIOS (Award modular)
doesn't mention any of these things at all.

Another source of my confusion is that my computer appears to work _now_
without setting any additional kernel options.  The problem I had (X11
turning the screen all black and never coming back) seems to have fixed
itself with an upgrade of either kernel, X11, or radeon firmware, I
don't remember which (and I probably never knew for sure which).

There are some 2 or 3 BIOS upgrades for my motherboard (GA-880GM-D2H)
but they are all ancient anyway, from 2011 - the same year I built the
computer, and the description Gigabyte gives sounds irrelevant to this
stuff, so I never changed the BIOS.

I have looked at the kernel docs you recommend, but they leave a lot to
be desired in the clarity department.  For example, the general
kernel-parameters doc lists a iommu=pt option, without explanation but
tagged with [X86][IA-64].  But the x86_64/boot-options doc doesn't
mention this value at all.

I guess my high-level confusion is that I don't understand the
relationships between all these bits of hardware, and I also don't
understand why the automatic setup that Linux does wouldn't result in a
normally working system.  So there's no aperture, so steal 64M of
addresses from RAM.  Big deal, no?  What's the problem?

Finally, in your last article in that earlier thread you said that I
should turn on multiple APICs in the kernel.  But I can find no kernel
option for that.  What's the exact CONFIG_* symbol to tweak?

Thanks.

-- 
Please *no* private Cc: on mailing lists and newsgroups
Personal signed mail: please _encrypt_ and sign
Don't clear-text sign:
http://primate.net/~itz/blog/the-problem-with-gpg-signatures.html

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Corbin Bird]

On 05/14/2017 11:58 PM, Ian Zimmerman wrote:
> On 2017-05-14 20:07, Corbin Bird wrote:
> 
>> Background / System Info : 2 systems.
>> Chipset 990FX, FX-9590 ( -march=bdver2 / Fam15h )
>> Chipset 790FX, PhenomII 980 ( -march=amdfam10 / Fam10h )
>> Gentoo x86_64, multilib, Kernel 4.9.x
>> IOMMU enabled in UEFI, BIOS.
> 
>> General Info : ( if you don't know already ... )
>>
>> The 990FX / 790FX Chipset doesn't have a GART / AGP Aperature or IOMMU
>> in it.  The CPU contains the original K8 IOMMU ( v1 ) engineered /
>> converted from a GART.
>>
>> Standard output in '/var/log/dmesg'.
>>
>>> [    0.000000] AGP: No AGP bridge found
>>> [    0.926911] PCI-DMA: Disabling AGP.
>>> [    0.927142] PCI-DMA: aperture base @ b4000000 size 65536 KB
>>> [    0.927263] PCI-DMA: using GART IOMMU.
>>> [    0.927376] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
> 
> I know we have been over this in an earlier thread "Another x11
> problem", but I am still very confused.
> 
> What BIOS do you have, on the computer that is BIOS based?  I assume you
> have one BIOS based and one EFI based, although you don't really say
> that.  I am asking because the setup screen of my BIOS (Award modular)
> doesn't mention any of these things at all.
> 
> Another source of my confusion is that my computer appears to work _now_
> without setting any additional kernel options.  The problem I had (X11
> turning the screen all black and never coming back) seems to have fixed
> itself with an upgrade of either kernel, X11, or radeon firmware, I
> don't remember which (and I probably never knew for sure which).
> 
> There are some 2 or 3 BIOS upgrades for my motherboard (GA-880GM-D2H)
> but they are all ancient anyway, from 2011 - the same year I built the
> computer, and the description Gigabyte gives sounds irrelevant to this
> stuff, so I never changed the BIOS.
> 
> I have looked at the kernel docs you recommend, but they leave a lot to
> be desired in the clarity department.  For example, the general
> kernel-parameters doc lists a iommu=pt option, without explanation but
> tagged with [X86][IA-64].  But the x86_64/boot-options doc doesn't
> mention this value at all.
> 
> I guess my high-level confusion is that I don't understand the
> relationships between all these bits of hardware, and I also don't
> understand why the automatic setup that Linux does wouldn't result in a
> normally working system.  So there's no aperture, so steal 64M of
> addresses from RAM.  Big deal, no?  What's the problem?
> 
> Finally, in your last article in that earlier thread you said that I
> should turn on multiple APICs in the kernel.  But I can find no kernel
> option for that.  What's the exact CONFIG_* symbol to tweak?
> 
> Thanks.
> 

----

Answer #1 :

Asrock, Extreme6, 990FX, UEFI, AMD FX-9590
Gigabyte, GA-MA790FX-UD5P, 790FX, BIOS, AMD PhenomII x4 980

Update the BIOS on that motherboard. Gigabyte doesn't state everything
done by their BIOS updates. The IOMMU enable option was added to the
BIOS at a later date on the GA-MA790FX-UD5P.

Gigabyte has long been known to ship 'broken for Linux' e820 firmware.
'e820' is basically the APG Aperature setup in the firmware.
Gigabyte kills OR overrides the IOMMU support in the firmware, to setup
the AGP Aperature.

Closed source / binary video drivers WANT an AGP Aperature.
The kernel wants an IOMMU on x86_64, as an IOMMU.
Squabbling over memory, and I/O, used for what, by whom, yes?

That is why I mentioned 'the monkey wrench'.

----

Answer #2 :

Here is what to look for, ( x2 APIC answer )

> 
> kernel 4.9.x, make menuconfig ... ( look just under "Symmetric multi-processing support" )
> 
>          Processor type and features  --->     
>   │ │                                                      [*] Symmetric multi-processing support                                                                                       │ │  
>   │ │                                                      [*] Support x2apic                                                                                                           │ │  
>   │ │                                                      [*] Enable MPS table                                                                                                         │ │  
>   │ │                                                      [ ] Support for extended (non-PC) x86 platforms                                                                              │ │  
>   │ │                                                      [ ] Intel Low Power Subsystem Support                                                                                        │ │  
>   │ │                                                      [ ] AMD ACPI2Platform devices support                                                                                        │ │  
>   │ │                                                      < > Intel SoC IOSF Sideband support for SoC platforms                                                                        │ │  
>   │ │                                                      [*] Single-depth WCHAN output                                                                                                │ │  
>   │ │                                                      [ ] Linux guest support  ----                                                                                                │ │  
>   │ │                                                          Processor family (AMD Piledriver)  --->                                                                                  │ │  
>   │ │                                                      [*] Old AMD GART IOMMU support                                                                                               │ │  
>   │ │                                                      [ ] IBM Calgary IOMMU support                                                                                                │ │  
>   │ │                                                      (8) Maximum number of CPUs                                                                                                   │ │  


>   │ │                                                      [*] Symmetric multi-processing support                                                                                       │ │  
>   │ │                                                      [*] Support x2apic                                                                                                           │ │  
>   │ │                                                      [*] Enable MPS table                                                                                                         │ │  
>   │ │                                                      [ ] Support for extended (non-PC) x86 platforms                                                                              │ │  
>   │ │                                                      [ ] Intel Low Power Subsystem Support                                                                                        │ │  
>   │ │                                                      [ ] AMD ACPI2Platform devices support                                                                                        │ │  
>   │ │                                                      < > Intel SoC IOSF Sideband support for SoC platforms                                                                        │ │  
>   │ │                                                      [*] Single-depth WCHAN output                                                                                                │ │  
>   │ │                                                      [ ] Linux guest support  ----                                                                                                │ │  
>   │ │                                                          Processor family (AMD 61xx/7x50/PhenomX3/X4/II/K10)  --->                                                                │ │  
>   │ │                                                      [*] Old AMD GART IOMMU support                                                                                               │ │  
>   │ │                                                      [ ] IBM Calgary IOMMU support                                                                                                │ │  
>   │ │                                                      (4) Maximum number of CPUs  


>   │ │                                                      [*] Symmetric multi-processing support                                                                                       │ │  
>   │ │                                                      [*] Support x2apic                                                                                                           │ │  
>   │ │                                                      [*] Enable MPS table                                                                                                         │ │  
>   │ │                                                      [ ] Support for extended (non-PC) x86 platforms                                                                              │ │  
>   │ │                                                      [ ] Intel Low Power Subsystem Support                                                                                        │ │  
>   │ │                                                      [ ] AMD ACPI2Platform devices support                                                                                        │ │  
>   │ │                                                      < > Intel SoC IOSF Sideband support for SoC platforms                                                                        │ │  
>   │ │                                                      [*] Single-depth WCHAN output                                                                                                │ │  
>   │ │                                                      [ ] Linux guest support  ----                                                                                                │ │  
>   │ │                                                          Processor family (AMD Barcelona)  --->                                                                                   │ │  
>   │ │                                                      [*] Old AMD GART IOMMU support   
>   │ │                                                      [ ] IBM Calgary IOMMU support
>   │ │                                                      (4) Maximum number of CPUs  

>   
> ( from '/usr/src/linux/.config', look at the fifth item in this list ... )
> #
> # Processor type and features
> #
> CONFIG_ZONE_DMA=y
> CONFIG_SMP=y
> CONFIG_X86_FEATURE_NAMES=y
> CONFIG_X86_FAST_FEATURE_TESTS=y
> CONFIG_X86_X2APIC=y
> CONFIG_X86_MPPARSE=y

-----

Check / set this kernel parameter for the PhenomII ( Fam10h / amdfam10
CPUs )
( from '/usr/src/linux/Documentation/kernel-parameters.txt' )

> 	pci=option[,option...]	[PCI] various PCI subsystem options:

> 		check_enable_amd_mmconf [X86] check for and enable
> 				properly configured MMIO access to PCI
> 				config space on AMD family 10h CPU

'/var/log/dmesg' has an error solved by setting this.

----

Corbin

Re: [gentoo-user] Issues with AMD_IOMMU

[R0b0t1]

On Sun, May 14, 2017 at 12:31 AM, Adam Carter <adamcarter3@gmail.com> wrote:
> Tried kernels 4.10.13 and 4.11, with
> CONFIG_GART_IOMMU=y
> CONFIG_IOMMU_HELPER=y
> CONFIG_IOMMU_API=y
> CONFIG_IOMMU_SUPPORT=y
> CONFIG_IOMMU_IOVA=y
> CONFIG_AMD_IOMMU=y
> CONFIG_AMD_IOMMU_V2=m
> Chipset is 990FX, and AFAICT the V2 is for the APU (bdver3 and 4 vintage).
>
> When I enable the IOMMU in the BIOS I get stack traces. Is anyone using
> AMD_IOMMU successfully? If so, any tips?

I don't mean to suggest the problem is intractable, but on fairly new
hardware (i7-4770K, MSI Z87-G45) there has been a regression in the
kernel's nested KVM support that makes me view VT-x/AMD-V VT-d/AMD-Vi
with a great deal of suspicion. In any case, devices are liable to
just spontaneously not work, even if the hardware reports it has been
properly initialized. This may require changes to the kernel and the
information needed to make those changes may or may not be available.

[gentoo-user] Re: Issues with AMD_IOMMU

[Ian Zimmerman]

On 2017-05-15 14:33, Corbin Bird wrote:

> Gigabyte has long been known to ship 'broken for Linux' e820 firmware.
> 'e820' is basically the APG Aperature setup in the firmware.
> Gigabyte kills OR overrides the IOMMU support in the firmware, to setup
> the AGP Aperature.
> 
> Closed source / binary video drivers WANT an AGP Aperature.

So is _that_ what it's all about?  Is the default assumption that I run
non-free video drivers?  Why?  I _never_ use these, and if you bet on me
using them in the future and win the bet, you can probably make a cool
million or two.

> The kernel wants an IOMMU on x86_64, as an IOMMU.

But it has one!  The GART_IOMMU.  It was even added specifically for
Linux, according to kernel.org discussions I remember seeing but can't
locate now.

I also found [1] , which I think is the clearest and most human-oriented
explanation of this issue, yet.  And ... the recommendation is pretty
much the opposite of yours.  The ball is in your court, sir.

> > CONFIG_X86_X2APIC=y

I did this long ago, and it has no effect - still only one is found,
according to dmesg.  As I wrote in the earlier thread.

Also, the Kconfig help for this option says it does something quite
different.  

> > CONFIG_X86_MPPARSE=y

Ok, this one I have not set so far.  Will read up on it and try,

[1]
http://bogdan.org.ua/2009/09/30/iommu-this-costs-you-64-mb-of-ram.html

-- 
Please *no* private Cc: on mailing lists and newsgroups
Personal signed mail: please _encrypt_ and sign
Don't clear-text sign:
http://primate.net/~itz/blog/the-problem-with-gpg-signatures.html

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Taiidan]

On 05/15/2017 12:58 AM, Ian Zimmerman wrote:

>> The 990FX / 790FX Chipset doesn't have a GART / AGP Aperature or IOMMU
>> in it.  The CPU contains the original K8 IOMMU ( v1 ) engineered /
>> converted from a GART.
>>
The 8 and 9 series (not 7) does have an IOMMU, AMD-Vi v1.26.

I have two 890 series (but the server flavor - SR5690 chipset) boards 
with an IOMMU and it works great, I play games in a VM with an attached 
graphics card on my libre firmware KGPE-D16 and devices are DMA restricted.

Most consumer boards don't properly implement this feature, in fact I 
have never seen one that did which is why I bought my coreboot (libre 
init variant) D16.
It wouldn't be that difficult to port coreboot to your board if you want 
this to work FYI.

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Taiidan]

On 05/15/2017 10:59 PM, Taiidan@gmx.com wrote:

> On 05/15/2017 12:58 AM, Ian Zimmerman wrote:
>
>>> The 990FX / 790FX Chipset doesn't have a GART / AGP Aperature or IOMMU
>>> in it.  The CPU contains the original K8 IOMMU ( v1 ) engineered /
>>> converted from a GART.
>>>
> The 8 and 9 series (not 7) does have an IOMMU, AMD-Vi v1.26.
>
> I have two 890 series (but the server flavor - SR5690 chipset) boards 
> with an IOMMU and it works great, I play games in a VM with an 
> attached graphics card on my libre firmware KGPE-D16 and devices are 
> DMA restricted.
>
> Most consumer boards don't properly implement this feature, in fact I 
> have never seen one that did which is why I bought my coreboot (libre 
> init variant) D16.
> It wouldn't be that difficult to port coreboot to your board if you 
> want this to work FYI.
>
>
To be clear, it is present on the northbridge chipset not the CPU - AMD 
documentation backs this up.

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Adam Carter]

[-- Attachment #1: Type: text/plain, Size: 674 bytes --]

Answer #1 :

>
> Asrock, Extreme6, 990FX, UEFI, AMD FX-9590
> Gigabyte, GA-MA790FX-UD5P, 790FX, BIOS, AMD PhenomII x4 980
>
>
Hi Corbin,

I noticed i didnt have x2apic enabled on my 990FX / FX-8350 system, so i've
rebuilt/rebooted etc, but;

# zgrep X2APIC /proc/config.gz
CONFIG_X86_X2APIC=y
# dmesg | grep -i x2apic
#

On my skylake box
# dmesg | grep -i x2apic
[    0.044148] DMAR-IR: Queued invalidation will be enabled to support
x2apic and Intr-remapping.
[    0.045802] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.045908] x2apic enabled
[    0.046011] Switched APIC routing to cluster x2apic.

What x2apic messages do you get on your 990FX / FX-9590 system?

[-- Attachment #2: Type: text/html, Size: 1104 bytes --]

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Corbin Bird]

On 05/15/2017 09:33 PM, Ian Zimmerman wrote:
> On 2017-05-15 14:33, Corbin Bird wrote:
> 
>> Gigabyte has long been known to ship 'broken for Linux' e820 firmware.
>> 'e820' is basically the APG Aperature setup in the firmware.
>> Gigabyte kills OR overrides the IOMMU support in the firmware, to setup
>> the AGP Aperature.
>>
>> Closed source / binary video drivers WANT an AGP Aperature.
> 
> So is _that_ what it's all about?  Is the default assumption that I run
> non-free video drivers?  Why?  I _never_ use these, and if you bet on me
> using them in the future and win the bet, you can probably make a cool
> million or two.
> 
>> The kernel wants an IOMMU on x86_64, as an IOMMU.
> 
> But it has one!  The GART_IOMMU.  It was even added specifically for
> Linux, according to kernel.org discussions I remember seeing but can't
> locate now.
> 
> I also found [1] , which I think is the clearest and most human-oriented
> explanation of this issue, yet.  And ... the recommendation is pretty
> much the opposite of yours.  The ball is in your court, sir.
> 
>>> CONFIG_X86_X2APIC=y
> 
> I did this long ago, and it has no effect - still only one is found,
> according to dmesg.  As I wrote in the earlier thread.
> 
> Also, the Kconfig help for this option says it does something quite
> different.  
> 
>>> CONFIG_X86_MPPARSE=y
> 
> Ok, this one I have not set so far.  Will read up on it and try,
> 
> [1]
> http://bogdan.org.ua/2009/09/30/iommu-this-costs-you-64-mb-of-ram.html
> 

That link, read the rest of it.

It says to leave it alone, let the kernel use it as an IOMMU.

----

I once compiled and tried to run a 64bit kernel with NO IOMMU support,
NO virt/vm support, used kernel parameters to kill all forms of IOMMU
... would not run at all ( hard lock / kernel panic ).

The error output made it very clear that an IOMMU was REQUIRED for
x86_64 ( 64bit ).

----

On AMD, with NO IOMMU kernel parameters, output in '/var/log/dmesg':
NOTE : IOMMU is enabled in the UEFI firmware.

> [    0.000000] AGP: Checking aperture...
> [    0.000000] AGP: No AGP bridge found
> [    0.000000] AGP: Node 0: aperture [bus addr 0xb4000000-0xb5ffffff] (32MB)
> [    0.000000] Aperture pointing to e820 RAM. Ignoring.
> [    0.000000] AGP: Your BIOS doesn't leave an aperture memory hole
> [    0.000000] AGP: Please enable the IOMMU option in the BIOS setup
> [    0.000000] AGP: This costs you 64MB of RAM
> [    0.000000] AGP: Mapping aperture over RAM [mem 0xb4000000-0xb7ffffff] (65536KB)

> [    0.926772] PCI: CLS 64 bytes, default 64
> [    0.926983] PCI-DMA: Disabling AGP.
> [    0.927215] PCI-DMA: aperture base @ b4000000 size 65536 KB
> [    0.927336] PCI-DMA: using GART IOMMU.
> [    0.927448] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture

Corbin

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Corbin Bird]

On 05/16/2017 01:34 AM, Adam Carter wrote:
> Answer #1 :
> 
> 
>     Asrock, Extreme6, 990FX, UEFI, AMD FX-9590
>     Gigabyte, GA-MA790FX-UD5P, 790FX, BIOS, AMD PhenomII x4 980
> 
> 
> Hi Corbin,
> 
> I noticed i didnt have x2apic enabled on my 990FX / FX-8350 system, so
> i've rebuilt/rebooted etc, but;
> 
> # zgrep X2APIC /proc/config.gz
> CONFIG_X86_X2APIC=y
> # dmesg | grep -i x2apic
> #
> 
> On my skylake box
> # dmesg | grep -i x2apic
> [    0.044148] DMAR-IR: Queued invalidation will be enabled to support
> x2apic and Intr-remapping.
> [    0.045802] DMAR-IR: Enabled IRQ remapping in x2apic mode
> [    0.045908] x2apic enabled
> [    0.046011] Switched APIC routing to cluster x2apic.
> 
> What x2apic messages do you get on your 990FX / FX-9590 system?


Interesting kernel parameter for you to try :
( from '/usr/src/linux/Documentation/kernel-parameters.txt' )

x2apic_phys	[X86-64,APIC] Use x2apic physical mode instead of
			default x2apic cluster mode on platforms
			supporting x2apic.

With that switch I have not been getting any output in 'dmesg' about
this at all.

Question :
Do you have this enabled in your kernel ...
>   [ ]   Support for Intel IOMMU using DMA Remapping Devices  


Corbin

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Corbin Bird]

On 05/15/2017 09:59 PM, Taiidan@gmx.com wrote:
> On 05/15/2017 12:58 AM, Ian Zimmerman wrote:
> 
>>> The 990FX / 790FX Chipset doesn't have a GART / AGP Aperature or IOMMU
>>> in it.  The CPU contains the original K8 IOMMU ( v1 ) engineered /
>>> converted from a GART.
>>>
> The 8 and 9 series (not 7) does have an IOMMU, AMD-Vi v1.26.
> 
> I have two 890 series (but the server flavor - SR5690 chipset) boards
> with an IOMMU and it works great, I play games in a VM with an attached
> graphics card on my libre firmware KGPE-D16 and devices are DMA restricted.
> 
> Most consumer boards don't properly implement this feature, in fact I
> have never seen one that did which is why I bought my coreboot (libre
> init variant) D16.
> It wouldn't be that difficult to port coreboot to your board if you want
> this to work FYI.
> 

Thank you for that info.

Corrections based on fact are appreciated.

---

How does one 'port' coreboot?

Last time I went to the coreboot site, I didn't see anything really
helpful to me.

Doesn't the 'CPU voltage table firmware blob' require signing NDA's?


Corbin

[gentoo-user] Re: Issues with AMD_IOMMU

[Ian Zimmerman]

On 2017-05-16 09:11, Corbin Bird wrote:

> > http://bogdan.org.ua/2009/09/30/iommu-this-costs-you-64-mb-of-ram.html
> 
> That link, read the rest of it.
> 
> It says to leave it alone, let the kernel use it as an IOMMU.
> 
> On AMD, with NO IOMMU kernel parameters, output in '/var/log/dmesg':
> NOTE : IOMMU is enabled in the UEFI firmware.
> 
> > [    0.000000] AGP: Checking aperture...
> > [    0.000000] AGP: No AGP bridge found
> > [    0.000000] AGP: Node 0: aperture [bus addr 0xb4000000-0xb5ffffff] (32MB)
> > [    0.000000] Aperture pointing to e820 RAM. Ignoring.
> > [    0.000000] AGP: Your BIOS doesn't leave an aperture memory hole
> > [    0.000000] AGP: Please enable the IOMMU option in the BIOS setup
> > [    0.000000] AGP: This costs you 64MB of RAM
> > [    0.000000] AGP: Mapping aperture over RAM [mem 0xb4000000-0xb7ffffff] (65536KB)
> 
> > [    0.926772] PCI: CLS 64 bytes, default 64
> > [    0.926983] PCI-DMA: Disabling AGP.
> > [    0.927215] PCI-DMA: aperture base @ b4000000 size 65536 KB
> > [    0.927336] PCI-DMA: using GART IOMMU.
> > [    0.927448] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture

And that is exactly what I get.

It was you who suggested starting to add iommu=blah to the kernel args.

-- 
Please *no* private Cc: on mailing lists and newsgroups
Personal signed mail: please _encrypt_ and sign
Don't clear-text sign:
http://primate.net/~itz/blog/the-problem-with-gpg-signatures.html

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Taiidan]

FYI:
IOMMU=pt means pass-through, as in no DMA protection.

AMD_IOMMU is for the bulldozer and piledriver based systems, v2 is for 
the newer excavator and beyond stuff that has vAPIC



On 05/16/2017 10:33 AM, Corbin Bird wrote:
> On 05/15/2017 09:59 PM, Taiidan@gmx.com wrote:
>> On 05/15/2017 12:58 AM, Ian Zimmerman wrote:
>>
>>>> The 990FX / 790FX Chipset doesn't have a GART / AGP Aperature or IOMMU
>>>> in it.  The CPU contains the original K8 IOMMU ( v1 ) engineered /
>>>> converted from a GART.
>>>>
>> The 8 and 9 series (not 7) does have an IOMMU, AMD-Vi v1.26.
>>
>> I have two 890 series (but the server flavor - SR5690 chipset) boards
>> with an IOMMU and it works great, I play games in a VM with an attached
>> graphics card on my libre firmware KGPE-D16 and devices are DMA restricted.
>>
>> Most consumer boards don't properly implement this feature, in fact I
>> have never seen one that did which is why I bought my coreboot (libre
>> init variant) D16.
>> It wouldn't be that difficult to port coreboot to your board if you want
>> this to work FYI.
>>
> Thank you for that info.
>
> Corrections based on fact are appreciated.
http://support.amd.com/TechDocs/43869.pdf
Hell yeah dude, this documentation backs up the fact that the IOMMU is 
on the northbridge - not the CPU.
>
> ---
>
> How does one 'port' coreboot?
>
> Last time I went to the coreboot site, I didn't see anything really
> helpful to me.
The documentation really sucks, its DIY or die - but I have never 
programmed anything before in my life and I figured out how to port boards.
> Doesn't the 'CPU voltage table firmware blob' require signing NDA's?
Naah you don't need that on most *good* systems, and 
bulldozer/piledriver era AMD was cool with releasing documentation.
>
>
> Corbin
https://www.coreboot.org/Developer_Manual
https://www.coreboot.org/Motherboard_Porting_Guide
You would start with the KCMA-D8, as it is the closest board - then 
change the superio, irq mappings, acpi etc. It takes some figuring out 
for yourself as there isn't really a detailed guide for it.

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Adam Carter]

[-- Attachment #1: Type: text/plain, Size: 2354 bytes --]

On Wed, May 17, 2017 at 12:25 AM, Corbin Bird <corbinbird@charter.net>
wrote:

> On 05/16/2017 01:34 AM, Adam Carter wrote:
> > Answer #1 :
> >
> >
> >     Asrock, Extreme6, 990FX, UEFI, AMD FX-9590
> >     Gigabyte, GA-MA790FX-UD5P, 790FX, BIOS, AMD PhenomII x4 980
> >
> >
> > Hi Corbin,
> >
> > I noticed i didnt have x2apic enabled on my 990FX / FX-8350 system, so
> > i've rebuilt/rebooted etc, but;
> >
> > # zgrep X2APIC /proc/config.gz
> > CONFIG_X86_X2APIC=y
> > # dmesg | grep -i x2apic
> > #
> >
> > On my skylake box
> > # dmesg | grep -i x2apic
> > [    0.044148] DMAR-IR: Queued invalidation will be enabled to support
> > x2apic and Intr-remapping.
> > [    0.045802] DMAR-IR: Enabled IRQ remapping in x2apic mode
> > [    0.045908] x2apic enabled
> > [    0.046011] Switched APIC routing to cluster x2apic.
> >
> > What x2apic messages do you get on your 990FX / FX-9590 system?
>
>
> Interesting kernel parameter for you to try :
> ( from '/usr/src/linux/Documentation/kernel-parameters.txt' )
>
> x2apic_phys     [X86-64,APIC] Use x2apic physical mode instead of
>                         default x2apic cluster mode on platforms
>                         supporting x2apic.
>
> With that switch I have not been getting any output in 'dmesg' about
> this at all.
>

I tried adding x2apic_phys, but dmesg output did not change.

From wikip;
"In 2012, AMD announced their *Advanced Virtual Interrupt Controller* (
*AVIC*) targeting interrupt overhead reduction in virtualization
environments.[33]
<https://en.wikipedia.org/wiki/X86_virtualization#cite_note-33> This
technology has materialize in hardware and (as announced) does not support
x2APIC <https://en.wikipedia.org/wiki/X2APIC>.[34]
<https://en.wikipedia.org/wiki/X86_virtualization#cite_note-34> In 2016,
AVIC is available on the AMD family 15h models 6Xh (Carrizo) processors and
newer.[35]" <https://en.wikipedia.org/wiki/X86_virtualization#cite_note-35>

So;
FX-8350 processors (bulldozer gen2) dont have AVIC
Carrizo processors (bulldozer gen4), have AVIC, but AVIC doesnt support
x2apic
seems reasonable to conclude that there's no x2apic on bulldozer gen2, and
that's why there's nothing in dmesg about it.


> Question :
> Do you have this enabled in your kernel ...
> >   [ ]   Support for Intel IOMMU using DMA Remapping Devices
>

On the Skylake box, yes, FX box, no.

[-- Attachment #2: Type: text/html, Size: 3630 bytes --]

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Corbin Bird]

On 05/19/2017 01:39 AM, Adam Carter wrote:
> On Wed, May 17, 2017 at 12:25 AM, Corbin Bird <corbinbird@charter.net
> <mailto:corbinbird@charter.net>> wrote:
> 
>     On 05/16/2017 01:34 AM, Adam Carter wrote:
>     > Answer #1 :
>     >
>     >
>     >     Asrock, Extreme6, 990FX, UEFI, AMD FX-9590
>     >     Gigabyte, GA-MA790FX-UD5P, 790FX, BIOS, AMD PhenomII x4 980
>     >
>     >
>     > Hi Corbin,
>     >
>     > I noticed i didnt have x2apic enabled on my 990FX / FX-8350 system, so
>     > i've rebuilt/rebooted etc, but;
>     >
>     > # zgrep X2APIC /proc/config.gz
>     > CONFIG_X86_X2APIC=y
>     > # dmesg | grep -i x2apic
>     > #
>     >
>     > On my skylake box
>     > # dmesg | grep -i x2apic
>     > [    0.044148] DMAR-IR: Queued invalidation will be enabled to support
>     > x2apic and Intr-remapping.
>     > [    0.045802] DMAR-IR: Enabled IRQ remapping in x2apic mode
>     > [    0.045908] x2apic enabled
>     > [    0.046011] Switched APIC routing to cluster x2apic.
>     >
>     > What x2apic messages do you get on your 990FX / FX-9590 system?
> 
> 
>     Interesting kernel parameter for you to try :
>     ( from '/usr/src/linux/Documentation/kernel-parameters.txt' )
> 
>     x2apic_phys     [X86-64,APIC] Use x2apic physical mode instead of
>                             default x2apic cluster mode on platforms
>                             supporting x2apic.
> 
>     With that switch I have not been getting any output in 'dmesg' about
>     this at all.
> 
> 
> I tried adding x2apic_phys, but dmesg output did not change.
> 
> From wikip;
> "In 2012, AMD announced their /Advanced Virtual Interrupt Controller/
> (/AVIC/) targeting interrupt overhead reduction in virtualization
> environments.^[33]
> <https://en.wikipedia.org/wiki/X86_virtualization#cite_note-33> This
> technology has materialize in hardware and (as announced) does not
> support x2APIC <https://en.wikipedia.org/wiki/X2APIC>.^[34]
> <https://en.wikipedia.org/wiki/X86_virtualization#cite_note-34> In 2016,
> AVIC is available on the AMD family 15h models 6Xh (Carrizo) processors
> and newer.^[35]"
> <https://en.wikipedia.org/wiki/X86_virtualization#cite_note-35>
>  
> So;
> FX-8350 processors (bulldozer gen2) dont have AVIC
> Carrizo processors (bulldozer gen4), have AVIC, but AVIC doesnt support
> x2apic
> seems reasonable to conclude that there's no x2apic on bulldozer gen2,
> and that's why there's nothing in dmesg about it.
> 
> 
>     Question :
>     Do you have this enabled in your kernel ...
>     >   [ ]   Support for Intel IOMMU using DMA Remapping Devices
> 
> 
> On the Skylake box, yes, FX box, no.

Thanks for the info. Will kill the kernel parameter on my box.

---

Tried an experiment, recompiled the kernel, no GART IOMMU support.
This was not enabled either.
 < >     AMD IOMMU Version 2 driver

CPU FX-9590, 990FX Chipset

Now getting this instead :

> [    0.988027] PCI: CLS 64 bytes, default 64
> [    0.988362] iommu: Adding device 0000:00:00.0 to group 0
> [    0.988530] iommu: Adding device 0000:00:02.0 to group 1
> [    0.988689] iommu: Adding device 0000:00:0a.0 to group 2
> [    0.988872] iommu: Adding device 0000:00:0d.0 to group 3
> [    0.989029] iommu: Adding device 0000:00:11.0 to group 4
> [    0.989194] iommu: Adding device 0000:00:12.0 to group 5
> [    0.989318] iommu: Adding device 0000:00:12.2 to group 5
> [    0.989484] iommu: Adding device 0000:00:13.0 to group 6
> [    0.989615] iommu: Adding device 0000:00:13.2 to group 6
> [    0.989780] iommu: Adding device 0000:00:14.0 to group 7
> [    0.989943] iommu: Adding device 0000:00:14.1 to group 8
> [    0.990128] iommu: Adding device 0000:00:14.3 to group 9
> [    0.990292] iommu: Adding device 0000:00:14.4 to group 10
> [    0.990458] iommu: Adding device 0000:00:14.5 to group 11
> [    0.990637] iommu: Adding device 0000:00:15.0 to group 12
> [    0.990763] iommu: Adding device 0000:00:15.1 to group 12
> [    0.990935] iommu: Adding device 0000:00:16.0 to group 13
> [    0.991058] iommu: Adding device 0000:00:16.2 to group 13
> [    0.991252] iommu: Adding device 0000:01:00.0 to group 14
> [    0.991387] iommu: Adding device 0000:01:00.1 to group 14
> [    0.991560] iommu: Adding device 0000:02:00.0 to group 15
> [    0.991742] iommu: Adding device 0000:03:00.0 to group 16
> [    0.991863] iommu: Adding device 0000:06:00.0 to group 12
> [    0.991982] iommu: Adding device 0000:07:04.0 to group 12
> [    1.063849] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
> [    1.063962] AMD-Vi: Interrupt remapping enabled
> [    1.064145] AMD-Vi: Lazy IO/TLB flushing enabled
> [    1.065331] perf: AMD NB counters detected
> [    1.065622] LVT offset 0 assigned for vector 0x400
> [    1.065836] perf: AMD IBS detected (0x000000ff)

And the Linux AGP Driver ( in-kernel ) is working now.

Now this is showing properly with lspci :
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD890S/RD990 I/O
Memory Management Unit (IOMMU)

And the GART IOMMU is no longer showing as an SMbus entry.

Corbin

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Adam Carter]

[-- Attachment #1: Type: text/plain, Size: 1047 bytes --]

<snip>

> > [    0.991863] iommu: Adding device 0000:06:00.0 to group 12
> > [    0.991982] iommu: Adding device 0000:07:04.0 to group 12
> > [    1.063849] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
> > [    1.063962] AMD-Vi: Interrupt remapping enabled
> > [    1.064145] AMD-Vi: Lazy IO/TLB flushing enabled
> > [    1.065331] perf: AMD NB counters detected
>

I'm similar, but have a couple of extra entries. I've read a little bit
about them, but so far am unable to determine if their existence indicates
a better or worse kernel config.

[    1.036309] AMD-Vi: Lazy IO/TLB flushing enabled
[    1.036419] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    1.036529] software IO TLB [mem 0xba61a000-0xbe61a000] (64MB) mapped at
[ffffa3b87a61a000-ffffa3b87e619fff]
[    1.036744] perf: AMD NB counters detected

And the Linux AGP Driver ( in-kernel ) is working now.
>
> Now this is showing properly with lspci :
> 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD890S/RD990 I/O
> Memory Management Unit (IOMMU)
>

Same.

[-- Attachment #2: Type: text/html, Size: 1575 bytes --]

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Taiidan]

Worse, ideally you wouldn't be using SWIOTLB but I don't know how to 
disable this without re-compiling the kernel.


On 05/21/2017 07:12 PM, Adam Carter wrote:
> <snip>
>
>>> [    0.991863] iommu: Adding device 0000:06:00.0 to group 12
>>> [    0.991982] iommu: Adding device 0000:07:04.0 to group 12
>>> [    1.063849] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
>>> [    1.063962] AMD-Vi: Interrupt remapping enabled
>>> [    1.064145] AMD-Vi: Lazy IO/TLB flushing enabled
>>> [    1.065331] perf: AMD NB counters detected
> q
> I'm similar, but have a couple of extra entries. I've read a little bit
> about them, but so far am unable to determine if their existence indicates
> a better or worse kernel config.
>
> [    1.036309] AMD-Vi: Lazy IO/TLB flushing enabled
> [    1.036419] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> [    1.036529] software IO TLB [mem 0xba61a000-0xbe61a000] (64MB) mapped at
> [ffffa3b87a61a000-ffffa3b87e619fff]
> [    1.036744] perf: AMD NB counters detected
>
> And the Linux AGP Driver ( in-kernel ) is working now.
>> Now this is showing properly with lspci :
>> 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD890S/RD990 I/O
>> Memory Management Unit (IOMMU)
>>
> Same.
>

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Adam Carter]

[-- Attachment #1: Type: text/plain, Size: 341 bytes --]

On Mon, May 22, 2017 at 7:15 PM, Taiidan@gmx.com <Taiidan@gmx.com> wrote:

> Worse, ideally you wouldn't be using SWIOTLB but I don't know how to
> disable this without re-compiling the kernel.
>

Can you point to some definitive documentation on this?

When i read https://wiki.gentoo.org/wiki/IOMMU_SWIOTLB i dont come to that
conclusion.

[-- Attachment #2: Type: text/html, Size: 845 bytes --]

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Corbin Bird]

On 05/21/2017 06:12 PM, Adam Carter wrote:
> <snip>
> 
>     > [    0.991863] iommu: Adding device 0000:06:00.0 to group 12
>     > [    0.991982] iommu: Adding device 0000:07:04.0 to group 12
>     > [    1.063849] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
>     > [    1.063962] AMD-Vi: Interrupt remapping enabled
>     > [    1.064145] AMD-Vi: Lazy IO/TLB flushing enabled
>     > [    1.065331] perf: AMD NB counters detected
> 
> 
> I'm similar, but have a couple of extra entries. I've read a little bit
> about them, but so far am unable to determine if their existence
> indicates a better or worse kernel config.
> 
> [    1.036309] AMD-Vi: Lazy IO/TLB flushing enabled
> [    1.036419] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> [    1.036529] software IO TLB [mem 0xba61a000-0xbe61a000] (64MB) mapped
> at [ffffa3b87a61a000-ffffa3b87e619fff]
> [    1.036744] perf: AMD NB counters detected
> 
>     And the Linux AGP Driver ( in-kernel ) is working now.
> 
>     Now this is showing properly with lspci :
>     00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD890S/RD990 I/O
>     Memory Management Unit (IOMMU)
> 
> 
> Same.

The (SWIOTLB) should not be the default.

What kernel parameters for IOMMUs are you using now?

The listed result I posted is with nothing related to IOMMUs in the
kernel parameters, and NO GART IOMMU support compiled.


Corbin

Re: [gentoo-user] Re: Issues with AMD_IOMMU

[Alan Grimes]

Here's the grub command line I was using on my Phenom II on a 990FX
board, I had only commented it out after upgrading to Ryzen....

#GRUB_CMDLINE_LINUX="check_enable_amd_mmconf amd_iommu=fullflush iommu=soft"


-- 
Strange Game.
The only winning move is not to play. 

Powers are not rights.