Re: [gentoo-amd64] Re: mtrr: base is not aligned

[Mark Knecht <markknecht@gmail.com>]

On 10/16/05, Duncan <1i5t5.duncan@cox.net> wrote:
> Mark Knecht posted
> <5bdc1c8b0510151118p447b72a9la959017a0de1dd08@mail.gmail.com>, excerpted
> below,  on Sat, 15 Oct 2005 11:18:12 -0700:
>
> >    Is anyone else seeign these troubling messages from tie to time?
> >
> > mtrr: base(0xe8020000) is not aligned on a size(0x400000) boundary
> > mtrr: type mismatch for e8000000,8000000 old: write-back new: write-combining
> > mtrr: type mismatch for e8000000,8000000 old: write-back new: write-combining
> > mtrr: type mismatch for e8000000,8000000 old: write-back new: write-combining
> > mtrr: base(0xe8020000) is not aligned on a size(0x400000) boundary
> > mtrr: type mismatch for e8000000,8000000 old: write-back new: write-combining
> > mtrr: base(0xe8020000) is not aligned on a size(0x400000) boundary
> > mtrr: type mismatch for e8000000,8000000 old: write-back new: write-combining
> > mtrr: base(0xe8020000) is not aligned on a size(0x400000) boundary
> > mtrr: type mismatch for e8000000,8000000 old: write-back new: write-combining
> >
> >    Is there somethign I should be doing to fix this?
>
> You may well know more about this than I do, but on the off chance this
> may be new to you and for others... (and because I'm googling and learning
> a bit in the process myself... =8^)

Duncan,
   Great post. I'm getting some really good stuff out of it.

   Time to fess up. Based on reading this and looking again this
morning I'm embarrassed to say that the machine that is failing is
actually an Intel P4HT machine, not an AMD64 machine. I do have an
AMD64 machine and I guess I blanked out at the moment I sent the
email. None the less I think the topic is valuable on either list and
I cannot imagine getting a more detailed and comprehensive response
from anyone. Thanks!
>
> mtrr=memory type range (or region) register.  This is definitely kernel
> domain we are talking about here, often video drivers (so xorg related as
> well).

Yes, this I knew. Until reading you post I thought it was only used
for video memory, and from my hardware background (PC chipset/AGP
stuff some years ago) specifically with write-combining. I see that it
really can have more wide ranging uses.
<SNIP>
>
> http://www.meduna.org/txt_mtrr_en.html
>
> Paraphrasing from the above link...
>
> Basically, the MTRRs determine the behavior of cache vs regular memory on
> memory-write, for various memory regions/ranges.
>
<SNIP>
>
> * Write-combining is sort of in-between the two above choices.  The data
> is allowed to sit in cache without updating main memory only a
> comparatively short period, in ordered to allow the possibility of
> combining several smaller writes into a larger, single, more efficient
> write.  (More efficient because each write has a set amount of overhead in
> setup and take-down time and data.  Thus, combining 8 128-byte
> transactions into a single 1KB transaction means 1/8 the overhead, thus
> more effective payload bandwidth, at a cost of more latency, due to
> waiting for several transactions to accumulate, provided of course they
> come in before the expiry time forces what's there to be written even if
> it's not yet a full size transfer.)

There was a totally related, but slightly additional use for write
combining at the hardware level. It was introduced by Intel when they
first wrote the AGP 1.0 spec. The idea at the time was that game
software doing 3D work does not naturally output memory chunks that
are the same width as the AGP bus. Typically they are smaller blocks,
like 2 bytes, 4 bytes, etc. For this reason, to keep transfers on the
AGP bus more efficient, write combining could be set up so that the
processor would hold a number of small writes in a single cache line
and only flush this cache line over the AGP bus to the video display
when the software moved off the cache line or over-wrote some dirty
part of the cache line. The idea was that there was no reason to send
a bunch of bytes in separate writes, using more AGP bus bandwidth,
when these writes could be sent as a group using less bandwidth. The
description above supports this but it's not specifically graphics
oriented.
>
> * There's also uncachable, which turns off caching for reads as well as
> writes.  This will be /very/ slow.
>
> Where graphics gets involved is that these MTRRs are often used to program
> access to video memory over the AGP or whatever bus.  The link above lists
> some of the (then) xfree86 operations that MTRR settings affect and by how
> much.

Yep.

>
> Another link with some interesting info on the kernel config option
> (CONFIG_MTRR) and the userland interface to MTRRs (/proc/mtrr) once
> enabled.  The two paragraphs below are excerpted:
>
> http://developer.osdl.org/dev/robustmutexes/src/fusyn.hg/Documentation/mtrr.txt

This one is interesting in that it opens my eyes to the possibility of
defining more MTRRs, a concept that I hadn't considered before.

<SNIP>

> Memory type region registers control the caching on newer Intel and non
> Intel processors. This function allows drivers to request an MTRR is
> added. The details and hardware specifics of each processor's
> implementation are hidden from the caller, but nevertheless the caller
> should expect to need to provide a power of two size on an equivalent
> power of two boundary.

The 'power of two' comments are interesting.
<SNIP>
>
> That last contains a mention of boundaries ("power of two size on an
> equivalent power of two boundary") that appears to pertain to your
> problem.

Seems to...

>
> So... now we know a bit about what MTRRs actually do (control the
> interaction between cache and a specified portion of memory for write
> transactions), what they are most often adjusted for (to increase graphics
> performance, by changing the way writes to graphics memory are cached),
> and can make a bit of sense out of the messages (the size doesn't match
> the required base address for the MTRR, something's trying to change the
> caching method, but using the wrong address and the adjustment is
> therefore getting a type mismatch).

OK.

>
> How does that translate into something you can do to fix it?
>
> Well, first, you can actually go take a look at /proc/mtrr (don't try to
> write anything to it, unless you are sure you know what you are doing, but
> reading it should be fine), and see if you can figure out which entry it's
> supposed to be changing, if there's one close to that address at all, or
> if not, what needs created.

Yes, on the IA32 machine that actually has the problem I see strange
results. It's a machine with 512MB, but possibly with shared memory?
Not sure. Anyway, look at this:

mark@dragonfly ~ $ cat /proc/mtrr
reg00: base=0x00000000 (   0MB), size=983552MB: write-back, count=1
reg01: base=0x1f000000 ( 496MB), size=983056MB: uncachable, count=1
mark@dragonfly ~ $

where as my AMD64 machine has a single MTRR operating and shows this:

mark@lightning ~/.wine/drive_c/VST $ cat /proc/mtrr
reg00: base=0x00000000 (   0MB), size= 512MB: write-back, count=1
mark@lightning ~/.wine/drive_c/VST $

It seems that the IA32 machine is having some trouble with these
registers. The size numbers look strange to me.

<SNIP>

>Whatever your video card driver is, that's
> probably (but not for certain) what's causing the log messages.

I agree. It's a built in Intel device. Part of the chipset I think:

dragonfly ~ # lspci
0000:00:00.0 Host bridge: Intel Corporation 82865G/PE/P DRAM
Controller/Host-Hub Interface (rev 02)
0000:00:02.0 VGA compatible controller: Intel Corporation 82865G
Integrated Graphics Controller (rev 02)

> Therefore, take a look at /var/log/Xorg.0.log, and see if you can match up
> any possible MTRR messages listed there.

No MTRR messages in Xorg.0.log....

>
> Next, take a look at the driver documentation and your xorg.conf file
> and boot loader config, and see what sort of adjustments you might need to
> make.
>
> Of course, if it's /not/ video driver related, you'll likely have to
> figure out what else is accessing the MTRRs and how to reconfigure it
> correctly.  Taking a wild guess, I'd say check anything that's likely to
> be using  DMA, thus, stuff like NICs or storage devices, and their drivers.

I'm thinking that the strange sizes on the first MTRR are what caused
the messages and possibly relate to something not really happy at boot
time. The first one shows up in this set of operations:

agpgart: Detected an Intel 865 Chipset.
agpgart: Detected 16252K stolen memory.
agpgart: AGP aperture is 128M @ 0xe8000000
[drm] Initialized drm 1.0.0 20040925
ACPI: PCI Interrupt 0000:00:02.0[A] -> GSI 16 (level, low) -> IRQ 19
mtrr: type mismatch for e8000000,8000000 old: write-back new: write-combining
[drm] Initialized i915 1.1.0 20040405 on minor 0: Intel Corporation
82865G Integrated Graphics Controller

Not sure if it's ACPI related or graphics related. I'm guessing the latter...

<SNIP>
>
> From an Opteron BIOS integrator's pdf @ amd... they recommend one of the
> variable MTRRs (there are some fixed ones covering the memory space from
> 640k to 1 MB as well, that must be what those common settings in BIOS must
> be for) be set to cover the entire physical memory range... And so I
> see... I have a gig of memory and see a 1024 MB MTRR set @ base-address
> 0x 0000 0000 (0 MB), type write-back (so read/write cacheable). That makes
> sense as it's telling the CPU(s, 2 in my case) that all of my main memory
> is fully cacheable, no special restrictions needed!
>
> Apparently, some CPUs only have two variable MTRRs, and if one is used to
> cover all of physical main-memory, that leaves only one available, which
> would be used by the video driver, so that's how Linux is normally setup.
> Again apparently, modern x86 (and presumably x86_64 as well) CPUs from
> both AMD and Intel have eight such MTRRs, so more ranges can be programed
> as needed.

Intel P4 with Hyper-threading here. I would assume it has a reasonable
set of these things.

<SNIP>
>
> Note that at least here (Tyan s2885 dual Opteron board), the BIOS actually
> has two related settings controlling the way > 3.5 GB of physical memory
> is mapped.  One apparently controls the actual memory addresses, whether
> they skip the 3.5-4 GB range or not, the other controls the MTRRs,
> continuous or not.  If the two don't match, it could mean all of memory is
> seen but not all of it is marked as cacheable, slowing access to the
> uncached memory range.

I have not investigated BIOS on this machine. Good point.

>
> ...  Another leap of understanding...  Remember those ranges need to be in
> power-of-2 sizes and on matching power-of-2 boundaries?  I happen to have
> a gig of memory, an even power of two, so one MTR covers it exactly.
> That wouldn't work for those with 3/4 gig or 1.5 or 3 gig or some such.
> OTOH, I noticed a count=1 at the end of the two ranges I have mapped in my
> /proc/mtrr, so it would appear that could be remedied by using say 3
> half-gig MTRs (count=3) stacked end-to-end, or a 1 gig and a half gig
> (thus two), to map that 1.5 gig area.  I'm not sure if the count= would
> mean it's using additional MTRRs or not, tho I'd expect it would indeed
> mean that.  Thus, non-power-of-two memory layouts will probably mean more
> MTRRs used to map the full memory MTR.
>
> (If there's anyone with an odd amount of memory that could verify, it'd be
> nice...)

I have an older Athlon XP 1600+ machine with 756MB. I don't pretend to
understand this exactly but here's the results:

gigastudio ~ # cat /proc/mtrr
reg00: base=0x00000000 (   0MB), size= 512MB: write-back, count=1
reg01: base=0x20000000 ( 512MB), size= 256MB: write-back, count=1
reg02: base=0xe8000000 (3712MB), size= 128MB: write-combining, count=1
reg07: base=0xf0000000 (3840MB), size= 128MB: write-combining, count=1
gigastudio ~ #

It looks like reg00 and reg01 make up the 756MB of main memory. The
video card is an NVidia with 128MB (I think....I'll have to go check)
so I assume that this is reg02. I do not understand reg07.

Somewhat disappointing, from this conversation, is my AMD64 machine:

mark@lightning ~/.wine/drive_c/VST $ cat /proc/mtrr
reg00: base=0x00000000 (   0MB), size= 512MB: write-back, count=1
mark@lightning ~/.wine/drive_c/VST $

Why no mtrr for the video adapter? Note that this machine is running
2.6.14-rc4-rt6. That's new today. There are no advanced drivers for
this kernel so possibly this gets turned on when the VGA driver is
loaded?
>
> My second MTRR is set to cover the 128 MB range dedicated to my video card
> (it's got 256 MB but on 128 MB is being used, it would appear, but it's no
> big deal since I don't do much 3D anyway because I'm running dual head
> video @ 2048x1536 each).  Here, it's the 128 MB beginning at exactly 3.5
> MB base address (0x e000 0000), write combining.
>
> OK...  0x e000 0000 is the 3.5 GB boundary, so your 0x e802 0000 is indeed
> in the PCI device area... 0x 800 0000 is 128 MB, 0x 2 0000 is 8 KB, so the
> base address it's attempting to use is 3.5 GB + 128 MB + 8 KB.  The
> requested size is 0x 40 0000 or 4 MB, so the closest boundary would be the
> 0x e800 0000 you see it trying for and getting the type mismatch.

And this is the address that my Via machine with 768MB shows.
<SNIP>
>
> That's about all I have ATM...  Hope this is as informative for others as
> it just was for me!  <g>  (And, if anyone's an expert on this stuff and I
> got it wrong, please inform me where!  Better to correct any mistakes I
> have now while it's new than after I build a bunch more suppositions on a
> faulty foundation!)
>

That was a huge contribution to my knowledge and I appreciate it
greatly. I think it's more than likely that there is something here
that should be reported to the LKML but I want to understand it just a
bit more first.

Thanks,
Mark

-- 
gentoo-amd64@gentoo.org mailing list