[gentoo-amd64] Advice On Enabling Transparent Hugepage

[gentoo-amd64] Advice On Enabling Transparent Hugepage

[Frank Peters]

During kernel configuration there is an option called CONFIG_TRANSPARENT_HUGEPAGE
which allows the use of Transparent Hugepages or application memory pages larger
than the traditional 4K size.  This option has been available for a while but
I've never enabled it.  However, it would seem like a good idea to use.

Searching for more information I can find no comments on actual Hugepage
performance for a simple desktop Linux system.  Can anyone verify or refute
the idea that Transparent Hugepages will lead to improvements on a desktop
system?  Does an application have to specifically request Hugepages or
does the allocation occur for all applications?

Frank Peters

Re: [gentoo-amd64] Advice On Enabling Transparent Hugepage

[Paul Hartman]

On Thu, May 2, 2013 at 3:33 PM, Frank Peters <frank.peters@comcast.net> wrote:
> During kernel configuration there is an option called CONFIG_TRANSPARENT_HUGEPAGE
> which allows the use of Transparent Hugepages or application memory pages larger
> than the traditional 4K size.  This option has been available for a while but
> I've never enabled it.  However, it would seem like a good idea to use.
>
> Searching for more information I can find no comments on actual Hugepage
> performance for a simple desktop Linux system.  Can anyone verify or refute
> the idea that Transparent Hugepages will lead to improvements on a desktop
> system?  Does an application have to specifically request Hugepages or
> does the allocation occur for all applications?

AFAIR there were claims of up to 10% performance improvement in
certain test scenarios, and that you don't need to do anything special
to exploit THP in existing software (but applications can be rewritten
in such a way as to specifically make better use of it).

There is a setting that lets you expose THP _only_ to those programs
which specifically request it, rather than backwards-compatible to all
programs if you are worried about that.

Check out the in-kernel documentation (vm/transhuge.txt) for details.

Anecdotal: I have turned it off and on and don't "feel" any difference
in the operation of my computer either way...

Re: [gentoo-amd64] Advice On Enabling Transparent Hugepage

[Frank Peters]

On Thu, 2 May 2013 16:05:10 -0500
Paul Hartman <paul.hartman+gentoo@gmail.com> wrote:

> 
> There is a setting that lets you expose THP _only_ to those programs
> which specifically request it, rather than backwards-compatible to all
> programs if you are worried about that.
> 

I don't really expect an answer, but what programs incorporate the
MADV_HUGEPAGE directive in their source code?  Since THP is relatively
new for the Linux kernel, it would seem that most Linux software has not
yet included MADV_HUGEPAGE and could not benefit from THP if it is
limited to madvise only.

Also, if a program does include MADV_HUGEPAGE would it need to be
recompiled with a THP enabled kernel?

These are some questions I will have to research.

However, the madvise manpage has this to say about THP:

"This feature is primarily  aimed  at  applications that use large mappings
of data and access large regions of that memory at a time (e.g., virtualization
systems  such  as  QEMU)."

I wonder if image or sound processing applications are using MADV_HUGEPAGE.

Frank Peters

[gentoo-amd64] Re: Advice On Enabling Transparent Hugepage

[Duncan]

Frank Peters posted on Thu, 02 May 2013 18:51:42 -0400 as excerpted:

> On Thu, 2 May 2013 16:05:10 -0500 Paul Hartman
> <paul.hartman+gentoo@gmail.com> wrote:
> 
>> There is a setting that lets you expose THP _only_ to those programs
>> which specifically request it, rather than backwards-compatible to all
>> programs if you are worried about that.
>> 
> I don't really expect an answer, but what programs incorporate the
> MADV_HUGEPAGE directive in their source code?  Since THP is relatively
> new for the Linux kernel, it would seem that most Linux software has not
> yet included MADV_HUGEPAGE and could not benefit from THP if it is
> limited to madvise only.

As suggested by the manpage you quote below, it's primarily big-memory 
applications, including databases, virtualization, etc.  Those apps stand 
to gain enough from it (and have enough big-money sponsored development 
behind them) that despite the feature being relatively new and Linux 
specific, their newest releases are likely to support it.

> Also, if a program does include MADV_HUGEPAGE would it need to be
> recompiled with a THP enabled kernel?

AFAIK it wouldn't actually need rebuilt against a supporting kernel, 
specifically, but it may need built against a supporting linux-headers 
package.  When built against a supporting linux-headers, the app would 
detect at runtime whether the running kernel has the feature available or 
not, and use it if so, but should still run just fine, only without that 
optimization, on earlier kernels or those built without the feature 
enabled.

> These are some questions I will have to research.
> 
> However, the madvise manpage has this to say about THP:
> 
> "This feature is primarily  aimed  at  applications that use large
> mappings of data and access large regions of that memory at a time
> (e.g., virtualization systems  such  as  QEMU)."
> 
> I wonder if image or sound processing applications are using
> MADV_HUGEPAGE.

I don't know about media apps.  It may be that despite their memory 
usage, their access pattern doesn't benefit from huge-pages... or maybe 
it does... I simply don't know.

Meanwhile, when you wrote the above you probably hadn't read my earlier 
reply as they likely crossed on the net, and I don't know if it 
specifically uses MADV_HUGEPAGE or not, but as I said in that reply, 
firefox definitely uses transparent huge-pages if they're generally 
available.  I know this due to the zero-huge-page related bug I had with 
an early pre-release of the kernel that enabled that, since it was firefox 
triggering that bug.  I temporarily turned transparent-hugepage support 
off due to that, but turned it back on again when the bug was fixed, 
before that kernel was released as a .0, and haven't had a huge-pages 
related problem since.  But due to that now fixed bug, I definitely know 
at least one app that makes use of huge-pages, here. =:^)

If you plan on doing a bunch of research on the subject, I strongly 
suggest that you look at the links I posted in the previous reply.  LWN's 
well respected in the Linux and particularly kernel community as a 
relatively authoritative source, and you could certainly do worse than LWN 
as a source for research on the subject, but may not be able to find 
better.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-amd64] Re: Advice On Enabling Transparent Hugepage

[Frank Peters]

On Fri, 3 May 2013 05:13:44 +0000 (UTC)
Duncan <1i5t5.duncan@cox.net> wrote:

>  
> firefox definitely uses transparent huge-pages if they're generally 
> available.  
>

According to this early benchmark at Phoronix, the improvement for
system-wide THP use is not that impressive:

http://www.phoronix.com/scan.php?page=article&item=linux_transparent_hugepages&num=1

I think it would probably be best to enable THP in the kernel limited
to MADV_HUGEPAGE only.

Frank Peters

Re: [gentoo-amd64] Re: Advice On Enabling Transparent Hugepage

[Volker Armin Hemmann]

Am 03.05.2013 18:12, schrieb Frank Peters:
> On Fri, 3 May 2013 05:13:44 +0000 (UTC)
> Duncan <1i5t5.duncan@cox.net> wrote:
>
>>  
>> firefox definitely uses transparent huge-pages if they're generally 
>> available.  
>>
> According to this early benchmark at Phoronix, the improvement for
> system-wide THP use is not that impressive:
>
> http://www.phoronix.com/scan.php?page=article&item=linux_transparent_hugepages&num=1
>
> I think it would probably be best to enable THP in the kernel limited
> to MADV_HUGEPAGE only.
>

why? it has no downsides. Apps use transparent hugepages without knowing
it (that is what the magical transparent means). It does not slow down
things.

Just turn it on.

[gentoo-amd64] Re: Advice On Enabling Transparent Hugepage

[Duncan]

Frank Peters posted on Thu, 02 May 2013 16:33:09 -0400 as excerpted:

> During kernel configuration there is an option called
> CONFIG_TRANSPARENT_HUGEPAGE which allows the use of Transparent
> Hugepages or application memory pages larger than the traditional 4K
> size.  This option has been available for a while but I've never enabled
> it.  However, it would seem like a good idea to use.
> 
> Searching for more information I can find no comments on actual Hugepage
> performance for a simple desktop Linux system.  Can anyone verify or
> refute the idea that Transparent Hugepages will lead to improvements on
> a desktop system?  Does an application have to specifically request
> Hugepages or does the allocation occur for all applications?

In addition to what Paul says (which is generally correct, AFAIK)...

As I understand it, the whole idea of "huge pages" is based on making 
better use of a relatively limited hardware resource, the TLB entries, 
translation lookaside buffer entries.  The TLB is a sort of memory 
mapping cache index, the memory mapping itself being between virtual and 
physical memory.  The technical details of TLB are beyond me, so I'll 
simply refer you to wikipedia:

http://en.wikipedia.org/wiki/Translation_lookaside_buffer

But basically, the TLB is an index with a limit of (from wikipedia) 
12-4096 entries, depending on architecture and specific CPU.

The key is that each memory page requires an entry, and normal memory 
pages are 4k in size, so normal TLB coverage is relatively small, a few 
megabytes of RAM can be indexed at a time, that's it.

Which is where huge pages come in, since each huge page entry requires 
the same index line as a normal page would, but indexes and allows fast 
access to a far larger bit of memory, often 2 MB.  If all TLB line 
entries were for huge-pages, then, coverage would be several GB instead 
of several MB.

The problem was to implement that in the kernel.  Originally, there was a 
huge page interface that individual apps could use, but only if they 
specifically requested it, which of course required that they be 
rewritten in ordered to do so.

Which is where the transparent comes in.  With transparent huge-pages, 
apps work as normal -- it's the kernel that manages the huge pages 
"transparently".

Practical value depends on your work load, but in general, it won't hurt.

Two further notes: 1) Because I regularly run mainline git kernels, I was 
exposed to the option early on, and based on the coverage in (2) below, I 
enabled them.  But at one point in a pre-release kernel there was a bug, 
of course long since fixed.  The details aren't important as it is long 
fixed, but due to that bug, I *DID* discover ONE app that ends up using 
huge-pages a LOT -- firefox!  Evidently, huge pages only really come into 
play when individual app memory usage is relatively high, and firefox hit 
that threshold, as well as some other requirements for huge pages, plus 
some corner-cases that were required to hit this particular bug (which 
was with zero-pages, in the second kernel that had huge-page support as 
they added zero-page support, a different "trick" that wasn't initially 
applied to huge-pages, only normal pages).

So for that period, I simply disabled huge-page support again, until the 
bug was fixed.  Then I enabled it and haven't disabled it since.

2) LWN has several articles covering kernel huge-page support as well as 
transparent-huge-pages, including a 5-part-series covering the initial 
non-transparent version, the first part (with further links to the other 
parts) of which is found here:

http://lwn.net/Articles/374424/

That'll let you get pretty deep into the technology, if you like.  
There's benchmarks, ways to discover what the TLBs are for your hardware, 
etc.

Here's a google search limited to site:lwn.net for transparent huge pages:

http://www.google.com/search?q=site%3Alwn.net+transparent+huge+pages

Of interest, down the first page of hits a bit you can see overage of the 
"adding a huge zero page" discussion, the source of the bug I mentioned 
above, again, now long fixed.

That should give you quite enough information to get you started. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman