[gentoo-user] zram / compcache, anyone?

[gentoo-user] zram / compcache, anyone?

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 168 bytes --]

Just stumbled upon this blog:

http://www.webupd8.org/2011/10/increased-performance-in-linux-with.html

anyone got any experience with zram/compcache on Gentoo?

Rgds,

[-- Attachment #2: Type: text/html, Size: 284 bytes --]

Re: [gentoo-user] zram / compcache, anyone?

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1443 bytes --]

Am 13.10.2011 03:52, schrieb Pandu Poluan:
> Just stumbled upon this blog:
> 
> http://www.webupd8.org/2011/10/increased-performance-in-linux-with.html
> 
> anyone got any experience with zram/compcache on Gentoo?
> 
> Rgds,
> 

Yes, I use it on my laptop (4GB RAM, typically 1-2GB swap used). It
works pretty well but I can't give you any hard figures.

I wrote my own init script for this. I can share it if you want.
Otherwise the sunrise, betagarden and mv overlays offer ebuilds for it.
I think the mv version is closest to mine.

What has been pretty confusing is that there are two versions: The
original one from Google(?) and the one in the mainline kernel. They
have different APIs (hint: if you have a userland tool instead of
manipulating /sys, it is the original version) and only the original
version can use a swap device as an additional backend for
uncompressable pages. With the mainline version (which I use), you can
only use zram as an additional swap device and give it a higher priority
than your normal swap.

/etc/fstab:
/dev/zram0 none swap sw,pri=1,discard 0 0
/dev/sda7 none swap sw,pri=0 0 0

Only drawback so far: When zram is full, putting the laptop into standby
takes longer, maybe 15s compared to 3s without. Sometimes this can lead
to timeouts and the kernel aborts the suspend operation with an error on
dmesg. Reattempting it then succeeds.

Regards,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] zram / compcache, anyone?

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1541 bytes --]

Am 13.10.2011 03:52, schrieb Pandu Poluan:
> Just stumbled upon this blog:
> 
> http://www.webupd8.org/2011/10/increased-performance-in-linux-with.html
> 
> anyone got any experience with zram/compcache on Gentoo?
> 
> Rgds,
> 

Hmm, it seems like my reply was eaten by the mail server. Apologies if
you receive this twice:

I use it on my laptop (4GB RAM, typically 1-2GB swap used). It
works pretty well but I can't give you any hard figures.

I wrote my own init script for this. I can share it if you want.
Otherwise the sunrise, betagarden and mv overlays offer ebuilds for it.
I think the mv version is closest to mine.

What has been pretty confusing is that there are two versions: The
original one from Google(?) and the one in the mainline kernel. They
have different APIs (hint: if you have a userland tool instead of
manipulating /sys, it is the original version) and only the original
version can use a swap device as an additional backend for
uncompressable pages. With the mainline version (which I use), you can
only use zram as an additional swap device and give it a higher priority
than your normal swap.

/etc/fstab:
/dev/zram0 none swap sw,pri=1,discard 0 0
/dev/sda7 none swap sw,pri=0 0 0

Only drawback so far: When zram is full, putting the laptop into standby
takes longer, maybe 15s compared to 3s without. Sometimes this can lead
to timeouts and the kernel aborts the suspend operation with an error on
dmesg. Reattempting it then succeeds.

Regards,
Florian Philipp




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] zram / compcache, anyone?

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 1943 bytes --]

On Oct 13, 2011 8:32 PM, "Florian Philipp" <lists@binarywings.net> wrote:
>
> Am 13.10.2011 03:52, schrieb Pandu Poluan:
> > Just stumbled upon this blog:
> >
> > http://www.webupd8.org/2011/10/increased-performance-in-linux-with.html
> >
> > anyone got any experience with zram/compcache on Gentoo?
> >
> > Rgds,
> >
>
> Hmm, it seems like my reply was eaten by the mail server. Apologies if
> you receive this twice:
>

In addition to "the dog ate my homework", now we have a new excuse, "the
server are my document" :-D

Ah, progress ;-)

> I use it on my laptop (4GB RAM, typically 1-2GB swap used). It
> works pretty well but I can't give you any hard figures.
>
> I wrote my own init script for this. I can share it if you want.
> Otherwise the sunrise, betagarden and mv overlays offer ebuilds for it.
> I think the mv version is closest to mine.
>

What makes the proliferation of ebuilds?

> What has been pretty confusing is that there are two versions: The
> original one from Google(?) and the one in the mainline kernel. They
> have different APIs (hint: if you have a userland tool instead of
> manipulating /sys, it is the original version) and only the original
> version can use a swap device as an additional backend for
> uncompressable pages. With the mainline version (which I use), you can
> only use zram as an additional swap device and give it a higher priority
> than your normal swap.
>

In the kernel? What .config knob should I twiddle?

I do prefer having zram support in the kernel.

> /etc/fstab:
> /dev/zram0 none swap sw,pri=1,discard 0 0
> /dev/sda7 none swap sw,pri=0 0 0
>
> Only drawback so far: When zram is full, putting the laptop into standby
> takes longer, maybe 15s compared to 3s without. Sometimes this can lead
> to timeouts and the kernel aborts the suspend operation with an error on
> dmesg. Reattempting it then succeeds.
>

Point taken. Do you think it's worth the slight annoyance?

Rgds,

[-- Attachment #2: Type: text/html, Size: 2475 bytes --]

Re: [gentoo-user] zram / compcache, anyone?

[Adam Carter]

> In the kernel? What .config knob should I twiddle?

Device Drivers -> Staging Drivers -> Compressed RAM block device

Re: [gentoo-user] zram / compcache, anyone?

[Adam Carter]

On Fri, Oct 14, 2011 at 12:29 PM, Adam Carter <adamcarter3@gmail.com> wrote:
>> In the kernel? What .config knob should I twiddle?
>
> Device Drivers -> Staging Drivers -> Compressed RAM block device

But Dynamic Compression of swap pages underneath looks interesting too
(check the help).
Compressed RAM block device = CONFIG_ZRAM
Dynamic Compression of swap pages = CONFIG_ZCACHE

Re: [gentoo-user] zram / compcache, anyone?

[Pandu Poluan]

[-- Attachment #1: Type: text/plain, Size: 585 bytes --]

On Oct 14, 2011 8:36 AM, "Adam Carter" <adamcarter3@gmail.com> wrote:
>
> On Fri, Oct 14, 2011 at 12:29 PM, Adam Carter <adamcarter3@gmail.com>
wrote:
> >> In the kernel? What .config knob should I twiddle?
> >
> > Device Drivers -> Staging Drivers -> Compressed RAM block device
>
> But Dynamic Compression of swap pages underneath looks interesting too
> (check the help).
> Compressed RAM block device = CONFIG_ZRAM
> Dynamic Compression of swap pages = CONFIG_ZCACHE
>

Ah, thanks for the tips!

make menuconfig will be the first order of the day when I reach my office
:-D

Rgds,

[-- Attachment #2: Type: text/html, Size: 824 bytes --]

Re: [gentoo-user] zram / compcache, anyone?

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 3171 bytes --]

Am 14.10.2011 03:33, schrieb Adam Carter:
> On Fri, Oct 14, 2011 at 12:29 PM, Adam Carter <adamcarter3@gmail.com> wrote:
>>> In the kernel? What .config knob should I twiddle?
>>
>> Device Drivers -> Staging Drivers -> Compressed RAM block device
> 
> But Dynamic Compression of swap pages underneath looks interesting too
> (check the help).
> Compressed RAM block device = CONFIG_ZRAM
> Dynamic Compression of swap pages = CONFIG_ZCACHE
> 

Okay, so just to verify I understand this correctly (I doubt it):

Cleancache (not to be confused with compcache) is an intermediate layer
using a transcendent memory framework [6] for pages that can disappear
from the cache at any time. When the kernel has a clean page backed by
persistent memory (i.e. a file that has not been changed), instead of
removing the page from memory when the space is needed, it can push it
into cleancache. When the kernel later loads the page again, it checks
the cache to see if it is still there.[1,4]

Cleancache requires changes on the filesystem implementations. The
patches I've seen did not include any changes to Ext4 or others.
Somewhere it was mentioned that a generic hook was included so most
filesystems would immediately use it. However, the patches listed on
kernelnewbies.org did not include anything that looked like it so I
guess it was not included in 3.0. Therefore cleancache looks pretty
useless now but might be really neat in the future.

Frontswap, on the other hand, can be used for persistent pages (i.e.
"normal" anonymous memory for program data).[4] When the kernel would
normally swap out the page, it now presents it to frontswap and either
frontswap accepts the page (i.e. store it compressed in memory) or
refuses it (after which the kernel will really swap it out).[5]
Frontswap does not seem to be part of linux-3.0. The article at [4] says
it is somewhat controversial.

The difference between frontswap and cleancache is that cleancache
accepts everything but might not keep it while frontswap does not accept
everything but guarantees to keep it.

The slightly older zram implementation, on the other hand, uses the same
interface as a block device (which can be formatted as a swap device)
and therefore cannot decline an offered page like frontswap can. Zram
was easier to implement and to get accepted by the kernel community but
it is less efficient because it might save pages which are uncompressable.

Zcache is a backend used by frontswap and cleancache. It basically
reuses zram to offer a storage service for the transcendent memory
framework.[2,3] Another backend is implemented in Xen, using hypervisor
memory.[1]

Any remarks on that summary?

Regards,
Florian Philipp

[1]
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=4fe4746ab694690af9f2ccb80184f5c575917c7f

[2]
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=9cc06bf88d554dd527ded26eab28eec6a0d0e3df

[3] http://lwn.net/Articles/397574/

[4] http://lwn.net/Articles/386090/

[5] http://lwn.net/Articles/386103/

[6] https://lwn.net/Articles/340080/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] zram / compcache, anyone?

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 3799 bytes --]

Am 14.10.2011 03:10, schrieb Pandu Poluan:
> 
> On Oct 13, 2011 8:32 PM, "Florian Philipp" <lists@binarywings.net
> <mailto:lists@binarywings.net>> wrote:
>>
>> Am 13.10.2011 03:52, schrieb Pandu Poluan:
>> > Just stumbled upon this blog:
>> >
>> > http://www.webupd8.org/2011/10/increased-performance-in-linux-with.html
>> >
>> > anyone got any experience with zram/compcache on Gentoo?
>> >
[...]
>> I use it on my laptop (4GB RAM, typically 1-2GB swap used). It
>> works pretty well but I can't give you any hard figures.
>>
[...]
>> Only drawback so far: When zram is full, putting the laptop into standby
>> takes longer, maybe 15s compared to 3s without. Sometimes this can lead
>> to timeouts and the kernel aborts the suspend operation with an error on
>> dmesg. Reattempting it then succeeds.
>>
> 
> Point taken. Do you think it's worth the slight annoyance?
> 
> Rgds,
> 

As a follow-up, this is what it looks like when suspend fails with full
zram:

PM: Syncing filesystems ... done.
PM: Preparing system for mem sleep
Freezing user space processes ... (elapsed 0.01 seconds) done.
Freezing remaining freezable tasks ...
Freezing of tasks failed after 20.00 seconds (1 tasks refusing to
freeze, wq_busy=0):
khugepaged      R  running task        0   522      2 0x00800000
 ffff88010255c600 ffff88010255c8b8 ffff88013602b7c0 ffffffff81026f33
 0000000000000000 00000000ffffffff ffff8800257ab308 ffff88013602b968
 ffff88013602b7e0 ffffffff81025d00 ffffea0000e8d378 0000000000000008
Call Trace:
 [<ffffffff81025d00>] ? ptep_clear_flush_young+0x20/0x30
 [<ffffffff810a6d18>] ? __pagevec_free+0x38/0x50
 [<ffffffff810ad4fc>] ? free_page_list+0xfc/0x110
 [<ffffffff810adabd>] ? shrink_page_list+0x14d/0x5e0
 [<ffffffff810ac40f>] ? isolate_lru_pages+0x19f/0x2b0
 [<ffffffff810ae4a9>] ? shrink_inactive_list+0x2c9/0x350
 [<ffffffff810a8f25>] ? determine_dirtyable_memory+0x15/0x30
 [<ffffffff810a8fce>] ? global_dirty_limits+0x2e/0x110
 [<ffffffff810a90eb>] ? throttle_vm_writeout+0x3b/0xa0
 [<ffffffff810aebf0>] ? shrink_zone+0x480/0x550
 [<ffffffff812a3a29>] ? i915_gem_inactive_shrink+0x169/0x1f0
 [<ffffffff81061d5c>] ? ktime_get_ts+0xac/0xe0
 [<ffffffff81093aa0>] ? delayacct_end+0x80/0xa0
 [<ffffffff810a429a>] ? zone_watermark_ok+0x1a/0x20
 [<ffffffff813e7fff>] ? schedule+0x8ef/0xae0
 [<ffffffff810a5af0>] ? page_alloc_cpu_notify+0x50/0x50
 [<ffffffff810a5b01>] ? drain_local_pages+0x11/0x20
 [<ffffffff810a7bd9>] ? __alloc_pages_nodemask+0x4e9/0x830
 [<ffffffff813e884d>] ? schedule_timeout+0x16d/0x260
 [<ffffffff81049bb0>] ? del_timer+0xa0/0xa0
 [<ffffffff813e8999>] ? schedule_timeout_interruptible+0x19/0x20
 [<ffffffff810db8f7>] ? khugepaged_alloc_hugepage+0xc7/0xf0
 [<ffffffff81059260>] ? wake_up_bit+0x40/0x40
 [<ffffffff810dbd0d>] ? khugepaged+0x8d/0x11f0
 [<ffffffff81059260>] ? wake_up_bit+0x40/0x40
 [<ffffffff810dbc80>] ? collect_mm_slot+0xa0/0xa0
 [<ffffffff81058df6>] ? kthread+0x96/0xa0
 [<ffffffff813ebb54>] ? kernel_thread_helper+0x4/0x10
 [<ffffffff81058d60>] ? kthread_worker_fn+0x180/0x180
 [<ffffffff813ebb50>] ? gs_change+0xb/0xb

Restarting tasks ... done

All things considered, I think it is worth it. This is my memory usage:
free -m
      total         used free shared buffers cached
Mem:  3753          3167  586      0      16    644
-/+ buffers/cache:  2505 1247
Swap: 8018          1487 6530

cat /proc/swaps
Filename   Type      Size    Used    Priority
/dev/zram0 partition 1921700 1520320 1
/dev/sda7  partition 6289412    2968 0

There is no observable slowdown, no swapping, no system freeze when
switching active windows. It feels like the system didn't swap at all. I
even increased swappiness to use more space for caching.

Regards,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] zram / compcache, anyone?

[Paul Hartman]

On Wed, Oct 12, 2011 at 8:52 PM, Pandu Poluan <pandu@poluan.info> wrote:
> Just stumbled upon this blog:
>
> http://www.webupd8.org/2011/10/increased-performance-in-linux-with.html
>
> anyone got any experience with zram/compcache on Gentoo?

I'm using zram in a gentoo server with only 256mb of RAM, only used
for a few weeks so far. It seems to work and the server hasn't crashed
yet. :) I have allocated 128MB of compressed swap (64x2, actually, to
theoretically utilize both CPU cores for compression at the same time)
followed by normal on-disk swap at lower priority. Usually my total
swap used is less than 128MB so the real disk swap is rarely touched.
It's difficult to say if there is any improved performance, but I
haven't experienced any slowdown, which occasionally I did when swap
became heavily used in the past. Keep in mind the 128MB zram is the
uncompressed size, so the actual amount of RAM used by this should be
much less, depending on contents of the swap. Some even recommend
using zram equal to the amount of RAM but that idea scares me.

After enabling the CONFIG_ZRAM module in kernel 3.0.6, I did this:

modprobe zram num_devices=2
echo $((64*1024*1024)) > /sys/block/zram0/disksize
echo 1 > /sys/block/zram0/reset
mkswap /dev/zram0
swapon -p 11 /dev/zram0

(repeat for /dev/zram1 and so on)

you can then swapoff your disk swap partition to empty it, then swapon
with lower priority than the zram swap devices.

Also note that zram is really just a generic compressed RAM drive. You
don't have to use it for swap, you can mkfs anything you like onto it,
to use as compressed tmp space or whatever... just know that it'll be
gone when you reboot.

I think zram can be beneficial in an environment where CPU power is
plentiful but RAM needs to be conserved (i.e. fitting more virtual
servers onto one physical box). I seem to recall there is a way for a
virtual host to utilize zram automatically/transparently for the
virtualized guests, but I don't personally know anything about that.

Re: [gentoo-user] zram / compcache, anyone?

[Florian Philipp]

[-- Attachment #1: Type: text/plain, Size: 1789 bytes --]

Am 18.10.2011 07:16, schrieb Paul Hartman:
> On Wed, Oct 12, 2011 at 8:52 PM, Pandu Poluan <pandu@poluan.info> wrote:
>> Just stumbled upon this blog:
>>
>> http://www.webupd8.org/2011/10/increased-performance-in-linux-with.html
>>
>> anyone got any experience with zram/compcache on Gentoo?
> 
> I'm using zram in a gentoo server with only 256mb of RAM, only used
> for a few weeks so far. It seems to work and the server hasn't crashed
> yet. :) I have allocated 128MB of compressed swap (64x2, actually, to
> theoretically utilize both CPU cores for compression at the same time)
> followed by normal on-disk swap at lower priority. Usually my total
> swap used is less than 128MB so the real disk swap is rarely touched.
> It's difficult to say if there is any improved performance, but I
> haven't experienced any slowdown, which occasionally I did when swap
> became heavily used in the past. Keep in mind the 128MB zram is the
> uncompressed size, so the actual amount of RAM used by this should be
> much less, depending on contents of the swap. Some even recommend
> using zram equal to the amount of RAM but that idea scares me.
> 
> After enabling the CONFIG_ZRAM module in kernel 3.0.6, I did this:
> 
> modprobe zram num_devices=2
> echo $((64*1024*1024)) > /sys/block/zram0/disksize
> echo 1 > /sys/block/zram0/reset
# sleep 1
> mkswap /dev/zram0
> swapon -p 11 /dev/zram0
> 

In my experience, it can be necessary to put a `sleep 1` between reset
and mkswap because the /dev/zram0 disappears and reappears after the
reset command.

Another remark: The kernel docs recommend using /bin/echo instead of
echo because problems are reported as write errors and the echo builtin
of bash doesn't check for that.

Regards,
Florian Philipp


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [gentoo-user] zram / compcache, anyone?

[Paul Hartman]

On Tue, Oct 18, 2011 at 12:53 AM, Florian Philipp <lists@binarywings.net> wrote:
> Am 18.10.2011 07:16, schrieb Paul Hartman:
>> On Wed, Oct 12, 2011 at 8:52 PM, Pandu Poluan <pandu@poluan.info> wrote:
>>> Just stumbled upon this blog:
>>>
>>> http://www.webupd8.org/2011/10/increased-performance-in-linux-with.html
>>>
>>> anyone got any experience with zram/compcache on Gentoo?
>>
>> I'm using zram in a gentoo server with only 256mb of RAM, only used
>> for a few weeks so far. It seems to work and the server hasn't crashed
>> yet. :) I have allocated 128MB of compressed swap (64x2, actually, to
>> theoretically utilize both CPU cores for compression at the same time)
>> followed by normal on-disk swap at lower priority. Usually my total
>> swap used is less than 128MB so the real disk swap is rarely touched.
>> It's difficult to say if there is any improved performance, but I
>> haven't experienced any slowdown, which occasionally I did when swap
>> became heavily used in the past. Keep in mind the 128MB zram is the
>> uncompressed size, so the actual amount of RAM used by this should be
>> much less, depending on contents of the swap. Some even recommend
>> using zram equal to the amount of RAM but that idea scares me.
>>
>> After enabling the CONFIG_ZRAM module in kernel 3.0.6, I did this:
>>
>> modprobe zram num_devices=2
>> echo $((64*1024*1024)) > /sys/block/zram0/disksize
>> echo 1 > /sys/block/zram0/reset
> # sleep 1
>> mkswap /dev/zram0
>> swapon -p 11 /dev/zram0
>>
>
> In my experience, it can be necessary to put a `sleep 1` between reset
> and mkswap because the /dev/zram0 disappears and reappears after the
> reset command.

Good to know, thanks. In my case I typed those commands manually, so
of course I didn't encounter any timing-related problem like that.

> Another remark: The kernel docs recommend using /bin/echo instead of
> echo because problems are reported as write errors and the echo builtin
> of bash doesn't check for that.

Also noted, thanks again.