[gentoo-user] Strive for zero swap usage?

[gentoo-user] Strive for zero swap usage?

[Grant]

[-- Attachment #1: Type: text/plain, Size: 201 bytes --]

Swap usage on Linux always seems a little tricky to me.  Should my
goal on a web server be zero swap usage, meaning the attached graph
should show no green lines at all if I'm doing it right?

- Grant

[-- Attachment #2: swap.png --]
[-- Type: image/png, Size: 54051 bytes --]

Re: [gentoo-user] Strive for zero swap usage?

[Rich Freeman]

On Thu, Oct 6, 2016 at 5:51 PM, Grant <emailgrant@gmail.com> wrote:
> Swap usage on Linux always seems a little tricky to me.  Should my
> goal on a web server be zero swap usage, meaning the attached graph
> should show no green lines at all if I'm doing it right?
>

No.  You want things that aren't in use to be swapped, like memory
leaks and such.  You don't want things that will be used to be
swapped.

-- 
Rich

Re: [gentoo-user] Strive for zero swap usage?

[Grant]

>> Swap usage on Linux always seems a little tricky to me.  Should my
>> goal on a web server be zero swap usage, meaning the attached graph
>> should show no green lines at all if I'm doing it right?
>>
>
> No.  You want things that aren't in use to be swapped, like memory
> leaks and such.  You don't want things that will be used to be
> swapped.


Does this look OK?  It looks to me like heavy swapping in and out with
plenty of free memory (minus buffers/cache).

# vmstat 1 20
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  1 155308  71328  29200 2437260    1    0   127    63    0    0 48
1 50  1  0
 0  1 155308  44444  29204 2465164    0    0 27772     0 1483 1236  1
1 74 24  0
 2  2 155480  32904  29220 2476392    0  172 33684   172 4166 2790 44
2 30 24  0
 4  0 155828  30332  29220 2479756    0  348 41388   348 5319 1884 76
2  1 21  0
 1  1 156184  37668  29244 2470752    0  356 28960   424 5033 3299 53
2 15 29  0
 0  2 156412  32912  29264 2477364    0  228 30976   252 2909 2516 14
2 56 29  0
 2  5 156640  29372  29400 2480328    0  228 23180   344 3949 3156 32
2 35 31  0
 1  4 157072  36372  29624 2469924    0  432 28032   488 4120 3289 50
2 10 37  0
 2  4 157260  37372  29812 2466036    0  188 12188   316 3789 2540 54
1  7 38  0
 2  3 157260  30236  29912 2467672    0    0 16372    48 3730 2077 57
2  8 34  0
 2  3 157304  29684  28940 2459688   24   44 10892    44 4271 2808 44
2 11 43  0
 5  2 157824  36448  28372 2440176    4  520 13036   620 4796 3185 56
2 10 33  0
 2  3 157928  34552  28516 2437032    4  104 19620   148 4446 4108 48
3 15 35  0
 2  6 157928  38620  28444 2385828  436    0  4808   136 4615 2851 39
5  6 51  0
 3  5 157928  36936  28604 2377112  732    0  8072    84 4773 2970 64
2  0 34  0
 5  7 157928  39068  28832 2375944   40    0  2040   492 4877 3341 71
2  1 26  0
 3  4 157928  81820  28984 2376868  188    0 19320     0 5122 6990 66
5  0 30  0

# free -h
             total       used       free     shared    buffers     cached
Mem:          3.9G       3.8G        41M        88M        20M       2.4G
-/+ buffers/cache:       1.4G       2.4G
Swap:         1.0G       154M       869M

- Grant

Re: [gentoo-user] Strive for zero swap usage?

[Grant]

>>> Swap usage on Linux always seems a little tricky to me.  Should my
>>> goal on a web server be zero swap usage, meaning the attached graph
>>> should show no green lines at all if I'm doing it right?
>>>
>>
>> No.  You want things that aren't in use to be swapped, like memory
>> leaks and such.  You don't want things that will be used to be
>> swapped.
>
>
> Does this look OK?  It looks to me like heavy swapping in and out with
> plenty of free memory (minus buffers/cache).


Or put another way, how do I know when swapping is a problem?  I'm
running munin so I can look over graphs of my system's characteristics
but I'm not sure what to look for to determine if I'm swapping
excessively.

- Grant


> # vmstat 1 20
> procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
>  0  1 155308  71328  29200 2437260    1    0   127    63    0    0 48
> 1 50  1  0
>  0  1 155308  44444  29204 2465164    0    0 27772     0 1483 1236  1
> 1 74 24  0
>  2  2 155480  32904  29220 2476392    0  172 33684   172 4166 2790 44
> 2 30 24  0
>  4  0 155828  30332  29220 2479756    0  348 41388   348 5319 1884 76
> 2  1 21  0
>  1  1 156184  37668  29244 2470752    0  356 28960   424 5033 3299 53
> 2 15 29  0
>  0  2 156412  32912  29264 2477364    0  228 30976   252 2909 2516 14
> 2 56 29  0
>  2  5 156640  29372  29400 2480328    0  228 23180   344 3949 3156 32
> 2 35 31  0
>  1  4 157072  36372  29624 2469924    0  432 28032   488 4120 3289 50
> 2 10 37  0
>  2  4 157260  37372  29812 2466036    0  188 12188   316 3789 2540 54
> 1  7 38  0
>  2  3 157260  30236  29912 2467672    0    0 16372    48 3730 2077 57
> 2  8 34  0
>  2  3 157304  29684  28940 2459688   24   44 10892    44 4271 2808 44
> 2 11 43  0
>  5  2 157824  36448  28372 2440176    4  520 13036   620 4796 3185 56
> 2 10 33  0
>  2  3 157928  34552  28516 2437032    4  104 19620   148 4446 4108 48
> 3 15 35  0
>  2  6 157928  38620  28444 2385828  436    0  4808   136 4615 2851 39
> 5  6 51  0
>  3  5 157928  36936  28604 2377112  732    0  8072    84 4773 2970 64
> 2  0 34  0
>  5  7 157928  39068  28832 2375944   40    0  2040   492 4877 3341 71
> 2  1 26  0
>  3  4 157928  81820  28984 2376868  188    0 19320     0 5122 6990 66
> 5  0 30  0
>
> # free -h
>              total       used       free     shared    buffers     cached
> Mem:          3.9G       3.8G        41M        88M        20M       2.4G
> -/+ buffers/cache:       1.4G       2.4G
> Swap:         1.0G       154M       869M

Re: [gentoo-user] Strive for zero swap usage?

[Michael Mol]

[-- Attachment #1: Type: text/plain, Size: 1906 bytes --]

On Friday, October 07, 2016 04:33:27 AM Grant wrote:
> >>> Swap usage on Linux always seems a little tricky to me.  Should my
> >>> goal on a web server be zero swap usage, meaning the attached graph
> >>> should show no green lines at all if I'm doing it right?
> >> 
> >> No.  You want things that aren't in use to be swapped, like memory
> >> leaks and such.  You don't want things that will be used to be
> >> swapped.
> > 
> > Does this look OK?  It looks to me like heavy swapping in and out with
> > plenty of free memory (minus buffers/cache).
> 
> Or put another way, how do I know when swapping is a problem?  I'm
> running munin so I can look over graphs of my system's characteristics
> but I'm not sure what to look for to determine if I'm swapping
> excessively.

"Swapping excessively" is inherently a use-case-specific problem, but it comes 
down to two questions:

* Do you notice your system spending time in iowait swapping data in while 
you're waiting on it? 
* Do you notice your system spending time in iowait swapping data out while 
you're waiting on it? (I.e. as it tries to make room for new memory 
allocations)

If the answer to those questions is yes, then you're swapping excessively. If 
not, you're not.

There are ways other than swap to find yourself in iowait, though. I wonder 
what might a good metric of combining iowait numbers with swap event counts. 
Swap events without iowait are likely imperceptible.

But it does all come down to perception and how you want to manage it. I have 
some nodes that swap a *lot*, but I don't care as long as they don't fall 
behind in their workload. And I have some nodes that I don't permit to swap at 
all, as that causes latency spikes that are difficult to nail down, or can cause 
snowballing cascade events across several nodes and processes that interact 
with each other.

-- 
:wq

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [gentoo-user] Strive for zero swap usage?

[Grant]

>> >>> Swap usage on Linux always seems a little tricky to me.  Should my
>> >>> goal on a web server be zero swap usage, meaning the attached graph
>> >>> should show no green lines at all if I'm doing it right?
>> >>
>> >> No.  You want things that aren't in use to be swapped, like memory
>> >> leaks and such.  You don't want things that will be used to be
>> >> swapped.
>> >
>> > Does this look OK?  It looks to me like heavy swapping in and out with
>> > plenty of free memory (minus buffers/cache).
>>
>> Or put another way, how do I know when swapping is a problem?  I'm
>> running munin so I can look over graphs of my system's characteristics
>> but I'm not sure what to look for to determine if I'm swapping
>> excessively.
>
> "Swapping excessively" is inherently a use-case-specific problem, but it comes
> down to two questions:
>
> * Do you notice your system spending time in iowait swapping data in while
> you're waiting on it?
> * Do you notice your system spending time in iowait swapping data out while
> you're waiting on it? (I.e. as it tries to make room for new memory
> allocations)


I can't find a good graph for iowait in munin.  Is watching wa in top
my best bet?

If I do find a correlation between iowait and web server response
times, should I just decrease memory usage until the problem goes
away?

What I do notice is that my web server's response time increases along
with the swapping peaks in the graph I posted before.

- Grant


> If the answer to those questions is yes, then you're swapping excessively. If
> not, you're not.
>
> There are ways other than swap to find yourself in iowait, though. I wonder
> what might a good metric of combining iowait numbers with swap event counts.
> Swap events without iowait are likely imperceptible.
>
> But it does all come down to perception and how you want to manage it. I have
> some nodes that swap a *lot*, but I don't care as long as they don't fall
> behind in their workload. And I have some nodes that I don't permit to swap at
> all, as that causes latency spikes that are difficult to nail down, or can cause
> snowballing cascade events across several nodes and processes that interact
> with each other.

Re: [gentoo-user] Strive for zero swap usage?

[Bill Kenworthy]

On 08/10/16 07:43, Grant wrote:
>>>>>> Swap usage on Linux always seems a little tricky to me.  Should my
>>>>>> goal on a web server be zero swap usage, meaning the attached graph
>>>>>> should show no green lines at all if I'm doing it right?
> ...

Have you tuned swappiness?

e.g.:
vm.swappiness=1
vm.vfs_cache_pressure=50

You can force minimal use of swap as well as turn it off.  I have tried
to go swap-less a few times and have come to the conclusion you are
trading occasional slow downs for the certainty of a crash and burn when
you hit an unexpected memory requirement.  I have swap (2xram - mostly
for hibernation), but tune it as above.

rattus ~ # free
              total        used        free      shared  buff/cache
available
Mem:        4041520     1605108      147588       19500     2288824
2083800
Swap:       4194300         356     4193944
rattus ~ #

BillK

Re: [gentoo-user] Strive for zero swap usage?

[Bill Kenworthy]

On 08/10/16 08:26, Bill Kenworthy wrote:
> On 08/10/16 07:43, Grant wrote:
>>>>>>> Swap usage on Linux always seems a little tricky to me.  Should my
>>>>>>> goal on a web server be zero swap usage, meaning the attached graph
>>>>>>> should show no green lines at all if I'm doing it right?
>> ...
> 
> Have you tuned swappiness?
> 
> e.g.:
> vm.swappiness=1
> vm.vfs_cache_pressure=50
> 
> You can force minimal use of swap as well as turn it off.  I have tried
> to go swap-less a few times and have come to the conclusion you are
> trading occasional slow downs for the certainty of a crash and burn when
> you hit an unexpected memory requirement.  I have swap (2xram - mostly
> for hibernation), but tune it as above.
> 
> rattus ~ # free
>               total        used        free      shared  buff/cache
> available
> Mem:        4041520     1605108      147588       19500     2288824
> 2083800
> Swap:       4194300         356     4193944
> rattus ~ #
> 
> BillK
> 
> 

Also, this is a read that may  help:
http://www.linuxjournal.com/article/10678

BillK

Re: [gentoo-user] Strive for zero swap usage?

[Grant]

>>>>>>> Swap usage on Linux always seems a little tricky to me.  Should my
>>>>>>> goal on a web server be zero swap usage, meaning the attached graph
>>>>>>> should show no green lines at all if I'm doing it right?
>> ...
>
> Have you tuned swappiness?


Thanks Bill I'll give swappiness a try too.

- Grant

Re: [gentoo-user] Strive for zero swap usage?

[Grant]

>> "Swapping excessively" is inherently a use-case-specific problem, but it comes
>> down to two questions:
>>
>> * Do you notice your system spending time in iowait swapping data in while
>> you're waiting on it?
>> * Do you notice your system spending time in iowait swapping data out while
>> you're waiting on it? (I.e. as it tries to make room for new memory
>> allocations)


I just ran sar from the sysstat package and this looks like a lot of
iowait to me:

00:00:02        CPU     %user     %nice   %system   %iowait    %steal     %idle
00:10:01        all     48.11      0.86      0.83      1.38      0.00     48.82
00:20:01        all     43.98      0.85      0.64      0.54      0.00     53.99
00:30:01        all     48.17      0.90      1.04      0.82      0.00     49.07
00:40:01        all     48.69      0.85      1.06      0.48      0.00     48.92
00:50:01        all     49.74      0.87      0.58      0.49      0.00     48.33
01:00:01        all     46.21      0.85      0.48      0.41      0.00     52.05
01:10:01        all     48.10      0.86      0.79      0.61      0.00     49.64
01:20:01        all     54.00      0.86      0.60      0.65      0.00     43.89
01:30:01        all     45.81      0.85      0.49      0.49      0.00     52.36
01:40:01        all     52.04      0.86      0.56      0.56      0.00     45.99
01:50:01        all     48.49      0.85      0.52      0.47      0.00     49.66
02:00:01        all     43.18      0.85      0.48      0.50      0.00     54.99
02:10:01        all     45.48      1.12      1.74     20.65      0.00     31.01
02:20:02        all     46.20     10.22      1.97      9.70      0.00     31.90
02:30:01        all     64.93      0.88      1.98     12.54      0.00     19.67
02:40:01        all     46.24      0.86      0.93      5.08      0.00     46.90
02:50:01        all     43.49      0.85      0.45      0.60      0.00     54.60
03:00:01        all     43.28      0.85      0.45      0.45      0.00     54.97
03:10:01        all     39.58      0.85      0.81      5.22      0.00     53.54
03:20:01        all     42.04      0.91      0.72      3.97      0.00     52.35
03:30:01        all     46.60      0.85      0.74      0.49      0.00     51.31
03:40:01        all     47.30      0.85      0.82      0.82      0.00     50.22
03:50:01        all     49.43      0.85      0.84      0.59      0.00     48.29
04:00:01        all     45.50      0.85      1.02      0.71      0.00     51.91
04:10:01        all     44.35      0.86      1.13      2.32      0.00     51.35
04:20:01        all     44.29      0.85      1.17      4.91      0.00     48.77
04:30:01        all     42.69      0.85      0.47      1.41      0.00     54.59
04:40:01        all     48.22      0.85      1.00      7.23      0.00     42.70
04:50:01        all     44.70      0.86      0.49      1.49      0.00     52.45
Average:        all     46.92      1.19      0.86      2.95      0.00     48.08


> If I do find a correlation between iowait and web server response
> times, should I just decrease memory usage until the problem goes
> away?
>
> What I do notice is that my web server's response time increases along
> with the swapping peaks in the graph I posted before.
>
>
>> There are ways other than swap to find yourself in iowait, though. I wonder
>> what might a good metric of combining iowait numbers with swap event counts.
>> Swap events without iowait are likely imperceptible.


I do see a clear correlation between iowait above and swap in on the
munin graph.  Is that enough to conclude that swap activity is slowing
down the system and I need to reduce memory usage or perhaps tune
swappiness?

- Grant

[gentoo-user] Re: Strive for zero swap usage?

[Kai Krakow]

Am Sat, 8 Oct 2016 05:09:46 -0700
schrieb Grant <emailgrant@gmail.com>:

> >> "Swapping excessively" is inherently a use-case-specific problem,
> >> but it comes down to two questions:
> >>
> >> * Do you notice your system spending time in iowait swapping data
> >> in while you're waiting on it?
> >> * Do you notice your system spending time in iowait swapping data
> >> out while you're waiting on it? (I.e. as it tries to make room for
> >> new memory allocations)  
> 
> 
> I just ran sar from the sysstat package and this looks like a lot of
> iowait to me:
> 
> 00:00:02        CPU     %user     %nice   %system   %iowait
> %steal     %idle 00:10:01        all     48.11      0.86
> 0.83      1.38      0.00     48.82 00:20:01        all     43.98
> 0.85      0.64      0.54      0.00     53.99 00:30:01        all
> 48.17      0.90      1.04      0.82      0.00     49.07
> 00:40:01        all     48.69      0.85      1.06      0.48
> 0.00     48.92 00:50:01        all     49.74      0.87      0.58
> 0.49      0.00     48.33 01:00:01        all     46.21      0.85
> 0.48      0.41      0.00     52.05 01:10:01        all     48.10
> 0.86      0.79      0.61      0.00     49.64 01:20:01        all
> 54.00      0.86      0.60      0.65      0.00     43.89
> 01:30:01        all     45.81      0.85      0.49      0.49
> 0.00     52.36 01:40:01        all     52.04      0.86      0.56
> 0.56      0.00     45.99 01:50:01        all     48.49      0.85
> 0.52      0.47      0.00     49.66 02:00:01        all     43.18
> 0.85      0.48      0.50      0.00     54.99 02:10:01        all
> 45.48      1.12      1.74     20.65      0.00     31.01
> 02:20:02        all     46.20     10.22      1.97      9.70
> 0.00     31.90 02:30:01        all     64.93      0.88      1.98
> 12.54      0.00     19.67 02:40:01        all     46.24
> 0.86      0.93      5.08      0.00     46.90 02:50:01        all
> 43.49      0.85      0.45      0.60      0.00     54.60
> 03:00:01        all     43.28      0.85      0.45      0.45
> 0.00     54.97 03:10:01        all     39.58      0.85      0.81
> 5.22      0.00     53.54 03:20:01        all     42.04      0.91
> 0.72      3.97      0.00     52.35 03:30:01        all     46.60
> 0.85      0.74      0.49      0.00     51.31 03:40:01        all
> 47.30      0.85      0.82      0.82      0.00     50.22
> 03:50:01        all     49.43      0.85      0.84      0.59
> 0.00     48.29 04:00:01        all     45.50      0.85      1.02
> 0.71      0.00     51.91 04:10:01        all     44.35      0.86
> 1.13      2.32      0.00     51.35 04:20:01        all     44.29
> 0.85      1.17      4.91      0.00     48.77 04:30:01        all
> 42.69      0.85      0.47      1.41      0.00     54.59
> 04:40:01        all     48.22      0.85      1.00      7.23
> 0.00     42.70 04:50:01        all     44.70      0.86      0.49
> 1.49      0.00     52.45 Average:        all     46.92      1.19
> 0.86      2.95      0.00     48.08
> 
> 
> > If I do find a correlation between iowait and web server response
> > times, should I just decrease memory usage until the problem goes
> > away?
> >
> > What I do notice is that my web server's response time increases
> > along with the swapping peaks in the graph I posted before.
> >
> >  
> >> There are ways other than swap to find yourself in iowait, though.
> >> I wonder what might a good metric of combining iowait numbers with
> >> swap event counts. Swap events without iowait are likely
> >> imperceptible.  
> 
> 
> I do see a clear correlation between iowait above and swap in on the
> munin graph.  Is that enough to conclude that swap activity is slowing
> down the system and I need to reduce memory usage or perhaps tune
> swappiness?

Looking at the times, it looks a lot like you are having higher iowait
only at around 2:00 and 4:20 which are pretty standard cron job times.
These probably run niced or ioniced. It's normal that you are seeing
higher iowait for such processes.

You may want to try setting your io scheduler to deadline (or even noop
if you are using a RAID controller with bbu and write cache). Since you
seem to prefer response times over throughput you should be using
deadline io scheduler anyways. Actually, don't use the default CFQ if
your server is virtualized. At least in my tests, CFQ seems to work a
lot against what virtualized IO seems to achieve.

I also suggest using maybe XFS as a filesystem. Which one are you using?

If your server is a web server and it starts swapping, there is not
much you can do against it. Tuning swappiness will probably not help at
all. Get more RAM or lower your memory usage. If, for example, MySQL
runs on the same host, either move it or lower it's memory usage.
Reduce the amount of apache application processes running at the same
time (PHP, Perl, whatever), use a layered application stack: One
frontend for handling static files, one middleware server for handling
requests over to PHP and doing the request dispatch queue, and reduce
memory/IO footprint of your backend.

-- 
Regards,
Kai

Replies to list-only preferred.

Re: [gentoo-user] Re: Strive for zero swap usage?

[Håkon Alstadheim]

Den 08. okt. 2016 21:27, skrev Kai Krakow:
>
> You may want to try setting your io scheduler to deadline (or even noop
> if you are using a RAID controller with bbu and write cache). Since you
> seem to prefer response times over throughput you should be using
> deadline io scheduler anyways. Actually, don't use the default CFQ if
> your server is virtualized. At least in my tests, CFQ seems to work a
> lot against what virtualized IO seems to achieve.
>
> I also suggest using maybe XFS as a filesystem. Which one are you using?
>
I second XFS and deadline, at least on RAID or on a VM. I got bitten by
XFS recently though, had a crash with the linux-4.8.0 bug which left my
root fs in a bad state. I had no xfs_repair in my initramfs,  (no help
from genkernel) and the systemrescueCD images from portage were stored
on the non-mountable fs :-). My GRML CD had a version of xfs_repair that
was too old to deal. Had to find a lap-top and plug it straight to the
outside line and get a systemrescueCD from some shady download-provider.
Believe it was sourceforge. Make sure you have an up-to date
systemrescueCD with recent-ish xfsprogs residing in you boot device if
you go XFS for your root-device. fsck.xfs is basically a no-op, so if
bad things happen, you mount to play back the journal, then umount, then
xfs_repair.

Packages:
app-admin/systemrescuecd-x86 (doctored to store latest .iso on your
boot-device, which you back up to memory-stick)
sys-boot/systemrescuecd-x86-grub
sys-fs/xfsprogs (up to date !)

Re: [gentoo-user] Re: Strive for zero swap usage?

[Grant]

> Looking at the times, it looks a lot like you are having higher iowait
> only at around 2:00 and 4:20 which are pretty standard cron job times.
> These probably run niced or ioniced. It's normal that you are seeing
> higher iowait for such processes.
>
> You may want to try setting your io scheduler to deadline (or even noop
> if you are using a RAID controller with bbu and write cache). Since you
> seem to prefer response times over throughput you should be using
> deadline io scheduler anyways. Actually, don't use the default CFQ if
> your server is virtualized. At least in my tests, CFQ seems to work a
> lot against what virtualized IO seems to achieve.


I'm using CFQ now, no virtualization.  I should use CFQ if I prefer
throughput and deadline for response time?


> I also suggest using maybe XFS as a filesystem. Which one are you using?


I'm using ext3 but I plan to move to ZFS.


> If your server is a web server and it starts swapping, there is not
> much you can do against it. Tuning swappiness will probably not help at
> all. Get more RAM or lower your memory usage. If, for example, MySQL
> runs on the same host, either move it or lower it's memory usage.
> Reduce the amount of apache application processes running at the same
> time (PHP, Perl, whatever), use a layered application stack: One
> frontend for handling static files, one middleware server for handling
> requests over to PHP and doing the request dispatch queue, and reduce
> memory/IO footprint of your backend.


Changing swappiness from 60 to 30 has drastically reduced swap usage
but I'm not sure how much it has done for iowait and response times.
I'll know more in a few days.  If swap usage stays very low and I'm
still not happy with the consistency of response times, I would think
reducing memory usage won't help.

- Grant

[gentoo-user] Re: Strive for zero swap usage?

[Kai Krakow]

Am Sun, 9 Oct 2016 06:25:20 -0700
schrieb Grant <emailgrant@gmail.com>:

> > Looking at the times, it looks a lot like you are having higher
> > iowait only at around 2:00 and 4:20 which are pretty standard cron
> > job times. These probably run niced or ioniced. It's normal that
> > you are seeing higher iowait for such processes.
> >
> > You may want to try setting your io scheduler to deadline (or even
> > noop if you are using a RAID controller with bbu and write cache).
> > Since you seem to prefer response times over throughput you should
> > be using deadline io scheduler anyways. Actually, don't use the
> > default CFQ if your server is virtualized. At least in my tests,
> > CFQ seems to work a lot against what virtualized IO seems to
> > achieve.  
> 
> 
> I'm using CFQ now, no virtualization.  I should use CFQ if I prefer
> throughput and deadline for response time?

Maybe not as easy as that. But I remember benchmarks found that
deadline or noop may give better web server performance. But that
depends on your application. I'm running a few virtual machines
concurrently on a few less hardware machines. Resources are sometimes a
bit overcommitted, and I see deadline gives much reduced iowait in the
machines in contrast to using cfq.

> > I also suggest using maybe XFS as a filesystem. Which one are you
> > using?  
> 
> 
> I'm using ext3 but I plan to move to ZFS.

XFS is much better at handling many parallel requests, and is also quite
good working with deadline. I strongly suggest against using ZFS if
swapping is already a problem for you even now.

> > If your server is a web server and it starts swapping, there is not
> > much you can do against it. Tuning swappiness will probably not
> > help at all. Get more RAM or lower your memory usage. If, for
> > example, MySQL runs on the same host, either move it or lower it's
> > memory usage. Reduce the amount of apache application processes
> > running at the same time (PHP, Perl, whatever), use a layered
> > application stack: One frontend for handling static files, one
> > middleware server for handling requests over to PHP and doing the
> > request dispatch queue, and reduce memory/IO footprint of your
> > backend.  
> 
> Changing swappiness from 60 to 30 has drastically reduced swap usage
> but I'm not sure how much it has done for iowait and response times.
> I'll know more in a few days.  If swap usage stays very low and I'm
> still not happy with the consistency of response times, I would think
> reducing memory usage won't help.

Yes, it may have reduced swap usage - but at what price? Reduced cache?
This can only increase iowait...


-- 
Regards,
Kai

Replies to list-only preferred.

Re: [gentoo-user] Strive for zero swap usage?

[Jeremi Piotrowski]

On Thu, Oct 06, 2016 at 03:31:59PM -0700, Grant wrote:
> 
> Does this look OK?  It looks to me like heavy swapping in and out with
> plenty of free memory (minus buffers/cache).
> 

Have you looked at the _units_ displayed by vmstat and munin? You're
looking at **kilobytes** being swapped and calling that _heavy swapping_...

In general your system shows that you have a lot of memory used as cache,
and I would guess that the kernel thinks its better to keep using that
memory for caching and swap out the occasional page of application data.
The alternative would be dropping some cache - might be worse for
performance. Swapping does not seem to be something that could be causing
any issues for you.

Do you even have an issue with response times of your webservers? If so
(which you haven't explicitly stated here) it would be better for you to
look at things from that end - what is taking long in the webserver and
what is causing those things to take that long.

As it stands, you're using anti-methods of performance analysis:

    Streetlight Anti-Method:

        1. Pick observability tools there are familiar
        2. Run tools
        3. Look for obvious issues

and
    
    Traffic Light Anti-Method

        1. Open dashboard (munin)
        2. All green? Assume everything is good
        3. Something red? Assume that's a problem

If you're really interested in this kind of stuff I recommend checking out
the materials that Brendan Gregg has gathered over the years.
http://www.brendangregg.com/methodology.html and other links on his page.

Re: [gentoo-user] Strive for zero swap usage?

[Michael Mol]

[-- Attachment #1: Type: text/plain, Size: 2956 bytes --]


On Friday, October 07, 2016 04:43:56 PM Grant wrote:
> >> >>> Swap usage on Linux always seems a little tricky to me.  Should my
> >> >>> goal on a web server be zero swap usage, meaning the attached graph
> >> >>> should show no green lines at all if I'm doing it right?
> >> >> 
> >> >> No.  You want things that aren't in use to be swapped, like memory
> >> >> leaks and such.  You don't want things that will be used to be
> >> >> swapped.
> >> > 
> >> > Does this look OK?  It looks to me like heavy swapping in and out with
> >> > plenty of free memory (minus buffers/cache).
> >> 
> >> Or put another way, how do I know when swapping is a problem?  I'm
> >> running munin so I can look over graphs of my system's characteristics
> >> but I'm not sure what to look for to determine if I'm swapping
> >> excessively.
> > 
> > "Swapping excessively" is inherently a use-case-specific problem, but it
> > comes down to two questions:
> > 
> > * Do you notice your system spending time in iowait swapping data in while
> > you're waiting on it?
> > * Do you notice your system spending time in iowait swapping data out
> > while
> > you're waiting on it? (I.e. as it tries to make room for new memory
> > allocations)
> 
> I can't find a good graph for iowait in munin.  Is watching wa in top
> my best bet?

I've never used Munin. I use Zabbix, which monitors iowait just fine.

> 
> If I do find a correlation between iowait and web server response
> times, should I just decrease memory usage until the problem goes
> away?

There's more than one cause for iowait, so you can't just assume iowait is 
related to memory consumption.

iowait can happen while waiting for files to load in from disk. In this case, 
freeing up memory to be used by the page cache can help.

iowait can happen while waiting for data to be written *to* disk. In this 
case, finding what's writing to disk and reducing that can help.

iowait can happen during swap. In this case, reducing things going into swap 
can help. You can reduce things going into swap by reducing vm.swappiness (I 
prefer to set it to 0, myself; swap will be used if and only if there isn't 
enough memory at the moment). You can reduce things going into swap by tuning 
applications to use less memory. (With a web server, there are going to be a 
lot of things to tune here. What is the webserver doing? What web server is it 
running? Is it running a dynamic application? What language?)

> 
> What I do notice is that my web server's response time increases along
> with the swapping peaks in the graph I posted before.

Then you're using too much memory, which is leading you to use too much swap, 
which is causing an I/O bottleneck for you. Can you tell me more about your 
stack? What is the web server doing? What HTTPd are you using? Are you running 
dynamic applications? What language? Are you using any caching?

-- 
:wq

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 473 bytes --]