Re: [gentoo-user] More memory?

Re: [gentoo-user] More memory?

[James Colannino]

Grant wrote:

>>Swap enables the system to have more memory pages for the running processes. If
>>the OS doesn't have enough memory for a process, it'll let the process know, or
>>simply kill it.
>>
>>If you don't want swap to be used, disable it. Remember that the system also
>>uses physical ram for caching disk, not only for user processes.
>>
>>If you don't use swap, your 'inactive' processes won't go into swap. Maybe the
>>OS prefers having more disk into cache than a process in physical ram. (there
>>must be rules in Linux for that). Then, when your usually-inactive process is
>>awakened, the OS must move its pages from disk to RAM, shown as a slow answer
>>from that process.
>>
>>Reaching your memory limit without swap, is the same as reaching the memory
>>limit with swap (physical + disk). Someone who maintains a system, must select a
>>reasonable amount of 'swap' according to the expected behaviour of that system.
>>What do you want, be able of running more processes than your physical RAM
>>enables -use swap- against some slowness, or keep on your physical RAM limit,
>>keeping a normal system speed for all processes?
>>
>>conclusion: you're absolutely free for disabling your swap. System doesn't
>>require swap for "some reason", as you said.
>>    
>>
>
>OK, I'm getting a better grip on memory and swap thanks to you guys. 
>Let me ask this another way though.
>
>I doesn't seem like disabling swap is such a good idea, especially on
>a commercial server and especially when I'm going to be running a
>second instance of the OS inside VMware.  What if my site gets a bunch
>of traffic for whatever reason all of a sudden?  Maybe I'm compiling
>my kernel at the same time, and running all kinds of tests in the
>VMware'd OS.  It seems like the swap needs to be there for times when
>the system is really strained and the memory fills up with active
>stuff.  I remember when I was just starting with Gentoo I forgot to
>enable the swap and kept getting an out of memory error during the
>bootstrap.
>
>Maybe a better way to phrase my question is:  Is it possible to set my
>server up so it will use swap when it needs it and then free it back
>up when it doesn't need it anymore?  What makes me think that is
>necessary is the fact that I see a very snappy response when browsing
>my site after a fresh reboot.  After it's been up for awhile, the swap
>starts to fill and it slows down.  Rebooting clears out the swap and
>the snaps return.
>
>All I'm trying to do here is keep my site nice and fast.
>  
>

Hey Grant, this may not help, but I'm reading posts about disk cache 
going into physical memory, and I thought of something.  Since you get 
better performance when you reboot, try this command as root:

#sync

See if that helps.  That basically flushes the cache and sends it to 
disk where it would eventually go anyway.  If so, then you know what's 
hogging all your memory.

James

-- 
My blog: http://www.crazydrclaw.com/
My homepage: http://james.colannino.org/

"Black holes are where God divided by zero." --Steven Wright


--
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] More memory?

[Bastian Balthazar Bux]

Grant wrote:

>>what kernel are you using ? in 2.6 yes definetively there is a way, also
>>some of the 2.4 series kernel was swapping too much, google for "swappines".
>>normally it's set to 60, lower that value and your swap will empty.
>>if you have a pre 2.4.19 kernel upgrade to a newer one also of the same
>>series.
>>    
>>
>
>I'm using 2.4.27-hardened, and I think swappiness is only available
>for 2.6.  Maybe it's time to move to 2.6, but this is a commercial
>server so I'm not so sure.
>
>  
>
If it's a commercial server and it's load under the 40%
it's easy, buy a disk and build a chrooted gentoo with hardened-dev-sources
then schedule a downtime of some hour to make the change.
Before to do this anyway it's better to figure out what really is the 
problem, try to isolate it,
reboot, cat somewhere that output of "top -b -n1" and diff one hour 
after and one day after,
try also "ps ax -eo user,vsize,size,command" and "man ps" I bet you will 
find what is strange

btw 2.4.27 it's *not* an old kernel.

>>Also it's not true with older kernels that running out of memory it's
>>safe, *really* it's not if you are also writing a lot to a disk.
>>    
>>
>
>Even if there is no crashing and the program just fails to load. 
>That's no good either.  I don't think disabling swap is the solution
>at all (we agree).
>
>- Grant
>
>--
>gentoo-user@gentoo.org mailing list
>
>
>  
>


--
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] More memory?

[Grant]

> > Maybe a better way to phrase my question is:  Is it possible to set my
> > server up so it will use swap when it needs it and then free it back
> > up when it doesn't need it anymore?  What makes me think that is
> > necessary is the fact that I see a very snappy response when browsing
> > my site after a fresh reboot.  After it's been up for awhile, the swap
> > starts to fill and it slows down.  Rebooting clears out the swap and
> > the snaps return.
> 
> Actually, that is the default behaviour.

I'm now using swap again for the first time since my last reboot. 
It's currently at 1036k, but that is guaranteed to keep increasing.

Here's what I don't understand.

total       used       free     shared    buffers     cached
Mem:           978        731        246          0        164        226
-/+ buffers/cache:        340        637
Swap:          494          1        493

If I'm reading that right, I'm only *using* using 340MB.  Why doesn't
the system get rid of some of the inactive stuff in memory so I don't
have to use more and more swap and slow down my system?

- Grant

> Uwe

--
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] More memory?

[Jerry McBride]

On Monday 06 December 2004 11:16 am, Grant wrote:
> > > Maybe a better way to phrase my question is:  Is it possible to set my
> > > server up so it will use swap when it needs it and then free it back
> > > up when it doesn't need it anymore?  What makes me think that is
> > > necessary is the fact that I see a very snappy response when browsing
> > > my site after a fresh reboot.  After it's been up for awhile, the swap
> > > starts to fill and it slows down.  Rebooting clears out the swap and
> > > the snaps return.
> >
> > Actually, that is the default behaviour.
>
> I'm now using swap again for the first time since my last reboot.
> It's currently at 1036k, but that is guaranteed to keep increasing.
>
> Here's what I don't understand.
>
> total       used       free     shared    buffers     cached
> Mem:           978        731        246          0        164        226
> -/+ buffers/cache:        340        637
> Swap:          494          1        493
>
> If I'm reading that right, I'm only *using* using 340MB.  Why doesn't
> the system get rid of some of the inactive stuff in memory so I don't
> have to use more and more swap and slow down my system?
>
>

Did I miss a previous post? What you don't see is that you are ONLY useing 1 
meg of swapper file... What slow down does that cause?

As for making the kernel use more of the free space in cache.... it will... 
when it needs it. As you load more and more onto the task heap, the buffers 
free space will begin to deminish... same for the free space in the memory 
pool.

As for clearing out swap.... instead of rebooting a perfectly running linux 
box, how about SWAPON/SWAPOFF? See the man pages...
-- 

******************************************************************************
                     Registered Linux User Number 185956
              FSF Associate Member number 2340 since 05/20/2004
             Join me in chat at #linux-users on irc.freenode.net
    Buy an Xbox for $149.00, run linux on it and Microsoft loses $150.00!
    12:10pm  up 57 days, 19:56,  8 users,  load average: 0.10, 0.14, 0.09

--
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] More memory?

[Billy]

Grant wrote:
> If I'm reading that right, I'm only *using* using 340MB.  Why doesn't
> the system get rid of some of the inactive stuff in memory so I don't
> have to use more and more swap and slow down my system?

that's what swap does. It moves inactive stuff onto disk so :

* apps have more ram - thus faster
* the OS can cache more data - thus faster

I'll post some of my systems, so you can see their swap usage. The 
machines with the most IO utilize the most swap. The one that serves 
purely dynamic pages uses very little swap - and it's apache server has 
fewer dynamic modules compiled in so that will help on memory foot print 
usage. However, swap usage isn't a bad thing. In fact, it most cases 
it's a really, really good thing.

[Terminal Server] gentoo running 2.6.x kernel. I'm doing an emerge, and 
I have 10 physically different people logged on the system doing work. 
Running on dual 1.6 Ghz opertons. High load average is mostly disk IO 
due to emerge.

  11:56:03 up 18 days, 12:04, 25 users,  load average: 7.65, 7.23, 5.63
           total       used       free     shared    buffers     cached
Mem:       2009       1591        417          0         57        341
-/+ buffers/cache:    1192        816
Swap:      4094       1218       2876

However, since I'm using the ck-sources, and most of the apps are 
interactive, no one even notices I'm doing anything funny.

[Web Server] suse running 2.4.x kernel, a peak of about 15 hits per 
second, and peak traffic of 670kbps. does not act slow. Running on 1.2 
Ghz P3. Also email virus scanner, SMTP and POP - I don't have peak usage 
on those services.

  11:52am  up 274 days, 18:38,  1 user,  load average: 0.00, 0.02, 0.02
           total       used       free     shared    buffers     cached
Mem:       1005        929         75          0        183        319
-/+ buffers/cache:     426        578
Swap:      1027        698        328

[secure commerce server] redhat running 2.4.x kernel, a peak of about 12 
hits per minute, and about 130 kbps. Also not slow, all pages dynamic. 
Running on 1.2 Ghz P3. Database on another system.

  12:17pm  up 67 days,  3:08,  2 users,  load average: 0.01, 0.06, 0.06
           total       used       free     shared    buffers     cached
Mem:        439        414         24          0        159        117
-/+ buffers/cache:     137        301
Swap:       509         37        472


--
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] More memory?

[Uwe Thiem]

On Monday 06 December 2004 18:16, Grant wrote:

> I'm now using swap again for the first time since my last reboot.
> It's currently at 1036k, but that is guaranteed to keep increasing.
>
> Here's what I don't understand.
>
> total       used       free     shared    buffers     cached
> Mem:           978        731        246          0        164        226
> -/+ buffers/cache:        340        637
> Swap:          494          1        493
>
> If I'm reading that right, I'm only *using* using 340MB.  Why doesn't
> the system get rid of some of the inactive stuff in memory so I don't
> have to use more and more swap and slow down my system?

Not exactly. You are misinterpreting the output.

You are using about 341MB = (731 - 164 - 226) MB. About 1MB of your swap is 
used for whatever reason. Maybe there was a short peak time (memory-wise) 
when some piece of software got swapped out (actually, paged out) and hasn't 
been used yet since.

If swap space usage goes up over time monitor your processes using tools like 
top, ps and such to check which process eats up memory. Then do as I 
originally suggested.

BTW, keep in mind that tools like top, ps, free,... are notoriously *wrong* on 
linux. They give you an idea of memory usage but don't take their exact 
numbers for truth. Question to the rest of the crowd: Did that change with 
kernel 2.6? 

Uwe

-- 
Alternative phrasing of the First Law of Thermodynamics:
If you eat it, and you don't burn it off, you'll sit on it.

http://www.uwix.iway.na (last updated: 20.06.2004)

--
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Re: More memory?

[Simon Windsor]

Hi

SWAP is not necessarily a bad thing. It is mealy a simple way of moving
unused code from memory so that it can be better used, ie disk caching.

Swap can be bad if you have insufficient memory to run your processes.

If your system has lots of disk activity, the OS will assign memory to
cache the disk. This may where you memory is going.

You say that your system is using perl. What else? Apache? MySQL/PgSQL?
etc ...

Please more info ..

All the best

Simon

On Fri, 2004-12-03 at 20:45 -0600, Gabriel M. Beddingfield wrote:
> Grant wrote:
> 
> >> If you don't want it to swap try 'man swapoff' and 'man fstab'
> > 
> > It seems like swap is there for a reason so I don't want to disable
> > it.  What I do want to be able to do is run my system in a manner that
> [snip]
> 
> Just turn it off.  You don't need it unless you run out of RAM.  Swap is
> just for "virtual RAM."  You don't *need* virtual RAM unless you run out of
> physical RAM.
> 
> You may still need 2GB RAM in the future... but turning off swap won't hurt
> anything.  Using 'swapoff' and 'swapon' will let you try-before-you-buy. 
> Editing fstab (comment out the line with the swap partition) will make it
> so for every boot-up.
> 
> What happens when you run out of RAM?  The application doesn't start up.  It
> doesn't crash the system.  Once when attempting a Gentoo install on a 486 I
> ran out of memory when using tar.  I didn't crash, it just didn't untar my
> file.  I turned on swap, and everything worked just fine.
> 
-- 
Simon Windsor

Eml: simon.windsor@cornfield.org.uk
Tel: 01454 617689
Mob: 07960 321599


--
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Re: More memory?

[Grant]

> SWAP is not necessarily a bad thing. It is mealy a simple way of moving
> unused code from memory so that it can be better used, ie disk caching.
> 
> Swap can be bad if you have insufficient memory to run your processes.
> 
> If your system has lots of disk activity, the OS will assign memory to
> cache the disk. This may where you memory is going.
> 
> You say that your system is using perl. What else? Apache? MySQL/PgSQL?
> etc ...

I'm running interchange (http://www.icdevgroup.org), apache2,
mod_perl2, and mysql.  I run courier-imap and postfix for email, but
I'm the only user.  I use php only for squirrelmail.

- Grant

> All the best
> 
> Simon

--
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Re: More memory?

[Tim Igoe]

Grant wrote:

>>SWAP is not necessarily a bad thing. It is mealy a simple way of moving
>>unused code from memory so that it can be better used, ie disk caching.
>>
>>Swap can be bad if you have insufficient memory to run your processes.
>>
>>If your system has lots of disk activity, the OS will assign memory to
>>cache the disk. This may where you memory is going.
>>
>>You say that your system is using perl. What else? Apache? MySQL/PgSQL?
>>etc ...
>>    
>>
>
>I'm running interchange (http://www.icdevgroup.org), apache2,
>mod_perl2, and mysql.  I run courier-imap and postfix for email, but
>I'm the only user.  I use php only for squirrelmail.
>
>  
>
Just a check here, it is the -latest- version of Squirrelmail. There was 
one version that would cause my server to use -all- its RAM and swap 
(and eventually just reboot) when i clicked reply (memory leak 
somewhere) which has since been sorted.

Just a thought.

Tim

>- Grant
>
>  
>
>>All the best
>>
>>Simon
>>    
>>
>
>--
>gentoo-user@gentoo.org mailing list
>
>  
>


--
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] Re: More memory?

[Grant]

> >>SWAP is not necessarily a bad thing. It is mealy a simple way of moving
> >>unused code from memory so that it can be better used, ie disk caching.
> >>
> >>Swap can be bad if you have insufficient memory to run your processes.
> >>
> >>If your system has lots of disk activity, the OS will assign memory to
> >>cache the disk. This may where you memory is going.
> >>
> >>You say that your system is using perl. What else? Apache? MySQL/PgSQL?
> >>etc ...
> >>
> >>
> >
> >I'm running interchange (http://www.icdevgroup.org), apache2,
> >mod_perl2, and mysql.  I run courier-imap and postfix for email, but
> >I'm the only user.  I use php only for squirrelmail.
> >
> >
> >
> Just a check here, it is the -latest- version of Squirrelmail. There was
> one version that would cause my server to use -all- its RAM and swap
> (and eventually just reboot) when i clicked reply (memory leak
> somewhere) which has since been sorted.
> 

I just synced and it looks like I'm not using the latest Squirrelmail.
 The output is a little strange though.  I'm going to start another
thread about that.  Thanks for the tip!

- Grant

> Just a thought.
> 
> Tim

--
gentoo-user@gentoo.org mailing list