Re: [gentoo-user] gentoo-box stopping services

Re: [gentoo-user] gentoo-box stopping services

[Stefan G. Weichinger]

additional thoughts:

Am 20.04.2010 14:01, schrieb Stefan G. Weichinger:

> I thought maybe the NIC has a problem?
> 
> Ethernet controller: Realtek Semiconductor Co., Ltd.
> RTL-8139/8139C/8139C+ (rev 10)
> 
> but as it doesn't lose its IP and config I think that is not the case here?

I noticed that both relevant kernel-modules were loaded as noted here:

http://www.mail-archive.com/netdev@vger.kernel.org/msg60241.html

I was able to "rmmod 8139cp" without losing network ...

This might just be a cosmetic issue, I just wanted to add that info.

Stefan

Re: [gentoo-user] gentoo-box stopping services

[Stroller]

On 20 Apr 2010, at 13:01, Stefan G. Weichinger wrote:
> ...
> One of my customers runs an old P3 as a mail-gateway and samba-server
> (yeah, I know ...) behind his firewall ...
>
> They simply don't want to swap hardware, they are happy ... until the
> following started to happen every week or so:

You emphasise how old the hardware is, but this really isn't a  
problem. As you say, one increasingly fears the death of a system  
which is getting so old, but I have two systems nearly as old running  
for years without hardware problems.

The questions I must ask are:

- How uptodate is the Gentoo software?
- Do you run updates regularly?
- Did you run any shortly before this started occurring?
- Have you run revdep-rebuild and stuff?
- Does the system have sufficient swap?

Stroller.
  

Re: [gentoo-user] gentoo-box stopping services

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 1248 bytes --]

On Tuesday 20 April 2010 13:01:52 Stefan G. Weichinger wrote:
> greetings, gentoo-users ...
> 
> One of my customers runs an old P3 as a mail-gateway and samba-server
> (yeah, I know ...) behind his firewall ...
> 
> They simply don't want to swap hardware, they are happy ... until the
> following started to happen every week or so:
> 
> The server goes offline, you can ping it OK but services like smbd,
> postfix, sshd all are not reachable anymore.
> 
> I "see" the open ports with nmap from my machine ... but they are shown
> as closed.
> 
> When the guy there restarts sshd on the server itself I am able to login
> again without a problem. There are no bad messages in dmesg and/or
> /var/log/messages.
> 
> But this seems to be related to the fact that syslog-ng also is inactive
> then ... so who should log errors ... ?
> 
> --
> 
> I thought maybe the NIC has a problem?
> 
> Ethernet controller: Realtek Semiconductor Co., Ltd.
> RTL-8139/8139C/8139C+ (rev 10)
> 
> but as it doesn't lose its IP and config I think that is not the case here?

Have you looked at dmesg in case there is something there that the kernel's 
spewed out?  Also, you haven't run out of space?  df -h 

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [gentoo-user] gentoo-box stopping services

[Stefan G. Weichinger]

Am 21.04.2010 00:07, schrieb Stroller:

> You emphasise how old the hardware is, but this really isn't a
> problem. As you say, one increasingly fears the death of a system
> which is getting so old, but I have two systems nearly as old running
> for years without hardware problems.

Yes, it does what it should do.

It's just that it gets more probable to have some strange and hidden
defects *maybe*.

But it would show other symptoms then, I assume.

> The questions I must ask are:
> 
> - How uptodate is the Gentoo software? - Do you run updates
> regularly? - Did you run any shortly before this started occurring? -
> Have you run revdep-rebuild and stuff? - Does the system have
> sufficient swap?

swap should be OK:

# free -m
             total       used       free     shared    buffers     cached
Mem:         501        484         17          0         16        241
-/+ buffers/cache:        226        275
Swap:          494        288        205

OK, a bit more RAM wouldn't hurt here.
But I am compiling stuff right now.

-

ad updates:

I was rather defensive there, I have to admit ..
Just like "never touch a running system" ...

I updated the relevant pkgs like postfix, samba, clamav ... but there
are around 60 pkgs to update today.

Stuff like glibc, udev, pam .... I will apply them now step by step ...

revdep-rebuild was OK before, I had checked that after the last updates
a few days ago.

My first idea was to upgrade the kernel to maybe catch some relevant
fixes, that was about a week ago.

There was no specific update triggering this, in fact I hadn't touched
that box for weeks when the responsible man called me to tell me about
the new problems ...

Stefan

Re: [gentoo-user] gentoo-box stopping services

[Stroller]

On 21 Apr 2010, at 08:28, Stefan G. Weichinger wrote:
>> ... Does the system have sufficient swap?
>
> swap should be OK:
>
> # free -m
>             total       used       free     shared    buffers      
> cached
> Mem:         501        484         17          0         16         
> 241
> -/+ buffers/cache:        226        275
> Swap:          494        288        205
>
> OK, a bit more RAM wouldn't hurt here.
> But I am compiling stuff right now.

I would add more.

Services mysteriously dying, surely that could be because the kernel  
is killing them off due to an out-of-memory condition?

Maybe you have changed to a different compiler version in the past,  
and this creates larger binaries?

That seems a bit tenuous, I don't know, but you can use a swapfile on  
Linux (i.e. you can add to the current swap without having to create  
an additional partition), and I doubt if it is hard to set up.

> ad updates:
>
> I was rather defensive there, I have to admit ..
> Just like "never touch a running system" ...

I seem to get into more trouble when I'm cautious with updates than I  
do when I just let 'em rip as often as possible. More frequent updates  
means fewer updates.

Stroller.

Re: [gentoo-user] gentoo-box stopping services

[Stefan G. Weichinger]

Am 21.04.2010 09:18, schrieb Mick:

> Have you looked at dmesg in case there is something there that the kernel's 
> spewed out?  Also, you haven't run out of space?  df -h 

checked both before even posting here:

nothing stinky in "dmesg", "df -h" shows enough free space on the
partitions.

Thanks, S

Re: [gentoo-user] gentoo-box stopping services

[Stefan G. Weichinger]

Am 21.04.2010 11:09, schrieb Stroller:

>> OK, a bit more RAM wouldn't hurt here.
>> But I am compiling stuff right now.
> 
> I would add more.
> 
> Services mysteriously dying, surely that could be because the kernel is
> killing them off due to an out-of-memory condition?

Shouldn't the kernel *swap* then ?

I suggested adding RAM there, sure ...

> Maybe you have changed to a different compiler version in the past, and
> this creates larger binaries?

current version of gcc: 4.3.2-r3 (04:13:52 18.04.2009)

So it is about one year old, no changes since then. The symptoms only
started some week ago or two ...

I rather wonder about that f-secure scanner "fsav" ... I had to manually
fiddle it onto the system and it takes quite much ram and cpu.

But it is installed since june 2009 as well.

> I seem to get into more trouble when I'm cautious with updates than I do
> when I just let 'em rip as often as possible. More frequent updates
> means fewer updates.

I have to find some sweet spot here. I have quite many servers out there
at customers sites ... noone so far pays me for regularly emerging world
there ... this has to change, you are right ;-)

Stefan

Re: [gentoo-user] gentoo-box stopping services

[Alan McKinnon]

On Thursday 22 April 2010 17:31:33 Stefan G. Weichinger wrote:
> Am 21.04.2010 11:09, schrieb Stroller:
> >> OK, a bit more RAM wouldn't hurt here.
> >> But I am compiling stuff right now.
> > 
> > I would add more.
> > 
> > Services mysteriously dying, surely that could be because the kernel is
> > killing them off due to an out-of-memory condition?
> 
> Shouldn't the kernel *swap* then ?

No, the OOM killer kicks in when the kernel has no more virtual memory, 
including swap. Either way, more RAM is the answer. Or fidn the app with the 
memory leak if you are unlucky enough to have one of those running around.

[snip]

-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] gentoo-box stopping services

[Stefan G. Weichinger]

Am 22.04.2010 17:50, schrieb Alan McKinnon:

>> Shouldn't the kernel *swap* then ?
> 
> No, the OOM killer kicks in when the kernel has no more virtual memory, 
> including swap. Either way, more RAM is the answer. Or fidn the app with the 
> memory leak if you are unlucky enough to have one of those running around.

The added swapfile with one GB won't help here for a start?

S

Re: [gentoo-user] gentoo-box stopping services

[Alan McKinnon]

On Thursday 22 April 2010 18:24:08 Stefan G. Weichinger wrote:
> Am 22.04.2010 17:50, schrieb Alan McKinnon:
> >> Shouldn't the kernel *swap* then ?
> > 
> > No, the OOM killer kicks in when the kernel has no more virtual memory,
> > including swap. Either way, more RAM is the answer. Or fidn the app with
> > the memory leak if you are unlucky enough to have one of those running
> > around.
> 
> The added swapfile with one GB won't help here for a start?

It will certainly help. If your core problem is simply not enough RAM, then 1G 
more might be all you need. You'd have to run checks and do some monitoring to 
see if performance is affected. 

I haven't followed the full thread so I don't know what you are running; and 
some daemons perform really badly if they have to touch swap. Apache for 
example, a busy MTA for another - disks are thousands of times slower than 
RAM, so if a webserver has to swap memory back in from disk, it almost 
instantly brings the server to a grinding halt.

On my web and mail servers I have no swap at all, they do have lots and lots 
of RAM; my Sybase database servers have enormous amounts of swap. Each server 
has been profiled so it is set up to be as close to ideal as I can determine.

-- 
alan dot mckinnon at gmail dot com

[gentoo-user] Re: gentoo-box stopping services

[walt]

On 04/22/2010 09:52 AM, Alan McKinnon wrote:

> On my web and mail servers I have no swap at all, they do have lots and lots
> of RAM; my Sybase database servers have enormous amounts of swap...

Hm.  Does that mean your database servers are allowed to be slower than the web
and mail servers?

Re: [gentoo-user] Re: gentoo-box stopping services

[Alan McKinnon]

On Thursday 22 April 2010 23:16:56 walt wrote:
> On 04/22/2010 09:52 AM, Alan McKinnon wrote:
> > On my web and mail servers I have no swap at all, they do have lots and
> > lots of RAM; my Sybase database servers have enormous amounts of swap...
> 
> Hm.  Does that mean your database servers are allowed to be slower than the
> web and mail servers?

No, it means that the database software intelligently handles swap.

An enterprise grade db like this one will load many megabytes of data in huge 
chunks on and off disk in one action. If you are dealing with tables 
containing many millions of rows, you really do want this behaviour. Without 
swap, the db will still load and unload many megs of data at a time on and off 
disk, so there is no appreciable downgrade of performance, but I do have the 
benefit of keeping data in virtual memory for longer using swap. The database 
code pages are never swapped by design - doing that is just plan stupid.

The web and mail servers on the other hand deal with millions of small files, 
on the order of 10k each. Allowing that to swap is catastrophic.


-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] gentoo-box stopping services

[Stroller]

On 22 Apr 2010, at 17:24, Stefan G. Weichinger wrote:

> Am 22.04.2010 17:50, schrieb Alan McKinnon:
>
>>> Shouldn't the kernel *swap* then ?
>>
>> No, the OOM killer kicks in when the kernel has no more virtual  
>> memory,
>> including swap. Either way, more RAM is the answer. Or fidn the app  
>> with the
>> memory leak if you are unlucky enough to have one of those running  
>> around.
>
> The added swapfile with one GB won't help here for a start?

Yes, it will. If it's running out of RAM+swap, then more swap will help.

Stroller.

Re: [gentoo-user] gentoo-box stopping services

[Stefan G. Weichinger]

Am 23.04.2010 05:28, schrieb Stroller:

>> The added swapfile with one GB won't help here for a start?
> 
> Yes, it will. If it's running out of RAM+swap, then more swap will
> help.

I watch the system now, only 6 MB of RAM free now ... the AV-scanners
grab the most ... I already deactivated f-secure (fsavd) because it
timed out while scanning a 10k text-mail :-(

Gotta get some RAM, yes.

S