public inbox for gentoo-user@lists.gentoo.org
 help / color / mirror / Atom feed
From: brettrsears@gmail.com
To: gentoo-user@lists.gentoo.org
Subject: Re: [gentoo-user] strange TCP timeout errors
Date: Wed, 7 Oct 2015 19:42:05 +0000	[thread overview]
Message-ID: <588564942-1444246926-cardhu_decombobulator_blackberry.rim.net-2145807130-@b25.c2.bise6.blackberry> (raw)
In-Reply-To: <561566EE.9000507@gmail.com>

YyyyYYuIIIIIU
Sent from my Verizon Wireless BlackBerry

-----Original Message-----
From: Alan McKinnon <alan.mckinnon@gmail.com>
Date: Wed, 7 Oct 2015 20:39:42 
To: <gentoo-user@lists.gentoo.org>
Reply-to: gentoo-user@lists.gentoo.org
Subject: Re: [gentoo-user] strange TCP timeout errors

On 07/10/2015 17:55, Grant wrote:
>>>>>> I've attached a PNG from Munin showing the TCP timeout errors on my
>>>>>> Gentoo server over the past month.  The data is expressed in timeouts
>>>>>> per second and that rate is shown to be steadily increasing over the
>>>>>> past month.  That seems strange to me.  Munin doesn't show any other
>>>>>> data point increasing like this over the time period.  Any ideas?
>>>>>>
>>>>>> - Grant
>>>>>>
>>>>>
>>>>> weird - does it reset on an interface restart or reboot?
>>>>
>>>> this would be my test #1
>>>
>>>
>>> I rebooted and the rate of errors has dropped off to almost nothing.
>>>
>>>
>>>>> Can you verify its not an artefact within munin (how?)
>>>>
>>>> In theory, a misconfigured graph can do this. Munin can draw many
>>>> different types of graph, including cumulative values. Even for a data
>>>> type like this which is X events per unit time, if you tell munin to add
>>>> them all up, it will do so and graph it.
>>>>
>>>> Qucik test is to look at the graph config.
>>>
>>>
>>> This graph lives in the "network" section of the munin web interface.
>>> There is no matching section in /etc/munin/plugin-conf.d/munin-node so
>>> it should be be using the default config.
>>>
>>> Any ideas based on this new info?
>>
>> A few :-)
>>
>>
>> I can't find the plugin that delivers that graph though. Maybe I just
>> don't have it, maybe it comes from contrib/
>>
>> What's your USE for munin?
> 
> 
> USE="apache cgi http mysql ssl syslog -asterisk -dhcpd -doc -ipmi
> -ipv6 -irc -java -memcached -minimal -postgres (-selinux) {-test}"
> 
> 
>> What do you have in "ls -al /etc/munin/plugins/"  ?


It's as I thought - your data is accurate but rrd has been given a
completely wrong method to derive the graphs.

Munin graphs for section "Network" do not have to be in a file called
"network" - it's just a category and the plugin defines what web-page
section it must be in. In your case, the relevant plugin is
netstat_multi which doesn't often get installed. It's data source is
"netstat -s" so grep that output for "timeout" to see it.

Timeouts are cumulative counters, they do not get less till they wrap
around. So to scale them, the plugin gets the rrd file to subtract
previous reading from current reading and divide by the time interval to
get the timeouts/sec. This is all done inside rrd when the data files
are updated (it's quite a lot of magic)

That plugin sets the graph type to DERIVE
(/etc/munin/plugins/netstat_multi around line 190. I feel it should be
GAUGE or COUNTER.

The proper reference on rrd is
http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html
and the munin docs are
https://munin.readthedocs.org/en/latest/index.html

You must edit the plugin file and IIRC recreate the rrd, you will lose
all past info (can't be helped).


[snip ls output]


> P.S. Any other good plugins you'd recommend?

http://gallery.munin-monitoring.org/

Monitoring is highly site-specific so recommendations aren't usually
worth much, but that gallery has LOTS of contributed plugins

-- 
Alan McKinnon
alan.mckinnon@gmail.com



  reply	other threads:[~2015-10-07 19:42 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-10-05 17:35 [gentoo-user] strange TCP timeout errors Grant
2015-10-05 22:57 ` Bill Kenworthy
2015-10-05 23:26   ` Alan McKinnon
2015-10-07 12:58     ` Grant
2015-10-07 14:22       ` Alan McKinnon
2015-10-07 15:55         ` Grant
2015-10-07 18:39           ` Alan McKinnon
2015-10-07 19:42             ` brettrsears [this message]
2015-10-07 22:25               ` Alan McKinnon
2015-10-09 14:15             ` Grant

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=588564942-1444246926-cardhu_decombobulator_blackberry.rim.net-2145807130-@b25.c2.bise6.blackberry \
    --to=brettrsears@gmail.com \
    --cc=gentoo-user@lists.gentoo.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox