[gentoo-user] syslog-ng filtering

[gentoo-user] syslog-ng filtering

[Ralph Slooten]

[-- Attachment #1: Type: text/plain, Size: 1154 bytes --]

Hi all,

Has anyone here worked out how to filter out syslog messages using syslog-ng
v3? The old syntax doesn't work (well complains bitterly about performance
and says to use regex), and no matter what I try I cannot get the new syntax
to work :-/ I have a syslog-ng server which logs to MySQL for multiple
clients in a network, however the database just keeps growing with
irrelevant data I'd prefer to just quietly ignore on the server side.

I'm trying to filter out (exclude) messages such as:
  (root) CMD (/root/bin/vmware-checker)
and
  (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons )

==============
filter myfilter {
        not match("regex" value("\/usr\/sbin\/run-crons"))
        and not match("regex" value("vmware-checker"));
}
log {
        source(src);
        source(remote);
        filter(myfilter);
        destination(d_mysql);
};
===============

However they just keep coming through the filter (ie: not matching the "not
match" filter). I've tried escaping the slashes, not escaping them ... even
partial words, but I obviously am missing something somewhere.

Anyone have any ideas?

Thanks in advance,
Ralph

[-- Attachment #2: Type: text/html, Size: 1450 bytes --]

Re: [gentoo-user] syslog-ng filtering

[Roy Wright]

On Mar 16, 2010, at 6:22 PM, Ralph Slooten wrote:

> Hi all,
> 
> Has anyone here worked out how to filter out syslog messages using syslog-ng v3? The old syntax doesn't work (well complains bitterly about performance and says to use regex), and no matter what I try I cannot get the new syntax to work :-/ I have a syslog-ng server which logs to MySQL for multiple clients in a network, however the database just keeps growing with irrelevant data I'd prefer to just quietly ignore on the server side. 
> 

I just started with the example at:

http://en.gentoo-wiki.com/wiki/Syslog-ng

HTH,
Roy

Re: [gentoo-user] syslog-ng filtering

[Ralph Slooten]

[-- Attachment #1: Type: text/plain, Size: 811 bytes --]

On 17 March 2010 13:00, Roy Wright <roy@wright.org> wrote:
>
> I just started with the example at:
> http://en.gentoo-wiki.com/wiki/Syslog-ng
>
> HTH,
> Roy

Thanks Roy, however they have the same syntax which isn't working on my
side.

filter f_shorewall { not match("regex" value("Shorewall")); }


I just tried a single rule (to make sure it wasn't my syntax):

filter killVmMessages {
        not match("regex" value("vmware-checker"));
};

yet the "(root) CMD (/root/bin/vmware-checker)" messages still go through?!

log {
        source(src);
        source(remote);
        filter(myfilter);
        filter(killVmMessages);
        destination(d_mysql);
};

I'm really stumped here. All other filters (non regex) works fine though,
such as facility() & host().

Are you able to filter by content?

Ralph

[-- Attachment #2: Type: text/html, Size: 1413 bytes --]

Re: [gentoo-user] syslog-ng filtering

[Fred Leon]

[-- Attachment #1: Type: text/plain, Size: 1494 bytes --]

Ralph Slooten <axllent@gmail.com> a écrit :

> On 17 March 2010 13:00, Roy Wright <roy@wright.org> wrote:
>>
>> I just started with the example at:
>> http://en.gentoo-wiki.com/wiki/Syslog-ng
>>
>> HTH,
>> Roy
>
> Thanks Roy, however they have the same syntax which isn't working on my
> side.
>
> filter f_shorewall { not match("regex" value("Shorewall")); }
>
>
> I just tried a single rule (to make sure it wasn't my syntax):
>
> filter killVmMessages {
>         not match("regex" value("vmware-checker"));
> };
>
> yet the "(root) CMD (/root/bin/vmware-checker)" messages still go through?!
>
> log {
>         source(src);
>         source(remote);
>         filter(myfilter);
>         filter(killVmMessages);
>         destination(d_mysql);
> };
>
> I'm really stumped here. All other filters (non regex) works fine though,
> such as facility() & host().
>
> Are you able to filter by content?
>
> Ralph
>

Perhaps you could try this which is working for me and let me filter  
all messages coming from iptables:

# firewall logging
destination iptables { file("/var/log/firewall/iptables.log"); };
filter f_iptables { message("iptables"); };
log { source(s_all); filter(f_iptables); destination(iptables); };

# all messages coming from kern
destination df_kern { file("/var/log/system/kern.log" ); };
filter f_kern { facility(kern) and not filter(f_iptables); };
log { source(s_all); filter(f_kern);destination(df_kern); };

Fred

[-- Attachment #2: Signature numérique PGP --]
[-- Type: application/pgp-signature, Size: 230 bytes --]

Re: [gentoo-user] syslog-ng filtering

[Alan McKinnon]

On Wednesday 17 March 2010 01:22:59 Ralph Slooten wrote:
> Hi all,
> 
> Has anyone here worked out how to filter out syslog messages using
> syslog-ng v3? The old syntax doesn't work (well complains bitterly about
> performance and says to use regex), and no matter what I try I cannot get
> the new syntax to work :-/ I have a syslog-ng server which logs to MySQL
> for multiple clients in a network, however the database just keeps growing
> with irrelevant data I'd prefer to just quietly ignore on the server side.
> 
> I'm trying to filter out (exclude) messages such as:
>   (root) CMD (/root/bin/vmware-checker)
> and
>   (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons )
> 
> ==============
> filter myfilter {
>         not match("regex" value("\/usr\/sbin\/run-crons"))
>         and not match("regex" value("vmware-checker"));
> }

Hah! this caught me out too.

The value of "value" cannot be anything arbitrary - syslog-ng has no clue what 
you mean. The value is a field name, either a pre-defined one, or something 
you defined using a parser. The docs are ambiguous on this, it's not clear 
that the supplied values are abstracts. You are truing to search for the 
string "regex" in a field called /usr/bin/vmware-checker.

Which obviously will not work.

I think you want:

match("\/usr\/sbin\/run-crons" value "MESSAGE")

Note that it is MESSAGE. You want the field name, not it's dereferenced value.



> log {
>         source(src);
>         source(remote);
>         filter(myfilter);
>         destination(d_mysql);
> };
> ===============
> 
> However they just keep coming through the filter (ie: not matching the "not
> match" filter). I've tried escaping the slashes, not escaping them ... even
> partial words, but I obviously am missing something somewhere.
> 
> Anyone have any ideas?
> 
> Thanks in advance,
> Ralph

-- 
alan dot mckinnon at gmail dot com

[gentoo-user] Re: syslog-ng filtering

[Robert Fekete]

That's right, the value() parameter specifies which part of the message to
check. This helps to cut down the performance cost of filtering, because there
is no need to process the entire message if you are filtering on the program
name, for example. 

Also, check the syslog-ng Administrator Guide
(http://www.balabit.com/support/documentation/?product=syslog-ng&type=all&language[en]=en&)
if you run into problems. And let me know if you do not find something that
should be in the guide so I can add it some time.

Regards, 

Robert Fekete
maintainer of the syslog-ng documentation
 

Re: [gentoo-user] syslog-ng filtering

[Ralph Slooten]

[-- Attachment #1: Type: text/plain, Size: 2899 bytes --]

Fantastic, you hit the nail right on the head! Works like a charm now.

Now I'm wondering how it is you found out that it was this way and not the
other? Robert maintains the documentation for rsync which I did look at, but
with 225 pages I wasn't able to find this useful piece of information. Man
syslog-ng.conf does not explain it either, in fact I searched Google and
found several "tutorials", none mentioning this ;-)

Maybe I'm the idiot here, however I thought that this was a common way of
getting rid of unwanted crud from the syslog?

Also, I just read the gentoo-wiki site page again and it says :

filter f_shorewall { not match("regex" value("Shorewall")); };  #
Filter everything except regex keyword Shorewall

Surely this is the exact same mistake I made? Either that or I'm reading it
wrong....





On 17 March 2010 23:39, Alan McKinnon <alan.mckinnon@gmail.com> wrote:

> On Wednesday 17 March 2010 01:22:59 Ralph Slooten wrote:
> > Hi all,
> >
> > Has anyone here worked out how to filter out syslog messages using
> > syslog-ng v3? The old syntax doesn't work (well complains bitterly about
> > performance and says to use regex), and no matter what I try I cannot get
> > the new syntax to work :-/ I have a syslog-ng server which logs to MySQL
> > for multiple clients in a network, however the database just keeps
> growing
> > with irrelevant data I'd prefer to just quietly ignore on the server
> side.
> >
> > I'm trying to filter out (exclude) messages such as:
> >   (root) CMD (/root/bin/vmware-checker)
> > and
> >   (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons )
> >
> > ==============
> > filter myfilter {
> >         not match("regex" value("\/usr\/sbin\/run-crons"))
> >         and not match("regex" value("vmware-checker"));
> > }
>
> Hah! this caught me out too.
>
> The value of "value" cannot be anything arbitrary - syslog-ng has no clue
> what
> you mean. The value is a field name, either a pre-defined one, or something
> you defined using a parser. The docs are ambiguous on this, it's not clear
> that the supplied values are abstracts. You are truing to search for the
> string "regex" in a field called /usr/bin/vmware-checker.
>
> Which obviously will not work.
>
> I think you want:
>
> match("\/usr\/sbin\/run-crons" value "MESSAGE")
>
> Note that it is MESSAGE. You want the field name, not it's dereferenced
> value.
>
>
>
> > log {
> >         source(src);
> >         source(remote);
> >         filter(myfilter);
> >         destination(d_mysql);
> > };
> > ===============
> >
> > However they just keep coming through the filter (ie: not matching the
> "not
> > match" filter). I've tried escaping the slashes, not escaping them ...
> even
> > partial words, but I obviously am missing something somewhere.
> >
> > Anyone have any ideas?
> >
> > Thanks in advance,
> > Ralph
>
> --
> alan dot mckinnon at gmail dot com
>

[-- Attachment #2: Type: text/html, Size: 4480 bytes --]

Re: [gentoo-user] syslog-ng filtering

[Alan McKinnon]

On Wednesday 17 March 2010 22:16:20 Ralph Slooten wrote:
> Fantastic, you hit the nail right on the head! Works like a charm now.
> 
> Now I'm wondering how it is you found out that it was this way and not the
> other? Robert maintains the documentation for rsync which I did look at,
> but with 225 pages I wasn't able to find this useful piece of information.
> Man syslog-ng.conf does not explain it either, in fact I searched Google
> and found several "tutorials", none mentioning this ;-)

I read documentation, man pages and google all day every day, some things just 
get intuitive :-)

Seriously though, there are a few hints. Syslog-ng's config file format was 
written by programmers for programmers to be understood by programmers. That 
may not have been the stated intent, but it is how things turned out. The 
syntax is exactly that of C, all the way down to braces and statement 
terminators. So, when reading the docs, I flicked the switch that puts my 
brain in C-mode.

Also, there's an example in the admin guide pdf chapter 3 "Configuring syslog-
ng", something like:

match("string" value(MESSAGE);

It says that MESSAGE is exactly that and must not be dereferenced with "$"

That was a dead give-away

> 
> Maybe I'm the idiot here, however I thought that this was a common way of
> getting rid of unwanted crud from the syslog?

It IS the ideal way to pre-filter logs based on the message content. Pre 
version 3, you could only match on the entire message, so the feature to be 
able to search just a user-defined chunk of the log entry is a major plus
 
> Also, I just read the gentoo-wiki site page again and it says :
> 
> filter f_shorewall { not match("regex" value("Shorewall")); };  #
> Filter everything except regex keyword Shorewall
> 
> Surely this is the exact same mistake I made? Either that or I'm reading it
> wrong....

No, you are not reading it wrong - the gentoo guide is wrong. It's a common 
mistake, as the syntax looks like it's a name-value pair. To my mind, the 
label "value" should instead be "field" or some synonym of that.

All the evidence indicates to me that the syntax makes sense once you "get" 
how it works, but most folks' initial assumption about it is wrong, and the 
developer never spotted his serious case of being blinded by his own 
understanding.

I see Robert responded here earlier. Perhaps he'll see this post and re-look 
at that section in a new light with a view to making a patch



> 
> On 17 March 2010 23:39, Alan McKinnon <alan.mckinnon@gmail.com> wrote:
> > On Wednesday 17 March 2010 01:22:59 Ralph Slooten wrote:
> > > Hi all,
> > > 
> > > Has anyone here worked out how to filter out syslog messages using
> > > syslog-ng v3? The old syntax doesn't work (well complains bitterly
> > > about performance and says to use regex), and no matter what I try I
> > > cannot get the new syntax to work :-/ I have a syslog-ng server which
> > > logs to MySQL for multiple clients in a network, however the database
> > > just keeps
> > 
> > growing
> > 
> > > with irrelevant data I'd prefer to just quietly ignore on the server
> > 
> > side.
> > 
> > > I'm trying to filter out (exclude) messages such as:
> > >   (root) CMD (/root/bin/vmware-checker)
> > > 
> > > and
> > > 
> > >   (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons )
> > > 
> > > ==============
> > > filter myfilter {
> > > 
> > >         not match("regex" value("\/usr\/sbin\/run-crons"))
> > >         and not match("regex" value("vmware-checker"));
> > > 
> > > }
> > 
> > Hah! this caught me out too.
> > 
> > The value of "value" cannot be anything arbitrary - syslog-ng has no clue
> > what
> > you mean. The value is a field name, either a pre-defined one, or
> > something you defined using a parser. The docs are ambiguous on this,
> > it's not clear that the supplied values are abstracts. You are truing to
> > search for the string "regex" in a field called /usr/bin/vmware-checker.
> > 
> > Which obviously will not work.
> > 
> > I think you want:
> > 
> > match("\/usr\/sbin\/run-crons" value "MESSAGE")
> > 
> > Note that it is MESSAGE. You want the field name, not it's dereferenced
> > value.
> > 
> > > log {
> > > 
> > >         source(src);
> > >         source(remote);
> > >         filter(myfilter);
> > >         destination(d_mysql);
> > > 
> > > };
> > > ===============
> > > 
> > > However they just keep coming through the filter (ie: not matching the
> > 
> > "not
> > 
> > > match" filter). I've tried escaping the slashes, not escaping them ...
> > 
> > even
> > 
> > > partial words, but I obviously am missing something somewhere.
> > > 
> > > Anyone have any ideas?
> > > 
> > > Thanks in advance,
> > > Ralph
> > 
> > --
> > alan dot mckinnon at gmail dot com

-- 
alan dot mckinnon at gmail dot com

[gentoo-user] Re: syslog-ng filtering

[Robert Fekete]

Alan McKinnon <alan.mckinnon <at> gmail.com> writes:
> 
> I see Robert responded here earlier. Perhaps he'll see this post and re-look 
> at that section in a new light with a view to making a patch
> 
Yes, there are some sections in the admin guide that sorely need a cleanup. I
hope to get to it before summer.

If you see other problems or missing stuff, feedback is most welcome at
documentation@balabit.com or frobert@balabit.hu.

Regards, 

Robert

Re: [gentoo-user] syslog-ng filtering

[Keith Dart]

=== On Thu, 03/18, Ralph Slooten wrote: ===
> Maybe I'm the idiot here, however I thought that this was a common
> way of getting rid of unwanted crud from the syslog?

===

Probably the best method is to not send it there in the first place. 

For example, the script run by cron, /usr/sbin/run-crons, has this line
in it:

[ -x /usr/bin/logger ] && /usr/bin/logger -i -p cron.info -t run-crons
"(`whoami`) CMD ($SCRIPT)"

You can comment that out and then those annoying run-cron entries won't
be logged.



-- Keith Dart

-- 
-- --------------------
Keith Dart
<keith@dartworks.biz>
=======================

Re: [gentoo-user] syslog-ng filtering

[Ralph Slooten]

[-- Attachment #1: Type: text/plain, Size: 536 bytes --]

On 18 March 2010 09:40, Keith Dart <keith@dartworks.biz> wrote:
>
> You can comment that out and then those annoying run-cron entries won't
> be logged.


Yes, dropping those entries on the client side is an option, however then I
have to do it for each client in the network. Doing it on the server means
just once... and it's all local network, no bandwidth isn't an issue either.

There are also some cron jobs I do want logged ~ things that run maybe
weekly or monthly, but some run every minute and really don't need to be
logged.

[-- Attachment #2: Type: text/html, Size: 818 bytes --]

Re: [gentoo-user] syslog-ng filtering

[Alan McKinnon]

On Wednesday 17 March 2010 23:43:39 Ralph Slooten wrote:
> On 18 March 2010 09:40, Keith Dart <keith@dartworks.biz> wrote:
> > You can comment that out and then those annoying run-cron entries won't
> > be logged.
> 
> Yes, dropping those entries on the client side is an option, however then I
> have to do it for each client in the network. Doing it on the server means
> just once... and it's all local network, no bandwidth isn't an issue
> either.
> 
> There are also some cron jobs I do want logged ~ things that run maybe
> weekly or monthly, but some run every minute and really don't need to be
> logged.

And you still have to cater for the case where some joker sends you heaps of 
unwanted stuff despite you repeatedly asking him not to.

Or, god forbid, you have to receive logs from Cisco kit.

-- 
alan dot mckinnon at gmail dot com