Re: [gentoo-user] aggregate logs into Elasticsearch

[Ralph Seichter <abbot@monksofcool.net>]

* Stefan G. Weichinger:

> Maybe I look into mongodb as well, for example I found this small
> howto: https://www.fluentd.org/guides/recipes/maillog-mongodb

That looks unnecessarily complicated to me. While you can of course move
data from an existing log file into MongoDB, I find configuring syslog
to use a MongoDB destination (in addition to your files or as a full
replacement) much easier.

See [1] section "Storing messages in a MongoDB database". I have also
done it with rsyslog, but that took a bit more work.

Here's a syslog-ng destination I use. Note that using uri() allows
passing parameters to modern MongoDB drivers which the older servers()
statement cannot cope with.

  destination d_mongo {
    mongodb(
      uri("mongodb://user:pw@hostname:27017/syslog?authSource=admin&ssl=true")
      collection("messages")
      value-pairs(
        scope("selected-macros" "nv-pairs")
        pair("DATE", datetime("$UNIXTIME"))
        pair("PID", int64("$PID"))
        pair("SEQNUM", int64("$SEQNUM"))
        exclude("HOST*")
        exclude("LEGACY*")
        exclude("SOURCE*")
        exclude("TAGS")
      )
    );
  };

Values are strings to begin with. This example excludes some values I am
not interested in, and performs type conversion on others, for example
mapping DATE to MongoDB's date/time data type (see ISODate) and PID to a
numeric value. Conversion can of course happen during analysis, but
since syslog-ng is smart enough to do it when writing data, I prefer
that.

[1] https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/37#TOPIC-956524

-Ralph