From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 2FC79138350 for ; Thu, 9 Apr 2020 08:42:33 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 8EABFE09DA; Thu, 9 Apr 2020 08:42:25 +0000 (UTC) Received: from oc.oops.co.at (oc.oops.co.at [85.90.244.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 2D5D3E0998 for ; Thu, 9 Apr 2020 08:42:24 +0000 (UTC) Received: from [IPv6:2001:470:6d:2d0:c726:8cc2:76de:16cf] (unknown [IPv6:2001:470:6d:2d0:c726:8cc2:76de:16cf]) by oc.oops.co.at (Postfix) with ESMTPSA id 4B0F88800C for ; Thu, 9 Apr 2020 10:42:22 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xunil.at; s=20190917; t=1586421742; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=/+yDFOze1rADT5GYIm6sYVuAe+hI6QPEY4XFF7g2m88=; b=r3EvUY4z4VeHINtait2Ab6jvsPk5Eal6myhcrmz5NqTKqt6D89O1oc3IscUgNznHRr1L9q NpKWYM4JlogVYFQjkqFikWuViz8wYZ1lWFk9Ts28NWcOXWnDk8qeG5VfX6LDUIz1pF2LhJ Jsy3kTeJx2iSjr5oROTYriuWZPg/MiPnKsuN9IPQVJy/10hkhlmkW3f7XtaetwXPA01VXh X72eygjVDLtSN9kByJyFMwCGyNjIzq57Sa0EWGJEBK6DkhP1FZP1tMpC66UaDEoaN292SB l484SDpKvmqfnhazwhQHxYEhCMTVICT5T4FYYzKXsMROcjoo53YpckYq/94C9g== Subject: Re: [gentoo-user] aggregate logs into Elasticsearch To: gentoo-user@lists.gentoo.org References: <2dd98a05-88d2-0899-0bcd-f064a5fad9e1@xunil.at> <87imigk1dn.fsf@wedjat.horus-it.com> <8e156bd2-0bc6-94bb-caba-dbac8f80bf35@xunil.at> <87tv1zfiwl.fsf@wedjat.horus-it.com> From: "Stefan G. Weichinger" Autocrypt: addr=lists@xunil.at; prefer-encrypt=mutual; keydata= mQINBE0XrckBEAC4eHs5mCLfmu+F8SMkkE1IfLvlIq1PcTlpg3VBinmOsInmvDbCwpDyqLv/ ABffa9K888XHlDdtJ6Py14uEXkoS6jW6V0VJV8w/Xi/Tjoa6iA64o1K5AGM3FqGjVsP7sF9j FmKr+RTa0Mi/9SSmQj5OkvqB0DGEh4qOZ9B2GLC+XsxnrlLj5k6WTvAHGNShV7N2jNn03/Fi VVS/pbMYDa6CsNoPcJBUFtq2ZY6t0hLe/armcCAyPxgBAXUM7Y54OmNrw9GcyVTKP9e5/h2W 3r9qqIP3Q3i9O37WdnmOWLs+DFyE5kLexVMf9WU6kGjciY46s1fjpJ/t5hfS4Jth9iE48HM3 kLpu1IQNN37ukgchJ/gxaAvgtFtNdgpZ9MTh28qYoiYRsAp2/aAwwHxCi387s8bhhSHB5vzV Qd+gzuqlgngwKirFXPRbZluIC6XLsPlfVM9YMPMFP2QfkvglyZYBj6aUG/XUsWicF0wzw72+ 5sr2CgqZr5X+gQF4lveKEwGl+qdBNqZShJQwPLWuUgp2BPvZnvEYaCis3J/GJTnn0hODMGuw 15ziSvL1MDrzA/QFHsGmZ96W7Zyuo/28j1hmfqFPA1MPku1//XpjtwrFhMMxoci46DHMUc1d J4uGQvV6TVtpII4378y9TdZJ24wojYe4LxzLEhbbqGnEhxHj7wARAQABtCVTdGVmYW4gRy4g V2VpY2hpbmdlciA8bGlzdHNAeHVuaWwuYXQ+iQJWBBMBCABAAhsjBwsJCAcDAgEGFQgCCQoL BBYCAwECHgECF4AWIQQSAQhEJOE+sVyWKVQpXLg9VdD85gUCXeTlEQUJEq5qxwAKCRApXLg9 VdD85nkoD/9GSX4EDu86pTCaac4Z86h1nko3VmiPcTq5x/K+xvmE09OrAURDZo7M17SV3cy/ ReqSpx5cprS55CjnzvGAsGeuACniaR2JkjIORxIh6Ptd0immj6tR6CFLh3KQ7AB/r4p7tbYv e8RnAI56zI39mE+A0urQCEM/VKsu3p7r/bN5SsBvS/1dGnDMTr/EhkL6Y45F6AOI9jgNePly mwH/DzuFxWDbBSxuV13Rz63xG6DSkoh2s9Oltu5fSRKG8GXcoMZHvUR1PlUO5ssKfzJcPtZi tSc6Xu7et38jXkxuMZOPA81c/bne2vYVvSCS84igE1C4VQbIDZXEnqnfIaEZnQClX5pcYZKS P21RM0mR3MH/u58I9W+kgS6+ChGakdWS73V5HJLnMndWvQHSCiWaaY7FMvg4+tTomG+i5Iit Y7bWP1Euo4Ly41hGVRzbwg78eG1p8bVESRdPvQovDZRwS/PxH7vYDRAKt/UfPV+IVeST7tIA ZO2UgG2tqKOD7jDyfMc/s7t2HptcUwEaTBONrx5qcXDpqRnwKAo2p2Ieaoj8Bl86lvYgnMGB tgr3TZVGHK2wDvJJbQyczEmk1gcJ/q1Imf2TPe/ncpRvSDSTWW1JjA7wi0+YltR+Lsh+cSGO zJxTc3nMz8QmVPNOQe/VMNfSzbIgYffgFlm4DpiYyi6IALkBDQRVpOS5AQgA0Ris43rFJyp8 mFfyaBSxi/ECVq5SplZL7MWESuhXh/j9L5HlmERt3JEtrwUJepXaVF/0j1nWHbpXqoE+JEaU Md7qUue+PTwJLAeFadUrAfVzHqNiAl4ZFmfW72FUNZRgTQWHh4/YmghsF9p9yRH/mSSAMO6f fTed2JmKFvoAfjqWkMDVKBHNxyZx9uRMW+lDfuT9cF8TVwT7G5Xuq45MGmN+ZRgB/WjywrnW Ysz5owuQZ32uMbvqpEKzRW9BWu9IEZXTovCE8vZ3u3yLlPBt7SuVh94FLM9jDxOehZwvb6en NI6nrOkP2JNFDFUaTduETxTQCbyL9aPhUHtk5G0D4QARAQABiQI8BBgBCAAmAhsMFiEEEgEI RCThPrFclilUKVy4PVXQ/OYFAl3k5REFCQohM9gACgkQKVy4PVXQ/OaWJhAAi20eR4gyuWTv DITGcL6WBf2zMjpi2N3KFYok6LEUv3OQrKvu/eHHo2Ij1UymOndTOYJjmgZq6lNWjPLrZ1uq e5kPwWCS6F7cEzlvOpFKh36Hl1tEs1nyYon7tJQG3D30GxM1UI9HJ+A+6I3NKaVrs6XzowxM nt+ibz1Z7EmKVkrociBo/YfP78R/R2duhDESq2LGI25p4BJiWzu4nETXgu1R5ukbGh+wr609 EzUnh+nzVDDTeJ7wOgCsJIHjbDTaxS505F2j5IbcydkJyCYBdGCyiWCctAEY7tJ4jYibv8fK s8ofBPA3lgRszIaFuBXhu4DB19ErOXOb2IG1rslxkucTORGd7dnPtQXHxHPiCPxKN+bsXCKJ EKqvG2WK/zJebRvwlBKl8mbI5fV24DRDDjvxOQayTY/xZzxv3NcjNo+EBAPxSkTgcJTYJYrS 2dWnZEWI0BqJhLHCqW86yOmQvULAKtaDaEiBAJ/zqzkYYc554QraOStleE1akxhcG7WftjsL ToTdGzifMREKZLpa4t949sA7xfaB9Xci/9mwQZ1ppWY9xQrW60tjuWFOtKf6f9Nf/BWZW2hj m7s4x7b6wkLHLisAvDkjIk7OKK53PBzn0tQkoHwsAv9B+YAVX7cm5OgRUwq/mVvt59ifpv3p 92n2rDvAa6onhncfNUFKWXe5AQ0EVaUDZwEIAJmyHlE18WMX+BoKrk9xs/leZhMFSrEj74TB LNlnRovA0bmUwU8fPyGxaj41hBtzt9LVmINCnwz/zevPL3AVRju/M9GlB6S0LYqFB0A1XpIo wMQRmjB2k3EBTFHryJgnklPB12zmt08mzaTtZMxpSv3x3YGaTOIAoBPrGl82mm+kJTNRYKCI mx6cWEGMU2gnBXDQjxheRrtPTpPy8TnE3bJ4lBVdqBaPKRHXSVKwIjyb3tjY7BAW1aSoho0J ouRIci+99Jb7r2vaNncxT/ujy98OnMkl1pKJZ5Dq71atHsCX8na/RU61RfRS2GHrXiwuaV+6 tV6Sn0rBEoJs8Y10gKEAEQEAAYkDWwQYAQgAJgIbAhYhBBIBCEQk4T6xXJYpVClcuD1V0Pzm BQJd5OURBQkKIRUqASnAXSAEGQEIAAYFAlWlA2cACgkQmoVqt0AbOule4Qf/W/9vKwy9kf+K 36zUzaombJ98sBh2P4XjJAXmqmb78rADDptyXSlBXbdQf+9VQqXU+thihyZY8WbPyFs7FJpu FzLrUJU3O0sK7QAEf/vjl3V6ADh9BWU4sAwVjVNwIlaDMNpLeKkduQChFZSuKqfNMmHI4v0o 7RjQi3t5JPvBTXpTqBki3Rn5yxg/zyvgPBEzTH5BuL5tH8Oj+s+dFB8icqvZejDW1cRRoSGc OT0ZVfm+7Tsv6P/ylHtiClCWgatMTnalh2YoLNGM9CdecT/ogN1tRTK2ZXmhB2DccqNdexK1 sGXGcixwTlntv1LISvNFmoBNFdS9aL2GqYHFIdEWAgkQKVy4PVXQ/ObWhQ//UoYoqK30Kvty 16mur7W7/I67eG9g+3b2fRzRVG5TtxRlQmCRcDSFETg92AFx5GQLxfviI5yIPDQw11gIV/Ee cSa6l78ESfjtwPAa6v0QEgVNQN4Qt5mN/FpKIvNbgCm/n1f7JbhG/O9rGs1lhLV494Tu+wx7 8296zSu8m2OF0xTVXKS04Vxm4ZCSXCZXK2VysWECzCc7Kkw2FLXVcZtHCJtaFWDJOWzJlJk2 Unv1msTes9NKCVf3iNh4T0TiH7lKKa95h9Suq0uHW9FE+oWspGuPmalXMapW0CBC4VLBPE4q OXOK5ZEuw7x/2NtITcTHILWzstnOqMNWmf5w4F0lLRXovdgEtTsxNW/iewtlfAWmta5PMTxq I8o8qAXdwf+5zNOaHJSyBPXPA4NcOALyxeyYaglaXlkvL4rj2lRBRS9S0h+nN01LUGJf0ig0 mzvDkN1GthYkUPu0xB+cD0kGLT7Pie1yIjHmf03DYNMG/yHdIrFv5QUoIueCP2xYOZdsqa41 fMgXxWXOxnqLOxBPWy0OAmcHB1V2vWpqRRSr+OlQkg4xcMH8lpodkXMKoGDjx6j0Qfrw9lT+ eYakxZKwJY8QBYh+LVVGBvPTm4cCfnoAZvpn0zgWEzX8n8O8PVDeRV4GJaTP5rM2+9KqwJOh +DnhhI6deD3l5NMmPMdZjrI= Message-ID: Date: Thu, 9 Apr 2020 10:42:21 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 In-Reply-To: <87tv1zfiwl.fsf@wedjat.horus-it.com> Content-Type: text/plain; charset=windows-1252 Content-Language: en-US Content-Transfer-Encoding: 7bit Authentication-Results: oc.oops.co.at; auth=pass smtp.auth=oc@oc.oops.co.at smtp.mailfrom=lists@xunil.at X-Archives-Salt: f01766ac-34a2-4934-83ea-abe58f268a2f X-Archives-Hash: 04e0c4b525390fbdf9052b14d8c92d92 Am 04.04.20 um 16:02 schrieb Ralph Seichter: > * Stefan G. Weichinger: > >> Maybe I look into mongodb as well, for example I found this small >> howto: https://www.fluentd.org/guides/recipes/maillog-mongodb > > That looks unnecessarily complicated to me. While you can of course move > data from an existing log file into MongoDB, I find configuring syslog > to use a MongoDB destination (in addition to your files or as a full > replacement) much easier. > > See [1] section "Storing messages in a MongoDB database". I have also > done it with rsyslog, but that took a bit more work. > > Here's a syslog-ng destination I use. Note that using uri() allows > passing parameters to modern MongoDB drivers which the older servers() > statement cannot cope with. > > destination d_mongo { > mongodb( > uri("mongodb://user:pw@hostname:27017/syslog?authSource=admin&ssl=true") > collection("messages") > value-pairs( > scope("selected-macros" "nv-pairs") > pair("DATE", datetime("$UNIXTIME")) > pair("PID", int64("$PID")) > pair("SEQNUM", int64("$SEQNUM")) > exclude("HOST*") > exclude("LEGACY*") > exclude("SOURCE*") > exclude("TAGS") > ) > ); > }; > > Values are strings to begin with. This example excludes some values I am > not interested in, and performs type conversion on others, for example > mapping DATE to MongoDB's date/time data type (see ISODate) and PID to a > numeric value. Conversion can of course happen during analysis, but > since syslog-ng is smart enough to do it when writing data, I prefer > that. > > [1] https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/37#TOPIC-956524 Thanks a lot ... I think I will postpone that project ;-)