From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) by finch.gentoo.org (Postfix) with ESMTP id 1C193138A87 for ; Wed, 25 Feb 2015 18:57:03 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id C63BFE0A00; Wed, 25 Feb 2015 18:56:59 +0000 (UTC) Received: from mout.gmx.net (mout.gmx.net [212.227.15.18]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id E93DFE09A2 for ; Wed, 25 Feb 2015 18:56:58 +0000 (UTC) Received: from marcec.fritz.box ([93.181.44.4]) by mail.gmx.com (mrgmx002) with ESMTPSA (Nemesis) id 0LcBvB-1Xi3790bfZ-00jbTn for ; Wed, 25 Feb 2015 19:56:50 +0100 Date: Wed, 25 Feb 2015 19:56:32 +0100 From: Marc Joliet To: gentoo-amd64@lists.gentoo.org Subject: Re: [gentoo-amd64] Re: Systemd migration: opinion and questions Message-ID: <20150225195632.2dbe5cda@marcec.fritz.box> In-Reply-To: References: <20150224211545.50495cc0@marcec.fritz.box> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.25; x86_64-pc-linux-gnu) Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-amd64@lists.gentoo.org Reply-to: gentoo-amd64@lists.gentoo.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/Uz85=v72F0CG3Fd3OC42hYr"; protocol="application/pgp-signature" X-Provags-ID: V03:K0:LO6B6iV8HrqaGMwhg21w2wvwnJcXd/ge/5WoyOY5Gh4tOC/5ZHl IlmrNfIn+5ye/EWt+2EGwlK1NvXahFA3Scs3TNxmb6oufXH9wh6BhnlJUmMcq0rCKSFCb0/ cg80MbQtfocZI8X0sO/9gzsjgEErYxgX299lvNVlDCOYkaBoOsZzC8FLJFm67d+Lg3YOLws Kq0+ON5Qtt9M11z8snI5w== X-UI-Out-Filterresults: notjunk:1; X-Archives-Salt: 8321539a-d7b1-40b1-8ef5-c69b3ce42cd0 X-Archives-Hash: 9a3f9e9c7d3d67b8815240e4e895cf6c --Sig_/Uz85=v72F0CG3Fd3OC42hYr Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Am Wed, 25 Feb 2015 10:13:14 +0000 (UTC) schrieb Duncan <1i5t5.duncan@cox.net>: > Rich Freeman posted on Tue, 24 Feb 2015 16:44:59 -0500 as excerpted: >=20 > > On Tue, Feb 24, 2015 at 3:15 PM, Marc Joliet wrote: > >> > >> =3D=3D Things I have *not* gotten rid of (yet) =3D=3D > >> > >> I plan on uninstalling syslog-ng, but haven't done so yet. I simply > >> feel better waiting a bit, even though I don't run it anymore. Man, I > >> feel silly after typing that... > >=20 > > Yeah, took me a while to uninstall it, but I don't find much value in > > running both. >=20 > First, welcome to systemd! =3D:^) I may respond to other parts too but=20 > this one's reasonably easy, while my choice is different, so I'll explain= =20 > what and why... >=20 > Intro: I chose to keep syslog-ng running here, while I (of course) have=20 > journald running as well. I believe the way I have configured both plays= =20 > to the advantages of each, while discounting the weak areas of each as=20 > the other one covers that area. >=20 > What: >=20 > 1) Journald is configured to use volatile storage (the /run/log/journal=20 > subdir, with /run being tmpfs) ONLY, effectively limiting it to single- > session. Right, I remember that from your thread a while back. > 2) Syslog-ng is configured pretty much as before, except that I have its= =20 > systemd USE flag turned on, which builds it with journald support so the= =20 > two cooperate a bit better. [...] > Meanwhile, for logs beyond one session, I still prefer traditional text- > based logs, tho I suppose that's partly because that's what I'm most used= =20 > to. But there's several advantages to it that I see: Personally, I don't mind the binary format, since the engineering advantages convinced me (constant-time access, lots of meta-data which you can filter = by, git-inspired chain of hashes for integrity checking, etc.). > a) If the system crashes, partially corrupted in-the-crash text logs can= =20 > be of at least some use after a reboot. Binary journals, not so much. As Rich mentioned, and as I remember reading myself (on the btrfs ML?), the journal will do its best to read back from corrupted logs and is supposedly pretty resilient. > b) As far as I could find, journald has absolutely no mechanism to filter= =20 > incoming (pre-storage) log entries, while syslog-ng has had (pre-storage)= =20 > filter mechanisms for ages. Journald seems to have all sorts of outgoing= =20 > post-storage filter-on-output options, some of which are pretty nifty,=20 > but there have been several times in the past where logs filled up with=20 > "noise", thousands of repeat entries for something or other that occurs=20 > regularly, say once a minute, that I simply do not care about and don't=20 > want in my logs at all, but which I can't directly stop at the source for= =20 > whatever reason. One of them was a kernel bug that triggered for a few=20 > kernel cycles, then eventually went away, another was a ksysguard=20 > triggered message, with that ksysguard graph updating every second, etc. = =20 > Unfortunately, while journald makes it easy to filter these out on=20 > output, I found no way of preventing their appearance in the journal at=20 > all. >=20 > But syslog-ng lets me dump them without ever actually logging them, or=20 > route them to a different log file if I prefer, keeping my primary logs=20 > clean. =3D:^) That is a very good point, I hadn't thought of that. However, I think that I prefer the post-filtering approach, since it is in principle the more flexible one. At work I sometimes mix the logs from different units (e.g., units A and B, and then units B, C, and D). But I realised today that journalctl has no negation operator! There is, however, an RFE for it. But regardless of what you use, I think that the worst offenders are servic= es that write logs themselves (I'm looking at you, samba). > As a counter-point, however, at least journald does compression to some=20 > degree, so tens of thousands of identical or nearly identical messages=20 > don't take up as much room as you might expect as they compress very=20 > well. =3D:^) I guess after the first instance, dups basically consist of= a=20 > timestamp and a pointer to the first message. So they don't take much=20 > space. Which is fortunate when some "noise" message starts appearing=20 > once a second... and you're storing it in tmpfs, thus in memory! =20 > Obviously persistent logs would be similarly space efficient, but I still= =20 > don't like the idea of not being able to filter out the noise, BEFORE it= =20 > hits permanent storage, so I just make sure it /doesn't/ hit permanent=20 > storage. =3D:^) Yeah, I think that's a good use case for pre-filtering: an overly noisy ser= vice whose logs you purposely want to ignore (for example, because you're still testing it - but then again, for that particular case you could probably use systemd-nspawn and log to an isolated journal). > c) I use btrfs for my primary filesystems, and btrfs and journald's=20 > binary-format journals don't play so well together. FWIW, the latest=20 > systemd-219 has lots of improvements for btrfs users, including some new= =20 > features that are btrfs-specific (which I never-the-less don't expect to= =20 > use right away as they're mostly VM/container oriented and I don't do=20 > much of that sort of thing, but I may well eventually use them), as well= =20 > as some changes that should make journaling on btrfs work somewhat=20 > better. So this point should be to some degree eliminated after that=20 > stuff stabilizes tho part of the fix is journald setting nocow on the=20 > journal files and that has other implications I'll skip further=20 > discussion of here. However, by limiting journald to tmpfs only, I still= =20 > get the best benefits of systemd/journald integration (like status=20 > including the last few log messages as mentioned above), while=20 > eliminating the entire class of btrfs/journald issues. Best of both=20 > worlds. =3D:^) Well, I'm on an SSD, but even on the laptop I haven't noticed any performan= ce issues (yet). Then again, I use autodefrag, so that probably helps. What's funny though is that the systemd news file (http://cgit.freedesktop.org/systemd/systemd/tree/NEWS) occasionally refers to non-btrfs file systems as "legacy file sysetms". At least, as a btrfs user I think it's funny :) . > d) Tho I don't do anything with it in my syslog-ng config here, syslog-ng= =20 > can actually make use of some of the extra information journald passes to= =20 > it, including trusted per-app filtering and routing. One of the problems= =20 > that the old syslog approach had was that anything that could log=20 > messages could claim to be anything it wanted, and syslog really couldn't= =20 > tell whether it was telling the truth or not, all it could do is sort and= =20 > report the messages based on what was claimed. A big bullet-point=20 > feature of systemd/journald is that because systemd enforces service=20 > separation via control-classes and the like, there's a much stronger=20 > guarantee that what systemd/journald says is service X, is REALLY service= =20 > X and not service Y! But that's not limited to journald. Syslog-ng has= =20 > for several versions now had the ability to filter, route and report on=20 > out-of-log-channel information such as the extra properties that journald= =20 > associates with each message, that make features such as systemctl status= =20 > reporting the last few log messages from that service possible, because=20 > systemd/journald actually tracks it, AND passes it on to other loggers as= =20 > well now, providing they're built to use the API provided by the systemd= =20 > library I mentioned above. And again, for gentooers, syslog-ng is built= =20 > with that API, against that library, based on USE=3Dsystemd. So once=20 > that's there, if desired, you can do all sorts of fancy filtering/log- > routing/etc based on information such as the application/service name,=20 > that systemd passes along to syslog-ng. >=20 > And again, with syslog-ng, this can be done BEFORE the actual logging if= =20 > desired, instead of logging everything and filtering only on output, as=20 > journald does. =3D:^) That's actually pretty cool :) . You're basically using the journal as a log accumulator and syslog-ng for routing and post-processing its messages. Whiles that's not how I want to do things, I still appreciate how cool that is :) (and the fact that the journal can be used that way). > e) I'm simply more comfortable dealing with text-based logs, since I'm=20 > used to them and can use whatever tools I want on them, as well as being= =20 > able to split/sort/route to multiple log files in as simple or complex a= =20 > setup as I want. =3D:^) >=20 > Summary: >=20 > For me, splitting up logging duties so journald does the collection, but= =20 > only does session journaling to tmpfs, while syslog-ng still handles the= =20 > "persistent" logging, gives me the best of both worlds. I love systemd/ > journald features such as log-messages-in-status and having seen how=20 > useful it is, would seriously find it hard to do without. But at the=20 > same time, I can't bring myself to accept persistent logging without the= =20 > ability to filter out the noise before it hits persistent storage, and=20 > journal storage on btrfs is problematic anyway, with both problems nicely= =20 > solved by handing off to syslog-ng for my persistent storage needs while= =20 > keeping journald for same-session use, so that's exactly what I do. =3D:^) Right :) . Greetings --=20 Marc Joliet -- "People who think they know everything really annoy those of us who know we don't" - Bjarne Stroustrup --Sig_/Uz85=v72F0CG3Fd3OC42hYr Content-Type: application/pgp-signature Content-Description: Digitale Signatur von OpenPGP -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJU7hruAAoJEL/Q5oYsiHj029AQAJgTeUk32rvRlFKMlcmltZ/8 ZAlvi7F90mBjDDU+M9wEXHM1HNSPMsgw4nzS8Soz77z+KFSLY33LGtk8RoIP64zx ea+CRaY/H0cwCdifTUZeqSP6AI6rfWAOH55jsekxVH0v1jjAOGP+f+iaWx1mnRCI UuxZXiruM3t4GKPV1FgpUTIApe/cbs5EjwdNpMbBk4BMPavFUMPshu4/IzTmQhEA FgD7yLw4h+fH8PpzFCxqU1fXxlVbiRm5ALMnT+AlkTDNoQoB/T4VY8+LiNt71GJl lBeJVpdZMs7ZFqU02CfvmSUPL02eqgVblEqujLqwGNIM7FHS0FptnPtSwp4yS/MG 3cBPwdME16DXnCJ7hRh4bJ+vBuzvjQes01JUL20HaqiYpyNcL6WDg9kj2POd3CDn /jUhxchcRjmXIpL9BLXKP7TcaYV8E82vXuwcjj2zJ0DbQmgBxDzOKS79iYFwkfa1 11HZJkaHXfdYR66E13HykalwxmTzV+uhTS8/oNz1WUHxGMyzYjJyH4PULJ5ENFpD zTvbS6VZbc3R4n30k84amhiu5X1fbSu/3ww1sDnlRgT1Masi8h8UcFZeUL8aJRpC LVPZ6Lqy3z/s8L2DyVK1wG8GuG4Q2MNHxmT/i5viaM++I5dWOGSJYIC3Oatazgtm nY0HHMxrBIEIPR8QdfFt =E1Od -----END PGP SIGNATURE----- --Sig_/Uz85=v72F0CG3Fd3OC42hYr--