Re: [gentoo-amd64] Re: Systemd migration: opinion and questions

[Marc Joliet <marcec@gmx.de>]

[-- Attachment #1: Type: text/plain, Size: 9690 bytes --]

Am Wed, 25 Feb 2015 10:13:14 +0000 (UTC)
schrieb Duncan <1i5t5.duncan@cox.net>:

> Rich Freeman posted on Tue, 24 Feb 2015 16:44:59 -0500 as excerpted:
> 
> > On Tue, Feb 24, 2015 at 3:15 PM, Marc Joliet <marcec@gmx.de> wrote:
> >>
> >> == Things I have *not* gotten rid of (yet) ==
> >>
> >> I plan on uninstalling syslog-ng, but haven't done so yet.  I simply
> >> feel better waiting a bit, even though I don't run it anymore.  Man, I
> >> feel silly after typing that...
> > 
> > Yeah, took me a while to uninstall it, but I don't find much value in
> > running both.
> 
> First, welcome to systemd! =:^)  I may respond to other parts too but 
> this one's reasonably easy, while my choice is different, so I'll explain 
> what and why...
> 
> Intro: I chose to keep syslog-ng running here, while I (of course) have 
> journald running as well.  I believe the way I have configured both plays 
> to the advantages of each, while discounting the weak areas of each as 
> the other one covers that area.
> 
> What:
> 
> 1) Journald is configured to use volatile storage (the /run/log/journal 
> subdir, with /run being tmpfs) ONLY, effectively limiting it to single-
> session.

Right, I remember that from your thread a while back.

> 2) Syslog-ng is configured pretty much as before, except that I have its 
> systemd USE flag turned on, which builds it with journald support so the 
> two cooperate a bit better.
[...]
> Meanwhile, for logs beyond one session, I still prefer traditional text-
> based logs, tho I suppose that's partly because that's what I'm most used 
> to.  But there's several advantages to it that I see:

Personally, I don't mind the binary format, since the engineering advantages
convinced me (constant-time access, lots of meta-data which you can filter by,
git-inspired chain of hashes for integrity checking, etc.).

> a) If the system crashes, partially corrupted in-the-crash text logs can 
> be of at least some use after a reboot.  Binary journals, not so much.

As Rich mentioned, and as I remember reading myself (on the btrfs ML?), the
journal will do its best to read back from corrupted logs and is supposedly
pretty resilient.

> b) As far as I could find, journald has absolutely no mechanism to filter 
> incoming (pre-storage) log entries, while syslog-ng has had (pre-storage) 
> filter mechanisms for ages.  Journald seems to have all sorts of outgoing 
> post-storage filter-on-output options, some of which are pretty nifty, 
> but there have been several times in the past where logs filled up with 
> "noise", thousands of repeat entries for something or other that occurs 
> regularly, say once a minute, that I simply do not care about and don't 
> want in my logs at all, but which I can't directly stop at the source for 
> whatever reason.  One of them was a kernel bug that triggered for a few 
> kernel cycles, then eventually went away, another was a ksysguard 
> triggered message, with that ksysguard graph updating every second, etc.  
> Unfortunately, while journald makes it easy to filter these out on 
> output, I found no way of preventing their appearance in the journal at 
> all.
> 
> But syslog-ng lets me dump them without ever actually logging them, or 
> route them to a different log file if I prefer, keeping my primary logs 
> clean. =:^)

That is a very good point, I hadn't thought of that.

However, I think that I prefer the post-filtering approach, since it is in
principle the more flexible one.  At work I sometimes mix the logs from
different units (e.g., units A and B, and then units B, C, and D).

But I realised today that journalctl has no negation operator!  There is,
however, an RFE for it.

But regardless of what you use, I think that the worst offenders are services
that write logs themselves (I'm looking at you, samba).

> As a counter-point, however, at least journald does compression to some 
> degree, so tens of thousands of identical or nearly identical messages 
> don't take up as much room as you might expect as they compress very 
> well. =:^)  I guess after the first instance, dups basically consist of a 
> timestamp and a pointer to the first message.  So they don't take much 
> space.  Which is fortunate when some "noise" message starts appearing 
> once a second... and you're storing it in tmpfs, thus in memory!  
> Obviously persistent logs would be similarly space efficient, but I still 
> don't like the idea of not being able to filter out the noise, BEFORE it 
> hits permanent storage, so I just make sure it /doesn't/ hit permanent 
> storage. =:^)

Yeah, I think that's a good use case for pre-filtering: an overly noisy service
whose logs you purposely want to ignore (for example, because you're still
testing it - but then again, for that particular case you could probably use
systemd-nspawn and log to an isolated journal).

> c) I use btrfs for my primary filesystems, and btrfs and journald's 
> binary-format journals don't play so well together.  FWIW, the latest 
> systemd-219 has lots of improvements for btrfs users, including some new 
> features that are btrfs-specific (which I never-the-less don't expect to 
> use right away as they're mostly VM/container oriented and I don't do 
> much of that sort of thing, but I may well eventually use them), as well 
> as some changes that should make journaling on btrfs work somewhat 
> better.  So this point should be to some degree eliminated after that 
> stuff stabilizes tho part of the fix is journald setting nocow on the 
> journal files and that has other implications I'll skip further 
> discussion of here.  However, by limiting journald to tmpfs only, I still 
> get the best benefits of systemd/journald integration (like status 
> including the last few log messages as mentioned above), while 
> eliminating the entire class of btrfs/journald issues.  Best of both 
> worlds. =:^)

Well, I'm on an SSD, but even on the laptop I haven't noticed any performance
issues (yet).  Then again, I use autodefrag, so that probably helps.

What's funny though is that the systemd news file
(http://cgit.freedesktop.org/systemd/systemd/tree/NEWS) occasionally
refers to non-btrfs file systems as "legacy file sysetms".  At least, as a
btrfs user I think it's funny :) .

> d) Tho I don't do anything with it in my syslog-ng config here, syslog-ng 
> can actually make use of some of the extra information journald passes to 
> it, including trusted per-app filtering and routing.  One of the problems 
> that the old syslog approach had was that anything that could log 
> messages could claim to be anything it wanted, and syslog really couldn't 
> tell whether it was telling the truth or not, all it could do is sort and 
> report the messages based on what was claimed.  A big bullet-point 
> feature of systemd/journald is that because systemd enforces service 
> separation via control-classes and the like, there's a much stronger 
> guarantee that what systemd/journald says is service X, is REALLY service 
> X and not service Y!  But that's not limited to journald.  Syslog-ng has 
> for several versions now had the ability to filter, route and report on 
> out-of-log-channel information such as the extra properties that journald 
> associates with each message, that make features such as systemctl status 
> reporting the last few log messages from that service possible, because 
> systemd/journald actually tracks it, AND passes it on to other loggers as 
> well now, providing they're built to use the API provided by the systemd 
> library I mentioned above.  And again, for gentooers, syslog-ng is built 
> with that API, against that library, based on USE=systemd.  So once 
> that's there, if desired, you can do all sorts of fancy filtering/log-
> routing/etc based on information such as the application/service name, 
> that systemd passes along to syslog-ng.
> 
> And again, with syslog-ng, this can be done BEFORE the actual logging if 
> desired, instead of logging everything and filtering only on output, as 
> journald does. =:^)

That's actually pretty cool :) . You're basically using the journal as a log
accumulator and syslog-ng for routing and post-processing its messages.

Whiles that's not how I want to do things, I still appreciate how cool that
is :) (and the fact that the journal can be used that way).

> e) I'm simply more comfortable dealing with text-based logs, since I'm 
> used to them and can use whatever tools I want on them, as well as being 
> able to split/sort/route to multiple log files in as simple or complex a 
> setup as I want. =:^)
> 
> Summary:
> 
> For me, splitting up logging duties so journald does the collection, but 
> only does session journaling to tmpfs, while syslog-ng still handles the 
> "persistent" logging, gives me the best of both worlds.  I love systemd/
> journald features such as log-messages-in-status and having seen how 
> useful it is, would seriously find it hard to do without.  But at the 
> same time, I can't bring myself to accept persistent logging without the 
> ability to filter out the noise before it hits persistent storage, and 
> journal storage on btrfs is problematic anyway, with both problems nicely 
> solved by handing off to syslog-ng for my persistent storage needs while 
> keeping journald for same-session use, so that's exactly what I do. =:^)

Right :) .

Greetings
-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 819 bytes --]