[gentoo-amd64] Systemd Was: Please get me straight about sysvinit vs. systemd [...]

[Duncan <1i5t5.duncan@cox.net>]

Canek Peláez Valdés posted on Mon, 03 Mar 2014 11:58:18 -0600 as
excerpted:

> On Mon, Mar 3, 2014 at 11:44 AM, Duncan <1i5t5.duncan@cox.net> wrote:

>> Meanwhile, for the record, the systemd and now udev folks have stated
>> that they would like to eventually merge udev fully into systemd, and
>> indeed, it's already shipped as a single tarball, but that udev is
>> likely to remain a separate binary that can be run stand-alone for some
>> time, because that's necessary in ordered to be able to keep a somewhat
>> small initramfs, with udev but without all the trappings of a
>> full-fledged systemd.
> 
> If you use dracut to generate your initramfs, you can get a full-fledged
> systemd inside it, so you can use the systemd in the initramfs to start
> the systemd in the real filesystem. I use it like that.

I had seen hints to that effect, but as I was still running openrc it 
didn't have any immediate bearing on my dracut and initr* config, beyond 
USE=-systemd for dracut.

>> (This said as an openrc user at least for the time being... even
>> apparently one of the only people actually running the live-git
>> openrc-9999, or at least all the bugs filed on it seem to be mine. 
>> I've suspected for some time that I'll eventually switch to systemd,
>> but was at least originally hoping to avoid it until it quits actively
>> blackholing nearly everything it comes across and had some reasonable
>> time to stabilize without gobbling something else up.  But when that'll
>> be... who knows?  And I'm getting an itch to try it one of these days,
>> or at least seriously read up on it with a view to _consider_ trying
>> it, tho if I do it'll likely still be against my better judgment, since
>> I don't see it really stabilizing any time soon and I had originally
>> planned to wait for that.  So I guess I sort of fall in the middle in
>> this debate.)
> 
> If you run OpenRC live, I think you'll be fine running systemd, even
> 209/210, which introduced so many changes I've been waiting to upgrade
> my systems.
> 
> Come to the dark side. We have cookies.

=:^)

As you /might/ have deduced by now from the hints of the past-tense "was" 
and "didn't" above, as well as the fact that I'm dredging up this 
slightly (10 days) stale thread to reply to...

Yeah, I'm actually replying from a systemd booted system, now.

As I was privately predicting for myself, but didn't want to promise even 
to myself[1], I was getting close to the tipping point and this thread, 
plus a couple extra days off, provided the trigger.

Since my experience may be of help to others considering the switch, here 
we go...

In addition to unfortunately not finding a proper good detail but not 
losing sight of the forest for the trees admin-level-targeted systemd 
guide, a problem I overcame by simply immersing myself in the idea, full 
systemd for admins series, multiple hour-ish videos, hours of thinking 
about it while doing other things, sleeping over what I was learning over 
several days...

There were five specific problems, three of which I've now overcome, with 
two less pressing ones to go:

1) Various fstab mount configuration issues (local-fs.target).

To my very pleasant surprise, systemd has a reasonably effective nested-
mount-dependency algorithm and can often figure out on its own which 
fstab configured mount-points are nested under and thus depend on 
others.  This actually simplified things considerably as one of my custom 
openrc services was basically a localmount that had to come after 
localmount but before some other service, in ordered to properly handle 
nested mountpoints, and systemd's nested-mount-dependency processing very 
nicely solved most of that automatically, without my having to worry 
about porting a custom service.

1a) Unfortunately, systemd's nested-mountpoint detection algorithm isn't 
smart enough to deal with symlinked paths, at least not in all cases.  
One of my mount-related problems was thus a bind-mount configured in 
fstab, that wasn't happening because the origin, on a different non-root 
filesystem, wasn't mounted yet, so the attempt to bind-mount it elsewhere 
failed, and local-fs.target was failing as a result of that failure.

I solved that one by changing the origin/device path for that fstab entry 
to point at the canonical path instead of the symlinked path.

1b) My fstab entry for tmpfs on /run had a size=2m option, plenty for 
/run on openrc, but NOT for systemd, as by the time it got to the /run 
remount it already had an 8-meg early-boot journal file there, so 
obviously the remount to 2-meg was failing!  Of course that was failing 
local-fs.target as well.

That one was hard to track down as I didn't expect systemd to be putting 
a journal there and thus couldn't imagine what the problem was, but once 
tracked down, given that I'm running 16-gig RAM, upping that to something 
a bit more reasonable for the new usage, size=512m, was an easy fix.

With those fixes, local-fs.target finally succeeded, and I could get 
farther in the boot, most services of course having an automatic 
dependence on local-fs.target.

2) Unit file .include deprecated, replaced by *.d/*.conf files.

2a) systemd-random-seed.service was failing, again due to a mount-point-
detection symlink issue -- my /var is on my rootfs, which is read-only 
mounted for normal operation (only rw mounted for updates and config 
changes), so /var/lib/ is a symlink to /home/var/lib, /home naturally 
being rw mounted in normal operation, and systemd was trying to read the 
random-seed off that symlinked /home location before it was mounted, 
because all it knew to check for was /var, which being on / was of course 
mounted.

This one I solved initially by creating a new systemd-random-seed.service 
file in /etc, .including the default service in /usr/systemd/system/, but 
with an added RequiresMountsFor= entry.

That worked, but later on I found a note in the TODO to document the fact 
that .include was deprecated and to point people to *.d/*.conf files 
instead.  So I decided to try that.  The problem with /that/ was that the 
TODO reference (and various other *.conf references I found) weren't 
fully fleshed out, and as a systemd newbie I had some difficulty trying 
to figure out what they were talking about.

But after absorbing systemd for a few days I finally realized that not 
yet fully documented allusions I was seeing were to (paths like, for this 
service) /etc/systemd/system/systemd-random-seed.service.d/whatever.conf.  
And after adding a *.conf file there with the appropriate

[Unit]
RequiresMountsFor=

... entry, I was able to delete my custom service file and return to 
using the shipped service file, appending the appropriate *.conf line to 
the shipped configuration that way.

2b) I wasn't satisfied with the default gpm unit file config either, but 
by the time I got to that, I knew .includes were deprecated, and was in 
the process of trying to figure out what the allusion to *.d/*.conf was 
all about.

But once I figured that out for random-seed, I was quickly able to add
/etc/systemd/system/gpm.service.d/execstart.conf , with a customized 
ExecStart= line. =:^)


3) Systemd network config.

This one took me awhile too.  After the above and a few other minor 
tweaks to general systemd/journald/logind configuration[2], the big 
problem was that I was still networkless.  I didn't want to install 
network manager or whatever for a simple single-NIC static IP config, I'm 
still on systemd-208 as I've not figured out the systemd/dracut thing 
yet, so I don't have the 209+ networkd yet, and the various static-ip-
configuring network.service units I could find via google (plus one on a 
systemd bug on bugs.g.o) all used ip, from iproute2, which I had never 
installed as the old ifconfig and route from net-tools had to that point 
done all I needed it to do.

IMO this one's likely the biggest current problem for the gentoo masses 
who might contemplate a systemd switch, as my other problems are due to 
my decidedly custom config or are reasonably covered on the wiki.g.o 
systemd page, while there's very likely quite a few gentooers still on 
ifconfig/net-tools and unfortunately there's very little network 
discussion on the wiki page at all, let alone a nice step-by-step in the 
gentoo documentation tradition, covering ifconfig/net-tools users among 
others. =:^(

I'm actually quite disappointed that the gentoo/systemd folks don't have 
some sort of virtual/network-manager package or the like setup, with the 
various network.service and similar files for each one, and good coverage 
on the wiki.  Given all the noise around systemd, and the vital role I'd 
guess a working network config plays for most gentoo boxes, you'd think 
that would be well covered.  Oh, well...  At least it appears now to be a 
quite temporary problem, to hopefully be solved with the networkd in 
210/211/whatever-the-next-unmasked/unblocked happens to be.

Anyway, a quick emerge --pretend iproute2 said it wasn't going to pull 
any other deps I didn't already have, and I decided a single additional 
reasonably core package that I really probably should have installed 
years ago was after all a pretty small price to pay, so I finally just 
bit the bullet and installed it, and then setup the network@.service file 
using the ip command from iproute2.

Not so painful after all, but it'll still be nice when I get 210 or 211 
or whatever installed, and can hopefully use the new networkd solution 
instead of the still unofficial custom hack I've installed to deal with 
networking for the time being.

Which brings me up to current, with those three issues behind me.

The remaining two issues are:

4) Dracut and systemd

4a) Dracut's USE=systemd, enable or disable?

What does that do and given that the current systemd-less initr* I'm 
running now works well booting regardless of whether init=systemd, init 
(openrc), or bash (rescue shell), will whatever changes systemd in the 
initr* brings be worth it, or not?

Of course since you mentioned that you're running with systemd in your 
init, I'm hoping you can be of help on this one. =:^)

4b) The ongoing systemd-209+ file "musical chairs" and the havoc that 
plays with dracut, thus the current >=systemd-209 blockage in dracut.

It'd sure be nice to have this resolved one way or another, particularly 
so given that 209+ brings the hopefully more final networkd solution to 
#3 above, as well.  I was actually hoping that the multi-device btrfs 
root kernel command-line rootflags issue was resolved so I could just 
drop the initr* thing now and thus wouldn't have to worry about dracut 
any more, but as I posted in the earlier post, unfortunately not.

5) A couple remaining minor service failures:

5a) ntp.service is still failing, for some reason.

While I was running ntp on the old mobo before I upgraded and thus 
haven't ever really run this one without it, I've no reason to think this 
machine's hardware is /bad/ at keeping time, so ntp isn't critical.  But 
it'll be nice to find whatever's going on there and fix it.

Based on an aside-comment I read when researching something else, I 
already have one lead to check on this one -- the comment said systemd's 
privatetmp option doesn't play well with /var/tmp being a symlink to /
tmp, and that's what I have here (with /tmp in turn being tmpfs), and I 
think the ntp unit might use privatetmp, so...  But I've yet to confirm, 
and then I'll have to look for alternatives, so...

5b) On openrc, I have a custom user-bootclean service that cleans up a 
few stray user files at boot, that systemd created an attempted port for, 
except it's failing.  I've not investigated that one at all yet, but just 
as I write this it occurs to me that the problem might be yet another 
symlinked path mount-dependency issue, so I now have one lead to 
investigate on that one, too.

5c) bind/named (not failing but reduced functionality)

There's a gentoo bug about this as well.  Openrc's named/bind initscript 
has an optional chroot feature.  Systemd has both a legacy chroot support 
feature and a newer kernel-namespaces-based chroot replacement, but the 
shipped named.service file doesn't make use of either one.  If nobody 
else gets to that bug before I get comfortable enough with systemd to try 
to address it myself, I'll try to come up with a systemd-style solution, 
potentially using namespaces, to parallel the gentoo/openrc named 
initscript's optional chroot feature.

https://bugs.gentoo.org/show_bug.cgi?id=496198


Other than that... a couple positives already. =:^)

1) As mentioned in problem #1 above, systemd has actually surprisingly 
good nested mountpoint dependency resolution.  My setup is complicated 
enough I really appreciate that, and now that I'm "dual-booting" openrc 
and systemd <g>, I definitely miss that feature on openrc!  =:^0

2) Speed, WOW!

I've had openrc's parallel boot option enabled for years, but systemd 
really does run circles around it!  I'd guess a lot of that's due to the 
socket-based activation along with the mount dependency resolution such 
that services that are only dependent on each other due to that can still 
start in parallel, and particularly with socket-based dependencies, the 
otherwise depending service can often actually fully initialize if it 
gets to that point before the depended service is up, as long as the 
socket is already active and buffering any data the depending service 
might try to write to it.

The fact that it's all C based and mostly declarative, with near-zero 
scripting and far /far/ less forking, thus short-circuiting a good 
portion of what even openrc's semi-C-based solution does, certainly helps 
as well.

And of course the fact that I'm on SSD, where all those services starting 
in parallel doesn't trigger a seek-storm on the spinning rust that's not 
there to have a seek-storm on, doesn't hurt!  (Systemd has a read-ahead 
service to counteract part of the parallel-init seek-storm issue, but 
spinning rust will still be a bit of a bottleneck.  I had switched to SSD 
and my boot times improved, but in hindsight on openrc, only to a point, 
because the service-dependency on openrc then became the bottleneck.  
With systemd's socket-based-activation and the like killing /that/ 
bottleneck too, the speed of SSD **REALLY** shows it's stuff!  I had no 
/idea/!)

I have a feeling that once I get settled in, I'll be doing far more full 
shutdowns, since boottimes really /are/ trivial on systemd.  And looking 
to the future, if logind ends up doing for kde5 login times what systemd 
did for boot times...  WOW!  The idea of shutdown and startup cycles 
being so trivial one might as well just shut down when going to microwave 
a snack, since bootup and session startup's literally mostly the time 
waiting for the human to type in the password...


---
Notes:

[1] Predicting but not promising, even to myself:  I've discovered that I 
work best without deadlines or too specific goals, which only tend to 
demotivate and drive procrastination.  Left to itself, at some point the 
drive to find some stable conclusion normally takes over, at which point 
it's generally a done deal, concluded far more rapidly and with more 
finality than it would be, had I tried to artificially drive it myself, 
or let others try to drive it for me.

[2] General configuration changes, well enough documented that I didn't 
have much problem with them:

(a)/etc/systemd/system/default.target symlink pointing at multi-user, 
since I login at the CLI and have XSESSION set so I can startx, so systemd 
would quick complaining about the missing *DM service.

(b) /etc/systemd/journald.conf:  Storage=volatile at least until I figure 
out whether I want to keep syslog-ng or not, appropriate RuntimeMaxUse 
and RuntimeKeepFree values, ForwardToConsole=yes and TTYPath=/dev/tty12, 
so I get the traditional TTY12 console log.

(c) Setting NAutoVTs and ReserveVT= in /etc/systemd/logind.conf

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman