[gentoo-amd64] Re: Systemd migration: opinion and questions

[Duncan <1i5t5.duncan@cox.net>]

Rich Freeman posted on Sun, 01 Mar 2015 14:13:53 -0500 as excerpted:

> On Sun, Mar 1, 2015 at 1:20 PM, Marc Joliet <marcec@gmx.de> wrote:
>>
>> Regardless: thoughts?
> 
> I'd probably just do this:
>> Am Sun, 1 Mar 2015 08:34:19 -0500 schrieb Rich Freeman
>> <rich0@gentoo.org>:
>>>
>>> The timer keeps running if you set the dependency on the service.  So,
>>> next time the timer runs, it will try again.  You might want to just
>>> set an hourly job and have it check for a successful run in the last
>>> day or whatever.
>>>
> You could of course trigger this from either the mount or hourly.
> Anytime you mount the drive or every hour systemd will run the service,
> and the service will see if it managed to do a backup/etc in the last
> day/week/whatever, and then run if appropriate.

This is actually how I setup several former cron-jobs as systemd timers, 
here, based on an hourly check somewhat similar to what most crons 
(including gentoo's for over 10 years now and mandrake's before that) are 
actually setup to do to get around the fact that cron won't on-its-own 
trigger after restart if the machine was down or cron not running when 
the configured time for a job ran.

Here's how I have it setup here.  Note that my initials are jed, and I 
use them regularly as a prefix/suffix to denote custom configs (here, 
systemd units) I've created myself, as opposed to those shipped in 
whatever package.


/etc/systemd/system/jed.hourly.timer:

[Unit]
Description=JED Hourly timer

[Timer]
OnBootSec=10min
OnUnitActiveSec=1h
Unit=jed.hourly.target

[Install]
WantedBy=timers.target


timers.target is a systemd default target with the purpose of starting 
all the various timers.  The install entry says to install a symlink 
pointing to jed.hourly.timer in the timers.target.wants subdir, which 
will of course cause timers.target to start it.

The 10 minute OnBootSec setting says wait 10 minutes after boot before 
activating this timer the first time.  The 1h OnUnitActiveSec setting 
says reactivate it every hour after its first activation.  Thus, we have 
an hourly timer, except that its first run is offset from boot by only 10 
minutes, not a full hour.

Meanwhile, what it actually does when it runs is activate 
jed.hourly.target -- that's the Unit= line.


/etc/systemd/system/jed.hourly.target:

[Unit]
Description=JED Hourly target
StopWhenUnneeded=yes


This one, like most targets, is pretty simple, its only purpose being a 
synchronization point, activating a bunch of services and/or other units, 
generally those with symlinks installed in the corresponding 
jed.hourly.target.wants subdir, at once.

Note the StopWhenUnneeded=yes setting.  The sole purpose of activating 
this thing is to trigger all its wants deps to run, and as soon as it has 
done that, it can stop again.  This lets the target "rest" between hourly 
triggerings.


Symlinks in /etc/systemd/jed.hourly.target.wants/:

jed.hourly.timestamp.service


That's it, just the one.  With just it there it would have in fact been 
just as easy for me to set jed.hourly.timer's 
Unit=jed.hourly.timestamp.service, and omitted jed.hourly.target 
entirely.  However, using the target is MUCH more flexible, as it allows 
me to add new services I want triggered hourly to the same target, as I 
decide I want/need them.

If you're familiar with the usual cron setup, in particular, the usual 
/etc/crond.hourly/ (or whatever it was, that's from memory as I replaced 
my cronjobs with timers and don't even have a cron installed, these 
days), that's EXACTLY the function /etc/systemd/jed.hourly.target.wants 
ends up filling, with the minor exception being that as I set it up, it 
triggers hourly, starting 10 minutes after boot, instead of hourly, at 
some arbitrary minute of the hour.  That actually suits my purposes 
better than an arbitrary time of the hour would, but if people really 
wanted more-cron-line specific minutes of the hour, that would be set 
using OnCalendar= instead of the boot+10min and hourly thereafter, that I 
used.


/etc/systemd/jed.hourly.timestamp.service:

[Unit]
Description=JED Hourly timestamp updater

[Service]
Type=oneshot
ExecStart=/etc/systemd/local-scripts/jed.hourly.timestamp

[Install]
WantedBy=jed.hourly.target


This service simply runs the script named in execstart, oneshot, meaning 
it runs until its done, then stops.  Which is what we want, since we're 
triggering it once per hour via the timers above.

The install section simply tells systemd where to put that symlink 
mentioned above, in the appropriate target.wants subdir, when the service 
is "installed".


/etc/systemd/local-scripts/jed.hourly.timestamp:

#!/bin/bash

stampfile=/var/log/jed.hourly.day.stamp

# if stamp is old or doesn't exist, update stampfile and run...
doit(){
	touch $stampfile
	systemctl start jed.daily.target
	exit
}

[[ -f stampfile ]] || doit

yesterday=$(date -d yesterday +"%Y%m%d%H%M")

stamp=$(date -r "$stampfile" +"%Y%m%d%H%M")

[[ $stamp ]] || doit

[[ $stamp -lt $yesterday ]] && doit

exit 0


This is where the custom magic happens and the real work gets done.  If 
you're familiar with the usual cronjob runscripts setup, this will look 
similar.  Basically, systemd timers, etc, magic above only sets up an 
hourly systemd timer, not a daily, weekly, etc.  This script actually 
sets up a daily trigger, based on a daily timestamp that's checked every 
hour to see if it has been 24 hours or longer since the last run, updated 
if so, and the daily trigger triggered.

As I said, the idea is very similar to that used with a standard cronjob 
runscripts setup, since that's exactly where I got it from. =:^)  Like 
the cronjob setup, if the system was down the hour the daily would have 
otherwise triggered, this script ensures that it's triggered the next 
time the hourly timer is run... which will normally be 10 minutes after 
boot due to the 24 hours elapsing when the system was down, and the boot
+10min setting in that first unit, jed.hourly.timer, above.

OK, so what does this script trigger?  systemctl start jed.daily.target

Again, I /could/ have setup a daily timer much like the hourly timer, and 
handled it that way.  Instead, I chose to use the hourly time to activate 
a service to run this script, handling the daily mechanism that way.

(FWIW I put the stampfile itself in /var/log primarily because that's a 
conveniently writable place to put it.  My /, including much of /var, is 
mounted read-only by default, while /var/log obviously must be writable.  
And thought about from a particular viewpoint, that timestamp /is/ a log 
in some fashion, since it effectively logs the last time the daily jobs 
should have run.)

OK, we're almost there now. =:^)  But tying up loose ends...


/etc/systemd/system/jed.daily.target:

[Unit]
Description=JED Daily Target
StopWhenUnneeded=yes


Pretty much identical to the hourly target as the functionality is the 
same, just with a different name, reflecting the different timing.


Symlinks in /etc/systemd/system/jed.daily.target.wants:

jed.daily.logrotate.service.


For the moment that's it.  I haven't needed weekly, monthly, etc, 
scheduling, so I've not created it.  But having done the hourly to daily 
thing, I could easily copy and convert that to weekly or whatever, 
triggered by the daily, just as daily is triggered by the hourly.

And to tie up the final loose end before I apply the above to the case of 
this thread, as well as to supply a sample logrotate systemd service for 
anyone else wanting to switch to systemd timer based triggering and get 
rid of cron's triggering (while noting that the systemd way to do it 
would be to use tmpfiles.d settings, but I still find logrotate useful 
for doing actual log rotation, thus keeping those settings separate from 
systemd's config)...


/etc/systemd/system/jed.daily.logrotate.service:

[Unit]
Description=JED daily logrotate

[Service]
Type=oneshot
Nice=10
ExecStart=/usr/sbin/logrotate --state /var/log/logrotate.state /etc/
logrotate.conf

[Install]
WantedBy=jed.daily.target


(Note the wrap of the execstart line...)

###################

OK, so how does all that apply to the case of this thread? =:^)

Simple enough.  Where I threw in that jed.hourly.timestamp.service systemd 
unit file (that is, symlinked into the hourly target.wants subdir) and 
jed.hourly.timestamp script that it called, you'd throw in another 
service, either in addition to the timestamp one I used, or if you don't 
want to use my daily solution, in place of it.


The key thing to understand here is that once you've setup a service to 
run a script you create yourself, you can have it do literally whatever 
you want it too, just as you would with any other script -- it's NO 
LONGER limited to the strict declarative style and options systemd 
provides -- you're ONLY using systemd to /start/ it. =:^)

So... suppose you want it to check hourly to see if it can and should 
trigger something else, but only run once a day... the timer and 
timestamp-script-based framework I have above already does the hourly 
check part, as well as the only running once a day part.

If you prefer that it check every 10 minutes, it's simple enough to 
change that hourly thing to 10 minutes.  If you like it hourly but want 
it to ALWAYS be hourly, including not checking the first time until an 
hour after boot, that's simple enough as well.  Similarly, if you want 
the final action to run only every two days, or twice a day, or 
whatever.  It's simple enough to change the timestamp mechanism to make 
that happen.

Meanwhile, with the basic framework already there, it should also be 
simple enough to plugin some conditional logic testing to see if the 
partition device-files are showing up or not.  If not, simply quit 
without updating the timestamp and wait for the next hourly or whatever 
check to roll around.  If so, and the timestamp is also expired, then run 
the scripted backup or whatever else you were wanting it to do.

... And of course you can rename them from jed.* to your own name of 
choice, too.  Jed.* is fine for my own usage, but it would look a bit 
ridiculous, and could well defeat its site-specific-id purpose if someone 
actually decided to ship it as part of some package, if that jed.* naming 
pattern remained as installed out of my own control.  But be sure to 
change the names /in/ the files too, not just the names /of/ the files, 
or systemd will be complaining about the stupid admin trying to give it 
nonsensical unit files to run. =:^P

Also, one last caution.  I chose to retype most of the above files 
manually, getting back into what they did and how as I did so, instead of 
copy/pasting them and then sitting there studying them to remember how 
they worked.  Not so boring that way, but there's possibly a few typos I 
missed.  They should be reasonably obvious and easy to fix, tho, if I did 
miss some...

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman