[gentoo-user] Bug in run-crons?

[gentoo-user] Bug in run-crons?

[Frank Steinmetzger]

[-- Attachment #1: Type: text/plain, Size: 1260 bytes --]

Hi, it’s me again, :)

I’m currently improving the cron setup on my Gentoo-powered NAS. I installed
zfs-auto-snapshot, which includes scripts for hourly, daily, weekly and
monthly creation of snapshots.

The NAS uses fcron for its cron daemon. In order for the scripts to work,
I enabled the run-crons script from sys-process/cronbase, which runs all
scripts in /etc/cron.{hourly,daily,...}. But I have the feeling that there
is a bug in the script:

It uses state files in /var/spool/cron/lastrun/ to know when each interval
was last run, so it only runs once per period. But: the age threshold for
the state file is period + 5 minutes. Shouldn’t that be period - 5 minutes?

My reasoning: assume run-crons is run hourly, at the 0 minute sharp. So at
the next run, the state file is exactly one hour old. Since this is not old
enough for the check, run-crons thinks that the last run is too recent and
ignores this period. As a result, each period is only run on every other
iteration.

Any thoughts on that?
Cheerio.

-- 
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

“Today I watched my first porn movie.” – “And?” – “I was so young back then.”

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Bug in run-crons?

[Rich Freeman]

On Sun, Dec 12, 2021 at 1:21 PM Frank Steinmetzger <Warp_7@gmx.de> wrote:
>
> It uses state files in /var/spool/cron/lastrun/ to know when each interval
> was last run, so it only runs once per period. But: the age threshold for
> the state file is period + 5 minutes. Shouldn’t that be period - 5 minutes?
>
> My reasoning: assume run-crons is run hourly, at the 0 minute sharp. So at
> the next run, the state file is exactly one hour old. Since this is not old
> enough for the check, run-crons thinks that the last run is too recent and
> ignores this period. As a result, each period is only run on every other
> iteration.

I don't use this, but I believe there should be an hourly crontab
entry that deletes the cron.hourly file, which would mean it gets run
on the next 10min cycle (or maybe sooner - I'm not sure if those jobs
are run in parallel or serial).

The overall result is that things will get run at traditional crontab
times no matter what, but if those are missed and enough time is
accrued then run-crons will catch this and run the job on the next
10min cycle, such as a missed daily overnight slot.

So if you don't have your computer on a 3AM then the job will get run
a bit more than 24h after the last time it was run.  Then if you leave
it on overnight that night it will run at 3AM despite it not being
24h, but at that point if you leave it on it will run at 3AM daily
since it will no longer be missed.  The fixed times serve to anchor
the actual execution times if you do leave the computer on.

-- 
Rich

Re: [gentoo-user] Bug in run-crons?

[Frank Steinmetzger]

[-- Attachment #1: Type: text/plain, Size: 1737 bytes --]

Am Sun, Dec 12, 2021 at 01:41:33PM -0500 schrieb Rich Freeman:
> On Sun, Dec 12, 2021 at 1:21 PM Frank Steinmetzger <Warp_7@gmx.de> wrote:
> >
> > It uses state files in /var/spool/cron/lastrun/ to know when each interval
> > was last run, so it only runs once per period. But: the age threshold for
> > the state file is period + 5 minutes. Shouldn’t that be period - 5 minutes?
> >
> > My reasoning: assume run-crons is run hourly, at the 0 minute sharp. So at
> > the next run, the state file is exactly one hour old. Since this is not old
> > enough for the check, run-crons thinks that the last run is too recent and
> > ignores this period. As a result, each period is only run on every other
> > iteration.
> 
> I don't use this, but I believe there should be an hourly crontab
> entry that deletes the cron.hourly file, which would mean it gets run
> on the next 10min cycle (or maybe sooner - I'm not sure if those jobs
> are run in parallel or serial).

The check that I mentioned above is actually the deletion which you mention:
run-crons looks for the state file for the given interval and - if it is old
enough - deletes it.

The part that executes the individual cron scripts is only executed if there
is no state file. The first thing it then does is to create a new state file.


In pseudo code:
1. look for period state file that is older than interval + 5 mins
found one?
    delete it

2. Look for period state file
none found?
    create state file
    execute cron scripts for that interval

-- 
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

The boss is a human just like everyone else, he just doesn’t know.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Bug in run-crons?

[Rich Freeman]

On Sun, Dec 12, 2021 at 2:07 PM Frank Steinmetzger <Warp_7@gmx.de> wrote:
>
> Am Sun, Dec 12, 2021 at 01:41:33PM -0500 schrieb Rich Freeman:
> > On Sun, Dec 12, 2021 at 1:21 PM Frank Steinmetzger <Warp_7@gmx.de> wrote:
> > >
> >
> > I don't use this, but I believe there should be an hourly crontab
> > entry that deletes the cron.hourly file, which would mean it gets run
> > on the next 10min cycle (or maybe sooner - I'm not sure if those jobs
> > are run in parallel or serial).
>
> The check that I mentioned above is actually the deletion which you mention:
> run-crons looks for the state file for the given interval and - if it is old
> enough - deletes it.

The check I'm talking about isn't in run-crons at all.  It is in
/etc/crontab.  It doesn't look at the age of the file and
unconditionally deletes it every hour:
59  *  * * *    rm -f /var/spool/cron/lastrun/cron.hourly

-- 
Rich

Re: [gentoo-user] Bug in run-crons?

[Frank Steinmetzger]

[-- Attachment #1: Type: text/plain, Size: 1703 bytes --]

Am Mon, Dec 13, 2021 at 03:18:44PM -0500 schrieb Rich Freeman:
> On Sun, Dec 12, 2021 at 2:07 PM Frank Steinmetzger <Warp_7@gmx.de> wrote:
> >
> > Am Sun, Dec 12, 2021 at 01:41:33PM -0500 schrieb Rich Freeman:
> > > On Sun, Dec 12, 2021 at 1:21 PM Frank Steinmetzger <Warp_7@gmx.de> wrote:
> > > >
> > >
> > > I don't use this, but I believe there should be an hourly crontab
> > > entry that deletes the cron.hourly file, which would mean it gets run
> > > on the next 10min cycle (or maybe sooner - I'm not sure if those jobs
> > > are run in parallel or serial).
> >
> > The check that I mentioned above is actually the deletion which you mention:
> > run-crons looks for the state file for the given interval and - if it is old
> > enough - deletes it.
> 
> The check I'm talking about isn't in run-crons at all.  It is in
> /etc/crontab.  It doesn't look at the age of the file and
> unconditionally deletes it every hour:
> 59  *  * * *    rm -f /var/spool/cron/lastrun/cron.hourly

I had a look at files and docs on the net again. Thus I found exactly those
rm entries in /etc/crontab, which by itself is not used by fcron. But after
I understood all the logic behind it, I added them to fcron to be run
serially before run-crons. Now everything is as I wanted it.

For the record: The checks in run-crons that I referred to earlier are
actually more for those cases in which the machine was powered off for a
while in order to restore cron completeness as early as possible after boot.

-- 
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

“Selfies are electronic masturbation.” — Karl Lagerfeld

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Bug in run-crons?

[Michael Orlitzky]

On Mon, 2021-12-13 at 22:19 +0100, Frank Steinmetzger wrote:
> 
> For the record: The checks in run-crons that I referred to earlier are
> actually more for those cases in which the machine was powered off for a
> while in order to restore cron completeness as early as possible after boot.
> 

The run-crons quackery has been causing problems since 2004:

  https://bugs.gentoo.org/69777

One-liners with run-parts (NOT run-crons) are a lot more predictable.

Re: [gentoo-user] Bug in run-crons?

[Frank Steinmetzger]

[-- Attachment #1: Type: text/plain, Size: 1216 bytes --]

Am Mon, Dec 13, 2021 at 04:33:32PM -0500 schrieb Michael Orlitzky:
> On Mon, 2021-12-13 at 22:19 +0100, Frank Steinmetzger wrote:
> > 
> > For the record: The checks in run-crons that I referred to earlier are
> > actually more for those cases in which the machine was powered off for a
> > while in order to restore cron completeness as early as possible after boot.
> > 
> 
> The run-crons quackery has been causing problems since 2004:
> 
>   https://bugs.gentoo.org/69777
> 
> One-liners with run-parts (NOT run-crons) are a lot more predictable.

Well I *could* disable run-crons altogether and add entries to fcron’s own
crontab which would run those scripts in /etc/cron.{hourly,daily,...}
instead.

However, I like predictable times at which those jobs will run. Especially
if one of them is a zfs scrub; the NAS is powered down for weeks, sometimes
months. And when I power it up, it’s for a reason. And that reason usually
is not a scrub, which—at the current zfs fill level—takes 10½ hours.

-- 
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

A preposition is something you should never end a sentence with.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Bug in run-crons?

[Michael Orlitzky]

On Mon, 2021-12-13 at 22:38 +0100, Frank Steinmetzger wrote:
> 
> Well I *could* disable run-crons altogether and add entries to fcron’s own
> crontab which would run those scripts in /etc/cron.{hourly,daily,...}
> instead.
> 
> However, I like predictable times at which those jobs will run. Especially
> if one of them is a zfs scrub; the NAS is powered down for weeks, sometimes
> months. And when I power it up, it’s for a reason. And that reason usually
> is not a scrub, which—at the current zfs fill level—takes 10½ hours.
> 

Why choose fcron then? It sounds like you have the same rationale as I
do: "no, I don't want to run the 4am backup job in the middle of the
business day just because it wasn't run at 4am."

If you pick a dumber cron, the crontab entries are run only at the
specified times.

Re: [gentoo-user] Bug in run-crons?

[Rich Freeman]

On Mon, Dec 13, 2021 at 4:42 PM Michael Orlitzky <mjo@gentoo.org> wrote:
>
> On Mon, 2021-12-13 at 22:38 +0100, Frank Steinmetzger wrote:
> >
> > Well I *could* disable run-crons altogether and add entries to fcron’s own
> > crontab which would run those scripts in /etc/cron.{hourly,daily,...}
> > instead.
> >
> > However, I like predictable times at which those jobs will run. Especially
> > if one of them is a zfs scrub; the NAS is powered down for weeks, sometimes
> > months. And when I power it up, it’s for a reason. And that reason usually
> > is not a scrub, which—at the current zfs fill level—takes 10½ hours.
> >
>
> Why choose fcron then? It sounds like you have the same rationale as I
> do: "no, I don't want to run the 4am backup job in the middle of the
> business day just because it wasn't run at 4am."
>
> If you pick a dumber cron, the crontab entries are run only at the
> specified times.

There really are no one-size-fits all solutions obviously.  You can
either have predictable times and if the machine is down things get
skipped, or you can run opportunistically in which case you lose
predictable times at least until the machine goes back to running
24x7.  The machine has no way to know if you'll leave it on overnight
the NEXT night - only if you did so the one before.

fcron is perfectly capable of running jobs at either set times or at
loosely-defined intervals that are compensated for if the machine is
off.  Systemd timers can also be set that way, and can also have
limits put on scheduling (daily nominally at 3AM but allowed between
11PM and 5AM or whatever).  Obviously the more expressive the crontab
equivalent is, the more you can tweak it to do what you want it to.

The default configs are a one-size-fits all approach that tend to do
the right thing if you leave the machine on 24x7, or if you run it at
random times.  If you want to specify exact behavior just edit the
config files and make it do so.

I'll also note that zfs scrubs checkpoint at shutdown and will
auto-resume where you leave off.  If they are involved multiple times
with the default options I think any attempt to scrub something that
is already being scrubbed is just a no-op.  Obviously if you don't
want all that IO during the day you'll have to do something more
clever - you can instruct it to pause and resume so you could have a
couple of crontab entries and scripts to do just that.

-- 
Rich

Re: [gentoo-user] Bug in run-crons?

[Frank Steinmetzger]

[-- Attachment #1: Type: text/plain, Size: 2686 bytes --]

Am Mon, Dec 13, 2021 at 04:54:30PM -0500 schrieb Rich Freeman:
> On Mon, Dec 13, 2021 at 4:42 PM Michael Orlitzky <mjo@gentoo.org> wrote:
> >
> > On Mon, 2021-12-13 at 22:38 +0100, Frank Steinmetzger wrote:
> > >
> > > Well I *could* disable run-crons altogether and add entries to fcron’s own
> > > crontab which would run those scripts in /etc/cron.{hourly,daily,...}
> > > instead.
> > >
> > > However, I like predictable times at which those jobs will run. Especially
> > > if one of them is a zfs scrub; the NAS is powered down for weeks, sometimes
> > > months. And when I power it up, it’s for a reason. And that reason usually
> > > is not a scrub, which—at the current zfs fill level—takes 10½ hours.
> > >
> >
> > Why choose fcron then? It sounds like you have the same rationale as I
> > do: "no, I don't want to run the 4am backup job in the middle of the
> > business day just because it wasn't run at 4am."

Fair point.
Fcron has a serialisation feature, so that for instance updatedb and mandb
don’t run at the same time. Me kinda liky.

> fcron is perfectly capable of running jobs at either set times or at
> loosely-defined intervals that are compensated for if the machine is
> off.

Indeed, I can say "@monthly * 0 *" to run a monthly job (like the scrub)
only at night. *pondering*

> Systemd timers can also be set that way, and can also have
> limits put on scheduling (daily nominally at 3AM but allowed between
> 11PM and 5AM or whatever).  Obviously the more expressive the crontab
> equivalent is, the more you can tweak it to do what you want it to.

I’ve gotten used to systemd on my Arch-based desktops. But the NAS is still
on good-ol’ openrc. :)

> I'll also note that zfs scrubs checkpoint at shutdown and will
> auto-resume where you leave off.

That’s a nice bit of knowledge.

> If they are involved multiple times
> with the default options I think any attempt to scrub something that
> is already being scrubbed is just a no-op.  Obviously if you don't
> want all that IO during the day you'll have to do something more
> clever - you can instruct it to pause and resume so you could have a
> couple of crontab entries and scripts to do just that.

Or, since I am the only user of that system, I could go back to my previous
way: just run the scrub manually every other month or so before I go to bed.
Because then I know that nothing will interfere with it and it won’t
interfere with anything else.

-- 
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

This message was written using only recycled electrons.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [gentoo-user] Bug in run-crons?

[Wols Lists]

On 13/12/2021 22:03, Frank Steinmetzger wrote:
>> If they are involved multiple times
>> with the default options I think any attempt to scrub something that
>> is already being scrubbed is just a no-op.  Obviously if you don't
>> want all that IO during the day you'll have to do something more
>> clever - you can instruct it to pause and resume so you could have a
>> couple of crontab entries and scripts to do just that.

> Or, since I am the only user of that system, I could go back to my previous
> way: just run the scrub manually every other month or so before I go to bed.
> Because then I know that nothing will interfere with it and it won’t
> interfere with anything else.

Yup. I was thinking that. The problem with scrubs et al auto-resuming or 
firing at boot is that disk i/o is then knackered for ages, and 
interferes with whatever it is you're actually trying to do :-)

Cheers,
Wol