[gentoo-user] journald refuses to put log files in /var/log/journal/

[gentoo-user] journald refuses to put log files in /var/log/journal/

[walt]

My main desktop machine is obviously having a brain fart :(

systemd-journald is allegedly obligated to write its journal files
to /var/log/journal/ *if* that directory exists, right?

Well, on my three other gentoo ~amd64 machines, that's exactly what
journald does.

But not on my everyday work machine, oh no.  I'd be daft to expect
my one main everyday machine to obey the rules, right?

On this machine (the one I'm using now) journald is writing its
files to /run/log/journal/ instead of /var/log/journal/

# ls -l /var/log/journal/
total 4
drwxr-sr-x 2 root systemd-journal-remote 4096 Sep 22 14:39 remote

#ls -l /var/log/journal/remote/
total 0


The *.conf files in /etc/systemd/ are the same on all machines:
all of the config items are commented out, as sys-apps/systemd
installed them.

So, why is this particular machine not behaving like the others?

Any debugging ideas would be most welcome.

Thanks.

Re: [gentoo-user] journald refuses to put log files in /var/log/journal/

[Rich Freeman]

On Mon, Sep 22, 2014 at 8:41 PM, walt <w41ter@gmail.com> wrote:
> On this machine (the one I'm using now) journald is writing its
> files to /run/log/journal/ instead of /var/log/journal/

Check the config:
/etc/systemd/journald.conf:

       Storage=
           Controls where to store journal data. One of "volatile",
"persistent", "auto" and "none". If
           "volatile", journal log data will be stored only in memory,
i.e. below the /run/log/journal hierarchy
           (which is created if needed). If "persistent", data will be
stored preferably on disk, i.e. below the
           /var/log/journal hierarchy (which is created if needed),
with a fallback to /run/log/journal (which is
           created if needed), during early boot and if the disk is
not writable.  "auto" is similar to
           "persistent" but the directory /var/log/journal is not
created if needed, so that its existence
           controls where log data goes.  "none" turns off all
storage, all log data received will be dropped.
           Forwarding to other targets, such as the console, the
kernel log buffer or a syslog daemon will still
           work however. Defaults to "auto".

As others frequently point out not everybody likes the journal so you
can use syslog and just keep the journal in /run where it will not be
persistent.  If that isn't the desire, make sure the config isn't set
to volatile.

--
Rich

Re: [gentoo-user] journald refuses to put log files in /var/log/journal/

[Canek Peláez Valdés]

On Mon, Sep 22, 2014 at 7:41 PM, walt <w41ter@gmail.com> wrote:
> My main desktop machine is obviously having a brain fart :(
>
> systemd-journald is allegedly obligated to write its journal files
> to /var/log/journal/ *if* that directory exists, right?
>
> Well, on my three other gentoo ~amd64 machines, that's exactly what
> journald does.
>
> But not on my everyday work machine, oh no.  I'd be daft to expect
> my one main everyday machine to obey the rules, right?
>
> On this machine (the one I'm using now) journald is writing its
> files to /run/log/journal/ instead of /var/log/journal/
>
> # ls -l /var/log/journal/
> total 4
> drwxr-sr-x 2 root systemd-journal-remote 4096 Sep 22 14:39 remote
>
> #ls -l /var/log/journal/remote/
> total 0
>
>
> The *.conf files in /etc/systemd/ are the same on all machines:
> all of the config items are commented out, as sys-apps/systemd
> installed them.
>
> So, why is this particular machine not behaving like the others?

Hi Walt; the relevant documentation is from man 8 systemd-journald:

"By default, the journal stores log data in /run/log/journal/. Since
/run/ is volatile, log data is lost at reboot. To make the data
persistent, it is sufficient to create /var/log/journal/ where
systemd-journald will then store the data."

So, in the failing machine the journal is not flushing its volatile
data to /var. I would suspect a permissions issue. Could you please
post the output from:

# ls -ld /var/log/journal

In my main machine, this is:

drwxr-sr-x 3 root systemd-journal 4096 Oct 28  2012 /var/log/journal

So its 2755; all permissions for root, read and execution (with SETGID
bit on), and read and execution for everyone else. The directory is
owned by root, and it's on the systemd-journal group.

Regards.
-- 
Canek Peláez Valdés
Profesor de asignatura, Facultad de Ciencias
Universidad Nacional Autónoma de México

[gentoo-user] Re: journald refuses to put log files in /var/log/journal/ [SOLVED]

[walt]

On 09/22/2014 08:50 PM, Canek Peláez Valdés wrote:
> On Mon, Sep 22, 2014 at 7:41 PM, walt <w41ter@gmail.com> wrote:
>> My main desktop machine is obviously having a brain fart :(
>>
>> systemd-journald is allegedly obligated to write its journal files
>> to /var/log/journal/ *if* that directory exists, right?
>>
>> Well, on my three other gentoo ~amd64 machines, that's exactly what
>> journald does.
>>
>> But not on my everyday work machine, oh no.  I'd be daft to expect
>> my one main everyday machine to obey the rules, right?
>>
>> On this machine (the one I'm using now) journald is writing its
>> files to /run/log/journal/ instead of /var/log/journal/
>>
>> # ls -l /var/log/journal/
>> total 4
>> drwxr-sr-x 2 root systemd-journal-remote 4096 Sep 22 14:39 remote
>>
>> #ls -l /var/log/journal/remote/
>> total 0
>>
>>
>> The *.conf files in /etc/systemd/ are the same on all machines:
>> all of the config items are commented out, as sys-apps/systemd
>> installed them.
>>
>> So, why is this particular machine not behaving like the others?
> 
> Hi Walt; the relevant documentation is from man 8 systemd-journald:
> 
> "By default, the journal stores log data in /run/log/journal/. Since
> /run/ is volatile, log data is lost at reboot. To make the data
> persistent, it is sufficient to create /var/log/journal/ where
> systemd-journald will then store the data."
> 
> So, in the failing machine the journal is not flushing its volatile
> data to /var. I would suspect a permissions issue. Could you please
> post the output from:
> 
> # ls -ld /var/log/journal
> 
> In my main machine, this is:
> 
> drwxr-sr-x 3 root systemd-journal 4096 Oct 28  2012 /var/log/journal
> 
> So its 2755; all permissions for root, read and execution (with SETGID
> bit on), and read and execution for everyone else. The directory is
> owned by root, and it's on the systemd-journal group.

Thanks, Rich and Canek.  I fixed the problem by accident while trying
to debug it.

I used systemctl to stop and restart systemd-journald, thinking I might
see some useful error messages.  But when systemd-journal started up
again the journal file was back in /var/log/journal where I want it :)

No idea why rebooting the machine didn't do the same thing.

Thanks.

Re: [gentoo-user] Re: journald refuses to put log files in /var/log/journal/ [SOLVED]

[Rich Freeman]

On Tue, Sep 23, 2014 at 10:27 AM, walt <w41ter@gmail.com> wrote:
>
> I used systemctl to stop and restart systemd-journald, thinking I might
> see some useful error messages.  But when systemd-journal started up
> again the journal file was back in /var/log/journal where I want it :)
>
> No idea why rebooting the machine didn't do the same thing.
>

Are you sure that it is solved, and that the problem won't recur on
the next reboot?

If it does, my next question (an educated guess, but a guess) would be
whether you're using an initramfs, and if so which one.  Dracut in
particular launches journald, but it should move its output to
/var/log/journal after pivoting to the new root.  It is actually nice
because your log contains early boot data which of course would not be
present in syslog unless it ended up in the ring buffer.  However,
perhaps something is going wrong with that.  I'd also look at anything
that might be causing issues with /var/log/journal when journald is
launched, such as that directory being on an unmounted filesystem and
there not being some dependency that causes journald to notice.
Systemd is pretty smart about spotting mount dependencies, but I've
seen it make mistakes in unusual configurations and I don't know what
the full logic is.

--
Rich

[gentoo-user] Re: journald refuses to put log files in /var/log/journal/ [NOT SOLVED]

[walt]

On 09/23/2014 07:46 AM, Rich Freeman wrote:
> On Tue, Sep 23, 2014 at 10:27 AM, walt <w41ter@gmail.com> wrote:
>>
>> I used systemctl to stop and restart systemd-journald, thinking I might
>> see some useful error messages.  But when systemd-journal started up
>> again the journal file was back in /var/log/journal where I want it :)
>>
>> No idea why rebooting the machine didn't do the same thing.
>>
> 
> Are you sure that it is solved, and that the problem won't recur on
> the next reboot?

<sigh>  After a reboot the journal file is back in /run/log/journal.

> If it does, my next question (an educated guess, but a guess) would be
> whether you're using an initramfs,

No, I never have.

> I'd also look at anything
> that might be causing issues with /var/log/journal when journald is
> launched, such as that directory being on an unmounted filesystem and
> there not being some dependency that causes journald to notice.

This particular machine has only root and swap partitions, so there's
nothing to remain unmounted during boot.

Having reassured myself with that claim, I now spot this journal message
(which appears only on the 'broken' machine):

Sep 23 07:40:46 a6 systemd[1]: Found ordering cycle on sysinit.target/start
Sep 23 07:40:46 a6 systemd[1]: Found dependency on local-fs.target/start
Sep 23 07:40:46 a6 systemd[1]: Found dependency on lvm.service/start
Sep 23 07:40:46 a6 systemd[1]: Found dependency on sysinit.target/start
Sep 23 07:40:46 a6 systemd[1]: Breaking ordering cycle by deleting job local-fs.target/start
Sep 23 07:40:46 a6 systemd[1]: Job local-fs.target/start deleted to break ordering cycle starting with sysinit.target/start

I don't understand everything about that message, but it seems to imply
that systemd may think that the local filesystems are not mounted(?)

Could this be causing my journald problem, maybe?

Re: [gentoo-user] Re: journald refuses to put log files in /var/log/journal/ [NOT SOLVED]

[Mike Gilbert]

On Tue, Sep 23, 2014 at 1:40 PM, walt <w41ter@gmail.com> wrote:
> On 09/23/2014 07:46 AM, Rich Freeman wrote:
>> On Tue, Sep 23, 2014 at 10:27 AM, walt <w41ter@gmail.com> wrote:
>>>
>>> I used systemctl to stop and restart systemd-journald, thinking I might
>>> see some useful error messages.  But when systemd-journal started up
>>> again the journal file was back in /var/log/journal where I want it :)
>>>
>>> No idea why rebooting the machine didn't do the same thing.
>>>
>>
>> Are you sure that it is solved, and that the problem won't recur on
>> the next reboot?
>
> <sigh>  After a reboot the journal file is back in /run/log/journal.
>
>> If it does, my next question (an educated guess, but a guess) would be
>> whether you're using an initramfs,
>
> No, I never have.
>
>> I'd also look at anything
>> that might be causing issues with /var/log/journal when journald is
>> launched, such as that directory being on an unmounted filesystem and
>> there not being some dependency that causes journald to notice.
>
> This particular machine has only root and swap partitions, so there's
> nothing to remain unmounted during boot.
>
> Having reassured myself with that claim, I now spot this journal message
> (which appears only on the 'broken' machine):
>
> Sep 23 07:40:46 a6 systemd[1]: Found ordering cycle on sysinit.target/start
> Sep 23 07:40:46 a6 systemd[1]: Found dependency on local-fs.target/start
> Sep 23 07:40:46 a6 systemd[1]: Found dependency on lvm.service/start
> Sep 23 07:40:46 a6 systemd[1]: Found dependency on sysinit.target/start
> Sep 23 07:40:46 a6 systemd[1]: Breaking ordering cycle by deleting job local-fs.target/start
> Sep 23 07:40:46 a6 systemd[1]: Job local-fs.target/start deleted to break ordering cycle starting with sysinit.target/start
>
> I don't understand everything about that message, but it seems to imply
> that systemd may think that the local filesystems are not mounted(?)
>
> Could this be causing my journald problem, maybe?

Where is lvm.service coming from? I suspect it is causing the ordering cycle.

Re: [gentoo-user] Re: journald refuses to put log files in /var/log/journal/ [NOT SOLVED]

[Rich Freeman]

On Tue, Sep 23, 2014 at 2:28 PM, Mike Gilbert <floppym@gentoo.org> wrote:
>>
>> I don't understand everything about that message, but it seems to imply
>> that systemd may think that the local filesystems are not mounted(?)
>>
>> Could this be causing my journald problem, maybe?
>
> Where is lvm.service coming from? I suspect it is causing the ordering cycle.
>

It doesn't mean that systemd thinks they aren't mounted - it means it
isn't trying to mount them before launching journald.  That might
include not remounting root as read-write if your kernel is mounting
it read-only (in the absence of an initramfs).  I'd have to look at
the units to see if that is part of that unit.

--
Rich

[gentoo-user] Re: journald refuses to put log files in /var/log/journal/ [REALLY SOLVED]

[walt]

On 09/23/2014 11:28 AM, Mike Gilbert wrote:
> On Tue, Sep 23, 2014 at 1:40 PM, walt <w41ter@gmail.com> wrote:
>> On 09/23/2014 07:46 AM, Rich Freeman wrote:
>>> On Tue, Sep 23, 2014 at 10:27 AM, walt <w41ter@gmail.com> wrote:
>>>>
>>>> I used systemctl to stop and restart systemd-journald, thinking I might
>>>> see some useful error messages.  But when systemd-journal started up
>>>> again the journal file was back in /var/log/journal where I want it :)
>>>>
>>>> No idea why rebooting the machine didn't do the same thing.
>>>>
>>>
>>> Are you sure that it is solved, and that the problem won't recur on
>>> the next reboot?
>>
>> <sigh>  After a reboot the journal file is back in /run/log/journal.
>>
>>> If it does, my next question (an educated guess, but a guess) would be
>>> whether you're using an initramfs,
>>
>> No, I never have.
>>
>>> I'd also look at anything
>>> that might be causing issues with /var/log/journal when journald is
>>> launched, such as that directory being on an unmounted filesystem and
>>> there not being some dependency that causes journald to notice.
>>
>> This particular machine has only root and swap partitions, so there's
>> nothing to remain unmounted during boot.
>>
>> Having reassured myself with that claim, I now spot this journal message
>> (which appears only on the 'broken' machine):
>>
>> Sep 23 07:40:46 a6 systemd[1]: Found ordering cycle on sysinit.target/start
>> Sep 23 07:40:46 a6 systemd[1]: Found dependency on local-fs.target/start
>> Sep 23 07:40:46 a6 systemd[1]: Found dependency on lvm.service/start
>> Sep 23 07:40:46 a6 systemd[1]: Found dependency on sysinit.target/start
>> Sep 23 07:40:46 a6 systemd[1]: Breaking ordering cycle by deleting job local-fs.target/start
>> Sep 23 07:40:46 a6 systemd[1]: Job local-fs.target/start deleted to break ordering cycle starting with sysinit.target/start
>>
>> I don't understand everything about that message, but it seems to imply
>> that systemd may think that the local filesystems are not mounted(?)
>>
>> Could this be causing my journald problem, maybe?
> 
> Where is lvm.service coming from? I suspect it is causing the ordering cycle.

Great pickup, thanks Mike.

I took lvm.service from either Arch or Fedora way back when gentoo first
introduced systemd.  For whatever reason my lvm volumes weren't being activated
during bootup.  The lvm.service file fixed that for a long time and caused no
trouble until recently.

I just disabled the lvm.service and rebooted:  the journal file is back in
/var/log/journal where it should be :)

Just as important, the lvm filesystem is properly mounted during bootup, so
somewhere along the line that problem got fixed independently of systemd.

Thanks again Mike, Rich, and Canek.  Excellent help and much appreciated.