[gentoo-user] syslog-ng: how to read the log files

[gentoo-user] syslog-ng: how to read the log files

[lee]

Hi,

how do you read the log files when using syslog-ng?

The log file seem to be some sort of binary that doesn't display too
well in less, and there doesn't seem to be any way to read them.


-- 
Again we must be afraid of speaking of daemons for fear that daemons
might swallow us.  Finally, this fear has become reasonable.

Re: [gentoo-user] syslog-ng: how to read the log files

[Rich Freeman]

On Tue, Feb 17, 2015 at 1:26 PM, lee <lee@yagibdah.de> wrote:
> Hi,
>
> how do you read the log files when using syslog-ng?
>
> The log file seem to be some sort of binary that doesn't display too
> well in less, and there doesn't seem to be any way to read them.

That's news to me.  Are you sure you're not looking at wtmp or
something like that (which isn't maintained by syslog)?

-- 
Rich

Re: [gentoo-user] syslog-ng: how to read the log files

[Alexander Kapshuk]

[-- Attachment #1: Type: text/plain, Size: 495 bytes --]

On Tue, Feb 17, 2015 at 8:26 PM, lee <lee@yagibdah.de> wrote:

> Hi,
>
> how do you read the log files when using syslog-ng?
>
> The log file seem to be some sort of binary that doesn't display too
> well in less, and there doesn't seem to be any way to read them.
>
>
> --
> Again we must be afraid of speaking of daemons for fear that daemons
> might swallow us.  Finally, this fear has become reasonable.
>
>
If you're talking about /var/log/messages, which is:
messages: data

I use cat(1).

[-- Attachment #2: Type: text/html, Size: 983 bytes --]

Re: [gentoo-user] syslog-ng: how to read the log files

[Matti Nykyri]

> On Feb 17, 2015, at 20:26, lee <lee@yagibdah.de> wrote:
> 
> Hi,
> 
> how do you read the log files when using syslog-ng?
> 
> The log file seem to be some sort of binary that doesn't display too
> well in less, and there doesn't seem to be any way to read them.

This was discussed earlier on this list... Actually what syslog-ng produces is plain text. There seemed to be a bug that creates some binary (i.e. unreadable characters) and that causes less to consider files to be binary and show them incorrectly.

To work around you can use -r flag with less, or replace/remove unreadable chars from log, or delete the log file.

-- 
-Matti

[gentoo-user] Re: syslog-ng: how to read the log files

[Nikos Chantziaras]

On 17/02/15 20:26, lee wrote:
> Hi,
>
> how do you read the log files when using syslog-ng?
>
> The log file seem to be some sort of binary that doesn't display too
> well in less, and there doesn't seem to be any way to read them.

As others said, it's probably a bug and /var/log/messages is actually 
really a text file.

FWIW, I'm on syslog-ng 3.6.2 and it seems to not have this bug.

Re: [gentoo-user] syslog-ng: how to read the log files

[Alan Mackenzie]

Hello, Lee.

On Tue, Feb 17, 2015 at 07:26:05PM +0100, lee wrote:
> Hi,

> how do you read the log files when using syslog-ng?

> The log file seem to be some sort of binary that doesn't display too
> well in less, and there doesn't seem to be any way to read them.

When I try "less /var/log/messages", less gives me what is basically a
hex dump of the file.  I'm assuming you see the same.

less searches part of the buffer (presumably the first few KB) and if it
finds non-printable characters, uses an input filter first to convert to
the hex dump.  This same filter is what enables less to expand
compressed files and man pages.

What I do is to disable this input filter with

    # LESSOPEN="" less /var/log/messages

.  It is evident that every now and then, syslog-ng writes a stream of
several hundred null bytes to /var/log/messages.  It seems to do this
when logging the system startup messages.  This is probably a bug.

By the way, the LESSOPEN="" trick can sometimes leave your display
corrupted, displaying wierd glyphs on the screen when you type.  To
restore your screen, output ^o.  To do this, type (blindly)

    # echo <ctrl-v><crtl-o><CR>

.

> -- 
> Again we must be afraid of speaking of daemons for fear that daemons
> might swallow us.  Finally, this fear has become reasonable.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: [gentoo-user] syslog-ng: how to read the log files

[Alexander Kapshuk]

[-- Attachment #1: Type: text/plain, Size: 679 bytes --]

On Tue, Feb 17, 2015 at 8:38 PM, Alexander Kapshuk <
alexander.kapshuk@gmail.com> wrote:

> On Tue, Feb 17, 2015 at 8:26 PM, lee <lee@yagibdah.de> wrote:
>
>> Hi,
>>
>> how do you read the log files when using syslog-ng?
>>
>> The log file seem to be some sort of binary that doesn't display too
>> well in less, and there doesn't seem to be any way to read them.
>>
>>
>> --
>> Again we must be afraid of speaking of daemons for fear that daemons
>> might swallow us.  Finally, this fear has become reasonable.
>>
>>
> If you're talking about /var/log/messages, which is:
> messages: data
>
> I use cat(1).
>
>
Just tried 'sed p /var/log/messages', which seems to work as well.

[-- Attachment #2: Type: text/html, Size: 1527 bytes --]

Re: [gentoo-user] syslog-ng: how to read the log files

[lee]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Lee.
>
> On Tue, Feb 17, 2015 at 07:26:05PM +0100, lee wrote:
>> Hi,
>
>> how do you read the log files when using syslog-ng?
>
>> The log file seem to be some sort of binary that doesn't display too
>> well in less, and there doesn't seem to be any way to read them.
>
> When I try "less /var/log/messages", less gives me what is basically a
> hex dump of the file.  I'm assuming you see the same.

Yes, that's what I was looking at.

> less searches part of the buffer (presumably the first few KB) and if it
> finds non-printable characters, uses an input filter first to convert to
> the hex dump.

Is that a new feature of less?  I've never had this problem with any
other file.  IIRC, unprintable characters, like null, used to be
displayed like ^@, and less always did a great job in preventing the
display from needing a reset without switching to an equivalent of
hexl-mode.


BTW, what happens when something writes to /var/log/messages?  I noticed
today that the default shorewall.conf that ships with gentoo has that
set as logfile for shorewall.  Shouldn't all messages going into
/var/log/messages go to syslog-ng instead when syslog-ng is used, with
nothing else writing to this file?


-- 
Again we must be afraid of speaking of daemons for fear that daemons
might swallow us.  Finally, this fear has become reasonable.

Re: [gentoo-user] syslog-ng: how to read the log files

[covici]

Alexander Kapshuk <alexander.kapshuk@gmail.com> wrote:

> On Tue, Feb 17, 2015 at 8:26 PM, lee <lee@yagibdah.de> wrote:
> 
> > Hi,
> >
> > how do you read the log files when using syslog-ng?
> >
> > The log file seem to be some sort of binary that doesn't display too
> > well in less, and there doesn't seem to be any way to read them.
> >
> >
> > --
> > Again we must be afraid of speaking of daemons for fear that daemons
> > might swallow us.  Finally, this fear has become reasonable.
> >
> >
> If you're talking about /var/log/messages, which is:
> messages: data
> 
> I use cat(1).

I wonder if the OP is using systemd and trying to read the journal
files?

-- 
Your life is like a penny.  You're going to lose it.  The question is:
How do
you spend it?

         John Covici
         covici@ccs.covici.com

Re: [gentoo-user] syslog-ng: how to read the log files

[Canek Peláez Valdés]

[-- Attachment #1: Type: text/plain, Size: 1528 bytes --]

On Tue, Feb 17, 2015 at 2:29 PM, <covici@ccs.covici.com> wrote:
>
> Alexander Kapshuk <alexander.kapshuk@gmail.com> wrote:
>
> > On Tue, Feb 17, 2015 at 8:26 PM, lee <lee@yagibdah.de> wrote:
> >
> > > Hi,
> > >
> > > how do you read the log files when using syslog-ng?
> > >
> > > The log file seem to be some sort of binary that doesn't display too
> > > well in less, and there doesn't seem to be any way to read them.
> > >
> > >
> > > --
> > > Again we must be afraid of speaking of daemons for fear that daemons
> > > might swallow us.  Finally, this fear has become reasonable.
> > >
> > >
> > If you're talking about /var/log/messages, which is:
> > messages: data
> >
> > I use cat(1).
>
> I wonder if the OP is using systemd and trying to read the journal
> files?

Those live under /var/lib/journal (which you need to create; Gentoo doesn't
do it by default last time I saw), in the directory named after the machine
ID (cat /etc/machine-id). And there are several journal files, of the kind:

system@1df50cd49c7f4a089c9414561f65aac7-000000000006f091-000507235df68768.journal

I think it would be really difficult to mix up that with /var/log/messages.

I think it's just that some part of /var/log/messages got corrupted
(happens a lot of times), and therefore /usr/bin/less identifies it as a
binary files since it contains non-printable characters.

Regards.
--
Canek Peláez Valdés
Profesor de asignatura, Facultad de Ciencias
Universidad Nacional Autónoma de México

[-- Attachment #2: Type: text/html, Size: 2021 bytes --]

Re: [gentoo-user] syslog-ng: how to read the log files

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 2497 bytes --]

On Tuesday 17 Feb 2015 19:17:20 lee wrote:
> Alan Mackenzie <acm@muc.de> writes:
> > Hello, Lee.
> > 
> > On Tue, Feb 17, 2015 at 07:26:05PM +0100, lee wrote:
> >> Hi,
> >> 
> >> how do you read the log files when using syslog-ng?
> >> 
> >> The log file seem to be some sort of binary that doesn't display too
> >> well in less, and there doesn't seem to be any way to read them.
> > 
> > When I try "less /var/log/messages", less gives me what is basically a
> > hex dump of the file.  I'm assuming you see the same.
> 
> Yes, that's what I was looking at.
> 
> > less searches part of the buffer (presumably the first few KB) and if it
> > finds non-printable characters, uses an input filter first to convert to
> > the hex dump.
> 
> Is that a new feature of less?  I've never had this problem with any
> other file.  IIRC, unprintable characters, like null, used to be
> displayed like ^@, and less always did a great job in preventing the
> display from needing a reset without switching to an equivalent of
> hexl-mode.
> 
> 
> BTW, what happens when something writes to /var/log/messages?  I noticed
> today that the default shorewall.conf that ships with gentoo has that
> set as logfile for shorewall.  Shouldn't all messages going into
> /var/log/messages go to syslog-ng instead when syslog-ng is used, with
> nothing else writing to this file?

It depends on what filters have been set in the configuration file of the 
application in question or syslog-ng.

I use less -L /var/log/messages to see the content of the log files in plain 
text.  At boot up I get a load of:

Feb 16 07:54:04 
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ kernel: Initializing cgroup subsys cpuset
Feb 16 07:54:04 
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

being printed up.  Perhaps I will disable cgroups in the kernel and see what 
gives.  I don't use containers anyway.

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [gentoo-user] syslog-ng: how to read the log files

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 1489 bytes --]

Am Tue, 17 Feb 2015 13:45:38 -0600
schrieb Canek Peláez Valdés <caneko@gmail.com>:

> On Tue, Feb 17, 2015 at 2:29 PM, <covici@ccs.covici.com> wrote:
> >
> > Alexander Kapshuk <alexander.kapshuk@gmail.com> wrote:
> >
> > > On Tue, Feb 17, 2015 at 8:26 PM, lee <lee@yagibdah.de> wrote:
> > >
> > > > Hi,
> > > >
> > > > how do you read the log files when using syslog-ng?
> > > >
> > > > The log file seem to be some sort of binary that doesn't display too
> > > > well in less, and there doesn't seem to be any way to read them.
> > > >
> > > >
> > > > --
> > > > Again we must be afraid of speaking of daemons for fear that daemons
> > > > might swallow us.  Finally, this fear has become reasonable.
> > > >
> > > >
> > > If you're talking about /var/log/messages, which is:
> > > messages: data
> > >
> > > I use cat(1).
> >
> > I wonder if the OP is using systemd and trying to read the journal
> > files?
> 
> Those live under /var/lib/journal (which you need to create; Gentoo doesn't
> do it by default last time I saw)
[...]

It did on my laptop after I migrated it to systemd over the weekend (on a whim,
no less -- apparently I'm adventurous?). Or, to be more precise, I didn't have
to create the directory myself. And wouldn't it be created at run-time, anyway?
That's what I would expect, at least.

[...]

-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-user] syslog-ng: how to read the log files

[Peter Humphrey]

On Tuesday 17 February 2015 20:41:06 Matti Nykyri wrote:
> > On Feb 17, 2015, at 20:26, lee <lee@yagibdah.de> wrote:
> > how do you read the log files when using syslog-ng?
> > The log file seem to be some sort of binary that doesn't display too
> > well in less, and there doesn't seem to be any way to read them.
> 
> This was discussed earlier on this list... Actually what syslog-ng
> produces is plain text. There seemed to be a bug that creates some
> binary (i.e. unreadable characters) and that causes less to consider
> files to be binary and show them incorrectly.

Yes, that was me. I found that something had marked /var/log/messages as 
a binary file. There's nothing in it that can't be read, no mysterious 
characters or anything; it's just marked as binary. All you have to do 
is to move it, then cat it back into place. I did that with no ill 
effects.

-- 
Rgds
Peter.

Re: [gentoo-user] syslog-ng: how to read the log files

[Peter Humphrey]

On Tuesday 17 February 2015 22:51:55 Peter Humphrey wrote:
> On Tuesday 17 February 2015 20:41:06 Matti Nykyri wrote:
> > > On Feb 17, 2015, at 20:26, lee <lee@yagibdah.de> wrote:
> > > how do you read the log files when using syslog-ng?
> > > The log file seem to be some sort of binary that doesn't display
> > > too
> > > well in less, and there doesn't seem to be any way to read them.
> > 
> > This was discussed earlier on this list... Actually what syslog-ng
> > produces is plain text. There seemed to be a bug that creates some
> > binary (i.e. unreadable characters) and that causes less to consider
> > files to be binary and show them incorrectly.
> 
> Yes, that was me. I found that something had marked /var/log/messages
> as a binary file. There's nothing in it that can't be read, no
> mysterious characters or anything; it's just marked as binary. All
> you have to do is to move it, then cat it back into place. I did that
> with no ill effects.

Actually, this is what I did, as I reported here on 26/12:
 
> 1.    Boot rescue system and mount main system
> 2.    # cd /mnt/main/var/log
> 3.    # mv messages messages.bin
> 4.    # strings messages.bin > messages
> 5.    # rm messages.bin
> 6.    Reboot.

-- 
Rgds
Peter.

Re: [gentoo-user] syslog-ng: how to read the log files

[Jan Sever]

On 02/18/2015 12:13 AM, Peter Humphrey wrote:
> On Tuesday 17 February 2015 22:51:55 Peter Humphrey wrote:
>> On Tuesday 17 February 2015 20:41:06 Matti Nykyri wrote:
>>>> On Feb 17, 2015, at 20:26, lee <lee@yagibdah.de> wrote:
>>>> how do you read the log files when using syslog-ng?
>>>> The log file seem to be some sort of binary that doesn't display
>>>> too
>>>> well in less, and there doesn't seem to be any way to read them.
>>>
>>> This was discussed earlier on this list... Actually what syslog-ng
>>> produces is plain text. There seemed to be a bug that creates some
>>> binary (i.e. unreadable characters) and that causes less to consider
>>> files to be binary and show them incorrectly.
>>
>> Yes, that was me. I found that something had marked /var/log/messages
>> as a binary file. There's nothing in it that can't be read, no
>> mysterious characters or anything; it's just marked as binary. All
>> you have to do is to move it, then cat it back into place. I did that
>> with no ill effects.
> 
> Actually, this is what I did, as I reported here on 26/12:
>  
>> 1.    Boot rescue system and mount main system
>> 2.    # cd /mnt/main/var/log
>> 3.    # mv messages messages.bin
>> 4.    # strings messages.bin > messages
>> 5.    # rm messages.bin
>> 6.    Reboot.
> 

When I had similar problem, I changed threaded(yes) to threaded(no)
in syslog-ng.conf and the problem disappeared. Maybe it helps you too.

-- 
Jan Sever

Re: [gentoo-user] syslog-ng: how to read the log files

[Rich Freeman]

On Tue, Feb 17, 2015 at 5:51 PM, Peter Humphrey <peter@prh.myzen.co.uk> wrote:
>
> Yes, that was me. I found that something had marked /var/log/messages as
> a binary file. There's nothing in it that can't be read, no mysterious
> characters or anything; it's just marked as binary. All you have to do
> is to move it, then cat it back into place. I did that with no ill
> effects.
>

Uh, of what binary "mark" are you speaking?  Seems likely that however
you processed the file stripped it of whatever was causing less to
consider it as binary.  I don't think cat alone would do anything to
the file, but I'm not certain of that.

-- 
Rich

Re: [gentoo-user] syslog-ng: how to read the log files

[Peter Humphrey]

On Wednesday 18 February 2015 00:23:19 Jan Sever wrote:
> On 02/18/2015 12:13 AM, Peter Humphrey wrote:
> > On Tuesday 17 February 2015 22:51:55 Peter Humphrey wrote:
> >> On Tuesday 17 February 2015 20:41:06 Matti Nykyri wrote:
> >>>> On Feb 17, 2015, at 20:26, lee <lee@yagibdah.de> wrote:
> >>>> how do you read the log files when using syslog-ng?
> >>>> The log file seem to be some sort of binary that doesn't display
> >>>> too
> >>>> well in less, and there doesn't seem to be any way to read them.
> >>> 
> >>> This was discussed earlier on this list... Actually what syslog-ng
> >>> produces is plain text. There seemed to be a bug that creates some
> >>> binary (i.e. unreadable characters) and that causes less to
> >>> consider
> >>> files to be binary and show them incorrectly.
> >> 
> >> Yes, that was me. I found that something had marked
> >> /var/log/messages
> >> as a binary file. There's nothing in it that can't be read, no
> >> mysterious characters or anything; it's just marked as binary. All
> >> you have to do is to move it, then cat it back into place. I did
> >> that
> >> with no ill effects.
> > 
> > Actually, this is what I did, as I reported here on 26/12:
> >> 1.    Boot rescue system and mount main system
> >> 2.    # cd /mnt/main/var/log
> >> 3.    # mv messages messages.bin
> >> 4.    # strings messages.bin > messages
> >> 5.    # rm messages.bin
> >> 6.    Reboot.
> 
> When I had similar problem, I changed threaded(yes) to threaded(no)
> in syslog-ng.conf and the problem disappeared. Maybe it helps you too.

Good idea. Sounds like a bug report is needed, unless it's already been 
superseded.

-- 
Rgds
Peter.

Re: [gentoo-user] syslog-ng: how to read the log files

[Peter Humphrey]

On Tuesday 17 February 2015 18:52:07 Rich Freeman wrote:
> On Tue, Feb 17, 2015 at 5:51 PM, Peter Humphrey 
<peter@prh.myzen.co.uk> wrote:
> > Yes, that was me. I found that something had marked
> > /var/log/messages as a binary file. There's nothing in it that
> > can't be read, no mysterious characters or anything; it's just
> > marked as binary. All you have to do is to move it, then cat it
> > back into place. I did that with no ill effects.
> 
> Uh, of what binary "mark" are you speaking?  Seems likely that however
> you processed the file stripped it of whatever was causing less to
> consider it as binary.  I don't think cat alone would do anything to
> the file, but I'm not certain of that.

I don't know. Are we talking magic here?

-- 
Rgds
Peter.

Re: [gentoo-user] syslog-ng: how to read the log files

[Mick]

[-- Attachment #1: Type: Text/Plain, Size: 1416 bytes --]

On Tuesday 17 Feb 2015 23:13:08 Peter Humphrey wrote:
> On Tuesday 17 February 2015 22:51:55 Peter Humphrey wrote:
> > On Tuesday 17 February 2015 20:41:06 Matti Nykyri wrote:
> > > > On Feb 17, 2015, at 20:26, lee <lee@yagibdah.de> wrote:
> > > > how do you read the log files when using syslog-ng?
> > > > The log file seem to be some sort of binary that doesn't display
> > > > too
> > > > well in less, and there doesn't seem to be any way to read them.
> > > 
> > > This was discussed earlier on this list... Actually what syslog-ng
> > > produces is plain text. There seemed to be a bug that creates some
> > > binary (i.e. unreadable characters) and that causes less to consider
> > > files to be binary and show them incorrectly.
> > 
> > Yes, that was me. I found that something had marked /var/log/messages
> > as a binary file. There's nothing in it that can't be read, no
> > mysterious characters or anything; it's just marked as binary. All
> > you have to do is to move it, then cat it back into place. I did that
> > with no ill effects.
> 
> Actually, this is what I did, as I reported here on 26/12:
> > 1.    Boot rescue system and mount main system
> > 2.    # cd /mnt/main/var/log
> > 3.    # mv messages messages.bin
> > 4.    # strings messages.bin > messages
> > 5.    # rm messages.bin
> > 6.    Reboot.

How often do you have to do this?

-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [gentoo-user] syslog-ng: how to read the log files

[Peter Humphrey]

On Wednesday 18 February 2015 07:38:46 Mick wrote:
> On Tuesday 17 Feb 2015 23:13:08 Peter Humphrey wrote:
> > Actually, this is what I did, as I reported here on 26/12:
> > > 1.    Boot rescue system and mount main system
> > > 2.    # cd /mnt/main/var/log
> > > 3.    # mv messages messages.bin
> > > 4.    # strings messages.bin > messages
> > > 5.    # rm messages.bin
> > > 6.    Reboot.
> 
> How often do you have to do this?

Just the once. Whatever bug caused it seems to have been fixed.

-- 
Rgds
Peter.

Re: [gentoo-user] syslog-ng: how to read the log files

[Stroller]

On Tue, 17 February 2015, at 6:26 pm, lee <lee@yagibdah.de> wrote:
> 
> The log file seem to be some sort of binary that doesn't display too
> well in less, and there doesn't seem to be any way to read them.

I believe this may be bug 406623.

https://bugs.gentoo.org/show_bug.cgi?id=406623

Note comment #2 - the "binary zero characters" are not visible with every editor, but if I recollect I was able to see them and delete them when I opened the log files in vim. I think they displayed as "@^" in vim.

Once you know where to look, you can also identify the binary characters using `hexdump -C /var/log/messages`.

I am now running app-admin/syslog-ng-3.4.8 and have threading enabled and the problem is now no longer occurring. 

You can establish whether you're affected by 406623 simply by deleting the character(s) (renaming the log file would probably work, too) and rebooting the system. My experience was that the text logfile is "turned to binary" on reboot - the binary characters were logged as part of the kernel's startup messages. This was repeatable and predictable.

Stroller.

Re: [gentoo-user] syslog-ng: how to read the log files

[gottlieb]

On Tue, Feb 17 2015, Canek Peláez Valdés wrote:

> On Tue, Feb 17, 2015 at 2:29 PM, <covici@ccs.covici.com> wrote:
>>
>> I wonder if the OP is using systemd and trying to read the journal
>> files?
>
> Those live under /var/lib/journal (which you need to create; Gentoo doesn't
> do it by default last time I saw),

Wow!  I just checked and indeed I do not have /var/lib/journal.
I run systemd (thanks to canek) and use journalctl, which I *thought*
was displaying the journal).

Need I make some changes?

thanks
allan
        

Re: [gentoo-user] syslog-ng: how to read the log files

[Canek Peláez Valdés]

[-- Attachment #1: Type: text/plain, Size: 1679 bytes --]

On Wed, Feb 18, 2015 at 12:22 PM, <gottlieb@nyu.edu> wrote:
>
> On Tue, Feb 17 2015, Canek Peláez Valdés wrote:
>
> > On Tue, Feb 17, 2015 at 2:29 PM, <covici@ccs.covici.com> wrote:
> >>
> >> I wonder if the OP is using systemd and trying to read the journal
> >> files?
> >
> > Those live under /var/lib/journal (which you need to create; Gentoo
doesn't
> > do it by default last time I saw),
>
> Wow!  I just checked and indeed I do not have /var/lib/journal.
> I run systemd (thanks to canek) and use journalctl, which I *thought*
> was displaying the journal).

The journal works without permanent storage (one more of its many
advantages); in that case, it keeps a small amount of logs in memory (you
can set how much memory to reserve for it).

> Need I make some changes?

Only if you want to have logs in permanent storage. In that case, you only
need to create the /var/log/journal dir with systemd-journal GID, and 2755
permissions (with setgid). systemd-journald will automatically rotate the
logs when they use 10% of the free disk available (you can also change
that).

Since the logs are compressed and indexed, each entry on them is accesible
in O(1), and they don't use that much space (with 280 megabytes reserved in
my laptop for journal logs, I have logs since Sep 20, 2014; that's 5 months
worth of logs, although my laptop doesn't run that many daemons).

Anyway, the journal works perfectly without permanent storage (as you can
see); if you are happy that way, you don't need to enable it.

Regards.
--
Canek Peláez Valdés
Profesor de asignatura, Facultad de Ciencias
Universidad Nacional Autónoma de México

[-- Attachment #2: Type: text/html, Size: 1974 bytes --]

systemd journal location (was: Re: [gentoo-user] syslog-ng: how to read the log files)

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 2442 bytes --]

Am Tue, 17 Feb 2015 23:31:26 +0100
schrieb Marc Joliet <marcec@gmx.de>:

> Am Tue, 17 Feb 2015 13:45:38 -0600
> schrieb Canek Peláez Valdés <caneko@gmail.com>:
> 
> > On Tue, Feb 17, 2015 at 2:29 PM, <covici@ccs.covici.com> wrote:
> > >
> > > Alexander Kapshuk <alexander.kapshuk@gmail.com> wrote:
> > >
> > > > On Tue, Feb 17, 2015 at 8:26 PM, lee <lee@yagibdah.de> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > how do you read the log files when using syslog-ng?
> > > > >
> > > > > The log file seem to be some sort of binary that doesn't display too
> > > > > well in less, and there doesn't seem to be any way to read them.
> > > > >
> > > > >
> > > > > --
> > > > > Again we must be afraid of speaking of daemons for fear that daemons
> > > > > might swallow us.  Finally, this fear has become reasonable.
> > > > >
> > > > >
> > > > If you're talking about /var/log/messages, which is:
> > > > messages: data
> > > >
> > > > I use cat(1).
> > >
> > > I wonder if the OP is using systemd and trying to read the journal
> > > files?
> > 
> > Those live under /var/lib/journal (which you need to create; Gentoo doesn't
> > do it by default last time I saw)
> [...]
> 
> It did on my laptop after I migrated it to systemd over the weekend (on a whim,
> no less -- apparently I'm adventurous?). Or, to be more precise, I didn't have
> to create the directory myself. And wouldn't it be created at run-time, anyway?
> That's what I would expect, at least.

Dammit, I *wanted* to mention that I didn't have my laptop there to look, and
now I regret not doing it, because I was *actually* thinking
of /var/log/journal/ (which I still didn't create by hand, BTW).

I mean, it still contains journal files, and systemd-journald(8) says its the
default *persistent* journal location.  However, it is structured differently
than what you showed, namely:

% tree /var/log/journal/
/var/log/journal/
├── b3a495d35e890b80816684a4521fc1cc
│   ├── system.journal
│   └── user-1000.journal
└── remote

So it creates a directory named after the machine ID, which contains a system
journal and one journal per user.  And if it receives logs from remote
machines, those go into the remote folder.

Just, uh, just so you know...

-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-user] syslog-ng: how to read the log files

[lee]

Stroller <stroller@stellar.eclipse.co.uk> writes:

> On Tue, 17 February 2015, at 6:26 pm, lee <lee@yagibdah.de> wrote:
>> 
>> The log file seem to be some sort of binary that doesn't display too
>> well in less, and there doesn't seem to be any way to read them.
>
> I believe this may be bug 406623.
>
> https://bugs.gentoo.org/show_bug.cgi?id=406623

That's almost three years old and should apparently be fixed?

> You can establish whether you're affected by 406623 simply by deleting
> the character(s) (renaming the log file would probably work, too) and
> rebooting the system. My experience was that the text logfile is
> "turned to binary" on reboot - the binary characters were logged as
> part of the kernel's startup messages. This was repeatable and
> predictable.

Maybe I'll try it tomorrow --- it's on a server at work which I plan to
reboot anyway.


-- 
Again we must be afraid of speaking of daemons for fear that daemons
might swallow us.  Finally, this fear has become reasonable.

Re: [gentoo-user] syslog-ng: how to read the log files

[lee]

covici@ccs.covici.com writes:

> Alexander Kapshuk <alexander.kapshuk@gmail.com> wrote:
>
>> On Tue, Feb 17, 2015 at 8:26 PM, lee <lee@yagibdah.de> wrote:
>> 
>> > Hi,
>> >
>> > how do you read the log files when using syslog-ng?
>> >
>> > The log file seem to be some sort of binary that doesn't display too
>> > well in less, and there doesn't seem to be any way to read them.
>> >
>> >
>> > --
>> > Again we must be afraid of speaking of daemons for fear that daemons
>> > might swallow us.  Finally, this fear has become reasonable.
>> >
>> >
>> If you're talking about /var/log/messages, which is:
>> messages: data
>> 
>> I use cat(1).
>
> I wonder if the OP is using systemd and trying to read the journal
> files?

Nooo, I hate systemd ...

What good are log files you can't read?  If syslog-ng would actually use
some binary format, I'd have switched to something else.


-- 
Again we must be afraid of speaking of daemons for fear that daemons
might swallow us.  Finally, this fear has become reasonable.

Re: [gentoo-user] syslog-ng: how to read the log files

[Stroller]

On Wed, 18 February 2015, at 8:40 pm, lee <lee@yagibdah.de> wrote:
>>> 
>>> The log file seem to be some sort of binary that doesn't display too
>>> well in less, and there doesn't seem to be any way to read them.
>> 
>> I believe this may be bug 406623.
>> 
>> https://bugs.gentoo.org/show_bug.cgi?id=406623
> 
> That's almost three years old and should apparently be fixed?

It's only been closed in the last few weeks. 

See for example, comment 36, November last year (i.e. 3 or 4 months old), "This isn't resolved unless commit f4ae768 is backported or >=3.5.6 is stabilised."

https://bugs.gentoo.org/show_bug.cgi?id=406623#c36

Since you haven't told us what version of syslog-ng you're running, I think it's reasonable to suspect you've not updated it recently.

Of course the characters could be left in your logfile from months ago, if you've not been rotating logs.

If it's not that bug, though, you should prolly file a new one.

Stroller.

Re: [gentoo-user] syslog-ng: how to read the log files

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 498 bytes --]

On Wed, 18 Feb 2015 21:49:54 +0100, lee wrote:

> > I wonder if the OP is using systemd and trying to read the journal
> > files?  
> 
> Nooo, I hate systemd ...
> 
> What good are log files you can't read?

You can't read syslog-ng log files without some reading software, usually
a combination of cat, grep and less. systemd does it all with journalctl.

There are good reasons to not use systemd, this isn't one of them.


-- 
Neil Bothwick

Weird enough for government work.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-user] syslog-ng: how to read the log files

[gottlieb]

On Wed, Feb 18 2015, Canek Peláez Valdés wrote:

> On Wed, Feb 18, 2015 at 12:22 PM, <gottlieb@nyu.edu> wrote:
>>
>> On Tue, Feb 17 2015, Canek Peláez Valdés wrote:
>>
>> > On Tue, Feb 17, 2015 at 2:29 PM, <covici@ccs.covici.com> wrote:
>> >>
>> >> I wonder if the OP is using systemd and trying to read the journal
>> >> files?
>> >
>> > Those live under /var/lib/journal (which you need to create; Gentoo
>> > doesn't do it by default last time I saw),
>>
>> Wow!  I just checked and indeed I do not have /var/lib/journal.
>> I run systemd (thanks to canek) and use journalctl, which I *thought*
>> was displaying the journal).
>
> The journal works without permanent storage (one more of its many
> advantages); in that case, it keeps a small amount of logs in memory (you
> can set how much memory to reserve for it).
>
>> Need I make some changes?
>
> Only if you want to have logs in permanent storage. In that case, you only
> need to create the /var/log/journal dir with systemd-journal GID, and 2755
> permissions (with setgid). systemd-journald will automatically rotate the
> logs when they use 10% of the free disk available (you can also change
> that).
>
> Since the logs are compressed and indexed, each entry on them is accesible
> in O(1), and they don't use that much space (with 280 megabytes reserved in
> my laptop for journal logs, I have logs since Sep 20, 2014; that's 5 months
> worth of logs, although my laptop doesn't run that many daemons).
>
> Anyway, the journal works perfectly without permanent storage (as you can
> see); if you are happy that way, you don't need to enable it.
>
> Regards.
> --
> Canek Peláez Valdés

Thank you for another lucid explanation.
allan

Re: [gentoo-user] syslog-ng: how to read the log files

[Fernando Rodriguez]

[-- Attachment #1: Type: text/plain, Size: 428 bytes --]

On Tuesday, February 17, 2015 7:26:05 PM lee wrote:
> Hi,
> 
> how do you read the log files when using syslog-ng?
> 
> The log file seem to be some sort of binary that doesn't display too
> well in less, and there doesn't seem to be any way to read them.
> 
> 
> 

You can just pipe the output of strings /var/log/messages to less. You can use 
strings(1) for systemd  journal files also.

-- 
Fernando Rodriguez

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-user] syslog-ng: how to read the log files

[Jan Sever]

[-- Attachment #1: Type: text/plain, Size: 605 bytes --]

On 02/19/2015 08:02 PM, Fernando Rodriguez wrote:
> On Tuesday, February 17, 2015 7:26:05 PM lee wrote:
>> Hi,
>>
>> how do you read the log files when using syslog-ng?
>>
>> The log file seem to be some sort of binary that doesn't display too
>> well in less, and there doesn't seem to be any way to read them.
>>
>>
>>
> 
> You can just pipe the output of strings /var/log/messages to less. You can
use 
> strings(1) for systemd  journal files also.

Yeah and you can check whether it contains any binary data by
diff <(strings /var/log/messages) /var/log/messages

-- 
Jan Sever
=

[-- Attachment #2: Type: text/html, Size: 751 bytes --]

Re: [gentoo-user] syslog-ng: how to read the log files

[lee]

Neil Bothwick <neil@digimed.co.uk> writes:

> On Wed, 18 Feb 2015 21:49:54 +0100, lee wrote:
>
>> > I wonder if the OP is using systemd and trying to read the journal
>> > files?  
>> 
>> Nooo, I hate systemd ...
>> 
>> What good are log files you can't read?
>
> You can't read syslog-ng log files without some reading software, usually
> a combination of cat, grep and less. systemd does it all with journalctl.
>
> There are good reasons to not use systemd, this isn't one of them.

To me it is one of the good reasons, and an important one.  Plain text
can usually always be read without further ado, be it from rescue
systems you booted or with software available on different operating
systems.  It can be also be processed with scripts and sent as email.
You can probably even read it on your cell phone.  You can still read
log files that were created 20 years ago when they are plain text.

Can you do all that with the binary files created by systemd?  I can't
even read them on a working system.


-- 
Again we must be afraid of speaking of daemons for fear that daemons
might swallow us.  Finally, this fear has become reasonable.

Re: [gentoo-user] syslog-ng: how to read the log files

[lee]

Stroller <stroller@stellar.eclipse.co.uk> writes:

> On Wed, 18 February 2015, at 8:40 pm, lee <lee@yagibdah.de> wrote:
>>>> 
>>>> The log file seem to be some sort of binary that doesn't display too
>>>> well in less, and there doesn't seem to be any way to read them.
>>> 
>>> I believe this may be bug 406623.
>>> 
>>> https://bugs.gentoo.org/show_bug.cgi?id=406623
>> 
>> That's almost three years old and should apparently be fixed?
>
> It's only been closed in the last few weeks. 

Still I wonder why it took so long to fix it.

> See for example, comment 36, November last year (i.e. 3 or 4 months old), "This isn't resolved unless commit f4ae768 is backported or >=3.5.6 is stabilised."
>
> https://bugs.gentoo.org/show_bug.cgi?id=406623#c36
>
> Since you haven't told us what version of syslog-ng you're running, I think it's reasonable to suspect you've not updated it recently.

The server was installed the week before the last, starting with the
latest live DVD.  It has been updated.  I can't tell what version it is
because it's at work.

> Of course the characters could be left in your logfile from months ago, if you've not been rotating logs.
>
> If it's not that bug, though, you should prolly file a new one.

Dunno, I've edited the file and removed the null characters.  Time will
tell whether new ones will be logged or not.


IIUC, syslog-ng handles rotating the logs.  Do I need to do something to
make it rotate them?


-- 
Again we must be afraid of speaking of daemons for fear that daemons
might swallow us.  Finally, this fear has become reasonable.

Re: [gentoo-user] syslog-ng: how to read the log files

[lee]

"Jan Sever" <n32@email.cz> writes:

> On 02/19/2015 08:02 PM, Fernando Rodriguez wrote:
>> On Tuesday, February 17, 2015 7:26:05 PM lee wrote:
>>> Hi,
>>>
>>> how do you read the log files when using syslog-ng?
>>>
>>> The log file seem to be some sort of binary that doesn't display too
>>> well in less, and there doesn't seem to be any way to read them.
>>>
>>>
>>>
>> 
>> You can just pipe the output of strings /var/log/messages to less. You can
> use 
>> strings(1) for systemd  journal files also.
>
> Yeah and you can check whether it contains any binary data by
> diff <(strings /var/log/messages) /var/log/messages

Well, yes; see it this way:

I never used syslog-ng before.  When I had to fix some problems, I
wanted to read the log files.  Less showed them in some hexl-mode, and
it has never done anything like that before.  I didn't even know that it
has this hexl-mode.

So I googled for how to read those log files because I figured that
syslog-ng perhaps uses some stupid binary format and that there might be
some program you're supposed to read them with.  That didn't turn up
anything and it really sucked.

How am I supposed to know that there's a combination of a three-year-old
bug and totally unexpected behaviour of less preventing me from reading
these logs?


Try something like 'less /boot/vmlinuz-3.17.8-gentoo-r1' and you don't
get hexl-mode.  Why such a surprise when trying to read a log file?


-- 
Again we must be afraid of speaking of daemons for fear that daemons
might swallow us.  Finally, this fear has become reasonable.

Re: [gentoo-user] syslog-ng: how to read the log files

[Canek Peláez Valdés]

[-- Attachment #1: Type: text/plain, Size: 1429 bytes --]

On Sun, Feb 22, 2015 at 6:41 PM, lee <lee@yagibdah.de> wrote:
>
> Neil Bothwick <neil@digimed.co.uk> writes:
>
> > On Wed, 18 Feb 2015 21:49:54 +0100, lee wrote:
> >
> >> > I wonder if the OP is using systemd and trying to read the journal
> >> > files?
> >>
> >> Nooo, I hate systemd ...
> >>
> >> What good are log files you can't read?
> >
> > You can't read syslog-ng log files without some reading software,
usually
> > a combination of cat, grep and less. systemd does it all with
journalctl.
> >
> > There are good reasons to not use systemd, this isn't one of them.
>
> To me it is one of the good reasons, and an important one.  Plain text
> can usually always be read without further ado, be it from rescue
> systems you booted or with software available on different operating
> systems.  It can be also be processed with scripts and sent as email.
> You can probably even read it on your cell phone.  You can still read
> log files that were created 20 years ago when they are plain text.
>
> Can you do all that with the binary files created by systemd?

Yes, you can.

> I can't even read them on a working system.

If that's true (which I highly doubt, more probably you don't know how to
read them), then it's a bug and should be reported and fixed.

Regards.
--
Canek Peláez Valdés
Profesor de asignatura, Facultad de Ciencias
Universidad Nacional Autónoma de México

[-- Attachment #2: Type: text/html, Size: 1819 bytes --]

Re: [gentoo-user] syslog-ng: how to read the log files

[Canek Peláez Valdés]

[-- Attachment #1: Type: text/plain, Size: 1867 bytes --]

On Sun, Feb 22, 2015 at 6:48 PM, lee <lee@yagibdah.de> wrote:
>
> Stroller <stroller@stellar.eclipse.co.uk> writes:
>
> > On Wed, 18 February 2015, at 8:40 pm, lee <lee@yagibdah.de> wrote:
> >>>>
> >>>> The log file seem to be some sort of binary that doesn't display too
> >>>> well in less, and there doesn't seem to be any way to read them.
> >>>
> >>> I believe this may be bug 406623.
> >>>
> >>> https://bugs.gentoo.org/show_bug.cgi?id=406623
> >>
> >> That's almost three years old and should apparently be fixed?
> >
> > It's only been closed in the last few weeks.
>
> Still I wonder why it took so long to fix it.
>
> > See for example, comment 36, November last year (i.e. 3 or 4 months
old), "This isn't resolved unless commit f4ae768 is backported or >=3.5.6
is stabilised."
> >
> > https://bugs.gentoo.org/show_bug.cgi?id=406623#c36
> >
> > Since you haven't told us what version of syslog-ng you're running, I
think it's reasonable to suspect you've not updated it recently.
>
> The server was installed the week before the last, starting with the
> latest live DVD.  It has been updated.  I can't tell what version it is
> because it's at work.
>
> > Of course the characters could be left in your logfile from months ago,
if you've not been rotating logs.
> >
> > If it's not that bug, though, you should prolly file a new one.
>
> Dunno, I've edited the file and removed the null characters.  Time will
> tell whether new ones will be logged or not.
>
>
> IIUC, syslog-ng handles rotating the logs.  Do I need to do something to
> make it rotate them?

syslog-ng, as long as I remember, has never rotated its logs. You need
logrotate (or something similar) to do it for you.

Regards.
--
Canek Peláez Valdés
Profesor de asignatura, Facultad de Ciencias
Universidad Nacional Autónoma de México

[-- Attachment #2: Type: text/html, Size: 2564 bytes --]

Re: [gentoo-user] syslog-ng: how to read the log files

[Rich Freeman]

On Sun, Feb 22, 2015 at 6:41 PM, lee <lee@yagibdah.de> wrote:
>
> To me it is one of the good reasons, and an important one.  Plain text
> can usually always be read without further ado, be it from rescue
> systems you booted or with software available on different operating
> systems.  It can be also be processed with scripts and sent as email.
> You can probably even read it on your cell phone.  You can still read
> log files that were created 20 years ago when they are plain text.

Doing any of that stuff requires the use of software capable of
reading text files.  It isn't like you can just interpret the magnetic
fields on your disk with your eyes.

Sure, there are a lot more utilities that can read text files than
journal files, but you just need to arrange to have them handy.
They'll be ubiquitous before long since every distro around will end
up needing them.

>
> Can you do all that with the binary files created by systemd?  I can't
> even read them on a working system.
>

You just type journalctl to read the live system logs.  For offline
use you just type journalctl --file=filename.  Or you can just run
strings on the file I imagine if you're desperate.  If it doesn't work
on a "working system" then your system isn't working.


-- 
Rich

Re: [gentoo-user] syslog-ng: how to read the log files

[Dale]

lee wrote:
> IIUC, syslog-ng handles rotating the logs. Do I need to do something
> to make it rotate them? 

I think you need this:

app-admin/logrotate

Then I think a cron package is needed to run that, set to daily here I
think.

Hope that helps.

Dale

:-)  :-) 

Re: [gentoo-user] syslog-ng: how to read the log files

[Peter Humphrey]

On Sunday 22 February 2015 20:57:43 Dale wrote:
> I think you need this:
> 
> app-admin/logrotate
> 
> Then I think a cron package is needed to run that, set to daily here I
> think.

It comes with logrotate:

/etc/cron.daily/logrotate

-- 
Rgds
Peter.

Re: [gentoo-user] syslog-ng: how to read the log files

[Dale]

Peter Humphrey wrote:
> On Sunday 22 February 2015 20:57:43 Dale wrote:
>> I think you need this:
>>
>> app-admin/logrotate
>>
>> Then I think a cron package is needed to run that, set to daily here I
>> think.
> It comes with logrotate:
>
> /etc/cron.daily/logrotate
>

The script does but if you don't have a cron package installed, nothing
will run to rotate the logs.  Maybe my message wasn't worded correctly? 
It's been a long week.  ;-)

Dale

:-)  :-)

Re: [gentoo-user] syslog-ng: how to read the log files

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 2217 bytes --]

Am Mon, 23 Feb 2015 00:41:50 +0100
schrieb lee <lee@yagibdah.de>:

> Neil Bothwick <neil@digimed.co.uk> writes:
> 
> > On Wed, 18 Feb 2015 21:49:54 +0100, lee wrote:
> >
> >> > I wonder if the OP is using systemd and trying to read the journal
> >> > files?  
> >> 
> >> Nooo, I hate systemd ...
> >> 
> >> What good are log files you can't read?
> >
> > You can't read syslog-ng log files without some reading software, usually
> > a combination of cat, grep and less. systemd does it all with journalctl.
> >
> > There are good reasons to not use systemd, this isn't one of them.
> 
> To me it is one of the good reasons, and an important one.  Plain text
> can usually always be read without further ado, be it from rescue
> systems you booted or with software available on different operating
> systems.  It can be also be processed with scripts and sent as email.
> You can probably even read it on your cell phone.  You can still read
> log files that were created 20 years ago when they are plain text.
> 
> Can you do all that with the binary files created by systemd?  I can't
> even read them on a working system.

What Canek and Rich already said is good, but I'll just add this: it's not like
you can't run a classic syslog implementation alongside the systemd journal.
On my systems, by *default*, syslog-ng kept working as usual, getting the logs
from the systemd journal.  If you want to go further, you can even configure
the journal to not store logs permanently, so that you *only* end up with
plain-text logs on your system (Duncan on gentoo-amd64 went this way).

So no, the format that the systemd journal uses is most decidedly *not* a reason
against using systemd.

Personally, I'm probably going to uninstall syslog-ng, because journalctl is
*such* a nice way to read logs, so why run something whose output I'll never
read again?  I recommend reading
http://0pointer.net/blog/projects/journalctl.html for examples of the kind of
stuff you can do that would be cumbersome, if not *impossible* with regular
syslog.

HTH
-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-user] syslog-ng: how to read the log files

[covici]

Marc Joliet <marcec@gmx.de> wrote:

> Am Mon, 23 Feb 2015 00:41:50 +0100
> schrieb lee <lee@yagibdah.de>:
> 
> > Neil Bothwick <neil@digimed.co.uk> writes:
> > 
> > > On Wed, 18 Feb 2015 21:49:54 +0100, lee wrote:
> > >
> > >> > I wonder if the OP is using systemd and trying to read the journal
> > >> > files?  
> > >> 
> > >> Nooo, I hate systemd ...
> > >> 
> > >> What good are log files you can't read?
> > >
> > > You can't read syslog-ng log files without some reading software, usually
> > > a combination of cat, grep and less. systemd does it all with journalctl.
> > >
> > > There are good reasons to not use systemd, this isn't one of them.
> > 
> > To me it is one of the good reasons, and an important one.  Plain text
> > can usually always be read without further ado, be it from rescue
> > systems you booted or with software available on different operating
> > systems.  It can be also be processed with scripts and sent as email.
> > You can probably even read it on your cell phone.  You can still read
> > log files that were created 20 years ago when they are plain text.
> > 
> > Can you do all that with the binary files created by systemd?  I can't
> > even read them on a working system.
> 
> What Canek and Rich already said is good, but I'll just add this: it's not like
> you can't run a classic syslog implementation alongside the systemd journal.
> On my systems, by *default*, syslog-ng kept working as usual, getting the logs
> from the systemd journal.  If you want to go further, you can even configure
> the journal to not store logs permanently, so that you *only* end up with
> plain-text logs on your system (Duncan on gentoo-amd64 went this way).
> 
> So no, the format that the systemd journal uses is most decidedly *not* a reason
> against using systemd.
> 
> Personally, I'm probably going to uninstall syslog-ng, because journalctl is
> *such* a nice way to read logs, so why run something whose output I'll never
> read again?  I recommend reading
> http://0pointer.net/blog/projects/journalctl.html for examples of the kind of
> stuff you can do that would be cumbersome, if not *impossible* with regular
> syslog.

Except that I get lots of messages about the system journal missing
messages when forwarding to syslog, so how can I make sure this does not
happening?

-- 
Your life is like a penny.  You're going to lose it.  The question is:
How do
you spend it?

         John Covici
         covici@ccs.covici.com

Re: [gentoo-user] syslog-ng: how to read the log files

[Peter Humphrey]

On Sunday 22 February 2015 22:28:07 Dale wrote:
> Peter Humphrey wrote:
> > On Sunday 22 February 2015 20:57:43 Dale wrote:
> >> I think you need this:
> >> 
> >> app-admin/logrotate
> >> 
> >> Then I think a cron package is needed to run that, set to daily
> >> here I think.
> > 
> > It comes with logrotate:
> > 
> > /etc/cron.daily/logrotate
> 
> The script does but if you don't have a cron package installed,
> nothing will run to rotate the logs.  Maybe my message wasn't worded
> correctly? It's been a long week.  ;-)

Ah, I see what you mean. My misread.

-- 
Rgds
Peter.

Re: [gentoo-user] syslog-ng: how to read the log files

[Canek Peláez Valdés]

[-- Attachment #1: Type: text/plain, Size: 3298 bytes --]

On Mon, Feb 23, 2015 at 3:41 AM, <covici@ccs.covici.com> wrote:
>
> Marc Joliet <marcec@gmx.de> wrote:
>
> > Am Mon, 23 Feb 2015 00:41:50 +0100
> > schrieb lee <lee@yagibdah.de>:
> >
> > > Neil Bothwick <neil@digimed.co.uk> writes:
> > >
> > > > On Wed, 18 Feb 2015 21:49:54 +0100, lee wrote:
> > > >
> > > >> > I wonder if the OP is using systemd and trying to read the
journal
> > > >> > files?
> > > >>
> > > >> Nooo, I hate systemd ...
> > > >>
> > > >> What good are log files you can't read?
> > > >
> > > > You can't read syslog-ng log files without some reading software,
usually
> > > > a combination of cat, grep and less. systemd does it all with
journalctl.
> > > >
> > > > There are good reasons to not use systemd, this isn't one of them.
> > >
> > > To me it is one of the good reasons, and an important one.  Plain text
> > > can usually always be read without further ado, be it from rescue
> > > systems you booted or with software available on different operating
> > > systems.  It can be also be processed with scripts and sent as email.
> > > You can probably even read it on your cell phone.  You can still read
> > > log files that were created 20 years ago when they are plain text.
> > >
> > > Can you do all that with the binary files created by systemd?  I can't
> > > even read them on a working system.
> >
> > What Canek and Rich already said is good, but I'll just add this: it's
not like
> > you can't run a classic syslog implementation alongside the systemd
journal.
> > On my systems, by *default*, syslog-ng kept working as usual, getting
the logs
> > from the systemd journal.  If you want to go further, you can even
configure
> > the journal to not store logs permanently, so that you *only* end up
with
> > plain-text logs on your system (Duncan on gentoo-amd64 went this way).
> >
> > So no, the format that the systemd journal uses is most decidedly *not*
a reason
> > against using systemd.
> >
> > Personally, I'm probably going to uninstall syslog-ng, because
journalctl is
> > *such* a nice way to read logs, so why run something whose output I'll
never
> > read again?  I recommend reading
> > http://0pointer.net/blog/projects/journalctl.html for examples of the
kind of
> > stuff you can do that would be cumbersome, if not *impossible* with
regular
> > syslog.
>
> Except that I get lots of messages about the system journal missing
> messages when forwarding to syslog, so how can I make sure this does not
> happening?

Could you please show those messages? systemd sends *everything* to the
journal, and then the journal (optionally) can send it too to a regular
syslog. In that sense, it's impossible for the journal to miss any message.

The only way in which the journal could miss messages is at very early boot
stages; but with a proper initramfs (like the ones generated with dracut),
even those get caught. You get to put an instance of systemd and the
journal inside the initramfs, and so it's available almost from the
beginning.

And if you use gummiboot, then you can even log from the moment the UEFI
firmware comes to life.

Regards.
--
Canek Peláez Valdés
Profesor de asignatura, Facultad de Ciencias
Universidad Nacional Autónoma de México

[-- Attachment #2: Type: text/html, Size: 4185 bytes --]

Re: [gentoo-user] syslog-ng: how to read the log files

[covici]

Canek Peláez Valdés <caneko@gmail.com> wrote:

> On Mon, Feb 23, 2015 at 3:41 AM, <covici@ccs.covici.com> wrote:
> >
> > Marc Joliet <marcec@gmx.de> wrote:
> >
> > > Am Mon, 23 Feb 2015 00:41:50 +0100
> > > schrieb lee <lee@yagibdah.de>:
> > >
> > > > Neil Bothwick <neil@digimed.co.uk> writes:
> > > >
> > > > > On Wed, 18 Feb 2015 21:49:54 +0100, lee wrote:
> > > > >
> > > > >> > I wonder if the OP is using systemd and trying to read the
> journal
> > > > >> > files?
> > > > >>
> > > > >> Nooo, I hate systemd ...
> > > > >>
> > > > >> What good are log files you can't read?
> > > > >
> > > > > You can't read syslog-ng log files without some reading software,
> usually
> > > > > a combination of cat, grep and less. systemd does it all with
> journalctl.
> > > > >
> > > > > There are good reasons to not use systemd, this isn't one of them.
> > > >
> > > > To me it is one of the good reasons, and an important one.  Plain text
> > > > can usually always be read without further ado, be it from rescue
> > > > systems you booted or with software available on different operating
> > > > systems.  It can be also be processed with scripts and sent as email.
> > > > You can probably even read it on your cell phone.  You can still read
> > > > log files that were created 20 years ago when they are plain text.
> > > >
> > > > Can you do all that with the binary files created by systemd?  I can't
> > > > even read them on a working system.
> > >
> > > What Canek and Rich already said is good, but I'll just add this: it's
> not like
> > > you can't run a classic syslog implementation alongside the systemd
> journal.
> > > On my systems, by *default*, syslog-ng kept working as usual, getting
> the logs
> > > from the systemd journal.  If you want to go further, you can even
> configure
> > > the journal to not store logs permanently, so that you *only* end up
> with
> > > plain-text logs on your system (Duncan on gentoo-amd64 went this way).
> > >
> > > So no, the format that the systemd journal uses is most decidedly *not*
> a reason
> > > against using systemd.
> > >
> > > Personally, I'm probably going to uninstall syslog-ng, because
> journalctl is
> > > *such* a nice way to read logs, so why run something whose output I'll
> never
> > > read again?  I recommend reading
> > > http://0pointer.net/blog/projects/journalctl.html for examples of the
> kind of
> > > stuff you can do that would be cumbersome, if not *impossible* with
> regular
> > > syslog.
> >
> > Except that I get lots of messages about the system journal missing
> > messages when forwarding to syslog, so how can I make sure this does not
> > happening?
> 
> Could you please show those messages? systemd sends *everything* to the
> journal, and then the journal (optionally) can send it too to a regular
> syslog. In that sense, it's impossible for the journal to miss any message.
> 
> The only way in which the journal could miss messages is at very early boot
> stages; but with a proper initramfs (like the ones generated with dracut),
> even those get caught. You get to put an instance of systemd and the
> journal inside the initramfs, and so it's available almost from the
> beginning.
> 
> And if you use gummiboot, then you can even log from the moment the UEFI
> firmware comes to life.

So, I get lots of messages in my regular syslog-ng /var/log/messages
like the following:
Feb 23 12:47:52 ccs.covici.com systemd-journal[715]: Forwarding to
syslog missed 15 messages.

So, I saw a post on Google to up the queue length, and I uped it to 200,
but no joy, still get the messages like the one above.


-- 
Your life is like a penny.  You're going to lose it.  The question is:
How do
you spend it?

         John Covici
         covici@ccs.covici.com

Re: [gentoo-user] syslog-ng: how to read the log files

[Canek Peláez Valdés]

[-- Attachment #1: Type: text/plain, Size: 4462 bytes --]

On Mon, Feb 23, 2015 at 11:49 AM, <covici@ccs.covici.com> wrote:
>
> Canek Peláez Valdés <caneko@gmail.com> wrote:
>
> > On Mon, Feb 23, 2015 at 3:41 AM, <covici@ccs.covici.com> wrote:
> > >
> > > Marc Joliet <marcec@gmx.de> wrote:
> > >
> > > > Am Mon, 23 Feb 2015 00:41:50 +0100
> > > > schrieb lee <lee@yagibdah.de>:
> > > >
> > > > > Neil Bothwick <neil@digimed.co.uk> writes:
> > > > >
> > > > > > On Wed, 18 Feb 2015 21:49:54 +0100, lee wrote:
> > > > > >
> > > > > >> > I wonder if the OP is using systemd and trying to read the
> > journal
> > > > > >> > files?
> > > > > >>
> > > > > >> Nooo, I hate systemd ...
> > > > > >>
> > > > > >> What good are log files you can't read?
> > > > > >
> > > > > > You can't read syslog-ng log files without some reading
software,
> > usually
> > > > > > a combination of cat, grep and less. systemd does it all with
> > journalctl.
> > > > > >
> > > > > > There are good reasons to not use systemd, this isn't one of
them.
> > > > >
> > > > > To me it is one of the good reasons, and an important one.  Plain
text
> > > > > can usually always be read without further ado, be it from rescue
> > > > > systems you booted or with software available on different
operating
> > > > > systems.  It can be also be processed with scripts and sent as
email.
> > > > > You can probably even read it on your cell phone.  You can still
read
> > > > > log files that were created 20 years ago when they are plain text.
> > > > >
> > > > > Can you do all that with the binary files created by systemd?  I
can't
> > > > > even read them on a working system.
> > > >
> > > > What Canek and Rich already said is good, but I'll just add this:
it's
> > not like
> > > > you can't run a classic syslog implementation alongside the systemd
> > journal.
> > > > On my systems, by *default*, syslog-ng kept working as usual,
getting
> > the logs
> > > > from the systemd journal.  If you want to go further, you can even
> > configure
> > > > the journal to not store logs permanently, so that you *only* end up
> > with
> > > > plain-text logs on your system (Duncan on gentoo-amd64 went this
way).
> > > >
> > > > So no, the format that the systemd journal uses is most decidedly
*not*
> > a reason
> > > > against using systemd.
> > > >
> > > > Personally, I'm probably going to uninstall syslog-ng, because
> > journalctl is
> > > > *such* a nice way to read logs, so why run something whose output
I'll
> > never
> > > > read again?  I recommend reading
> > > > http://0pointer.net/blog/projects/journalctl.html for examples of
the
> > kind of
> > > > stuff you can do that would be cumbersome, if not *impossible* with
> > regular
> > > > syslog.
> > >
> > > Except that I get lots of messages about the system journal missing
> > > messages when forwarding to syslog, so how can I make sure this does
not
> > > happening?
> >
> > Could you please show those messages? systemd sends *everything* to the
> > journal, and then the journal (optionally) can send it too to a regular
> > syslog. In that sense, it's impossible for the journal to miss any
message.
> >
> > The only way in which the journal could miss messages is at very early
boot
> > stages; but with a proper initramfs (like the ones generated with
dracut),
> > even those get caught. You get to put an instance of systemd and the
> > journal inside the initramfs, and so it's available almost from the
> > beginning.
> >
> > And if you use gummiboot, then you can even log from the moment the UEFI
> > firmware comes to life.
>
> So, I get lots of messages in my regular syslog-ng /var/log/messages
> like the following:
> Feb 23 12:47:52 ccs.covici.com systemd-journal[715]: Forwarding to
> syslog missed 15 messages.
>
> So, I saw a post on Google to up the queue length, and I uped it to 200,
> but no joy, still get the messages like the one above.

Are you using the unit file provided by syslog-ng (systemd-delta doesn't
mention syslog)? Also, is /etc/systemd/system/syslog.service is a link
to /usr/lib/systemd/system/syslog-ng.service?

I do, and I don't get any of those messages. I use the default journal
configuration. According to [1], this should be fixed.

Regards.

https://github.com/balabit/syslog-ng/issues/314
--
Canek Peláez Valdés
Profesor de asignatura, Facultad de Ciencias
Universidad Nacional Autónoma de México

[-- Attachment #2: Type: text/html, Size: 6183 bytes --]

Re: [gentoo-user] syslog-ng: how to read the log files

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 5545 bytes --]

Am Mon, 23 Feb 2015 12:10:18 -0600
schrieb Canek Peláez Valdés <caneko@gmail.com>:

> On Mon, Feb 23, 2015 at 11:49 AM, <covici@ccs.covici.com> wrote:
> >
> > Canek Peláez Valdés <caneko@gmail.com> wrote:
> >
> > > On Mon, Feb 23, 2015 at 3:41 AM, <covici@ccs.covici.com> wrote:
> > > >
> > > > Marc Joliet <marcec@gmx.de> wrote:
> > > >
> > > > > Am Mon, 23 Feb 2015 00:41:50 +0100
> > > > > schrieb lee <lee@yagibdah.de>:
> > > > >
> > > > > > Neil Bothwick <neil@digimed.co.uk> writes:
> > > > > >
> > > > > > > On Wed, 18 Feb 2015 21:49:54 +0100, lee wrote:
> > > > > > >
> > > > > > >> > I wonder if the OP is using systemd and trying to read the
> > > journal
> > > > > > >> > files?
> > > > > > >>
> > > > > > >> Nooo, I hate systemd ...
> > > > > > >>
> > > > > > >> What good are log files you can't read?
> > > > > > >
> > > > > > > You can't read syslog-ng log files without some reading
> software,
> > > usually
> > > > > > > a combination of cat, grep and less. systemd does it all with
> > > journalctl.
> > > > > > >
> > > > > > > There are good reasons to not use systemd, this isn't one of
> them.
> > > > > >
> > > > > > To me it is one of the good reasons, and an important one.  Plain
> text
> > > > > > can usually always be read without further ado, be it from rescue
> > > > > > systems you booted or with software available on different
> operating
> > > > > > systems.  It can be also be processed with scripts and sent as
> email.
> > > > > > You can probably even read it on your cell phone.  You can still
> read
> > > > > > log files that were created 20 years ago when they are plain text.
> > > > > >
> > > > > > Can you do all that with the binary files created by systemd?  I
> can't
> > > > > > even read them on a working system.
> > > > >
> > > > > What Canek and Rich already said is good, but I'll just add this:
> it's
> > > not like
> > > > > you can't run a classic syslog implementation alongside the systemd
> > > journal.
> > > > > On my systems, by *default*, syslog-ng kept working as usual,
> getting
> > > the logs
> > > > > from the systemd journal.  If you want to go further, you can even
> > > configure
> > > > > the journal to not store logs permanently, so that you *only* end up
> > > with
> > > > > plain-text logs on your system (Duncan on gentoo-amd64 went this
> way).
> > > > >
> > > > > So no, the format that the systemd journal uses is most decidedly
> *not*
> > > a reason
> > > > > against using systemd.
> > > > >
> > > > > Personally, I'm probably going to uninstall syslog-ng, because
> > > journalctl is
> > > > > *such* a nice way to read logs, so why run something whose output
> I'll
> > > never
> > > > > read again?  I recommend reading
> > > > > http://0pointer.net/blog/projects/journalctl.html for examples of
> the
> > > kind of
> > > > > stuff you can do that would be cumbersome, if not *impossible* with
> > > regular
> > > > > syslog.
> > > >
> > > > Except that I get lots of messages about the system journal missing
> > > > messages when forwarding to syslog, so how can I make sure this does
> not
> > > > happening?
> > >
> > > Could you please show those messages? systemd sends *everything* to the
> > > journal, and then the journal (optionally) can send it too to a regular
> > > syslog. In that sense, it's impossible for the journal to miss any
> message.
> > >
> > > The only way in which the journal could miss messages is at very early
> boot
> > > stages; but with a proper initramfs (like the ones generated with
> dracut),
> > > even those get caught. You get to put an instance of systemd and the
> > > journal inside the initramfs, and so it's available almost from the
> > > beginning.
> > >
> > > And if you use gummiboot, then you can even log from the moment the UEFI
> > > firmware comes to life.
> >
> > So, I get lots of messages in my regular syslog-ng /var/log/messages
> > like the following:
> > Feb 23 12:47:52 ccs.covici.com systemd-journal[715]: Forwarding to
> > syslog missed 15 messages.
> >
> > So, I saw a post on Google to up the queue length, and I uped it to 200,
> > but no joy, still get the messages like the one above.
> 
> Are you using the unit file provided by syslog-ng (systemd-delta doesn't
> mention syslog)? Also, is /etc/systemd/system/syslog.service is a link
> to /usr/lib/systemd/system/syslog-ng.service?
> 
> I do, and I don't get any of those messages. I use the default journal
> configuration. According to [1], this should be fixed.

I remember getting a small number of messages like that, too, on my laptop.
However, it's at the university, so I can't check now to see what types of
messages were missed (if any; if I understand [1] correctly, those messages are
most likely bogus?).

But yeah, that's any idea, Covici: see what's in /var/log/messages, compare that
to the journalctl output, and check if any messages were actually missed ("diff
-U" might be of help here).  And if/once you did that, what kinds of messages
were missed, if any?  If those messages really are bogus, you shouldn't see any
differences between the two.

> Regards.
> 
> https://github.com/balabit/syslog-ng/issues/314

Note that that fix would only be in the ~arch version of syslog-ng, the current
stable version (3.4.8) is a few months too old.

-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-user] syslog-ng: how to read the log files

[covici]

Canek Peláez Valdés <caneko@gmail.com> wrote:

> On Mon, Feb 23, 2015 at 11:49 AM, <covici@ccs.covici.com> wrote:
> >
> > Canek Peláez Valdés <caneko@gmail.com> wrote:
> >
> > > On Mon, Feb 23, 2015 at 3:41 AM, <covici@ccs.covici.com> wrote:
> > > >
> > > > Marc Joliet <marcec@gmx.de> wrote:
> > > >
> > > > > Am Mon, 23 Feb 2015 00:41:50 +0100
> > > > > schrieb lee <lee@yagibdah.de>:
> > > > >
> > > > > > Neil Bothwick <neil@digimed.co.uk> writes:
> > > > > >
> > > > > > > On Wed, 18 Feb 2015 21:49:54 +0100, lee wrote:
> > > > > > >
> > > > > > >> > I wonder if the OP is using systemd and trying to read the
> > > journal
> > > > > > >> > files?
> > > > > > >>
> > > > > > >> Nooo, I hate systemd ...
> > > > > > >>
> > > > > > >> What good are log files you can't read?
> > > > > > >
> > > > > > > You can't read syslog-ng log files without some reading
> software,
> > > usually
> > > > > > > a combination of cat, grep and less. systemd does it all with
> > > journalctl.
> > > > > > >
> > > > > > > There are good reasons to not use systemd, this isn't one of
> them.
> > > > > >
> > > > > > To me it is one of the good reasons, and an important one.  Plain
> text
> > > > > > can usually always be read without further ado, be it from rescue
> > > > > > systems you booted or with software available on different
> operating
> > > > > > systems.  It can be also be processed with scripts and sent as
> email.
> > > > > > You can probably even read it on your cell phone.  You can still
> read
> > > > > > log files that were created 20 years ago when they are plain text.
> > > > > >
> > > > > > Can you do all that with the binary files created by systemd?  I
> can't
> > > > > > even read them on a working system.
> > > > >
> > > > > What Canek and Rich already said is good, but I'll just add this:
> it's
> > > not like
> > > > > you can't run a classic syslog implementation alongside the systemd
> > > journal.
> > > > > On my systems, by *default*, syslog-ng kept working as usual,
> getting
> > > the logs
> > > > > from the systemd journal.  If you want to go further, you can even
> > > configure
> > > > > the journal to not store logs permanently, so that you *only* end up
> > > with
> > > > > plain-text logs on your system (Duncan on gentoo-amd64 went this
> way).
> > > > >
> > > > > So no, the format that the systemd journal uses is most decidedly
> *not*
> > > a reason
> > > > > against using systemd.
> > > > >
> > > > > Personally, I'm probably going to uninstall syslog-ng, because
> > > journalctl is
> > > > > *such* a nice way to read logs, so why run something whose output
> I'll
> > > never
> > > > > read again?  I recommend reading
> > > > > http://0pointer.net/blog/projects/journalctl.html for examples of
> the
> > > kind of
> > > > > stuff you can do that would be cumbersome, if not *impossible* with
> > > regular
> > > > > syslog.
> > > >
> > > > Except that I get lots of messages about the system journal missing
> > > > messages when forwarding to syslog, so how can I make sure this does
> not
> > > > happening?
> > >
> > > Could you please show those messages? systemd sends *everything* to the
> > > journal, and then the journal (optionally) can send it too to a regular
> > > syslog. In that sense, it's impossible for the journal to miss any
> message.
> > >
> > > The only way in which the journal could miss messages is at very early
> boot
> > > stages; but with a proper initramfs (like the ones generated with
> dracut),
> > > even those get caught. You get to put an instance of systemd and the
> > > journal inside the initramfs, and so it's available almost from the
> > > beginning.
> > >
> > > And if you use gummiboot, then you can even log from the moment the UEFI
> > > firmware comes to life.
> >
> > So, I get lots of messages in my regular syslog-ng /var/log/messages
> > like the following:
> > Feb 23 12:47:52 ccs.covici.com systemd-journal[715]: Forwarding to
> > syslog missed 15 messages.
> >
> > So, I saw a post on Google to up the queue length, and I uped it to 200,
> > but no joy, still get the messages like the one above.
> 
> Are you using the unit file provided by syslog-ng (systemd-delta doesn't
> mention syslog)? Also, is /etc/systemd/system/syslog.service is a link
> to /usr/lib/systemd/system/syslog-ng.service?
> 
> I do, and I don't get any of those messages. I use the default journal
> configuration. According to [1], this should be fixed.
> 
> Regards.
> 
> https://github.com/balabit/syslog-ng/issues/314

At the time when I did this there was no syslog-ng.service in
/usr/lib/systemd/system, now there is, but my unit file is like this:

[Unit]
Description=System Logger Daemon
Documentation=man:syslog-ng(8)

[Service]
Sockets=syslog.socket
ExecStart=/usr/sbin/syslog-ng -F
ExecReload=/bin/kill -HUP $MAINPID
#Restart=on-failure

[Install]
WantedBy=multi-user.target
Alias=syslog.service

Is there a reason why this should not work?

-- 
Your life is like a penny.  You're going to lose it.  The question is:
How do
you spend it?

         John Covici
         covici@ccs.covici.com

Re: [gentoo-user] syslog-ng: how to read the log files

[covici]

Marc Joliet <marcec@gmx.de> wrote:

> Am Mon, 23 Feb 2015 12:10:18 -0600
> schrieb Canek Peláez Valdés <caneko@gmail.com>:
> 
> > On Mon, Feb 23, 2015 at 11:49 AM, <covici@ccs.covici.com> wrote:
> > >
> > > Canek Peláez Valdés <caneko@gmail.com> wrote:
> > >
> > > > On Mon, Feb 23, 2015 at 3:41 AM, <covici@ccs.covici.com> wrote:
> > > > >
> > > > > Marc Joliet <marcec@gmx.de> wrote:
> > > > >
> > > > > > Am Mon, 23 Feb 2015 00:41:50 +0100
> > > > > > schrieb lee <lee@yagibdah.de>:
> > > > > >
> > > > > > > Neil Bothwick <neil@digimed.co.uk> writes:
> > > > > > >
> > > > > > > > On Wed, 18 Feb 2015 21:49:54 +0100, lee wrote:
> > > > > > > >
> > > > > > > >> > I wonder if the OP is using systemd and trying to read the
> > > > journal
> > > > > > > >> > files?
> > > > > > > >>
> > > > > > > >> Nooo, I hate systemd ...
> > > > > > > >>
> > > > > > > >> What good are log files you can't read?
> > > > > > > >
> > > > > > > > You can't read syslog-ng log files without some reading
> > software,
> > > > usually
> > > > > > > > a combination of cat, grep and less. systemd does it all with
> > > > journalctl.
> > > > > > > >
> > > > > > > > There are good reasons to not use systemd, this isn't one of
> > them.
> > > > > > >
> > > > > > > To me it is one of the good reasons, and an important one.  Plain
> > text
> > > > > > > can usually always be read without further ado, be it from rescue
> > > > > > > systems you booted or with software available on different
> > operating
> > > > > > > systems.  It can be also be processed with scripts and sent as
> > email.
> > > > > > > You can probably even read it on your cell phone.  You can still
> > read
> > > > > > > log files that were created 20 years ago when they are plain text.
> > > > > > >
> > > > > > > Can you do all that with the binary files created by systemd?  I
> > can't
> > > > > > > even read them on a working system.
> > > > > >
> > > > > > What Canek and Rich already said is good, but I'll just add this:
> > it's
> > > > not like
> > > > > > you can't run a classic syslog implementation alongside the systemd
> > > > journal.
> > > > > > On my systems, by *default*, syslog-ng kept working as usual,
> > getting
> > > > the logs
> > > > > > from the systemd journal.  If you want to go further, you can even
> > > > configure
> > > > > > the journal to not store logs permanently, so that you *only* end up
> > > > with
> > > > > > plain-text logs on your system (Duncan on gentoo-amd64 went this
> > way).
> > > > > >
> > > > > > So no, the format that the systemd journal uses is most decidedly
> > *not*
> > > > a reason
> > > > > > against using systemd.
> > > > > >
> > > > > > Personally, I'm probably going to uninstall syslog-ng, because
> > > > journalctl is
> > > > > > *such* a nice way to read logs, so why run something whose output
> > I'll
> > > > never
> > > > > > read again?  I recommend reading
> > > > > > http://0pointer.net/blog/projects/journalctl.html for examples of
> > the
> > > > kind of
> > > > > > stuff you can do that would be cumbersome, if not *impossible* with
> > > > regular
> > > > > > syslog.
> > > > >
> > > > > Except that I get lots of messages about the system journal missing
> > > > > messages when forwarding to syslog, so how can I make sure this does
> > not
> > > > > happening?
> > > >
> > > > Could you please show those messages? systemd sends *everything* to the
> > > > journal, and then the journal (optionally) can send it too to a regular
> > > > syslog. In that sense, it's impossible for the journal to miss any
> > message.
> > > >
> > > > The only way in which the journal could miss messages is at very early
> > boot
> > > > stages; but with a proper initramfs (like the ones generated with
> > dracut),
> > > > even those get caught. You get to put an instance of systemd and the
> > > > journal inside the initramfs, and so it's available almost from the
> > > > beginning.
> > > >
> > > > And if you use gummiboot, then you can even log from the moment the UEFI
> > > > firmware comes to life.
> > >
> > > So, I get lots of messages in my regular syslog-ng /var/log/messages
> > > like the following:
> > > Feb 23 12:47:52 ccs.covici.com systemd-journal[715]: Forwarding to
> > > syslog missed 15 messages.
> > >
> > > So, I saw a post on Google to up the queue length, and I uped it to 200,
> > > but no joy, still get the messages like the one above.
> > 
> > Are you using the unit file provided by syslog-ng (systemd-delta doesn't
> > mention syslog)? Also, is /etc/systemd/system/syslog.service is a link
> > to /usr/lib/systemd/system/syslog-ng.service?
> > 
> > I do, and I don't get any of those messages. I use the default journal
> > configuration. According to [1], this should be fixed.
> 
> I remember getting a small number of messages like that, too, on my laptop.
> However, it's at the university, so I can't check now to see what types of
> messages were missed (if any; if I understand [1] correctly, those messages are
> most likely bogus?).
> 
> But yeah, that's any idea, Covici: see what's in /var/log/messages, compare that
> to the journalctl output, and check if any messages were actually missed ("diff
> -U" might be of help here).  And if/once you did that, what kinds of messages
> were missed, if any?  If those messages really are bogus, you shouldn't see any
> differences between the two.
> 
> > Regards.
> > 
> > https://github.com/balabit/syslog-ng/issues/314
> 
> Note that that fix would only be in the ~arch version of syslog-ng, the current
> stable version (3.4.8) is a few months too old.

I am up to 3.6 something, so the fix should be there.  But my unit file
is different, so that remains to check.

-- 
Your life is like a penny.  You're going to lose it.  The question is:
How do
you spend it?

         John Covici
         covici@ccs.covici.com

Re: [gentoo-user] syslog-ng: how to read the log files

[Canek Peláez Valdés]

[-- Attachment #1: Type: text/plain, Size: 6654 bytes --]

On Mon, Feb 23, 2015 at 1:31 PM, <covici@ccs.covici.com> wrote:
>
> Marc Joliet <marcec@gmx.de> wrote:
>
> > Am Mon, 23 Feb 2015 12:10:18 -0600
> > schrieb Canek Peláez Valdés <caneko@gmail.com>:
> >
> > > On Mon, Feb 23, 2015 at 11:49 AM, <covici@ccs.covici.com> wrote:
> > > >
> > > > Canek Peláez Valdés <caneko@gmail.com> wrote:
> > > >
> > > > > On Mon, Feb 23, 2015 at 3:41 AM, <covici@ccs.covici.com> wrote:
> > > > > >
> > > > > > Marc Joliet <marcec@gmx.de> wrote:
> > > > > >
> > > > > > > Am Mon, 23 Feb 2015 00:41:50 +0100
> > > > > > > schrieb lee <lee@yagibdah.de>:
> > > > > > >
> > > > > > > > Neil Bothwick <neil@digimed.co.uk> writes:
> > > > > > > >
> > > > > > > > > On Wed, 18 Feb 2015 21:49:54 +0100, lee wrote:
> > > > > > > > >
> > > > > > > > >> > I wonder if the OP is using systemd and trying to read
the
> > > > > journal
> > > > > > > > >> > files?
> > > > > > > > >>
> > > > > > > > >> Nooo, I hate systemd ...
> > > > > > > > >>
> > > > > > > > >> What good are log files you can't read?
> > > > > > > > >
> > > > > > > > > You can't read syslog-ng log files without some reading
> > > software,
> > > > > usually
> > > > > > > > > a combination of cat, grep and less. systemd does it all
with
> > > > > journalctl.
> > > > > > > > >
> > > > > > > > > There are good reasons to not use systemd, this isn't one
of
> > > them.
> > > > > > > >
> > > > > > > > To me it is one of the good reasons, and an important one.
Plain
> > > text
> > > > > > > > can usually always be read without further ado, be it from
rescue
> > > > > > > > systems you booted or with software available on different
> > > operating
> > > > > > > > systems.  It can be also be processed with scripts and sent
as
> > > email.
> > > > > > > > You can probably even read it on your cell phone.  You can
still
> > > read
> > > > > > > > log files that were created 20 years ago when they are
plain text.
> > > > > > > >
> > > > > > > > Can you do all that with the binary files created by
systemd?  I
> > > can't
> > > > > > > > even read them on a working system.
> > > > > > >
> > > > > > > What Canek and Rich already said is good, but I'll just add
this:
> > > it's
> > > > > not like
> > > > > > > you can't run a classic syslog implementation alongside the
systemd
> > > > > journal.
> > > > > > > On my systems, by *default*, syslog-ng kept working as usual,
> > > getting
> > > > > the logs
> > > > > > > from the systemd journal.  If you want to go further, you can
even
> > > > > configure
> > > > > > > the journal to not store logs permanently, so that you *only*
end up
> > > > > with
> > > > > > > plain-text logs on your system (Duncan on gentoo-amd64 went
this
> > > way).
> > > > > > >
> > > > > > > So no, the format that the systemd journal uses is most
decidedly
> > > *not*
> > > > > a reason
> > > > > > > against using systemd.
> > > > > > >
> > > > > > > Personally, I'm probably going to uninstall syslog-ng, because
> > > > > journalctl is
> > > > > > > *such* a nice way to read logs, so why run something whose
output
> > > I'll
> > > > > never
> > > > > > > read again?  I recommend reading
> > > > > > > http://0pointer.net/blog/projects/journalctl.html for
examples of
> > > the
> > > > > kind of
> > > > > > > stuff you can do that would be cumbersome, if not
*impossible* with
> > > > > regular
> > > > > > > syslog.
> > > > > >
> > > > > > Except that I get lots of messages about the system journal
missing
> > > > > > messages when forwarding to syslog, so how can I make sure this
does
> > > not
> > > > > > happening?
> > > > >
> > > > > Could you please show those messages? systemd sends *everything*
to the
> > > > > journal, and then the journal (optionally) can send it too to a
regular
> > > > > syslog. In that sense, it's impossible for the journal to miss any
> > > message.
> > > > >
> > > > > The only way in which the journal could miss messages is at very
early
> > > boot
> > > > > stages; but with a proper initramfs (like the ones generated with
> > > dracut),
> > > > > even those get caught. You get to put an instance of systemd and
the
> > > > > journal inside the initramfs, and so it's available almost from
the
> > > > > beginning.
> > > > >
> > > > > And if you use gummiboot, then you can even log from the moment
the UEFI
> > > > > firmware comes to life.
> > > >
> > > > So, I get lots of messages in my regular syslog-ng /var/log/messages
> > > > like the following:
> > > > Feb 23 12:47:52 ccs.covici.com systemd-journal[715]: Forwarding to
> > > > syslog missed 15 messages.
> > > >
> > > > So, I saw a post on Google to up the queue length, and I uped it to
200,
> > > > but no joy, still get the messages like the one above.
> > >
> > > Are you using the unit file provided by syslog-ng (systemd-delta
doesn't
> > > mention syslog)? Also, is /etc/systemd/system/syslog.service is a link
> > > to /usr/lib/systemd/system/syslog-ng.service?
> > >
> > > I do, and I don't get any of those messages. I use the default journal
> > > configuration. According to [1], this should be fixed.
> >
> > I remember getting a small number of messages like that, too, on my
laptop.
> > However, it's at the university, so I can't check now to see what types
of
> > messages were missed (if any; if I understand [1] correctly, those
messages are
> > most likely bogus?).
> >
> > But yeah, that's any idea, Covici: see what's in /var/log/messages,
compare that
> > to the journalctl output, and check if any messages were actually
missed ("diff
> > -U" might be of help here).  And if/once you did that, what kinds of
messages
> > were missed, if any?  If those messages really are bogus, you shouldn't
see any
> > differences between the two.
> >
> > > Regards.
> > >
> > > https://github.com/balabit/syslog-ng/issues/314
> >
> > Note that that fix would only be in the ~arch version of syslog-ng, the
current
> > stable version (3.4.8) is a few months too old.
>
> I am up to 3.6 something, so the fix should be there.  But my unit file
> is different, so that remains to check.

I would try the provided unit file. It seems that the only difference with
yours is that it doesn't comment the Restart=on-failure line, and that it
has StandardOutput=null.

I think the general idea is always to use upstream's unit files. They write
the software, supposedly they should know better.

Regards.
--
Canek Peláez Valdés
Profesor de asignatura, Facultad de Ciencias
Universidad Nacional Autónoma de México

[-- Attachment #2: Type: text/html, Size: 9785 bytes --]

Re: [gentoo-user] syslog-ng: how to read the log files

[covici]

Canek Peláez Valdés <caneko@gmail.com> wrote:

> On Mon, Feb 23, 2015 at 1:31 PM, <covici@ccs.covici.com> wrote:
> >
> > Marc Joliet <marcec@gmx.de> wrote:
> >
> > > Am Mon, 23 Feb 2015 12:10:18 -0600
> > > schrieb Canek Peláez Valdés <caneko@gmail.com>:
> > >
> > > > On Mon, Feb 23, 2015 at 11:49 AM, <covici@ccs.covici.com> wrote:
> > > > >
> > > > > Canek Peláez Valdés <caneko@gmail.com> wrote:
> > > > >
> > > > > > On Mon, Feb 23, 2015 at 3:41 AM, <covici@ccs.covici.com> wrote:
> > > > > > >
> > > > > > > Marc Joliet <marcec@gmx.de> wrote:
> > > > > > >
> > > > > > > > Am Mon, 23 Feb 2015 00:41:50 +0100
> > > > > > > > schrieb lee <lee@yagibdah.de>:
> > > > > > > >
> > > > > > > > > Neil Bothwick <neil@digimed.co.uk> writes:
> > > > > > > > >
> > > > > > > > > > On Wed, 18 Feb 2015 21:49:54 +0100, lee wrote:
> > > > > > > > > >
> > > > > > > > > >> > I wonder if the OP is using systemd and trying to read
> the
> > > > > > journal
> > > > > > > > > >> > files?
> > > > > > > > > >>
> > > > > > > > > >> Nooo, I hate systemd ...
> > > > > > > > > >>
> > > > > > > > > >> What good are log files you can't read?
> > > > > > > > > >
> > > > > > > > > > You can't read syslog-ng log files without some reading
> > > > software,
> > > > > > usually
> > > > > > > > > > a combination of cat, grep and less. systemd does it all
> with
> > > > > > journalctl.
> > > > > > > > > >
> > > > > > > > > > There are good reasons to not use systemd, this isn't one
> of
> > > > them.
> > > > > > > > >
> > > > > > > > > To me it is one of the good reasons, and an important one.
> Plain
> > > > text
> > > > > > > > > can usually always be read without further ado, be it from
> rescue
> > > > > > > > > systems you booted or with software available on different
> > > > operating
> > > > > > > > > systems.  It can be also be processed with scripts and sent
> as
> > > > email.
> > > > > > > > > You can probably even read it on your cell phone.  You can
> still
> > > > read
> > > > > > > > > log files that were created 20 years ago when they are
> plain text.
> > > > > > > > >
> > > > > > > > > Can you do all that with the binary files created by
> systemd?  I
> > > > can't
> > > > > > > > > even read them on a working system.
> > > > > > > >
> > > > > > > > What Canek and Rich already said is good, but I'll just add
> this:
> > > > it's
> > > > > > not like
> > > > > > > > you can't run a classic syslog implementation alongside the
> systemd
> > > > > > journal.
> > > > > > > > On my systems, by *default*, syslog-ng kept working as usual,
> > > > getting
> > > > > > the logs
> > > > > > > > from the systemd journal.  If you want to go further, you can
> even
> > > > > > configure
> > > > > > > > the journal to not store logs permanently, so that you *only*
> end up
> > > > > > with
> > > > > > > > plain-text logs on your system (Duncan on gentoo-amd64 went
> this
> > > > way).
> > > > > > > >
> > > > > > > > So no, the format that the systemd journal uses is most
> decidedly
> > > > *not*
> > > > > > a reason
> > > > > > > > against using systemd.
> > > > > > > >
> > > > > > > > Personally, I'm probably going to uninstall syslog-ng, because
> > > > > > journalctl is
> > > > > > > > *such* a nice way to read logs, so why run something whose
> output
> > > > I'll
> > > > > > never
> > > > > > > > read again?  I recommend reading
> > > > > > > > http://0pointer.net/blog/projects/journalctl.html for
> examples of
> > > > the
> > > > > > kind of
> > > > > > > > stuff you can do that would be cumbersome, if not
> *impossible* with
> > > > > > regular
> > > > > > > > syslog.
> > > > > > >
> > > > > > > Except that I get lots of messages about the system journal
> missing
> > > > > > > messages when forwarding to syslog, so how can I make sure this
> does
> > > > not
> > > > > > > happening?
> > > > > >
> > > > > > Could you please show those messages? systemd sends *everything*
> to the
> > > > > > journal, and then the journal (optionally) can send it too to a
> regular
> > > > > > syslog. In that sense, it's impossible for the journal to miss any
> > > > message.
> > > > > >
> > > > > > The only way in which the journal could miss messages is at very
> early
> > > > boot
> > > > > > stages; but with a proper initramfs (like the ones generated with
> > > > dracut),
> > > > > > even those get caught. You get to put an instance of systemd and
> the
> > > > > > journal inside the initramfs, and so it's available almost from
> the
> > > > > > beginning.
> > > > > >
> > > > > > And if you use gummiboot, then you can even log from the moment
> the UEFI
> > > > > > firmware comes to life.
> > > > >
> > > > > So, I get lots of messages in my regular syslog-ng /var/log/messages
> > > > > like the following:
> > > > > Feb 23 12:47:52 ccs.covici.com systemd-journal[715]: Forwarding to
> > > > > syslog missed 15 messages.
> > > > >
> > > > > So, I saw a post on Google to up the queue length, and I uped it to
> 200,
> > > > > but no joy, still get the messages like the one above.
> > > >
> > > > Are you using the unit file provided by syslog-ng (systemd-delta
> doesn't
> > > > mention syslog)? Also, is /etc/systemd/system/syslog.service is a link
> > > > to /usr/lib/systemd/system/syslog-ng.service?
> > > >
> > > > I do, and I don't get any of those messages. I use the default journal
> > > > configuration. According to [1], this should be fixed.
> > >
> > > I remember getting a small number of messages like that, too, on my
> laptop.
> > > However, it's at the university, so I can't check now to see what types
> of
> > > messages were missed (if any; if I understand [1] correctly, those
> messages are
> > > most likely bogus?).
> > >
> > > But yeah, that's any idea, Covici: see what's in /var/log/messages,
> compare that
> > > to the journalctl output, and check if any messages were actually
> missed ("diff
> > > -U" might be of help here).  And if/once you did that, what kinds of
> messages
> > > were missed, if any?  If those messages really are bogus, you shouldn't
> see any
> > > differences between the two.
> > >
> > > > Regards.
> > > >
> > > > https://github.com/balabit/syslog-ng/issues/314
> > >
> > > Note that that fix would only be in the ~arch version of syslog-ng, the
> current
> > > stable version (3.4.8) is a few months too old.
> >
> > I am up to 3.6 something, so the fix should be there.  But my unit file
> > is different, so that remains to check.
> 
> I would try the provided unit file. It seems that the only difference with
> yours is that it doesn't comment the Restart=on-failure line, and that it
> has StandardOutput=null.
> 
> I think the general idea is always to use upstream's unit files. They write
> the software, supposedly they should know better.

I agree, but at the time, there was none, so I had to find something  on
the internet, maybe from arch or somewhere.  I can certainly try the one
provided.


-- 
Your life is like a penny.  You're going to lose it.  The question is:
How do
you spend it?

         John Covici
         covici@ccs.covici.com

Re: [gentoo-user] syslog-ng: how to read the log files

[covici]

Canek Peláez Valdés <caneko@gmail.com> wrote:

> On Mon, Feb 23, 2015 at 1:31 PM, <covici@ccs.covici.com> wrote:
> >
> > Marc Joliet <marcec@gmx.de> wrote:
> >
> > > Am Mon, 23 Feb 2015 12:10:18 -0600
> > > schrieb Canek Peláez Valdés <caneko@gmail.com>:
> > >
> > > > On Mon, Feb 23, 2015 at 11:49 AM, <covici@ccs.covici.com> wrote:
> > > > >
> > > > > Canek Peláez Valdés <caneko@gmail.com> wrote:
> > > > >
> > > > > > On Mon, Feb 23, 2015 at 3:41 AM, <covici@ccs.covici.com> wrote:
> > > > > > >
> > > > > > > Marc Joliet <marcec@gmx.de> wrote:
> > > > > > >
> > > > > > > > Am Mon, 23 Feb 2015 00:41:50 +0100
> > > > > > > > schrieb lee <lee@yagibdah.de>:
> > > > > > > >
> > > > > > > > > Neil Bothwick <neil@digimed.co.uk> writes:
> > > > > > > > >
> > > > > > > > > > On Wed, 18 Feb 2015 21:49:54 +0100, lee wrote:
> > > > > > > > > >
> > > > > > > > > >> > I wonder if the OP is using systemd and trying to read
> the
> > > > > > journal
> > > > > > > > > >> > files?
> > > > > > > > > >>
> > > > > > > > > >> Nooo, I hate systemd ...
> > > > > > > > > >>
> > > > > > > > > >> What good are log files you can't read?
> > > > > > > > > >
> > > > > > > > > > You can't read syslog-ng log files without some reading
> > > > software,
> > > > > > usually
> > > > > > > > > > a combination of cat, grep and less. systemd does it all
> with
> > > > > > journalctl.
> > > > > > > > > >
> > > > > > > > > > There are good reasons to not use systemd, this isn't one
> of
> > > > them.
> > > > > > > > >
> > > > > > > > > To me it is one of the good reasons, and an important one.
> Plain
> > > > text
> > > > > > > > > can usually always be read without further ado, be it from
> rescue
> > > > > > > > > systems you booted or with software available on different
> > > > operating
> > > > > > > > > systems.  It can be also be processed with scripts and sent
> as
> > > > email.
> > > > > > > > > You can probably even read it on your cell phone.  You can
> still
> > > > read
> > > > > > > > > log files that were created 20 years ago when they are
> plain text.
> > > > > > > > >
> > > > > > > > > Can you do all that with the binary files created by
> systemd?  I
> > > > can't
> > > > > > > > > even read them on a working system.
> > > > > > > >
> > > > > > > > What Canek and Rich already said is good, but I'll just add
> this:
> > > > it's
> > > > > > not like
> > > > > > > > you can't run a classic syslog implementation alongside the
> systemd
> > > > > > journal.
> > > > > > > > On my systems, by *default*, syslog-ng kept working as usual,
> > > > getting
> > > > > > the logs
> > > > > > > > from the systemd journal.  If you want to go further, you can
> even
> > > > > > configure
> > > > > > > > the journal to not store logs permanently, so that you *only*
> end up
> > > > > > with
> > > > > > > > plain-text logs on your system (Duncan on gentoo-amd64 went
> this
> > > > way).
> > > > > > > >
> > > > > > > > So no, the format that the systemd journal uses is most
> decidedly
> > > > *not*
> > > > > > a reason
> > > > > > > > against using systemd.
> > > > > > > >
> > > > > > > > Personally, I'm probably going to uninstall syslog-ng, because
> > > > > > journalctl is
> > > > > > > > *such* a nice way to read logs, so why run something whose
> output
> > > > I'll
> > > > > > never
> > > > > > > > read again?  I recommend reading
> > > > > > > > http://0pointer.net/blog/projects/journalctl.html for
> examples of
> > > > the
> > > > > > kind of
> > > > > > > > stuff you can do that would be cumbersome, if not
> *impossible* with
> > > > > > regular
> > > > > > > > syslog.
> > > > > > >
> > > > > > > Except that I get lots of messages about the system journal
> missing
> > > > > > > messages when forwarding to syslog, so how can I make sure this
> does
> > > > not
> > > > > > > happening?
> > > > > >
> > > > > > Could you please show those messages? systemd sends *everything*
> to the
> > > > > > journal, and then the journal (optionally) can send it too to a
> regular
> > > > > > syslog. In that sense, it's impossible for the journal to miss any
> > > > message.
> > > > > >
> > > > > > The only way in which the journal could miss messages is at very
> early
> > > > boot
> > > > > > stages; but with a proper initramfs (like the ones generated with
> > > > dracut),
> > > > > > even those get caught. You get to put an instance of systemd and
> the
> > > > > > journal inside the initramfs, and so it's available almost from
> the
> > > > > > beginning.
> > > > > >
> > > > > > And if you use gummiboot, then you can even log from the moment
> the UEFI
> > > > > > firmware comes to life.
> > > > >
> > > > > So, I get lots of messages in my regular syslog-ng /var/log/messages
> > > > > like the following:
> > > > > Feb 23 12:47:52 ccs.covici.com systemd-journal[715]: Forwarding to
> > > > > syslog missed 15 messages.
> > > > >
> > > > > So, I saw a post on Google to up the queue length, and I uped it to
> 200,
> > > > > but no joy, still get the messages like the one above.
> > > >
> > > > Are you using the unit file provided by syslog-ng (systemd-delta
> doesn't
> > > > mention syslog)? Also, is /etc/systemd/system/syslog.service is a link
> > > > to /usr/lib/systemd/system/syslog-ng.service?
> > > >
> > > > I do, and I don't get any of those messages. I use the default journal
> > > > configuration. According to [1], this should be fixed.
> > >
> > > I remember getting a small number of messages like that, too, on my
> laptop.
> > > However, it's at the university, so I can't check now to see what types
> of
> > > messages were missed (if any; if I understand [1] correctly, those
> messages are
> > > most likely bogus?).
> > >
> > > But yeah, that's any idea, Covici: see what's in /var/log/messages,
> compare that
> > > to the journalctl output, and check if any messages were actually
> missed ("diff
> > > -U" might be of help here).  And if/once you did that, what kinds of
> messages
> > > were missed, if any?  If those messages really are bogus, you shouldn't
> see any
> > > differences between the two.
> > >
> > > > Regards.
> > > >
> > > > https://github.com/balabit/syslog-ng/issues/314
> > >
> > > Note that that fix would only be in the ~arch version of syslog-ng, the
> current
> > > stable version (3.4.8) is a few months too old.
> >
> > I am up to 3.6 something, so the fix should be there.  But my unit file
> > is different, so that remains to check.
> 
> I would try the provided unit file. It seems that the only difference with
> yours is that it doesn't comment the Restart=on-failure line, and that it
> has StandardOutput=null.
> 
> I think the general idea is always to use upstream's unit files. They write
> the software, supposedly they should know better.

I did change the unit file, but no joy, I still get messages like this:
Feb 23 18:16:05 ccs.covici.com systemd-journal[715]: Forwarding to
syslog missed 13 messages.


-- 
Your life is like a penny.  You're going to lose it.  The question is:
How do
you spend it?

         John Covici
         covici@ccs.covici.com

Re: [gentoo-user] syslog-ng: how to read the log files

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 561 bytes --]

On Mon, 23 Feb 2015 18:18:36 -0500, covici@ccs.covici.com wrote:

> I did change the unit file, but no joy, I still get messages like this:
> Feb 23 18:16:05 ccs.covici.com systemd-journal[715]: Forwarding to
> syslog missed 13 messages.

I used to get messages like that. Sometimes substantial numbers of
messages, 100+ was far from uncommon. But the last such message in my
journal was on November 6th.

That's on my laptop, my desktop doesn't have a single such message.


-- 
Neil Bothwick

Disinformation is not as good as datinformation.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [gentoo-user] syslog-ng: how to read the log files

[Peter Humphrey]

On Monday 23 February 2015 23:29:49 Neil Bothwick wrote:
> On Mon, 23 Feb 2015 18:18:36 -0500, covici@ccs.covici.com wrote:
> > I did change the unit file, but no joy, I still get messages like
> > this: Feb 23 18:16:05 ccs.covici.com systemd-journal[715]:
> > Forwarding to syslog missed 13 messages.
> 
> I used to get messages like that. Sometimes substantial numbers of
> messages, 100+ was far from uncommon. But the last such message in my
> journal was on November 6th.
> 
> That's on my laptop, my desktop doesn't have a single such message.

Thank Goodness! Someone who knows enough to trim out the bits of the 
message he's not replying to.

Why do you others make me page-down eight times to find what you've 
written in reply to the last three lines of the preceding message?

-- 
Rgds
Peter.

Re: [gentoo-user] syslog-ng: how to read the log files

[Marc Joliet]

[-- Attachment #1: Type: text/plain, Size: 1116 bytes --]

Am Tue, 24 Feb 2015 00:50:30 +0000
schrieb Peter Humphrey <peter@prh.myzen.co.uk>:

> On Monday 23 February 2015 23:29:49 Neil Bothwick wrote:
> > On Mon, 23 Feb 2015 18:18:36 -0500, covici@ccs.covici.com wrote:
> > > I did change the unit file, but no joy, I still get messages like
> > > this: Feb 23 18:16:05 ccs.covici.com systemd-journal[715]:
> > > Forwarding to syslog missed 13 messages.
> > 
> > I used to get messages like that. Sometimes substantial numbers of
> > messages, 100+ was far from uncommon. But the last such message in my
> > journal was on November 6th.
> > 
> > That's on my laptop, my desktop doesn't have a single such message.
> 
> Thank Goodness! Someone who knows enough to trim out the bits of the 
> message he's not replying to.
> 
> Why do you others make me page-down eight times to find what you've 
> written in reply to the last three lines of the preceding message?

*Checks to see if he's guilty.*

*Hangs head in shame.*

-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [gentoo-user] syslog-ng: how to read the log files

[Matti Nykyri]

> On Feb 24, 2015, at 2:50, Peter Humphrey <peter@prh.myzen.co.uk> wrote:
> 
> Thank Goodness! Someone who knows enough to trim out the bits of the 
> message he's not replying to.
> 
> Why do you others make me page-down eight times to find what you've 
> written in reply to the last three lines of the preceding message?

+1

-- 
-Matti

Re: [gentoo-user] syslog-ng: how to read the log files

[Stroller]

On Sun, 22 February 2015, at 11:48 pm, lee <lee@yagibdah.de> wrote:
> 
>>>> I believe this may be bug 406623.
>>>> 
>>>> https://bugs.gentoo.org/show_bug.cgi?id=406623
>>> 
>>> That's almost three years old and should apparently be fixed?
>> 
>> It's only been closed in the last few weeks. 
> 
> Still I wonder why it took so long to fix it.

That's hardly unusual - Gentoo is massively understaffed.

Even version bumps may sometimes take weeks to be actioned.

Stroller.

Re: [gentoo-user] syslog-ng: how to read the log files

[lee]

Canek Peláez Valdés <caneko@gmail.com> writes:

> On Sun, Feb 22, 2015 at 6:41 PM, lee <lee@yagibdah.de> wrote:
>>
>> Neil Bothwick <neil@digimed.co.uk> writes:
>>
>> > On Wed, 18 Feb 2015 21:49:54 +0100, lee wrote:
>> >
>> >> > I wonder if the OP is using systemd and trying to read the journal
>> >> > files?
>> >>
>> >> Nooo, I hate systemd ...
>> >>
>> >> What good are log files you can't read?
>> >
>> > You can't read syslog-ng log files without some reading software,
> usually
>> > a combination of cat, grep and less. systemd does it all with
> journalctl.
>> >
>> > There are good reasons to not use systemd, this isn't one of them.
>>
>> To me it is one of the good reasons, and an important one.  Plain text
>> can usually always be read without further ado, be it from rescue
>> systems you booted or with software available on different operating
>> systems.  It can be also be processed with scripts and sent as email.
>> You can probably even read it on your cell phone.  You can still read
>> log files that were created 20 years ago when they are plain text.
>>
>> Can you do all that with the binary files created by systemd?
>
> Yes, you can.

You can predict the next 20 years?

>> I can't even read them on a working system.
>
> If that's true (which I highly doubt, more probably you don't know how to
> read them), then it's a bug and should be reported and fixed.

I read log files with less.  The bug is that systemd uses some sort of
binary files, and they aren't going to fix it.  They even won't fix
their misunderstanding of what "disabled" means.  So why make bug
reports?


-- 
Again we must be afraid of speaking of daemons for fear that daemons
might swallow us.  Finally, this fear has become reasonable.

Re: [gentoo-user] syslog-ng: how to read the log files

[lee]

Rich Freeman <rich0@gentoo.org> writes:

> On Sun, Feb 22, 2015 at 6:41 PM, lee <lee@yagibdah.de> wrote:
>>
>> To me it is one of the good reasons, and an important one.  Plain text
>> can usually always be read without further ado, be it from rescue
>> systems you booted or with software available on different operating
>> systems.  It can be also be processed with scripts and sent as email.
>> You can probably even read it on your cell phone.  You can still read
>> log files that were created 20 years ago when they are plain text.
>
> Doing any of that stuff requires the use of software capable of
> reading text files.  It isn't like you can just interpret the magnetic
> fields on your disk with your eyes.

Yes, and it doesn't seem very likely that it'll become impossible to
read text files in the next 20 years.

> Sure, there are a lot more utilities that can read text files than
> journal files, but you just need to arrange to have them handy.
> They'll be ubiquitous before long since every distro around will end
> up needing them.

Hopefully not, systemd is a bad thing for many reasons.

>> Can you do all that with the binary files created by systemd?  I can't
>> even read them on a working system.
>>
>
> You just type journalctl to read the live system logs.  For offline
> use you just type journalctl --file=filename.  Or you can just run
> strings on the file I imagine if you're desperate.  If it doesn't work
> on a "working system" then your system isn't working.

See, ppl already claim that when something that comes from systemd isn't
working, then the system isn't working.  Unfortunately, they overlook
that when things systemd don't work by design, it's bad design or a
problem of systemd rather than the system not working.


-- 
Again we must be afraid of speaking of daemons for fear that daemons
might swallow us.  Finally, this fear has become reasonable.

Re: [gentoo-user] syslog-ng: how to read the log files

[lee]

Marc Joliet <marcec@gmx.de> writes:

>> Can you do all that with the binary files created by systemd?  I can't
>> even read them on a working system.
>
> What Canek and Rich already said is good, but I'll just add this: it's not like
> you can't run a classic syslog implementation alongside the systemd journal.
> On my systems, by *default*, syslog-ng kept working as usual, getting the logs
> from the systemd journal.  If you want to go further, you can even configure
> the journal to not store logs permanently, so that you *only* end up with
> plain-text logs on your system (Duncan on gentoo-amd64 went this way).
>
> So no, the format that the systemd journal uses is most decidedly *not* a reason
> against using systemd.

It is only one of the many reasons.  I don't find it advantageous to
have to waste additional resources to be able to read the log files.

> Personally, I'm probably going to uninstall syslog-ng, because journalctl is
> *such* a nice way to read logs, so why run something whose output I'll never
> read again?

If you like it, nobody prevents you from using it.  It's good to have
many options.  Just don't force others to use it as well.


-- 
Again we must be afraid of speaking of daemons for fear that daemons
might swallow us.  Finally, this fear has become reasonable.

Re: [gentoo-user] syslog-ng: how to read the log files

[Rich Freeman]

On Mon, May 4, 2015 at 2:14 AM, lee <lee@yagibdah.de> wrote:
> Marc Joliet <marcec@gmx.de> writes:
>
>> Personally, I'm probably going to uninstall syslog-ng, because journalctl is
>> *such* a nice way to read logs, so why run something whose output I'll never
>> read again?
>
> If you like it, nobody prevents you from using it.  It's good to have
> many options.  Just don't force others to use it as well.
>

Who is forcing anybody to use anything?  Did Lennart break into your
house with an RHEL 7 disk and force you to install it at gunpoint or
something?  You did a great job holding out under the torture - that
would explain your 2.5 month absence from this long-dead thread.
Fortunately, while you were gone nobody treecleaned sysvinit, not that
treecleaning a package prevents anybody from using it.

-- 
Rich

Re: [gentoo-user] syslog-ng: how to read the log files

[Tom H]

On Mon, May 4, 2015 at 1:57 AM, lee <lee@yagibdah.de> wrote:
> Canek Peláez Valdés <caneko@gmail.com> writes:
>> On Sun, Feb 22, 2015 at 6:41 PM, lee <lee@yagibdah.de> wrote:
>>>
>>> I can't even read them on a working system.
>>
>> If that's true (which I highly doubt, more probably you don't know how to
>> read them), then it's a bug and should be reported and fixed.
>
> I read log files with less. The bug is that systemd uses some sort of
> binary files, and they aren't going to fix it. They even won't fix
> their misunderstanding of what "disabled" means. So why make bug
> reports?

The systemd developers' use of disable/mask isn't wrong simply because
you disagree with them.

"systemctl disable unit" is the same as "blacklist module": the
unit/module can be loaded manually or as a dependency.

"systemctl mask unit" is the same as "install module /bin/true": the
unit/module can't be loaded.

Re: [gentoo-user] syslog-ng: how to read the log files

[lee]

Tom H <tomh0665@gmail.com> writes:

> On Mon, May 4, 2015 at 1:57 AM, lee <lee@yagibdah.de> wrote:
>> Canek Peláez Valdés <caneko@gmail.com> writes:
>>> On Sun, Feb 22, 2015 at 6:41 PM, lee <lee@yagibdah.de> wrote:
>>>>
>>>> I can't even read them on a working system.
>>>
>>> If that's true (which I highly doubt, more probably you don't know how to
>>> read them), then it's a bug and should be reported and fixed.
>>
>> I read log files with less. The bug is that systemd uses some sort of
>> binary files, and they aren't going to fix it. They even won't fix
>> their misunderstanding of what "disabled" means. So why make bug
>> reports?
>
> The systemd developers' use of disable/mask isn't wrong simply because
> you disagree with them.

No, it's wrong because they don't know what "disabled" means.  Feel free
to look into dictionaries and to examine the use of the word "disabled"
in it's language to find out what it means.


-- 
Again we must be afraid of speaking of daemons for fear that daemons
might swallow us.  Finally, this fear has become reasonable.

Re: [gentoo-user] syslog-ng: how to read the log files

[lee]

Rich Freeman <rich0@gentoo.org> writes:

> On Mon, May 4, 2015 at 2:14 AM, lee <lee@yagibdah.de> wrote:
>> Marc Joliet <marcec@gmx.de> writes:
>>
>>> Personally, I'm probably going to uninstall syslog-ng, because journalctl is
>>> *such* a nice way to read logs, so why run something whose output I'll never
>>> read again?
>>
>> If you like it, nobody prevents you from using it.  It's good to have
>> many options.  Just don't force others to use it as well.
>>
>
> Who is forcing anybody to use anything?

Look around and you will find that systemd has taken over Linux, with a
few exceptions as in distributions like Gentoo.  The taking over will
probably continue until you cannot use Linux anymore without using
systemd.  If that's not forcing, then what is it?


-- 
Again we must be afraid of speaking of daemons for fear that daemons
might swallow us.  Finally, this fear has become reasonable.

Re: [gentoo-user] syslog-ng: how to read the log files

[Peter Humphrey]

[-- Attachment #1: Type: text/plain, Size: 1078 bytes --]

On Saturday 09 May 2015 17:01:00 lee wrote:
> Tom H <tomh0665@gmail.com> writes:
> > The systemd developers' use of disable/mask isn't wrong simply because
> > you disagree with them.
> 
> No, it's wrong because they don't know what "disabled" means.  Feel free
> to look into dictionaries and to examine the use of the word "disabled"
> in it's language to find out what it means.

I've been through this before, and a certain contributor to e-mail lists is 
still in my kill filter because of it. Some computing people, mostly American 
in my experience, insist that "disabled" means the same as "switched off". No 
amount of pointing out the error of this makes any difference. They merely 
shrug and cite custom and practice. It was never custom or practice in my 
patch of the forest.

Incidentally, there's another stupidity in an ancient CPU instruction set, I 
think 8080. If I move something from A to B it's no longer at A, but in the 
mov instruction it finishes up in both places.

Sometimes I wish the language were still extended only by scholars.

-- 
Rgds
Peter

[-- Attachment #2: Type: text/html, Size: 4217 bytes --]

Re: [gentoo-user] syslog-ng: how to read the log files

[Rich Freeman]

On Sat, May 9, 2015 at 11:08 AM, lee <lee@yagibdah.de> wrote:
> Rich Freeman <rich0@gentoo.org> writes:
>>
>> Who is forcing anybody to use anything?
>
> Look around and you will find that systemd has taken over Linux, with a
> few exceptions as in distributions like Gentoo.  The taking over will
> probably continue until you cannot use Linux anymore without using
> systemd.  If that's not forcing, then what is it?

They're not forcing you to use Linux at all, let alone Linux with systemd.

Fedora v13 never contained systemd as far as I'm aware.  Today it
still works EXACTLY as it did then.  Nobody is preventing you from
using that.  The same is true for any other distro that has adopted
systemd.

Your complaint isn't that people are forcing you to use systemd.  Your
complaint is that somebody isn't building a linux distro to your
personal specifications free of charge.  With FOSS we all contribute
code that anybody can use for as long as they wish.  The fact that you
don't want to use the code that somebody wrote a few years ago and
would rather they write updated software (perhaps with security
vulnerabilities removed, features added, and so on) doesn't obligate
others to create it for you.

And that is my issue with this line of argument.  It assumes that you
have a right to demand that others create free software for you, and
that they do it to your specifications.

I get your frustration.  There have been FOSS projects that were
discontinued that I've thought were really valuable.  However, while I
mourn their indefinite slumber, I'm not going to complain that the
devs chose to move on.  I never paid them for what they gave me in the
first place, and I have no right to demand more.  The same is true of
whoever maintains your init scripts.  If you have a support contract
that requires somebody to backport fixes to whatever you're using for
10 years, and they're breaking that contract, then I can only agree
that you're in the right.  Short of that, you've gotten what you've
paid for.  I understand your frustration, but I don't think the use of
terms like "force" is justified.

-- 
Rich

[gentoo-user] Re: syslog-ng: how to read the log files

[»Q«]

Bug Description:  gentoo-user subscribers should be made aware of the
    possible systemd apocalypse

Status:  RESOLVED FIXED

Resolution Summary:  Countless posts have been made to the list about
    this, emphasising its severity with helpful language such as
    "shoved down our throats", "forced", and "complete takeover of
    Linux".  Everyone paying any attention to the list is now fully
    aware.

Comment #1 from ConcernedUser -

    I think maybe this bug has been closed prematurely.  ISTM there are
    still people on the list who don't quite get how very bad it could
    be.  Shouldn't we try just a few more posts to make sure?   E.g.,
    I'm not sure we've tried the phrase "destruction of all your
    precious freedom" yet.  (If we have, I missed it, so maybe somebody
    else missed it too.)

Comment #2 from »Q« - 

    Stop it.  Just stop it.