[gentoo-user] logs in the browser?

[gentoo-user] logs in the browser?

[Stefan G. Weichinger]

As I see that syslog-thread ...

I think of setting up something for apache that allows a client to
browse and search through the postfix-logs for his domain.

Any good hints what to use for that purpose?

Stefan

Re: [gentoo-user] logs in the browser?

[Canek Peláez Valdés]

[-- Attachment #1: Type: text/plain, Size: 747 bytes --]

On Mon, Feb 23, 2015 at 4:20 PM, Stefan G. Weichinger <lists@xunil.at>
wrote:
>
>
> As I see that syslog-thread ...
>
> I think of setting up something for apache that allows a client to
> browse and search through the postfix-logs for his domain.
>
> Any good hints what to use for that purpose?

Stefan, if you already have systemd (which I believe you do), why don't you
compile in the support for microhttpd and use the journal? This is the
exact scenario for which systemd-journal-gatewayd[1] was written.

Regards.

[1]
http://www.freedesktop.org/software/systemd/man/systemd-journal-gatewayd.service.html
--
Canek Peláez Valdés
Profesor de asignatura, Facultad de Ciencias
Universidad Nacional Autónoma de México

[-- Attachment #2: Type: text/html, Size: 989 bytes --]

Re: [gentoo-user] logs in the browser?

[Stefan G. Weichinger]

Am 24.02.2015 um 03:14 schrieb Canek Peláez Valdés:

> Stefan, if you already have systemd (which I believe you do), why don't you
> compile in the support for microhttpd and use the journal? This is the
> exact scenario for which systemd-journal-gatewayd[1] was written.

very good ... enabled it on one of my machines, looks good.

I am unsure if it will be possible to see only postfix.service (easier)
AND only the lines relevant for the domain of the specific customer by
doing this?

thanks, Stefan

Re: [gentoo-user] logs in the browser?

[Rich Freeman]

On Tue, Feb 24, 2015 at 4:50 AM, Stefan G. Weichinger <lists@xunil.at> wrote:
> Am 24.02.2015 um 03:14 schrieb Canek Peláez Valdés:
>
>> Stefan, if you already have systemd (which I believe you do), why don't you
>> compile in the support for microhttpd and use the journal? This is the
>> exact scenario for which systemd-journal-gatewayd[1] was written.
>
> very good ... enabled it on one of my machines, looks good.
>
> I am unsure if it will be possible to see only postfix.service (easier)

I suspect this is trivial - it looks like something like this would work:
http://.../entries?_SYSTEMD_UNIT=postfix.service

(note, you might need to tweak that - I haven't used the http gateway
personally and am going from the manpage)

> AND only the lines relevant for the domain of the specific customer by
> doing this?

I think you're going to be stuck here unless they come from different
machines or something like that.  Obviously you can pipe the output
through grep but journald will only pre-filter the output using
journal fields, like facility, priority, etc.  syslog only provided a
few fields for clients to specify, and this is probably because in the
end the data just got dumped to a text file so that it wasn't
searchable by field anyway.  It would be nice if they extended the
syslog protocol for systemd and made it possible for clients to
specify additional fields, but obviously the client would need to
support this (likely sending logs over dbus or such).

The http gateway seems like it is intended more as a transport
mechanism with some usability for ad-hoc human viewing.  It isn't a
full-fledged log analysis tool.  The fact that journald can output in
JSON with uuids for each entry should make it far easier to parse its
logs with an analysis tool, but I think all those vendors are playing
catch-up.  I suspect they'll support it fairly soon once they see
everybody using it.  From a machine parsing standpoint the fielded
binary format makes a lot more sense.

-- 
Rich

Re: [gentoo-user] logs in the browser?

[Stefan G. Weichinger]

On 24.02.2015 13:14, Rich Freeman wrote:

> I suspect this is trivial - it looks like something like this would work:
> http://.../entries?_SYSTEMD_UNIT=postfix.service

Yes, correct, as I thought this is the easy part.

Works:

http://mythtv.local:19531/entries?_SYSTEMD_UNIT=postfix.service

(using my mythbackend as test target).


> (note, you might need to tweak that - I haven't used the http gateway
> personally and am going from the manpage)
> 
>> AND only the lines relevant for the domain of the specific customer by
>> doing this?
> 
> I think you're going to be stuck here unless they come from different
> machines or something like that.  Obviously you can pipe the output
> through grep but journald will only pre-filter the output using
> journal fields, like facility, priority, etc.  syslog only provided a
> few fields for clients to specify, and this is probably because in the
> end the data just got dumped to a text file so that it wasn't
> searchable by field anyway.  It would be nice if they extended the
> syslog protocol for systemd and made it possible for clients to
> specify additional fields, but obviously the client would need to
> support this (likely sending logs over dbus or such).
> 
> The http gateway seems like it is intended more as a transport
> mechanism with some usability for ad-hoc human viewing.  It isn't a
> full-fledged log analysis tool.  The fact that journald can output in
> JSON with uuids for each entry should make it far easier to parse its
> logs with an analysis tool, but I think all those vendors are playing
> catch-up.  I suspect they'll support it fairly soon once they see
> everybody using it.  From a machine parsing standpoint the fielded
> binary format makes a lot more sense.

Maybe I could set up some other web-app that (a) looks at the link
pointing to the postfix.service-logs and (b) filters them?

I could post to the systemd-devel-ml ... btw ;-)

Stefan

Re: [gentoo-user] logs in the browser?

[Canek Peláez Valdés]

[-- Attachment #1: Type: text/plain, Size: 949 bytes --]

On Tue, Feb 24, 2015 at 10:33 AM, Stefan G. Weichinger <lists@xunil.at>
wrote:
[ ... ]
> Maybe I could set up some other web-app that (a) looks at the link
> pointing to the postfix.service-logs and (b) filters them?

(With my programmer's hat on): I think the easiest way would be to create a
little client that downloads the logs in JSON format, and then do the
filtering off-site. If I'm not mistaken, libmicrohttpd supports
"Accept-Encoding: gzip", and therefore the used bandwidth should not be a
lot.

Also, you can get the last cursor from the journal the first time, and next
time you download logs, you start from that cursor on, so you don't
download everything again.

I don't see many advantages on doing the filtering on-site. Specially if,
after a while, you are handling several servers.

Regards.
--
Canek Peláez Valdés
Profesor de asignatura, Facultad de Ciencias
Universidad Nacional Autónoma de México

[-- Attachment #2: Type: text/html, Size: 1109 bytes --]

Re: [gentoo-user] logs in the browser?

[Rich Freeman]

On Tue, Feb 24, 2015 at 10:33 AM, Stefan G. Weichinger <lists@xunil.at> wrote:
> Maybe I could set up some other web-app that (a) looks at the link
> pointing to the postfix.service-logs and (b) filters them?
>
> I could post to the systemd-devel-ml ... btw ;-)
>

Seems like there should be a systemd-users mailing list, actually.
This sort of situation is completely distro-agnostic.

You certainly could design such an application.  If you do so I'd
consider pulling the journald logs in JSON format.  I'd also see if
somebody actually has written a journald library/class/etc for your
language of choice - it seems like that is the sort of thing that is
likely to exist soon if not already.  One of the goals journald (and
systemd) is to provide more of an API for services so that there is
less parsing of text files and communicating via signals/etc.  I'm
sure with appropriate permissions a process could just obtain log
entries via dbus, and using cursors poll for new entries (or maybe
there is a push/stream mechanism).

Really though it seems like the solution is a generic log monitor with
rules for such things, with the monitor utilizing the JSON data from
journald for richer metadata/efficiency/accuracy.

-- 
Rich

Re: [gentoo-user] logs in the browser?

[Stefan G. Weichinger]

On 24.02.2015 17:01, Rich Freeman wrote:

> Seems like there should be a systemd-users mailing list, actually.
> This sort of situation is completely distro-agnostic.

Yes! And systemd-devel ml is always kind of "they will laugh at me and
say ugly things!" ;-)

> You certainly could design such an application.  If you do so I'd
> consider pulling the journald logs in JSON format.  I'd also see if
> somebody actually has written a journald library/class/etc for your
> language of choice - it seems like that is the sort of thing that is
> likely to exist soon if not already.  One of the goals journald (and
> systemd) is to provide more of an API for services so that there is
> less parsing of text files and communicating via signals/etc.  I'm
> sure with appropriate permissions a process could just obtain log
> entries via dbus, and using cursors poll for new entries (or maybe
> there is a push/stream mechanism).
> 
> Really though it seems like the solution is a generic log monitor with
> rules for such things, with the monitor utilizing the JSON data from
> journald for richer metadata/efficiency/accuracy.

Well, it might be a nice challenge to do so ... sure!

The real world challenge is that I have to provide the postfix-logs for
one domain on one server (so far) for one "admin" to browse through
(without giving him ssh-access ... it's a noob ...)

No overkill paid here.

But maybe I ask on systemd-devel-ml ...

Stefan

Re: [gentoo-user] logs in the browser?

[Rich Freeman]

On Tue, Feb 24, 2015 at 3:27 PM, Stefan G. Weichinger <lists@xunil.at> wrote:
> On 24.02.2015 17:01, Rich Freeman wrote:
>
>> Seems like there should be a systemd-users mailing list, actually.
>> This sort of situation is completely distro-agnostic.
>
> Yes! And systemd-devel ml is always kind of "they will laugh at me and
> say ugly things!" ;-)
>

Honestly, I've never received treatment like this on the systemd lists
or irc channel.  However, when I post there it tends to be about
specific bugs with details/etc.  You're not really talking about
systemd development, but rather questions around best practices for
using it, discussion, etc.  It would be like posting this thread on
gentoo-dev - it isn't the purpose of the list, but such discussion is
completely welcome here.  If we were discussing an eclass change, on
the other hand, gentoo-dev would be the right place.

You might try asking on #systemd.  That doesn't really have a defined
topic afaict, and discussion is a lot more free.  Of course, the
downside to irc is you only hear from people online at the moment.
You might ask about a user mailing list.

-- 
Rich