[gentoo-user] problem filtering portage messages...

[gentoo-user] problem filtering portage messages...

[Jarry]

Hi,
I have a log-server (syslog-ng) collecting logs from my servers.
Everything works for standard logs: clients forward them to server
where they are filtered (based on facility or application) and
splitted into a few files (for each client).

The problem is with portage logs: I can not find any pattern
I could use to filter them out. For example, this is small part
of /var/log/emerge.log:

1250923552:  *** emerge  sync
1250923552:  === sync
1250923552: >>> Starting rsync with rsync://140.211.166.165/gentoo-portage
1250923665: === Sync completed with rsync://140.211.166.165/gentoo-portage
1250923666:  *** terminating.
1250923760: Started emerge on: Aug 22, 2009 06:49:20
1250923760:  *** emerge  depclean
1250923760:  >>> depclean
...

Can I somehow force portage to write all its logs (summary.log,
emerge-fetch.log, emerge.log) in standard syslog format RFC-3164?
I mean with all that stuff like priority (facility), header
(timestamp, hostname), application, etc. Even better if portage
could write its logs to /dev/log...

Jarry

-- 
_______________________________________________________________
This mailbox accepts e-mails only from selected mailing-lists!
Everything else is considered to be spam and therefore deleted.

Re: [gentoo-user] problem filtering portage messages...

[Albert Hopkins]

On Sat, 2009-08-22 at 10:39 +0200, Jarry wrote:
> Hi,
> I have a log-server (syslog-ng) collecting logs from my servers.
> Everything works for standard logs: clients forward them to server
> where they are filtered (based on facility or application) and
> splitted into a few files (for each client).
> 
> The problem is with portage logs: I can not find any pattern
> I could use to filter them out. For example, this is small part
> of /var/log/emerge.log:
> 
> 1250923552:  *** emerge  sync
> 1250923552:  === sync
> 1250923552: >>> Starting rsync with rsync://140.211.166.165/gentoo-portage
> 1250923665: === Sync completed with rsync://140.211.166.165/gentoo-portage
> 1250923666:  *** terminating.
> 1250923760: Started emerge on: Aug 22, 2009 06:49:20
> 1250923760:  *** emerge  depclean
> 1250923760:  >>> depclean
> ...
> 
> Can I somehow force portage to write all its logs (summary.log,
> emerge-fetch.log, emerge.log) in standard syslog format RFC-3164?
> I mean with all that stuff like priority (facility), header
> (timestamp, hostname), application, etc. Even better if portage
> could write its logs to /dev/log...

Portage doesn't use syslog format (that's why it doesn't log to
syslog ;)

There is the portrage elog system but's kinda a misnomer (maybe should
be called portage message system).  You can change
PORTAGE_ELOG_SYSTEM="syslog" but it won't do what you are expecting.

I'm not sure what you are trying to "filter out".  Portage logs are
relatively machine-readable (e.g. qlop).

If you don't mind getting your hands dirty, portage is written mostly in
python and bash so it shouldn't be that difficult to dig into the parts
that handle emerge.log and emerge-fetch.log and convert them to syslog.

-a

Re: [gentoo-user] problem filtering portage messages...

[Jarry]

Albert Hopkins wrote:
> On Sat, 2009-08-22 at 10:39 +0200, Jarry wrote:
>> The problem is with portage logs: I can not find any pattern
>> I could use to filter them out.

> I'm not sure what you are trying to "filter out".  Portage logs are
> relatively machine-readable (e.g. qlop).

syslog-ng in client mode does not open new connection for every
source (be it /dev/log, /proc/kmsg/, /var/log/messages, etc.).
Instead of that, it opens only one connection and "pumps" logs
to server in one stream. It is a log-server's job to split it out.

The first quite natural filter-rule is to split logs according
to client-IP (or hostname). The second level is to split logs
according to application. I do not want to have logs from kernel,
sshd, apache, ntp, portage, ftp, dns, mixed together in one file.

And that is a problem. I can easily identify logs from i.e. sshd:
 > 2009-08-22T10:54:30+00:00 obelix sshd[6984]: Accepted...
Simply I use filter "sshd*" for the 3rd field (application name).

Or in case of ntpd-logs I put filter for "ntpd*":
 > 2009-08-22T10:54:48+00:00 nitino ntpd[4421]: synchronized...

But what do I have to do with this:
 > 1250867551:  *** Finished. Cleaning up...
or:
 > 1250867528: === Sync completed
How can a log-server indentify it as portage-log so that it could
save it into /var/log/$HOST/emerge.log?

> If you don't mind getting your hands dirty, portage is written mostly in
> python and bash so it shouldn't be that difficult to dig into the parts
> that handle emerge.log and emerge-fetch.log and convert them to syslog.

Well, I'm affraid this is beyond my skill. I've looked at those
scripts, but they seem to me to be really complicated...

Jarry

-- 
_______________________________________________________________
This mailbox accepts e-mails only from selected mailing-lists!
Everything else is considered to be spam and therefore deleted.

Re: [gentoo-user] problem filtering portage messages...

[Alan McKinnon]

On Saturday 22 August 2009 13:25:12 Jarry wrote:
> The first quite natural filter-rule is to split logs according
> to client-IP (or hostname). The second level is to split logs
> according to application. I do not want to have logs from kernel,
> sshd, apache, ntp, portage, ftp, dns, mixed together in one file.

The solution is quite obvious: portage does not do what you want.

So, either you or someone else, must add a feature so that it can do what you 
want. File a bug request.

-- 
alan dot mckinnon at gmail dot com

Re: [gentoo-user] problem filtering portage messages...

[Robin Atwood]

On Saturday 22 August 2009, Jarry wrote:
> Hi,
> I have a log-server (syslog-ng) collecting logs from my servers.
> Everything works for standard logs: clients forward them to server
> where they are filtered (based on facility or application) and
> splitted into a few files (for each client).
>
> The problem is with portage logs: I can not find any pattern
> I could use to filter them out. For example, this is small part
> of /var/log/emerge.log:
>
> 1250923552:  *** emerge  sync
> 1250923552:  === sync
> 1250923552: >>> Starting rsync with rsync://140.211.166.165/gentoo-portage
> 1250923665: === Sync completed with rsync://140.211.166.165/gentoo-portage
> 1250923666:  *** terminating.
> 1250923760: Started emerge on: Aug 22, 2009 06:49:20
> 1250923760:  *** emerge  depclean
> 1250923760:  >>> depclean
> ...
>
> Can I somehow force portage to write all its logs (summary.log,
> emerge-fetch.log, emerge.log) in standard syslog format RFC-3164?
> I mean with all that stuff like priority (facility), header
> (timestamp, hostname), application, etc. Even better if portage
> could write its logs to /dev/log...

This came up on the Gentoo portage forum recently. What I do is this:

1. In /etc/conf.d/local.start add a line
tail -F /var/log/emerge.log | awk '{$1="";print | "logger -t emerge -p 
local5.info"}' &

2. In /etc/syslog-ng/syslog-ng.conf add lines to the effect of:

destination emerge { file("/var/log/emerge"); };
filter f_emerge { program(emerge); };

filter f_messages { level(info..emerg) 
        and not facility(auth, authpriv, mail, news, cron, local0, local6)
				and not program(emerge); };

log { source(src); filter(f_emerge); destination(emerge); };

Restart syslog-ng and there you go. :)

HTH
-Robin
-- 
----------------------------------------------------------------------
Robin Atwood.

"Ship me somewheres east of Suez, where the best is like the worst,
 Where there ain't no Ten Commandments an' a man can raise a thirst"
         from "Mandalay" by Rudyard Kipling
----------------------------------------------------------------------

Re: [gentoo-user] problem filtering portage messages...

[Jarry]

Robin Atwood wrote:

> 1. In /etc/conf.d/local.start add a line
> tail -F /var/log/emerge.log | awk '{$1="";print | "logger -t emerge -p 
> local5.info"}' &

Thanks. First I'll try to figure out what this line actually means! :-)

Jarry

-- 
_______________________________________________________________
This mailbox accepts e-mails only from selected mailing-lists!
Everything else is considered to be spam and therefore deleted.

Re: [gentoo-user] problem filtering portage messages...

[Robin Atwood]

On Saturday 22 August 2009, Jarry wrote:
> Robin Atwood wrote:
> > 1. In /etc/conf.d/local.start add a line
> > tail -F /var/log/emerge.log | awk '{$1="";print | "logger -t emerge -p
> > local5.info"}' &
>
> Thanks. First I'll try to figure out what this line actually means! :-)

This was devised by Mike Hunt on the Portage forum (see 
http://forums.gentoo.org/viewtopic-t-788086-highlight-.html). It tails the 
default emerge log into an awk stage which deletes the Unix time stamp, and 
then logs the records into the standard syslog with the priority set to 
facility "local5" and level "info"; these can be whatever works for you.

-Robin
-- 
----------------------------------------------------------------------
Robin Atwood.

"Ship me somewheres east of Suez, where the best is like the worst,
 Where there ain't no Ten Commandments an' a man can raise a thirst"
         from "Mandalay" by Rudyard Kipling
----------------------------------------------------------------------