[gentoo-user] Rationalizing log files

[gentoo-user] Rationalizing log files

[Walter Dnes]

  On another thread, I had to dive into into /var/log/messages, and I
realized that it was not being rotated.  It's 32 megabytes+, most of
which is iptables reject messages for Facebook trackers.  What do I need
to do to get log rotation working?

/etc/logrotate.conf

########################################################################

#
# Default logrotate(8) configuration file for Gentoo Linux.
# See "man logrotate" for details.

# rotate log files weekly.
weekly
#daily

# keep 4 weeks worth of backlogs.
rotate 4

# create new (empty) log files after rotating old ones.
create

# use date as a suffix of the rotated file.
dateext

# compress rotated log files.
compress

notifempty
nomail
noolddir

# packages can drop log rotation information into this directory.
include /etc/logrotate.d

# no packages own wtmp and btmp -- we'll rotate them here.
/var/log/wtmp {
    monthly
    create 0664 root utmp
    minsize 1M
    rotate 1
}
/var/log/btmp {
    missingok
    monthly
    create 0600 root utmp
    rotate 1
}

# system-specific logs may be also be configured here.

########################################################################

  /etc/logrotate.d contains...
dcron  elog-save-summary  hibernate-script  openrc  rsyncd  syslog-ng

########################################################################

  And maybe either stop logging Facebook, or else log iptables messages
to a separate file (how is that done?).  The Facebook tracker messages
are generated by iptables rules...

-A INPUT -s 31.13.24.0/21 -j FECESBOOK
-A INPUT -s 31.13.64.0/18 -j FECESBOOK
-A INPUT -s 66.220.144.0/20 -j FECESBOOK
-A INPUT -s 69.63.176.0/20 -j FECESBOOK
-A INPUT -s 69.171.224.0/19 -j FECESBOOK
-A INPUT -s 74.119.76.0/22 -j FECESBOOK
-A INPUT -s 103.4.96.0/22 -j FECESBOOK
-A INPUT -s 173.252.64.0/18 -j FECESBOOK
-A INPUT -s 204.15.20.0/22 -j FECESBOOK

-A OUTPUT -d 31.13.24.0/21 -j FECESBOOK
-A OUTPUT -d 31.13.64.0/18 -j FECESBOOK
-A OUTPUT -d 66.220.144.0/20 -j FECESBOOK
-A OUTPUT -d 69.63.176.0/20 -j FECESBOOK
-A OUTPUT -d 69.171.224.0/19 -j FECESBOOK
-A OUTPUT -d 74.119.76.0/22 -j FECESBOOK
-A OUTPUT -d 103.4.96.0/22 -j FECESBOOK
-A OUTPUT -d 173.252.64.0/18 -j FECESBOOK
-A OUTPUT -d 204.15.20.0/22 -j FECESBOOK

-A FECESBOOK -j LOG --log-prefix "FECESBOOK:" --log-level 6
-A FECESBOOK -j REJECT --reject-with icmp-port-unreachable

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [gentoo-user] Rationalizing log files

[Dale]

Walter Dnes wrote:
>   On another thread, I had to dive into into /var/log/messages, and I
> realized that it was not being rotated.  It's 32 megabytes+, most of
> which is iptables reject messages for Facebook trackers.  What do I need
> to do to get log rotation working?
>
> /etc/logrotate.conf
>
> ########################################################################
>
> #
> # Default logrotate(8) configuration file for Gentoo Linux.
> # See "man logrotate" for details.
>
> # rotate log files weekly.
> weekly
> #daily
>
> # keep 4 weeks worth of backlogs.
> rotate 4
>
> # create new (empty) log files after rotating old ones.
> create
>
> # use date as a suffix of the rotated file.
> dateext
>
> # compress rotated log files.
> compress
>
> notifempty
> nomail
> noolddir
>
> # packages can drop log rotation information into this directory.
> include /etc/logrotate.d
>
> # no packages own wtmp and btmp -- we'll rotate them here.
> /var/log/wtmp {
>     monthly
>     create 0664 root utmp
>     minsize 1M
>     rotate 1
> }
> /var/log/btmp {
>     missingok
>     monthly
>     create 0600 root utmp
>     rotate 1
> }
>
> # system-specific logs may be also be configured here.
>
> ########################################################################
>
>   /etc/logrotate.d contains...
> dcron  elog-save-summary  hibernate-script  openrc  rsyncd  syslog-ng
>
> ########################################################################
>
>   And maybe either stop logging Facebook, or else log iptables messages
> to a separate file (how is that done?).  The Facebook tracker messages
> are generated by iptables rules...
>
> -A INPUT -s 31.13.24.0/21 -j FECESBOOK
> -A INPUT -s 31.13.64.0/18 -j FECESBOOK
> -A INPUT -s 66.220.144.0/20 -j FECESBOOK
> -A INPUT -s 69.63.176.0/20 -j FECESBOOK
> -A INPUT -s 69.171.224.0/19 -j FECESBOOK
> -A INPUT -s 74.119.76.0/22 -j FECESBOOK
> -A INPUT -s 103.4.96.0/22 -j FECESBOOK
> -A INPUT -s 173.252.64.0/18 -j FECESBOOK
> -A INPUT -s 204.15.20.0/22 -j FECESBOOK
>
> -A OUTPUT -d 31.13.24.0/21 -j FECESBOOK
> -A OUTPUT -d 31.13.64.0/18 -j FECESBOOK
> -A OUTPUT -d 66.220.144.0/20 -j FECESBOOK
> -A OUTPUT -d 69.63.176.0/20 -j FECESBOOK
> -A OUTPUT -d 69.171.224.0/19 -j FECESBOOK
> -A OUTPUT -d 74.119.76.0/22 -j FECESBOOK
> -A OUTPUT -d 103.4.96.0/22 -j FECESBOOK
> -A OUTPUT -d 173.252.64.0/18 -j FECESBOOK
> -A OUTPUT -d 204.15.20.0/22 -j FECESBOOK
>
> -A FECESBOOK -j LOG --log-prefix "FECESBOOK:" --log-level 6
> -A FECESBOOK -j REJECT --reject-with icmp-port-unreachable
>


I may be missing something but this is what I could find on my system. 


root@fireball / # cat /etc/cron.daily/logrotate
#!/bin/sh

/usr/bin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
    /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit $EXITVALUE
root@fireball / # cat /etc/logrotate.d/syslog-ng
#
# Syslog-ng logrotate snippet for Gentoo Linux
# contributed by Michael Sterrett
#

/var/log/messages {
    delaycompress
    missingok
    sharedscripts
    postrotate
        /etc/init.d/syslog-ng reload > /dev/null 2>&1 || true
    endscript
}
root@fireball / #


Basically, it's two files, that I can find anyway.  One is to run it as
a cron and the other tells it what to rotate.  If you duplicate that, it
should help.  Of course, make sure whatever cron you are using is
running as well.

Hope that helps.

Dale

:-)  :-)

Re: [gentoo-user] Rationalizing log files

[Walter Dnes]

On Thu, May 13, 2021 at 03:42:44AM -0500, Dale wrote

> Basically, it's two files, that I can find anyway.  One is to run it as
> a cron and the other tells it what to rotate.  If you duplicate that, it
> should help.  Of course, make sure whatever cron you are using is
> running as well.
> 
> Hope that helps.

  Strange.  My files match yours.  Manual rotation did not work.  I
inserted...

maxsize 8M

...into syslog-ng and ran logrotate, which finally worked.

-- 
Walter Dnes <waltdnes@waltdnes.org>
I don't run "desktop environments"; I run useful applications

Re: [gentoo-user] Rationalizing log files

[Mark Knecht]

[-- Attachment #1: Type: text/plain, Size: 773 bytes --]

On Thu, May 13, 2021 at 12:58 AM Walter Dnes <waltdnes@waltdnes.org> wrote:
>
<SNIP>
> # no packages own wtmp and btmp -- we'll rotate them here.
> /var/log/wtmp {
>     monthly
>     create 0664 root utmp
>     minsize 1M
>     rotate 1
> }
> /var/log/btmp {
>     missingok
>     monthly
>     create 0600 root utmp
>     rotate 1
> }
<SNIP>

As you reported 'roughly monthly' failures my guess would be the above two
sections

>   And maybe either stop logging Facebook, or else log iptables messages
> to a separate file (how is that done?).  The Facebook tracker messages
> are generated by iptables rules...

Don't log what you're not interested in. If your disk is getting filled up
with billions of Facebook issues then limit how much of that you track.

HTH,
Mark

[-- Attachment #2: Type: text/html, Size: 1108 bytes --]

Re: [gentoo-user] Rationalizing log files

[Mark Knecht]

[-- Attachment #1: Type: text/plain, Size: 851 bytes --]

On Thu, May 13, 2021 at 6:20 AM Mark Knecht <markknecht@gmail.com> wrote:
>
>
>
> On Thu, May 13, 2021 at 12:58 AM Walter Dnes <waltdnes@waltdnes.org>
wrote:
> >
> <SNIP>
> > # no packages own wtmp and btmp -- we'll rotate them here.
> > /var/log/wtmp {
> >     monthly
> >     create 0664 root utmp
> >     minsize 1M
> >     rotate 1
> > }
> > /var/log/btmp {
> >     missingok
> >     monthly
> >     create 0600 root utmp
> >     rotate 1
> > }
> <SNIP>
>
> As you reported 'roughly monthly' failures my guess would be the above
two sections
>

One additional thought: If the above sections are involved and if it's a
bug then you might find it faster changing the above to daily vs monthly.

If you were to try this then do them one at a time and change the rotate
number to 30 or 40 to keep the data for the month. (I think...)

Good luck,
Mark

[-- Attachment #2: Type: text/html, Size: 1317 bytes --]

Re: [gentoo-user] Rationalizing log files

[David Haller]

Hello,

On Thu, 13 May 2021, Walter Dnes wrote:
[..]
>  And maybe either stop logging Facebook, or else log iptables messages
>to a separate file (how is that done?).  The Facebook tracker messages
>are generated by iptables rules...
>
>-A INPUT -s 31.13.24.0/21 -j FECESBOOK
>-A INPUT -s 31.13.64.0/18 -j FECESBOOK
[..]
>-A OUTPUT -d 31.13.24.0/21 -j FECESBOOK
>-A OUTPUT -d 31.13.64.0/18 -j FECESBOOK

FWIW:

For one: why not filter the iptables messages into a seperate logfile?

E.g. for syslog-ng (you'll need to add the filter to other
filters/log, having them in the filters you can use those
more intuitively):

====
filter f_iptables   { facility(kern) and message("IN=") and message("OUT="); };
filter f_console    { ... and not filter(f_iptables); }
filter f_messages   { ... and not filter(f_iptables); };
filter f_warn       { ... and not filter(f_iptables); };
[..]
log { source(src); source(chroots); filter(f_messages); destination(messages); };

# Firewall (iptables) messages in one file:
destination firewall { file("/var/log/firewall" suppress(30)); };
log { source(src); source(chroots); filter(f_iptables); destination(firewall); };
====

You might be logging more specifically, so you could add more specific
filters. That's what those filters (and log-prefixes in iptables) are
for after all :)

Also add a matching logrotate entry:

====
/var/log/firewall {
    delaycompress
    missingok
    notifempty
    size +4096k
    sharedscripts
    postrotate
        /etc/init.d/syslog-ng reload > /dev/null 2>&1 || true
    endscript
}
====

or some such as /etc/logrotate.d/firewall (or however you name you
iptables-logfile.

And second, how about setting up a local dnsmasq to send all fb-crap
to NXDOMAIN on the DNS-level?

==== dnsmasq.conf or e.g. /etc/dnsmasq.d/blocklist.conf [1] ====
address=/fb.com/
address=/fb.me/
address=/facebook.net/
address=/facebook.de/
address=/facebook.fr/
address=/facebook.co.uk/
address=/facebook.com/
address=/fbcdn.net/
address=/instagram.com/
address=/instagram.de/
address=/whatsapp.de/
address=/whatsapp.com/
address=/whatsapp.net/
====

That has the effect that all (sub-)domains with those names give
NXDOMAIN, i.e. are non-existant. Compare to:

$ nslookup there.is.no.such.domain.invalid

Depending on what sites you visit, you might add more domains like
e.g. facebook.ca, facebook.mx, facebook.es or whatever fb-domains
sites that you visit include...

Just as ideas,
-dnh

[1] you'll need a matching conf-dir or conf-file directive, preferably
    at the end of the main /etc/dnsmasq.conf then, I use:
    ====
    conf-dir=/etc/dnsmasq.d,*.conf
    ====
    which includes all *.conf files from /etc/dnsmasq.d/ (and ignores
    other files there like *.conf~ or Makefile or whatnot, so you can
    be creative and e.g. generate your blocklist from a simple list of
    domains ;) E.g.:
    ==== /etc/dnsmasq.d/Makefile
    all: blocklist.conf
    blocklist.conf: blocklist.conf.in
    	sort -u $< | sed 's@\(.*\)@address=/&/@' > $@
    ====

    You get the ideas ;) (and if not: ask!)

-- 
Of course. Anything with more than 2 buttons is too complex. This includes
things with 2 or less buttons. This may include clothing.  -- Satya