[gentoo-user] {OT} Strange apache2 access_log entries

[gentoo-user] {OT} Strange apache2 access_log entries

[Grant]

Does anyone else get entries like this in their apache2 access_log:

127.0.0.1 - - [26/Sep/2007:03:10:08 -0700] "GET /" 400 470

I get a whole slew of them every day.  They always show up in batches
and each entry in a batch is logged at almost the same second.

- Grant
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] {OT} Strange apache2 access_log entries

[Albert Hopkins]

On Wed, 2007-09-26 at 13:30 -0700, Grant wrote:
> Does anyone else get entries like this in their apache2 access_log:
> 
> 127.0.0.1 - - [26/Sep/2007:03:10:08 -0700] "GET /" 400 470
> 
> I get a whole slew of them every day.  They always show up in batches
> and each entry in a batch is logged at almost the same second.
> 

Connection to/from localhost.  Do you have some process running on the
same server that's doing monitoring?  The 400 reply is even more
interesting.  I think the request should be "GET / HTTP/1.1" or similar
which is probably why it is returning a 400.



-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] {OT} Strange apache2 access_log entries

[Grant]

> > Does anyone else get entries like this in their apache2 access_log:
> >
> > 127.0.0.1 - - [26/Sep/2007:03:10:08 -0700] "GET /" 400 470
> >
> > I get a whole slew of them every day.  They always show up in batches
> > and each entry in a batch is logged at almost the same second.
> >
>
> Connection to/from localhost.  Do you have some process running on the
> same server that's doing monitoring?  The 400 reply is even more
> interesting.  I think the request should be "GET / HTTP/1.1" or similar
> which is probably why it is returning a 400.

I'm not doing any sort of monitoring like that.  What is that "470"?
I noticed the log entries always include that, at least for the last
10 days.

- Grant
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] {OT} Strange apache2 access_log entries

[Grant]

> > Does anyone else get entries like this in their apache2 access_log:
> >
> > 127.0.0.1 - - [26/Sep/2007:03:10:08 -0700] "GET /" 400 470
> >
> > I get a whole slew of them every day.  They always show up in batches
> > and each entry in a batch is logged at almost the same second.
> That make sense, since 400 means 'bad request' the culprit probably
> fails a preset number of times and then gives up.  Perhaps 127.0.0.1 is
> the setting for something in the absence of a sane configuration - in
> other words, it might be tricky to track this one down.  You'll have to
> let us know what gurific sleuthing techniques you employ to track down
> the bad guys.

What do you mean by "bad guys"?

I made a mistake in my initial post.  The 127.0.0.1 entries always
show up in ssl_access_log, not access_log.

Also, I noticed that a huge block of them always appears at the very
beginning of each day's ssl_access_log at exactly 3:10AM.

> You should perhaps use combined logging so you get more information,
> like the user agent and such.  right now you're using 'common' logging
> which has the additional disadvantage that it doesn't give you
> particularly useful information if you decide to use a statistical
> analyzer like awstats on your archive of logs from the past umpteen
> years.  The user agent might be useful for debugging purposes.

I switched ssl_access_log temporarily to the combined format, and it
was definitely working, but the 127.0.0.1 error looked exactly as it
did in common format with no extra information.

> You might also consider running tcpdump for a few hours or so, or
> something, and have it watch for that port and interface and run ps or
> something if you get output from it.   Or use iptables logging for the
> job, if you'd rather do that.

Any specific commands or even just certain parameters I should look into?

- Grant
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] {OT} Strange apache2 access_log entries

[Albert Hopkins]

On Wed, 2007-09-26 at 18:51 -0700, Grant wrote:
> I'm not doing any sort of monitoring like that.  What is that "470"?
> I noticed the log entries always include that, at least for the last
> 10 days.

470 is the size of the HTTP response (read
http://httpd.apache.org/docs/1.3/logs.html).  The client is coming from
the loopback device, i.e. the same machine as the server.  So it's
something running on that machine talking to the server.

Are you sure you know what's running on your machine?

As another poster said, change your logging format and you should get
more information.  See the above link and check your config.

-a

--
Albert W. Hopkins

-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] {OT} Strange apache2 access_log entries

[Daniel Iliev]

On Wed, 26 Sep 2007 18:51:31 -0700
Grant <emailgrant@gmail.com> wrote:

> > > Does anyone else get entries like this in their apache2
> > > access_log:
> > >
> > > 127.0.0.1 - - [26/Sep/2007:03:10:08 -0700] "GET /" 400 470
> > >
> > > I get a whole slew of them every day.  They always show up in
> > > batches and each entry in a batch is logged at almost the same
> > > second.
> > >
> >
> > Connection to/from localhost.  Do you have some process running on
> > the same server that's doing monitoring?  The 400 reply is even more
> > interesting.  I think the request should be "GET / HTTP/1.1" or
> > similar which is probably why it is returning a 400.
> 
> I'm not doing any sort of monitoring like that.  What is that "470"?
> I noticed the log entries always include that, at least for the last
> 10 days.
> 
> - Grant


To be sure, you have to check the fields defined in your log format
variable in the configuration. The date and source IP address fields are
obvious. The dashes mean "field is empty". My guess is that one of
those is for http_user. 470 most probably is the http error code (means
bad request) and the last field with value "470" would be "bytes sent". 

Again: I'm just guessing. If it was up to me, I'd  check the config
files out and do some googling to get the meaning of those fields and
their values.


-- 
Best regards,
Daniel
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] {OT} [typo] Strange apache2 access_log entries

[Daniel Iliev]

> those is for http_user. 470 most probably is the http error code
> (means bad request) and the last field with value "470" would be
> "bytes sent". 
> 

The first line should read
 "those is for http_user. 400 most probably is the http error code".
400 instead of 470. Sorry for the typo.

-- 
Best regards,
Daniel
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] {OT} Strange apache2 access_log entries

[Grant]

> > I'm not doing any sort of monitoring like that.  What is that "470"?
> > I noticed the log entries always include that, at least for the last
> > 10 days.
>
> 470 is the size of the HTTP response (read
> http://httpd.apache.org/docs/1.3/logs.html).  The client is coming from
> the loopback device, i.e. the same machine as the server.  So it's
> something running on that machine talking to the server.
>
> Are you sure you know what's running on your machine?

I sure hope so.  It's a dedicated machine.

> As another poster said, change your logging format and you should get
> more information.  See the above link and check your config.

I tried combined logging format, verified it was working, but no more
information was printed to the log file for those errors.

What else can I do to track this down?

- Grant
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] {OT} Strange apache2 access_log entries

[Szénási István]

> > Does anyone else get entries like this in their apache2 access_log:
> >
> > 127.0.0.1 - - [26/Sep/2007:03:10:08 -0700] "GET /" 400 470
> >
> > I get a whole slew of them every day.  They always show up in batches
> > and each entry in a batch is logged at almost the same second.

Have you tried the netstat -p ?
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] {OT} Strange apache2 access_log entries

[Naga]

On Wednesday 26 September 2007 22:30:10 Grant wrote:
> Does anyone else get entries like this in their apache2 access_log:
>
> 127.0.0.1 - - [26/Sep/2007:03:10:08 -0700] "GET /" 400 470

Apache-2.6 does this as a internal signaling thing (don't remember the note 
about it now).

If you change the log to include "User-Agent" you should see something like 
below:
localhost - - [30/Aug/2007:03:11:33 +0200] "GET / HTTP/1.0" 401 
467 "-" "Apache (internal dummy connection)"

-- 
Naga
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] {OT} Strange apache2 access_log entries

[Neil Bothwick]

[-- Attachment #1: Type: text/plain, Size: 607 bytes --]

On Wed, 26 Sep 2007 18:51:31 -0700, Grant wrote:

> > Connection to/from localhost.  Do you have some process running on the
> > same server that's doing monitoring?  The 400 reply is even more
> > interesting.  I think the request should be "GET / HTTP/1.1" or
> > similar which is probably why it is returning a 400.  
> 
> I'm not doing any sort of monitoring like that.  What is that "470"?
> I noticed the log entries always include that, at least for the last
> 10 days.

Was this around the time your host leaked the passwords?

-- 
Neil Bothwick

The best antiques are old friends.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [gentoo-user] {OT} Strange apache2 access_log entries

[Grant]

> > > Does anyone else get entries like this in their apache2 access_log:
> > >
> > > 127.0.0.1 - - [26/Sep/2007:03:10:08 -0700] "GET /" 400 470
> > >
> > > I get a whole slew of them every day.  They always show up in batches
> > > and each entry in a batch is logged at almost the same second.
>
> Have you tried the netstat -p ?

It looks like netstat -p only gives me a snapshot and I can't seem to
predict when these errors will show up.  Can I have it run
continuously and keep the output or something?

- Grant
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] {OT} Strange apache2 access_log entries

[Grant]

> > > Connection to/from localhost.  Do you have some process running on the
> > > same server that's doing monitoring?  The 400 reply is even more
> > > interesting.  I think the request should be "GET / HTTP/1.1" or
> > > similar which is probably why it is returning a 400.
> >
> > I'm not doing any sort of monitoring like that.  What is that "470"?
> > I noticed the log entries always include that, at least for the last
> > 10 days.
>
> Was this around the time your host leaked the passwords?
>
> --
> Neil Bothwick

Hi Neil,

No, it was happening at least 2.5 days before the host problem.

- Grant
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] {OT} Strange apache2 access_log entries

[Szénási István]

> It looks like netstat -p only gives me a snapshot and I can't seem to
> predict when these errors will show up.  Can I have it run
> continuously and keep the output or something?

You can run it continously with the -c option.
You can simple redirect the output, but if you don't want a large
file, you should filter it.

Here's 2 examples:
	netstat --protocol=inet -c > inet_log.txt
	netstat --protocol=inet -c | grep localhost > inet

The first example will show all connections with the inet protocol family.
The second will show the rows only which contains the "localhost" string.
-- 
gentoo-user@gentoo.org mailing list

Re: [gentoo-user] {OT} Strange apache2 access_log entries

[Grant]

> > Does anyone else get entries like this in their apache2 access_log:
> >
> > 127.0.0.1 - - [26/Sep/2007:03:10:08 -0700] "GET /" 400 470
>
> Apache-2.6 does this as a internal signaling thing (don't remember the note
> about it now).
>
> If you change the log to include "User-Agent" you should see something like
> below:
> localhost - - [30/Aug/2007:03:11:33 +0200] "GET / HTTP/1.0" 401
> 467 "-" "Apache (internal dummy connection)"

I actually get this with combined logging which includes the referrer and UA:

127.0.0.1 - - [27/Sep/2007:07:32:09 -0700] "GET /" 400 470 "-" "-"

- Grant
-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: {OT} Strange apache2 access_log entries

[Grant]

> Does anyone else get entries like this in their apache2 access_log:
>
> 127.0.0.1 - - [26/Sep/2007:03:10:08 -0700] "GET /" 400 470
>
> I get a whole slew of them every day.  They always show up in batches
> and each entry in a batch is logged at almost the same second.

I've got this narrowed down to the vhost listening on port 444.  If I
remove "Listen 444" the error completely disappears.  That vhost has a
pretty standard config:

Listen 444
<VirtualHost *:444>
        Include /etc/apache2/vhosts.d/default_vhost.include
        ErrorLog /var/log/apache2/ssl_error_log
        <IfModule mod_log_config.c>
                TransferLog /var/log/apache2/ssl_access_log
        </IfModule>
        SSLEngine on
        SSLCipherSuite
ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
        SSLCertificateFile /etc/apache2/ssl/www.mydomain.com.crt
        SSLCertificateKeyFile /etc/apache2/ssl/www.mydomain.com.key
        <Files ~ "\.(cgi|shtml|phtml|php?)$">
                SSLOptions +StdEnvVars
        </Files>
        <IfModule mod_setenvif.c>
                BrowserMatch ".*MSIE.*" \
                nokeepalive ssl-unclean-shutdown \
                downgrade-1.0 force-response-1.0
        </IfModule>
        <IfModule mod_log_config.c>
                CustomLog /var/log/apache2/ssl_request_log \
                "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
        </IfModule>
        <Directory "/path/to/htdocs">
                AuthType Basic
                AuthName "Administration"
                AuthUserFile /path/to/passwords
                Require user myuser
                Options None
                AllowOverride None
                Order allow,deny
                Allow from all
        </Directory>
</VirtualHost>

This may have started when I upgraded to apache-2.2.  Any ideas?

- Grant
-- 
gentoo-user@gentoo.org mailing list

[gentoo-user] Re: {OT} Strange apache2 access_log entries

[Grant]

> Does anyone else get entries like this in their apache2 access_log:
>
> 127.0.0.1 - - [26/Sep/2007:03:10:08 -0700] "GET /" 400 470
>
> I get a whole slew of them every day.  They always show up in batches
> and each entry in a batch is logged at almost the same second.
>
> - Grant

This is fully explained in this very short document:

http://wiki.apache.org/httpd/InternalDummyConnection

- Grant
-- 
gentoo-user@gentoo.org mailing list