[gentoo-user] Dealing with scrapers - Help!

[gentoo-user] Dealing with scrapers - Help!

[Grant]

My apache web server has been very slow lately and webalizer charts
show page accesses at 5x normal with other stats normal.  I'm thinking
scrapers?  How do you guys deal with this?  Do you identify the IP
(how?) and ban it (how?)?

- Grant

[gentoo-user] Re: Dealing with scrapers - Help!

[Grant]

> My apache web server has been very slow lately and webalizer charts
> show page accesses at 5x normal with other stats normal.  I'm thinking
> scrapers?  How do you guys deal with this?  Do you identify the IP
> (how?) and ban it (how?)?
>
> - Grant

I used netstat to identify the IP and I see that I can use it with
"deny from" in httpd.conf.  It seems to be over now, but this type of
thing happens periodically.  How can I be alerted to this type of
situation when it starts so I can block the IP right away?

- Grant

Re: [gentoo-user] Re: Dealing with scrapers - Help!

[Mick]

[-- Attachment #1: Type: text/plain, Size: 900 bytes --]

On Saturday 09 August 2008, Grant wrote:
> > My apache web server has been very slow lately and webalizer charts
> > show page accesses at 5x normal with other stats normal.  I'm thinking
> > scrapers?  How do you guys deal with this?  Do you identify the IP
> > (how?) and ban it (how?)?
> >
> > - Grant
>
> I used netstat to identify the IP and I see that I can use it with
> "deny from" in httpd.conf.  It seems to be over now, but this type of
> thing happens periodically.  How can I be alerted to this type of
> situation when it starts so I can block the IP right away?

You will need to configure quotas probably using something like:

http://www.howtoforge.com/mod_cband_apache2_bandwidth_quota_throttling

Not sure if it is possible to differentiate between rogue and legit clients, 
other than by checking your logs to see what was blocked.

HTH.
-- 
Regards,
Mick

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [gentoo-user] Re: Dealing with scrapers - Help!

[Grant]

>> > My apache web server has been very slow lately and webalizer charts
>> > show page accesses at 5x normal with other stats normal.  I'm thinking
>> > scrapers?  How do you guys deal with this?  Do you identify the IP
>> > (how?) and ban it (how?)?
>> >
>> > - Grant
>>
>> I used netstat to identify the IP and I see that I can use it with
>> "deny from" in httpd.conf.  It seems to be over now, but this type of
>> thing happens periodically.  How can I be alerted to this type of
>> situation when it starts so I can block the IP right away?
>
> You will need to configure quotas probably using something like:
>
> http://www.howtoforge.com/mod_cband_apache2_bandwidth_quota_throttling
>
> Not sure if it is possible to differentiate between rogue and legit clients,
> other than by checking your logs to see what was blocked.

Turns out it was a "legit" bot.  Watch out for this one:

Mozilla/5.0 (compatible; discobot/1.0;
+http://discoveryengine.com/discobot.html)

It's bad that a single IP can bring down my http isn't it?

- Grant