Re: [gentoo-user] Re: Dealing with scrapers - Help!

[Grant <emailgrant@gmail.com>]

>> > My apache web server has been very slow lately and webalizer charts
>> > show page accesses at 5x normal with other stats normal.  I'm thinking
>> > scrapers?  How do you guys deal with this?  Do you identify the IP
>> > (how?) and ban it (how?)?
>> >
>> > - Grant
>>
>> I used netstat to identify the IP and I see that I can use it with
>> "deny from" in httpd.conf.  It seems to be over now, but this type of
>> thing happens periodically.  How can I be alerted to this type of
>> situation when it starts so I can block the IP right away?
>
> You will need to configure quotas probably using something like:
>
> http://www.howtoforge.com/mod_cband_apache2_bandwidth_quota_throttling
>
> Not sure if it is possible to differentiate between rogue and legit clients,
> other than by checking your logs to see what was blocked.

Turns out it was a "legit" bot.  Watch out for this one:

Mozilla/5.0 (compatible; discobot/1.0;
+http://discoveryengine.com/discobot.html)

It's bad that a single IP can bring down my http isn't it?

- Grant