I have this little script called leech-detector that I run on log files every now and then:
aschroeder@thinkmo:~$ leech-detector < logs/access.log | head
66.249.65.101 2654 11K 9% 8.7s 200 (86%), 404 (7%), 302 (2%), 304 (2%), 301 (0%), 501 (0%)
72.30.142.82 1055 14K 3% 22.0s 200 (91%), 304 (2%), 404 (2%), 301 (2%), 302 (0%)
87.250.253.241 979 20K 3% 23.7s 200 (96%), 404 (1%), 301 (1%)
67.218.116.134 868 12K 2% 26.7s 200 (85%), 301 (9%), 404 (3%), 302 (1%), 403 (0%)
72.14.199.225 413 15K 1% 56.1s 200 (50%), 304 (41%), 301 (7%)
67.195.110.173 293 18K 1% 78.6s 200 (76%), 304 (7%), 301 (7%), 404 (6%), 302 (2%)So, what do we have? Using whois on them:
I guess that assuming 20% search engine traffic is still a good estimate.
Actually, when grouping by user agent instead of the IP number using bot-analyze, we get an even higher number. Apparently the Microsoft bot is distributed over various IP numbers so that it doesn’t show up in the list above:
aschroeder@thinkmo:~$ bot-analyze < logs/access.log | head
---------------------------------Hits-------Actions
Total 29979 100% 0%
---------------------------------------------------
msnbot 7437 24% 1%
Googlebot 2817 9% 0%
robot 950 3% 3%
Exabot 137 0% 2%
robots 49 0% 0%
qihoobot 26 0% 30%
MSNBOT 22 0% 0%That’s a full 33% of my hits for search engines. Then again, I pay for bandwidth and not hits. Should I rewrite this to show bandwidth?
I wonder how bigger sites handle it. Perhaps I could decide that 3% hits is enough, and therefore serve a 503 Service Unavailable response for 10 in 11 bot requests? In order to implement this, my script would still have to start up, however, eating CPU cycles and disc access.
Or is 33% a sign of bad design decisions I made? Am I sending the bots around in loops, changing the URLs all the time, making them think they have discovered new pages they have not indexed, yet?