Diary SiteMap RecentChanges About Contact 2009-10 Calendar

Search:

Matching Pages:

2009-10-15 Network Traffic

I have this little script called leech-detector that I run on log files every now and then:

aschroeder@thinkmo:~$ leech-detector < logs/access.log | head
       66.249.65.101       2654    11K   9%    8.7s  200 (86%), 404 (7%), 302 (2%), 304 (2%), 301 (0%), 501 (0%)
        72.30.142.82       1055    14K   3%   22.0s  200 (91%), 304 (2%), 404 (2%), 301 (2%), 302 (0%)
      87.250.253.241        979    20K   3%   23.7s  200 (96%), 404 (1%), 301 (1%)
      67.218.116.134        868    12K   2%   26.7s  200 (85%), 301 (9%), 404 (3%), 302 (1%), 403 (0%)
       72.14.199.225        413    15K   1%   56.1s  200 (50%), 304 (41%), 301 (7%)
      67.195.110.173        293    18K   1%   78.6s  200 (76%), 304 (7%), 301 (7%), 404 (6%), 302 (2%)

So, what do we have? Using whois on them:

66.249.65.101
Google (“GOGL”)
72.30.142.82
Inktomi (“INKT” – Yahoo, I think)
87.250.253.241
Yandex (“YANDEX-252” a Russian search engine?)
67.218.116.134
Layer42 (“LAYER-1” – huh? a “web solution provider” – maybe a caching proxy for their network?)
72.14.199.225
Google (“GOGL” again)
67.195.110.173
Yahoo (“YHOO”)

I guess that assuming 20% search engine traffic is still a good estimate.

Actually, when grouping by user agent instead of the IP number using bot-analyze, we get an even higher number. Apparently the Microsoft bot is distributed over various IP numbers so that it doesn’t show up in the list above:

aschroeder@thinkmo:~$ bot-analyze < logs/access.log | head
    ---------------------------------Hits-------Actions
                         Total      29979   100%     0%
    ---------------------------------------------------
                        msnbot       7437    24%     1%
                     Googlebot       2817     9%     0%
                         robot        950     3%     3%
                        Exabot        137     0%     2%
                        robots         49     0%     0%
                      qihoobot         26     0%    30%
                        MSNBOT         22     0%     0%

That’s a full 33% of my hits for search engines. Then again, I pay for bandwidth and not hits. Should I rewrite this to show bandwidth?

I wonder how bigger sites handle it. Perhaps I could decide that 3% hits is enough, and therefore serve a 503 Service Unavailable response for 10 in 11 bot requests? In order to implement this, my script would still have to start up, however, eating CPU cycles and disc access.

Or is 33% a sign of bad design decisions I made? Am I sending the bots around in loops, changing the URLs all the time, making them think they have discovered new pages they have not indexed, yet?

Tags:

Read Comments (14)

Show Google +1