Diary SiteMap RecentChanges About Contact 2012-02 Calendar

Search:

Matching Pages:

2012-02-20 Network Traffic

Last edit

Summary: I'm surprised – bots only amount to 9% of my traffic? Great! :)

Changed:

< I'm surprised -- bots only amount to 9% of my traffic? Great! :)

to

> I'm surprised -- bots are responsible for 14% of all my hits. That's better than
> [[Comments on 2009-10-15 Network Traffic|what I saw in 2009]].
> The way I have set up
my web pages, bots should not crawl "actions" (URLS containing the ##action## parameter) -- and yet 9% of them do it, and superfeedr does it most of all. I should investigate, I guess.


Back in 2009 I wondered about network traffic. Decided to take another look. Yesterday’s log file via bot-analyze:

aschroeder@thinkmo:~$ bot-analyze < logs/access.log.1 | head
    ----------------------------Bandwidth-------Hits-------Actions
                     Everybody      2177M     113317
                      All Bots       274M      16154   100%     9%
    --------------------------------------------------------------
                www.google.com     92390K       5542    34%     2%
                    yandex.com     63545K       3954    24%     9%
                           bot     33803K       2309    14%     1%
                superfeedr.com     31740K       1091     6%    63%
                  www.bing.com     14161K        525     3%     4%
                    ahrefs.com      9251K        465     2%     2%

I’m surprised – bots are responsible for 14% of all my hits. That’s better than what I saw in 2009.

The way I have set up my web pages, bots should not crawl “actions” (URLS containing the action parameter) – and yet 9% of them do it, and superfeedr does it most of all. I should investigate, I guess.

Tags: RSS RSS

Add Comment

Show Google +1