Diary SiteMap RecentChanges About Contact 2012-02 Calendar

Search:

Matching Pages:

2012-02-20 Network Traffic

Last edit

Summary: I'm surprised – bots only amount to 9% of my traffic? Great! :)

Changed:

< I'm surprised -- bots only amount to 9% of my traffic? Great! :)

to

> I'm surprised -- bots are responsible for 14% of all my hits. That's better than
> [[Comments on 2009-10-15 Network Traffic|what I saw in 2009]].
> The way I have set up
my web pages, bots should not crawl "actions" (URLS containing the ##action## parameter) -- and yet 9% of them do it, and superfeedr does it most of all. I should investigate, I guess.


Back in 2009 I wondered about network traffic. Decided to take another look. Yesterday’s log file via bot-analyze:

aschroeder@thinkmo:~$ bot-analyze < logs/access.log.1 | head
    ----------------------------Bandwidth-------Hits-------Actions
                     Everybody      2177M     113317
                      All Bots       274M      16154   100%     9%
    --------------------------------------------------------------
                www.google.com     92390K       5542    34%     2%
                    yandex.com     63545K       3954    24%     9%
                           bot     33803K       2309    14%     1%
                superfeedr.com     31740K       1091     6%    63%
                  www.bing.com     14161K        525     3%     4%
                    ahrefs.com      9251K        465     2%     2%

I’m surprised – bots are responsible for 14% of all my hits. That’s better than what I saw in 2009.

The way I have set up my web pages, bots should not crawl “actions” (URLS containing the action parameter) – and yet 9% of them do it, and superfeedr does it most of all. I should investigate, I guess.

Tags: RSS RSS

Show Google +1

Comments


Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. See Info for text formatting rules. You can edit the comment page if you need to fix typos. You can subscribe to new comments by email without leaving a comment.

A carefully selected set of emoji for you to copy and paste: • ✕ ✓ ✝ ☠ ☢ ☣ ⚛ ☭ ☮ ☯ ⚒ ⚓ ⚔⚙ ★ ☆ ✨ 🌟 🐣 🐤 🐥 🐦 🐧 🐨 🐷 🐻 🐼 🐢 🐝 🐛 🐙 🐒 🐌 🐋 👑 ✊ 👊 ✌ 👋 👌 👍 👎 👏 👸 🍵 🍷 👹 👺 👻 👽 👾 👿 💀 ❤ ❦ ♥ 💔 📓 📖 📜 📝 🔒 🔓 🔔 🔥 🔨 🔪 🔫 🔮 😁 😂 😃 😄 😅 😆 😉 😊 😋 😌 😍 😏 😒 😓 😔 😖 😘 😚 😜 😝 😞 😠 😡 😢 😣 😤 😥 😨 😩 😪 😫 😭 😰 😱 😲 😳 😵 😷 😸 😹 😺 😻 😼 😽 😾 😿 🙀 🙇 🙈 🙉 🙊 🚶 🚲 🚀 🚽

To save this page you must answer this question:

Please say HELLO.