2009-06-12 Referrers

I wrote a little Perl script that parses Apache log files and counts referrers. It’s pretty specific for this site, since it tries to determine for both site and referrer whether the log entry is about Emacs Wiki, my blog, Oddmuse, etc.

The numbers for my blog in yesterday’s 24h period: 75307, which is 5% of the total number of hits in the log file. Of these, 50276 hits had no referrer information, 13196 came from my own site.

Now, to understand this, one needs to understand that most of the traffic on the web is caused by search engines. At least for my site!

aschroeder@thinkmo:~$ egrep 'GET /(cgi-bin/)?alex' logs/access.log.1 | wc -l
aschroeder@thinkmo:~$ egrep 'GET /(cgi-bin/)?alex.*google' logs/access.log.1 | wc -l

Fully 24% of all my traffic is Google related!

But now, some numbers: I still have 75307-50276-13196=11835 hits to explain! :)

And many other small hits.



I am probably responsible for some of the Bloglines.com stuff, as I use that particular feed aggregator.

– Adrian 2009-06-14 19:24 UTC

Ah! :) I switched from Bloglines to Google Reader a few weeks ago and was surprised other people still use it.

AlexSchroeder 2009-06-14 22:26 UTC

Yeah, I am slow with the technology. Google Reader does make some sense, though, since I am already using Gmail and Google Calendar...

– Adrian 2009-06-16 13:19 UTC

This is interesting. Thanks.

I’m surprised how much syndication directs traffic. I actually thought search engines would be the big draw to your site. Although this confirms that if you write a good blog and even if it has the syndication technology – a feed – doesn’t mean the visitors will find you. You can only get traffic if you advertise your blog by adding it to to a collective feed (an aggregator called a “planet”) or by commenting on other blogs.

AaronHawley 2009-06-17 16:24 UTC

Plus it makes me wonder whether 25% of all the hits and more for search engines is worth the price since I’m getting relatively few referrers. One would have to figure out a “conversion rate” for visitors from Google. My guess that I should introduce some sort of rate-limit for bots. What do you think?

AlexSchroeder 2009-06-17 22:24 UTC

I am always spellbound how little time it takes Google to find a new page and have it show up in search results. In my mind, I imagine that Google’s bot cluster is crawling all of the Web all of the time. With the development of blogging and their news media interface, I predict they are constantly pinging pages to find changes and new pages.

Yeah, if only you could rate-limit a bot with the ratio of traffic their search engine brings you? :)

AaronHawley 2009-06-18 17:36 UTC

Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. Note: in order to facilitate peer review and fight vandalism, we will store your IP number for a number of days. See Privacy Policy for more information. See Info for text formatting rules. You can edit the comment page if you need to fix typos. You can subscribe to new comments by email without leaving a comment.

To save this page you must answer this question:

Please say HELLO.