2018-02-08 Write Bots Carefully

Today was one of those days. Load climbing up to 40 because of some idiot writing a bot following every single link on Emacs Wiki. Don’t do that! Be mindful all the pitfalls out there:

  1. robots.txt for disallowed URLs, crawl delay, and more
  2. meta tags for robots in the HTML you’re getting in order to decide whether to follow any links or not
  3. using common sense
  4. keeping an eye on the logs

Don’t be an idiot.

Munin graph showing load

Tags:

Comments


Please make sure you contribute only your own work, or work licensed under the GNU Free Documentation License. Note: in order to facilitate peer review and fight vandalism, we will store your IP number for a number of days. See Privacy for more information. See Info for text formatting rules. You can edit the comment page if you need to fix typos. You can subscribe to new comments by email without leaving a comment.

To save this page you must answer this question:

Please say HELLO.