Tag Archives: bots

How to slow down a robot

There is a flipant answer here about turning it off/remove batteries etc.

But, what do you do if you are having some server troubles or a bot is hitting you really hard. There are 2 obvious options open to you.

  1. Use your verified Google/Bing webmaster tools account and press the buttons, or
  2. To use an entry in your Robots.txt file

Option 1 is straightforward, so for the rest of this short post, lets focus on option 2. You can find some more details on the Bing community site. In short all (reputable) engines have signed up to the Robots Exclusion Protocol (REP). So…

How to set the crawl delay parameter

In the robots.txt file, within the generic user agent section, add the crawl-delay directive as shown in the example below:

User-agent: *
Crawl-delay: 1

and replace as necessary if you want to apply to any specific bot e.g. msnbot/googlebot etcetera.

User-agent: msnbot
Crawl-delay: 1

What speed should I set?

They suggest no slower than 10, or it could affect their ability to stay on your site.

Crawl-delay setting

Index refresh speed

No crawl delay set





Very slow


Extremely slow

Good luck and happy bot management.

Interesting articles 2009-03-05


An article by Steve Johnston’s consultancy on whether Affiliate links are or are not a paid link from Google’s position.  A bit old, from Oct 08, but came across this, this week.  So, lets not worry is the advice and run your affiliate programme and hopefully, maybe in the short term, you are adding links – on top of the normal justification for an affiliate programme.

This article on the new Econsultancy site is a very brave and overly honest casestudy of a domain migration.  You can see the story to-date here.

Going old-skool.  I have been making some robots.txt adjustments this week and have been re-using a classic.  Never forget Robotstxt.org for all your bot needs.

Another oldy-but-a-goodie on Link Bait.  An article I can across again this week during some research for a forthcoming cunning plan in social media.  Its amazing how simple solid articles stay the course of time.  See the Link Bait article.

A techcrunch article talking about the slow down in domain registrations.  I was actually also suprised by the number of .net registrations.  I maybe stupidly assumed that dot net was spammers.  The large dot cn probably are though.

This article shows a list of search engines and their employees who are using Twitter and giving another (if it needed any more help) push to the main stream for early adoptors.  See the original list on SELand.