title: Detecting well behaved / well known bots
body: <p>I found this question very interesting : <a href="https://stackoverflow.com/questions/341695/programmatic-bot-detection">Programmatic Bot Detection</a>
  I have a very similar question, but I'm not bothered about 'badly behaved bots'.</p>
  
  <p>I am tracking (in addition to google analytics) the following per visit :</p>
  
  <ul>
  <li>Entry URL</li>
  <li>Referer</li>
  <li>UserAgent</li>
  <li>Adwords (by means of query string)</li>
  <li>Whether or not the user made a purchase</li>
  <li>etc.</li>
  </ul>
  
  <p>The problem is that to calculate any kind of conversion rate I'm ending up with lots of 'bot' visits that are greatly skewing my results.</p>
  
  <p>I'd like to ignore as many as possible bot visits, but I want a solution that I don't need to monitor too closely, and that won't in itself be a performance hog and preferably still work if someone has javascript disabled.</p>
  
  <p>Are there good published lists of the top 100 bots or so? I did find a list at <a href="http://www.user-agents.org/" rel="nofollow noreferrer">http://www.user-agents.org/</a> but that appears to contain hundreds if not thousands of bots. I don't want to check every referer against thousands of links.</p>
  
  <p>Here is the current googlebot UserAgent. How often does it change?</p>
  
  <pre><code> Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  </code></pre>