title: Methods for preventing search engines from indexing irrelevant content on a page
body: <p>I'm looking for ways to prevent indexing of parts of a page. Specifically, comments on a page, since they weigh up entries a lot based on what users have written. This makes a Google search on the page return lots of irrelevant pages.</p>
  
  <p>Here are the options I'm considering so far:</p>
  
  <p><strong>1)</strong> Load comments using JavaScript to prevent search engines from seeing them.</p>
  
  <p><strong>2)</strong> Use user agent sniffing to simply not output comments for crawlers.</p>
  
  <p><strong>3)</strong> Use search engine-specific markup to hide parts of the page. This solution seems quirky at best, though. Allegedly, this can be done to prevent Yahoo! indexing specific content:</p>
  
  <pre><code>&lt;div class="robots-nocontent"&gt;
  This content will not be indexed!
  &lt;/div&gt;
  </code></pre>
  
  <p>Which is a very ugly way to do it. I read about a Google solution that looks better, but I believe it only works with Google Search Appliance (can someone confirm this?):</p>
  
  <pre><code>&lt;!--googleoff: all--&gt;
  This content will not be indexed!
  &lt;!--googleon: all--&gt;
  </code></pre>
  
  <hr>
  
  <p>Does anyone have other methods to recommend? Which of the three above would be the best way to go? Personally, I'm leaning towards #2 since while it might not work for all search engines, it's easy to target the biggest ones. And it has no side-effect on users, unless they're deliberately trying to impersonate a web crawler.</p>