What are the implications of duplicate content issues with Drupal comments? Say we have a node called node/123 with comments enabled. When comments are made, additional URLs are generated /comment/1, /comment/2 ... and /comment/reply/2/1 etc. Unfortunately, these new URLs are a page that contains a full copy of the content in the original page /node/123 or the comment in reply. Core's robots.txt only has Disallow: /comment/reply/

Would there be any implications to adding the following to the robots.txt file? Would this be recommended (or even post a core issue pointing this problem out?)

# Paths (clean URLs)
Disallow: /comment/
Disallow: /comment*/
Disallow: /comment/reply/
Disallow: /comment/reply*/
...
# Paths (no clean URLs)
Disallow: /?q=comment/
Disallow: /?q=comment*/
Disallow: /?q=comment/reply/
Disallow: /?q=comment/reply*/
After investigating, I don't think there's an SEO problem here. It's an annoyance for sure but it's not a problem and here's why:

As per Google's recommendations here http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html on canonical URLs and duplicate content, you will not have any problems if you have duplicate content only if you properly specify your canonical URLs. Also, Google (and most probably all other search engines) will only display the canonical URL in search results and not the other URLs.

In Drupal's case, the canonical URLs are properly set. For example, I have created a test article (node/2) and added comments and replies to it and got the following pages:


/node/2
/comment/2
/comment/3


After inspecting their generated source code, they all have the same &lt;link rel="canonical" href="/node/2" /&gt; in their source code properly created. So basically only the main article page will display in the search results.

Now, whether to modify core to only generate a single page or to have the robots.txt file patched as you specified is beyond me. I'm sure there will be lots of considerations and special cases to consider before implementing any of the above and since there really isn't a problem maybe it's best to keep things as is.

Cheers!

Note: I'm using Drupal 7.17 (latest version as of today).