<p>Let's say I have a web site for hosting community generated content that targets a very specific set of users.  Now, let's say in the interest of fostering a better community I have an off-topic area where community members can post or talk about anything they want, regardless of the site's main theme.</p>

<p>Now, I <em>want</em> most of the content to get indexed by Google.  The notable exception is the off-topic content.  Each thread has it's own page, but all the threads are listed in the same folder so I can't just exclude search engines from a folder somewhere.  It has to be per-page.  A traditional robots.txt file would get huge, so how else could I accomplish this?</p>

[{'answer_id': 43980, 'author': 'James Marshall', 'author_id': 1025, 'author_profile': 'https://Stackoverflow.com/users/1025', 'pm_score': 2, 'selected': False, 'text': '<p>If using Apache I\'d use mod-rewrite to alias robots.txt to a script that could dynamically generate the necessary content.</p>\n\n<p>Edit: If using IIS you could use <a href="http://www.isapirewrite.com/" rel="nofollow noreferrer">ISAPIrewrite</a> to do the same.</p>\n'}, {'answer_id': 43983, 'author': 'UnkwnTech', 'author_id': 115, 'author_profile': 'https://Stackoverflow.com/users/115', 'pm_score': 6, 'selected': True, 'text': '<p>This will work for all well-behaving search engines, just add it to the <code>&lt;head&gt;</code>:</p>\n\n<pre><code>&lt;meta name="robots" content="noindex, nofollow" /&gt;\n</code></pre>\n'}, {'answer_id': 43986, 'author': 'Ian Nelson', 'author_id': 2084, 'author_profile': 'https://Stackoverflow.com/users/2084', 'pm_score': 0, 'selected': False, 'text': "<p>Simlarly to @James Marshall's suggestion - in ASP.NET you could use an HttpHandler to redirect calls to robots.txt to a script which generated the content.</p>\n"}, {'answer_id': 16476362, 'author': 'Dev Technosys', 'author_id': 2368896, 'author_profile': 'https://Stackoverflow.com/users/2368896', 'pm_score': -1, 'selected': False, 'text': '<p>You can disallow search engines to read or index your content by restricting robot meta tags. In this way, spider will consider your instructions and will index only such pages that you want.</p>\n'}, {'answer_id': 19090271, 'author': 'Ajay Prasad', 'author_id': 2768704, 'author_profile': 'https://Stackoverflow.com/users/2768704', 'pm_score': 1, 'selected': False, 'text': '<p>You can implement it by substituting robots.txt with dynamic script generating the output. \nWith Apache You could make simple .htaccess rule to acheive that.</p>\n\n<pre><code>RewriteRule  ^robots\\.txt$ /robots.php [NC,L]\n</code></pre>\n'}, {'answer_id': 21626326, 'author': 'Manchun', 'author_id': 3283431, 'author_profile': 'https://Stackoverflow.com/users/3283431', 'pm_score': -1, 'selected': False, 'text': '<p>block dynamic webpage by robots.txt  use this code </p>\n\n<hr>\n\n<p>User-agent: *</p>\n\n<p>Disallow: /setnewsprefs?</p>\n\n<p>Disallow: /index.html?</p>\n\n<p>Disallow: /?</p>\n\n<p>Allow: /?hl=</p>\n\n<p>Disallow: /?hl=*&amp;</p>\n'}, {'answer_id': 23827101, 'author': 'rational-space', 'author_id': 2364957, 'author_profile': 'https://Stackoverflow.com/users/2364957', 'pm_score': 0, 'selected': False, 'text': '<p>Just for that thread , make sure your head contains a noindex meta tag. Thats one more way to tell search engines not to crawl your page other than blocking in robots.txt</p>\n'}, {'answer_id': 24505953, 'author': 'SimilarWeb API', 'author_id': 3742357, 'author_profile': 'https://Stackoverflow.com/users/3742357', 'pm_score': 0, 'selected': False, 'text': '<p>Just keep in mind that a robots.txt disallow will NOT prevent Google from indexing pages that have links from external sites, all it does is prevent crawling internally. See <a href="http://www.webmasterworld.com/google/4490125.htm" rel="nofollow">http://www.webmasterworld.com/google/4490125.htm</a> or <a href="http://www.stonetemple.com/articles/interview-matt-cutts.shtml" rel="nofollow">http://www.stonetemple.com/articles/interview-matt-cutts.shtml</a>. </p>\n'}]
['https://Stackoverflow.com/questions/43971', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/3043/']