<p>Say I have a site on <a href="http://example.com" rel="noreferrer">http://example.com</a>. I would really like allowing bots to see the home page, but any other page need to blocked as it is pointless to spider. In other words</p>

<p><a href="http://example.com" rel="noreferrer">http://example.com</a> &amp; <a href="http://example.com/" rel="noreferrer">http://example.com/</a> should be allowed, but 
<a href="http://example.com/anything" rel="noreferrer">http://example.com/anything</a> and <a href="http://example.com/someendpoint.aspx" rel="noreferrer">http://example.com/someendpoint.aspx</a> should be blocked.</p>

<p>Further it would be great if I can allow certain query strings to passthrough to the home page:
<a href="http://example.com?okparam=true" rel="noreferrer">http://example.com?okparam=true</a> </p>

<p>but not
<a href="http://example.com?anythingbutokparam=true" rel="noreferrer">http://example.com?anythingbutokparam=true</a></p>

[{'answer_id': 43436, 'author': 'Biri', 'author_id': 968, 'author_profile': 'https://Stackoverflow.com/users/968', 'pm_score': 0, 'selected': False, 'text': "<p>Basic robots.txt:</p>\n\n<pre><code>Disallow: /subdir/\n</code></pre>\n\n<p>I don't think that you can create an expression saying 'everything but the root', you have to fill in all sub directories.</p>\n\n<p>The query string limitation is also not possible from robots.txt. You have to do it in the background code (the processing part), or maybe with server rewrite-rules.</p>\n"}, {'answer_id': 43454, 'author': 'UnkwnTech', 'author_id': 115, 'author_profile': 'https://Stackoverflow.com/users/115', 'pm_score': 0, 'selected': False, 'text': '<pre><code>Disallow: *\nAllow: index.ext\n</code></pre>\n\n<p>If I remember correctly the second clause should override the first.</p>\n'}, {'answer_id': 43838, 'author': 'ceejayoz', 'author_id': 1902010, 'author_profile': 'https://Stackoverflow.com/users/1902010', 'pm_score': 1, 'selected': False, 'text': '<p><a href="http://google.com/webmasters/tools" rel="nofollow noreferrer">Google\'s Webmaster Tools</a> report that disallow always takes precedence over allow, so there\'s no easy way of doing this in a <code>robots.txt</code> file.</p>\n\n<p>You could accomplish this by puting a <code>noindex,nofollow</code> <code>META</code> tag in the HTML every page but the home page.</p>\n'}, {'answer_id': 43848, 'author': 'hakan', 'author_id': 3993, 'author_profile': 'https://Stackoverflow.com/users/3993', 'pm_score': 0, 'selected': False, 'text': '<p>As far as I know, not all the crawlers support Allow tag. One possible solution might be putting everything except the home page into another folder and disallowing that folder.</p>\n'}, {'answer_id': 44711, 'author': 'Boaz', 'author_id': 2892, 'author_profile': 'https://Stackoverflow.com/users/2892', 'pm_score': 7, 'selected': True, 'text': '<p>So after some research, here is what I found  - a solution acceptable by the major search providers: <a href="http://www.google.com/support/webmasters/bin/answer.py?answer=40367" rel="noreferrer">google</a> , <a href="http://help.yahoo.com/l/us/yahoo/search/webcrawler/slurp-02.html" rel="noreferrer">yahoo</a> &amp; msn (I could on find a validator here) :</p>\n\n<pre><code>User-Agent: *\nDisallow: /*\nAllow: /?okparam=\nAllow: /$\n</code></pre>\n\n<p>The trick is using the $ to mark the end of URL.</p>\n'}]
['https://Stackoverflow.com/questions/43427', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/2892/']