Unable to manage robots.txt in Craft I would like my robots.txt file to be generated by Craft. Things that I have tried that don't work:


a craft/templates/robots.txt template
a craft/templates/robots.txt.twig template
a route for robots.txt that points to a robots.twig template


Both of these work fine for a foo.txt file. The error I get is 404,  not the Craft 404, but the system one - nginx.
You can simply create a 'robots.txt' file in your templates folder and populate it using twig (templates do not need to end in .twig). Not sure what you're trying to control exactly, but something like this should technically work.

User-agent: *
{% set entries = craft.entries.limit(null) %} 
{% for entry in entries %}
    Disallow: {{ entry.uri }}
{% endfor %}


Terrible example but you get the idea. Then just delete the robots.txt file that is in your public folder.