User-agent: *
Disallow: /*/print$
# Don't allow indexing of search results
Disallow: /search?
# We only allow indexing of the licence-finder landing page
Disallow: /licence-finder/
Allow: /licence-finder
# We only allow indexing of the finance finder landing page
Disallow: /business-finance-support-finder/*
Allow: /business-finance-support-finder
Disallow: /apply-for-a-licence
# Don't allow indexing of experimental performance platform dashboards
Disallow: /performance/experimental/*
# Don't allow indexing of user needs pages
Disallow: /info/*
Sitemap: https://www.gov.uk/sitemap.xml
Crawl-delay: 0.5

user-agent: AhrefsBot
disallow: /

User-agent: *
Disallow: /*/print$
# Don't allow indexing of search results
Disallow: /search?
# We only allow indexing of the licence-finder landing page
Disallow: /licence-finder/
Allow: /licence-finder
# We only allow indexing of the finance finder landing page
Disallow: /business-finance-support-finder/*
Allow: /business-finance-support-finder
Disallow: /apply-for-a-licence
# Don't allow indexing of experimental performance platform dashboards
Disallow: /performance/experimental/*
# Don't allow indexing of user needs pages
Disallow: /info/*
Sitemap: https://www.gov.uk/sitemap.xml
Crawl-delay: 0.5

# https://ahrefs.com/robot/ crawls the site frequently
User-agent: AhrefsBot
Crawl-delay: 10

Allow AhrefsBot to crawl very slowly
Allow AhrefsBot to crawl very slowly

@bradleywright [noted][1] in #408 that it would be nice if we tried
to remain mostly open.

This commit sets a long Crawl-delay which will allow the bot to crawl
us without impacting the site for actual users.

If this causes problems for users we should revert it.

[1]: https://github.com/alphagov/static/pull/408#issuecomment-44185309
