robots.txt vs language path prefix The default robots.txt shipped with Drupal contains rules such as this:

Disallow: /user/register/

If I understand correctly, the leading slash means this really has to match the beginning of the path. E.g. a path like /xyz/user/register/ would not match the rule. (Please correct me if I'm wrong)

(For the ending slash I am not sure)

With the language path prefix option, Drupal will have urls like /fr/user/register etc.

It can be assumed that if we want to exclude /user/register, we also want to exclude /fr/user/register and other language versions.

However, a /*/user/register/ might be too generic, as it also would match other urls which we don't want to exclude.

Questions:


Will Drupal's default robots.txt also exclude language-specific versions of the targeted urls? (according to my own assumption it does not).
What would be the simplest / cleanest way to exclude all language versions for a given path?


I think this applies equally to Drupal 7 and Drupal 8.
Whatever you are unhappy with, with Drupal's default robots.txt, delete it, and install RobotsTxt to provide an UI to let your SEO person edit the robots.txt according your needs.


  Use this module when you are running multiple Drupal sites from a
  single code base (multisite) and you need a different robots.txt file
  for each one. This module generates the robots.txt file dynamically
  and gives you the chance to edit it, on a per-site basis, from the web
  UI.




Look what I found. This duplicate issue on drupal.org robots.txt: add wildcarded paths for multilingual sites leads to this open issue Fix path matching in robots.txt, which actually could need your input.

But basically some commenters seem to recommend exactly what you came up with, wildcarding the path prefix.