robots.txt encoding issue I'm in charge of a website for my company, which runs Drupal 7.66 (PHP 5.3.3), for a few years now. Since a few months, I discovered a strange behavior regarding the robots.txt file. In some case, the encoding is totally messed up.

Here is a brief description of the problem:


If I am connected to the website: no problem, the robots.txt file is well displayed in the browser
If I am NOT connected to the website (like all our users and the search engine robots): it randomly appears broken, and I identify that as an encoding issue. By “randomly”, I mean that sometimes when I check the robots.txt while not being connected, its ok, but sometimes it's broken. I can't find a pattern that will allow to reproduce the error.


The robots.txt is managed through admin/config/search/robotstxt thanks to the module RobotsTxt version 7.x-1.2

Here is a screenshot to illustrate the issue (I had to hide some parts of the images):



Of course, the webmarketing team noticed me that Google can't crawl the robots.txt file because of that issue:



I made quite a lot of searchs on drupal.org, Stackoverflow, Google... but couldn't find anything related.
I am not a developer, but I work with a guy which is specialized in Drupal and he can not figure it out either.

Any idea? Of course if you need any further information, please let me know.

Thank you.
Problem solved.

Thanks to @hansfn comment, I found a similar issue on drupal.org: Doesn't work for anonymous users

I told about it to the developer I work with and he updated the RobotsTxt module from the version 7.x-1.2 to the version 7.x-1.4, and it solved the problem.