text:  So developers came up with a solution, a hidden plain text file in the backend software code of every website. It was intended for bots. It became known as robots.txt. And a robot.txt file then kind of puts a sign in front of that website to say, if you're a robot, sort of this visitor, you need to abide by the rules here. This is where you are or aren't welcome. This is what you can and can't do.