text:  That's Colleen Chen of UC Berkeley Law School, who teaches classes on how AI is changing the web. Over the years, the robots.txt page became something of a social contract for the entire internet. Tech giants like Google and Facebook adopted it. And even though it had no legal teeth, it was respected. Say there's a corporate or administrative page you don't want showing up on Google, put it in the file. It helped hold the entire internet together, says former Google engineer Jacob Hoffman Andrews. That system has remarkably worked well for 30 years.