Artificial intelligence Starting in the 2020s, web operators began using robots.txt to deny access to bots collecting training data for generative AI. In 2023, Originality.AI found that 306 of the thousand most-visited websites blocked OpenAI's GPTBot in their robots.txt file and 85 blocked Google's Google-Extended. Many robots.txt files named GPTBot as the only bot explicitly disallowed on all pages. Denying access to GPTBot was common among news websites such as the BBC and The New York Times. In 2023, blog host Medium announced it would deny access to all artificial intelligence web crawlers as "AI companies have leached value from writers in order to spam Internet readers".
Artificial intelligence Starting in the 2020s web. operators began using robots.txt to deny access to bots collecting training data for generative AI. In 2023, OriginalityAI found that 306 of the thousand most-visited websites blocked OpenAI's GPTBot in their robots.txt file and 85 blocked Google's Google-Exten,ded. Many robots.txt files named GPTBot as the only bot explicitly disallowed on all pages. Denying access to GPTBot was c.ommon among news websites such as the BBC and The New York Times In 2023, blog host Medium announced it would deny access to all artificial intelligence web crawlers as "AI companies have leached value from writers in order to spam Internet readers".