# The intense battle to stop AI bots from taking over the internet

Artificial intelligence systems need to be trained on text – which has led their creators to gather up words from right across the web

### Your support helps us to tell the story

From reproductive rights to climate change to Big Tech, The Independent is on the ground when the story is developing. Whether it's investigating the financials of Elon Musk's pro-Trump PAC or producing our latest documentary, 'The A Word', which shines a light on the American women fighting for reproductive rights, we know how important it is to parse out the facts from the messaging.

At such a critical moment in US history, we need reporters on the ground. Your donation allows us to keep sending journalists to speak to both sides of the story.

The Independent is trusted by Americans across the entire political spectrum. And unlike many other quality news outlets, we choose not to lock Americans out of our reporting and analysis with paywalls. We believe quality journalism should be available to everyone, paid for by those who can afford it.

**Your support makes all the difference.**

A number of companies have taken major steps to stop scrapers from attempting to take their text.

It is the latest front in an ongoing and apparently escalating battle between websites that allow people to read text and the AI companies that wish to use it to build their new tools.

The rise of artificial intelligence has brought a number of companies looking to train new and smarter AI technologies. But the large language model systems that underpin many of them – such as ChatGPT – require vast amounts of text to be trained.

That has led some companies to scrape text from the web so that it can be fed into those systems for that training. That in turn has led to frustration from the owners of text-based websites, who argue not only that the companies do not have permission to use their data, but also that it is slowing down the performance of the internet.

Elon Musk, for instance, has repeatedly suggested that X, formerly Twitter, gets a huge amount of traffic from such scraping systems. X is one of many sites that have introduced strict “rate limiting” rules, which try and restrict bots from reloading its site too much – though some have suggested that has also been used to disguise problems with X’s seemingly troubled website.

Last week, Reddit introduced a host of changes that attempted to block bots from scraping its website. It said that it too would use rate limiting, as well as blocking unknown bots and instructing such systems to stay away from its website.

It noted that those rules could potentially limit other automated systems that are important for transparency, such as the Internet Archive, which saves web pages for later access. But it insisted that important tools for researchers would still have access to Reddit.

“Anyone accessing Reddit content must abide by our policies, including those in place to protect redditors. We are selective about who we work with and trust with large-scale access to Reddit content,” it said when it introduced those new rules.

Some companies have entered into deals to give AI companies access to their or their users’ data. Both OpenAI and Google have signed deals with Reddit so that they can take its users’ posts for training their artificial intelligence systems, for instance.

Others have launched legal proceedings. The *New York Times *has sued OpenAI and Microsoft over its artificial intelligence systems, arguing that it has infringed on the paper’s copyright by using its articles to train them.

Now internet infrastructure company Cloudflare has introduced a range of similar tools, and told customers that it is a way of declaring their “AIndependence”. All Cloudflare customers will get an “easy button” to “block all AI bots”, it said.

Last year, Cloudflare had introduced a change to block AI bots that “behave well”. Despite the fact that system was intended at bots that do follow the rules, Cloudflare’s customers “overwhelmingly” decide to block them, it said.

Now the company has introduced a feature that will forcefully block all known bots. It will look for fingerprints of scrapers and stop them ever visiting websites, it said.

## Join our commenting forum

Join thought-provoking conversations, follow other Independent readers and see their replies

Comments