The recent news surrounding the AI industry has been rather depressing, and perhaps dystopic to some degree. Microsoft’s head of AI infamously claimed that all data is for AI to use, while other AI companies sneak past rules to scrape for data in order to feed their generative AI models. Today, AI companies have pretty much collectively waged a war against creators – but there’s a solution.
Cloudflare Comes To The Rescue
Cloudflare is one of the largest provider of CDNs (content delivery networks) and various internet infrastructures on the planet, which many websites rely on. It recently announced a new toggle switch designed to block all AI bots, and the feature is now available to all Cloudflare customers, including free tier users. The company cited data from its customers that a significant majority of users (85.2% of them) chose to block AI scrapers and crawlers from siphoning their content for AI training.
For website owners, all it takes is one switch via the Cloudflare dashboard and their website should be protected from AI bots. In its blog, Cloudflare says it detects fingerprints and uses machine learning to identify offenders, and there are lots of them, as the graph below denotes. Some of the most active bots include Bytespider, which is owned by TikTok parent company ByteDance and is reportedly used to train its LLMs; GPTBot is another big offender, which powers OpenAI’s ChatGPT.
On top of that, Cloudflare is offering an option for its customers to report any suspicious bot activity should they manage to escape the detection. “We fear that some AI companies intent on circumventing rules to access content will persistently adapt to evade bot detection,” the company wrote in its blog post, as it assures to “keep the Internet a place where content creators can thrive.”
Pokdepinion: AI companies has gotten away with a lot of things lately, hopefully this acts as a deterrent to them.
