End of Free Scraping: Cloudflare Allows Blocking Data Collection for AI Agents

Website owners now have a technical tool to protect their copyrights. On September 16, 2026, Cloudflare implemented granular control over AI crawlers (bots). Site administrators can now segregate traffic: allowing search engine indexing but strictly blocking bots harvesting texts for LLM training (AI Training) or executing tasks for autonomous AI agents.

This is the technological answer to the protracted lawsuits brought by media corporations against OpenAI and Anthropic. Web traffic is no longer "free raw material." Macroeconomically, this means monetizing access to high-quality data. If labs want to train neural networks on fresh content, they will have to pay for access via official APIs or sign licensing agreements. The AI industry has lost the ability to vacuum the internet with impunity under the guise of search indexing.

Source: Cloudflare