Let’s face it—robots.txt wasn’t designed for the age of AI crawlers, which are ravenously consuming content across the web. For creators, it’s tough to swallow that their hard work is being used, often for free, to train AI models.

Cloudflare’s latest feature now allows websites to block AI models or bots with a simple click. If you’ve ever had to prove you’re human before accessing a site, that’s part of the toolkit Cloudflare is offering to help publishers stop the constant battle of restricting access.

While this might be a win for creators in the short term, there’s a lingering question: Will limiting access to AI crawlers make it harder for your content to be found in AI-powered answer engines like Perplexity AI? Only time will tell, but for now, the choice is yours.

Cloudflare moves to end free, endless AI scraping with one-click blocking

Cloudflare may charge an app store-like fee for its AI-scraping data marketplace.