On August 4, 2025, Cloudflare said the AI search company Perplexity used undeclared crawlers to collect content from sites that had asked not to be crawled. Cloudflare said the activity reached 3 to 6 million stealth requests a day across tens of thousands of domains. It removed Perplexity from its list of verified bots and added signatures to block the traffic.
What is going on
Most well-behaved bots identify themselves and respect robots.txt, a small text file where a site states which pages bots may fetch. It works on trust, like a sign on a shop door. Nothing physically stops someone from walking past the sign.
According to Cloudflare, when Perplexity's declared crawler was blocked, undeclared crawlers appeared instead. They presented themselves as Chrome on macOS, a normal browser, and rotated between different IP addresses and networks (ASNs) to avoid being blocked. That makes the traffic look like ordinary human visitors rather than a bot.
Who is affected
This is not a software vulnerability, so there is no version to patch. It affects any public website that wants a say in whether AI services copy its content.
Small sites are included: Cloudflare described the activity spanning tens of thousands of domains. If your only control is a robots.txt file or a simple block on a known bot name, that control can be bypassed by crawlers that do not declare themselves.
What to do now
- Keep `robots.txt` in place, but do not rely on it alone. It tells honest bots your rules and nothing more.
- Turn on your CDN or host's AI-crawler blocking rule. Cloudflare and other providers offer a free one-click option to block AI crawlers.
- Use bot management rather than name-based blocking. Blocking a user agent string is easy to evade; behaviour-based detection is not.
- Check your logs for odd patterns. Many requests from changing networks, all claiming to be Chrome on macOS, hitting page after page, are worth a look.
- Decide what you actually want. Some owners welcome AI traffic for visibility. Make it a choice rather than a default.
How to check your own site
Crawler control is one layer; the basics underneath still matter. The free Website Security Check looks at your site from the outside and flags outdated CMS versions, known-vulnerable plugins, missing security headers, weak TLS and email-authentication gaps, so you can see what a passing bot or scanner sees.
Source: Cloudflare blog, August 2025.
Comments
No comments yet.