Simon Willison’s Weblog

Subscribe

Cloudflare Radar: AI Insights (via) Cloudflare launched this dashboard back in February, incorporating traffic analysis from Cloudflare's network along with insights from their popular 1.1.1.1 DNS service.

I found this chart particularly interesting, showing which documented AI crawlers are most active collecting training data - lead by GPTBot, ClaudeBot and Meta-ExternalAgent:

Line chart showing HTTP traffic by bot over time from August 26 to September 1. HTTP traffic by bot - HTTP request trends for top five most active AI bots. Crawl purpose: Training. GPTBot 31.7% (orange line), ClaudeBot 27.1% (blue line), Meta-ExternalAgent 25.3% (light blue line), Bytespider 9.3% (yellow-green line), Applebot 5.2% (green line). Max scale shown on y-axis. X-axis shows dates: Tue, Aug 26, Wed, Aug 27, Thu, Aug 28, Fri, Aug 29, Sat, Aug 30, Sun, Aug 31, Mon, Sep 1. Top right shows Crawl purpose dropdown set to "Training" with X and checkmark buttons.

Cloudflare's DNS data also hints at the popularity of different services. ChatGPT holds the first place, which is unsurprising - but second place is a hotly contested race between Claude and Perplexity and #4/#5/#6 is contested by GitHub Copilot, Perplexity, and Codeium/Windsurf.

Google Gemini comes in 7th, though since this is DNS based I imagine this is undercounting instances of Gemini on google.com as opposed to gemini.google.com.

Line chart showing generative AI services popularity rankings over time. Title: "Generative AI services popularity" with subtitle "Top 10 services based on 1.1.1.1 DNS resolver traffic" and question mark and share icons. Legend shows: ChatGPT/OpenAI (dark blue), Character.AI (light blue), Claude/Anthropic (orange), Perplexity (olive green), GitHub Copilot (green), Codeium/Windsurf AI (pink), Google Gemini (purple), QuillBot (red), Grok/xAI (brown), DeepSeek (yellow). Y-axis shows ranks #1-#10, X-axis shows dates from Mon, Aug 25 to Mon, Sep 1 (partially visible). ChatGPT maintains #1 position throughout. Other services show various ranking changes over the week-long period.

Posted 1st September 2025 at 5:06 pm

Recent articles

Monthly briefing

Sponsor me for $10/month and get a curated email digest of the month's most important LLM developments.

Pay me to send you less!

Sponsor & subscribe