Simon Willison’s Weblog

On Search: Robots. Everything you need to know about writing a tool to crawl the web

Posted 8th December 2003 at 7:28 pm

Recent articles

An Introduction to Google’s Approach to AI Agent Security - 15th June 2025
Design Patterns for Securing LLM Agents against Prompt Injections - 13th June 2025
Comma v0.1 1T and 2T - 7B LLMs trained on openly licensed text - 7th June 2025

Monthly briefing

Sponsor me for $10/month and get a curated email digest of the month's most important LLM developments.

Pay me to send you less!

Sponsor & subscribe