Simon Willison’s Weblog

Subscribe

On Search: Robots. Everything you need to know about writing a tool to crawl the web