Simon Willison’s Weblog

Subscribe

Why does Google use “Allow” in robots.txt, when the standard seems to be “Disallow?”

4th February 2012

My answer to Why does Google use “Allow” in robots.txt, when the standard seems to be “Disallow?” on Quora

The Disallow command prevents search engines from crawling your site.

The Allow command allows them to crawl your site.

If you’re using the Google Webmaster tools, you probably want Google to crawl your site.

Am I misunderstanding your question?

Posted 4th February 2012 at 9:45 am · Follow me on Mastodon, Bluesky, Twitter or subscribe to my newsletter

More recent articles

This is Why does Google use “Allow” in robots.txt, when the standard seems to be “Disallow?” by Simon Willison, posted on 4th February 2012.

Next: How can I install Django in a server without shell access?

Previous: Does YCombinator fund Start-ups interested in building electronic gadgets?

Monthly briefing

Sponsor me for $10/month and get a curated email digest of the month's most important LLM developments.

Pay me to send you less!

Sponsor & subscribe