Simon Willison’s Weblog

Subscribe

5 items tagged “read-the-docs”

2024

AI crawlers need to be more respectful (via) Eric Holscher:

At Read the Docs, we host documentation for many projects and are generally bot friendly, but the behavior of AI crawlers is currently causing us problems. We have noticed AI crawlers aggressively pulling content, seemingly without basic checks against abuse.

One crawler downloaded 73 TB of zipped HTML files just in Month, racking up $5,000 in bandwidth charges!

# 25th July 2024, 8:02 pm / eric-holscher, ethics, ai, read-the-docs

2022

Weeknotes: python_requires, documentation SEO

Visit Weeknotes: python_requires, documentation SEO

Fixed Datasette on Python 3.6 for the last time. Worked on documentation infrastructure improvements. Spent some time with Fly Volumes.

[... 1,497 words]

How I build a feature

I’m maintaining a lot of different projects at the moment. I thought it would be useful to describe the process I use for adding a new feature to one of them, using the new sqlite-utils create-database command as an example.

[... 2,779 words]

2021

Adding Sphinx autodoc to a project, and configuring Read The Docs to build it. My TIL notes from figuring out how to use sphinx-autodoc for the sqlite-utils reference documentation today.

# 11th August 2021, 1:21 am / documentation, sqlite-utils, sphinx-docs, read-the-docs

2018

The subset of reStructuredText worth committing to memory

reStructuredText is the standard for documentation in the Python world.

[... 1,186 words]