Simon Willison’s Weblog

Newsletter
Atom feed for datasette Random

1,568 posts tagged “datasette”

Datasette is an open source tool for exploring and publishing data.

2026

Datasette 1.0a43 added the ability for plugins to implement streaming export formats. Datasette core supports streaming CSV and TSV - you can export millions of lines of data from a table without needing to load it all into memory, because those two file formats support streaming data in chunks. The register_output_renderer() plugin hook can now be used to implement the same pattern for other formats.

I've learned never to release a new plugin feature without accompanying it with a plugin that uses it.

Claude Fable 5.2 suggested the Apache Arrow streaming format as a good option, and built this plugin in almost a single shot.

Add the plugin to Datasette (datasette install datasette-arrows) and the export dialog gains this new Arrow option:

Export data modal for the threads table. The TSV and CSV exports are now joined by Arrow, showing "All 927 rows" and the text Apache Arrow IPC stream, for DuckDB, pandas, Polars and other tools. There is a Download button and a Copy URL button.

This screenshot is from Datasette's main branch - the most recent release doesn't yet group streaming formats such as Arrow with the TSV and CSV in the "All rows in this table" section, but that's coming in the next alpha release.

Running the full server-side Datasette Python app in the browser with service workers and Pyodide

Visit Running the full server-side Datasette Python app in the browser with service workers and Pyodide

Datasette Lite is my port of the Datasette Python application to run directly in the browser using WebAssembly and Pyodide. I’ve released a complete rewrite of Datasette Lite using service workers, and it’s now almost indistinguishable from the real server-side application.

[... 1,951 words]

The first new Datasette geospatial plugin in quite a while! This one combines the excellent MapLibre GL library with OpenFreeMap, the free vector map server operated by Zsolt Ero (you should sponsor Zsolt on GitHub.)

I built most of this with a single prompt to Claude Fable 5.2:

Look at datasette-cluster-map and datasette-leaflet-geojson plugins, then look at ~/dev/tools/openfreemap-demo.html - then build a brand new Datasette plugin that uses maplibreGL and openfreemap and combines the features of those other two plugins - but be creative and think outside the box for other useful features the plugin might have

"Be creative and think outside the box" resulted in a few neat ideas - my favorite is that it displays SVG previews of GeoJSON shapes, then loads a WebGL preview only when they scroll into view.

I also had Claude build a comprehensive demo which works in the latest Datasette Lite, so you can visit this link to load the plugin and a demo database in your browser via Pyodide and WebAssembly.

I like this demo page showing both GeoJSON shapes and lat/lon points on the same map:

Screenshot of a web map interface. Toolbar: "View" set to "Clusters", "Colour by" set to "continent", plus "Filter to this area" and "Download GeoJSON" buttons. A world map shows countries outlined and shaded by continent, with coloured dots and numbered cluster circles. A popup for a point in South America reads: capital Caracas, country Venezuela, continent South America, capital_population 2985000, latitude 10.502944, longitude -66.918983. Legend: Africa (52), Asia (45), Europe (39), North America (16), South America (13), Oceania (6). Text: "All 171 rows are on the map". Below, a table with columns country, continent, capital_population, latitude, longitude, boundary. First row: Afghanistan, Asia, 3277000, 34.518636, 69.181314, with a small map in the boundary column showing Afghanistan outlined and shaded in blue.

I've been leaning hard into Claude Fable 5.2 to help knock things off the 1.0 milestone, which is now 53% complete.

The biggest 1.0 item in this release is streaming support for output renderers - the built-in TSV and CSV export mechanisms now use this as well, and plugins can register themselves as capable of streaming every row in a table in one go via the new Response.stream() method.

There are a couple of security fixes, one of which I deemed deserving of a GitHub advisory: private stored queries could be exposed, granting access to the title and SQL of the query (but not allowing that query to be executed).

There are dozens of smaller improvements. The row page JSON representation finally has a design suitable for 1.0. A new <datasette-copy-button> Web Component provides copy-schema in various places. JSON in columns on the row page can be pretty-printed. There's a new DatasetteModal.confirm() JavaScript utility method. Plus dozens of bug fixes, including contributions by Zihan Dai, Peter Olkhovets, Ilya Khalafi, and Ankita Advitot.

The feature I'm most excited about is a pretty tiny one: I've added CSS multi-page view transitions to Datasette! Here's the commit - key CSS is as follows:

/* Enable cross-document view transitions,
   for anyone without prefers-reduced-motion */
@media (prefers-reduced-motion: no-preference) {
    @view-transition {
        navigation: auto;
    }
}
/* .facet-results should morph from old position to new */
html:active-view-transition .content > .facet-results {
    view-transition-name: facet-results;
}
/* So should a .table-wrapper, provided it contains .rows-and-columns */
html:active-view-transition .table-wrapper:has(> table.rows-and-columns) {
    view-transition-name: rows-table;
}

With that in place, navigations within Datasette - still staunchly a multi-page application - feel more like an SPA. This works on Safari and Chrome at the moment and I believe supports for Firefox is due to land in Firefox 162 on December 8th 2026 (seeing as they are no longer feature-flagged in Firefox Nightly.)

I backported a minor security fix from 1.0a43:

Fixed a security issue where a database registered using add_database(db, name=..., route=...) with a route that differs from its name could be downloaded from /<route>.db by actors who were denied access to it. [...]

Datasette Apps are sandboxed HTML and JavaScript applications that live inside Datasette - think Claude Artifacts, but these ones can run SQL queries against your data.

0.4a0 adds support for linking: you can link to an app with #fragments in the URL and the app can see and use those, plus supported URL schemes are reported back to Datasette Agent so it can help write apps that link to each other.

This means you can have e.g. an app that shows details about a specific record, and a completely separate app that searches those records and links to their representations in the first app.

Accumulated small fixes for Datasette Agent, my agent plugin for Datasette that lets you ask questions about your data. Most notably, it now adds an agent prompt directly to the Datasette homepage, with support for hash-mentions of tables and queries:

Datasette homepage. Below the search box is a Datasette Agent heading with a text input and an ask button. User has typed: Tell me about #re - the box is showing autocomplete results for the repos and releases tables.

datasette-llm is the intermediate plugin for Datasette that supports configuring LLMs for use with other Datasette plugins.

This release adds support for OpenAI-compatible endpoints, so you can configure Datasette to use an LLM hosted anywhere that talks the OpenAI pseudo-standard web protocol.

This means that if you're running local models with LM Studio or Ollama or llama.cpp (which just hit v1.0.0) you can run Datasette via uvx like this:

wget https://datasette.io/content.db

uvx --with datasette-agent datasette \
  --root --secret 1 \
  -s plugins.datasette-llm.default_model '{
    "model": "lmstudio-qwen",
    "model_name": "qwen/qwen3.8-27b",
    "api_base": "http://localhost:1234/v1"
  }' content.db

And you'll get a local server running a Datasette Agent instance powered by your local model!

Screenshot of a chat interface. Header bar reads "home" on the left and "root" with a hamburger menu icon on the right. Below is a back arrow, the heading "Chat", a model label "lmstudio-qwen" and an "EXPORT" button. A blue user message bubble says "how many repos and how many releases are there?" Below it is a collapsed tool call "list_databases_and_tables" and an expanded tool call "sql_query". Its REQUEST section shows database "content", display "both" and the SQL: SELECT (SELECT COUNT(*) FROM repos) AS repo_count, (SELECT COUNT(*) FROM releases) AS release_count. Its RESPONSE section shows columns repo_count and release_count with a single row of 200 and 1889. Below that is a table with columns repo_count and release_count and one row: 200, 1889, followed by a "View SQL query" button. The assistant reply reads "There are 200 repos and 1,889 releases in the content database." At the bottom is a text input with placeholder "Type a message... (# to tag a table or query)" and a blue "Send" button.

A new release of Datasette's longest untouched plugin, for rendering charts (using Vega-Lite) against data in a table or custom query. The last release was in July 2018!

I finally bumped it for compatibility with the latest Datasette alphas, plus some cosmetic tweaks to make it work better with Datasette Lite.

(Datasette Lite just gained the ability to run JavaScript plugins like this! I'll write more about that soon.)

Since it works with Datasette Lite, the live demo in the README now links to this demo on Datasette Lite.

Screenshot of a charting tool. A panel headed "Charting options" has radio buttons for "Bar" (selected), "Line" and "Scatter". Below are select menus: "X column" set to "airline" with "Type" set to "Label", and "Y column" set to "fatalities_00_14" with "Type" set to "Numeric". A "Swap X and Y" button sits beneath those, followed by "Color" and "Size" menus both set to "-- none --". Underneath the panel is a blue bar chart with airline names along the X axis and fatalities_00_14 on the Y axis, running from 0 to 550 - most airlines have no bar, a handful have bars, and Malaysia Airlines is the tallest.

One of the oldest Datasette plugins, and still one of my favorites. It now works with Datasette 1.0a42, and depends on the new datasette-leaflet to ensure OpenStreetMap tiles are requested with a referrer header.

Here's a map of power plants owned by California's PG&E, also demonstrating Datasette's recently redesigned facets:

Screenshot of a faceted map interface. Three facet panels at the top: "country_long 1" with United States of America 81 selected; "owner 1" with Pacific Gas & Electric Co. 81 selected; "primary_fuel 5" listing Hydro 63, Solar 10, Gas 5, Storage 2, Nuclear 1. Each primary_fuel row has a pale blue horizontal bar behind its label, sized in proportion to its count: Hydro's bar fills most of the row width, Solar's is a short sliver, Gas's is shorter still, and Storage and Nuclear have only a thin strip at the left edge. Below is a Leaflet map with + and − zoom controls showing California and Nevada, with marker clusters across northern and central California: 9 near Redding, 21 near Chico, 2 to the west of Chico, 13 northeast of Sacramento, 7 east of Stockton, 6 at Oakland, 10 at Fresno, 9 west of Visalia, 2 north of Bakersfield, plus two individual blue pin markers on the coast, one in the far north and one north of Santa Maria. Attribution: Leaflet | © OpenStreetMap contributors

OpenStreetMap have a very reasonable policy that hits to their hosted tile server should send a referrer header. Datasette table pages used the Referrer-Policy: no-referrer HTTP header, causing plugins that loaded tiles from OpenStreetMap to violate that policy.

datasette-leaflet is the plugin that loads a bundled Leaflet so other Datasette plugins can depend on it. It now sets the referrer policy on the tile requests themselves to strict-origin-when-cross-origin, passing the host but not the full path to the page.

Hide your slop PR descriptions behind <details><summary>

Visit Hide your slop PR descriptions behind <details><summary>

Most of the changes I make to my projects these days come in the form of PRs opened for me by Claude Code or Codex. The PR is my unit of work—I use them to bundle commits together, iterating on them until a feature is complete, document decisions and then merge the finished result. Today’s coding agents are great at writing PR descriptions.

[... 987 words]

datasette #3046: Don’t serve link: rel=next HTTP headers to Firefox for HTML pages. I was looking at the Firefox network panel today when I noticed that requests to a Datasette table page were triggering an HTTP prefetch to the next page of rows in that table.

This was happening because Datasette pages serve an HTTP link header like this:

link: <https://latest.datasette.io/fixtures/compound_three_primary_keys?_next=a%2Cd%2Cv>; rel="next"

This is bad. A tiny fraction of the visitors to one of those pages will click the "next" link, which means for everyone else their browser is fetching unnecessary data and the Datasette server is having to render an entire page that then gets thrown away.

I dug in and learned that Firefox has an issue that has been open for 24 years about this behavior...

And one of the early comments on that issue links to my blog post from 2002 complaining about this feature. I hadn't just forgotten that I had written about it, I had forgotten the feature entirely!

# 7th November 2026, 11:57 pm / firefox, html, mozilla, datasette

Fixes a bug that affected my agent.datasette.io demo site, where the set of available LLM models was not correctly restricted to just those in the configuration.

Datasette Apps adds support for hosting and editing custom HTML applications inside of Datasette.

These can be created by hand, but it's easier to create them using LLMs via Datasette Agent. This release improves that integration:

  • Apps can now be tagged in Datasette Agent chat messages by typing #, which autocompletes from the apps the user has permission to view.
  • New app_add_stored_queries() and app_remove_stored_queries() tools let the agent manage which stored queries an app is allowed to call. Granting a write query requires approval from the user first.
  • app_create() now checks that each requested stored query exists and that the user can view it, and asks the user to approve any write queries before the app is created.
  • The "View app" buttons shown by the agent tools, and the links on # app references, now open the app full screen in the Datasette Agent panel.

This is a significant release of Datasette Agent. The most notable new feature is called the panel - it lets the agent chat open and then sit next to pages from Datasette, for example a table or query page.

When combined with datasette-apps this means you can prompt a database-backed app into existence and then modify it from the chat that sits next to that app.

Screenshot of a split-screen chat interface titled "evaluate llm prices" (model tag "gpt-6-luna"): the conversation runs in a narrow column on the left, and a wide side panel on the right displays the app the assistant built, with "EXPORT" and "Close panel" buttons above it. In the chat, blue user bubbles read "Build an app to help explore this data" and "give it a more hacker aesthetic design", each followed by collapsed tool calls (app_create, app_render, app_view, app_str_replace), "LLM Price Explorer v1." and "v2." entries with "View app" and "Edit" buttons, and short assistant replies describing the app and its new dark terminal-inspired theme. The right panel shows the resulting "LLM Price Explorer" in neon green monospace on black: stat cards for 185 models from 12 providers with workload costs from $0.042 (Jev 1.13) to $300.00 (o1 Pro), filter and workload cost estimate controls, and a table of matching models with input, output and cached input prices per million tokens.

You can try it by signing in with your GitHub account to agent.datasette.io.

I have some big upgrades to Datasette Agent in the pipeline, but in the meantime here's a tiny user experience improvement to the underlying datasette-llm plugin.

It now picks up the default model used by LLM, which means you can start using Datasette Agent without needing to explicitly configure a default model:

uvx --with datasette-agent datasette content.db --internal internal.db

Previously you would also need to specify a model with -s plugins.datasette-agent.default_model gpt-6-astra.

When I switched LLM over to httpx2 in LLM 0.33 I forget that this would break any plugins which used httpx without directly depending on it. llm-tools-datasette was one of those plugins, now fixed.

My plugin for detecting and listing all Datasette instances on the current machine now works with the latest version, and displays the detected PID as well. Knowing the PID makes it easier to attach the new Python 3.15 profiler.

None

I saw Parseable in a Show HN today - it's a new observability platform with both an open source (AGPL) Rust implementation (a single ~180MB binary), an "Enterprise" version with extra features and a cloud hosted option.

Since Datasette 1.0a41 added OpenTelemetry support (thanks, Alex Garcia), I decided to fire up Codex and have it figure out how to run Parseable and feed it traces from Datasette.

Here's my (human-written) TIL showing the patterns that worked, and here's a screenshot of a Datasette trace displayed within the Parseable localhost web application:

Screenshot of a trace detail view in an observability web app, with a span waterfall overlaid on a dimmed navigation sidebar and filter column. Dimmed sidebar: breadcrumb "Community > Traces > datas" (cut off), search box "Search... ⌘K", nav items "Home", "Ingest telemetry", section "ANALYZE": "Keystone", "Dashboards", "SQL Editor", section "OBSERVE": "Logs", "Metrics", "Traces" (selected), "APM", "Agents", section "MONITOR": "Alerts", "Errors", section "DATA": "Datasets", and at the bottom "Settings", "Book a call", "Support". Dimmed filter column, cut off at the right edge: "Search fi", "Core", "Log format", "User agent", "Source IPs", "Error", "Service", "service.ins", "service.na", "datasett" (checked), "Span", "Database", "HTTP", "http.reque", "NULL" (unchecked), "GET" (checked), "http.respo", "http.route", "Server", "server.add", "Telemetry", "URL", "All fields". Trace panel header: "Trace detail > 6f819a2170bcd1e91c6ea3ae236ec60b" with a copy icon, a "Related logs" button and a close X. Summary: "Start time 6:56 PM, Oct 6, 2026 UTC", "Duration 40.9 ms", "Spans 247". A minimap with axis "0ns 10.2ms 20.5ms 30.7ms 40.9ms" shows many short span bars cascading diagonally from top left toward the lower middle, with a few longer bars. Below is a span table with a "Span name" header, a "Search spans..." box, collapse and expand buttons, and a timeline axis "0ns 10.2ms 20.5ms 30.7ms 40.9ms". Rows (name, service, duration): root span with collapse toggle "123", "GET /..." "datasette..." 40.9ms spanning the full timeline; then alternating rows where each "db.query" has a collapse toggle "1": db.query datasette-local 679µs, db.query.execute datasette-local 278µs, db.query datasette-local 341µs, db.query.execute datasette-local 55µs, db.query datasette-local 357µs, db.query.execute datasette-local 197µs, db.query datasette-local 330µs, db.query.execute datasette-local 73µs, db.query datasette-local 2.26ms, db.query.execute datasette-local 2.02ms, db.query datasette-local 232µs, db.query.execute datasette-local 64µs, db.query datasette-local 207µs, db.query.execute datasette-local 71µs, db.query datasette-local 6.11ms. The child span bars start progressively later across the early part of the timeline.

A minor fix for compatibility with the latest Datasette alphas. This meant we could upgrade the datasette.io site to Datasette 1.0a41.

Alec Garcia added support for OpenTelemetry to Datasette in this release.

I've also refactored all of Datasette's modal dialogs to a single Web Component, which is now documented for other plugins to use.

  • Explain plans now work on read-only stored-query pages.

I upgraded datasette.simonwillison.net to Datasette 1.0a40, which inspired me to ship a new version of this explain plugin.

I run this GitHub login plugin on the agent.datasette.io demo site and I noticed that my authenticated sessions weren't lasting very long. It turned out that the plugin was setting cookies without a Max-Age parameter, so they were expiring at the end of a browser session (which in Mobile Safari seems to happen pretty often, independently of how you are using the app.)

I fixed that in #80 and, since this plugin has been around for quite a while and is tested against both Datasette 0.65.x and Datasette 1.0ax, I decided to bump it up to a 1.0 release. I'm trying to get better at promoting stable plugins to 1.0.

Same security fix as 0.65.5, plus some neat new features and bug fixes:

  • Plugins can now launch and manage background tasks using the new datasette.add_background_task() method. Thanks, Alex Garcia.
  • I've migrated Datasette to httpx2 for features like the internal datasette.client.get() method.
  • A whole lot of bug fixes, many of them stemming from a recent effort to triage issues for a 1.0 stable release.

Security fix for an issue where a trailing newline in a requested table name could bypass table permissions and expose private rows, reported by dpfkdlemtp in GHSA-h547-rmjf-5m2m.

Datasette 1.0a39 and 0.65.4 security releases. Today we're releasing two new security patch versions of Datasette: 1.0a39 and 0.65.4 - one for the current alpha series and one for the stable 0.65.x family.

These are security fixes which you should apply if you are running a Datasette instance on the public web - in particular if that instance mixes both public and private tables.

Following issues reported by Sevban Dönmez, Alex Garcia and I ran an extensive audit of Datasette using Claude Fable 5.1, GPT-5.6, and GPT-6 Astra. We then spent almost a week collaborating on and reviewing the fixes.

They helped find some very subtle bugs. We'll be incorporating security audits by frontier models into all of our development work going forward.

Alex came up with a way of splitting the work which I found extremely productive:

Alex Garcia and I worked together running and then responding to the audit, working in a shared private repository. For most of the issues we split the work: one of us would create the automated tests highlighting the issue, then the other would implement the fix. This ensured that two separate humans had eyes on each of the issues, in addition to our coding agents running different models.

# 11th September 2026, 3:27 am / releases, security, ai, datasette, generative-ai, llms, agentic-engineering, ai-security-research

  • Sets force_https=true in fly.toml. #31
  • Fix for Volume could not be found bug. #32
  • Compatible with app-scoped deploy tokens. #34