Simon Willison’s Weblog

Weeknotes: Trapped in an eternal refactor

16th December 2021

I’m still working on refactoring Datasette’s table view. In doing so I spun out a new plugin, datasette-pretty-traces, which improves Datasette’s tooling for seeing the SQL that was executed to build a specific page.

datasette-pretty-traces

I love tools like the Django Debug Toolbar which help show what’s going on under the hood of an application (see also the Tikibar, a run-in-production alternative we built at Eventbrite).

Datasette has long had a ?_trace=1 option for outputting debug information about SQL queries executed to build a page, but the output is a big block of JSON in the page footer, example here.

For the table view refactor project I decided it was time to make this more readable, so I built a plugin that runs some JavaScript to spot that output and turn it into something a bit more legible:

You can try it out here.

I’m becoming increasingly comfortable with the idea that it’s OK to ignore all of the current batch of JavaScript frameworks and libraries and just write code that uses the default browser APIs. Browser APIs are pretty great these days, especially given things like backtick literals for multi-line strings!

I’ll probably merge this into Datasette core at some point in the future, but a neat thing about having plugin support is I can dash out initial versions of things like this without needing to polish them up and include them in a formal release of the parent project.

Progress on the eternal refactor

Issue 1518, split from issue 878, is the all-consuming refactor.

Datasette’s table view is the most important page in the application: it’s the interface that lets you browse a table, filter it, search it, run faceting against it and export it out as other formats.

It’s the nastiest code in the entire project, having grown to over a thousand lines of Python. While it has very thorough tests, the actual code itself is unwieldy enough that it’s slowing down progress on all kinds of things I want to get done before I ship Datasette 1.0.

So I’m picking away at it. I’ve broken the underlying tests up into two modules (test_table_api.py and test_table_html.py) and I’ve made some small improvements, but I’ve also spun up some not-yet-committed prototypes both against my experimental asyncinject library and a new experiment that involves something that, if you squint at it, looks a tiny bit like a new ORM. I do not want to build a new ORM!

I’m not happy with any of this yet, and it’s definitely blocking my progress on other things. I’ll just have to keep on chipping away and see if I can get to a breakthrough.

Releases this week

• datasette-pretty-traces: 0.2.1—(3 releases total)—2021-12-13
  Prettier formatting for ?_trace=1 traces

TIL this week

• Using C_INCLUDE_PATH to install Python packages
• Using lsof on macOS
• Registering the same Pluggy hook multiple times in a single file