Simon Willison’s Weblog

Supercharge the One Person Framework with SQLite: Rails World 2024 (via) Stephen Margheim shares an annotated transcript of the YouTube video of his recent talk at this year's Rails World conference in Toronto.

The Rails community is leaning hard into SQLite right now. Stephen's talk is some of the most effective evangelism I've seen anywhere for SQLite as a production database for web applications, highlighting several new changes in Rails 8:

... there are two additions coming with Rails 8 that merit closer consideration. Because these changes make Rails 8 the first version of Rails (and, as far as I know, the first version of any web framework) that provides a fully production-ready SQLite experience out-of-the-box.

Those changes: Ensure SQLite transaction default to IMMEDIATE mode to avoid "database is locked" errors when a deferred transaction attempts to upgrade itself with a write lock (discussed here previously, and added to Datasette 1.0a14 in August) and SQLite non-GVL-blocking, fair retry interval busy handler - a lower-level change that ensures SQLite's busy handler doesn't hold Ruby's Global VM Lock (the Ruby version of Python's GIL) while a thread is waiting on a SQLite lock.

The rest of the talk makes a passionate and convincing case for SQLite as an option for production deployments, in line with the Rails goal of being a One Person Framework - "a toolkit so powerful that it allows a single individual to create modern applications upon which they might build a competitive business".

Back in April Stephen published SQLite on Rails: The how and why of optimal performance describing some of these challenges in more detail (including the best explanation I've seen anywhere of BEGIN IMMEDIATE TRANSACTION) and promising:

Unfortunately, running SQLite on Rails out-of-the-box isn’t viable today. But, with a bit of tweaking and fine-tuning, you can ship a very performant, resilient Rails application with SQLite. And my personal goal for Rails 8 is to make the out-of-the-box experience fully production-ready.

It looks like he achieved that goal!

# 16th October 2024, 10:24 pm / scaling, rails, sqlite, ruby, gil

What’s New in Ruby on Rails 8 (via)

Rails 8 takes SQLite from a lightweight development tool to a reliable choice for production use, thanks to extensive work on the SQLite adapter and Ruby driver.

With the introduction of the solid adapters discussed above, SQLite now has the capability to power Action Cable, Rails.cache, and Active Job effectively, expanding its role beyond just prototyping or testing environments. [...]

• Transactions default to IMMEDIATE mode to improve concurrency.

Also included in Rails 8: Kamal, a new automated deployment system by 37signals for self-hosting web applications on hardware or virtual servers:

Kamal basically is Capistrano for Containers, without the need to carefully prepare servers in advance. No need to ensure that the servers have just the right version of Ruby or other dependencies you need. That all lives in the Docker image now. You can boot a brand new Ubuntu (or whatever) server, add it to the list of servers in Kamal, and it’ll be auto-provisioned with Docker, and run right away.

More from the official blog post about the release:

At 37signals, we're building a growing suite of apps that use SQLite in production with ONCE. There are now thousands of installations of both Campfire and Writebook running in the wild that all run SQLite. This has meant a lot of real-world pressure on ensuring that Rails (and Ruby) is working that wonderful file-based database as well as it can be. Through proper defaults like WAL and IMMEDIATE mode. Special thanks to Stephen Margheim for a slew of such improvements and Mike Dalessio for solving a last-minute SQLite file corruption issue in the Ruby driver.

# 7th October 2024, 7:17 pm / docker, ruby, sqlite, 37-signals, rails

How does Sidekiq really work? (via) I really like this category of blog post: Dan Svetlov took the time to explore the Sidekiq message queue’s implementation and then wrote it up in depth.

# 5th February 2024, 5:20 pm / ruby, sidekiq, queues

Build impossible programs. Delightful talk by Julia Evans describing how she went about building a Ruby profiler in Rust despite having no knowledge of Ruby internals and only beginner’s knowledge of Rust.

# 19th September 2018, 6:38 pm / ruby, rust, julia-evans

Ruby-style Blocks in Python. Yes, yes, yes, yes. A proposal for muli-line lambda support in Python that doesn’t trip up on significant whitespace. If this gets in before the proposed feature freeze I’ll be a very happy Pythonista. UPDATE: This is a post from over a year ago, and it looks like the proposal has since stalled.

# 23rd April 2010, 11:19 am / python, blocks, ruby

grammar.coffee (via) The annotated grammar for CoffeeScript, a new language that compiles to JavaScript developed by DocumentCloud’s Jeremy Ashkenas. The linked page is generated using Jeremy’s Docco tool for literate programming, also written in CoffeeScript. CoffeeScript itself is implemented in CoffeeScript, using a bootstrap compiler originally written in Ruby.

# 8th March 2010, 7:27 pm / compilers, javascript, coffeescript, documentcloud, jeremy-ashkenas, docco, literateprogramming, programming, selfhosting, ruby

twitter-text-conformance (via) This is a neat idea: Twitter have released open source libraries for parsing standard tweet syntax in Ruby and Java, but they’ve also released a set of YAML unit tests aimed at anyone who wants to implement the same parsing logic in other languages.

# 6th February 2010, 3:39 pm / twitter, ruby, java, testing, yaml

rlisagor’s freshen. A Python clone of Ruby’s innovative Cucumber testing framework. Tests are defined as a set of plain-text scenarios, which are then executed by being matched against test functions decorated with regular expressions. Has anyone used this or Cucumber? I’m intrigued but unconvinced—are the plain text scenarios really a useful way of defining tests?

# 5th January 2010, 7:30 pm / cucumber, testing, bdd, ruby, python, freshen, unittests

clarity. A web interface for tailing and grepping the log files in /var/log, written in Ruby and EventMachine.

# 4th November 2009, 10:36 pm / clarity, ruby, eventmachine, logging

Introducing Resque. A new background worker management queue developed at GitHub, using Redis for the persistence layer. The blog post explains both the design and the shortcomings of previous solutions at length. Within 24 hours of the release code an external developer, Adam Cooke, has completely reskinned the UI.

# 4th November 2009, 8:20 pm / resque, open-source, redis, github, queue, workers, ruby, sinatra

How We Made GitHub Fast. Detailed overview of the new GitHub architecture. It’s a lot more complicated than I would have expected—lots of moving parts are involved in ensuring they can scale horizontally when they need to. Interesting components include nginx, Unicorn, Rails, DRBD, HAProxy, Redis, Erlang, memcached, SSH, git and a bunch of interesting new open source projects produced by the GitHub team such as BERT/Ernie and ProxyMachine.

# 21st October 2009, 9:14 pm / github, scaling, nginx, unicorn, rails, drbd, haproxy, replication, redis, erlang, memcached, ssh, git, proxymachine, ruby, bert, ernie

MySQL backups with EBS snapshots. Assaf Arkin’s 45 line ruby script shows how to lock tables / XFS freeze / create an EBS snapshot / unfreeze and unlock, with hourly snapshots preserved for the past 24 hours and daily snapshots for the past week. Is an EBS snapshot enough to restore your data to somewhere other than EC2 though?

# 13th October 2009, 12:34 pm / assaf-arkin, ruby, ec2, mysql, ebs, cloud, backups

XSS Protection by Default in Rails 3.0. Fantastic news—congratulations, Rails core team.

# 8th October 2009, 4:35 pm / xss, rails, ruby, security

I like Unicorn because it’s Unix. Ryan Tomayko analyses Unicorn, a new, pre-forking Ruby HTTP server that makes extensive use of Unix syscalls and idioms, and asks why dynamic language programmers don’t take advantage of these more often.

# 7th October 2009, 11:42 am / ruby, exec, fork, programming, ryan-tomayko, unicorn, unix

cloud-crowd. New parallel processing worker/job queue system with a strikingly elegant architecture. The central server is an HTTP server that manages job requests, which are farmed out to a number of node HTTP servers which fork off worker processes to do the work. All communication is webhook-style JSON, and the servers are implemented in Sinatra and Thin using a tiny amount of code. The web-based monitoring interface is simply beautiful, using canvas to display graphs showing the system’s overall activity.

# 21st September 2009, 11:09 pm / cloudcrowd, webhooks, json, http, message-queues, workers, sinatra, thin, ruby, canvas

homebrew. Exciting alternative to MacPorts for compiling software on OS X—homebrew avoids sudo and defines packages as simple Ruby scripts, shared and distributed using Git.

# 21st September 2009, 6:51 pm / homebrew, osx, git, ruby, macports

Webhooks behind the firewall with Reverse HTTP. Hookout is a Ruby / rack adapter that lets you serve a web application from behind a firewall, by binding to a Reverse HTTP proxy running on the internet (such as the free one provided by reversehttp.net). Useful for far more than just webhooks, this means you can easily expose any Ruby web service to the outside world. An implementation of this as a general purpose proxy server would make it useful for applications written in any language.

# 22nd July 2009, 1:46 pm / webhooks, hookout, ruby, reversehttp, comet

Phusion Passenger for nginx. Passenger (aka mod_rails / mod_rack) enables easy deployment of Rails and Ruby apps under Apache... and the latest version adds support for nginx as well. It works as an HTTP proxy and process manager, spawning worker processes and forwarding HTTP requests to them via a request queue. It can also handle Python WSGI applications—anyone tried it out for that yet?

# 20th April 2009, 4:53 am / nginx, passenger, python, rails, deployment, apache, ruby

Twitter: blaming Ruby for their mistakes? The comments on the entry include replies from Twitter employees and the RabbitMQ consultant they brought in, and provide a full rebuttal to the various accusations of NIH that were thrown around recently.

# 6th April 2009, 11:06 am / twitter, nih, rabbitmq, ruby

Ruby on Rails 2.3 Release Notes. I’m impressed with how thoroughly Rails has embraced Rack (Ruby’s standardised web framework API, inspired by Python’s WSGI).

# 15th March 2009, 1:22 pm / rack, ruby, rails, python, wsgi

CloudMade: A Summary of the Future of Mapping. CloudMade are now offering commercially supported APIs on top of OpenStreetMap, including geocoding, routing and tile access libraries in Python/Ruby/Java and a very neat theming tool that lets you design your own map styles. This is really going to kick innovation around OpenStreetMap up a notch.

# 17th February 2009, 11:25 am / openstreetmap, cloudmade, mapping, python, ruby, java, geocoding, routing, tiles