Simon Willison’s Weblog

Supercharge the One Person Framework with SQLite: Rails World 2024 (via) Stephen Margheim shares an annotated transcript of the YouTube video of his recent talk at this year's Rails World conference in Toronto.

The Rails community is leaning hard into SQLite right now. Stephen's talk is some of the most effective evangelism I've seen anywhere for SQLite as a production database for web applications, highlighting several new changes in Rails 8:

... there are two additions coming with Rails 8 that merit closer consideration. Because these changes make Rails 8 the first version of Rails (and, as far as I know, the first version of any web framework) that provides a fully production-ready SQLite experience out-of-the-box.

Those changes: Ensure SQLite transaction default to IMMEDIATE mode to avoid "database is locked" errors when a deferred transaction attempts to upgrade itself with a write lock (discussed here previously, and added to Datasette 1.0a14 in August) and SQLite non-GVL-blocking, fair retry interval busy handler - a lower-level change that ensures SQLite's busy handler doesn't hold Ruby's Global VM Lock (the Ruby version of Python's GIL) while a thread is waiting on a SQLite lock.

The rest of the talk makes a passionate and convincing case for SQLite as an option for production deployments, in line with the Rails goal of being a One Person Framework - "a toolkit so powerful that it allows a single individual to create modern applications upon which they might build a competitive business".

Back in April Stephen published SQLite on Rails: The how and why of optimal performance describing some of these challenges in more detail (including the best explanation I've seen anywhere of BEGIN IMMEDIATE TRANSACTION) and promising:

Unfortunately, running SQLite on Rails out-of-the-box isn’t viable today. But, with a bit of tweaking and fine-tuning, you can ship a very performant, resilient Rails application with SQLite. And my personal goal for Rails 8 is to make the out-of-the-box experience fully production-ready.

It looks like he achieved that goal!

# 16th October 2024, 10:24 pm / scaling, rails, sqlite, ruby, gil

What’s New in Ruby on Rails 8 (via)

Rails 8 takes SQLite from a lightweight development tool to a reliable choice for production use, thanks to extensive work on the SQLite adapter and Ruby driver.

With the introduction of the solid adapters discussed above, SQLite now has the capability to power Action Cable, Rails.cache, and Active Job effectively, expanding its role beyond just prototyping or testing environments. [...]

• Transactions default to IMMEDIATE mode to improve concurrency.

Also included in Rails 8: Kamal, a new automated deployment system by 37signals for self-hosting web applications on hardware or virtual servers:

Kamal basically is Capistrano for Containers, without the need to carefully prepare servers in advance. No need to ensure that the servers have just the right version of Ruby or other dependencies you need. That all lives in the Docker image now. You can boot a brand new Ubuntu (or whatever) server, add it to the list of servers in Kamal, and it’ll be auto-provisioned with Docker, and run right away.

More from the official blog post about the release:

At 37signals, we're building a growing suite of apps that use SQLite in production with ONCE. There are now thousands of installations of both Campfire and Writebook running in the wild that all run SQLite. This has meant a lot of real-world pressure on ensuring that Rails (and Ruby) is working that wonderful file-based database as well as it can be. Through proper defaults like WAL and IMMEDIATE mode. Special thanks to Stephen Margheim for a slew of such improvements and Mike Dalessio for solving a last-minute SQLite file corruption issue in the Ruby driver.

# 7th October 2024, 7:17 pm / docker, ruby, sqlite, 37-signals, rails

Running GitHub on Rails 6.0. Back in 2019 Eileen M. Uchitelle explained how GitHub upgraded everything in production to Rails 6.0 within 1.5 weeks of the stable release. There’s a trick in here I really like: they have an automated weekly job which fetches the latest Rails main branch and runs the full GitHub test suite against it, giving them super-early warnings about anything that might break and letting them provide feedback to upstream about unintended regressions.

# 6th August 2021, 4:30 pm / rails, continuous-integration, github

CSRF: Flash + 307 redirect = Game Over. Here’s the exploit that Django and Rails both just released fixes for. It’s actually a flaw in the Flash player. Flash isn’t meant to be able to make cross-domain HTTP requests with custom HTTP headers unless the crossdomain.xml file on the other domain allows them to, but it turns out a 307 redirect (like a 302, but allows POST data to be forwarded) confuses the Flash player in to not checking the crossdomain.xml on the host it is being redirect to.

# 10th February 2011, 10:07 pm / crossdomainxml, csrf, django, flash, rails, security, recovered

Hookbox (via) For most web projects, I believe implementing any real-time comet features on a separate stack from the rest of the application makes sense—keep using Rails, Django or PHP for the bulk of the application logic, and offload any WebSocket or Comet requests to a separate stack built on top of something like Node.js, Twisted, EventMachine or Jetty. Hookbox is the best example of that philosophy I’ve yet seen—it’s a Comet server that makes WebHook requests back to your regular application stack to check if a user has permission to publish or subscribe to a given channel. “The key insight is that all application development with hookbox happens either in JavaScript or in the native language of the web application itself”.

# 29th July 2010, 9:48 am / comet, django, hookbox, javascript, michael-carter, php, rails, webhooks, recovered

Node.js, redis, and resque (via) Paul Gross has been experimenting with Node.js proxies for allowing web applications to be upgraded without missing any requests. Here he places all incoming HTTP requests in a redis queue, then has his backend Rails servers consume requests from the queue and push the responses back on to a queue for Node to deliver. When the backend application is upgraded, requests remain in the queue and users see a few seconds of delay before their request is handled. It’s not production ready yet (POST requests aren’t handled, for example) but it’s a very interesting approach.

# 28th February 2010, 11:02 pm / node, nodejs, javascript, highavailability, http, rails, redis

Play framework for Java. I’m genuinely impressed by this—it’s a full stack web framework for Java that actually does feel a lot like Django or Rails. Best feature: code changes are automatically detected and reloaded by the development web server, giving you the same save-and-refresh workflow you get in Django (no need to compile and redeploy to try out your latest changes).

# 25th October 2009, 11:21 pm / play, java, frameworks, web, django, rails

How We Made GitHub Fast. Detailed overview of the new GitHub architecture. It’s a lot more complicated than I would have expected—lots of moving parts are involved in ensuring they can scale horizontally when they need to. Interesting components include nginx, Unicorn, Rails, DRBD, HAProxy, Redis, Erlang, memcached, SSH, git and a bunch of interesting new open source projects produced by the GitHub team such as BERT/Ernie and ProxyMachine.

# 21st October 2009, 9:14 pm / github, scaling, nginx, unicorn, rails, drbd, haproxy, replication, redis, erlang, memcached, ssh, git, proxymachine, ruby, bert, ernie

nginx_http_push_module. More clever design with webhooks—here’s an nginx module that provides a comet endpoint URL which will hang until a back end process POSTs to another URL on the same server. This makes it much easier to build asynchronous comet apps using regular synchronous web frameworks such as Django, PHP and Rails.

# 17th October 2009, 4:48 pm / nginx, comet, webhooks, django, php, rails

XSS Protection by Default in Rails 3.0. Fantastic news—congratulations, Rails core team.

# 8th October 2009, 4:35 pm / xss, rails, ruby, security

Ravelry. Tim Bray interviews Casey Forbes, the single engineer behind Ravelry, the knitting community that serves 10 million Rails requests a day using just seven physical servers, MySQL, Sphinx, memcached, nginx, haproxy, passenger and Tokyo Cabinet.

# 3rd September 2009, 6:50 pm / scaling, ravelry, tim-bray, caseyforbes, tokyocabinet, tokyotyrant, rails, mysql, sphinx-search, memcached, nginx, haproxy, passenger

cache-money. A “write-through caching library for ActiveRecord”, maintained by Nick Kallen from Twitter. Queries hit memcached first, and caches are automatically kept up-to-date when objects are created, updated and deleted. Only some queries are supported—joins and comparisons won’t hit the cache, for example.

# 28th June 2009, 3:17 pm / activerecord, caching, rails, twitter, cachemoney, memcached

whine flu, railsmalefail 2009. Danny quotes the smartest take on the CouchDB/pr0n controversy: “It’s about presenting women as ’the other,’ not ’us.’ It would have been just as offensive if all the women shown were domineering mothers in aprons, shaking their fingers and threatening with rolling pins.”

# 29th April 2009, 11:39 am / danny-obrien, rails, couchdb, sexism

Phusion Passenger for nginx. Passenger (aka mod_rails / mod_rack) enables easy deployment of Rails and Ruby apps under Apache... and the latest version adds support for nginx as well. It works as an HTTP proxy and process manager, spawning worker processes and forwarding HTTP requests to them via a request queue. It can also handle Python WSGI applications—anyone tried it out for that yet?

# 20th April 2009, 4:53 am / nginx, passenger, python, rails, deployment, apache, ruby

Ruby on Rails 2.3 Release Notes. I’m impressed with how thoroughly Rails has embraced Rack (Ruby’s standardised web framework API, inspired by Python’s WSGI).

# 15th March 2009, 1:22 pm / rack, ruby, rails, python, wsgi

Building and Scaling a Startup on Rails: 12 Things We Learned the Hard Way. Lessons learned from Posterous. Some good advice in here, in particular “Memcache later: If you memcache first, you will never feel the pain and never learn how bad your database indexes and Rails queries are”. Also recommends using job queues for offline processing of anything that takes more than 200ms.

# 23rd February 2009, 8:28 am / message-queues, rails, scaling, posterous, memcache

Infrastructure for Modern Web Sites. Leonard’s thoughts on what the next generation of web frameworks should aim to provide.

# 29th January 2009, 1:36 pm / frameworks, leonardlin, django, rails, sysadmin, infrastructure