Simon Willison’s Weblog

Leader Election With S3 Conditional Writes (via) Amazon S3 added support for conditional writes last week, so you can now write a key to S3 with a reliable failure if someone else has has already created it.

This is a big deal. It reminds me of the time in 2020 when S3 added read-after-write consistency, an astonishing piece of distributed systems engineering.

Gunnar Morling demonstrates how this can be used to implement a distributed leader election system. The core flow looks like this:

• Scan an S3 bucket for files matching lock_* - like lock_0000000001.json. If the highest number contains {"expired": false} then that is the leader
• If the highest lock has expired, attempt to become the leader yourself: increment that lock ID and then attempt to create lock_0000000002.json with a PUT request that includes the new If-None-Match: * header - set the file content to {"expired": false}
• If that succeeds, you are the leader! If not then someone else beat you to it.
• To resign from leadership, update the file with {"expired": true}

There's a bit more to it than that - Gunnar also describes how to implement lock validity timeouts such that a crashed leader doesn't leave the system leaderless.

# 30th August 2024, 11:13 pm / architecture, aws, s3

(Almost) Every infrastructure decision I endorse or regret after 4 years running infrastructure at a startup (via) Absolutely fascinating post by Jack Lindamood talking about services, tools and processes used by his startup and which ones turned out to work well v.s. which ones are now regretted.

I’d love to see more companies produce lists like this.

# 10th February 2024, 5:51 am / architecture, infrastructure, startups

Architecture Notes: Datasette (via) I was interviewed for the first edition of Architecture Notes—a new publication (website and newsletter) about software architecture created by Mahdi Yusuf. We covered a bunch of topics in detail: ASGI, SQLIte and asyncio, Baked Data, plugin hook design, Python in WebAssembly, Python in an Electron app and more. Mahdi also turned my scrappy diagrams into beautiful illustrations for the piece.

# 27th May 2022, 3:20 pm / architecture, datasette

Serverless Microservice Patterns for AWS (via) A handy collection of 19 architectural patterns for AWS Lambda collected by Jeremy Daly.

# 12th June 2019, 12:13 am / architecture, aws, lambda, patterns

How the Citus distributed database rebalances your data. Citus is a fascinating implementation of database sharding built on top of PostgreSQL primitives. PostgreSQL 10 introduced extremely flexible logical replication—in this post Craig Kerstiens explains how Citus use this new ability to re-balance shards (e.g. when you move from two to four physical PostgreSQL nodes) without downtime.

# 1st February 2018, 10:50 pm / architecture, postgresql, sharding, zero-downtime, craig-kerstiens

Twitter, an Evolving Architecture. The most detailed write-up of Twitter’s current architecture I’ve seen, explaining the four layers of cache (all memcached) used by the Twitter API.

# 28th June 2009, 3:09 pm / architecture, caching, memcached, twitter

Twitter, or Architecture Will Not Save You. Kellan is not an armchair architect. He also doesn’t mention Rails once. Well worth reading.

# 29th May 2008, 1:16 am / architecture, kellan-elliott-mccrea, rails, twitter

The GigaOM Interview: Mark Zuckerberg. Some interesting titbits on Facebook’s architecture.

# 11th March 2008, 5:41 am / architecture, facebook, mark-zuckerberg, scaling