Simon Willison’s Weblog

Introducing Digg’s IDDB Infrastructure. IDDB is Digg’s new infrastructure component for sharding data across multiple databases, with support for both MySQL and memcachedb. “The DiggBar and URL minifying service is powered by a 16 machine IDDB cluster, which includes 8 write masters in the index and 8 MySQL storage nodes.”

# 3rd April 2009, 8:42 pm / databases, digg, iddb, joe-stump, memcachedb, mysql, scaling, sharding

Techniques for safely consuming external HTTP on demand? I asked this question on programming.reddit.com yesterday and got some really insightful answers, including Joe Stump from Digg describing how Digg Images uses Danga’s Gearman worker queue.

# 15th December 2007, 12:29 pm / askreddit, danga, digg, gearman, http, joe-stump, queue, reddit, scaling, workers