Simon Willison’s Weblog

djng—a Django powered microframework

19th May 2009

djng is nearly two weeks old now, so it’s about time I wrote a bit about the project.

I presented a keynote at EuroDjangoCon in Prague earlier this month entitled Django Heresies. The talk followed the noble DjangoCon tradition (established last year with the help of Mark Ramm and Cal Henderson) of pointing a spotlight at Django’s flaws. In my case, it was a chance to apply the benefit of hindsight to some of the design decisions I helped make back at the Lawrence Journal-World in 2004.

I took a few cheap shots at things like the {% endifequal %} tag and error silencing in the template system, but the three substantial topics in my talk were class-based generic views (I’m a fan), my hatred of settings.py and my interest in turtles all the way down.

Why I hate settings.py

In the talk, I justified my dislike for settings.py by revisiting the problems behind PHP’s magic quotes feature (finally going away for good in PHP 6). Magic quotes were one of the main reasons I switched to Python from PHP.

My main problem with magic quotes was that they made it extremely difficult to write reusable PHP code. The feature was configured globally, which lead to a quandary. What if you have two libraries, one expecting magic quotes on and the other expecting it off? Your library could check get_magic_quotes_gpc() and stripslashes() from input if the setting was turned on, but this would break in the presence of the common idiom where stripslashes() is applied to all incoming $_GET and $_POST data.

Unfortunately, global settings configured using settings.py have a similar smell to them. Middleware and context processors are the best example here—a specific setting might be needed by just one installed application, but the effects are felt by everything in the system. While I haven’t yet seen two “reusable” Django apps that require conflicting settings, per-application settings are an obvious use case that settings.py fails to cover.

Global impact aside, my bigger problem with settings.py is that I almost always end up wanting to reconfigure them at run-time.

This is possible in Django today, but comes at a price:

• Only some settings can actually be changed at run-time—others (such as USE_I18N) are lazily evaluated once and irreversibly reconfigure parts of Django’s plumbing. Figuring out which ones can be changed requires exploration of Django’s source code.
• If you change a setting, you need to reliably change it back at the end of a request or your application will behave strangely. Uncaught exceptions could cause problems here, unless you remember to wrap dynamic setting changes in a try/finally block.
• Changing a setting isn’t thread-safe (without doing some extra work).

Almost every setting in Django has legitimate use-cases for modification at run-time. Here are just a few examples:

• Requests from mobile phones may need a different TEMPLATE_DIRS setting, to load the mobile-specific templates in preference to the site defaults.
• Some sites offer premium accounts which in turn gain access to more reliable servers. Premium users might get to send e-mail via a separate pool of SMTP servers, for example.
• Some sections of code may want to use a different cache backend, or talk to a different set of memcache servers—to reduce the chance of one rapidly changing component causing other component’s cache entries to expire too early.
• Errors in one area of a site might need to be sent to a different team of developers.
• Admin users might want DEBUG=True, while regular site visitors get DEBUG=False.

Finally, settings.py is behind the dreaded “Settings cannot be imported, because environment variable DJANGO_SETTINGS_MODULE is undefined” exception. Yuck.

Turtles all the way down

The final section of the talk was about turtles. More precisely, it was about their role as an “infinite regression belief about cosmology and the nature of the universe”. I want to apply that idea to Django.

My favourite thing about Django is something I’ve started to call the “Django Contract”: the idea that a Django view is a callable which takes a request object and returns a response object. I want to expand that concept to other parts of Django as well:

• URLconf: takes a request, dispatches based on request.path, returns a response.
• Application: takes a request, returns a response
• Middleware: takes a request, returns a response (conditionally transforming either)
• Django-powered site: hooked in to mod_wsgi/FastCGI/a Python web server, takes a request, returns a response

So instead of a Django site consisting of a settings.py, urls.py and various applications and middlewares, a site would just be a callable that obeys the Django Contract and composes together dozens of other callables.

At this point, Django starts to look a lot like WSGI. What if WSGI and the Django Contract were interchangeable? WSGI is a wrapper around HTTP, so what if that could be swapped in and out (through proxies) as well? Django, WSGI and HTTP, three breeds of turtle arranged on top of each other in various configurations. Turtles all the way down.

djng

djng is my experiment to see what Django would like without settings.py and with a whole lot more turtles. It’s Yet Another Python Microframework.

What’s a microframework? The best examples are probably web.py (itself a result of Aaron Swartz’s frustrations with Django) and Sinatra, my all time favourite example of Ruby DSL design. More recent examples in Python include juno, newf, mnml and itty.

Microframeworks let you build an entire web application in a single file, usually with only one import statement. They are becoming increasingly popular for building small, self-contained applications that perform only one task—Service Oriented Architecture reborn as a combination of the Unix development philosophy and RESTful API design. I first saw this idea expressed in code by Anders Pearson and Ian Bicking back in 2005.

Unlike most microframeworks, djng has a pretty big dependency: Django itself. The plan is to reuse everything I like about Django (the templates, the ORM, view functions, the form library etc) while replacing just the top level plumbing and removing the requirement for separate settings.py and urls.py files.

This is what “Hello, world” looks like in in djng:

import djng

def index(request):
    return djng.Response('Hello, world')

if __name__ == '__main__':
    djng.serve(index, '0.0.0.0', 8888)

djng.Response is an alias for Django’s HttpResponse. djng.serve is a utility function which converts up anything fulfilling the Django Contract in to a WSGI application, then exposes it over HTTP.

Let’s add URL routing to the example:

app = djng.Router(
    (r'^hello$', lambda request: djng.Response('Hello, world')),
    (r'^goodbye$', lambda request: djng.Response('Goodbye, world')),
)

if __name__ == '__main__':
    djng.serve(app, '0.0.0.0', 8888)

The implementation of djng.Router is just a few lines of glue code adding a nicer API to Django’s internal RegexURLResolver class.

Services, not settings

The trickiest problem I still need to solve is how to replace settings.py. A group of developers (including Adrian, Armin, Alex and myself) had an excellent brainstorming session at EuroDjangoCon about this. We realised that most of the stuff in settings.py can be recast as configuring services which Django makes available to the applications it is hosting. Services like the following:

• Caching
• Templating
• Sending e-mail
• Sessions
• Database connection—django.db.connection
• Higher level ORM
• File storage

Each of the above needs to be configured, and each also might need to be reconfigured at runtime. Django already points in this direction by providing hooks for adding custom backends for caching, template loading, file storage and session support. What’s missing is an official way of swapping in different backends at runtime.

I’m currently leaning towards the idea of a “stack” of service implementations, one for each of the service categories listed above. A new implementation could be pushed on to the stack at any time during the Django request/response cycle, and will be automatically popped back off again before the next request is processed (all in a thread-safe manner). Applications would also be able to instantiate and use a particular service implementation directly should they need to do so.

A few days ago I heard about Contextual, which appears to be trying to solve a similar problem. Just a few minutes ago I stumbled across paste.registry’s StackedObjectProxy which seems to be exactly what I’ve been busily reinventing.

My current rough thoughts on an API for this can be found in services_api_ideas.txt. I’m eager to hear suggestions on how to tackle this problem.

djng is very much an experiment at the moment—I wouldn’t suggest building anything against it unless you’re willing to maintain your own fork. That said, the code is all on GitHub partly because I want people to fork it and experiment with their own API concepts as much as possible.

If you’re interested in exploring these concepts with me, please join me on the brand new djng mailing list.