Simon Willison's Weblog Entries

Facebook Usernames and OpenID

2009-06-13T17:01:00Z

Today's launch of Facebook Usernames provides an obvious and exciting opportunity for Facebook to become an OpenID provider. Facebook have clearly demonstrated their interest in becoming the key online identity for their users, and the new usernames feature is their acknowledgement that URL-based identities are an important component of that, no doubt driven in part by Twitter making usernames trendy again.

It's interesting to consider Facebook's history with regards to OpenID and single sign on in general. When I started publicly advocating for OpenID back in 2007, my primary worry was that someone would solve the SSO problem in a proprietary way, irreparably damaging the decentralised nature of the Web - just as Microsoft had attempted a few years earlier with Passport.

When Facebook Connect was announced a year ago it seemed like my worst fears had become realised. Facebook Connect's user experience was a huge improvement over OpenID - with only one provider, the sign in UI could be reduced to a single button. Their use of a popup window for the sign in flow was inspired - various usability studies have since shown that users are much more likely to complete a SSO flow if they can see the site they are signing in to in a background window.

Thankfully, Facebook seem to understand that the industry isn't willing to accept a single SSO provider, no matter how smooth their implementation. Mark Zuckerberg made reassuring noises about OpenID support at both FOWA 2008 and SxSW 2009, but things really stepped up earlier this year when Facebook joined the OpenID Foundation Board (accompanied by a substantial financial donation). Facebook's board representative, Luke Shepherd, is an excellent addition and brings a refreshingly user-centric approach to OpenID. Luke was previously responsible for much of the work on Facebook Connect and has been advocating OpenID inside Facebook for a long time.

Facebook may not have committed to becoming a provider yet (at least not in public), but their decision to become a consumer first is another interesting data point. They may be trying to avoid the common criticism thrown at companies who provide but don't consume - if they're not willing to eat their own dog food, why should anyone else?

At any rate, their consumer implementation is fascinating. It's live right now, even though there's no OpenID login box anywhere to be seen on the site. Instead, Facebook take advantage of the little known checkid_immediate mode. Once you've associated your OpenID with your Facebook account (using the "Linked Accounts" section of the settings pane) Facebook sets a cookie remembering your OpenID provider, which persists even after you log out of Facebook. When you later visit the Facebook homepage, a checkid_immediate request is silently sent to your provider, logging you in automatically if you are already authenticated there.

While it's great to see innovation with OpenID at such a large scale, I'm not at all convinced that they've got this right. The feature is virtually invisible to users (it took me a bunch of research to figure out how to use it) and not at all intuitive - if I've logged out of Facebook, how come visiting the home page logs me straight back in again? I guess this is why Luke is keen on exploring single sign out with OpenID. It sounds like the current OpenID consumer support is principally intended as a developer preview, and I'm looking forward to seeing how they change it based on ongoing user research.

As OpenID provider implementation is an obvious next step that can't be that far off - I wouldn't be surprised to hear an announcement within a month or two.

HTTP redirect codes

As an aside, I decided to check that Facebook were using the correct 3xx HTTP status code to redirect from my old profile page to my new one. I was horrified to discover that they are using a 200 code, followed by a chunk of JavaScript to implement the redirect! The situation for logged out users is better but still fundamentally flawed: if you enable your public search listing (using an option tucked away on www.facebook.com/privacy/?view=search) and curl -i your old profile URL you get a 302 Found, when the correct status code is clearly a 301 Moved Permanently.

One final note: it almost goes without saying, but one of the best things about OpenID is that you can register a real domain name that you can own, instead of just having another URL on Facebook.

djng - a Django powered microframework

2009-05-19T00:13:31Z

djng is nearly two weeks old now, so it's about time I wrote a bit about the project.

I presented a keynote at EuroDjangoCon in Prague earlier this month entitled Django Heresies. The talk followed the noble DjangoCon tradition (established last year with the help of Mark Ramm and Cal Henderson) of pointing a spotlight at Django's flaws. In my case, it was a chance to apply the benefit of hindsight to some of the design decisions I helped make back at the Lawrence Journal-World in 2004.

I took a few cheap shots at things like the {% endifequal %} tag and error silencing in the template system, but the three substantial topics in my talk were class-based generic views (I'm a fan), my hatred of settings.py and my interest in turtles all the way down.

Why I hate settings.py

In the talk, I justified my dislike for settings.py by revisiting the problems behind PHP's magic quotes feature (finally going away for good in PHP 6). Magic quotes were one of the main reasons I switched to Python from PHP.

My main problem with magic quotes was that they made it extremely difficult to write reusable PHP code. The feature was configured globally, which lead to a quandary. What if you have two libraries, one expecting magic quotes on and the other expecting it off? Your library could check get_magic_quotes_gpc() and stripslashes() from input if the setting was turned on, but this would break in the presence of the common idiom where stripslashes() is applied to all incoming $_GET and $_POST data.

Unfortunately, global settings configured using settings.py have a similar smell to them. Middleware and context processors are the best example here - a specific setting might be needed by just one installed application, but the effects are felt by everything in the system. While I haven't yet seen two "reusable" Django apps that require conflicting settings, per-application settings are an obvious use case that settings.py fails to cover.

Global impact aside, my bigger problem with settings.py is that I almost always end up wanting to reconfigure them at run-time.

This is possible in Django today, but comes at a price:

Only some settings can actually be changed at run-time - others (such as USE_I18N) are lazily evaluated once and irreversibly reconfigure parts of Django's plumbing. Figuring out which ones can be changed requires exploration of Django's source code.
If you change a setting, you need to reliably change it back at the end of a request or your application will behave strangely. Uncaught exceptions could cause problems here, unless you remember to wrap dynamic setting changes in a try/finally block.
Changing a setting isn't thread-safe (without doing some extra work).

Almost every setting in Django has legitimate use-cases for modification at run-time. Here are just a few examples:

Requests from mobile phones may need a different TEMPLATE_DIRS setting, to load the mobile-specific templates in preference to the site defaults.
Some sites offer premium accounts which in turn gain access to more reliable servers. Premium users might get to send e-mail via a separate pool of SMTP servers, for example.
Some sections of code may want to use a different cache backend, or talk to a different set of memcache servers - to reduce the chance of one rapidly changing component causing other component's cache entries to expire too early.
Errors in one area of a site might need to be sent to a different team of developers.
Admin users might want DEBUG=True, while regular site visitors get DEBUG=False.

Finally, settings.py is behind the dreaded "Settings cannot be imported, because environment variable DJANGO_SETTINGS_MODULE is undefined" exception. Yuck.

Turtles all the way down

The final section of the talk was about turtles. More precisely, it was about their role as an "infinite regression belief about cosmology and the nature of the universe". I want to apply that idea to Django.

My favourite thing about Django is something I've started to call the "Django Contract": the idea that a Django view is a callable which takes a request object and returns a response object. I want to expand that concept to other parts of Django as well:

URLconf: takes a request, dispatches based on request.path, returns a response.
Application: takes a request, returns a response
Middleware: takes a request, returns a response (conditionally transforming either)
Django-powered site: hooked in to mod_wsgi/FastCGI/a Python web server, takes a request, returns a response

So instead of a Django site consisting of a settings.py, urls.py and various applications and middlewares, a site would just be a callable that obeys the Django Contract and composes together dozens of other callables.

At this point, Django starts to look a lot like WSGI. What if WSGI and the Django Contract were interchangeable? WSGI is a wrapper around HTTP, so what if that could be swapped in and out (through proxies) as well? Django, WSGI and HTTP, three breeds of turtle arranged on top of each other in various configurations. Turtles all the way down.

djng

djng is my experiment to see what Django would like without settings.py and with a whole lot more turtles. It's Yet Another Python Microframework.

What's a microframework? The best examples are probably web.py (itself a result of Aaron Swartz's frustrations with Django) and Sinatra, my all time favourite example of Ruby DSL design. More recent examples in Python include juno, newf, mnml and itty.

Microframeworks let you build an entire web application in a single file, usually with only one import statement. They are becoming increasingly popular for building small, self-contained applications that perform only one task - Service Oriented Architecture reborn as a combination of the Unix development philosophy and RESTful API design. I first saw this idea expressed in code by Anders Pearson and Ian Bicking back in 2005.

Unlike most microframeworks, djng has a pretty big dependency: Django itself. The plan is to reuse everything I like about Django (the templates, the ORM, view functions, the form library etc) while replacing just the top level plumbing and removing the requirement for separate settings.py and urls.py files.

This is what "Hello, world" looks like in in djng:

import djng

def index(request):
    return djng.Response('Hello, world')

if __name__ == '__main__':
    djng.serve(index, '0.0.0.0', 8888)

djng.Response is an alias for Django's HttpResponse. djng.serve is a utility function which converts up anything fulfilling the Django Contract in to a WSGI application, then exposes it over HTTP.

Let's add URL routing to the example:

app = djng.Router(
    (r'^hello$', lambda request: djng.Response('Hello, world')),
    (r'^goodbye$', lambda request: djng.Response('Goodbye, world')),
)

if __name__ == '__main__':
    djng.serve(app, '0.0.0.0', 8888)

The implementation of djng.Router is just a few lines of glue code adding a nicer API to Django's internal RegexURLResolver class.

Services, not settings

The trickiest problem I still need to solve is how to replace settings.py. A group of developers (including Adrian, Armin, Alex and myself) had an excellent brainstorming session at EuroDjangoCon about this. We realised that most of the stuff in settings.py can be recast as configuring services which Django makes available to the applications it is hosting. Services like the following:

Caching
Templating
Sending e-mail
Sessions
Database connection - django.db.connection
Higher level ORM
File storage

Each of the above needs to be configured, and each also might need to be reconfigured at runtime. Django already points in this direction by providing hooks for adding custom backends for caching, template loading, file storage and session support. What's missing is an official way of swapping in different backends at runtime.

I'm currently leaning towards the idea of a "stack" of service implementations, one for each of the service categories listed above. A new implementation could be pushed on to the stack at any time during the Django request/response cycle, and will be automatically popped back off again before the next request is processed (all in a thread-safe manner). Applications would also be able to instantiate and use a particular service implementation directly should they need to do so.

A few days ago I heard about Contextual, which appears to be trying to solve a similar problem. Just a few minutes ago I stumbled across paste.registry's StackedObjectProxy which seems to be exactly what I've been busily reinventing.

My current rough thoughts on an API for this can be found in services_api_ideas.txt. I'm eager to hear suggestions on how to tackle this problem.

djng is very much an experiment at the moment - I wouldn't suggest building anything against it unless you're willing to maintain your own fork. That said, the code is all on GitHub partly because I want people to fork it and experiment with their own API concepts as much as possible.

If you're interested in exploring these concepts with me, please join me on the brand new djng mailing list.

rev=canonical bookmarklet and designing shorter URLs

2009-04-11T17:37:55Z

I've watched the proliferation of URL shortening services over the past year with a certain amount of dismay. I care about the health of the web and try to ensure that URLs I am responsible will last for as long as possible, and I think it's very unlikely that all of these new services will still be around in twenty years time. Last month I suggested that the Internet Archive start mirroring redirect databases, and last week I was pleased to hear that Archiveteam, a different organisation, had already started crawling.

The most recent discussion was kicked off by Joshua Schachter and Dave Winer, and a solution has emerged driven by some lightning fast hacking by Kellan Elliott-McCrea. The idea is simple: sites get to chose their preferred source of shortened URLs (including self-hosted solutions) and specify it from individual pages using <link rev="canonical" href="... shorter URL here ...">.

By hosting their own shorteners, the reliability should match that of the host site - and the amount of damage caused by a major shortener going missing can be dramatically reduced.

I've been experimenting with this new pattern today. Here are a few small contributions to the wider discussion.

A URL shortening bookmarklet

Kellan's rev=canonical service exposes rev=canonical links using a server-side script running on App Engine. An obvious next step is to distil that logic in to a bookmarklet. I decided to combine the rev=canonical logic with my json-tinyurl web service (also on App Engine), which allows browsers to lookup or create TinyURLs using a cross-domain JSONP request. The resulting bookmarklet will display the site's rev=canonical link if it exists, or create and display a TinyURL link otherwise:

Bookmarklet: Shorten (drag to your browser toolbar)

You can also grab the uncompressed source code.

Designing short URLs

I've also implemented rev=canonical on this site. I ended up buying a new domain for this, since simonwillison.net is both difficult to spell and 17 characters long. I ended up going with swtiny.eu - 9 characters, and keeping tiny in the domain helps people guess the nature of the site from just the URLs it generates. Be warned: the DNS doesn't appear to have finished resolving yet.

For the path component, I turned to a variant of base 62 encoding. Decimal integers are represented using 10 digits (0-9), but base 62 uses those digits plus the letters of the alphabet in both lower and upper case. A 13 character integer such as 7250397214971 compresses down to just 8 characters (CDeIPpOD) using base62. My baseconv.py module implements base62, among others. I considered using base 57 by excluding o, O, 0, 1 and l as being too easily confused but decided against it.

This site has three key types of content: entries, blogmarks and quotations. Each one is a separate Django model, and hence each has its own underlying database table and individual ID sequence. Since the IDs overlap, I need a way of separating out the shortened URLs for each content type.

I decided to spend a byte on namespacing my shortened URLs. A prefix of E means an entry, Q means a quotation and B means a blogmark. For example:

http://swtiny.eu/EZ8: Entry with ID 1584
http://swtiny.eu/BBEQ: Blogmark with ID 4108
http://swtiny.eu/QE5: Quotation with ID 279

By using upper case letters for the prefixes, I can later define custom paths starting with a lower case letter. I also have another 23 upper case prefix letters reserved in case I need them.

I asked on Twitter and consensus opinion was that a 301 permanent redirect was the right thing to do (as opposed to a 302), both for SEO reasons and because the content will never exist at the shorter URL.

Implementation using Django and nginx

I run all of my Django sites using Apache and mod_wsgi, proxied behind nginx. Each site gets an Apache running on a high port, and nginx deals with virtual host configuration (proxying each domain to a different Apache backend) and static file serving. I didn't want to set up a full Django site just to run swtiny.eu, especially since my existing blog engine was required in order to resolve the shortened URLs.

Instead, I implemented the shortened URL direction as just another view within my existing site: http://simonwillison.net/shorter/EZ8. I then configured nginx to invisibly requests to swtiny.eu through to that URL. The correct incantation took a while to figure out, so here's the relevant section of my nginx.conf:

server {
    listen 80;
    server_name www.swtiny.eu swtiny.eu;
    location / {
        rewrite (.*) /shorter$1 break;
        proxy_pass http://simonwillison.net;
        proxy_redirect off;
    }
}

proxy_redirect off is needed to prevent nginx from replacing simonwillison.net in the resulting location header with swtiny.eu. My Django view code is relatively shonky, but if you're interested you can find it here.

The nice thing about this approach is that it makes it trivial to add custom URL shortening domains to other projects - a quick view function and a few lines of nginx configuration are all that is needed.

Update: The bookmarklet now supports the rev attribute on A elements as well - thanks for the suggestion, Jeremy.

List of SxSW 2009 panels with "social" in the title

2009-03-14T23:02:54Z

A Hard Sell? Social Media & Your Boss
Can Social Media End Racism?
Digital Urbanites: How To Become Part of the New Social Capital
The Future Of Social Networks
How Social Networks Are Killing the Revolution
Making Whuffie: Raising Social Capital in Online Communities
The Mix at Six Hosted by Social Media Group
Mobile Social SXSW BBQ
My Boss Doesn't Get It: Championing Social Media to the Man
PBS' Interactive Social Media & Online Video Studio
The Search for a More Social Web
Security for the Social Set
Social Engineering: Scam Your Way Into Anything or From Anybody
Social Gamers: Away From the Keyboard
Social Media For Social Good
Social Media Marketing
Social Media Marketing: An Hour a Day
Social Media Nonprofit ROI Poetry Slam
Social Media: If You Liked it, Then You Should Have Put a Digg on it...
Social Networking in Health: e-Patients, Data & Privacy
Social Patterns and Antipatterns For the Win
Suxorz '09: The Ten Worst Social Media Campaigns
Twitter for Marketers: Is It Still Social Media?
Using GPS & Location to Enhance Social Networking
Using the New Digital Social Media to Accelerate Sustainability

A few notes on the Guardian Open Platform

2009-03-10T14:28:39Z

This morning we launched the Guardian Open Platform at a well attended event in our new offices in Kings Place. This is one of the main projects I've been helping out with since joining the Guardian last year, and it's fantastic to finally have it out in the open.

There are two components to the launch today: the Content API and the Data Store. I'll describe the Data Store first as it deserves not to get buried in the discussion about its larger cousin.

The Data Store

Simon Rogers is the Guardian news editor who is principally responsible for gathering data about the world. If you ever see an infographic in the paper, the chances are Simon had a hand in researching the data for it. His delicious feed is a positive gold mine.

As of today, a sizeable portion the data he collects for the newspaper will also be published online. As a starting point, we're publishing over 80 data sets, all using Google Spreadsheets which means it's all accessible through the Spreadsheets Data API.

Here's Simon's take on it, from Welcome to the Datablog:

Everyday we work with datasets from around the world. We have had to check this data and make sure it's the best we can get, from the most credible sources. But then it lives for the moment of the paper's publication and afterward disappears into a hard drive, rarely to emerge again before updating a year later.

So, together with its companion site, the Data Store – a directory of all the stats we post – we are opening up that data for everyone. Whenever we come across something interesting or relevant or useful, we'll post it up here and let you know what we're planning to do with it.

It's worth spending quite a while digging around the data. Most sets come with a full description, including where the data was sourced from. New data sets will be announced on the Datablog, which is cleverly subtitled "Facts are sacred".

The Content API

The Content API provides REST-ish access to over a million items of content, mostly from the last decade but with a few gems that are a little bit older. Various types of content are available - article is the most common, but you can grab information (though not necessarily content) about audio, video, galleries and more. You can retrieve 50 items at a time, and pagination is unlimited (provided you stay below the API's rate limit).

Articles are provided with their full body content, though this does not currently include any HTML tags (a known issue). It's a good idea to review our terms and conditions, but you should know that if you opt to republish our article bodies on your site we may ask you to include our ads alongside our content in the future.

We serve 15 minute HTTP cache headers, but you are allowed to store our content for up to 24 hours. You really, really don't want to store content for longer than that, as in addition to violating our T&Cs you might find yourself inadvertently publishing an article that has been retracted for legal reasons. UK libel laws can be pretty scary.

In addition to regular search, you can also filter our content using tags. Tags are a core aspect of the Guardian's R2 platform, being used for keywords, contributors, "series" (used to implement blogs), content types and more. Every item returned by the API includes tags, and the tags can be used to further filter the results.

We also return a list of filters at the bottom of each page of search results showing the tags that could be used to filter that result set, ordered by the number of results (you may have seen this feature referred to as faceted search or guided navigation). Handy tip: you can use ?count=0 in your search API key to turn off results entirely and just get back the filters section. The race is on to be first to release a tag relationship browser based on this feature.

API responses can be had in custom XML, JSON or Atom. The Atom format is the least mature at the moment, and we'd welcome suggestions for improving it from the community.

I released a Python client library for the API this morning, and we also have libraries for Ruby, Java and PHP.

We also have an API Explorer (written in JavaScript and jQuery, hosted on the same domain as the API so that it can make Ajax requests) but you'll need an API key to try it out.

The bad news

The response to the API release has been terrific (check out what Tom Watson had to say), but as a result it's likely that API key provisions will be significantly lower than the overall demand for them. Please bear with us while we work towards a more widely accessible release.

Pragmatism, purity and JSON content types

2009-02-06T10:19:55Z

I started a conversation about this on Twitter the other day, but Twitter is a horrible place to have an archived discussion so I'm going to try again here.

If you're producing a JSON API for other people to use (as opposed to an API that's only really meant for your own local Ajax responses), you need to decide which Content-Type to use. The best option is not entirely obvious.

RFC 4672 defines JSON and reserves application/json as the preferred media type. The problem is that most browsers will prompt you to download the file rather than displaying it inline (as they would for text/plain or application/javascript). One of my favourite qualities of REST-style APIs is that they enable exploration and debugging using just a browser - using application/json throws a big, frustrating road block in the way. There are ways of telling your browser to treat application/json in the same way as text/plain but that doesn't really help you if your aim is to create an API that's easy for other developers to use.

It's also worth mentioning that if you are returning JSONP (with an extra callback function wrapped around the JSON response to enable the dynamic script tag hack) you HAVE to serve as application/javascript - otherwise the script you are providing won't be executed by the browser. Don't forget to include charset=UTF8 as well (for both types of response).

So, it's pragmatism v.s. purity. The correct thing to do is to return application/json, but doing so makes your API harder for developers to use.

In a brief, non-comprehensive review of some existing JSON APIs (FriendFeed, Flickr, Google Social Graph etc) I couldn't find any that were using application/json, presumably for this exact reason.

Using the Accept: header

The Accept: header is one of my least favourite parts of HTTP. I like to be confident that if I send a URL to someone, they'll get back exactly the same bytes as I did when I retrieved it myself (I distrust language negotiation for the same reason). However, a number of people suggested it on Twitter and it looks like it could be a useful solution to this problem.

I'm currently considering the following: ONLY use the application/json Content-Type in reply to requests that include application/json in their Accept header - essentially allowing clients that care about the correct content type to opt-in to receiving it. Everyone else (browsers included) gets application/javascript, which is less correct (though not an all-out lie, since JSON is a subset of JavaScript) but solves the usability problem.

A couple of things worry me about this. Firstly, is this a reasonable thing to use Accept for? Secondly, is there a chance that browsers might add application/json to their Accept header at some point in the future? Safari currently sends text/xml,application/xml,application/xhtml+xml,text/html; q=0.9,text/plain; q=0.8,image/png,*/*; q=0.5 while Firefox sends text/html,application/xhtml+xml,application/xml; q=0.9,*/*; q=0.8. Would it be smarter to look for */* and serve the incorrect Content-Type to those requests and the correct one to everything else?

An alternative is to simply allow people to specify "JSON with a browsable Content-Type" as an alternative format option, or to enable a "pretty=1" query string argument which returns the response as text/plain and potentially pretty prints it as well. I haven't yet decided if this is better than messing around with the Accept header.

Rate limiting with memcached

2009-01-07T22:27:08Z

On Monday, several high profile "celebrity" Twitter accounts started spouting nonsense, the victims of stolen passwords. Wired has the full story - someone ran a dictionary attack against a Twitter staff member, discovered their password and used Twitter's admin tools to reset the passwords on the accounts they wanted to steal.

The Twitter incident got me thinking about rate limiting again. I've been wanting a good general solution to this problem for quite a while, for API projects as well as security. Django Snippets has an answer, but it works by storing access information in the database and requires you to run a periodic purge command to clean up the old records.

I'm strongly averse to writing to the database for every hit. For most web applications reads scale easily, but writes don't. I also want to avoid filling my database with administrative gunk (I dislike database backed sessions for the same reason). But rate limiting relies on storing state, so there has to be some kind of persistence.

Using memcached counters

I think I've found a solution, thanks to memcached and in particular the incr command. incr lets you atomically increment an already existing counter, simply by specifying its key. add can be used to create that counter - it will fail silently if the provided key already exists.

Let's say we want to limit a user to 10 hits every minute. A naive implementation would be to create a memcached counter for hits from that user's IP address in a specific minute. The counter key might look like this:

ratelimit_72.26.203.98_2009-01-07-21:45

Increment that counter for every hit, and if it exceeds 10 block the request.

What if the user makes ten requests all in the last second of the minute, then another ten a second later? The rate limiter will let them off. For many cases this is probably acceptable, but we can improve things with a slightly more complex strategy. Let's say we want to allow up to 30 requests every five minutes. Instead of maintaining one counter, we can maintain five - one for each of the past five minutes (older counters than that are allowed to expire). After a few minutes we might end up with counters that look like this:

ratelimit_72.26.203.98_2009-01-07-21:45 = 13
ratelimit_72.26.203.98_2009-01-07-21:46 = 7
ratelimit_72.26.203.98_2009-01-07-21:47 = 11

Now, on every request we work out the keys for the past five minutes and use get_multi to retrieve them. If the sum of those counters exceeds the maximum allowed for that time period, we block the request.

Are there any obvious flaws to this approach? I'm pretty happy with it - it cleans up after itself (old counters quietly expire from the cache), it shouldn't use much resources (just five active cache keys per unique IP address at any one time) and if the cache is lost the only snag is that a few clients might go slightly over their rate limit. I don't think it's possible for an attacker to force the counters to expire early.

An implementation for Django

I've put together an example implementation of this algorithm using Django, hosted on GitHub. The readme.txt file shows how it works - basic usage is via a simple decorator:

from ratelimitcache import ratelimit

@ratelimit(minutes = 3, requests = 20)
def myview(request):
    # ...
    return HttpResponse('...')

Python decorators are typically functions, but ratelimit is actually a class. This means it can be customised by subclassing it, and the class provides a number of methods designed to be over-ridden. I've provided an example of this in the module itself - ratelimit_post, a decorator which only limits on POST requests and can optionally couple the rate limiting to an individual POST field. Here's the complete implementation:

class ratelimit_post(ratelimit):
    "Rate limit POSTs - can be used to protect a login form"
    key_field = None # If provided, this POST var will affect the rate limit
    
    def should_ratelimit(self, request):
        return request.method == 'POST'
    
    def key_extra(self, request):
        # IP address and key_field (if it is set)
        extra = super(ratelimit_post, self).key_extra(request)
        if self.key_field:
            value = sha.new(request.POST.get(self.key_field, '')).hexdigest()
            extra += '-' + value
        return extra

And here's how you would use it to limit the number of times a specific IP address can attempt to log in as a particular user:

@ratelimit_post(minutes = 3, requests = 10, key_field = 'username')
def login(request):
    # ...
    return HttpResponse('...')

The should_ratelimit() method is called before any other rate limiting logic. The default implementation returns True, but here we only want to apply rate limits to POST requests. The key_extra() method is used to compose the keys used for the counter - by default this just includes the request's IP address, but in ratelimit_post we can optionally include the value of a POST field (for example the username). We could include things like the request path here to apply different rate limit counters to different URLs.

Finally, the readme.txt includes ratelimit_with_logging, an example that over-rides the disallowed() view returned when a rate limiting condition fails and writes an audit note to a database (less overhead than writing for every request).

I've been a fan of customisation via subclassing ever since I got to know the new Django admin system, and I've been using it in a bunch of projects. It's a great way to create reusable pieces of code.

DjangoCon and PyCon UK

2008-09-15T15:20:20Z

September is a big month for conferences. DjangoCon was a weekend ago in Mountain View (forcing me to miss both d.Construct and BarCamp Brighton), PyCon UK was this weekend in Birmingham, I'm writing this from @media Ajax and BarCamp London 5 is coming up over another weekend at the end of this month. As always, I've been posting details of upcoming talks and notes and materials from previous ones on my talks page.

DjangoCon went really, really well. Huge thanks to conference chair Robert Lofthouse for pulling it all together in just two months and Leslie Hawthorne for making it all happen from Google's end. Google's facitilies were superb: the AV team were the best I've ever worked with and an army of Google volunteers made sure everything went smoothly. It's hard to see how it could have gone better; the principle complaint we got was that at only two days it was hard to justify the travel, something which future DjangoCons will definitely address.

Every session was recorded and the videos ~~should be going up on YouTube shortly~~ are now up on YouTube. For the impatient, you can subscribe to an Atom feed of a YouTube search for "DjangoCon". I recommend starting with Cal Henderson's keynote "Why I hate Django" which was both funny and insightful in equal parts. Malcolm's talk on ORM internals was another personal favourite.

PyCon UK was the second I've attended, but last year I only stayed for the first day. This time I stuck around and was enormously impressed by the grassroots feel of the conference and the enthusiastic atmosphere. I presented a tutorial on extending the Django admin and a lightning talk on Zeppelins, prepared two hours in advance after Jacob mentioned that the lightning talks were tending too much towards the technical side. It went down very well; I'm tempted to extend it to a half hour session for BarCamp London.

Unlike most conferences I attend, PyCon tickets included a sit-down dinner for all attendees complete with a "dramatic lecture" on the Lunar Society presented by Andrew Lound. This was a great fit for the conference, both for the Birmingham connection and the many analogies to the modern open source community - loose collaboration, patent concerns and what you might call an 18th century equivalent of the modern hacker ethic.

Next year the PyCon UK team will be hosting EuroPython, and I'm certain they'll do an excellent job of it. Meanwhile, Rob has already started making plans for a Euro DjangoCon in around six months time, probably taking place in Prague.

Announcing dmigrations

2008-09-03T19:17:01Z

The team at Global Radio (formerly GCap Media) is the largest group of Django developers I've personally worked with, consisting of 14 developers split into two scrum teams, all contributing to the same overall codebase.

Working with that many developers makes smart tools and processes essential, and in some cases we've had to develop our own. Today, we're releasing one of them as an open source project.

dmigrations is a Django migrations tool. It addresses a common problem in Django development: if you change a model after creating the database tables for it with syncdb, how do you reflect those changes in your database tables without blowing away your existing data and starting again from scratch?

django-evolution attempts to address this problem the clever way, by detecting changes to models that are not yet reflected in the database schema and figuring out what needs to be done to bring the two back in sync. In contrast, dmigrations takes the stupid approach: it requires you to explicitly state the changes in a sequence of migrations, which will be applied in turn to bring a database up to the most recent state that reflects the underlying models.

This means extra work for developers who create migrations, but it also makes the whole process completely transparent - for our projects, we decided to go with the simplest system that could possibly work.

The interface to dmigrations is a pair of custom Django commands. The first, ./manage.py dmigrate, provides a set of command for listing, applying and unapplying (reverting) migrations. This entirely replaces Django's syncdb command.

The second, ./manage.py dmigration, provides commands for code-generating new migrations. It turns out that most migrations fit a set of common patterns: add a new table, add the tables for a new Django application, add a column to an existing table, add an index. These common cases are handled by dmigration; if you want to do something more complex (rename a column while transforming its data for example) you'll need to write a custom migration class.

The dmigrations tutorial provides a full introduction to both of these commands, as well as hints on writing your own custom migrations. Since migrations are just classes, one of our hopes is that external developers will write extra migration classes for operations like "rename column" - things that currently require a one-off custom migration.

dmigrations is actually the third iteration of our in-house migrations system. The first, smigrations, was designed to do the least amount of work possible to give us a controlled way of applying changes to our database schemas. The 's' stood for 'simple'. The second version (dmigrations) written by Tomasz Wegrzanowski consisted of a major upgrade to smigrations that addressed many of the frustrations we found when using it with branched development, in particular the problem of migrations in two branches conflicting with each other. The 'd' stood for 'distributed'.

Version three, released today, is my refactoring of dmigrations to de-couple it from the rest of our codebase. I've also stubbed out hooks for adding support for alternative database engines; dmigrations only supports MySQL out of the box, but I'm keen on getting it working with other databases now that it's out in the wild. Patches welcome!

How does this fit in with South and django-evolution?

That's an excellent question. We'll be discussing all three systems on the schema evolution panel at DjangoCon this weekend. I would love to see co-operation between the projects; at the very least I'd like to see the emergence of a standard Django-style abstraction library for create/alter table statements (something we punted on entirely with dmigrations). You'll certainly be hearing a lot more about migrations in Django after the conference.

Back to full-time employment

2008-08-22T17:26:39Z

I've been freelance for a year and a half now, and it's been a great deal of fun. For me, being freelance meant having the freedom to pursue all sorts of different interests - technical writing, public speaking, Django, OpenID, JavaScript - and the opportunity to work with some really fantastic people.

It was going to take a very special opportunity to pull me back in to full-time employment, but I believe I've found that opportunity at the Guardian. I'll be joining them full time (well, four days a week) in mid-October as a software architect, collaborating with their development team on some ambitious API projects. The Guardian have access to a lot of interesting data and I can't wait to get stuck in to it. Since they're a newspaper, it shouldn't be a surprise that they scooped me to the story.

I'll be particularly sorry to say good-bye to the outstanding team I've been working with at GCap. I'm looking forward to talking about some of the things we've been working together over the next few weeks.

The point of "Open" in OpenID

2008-06-24T08:12:23Z

TechCrunch report that Microsoft are accepting OpenID for their new HealthVault site, but with a catch: you can only use OpenIDs from two providers: Trustbearer (who offer two-factor authentication using a hardware token) and Verisign. "Whatever happened to the Open in OpenID?", asks TechCrunch's Jason Kincaid.

Microsoft's decision is a beautiful example of the Open in action, and I fully support it.

You have to remember that behind the excitement and marketing OpenID is a protocol, just like SMTP or HTTP. All OpenID actually provides is a mechanism for asserting ownership over a URL and then "proving" that assertion. We can build a pyramid of interesting things on top of this, but that assertion is really all OpenID gives us (well, that and a globally unique identifier). In internet theory terms, it's a dumb network: the protocol just concentrates on passing assertions around; it's up to the endpoints to set policies and invent interesting applications.

Open means that providers and consumers are free to use the protocol in whatever way they wish. If they want to only accept OpenID from a trusted subset of providers, they can go ahead. If they only want to pass OpenID details around behind the corporate firewall (great for gluing together an SSO network from open-source components) they can knock themselves out. Just like SMTP or HTTP, the protocol does not imply any rules about where or how it should be used.

HealthVault have clearly made this decision due to security concerns - not over the OpenID protocol itself, but the providers that their users might choose to trust. By accepting OpenID on your site you are outsourcing the security of your users to an unknown third party, and you can't guarantee that your users picked a good home for their OpenID. If you're a bank or a healthcare provider that's not a risk you want to take; whitelisting providers that you have audited for security means you don't have to rule out OpenID entirely.

I have a very simple rule of thumb for whether or not a site should consider whitelisting OpenID providers: does the site offer a "forgotten password" feature that e-mails the user a login token? If it does, then the owners have already made the decision to outsource the security of their users to whoever they picked as an e-mail provider. If they don't (banks are a good example here) they should continue that policy decision and consider using an OpenID provider whitelist.

I've been using the example of banks potentially accepting OpenID only from security audited providers in my talks on OpenID for at least the past year. Now I can finally provide a real-world example.

Debugging Django

2008-05-22T00:35:40Z

I gave a talk on Debugging Django applications at Monday's inaugural meeting of DJUGL, the London Django Users Group. I wanted to talk about something that wasn't particularly well documented elsewhere, so I pitched the talk as "Bug Driven Development" - what happens when Test Driven Development goes the way of this unfortunate pony.

The slides are up on SlideShare, but don't provide quite enough context so I'm going to cover the tips in full here.

Making the most of the error page

Django's default error page is great - it provides a detailed traceback with local variables, lets you expand out the lines of code around the problem, provides a plain text exception suitable for e-mailing to colleagues and even a one-click button to send details to http://dpaste.com/ so you can go and talk about the error on IRC. It also serves the same purpose as phpinfo() - it shows you your application's settings, the GET, POST and COOKIE data from the request and the all important META fields assembled from the HTTP environment (great for remembering how to miss-spell HTTP_REFERER).

Useful tip number one is that you can trigger the error page from any view just by adding the following line:

assert False

You can serve up an expression with the assertion as well; it will be displayed at the top of the error page:

assert False, request.GET

One particularly useful place to use this is when you are building a complex form. If you want to see the data that was submitted, drop an assert False in to the view that the form targets and use the error page to inspect the data.

Logging to the development server console

If you want to know what's going on inside a view, the quickest way is to drop in a print statement. The development server outputs any print statements directly to the terminal; it's the server-side alternative to a JavaScript alert().

If you want to be a bit more sophisticated with your logging, it's worth turning to Python's logging module (part of the standard library). You can configure it in your settings.py:

import logging
logging.basicConfig(
    level = logging.DEBUG,
    format = '%(asctime)s %(levelname)s %(message)s',
)

Then call it from any of your views:

def my_view(request):
    import logging
    logging.debug("A log message")
    ...

Again, this will log things to the terminal where the development server is running. If you want to log things to a file you can do so by extending the basicConfig call:

logging.basicConfig(
    level = logging.DEBUG,
    format = '%(asctime)s %(levelname)s %(message)s',
    filename = '/tmp/myapp.log',
    filemode = 'w'
)

You can then use tail -f /tmp/myapp.log to see log lines being appended to that file in real-time. This can be used in production as well as development.

The above just scratches the surface of Python's logging module; with a bit of digging around in the documentation you can use it to rotate log files, send log messages over the network and even POST log events to an HTTP server somewhere.

Often you find yourself dealing with an error that only occurs in certain circumstances - a function might be called from dozens of different places in your program but only runs in to trouble in a very specific case. You can use the traceback module to log the current stack, which will allow you to tell how a function was called when something went wrong:

import logging, traceback, pprint

def my_buggy_function(arg):
    ...
    if error_condition:
        stack = pprint.pformat(traceback.extract_stack())
        logging.debug('An error occurred: %s' % stack)

The tuple returned by traceback.extract_stack() includes line numbers, function names and paths to Python files so you can use it to reconstruct a good amount of information about your program.

Using the debugger

By far the most powerful weapon in my debugging toolkit is the Python debugger, pdb. Again, this ships with the standard library so there's nothing extra to install. pdb is a command line debugger (if you want a GUI options include PyEclipse and Komodo, but I haven't used either myself). There are a bunch of ways to activate pdb, but the most straight forward is to simply drop the following line directly in to a Django view function:

import pdb; pdb.set_trace()

If you try to load that page in your browser, the browser will hang - the page will appear to be loading extremely slowly. What's actually happened is the developer server has paused execution and thrown up the pdb interface - you can switch over to your console and start interacting directly with the server mid view.

Did I mention you should never, ever leave this on in production?

So, you've got a hung development server and a pdb prompt. What can you do with it? The answer is pretty much anything. I won't provide a full pdb tutorial here (this is a good introduction), but the commands I find most useful are the following:

list: Shows the lines of source code around your current point of execution. You can run it multiple times to increase the amount of source code displayed.
n: Execute the next line
s: Same as n, but steps in to any functions that are called. You can quickly get lost in a twisty maze of code with this command, but that's OK because...
r: Continues execution until the current function returns
u: Goes UP one level in the stack - so you can see the function that called the function you are currently in
d: Goes DOWN again
locals(): not a pdb command, but handy for seeing what's in your current scope

The pdb docs have a full list of commands.

The pdb prompt doubles up as a full Python interactive shell, so you can not only access variables but you can modify them, call functions and generally mess around with the internals of your application as much as you like, while it's running. It's kind of a poor man's imitation of being a Smalltalk developer.

Remember though, the whole time you're messing around in pdb your browser is still stuck there, waiting for the HTTP request to come back. If you hit "c" (for continue) your application will kick in again, the request will be served and your browser will breathe a sigh of relief.

Thankfully you don't have to use pdb in a way that freezes your development server; it also works great in the interactive shell. If you've got a buggy function, one way to explore it is to run it interactively, then use the following idiom:

>>> def function_that_raises_an_exception():
...   assert False
... 
>>> function_that_raises_an_exception()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in function_that_raises_an_exception
AssertionError
>>> import pdb; pdb.pm()
> <stdin>(2)function_that_raises_an_exception()
(Pdb)

pdb.pm() stands for post-mortem, and is probably my favourite feature of the debugger - it lets you jump back in to debug the most recently raised exception, even if you hadn't imported pdb at the time the exception was raised.

One last pdb tip: you can use it to debug Python command line scripts such as Django's custom ./manage.py commands. The trick is to run the script like this:

python -i manage.py buggy_command

The -i argument causes Python to drop in to the interactive prompt after executing the script. If the script raised an exception, you can then use pdb.pm() to debug it.

Handling errors in production

Django's default behaviour in production (that is, when the DEBUG setting is set to False) is to e-mail exception reports to anyone listed in the ADMINS section. You can also turn on e-mail reports on every 404 error with the SEND_BROKEN_LINK_EMAILS setting, which will send them to addresses in the MANAGERS setting. As far as I know these settings don't do anything else - they're a pretty ancient bit of Django.

On a high traffic site you probably don't want to be e-mailed on every server error. One neat alternative is David Cramer's django-db-log, which logs exceptions to a database table. It cleverly uses an MD5 hash of the traceback to aggregate many reports of the same error. More importantly though, it acts as a really straight forward example of how to use Django middleware's process_exception hook to roll your own error reporting. Take a look at the code to see how simple this is.

More useful middleware

In the talk I demoed a couple of other handy pieces of middleware. The first was the ProfilerMiddleware (one of several profiling tools on Django Snippets) which allows you to add ?prof to the end of any URL to see the output of Python's cProfile module run against that request. The second is one that I've just released: DebugFooter, which adds a footer showing exactly which templates were loaded from where (handy for debugging complex template paths) as well as every executed SQL query and how long each one took.

Abusing the test client

A final tip for exploring your application interactively is to learn to use Django's TestClient. Although designed for use in unit tests, this tool is equally useful for use at the interactive prompt. It allows you to simulate an in-process request against your application from within your Python code. Here's an example:

>>> from django.test.client import Client
>>> c = Client()
>>> response = c.get("/") # The homepage
>>> response
<django.http.HttpResponse object at 0x2300470>
>>> print response
Vary: Cookie
Content-Type: text/html; charset=utf-8

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
    "http://www.w3.org/TR/html4/strict.dtd">
<html>
...

The response object you get back is the HttpResponse returned by the view, ready to be explored interactively.

There's another function from the unit testing tools that can help with interactively exploring an application: setup_test_environment(). This function monkey-patches in some additional hooks used by the unit tests, including one that intercepts template render calls and adds information on them to the request object. Here's an example:

>>> from django.test.utils import setup_test_environment
>>> setup_test_environment()
>>> from django.test.client import Client
>>> c = Client()
>>> response = c.get("/")
>>> response.template
[<django.template.Template object at 0x2723dd0>,
 <django.template.Template object at 0x2723f30>,
 <django.template.Template object at 0x273ee10>]
>>> response.context
[ list of Context objects ]

This allows you to explore not just the HTML returned by a view, but also the templates and contexts that were used to render it.

Your tips welcome

If you have any useful tips on debugging Django applications, please share them in the comments on this entry.

jQuery style chaining with the Django ORM

2008-05-01T12:31:17Z

Django's ORM is, in my opinion, the unsung gem of the framework. For the subset of SQL that's used in most web applications it's very hard to beat. It's a beautiful piece of API design, and I tip my hat to the people who designed and built it.

Lazy evaluation

If you haven't spent much time with the ORM, two key features are lazy evaluation and chaining. Consider the following statement:

entries = Entry.objects.all()

Assuming you have created an Entry model of some sort, the above statement will create a Django QuerySet object representing all of the entries in the database. It will not result in the execution of any SQL - QuerySets are lazily evaluated, and are only executed at the last possible moment. The most common situation in which SQL will be executed is when the object is used for iteration:

for entry in entries:
    print entry.title

This usually happens in a template:

<ul>
{% for entry in entries %}
  <li>{{ entry.title }}</li>
{% endfor %}
</ul>

Lazy evaluation works nicely with template fragment caching - even if you pass a QuerySet to a template it won't be executed if the fragment it is used in can be served from the cache.

You can modify QuerySets as many times as you like before they are executed:

entries = Entry.objects.all()
today = datetime.date.today()
entries_this_year = entries.filter(
    posted__year = today.year
)
entries_last_year = entries.filter(
    posted__year = today.year - 1
)

Again, no SQL has been executed, but we now have two QuerySets which, when iterated, will produce the desired result.

Chaining

Chaining comes in when you want to apply multiple modifications to a QuerySet. Here are blog entries from 2006 that weren't posted in January:

Entry.objects.filter(
    posted__year = 2006
).exclude(posted__month = 1)

And here's entries from that year posted to the category named "Personal", ordered by title:

Entry.objects.filter(
    posted__year = 2006
).filter(
    category__name = "Personal"
).order_by('title')

The above can also be expressed like this:

Entry.objects.filter(
    posted__year = 2006,
    category__name = "Personal"
).order_by('title')

Chaining in jQuery

The parallels to jQuery are pretty clear. The jQuery API is built around chaining, and the jQuery animation library even uses a form of lazy evaluation to automatically queue up effects to run in sequence:

jQuery('div#message').addClass(
	'borderfade'
).animate({
   'borderWidth': '+10px'
}, 1000).fadeOut();

One of the neatest things about jQuery is the plugin model, which takes advantage of JavaScript's prototype inheritance and makes it trivially easy to add new chainable methods. If we wanted to package the above dumb effect up as a plugin, we could do so like this:

jQuery.fn.dumbBorderFade = function() {
    return this.addClass(
        'borderfade'
    ).animate({
       'borderWidth': '+10px'
    }, 1000).fadeOut();
};

Now we can apply it to an element like so:

jQuery('div#message').dumbBorderFade();

Custom QuerySet methods in Django

Django supports adding custom methods for accessing the ORM through the ability to implement a custom Manager. In the above examples, Entry.objects is the Manager. The downside of this approach is that methods added to a manager can only be used at the beginning of the chain.

Luckily, Managers also provide a hook for returning a custom QuerySet. This means we can create our own QuerySet subclass and add new methods to it, in a way that's reminiscent of jQuery:

from django.db import models
from django.db.models.query import QuerySet
import datetime

class EntryQuerySet(QuerySet):
    def on_date(self, date):
        next = date + datetime.timedelta(days = 1)
        return self.filter(
            posted__gt = date,
            posted__lt = next
        )

class EntryManager(models.Manager):
    def get_query_set(self):
        return EntryQuerySet(self.model)

class Entry(models.Model):
    ...
    objects = EntryManager()

The above gives us a new method on the QuerySets returned by Entry.objects called on_date(), which lets us filter entries down to those posted on a specific date. Now we can run queries like the following:

Entry.objects.filter(
    category__name = 'Personal'
).on_date(datetime.date(2008, 5, 1))

Reducing the boilerplate

This method works fine, but it requires quite a bit of boilerplate code - a QuerySet subclass and a Manager subclass plus the wiring to pull them all together. Wouldn't it be neat if you could declare the extra QuerySet methods inside the model definition itself?

It turns out you can, and it's surprisingly easy. Here's the syntax I came up with:

from django.db.models.query import QuerySet

class Entry(models.Model):
   ...
   objects = QuerySetManager()
   ...
   class QuerySet(QuerySet):
       def on_date(self, date):
           return self.filter(
               ...
           )

Here I've made the custom QuerySet class an inner class of the model definition. I've also replaced the default manager with a QuerySetManager. All this class does is return the QuerySet inner class for the current model from get_query_set. The implementation looks like this:

class QuerySetManager(models.Manager):
    def get_query_set(self):
        return self.model.QuerySet(self.model)

I'm pretty happy with this; it makes it trivial to add custom QuerySet methods and does so without any monkeypatching or deep reliance on Django ORM internals. I think the ease with which this can be achieved is a testament to the quality of the ORM API.

wikinear.com, OAuth and Fire Eagle

2008-03-22T14:34:53Z

I'm pleased to announce wikinear.com. It's a simple site that does just one thing: show you a list of the five Wikipedia pages that are geographically closest to your current location. It's designed (or not-designed) to be used mainly from mobile phones.

You'll need a Fire Eagle invitation code to use the site. ~~I've got four spare; the first four comments to ask for one can have them~~ my invites are all accounted for. If you don't have a Fire Eagle account you'll have to make do with this screenshot instead.

The idea for the site came from living in Oxford for a year. The city is full of beautiful old historic buildings (many of them colleges), but very few of them are labelled or signposted. With wikinear.com and a GPS hooked up to Fire Eagle, I can pull out my phone and see a list of the closest points of interest, plotted on a handy map.

Under the hood the site combines a number of interesting technologies: OAuth, Fire Eagle, GeoNames and the new Google Static Maps API.

OAuth

OAuth was originally designed to solve a problem with OpenID: in an authentication protocol based on browser redirects, how do you authenticate a desktop or command-line application? As it turned out, the solution to that problem solved a bunch of other problems that are unrelated to OpenID, so OAuth now exists as very much its own thing. In essence, it lets users delegate permission to perform actions on their behalf, without having to hand their regular authentication credentials (e.g. username and password) over to a third-party piece of software.

If you've ever used a Flickr application that sends you back to Flickr to ask permission to view your private photos you'll understand what OAuth does straight away. Before OAuth, sites had to invent their own solutions to this problem - complete with smart security measures, their own UI flow and libraries for developers wishing to access their protected APIs. OAuth provides a ready-made solution, complete with tested libraries in a bunch of languages.

If you want to securely expose your user's private data via an API, OAuth is a no-brainer. I expect to see a lot more of it over the next year.

Fire Eagle

Launched at ETech a few weeks ago, Fire Eagle is a service with enormous potential. You can watch Tom Coates explain it in ten minutes in this video from the conference, but the short version is that Fire Eagle acts as a location broker. It consists of two key OAuth-protected APIs: one for setting the geographical location of a user, and another for retrieving that location.

This leads to a neat separation of concerns. On the one hand are the applications that attempt to figure out your location - GPS receivers, WiFi maps, mobile phones that triangulate nearby cell towers, or even sites that know where you are because you told them (Dopplr and Upcoming, for example, or the Fire Eagle site itself). On the other hand are the applications that do something useful with your location - from restaurant review sites, traffic alert services, friend finders and ARGs down to trivial applications like wikinear.com.

As a developer, this is really exciting. I can build location-based services without having to solve the much bigger problem of figuring out where my users are. Even better, wikinear.com becomes incrementally more useful every time someone builds a new tool for passing location information to Fire Eagle, without me having to do anything at all.

Obviously privacy is a huge concern when dealing with this kind of data. That's where the Fire Eagle application itself comes in: it provides a simple suite of tools for users to manage the applications that can access their location. Applications can be permitted to access different levels of accuracy or disabled entirely, and there's a "Hide" button for disabling all applications at once.

Disclaimer: I worked on an early prototype of Fire Eagle as my last project at Yahoo! before leaving in January 2007, but the product that has launched has changed enormously and is entirely the work of the current Fire Eagle team. wikinear.com is inspired by part of that early prototype.

Wikipedia and GeoNames

Wikipedia has a thriving community of geo-hackers, mainly focused around the Maps, Geographical coordinates and Wikipedia-World wiki projects. Many Wikipedia pages (Brighton, for example) have their co-ordinates in the top-right, added using a bewildering array of macros and markup extensions. You can browse through the huge collection of geotagged pages using this KML-powered Google Maps tool - zoom in and wait a few seconds to load in more markers.

The wonderful GeoNames (also used on djangopeople.net) includes an API for querying Wikipedia by location, based on 610,000 articles extracted from a Wikipedia data dump. This was a huge relief when I found it, as "order by distance from X" is actually pretty tricky to do efficiently; I've used expanding bounding box searches in the past but I'd love to hear about more effective solutions.

Google Static Maps

A long-term criticism of the Google Maps API is that it requires JavaScript to display anything at all - once you've committed to using it, you're going to have trouble implementing unobtrusive scripting (although you can work around the problem to some extent). Yahoo! Maps has long been better in this regard, but their map image API is a bit of a pain to use - you have to do an initial call to get back the URL to an image embedded in an XML file, then extract that URL and send it to the browser.

Launched last month, Google's Static Maps API is a big improvement. As with Google Charts, you need only construct a URL to the image to have it dynamically generated on the fly. You can also specify markers, and optionally omit the initial latitude/longitude/zoom to indicate that you want a best fit for the markers you are displaying. There's even a flag for a "mobile optimised" image which I'm using for wikinear.com.

Mixing it all together

Excluding templates, the entire application comes in at less than 200 lines of code and took around two hours to build. The only persistence is a couple of cookies for storing Fire Eagle tokens; Django's database layer isn't even configured (and user locations aren't logged anywhere, which is great from a privacy point of view). I suppose it's a classic mashup - Fire Eagle + OAuth + Wikipedia + GeoNames + Google Static Maps = wikinear.com. Despite its simplicity (or maybe because if it), I think it's a neat demonstration of the kind of applications Fire Eagle enables.

Django People: OpenID and microformats

2008-01-24T02:02:19Z

In hindsight, it was a mistake to launch Django People without support for OpenID. It was on the original feature list, but in the end I decided to cut any feature that wasn't completely essential in order to get the site launched before it drowned in an ocean of "wouldn't-it-be-cool-ifs".

I thought that, once launched, the site would see a small amount of activity from a few interested parties and I'd have plenty of time to catch up on the feature backlog. What I didn't expect was that over 750 people would create profiles within the first 24 hours!

So, I spent a few hours this evening integrating my current development version of django-openid, which thankfully had about 80% of the code needed to integrate with Django's built-in authentication mechanism already written. Sadly the other 20% is either incomplete or a bit of a mess, but I've checked it in to a branch on Google Code for anyone who's interested.

Anyway, there are a few new features on the site of interest to OpenID users:

When signing up for a new account, you now have the option to start by signing in with an OpenID. If you do this, you'll be able to complete the signup form without having to pick a password. If your OpenID provider supports simple registration the name, e-mail address and username fields will be filled in for you.
If you already have an existing account, you can associate one or more OpenIDs with that account. You'll then be able to use any of them to sign in to the account. Why multiple OpenIDs instead of just one? Two reasons: firstly, it opens the potential for doing interesting things with multiple OpenIDs from different providers in the future; secondly, it gives you a fallback for if one of your OpenID providers becomes unavailable.
You can freely add and remove OpenIDs from your associations, with one exception: the site won't let you delete your last OpenID if your account doesn't also have a password associated with it, to prevent you from locking yourself out.
While I decided that I didn't want Django People to become yet another OpenID provider, I do want to give people the ability to use their profile page on the site as an OpenID - so that they can prove that they own it (see my recent post on identity projection). To that end, the new account settings page lets advanced OpenID users set up an openid.server and openid.delegate for their profile page, as described in my blog entry from just over a year ago.

One caveat: the site only supports OpenID 1.1, at least for the moment. I had originally planned to go for OpenID 2.0, but demand was such that I decided to get what I had up and running rather than digging in to the OpenID 2.0 libraries.

Microformats

While I was messing around with OpenID, Natalie was updating the site's templates to clean up the crufty code I'd introduced and add some microformatted goodness. The site now uses hCard where you would expect it (country listing pages, skill listing pages and the new search interface) and the profile pages have been updated with a healthy dose of XFN (just rel="me", since there isn't a relevant microformat for "people who live nearby") and Rel-Tag. On Jeremy Keith's suggestion, the profile pages also use hResume - all the more reason to add the Django projects you've worked on to your profile's portfolio.

As usual, post feedback and bug reports as comments on this entry.