Simon Willison’s Weblog

Comet works, and it’s easier than you think

5th December 2007

I gave a talk this morning at the Yahoo! Web Developer Summit on Comet, cometd and Bayeux.

I’ve been trying to keep up with Comet ever since Alex coined the term last year, but it’s only in the past few weeks that I’ve actually found some time to play with it myself. I was very impressed with what I found: the open source infrastructure for building and deploying Comet applications is surprisingly mature, and with just a few more improvements I can see Comet achieving much more widespread use.

Comet is an umbrella term for any technique that allows a web server to “push” events down to a browser. You can think of it as an alternative to Ajax polling, with the benefit that events are relayed in almost real-time and “wasted” requests (when nothing has changed) are massively reduced. The name doesn’t stand for anything; it’s named after an American kitchen cleaner (a joke on Ajax).

When you consider the hacks involved in getting Comet to work across the four major browsers it’s miraculous that it works at all. In the talk I tried to illustrate the insanity with examples of browser hacks, mainly to show how totally absurd it all was. But while the solutions are mostly pretty terrifying, the fact that we’re dealing with JavaScript (rather than CSS) means that we can abstract all of the nastiness away, ready to be replaced later on when browsers start introducing native support. Abstracting away the nastiness until the browsers catch up is something of an unofficial mission statement for the Dojo project, so you won’t be surprised to hear that Dojo offers excellent support for Comet.

The Bayeux protocol

For me the most exciting Comet development is the invention of the Bayeux protocol. Bayeux defines a standard protocol for Comet clients (both browsers and others) to communicate with a dedicated Comet server, using a simple but powerful publisher/subscriber architecture based around the concept of named channels. Clients can connect to a Bayeux server, subscribe to one or more channels, and then publish messages targeted at a channel. The server’s job is to relay those messages to all clients subscribed to that channel.

There are a number of things to like about this setup. Firstly, it means that all of the difficult parts of Comet (relaying real-time messages, coping with huge numbers of simultaneous connections) are kept separate from your regular architecture. You can keep developing and deploying applications in your preferred framework (Django, Rails, PHP on Apache or whatever) while the Bayeux server sits there as essentially a black box—clients can subscribe to it and you can use it to publish messages all without needing to customise the Comet server at all. If the Bayeux server implementation you chose doesn’t work out for some reason you can swap it straight out for something else that supports the same protocol.

The one thing missing from Bayeux at the moment is authentication. Out of the box, a Bayeux server will relay messages to a specific channel from any client to any other set of clients. This is an obvious flaw: if you’re running a site where events such as “a new comment has been added” are broadcast out via Comet, you don’t want just anyone to be able to imitate such messages and have them sent to all of your subscribed clients. Right now, Bayeux leaves it up to the individual servers to figure out how they will deal with this. I’d like to see the specification address this directly as without it a Bayeux server isn’t much use in a real-world environment.

Getting started with Comet

If I’ve piqued your interest, the good news is that getting started with Comet is really, really easy. All you need is a running server that supports the Bayeux protocol. I tried out a number of options, but the first one that worked straight out of the box was Jetty 6.1, a Java web server that uses continuations to achieve high concurrent performance. Here’s how to get Jetty up and running in a few easy steps:

1. Download Jetty 6.1 (the first version to include a cometd implementation) from www.mortbay.org.
2. Install Maven, a free build tool for Java (the download page has installation instructions).
3. Unzip Jetty (you can put it anywhere).
4. cd jetty-6.1.6/contrib/cometd/demo
5. mvn jetty:run—the first time you run this it will download a bunch of dependencies and then start up a Jetty server on port 8080.
6. Navigate to http://localhost:8080/ and you should see an index page linking to a number of Comet demo applications.

Once you’ve played with the demos, building your own application is almost as easy. The code for the bundled examples lives in jetty-6.1.6/contrib/cometd/demo/src/main/webapp/examples—I simply created a copy of the entire chat directory and started building my own experiments from there.

Using dojox.cometd

If you look through the source of the chat application, it quickly becomes apparent that most of Comet boils down to just three methods:

• dojox.cometd.init(comet_server_url) initialises a connection to the given Comet server. The Bayeux handshake protocol is used to establish the most appropriate Comet method for the connecting browser; you don’t have to worry about the details of the Comet connection at all.
• dojox.cometd.subscribe(’/channel’, callback) subscribes a callback function to a named channel. Any time a message is sent to that channel the function will be called.
• dojox.cometd.publish(’/channel’, json_object) sends (publishes) a new message to a named channel. The message can be any valid JSON data structure.

There are a few other methods relating to batching requests and disconnecting from the server, but the above three make up the bulk of any Comet application.

A Comet slideshow

In preparation for my talk I decided that I’d try to present my slides using a small Comet application. I ended up building a very simple slideshow tool—I exported my Keynote presentation as a sequence of images, then wrote a Comet client that listened for “show this slide” messages and another client (a master, which acted as my presenter view) that could publish those messages. You can see a screenshot of the master client on Flickr; in addition to “next” and “previous” buttons it also shows a small preview of the upcoming slide.

The code for the client application (which I had running on the main projector screen, and also encouraged the audience to load on their laptops) boiled down to just a few lines of code. Here’s the slideshow client in its entirety:

dojo.require("dojox.cometd");
jQuery(function($) {
    dojox.cometd.init("http://example.com/cometd");
    dojox.cometd.subscribe("/slideshow/change", function(comet) {
        $('#currentSlide').attr('src', comet.data.src);
    });
});

The master client was only a little more complicated, due to the need to keep track of the full list of slide URLs (non-sequential, to discourage the audience from skipping ahead) as well as display the preview. That said, the core Comet functionality was wrapped up in a single function:

function publishSlide(src) {
    dojox.cometd.publish("/slideshow/change", {
        'src': src
    });
}

The entire application took less than an hour to put together, which I think is a testament to the quality of the Bayeux implementation present in both Jetty and Dojo.

The future

Before taking a detailed look at Comet, my assumption was that the amount of complexity involved meant it was out of bounds to all but the most dedicated JavaScript hackers. I’m pleased to admit that I was wrong: Comet is probably about 90% of the way to being usable for mainstream projects, and the few remaining barriers (Bayeux authentication chief amongst them) are likely to be solved before too long. I expect to see many more sites start deploying Comet powered features over the next twelve months.