Simon Willison’s Weblog

MongoDB—Capped Collections. Collections with a size limit that automatically expire older entries are interesting—useful for things like a “recent searches on this site” feature.

# 7th June 2009, 12:50 pm / mongodb, cappedcollections, search

Haystack (via) A brand new modular search plugin for Django, by Daniel Lindsley. The interface is modelled after the Django ORM (complete with declarative classes for defining your search schema) and it ships with backends for both Solr and pure-python Whoosh, with more on the way. Excellent documentation.

# 17th April 2009, 9:53 pm / django, python, search, haystack, whoosh, solr, orm, daniel-lindsley

Digg Search: Now With 99.987% Less Suck. Really nice implementation of faceted search, still using Lucene and Solr under the hood.

# 10th April 2009, 10:17 pm / search, lucene, solr, digg, facets, full-text-search

Sphinx 0.9.9-rc2 is out. Interesting new feature: the Sphinx search server now supports the MySQL binary protocol, so you can talk to it using a regular MySQL client library and fire off search queries using SELECT syntax and the new SphinxQL query language.

# 8th April 2009, 1:59 pm / sphinx-search, mysql, sql, search, full-text-search

Guardian + Lucene = Similar Articles + Categorisation. Alf Eaton loaded 13,000 Guardian articles tagged Science in to Solr and Lucene and is using Solr’s MoreLikeThisHandler to find related articles and automatically apply Guardian tags to Nature News articles.

# 11th March 2009, 12:53 pm / alf-eaton, solr, lucene, guardian, openplatform, naturenews, search, full-text-search

How search.twitter.com uses Varnish. Includes examples of the configuration options they use.

# 2nd March 2009, 5:08 pm / twitter, varnish, search, caching

django-springsteen and Distributed Search. Will Larson’s Django search library currently just talks to Yahoo! BOSS, but is designed to be extensible for other external search services. Interestingly, it uses threads to fire off several HTTP requests in parallel from within the Django view.

# 25th February 2009, 10:28 pm / django, python, will-larson, djangospringsteen, yahooboss, search, http, threads, concurrency

Xapian performance comparision with Whoosh. Whoosh appears to be around four times slower than Xapian for indexing and empty cache searches, but Xapian with a full cache blows Whoosh out of the water (5408 searches/second compared to 26.3). Considering how fast Xapian is, that’s still a pretty impressive result for the pure-Python Whoosh.

# 14th February 2009, 1:15 pm / whoosh, xapian, search, full-text-search, python, richard-boulton

Whoosh. A brand new, pure-python full text indexing engine (think Lucene). Claims to offer performance in the same league as wrappers to C or Java libraries. If this works as well as it claims it will be an excellent tool for adding search to projects that wish to avoid a dependency on an external engine.

# 12th February 2009, 12:49 pm / search, python, whoosh, lucene, open-source, full-text-search

Introduction to Information Retrieval (via) This looks excellent—a modern guide to implementing search engines written by some of the engineers behind Yahoo! Search. The full text is available online, but it looks like it’s well worth investing in the dead tree edition.

# 9th February 2009, 8:54 pm / search, yahoosearch, books, freebooks

Announcing the Article Search API. The most interesting API from the NYTimes yet—search against 2.8 million articles from 1981 until today using 35 searchable fields and get back detailed metadata as well as the first paragraph of the articles themselves.

# 5th February 2009, 11:06 pm / new-york-times, apis, newspapers, search

solango. Another attempt at a Django/Solr integration library, based on code written for “a top 20 newspaper site” (I’d love to know which one). This is well documented, uses a registration model clearly inspired by the Django admin which keeps search related metadata out of your regular models and includes management commands for re-indexing and generating Solr schema.xml files.

# 4th February 2009, 12:22 pm / solr, django, python, lucene, search

All you ever wanted to know about writing bloom filters. This helped me understand a key use case for bloom filters: reducing the impact of the “worst case search is when there are no matching results so everything gets scanned” problem.

# 30th January 2009, 8:26 am / bloom-filters, search, jonathan-ellis

How-to: Full-text search in Google App Engine. Use search.SearchableModel instead of db.Model—it’s pretty rough at the moment which is probably why it’s still undocumented.

# 27th June 2008, 8:25 am / googleappengine, appengine, full-text-search, python, search

Google AJAX Search API: Flash and Server Side Access. Over a year after Google shot down their SOAP Search API, they’ve quietly released a JSON based one under the guise of supporting “Flash and other non JavaScript environments”. Comes with the strange requirement that an HTTP referer be sent with every request; the API key is optional.

# 22nd April 2008, 7:16 pm / google, soap, ajax, json, search, web-services, apis

In-Depth django-sphinx Tutorial. Another neat Django extension from the guys at Curse: easy integration with the sphinx full text search engine.

# 5th March 2008, 12:03 am / django, curse, python, sphinx-search, search, david-cramer

pysolr. Python wrapper for Solr, the search web service wrapper for Lucene. One thing I’m not clear on: do you need to configure Solr with the fields you’ll be indexing in advance, or can Solr create new fields on the fly to match the data you send it?

# 9th January 2008, 8:50 pm / solr, search, apache, python, lucene, pysolr

Opera 9.5 alpha, Kestrel, released. “With history search, Opera creates a full-text index of each and every page you visit, and when you go to the address bar, you can simply start entering words you know have been on pages you’ve visited before, and items matching your search show up.” I just tried this; it’s magic. I’m switching back to Opera from Camino.

# 16th September 2007, 8:34 pm / opera, camino, browsers, history, search, kestrel, full-text-search

django-sphinx (via) More code from Curse Gaming; this time a really nice API for adding Sphinx full-text search to a Django model.

# 9th September 2007, 12:35 am / django, python, orm, search, sphinx-search, cursegaming, david-cramer, full-text-search

Grub. Jimmy Wales just announced at OSCON that Wikia have acquired Grub from LookSmart, and will be releasing it as open source.

# 27th July 2007, 5:24 pm / search, oscon07, looksmart, open-source, oscon, jimmywales, grub, wikia

Apache Solr 1.1. Solr is the search Web Service built on top of Lucene. The latest release introduces JSON, Python and Ruby response formats in addition to XML.

# 13th January 2007, 1:16 am / json, python, ruby, xml, webservice, search, lucene, solr

Implementing Full Text Indexing with PostgreSQL (via) TSearch2 tutorial.

# 13th August 2004, 3:44 pm / postgresql, search