blog.omdb-beta.org

If your application relies on server-based live searches, you’ll notice that using database queries is far too slow. On OMBD we implemented our first live search tests (some time back in May 2006) with LIKE statements and MySQL full text columns. But both solutions were far too slow, leaving us wondering, how to speed up live searches. Fortunately for us, we discovered Ferret, maybe the only search engine for ruby worth mentioning as of this day.

We used Ferret as an indexer to find the appropriate ActiveRecord-Objects, basically just storing the class and id of the AR-Objects. So, whenever a live search is performed by a user, the ferret index will be queried and the appropriate AR-Objects fetched from the database.

If you are still using SQL-Statements for searches, you should really take a look at the acts_as_ferret plugin for rails. It encapsulates the logic i described above. Jens did a fantastic job writing a plugin that allows you to include a really fast search engine into your project with ease.

However, if you want to do a live search there is no need to fetch the whole object from the database. Most of the time a live search result is just a small title or a name. In OMDB we’ve implemented a lot of live searches, so for example if you want to attach a new cast to a movie, you do a livesearch for a person and for a job. If you’re adding a lot of casts, thats a whole lot of SQL statements. So we thought about removing these SQL Statements by using Daves LazyDocs.

A LazyDoc is one of the newer features of ferret. Whenever you store something in the ferret index you basically just store fields and values, a hash that is. Ferret will return those hashes as results to you, whenever you run a query. So we added a lot more fields to the ferret index, allowing us to fetch all the data we need for our live searches.

In order to keep our templates DRY we wanted to make sure a template does not need to know whether it is currently working with a LazyDoc or with a real AR-Object. Therefore we implemented a small extension to the LazyDoc class.

First of all we need some way to convert a LazyDoc to an AR-Object. We’re storing the class of the AR-Object as the ferret key :type, and the database-id as :object_id, therefore these small methods (extension to LazyDoc) will return the class and the id of the stored object.

def class
  self[:type].constantize
end

def id
  self[:object_id].to_i
end

Having the class and the id we can then fetch the object from the database.

def to_o
  self.class.find( self[:object_id].to_i )
end

Now we can convert a LazyDoc to an AR-Object by calling the to_o method. Most of the AR-Object data is stored in the index, but some complex information or relations cannot or should not be stored in ferret – at least for now. So if an information is missing in the index, we can fallback to the AR-Object and get the information from there Furthermore it would be great to be able to use all of these nifty methods that ActiveRecord is creating automatically from the database fields. A LazyDoc does not know about these methods. But by implementing a method_missing method for LazyDocs, we can automatically fall back on the AR-Object, whenever neccessary.

def method_missing( method_name, *args )
   # Fetch the information from the ferret index.
   value = self[method_name]
   # Return the information from the database, 
   # if there is nothing in the ferret index
   # A LazyDoc - like a Hash - will return nil, if the
   # key is unknown.
   to_o.send(method_name, *args ) if value.nil?
end

Now our live searches can use LazyDocs, reducing our sql statements for live searches to almost zero. It increased the performance for live searches dramatically. Our next step will be to memchache LazyDocs, so we don’t need to fetch the information from disk for every search.