I’ve been working with MongoDB for a while. For most of the part, I like it. The syntax is expressive and simple, which makes it very friendly for new developers. After working with it for a while, I gradually realized that some queries are expensive to carry out, and some are unnecessary. So I tried to optimize my queries with various techniques. Here’s what I’ve found.
When you make a query to a MongoDB collection, MongoDB will perform a full collection scan. If the collection is big, this will take a lot of processing power and time. With indexing, you can tell MongoDB what fields you will need ahead of time. When you perform the query later, instead of searching every field of the collection, MongoDB will look for the specific field directly.
Consider this situation: You want to perform a text search against the database, your target content resides in the the
description field. With indexing, you can tremendously increase the search speed by avoiding scanning every
description field of the collection.
First, you add the index to the mongoose schema:
Then in your router controller, you can perform the search:
This is how you can avoid expensive queries to the database.
Mongoose indexing is handy. However, there are some limitations. First, adding too many indices can slow down writing records to the the data base. Each index you add takes additional processing power. Second, you don’t know ahead of time all the indices you need.
Don’t fret it, here’s the good news: Sometimes we don’t need to query the database at all!
Sounds crazy? Keep reading.
The way of reducing queries is through caching. If a query is made, there’s no need to perform it again upon new requests within seconds, since the content won’t change. So, the main idea is, when a query is made, we cache the results, when the same query requests come in later, we send back the content from the cache.
The caching tool we’re going to use is Redis. Redis is an in-memory database, it operates in the RAM of the server, so it’s super fast.
To work with Redis in nodeJS, we need to install the
redis npm module.
Redis save the cache content under key-value pairs. To save a record, you write:
Here is how to get a record out:
Then we setup the connection:
Suppose we cache all queries. This is just for educational purpose, normally we wouldn’t do this. Where should we implement the cache logic? Since the mongoose model deals with all queries, it seems like a fit candidate. In order to inject the custom cache functionality, we need to monkey patch the prototype methods of mongoose query objects. Here’s how it’s done:
Now we have a working cache functionality, but it caches all queries under the same hashKey. Let’s fix this.
We will add a new method to the prototype chain of the mongoose model object, so that every model instance can access this method and toggle the cache state. Here’s the code:
Then we need to configure the
exec() method accordingly.
Now, in the route controller, we can call the
cache() method on the mongoose query object and pass in the hash key.
Sometimes you don’t want to keep the cache. For example, after the user add a record, we need to show them the new content immediately. To do this we need to flash the cache once the post request is completed.
We’ll create a middleware to handle the delete process. When a route needs to auto-delete the cache, just apply the middleware to it.
Here’s how we use this middleware: