Mongoose Tip $Push

22 Apr 2017

I've been working with Mongodb for the last year at my software engineering job. At first, I was hesitant. To be frank, I was a bit of a Mongodb hater (for no good reason).

Since then I've matured (some what :) ) as a software engineer. I realized my bias was holding me back from using an amazing piece of technology.

I want to share a quick tip I learned. The tip is to use $push when adding records to an array. Some examples I've seen get the entire document from Mongo, use Node.js to push the element into the array, and save it back to Mongo.

While this approach works, it has many downsides. One downside is the entire document is transferred on the wire. This is a waste of bandwidth. Another downside is all of the JSON serialization and deserialization that occurs. Node.js can handle this, but while JSON.parse or JSON.stringify are on the CPU stack, NO other work is getting done.

Another downside of read, push, save technique is it is not an atomic operation.

Thanks Dan for pointing this out!

$push on the other hand is an atomic operation. So you won't lose data if someone else is writing to the same array at the same time.

Let's get right to some example code.

To follow along:

'use strict';

const mongoose = require('mongoose');
mongoose.Promise = global.Promise;
const _ = require('lodash');
const async = require('async');

mongoose.connect('mongodb://localhost/test');

const Restaurant = mongoose.model('Restaurant', {
  address: {
    building: String,
    coord: [Number],
    street: String,
    zipcode: String
  },
  borough: String,
  cuisine: String,
  grades: [{
    date: Date,
    grade: String,
    score: Number
  }],
  name: String,
  restaurant_id: String
});

function makeNewGrade() {
  return {
    date: new Date(),
    grade: _.sample(['A','B','C', 'D']),
    score: _.sample(['1', '2', '3', '4', '5'])
  };
}

const numberOfGrades = process.env.NUM_GRADES || 10;
const restaurantId = '30075445';

function pushIteree(num, cb) {
  let newGrade = makeNewGrade();
  Restaurant.updateOne({
    restaurant_id: restaurantId
  }, {
    $push: {
      grades: newGrade
    }
  }).exec(cb);
}

function getSaveIteree(num, cb) {
  Restaurant.findOne({
    restaurant_id: restaurantId
  }).exec((err, restaurant) => {
    if (err) {
      return cb(err);
    }
    if (!restaurant) {
      return cb(new Error('Restaurant not found'));
    }
    restaurant.grades.push(makeNewGrade());
    restaurant.save(cb);
  });
}

var iteree;

if (process.env.PUSH) {
  console.log('running with $push');
  iteree = pushIteree;
} else {
  console.log('running with get/save technique');
  iteree = getSaveIteree;
}

async.each(_.range(numberOfGrades), iteree, (err) => {
  console.log('Grades processed');
  process.exit();
});

Most examples I see on the internet are a variation of the getSaveIteree function. The task is to save an element into an array of a document. It performs a lookup using findOne, Array.push's the element, and saves the document back to the database.

A cleaner and more effective way is to use $push. The pushIteree function uses updateOne with the $push operation. This implementation doesn't require any JSON parsing in the node.js process. Almost all of the workload is put on Mongo. Mongo is better fit for high CPU operations.

Naive and simple benchmarks

This isn't a real world app example, BUT you probably know places in your app that uses the getSaveIteree pattern.

Mongoose push vs get save

The first two runs are using the getSaveIteree function. The first run happens in an acceptable amount of time. The second does not. The problem will get worse with each run because the amount of data transferred and parsed grows. Eventually the process runs out of memory.

The third, fourth, and fifth runs use the pushIteree function. As you can see, they both run in the same amount of time. The process doesn't have to deal with the overhead of parsing the document into memory each time.

Summary

That's it for this tip!

What's your number one Mongoose or Mongodb question?

Content is published under this license.