Mongodb errors

(by nicholas, last updated: mar 2013)

There are three distinct problems we have with mongodb.

1. "Execution expiration" and "Connection closed" caused by mongoid counts.

2. "TypeError: No c decoder"

3. Slow queries due to lack of indices

Back in december the application was unusable due to 3 and 1 occurring simultaneously. Back then we migrated off of MongoHQ to be able to see attach monitoring systems to it and dig deeper into what may cause these slowdowns.

Slow queries due to lack of indices

When we migrated we were able to run "dex", a log analyzer that recommends indices. We added a set of indices to the database and the application became usable, however there were other problems. It's important to note though that we have added a lot of indices and although we still have slow queries, the slowest one we've recorded is 6 seconds. nothing close to 30seconds that cause timeouts.

current status: resolved to adequate standards, optimizing these 6s queries is not the most important thing right now. We'll defer that.

TypeError: No c decoder

"TypeError: No c decoder" started occurring and is currently not addressed, they occur on mentions which are inherently temporal data and my belief is that this is a data consistency issue (aka, we are writing crap into it and we fail to get this crap back out).

current status: unresolved but temporal, solving this requires some more research.

"Execution expiration" and "Connection closed" caused by mongoid counts.

Which brings me to the reason I wrote this page. After the indices were added I investigated the mongo database statistics using tools like mms.10gen.com and the mongo shell. Our db statistics have some interesting figures and while some numbers are large they appear to be normal according to mongodb documentation. Except for this one: https://gist.github.com/noverloop/d503b2dd39c10fd791b1#file-gistfile1-txt-L152

According to mongodb documentation this is caused by the application requesting that. So what happens is that our application is downloading a large amount of data (using a cursor, which is the same thing as an iterator). code like

@pages.each do
end

So the other issue we are running in is this http://stackoverflow.com/questions/9778420/mongo-count-really-slow-when-there-are-millions-of-records

Basically our database becomes big and stuff like

@pages.count
@pages.blank?
@pages.empty?

become really slow because they are using that count-operation.

This is what we have to remove from our application, unfortunately it's hard to know when mongoid is using this operation so we have to rely on exceptional errors pointing us to it.