Running a simple logging program, how to diagnose spikes mongod's CPU activity? I'm running mongodb 1.4.7 - I've a node server set up which logs incoming requests in a logs collection:

{'u':'user123', 'l':'log data'}

In addition, once the request happens, the server will check if a users collection contains a user123 in this case via the .find({'u': 'user123'})

Running top I see this:

PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
4709 mongodb   20   0 1568m 435m 303m R 176.3 11.0 370:22.58 mongod


This, however, only happens every so often, most of the time it sits at about 1-2% CPU and running mongotop shows that mongo isn't particularly stressed:

           mdb.users         1ms         1ms         0ms
           mdb.impressions   1ms         1ms         0ms


I'm getting a decent amount of traffic. What steps can I take to identify the source of resource usage spikes?
There is no one simple answer here, it will very much depend on your usage.  CPU can be consumed in many ways.  I'll list the most common (in my experience with MongoDB) and how you can try to determine if this is the case for your spikes.  

Before I do though, I will say that a CPU spike is not necessarily a problem - people get obsessed about spikes in CPU, increases in the load metric, but what really matters is the performance of your application - is it actually suffering during these spikes?  If not, then it's probably not that important - by all means monitor it and be aware in case things get worse and you do see an impact, but spikes happen on busy systems without it being a problem per se.

Common Causes (and how to narrow it down):

1 - A periodic, intensive database operation

Table scans (non-indexed queries), massive index updates, in-memory sorts and probably a few others can all tax a database process from the CPU side of things.  To determine if that is the cause here, there are a few tools that you should be using:


MMS Monitoring


This is going to come up in this list more than once, but specifically you will want to look at the number of ops being executed at a given time, btree hits, and the hardware stats (munin-node required).  These will give you an insight into the work that the database is doing.  User CPU is what you have listed above, but is there a system CPU component too?  MMS will graph them over time and let you correlate the spikes in CPU with other events in general.


Examine the mongod logs


If you know the times that you are seeing the spikes, take a look in the logs - are there any slow operations around that time? Any index builds?  If the system is being generally slowed down there might be a lot of false positives in here, so look for trigger events, common themes etc.

The excellent mtools by Thomas Rueckstiess can make this analysis significantly easier, allowing you to graph events, look at specific slices of time etc.


mongotop


This will tell you what databases and collections MongoDB is spending time on.  Now, if you only have one collection and no replication, then this will not be terribly helpful, but if you have a multitude to pick from, then it can be very effective at narrowing things down, especially if you combine it with the data from MMS to get you specific time frames to analyze.  Unlike MMS though, you have to run this in advance of the issue so that the timeframe of your suspicious spike is covered.


mongostat 


This tool provides data similar to that in MMS, but at a per-second granularity (MMS Monitoring is 60 seconds), so if the spikes are short lived, this can be a valuable piece of additional data.  Similar to mongotop, it must be run manually beforehand to cover an event and then the data gathered/examined.

2 - An IO Bottleneck

Slow IO can cause CPU usage to spike.  Again, MMS Monitoring with munin-node installed will give you an idea as to the nature of the CPU usage - if IOWait is high then you will want to pay attention to the whether the traffic is read or write bound (or both) and looking at the page fault levels can also be enlightening (high levels can mean that data is not fitting into memory, or "cold" data is being paged in at that time).

Essentially, all of the same tools are in play here as in the first scenario, but this time you can add:


IOStat


IOStat deserves an essay in and of itself, so I will be brief - you will want to look at %util as a broad first measure, then look for further information to determine the source of the slowness.  The await metric can tell you if queueing is an issue, svctime can tell you if the device is being slow, high reads or writes at the time of the spikes can tell you what particular IO limit you are running into.  Again, similar to the mongostat/mongotop tools, you can run this command to a much finer granularity than you will see in MMS.

3 - Connection Related Churn

This is somewhat more rare, but a badly configured driver, or an application that gets out of control can sometimes cause a massive amount of connections to be opened/closed in a small time frame while leaving the concurrent connection level fairly constant.  You can use a combination of MMS (connection graph) and analysis of the log files (as long as you are not running with --quiet) to track this down.

There are quite a few other things to look at here, but those are where I would start, and if you master the tools listed you will be well on your way to troubleshooting those other possibilities.