Socket recv() timeout on secondary nodes
Just a shot in the dark. Could it be a problem with the primary servicing too many connectionss simultaneously?



This information comes from the mongodb 202 training.

If a primary can no longer service requests, there will be 


socket exceptions
timeouts
the over servers can no longer ping the primary possibly causing an election
--maxConns limit connections to mongos can insulate mongods from getting into unpredicatble states


From MongoDB best practices

Connections MongoDB drivers implement connection pooling to facilitate efficient use of resources. Each connection consumes 1MB of RAM, so be careful to monitor the total number of connections so they do not over-whelm the available RAM and reduce the availablememory for the working set. This typically happenswhen client applicationsdo not properly close their connections, or with Java in particular, that relies ongarbage collection to close the connections.

If there are too many connections open on the primary at any one given time, secondaries, administrative, clients, etc, then your primary could also be running out of memory. 

Here is an example layout, with connections coming from drivers, secondaries, to and from different processes. This can all quickly add up depending on the number of clients and machines in your infrastructure!