3D Manipulator only selects a handful of times
After looking through the 3d-manipulator questions this helped me find the solution. 

It turns out I somehow locked my objects to another object. I don't know how I did it but after clearing all the objects from having an object they were locked to it fixed the problem. 

My view field looks like this now: