Outside sub panel between meter and breaker box
From the description, not a sub-panel, but a main disconnect switch. 

Unclear why you are having trouble switching it off - if having trouble switching it ON, you might have a problem causing it to trip (they are almost always circuit breakers, not "merely switches")

It probably will take considerable force to switch off, but you should not need (nor use) a prybar or hammer on it. ;-) A forceful and determined motion of the hand towards the "off" direction should suffice.