Subdivide vs subdivision modifier
When subdividing only the vertices inside the circle, you created N-Gons. That means that some of your faces had more (or maybe less) than 4 vertices. 

As you have guessed, the catmull-clark interpolation for subidivison does not work well with n-gons and can produce a messy result.

There are multiple ways to locally add some finer details when using a subdivision surface modifier but the one you used will produce a bad shading if you do not retopologize the area to avoid n-gons.