Why does air seem to affect local surface temperatures so much?
The surface temperature is the temperature of the air at the surface (rather than the temperature of the ground itself).  This temperature is typically taken between 1 and 2 m off the ground, and this is the temperature we feel when we are outside.  Simply put, the reason air affects this temperature, sometimes radically, is because temperature is a property of the air (it is related to the mean kinetic energy of the molecules comprising the air). 

If we are curious about the change in temperature we experience, we might write $\frac{DT}{Dt}$, where $\frac{D}{Dt}$ is a material derivative and $T$ is temperature.  This change in temperature is Lagrangian, or in other words it is the temperature of a parcel of air as it moves around.  We can expand this as $$\frac{DT}{Dt} = \frac{\partial T}{\partial t} + \mathbf{u}\cdot\nabla T$$ What this equation says to us is this that the change in temperature can be related to two effects:


The change in temperature of air at a fixed location in space (e.g. outside your house), and
The transport of different temperature air to your location by the wind.


The first bullet point is going to be modified by energy transfer between the ground and the air (conduction) and radiative losses (radiation emission).  These changes are on the scale of how temperature changes between morning and afternoon -- nothing too radical.

The second bullet point is where the radical changes can take place.  Temperature is a property of the air, so if new air moves in to your location with a different temperature, that will strongly determine the local surface temperature. 



You mention the polar vortex, which is a often misused term which has recently come into popular vocabulary.  The polar vortex is primarily a mid-upper atmospheric circulation above the poles.  When you hear it on TV you are most likely hearing it misused and the generally just mean a really strong cold front with a (perhaps unseasonably) cold arctic continental airmass behind it.  A front, in simple terms, is an enhanced surface temperature gradient that separates airmasses with different properties.  The key feature is a strong temperature gradient -- a relatively large change in surface temperature over a relatively small distance.

This goes back to the earlier discussion on the primary contribution to radical changes in temperature being the wind bringing in air with a different temperature.  When a cold front arrives, it is being driven by the wind and this is an example of the transport of temperature by the wind (cold air advection).  



In summary, air affects local surface temperatues so strongly because temperature is a property of the air.  If the air currently around your town is replaced by new air, that air will have its own temperature and so you will experience a change in temperature -- if the new air is radically colder, you will experience a radical temperature change.