What setting causes OS X to let you resize windows behind the dock?
No application will ever slide “on top” of the Dock. You can resize applications that might cover part of the dock and they will remain below the dock. If you move or hit the “zoom” button, the OS X window manager will automatically resize the window to remove the part that is covered by the dock. 

A different behavior can be observed if the dock is set to “Auto-Hide” (cmd+alt+d). Since the Dock is now a “temporary visible” object, windows will cover and use that part, regardless of the Dock’s visibility at the time of the resize.

UPDATE: If the Dock is always visible, are you telling me that you can’t grab any window (in this example a Finder window) and drag it like in the picture? (The red arrow is the mouse pointer that the SSHOT didn’t capture). I was clicking and dragging below the dock.



UPDATE II: As we started investigating, turns out that the following is true: 

If OS X has a secondary display attached and this display’s vertical resolution is higher than the primary display (where the menubar and dock are), then the Dock will let you resize “behind” it. I don’t have enough screen combinations to test exactly when this happens, but it’s clear to me that it only happens when there’s an external display. 

Next is a shot of my arrangement. As you can see in the picture, the “gap” between the big wide screen and the vertical one on the right is what I think causes the Dock to behave like this. It appears as if the window manager takes the highest vertical size when determining the bounds. I don’t know if this is a feature or a bug.



UPDATE III*: Ok, for the sake of testing, I went ahead and modified my Screen’s arrangement (see picture). I exaggerated the placement because I wanted to make sure that there’s no weird offset acting. With the screens arranged like in the picture, my dock will not let me resize below it.



This possibly solves the “mystery” of why does this happen. I hope you can replicate it and/or find a way to work around the issue. For the record, I rarely run into this because I usually either auto hide the dock or use SizeUp to move/arrange my windows, and it doesn’t let me put windows behind the dock.