How does "Restrict background data" work? In "Settings"->"Data Usage" there is a "Restrict background data" checkbox.

How does it work, exactly? What does "background" mean here? And is this setting enforced by the OS, or is it a hint that the apps are free to disregard if coded improperly?
First, that setting is enforced by the OS (so it's not for the app to decide whether to observe that setting). Put in easy words: Android won't let an app access the network unless it's running in foreground (i.e. the user is interacting with it) with the restriction enabled for it.

Explained with a plastic example: Restrict background data on the Google Play Store app (as I did), it won't anymore check for updates in background while you are on mobile data. But it will do so when you're using WiFi. Also, when you call up the app to install something, it will work as long as you keep it in foreground (i.e. not switch to another app while it's still downloading).