Can I make the caps-lock setting stick, for external USB keyboard in Mountain Lion?
It should stick even on Mountain Lion.

Two suggestions.

First, you might use Onyx to validate your preferences.  

Second, try deleting ~/Library/Preferences/com.apple.systempreferences.plist and logging out and logging in again.  

In either case, the pref file might be messed up and that could prevent it from being saved.