How to remove or disable a default keyboard layout?
For macOS 10.8 and later:

Change the current input source to your custom keyboard layout.
Open ~/Library/Preferences/com.apple.HIToolbox.plist (in 10.9) or ~/Library/Preferences/ByHost/com.apple.HIToolbox.*.plist (in 10.8 and earlier). You can convert the plist to XML with plutil -convert xml1.
Remove the input source or input sources you want to disable from the AppleEnabledInputSources dictionary. If there is an AppleDefaultAsciiInputSource key, remove it.
Restart.

For macOS 10.7 and earlier:

Enable an input source that is not like your physical keyboard layout from System Preferences (I used Afghan Dari).
Run f=~/Library/Preferences/ByHost/com.apple.HIToolbox*.plist; plutil -convert xml1 $f; open $f -e.
Remove other input sources except the temporarily enabled keyboard layout and your custom keyboard layout.
Log out and back in.
Remove the temporarily enabled keyboard layout from the property list.
Log out and back in.