Mavericks - terminal change default printer paper size
The setting in System Preferences changes the DefaultPaperID key in ~/Library/Preferences/org.cups.PrintingPrefs.plist. Try to change that key instead:

defaults write org.cups.PrintingPrefs DefaultPaperID 154x216


Quit and reopen applications to apply the changes. 

defaults read com.apple.print.custompapers shows the IDs of custom paper sizes.