Where does Messages.app store its "Play sound effects" setting?
Since Messages.app is a sandboxed app, the preferences would be under ~/Library/Containers/:

~/Library/Containers/com.apple.iChat/Data/Library/Preferences/com.apple.iChat.plist


or look through ~/Library/Containers/com.apple.*iChat*/.