What methods can be used to find hidden options in OSX softwares?
It involves a lot of trial and a lot of error, but you can be helped along in the process by running the strings command against the binary for application, directing the results to a file, and then reading that output looking for hyphenated or InnerCapitalizedWords that might suggest a  candidate for a hidden preference.

Rob Griffeths wrote an article at MacWorld that gives some helpful suggestions for finding hidden preferences.