How does one identify relevant variables when performing research on perception? Suppose, hypothetically, that I wanted to do a blind taste test into preference for Coke or Pepsi. I organize hundreds of volunteers and find that a certain group prefers one and another group the other. 

As I work for some marketing company, they ask me to account for this. After all, they reason, preference for Coke vs Pepsi is not random. 

Perhaps there's a socioeconomic explanation, perhaps it relates to gender, hormones, age or ethnicity. Who knows! There are hundreds of variables commonly studied in the academic community.

Even if I find a couple of variables that correlate, say hypothetically, BMI and employment status, I have no way to determine if those are "good" variables or if this is just a coincidence.

How are researchers in psych experiments trained to deal with this sort of scenario?
Researchers are trained to address this problem by using literature reviews to isolate the specific variables that should be of most interest. At the start of research you generally examine the knowns and unknowns in the particular area of interest, so as to determine the state of the art in that area. Once you know the state of the art you have a better idea of how to proceed with the research. 

For example, if you found that something was known about your research area then you could use this literature to justify looking at a specific set of variables when examining the problem. This relevant prior literature need not have examined the exact same area as the problem you are looking at. For example, if you were looking at factors that influence preferences for coke and pepsi you could look at research that examined brand preferences, or even something more fundamental such as the factors that lead to positive beliefs and attitudes. This research would give you an idea as to which particular variables were most deserving of consideration.

Depending on the extent that you could argue that little was known about the particular problem,  you could also consider doing exploratory or descriptive research to attempt to identify or categorize the different variables involved. In this case you might use an interview to get some factors that set up your experiment and then use experiment to assess their relative impacts on the behaviour.