Yosemite randomly switches audio output to displayport
System Preferences builds a preferred list for your audio outputs and inputs based on which ones you pick over others.

To work around this you will need to manually select the audio output you would rather use once when the undesired one is available and again when it isn't. This method will work in most (but not all) cases.