Problem with lifelines in "Who Wants To Be A Millionaire" board game
The hit-and-miss nature of the Ask The Audience seems like it is right.  There is no mention in the rules that the answer that a player gives for Ask The Audience must match the answer they ultimately decide to go with.  If you look at the rules for Phone A Friend (which appears to work similarly) it states 


  this card does not have to match what the “Friend” may have answered
  on his/her own Console


Since in the real game show asking the audience is really just asking a bunch of random people what they think, I believe the "problem" you have encountered in many ways actually approximates how a real random audience would work.  You have some that know the answer (and are helpful people), some who aren't sure, some who are trolls and saboteurs (even the Phone A Friend lifeline could be used to mess somebody up on purpose, which is partially why I believe that sabotage is ok).

Also, just like in the game show, there is no guarantee that what the audience says is the right answer is really correct.  The contestant has to decide how much they really want to believe the audience and then make their decision.