title: How do I evaluate the effectiveness of an algorithm that predicts probabilities?
body: <p>I need to evaluate the effectiveness of algorithms which predict the probability of something occurring.</p>
  
  <p>My current approach is to use "root mean squared error", ie. the square root of the mean of the errors squared, where the error is <em>1.0-prediction</em> if the event occurred, or <em>prediction</em> if the event did not occur.</p>
  
  <p>The algorithms have no specific applications, but a common one will be to come up with a prediction of an event occurring for each of a variety of options, and then selecting the option that maximizes this probability.  The benefit to us is directly proportional to the rate at which the desired event occurs among the options that have the highest predicted probabilities.</p>
  
  <p>It has been suggested that RMSE may not be the best option for this, and I'm interested in the opinions of others.</p>