Finding Statistical Signifigance for a Classifier I recently did work in the area of machine learning for one of my jobs and was able to build a classifier which evaluated to an F score of 85%. I also have access to correctly and incorrectly classified instances. My boss would like to make sure my classifier is good and and test it for statistical significance. I'm not quite sure what he means by this. I have taken a course in statistics quite some time ago where I did hypothesis testing and stuff with confidence intervals. Do you think that's what he means or does he want me to compare it with other algorithms? I don't want to say what kind of data set I am working with but it's comparable to something such as predicting spam or ham for email messages. 

Any help or guidance on this question would be greatly appreciated. I am pretty new to the area of computer science research. But am I right in saying that statistics are used quite a lot to evaluate these types of things?

Thank you!
You can take a look at McNemar's Test or a Bayesian Testing. If you are comparing two or more classifiers, you can perform a to evaluate their performances.

Evaluating learning is actually a pretty wide research topic. A good book to refer to would be Japkowicz-Shah. This is a really nice book on evaluating classifiers and performances. To get some idea about the topic, take a look at this ICMLA talk.

If this is a one time thing, get done with McNemar or Bayesian test, however if you continue to work in the field, do see the book.

P.S. ask you boss if he means any real stat tests (like those mentioned above). Once I was working on a classifier, my professor asked me to evaluate its performance, after a long period of pain and agony of cluelessness, it turned out he just wanted me to test it on other 'toy problems'.