Testing Model Performance
You should provide more information about your problem. How large is your data set? Do you have hyper-parameters? What is your classifier?

Normally, you should split your entire data set into three parts: training, validation and testing sets. The testing set you should never use before the model is finally tuned. If you data set is not large enough, you should you use the cross-validation method to measure the error of your classifier. 

There are several metrics to assess a performance of your classifier: accuracy, precision and recall, Cohen's kappa, F1-score, etc.

If you are working with python and scikit-learn, I would recommend to check this book. It is nicely written and explains basics of model assessments.