How do I assess which sentiment classifier is best for my project?
A couple of important points:


Sentiment analysis is not an exact science. Two people, reading the same text in different contexts will come to different conclusions about sentiment, especially on borderline cases. Perhaps text has complex grammar, or has a metaphor or simile in it where it helps to understand what is actually being compared.
The ground truth for sentiment data sets is established by people. The best you can hope for from any trained ML classifier is that it closely matches the opinion of those people when it predicts on new data.
Your different sentiment analysis tools could well vary in quality, because they will have been built with different technologies at different times, and with different training data.


There are a couple of ways I can think of that you could assess the classifiers for your purpose:

1. Test against your own labelled test data

It is important to source this yourself to ensure it does not overlap with training data used by any of the models you want to compare (if it did overlap then it gives a big advantage to any model that trained using it). You will need to collect ground truth data on sentiment for all the texts.

Before you test, it is a good idea to pick a metric that you care about for your intended use. Accuracy is not the only metric, but it is probably fine for your purpose. You might choose other metrics if there are different costs associated with incorrect classifications - e.g. if assigning negative sentiment needs to be done cautiously.

Then run each classifier across your test set and calculate the metric. The classifier with the best metric is your best guess at the one you should use.

If you have a large enough test set, you can split it into parts and get a measure of the estimate error in your metric. This will help you see whether differences in the test metrics are significant.

This can be the best approach, especially if you can source data that you know matches your project goals. However, it would take a long time and a lot of effort.

If you are in a hurry, you could just search for and use someone else's dataset e.g I found these ones on a quick web search: a Kaggle competition and a free sample hosted by Niek Sanders (I have no idea of the quality of these - you should sample them and see if the data is useful for you). There is a risk though that these data sets were used to train one of your classifiers, so it would give you a false high rating.

2. Read up on associated reports and papers

Each of the sentiment analysers should explain the model class and data used to train it in documentation. Quite often you can find papers comparing the different approaches and quoting accuracy scores on a standard data set. If you are lucky, you will find enough comparisons that you get some sense of which algorithms are considered "cutting edge" and which are out-dated.

3. Collect text that is classified differently by algorithms you are considering, and get feedback

A variation of (1), you can do this at any stage, and it might work well in the context of a live product. For all the data that is classified, log how different classifiers respond to it, and where there is discrepancy, save it for later assessment. You will need some people to read and help classify the text (ideally without seeing what the classifiers thought), and you can collect metric data over time and use the classifier that does best once you have collected enough examples to make a clear decision.



Ultimately what matters is the results of using your classifier in a project. Just labelling data has no inherent purpose. It is the consequences of those labels that matter. So you need to be driven by results. If you use one classifier and get feedback that the product is not performing well, you may be able to trial another one. If your product is used by many people, and enough are giving feedback of quality, you could even A/B test a couple of classifiers in production.