Visualizing F-score differences in information extraction
Yes, I think that's a sound approach and a good way to compare different systems. 

A ROC curve comparison is usually more informative than the raw performance scores, but it's still quite general. In case you want to observe even more detail, you could also try to look at specific groups of instances. One way to do that is to count for every instance how many systems correctly classify it: an instance almost always correctly classified is "easy", and conversely an instance which is almost always misclassified is "hard". It's often interesting to look at what happens specifically for the "hard" instances with the different systems. You could take a subset of "hard" instances and calculate the performance or ROC curve only on those, in order to distinguish more precisely the best systems. 

For the record, if it makes sense for your task you might also want to consider more flexible scoring methods for text spans: currently it seems that your evaluation considers an answer correct only of the exact span is predicted. You could consider counting the fact that a span is partially correct, for instance by counting the number of tokens correctly annotated.