title: Computing ROC curve to verify the quality of segmentation algorithm
body: <p>1) Let's say I have a set of 100 images, from which I manually extracted the foreground and the background (ground truth).</p>
  
  <p>2) Now I have a segmentation algorithm algorithm which I apply to the previous dataset. The results depend on several algorithm parameters (let's assume I do color segmentation, so the results depend on different color space thresholds in use). The result of the segmentation is a binary mask.</p>
  
  <p>3) So for each of the 100 images I have a ground truth and one prediction. I can easily compute the confusion matrix per image or the final confusion matrix for all the images.</p>
  
  <p>My question: How can I compute the ROC curve?</p>
  
  <p>Ideas:</p>
  
  <p>a) I am thinking to have one ROC curve for each parameter of the algorithm. So I keep N-1 parameters constant  and I vary the other in a normalized scale [0,1]. So for every change in [0,1] I need to re-run the segmentation for all images and compute the TPR and FPR which is a point in the ROC curve. But in this approach I could never evaluate the classifier for all parameters at the same time.</p>
  
  <p>b) Maybe the Roc curve is not the ideal evaluation metric for this case and I have to use something else eg F1 score.</p>
  
  <p>This is a well known <a href="http://www.hpl.hp.com/techreports/2003/HPL-2003-4.pdf" rel="nofollow">tutorial on ROC curves</a> but in my case I can't compute those scores shown in Figure 3 (they seem to be a metric that shows how confident you are for your classification result, if I understood correctly).</p>