1. a method of classifying markings on images in a document: according to marking type, the method comprising: supplying the document containing the images to a segmenter; breaking the images received by the segmenter into fragments of foreground pixel structures that are identified as being likely to be of the same marking type by finding connected components, extracting near horizontal or vertical rule lines, and splitting at least some connected components to obtain the fragments; supplying the fragments to a classifier, the classifier providing a category score to each fragment, wherein the classifier is trained from ground truth images whose pixels are labeled according to known marking types; and assigning a same label to all pixels in a fragment when the fragment is classified by the classifier.