anchor.diff:  
   .Supported agreement measures
   |====
  -| Measure | Type | Short description |
  +| Measure | Type | Short description
   
   | Cohen's kappa
   | Coding
  -| Chance-corrected inter-annotator agreement for two annotators. The measure assumes a different probability distribution for all raters.
  +| Chance-corrected inter-annotator agreement for two annotators. The measure assumes a different probability distribution for all raters. Incomplete annotations are always excluded.
   
   | Fleiss' kappa
   | Coding
  -| Generalization of Scott's pi-measure for calculating a chance-corrected inter-rater agreement for multiple raters, which is known as Fleiss' kappa and Carletta's K. The measure assumes the same probability distribution for all raters.
  +| Generalization of Scott's pi-measure for calculating a chance-corrected inter-rater agreement for multiple raters, which is known as Fleiss' kappa and Carletta's K. The measure assumes the same probability distribution for all raters. Incomplete annotations are always excluded.
   
   | Krippendorff's alpha (nominal)
   | Coding
  
positive.code: 
  |====
  
    
  CAUTION: Multiple interpretations in the form of stacked annotations are not supported in the agreement 
        calculation! This also includes relations for which source or targets spans are stacked.
  
  
negative.code: 
  Unitizing measures basically work by internally concatenating all documents into a single long virtual document and then consider partial overlaps of annotations from different annotations. I.e. there is no averaging over documents. The partial overlap agreement is calculated based on character positions, not on token positions. So if one annotator annotates *the blackboard* and another annotator just *blackboard*, then the partial overlap is comparatively high because *blackboard* is a longish word. Relation and chain layers are presently not supported by the unitizing measures.
  
  == Incomplete annotations
  
  When working with coding measures, there is the concept of *incomplete annotations*. For a given
  position, the annotation is incomplete if at least one annotator has *not* provided a label. In the
  case of the pairwise comparisons that are used to generate the agreement table, this means that one
  annotator has produced a label and the other annotator has not. Due to the way that positions are
  generated, it also means that if one annotator annotates *the blackboard* and another annotator just
  *blackboard*, we are actually dealing with two positions (*the blackboard*, offsets 0-15 and 
  *blackboard*, offsets 4-14), and both of them are incompletely annotated. Some measurs cannot deal
  with incomplete annotations because they require that every annotator has produced an annotation. In these
  cases, the incomplete annotations are *excluded* from the agreement calculation. The effect is that
  in the *(the) blackboard* example, there is actually no data to be compared. If we augment that
  example with some other word on which the annotators agree, then only this word is considered, 
  meaning that we have a perfect agreement despite the annotators not having agreed on *(the) blackboard*.
  Thus, one should avoid measure that cannot deal with incomplete annotations such as Fleiss' kappa
  and Cohen's kappa except for tasks such as part-of-speech tagging where it is known that positions
  are the same for all annotators and all annotators are required (not expected) to provide an annotation.
  
  The agreement calculations considers an unset feature (with a `null` value) to be equivalent to a
  feature with the value of an empty string. Empty strings are considered valid labels and are not
  excluded from agreement calculation. Thus, an *incomplete* annotation is not one where the label is
  missing, but rather one where the entire annotation is missing.
  
  In general, it is a good idea to use at least a measure that supports incomplete data (i.e. missing
  labels) or even a unitizing measure which is able to produce partial agreement scores.
  
  .Possible combinations for agreement
  |====
  