anchor.diff:  
   Unitizing measures basically work by internally concatenating all documents into a single long virtual document and then consider partial overlaps of annotations from different annotations. I.e. there is no averaging over documents. The partial overlap agreement is calculated based on character positions, not on token positions. So if one annotator annotates *the blackboard* and another annotator just *blackboard*, then the partial overlap is comparatively high because *blackboard* is a longish word. Relation and chain layers are presently not supported by the unitizing measures.
   
  -Agreement is calculated in two steps:
  +== Incomplete annotations
   
  -. *Generation of positions and configuration sets* - all documents are scanned for annotations and 
  -   annotations located at the same positions are collected in configuration sets. To determine if
  -   two annotations are at the same position, different approaches are used depending on the layer
  -   type. For a span layer, the begin and end offsets are used. For a relation layer, the begin and end
  -   offsets of the source and target annotation are used. Chains are currently not supported. 
  -. *Calculation of pairwise agreement* - based on the generated configuration sets, agreement is calculated.
  -  There are two cases where a configuration set may be omitted from the pairwise agreement calculation:
  -.. one of the users did not make an annotation at the position;
  -.. one or both of the users did not assign a value to the feature on which agreement is calculated
  -   at the position.
  -
  -The lower part of the agreement matrix displays how many configuration sets were used to calculate
  -agreement and how many were found in total. The upper part of the agreement matrix displays the
  -pairwise Cohen's kappa scores.
  +When working with coding measures, there is the concept of *incomplete annotations*. For a given
  +position, the annotation is incomplete if at least one annotator has *not* provided a label. In the
  +case of the pairwise comparisons that are used to generate the agreement table, this means that one
  +annotator has produced a label and the other annotator has not. Due to the way that positions are
  +generated, it also means that if one annotator annotates *the blackboard* and another annotator just
  +*blackboard*, we are actually dealing with two positions (*the blackboard*, offsets 0-15 and 
  +*blackboard*, offsets 4-14), and both of them are incompletely annotated. Some measurs cannot deal
  +with incomplete annotations because they require that every annotator has produced an annotation. In these
  +cases, the incomplete annotations are *excluded* from the agreement calculation. The effect is that
  +in the *(the) blackboard* example, there is actually no data to be compared. If we augment that
  +example with some other word on which the annotators agree, then only this word is considered, 
  +meaning that we have a perfect agreement despite the annotators not having agreed on *(the) blackboard*.
  +Thus, one should avoid measure that cannot deal with incomplete annotations such as Fleiss' kappa
  +and Cohen's kappa except for tasks such as part-of-speech tagging where it is known that positions
  +are the same for all annotators and all annotators are required (not expected) to provide an annotation.
   
   The agreement calculations considers an unset feature (with a `null` value) to be equivalent to a
   feature with the value of an empty string. Empty strings are considered valid labels and are not
  -excluded from agreement calculation.
  +excluded from agreement calculation. Thus, an *incomplete* annotation is not one where the label is
  +missing, but rather one where the entire annotation is missing.
   
  -Annotations for a given position are considered complete when both annotators have made an
  -annotation. Unless the agreement measure supports `null` values (i.e. missing annotations),
  -incomplete annotations are implicitly excluded from the agreement calculation. If the agreement
  -measure does support incomplete annotations, then excluding them or not is the users' choice.
  +In general, it is a good idea to use at least a measure that supports incomplete data (i.e. missing
  +labels) or even a unitizing measure which is able to produce partial agreement scores.
   
   .Possible combinations for agreement
   |====
  
positive.code: 
  .Supported agreement measures
  |====
  | Measure | Type | Short description |
  
  | Cohen's kappa
  | Coding
  | Chance-corrected inter-annotator agreement for two annotators. The measure assumes a different probability distribution for all raters.
  
  | Fleiss' kappa
  | Coding
  | Generalization of Scott's pi-measure for calculating a chance-corrected inter-rater agreement for multiple raters, which is known as Fleiss' kappa and Carletta's K. The measure assumes the same probability distribution for all raters.
  
  | Krippendorff's alpha (nominal)
  | Coding
  
negative.code: 
  |====
  
  == Stacked annotations
  
  Multiple interpretations in the form of stacked annotations are not supported in the agreement 
  calculation! This also includes relations for which source or targets spans are stacked.
  
  
  == Pairwise agreement matrix
  
  The lower part of the agreement matrix displays how many configuration sets were used to calculate
  agreement and how many were found in total. The upper part of the agreement matrix displays the
  pairwise agreement scores.
  
  Annotations for a given position are considered complete when both annotators have made an
  annotation. Unless the agreement measure supports `null` values (i.e. missing annotations),
  incomplete annotations are implicitly excluded from the agreement calculation. If the agreement
  measure does support incomplete annotations, then excluding them or not is the users' choice.
  