Question on bootstrap sampling
It simply depends what you count as your object of interest: from your description the unit can be either document or annotation. Your method describes using the document as unit, it's fine as long as the tests you plan to do are compatible with this.
Another option is to use the annotation as unit: in this case you would pick 60% of the 44k annotations every time, so you would have a mix of annotations from multiple documents. Depending what you test exactly, this might be an issue, in particular I don't see how you would count False Negative cases in this way.
Since you have text documents of varying size (I assume), you could also consider different options: sentence, paragraph, block of N sentences, etc.