What does the notation mAP@[.5:.95] mean?
mAP@[.5:.95](someone denoted mAP@[.5,.95]) means average mAP over different IoU thresholds, from 0.5 to 0.95, step 0.05 (0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95).

There is
an associated MS COCO challenge with a new evaluation
metric, that averages mAP over different IoU thresholds,
from 0.5 to 0.95 (written as “0.5:0.95”). [Ref]


We evaluate the
mAP averaged for IoU ∈ [0.5 : 0.05 : 0.95] (COCO’s
standard metric, simply denoted as mAP@[.5, .95])
and mAP@0.5 (PASCAL VOC’s metric). [Ref]


To evaluate our final detections, we use the official
COCO API [20], which measures mAP averaged over IOU
thresholds in [0.5 : 0.05 : 0.95], amongst other metrics. [Ref]

BTW, the source code of coco shows exactly what mAP@[.5:.95] is doing:
self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True)
References

cocoapi

Inside-Outside Net: Detecting Objects in Context with Skip Pooling and
Recurrent Neural Networks

Faster R-CNN: Towards Real-Time Object
Detection with Region Proposal Networks

Speed/accuracy trade-offs for modern convolutional object detectors