When training an object detection network for one class, should I include empty images in the dataset? I fine tuned MobileNetSSD for object detection using a dataset with just one class (~4000 images). All the training images include at least one bounding box related to that class (no empty images). By following the example with the VOC dataset, the labelmap includes two classes, the background and my custom class. However, as I mentioned, there are no annotations related to the background and I  am not sure if there should be any.

Now my fine tuned network performs very well when objects belonging to my class are present, however there are some false detections with very high confidence when the class is not present. Can this be related to the fact that I don't have empty images in my training set?
It is correct that to eliminate the false positives, the training must include images that reflect true negative cases. The objective of training is not this.


  Find the objects of class $O$ in scene $X$.


It is this.


  Find any objects of class $O$ in scene $X$.


Training is effective when sample $S$ is drawn according to the same distribution as will be used for drawing at test time or during actual use.

It is not necessary to annotate the background if the loss function is chosen so that finding nothing when nothing there is indicated as a zero loss and a proper distribution of those cases exist in the training sample.