Preparing custom dataset for object detection using ML
I have the following suggestion:


The size of your data-set is small, you should increase the sample size. You also have to use data augmentation methods for improving the performance, translation, orientation and blurring are highly recommended. 
For your task I guess the most popular approach is what which is discussed in YOLO paper, and it is highly recommended using that if you want to have object detection not just localization and recognition for good performance.
I also suggest you providing real data for your task. The images of first row are artificially made. You have to provide real data which your classifier is going to face. The distribution of your test time should be like validation and train time if you want to have good performance.