Faster RCNN tensorflow object detection API : dealing with big images
It looks to me like you are training on images with a different aspect ratio than what you are testing on (square vs not square) --- this could lead to a significant degradation in quality.  
Though to be honest I'm a bit surprised that the results could be really bad, if you are just visually evaluating, maybe you also have to turn down the score thresholds for visualization.
I have images of a big size (6000x4000). I want to train FasterRCNN to detect quite small object (tipycally between 50 150 pixels). So for memory purpose I crop the images to 1000x1000. The training is ok. When I test the model on the 1000x1000 the results are really good. When I test the model on images of 6000x4000 the result are really bad...
I guess it is the region proposal step, but I don't know what I am doing wrong (the keep_aspect_ratio_resizer max_dimension is fix to 12000)...
Thanks for your help !