Training object detection model too long causes model to crack
After some research (and a lot of experiments) I have come to some conclusions which might help others facing the same problem.

Failing to detect any object probably means that the model has collapsed in some part of training. A possible indication of this in my case was some extremely high values of loss values when training. While I have models that this happened and does not seem to affect the model too much (at least not too much to make it useless) an increasing amount of these values is a good sign that something is fatally wrong. Notice that the frequency of appearance of these values is not necessarily increasing all the time.
More classes the better: After the first period I was conducting my experiments with just 1 class. While this is legitimate and nothing has been mentioned (to my knowledge) about avoiding it (I even read tutorials using 1 class object detection) this also intensified the crack rate. Using 2 classes seems to partially alleviate the problem. When I trained my model with 11 classes the problem seems to vanish.
Small variation in bbox's sizes helps: Experimenting with bounding boxes varying a lot in size I came to the conclusion that largely homogeneous bbox's sizes help to also alleviate the issue.
More classes may increase performance: I have noticed also that while 11 classes problem seem quite heavier task than 1 class due to the limitations in training the model with 1 class performance is some cases was higher in the former case.

My final conclusion is that my model for some reason was experiencing at some point an overflow on some weights that prevent it from returning to the normal again. The fact that I have augmented my data (maybe similar images were fed one after the other and intensified certain weights for example) maybe increased the frequency of the problem occurrence but the problem was still present in non-augmented dataset.
I am using object detection from tensorflow. 
Some background information about the problem
I was facing a challenge since I am interested in training a model which will among other things detect also weapons. My regular point of view is from a drone but I couldn't find any publicly available dataset from this point of view containing weapons. So, I collected images with weapons and tried to imitate as much as possible the drone point of view. 

One of my first thought was to pad my images (to make smaller my actual bounding boxes inside them in respect with the image itself). 
A second though was to also rotate my images. Since tensorflow has not implemented this feature I made a simple code to create prerotated images. This has limitation but anyway I gave it a shot.

I experimented with images of people at the beach and tried to imitate the actual problem: 

I am using relatively close images of people at the beach for training
And at the same time using relatively distant images of people at the beach for testing purposes.

The actual problem
At one of my experiments I trained my model for longer than usual and noticed that it really blew it. To be more specific: 

I have an initial 140 "close frame" images for training. 
From them I had created 4x140=560 images with small padding (resulting image is up to 1.6 times the original in the biggest version) and rotation (up to 5 degrees rotation) resized version.
I used this dataset of 560 images to train my model for almost 800k iterations using Faster R-CNN model.
I exported the model in various iterations:
At 150k, 300k, 500k, 700k and used to test it on my test images.

I have also used for comparison models which was trained with the original resized images to check the differences.
Un-expected results

The first peculiar observation is that my model at 500k and above it does not detect any person at all. (First thought was overtraining)
I tried it on the actual training images to check this and here also did not detect anything at my testing images. 
Also, I would expect my "enhanced" dataset to perform better than the simply resized one but this does not seem the case also. (This though is a minor problem which seem scientifically rational).

So, my question is how is that even possible? Training for too long can lead to being unable to detect training samples also? Is there any theoretical explanation for this? (Apart from errors in implementation etc which I think is not really probable but cannot be excluded in any case).
P.S.1
My resized version of the image follows the same resize conditions of Faster R-CNN with min. dim. being 600 pixels and max. one 1024 pixels. So, this should not be any resizing problem.
P.S.2
I am using tensorflow but I don't think this is tensorflow specific issue. So, I did not include it in tags.