Training a CNN from scratch over COCO dataset I am using Tensorflow Object Detection API for training a CNN from scratch on COCO dataset. I need to use this specific configuration.
There is no pre-trained model on COCO with that configuration and this is the reason why I am training from scratch.

However, after 1 week of training and evaluating each checkpoint generated by the training phase this is how my learning phase appears on Tensorboard:



Thus, my questions are:


does anyone know how many iterations approximately will be necessary? Right now I did more than 500'000 iterations.
How can be possible that after 500'000 the evaluation is 0,8%? I would expected something like 60-70%.
Why does there is a sudden drop after 500k iterations? I thought that the eval was supposed to converge to some limit. (this is what SGD should do)
Is there any 'trick' to speed up the training phase? (ex: increasing the learning rate, etc).
It's hard to know for sure what's gone wrong, but here are some possibilities:


The problem is difficult. The COCO  paper reports that a typical object covers just 4-6% of the image. A randomly initialized model is therefore likely to do extremely poorly, with an expected precision of between 4 and 6% for detecting the object of interest in a frame. You also have 90 classes in your configuration file. It's not clear whether the model has access to the correct label, but if it's also inferring the class, we'd expect initial accuracy somewhere around 0.06%. That's actually about the precision of your starting model.
You're training on mini-batches of size 32. It's not clear to me from your config whether the lower axis of that graph is iterations or epochs. If it's the former, then your model will have seen only 1.5 million examples during the entire training period. For a problem this hard, that's not nearly enough. Indeed, COCO contains only 328k unique examples, so the model would have seen each of them just 5 times (corresponding to 5 epochs). If you've done 500,000 epochs though, then that ought to be enough.
It's possible that the hyper-parameters are not well set. I'm not an expert at training CNNs, but deep networks are notoriously finicky. I have difficulty reading your config file, but your learning rate looks reasonable to me. It also appears to undergo exponential decay over time though, and it's not clear to me that you want to be doing that, or whether the schedule you're using for the decay makes sense. This might be worth reviewing.