Relation between hyperparamters and training set for an object detection model
A major difference between the first and the second model you trained is the size of the data assuming that the model is not pretrained. Increased data, of course, needs increased epochs. According, the batch size must also increase.

Batch Size:


While training on the smaller dataset, a batch size of 10 yielded better results. The errors were averaged over 10 samples and then back-propagated through the model. But now for the larger dataset, the batch size remains the same and hence only little optimization occurs as the error is averaged over 10 samples only ( which is relatively smaller for a large dataset ).


Learning Rate:


For the larger dataset, the number of epochs is increased. The purpose of the learning rate is to scale the gradients of the loss with respect to the parameter. A smaller learning rate always helps as it prevents the overshooting of the minima of the loss function. I would insist you increase the learning rate so that the optimization does not diminish as we are having a larger number of epochs. Gradually decrease the learning rate, as the loss approaches its minima.



  If you are training a popular architecture ( like Inception, VGG, etc. ) and that too on datasets like ImageNet, COCO with little modifications, you should definitely read the research papers published on various problems as they would provide a better start to the training.