Mask RCNN: Random predictions during inference for the same image
The problem was that some layers were not getting their weights loaded and were being initialised randomly. The random initialisation of those weights were the cause of random predictions during inference.