Training object detection models from the scratch without using pre-trained models and weights
Maybe you could first train the model on a large sized flower dataset, that way the conv layer will be optimised to detect flowers and then do transfer learning on that model using your custom dataset freezing all but the last layer.
I am not sure though, 25 images seem way too less. Try data augmentation on those images like horizontal and vertical flips, noise, shear etc.