Bib number recognition using Keras
We can break down the task into two parts. 

In the first part, our aim will be to localize regions where the bibs could be found. This is an object detection task, so you can use YOLO, Faster RCNN etc. YOLO could be a better choice as it is fast and efficient. We don't need semantic segmentation here. A number of GitHub repositories are available which enable to train a custom dataset for detecting bibs in images.

In the second part, as mentioned in the question, a CNN trained on MNIST will not be enough. We will need an OCR ( Optical Character Recognition ) which would get the text ( number ) from the image.


  The localizer will produce bounding box coordinates. The image will be
  then cropped by those coordinates. This cropped image could be passed
  into an OCR engine and finally, you will get the number present on the
  bib.


For training YOLO on a custom dataset, see here.

For an OCR engine, see here.