Normalization for well known data sets like coco-text and total text data set
Find the largest height and width amongst all the images. Let us call it H and W respectively. It is true that you cannot resize the images, but say if you have an image of height h and width w where h &lt; H, w &lt; W. To the right of the image append W - w number of columns and at the bottom of the image append H - h number of rows having some constant value (0 is okay for grey-scale and B/W images and 0 for each of the channel in case of colour images).

In this way all the images will be of same size. Since you appending at the right and bottom of the image, the annotations will not lose its meaning in the transformed image in terms of the position and content of the text to recognised.

You could also try pixelRNN kind of ideas after you are done with DNN. RNN can handle variable length inputs and in your case it will be sequence of pixels. Here you don't need to append rows and columns to the image.