How to properly rotate image and labels for semantic segmentation data augmentation in Tensorflow?
I have found the following solution. 

Labels

Here is a snippet filling an empty space in label with ignore label.

IGNORE_LABEL = 255
label = tf.subtract(label, IGNORE_LABEL)
label = tf.contib.image.rotate(label, angle)
label = tf.add(label, IGNORE_LABEL)


This way, by subtracting IGNORE_LABEL, you shift the interval to [-255, 0]. When label (segmentation labels) is rotated, empty spaces are filled with zeros. By adding back IGNORE_LABEL, you shift interval back to [0, 255] meaning that empty space is set to IGNORE_LABEL.

Images

Similar trick can be applied to image rotation to fill empty space with preferred value.