Unsupervised learning if existing image captions match the images
A thought:

Maybe an approach like Show and Tell: A neural image caption generator would be useful to look at. There they use exactly as you say a combination of CNN (for image) features and LSTM (for text) + search to generate text given the features from the image. 

It is possible that then a discriminator can be used to learn whether 'generated text' is close enough to 'given label' in order to give the final result.

It is also possible that you do not need the discriminator, and instead of using the output of the show and tell network directly (like removing the last linear layer) and feed one of the earlier steps (which represents a probability distribution of possible texts to generate) into the discriminator so that it has more possible combinations of text outputs to work with...

This is probably more supervised than you want, but maybe starting with pre-trained weights of this model can get you somewhere faster.