CNN only performs well when split into 2 models
You could try saving the model after training the one it performs best on. Then restore it and try feeding in the 2nd set.

saver = tf.train.Saver()


Then at the end of training, add the line

save_path = saver.save(sess, "/your/file/path.ckpt")


Then to restore.

with tf.Session() as sess:
      saver.restore(sess, "/your/file/path.ckpt")