Keras custom loss using multiple input
When you are in the inference phase, there is no loss to be calculated unless you are using a validation set to fine tune your model on.
Using two models with shared layers in between is fine yet I suggest to start training the model and don't waste your time writing perfect code.