Train a cascaded network with a classifier in between
So basically, you want to do a classification task followed by a reconstruction task.
Here is what I suggest (I do not pretend this the absolute best solution, but it's how I would approach this problem) :
You can create a single task that does Classification--> Reconstruction with a single loss. Let's still separate this network in two and call net_class the part that does classification , and net_reconstruct the part performing reconstruction.
Let's say your classification network predicts {'B': 0.1, 'C': 0.9). Instead of using only image 'C' for reconstruction, I would feed both pairs (A-B and A-C) to the second network and compute a reconstruction loss L (I'm not an expert in reconstruction, but I guess there are some classical losses in this).
Therefore, you would compute two losses L(A-B) and L(A-C).
My total loss would be 0.1 * L(A-B) + 0.9 L(A-C). This way, you would train net_class to choose the pairing that minimizes the reconstruction loss and you would still train net_reconstruct to minimize both losses, and the loss is continuous (and therefore, differentiable according to AI experts ;) ).
The idea behind this loss is three-fold :
1 - Improving the reconstructor makes the loss go down (since both L(A-B) and L(A-C) would decrease. therefore, this loss should make your reconstructor converge towards something you want.
2 - Let's imagine your reconstructor is pretty much trained (L(A-B) and L(A-C) are relatively low). Then, your classifier has an incentive to predict the class which has the lowest reconstruction loss.
3 - Now, your reconstructor and your classifier will train at the same time. You can expect, at the end of the training, to have a classifier that would output pretty much binary results (like 0.998 vs 0.002). AAt that point, your reconstructor will almost only train on the scene associated with the 0.998 ouput. This should not be a problem, since, if I understood correctly your problem, you want to perform the reconstruction part only for the top classified scene.
Note that this method also works if you're not performing deep learning for the reconstruction part.
If you want some inspiration on this kind of topic, I recommend you read some blog posts about GANs (Generative Adversarial Networks). They use the same two stage - one loss trick (with some slight differences of course, but the ideas are very close).
Good luck !
I am attempting to do a two-fold task. The input is an image and based on the input I want to pick another image from a set of images (classification task) and then use both the images to obtain an output tensor. Clearly, I can train both the models separately if I know the ground truth of which image I should pick from that set. But, I only have the output tensor ground truth.
The problem, as it appears to me, is that if we employ a classification layer, the gradients will not be differentiable anymore. How do I deal with this problem? Is there literature which uses this kind of architecture for any application? TIA
More details: I have multiple images of an object/scene and I want to use two of those images for some kind of reconstruction problem. To maximize the performance of reconstruction, I want to smartly choose the second image if I am given the first image. For eg., I have three images A, B, C and using AC gives the best result. I need a model which given A predicts C and then using AC I can achieve the reconstruction. Is the task clear now? I do not have ground truth which says AC is better than AB. Is the task clear now?