Discriminator of a Conditional GAN with continuous labels
I think the answer is, you have to either create a very large number of categories, or do it completely differently than with the code I provided.  The paper as a good solution to the problem.