How to create a multi-dimensional softmax output in Tensorflow? I came across this research paper released by YouTube, on how they use deep learning neural networks for recommendations. It's located here: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45530.pdf

In the paper, the candidate generation neural network model outputs a softmax with 256 dimensions, which acts as an "output embedding" of each of the 1M video classes.

How is this possible to implement in tensorflow, for example? Isn't softmax supposed to be only 1-Dimensional. If the model outputs an "embedding" like this, as they say it does, how would the training data's labels be formatted as 256-dimensional? In other words, how do they compute the 256-dimensional vector for each of the videos in their training dataset?

Thank you so much for your time and help, guys!
You are confusing "dimensions" with "order of tensor". A softmax with 256 different categories is a 256 dimensional vector, but is also a tensor with order 1 (whilst a matrix is a tensor of order 2). The paper is using the technical terms correctly, so the 256 dimensional vector is just a normal vector with 256 scalar entries.

Therefore a 256-dimensional softmax in TensorFlow is typically an output layer that looks something like this:

y = tf.nn.softmax(tf.matmul(h, W) + b)


where h is the last hidden layer, W is the weight matrix n x 256, and b is the bias 1 x 256 vector.


  In the paper, the candidate generation neural network model outputs a softmax with 256 dimensions, which acts as an "output embedding" of each of the 1M video classes


That is a description of the training process that compresses 1M different inputs to 256-dimensional output for use as an embedding for recommendation matches. The softmax is at the output, and as far as I can see is just a normal softmax classifier output as seen in many other classifier networks (except the result is not technically being used to classify anything). I am not clear on what supervision data was used or on what the input representation was. However, I don't think it likely that 1M "classes" ever appear as e.g. 1-hot encoding, because that would not scale out usefully to the many other millions of videos - the point of the embedding is to turn disparate features of the videos into something that be used as a similarity measure, that can be run on any video stored in YouTube.