Tensorflow Calculate error for a single neuron
Easy, you have already done it.

mse = tf.reduce_mean(tf.squared_difference(out, Y))

This command means you are taking the mean of all the neurons over all the examples.

error = tf.squared_difference(out, Y)

This command will give you the error over all neurons over all examples. But make sure you calculate against the correct axes.