What is the best activation function for the embedding layer in a deep auto-encoder?
It is hard to tell what exactly is better because these are hyperparameters. However, the sigmoid activation function is closer to biological neurons.
In the paper below, Bengio demonstrates why ReLU activation functions are better for hidden layers. In summary, they increase the sparsity of calculations (matrix in each layer shod multiply to its relative weights), and because of that, it made it possible for data to classify faster and easier.
http://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf