Why Do Neural Networks Prefer Dense Features Reading through this tf-ranking notebook there is an interesting note that typically networks take in dense features, this is true in my general experience. But I am wondering why, if I were to pass in a categorical feature as a one hot encoded representation would it be worse?


  https://colab.research.google.com/github/tensorflow/ranking/blob/master/tensorflow_ranking/examples/handling_sparse_features.ipynb#scrollTo=aQ-VTA56sOTA
A one-hot encoded feature will act like a lookup table, picking out from the first layer in the network a dense representation for that categorical value. The first layer transforms the one-hot into a dense representation via matrix multiplication.