Why Do Neural Networks Prefer Dense Features
A one-hot encoded feature will act like a lookup table, picking out from the first layer in the network a dense representation for that categorical value. The first layer transforms the one-hot into a dense representation via matrix multiplication.