How to reduce dimensionality of 3.2B categorical features?
Based on this article, I have decided to run with an autoencoder (tanh, tanh, tanh) neural net as opposed to PCA.


  "[for] thousands of dimensions the autoencoder has a better chance of unpacking the structure and storing it in the hidden nodes by finding hidden features ... [Also,] in contrast to PCA, the autoencoder has all the information from the original data compressed in to the reduced layer. [Finally,] the autoencoder is better at reconstructing the original data set than PCA when k is small, however the error converges as k increases. For very large data sets this difference will be larger and means a smaller data set could be used for the same error as PCA."
  
  https://www.r-bloggers.com/pca-vs-autoencoders-for-dimensionality-reduction/


Autoencoders for dimensionality reduction are also explained in OReilly's TensorFlow book.



UPDATE:
It looks like an embeddings layer as the first hidden layer would also be a good start to reducing dimensionality.

UPDATE: keras pooling layers also seem relevant