Two-tower net does not learn when made deep
I now have a working network. It turned out that the gradients were all zeros after only about 3000 update step. I tried two approaches to fix this - using Batch Normalization after each activation function in the feed-forward net and changing the activation function from ReLU to Leaky ReLU. Both worked, and I ended up using the Leaky ReLU without normalization.
For the full thread in the PyTorch discussion forum here