What does "Wide" vs. "Deep" mean in the context of Neural Networks?
The width refers to the number of neurons in a layer. The depth refers to the number of layers. 

Have a look at the following question regarding the impact of these hyper-parameters on the performance of the neural network: https://stats.stackexchange.com/q/214360/82135.