Should I remove the units of a neural network or increase dropout?
There is no incentive to increase the size of the model for not reason. If a model of size x gives the best possible performance, there is no reason to use a model of size 2*x with 0.5 dropout during training. Usually we want to find the smallest possible model with the best performance. Inflating the model just results in higher computational requirements.

You are basically suggesting to use dropout to allow the network to learn the same features more than once (creating a redundancies in the model). That is not the purpose of dropout.
Dropout is used to enable the network break unnecessary correlations which occur in the training set. For example if class 1 is the only one that contains in it features A and B, but all the training samples always feature them together. The dropout process can make the model realize that even just one of them is enough to point to class 1.