Should I set higher dropout prob if there are plenty of data?
You are right, increasing the dropout proportion will help.

However, this looks like a setting where early stopping will be a very good choice. Although dropout, weight decay and batch-norm can work propperly, the fact that you easily overfit your training set would make it an appropiate scenario to try early stopping.

In addition, as the neural network takes very short to be trained you can train many of them (on some subset of the training set, making them weak learners) and create an ensemble to make the final predictions.