Big number of false positives in binary classification
It's hard to know, from just the information you've given us.  It could be any number of things: overfitting, insufficient training data, class imbalance, or something else.  The first step is probably to figure out the source of the problem, then you can explore the solutions.

There are standard diagnostics to check for overfitting (cross-validation, comparing accuracy on the validation set to accuracy on the training set, etc.). I'd suggest you use them to diagnose whether the network seems to be overfitting.  There are many ways to combat overfitting: regularization, early-stop training, dropout, etc.

There are also ways to check if the problem is that you just don't have enough training data.  For instance, you can take different-size subsets of your training set (10K, 20K, 30K, 40K, 50K) and plot how accuracy depends on the size of the training set.  If accuracy seems to be gradually improving as training set size increases and doesn't appear to have reached a plateau yet, that's a suggestion that obtaining more training data might be useful.

Finally, class imbalance might be a factor.  When you have 5 times as many non-food instances as food instances, that's call class imbalance.  There's lots written on that subject and many techniques (undersampling the minority class, oversampling the majority class, using class weights, adjusting the threshold for classification, and more).  I suggest you do some reading about that topic.  It's not necessarily true that you need to train on a training set with balanced classes; that's one strategy, but it's not the only one, and it may not be optimal.

Hopefully this will help you narrow down what might be going on and then ask a more focused question.