Neural Net Accuracy: Test Set vs Real World Data
Many resources teach the process of splitting data into training, validation and test sets. This is what you want to do for "closed" datasets where it's not possible to get additional data.

This assumption of a closed dataset is often not true in the real world, where it may be feasible to collect more data. Statistically speaking, it is a lot more desirable to define a test set as a new data sample that was collected separately from your training data. This might be more representative of how the model will behave in production, but sometimes even this is not enough:

A few weeks back I built an image classifier for cars. I trained it using a mix of existing datasets and the results of a web scrape. Ultimately, it was deployed through an iOS app where it was supposed to do make predictions in real time. In this case, it was not enough to just create a test split or collect a new sample from the web. We needed to shoot our own images that were representative for the use case in order to make realistic assumptions about the app's performance.