How to determine if dataset is a suitable representation of the context? How can I determine if the data I have collected is a good enough representation of the context? For example, I am working on an object detection system and have been building an image dataset. How can I know if my dataset represents the task? For example, I need to account for instances where the object is close up and far away. But what about contexts/situations I have missed, or not even considered? Is there an art to building datasets?
Before collecting the data, domain knowledge is used to determine plausible variations the task may present. It is not necessary to capture in data all the variations experts may point out (for example, some may be syntheticaly constructed). But domain knowledge indeed dictates the variations a certain task may have to take account for.
That being said, one then gathers the data trying to match plausible variations in the percent that domain experts dictate. Again some may even be syntheticaly constructed a-posteriori. Then we have a representative dataset.
In case the dataset proves inadequate, this means domain knowledge needs to be updated with new information about the task.