what is the meaning of independent and identically distribution of samples in a dataset for neuralnetworks
Once the data is with us, the best we can do is shuffling the samples to make them Identically distributed assuming a hypothetical process behind it that generates all the samples.
Let's take the example of MNIST digits, the hypothetical process is our similar knowledge of digits, so most of us will write a specific digit in a similar way.
Now let's test the MNIST model on Japanese MNIST, it will fail as the assumption is broken.
Again, mix both the MNIST, Shuffle it, Split it and Train it. It will learn the pattern and can predict the digits.
Once again, both the train/test have a similar probability distribution of pixels.
While training the model,
If the process is incremental e.g. Gradient Descent, we try to assure that each batch represents the parent distribution. Again this is achieved by Shuffling esp. Stratified shuffling.