How can we get small test error reducing only train error? My question is about mathematical part of machine learning algorithms, especially about using it in neural networks. We train network reducing train error and I was thinking about how then test error is also small (discarding overfitting and underfitting), what is mathematical proof for it. I found about that in this book http://www.deeplearningbook.org, chapter 5.2. Here is a part of that chapter:


  How can we affect performance on the test set when we can observe only
  the training set? The field of statistical learning theory provides
  some answers. If the training and the test set are collected
  arbitrarily, there is indeed little we can do. If we are allowed to
  make some assumptions about how the training and test set are
  collected, then we can make some progress. The training and test data
  are generated by a probability distribution over datasets called the
  data-generating process. We typically make a set of assumptions known
  collectively as the i.i.d. assumptions. These assumptions are that the
  examples in each dataset are independent from each other, and that the
  training set and test set are identically distributed, drawn from the
  same probability distribution as each other. This assumption enables
  us to describe the data-generating process with a probability
  distribution over a single example. The same distribution is then used
  to generate every train example and every test example.


I want to transfer this theoretical part to a more concrete example for better understanding. Assume we have images and we need to select train and test datasets. My questions are


What the author means by "example"? Is it every image? If yes, is that mean, that we consider every image as a random variable and every time it selected or not?
I totally can't understand why we need to do this assumption: "the examples in each dataset are independent from each other". Maybe I misunderstood the example part in my first question and it's come from that.
If the training set and test set are identically distributed why always in practice the training set is about 90% of all data and the test set is 10%, I can't see identical distribution in this situation.


Also, I would be glad if you will give me some references to read about this in other books/topics.
Yes, if you are classifying images, each example would be an image.  There is some distribution on images that are likely to arise in practice.

We need independence and identical distributions because without it we can't learn.  If we practice five times, but then are tested on something entirely different, our practice might not be useful.

The distribution refers to the distribution of each single image.  That has nothing to do with how many images you draw from that distribution.  Imagine flipping a biased coin.  I could flip the coin 100 times; each outcome would be a sample from that (biased) distribution, and that's true whether we have 10 coin flips or 1000 coin flips.