What does it mean for the training data to be generated by a probability distribution over datasets I was reading the Deep Learning book and came across the following para (page 109, second para):


  The training and test data are generated by a probability distribution over datasets called the data-generating process. We typically make a set of assumptions known collectively as the i.i.d. assumptions. These assumptions are that the examples in each dataset are independent from each other and that the training set and test set are identically distributed, drawn from the same probability distribution as each other. This assumption enables us to describe the data-generating process with a probability distribution over a single example.The same distribution is then used to generate every train example and every test example. We call that shared underlying distribution the data-generating distribution, denoted $p_{\text{data}}$. This probabilistic framework and the i.i.d. assumptions enables us to mathematically study the relationship between training error and test error.


Can somebody please explain to me the meaning of this paragraph?

On page 122 the last paragraph, it also gives an example 


  a set of samples $\{x(1), \dots, x(m) \}$ that are independently and identically distributed according to a Bernoulli distribution with mean $\theta$.


What does this mean?
There is a common assumption that data that is being modeled is independent and identically distributed (i.i.d.) samples from a probability distribution. There is the same underlying probability distribution for both the training and test datasets. And each sample is independent of the other samples.

Examples of these assumptions being violated:


Data is generated by a completely random process such as random walk.
The training and test dataset come from different probability distributions, either completely different probability distributions or the same probability distribution with different parameters.
Samples are not independent. An example of dependent samples is dealing cards from a single deck, the probability of later cards is dependent on previously dealt cards.


The model fitting process only has access to the data samples, not the underlying probability distribution. Parametric modeling fitting makes a guess about the functional form of that probability distribution (e.g., Bernoulli or Gaussian) and then, estimates the associated parameters.