In probability theory, Hoeffding's inequality provides an upper bound on the probability that the sum of bounded independent random variables deviates from its expected value by more than a certain amount. Hoeffding's inequality was proven by Wassily Hoeffding in 1963. Hoeffding's inequality is a special case of the Azuma–Hoeffding inequality and McDiarmid's inequality. It is similar to the Chernoff bound, but tends to be less sharp, in particular when the variance of the random variables is small. It is similar to, but incomparable with, one of Bernstein's inequalities. Statement Let be independent random variables such that almost surely. Consider the sum of these random variables, Then Hoeffding's theorem states that, for all , Here is the expected value of . Note that the inequalities also hold when the have been obtained using sampling without replacement; in this case the random variables are not independent anymore. A proof of this statement can be found in Hoeffding's paper. For slightly better bounds in the case of sampling without replacement, see for instance the paper by . Example Suppose and for all i. This can occur when Xi are independent Bernoulli random variables, though they need not be identically distributed. Then we get the inequality for all . This is a version of the additive Chernoff bound which is more general, since it allows for random variables that take values between zero and one, but also weaker, since the Chernoff bound gives a better tail bound when the random variables have small variance. General case of sub-Gaussian random variables The proof of Hoeffding's inequality can be generalized to any sub-Gaussian distribution. In fact, the main lemma used in the proof, Hoeffding's lemma, implies that bounded random variables are sub-Gaussian. A random variable is called sub-Gaussian, if for some c>0. For a random variable , the following norm is finite if and only if is sub-Gaussian: Then let be zero-mean independent sub-Gaussian random variables, the general version of the Hoeffding's inequality states that: where c > 0 is an absolute constant. Proof The proof of Hoeffding's inequality follows similarly to concentration inequalities like Chernoff bounds. The main difference is the use of Hoeffding's Lemma: Suppose is a real random variable such that almost surely. Then Using this lemma, we can prove Hoeffding's inequality. As in the theorem statement, suppose are independent random variables such that almost surely for all i, and let . Then