How to calculate the redundancy of a language I originally asked this question on the information security stack exchange but I think it's better suited here.

I'm trying to calculating the unicity distance for a cipher applied to a language I wrote and I'm having trouble understanding the concept of redundancy in a language.

From this book 


  The redundancy of a language severely reduces the amount of information conveyed with each character and the rate of a language is defined as the average number of bits of information contained in each character of a message, i.e. $\frac{H(X)}{N}$ where $N$ is the number of characters in the message. 


You need to generate a frequency table for a language to calculate $H(X)$ but an accurate frequency table requires a long message. Won't this cause all redundancies to tend towards 0 since $N$ can be arbitrarily long? How exactly is the redundancy for English, 1 to 1.5 given by this book, calculated? What value of $N$ do they use?
Not really, for example if your language has $m$ letters, and the letters are independent identically distributed then $H(X)=H(X_1,\ldots,X_N)=N H(X_1)$ where $H(X_1)$ is is the entropy of the single letter distribution $(p(1),\ldots,p(m)),$ where $P[X_k=u]=P[X_1=u]=p(u)$ for any $k\geq 1.$

This is because for independent random variables
$$
H(X_1,\ldots,X_n)=
$$
$$
=H(X_1)+H(X_2|X_1)+H(X_3|X_1 X_2)+\cdots+H(X_N|X_1X_2\cdots X_{N-1})=
$$
$$
=H(X_1)+H(X_2)+\cdots+H(X_N).
$$

So $X$ in that statement refers to the sequence of $N$ characters.

If your language is first order markov then
$$
H(X_1,\ldots,X_n)=
$$
$$
=H(X_1)+H(X_2|X_1)+H(X_3|X_1 X_2)+\cdots+H(X_N|X_1X_2\cdots X_{N-1})=
$$
$$
=H(X_1)+H(X_2|X_1)+H(X_3|X_2)+\cdots+H(X_N|X_{N-1})\geq (N-1)H(M).
$$
where $H(M)$ is the entropy of the Markov chain, i.e.,
$$
H(M)=\sum_{s\in S} \mu_s H(X_{n+1}|X_n=s)
$$
where $\mu_s$ is the steady state probability of being in the state $s$. Of course we need to assume a stationary Markov chain here.

See Shannon's classical paper Prediction and Entropy of Printed English (here )for a number of ways of calculating the entropy.