title: Shannon entropy to mutual information
body: <p>I have some statistics over some properties like:</p>
  
  <pre><code>1st iter : p1:10 p2:0 p3:12 p4:33 p5:0.17 p6:ok p8:133 p9:89
  2nd iter : p1:43 p2:1 p6:ok p8:12 p9:33
  3rd iter : p1:14 p2:0 p3:33 p5:0.13 p9:2
  ...
  
  (p1 -&gt; number of tries, p2 -&gt; try done well, p3..pN -&gt; properties of try).
  </code></pre>
  
  <p>I need to calculate the amount of information of each property. 
  After some procedures of quantization (for ex. to 10 levels) to make all input numbers on the same level the input file starts to look like:</p>
  
  <pre><code>p0: 4 3 2 4 5 5 6 7
  p3: 4 5 3 3   
  p4: 5 3 3 2 1 2 3 
  ...
  </code></pre>
  
  <p>Where <code>p(0) = funct(p1,p2)</code>.</p>
  
  <p>Not every input line got every <code>pK</code> so <code>len(pk) &lt;= len(p0)</code>.</p>
  
  <p>Now I know how to calculate entropy of each property via Shannon entropy for each line. I need to calculate mutual information from here. </p>
  
  <p>Calculation of joint entropy for mutual information <code>I(p0,pK)</code> is stuck because of different lengths.</p>
  
  <p>I'm calculating entropy for one element like this:</p>
  
  <pre><code>def entropy(x):
      probs = [np.mean(x == c) for c in set(x)]
      return np.sum(-p * np.log2(p) for p in probs)
  </code></pre>
  
  <p>So, for joint I need to use <code>product</code> to generate input array <code>x</code> and use <code>zip(p0,pk)</code> instead of <code>set(x)</code>?</p>