'Returns the entropy over labellings of the given sequence. This is
given by::
H(O) = - sum_S Pr(S | O) log Pr(S | O)
where the summation ranges over all state sequences, S. Let
*Z = Pr(O) = sum_S Pr(S, O)}* where the summation ranges over all state
sequences and O is the observation sequence. As such the entropy can
be re-expressed as::
H = - sum_S Pr(S | O) log [ Pr(S, O) / Z ]
= log Z - sum_S Pr(S | O) log Pr(S, 0)
= log Z - sum_S Pr(S | O) [ log Pr(S_0) + sum_t Pr(S_t | S_{t-1}) + sum_t Pr(O_t | S_t) ]
The order of summation for the log terms can be flipped, allowing
dynamic programming to be used to calculate the entropy. Specifically,
we use the forward and backward probabilities (alpha, beta) giving::
H = log Z - sum_s0 alpha_0(s0) beta_0(s0) / Z * log Pr(s0)
+ sum_t,si,sj alpha_t(si) Pr(sj | si) Pr(O_t+1 | sj) beta_t(sj) / Z * log Pr(sj | si)
+ sum_t,st alpha_t(st) beta_t(st) / Z * log Pr(O_t | st)
This simply uses alpha and beta to find the probabilities of partial
sequences, constrained to include the given state(s) at some point in
time.'

def entropy(self, unlabeled_sequence):

    unlabeled_sequence = self._transform(unlabeled_sequence)
    T = len(unlabeled_sequence)
    N = len(self._states)
    alpha = self._forward_probability(unlabeled_sequence)
    beta = self._backward_probability(unlabeled_sequence)
    normalisation = logsumexp2(alpha[(T - 1)])
    entropy = normalisation
    for (i, state) in enumerate(self._states):
        p = (2 ** ((alpha[(0, i)] + beta[(0, i)]) - normalisation))
        entropy -= (p * self._priors.logprob(state))
    for t0 in range((T - 1)):
        t1 = (t0 + 1)
        for (i0, s0) in enumerate(self._states):
            for (i1, s1) in enumerate(self._states):
                p = (2 ** ((((alpha[(t0, i0)] + self._transitions[s0].logprob(s1)) + self._outputs[s1].logprob(unlabeled_sequence[t1][_TEXT])) + beta[(t1, i1)]) - normalisation))
                entropy -= (p * self._transitions[s0].logprob(s1))
    for t in range(T):
        for (i, state) in enumerate(self._states):
            p = (2 ** ((alpha[(t, i)] + beta[(t, i)]) - normalisation))
            entropy -= (p * self._outputs[state].logprob(unlabeled_sequence[t][_TEXT]))
    return entropy
