Compare Hidden Markov Model's sample with ground truth data I have a time-serie and I fit different HMMs on it, each with a different number of hidden states.
Now after sampling from the models , I'd like to compare the results with the ground truth data and find the model that gets closer to the real world data in the original time-serie.
For now I simply compared visually the distribution of the values generated by the HMMs and the distribution of values in the time-serie, but I'd like to compute a number indicating which model generates better samples.
Compute the likelihood of the observed data, for each model.  Then higher the likelihood, the better the fit.  The likelihood is just the probability that the model assigns to the observed data, which for HMMs can be computed using dynamic programming.
Be prepared that the more complex the model, the higher the likelihood will be, but that doesn't necessarily mean the model is &quot;better&quot; -- you run the risk of overfitting.  Larger HMMs may fit the training data better, but if the HMM is too large, it might perform poorly on new data because of overfitting.