Estimating probability from generative model Assume that we have access to a generative model $G$, that takes as input a state-action pair $(s,a)$ and outputs a sample consisting of a successor state-observation pair $(s',o)$, that is, $(s',o)\sim G(s,a)$.

It can be assumed that the states, actions, and observations all take on a finite number of values (can be positive integers).

Is it possible to use the generative model to estimate the probability $P(o|s,s',a)$?



My thoughts: 

Fix $s_0,a_0,s_0'$. 


  
  Generate $(s',o)\sim G(s_0,a_0)$
  If $s'=s_0'$, add a particle to bin $b(o)$
  Return to step 1
  


Continue the above three steps for $M$ runs so that we have $M$ particles in different bins (each bin corresponding to an observation we have seen so far). Now how do we convert the set of particles to probabilities?
Think of this as estimating a probability of some outcome, based on observations of many trials.  Suppose we flip a coin 1000 times, and it comes up heads 327 times.  What is your best estimate for the probability that the next flip will come up heads?  $0.327$, right?  You get that by dividing the number of heads by the number of trials.

What if you rolled a six-sized die 1000 times, and it came up a "one" 327 times?  What is your best estimate for the probability that it comes up "one"?  The same thing -- $0.327$, right?

It's the same here.  Each time that you generate $(s',o)$ and find that $s'=s'_0$, think of that as a trial -- the outcome of the trial is the observed value of $o$.  (Ignore all iterations where $s' \ne s'_0$; they're not even a trial.)  So, if you have a particular bin, say the bin corresponding to $o=1$, then you count the number of trials where you observed that $o$ was $1$, and divide by the total number of trials.  That's your estimate for the desired probability.