independently of xt - thusif we define and let be the -th time that the chain returns to state then we can break up the time interval into independent cycles of lengths ci - now suppose that during the -th cycle reward ri - % -ti- (xt = - is receivedwhere (iis some fixed reward for visiting state { and ( is discounting factor clearly{(ci ri )is renewal reward process figure shows the outcomes of pairs (cr)using ( ( ( ( and reward cycle length figure each circle represents (cycle lengthrewardpair the varying circle sizes indicate the number of occurrences for given pair for example( , is the most likely pair hereoccurring out of times it corresponds to the cycle path the long-run average reward is estimated as for our data but how accurate is this estimatefigure shows density plot of the bootstrapped ratio estimateswhere we independently resampled the data pairs times density long-run average reward figure density plot of the bootstrapped ratio estimates for the markov chain renewal reward process