How to generate plot of reward and its variance?
To estimate the variance, you probably need to run your algorithm multiple times and keep track of the return for each of these runs. From these multiple returns, you can estimate the variance.

Once you have the standard deviation (or variance) of the return, you can plot something like your plots using matplotlib's fill_between function.