Is there a good and easy paper to code policy gradient algorithms (REINFORCE) from scratch?
this is maybe not a paper but this article is dope. 

I got 2 link in below to help you understand about policy gradient algorithms especially Reinforce Learning. Both article have a good explanation about that algorithms and not only an explanation, also contain a good example about that. They really put their effort into it so i'm sure if you've read it and really understand it, you can code from scratch. Hope this is help you. 

If you think this answer is helping you, i will grateful if you make my answer as accepted. Just kidding, have a nice day.... 

(no i'm not kidding)

https://towardsdatascience.com/policy-gradients-in-a-nutshell-8b72f9743c5d by Sanyam Kapoor on Towardsdatascience.com

https://medium.com/@jonathan_hui/rl-policy-gradients-explained-9b13b688b146 by Jonathan Hui on Medium.com