Update in REINFORCE algorithm - step-wise or episode-wise?
Hi Seewoo Lee and welcome to our community!

The essence of your observation is that Sutton's version of REINFORCE is taking into consideration all of the trajectory to compute the returns while in the pytorch version only the future is taken into consideration, hence going in reverse to sum the future rewards and ignore the previous rewards. The consequence is that future actions are not punished for early mistakes. The guys at OpenAI refer to this as reward-to-go but personally, I find that it resembles Monte Carlo On Policy Control without Exploring Starts or First Visit from Sutton's book.

You can find more on REINFORCE and Policy Gradient in Spinning Up RL: Part3: Intro to policy gradient -  Don't let the past distract you.

Also, something to note is that even in Sutton's version, the whole trajectory is unrolled, i.e. the episode completes, and then the weights get updated. Otherwise it stops being  a Monte Carlo Method and it becomes a TD method. In addition, you can't make a change on a single point because sampling is a non differentiable operation, instead, the gradient is estimated by collecting a lot of trajectories.