pytorch optim.SGD with momentum how to check "velocity"?
I have a noob question: from the SGD doc they provided the equation of SGD with momentum, which indicates that apart from current gradient weight.grad, we also need to save the velocity from the previous step (something like weight.prev_v?). I know nn.Parameter object has .data and .grad attributes, but does it also saves a .prev_v? Do you know how pytorch works? Thanks!
Edit: Basically I'd like to know where does pytorch save velocity from the previous step?

Those are stored inside the state attribute of the optimizer. In the case of torch.optim.SGD the momentum values are stored a dictionary under the 'momentum_buffer' key, as you can see in the source code.
Here is a minimal example:
&gt;&gt;&gt; m = nn.Linear(10,10)
&gt;&gt;&gt; optim = torch.optim.SGD(m.parameters(), lr=1.e-3, momentum=.9)
&gt;&gt;&gt; m(torch.rand(1, 10)).mean().backward()
&gt;&gt;&gt; optim.step()

&gt;&gt;&gt; optim.state
defaultdict(dict, {0: {}, Parameter containing: ...})

&gt;&gt;&gt; list(optim.state.values())[0]
{'momentum_buffer': tensor([...])}

