What considerations should be taken into account with directly setting Pytorch model variables and optimizer parameters?
From the Pytorch forums 
https://discuss.pytorch.org/t/layer-weight-vs-weight-data/24271/2
it is mentioned that setting variable weights directly could result in 

"Using .data on the other side would work, but is generally not recommended, as changing it after the model was used would yield weird results and autograd cannot throw an error."

I am wonder what would cause weird results. Also, I am also thinking of setting optimizer parameters directly, specifically the momentum/sum of gradients for optimizers which have those parameters. Is there any considerations needed for that case as well? 

It is perfectly legal to update the PyTorch layer weights.
Check this out how we can alter the weights no problem:

lin = nn.Linear(10, 2)
torch.nn.init.xavier_uniform_(lin.weight)


The upper code actually calls with torch.no_grad():

def _no_grad_uniform_(tensor, a, b):
    with torch.no_grad():
        return tensor.uniform_(a, b)


See how the torch.no_grad() will help us in the next example.

lin = nn.Linear(10, 2)
with torch.no_grad():
    lin.weight[0][0] = 1.

x = torch.randn(1, 10)
output = lin(x)
output.mean().backward()


And if we don't use it:

lin = nn.Linear(10, 2)
lin.weight[0][0] = 1.
x = torch.randn(1, 10)
output = lin(x)
output.mean().backward()


We end in: 


  RuntimeError: leaf variable has been moved into the graph interior


So you can do it but inside with torch.no_grad():.
This is because every action we do on PyTorch tensor will get caught, if requires grad is set to True.

In case we do lin.weight[0][0] = 1. we will catch grad_fn=&lt;CopySlices&gt;.
The thing is we don't need that to be caught as this is part of our layer setup and not our calculation.
