Pytorch What's the difference between define layer in __init__() and directly use in forward()?
What is the difference between the method that define layers in __init__() function, call layer in forward later and the method that directly use layer in forward() function ?
Should I define every layer in my compute graph in constructed function(eg. __init__) before I write my compute graph?
Could I direct define and use them in forward()? 

Everything which contains weights which you want to be trained during the training process should be defined in your __init__ method.

You don't need do define activation functions like softmax, ReLU or sigmoid in your __init__, you can just call them in forward. 

Dropout layers for example also don't need to be defined in __init__, they can just be called in forward too. [However defining them in your __init__ has the advantage that they can be switched off easier during evaluation (by calling eval() on your model). You can see an example of both versions here.

Hope this is clear. Just ask if you have any further questions.
