How to choose the number of output channels in a convolutional layer?
I recommend you reading the  guide to convolution arithmetic for deep learning . There you can find very well written explanations about calculating the about size of your layers depending on kernel size, stride, dilatation, etc. 

Further you can easily get your intermediate shapes in pytorch by adding a simple print(x.shape) statement in your forward pass and adapting your number of neurons in your fully connected layers. 

Last but not least. When you cange your input size from 32x32 to 64x64 your output of your final convolutional layer will also have approximately doubled size (depends on kernel size and padding) in each dimension (height, width) and hence you quadruple (double x double) the number of neurons needed in your linear layer.