title: Tensorflow load pre-trained model use different optimizer
body: <p>I want to load a pre-trained model (optimized by AdadeltaOptimizer) and continue training with SGD (GradientDescentOptimizer). The models are saved and loaded with <a href="http://tensorlayer.readthedocs.io/en/latest/modules/files.html" rel="nofollow noreferrer">tensorlayer API</a>:</p>
  
  <p>save model:</p>
  
  <pre><code>import tensorlayer as tl
  tl.files.save_npz(network.all_params,
                    name=model_dir + "model-%d.npz" % global_step)
  </code></pre>
  
  <p>load model:</p>
  
  <pre><code>load_params = tl.files.load_npz(path=resume_dir + '/', name=model_name)
  tl.files.assign_params(sess, load_params, network)
  </code></pre>
  
  <p>If I continue training with adadelta, the training loss (cross entropy) looks normal (start at a close value as the loaded model). However, if I change the optimizer to SGD, the training loss would be as large as a newly initialized model. </p>
  
  <p>I took a look at the <code>model-xxx.npz</code> file from <code>tl.files.save_npz</code>. It only saves all model parameters as ndarray. I'm not sure how the optimizer or learning rate is involved here.</p>