title: Changing optimizer or lr after loading model yields strange results
body: <p>I'm using the latest Keras with Tensorflow backend (Python 3.6)</p>
  
  <p>I'm loading a model that had a training accuracy at around 86% when I last trained it.</p>
  
  <p>The orginal optimizer that I used was :</p>
  
  <pre><code>r_optimizer = optimizer=Adam(lr=0.0001, decay = .02)
  model.compile(optimizer= r_optimizer,
                loss='categorical_crossentropy', metrics = ['accuracy'])
  </code></pre>
  
  <p>If I load the model and continue training without recompiling, my
  accuracy would stay around 86% (even after 10 or so more epochs).
  So I wanted to try changing the learning rate or optimizer.</p>
  
  <p>If I recompile the model and try to change the learning rate or the
  optimizer as  follows:</p>
  
  <pre><code>new_optimizer = optimizer=Adam(lr=0.001, decay = .02)
  </code></pre>
  
  <p>or to this one:</p>
  
  <pre><code>sgd = optimizers.SGD(lr= .0001)
  </code></pre>
  
  <p>and then compile:</p>
  
  <pre><code>model.compile(optimizer= new_optimizer ,
                loss='categorical_crossentropy', metrics = ['accuracy'])
  
  model.fit ....
  </code></pre>
  
  <p>The accuracy would reset to around 15% - 20%, instead of starting around 86%,
  and my loss would be much higher. 
  Even if I used a small learning rate, and recompiled, I would still start
  off from a very low accuracy.
  From browsing the internet it seems some optimizers like ADAM or RMSPROP have
  a problem with resetting weights after recompiling (can't find the link at the moment)</p>
  
  <p>So I did some digging and tried to reset my optimizer without recompiling as follows:</p>
  
  <pre><code>    model = load_model(load_path)
      sgd = optimizers.SGD(lr=1.0) # very high for testing
      model.optimizer = sgd   #change optimizer
  
      #fit for training
  
       history =model.fit_generator(
       train_gen,
      steps_per_epoch = r_steps_per_epoch,
      epochs = r_epochs,
      validation_data=valid_gen,
      validation_steps= np.ceil(len(valid_gen.filenames)/r_batch_size),
      callbacks = callbacks,
      shuffle= True,
      verbose = 1)
  </code></pre>
  
  <p>However, these changes don't seem to be reflected in my training.
  Despite raising the <code>lr</code> significantly, I'm still floundering around 86% with the same loss. During each epoch, I'm seeing very little loss or accuracy movement. I would expect the loss to be a lot more volatile. 
  This leads me to believe that my change in optimizer and lr isn't being
  realized by the model. </p>
  
  <p>Any idea what I could be doing wrong?</p>