Is it possible to output a variable clock using Fast PWM?
I'm not sure whether this is the real issue you have, but I spot an issue which can be the issue you have.

You wrote something like this:

if (pulseFromEncoder)
  if (direction1)
    OCR1A -= 100;
  else
    OCR1A += 100;


Now, the OCR1A is a unsigned 16 bit integer. A value of 100 corresponds to 1.6ms of delay on the timer (see comment from Edgar Bonet below). This is what happens when you rotate in direction 2:

        OCR1A  Time delay
Rotate:   300      4.8ms
Rotate:   200      3.2ms
Rotate:   100      1.6ms
Rotate:     0      0ms
Rotate: 65436   1047ms


See? The value rolls over and the time completely changes.

Now, maybe the issues are other (for instance changing the mode to 15 instead of 4 like EdgarBonet said (no idea on what this means, you should double check the differences on the atmega datasheet), but I strongly suggest you to fix this issue, for instance this way:

const uint16_t OCR1A_Min = 0
const uint16_t OCR1A_Max = 20000
const uint16_t OCR1A_Step = 100

...

if (pulseFromEncoder)
  if (direction1)
  {
    if (OCR1A &gt; OCR1A_Min + OCR1A_Step)
      OCR1A -= OCR1A_Step;
    else
      OCR1A = OCR1A_Min;
  }
  else
  {
    if (OCR1A &lt; OCR1A_Max - OCR1A_Step)
      OCR1A += OCR1A_Step;
    else
      OCR1A = OCR1A_Max;
  }


Sidenote: doesn't your encoder need debounce?