Policy Adjustment in Markov Decision Process
The set of optimal policies for one MDP may be completely different than for another MDP even if the two MPDs differ only in the rewards and/or transition probabilities.

One thing you might try in order to save resources in some cases when you change the MDP (e.g. when the set of optimal policies does not change too much) is to use policy iteration to find optimal policies and to start it for the new MDP from an optimal policy for the old MDP. Theoretically, you should bear in mind that there are exponential time examples for policy iteration that start and finish at policies that differ in only one action, but these are pathological examples that will almost certainly not arise in practice.

You may find some relevant information if you search for "sensitivity analysis" or "perturbation analysis" for MDPs.