**Problem description:**

Suppose we have an environment, where a reward at time step $t$ is dependent not only on the current action, but also on previous action in the following way:

* if current action == previous action, you get reward = $R(a,s)$
* if current action != previous action, you get reward = $R(a,s) - \text{penalty}$

In this environment, switching actions bears a significant cost. We would like the RL algorithm to learn optimal actions under the constraint that switching action is costly, i.e. we would like to stay in selected action as long as possible.

The penalty is significantly higher than an immediate reward, so if we do not take it into account, the model evaluation will have a negative total reward with almost 100% probability, since the agent will be constantly switching and extracting rewards from environment that are smaller than the cost of switching actions.

Action space is small (2 actions: left, right). I'm trying to beat this game with PPO (Proximal Policy Optimization)

**Questions**

* How one might address this constraint: i.e. explicitly make the agent learn that switching is costly and it's worth sitting in one action even if immediate rewards are negative?
* How can you make the RL algorithm learn that it's not the reward term $R(a\_t|s\_t)$ that is negative, and thus decreasing $Q(a\_t|s\_t)$ and $V(s\_t)$, but it's the penalty term (taking the action that is different from the previous action at step $t-1$) that is pushing total reward down?