PPO agent for vehicle control does not learn to stop at traffic lights
The way I dealt with it was by giving a (very) strong negative reward when committing the mistake (here going under a red light) and the agent should learn to do not do this mistake anymore.
It is often better to change actions, rewards or environment rather than acting on hyperparameters in my opinion, but I may be mistaken.