reinforcement learning in gridworld with subgoals Andrew Ng, Daishi Harada, Stuart Russell published a conference paper entitled Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping.

There is a specific example there that I am extremely curious/interested about. It is in Figure 2(a) of the paper:



It is about a 5x5 gridworld with start and goal states in opposite corners. The catch is, the agent must learn to go from start to end BY VISITING specific subgoals 1,2,3,4 IN ORDER. 

Has anyone seen/understood the code for this? I want to know how the reward function/shaping is given in this kind of problem. 

I am interested to know how the flow of this modification to the grid world is written.
The subgoals are encoded in external rules to determine if a reward is given and in what amount. The creation of the rules and the used features is done manually by experts. There is no algorithm which calculates the reward function by itself.