Thus, the 'decision maker' started with random sampling to learn from trial-and-error. From iteration 1, the possibility to use the updated policy model 'decision maker' gradually increased, and from iteration 10, the reaction pathway was optimised only from the policy model. Suggested by Schreck et al., not only does the expected cost of a molecule depend on the molecule itself, but also the depth of the molecule in the pathway. If the molecule requires a long synthesis pathway, whether the pathway of a molecule reaches maximum-depth molecules or building-block molecules relies on its residual depth, i.e. the maximum allowed depth (10 steps) subtracted by the current depth from the target molecule. To learn from the simulated experience, the residual depth and the corresponding expected costs of all molecules in the pathway were collected following the designed penalty rules. This did not include the side-