Reinforcement Learning with Adaptive Action Magnitude

How to select an action in a state if the action does not necesarily cause the environment to change state?

Given 10 states ($$S_0$$ to $$S_9$$) and in each state $$i$$ there are two actions defined $$(1,-1)$$. $$1$$ increases a parameter of the environment and $$-1$$ decreases it.

For example, if the parameter is speed and it is currently 1 rad/s, corresponding to $${S_i}$$, an action can be either increase or decrease this and therefore transition to the next state, ideally $$S_{i-1}$$ or $$S_{i+1}$$.

It is unclear how to formulate a reinforcement learning problem in this case. The problem is the same magnitude of the parameter change through an action.

The range of the parameter is (3, 25) and step size is 1. The problem is thath the responce of the environment is not the same in every satate. In some states a parameter change with magnitude 1 results in a state transition in some state the magintude proves to be to small to provoke a transition change.

For example, if the envitonment is instate $$S_1$$ and action 1 is applied, how can the magintude of the paremter be adopted in a way which assures that the environment will transition to state $$S_2$$? How can the step in the paremeter change be made adaptive? Actually my environment is uncertain that is why I don't know exactly whether this action will take me to next state or not.

For your information, I am using Q learning off policy algorithm. Suppose state 9 is the goal state and my Q table is $$10 \times 2$$.

There are more possible approaches to tackle this problem:

1. Use a reinforcement learning method which can cope with a continous state space. This would eliminate the need for discretizing the state space which in turn, if I understood correctly, leads to problems in transitioning between states. You can also consider selecting a reinforcement learning method that can cope with a continous state space and a continous action space.

2. Actions do not have to be immediate. If your action 1 is transitioning form $$S_i$$ to $$S_{i+1}$$ you can have a function wich continously increases the spead until the next state is reached and only then you consider the execution of the action complete. Furthermore, the magnitude of the actions can be dependent on the current state, if it assures state transition.

3. You can add more actions. Use RL for your advantage and define not just $$+1$$ and $$-1$$ but $$+0.1$$, $$0.3$$, $$+0.5$$, $$+0.7$$, $$+1$$ (and same for deacreasing the speed) etc. Add a negative reward for all actions which do not cause a state transition or which cause higher jump then needed. However, care must be taken to make sure you have the same speed fro each sampling. Eg. if you are in $$S_1$$ and the action +0.1 does not cause a state transition to $$S_2$$ you have to reset the state (if possible) or fail the epoch, since the state you are in is not only dependent on the action and the previous state. In other words the speed of e.g. $$1 rad/s$$ (with a tolarance band) defined as $$S_1$$ will be slightly higher and ending up in $$S_2$$ after applying a slight increase in the speed takes less change in the speed than from $$S_1$$ without the "unnoticed" speed change.

• Ok thank you very much for a very nice explanation. Actually in my problem, I have finite states available (say 10). There are total 20 actions (say discrete value from 3 to 23). I know state 1 is bad and state 10 is good. In each state, i can only try few actions (not all 20). The problem actually lies here. When I do simulation, I know if I increase action by some number relative to previous action I will transition to next state. But In real implementation, it does not. Can I do like binary decisions (1 means keep increasing action and -1 keep decreasing until I transition to next state. – adnan jaffar Dec 14 '18 at 21:23
• So you are training in a virtual environment (simulation) which does not reflect your real setup used in the expoitation? – 50k4 Dec 14 '18 at 21:26
• Yes, I still did not implement it in real but i am worried before implementation. My real environment is patient and so I can only try when I am confident about my algorithm. The virtual environment is somehow relating with real but not 100 percent. – adnan jaffar Dec 14 '18 at 21:29
• Can I make Q table like states * decision actions and decision actions are +1,-1. 1 means increase the previous action by some number and -1 decrease it by some number until i reach to next state. and I will update Q table only when i transition to next state. – adnan jaffar Dec 14 '18 at 21:36
• As stated in point 2, the action does not have to be immediate and if it brings your environment to the same state in reality as in simulation it does not have to be the same implementation or it does not have to act in the same way. – 50k4 Dec 14 '18 at 21:43
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 0, "mathjax_display_tex": 0, "mathjax_asciimath": 0, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0, "katex": 0, "math-container": 27, "wp-katex-eq": 0, "align": 0, "equation": 0, "x-ck12": 0, "texerror": 0, "math_score": 0.9191002249717712, "perplexity": 419.2617456793214}, "config": {"markdown_headings": false, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "plain"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "warc_path": "s3://commoncrawl/crawl-data/CC-MAIN-2021-21/segments/1620243989018.90/warc/CC-MAIN-20210509213453-20210510003453-00532.warc.gz"}