# Questions tagged [temporal-difference-methods]

For questions related to the temporal-difference reinforcement learning (RL) algorithms, which is a class of model-free (that is, they do not use the transition and reward function of the MDP) RL algorithms which learn by bootstrapping from the current estimate of the value function (that is, they use one estimate to update another estimate).

62 questions
Filter by
Sorted by
Tagged with
22 views

### Derivation of weights for linear function approximation of Temporal difference method

https://web.stanford.edu/class/cs234/slides/lecture5.pdf I am reading the above material. On page 38, the update for the parameters for the linear function approximation of TD(0) is given. I have a ...
52 views

71 views

### How should I implement the state transition when it is a Gaussian distribution?

I am reading this paper Anxiety, Avoidance and Sequential Evaluation and is confused about the implementation of a specific lab study. Namely, the authors model what is called the Balloon task using a ...
127 views

### When calculating the cost in deep Q-learning, do we use both the input and target states?

I just finished Andrew Ngs's deep learning specialization, but RL was not covered, so I don't know the basics of RL. So, I have been having trouble understanding the cost function in deep Q-learning. ...
117 views

### Is the expected value we sample in TD-learning action-value Q or state-value V?

Both MC and TD are model-free and they both follow a sample trajectory (in the case of TD, the trajectory is cut-short) to estimate the return (we basically are sampling Q values). Other than that, ...
422 views

### How to determine if Q-learning has converged in practice？

I am using Q-learning and SARSA to solve a problem. The agent learns to go from the start to the goal without falling in the holes. At each state, I can choose the action corresponding to the maximum ...
203 views

86 views

### Why does TD Learning require Markovian domains?

One of my friends and I were discussing the differences between Dynamic Programming, Monte-Carlo, and Temporal Difference (TD) Learning as policy evaluation methods - and we agreed on the fact that ...
41 views

### If the transition model is available, why would we use sample-based algorithms?

Sample-based algorithms, like Monte Carlo Algorithms and TD-Learning, are often presented as useful since they do not require a transition model. Assuming I do have access to a transition model, are ...
93 views

### Into which subcategories can reinforcement learning be divided?

In the course of a scientific work, I will discuss the different types of reinforcement learning. However, I have difficulties to find these different types. So, into which subcategories can ...
33 views

### Why do bootstrapping methods produce nonstationary targets more than non-bootstrapping methods?

The following quote is taken from the beginning of the chapter on "Approximate Solution Methods" (p. 198) in "Reinforcement Learning" by Sutton & Barto (2018): reinforcement ...
61 views

### Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value?

Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value? Why can't we forget the learning rate and temporal difference? Here's the update formula.
439 views

### What is the bias-variance trade-off in reinforcement learning?

I am watching DeepMind's video lecture series on reinforcement learning, and when I was watching the video of model-free RL, the instructor said the Monte Carlo methods have less bias than temporal-...
56 views

### Why does the n-step return being zero result in high variance in off policy n-step TD?

In the paragraph given between eq 7.12 and 7.13 in Sutton & Barto's book: $G_{t:h} = R_{t+1} + G_{t+1:h} , t < h < T$ where $G_{h:h} = V_{h-1}(S_h)$. (Recall that this return is used at ...
94 views

48 views

### What is correct update when the some indexes are not available?

To update the Q table Q-learning takes the arg max of the Q values - the state, value mappings. For example, in tic tac toe the state XOX OXO -X- contains two ...
591 views

### What are the conditions of convergence of temporal-difference learning?

In reinforcement learning, temporal difference seem to update the value function in each new iteration of experience absorbed from the environment. What would be the conditions for temporal-...
160 views

### In what RL algorithm category is MiniMax?

Q-learning is a temporal-difference method and Monte Carlo tree search is a Monte Carlo method. In what category is MiniMax?
{"ft_lang_label":"__label__en","ft_lang_prob":0.8971885,"math_prob":0.7990847,"size":6237,"snap":"2021-43-2021-49","text_gpt3_token_len":1498,"char_repetition_ratio":0.123696454,"word_repetition_ratio":0.0020470829,"special_character_ratio":0.25012025,"punctuation_ratio":0.14308552,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.97187287,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2021-12-06T03:32:00Z\",\"WARC-Record-ID\":\"<urn:uuid:a0c6a87a-e178-42c4-8154-940107793dec>\",\"Content-Length\":\"277186\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:389b977d-685b-4294-88a2-fdfe9c6587ac>\",\"WARC-Concurrent-To\":\"<urn:uuid:47dd4b5a-e0a0-4ea8-8c33-9b0c20e0e2e7>\",\"WARC-IP-Address\":\"151.101.129.69\",\"WARC-Target-URI\":\"https://ai.stackexchange.com/questions/tagged/temporal-difference-methods\",\"WARC-Payload-Digest\":\"sha1:3Q4GCOJEUV3XUDWBY4ECTGJQVX7P37C4\",\"WARC-Block-Digest\":\"sha1:HBJAM2RJOBGLJHJ4YU5RD2IIQOPNR6LH\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2021/CC-MAIN-2021-49/CC-MAIN-2021-49_segments_1637964363229.84_warc_CC-MAIN-20211206012231-20211206042231-00115.warc.gz\"}"}