With reference to the TD learning algorithm proposed by Sutton and Barto which is given by the equations:

$$V_i(t+1) = V_i (t)+ \beta \bigg(\lambda(t+1)+\gamma \bigg[\sum_{j}V_j(t)X_j(t+1)\bigg]-\bigg[\sum_{j}V_j(t)X_j(t) \bigg] \bigg)\alpha\bar{X}_i(t+1),\\ \bar{X}_i(t+1)=\bar{X}_i(t)+\delta\big(X_i(t)-\bar{X}_i(t)\big)$$ I have the following doubts:

  1. If I want to simulate the algorithm in a standalone environment then how do I generate the reward signal $\lambda(t+1)$?
  2. How is $\lambda(t+1)$ related to the conditioning stimulus and the unconditioned stimulus?

For example, if I wanted to simulate the facilitation of a remote association by an intervening stimulus in the TD model as shown in the fig. below then will it suffice if I consider "lambda" to be a signal as represented by US ? simulation

I have been able to design suitable CSA and CSB. However, when I use a $\lambda$ as specified by US in the image, I don't get the result that is shown in the trials. What could possibly go wrong in the formulation of the reward?

The equations can be found in chapter 12 of the book by Sutton & Barto, 1990. The chapter is titled "Time-Derivative Models of Pavlovian Reinforcement".

Sutton, R. S., & Barto, A. G. (1990). Learning and computational neuroscience: foundations of adaptive networks. A/1 IT Press, Cambridge, MA, 497-437.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.