In the Temporal difference learning algorithm (TD-learning), an agent seeks to predict the total value of future rewards that will be received during the current trial. The agent updates the prediction as the trial progresses, based on conditioned stimuli (CS) that have been seen so far. Specifically, the prediction is calculated as a weighted sum of the intensities of past CS presentation. At time $t$, the prediction is calculated as

$$v(t) = \sum_{z=0}^t w(z)u(t-z)$$

where $v(t)$ is the prediction at time $t$, $u(t)$ is the CS intensity at time $t$, and $w$ is an array of weights.

My question is: why are we summing the terms $w(z)u(t-z)$, rather than $w(z)u(z)$, i.e. why isn't each weight associated with the CS strength at a particular time?

[Thanks to user honi's reply, I subsequently understood that each element of the weight array is associated with an elapsed time since seeing the CS. For example, $w(3)$ is the weight given to the CS strength experienced three units of time ago.]

[Original phrasing of the question: I am reading Theoretical Neuroscience by Dayan and Abbot, and I am confused by the use of $w(t)$ in equations (9.6) and (9.7). In the former, it is a window function (linear filter) and in the latter it is the weight function. Is this (a) an unfortunate use of the same name for two different things, or (b) is the weight function really used as a linear filter to calculate $v(t)$?]

  • 2
    $\begingroup$ Welcome. Do realize not everyone owns the referenced book. Mentioning formula numbers is not really helpful. Instead, a clear explanation of the context, acronyms and symbols is necessary for folks to understand your question. Personally, I'm at a loss at what you are asking. $\endgroup$ – AliceD Feb 21 '18 at 23:03
  • $\begingroup$ Thanks for the advice, AliceD. - I wasn't under the illusion that everyone owned this book, just hoping to connect with one person who did and could help me out - and it worked! :-) $\endgroup$ – Nasorenga Feb 23 '18 at 19:44
  • $\begingroup$ Do realize any post should be I interesting for the community at large. The fact you are helped out doesn't mean it's on topic here. @Honi perhaps you can update the question? $\endgroup$ – AliceD Feb 23 '18 at 19:55

They are the same in the two equations. See equation 9.3: v = w*u. w and u are bolded in that equation to indicate that they are vectors. In equation 9.3, they are vectors of simultaneously presented stimuli, but you can apply the same equation if u is a vector of a single stimulus over time and w are the weights of each time step of that stimulus. Indeed, a linear filter is simply a way to weight different time steps of a time-varying input. Note that v in equation 9.3 is a single value, whereas v in equation 9.6 is a function of t, i.e. it has a different value at each time point depending on what part of u is currently occurring.

| improve this answer | |
  • $\begingroup$ Thanks honi, I get it now! -- At any time t, when calculating the prediction v(t), w(z) is used to weigh the contribution of u(t-z), the value that the stimulus had z ticks earlier. $\endgroup$ – Nasorenga Feb 23 '18 at 19:39
  • $\begingroup$ yep. do you mind accepting my answer if it resolved your problem? $\endgroup$ – honi Feb 23 '18 at 21:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.