# In TD learning, is the weight function used as a linear filter?

In the Temporal difference learning algorithm (TD-learning), an agent seeks to predict the total value of future rewards that will be received during the current trial. The agent updates the prediction as the trial progresses, based on conditioned stimuli (CS) that have been seen so far. Specifically, the prediction is calculated as a weighted sum of the intensities of past CS presentation. At time $t$, the prediction is calculated as

$$v(t) = \sum_{z=0}^t w(z)u(t-z)$$

where $v(t)$ is the prediction at time $t$, $u(t)$ is the CS intensity at time $t$, and $w$ is an array of weights.

My question is: why are we summing the terms $w(z)u(t-z)$, rather than $w(z)u(z)$, i.e. why isn't each weight associated with the CS strength at a particular time?

[Thanks to user honi's reply, I subsequently understood that each element of the weight array is associated with an elapsed time since seeing the CS. For example, $w(3)$ is the weight given to the CS strength experienced three units of time ago.]

[Original phrasing of the question: I am reading Theoretical Neuroscience by Dayan and Abbot, and I am confused by the use of $w(t)$ in equations (9.6) and (9.7). In the former, it is a window function (linear filter) and in the latter it is the weight function. Is this (a) an unfortunate use of the same name for two different things, or (b) is the weight function really used as a linear filter to calculate $v(t)$?]

• Welcome. Do realize not everyone owns the referenced book. Mentioning formula numbers is not really helpful. Instead, a clear explanation of the context, acronyms and symbols is necessary for folks to understand your question. Personally, I'm at a loss at what you are asking. – AliceD Feb 21 '18 at 23:03
• Thanks for the advice, AliceD. - I wasn't under the illusion that everyone owned this book, just hoping to connect with one person who did and could help me out - and it worked! :-) – Nasorenga Feb 23 '18 at 19:44
• Do realize any post should be I interesting for the community at large. The fact you are helped out doesn't mean it's on topic here. @Honi perhaps you can update the question? – AliceD Feb 23 '18 at 19:55