I'm reading Reinforcement learning and causal models by Sam Gershman, which states that TD-learning provides an account of second-order conditioning that provides an explanation for the phenomenon occurred in trace learning (pg. 7, 3rd paragraph). The idea is as follows:

In trace conditioning, a reward is presented after some cue A with some relatively large time delay. If a second cue, B, is presented during this gap, this facilitates the learning of a conditional response to A. He says that this is because B receives a large positive value which generates a large positive prediction error at the offset of A. I do not see why this is true. If you look at TD-learning, the values of A and B should strictly be a function of their temporal distance from the reward, i.e. $$\gamma^{T-t} r_T$$, where $t$ is the current time and $T$ is the time of the reward. He goes on to say that this is related to how the prediction error is changed with the presence of B. Could somebody explain to me why having B increases the prediction error?

  • 1
    $\begingroup$ Hi, welcome at CogSci and very great question! Looking forward to an answer :) $\endgroup$ May 21 '17 at 7:08
  • $\begingroup$ I'm not sure there is a definitive answer to this, since in my experience the way the model explains phenomena such as second-order conditioning is somewhat up to the modeller. It's then an empirical question as to whether that explanation is correct. $\endgroup$ Mar 31 '18 at 1:01

See my comment above, but it is possible to assume that B acquires the same reward value as the original reward, so B can then be treated as a reward, which in turn brings A closer to the reward event (now at time B). Does that make sense? In other words, the addition of B allows the reward value to accrue to B and thus brings A closer to reward.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.