I've been working on modeling some phenomena involving real-time control in an environment with inherent rewards (specifically, playing a 'pong'-like game), and it's increasingly looking like reinforcement learning by itself won't cut it computationally (I'm currently using a temporal difference back propagation neural network).

One possible supplemental learning mechanism is to also have the model predict the environment's future state, from which it can learn from in a supervised manner using standard feed forward back propagation.

My current thinking on synthesizing these mechanisms is to have the input layer feed into a hidden layer, which in turn feeds into both a reward predicting layer and a separate state predicting layer. In training this net, I simply change the weights via reinforcement learning first and then change them again to account for the state prediction error via back prop.

So my question is this: Are there any problems you can foresee arising from this architecture? Additionally, has this combination of learning mechanisms been done before, and if so was it done in a similar way?


I'm not sure I fully understand your design; perhaps you can clarify what you want your network to learn, why TD-learning "isn't cutting it", and what you mean by 'reinforcement' and 'prediction' learning. In particular, TD-learning is a reinforcement learning model, and it does reward based on predicted (and not just observed) outcomes. However, you seem to be describing reinforcement and prediction learning as orthogonal models, so again, I'm not sure I understand correctly.

As a general suggestion, you might consider using an Elman/Jordan network (e.g. recurrent neural network/RNN). Rather than relying on knowledge of only the current state to make a prediction about the next state, RNN can learn to recognize sequences of events. This is especially useful for predicting future states in a task that unfolds over time (e.g. [1]). I suggest this mostly because you say your task is a 'real-time control' task, but without more knowledge of your task I don't really know if this is appropriate.

As for your suggestion of using two different learning mechanisms to modify a single set of weights-- I don't have the answer, but it seems counterintuitive to me. You're using two different optimization techniques on a single set of data. If the techniques disagree, your network will probably never learn its connection weights. If they agree (i.e., converge on the same answer) then I'm not certain you're adding any value by having two learning mechanisms.

[1] Elman (1990). Finding structure in time. Cognitive Science, 14, 179-211. Retrieved from http://synapse.cs.byu.edu/~dan/678/papers/Recurrent/Elman.pdf

  • $\begingroup$ Point taken; I've gone over this problem enough in my head that I was unaware how many terms I left unspecified. I'll clarify my question shortly and add a diagram of the intended network architecture. On a side note, this seems to serve more as a comment than an answer. $\endgroup$ – zergylord Jan 26 '12 at 3:49
  • $\begingroup$ half comment, half answer-- yes, i can forsee problems arising from that architecture. but moreover, it just didn't fit in the comment box. $\endgroup$ – Jeff Jan 26 '12 at 4:04
  • $\begingroup$ I made a few corrections in order to clear up ambiguities. To be clear, both proposed learning mechanisms are predicting something, but predicting the next state (a matrix of pixels in this case) doesn't require TD since the correct answer is immediately visible in the next time step. $\endgroup$ – zergylord Jan 26 '12 at 4:38
  • $\begingroup$ Are you currently using a RNN or feedforward? A RNN should learn to associate trajectories of the ball with a specific state of the paddle. Sorry if this doesn't help... $\endgroup$ – Jeff Jan 26 '12 at 14:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.