I'm playing around with social learning of near-optimal behavioral rules on a set of agents. The idea is roughly that given an income process (or technology process, depending on the question) an optimal nonlinear, intertemporal policy rule exists. Assume this rule can be approximated closely by a linear function. Agents would like to learn this policy rule, and a first pass is to have them learn the rule simply by experimentation. "In autarky," i.e. without any information exchange with other agents, an agent would try a rule for some time, use some metric to determine how well it does against other rules he/she has tried, and perhaps reassess, perhaps try an entirely different rule via experimentation. This agent only observes his/her own history.

A second pass is to allow the agent access to all other agents' histories. Presumably this would speed up learning. A third pass might be to put these agents on an information network of some sort.

I've been perusing literature on social learning, but am not entirely sure the frameworks I am looking at are exactly what I want. Many of them appear to be Bayesian learning about a hidden state of nature, for which everyone has a private signal. I'm actively reviewing literature right now, but as I do, does anyone have any thoughts/suggestions?

  • $\begingroup$ You might look into the AI literature. $\endgroup$ – Jason B Oct 31 '11 at 20:23
  • $\begingroup$ Agree on the AI. Bayesian networks and genetic algorithms come into this as well. $\endgroup$ – Turukawa Oct 31 '11 at 21:54
  • $\begingroup$ Any particular references? $\endgroup$ – Nate Nov 1 '11 at 3:48

Take a look at POMDP's - partially observable markov decision processes.

If you have a value function (income) that is known for agents in various states, and you are trying to identify the optimal policy then Bellman's equation which is at the heart of POMDP's will help you identify this policy.

These tools are part of a class of re-inforcement learning algorithms (in fact, they are used quite often for robotics). So they map very neatly to the framework you have identified (agents, a reward function, and a state/action space)

Another angle of attack would be using genetic algorithms in your optimization procedure.

  • 1
    $\begingroup$ +1 reinforcement learning algorithms are really something that the OP should consider. Keywords to find references: Q-learning, kearns algorithm, planning problem, approximation for markov decision processes. For instance you can look at G. Tesauro papers for a start. $\endgroup$ – Sylvain Peyronnet Nov 22 '11 at 23:59
  • $\begingroup$ Thanks, I will be looking further into this. After perusing the literature for a while (there are a lot of nice things in the Handbook of Computational Economics, Vol. 2, that I missed the first time I read through it a long time ago), I think I've settled on an approach. Thanks everybody! $\endgroup$ – Nate Nov 24 '11 at 4:59

Here are two computational approaches which might work:

I. Artificial neural network

A neural network consists of an interconnected group of artificial neurons, and it processes information using a connectionist approach to computation. In most cases an ANN is an adaptive system that changes its structure based on external or internal information that flows through the network during the learning phase. Modern neural networks are non-linear statistical data modelling tools. They are usually used to model complex relationships between inputs and outputs or to find patterns in data. Perhaps the greatest advantage of ANNs is their ability to be used as an arbitrary function approximation mechanism which 'learns' from observed data. However, using them is not so straightforward and a relatively good understanding of the underlying theory is essential.

  • Choice of model: This will depend on the data representation and the application. Overly complex models tend to lead to problems with learning.
  • Learning algorithm: There are numerous trade-offs between learning algorithms. Almost any algorithm will work well with the correct hyper-parameters for training on a particular fixed dataset. However selecting and tuning an algorithm for training on unseen data requires a significant amount of experimentation.
  • Robustness: If the model, cost function and learning algorithm are selected appropriately the resulting ANN can be extremely robust.

II. Support vector machine

A set of related supervised learning methods that analyse data and recognize patterns, used for classification and regression analysis. The standard SVM takes a set of input data, and predicts, for each given input, which of two possible classes the input is a member of, which makes the SVM a non-probabilistic binary linear classifier. Since an SVM is a classifier, then given a set of training examples, each marked as belonging to one of two categories, an SVM training algorithm builds a model that predicts whether a new example falls into one category or the other. Intuitively, an SVM model is a representation of the examples as points in space, mapped so that the examples of the separate categories are divided by a clear gap that is as wide as possible. New examples are then mapped into that same space and predicted to belong to a category based on which side of the gap they fall on.

Neural Networks have been used to create highly competitive computer-players for the open-source FreeCiv. There, Neural Networks are used with Monte Carlo Methods which I've also used in simulating economic interactions in games.

Not sure if this is entirely what you're looking for but might be a start?

  • 1
    $\begingroup$ Thanks for your reply. This isn't exactly what I'm looking for -- at this stage, I'm more looking for examples of simpler learning mechanisms, ideally in some published economics papers. I appreciate the pointers, however; thanks! $\endgroup$ – Nate Nov 2 '11 at 16:51
  • $\begingroup$ @Nathan - do you mean actual working algorithms with papers showing proof, or just general economics using these types of models? $\endgroup$ – Turukawa Nov 2 '11 at 17:49
  • $\begingroup$ at the end of the day I'm looking for papers that used mechanisms I could use; ideally papers I could cite in a literature review. $\endgroup$ – Nate Nov 15 '11 at 15:56
  • $\begingroup$ Just wanted to say again -- thanks for the above pointers! The article on FreeCiv is particularly interesting -- especially because I had a tough time beating the most recent version of it :) Thanks again for the time and effort you put into your response! I think I'll be using these down the road a bit. $\endgroup$ – Nate Nov 24 '11 at 5:01

Many key algorithms are summarized on the ACE Research Area: Learning and the Embodied Mind website.

ACE = Agent-based Computational Economics

  • $\begingroup$ Leigh Tesfatsion's website is one everyone should peruse from time to time -- absolutely fantastic resource. I spent a bit of time there myself as I was looking into this. $\endgroup$ – Nate Nov 24 '11 at 5:03
  • 1
    $\begingroup$ Yes, it is a great website for behavioral economics and understanding agent based motivation. @Sylvain Peyronnet there is a great deal of material on this website, are there particular entries that you could mention in your answer, and why? $\endgroup$ – Feral Oink Nov 25 '11 at 4:33

This paper by Glazer and Rubinstein, while not strictly relevant to your research, uses a particular algorithmic model of agent behaviour and analyses its effect on the implementability of different mechanisms. The algorithm itself might be of interest to you - I think variants of it could be both realistic and easy to use in applied models.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy