What are the key algorithms for learning optimal behavior of economic agents?

Question

I'm playing around with social learning of near-optimal behavioral rules on a set of agents. The idea is roughly that given an income process (or technology process, depending on the question) an optimal nonlinear, intertemporal policy rule exists. Assume this rule can be approximated closely by a linear function. Agents would like to learn this policy rule, and a first pass is to have them learn the rule simply by experimentation. "In autarky," i.e. without any information exchange with other agents, an agent would try a rule for some time, use some metric to determine how well it does against other rules he/she has tried, and perhaps reassess, perhaps try an entirely different rule via experimentation. This agent only observes his/her own history.

A second pass is to allow the agent access to all other agents' histories. Presumably this would speed up learning. A third pass might be to put these agents on an information network of some sort.

I've been perusing literature on social learning, but am not entirely sure the frameworks I am looking at are exactly what I want. Many of them appear to be Bayesian learning about a hidden state of nature, for which everyone has a private signal. I'm actively reviewing literature right now, but as I do, does anyone have any thoughts/suggestions?

Answer 1

Take a look at POMDP's - partially observable markov decision processes.

If you have a value function (income) that is known for agents in various states, and you are trying to identify the optimal policy then Bellman's equation which is at the heart of POMDP's will help you identify this policy.

These tools are part of a class of re-inforcement learning algorithms (in fact, they are used quite often for robotics). So they map very neatly to the framework you have identified (agents, a reward function, and a state/action space)

Another angle of attack would be using genetic algorithms in your optimization procedure.

Answer 2

Here are two computational approaches which might work:

I. Artificial neural network

A neural network consists of an interconnected group of artificial neurons, and it processes information using a connectionist approach to computation. In most cases an ANN is an adaptive system that changes its structure based on external or internal information that flows through the network during the learning phase. Modern neural networks are non-linear statistical data modelling tools. They are usually used to model complex relationships between inputs and outputs or to find patterns in data. Perhaps the greatest advantage of ANNs is their ability to be used as an arbitrary function approximation mechanism which 'learns' from observed data. However, using them is not so straightforward and a relatively good understanding of the underlying theory is essential.

• Choice of model: This will depend on the data representation and the application. Overly complex models tend to lead to problems with learning.
• Learning algorithm: There are numerous trade-offs between learning algorithms. Almost any algorithm will work well with the correct hyper-parameters for training on a particular fixed dataset. However selecting and tuning an algorithm for training on unseen data requires a significant amount of experimentation.
• Robustness: If the model, cost function and learning algorithm are selected appropriately the resulting ANN can be extremely robust.

II. Support vector machine

A set of related supervised learning methods that analyse data and recognize patterns, used for classification and regression analysis. The standard SVM takes a set of input data, and predicts, for each given input, which of two possible classes the input is a member of, which makes the SVM a non-probabilistic binary linear classifier. Since an SVM is a classifier, then given a set of training examples, each marked as belonging to one of two categories, an SVM training algorithm builds a model that predicts whether a new example falls into one category or the other. Intuitively, an SVM model is a representation of the examples as points in space, mapped so that the examples of the separate categories are divided by a clear gap that is as wide as possible. New examples are then mapped into that same space and predicted to belong to a category based on which side of the gap they fall on.

Neural Networks have been used to create highly competitive computer-players for the open-source FreeCiv. There, Neural Networks are used with Monte Carlo Methods which I've also used in simulating economic interactions in games.

Not sure if this is entirely what you're looking for but might be a start?

Answer 3

Many key algorithms are summarized on the ACE Research Area: Learning and the Embodied Mind website.

ACE = Agent-based Computational Economics

Answer 4

This paper by Glazer and Rubinstein, while not strictly relevant to your research, uses a particular algorithmic model of agent behaviour and analyses its effect on the implementability of different mechanisms. The algorithm itself might be of interest to you - I think variants of it could be both realistic and easy to use in applied models.