I'm a mathematics and physics student very much interested in cognitive science. Recently I've been hearing about "a new approach" in cognitive science via dynamical system theory.
Given your background and interest in modeling, I would highly recommend The Cambridge Handbook of Computational Psychology. The book provides an overview for several of the prominent modeling paradigms in cogsci, including dynamical systems, as well as many concrete examples--albeit most using other computational paradigms.
Dynamical systems, to my knowledge, are most used in the field of motor control. Though to be honest, dyn systems is not 'my' paradigm, and most of my knowledge comes from the aforementioned book.
In particular, the chapter "Dynamical Systems Approaches to Cognition" by G. Schoner gives this example that has always stuck in my mind:
A highly illustrative example comes from the orientation behaviors of the common house fly (Reichardt & Poggio, 1976; Poggio & Reichardt, 1976). Flies orient toward moving objects, which they chase as part of their mating behavior. Detailed analysis revealed that the circuitry underlying this behavior forms a simple controller: a motion detection system fed by luminance changes on the fly’s facet eye drives the flight motor, generating an amount of torque that is a function of where on the sensory surface motion was detected. If the speck of motion is detected on the right, a torque to the right is generated. If the speck is detected on the left, a torque to the left is generated. The level of torque passes through zero when the speck is right ahead. The torque changes the flight direction of the fly, which in turn changes the location on the facet eye at which the moving stimulus is detected. Given the aerodynamics of flies, the torque and its on-line updating generate an orientation behavior, in which the insect orients its flight into the direction in which a moving stimulus is detected.
I have a similar background to you, and have found a lot of interesting things in evolutionary game theory (you can follow links from my profile for more). But on the specific content of your question: I have come across to uses of dynamic systems on the opposite ends of cognition. Beer's work on modeling minimal cognition, and Busemeyer & Townsend's work on human decision making.
Beer's work models simple cognitive agents as dynamic recurrent neural networks. He then either hand-builds the agents, or evolves them, and analyzes them from a situated cognition perspective by treating them as dynamic systems.
Beer, R. D. (1995). On the Dynamics of Small Continuous-Time Recurrent Neural Networks. Adaptive Behavior, 3(4), 469-509. doi:10.1177/105971239500300405
Beer, R. D. (2003). The Dynamics of Active Categorical Perception in an Evolved Model Agent. Adaptive Behavior, 11(4), 209-243. doi:10.1177/1059712303114001
Minimal cognition research tends to use dynamic systems much more heavily than other fields I am familiar with. This blog provides a very thorough survey of papers in the field. A short 'manifesto':
Busemeyer & Towsend on the other hand try to describe how humans deliberate and make decisions. One of the key aspects they want to capture is not just the outcome of human decision making (with all of its flaws and irrationality) but also the deliberation time. To do this, they introduced decision field theory which models human decision making as a stochastic dynamic process.
Busemeyer, J. R., & Townsend, J. T. (1993) Decision Field Theory: A dynamic cognition approach to decision making. Psychological Review, 100, 432-459. [pdf]
Busemeyer, J. R. & Diederich, A. (2002) Survey of decision field theory. Mathematical Social Sciences, 43, 345-370. [pdf]
Esther Thelen and Linda Smith wrote a nice book examining human development from a dynamic systems approach:
https://mitpress.mit.edu/books/dynamic-systems-approach-development-cognition-and-action
Great to hear you're interested in applying your quantitative background to cognitive science; the field can definitely use more individuals like you!
I'm not sure to what degree these models might be meet your definition of "dynamical systems", but here are some off the top of my head:
From my social psychology perspective, there has been some computational modelling work on things like attitudinal influence dynamics. See: Nowak, A., Szamrej, J., & Latané, B. (1990). From private attitude to public opinion: a dynamic theory of social impact. Psychological Review, 97, 362-476.
Robin Vallacher and Andrzej Nowak crop up a lot in this field, and they're a joy to read. Perhaps this is less up your alley, since social psychologists are less interested in formulating differential equations and more interested in just applying the ideas of complexity science to their research methods, but it's certainly a step forward from what's been going on in the field so far. If anyone's interested in those broad concepts, try reading: Nowak, A. (2004). Dynamical Minimalism: Why Less is More in Psychology. Personality and Social Psychology Review, 8, 183-192.
