What topics would fall under theoretical cognitive science, if there could be such a term? I am referring to the more mathematical, foundational topics vs. topics dealing with specific cognitive phenomena.
I would imagine that artificial intelligence and any other kind of cognitive modelling of behaviour or the brain would fall under that category. Essentially, anything that isn't about directly describing human behaviour, but is about modelling it computationally.
Look up the research on Theoretical Neuroscience, it's really quite fascinating. There is also an going project about literally building a brain: http://en.wikipedia.org/wiki/Blue_Brain_Project
It's really neat stuff.