I think we should first clarify the difference between simple mathematical operators and a complete formal axiomatic system. I think it is very clear that exponentiation, square root or multiplication operations are taught. They represent a higher level cognitive ability than simple addition and subtraction.
In the level of more innate abilities, we can still claim that some numeric tendencies do exist in the human infant. Moreover, these are shown to exist in primates, and even rodents (for instance, see Brannon & Terrace, 1998). It is argued that in infants, numbers are represented in two distinct systems; one is responsible for small numbers and their relations, and the second responsible for mass amounts of items (have a look at Feigenson, Dehaene, Spelke, 2004). Briefly speaking, infants can differentiate 2 items from 3 items or 1 item; on the other hand they can distinguish 40 items from 80 items better than they could 60 or 70 items. So this second system operates on a relatively macro level. Moreover, first system not only provides a comparison operator, but addition also. So, infants can understand that two individual objects make 2 objects. For instance, suppose you have a box and three dolls in it. If you remove two of them, they can understand that there is still one more in it. To summarize, numeric abilities at least partially seem to have innate roots.
This is not surprising at all. In evolutionary terms, it is not hard to see that counting objects can be essential, up to a certain point. For instance, it is very important that prey should recognize whether there is a single hunter or two of them. As another example, it is equally important to recognize whether there are 100 or 200 footsteps on the ground, but not that important whether there is 100 or 105 footsteps. There are surely cognitive abilities with innate roots; one other such ability is human face recognition for example. Infants can detect an upright human face right after they are born.
How is such information represented? It does not have to be in an axiomatic fashion. Neurons have a very simple structure themselves, but networks of neurons are capable of great things. They can be represented in states of a dynamical system, like associative memory. Or they can be represented by simple feedforward structures. It is very hard to distinguish that. My primitive claim is that assuming innate formal axiomatic systems is against the rule of parsimony.