Neurobiology and Evolution are not exactly my field of expertise, but I'll try to answer anyway.
Are there any neurotransmitters that appeared only in human evolution history?
Probably not. The common human neurotransmitters we know of, are also found in other animals. More surprisingly, most of them are also found in organisms with no nervous-system, such as microorganisms and plants. This has even led some to advocate the change of the name neurotransmitters to biomediaters (Roshchina, 2010).
It is believed that most of the changes in the nervous system occur at the circuitry level, and are not the introduction of new types of neurotransmitters. For example, Venter et al (1988) write:
The presence of hormones, neurotransmitters, their receptors and biosynthetic and degradative enzymes is clearly not only associated with the present and the recent past but with the past several hundred million years. Evidence is mounting which indicates substantial conservation of protein structure and function of these receptors and enzymes over these tremendous periods of time. These findings indicate that the evolution and development of the nervous system was not dependent upon the formation of new or better transmitter substances, receptor proteins, transducers and effector proteins but involved better utilization of these highly developed elements in creating advanced and refined circuitry.
And Winer and Larue (1996) write:
Many features in the mammalian sensory thalamus, such as the types of neurons, their connections, or their neurotransmitters, are conserved in evolution.
However, it is important to remember that there are changes related to the neurotransmitters and how they are used in the brain. For example, Burki and Kaessmann (2004) showed that one gene encoding the enzyme glutamate dehydrogenase, which is important in the neurotransmitter glutamate recycling process has appeared less than 23 million years ago in our evolution. Grailhe et al (2001) show that one type of Serotonin recceptor has disappeared somewhere in the evolution between rodents and humans.
- Roshchina, V. V. (2010). Evolutionary considerations of neurotransmitters in microbial, plant, and animal cells. In Microbial Endocrinology (pp. 17-52). Springer New York.
- Venter, J. C., Di Porzio, U., Robinson, D. A., Shreeve, S. M., Lai, J., Kerlavage, A. R., ... & Fraser, C. M. (1988). Evolution of neurotransmitter receptor systems. Progress in neurobiology, 30(2-3), 105.
- Winer, J. A., & Larue, D. T. (1996). Evolution of GABAergic circuitry in the mammalian medial geniculate body. Proceedings of the National Academy of Sciences, 93(7), 3083-3087.
- Burki, F., & Kaessmann, H. (2004). Birth and adaptive evolution of a hominoid gene that supports high neurotransmitter flux. Nature genetics, 36(10), 1061-1063.
- Grailhe, R., Grabtree, G. W., & Hen, R. (2001). Human 5-HT5 receptors: the 5-HT5A receptor is functional but the 5-HT 5B receptor was lost during mammalian evolution. European journal of pharmacology, 418(3), 157-167.