When one is either learning or working with a new concept that is -
complex, untangle, counter-intuitive, tedious, multi-dimensional,
boring and difficult, it appears mood is altered to a lower state,
very similar to that of a depressive state, however is temporary,
until another concepts comes along (still new) that is simple, clear
and straight forward.
Are there any studies that show correlation to dopamine and serotonin
in such a case
Yes, there are studies that suggest that failure can lead to reduced dopamine and seratonin. Essentially, dopamine production is argued to be produced by reward . One type of reward is success at at something. By extension, where there is no success, less dopamine is therefore produced. As dopamine provides energy and a motivation to act  individuals have less motivation and energy. Further, in parallel more they may produce more cortisol, leading to a them developing a depressive state [e.g., 2]
more importantly, are there any studies or know
methods that reverse this, causing instead increased dopamine and
serotonin during such events?
Yes, there are some known methods that you can use to increase serotonin/dopamine before/during these sorts of events:
- Break down the task so that you get a dopamine boost everytime you
get one part done [e.g., 1]
- Manage your expectations - much of reward is about expectations. If
you want to be better than before when doing something, as opposed
to be the best, then you are much more likely to find something
rewarding, and produce more dopamine .
- Listen to music - in some cases this can raise dopamine levels to
ensure that you have sufficient motivation to work 
- Consume substances that will increase dopamine( e.g., st. johns wort)
or dopamines precursors (e.g., Tyrosine).
- This article provides a set of potentially useful approaches that
may be of use for increasing dopamine. However, before adopting them
I would investigate the literature cited.
- Exercise prior to the task 
I am happy to clarify or expand on any part of this answer if you want, so just let me know.
 Hoebel, B. G., Rada, P. V., Mark, G. P. and Pothos, E. N. (1999) 'Neural systems for reinforcement and inhibition of behavior: Relevance to eating, addiction, and depression'.
 Salimpoor, V. N., Benovoy, M., Larcher, K., Dagher, A. and Zatorre, R. J. (2011) 'Anatomically distinct dopamine release during anticipation and experience of peak emotion to music', Nat Neurosci, 14(2), 257-262.
 Young, S. N. (2007). "How to increase serotonin in the human brain without drugs." Journal of Psychiatry & Neuroscience : JPN 32(6): 394-399.