Short answer:
Interestingly, even though the notion of projection has been intuitively appealing and plausible to many, the psychoanalytic explanation of how projection works has always been vague.
Long answer:
Sigmund Freud (1915) defined the "mechanism of projection" by explaining that engaging in it, the "ego" of a person "expels whatever within itself becomes a cause of unpleasure" (p. 136).
In a similar way, Anna Freud, who later worked on clarifying various defense mechanisms further, thought that projection allows people to get rid of painful thoughts: "The effect of the mechanism of projection is to break the connection between the ideational representatives of dangerous instinctual impulses and the ego" (p. 122).
Critics have noted that psychoanalytic writings have never explicated how exactly (by which process) projecting negative thoughts to others helps them in disconnecting these thoughts from the self.
Pointing to these criticisms and noting that empirical evidence in support of defensive projection has been scarce, Baumeister, Dale, and Sommer (1998, p. 1091), summarize
In retrospect, it was never clear how seeing another person as dishonest (for example) would enable the individual to avoid recognizing his or her own dishonesty. The notion that projection would effectively mask one’s own bad traits was perhaps incoherent.
Newman, Duff, and Baumeister (1997) propose and test an alternative, social-cognitive account of defensive projection. They posit that people often try hard to suppress unwanted thoughts. However, ironically, doing so makes such tthoughts all the more accessible (because when you try not to think about something, you constantly think about it). These accessible thoughts can then also color people's impression of others. This would then just be a by-product of trying to suppress unwanted thoughts about the self and not the core of a defense mechanism.
In addition, it is important to note that the term projection is also used in a more general sense, to describe people's tendency to overestimate the presence of their own characteristics, beliefs, and attitudes in others. This tendency results, for example, in the false consensus effect (see this earlier question), for which there is ample evidence. This general notion of projection is different from the Freudian one in that it is not limited to negative attributes but happens also for positive attributes the self. Various cognitive and motivational mechanisms have been proposed for this more general phenomenon (see Baumeister, et al., 1998, for example, for some pointers).
References
Baumeister, R. F., Dale, K., & Sommer, K. L. (1998). Freudian defense mechanisms and empirical findings in modern social psychology: Reaction formation, projection, displacement, undoing, isolation, sublimation, and denial. Journal of Personality, Defense mechanisms in contemporary personality research, 66, 1081–1124. doi:10.1111/1467-6494.00043
Freud, A. (1936/1992). The ego and the mechanisms of defence. London: Karnac Books.
Freud, S. (1915). Instincts and their Vicissitudes. The Standard Edition of the Complete Psychological Works of Sigmund Freud, Volume XIV (1914-1916): On the History of the Psycho-Analytic Movement, Papers on Metapsychology and Other Works, 109-140
Newman, L. S., Duff, K. J., & Baumeister, R. F. (1997). A new look at defensive projection: Thought suppression, accessibility, and biased person perception. Journal of Personality and Social Psychology, 72, 980–1001. doi:10.1037/0022-3514.72.5.980
