@JeromyAnglim's answer to your linked question says most of what I'd have wanted to say in response to your original (title) question. Difficulty of any given question in terms of item response theory is generally defined as the probability of answering it correctly at a given level of a (often latent) relevant skill. This definition of difficulty can be applied to an aggregate of questions as well (e.g., the probability of getting an "A" as one's final grade in a math class).
Mathematics may have a reputation for being a relatively "hard" subject because it relies more directly on quantitative, spatial, and logical reasoning than on verbal, social, or emotional reasoning. This may not be the case for many other subjects. These kinds of reasoning and the aptitudes associated with them aren't completely independent, but they are somewhat independent (i.e., intelligence as a whole is partly multidimensional, though the extent to which this is true is debated hotly), and may certainly be practiced somewhat independently. The latter set of reasoning skills (those not-so-relevant to math) may be practiced more frequently in everyday experience, especially during adolescence, and to a probably lesser extent, youth in general. Furthermore, those with special strengths in verbal, social, or emotional reasoning (and without corresponding strengths in math-relevant aptitudes, however rare that may be) may have more influence over the reputation of the subject. Thus the popular reputation may disproportionately reflect the opinions of those who are used to feeling stronger in other subjects, even though they may still be relatively strong in math as well.
A second major factor in both willingness to learn and the experience of interest is stereotype threat. This is one of probably many causes of academic anxiety in topics such as math—in fact, stereotype threat in math may be one of the most commonly studied varieties of the general social phenomenon. This may be due to Western society's many demographic stereotypes and prejudices with respect to quantitative reasoning, most notably ethnicity and gender. Stereotype threat research describes a complex process beginning with incidental membership in a group with a relevant stereotype. When stereotyped individuals are aware of a negative stereotype (e.g., their group is bad at math), they often fear that they will confirm the stereotype through their behaviors (e.g., earning a bad grade in math). This fear interferes with performance in often already high-pressure testing environments, often making the feared outcomes harder to avoid (an example of a self-fulfilling prophecy). Fear also reduces the capacity to enjoy the challenge of a test or other learning opportunity; enjoyment and self-confidence are crucial to the experience of intrinsic motivation.
Loss of interest and motivation, especially when due to fear, may often motivate use of cognitive coping mechanisms in defense of one's ego. One such mechanism is disparagement of the source of anxiety. In the all-too-common case of a person who experiences stereotype threat regarding his or (more likely) her mathematical ability, one may choose (not necessarily consciously) to see mathematics classes as inherently uninteresting, unfair, and difficult, even though none of these are necessarily true (for instance, imagine taking a middle school math class at your age). Naturally, one who believes these things is more likely to voice these things, particularly in as much as doing so may afford opportunities for ego compensation, or may be necessary to further offset others' negative judgments.
Stereotype threat theory has received some pretty serious criticism via empirical study (e.g., it's not the only cause of differences; (Sackett, Hardison, & Cullen, 2004). Nonetheless, it seems to have survived the test of meta-analysis reasonably well (Nadler & Clark, 2011). These analyses have revealed several moderators, including strength of identification with mathematics among women (Nguyen & Ryan, 2008).
References
- Nadler, J. T., & Clark, M. H. (2011). Stereotype threat: A meta‐analysis comparing African Americans to Hispanic Americans. Journal of Applied Social Psychology, 41(4), 872–890.
- Nguyen, H. H. D., & Ryan, A. M. (2008). Does stereotype threat affect test performance of minorities and women? A meta-analysis of experimental evidence. Journal of Applied Psychology, 93(6), 1314–1334. Retrieved from http://www.researchgate.net/publication/23489223_Does_stereotype_threat_affect_test_performance_of_minorities_and_women_A_meta-analysis_of_experimental_evidence/file/60b7d51a8da2aeacb0.pdf.
- Sackett, P. R., Hardison, C. M., & Cullen, M. J. (2004). On interpreting stereotype threat as accounting for African American-White differences on cognitive tests. American Psychologist, 59(1), 7–13. Retrieved from http://www.asc.upenn.edu/usr/ogandy/C45405%20resources/Sackett%20et%20al%20stereotype%20threat.pdf.
