11
$\begingroup$

On Programmers SE there is a popular question about scientific evidence for whether some people either are or are not able to program.

One person cites an unpublished manuscript by Dehnadi and Bornat (2006) who state in an abstract:

All teachers of programming find that their results display a 'double hump'. It is as if there are two populations: those who can, and those who cannot, each with its own independent bell curve.

That said, educators have choice in how individual assessments are scored, how assessments are combined, and whether and how any final scores are scaled. These could all influence the shape of the final grade distribution.

I also imagine that to get a double hump would suggest that the variables that best predict student performance (e.g., time spent on subject, general ability, subject-specific ability) would have to either show a double hump or show a discontinuous relationship with performance. In particular, I can see that students that drop out of a subject without formally un-enrolling are likely to fail very badly and appear in a separate hump. I also suspect that certain types of subjects where answers are clearly correct or incorrect and where intuition from everyday life is less directly relevant may also give rise to a process where it appears as if people can either do the subject or they can't.

Questions

  • What is the distribution of grades in university subjects?
  • To what extent does the distribution of grades in programming subjects differ from non-programming subjects? Is it bimodal?
  • Is there any theories or empirical evidence about what leads to bimodal grade distributions?

References

  • Dehnadi, S. & Bornat, R. (2006). The camel has two humps (working title). Middlesex University, UK. PDF and summary page.
Improve this question
$\endgroup$
1

3 Answers 3

Reset to default
5
$\begingroup$

Our university has a public grade distribution database, so I did a really quick analysis on some historical data to see if there was any support for this idea. To my surprise, it appears that there might be. But, my analysis is very limited.

I downloaded the grade distributions for all intro psychology and intro computer science classes from 2010-2014. This gave me the grades of about 15,000 psychology students and about 1,500 computer science students. I summed up the count for each letter grade, and then normalized to a percentage of students. Apologies for the somewhat ugly chart, but here's what I found:

enter image description here

The distribution for letter grades seems pretty consistent across both disciplines, but what jumps out to me is the assignment of "W" grades, which is the grade received when a student withdraws from the course after being enrolled for at least the first week of classes. About 20% of all intro CS students withdraw from the class, while less than 5% of intro psych students do. This might indicate that a larger number of CS students find the course to be too challenging for them, and withdraw rather than get a low grade.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Do you (still) have the data in numerical format somewhere? I'd hate to have to reverse it from the graph... $\endgroup$
    – got trolled too much this week
    Commented Dec 16, 2017 at 1:47
2
$\begingroup$

According to the research of Elizabeth Patitsas, the distribution is not bimodal. She sampled 778 University of British Columbia CS courses, as well as 5 University of Toronto CS courses and found the vast majority to not be bimodal:

An estimated 85.1% of the final grades in UBC's undergrad CS classes are normally-distributed. 5.8% of the classes tested as being bimodal, which isn't a whole lot more than the false positive rate I'd expect to see (5%).

You can read more about these results in the blog post linked above or in "Evidence That Computer Science Grades Are Not Bimodal" by Elizabeth Patitsas, Jesse Berlin, Michelle Craig and Steve Easterbrook.

To make things worse, when discussing this, people who believe the distribution is bimodal are more likely to see a bimodal distribution in a normally distributed sampling:

We delayed the debriefing until after our analysis was complete, on the assumption you’d want to know the preliminary results of the study. We indeed found that participants who agreed more strongly with the statement “Some students are innately predisposed to do better at CS than others” were statistically significantly more likely to label ambiguous distributions as bimodal.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Nice find; the psychology of the observer explained (in the paper abstract):"A random half of participants were primed to think about the fact that CS grades are commonly thought to be bimodal; these participants were more likely to label ambiguous dis- tributions as \bimodal". Participants were also more likely to label distributions as bimodal if they believed that some students are innately predisposed to do better at CS. These results suggest that bimodal grades are instructional folklore in CS, caused by conrmation bias and instructor beliefs about their students." $\endgroup$
    – got trolled too much this week
    Commented Dec 16, 2017 at 0:44
  • $\begingroup$ I do have to wonder if the UBC data is representative though. They do mention a paper with an over-lapping set of authors (Zingaro et al.) in which the exam is turned into many fine-grained questions. So, if this practice is commonly applied at UBC (but not in other places), then UBC data is not representative. Ah, never mind the UBC data is from 1996 to 2013, while the Zingaro paper is from 2012, so not much chance it was applied unless it was "prior art" that nobody thought to publish about for years (unlikely). $\endgroup$
    – got trolled too much this week
    Commented Dec 16, 2017 at 1:29
  • $\begingroup$ @Fizz fair criticism which she acknowledges in her blog post. I guess we could do the same tests on Josh's data? $\endgroup$
    – Seanny123
    Commented Dec 16, 2017 at 1:31
  • $\begingroup$ Good point on trying that with different data. The methodological question is what do we do with the large number of withdrawals. The Patitsas paper says NOTHING about that... which is probably the biggest question mark, now that I realized it. $\endgroup$
    – got trolled too much this week
    Commented Dec 16, 2017 at 1:39
  • $\begingroup$ I'm pretty sure the 2014 meta-analysis counted withdrawals as failed. That's because they have ~30% average failure for US (it has a graph by country as well), so with Josh's data, you'd definitely have to add F+W to get anywhere near that 30% number. Also, at some US institution I'm familiar with, the D's would be failure as well because the students need at least a C to enroll in subsequent CS classes. The Canadian failure rate in the meta-analysis was only 20%, by the way. $\endgroup$
    – got trolled too much this week
    Commented Dec 16, 2017 at 2:24
2
$\begingroup$

(This is a bit brief given the complexity of the question. I'll expand later if I can summarize more of the findings.)

Probably the only thing which is certain is the high-failure rate in CS1-level classes, worldwide. From a recent (2014) meta-analysis:

In this paper, we answer the call for further substantial evidence on the CS1 failure rate phenomenon, by performing a systematic review of introductory programming literature, and a statistical analysis on pass rate data extracted from relevant articles. Pass rates describing the outcomes of 161 CS1 courses that ran in 15 different countries, across 51 institutions were extracted and analysed. An almost identical mean worldwide pass rate of 67.7% was found. Moderator analysis revealed significant, but perhaps not substantial differences in pass rates based upon: grade level, country, and class size. However, pass rates were found not to have significantly differed over time, or based upon the programming language taught in the course.

There was a prior world-wide study in 2007 (using online self-reported data) which agreed with this number (~33% failure rate), although the distribution observed for the passing students was different:

enter image description here

Alas neither paper compares this number with other fields, but the 2007 one also calculated that graduation from CS tertiary education is low worldwide, 26.8% so getting past CS1 hardly guarantees smooth sailing thereafter.

enter image description here

The latter [counterproductively 3D] graph is based on UNESCO data, which is why North America is absent (the US boycott[ed] UNESCO.)

There is however one paper which does state that:

Studies in Ireland [1] have shown that Computer Science has the highest failure rate, giving a 26.9% non-completion rate for Computing courses.

[... citing:]

[1] Mark Morgan, Rita Flanagan and Thomas Kellaghan, "Higher Education Report A Study of Non-Completion in Undergraduate University Courses" published February 2001

Several theories for the CS1 failure rate have been advanced: genetic basis, "learning edge momentum", and/or stumbling blocks but it's too early to tell if any of these have convincing empirical support. (In fact the last paper argues that insofar there isn't a good data gathering strategy that would even allow one to one to decide.) And one academic blog suggests (based on its own data) that none might be correct. Yet another paper suggests that (Duckworth's) grit plays a substantial role. And with respect to the final grade being an aggregation one paper (based on a single course) suggests that students' bimodal final grades come from the practical side of the evaluation:

enter image description here

Alas I was not able to find other papers looking into this aspect, however it does look to me that the grit paper does point in the same direction; it found that grit was a much better predictor than ACT (itself considered a proxy for IQ) for CS1 success.

The humps paper mentiond in the question is not taken very seriously (most of these papers dont' cite it), and even by its own authors themselves these days. It also does not seem to have been the first to observe the grade distribution phenomenon. I'm not entirely sure why it became popular on-line. Its strong language and/or the SE-effect maybe. For instance this 2005 paper noted the 30% to 40% failure rate as well.

Also here's a graph with the failure rate by country from the 2014 meta-analysis:

enter image description here

The authors note that the top 3 failure leaders Portugal, Germany and Brazil were represented by a pretty small sample in this study, so the data for those countries cannot really be relied upon. On the other hand, "of the 4 countries which made 80% of the sample, Finland was found to have the lowest pass rate 57.7%". The other three countries in this 80% bulk of the data, US, UK, and Australia had statistically indistinguishable rates.

Also, the methodology should have been more upfront in this meta-analysis but they do say "Currently we have found in this study 3/10 students do not complete, or fail CS1." So they definitely count withdrawals as failure, basically, for the statics they reported.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.