1
$\begingroup$

I posted this question in the programming forums but think I might get some answers if I post here as well... hope that's ok.

I'm trying to follow the steps in this paper to reproduce the excel table shown, but for some reason I'm unable to reproduce the 'estimated' values on the right hand side of the sheet, despite correctly calculating c for each of the six levels. I'm confused as to how the author arrived at those values, so I would appreciate if anyone could point out what's going on.

$\endgroup$
1
$\begingroup$

The cross posting make it hard to answer, but I would rather the points here ...

Apart from not running on my machine (something about convert_to_proportions that I don't care to debug or understand) the code you posted seems conceptually correct. Looking at an abridged version of the code

### Input the signal and noise values
## "sure signal", "likely signal", "guess signal", ... ... "sure noise"
signal = [1230, 496, 358, 272, 215, 165]
noise = [111, 216, 349, 540, 625, 895]

# cumulate them
csignal, cnoise = np.cumsum(signal), np.cumsum(noise)

# convert them to proportions and correct so that all but the last elements will never equal 1
propsignal = convert_to_proportions(csignal)
propnoise = convert_to_proportions(cnoise)

# calculate d' for all, except for the last element which is always 1.
d_primes = [get_d_prime(i[0], i[1]) for i in zip(propsignal[0:-1], propnoise[0:-1])]

# calculate c in the same way
cs = [get_c(i[0], i[1]) for i in zip(propsignal[0:-1], propnoise[0:-1])]

#estimate the noise and signal curves. 
#*** This is where my numbers disagree with those in the paper.
estimated_noise = [stats.norm.cdf(-c) for c in cs]
estimated_signal = [stats.norm.cdf(np.mean(d_primes) - c) for c in cs]

you do not actually need to calculate cs. I suggest instead just do

cs = np.arange(-10, 10, 0.01)

This way cs varies over the entire range of interest and makes a smooth curve and you avoid issues from interpolating between the points. When you do this, the fitted curve passes very close to the third data point, but in the manuscript the fitted curve passes very close to the fourth data point. The discrepancy between the paper and your plot seems to arise from

stats.norm.cdf(np.mean(d_primes) - c)

It turns out that np.mean(d_primes) is very close to d_primes[2] (i.e., the $d^\prime$ associated with the third category). If we replace the offending line with

stats.norm.cdf(d_primes[3] - c)

everything seems to line up.

From a conceptual standpoint, I think that d_primes[3] is the best estimate of $d^\prime$ since that corresponds to the division between possible signal and possible noise on the 6 point rating scale. That said, there may be a method for which np.mean(d_primes) is the best choice. It is also worth noting that the numbers in the excel table in the paper do not match the figure in the paper.

In summary, if you want to get the numbers in the table, use np.mean(d_primes), but if you want the figure to match use d_primes[3].

$\endgroup$
  • $\begingroup$ Wow, thanks a lot for the help! The reason I chose np.mean(d_primes) is because cell J14 in the author's spreadsheet looks like it is calculated as the mean of the d` array, which they are then using to calculate the estimated measures in the adjacent columns. That said, I happened to notice that by chance - it wasn't explicitly written so it could be wrong. I don't suppose you know of any other resources to guide through this process? I haven't been able to find much. $\endgroup$ – fffrost Jun 4 '18 at 21:33
  • $\begingroup$ @fffrost I changed my mind I like d_primes[3] much better than np.mean(d_primes) (see edit). If you want to match the numbers in the table, use the mean, but if you want to match the figure use the 3rd value. $\endgroup$ – StrongBad Jun 4 '18 at 21:41
  • $\begingroup$ Ah that makes a lot of sense (about d_primes[3] being the best estimate of d`), didn't really consider it before. I will go with this rather than the mean, despite the author's value in the table - I guess it's a mistake or something. $\endgroup$ – fffrost Jun 4 '18 at 21:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.