correlation coefficients and true score?

Topic: Validity and reliability - issues and discussion Date: 27 November 2006 11:36 AM
Subject: correlation coefficients and true score? Author: Mondy, Steven
Last night, I was reading about how knowing the correlation coefficient (positive, negative, and no relationship) is quite a useful way to determine the reliability of a test. The study book was talking about how highly correlated scores can mean high reliability. It got me thinking about specific ways of gauging the reliability of our tests and test items. It was suggested that we could do things like give the same test twice (as long as not too much time has elapsed between the administration of the tests), or have such things as split tests, where we compare the two parts of one test. Up until this point, I didn’t really see the value of giving a test more than once in relatively the same, if not exactly the same format. I thought that it might be a waste of time, as I have been always encouraged to believe a test must be held back in extreme secrecy to not influence cheating and studying for specific items. According to the study materials, we see that there may be a change in the scores in that they may be slightly higher, but we can account for this in working out the students overall performance, and correlate the scores to see how that student performed in relation to other students doing the same test (norm referencing?). Then we can see if it has been error from the test format or from other things such as communicative ability, personal attributes, or other random factors (study materials: module 4).
Another interesting thing that I came across was the comparison of the two students with different scores, and being able to say that a student with a slightly higher score in a test may not be better than the student with the lower score, if we take into account their true score (true score=observed score +-measurement error). This was quite striking as I had assumed that ranking students in order of best to worst according to their raw scores was sufficient to get an idea of general performance. This idea of true score has made me rethink how I look at data, especially in terms of what judgments I make when I see the scores of two or more students are very close. What makes the difference between an A and an A- for example, if the range overlaps so significantly?
What do you think about correlation coefficients and true score?
Steven Mondy