Reliability in once-off tests

Topic: Validity and reliability - issues and discussion Date: 26 November 2006 5:45 PM
Subject: Reliability in once-off tests Author: Mondy, Steven


How do we ensure reliability in a test we make and use only once? Often I find myself in a situation in which I’m unable to use a test again in the same kind of context, or with students with similar abilities. I often find that I may need to wait 2-3 years before actually being able to teach the same content. By that time, the tests lose validity? Reliability is closely related to having items that are consistent over time. Hughes and Bachman talk about creating item banks, which store items that we know are reliable, in order to reduce the effort that’s needed to make tests. I guess this is one way to be more certain that whether a test is a one-time thing, or to be used again, the content of the test will be reliable. Also, we shouldn’t understand a test only in terms of a complete, unchangeable entity, but a series of well-formulated items that put together correctly, can be reliable and valid for each and every time we use them. So, what I’m trying to say here is, whenever we make a test, we should try to create a test that may have items that are often used again and again, but not necessarily tests as a whole that are used again and again. Is it really possible to use an intact, unchangeable test in exactly the same way with new students anyway?

I’ve also been thinking of how one can develop tests that are specifically used more as a teaching tool, rather than a measurement or evaluation instrument. If students are prevented from seeing a test and analyzing the mistakes that are made in order to solely ensure reliability, then the student is missing a valuable learning experience. I think that tests can be used as a way to have students experience giving an answer, and then checking their answers with a model, or having an instructed lesson following the tests administration (looking over the actual test) in order to clarify the main points of the test within a student’s mind. Of course, we may need to forgo the ability to use the test in the future, but what can be gained may outweigh the possible disadvantages. This may be where the idea of item banks may help?! What do you think?
Steven Mondy