As defined by the National Council on Measurement in Education (NCME), psychometrics refers to the field in psychology and education that is devoted to testing, measurement, assessment, and related activities. Subject Matters interviewed Jitendra Pradhan, director of CEM’s Psychometric Services Unit, to gain a better understanding of this field of study and the critical role his unit plays in the test development process.
Q: What is psychometrics?
A: Psychometrics is the study of mental measurement (as opposed to a physical measurement). In particular, educational measurement is a type of psychometrics that has to do with measuring knowledge, skills, abilities, and educational achievement. There is a wide range of duties that psychometricians perform on a daily basis to solve issues that come up in educational measurement. We analyze items to determine the level of difficulty, determine why people will answer certain questions in a certain way and how this affects test taking, and identify whether a test truly measures someone’s ability to master a skill, among many other activities.
Q: What are some nuances to determining quality test questions?
A: Quality for test questions means that they do a good job of separating people who know the material from people who don’t. Difficulty is sometimes related to quality: some questions that are very hard or very easy are that way because they are badly written. But others are very hard or very easy because they deal with topics that are either very challenging or very well known.
Q: If you get the question right, does it mean the test question is easy?
A: Not necessarily. Any one question might be easy for any one person. We look at groups of people, not individuals. If everyone gets a question wrong, it means the question is probably too difficult (or it is a badly written question, or it was mis-keyed, or the practice has changed). How a group of people answers a question gives us a lot of information to work with.
Q: How does the potential for student guessing affect your work?
A: As far as guessing goes, research has shown that most test-takers do not randomly guess. In fact, multiple choice questions with three options can work as well as multiple choice questions with four options.
Q: How are test scores determined?
A: Usually for multiple choice tests, the test score is the number of questions that a test-taker answers correctly. However, the test score for our exams is not reported to test-takers. Instead, we provide a letter grade. The reason for this is that our tests are criterion-referenced tests, which means that the test scores are divided into performance categories based on whether candidates met certain standards. This allows us to interpret the results with more clarity and accuracy.
Q: Can you explain how you know for sure that a test is reliable and valid?
A: The best analogy I can think of is a thermometer that you use to measure your body’s temperature. If you took your temperature repeatedly with the same thermometer you would hope that the reading would be almost the same each time—maybe different within a degree or so. But if the thermometer always varied by a few degrees or more, you would discard the thermometer as being unreliable. Similarly, we would expect that if an examinee took the same test (which is the measuring instrument in this case) a second time, their scores should be close to the score on the first attempt. Our job is to estimate how much variation in scores you can expect, if it were possible to have repeated testing. If you can expect or measure consistent scoring, the test is reliable. If not, the test needs adjustment. With validity, you are asking how good the evidence is for what you want to use the test score for. For example, if a mathematics test has many word problems that depend heavily on reading comprehension, then the test score might reflect reading ability as much as mathematics ability. In this case, using the test score to make decisions about whether someone has mathematics ability would be questionable.
Q: Anything else you would like our readers to know or where can they find more information on this subject?
A: One of the best resources I have found is available freely on the internet, produced by the National Council on Measurement in Education. <hyperlink: http://www.ncme.org/ncme/NCME/Publication/ITEMS/NCME/Publication/ITEMS.aspx?hkey=2e36be6e-79cd-420f-bce3-51ded8d8ab68 >
All different topics in measurement are covered, however these are research papers requiring some background in measurement.
An easier reference for faculty wanting to improve their knowledge of assessment in the classroom would be Assessment of Student Achievement by C. Keith Waugh and Norman E. Gronlund.