TEST RELIABILITY

TEST RELIABILITY

Reliability

In statistics, reliability is the consistency of a measuring instrument. This can either be whether the instrument gives or are likely to give the same measurement or in the case of more subjective instruments, whether two independent assessors give similar scores.

Reliability is the extent to which a test is repeatable and yields consistent scores. In order to be valid, a test must be reliable; but reliability does not guarantee validity. All measurement procedures have the potential for error, so the aim is to minimize it. The goal of estimating reliability (consistency) is to determine how much of the variability in test scores is due to measurement error and how much is due to variability in true scores.

For example: If a person takes a personality assessment today and scores high in a trait like dominance, and we retest that same person after six weeks, if the person again scores high in dominance than we can say that the test is reliable.  If however, the individual scored low in dominance, we would have to conclude that the measure was inaccurate and unreliable. 

Reliability can be improved by getting repeated measurements using the same test and getting many different measures using slightly different techniques and methods - e.g. We would not consider one multiple-choice exam question to be a reliable basis for testing your knowledge of "individual differences".  Many questions are asked in many different formats (e.g., exam, essay, presentation) to help provide a more reliable score.

Validity

Validity is the extent to which a test measures what it is supposed to measure.  Validity is a subjective judgment made on the basis of experience and empirical indicators. Validity asks "Is the test measuring what you think it’s measuring?” For example, we might define "violence" as an act intended to cause harm to another person (a conceptual definition) but the operational definition might be seeing:

• how many times a child hits a doll
• how  often a child pushes to the front of the queue
• How many physical scraps he/she gets into in the playground.

Are these valid measures of aggression?  i.e., how well does the operational definition match the conceptual definition?

Difference between Reliability and Validity

Validity and reliability are two terms that go hand in hand in any forms of testing.  It is important to understand the difference between the two terms.  Validity basically refers to research that provides evidence that a test actually measures what it is supposed to measure.  Reliability is more of a measure of consistency. In terms of accuracy and precision, reliability is precision, while validity is accuracy.

Bathroom scale analogy

An often-used example used to elucidate the difference between reliability and validity in the experimental sciences is a common bathroom scale. If someone that weighs 200 lbs. steps on the scale 10 times and it reads "200" each time, then the measurement is reliable and valid. If the scale consistently reads "150", then it is not valid, but it is still reliable because the measurement is very consistent. If the scale varied a lot around 200 (190, 205, 192, 209, etc.), then the scale could be considered valid but not reliable.

TYPES OF RELIABILITY

Reliability may be estimated through a variety of methods that fall into two types: Single-administration (split half and internal consistency method) and multiple-administration (test-retest and parallel form method). Multiple-administration methods require that two assessments are administered.

Each of these estimation methods is sensitive to different sources of error and so might not be expected to be equal. Reliability estimates from one sample might differ from those of a second sample if the second sample is drawn from a different population. (This is true of measures of all types--yardsticks might measure houses well yet have poor reliability when used to measure the lengths of insects.)

Inter-Rater or Inter-Observer Reliability

Inter-Rater or Inter-Observer Reliability measures homogeneity, is administering the same form to the same people by two or more raters/interviewers so as to establish the extent of consensus on use of the instrument by those who administer it. Used to assess the degree to which different raters/observers give consistent estimates of the same phenomenon.

There are two major ways to actually estimate inter-rater reliability. If your measurement consists of categories -- the raters are checking off which category each observation falls in -- you can calculate the percent of agreement between the raters. For instance, let's say you had 100 observations that were being rated by two raters. For each observation, the rater could check one of three categories. Imagine that on 86 of the 100 observations the raters checked the same category. In this case, the percent of agreement would be 86%. OK, it's a crude measure, but it does give an idea of how much agreement exists, and it works no matter how many categories are used for each observation.

The other major way to estimate inter-rater reliability is appropriate when the measure is a continuous one. There, all you need to do is calculate the correlation between the ratings of the two observers. For instance, they might be rating the overall level of activity in a classroom on a 1-to-7 scale. You could have them give their rating at regular time intervals (e.g., every 30 seconds). The correlation between these ratings would give you an estimate of the reliability or consistency between the raters.

Test-retest Reliability

The most commonly used method of determining reliability is through the test-retest method which measures stability over time.  The test-retest method of estimating a test's reliability involves administering the test to the same group of people at least twice.  Then the first set of scores is correlated with the second set of scores to determine if the scores on the first test are related to the scores on the second test.  Correlations range between 0 (low reliability) and 1 (high reliability), (highly unlikely they will be negative!).

This approach assumes that there is no substantial change in the construct being measured between the two occasions. The amount of time allowed between measures is critical. We know that if we measure the same thing twice that the correlation between the two observations will depend in part by how much time elapses between the two measurement occasions. The shorter the time gap, the higher the correlation; the longer the time gap, the lower the correlation.

Split-Half Reliability

In split-half reliability we randomly divide all items that purport to measure the same construct into two sets. We administer the entire instrument to a sample of people and calculate the total score for each randomly divided half. The split-half reliability estimate, as shown in the figure, is simply the correlation between these two total scores. In the example it is .87. It is the relationship between half the items and the other half. The random distribution for split half reliability can be done by software.

Parallel-Forms Reliability

In parallel forms reliability we first have to create two parallel forms. One way to accomplish this is to create a large set of questions that address the same construct and then randomly divide the questions into two sets. We administer both instruments to the same sample of people. The correlation between the two parallel forms is the estimate of reliability. One major problem with this approach is that we have to be able to generate lots of items that reflect the same construct.

Furthermore, this approach makes the assumption that the randomly divided halves are parallel or equivalent. Even by chance this will sometimes not be the case. The parallel forms approach is very similar to the split-half reliability. The major difference is that parallel forms are constructed so that the two forms can be used independent of each other and considered equivalent measures. For instance, we might be concerned about a testing threat to internal validity. If we use Form A for the pretest and Form B for the posttest, we minimize that problem. It would even be better if we randomly assign individuals to receive Form A or B on the pretest and then switch them on the posttest.

Internal Consistency Reliability

It is used to assess the consistency of results across items within a test. In internal consistency reliability estimation we use our single measurement instrument administered to a group of people on one occasion to estimate reliability. In effect we judge the reliability of the instrument by estimating how well the items that reflect the same construct yield similar results. We are looking at how consistent the results are for different items for the same construct within the measure. There are a wide variety of internal consistency measures that can be used and one of them is Average Inter-item Correlation.

Average Inter-item Correlation: The average inter-item correlation uses all of the items on our instrument that are designed to measure the same construct. We first compute the correlation between each pair of items, as illustrated in the figure. For example, if we have six items we will have 15 different item pairings (i.e., 15 correlations). The average inter item correlation is simply the average or mean of all these correlations. In the example, we find an average inter-item correlation of .90 with the individual correlations ranging from .84 to .95.

How reliable should tests be?  Some reliability guidelines

.90 = high reliability

.80 = moderate reliability

.70 = low reliability

• High reliability is required when tests are used to make important decisions, individuals are sorted into many different categories based upon relatively small individual differences e.g. intelligence (Most standardized tests of intelligence report reliability estimates around .90).
• Lower reliability is acceptable when tests are used for preliminary rather than final decisions, tests are used to sort people into a small number of groups based on gross individual differences e.g. height or sociability /extraversion.
• Reliability estimates below .60 are usually regarded as unacceptably low.

Sources of Error or Sources of Unreliability

• Respondent’s or subject’s mood, fatigue, or motivation which effect his or her responses
• Observer’s measurements, which can be influenced by the same factor affecting the subject’s responses
• The condition under which measurement is made, which may produce responses which do not reflect the true scores. Measurement errors are essentially random: a person’s test score might not reflect the true score because they were sick, anxious, in a noisy room, etc.
• Problems with the measurement instrument, such as poorly worded questions in an interview
• Processing problems such as simple coding or mechanical errors