module.gif (6711 bytes)

 

Interobserver Reliability

Goal: Understandwhy reliability measures are needed and how they are computed in observation studies.

Objective 1:

Discuss the reasons why a researcher may want two or more people observing the same event.

Benchmark 1.1:

Give an accurate explanation of the main reason for using multiple observers in research.

Objective 2:

Define "Interobserver Reliability".

Benchmark 2.1:

Provide a correct definition of the concept of interobserver reliability as used in psychological research.

Objective 3:

Define "Interobserver Reliability".

Benchmark 3.1:

Give a correct and complete rationale for computing and reporting levels of interobserver reliability in psychological research.

Benchmark 3.2:

Explain why high reliability may not mean that the observations are an accurate indication of what occured.

 Because of the potential for an individual observer's past experience and sensory and perceptual capabilities to bias what is attended to, researchers often plan to have two or more people collecting observational data at the same time. A measure of the validity and objectivity of the data then is the extent to which the observers agreed about what happened. If there is a very low level of agreement about what happened in a given situation, then we cannot have much confidence in any of the individual reports. High agreement levels do not always indicate accurate observations, for example, if two observers share a bias. Nevertheless, the degree of agreement usually is measured and reported as part of an observation study.

The level of agreement between observers viewing the same event is referred to as the "interobserver reliability". The formula for computing the interobserver reliability is:

Number of agreements /Number of opportunities to agree x 100

For example, if two observers in a classroom were asked to record, in each minute for twenty minutes, whether or not a student was out of their seat, and their notes showed agreement for 16 of the 20 minutes, then the reliability would be:

16 /20 x 100 = 80% reliability