Every decision you make as a behavior analyst rests on data. Before you ask "Is the intervention working?", you have to ask a quieter question: "Can I trust these numbers?" This lesson gives you the three yardsticks for answering it.
Why it matters
Untrustworthy data lead to confident but wrong decisions. You might keep an ineffective procedure because the graph looks good, or drop an effective one because an observer slowly changed how she scored. Clients, caregivers, and funders rely on your graphs, so judging measurement quality is part of providing effective, ethical services.
Practice vignettes often describe a data problem and ask you to name it: a measure that misses the point (validity), numbers that do not match what really happened (accuracy), or numbers that would not repeat if the same events were measured again (reliability).
Key ideas
- Validity: are you measuring the right thing? Valid measurement directly captures a socially significant behavior, along the dimension that matters for the question, at times and places that represent the behavior of interest.
- Accuracy: do the numbers match reality? Measurement is accurate when the observed values match the true values of the events. To judge accuracy you need a true value to compare against, which is usually obtained with special care, such as repeated careful scoring of a video or a calibrated instrument.
- Reliability: are the numbers consistent? Measurement is reliable when measuring the same events again produces the same values.
- Reliable does not mean accurate. An observer can make the same error every time. But measurement that is consistently accurate is, by definition, also reliable.
- Accurate and reliable does not mean valid. You can count something perfectly and still be counting the wrong thing.
Common threats to validity
- Indirect measurement: relying on interviews, rating scales, or reports instead of observing the behavior itself.
- Measuring the wrong dimension: for example, recording how long a student sits at a desk when the goal is how many problems she solves correctly.
- Measurement artifacts: data that reflect the measurement method rather than the behavior. Sources include discontinuous methods that over- or underestimate, observation periods scheduled when the behavior rarely (or always) happens, and scales that cap or floor the data.
Common threats to accuracy and reliability
- Poorly designed measurement systems, such as a cumbersome data sheet or too many behaviors to score at once.
- Inadequate observer training, with no competency criterion before collecting real data.
- Unintended influences on observers: expectations about how the data "should" look, observer reactivity (scoring differently because someone is checking), and observer drift (gradually changing how the definition is applied over time).
Definitions
- Validity: the degree to which the data directly reflect the behavior and dimension of interest under the conditions that matter.
- Accuracy: the degree to which observed values match the true values of the events measured.
- Reliability: the consistency of measurement; repeated measurement of the same events yields the same values.
- Measurement artifact: an apparent feature of the data that comes from how behavior was measured, not from the behavior itself.
- Observer drift: an unintended, gradual change in how an observer applies a measurement system or definition.
- Interobserver agreement (IOA): the degree to which two or more observers who independently measure the same events report the same values.
Examples from a session
Three yardsticks, three problems
- Validity: A clinic tracks "minutes on the communication app" for Amara, age 6, when the goal is independent mands. Time on the app can rise while requesting stays flat. The measure is precise, but it is not the behavior of interest.
- Accuracy: An RBT records 18 instances of elopement for Mr. Ortiz, an adult in a day program. A careful review of the session video shows 25. The observed value does not match the true value.
- Reliability: A supervisor rescores the same video of Theo, age 12, three times and gets three very different counts of call-outs. The measurement is inconsistent, so no single count can be trusted.
- Observer drift: Months into a school program, an RBT begins counting loud humming as "vocal disruption," which the definition excludes. Nothing about the student changed; the scoring did.
- Non-example (not drift): An RBT scores more call-outs only on days the BCBA is in the room. The change follows the checking, not the passage of time, so it is observer reactivity.
Practical safeguards follow directly from the threats: write clear definitions with examples and non-examples, train observers to a criterion with practice videos, keep data systems simple, retrain and recheck observers periodically, and schedule observations when and where the behavior matters.
Common mistakes / exam traps
- Trusting matching totals. Total count IOA compares only the two totals, so observers who each score 10 of 20 intervals can still disagree on which ones. Interval-by-interval IOA divides the intervals where both agree (both scored or both blank) by all intervals.
- Mixing up accuracy and reliability. Ask: was the measured value compared with a true value (accuracy), or with a repeat measurement of the same events (reliability)?
- Forgetting validity is about the target. If the stem says the data are "consistent and match the video" but the measure misses the goal behavior, the problem is validity.
Exam trap
High IOA vs. accuracy: IOA compares one observer with another; accuracy compares the record with a true value. Two observers trained on the same flawed definition can agree perfectly and both be wrong, so high IOA raises believability, not truth.
Exam trap
Reliability vs. accuracy vs. validity: reliability is necessary but not sufficient for accuracy, and neither one guarantees validity. Work through the questions in order: Right behavior and dimension? Matches the true value? Repeats when re-measured?
Quick check
Ana's team wants to reduce her afternoon tantrums at home, which happen mostly between 3:00 and 6:00 p.m. Data are collected only from 9:00 to 9:30 a.m., when a therapist is available. The recorded counts are consistently near zero. What is the main concern with these data?