Your supervisor's decisions rest on two things: the data you bring back and the plan being run as written. If either is off, the decisions built on it are off too.
Why it matters
A graph showing a skill improving leads the team to fade prompts or move on; a graph showing problem behavior staying high leads to a plan change. Those choices only help the learner if two things are true:
- The data reflect what really happened. If the numbers are wrong, the graph tells a false story.
- The plan was run the way it is written. If the procedure changed in practice, the team cannot tell whether the plan works, because the plan was never really tested.
When either one fails, the learner pays for it. An effective plan may be dropped, an ineffective one may continue for months, or a skill may be marked mastered before the learner can really do it.
Key ideas
- Good data are accurate and reliable. Accurate data match what actually happened. Reliable data come out the same when the same behavior is measured again in the same way. Data can be consistent and still be wrong, for example if an observer makes the same mistake every time.
- Interobserver agreement (IOA) is a check, not a guarantee. When two people record the same session independently and their numbers agree, the team can trust the data more. High agreement does not prove both observers are correct; they could share the same misunderstanding of the definition.
- Procedural fidelity means doing the plan as written. It is the degree to which a procedure is carried out the way the behavior plan or skill program describes, every step, every time. It is also called treatment integrity.
- Low fidelity hides what is working. If behavior changes, the team cannot tell whether the plan or the deviation caused it. If behavior does not change, the team might wrongly blame a plan that was never fully used.
- Small changes add up. Reinforcing prompted responses when the plan says independent only can build prompt dependence. Attention for a behavior the plan says to stop reinforcing can keep it going. Skipping a step "because we were running late" means the learner gets a different program than the one being graphed.
- Your job is to prevent, notice, and report. Follow the plan exactly, record in the moment, ask when a definition or step is unclear, and tell your supervisor about errors honestly and promptly. Do not quietly fix data or change procedures on your own.
Definitions
- Accuracy: the extent to which recorded values match the true values of the behavior that occurred.
- Reliability: the consistency of measurement; repeated measurement of the same behavior yields the same values.
- Interobserver agreement (IOA): the degree to which two observers who independently measure the same behavior report the same values. Many teams treat about 80% as a minimum.
- Procedural fidelity (treatment integrity): the extent to which an intervention is implemented as planned.
- Observer drift: an unplanned shift, over time, in how an observer applies an operational definition.
- Observer reactivity: a change in how someone records data because they know their data are being checked or observed.
Examples from a session
Three ways things go wrong
- Late recording. At a busy preschool, Jordan, an RBT, plans to write down Mia's requests "after circle time" and then estimates 10. The real number was 4. The graph now shows a jump that never happened. If estimates like this keep going, the supervisor may decide the skill is mastered and move on too soon.
- Observer drift. Over three weeks, Priya slowly starts counting a 15-year-old client's loud sighs as "verbal protest," even though the definition covers only yelling or saying "no" to an instruction. Protest seems to rise, and the team considers a plan change that is not needed.
- Fidelity slipping. The plan for 8-year-old Leo says to provide no attention for whining and to give praise and access to a toy when he asks calmly. At home, his RBT starts saying "I know, buddy, almost done" whenever Leo whines. Whining stays high, and the graph makes the plan look ineffective when it was not fully run.
- Non-example. A fire drill clears the clinic for 10 minutes. Ana, an RBT, marks that interval "not observed" and tells her supervisor. The record has a gap, but it is not unreliable: nothing in it is false.
The fix starts with the RBT: record in real time, check the definition, follow the written steps, and tell the supervisor what happened. In a fidelity check, your supervisor watches you run the plan and scores each step; the score is steps done as written ÷ steps scored × 100, and a skipped step counts as an error. Treat it as support.
Common mistakes / exam traps
- "Improving" the plan on your own. Even a change you believe is helpful lowers procedural fidelity. Suggest it to your supervisor instead.
- Filling in or rounding data after the session. Estimates feel close but are not measurement. Writing down data that you did not actually observe is also an ethics issue, not just a quality issue. Report missed data as missed.
- Hiding mistakes. A missed step reported honestly is useful information; a hidden one leads to decisions based on a false picture.
Exam trap
Reliability vs accuracy. High IOA means two observers agree with each other; accuracy means the record matches what really happened. Two observers trained on the same unclear definition can agree 100% and both be wrong.
Exam trap
Unreliable data vs poor procedural fidelity. Ask what went wrong: the record (counted late, guessed, drifting definition) or the plan as delivered (steps skipped, extra reinforcement added). Either way, record what actually happened and tell your supervisor; never quietly fix the data.
Quick check
Omar's data show his hand-raising has not improved in four weeks. The supervisor then watches a session and sees that his RBT often praises him for calling out instead of raising his hand. What is the main risk of this low procedural fidelity?
Related topics
- Continuous measurement: frequency, rate, duration, latency, and IRT (A.1)
- Recording session data and keeping graphs up to date (A.4)
- Writing observable, measurable descriptions of behavior (A.5)
- Spotting trends in graphed data (A.7)
- Seeking clinical direction from your supervisor (E.2)
- Describing sessions objectively in session notes (E.4)