You would never keep a client in a program without looking at the data. Supervision deserves the same treatment. Your observations, feedback, and training sessions are an intervention, and the only way to know whether that intervention works is to measure its effects.
Why it matters
Supervision exists to produce two things: skilled supervisees and better outcomes for the clients they serve. Logging hours and holding meetings does not guarantee either. A supervisor who never checks may keep using a feedback style that confuses staff, or may miss that one RBT's sessions have drifted far from the written plan.
The ethics code for behavior analysts treats this as a standing duty: keep checking whether your supervision helps, write down what you find, and change course promptly when the data call for it. Practice items on this task ask which measure best shows that supervision is working, or what a supervisor should do after reviewing evaluation data.
Key ideas
- Supervision is the independent variable. Your supervision practices are what you change. Supervisee behavior and client progress are the dependent variables you measure.
- Measure at more than one level.
- Supervisee performance: direct observation of the skills you trained, such as procedural integrity on a checklist, accurate data collection, or competency checks after behavioral skills training.
- Client outcomes: progress on goals, rate of skill acquisition, and changes in problem behavior for the clients your supervisees serve.
- Social validity: supervisee, client, and caregiver views on whether supervision is acceptable, useful, and respectful.
- Prefer direct measures. A supervisee saying "I feel confident" is useful information, but watching them implement the procedure accurately is stronger evidence of skill.
- Collect baseline and repeated measures. Measure supervisee performance before you change your approach, then keep measuring. Without a reference point, you cannot tell whether your change helped.
- Monitor your own behavior too. Track whether you delivered supervision as planned: observations completed, feedback given promptly, goals reviewed. This is the fidelity of your supervision.
- Ask for feedback, and make it safe to give. Supervisees may hesitate to criticize the person who signs their hours. Anonymous surveys or a neutral third party can help.
- Close the loop. Evaluation only matters if it changes something. Use the data to keep, modify, or replace supervision practices, then measure again.
- Check maintenance and generalization. A skill that appears only when you are in the room has not been fully trained. Probe performance later and with other clients or settings.
Definitions
- Supervision evaluation: the ongoing, data-based review of whether supervision practices are improving supervisee performance and client outcomes.
- Supervisee performance measures: direct measures of the skills a supervisee is expected to show, such as integrity scores or competency checklists.
- Client outcome measures: data on client progress that reflect, in part, the quality of the services supervisees deliver.
- Social validity: the extent to which the goals, procedures, and results of supervision are seen as acceptable and important by the people involved.
- Self-monitoring: a supervisor observing and recording their own supervision behavior to check that it happens as planned.
Examples from a session
Three supervisors, three data sources
- Supervisee performance: Lena, a BCBA at a clinic, supervises four RBTs running discrete trial teaching. Integrity checks average about 70%. She switches from verbal feedback alone to behavioral skills training with modeling and rehearsal. Over the next month, integrity rises above 90% for all four, and she keeps graphing it.
- Client outcomes: Omar supervises home-based staff for several families. After he adds a monthly video review of each staff member's sessions, he compares the share of clients' short-term goals met per quarter before and after the change. More goals are met, which supports keeping the reviews.
- Social validity: Grace consults with paraprofessionals at a middle school. Her anonymous end-of-term survey says her feedback is hard to act on. She starts ending each observation with one written, specific next step, checks that the step appears in her notes, and surveys again next term.
Notice that each supervisor treats the data as feedback on their own behavior, not only on the supervisee's.
Non-example
Rosa, a BCBA, notes that every RBT she supervises received the required supervision hours this year and calls her supervision effective. Hours show how much supervision happened, not what it changed. She has no data on supervisee skills or client progress.
Common mistakes / exam traps
- Counting hours as outcomes. Meeting the required supervision hours shows how much supervision happened, not whether it worked.
- Relying only on satisfaction surveys. Supervisees can enjoy supervision that does not change their performance. Pair social validity data with direct measures.
- Relying on supervisee self-report of skill. Confidence and competence are different. Observe the skill.
- Evaluating only at the end. A single end-of-year review is too late to fix problems. Measure repeatedly.
- Blaming the supervisee by default. When performance stays low, check your own supervision and the work environment, for example with a performance diagnostic, before concluding the person cannot learn the skill.
- Ignoring client data. Improved supervisee integrity is good news, but if clients still are not progressing, the treatment plan itself may need review.
- Changing everything at once. If you add modeling, new checklists, and extra meetings together, you will not know which change helped.
Exam trap
Direct measures vs. indirect measures. The discriminating feature: a direct measure records what the supervisee does with clients or what clients achieve. Hours logged, quiz scores, and satisfaction ratings are indirect, so they rarely answer "did supervision work?"
Exam trap
Supervisee level vs. client level. High, reliable integrity with flat client progress does not make supervision a failure. The discriminating feature: integrity shows the supervision effect; flat client data then point to the treatment plan.
Quick check
In one week, Theo, a BCBA, adds modeling, a new integrity checklist, and an extra weekly meeting to his supervision. His RBTs' integrity on a token economy rises from 70% to 92%. What can he conclude?
Related topics
- Setting supervision goals from an assessment of the supervisee (I.4)
- Evidence-based, culturally responsive performance management (I.5)
- Function-based performance diagnostics for staff (I.6)
- Measuring procedural integrity: accuracy, dosage, and practical limits (C.12)
- Data-based decisions about effectiveness and when to modify (H.7)