By Finn Toompuu · Founder, Qualidact · Published
Imagine a course with a 94% pass rate. It looks like success. But before that figure goes into a report, ask what it actually shows. The answer depends on what learners were assessed on and what you claim they can now do.
Three different things
Course completion shows that learners met the recorded completion requirements. In an online course, that might mean logging in, working through the material and reaching the end. Completion alone does not establish what they learned.
Assessment performance shows how learners performed on a particular set of tasks. Passing means they met the assessment’s pass criteria. What that result supports depends on the tasks, the scoring and the conditions under which the assessment took place.
Demonstrated competence requires evidence that learners can perform the relevant task to the required standard in appropriate conditions. An observed performance can provide such evidence, but the claim should remain proportionate to the range of tasks and conditions assessed.
An illustrative example
Consider a troubleshooting course for equipment technicians, assessed entirely through factual recall: component names, fault codes, the correct order of a standard procedure. Learners score well.
Those results provide evidence of the knowledge sampled by the test. They do not establish that learners can work out why a system is failing when symptoms are ambiguous, choose between competing explanations or justify a decision under time pressure. Recognising a correct answer, explaining a decision and performing a task provide different kinds of evidence. If the course claims to build troubleshooting capability, this assessment supports only part of that claim.
Where multiple-choice fits
Multiple-choice questions can be useful tools. They are efficient to score and can check foundational knowledge and understanding of rules or procedures. Well-written scenario-based items can also probe judgement. The issue is the fit between the tasks and the claim. Where the claim includes practical performance, direct evidence may be needed: a simulation, an observed task or a work-based sample. A structured explanation of reasoning can add evidence about how a learner reaches a decision.
Three questions to ask
- What capability are we claiming? Write it as an observable action, not a topic. “Understands fault diagnosis” is vague; “diagnoses a common fault using the standard process” is testable.
- Could someone pass without being able to do this? If a learner could succeed through memorisation, guessing or recognition alone, passing does not establish that they can perform the claimed task.
- What evidence would change our confidence? Identify what you would need to see to make the intended decision with justified confidence, then check whether the current assessment provides it. The evidence required should reflect the consequences of getting that decision wrong.
Why it matters
Reported results inform decisions about who is deployed, who is signed off and where further training is needed. Confidence in those decisions should rest on what the evidence can bear.
Further reading: AERA, APA & NCME (2014), Standards for Educational and Psychological Testing, Chapter 1: Validity.