Written and maintained by CASRAI Editorial Board
Last updated
In medical device and Software as a Medical Device (SaMD) regulatory submissions, formative vs. summative usability testing defines the two sequential phases of empirical user evaluation mandated by the FDA Human Factors and Usability Engineering Guidance (2016) and international standard IEC 62366-1:2015+AMD1:2020 (supported by technical report IEC/TR 62366-2:2016). While both phases evaluate how intended users interact with a medical device’s User Interface (UI)—including hardware controls, visual/auditory alarms, software screens, packaging, on-device labels, Instructions for Use (IFU), and training—they serve fundamentally different engineering and regulatory functions.
What Is the Difference Between Formative Usability Evaluation and Summative Human Factors Validation Testing?
Under 21 CFR § 820.30 (Quality System Regulation / QMSR ISO 13485 alignment) and ISO 14971 medical device risk management, usability testing bridges design input refinement and final design validation:
- Formative Usability Evaluation (Diagnostic & Iterative — IEC 62366-1 Clause 5.7 / FDA Section 7): Conducted throughout the design and development phase before design freeze. Using early wireframes, 3D-printed enclosures, breadboard prototypes, or pre-freeze software builds, human factors engineers test small groups (typically 5 to 8 participants per iteration) to identify unanticipated use hazards, update the Use FMEA (uFMEA) and Use-Related Risk Analysis (URRA), compare competing UI layouts, and refine instructions. Moderators may use concurrent “think-aloud” protocols and intervene mid-task to explore design alternatives.
- Summative Human Factors Validation Testing (Confirmative & Regulatory — IEC 62366-1 Clause 5.9 / FDA Section 8): Conducted after UI design freeze as the capstone validation study included in an FDA 510(k), De Novo, or Premarket Approval (PMA) HFE/UE Report. The study must use a production-equivalent device, final labeling/IFU, realistic simulated-use environmental conditions (lighting, ambient noise, PPE/gloves, distractions), and a minimum of 15 representative U.S. participants per distinct user group performing every critical task without moderator assistance.
Why Does FDA Human Factors Guidance Require at Least 15 Representative Users per User Group in Summative Testing?
In Section 8.1.3 of Applying Human Factors and Usability Engineering to Medical Devices (February 2016), the FDA specifies that summative validation testing must include at least 15 representative U.S. participants for each distinct user group. This requirement is grounded in both binomial probability theory and Laura Faulkner’s landmark empirical study (Behavior Research Methods, Instruments, & Computers, 2003, 35(3):379–383):
If a specific critical-task use error has a true underlying probability $p$ of occurring in the intended user population, the probability $P_{text{detect}}$ of observing that use error at least once across $n$ independent test participants is:
P_detect = 1 - (1 - p)^n| Sample Size ($n$ Users per Group) | Detection Probability if $p = 0.20$ | Detection Probability if $p = 0.15$ | Detection Probability if $p = 0.10$ | Faulkner (2003) Empirical Mean % Problems Found (Min–Max Range) |
|---|---|---|---|---|
| $n = 5$ (Typical Formative Round) | $67.2%$ | $55.6%$ | $41.0%$ | Mean $85.6%$ (Range: $55%text{–}99%$) |
| $n = 10$ | $89.3%$ | $80.3%$ | $65.1%$ | Mean $94.7%$ (Range: $82%text{–}100%$) |
| $n = 15$ (FDA Summative Minimum) | $96.5%$ | $91.3%$ | $79.4%$ | Mean $97.1%$ (Range: $90%text{–}100%$) |
| $n = 20$ | $98.8%$ | $96.1%$ | $87.8%$ | Mean $98.4%$ (Range: $95%text{–}100%$) |
Rules for Counting User Groups ($15 times G$)
- Distinct Roles or Capabilities Require Separate 15-User Cohorts: You cannot pool 8 lay patients and 7 registered nurses to reach 15 total participants. If an home-use infusion pump is set up by a home-health nurse and operated daily by an adult patient or lay caregiver, nurses and lay users have distinct knowledge, training, and tasks—requiring at least 30 participants total (15 nurses + 15 lay users).
- Pediatric and Age/Impairment Stratification: If a device is indicated for both adolescent self-injection and elderly patients with rheumatoid arthritis (dexterity impairment) or diabetic retinopathy (visual impairment), each clinically distinct population constitutes a separate 15-user group.
- Multi-Person Collaborative Operation: When a device requires two distinct roles working together simultaneously (e.g., a sterile surgical-field scrub nurse and a non-sterile circulating nurse operating an electrosurgical console), FDA Section 8.1.3 requires 15 representative two-person teams (30 individuals).
Why Do IEC 62366-1 and the FDA Use the Term “Use Error” Rather Than “User Error”?
Both IEC 62366-1:2015+AMD1:2020 (Clause 3.21) and the FDA 2016 Human Factors Guidance (Section 3.12) define a use error as “a user action or lack of user action while using the medical device that leads to a different result than that intended by the manufacturer or expected by the user.” Regulatory reviewers routinely issue Additional Information (AI) deficiency letters when a submission attributes a summative test failure to “user error,” “participant carelessness,” or “failure to read the IFU.”
The distinction is foundational to human factors engineering:
- Perception–Cognition–Action (PCA) Root Cause Model: Human beings interact with a device by perceiving sensory inputs (display contrast, alarm pitch, syringe graduations), cognitively processing those inputs (interpreting units such as $text{mg/mL}$ vs. $text{mL/hr}$), and executing a physical action (pressing a softkey or twisting a luer lock). When an error occurs, its root cause lies in a mismatch between the UI design and human capabilities under clinical stress.
- Hierarchy of Risk Control (ISO 14971 & IEC 62366-1 Clause 4.1.2): Because a use error is classified as a design-induced hazard, manufacturers must mitigate it using the strict three-tier risk control priority order: (1) Inherent safety by design (e.g., physical mechanical keying so a tubing connector cannot physically fit into the wrong port, or forcing functions in SaMD), (2) Protective measures in the device itself (e.g., automated air-in-line occlusion sensors and audible/visual high-priority IEC 60601-1-8 alarms), and (3) Information for safety and training (warnings in the IFU or on-device labels—considered the weakest and least reliable control).
- Qualitative Root-Cause Evaluation Over Pass/Fail Percentage Cutoffs: Neither the FDA nor IEC 62366-1 permits pre-set numerical pass/fail success rate thresholds (such as “90% task success passes summative validation”). Even a single observed use error, close call (near miss recovered by the user before harm), or operational difficulty on a critical task among 15 users must be investigated during the post-task subjective debriefing to determine its UI root cause and verify whether the residual risk is acceptable in the ISO 14971 Risk Management File.
Frequently Asked Questions
What is the difference between formative usability evaluation and summative human factors validation testing?
Formative usability evaluation (IEC 62366-1 Clause 5.7; FDA 2016 Guidance Section 7) is conducted iteratively during medical device design using prototypes and small sample sizes (typically 5–8 users per round) with active moderator probing to discover unanticipated use errors and refine the user interface. Summative human factors validation testing (IEC 62366-1 Clause 5.9; FDA Section 8) is conducted after design freeze on the final production-equivalent device, labeling, and training program across at least 15 representative users per distinct user group without moderator assistance, providing objective validation evidence that all critical tasks can be performed safely and effectively.
Why does FDA Human Factors guidance require at least 15 representative users per user group in summative testing?
The FDA’s 15-participant-per-group requirement in Section 8.1.3 of its 2016 guidance is grounded in empirical human factors research by Laura Faulkner (2003) and binomial probability modeling: $P(text{detecting at least one occurrence}) = 1 – (1 – p)^n$. While testing $n = 5$ users detects an average of $85%$ of usability problems, random 5-user subsamples can detect as few as $55%$. Testing $n = 15$ representative users per distinct user group increases the mean detection rate to $97%$ (with a minimum of $90%$ across subsamples) and gives a $79.4%$ probability of observing a use error whose true population incidence is $p = 0.10$.
Why do IEC 62366-1 and the FDA use the term ‘use error’ rather than ‘user error’?
Both IEC 62366-1:2015+AMD1:2020 (Clause 3.21) and the FDA 2016 Human Factors guidance explicitly reject the phrase ‘user error’ because it implies human blame, negligence, or incompetence—which historically led manufacturers to dismiss test failures or rely solely on warning labels. The regulatory term ‘use error’ defines an action or omission during device use that yields an unintended result due to an interaction flaw between the user, the use environment, and the device User Interface (perceptual, cognitive, or physical overload), requiring engineering mitigation via inherent safety by design under ISO 14971.
Sources
- U.S. Food and Drug Administration (FDA). Applying Human Factors and Usability Engineering to Medical Devices: Guidance for Industry and Food and Drug Administration Staff (February 3, 2016).
- International Electrotechnical Commission (IEC). IEC 62366-1:2015+AMD1:2020 — Medical devices — Part 1: Application of usability engineering to medical devices.
- Faulkner L. Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments, & Computers (2003;35(3):379–383).
