Tipping-Point Analysis for Missing Outcome Data

Tipping-point analysis stress-tests a study’s conclusion against its own missing outcome data: it systematically varies the assumed outcome for the subjects you never observed until the result flips, then reports exactly how extreme that assumption would have to be. Instead of picking one imputation method and hoping it is close enough to the truth, the analyst asks a sharper question — how far from the observed pattern would the missing data need to depart before the study’s headline finding stops holding up? If the answer is “wildly implausible,” the conclusion is robust. If the answer is “a value not that different from what a plausible dropout mechanism predicts,” the conclusion is fragile, and that fragility is exactly what a reviewer, funder, or regulator needs to see stated plainly rather than left inside a single complete-case p-value.

Why Missing Outcome Data Threatens a Conclusion in the First Place

Every analysis of incomplete data rests on an assumption about why the data are missing. The standard three-way split, going back to Rubin’s 1976 framework, is:

• Missing completely at random (MCAR) — missingness has nothing to do with any variable, observed or not. Complete-case analysis is unbiased here, but this is the least plausible assumption in a real trial.
• Missing at random (MAR) — missingness depends only on observed data (baseline covariates, earlier visit outcomes), not on the unobserved outcome itself once those observed variables are accounted for. Most standard multiple-imputation and mixed-model approaches assume this.
• Missing not at random (MNAR) — missingness depends on the unobserved outcome itself, even after conditioning on everything observed. A patient who stops attending visits because the treatment isn’t working is the textbook case, and it is common enough in real trials that regulators treat it as the default concern rather than the exception.

The problem is that MAR and MNAR are statistically indistinguishable from the observed data alone — no test tells you which one holds. A trial with meaningful attrition or a documented pattern of non-response that correlates with treatment arm cannot rule out MNAR on the data it has. That is precisely the gap tipping-point analysis is built to probe: not to prove MNAR is happening, but to show what the reader’s own conclusion would require to still be true if it were.

Why a Single Imputation Method Doesn’t Settle This

Multiple imputation, mixed models for repeated measures, and intention-to-treat handling of missing follow-up all give a single point estimate under one specific assumption about the missing-data mechanism — typically MAR. That is a reasonable primary analysis, and it is what most statistical analysis plans pre-specify. But a single number under a single assumption cannot, by itself, tell a reader how sensitive that number is to the assumption being wrong. Two trials can report an identical significant p-value from an identical primary analysis and differ enormously in how much a plausible MNAR departure would threaten that result — and nothing in the primary analysis reveals which trial that is. Sensitivity analysis exists to close that gap, and tipping-point analysis is the most direct, least assumption-laden form of it: rather than picking one alternative missing-data model and rerunning the analysis once, it sweeps a full range of assumed values and reports where the boundary sits.

How Tipping-Point Analysis Works, Step by Step

1. Start from the primary analysis — the complete-case or MAR-based result the study actually reports, including its significance threshold and direction of effect.
2. Identify the missing outcomes to stress-test — usually the arm(s) with the outcomes never observed due to dropout, non-response, loss to follow-up, or device/assay failure.
3. Define a plausible range of alternative values for those missing outcomes — for a binary outcome, this is a response rate from 0% to 100% among the missing subjects; for a continuous outcome, it is a shift (delta) applied to the value that would otherwise have been imputed under MAR.
4. Recompute the primary test statistic at each assumed value, holding everything else in the analysis fixed.
5. Find the tipping point — the specific assumed value at which the conclusion changes (typically where the p-value crosses the significance threshold, or where an effect estimate crosses zero or a pre-specified margin).
6. Judge plausibility, not just distance — the tipping point is only informative once it’s compared against what is clinically or mechanistically plausible for that dropout population. A tipping point that requires an implausibly large departure from the observed pattern supports the primary conclusion; one that sits close to what a known dropout mechanism would predict undermines it.

This is the same logic behind Rosenbaum-style sensitivity bounds in observational causal inference — instead of assuming away unmeasured confounding, you report how strong an unmeasured confounder would need to be to overturn the result. Tipping-point analysis for missing data applies that same “how extreme would it have to be” question to the missing-outcome problem specifically.

Worked Example: A Two-Arm Trial With Differential Dropout

The following example is an illustrative composite, not a real trial — every number below comes from a seeded, deterministic Node.js simulation written for this guide (full reproducibility details at the end of the article). It is designed to show a realistic pattern: dropout that is more common among non-responders, and more common still in the treatment arm — a classic MNAR mechanism (patients who don’t feel the treatment working are more likely to stop showing up).

A two-arm trial randomizes 150 patients per arm to treatment or control, with a binary “responded” outcome.

Arm	Randomized	Observed	Missing	Observed responders	Observed response rate
Control	150	140	10	48	34.29%
Treatment	150	128	22	61	47.66%

The complete-case analysis (a two-proportion z-test on observed cases only) shows a 13.37 percentage-point difference in response rate, z = 2.226, two-sided p = 0.0260 — a statistically significant result at the conventional 0.05 threshold, and the number that would typically headline the trial’s results.

The tipping-point sweep holds the 10 missing control-arm subjects at the observed control response rate (a neutral, MAR-style imputation) and varies k, the assumed number of responders among the 22 missing treatment-arm subjects, from 0 (none of them responded — the most pessimistic plausible assumption) to 22 (all of them responded — the most optimistic). The two-proportion z-test is recomputed at every step:

k (assumed responders among 22 missing)	Implied treatment response rate	Two-sided p-value
0 (worst case)	40.67%	0.2326
3	42.67%	0.1227
6	44.67%	0.0586
7 — tipping point	45.33%	0.0448
14	50.00%	0.0050
22 (best case)	55.33%	0.0002

The tipping point sits at k = 7. The trial’s significant complete-case conclusion holds up only if at least 7 of the 22 missing treatment-arm patients (31.8%) actually responded to treatment. Below that — six or fewer responders among the missing, a response rate under about 27% — the result no longer clears p < 0.05.

Whether that is reassuring or alarming depends entirely on plausibility, which is the whole point of running the analysis. A 31.8% response rate among the dropouts is not an extreme assumption on its face — it sits below the observed treatment-arm rate (47.66%) but close to the observed control-arm rate (34.29%). Given that dropout in this scenario is disproportionately concentrated among non-responders, a true response rate among the missing that falls below 31.8% is a genuinely plausible reading of the same mechanism that produced the dropout in the first place. That is the honest conclusion a tipping-point analysis is supposed to surface: this particular result, while nominally significant on complete cases, is only moderately robust to a dropout pattern the trial’s own data already hints at.

Extending to Both Arms: Tipping-Point Maps

The worked example above holds the control arm fixed and sweeps only the treatment arm, which is the simplest version and the easiest to report in a single sentence. The fuller method, formalized by Liublinska and Rubin (2014) for a partially missing binary outcome in a two-arm randomized trial, sweeps both arms’ missing-subject response rates simultaneously and plots the resulting significance boundary as a two-dimensional “tipping-point map” — a curve across the grid of (control missing-rate assumption, treatment missing-rate assumption) pairs along which the p-value crosses 0.05. Anywhere on one side of that curve the conclusion holds; anywhere on the other side it doesn’t. This is more informative than the single-arm sweep because it doesn’t force an arbitrary neutral assumption onto the arm you aren’t stress-testing — both arms’ missingness can be driven by the same MNAR mechanism at once, and a two-way map shows the full boundary rather than one slice of it.

Continuous Outcomes: Delta-Adjustment and Reference-Based Tipping Points

For a continuous outcome (a change-from-baseline score, a lab value, a survival time), the same logic applies with a delta-adjustment instead of a swept response rate: missing values are first imputed under a standard MAR-consistent model (e.g., multiple imputation or a mixed model for repeated measures), then a shift — delta — is added to those imputed values, and delta is swept until the treatment effect’s confidence interval crosses the null. White, Joseph, and Best (2020) formalized a causal modelling framework that connects this delta-adjustment approach to reference-based imputation methods (where the shift represents “behaving like the control arm after dropout” or similar clinically motivated benchmarks), which gives the swept delta a concrete real-world interpretation rather than an arbitrary number on a scale.

How to Report a Tipping-Point Analysis

• State the primary result and its assumption explicitly — the p-value or effect estimate the trial reports, and which missing-data mechanism (typically MAR) it assumes. Framing this against the trial’s pre-specified estimand makes the sensitivity analysis’s role unambiguous: it isn’t a second primary analysis, it’s a stress test of the first one.
• Report the tipping point in the same units the reader already understands — a response-rate percentage, a raw delta on the outcome scale, or a fraction of the missing subjects — not just an abstract sensitivity-parameter value.
• State explicitly how plausible that tipping value is, and on what basis — observed dropout-reason data, baseline characteristics of dropouts versus completers, or comparison against the difference in observed rates between arms, exactly as in the worked example above. A tipping point reported without a plausibility judgment is half the analysis.
• Don’t present it as proof either way. A tipping-point analysis cannot prove MNAR is or isn’t happening — the missing data alone never can. It can only quantify how much of a departure from the primary assumption the conclusion can absorb before it changes, which is a different and more honest claim than either “the result is definitely valid” or “the result is definitely biased.”

Common Mistakes

• Sweeping an implausibly narrow range. If the swept values never get near a value a real dropout mechanism could plausibly produce, the sensitivity analysis will always “pass” — which tells the reader nothing. The range needs to include values a skeptical reviewer would actually propose.
• Reporting only that the conclusion “held” or “didn’t hold,” without the tipping value itself. The number is the whole point; a binary pass/fail collapses the analysis back into exactly the single-imputation problem it exists to avoid.
• Treating the tipping point as an estimate of the true effect. It is a boundary condition on the conclusion, not a corrected effect size — conflating the two overstates what the analysis shows.
• Skipping a plausibility argument entirely. A tipping point of “10% of missing subjects would need to have failed” and one of “99% of missing subjects would need to have failed” both look like a single sentence in a manuscript, but they mean opposite things about how worried a reader should be. Without an explicit plausibility judgment, both get read the same way.
• Confusing this with a formal MNAR estimation method. Tipping-point analysis is a stress test of an existing conclusion, not a standalone model for the true missing-data mechanism — it deliberately avoids claiming to know the true mechanism, which is its strength for reporting and its limit as an estimation tool.

Frequently Asked Questions

Is tipping-point analysis the same as a sensitivity analysis?

Tipping-point analysis is one specific, widely used form of sensitivity analysis for missing outcome data — “sensitivity analysis” is the broader category, which also includes pattern-mixture models, selection models, and single-scenario delta-adjustment without a full sweep. What distinguishes tipping-point analysis is that it sweeps a full range of assumptions and reports the exact boundary value, rather than testing one alternative assumption at a time.

Do regulators expect to see tipping-point analysis?

Sensitivity analysis of missing-data assumptions is a standard expectation in confirmatory clinical trials, reflected in the ICH E9(R1) estimands and sensitivity analysis framework and discussed at length in the National Research Council’s 2010 report on missing data in clinical trials, which devotes a full chapter to sensitivity-analysis principles and methods. Tipping-point analysis specifically has become one of the more commonly used ways to satisfy that expectation because it is intuitive to present to a non-statistical reviewer — a single number (“the finding holds unless at least X% of the missing patients actually did Y”) communicates the robustness question directly.

Can tipping-point analysis be used outside clinical trials?

Yes — the same logic applies to any study with outcome data missing from a subset of a comparison, including survey research with differential non-response between groups and observational studies with differential loss to follow-up. The mechanics (sweep an assumed value for the missing subjects, find where the conclusion flips) don’t depend on randomization; only the plausibility judgment at the end changes, since observational studies often have less direct evidence about why data are missing.

What counts as a “reassuring” tipping point versus a “concerning” one?

There’s no universal cutoff — it’s a judgment call informed by what’s actually known about the missing subjects, not a fixed percentage. A tipping point that would require an assumption more extreme than anything suggested by the observed dropout-reason data, baseline comparisons of dropouts versus completers, or the difference already observed between arms, supports the conclusion. A tipping point that sits within the range those same sources suggest is plausible does not.

Reproducibility Note

Every number in the worked example above — the simulated trial data, the complete-case two-proportion z-test, and the full k = 0-to-22 tipping-point sweep — came from a self-contained, seeded Node.js simulation (a mulberry32 pseudo-random generator with a fixed integer seed, an explicit MNAR dropout mechanism weighting non-responders more heavily, and a standard pooled two-proportion z-test with a closed-form normal-CDF approximation for the p-values) rather than an assumed or hand-picked result. It is exactly reproducible from the same seed and is presented as an illustrative composite for teaching the method, not as data from any real trial.