Best-Worst Scaling: Case 1, 2, and 3 Designs

Best-worst scaling (BWS) is a survey method that asks respondents to identify the best and worst item in a small set, rather than rating every item on a scale. It was developed by Jordan Louviere and colleagues beginning in the late 1980s and formalized into a three-case typology by Louviere, Flynn, and Marley, whose 2015 book Best-Worst Scaling: Theory, Methods and Applications (Cambridge University Press) is still the field’s reference text. The three cases — object, profile, and multi-profile — look similar on the survey page but measure fundamentally different things and call for different analysis. This guide distinguishes the three, works through a Case 1 choice task with a fully computed example, and covers the move from simple counting to formal choice modeling.

What best-worst scaling measures, and why researchers use it

Every BWS task shows a respondent a small subset of items (usually 3–5) and asks two questions: which one is best (most important, most preferred, most agreed-with) and which one is worst. Forcing a choice at both ends, rather than allowing every item a high rating, avoids the scale-use problems that plague Likert-type batteries — acquiescence bias, central tendency, and the ceiling effect where everything gets rated “important.” See Likert Scale Survey Design & Statistical Analysis for how those problems show up in rating-scale data specifically. Because BWS produces a ranking-consistent choice on every task, it yields an interval-level preference scale from ordinal choice data — useful context: Levels of Measurement: Nominal, Ordinal, Interval and Ratio.

What varies across the three cases is what’s in the set being judged: single unstructured items, attribute levels drawn from one profile, or complete multi-attribute profiles. Get this wrong and the resulting scores answer a different question than the one you meant to ask.

Case 1: the object case

Case 1 (also just called “BWS” or MaxDiff in casual usage, though MaxDiff proper is closer to Case 2) presents a set of standalone objects — statements, brand names, symptoms, policy priorities, service features described as single items — with no internal attribute structure. The respondent picks the best and worst of that set on a single criterion (importance, agreement, preference).

• What it measures: a one-dimensional priority or importance ordering across a list of discrete things.
• Typical use: prioritizing training topics, ranking service-quality attributes, scoring agreement with a bank of statements, identifying which of many symptoms most affects quality of life.
• Design: items are assigned to blocks (subsets shown together) using a balanced incomplete block design (BIBD), so every item appears the same number of times and, ideally, every pair of items co-occurs an equal number of times.
• Analysis: counting-based scores (below) are often sufficient; multinomial logit or hierarchical Bayes modeling is optional, used when individual-level scores or covariate effects are needed.

Case 2: the profile case

Case 2 presents a single multi-attribute profile — one product, one job posting, one policy proposal — and asks the respondent to pick the best and worst attribute level within that one profile (e.g., for one described job: “remote work,” “$65k salary,” “4 weeks PTO,” “flexible hours” — which one matters most, which least). This is the design most commercial researchers mean when they say “MaxDiff.”

• What it measures: the relative importance of attribute levels belonging to a single profile, not preference between competing profiles.
• Typical use: feature-importance studies, message testing (which claim within one ad matters most), employee-benefits trade-off studies.
• Design: orthogonal main-effects plans or D-efficient designs generate the attribute-level sets shown per task, similar to conjoint design methods.
• Analysis: typically modeled (conditional logit or hierarchical Bayes) rather than simply counted, because levels belong to different attributes and aren’t directly comparable by raw frequency alone.

Case 3: the multi-profile case

Case 3 shows several complete multi-attribute profiles side by side — three or more full product configurations, job offers, or treatment options — and asks which whole profile is best and which is worst. This extends a standard discrete choice experiment (DCE), which normally asks only for the single best choice, by adding a worst pick to the same task.

• What it measures: preference between complete alternatives, the same underlying construct a DCE measures, with an added worst-choice signal per task that increases the information collected from each respondent.
• Typical use: anywhere you would already run a discrete choice experiment or choice-based conjoint study and want more preference information per task without adding tasks. See Discrete Choice Experiments: Attributes, Levels, and Experimental Design for the design and analysis mechanics Case 3 builds on directly.
• Design: standard DCE design approaches (fractional factorial, D-efficient) apply unchanged — Case 3 is a DCE with an extra response captured per task.
• Analysis: conditional logit, mixed logit, or hierarchical Bayes, identical to conventional choice modeling, typically estimated jointly across the best and worst responses.

Choosing the right case for your question

Worked example: a Case 1 choice-task walkthrough

Illustrative example. The scenario, items, and respondent data below are a synthesized, illustrative simulation built to demonstrate the mechanics of Case 1 design and counting analysis — not a real study, and not attributed to any real institution or dataset. All figures were generated by a seeded, reproducible simulation script and tallied programmatically, not hand-typed.

Suppose a research office wants to prioritize five candidate topics for a data-management training series:

• A — Data management plans
• B — File naming conventions
• C — Metadata standards
• D — Data sharing repositories
• E — Backup and version control

With 5 items, a balanced incomplete block design showing 3 items per task uses every possible 3-item subset of the 5: C(5,3) = 10 blocks, with each item appearing in C(4,2) = 6 blocks. That’s a complete, naturally balanced design at this small a scale — larger item sets use a BIBD table or generator rather than all subsets, since the number of subsets grows fast.

In this illustrative simulation, 40 synthetic respondents each completed all 10 blocks (400 choice tasks total), with best/worst picks drawn from a standard random-utility simulation (true item utility plus random noise, best = highest realized utility in the block, worst = lowest) — the same generative logic that later justifies fitting a logit model to real data of this shape. Tallying best and worst counts per item across all tasks:

Two counting statistics come straight out of that tally, no modeling required:

• B−W score: best count minus worst count for each item. Dividing by the number of times the item appeared (its standardized score) makes items comparable even when a design doesn’t show every item equally often.
• Square-root counting score: √(best% ÷ worst%), an alternative that some BWS practitioners prefer because it approximates the ratio-scale property of formally modeled utilities more closely than the linear B−W difference. It’s undefined when worst% is zero, as it is for Item A here in a smaller sample — a modeling approach (below) doesn’t have that edge case.

Reading the standardized scores: Item A (data management plans) is the clear priority — chosen best in 62% of the tasks it appeared in, worst in only 3%. Item E (backup and version control) sits at the opposite end. Item D (data sharing repositories) sits near zero, meaning respondents split roughly evenly on whether it was the best or worst item in the sets it appeared in — genuinely middling, not simply unmeasured.

From counting to modeling

Counting scores are a legitimate, defensible endpoint for Case 1 studies where the goal is a simple, transparent priority ranking — they require no statistical software and are easy to explain to a non-technical stakeholder. They fall short when you need standard errors, individual-level (rather than sample-average) scores, or the effect of a respondent covariate (e.g., does priority differ by career stage).

The modeling alternative treats each best/worst pick as two observations from a multinomial logit process operating on the same underlying utility scale: the “best” pick is a standard discrete choice among the items in the block, and the “worst” pick is modeled as a choice among the remaining items after removing the item chosen best (the “worst” pick, formally, is the “best” choice of which item is least preferred). Fitting this jointly with maximum likelihood produces the same ordering as the counting scores in well-behaved data, plus proper confidence intervals. Where preference heterogeneity across respondents matters, hierarchical Bayes (HB) estimation produces individual-level utility scores per respondent rather than one pooled sample score — the standard approach in commercial MaxDiff (Case 2) work, and increasingly used for Case 1 and Case 3 studies too when segment-level differences are the actual research question.

Designing the block structure

A few practical constraints apply across all three cases:

• Block size: 3–5 items per task is standard. Fewer than 3 removes the point of a best/worst forced choice; more than 5–6 raises respondent burden and encourages heuristics rather than genuine evaluation.
• Number of blocks: enough that every item appears a reasonable number of times (commonly 3–5 exposures per item at minimum for stable counting scores) and, for a proper BIBD, that every pair of items co-occurs an equal or near-equal number of times.
• Respondent burden: showing every respondent every block (as in the worked example above) is only practical for small item sets. Larger studies split the full block design across respondents so each sees a manageable subset, relying on the design’s balance property to keep the pooled counts comparable across items.
• Software: dedicated conjoint/MaxDiff platforms (e.g., Sawtooth Software, Qualtrics’ MaxDiff module) generate BIBDs and D-efficient designs automatically and handle HB estimation; R’s support.BWS and idefix packages cover open-source design generation and analysis for researchers who want to build the pipeline themselves.

Common mistakes

• Mixing cases in one instrument. Asking respondents to best/worst compare a mix of standalone statements and multi-attribute profiles in the same battery muddies exactly what’s being measured — pick one case and design the whole instrument around it.
• Treating raw counts as final without checking pair balance. If the block design doesn’t give every item roughly equal exposure and equal pairing with every other item, comparing raw best/worst counts across items is comparing unlike quantities — verify the design’s balance before trusting the scores.
• Skipping the worst pick. A “pick only your top choice” task from the same item sets is a different, weaker method (partial ranking) — the worst pick is what gives BWS most of its statistical efficiency per task relative to plain ranking or rating.

Frequently asked questions

Is best-worst scaling the same as MaxDiff?

“MaxDiff” is the common commercial-research name for best-worst scaling, most often used specifically for Case 2 (the profile case, judging attribute levels within one profile). The three-case framework (object, profile, multi-profile) is the more precise academic terminology and covers a broader set of designs than commercial MaxDiff software typically ships with.

How many items should I test in a Case 1 study?

There’s no hard ceiling — BWS scales to lists that would be unwieldy as a rating-scale battery, since each task only shows a small subset. What matters is designing enough blocks that every item gets adequate exposure (see “Designing the block structure” above), not limiting the total item pool.

Do I need special software to run a BIBD-based BWS study?

Not strictly. For small item sets, a balanced design can be constructed by hand or with the full-subset method used in the worked example above. For larger sets, a BIBD generator or a dedicated conjoint platform is the practical choice — hand-building loses balance quickly as item count grows.

Can Case 1 results be compared across two different studies?

Only cautiously. B−W and standardized scores are sample- and design-relative, not an absolute scale — comparing scores across two studies with different item sets, block designs, or respondent populations requires the same care as comparing raw scores across two different Likert instruments.

When should I use Case 3 instead of a standard discrete choice experiment?

When you’re already planning a DCE and want more information per task at low added respondent burden — adding a worst pick to an existing choice task is a small design change that increases the statistical information captured without adding more choice tasks to the survey.

Related CASRAI resources

• Discrete Choice Experiments: Attributes, Levels, and Experimental Design
• Likert Scale Survey Design & Statistical Analysis
• Questionnaire Design: Writing Survey Questions That Work
• Levels of Measurement: Nominal, Ordinal, Interval and Ratio
• Sampling Methods: Probability and Non-Probability Types
• Research Methods & Statistics