Rank Order Scale Examples and How to Score Them

Q: How do you score a rank order question?

Sum or average each item’s ranks across respondents. With raw ranks, the lowest mean is the most preferred; with SurveyMonkey’s reverse weights (5-4-3-2-1 for five items), the highest average is the most preferred.

Q: Can I use a rank order question in a scored quiz?

Not on Qualtrics: its documentation states that rank order questions are not compatible with scoring.

Q: How many items should a ranking question have?

SurveyMonkey recommends about 5 choices, although it allows up to 250. Qualtrics recommends drag and drop for shorter lists.

Q: Is a rank order scale the same as a rating scale?

No. A rating scale scores each item on its own, so two items can receive the same score. A rank order scale compares the items against each other. For other question formats, see the comparison of open-ended and closed-ended questions and the guide to quantitative survey question examples .

A rank order scale asks respondents to put a set of items in order, from most to least preferred, important or relevant. This guide shows working rank order scale examples (question wording, the formats survey platforms offer), then does the part most examples skip: scoring one small data set two ways and analysing it with a Friedman test, with every number computed on the page.

What is a rank order scale?

A rank order scale is a question format in which a respondent compares items against each other and assigns each a position: 1st, 2nd, 3rd and so on. Qualtrics describes its rank order question as one that gives respondents “the unique opportunity to rank a set of items against each other” and as a way “to determine each respondent’s order of preference for a list of items”. SurveyMonkey describes its Ranking question as asking respondents “to compare items to each other by placing them in order of preference.”

The data are ordinal: the ranks tell you the order of the items, not how far apart they are. A respondent who ranks Item A first and Item B second may like them almost equally or may strongly prefer A. This is the main difference from a rating scale such as a Likert-type item (see examples of ordinal survey questions), where each item is scored on its own.

Rank order scale examples (survey question wording)

The three wordings below are illustrative examples written for this guide, not taken from any named survey.

Preference ranking. “Rank the following features of a reference manager from most important (1) to least important (5): search speed, export formats, collaboration, cost, support.”
Priority ranking. “Rank these five lab-support services in the order you would want them improved, starting with 1 for the first priority.”
Criteria ranking. “Rank the factors that most influenced where you submitted your last manuscript, with 1 for the strongest influence.”

Formats platforms offer

Qualtrics lists four formats for its rank order question, and the choice affects how respondents answer:

Format	How the respondent answers
Drag and drop	Drags items into the preferred order. Qualtrics calls it “appropriate for shorter lists where you want respondents to rank each item.”
Radio buttons	Selects a rank for each item from columns of possible rankings.
Text box	Types the preferred ranking for each item.
Select box	Uses arrows to move items up or down the list.

SurveyMonkey allows up to 250 options in a Ranking question but advises you to “limit the number of ranking choices to about 5.” A short list is also easier for respondents to order consistently. Two other platform facts to know before you build the question: Qualtrics states that rank order questions “are not compatible with scoring”, and a rank order question the respondent does not interact with is recorded as unanswered.

Worked example: scoring a rank order question two ways

This is an illustrative data set, not a real survey. Ten respondents each ranked five features from 1 (most important) to 5 (least important). Every respondent used each rank exactly once.

Respondent	Search speed	Export formats	Collaboration	Cost	Support
1	1	3	2	5	4
2	2	1	3	5	4
3	1	2	4	5	3
4	3	1	2	4	5
5	1	4	2	5	3
6	2	3	1	4	5
7	1	2	3	5	4
8	4	1	2	5	3
9	2	1	3	4	5
10	1	3	2	4	5
Sum of ranks	18	21	24	46	41

Method 1: mean raw rank (lower is better)

Divide each column sum by the 10 respondents. Search speed: 18 / 10 = 1.8. Export formats: 21 / 10 = 2.1. Collaboration: 24 / 10 = 2.4. Cost: 46 / 10 = 4.6. Support: 41 / 10 = 4.1. Because rank 1 means most important, the smallest mean rank is the most preferred item.

Method 2: reverse-weighted average (higher is better)

SurveyMonkey’s average ranking reverses the weights. With five choices, a #1 ranking is worth 5, #2 is worth 4, down to #5, which is worth 1; the average is the weighted sum of response counts divided by the total response count, and “the answer choice with the largest average ranking is the most preferred choice.” SurveyMonkey states that you cannot change the default weights. For Search speed, the rank counts are 5, 3, 1, 1 and 0 for ranks 1 to 5:

(5 × 5 + 3 × 4 + 1 × 3 + 1 × 2 + 0 × 1) / 10 = (25 + 12 + 3 + 2 + 0) / 10 = 4.2

Applying the same arithmetic to every item gives 4.2, 3.9, 3.6, 1.4 and 1.9. With complete rankings the two methods are mirror images: reverse-weighted average = 6 minus mean raw rank (6 – 1.8 = 4.2). They agree on the order of the items; only the direction of “better” differs, which is why a report must say which one it uses.

Feature	Sum of ranks	Mean raw rank (lower = preferred)	Reverse-weighted average (higher = preferred)
Search speed	18	1.8	4.2
Export formats	21	2.1	3.9
Collaboration	24	2.4	3.6
Support	41	4.1	1.9
Cost	46	4.6	1.4

Interpretation: reading the results

What you see	How to read it
Mean raw rank near 1	Respondents placed the item near the top on average (Search speed, 1.8).
Mean raw rank near the number of items	Respondents placed the item near the bottom (Cost, 4.6 of 5).
Two items with close means	The order between them is weak; Search speed (1.8) and Export formats (2.1) differ by 0.3 of a rank.
Friedman test with a small p-value	The ranks differ across items more than chance ordering would produce; it does not say which items differ.

For the example data, the Friedman statistic is χ² = (12 / (10 × 5 × 6)) × (18² + 21² + 24² + 46² + 41²) – 3 × 10 × 6 = 0.04 × 5,138 – 180 = 25.52, with 5 – 1 = 4 degrees of freedom and p ≈ 0.00004. In R, friedman.test “performs a Friedman rank sum test with unreplicated blocked data”: when the data are a matrix, the columns are the groups and the rows are the blocks, so respondents are blocks and items are groups.

Assumptions and common mistakes

Ranks are ipsative. Every respondent’s ranks sum to the same total (here 1 + 2 + 3 + 4 + 5 = 15, so 150 across 10 respondents). One item rising in rank forces others to fall, so the items cannot be treated as independent measurements.
Ranks show order, not distance. Do not average them as if the gaps between positions were equal and then interpret differences in means as sizes of preference. If you need strength of preference, use a rating scale or a design such as best-worst scaling.
Too many items. SurveyMonkey advises about 5 choices; Qualtrics recommends drag and drop for shorter lists.
Item order effects. Pew Research Center notes that rotating or randomizing means items are not asked in the same order to each respondent; rotate the item list so the starting order does not favour one item.
Unanswered ranks. A skipped question is recorded as unanswered on Qualtrics; SurveyMonkey excludes N/A responses from the average. Decide how to treat partial rankings before you analyse.
Mixing the two scores. Report whether lower (mean raw rank) or higher (reverse-weighted average) means preferred.

How to report it (APA example)

Report the test statistic, degrees of freedom, sample size and p-value, then the descriptive ranks. For the example data:

“A Friedman test showed that the ranks of the five features differed, χ²(4, N = 10) = 25.52, p < .001. Mean ranks were lowest for search speed (M = 1.8) and highest for cost (M = 4.6), where 1 = most important.”

State the direction of the scale, the number of items and whether any respondents were excluded for incomplete rankings. For correlations between two sets of ranks, see how to calculate Spearman correlation.

How to calculate it in Excel, R or Python

Each snippet assumes the ranks sit in a table with one row per respondent and one column per item (Excel columns B to F, rows 2 to 11).

Excel

Mean raw rank:            =AVERAGE(B2:B11)
Reverse-weighted average: =6-AVERAGE(B2:B11)
Friedman p-value:         =CHISQ.DIST.RT(25.52, 4)

Compute the Friedman statistic itself from the column sums: =12/(10*5*6)*SUMSQ(B13:F13)-3*10*6, where row 13 holds the column sums.

R

ranks <- matrix(c(1,3,2,5,4, 2,1,3,5,4, 1,2,4,5,3, 3,1,2,4,5, 1,4,2,5,3,
                  2,3,1,4,5, 1,2,3,5,4, 4,1,2,5,3, 2,1,3,4,5, 1,3,2,4,5),
                nrow = 10, byrow = TRUE)
colMeans(ranks)          # mean raw rank per item
friedman.test(ranks)     # rows = respondents (blocks), columns = items

Python

import numpy as np
from scipy.stats import friedmanchisquare

ranks = np.array([[1,3,2,5,4],[2,1,3,5,4],[1,2,4,5,3],[3,1,2,4,5],[1,4,2,5,3],
                  [2,3,1,4,5],[1,2,3,5,4],[4,1,2,5,3],[2,1,3,4,5],[1,3,2,4,5]])
print(ranks.mean(axis=0))                        # mean raw rank per item
print(friedmanchisquare(*ranks.T))               # statistic 25.52, p about 0.00004

Frequently asked questions

How do you score a rank order question?

Sum or average each item’s ranks across respondents. With raw ranks, the lowest mean is the most preferred; with SurveyMonkey’s reverse weights (5-4-3-2-1 for five items), the highest average is the most preferred.

Can I use a rank order question in a scored quiz?