Written and maintained by CASRAI Editorial Board
Published
On this page
NDCG@k (normalized discounted cumulative gain at k) scores a ranked list by looking only at its top k positions: each result’s graded relevance is divided by a logarithmic discount for its rank, the sum is called DCG@k, and that sum is divided by the DCG@k of the best possible ordering (IDCG@k). The result lies between 0 and 1, where 1 means the top k are in ideal order. The catch for research use is that libraries do not all use the same gain formula, so the same ranking can produce different numbers.
What NDCG@k is used for in research
NDCG@k is a ranking-quality metric for systems that return an ordered list: search engines, retrieval components of question-answering pipelines, recommender systems and learning-to-rank models. It is used instead of a yes/no metric when relevance has grades (for example 0 = irrelevant, 3 = perfect) and when position matters, because a relevant result at rank 1 should count for more than the same result at rank 10. The scikit-learn documentation for ndcg_score cites Järvelin and Kekäläinen (2002), ACM Transactions on Information Systems 20(4), as the reference for the metric, and the Keras RS and PyTorch Ignite documentation both define it as DCG@k divided by IDCG@k.
The “at k” matters as much as the “NDCG”. Only the first k positions of the list are scored, so NDCG@5 and NDCG@10 of the same system are different numbers and must never be mixed in one results table. When you report a result, the minimum a reader needs is k, the gain function, the discount function, how ties and empty relevance lists were handled, and the library version. Treat those as part of the method, in the same way you would record the software version for any computationally reproducible analysis.
The formula
The Keras RS documentation gives the per-list definition with a pluggable gain and discount:
- nDCG@k = DCG@k / IDCG@k
- DCG@k = the sum for i = 1 to k of gain(y’i) / rank_discount(i), where y’i is the true relevance of the item at rank i in the order induced by the predicted scores
- IDCG@k is the same sum over the ideal ranking, where the true labels are sorted from highest to lowest
- Keras RS defaults: gain = 2y − 1 and discount = 1 / log2(rank + 1)
The linear form used in many textbooks and on Wikipedia’s DCG page is DCGp = the sum of reli / log2(i + 1); the same page notes the alternative (2rel − 1) / log2(i + 1), and that the two coincide when relevance is binary. With graded relevance they differ, and that difference is the main reason numbers from different tools disagree.
How to access NDCG@k in scikit-learn, Keras RS and PyTorch Ignite
You do not need to implement the metric yourself. Three widely used Python libraries ship it, and each installs from its package index.
scikit-learn
The scikit-learn installation page gives pip install -U scikit-learn (and pip3 install -U scikit-learn on Linux), recommended inside a virtual environment, or the conda route conda create -n sklearn-env -c conda-forge scikit-learn followed by conda activate sklearn-env. The metric is sklearn.metrics.ndcg_score(y_true, y_score, *, k=None, sample_weight=None, ignore_ties=False). It returns a float between 0 and 1 averaged over samples, where each row of y_true and y_score is one query.
Keras RS
The Keras RS package is published on PyPI as keras-rs, so the install command is pip install keras-rs. The class is keras_rs.metrics.NDCG(k=None, gain_fn=..., rank_discount_fn=..., **kwargs), and its documentation shows it called directly on arrays of shape (batch_size, list_size), as in keras_rs.metrics.NDCG()(y_true=labels, y_pred=scores). With k=None the whole list is scored.
PyTorch Ignite
The Ignite package is published on PyPI as pytorch-ignite, so the install command is pip install pytorch-ignite. The class is ignite.metrics.rec_sys.NDCG(top_k, ignore_zero_hits=True, relevance_threshold=1.0, gain_function='exp_rank', ...). Unlike the other two, top_k is a sorted list of integers, so one pass returns NDCG at several cutoffs. Its update method takes (y_pred, y) tensors, both of shape (batch, num_items).
How to compute NDCG@k step by step
This section works one small example by hand and then shows the library calls. The ranking below is an illustrative composite invented for this guide, not data from a real system; every number was computed from the formula with a short script and checked against the scikit-learn documentation’s own examples.
Suppose a retrieval system returned five documents for one query, in this order, with human relevance grades of 2, 3, 0, 1 and 3 (grade 0 is irrelevant). The judged pool for the query also holds three documents that were not returned, graded 2, 1 and 0, so the full pool of eight judged documents carries the grades 3, 3, 2, 2, 1, 1, 0, 0.
- List the returned grades in rank order. Rank 1 to 5 carry grades 2, 3, 0, 1, 3.
- Compute the discount for each rank. With 1 / log2(rank + 1) the discounts are 1.000, 0.631, 0.500, 0.431 and 0.387.
- Choose a gain function. Linear gain uses the grade as is; exponential gain uses 2grade − 1 (Keras RS and Ignite defaults).
- Sum gain times discount over the first k ranks to get DCG@k. With linear gain and k = 3: 2 × 1.000 + 3 × 0.631 + 0 × 0.500 = 3.893.
- Sort the whole pool by grade, descending, and compute IDCG@k the same way. The ideal top 3 is 3, 3, 2: 3 × 1.000 + 3 × 0.631 + 2 × 0.500 = 5.893.
- Divide. NDCG@3 = 3.893 / 5.893 = 0.661 (linear gain).
Repeating the arithmetic for other cutoffs and for the exponential gain gives these results:
| k | DCG@k (linear) | IDCG@k (linear) | NDCG@k (linear) | NDCG@k (2grade − 1) |
|---|---|---|---|---|
| 1 | 2.000 | 3.000 | 0.667 | 0.429 |
| 3 | 3.893 | 5.893 | 0.661 | 0.574 |
| 5 | 5.484 | 7.141 | 0.768 | 0.723 |
Two things are visible. First, the exponential gain punishes the grade-2 document at rank 1 harder (0.429 against 0.667 at k = 1), because it widens the distance between grade 2 and grade 3. Second, NDCG@5 is higher than NDCG@3 here because the grade-3 document at rank 5 is credited at a discount of 0.387 while the ideal list’s tail is made of lower grades, so adding positions can raise the score as well as lower it.
The same calculation in code
scikit-learn takes one row per query. The example from its documentation, with relevance grades 10, 0, 0, 1, 5 and predicted scores that rank the grade-5 item first:
import numpy as np
from sklearn.metrics import ndcg_score
true_relevance = np.asarray([[10, 0, 0, 1, 5]])
scores = np.asarray([[.05, 1.1, 1., .5, .0]])
ndcg_score(true_relevance, scores) # about 0.49
ndcg_score(true_relevance, scores, k=4) # about 0.35
The same page reports that scoring the true relevances against themselves, ndcg_score(true_relevance, true_relevance, k=4), returns 1.0. In Keras RS the call is a direct function-style call on label and score arrays, and for lists of uneven length it accepts y_true={"labels": labels, "mask": mask}. In Ignite you create NDCG(top_k=[1, 2, 3, 4]), attach it to an evaluator with metric.attach(evaluator, "ndcg"), and read state.metrics["ndcg"], which comes back as a list in the order of top_k.
How to read the output
A value of 1.0 means that, within the top k, the system ordered the results exactly as the judgments would. A value near 0 means almost all of the gain was missed or placed low. Because the denominator is the ideal ordering for the same list, NDCG is comparable across queries that have different numbers of relevant documents, which raw DCG is not; that is the whole purpose of the normalization.
Three reading rules keep the number honest:
- Compare like with like. Same k, same gain function, same judged pool, same library semantics for ties and empty lists. The scikit-learn documentation notes that negative relevance values can give results outside the 0 to 1 range, so a value outside it is a data problem, not a very good system.
- Remember what the metric ignores. The Wikipedia DCG article lists limitations: a list of three relevant documents and the same list followed by an irrelevant one score the same on unnormalized DCG, and missing documents are not penalized unless lists have a fixed size. Recomputing it confirms the second point: with linear gain, [1, 1, 1] and [1, 1, 1, 0] both give an NDCG of 1.0 against the same ideal. Pair NDCG with a recall-style measure if missing relevant documents matter to your question.
- Know how the ideal list was built. The ideal ordering is built from the documents that were judged. When judgments are partial, a different pool makes a different IDCG, so a score is only as meaningful as the judged pool behind it.
The scikit-learn tie example shows how much the tie rule can move a number. With y_true = [[10, 0, 0, 1, 5]], scores [[1, 0, 0, 0, 1]] and k = 1, the default (ties averaged) gives 0.75, which is (10/10 + 5/10) / 2; with ignore_ties=True it gives 0.5. The documentation warns that ignore_ties=True gives wrong results if ties are present, so use it only when scores are known to be tie-free.
Troubleshooting NDCG@k
Different libraries give different numbers for the same ranking
Check the gain function first. Keras RS and Ignite default to 2rel − 1; Ignite offers gain_function='linear_rank' for the raw grade, and Keras RS accepts a custom gain_fn such as lambda y: y. The scikit-learn function page does not state its gain or log base. Recomputing its documented examples (0.49 at full length and 0.35 at k = 4) shows they are reproduced by linear gain with a base-2 discount. For binary labels the choice does not matter; for grades above 1 it does.
The score changes between runs
Keras RS defaults to shuffle_ties=True, which randomly shuffles scores before sorting to break ties, and takes a seed argument. If your model emits tied scores, set a seed and report it; the reproducible AI experiment checklist applies directly. If ties are rare, the variation should be negligible, and a large run-to-run swing is itself a sign the model’s scores are saturated.
A query with no relevant documents
When the ideal DCG is zero the ratio is undefined, and libraries choose differently. Keras RS defines nDCG@k as 0 when IDCG@k is 0. Ignite’s default ignore_zero_hits=True excludes such queries from the average, and ignore_zero_hits=False counts them as 0. In the Ignite documentation’s example, the same batch gives [0.0, 0.38, 0.38, 0.65] with the default and [0.0, 0.19, 0.19, 0.32] with ignore_zero_hits=False, because the second query has all-zero labels. Whichever rule you use, state it.
Labels below the relevance threshold count as nothing in Ignite
Ignite’s relevance_threshold defaults to 1.0: labels below it contribute 0 to DCG and IDCG. If your grading scale includes fractional labels below 1, lower the threshold deliberately instead of discovering later that those labels were dropped.
Shape errors
Ignite requires y_pred and y to both have shape (batch, num_items); Keras RS examples use (batch_size, list_size); scikit-learn expects one row per query. A flat vector for a single query needs to be reshaped to one row first.
Free alternatives to NDCG@k
Where the question is which tool to compute NDCG@k with, the three libraries above are the documented, pip-installable options. This table is about how you call each one; the defaults that decide whether numbers are comparable are in the data table further down the page.
| Library | Call | Input layout | How k is set |
|---|---|---|---|
| scikit-learn | sklearn.metrics.ndcg_score |
One row per query, y_true and y_score |
k=None (all) or an integer |
| Keras RS | keras_rs.metrics.NDCG |
(batch_size, list_size) labels and scores, optional mask | k=None (all) or an integer |
| PyTorch Ignite | ignite.metrics.rec_sys.NDCG |
(batch, num_items) tensors through an engine | top_k: a sorted list of integers |
If your question does not need graded relevance or a rank discount, a binary, unweighted measure may be enough, but choose it on the research question and report why; NDCG is not automatically better. For a wider view of how to document an evaluation setup, see the model evaluation suite entry, and for related software guides browse the research tools hub.
Sources
- scikit-learn: sklearn.metrics.ndcg_score, checked 2026-11-17
- scikit-learn: Installing scikit-learn, checked 2026-11-17
- Keras RS: nDCG metric, checked 2026-11-17
- Python Package Index: keras-rs, checked 2026-11-17
- PyTorch Ignite: ignite.metrics.rec_sys.NDCG, checked 2026-11-17
- Python Package Index: pytorch-ignite, checked 2026-11-17
- Wikipedia: Discounted cumulative gain, checked 2026-11-17
Data: NDCG@k defaults in scikit-learn, Keras RS and PyTorch Ignite
Compares how three Python libraries define NDCG@k: gain function, rank discount, tie handling and behaviour when a list has no relevant item. Compiled from each library's official documentation page; the scikit-learn gain and discount cell reports a recomputation of its documented examples, not a statement on the page.
| Library | Default gain | Rank discount | Tie handling | List with no relevant item |
|---|---|---|---|---|
| scikit-learn ndcg_score | Not stated on the function page; its documented examples are reproduced by linear gain | Not stated on the function page; examples match log base 2 | Ties averaged by default; ignore_ties=True is wrong if ties are present | Not stated on the function page |
| Keras RS NDCG | 2**y - 1 (gain_fn) | 1 / log2(rank + 1) (rank_discount_fn) | shuffle_ties=True randomly shuffles scores before sorting; seed argument | nDCG@k defined as 0 when IDCG@k is 0 |
| PyTorch Ignite NDCG | exp_rank: 2^relevance - 1 (linear_rank available) | 1 / log2(i + 1), fixed | Not stated on the class page | Excluded when ignore_zero_hits=True (default); counted as 0 when False |
Sources
- scikit-learn: sklearn.metrics.ndcg_score (accessed 2026-11-17)
- Keras RS: nDCG metric (accessed 2026-11-17)
- PyTorch Ignite: ignite.metrics.rec_sys.NDCG (accessed 2026-11-17)
Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.







