Skip to main content
v2026.11,986 entries · CC-BY 4.0

GLUE Benchmark: A Guide to General Language Understanding Evaluation

Understand the GLUE benchmark, the pivotal NLP dataset suite that accelerated the development of general-purpose language models.

Written and maintained by CASRAI Editorial Board

Published · Last updated

On this page

GLUE Benchmark: A Guide to General Language Understanding Evaluation

The General Language Understanding Evaluation (GLUE) benchmark is a foundational suite of tools and datasets designed to train, evaluate, and compare Natural Language Processing (NLP) models. Introduced in 2018, it shifted AI research focus from building narrow, single-task models to developing general-purpose language models that can understand language across a wide variety of contexts.

What it is used for in research

GLUE provides a standardized framework to measure how well an AI can process language flexibly. It offers a single-number score and a public leaderboard, allowing researchers to objectively rank model performance on various natural language understanding (NLU) tasks.

How to access the GLUE benchmark

Researchers can access the benchmark and its datasets through the official GLUE Benchmark website, which hosts instructions, data downloads, and the leaderboard.

The GLUE Benchmark Tasks

Task Category Task Name Description
Single-Sentence CoLA Corpus of Linguistic Acceptability: Tests grammatical correctness.
Single-Sentence SST-2 Stanford Sentiment Treebank: Sentiment analysis for positive/negative classification.
Similarity MRPC Microsoft Research Paraphrase Corpus: Paraphrase detection.
Similarity QQP Quora Question Pairs: Semantic equivalence detection.
Similarity STS-B Semantic Textual Similarity Benchmark: Degree of semantic similarity.
Inference MNLI Multi-Genre Natural Language Inference: Entailment, contradiction, neutral classification.
Inference QNLI Question-answering NLI: Determining context relevance.
Inference RTE Recognizing Textual Entailment: Small-scale entailment task.
Inference WNLI Winograd NLI: Pronoun resolution task.

Impact on AI research

GLUE played a massive role in the “Transformer revolution.” Performance on this benchmark proved the incredible power of architectures like BERT and RoBERTa, showing they could outperform non-expert human baselines on general understanding tasks.

Evolution to SuperGLUE

Because modern LLMs quickly saturated the original GLUE benchmark, the research team launched SuperGLUE. SuperGLUE features a much harder set of tasks, including complex reasoning and reading comprehension, to challenge more advanced language models.

FAQ

Is GLUE still the standard for NLP?

While foundational, GLUE is now considered saturated. SuperGLUE and other domain-specific benchmarks are generally used for evaluating the latest generation of large language models.

Sources

Data: GLUE Benchmark Task Overview

This dataset provides a comprehensive overview of the nine tasks that constitute the General Language Understanding Evaluation (GLUE) benchmark, detailing the objective and category for each task.

GLUE Benchmark Task Overview
TaskCategoryObjective
CoLASingle-SentenceGrammatical correctness
SST-2Single-SentenceSentiment analysis
MNLIInferenceEntailment, contradiction, neutral classification

Sources

Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.

Follow CASRAI

Research-administration guidance, standards updates and independent tool reviews.

Ask CASRAI · free to try

Ask about GLUE Benchmark: A Guide to General Language Understanding Evaluation

2 free questions a day, no account.

An AI assistant for research: funder rules, compliance, methods and publishing. It links the sources behind its answers and labels answers that come from a web search.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays open to read, with no account and no card.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →