GSEA: How It Works and How to Read the Output

Who this is for: researchers with a ranked gene list (for example from an RNA-seq comparison) who need to run gene set enrichment analysis (GSEA) and read its output: NES, FDR q-value and leading-edge genes. Facts below were read from the sources listed at the end on 2026-11-07.

What GSEA is used for in research

GSEA (Gene Set Enrichment Analysis) tests whether an a priori defined set of genes shows statistically significant, concordant differences between two biological states. The method was introduced by Subramanian, Tamayo, Mootha and colleagues in PNAS 2005;102(43):15545-15550. The GSEA software and the Molecular Signatures Database (MSigDB) of annotated gene sets are run by the Broad Institute and UC San Diego and are NCI/NIH funded.

The key difference from a thresholded approach is that GSEA works on the whole ranked list. No gene is dropped for missing a p-value or fold-change cutoff, so a coordinated, modest shift across many genes in one pathway can be detected.

How the algorithm works

Genes are ranked by their correlation with the phenotype, using any suitable ranking metric.
An enrichment score (ES) is calculated for each gene set: the maximum deviation from zero of a running sum (a weighted Kolmogorov-Smirnov-like statistic) as you walk down the ranked list.
Significance is estimated by permuting phenotype labels (the original authors used 1,000 permutations).
The ES is normalized for gene set size to give the normalized enrichment score (NES), and a false discovery rate (FDR) is estimated by comparing observed NES values with the null distribution.

GSEA versus GO analysis (and other enrichment methods)

What is the difference between GSEA and GO analysis?

GSEA takes a full ranked list and asks whether a gene set is concentrated at the top or bottom of it. A typical Gene Ontology (GO) enrichment analysis takes a thresholded list of genes (for example, those called differentially expressed) and asks whether a GO term is over-represented in that list. This second approach is called over-representation analysis (ORA). The two answer different questions and can disagree; see the Enrichr guide for an ORA tool and differential gene expression analysis for how the input list is produced.

What is a gene ontology enrichment analysis?

It is an enrichment test in which the gene sets are Gene Ontology terms (biological process, molecular function, cellular component). The statistical approach can be ORA on a thresholded list or GSEA-style on a ranked list; the gene-set source and the test are separate choices. Other gene-set sources include pathway resources such as Reactome and the Kyoto Encyclopedia of Genes and Genomes (KEGG), and the MSigDB collections.

What is the difference between GSEA and ssGSEA?

Standard GSEA compares two phenotype groups and uses permutation of phenotype labels for significance. Single-sample GSEA (ssGSEA) scores each sample independently against a gene set, so no phenotype labels are required. In GSEApy, the gp.ssgsea output table has the columns Name, Term, ES and NES; setting permutation_num above 0 makes it behave like prerank.

Install or access (official sources only)

Broad GSEA and MSigDB: gsea-msigdb.org. Downloading the software, using the web tools and viewing MSigDB gene sets require free registration, which the project uses for funding reports. Documentation is at docs.gsea-msigdb.org, including a page on GSEA and RNA-seq.
fgsea (R/Bioconductor): an implementation of a fast gene set enrichment algorithm; version 1.38.0 on Bioconductor 3.23 when checked. Install:
```
if (!require("BiocManager", quietly = TRUE)) install.packages("BiocManager")
BiocManager::install("fgsea")
```
GSEApy (Python): documented at gseapy.readthedocs.io (v1.x; 1.3.0 shown in the example page).

Step-by-step: run a preranked GSEA

This route uses a ranked gene list you have already computed, which is the usual case for RNA-seq. The inputs are a ranking of genes (one score per gene, any suitable metric such as a differential-expression statistic) and a collection of gene sets.

GSEA in R with fgsea

Load the package: library(fgsea).
Prepare your gene sets as a list of pathways and your ranking as the stats input. The package vignette passes its example data as pathways = examplePathways and stats = exampleRanks; check the object structure against the vignette before using your own data.
Set a seed so permutation results are reproducible (the vignette uses set.seed(42)).

Run the test with a gene set size window:

fgseaRes <- fgsea(pathways = examplePathways,
                  stats    = exampleRanks,
                  minSize  = 15,
                  maxSize  = 500)

Sort by p-value to view the strongest results: head(fgseaRes[order(pval), ]).

The vignette is “Using fgsea package” on the fgsea Bioconductor page. To cite fgsea: Korotkevich G, Sukhov V, Golikov N, Sergushichev A, doi:10.18129/B9.bioc.fgsea.

GSEA in Python with GSEApy

GSEApy provides gp.prerank for a ranked list and gp.gsea for an expression matrix with phenotype labels.

gp.prerank takes rnk (the ranking), gene_sets, and parameters including threads, min_size=5, max_size=1000, permutation_num=1000, outdir=None and seed=6. The default method is ‘permutation’; ‘multilevel’ follows the fgsea-style approach.
gp.gsea takes data (a DataFrame or a .gct file), gene_sets, and cls (a .cls file), with permutation_type='phenotype', permutation_num=1000 and method='signal_to_noise'.
gene_sets accepts an Enrichr library name, a path to a .gmt file, or a Python dictionary.

What is a gmt file?

A .gmt file is a gene set file: each entry is a named gene set with its member genes. GSEApy accepts a path to one as gene_sets. MSigDB gene set collections are available from the Broad site after registration. The full file format specification is in the Broad Data Formats documentation.

Output and how to read it

GSEApy returns a results table (res2d) with these columns: NES, FDR q-val (Benjamini-Hochberg), NOM p-val, FWER p-val (permutation method only), log2err (multilevel method only), Lead_genes (semicolon-separated), Tag % and Gene %. The fgsea table has pathway, pval, padj, log2err, ES, NES, size and leadingEdge.

NES in GSEA

The normalized enrichment score is the enrichment score normalized for gene set size, so scores can be compared across gene sets of different sizes. The sign tells you which end of the ranked list the set is concentrated at. A multilevel-method NES in GSEApy is not comparable with a permutation-method NES, so do not mix the two in one comparison.

FDR q-value, nominal p-value and FWER

The FDR q-value estimates the proportion of false positives among results at or above a given NES, from observed versus null NES. The nominal p-value is the uncorrected permutation p-value for a single gene set; because many sets are tested at once, the FDR (padj in fgsea) is the figure to report. For the underlying logic of correcting for many tests, see Bonferroni correction and multiple comparisons.

What are the leading-edge genes in GSEA?

The leading-edge subset of a gene set is the group of its genes that appear in the ranked list at or before the point where the running sum reaches its maximum deviation from zero (Subramanian et al. 2005). These are the genes that account for the enrichment signal. In GSEApy they are in the Lead_genes column; in fgsea they are in the leadingEdge list column. Reading leading-edge genes across several significant sets shows whether those sets are driven by the same few genes.

Troubleshooting

Different results between runs: permutation p-values depend on random permutations. Fix the seed (GSEApy default seed=6; set one explicitly in R).
Expected gene sets missing: sets outside the size window are excluded. GSEApy defaults are min_size=5 and max_size=1000; the fgsea example uses minSize = 15 and maxSize = 500.
NES values differ between tools: check the method. Multilevel and permutation NES are not comparable in GSEApy.
No phenotype labels: use a preranked analysis, or ssGSEA for per-sample scores.

Free alternatives

Tool	Interface	Input options	Source
Broad GSEA	Desktop software and web tools; free registration	MSigDB gene sets	gsea-msigdb.org
fgsea	R (Bioconductor)	Pathways list plus ranked stats	bioconductor.org/packages/fgsea
GSEApy	Python	Ranked list (prerank), expression matrix with labels (gsea), single samples (ssgsea); Enrichr library, .gmt or dict	gseapy.readthedocs.io

FAQ

What does GSEA stand for?

Gene Set Enrichment Analysis: a method that tests whether a predefined gene set shows concordant differences between two biological states.

Do I need a significance cutoff before running GSEA?

No. GSEA uses the whole ranked list; the ranking can be any suitable metric of association with the phenotype.

Is registration required for the Broad software?

Yes, free registration is required to download the software, use the web tools or view MSigDB gene sets.

Can GSEA be run without phenotype labels?

Yes, with a preranked list or with ssGSEA, which scores each sample independently.

Sources (read 2026-11-07)

GSEA-MSigDB home: https://www.gsea-msigdb.org/gsea/index.jsp
Subramanian A, et al. Gene set enrichment analysis. PNAS 2005;102(43):15545-15550, PMID 16199517: PMC1239896
fgsea, Bioconductor: package page and vignette
GSEApy documentation: gseapy.readthedocs.io

GSEA (Gene Set Enrichment Analysis): How It Works and How to Read the Output