Written and maintained by CASRAI Editorial Board
Last updated
High-dimensional data is data with a large number of variables (features, columns) relative to the number of observations. In the strict statistical sense it means p is greater than N: more variables than observations. In algorithm design it also means a dimension D large enough that distance-based and tree-based methods stop working well, even when N is large. This guide covers both meanings, the arithmetic that makes p greater than N a problem, and the standard remedies.
What is high-dimensional data? (definition)
Write N for the number of observations (rows) and p for the number of variables (columns). Data is high-dimensional in the statistical sense when p exceeds N. A table of 30 patients with 50 measurements each is high-dimensional; a table of 100,000 customers with 10,000 variables is wide, but it is not high-dimensional in this sense because N is still larger than p.
A second meaning comes from computing. The scikit-learn documentation says a KD tree is “very fast for low-dimensional (D < 20) neighbors searches” but “becomes inefficient as D grows very large”, and it calls this one manifestation of the “curse of dimensionality”. So a dataset can be high-dimensional for a nearest-neighbour search at D = 25 even with millions of rows. Which meaning applies depends on the method you intend to use.
High-dimensional data formula: p, N and the p/N ratio
There is no single formula; the working quantities are:
- N: number of observations (rows of the design matrix X).
- p: number of predictors (columns of X, not counting the intercept).
- p/N: the ratio of predictors to observations. Values above 1 mean high-dimensional in the statistical sense.
- rank(X): the number of linearly independent columns. It can never exceed the smaller of N and p+1 (with an intercept column).
Ordinary least squares estimates coefficients as (X’X)-1X’y. That inverse exists only when X’X has full rank, which needs rank(X) equal to the number of estimated parameters, p+1. Because rank(X) is at most N, you need N of at least p+1.
Worked example: why p = 50 and N = 30 breaks least squares
Suppose a study measures 50 variables on 30 samples, so X has 30 rows and 51 columns once an intercept is added.
- rank(X) is at most min(30, 51) = 30.
- X’X is a 51 × 51 matrix, but its rank equals rank(X), so it is at most 30. A 51 × 51 matrix of rank at most 30 is singular, so (X’X)-1 does not exist.
- There are 51 unknown coefficients and only 30 equations, so 51 – 30 = 21 or more free directions remain: infinitely many coefficient vectors fit the training data exactly.
- Any one of them gives zero training error, which says nothing about prediction on new samples.
Interpretation: what p and N tell you
| Situation | What it means for ordinary least squares |
|---|---|
| N at least p+1, columns independent | X’X is invertible; the fit is defined |
| N equal to p+1 | Exactly as many parameters as observations: the model interpolates the data |
| p greater than N | X’X is singular; infinitely many exact fits; regularization or dimension reduction is required |
| D below about 20 (nearest-neighbour search) | Tree searches cost about O[D log(N)] per scikit-learn |
| D above about 15 (scikit-learn algorithm=”auto”) | Brute force is chosen because the intrinsic dimensionality is “generally too high for tree-based methods” |
The glmnet documentation adds a classification example: logistic regression “is often plagued with degeneracies when p > N and exhibits wild behavior even when N is close to p”. Trouble therefore starts before p crosses N.
Why high-dimensional data breaks ordinary methods
Singular design matrix
As shown above, least squares has no unique solution once p+1 exceeds N.
Curse of dimensionality
Per scikit-learn, nearest-neighbour classification “becomes less effective” in high-dimensional parameter spaces, and tree-based neighbour search cost rises from roughly O[D log(N)] for small D to nearly O[DN] for larger D, at which point it can be slower than brute force. The practical effect is that adding variables can hurt rather than help.
How to handle high-dimensional data
Dimension reduction with PCA
Principal component analysis decomposes the data into successive orthogonal components that explain a maximum amount of variance; you keep the leading components and drop those with small singular values. Two details from the scikit-learn documentation matter: PCA centers but does not scale each feature, so standardize first if variables are on different units, and the number of components is bounded by the number of features. See our guide to PCA in R for the full workflow, and the comparison of UMAP and t-SNE for nonlinear projections used mainly for visualization.
Regularization
Penalized regression adds a constraint so a unique solution exists even when p exceeds N. In glmnet the mixing parameter alpha ranges from 0 to 1: alpha = 1 is the lasso (the default) and alpha = 0 is ridge, with values between giving the elastic net, which “regularizes and selects variables as well”. Read ridge versus lasso regression and the elastic net guide to choose between them.
Methods that tolerate many variables
Tree ensembles such as those in our random forests in R guide sample variables at each split, and distance-based clustering is often run after a reduction step; see what is k-means clustering for how the method uses distances.
Assumptions and common mistakes
- Judging by N alone. A large sample does not remove the problem; what matters is p relative to N. Check that p + 1 does not exceed N before fitting ordinary least squares, because beyond that point the design matrix is singular and there is no unique solution.
- Forgetting to standardize before PCA. Per the scikit-learn documentation, PCA centers each feature but does not scale it, so variables measured on larger scales dominate the components unless you standardize first.
- Assuming the default neighbour search stays fast. Tree-based neighbour search is efficient for small D but its cost approaches that of brute force as D grows, so adding variables can slow the method as well as weaken it.
- Expecting every penalty to select variables. In glmnet, alpha = 1 (the lasso) and alpha = 0 (ridge) behave differently; ridge shrinks coefficients without dropping variables, whereas the lasso and elastic net perform selection. Pick the mixing value deliberately rather than relying on the default.
- Treating a dimension-reduction step as free. The number of PCA components is bounded by the number of features, and you decide how many to keep; report that choice rather than leaving it implicit.
How to report high-dimensional data (APA example)
State N, p and how you handled p relative to N, so readers can judge the analysis. A sentence of the following form works: “The dataset contained 50 predictors and 30 observations (p > N); predictors were standardized and reduced with principal component analysis before modeling.” Cite a textbook for the method in author-date form. For a work with three or more authors APA 7th edition uses the first author plus “et al.” in every in-text citation: (Hastie et al., 2009). The matching reference-list entry follows the APA 7th book pattern:
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer.
How to calculate it in Excel, R or Python
Excel
Put the data in A2:AX31 (30 rows, 50 columns). The ratio is =COLUMNS(A2:AX31)/ROWS(A2:AX31), which returns 1.67; a value above 1 flags p greater than N. The largest possible rank is =MIN(ROWS(A2:AX31),COLUMNS(A2:AX31)+1), which returns 30.
R
set.seed(1)
X <- matrix(rnorm(30 * 50), nrow = 30, ncol = 50)
p <- ncol(X); N <- nrow(X)
p / N # 1.666667
qr(cbind(1, X))$rank # 30, less than 51 columns
pc <- prcomp(X, center = TRUE, scale. = TRUE)
length(pc$sdev) # 30: components cannot exceed min(N, p)Python
import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
rng = np.random.default_rng(1)
X = rng.normal(size=(30, 50))
N, p = X.shape
print(p / N) # 1.666...
print(np.linalg.matrix_rank(np.c_[np.ones(N), X])) # 30, below 51
Z = StandardScaler().fit_transform(X) # PCA centers but does not scale
pca = PCA(n_components=10).fit(Z)
print(pca.explained_variance_ratio_.sum())Frequently asked questions
Is high-dimensional data the same as big data?
No. Big data refers to volume, usually many observations. High-dimensional data refers to many variables relative to observations. A table with 10,000 variables and 100,000 rows is large but has p well below N, so least squares remains defined.
How many features is high dimensional?
There is no universal count. Statistically it is p greater than N, with trouble beginning as N approaches p. For nearest-neighbour search, scikit-learn treats about 20 dimensions as the point where tree methods lose their advantage and switches to brute force above 15 when the algorithm is set to auto.
Can I use ordinary least squares when p exceeds N?
Not for a unique solution: X’X is singular. Use regularization (ridge, lasso, elastic net) or reduce dimensions first.
Should I scale before PCA?
Usually yes when variables use different units, because scikit-learn’s PCA centers the data but does not scale each feature.
Sources
- scikit-learn documentation, “Nearest Neighbors”: https://scikit-learn.org/stable/modules/neighbors.html (checked 2026-10-29).
- scikit-learn documentation, “Decomposing signals in components (matrix factorization problems)”: https://scikit-learn.org/stable/modules/decomposition.html (checked 2026-10-29).
- glmnet vignette, “An Introduction to glmnet”: https://glmnet.stanford.edu/articles/glmnet.html (checked 2026-10-29).
- Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning (2nd ed.). Springer (textbook reference for the p much larger than N setting).
- APA reference patterns: Publication Manual of the APA, 7th ed., official reference examples, https://apastyle.apa.org/instructional-aids/reference-examples.pdf (checked 2026-10).
