Elastic Net Regression: The Alpha Parameter, Correlated Predictors, and Cross-Validation

Elastic net regression combines the two penalty terms used by ridge and LASSO regression into a single model, controlled by a mixing parameter usually called alpha (α). It exists because ridge and LASSO each solve part of the regularization problem and each have a specific, well-documented weakness — ridge shrinks correlated predictors together but never drops any of them, while LASSO drops predictors but does so unstably when they are correlated with each other. Elastic net was introduced by Zou and Hastie (2005) specifically to fix LASSO’s behavior on correlated predictors while keeping most of its ability to produce a sparse, interpretable model.

The penalty: what ridge and LASSO each add to ordinary least squares

Ordinary least squares regression minimizes the residual sum of squares with no penalty on the size of the coefficients. Ridge regression and LASSO both add a penalty term to that objective, multiplied by a tuning parameter λ (lambda) that controls how strongly the penalty is enforced:

• Ridge regression penalizes the sum of squared coefficients (an L2 penalty). This shrinks all coefficients toward zero — more for weakly-informative predictors, less for strongly-informative ones — but never shrinks a coefficient exactly to zero. Ridge keeps every predictor in the model.
• LASSO (Least Absolute Shrinkage and Selection Operator) penalizes the sum of the absolute values of the coefficients (an L1 penalty). Because of the geometry of that penalty, LASSO can shrink coefficients exactly to zero, which performs variable selection as part of fitting the model — a predictor with a zero coefficient is effectively dropped.

Both penalties trade a small amount of bias for a reduction in variance, which is the same bias-variance tradeoff behind every regularization method: an unpenalized model fit on limited or highly correlated data can have very low bias but wildly unstable coefficient estimates, and a modest, well-chosen penalty usually predicts better on new data even though no individual coefficient is exactly right.

The alpha mixing parameter

Elastic net’s penalty is a weighted combination of the ridge (L2) and LASSO (L1) penalties, and α is the weight:

penalty = α × (L1 penalty) + (1 − α) × (L2 penalty)

In this common parameterization (the one glmnet in R and scikit-learn’s ElasticNet in Python both use):

• α = 1 reduces the penalty to pure LASSO.
• α = 0 reduces the penalty to pure ridge.
• 0 < α < 1 is elastic net proper — a blend of both penalties in the proportion α specifies.

λ still controls the overall strength of whichever blended penalty α defines — λ = 0 recovers ordinary least squares regardless of α, and larger λ shrinks coefficients harder. Elastic net therefore has two hyperparameters to choose, not one, which is the direct cost of gaining the L1/L2 blend: α (how LASSO-like vs. ridge-like the penalty is) and λ (how strong the penalty is).

Why elastic net handles correlated predictors better than LASSO alone

LASSO’s specific weakness with correlated predictors is well documented and is the direct motivation for elastic net, not an incidental side benefit. When two or more predictors are highly correlated, LASSO’s L1 penalty tends to pick one of them somewhat arbitrarily and shrink the others’ coefficients to zero, rather than distributing the effect across the correlated group. Which one gets picked can depend on small perturbations in the sample — a different bootstrap resample of the same data can select a different member of the correlated group, which makes the “selected” model unstable and hard to interpret substantively (this is the same broad problem discussed on this site’s page on multicollinearity and VIF in regression, though the mechanism there is inflated standard errors in an unpenalized model rather than unstable selection in a penalized one). Zou and Hastie also showed that when the number of predictors p exceeds the number of observations n, LASSO can select at most n non-zero coefficients — a hard ceiling that has nothing to do with how many predictors actually matter.

Elastic net’s ridge component fixes both problems through what Zou and Hastie call the grouping effect: because the L2 penalty shrinks correlated predictors together rather than arbitrarily favoring one, elastic net tends to select or exclude a correlated group as a whole rather than picking one representative and dropping the rest. The practical result is a model whose coefficient pattern is more stable across resamples and more defensible substantively — “these three correlated measures of the same underlying construct all got small, similar coefficients” is usually a more honest description of the data than “this one measure mattered and the other two, which correlate with it at r > 0.9, did not.” Elastic net also removes the n-predictor selection ceiling that pure LASSO has in high-dimensional (p > n) settings, which is one of the two problems the original elastic net paper was written to solve.

Cross-validation for selecting lambda and alpha

Neither λ nor α is estimated by the model fit itself — both are tuned by cross-validation, the same way a single LASSO or ridge λ would be, just over a larger search space because there are now two dimensions instead of one.

The standard approach: a grid search over alpha, with k-fold CV for lambda at each alpha

The typical workflow, and the one both glmnet and scikit-learn’s ElasticNetCV implement directly:

1. Fix a small grid of candidate α values (e.g. 0.1, 0.3, 0.5, 0.7, 0.9, or a finer grid if compute allows).
2. For each candidate α, run k-fold cross-validation (10-fold is a common default) across a path of candidate λ values, and record the cross-validated prediction error (mean squared error for a continuous outcome, deviance for a generalized linear model) at each λ.
3. Within each α, pick λ either at the minimum cross-validated error (lambda.min in glmnet‘s terminology) or at the largest λ whose error is within one standard error of that minimum (lambda.1se) — the “one-standard-error rule” deliberately favors a simpler, more heavily-penalized model when the extra complexity of the minimum-error model isn’t earning a meaningfully better score. This matters more for elastic net than it does for a single-penalty model, since a slightly-larger λ at a given α can meaningfully change which correlated-group members survive.
4. Compare the best cross-validated error achieved at each α, and select the (α, λ) pair with the best overall score, not just the best λ at a single, guessed α.

Why this needs to be nested, not a single flat cross-validation

If the same cross-validation folds used to pick α and λ are then reused to report the model’s performance, the reported error is optimistic — the hyperparameters were chosen specifically because they performed well on those folds, so evaluating on the same folds double-dips. The honest version is nested cross-validation: an outer loop holds out a test fold that never participates in any hyperparameter selection, and an inner loop (the grid search described above) tunes α and λ using only the remaining data, repeated across each outer fold. The outer-loop performance, averaged across folds, is the number that should be reported as the model’s expected performance on new data; the inner-loop selection is only for choosing the hyperparameters to use in the final model, which is then typically refit on the full dataset at the selected (α, λ) once tuning is done. Reporting only the inner-loop (tuning) cross-validated error as if it were an honest estimate of out-of-sample performance is a common and specific mistake with penalized regression — the tuning process searched over enough (α, λ) combinations that some combination is likely to look good on any particular set of folds by chance alone.

Practical notes

• Standardize predictors before fitting. Both the L1 and L2 penalties are scale-dependent — a predictor measured in larger units gets penalized more heavily purely because of its scale, not its actual relationship to the outcome. glmnet standardizes internally by default and reports coefficients back on the original scale; scikit-learn’s ElasticNet does not standardize automatically and expects the caller to do it (e.g. with a StandardScaler in a pipeline, fit only on the training fold to avoid leakage).
• Report the selected α and λ, not just the final coefficients. A reviewer or reader cannot evaluate how much shrinkage and how much LASSO-vs-ridge blending the reported model reflects without both numbers, and re-running the same procedure without them isn’t reproducible.
• A grid search over α and λ jointly is more compute-intensive than tuning a single λ, but the search is embarrassingly parallel across folds and α values, and both major implementations (glmnet::cv.glmnet, scikit-learn’s ElasticNetCV) handle the λ path efficiently via warm starts, so the added cost is usually modest relative to the stability gained.

When elastic net is (and isn’t) the right choice

Elastic net is a reasonable default when predictors are numerous, at least moderately correlated, and a sparse, interpretable model is desirable — the combination LASSO alone struggles with. It is less necessary when predictors are close to uncorrelated (plain LASSO’s grouping problem doesn’t arise) or when there is no reason to want variable selection at all and ridge’s “shrink everything, drop nothing” behavior is actually preferred, such as when every candidate predictor has an established theoretical basis for inclusion. It is not a substitute for stepwise variable selection in the sense of doing the same job better — it solves a related but different problem (penalized, simultaneous selection via a single convex optimization, rather than a sequence of significance tests) and does not carry stepwise regression’s specific inflated Type I error and instability problems, though a penalized model’s standard errors still require care before being reported as though from an unpenalized regression. See this site’s broader regression analysis guide for how ridge, LASSO, and elastic net fit into the wider set of regression approaches and their assumptions.

Frequently asked questions

Is elastic net always better than LASSO or ridge alone?

Not universally — it is a generalization that includes both as special cases (α = 1 is LASSO, α = 0 is ridge), so a properly cross-validated elastic net search will select α close to 1 or 0 if pure LASSO or pure ridge genuinely fits the cross-validated error best for that dataset. Its practical advantage shows up specifically when predictors are correlated and some intermediate α value outperforms both extremes, which is common in practice but not guaranteed for every dataset.

What does it mean if cross-validation selects alpha near 0 or near 1?

It means the tuning process found the ridge-like or LASSO-like end of the blend to fit the data best given the candidate grid tested — effectively, the cross-validation is telling you the data didn’t benefit from mixing the two penalties. This is a legitimate result, not a sign the procedure failed; it simply means plain ridge or plain LASSO would have been an adequate and simpler choice for this particular dataset.

Can elastic net be used for logistic or other generalized linear models, not just linear regression?

Yes. Both glmnet and scikit-learn support elastic-net-penalized logistic regression and other generalized linear model families; the penalty term is added to the model’s deviance rather than the residual sum of squares, but the α/λ mixing and tuning logic is the same. See this site’s guide to logistic regression in R for the unpenalized case that elastic net logistic regression extends.

Does elastic net eliminate the need to check for multicollinearity beforehand?

No — it changes what multicollinearity does to the model (grouped shrinkage instead of unstable single-predictor selection or inflated standard errors) rather than removing the underlying redundancy in the predictors. Understanding which predictors are correlated, and why, is still useful for interpreting which variables end up grouped together in the fitted model’s coefficient pattern.

How many candidate alpha values are usually enough for the grid search?

There is no universal number; a coarse grid (e.g. five to ten values spanning 0 to 1) is a common practical starting point, refined with a finer grid around whichever region the coarse search favors if the extra precision is worth the added compute. The right resolution depends on how sensitive the cross-validated error actually is to α for the dataset at hand, which is itself only visible after running the coarse search.

Sources

• Zou, H., and Hastie, T. (2005). “Regularization and Variable Selection via the Elastic Net.” Journal of the Royal Statistical Society, Series B, 67(2), 301–320.
• Tibshirani, R. (1996). “Regression Shrinkage and Selection via the Lasso.” Journal of the Royal Statistical Society, Series B, 58(1), 267–288.
• Hoerl, A.E., and Kennard, R.W. (1970). “Ridge Regression: Biased Estimation for Nonorthogonal Problems.” Technometrics, 12(1), 55–67.
• Friedman, J., Hastie, T., and Tibshirani, R. (2010). “Regularization Paths for Generalized Linear Models via Coordinate Descent.” Journal of Statistical Software, 33(1), 1–22. (The glmnet algorithm paper.)
• Hastie, T., Tibshirani, R., and Friedman, J. The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer.