Written and maintained by CASRAI Editorial Board
Published
On this page
AutoAttack is a fixed ensemble of four adversarial attacks, written by Francesco Croce and Matthias Hein, that estimates the adversarial robustness of an image classifier without any attack tuning on your side. You install it from its GitHub repository, wrap your model so it returns logits, choose a norm and an epsilon, and run it; the reported robust accuracy is the share of test images the classifier still gets right after the strongest of the four attacks has had its turn. It is the evaluation that RobustBench adopted as its standard in October 2020.
What AutoAttack is used for in research
AutoAttack is used to check whether a claimed defense against adversarial examples really holds. The method was introduced in the ICML 2020 paper “Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks” (arXiv 2003.01690). The paper’s starting point is that robustness evaluations are often insufficient, mainly because attack hyperparameters are improperly tuned or because the defense masks or obfuscates gradients. Run on more than 50 models from the literature, the ensemble gave a lower robust accuracy than the one reported in all but one case, often by more than 10 percentage points, and it identified several broken defenses.
The ensemble has four members, and the README fixes all of their hyperparameters so that no tuning is needed for each new classifier:
- APGD-CE: a step-size-free version of projected gradient descent (PGD) on the cross-entropy loss.
- APGD-DLR: the same step-size-free PGD on the DLR (difference of logits ratio) loss.
- FAB: an attack that minimises the norm of the adversarial perturbation (Croce and Hein, 2019).
- Square Attack: a query-efficient black-box attack (Andriushchenko et al., 2019) that does not use gradients.
Combining gradient-based and gradient-free attacks is the point of the design. A defense that blunts one family of attack rarely blunts all four, so the lowest robust accuracy across the ensemble is a more honest figure than a single PGD run. For the wider picture of how model robustness is defined, see robustness; for evaluation of language models rather than image classifiers, the pages on garak and HarmBench cover the red-teaming side, and data poisoning covers attacks on the training data instead of the inputs.
Standard, plus and rand versions
The version is chosen with the version argument when you create the AutoAttack object. The attack lists below are those given in the project README.
| Version | Attacks run | Intended for |
|---|---|---|
standard |
Untargeted APGD-CE (no restarts); targeted APGD-DLR (9 target classes); targeted FAB (9 target classes); Square Attack (5000 queries) | The default evaluation used for the README’s defense tables and for RobustBench |
plus |
Untargeted APGD-CE, APGD-DLR and FAB (5 restarts each); Square Attack (5000 queries); targeted APGD-DLR and FAB (9 target classes) | A more expensive evaluation |
rand |
Untargeted APGD-CE and APGD-DLR (no restarts, 20 iterations for Expectation over Transformation) | Classifiers with stochastic components |
custom |
Whatever you set in attacks_to_run and the per-attack restart counts |
Cases where you need to change the attack list |
The targeted variants of APGD-DLR and FAB were added in the August 2020 update so that the evaluation scales to datasets with many classes and runs faster and more accurately. An L1 version, including extensions of APGD and Square Attack, followed in March 2021 (Croce and Hein, 2021, arXiv 2103.01208).
How to install AutoAttack
AutoAttack is not distributed as a packaged release; the README installs it straight from the GitHub repository of its authors. You need a working PyTorch or TensorFlow setup for your own model first, since AutoAttack only wraps the model you already have.
- Open a terminal in the Python environment where your model runs.
- Run
pip install git+https://github.com/fra31/auto-attack. - Check the import with
from autoattack import AutoAttack. For TensorFlow models you also import the adapter modules,utils_tffor TensorFlow 1.x orutils_tf2for TensorFlow 2.x, from the same package.
The repository also contains an examples/ folder. Its eval.py script runs the standard evaluation on a pretrained PyTorch model on CIFAR-10, and accepts --version=standard or --version=plus and an optional --individual flag that runs every attack on the full test set.
How to run AutoAttack step by step
The steps below follow the README for a PyTorch classifier. Your model, images and labels are placeholders for your own objects.
- Make the model return logits. AutoAttack expects the pre-softmax output. The
forward_passyou give it must return logits and accept inputs with components in [0, 1] in NCHW layout. Put the network in evaluation mode before you start, so that dropout or batch-norm updates do not make the logits change between runs; the library raises a warning when they do. - Choose the threat model. Set
normto'Linf','L2'or'L1', and setepsto the bound on the perturbation norm. The README’s CIFAR-10 Linf table uses eps = 8/255, the MNIST Linf table uses eps = 0.3 and the CIFAR-10 L2 table uses eps = 0.5; use the epsilon your defense claims, not a convenient one. - Create the adversary.
from autoattack import AutoAttack adversary = AutoAttack(forward_pass, norm='Linf', eps=8/255, version='standard') - Run the standard evaluation. The attacks run sequentially on batches of size
bs:x_adv = adversary.run_standard_evaluation(images, labels, bs=batch_size) - Optionally run each attack on its own.
dict_adv = adversary.run_standard_evaluation_individual(images, labels, bs=batch_size)This returns a dictionary with the adversarial examples found by each attack.
- Optionally restrict or customise the attacks. Setting
adversary.attacks_to_run = ['apgd-ce']runs a subset. Withversion='custom'you can set, for example,adversary.attacks_to_run = ['apgd-ce', 'fab']and changeadversary.apgd.n_restartsandadversary.fab.n_restarts. - Fix the seed and write a log.
adversary.seed = 0uses the same random seed for all attacks; otherwise each attack picks its own. Passinglog_path=/path/to/logfile.txtwhen you create the object logs the intermediate results.
TensorFlow models
For TensorFlow 1.x, wrap the model with utils_tf.ModelAdapter(logits, x_input, y_input, sess) (inputs in NHWC layout) and pass is_tf_model=True to AutoAttack. For TensorFlow 2.x, use utils_tf2.ModelAdapter(tf_model), where tf_model is a Keras model without the final softmax activation, and again pass is_tf_model=True. After that the evaluation runs in the same way as for PyTorch models.
Randomized defenses
If the classifier has stochastic components, create the adversary with version='rand'. This combines APGD with Expectation over Transformation as in Athalye et al. (2018), and the README lists 20 EoT iterations for each of its two attacks.
How to read the AutoAttack output
The call returns adversarial images, and the quantity you report is robust accuracy: the fraction of your evaluation images that the classifier still labels correctly after the attack. Always report it next to clean accuracy, the norm, the epsilon and the version, because a robust accuracy number means nothing without its threat model. The project’s own tables use the same layout: clean accuracy, the robust accuracy reported by the defense’s authors, and the robust accuracy under AutoAttack, with the AutoAttack figure computed on the full test set.
The gap between the reported and the AutoAttack column is the useful signal. In the data table below, taken from the AutoAttack repository’s CIFAR-10 Linf results at eps = 8/255, several defenses lose very little under AutoAttack (the 65.87 reported versus 65.88 under AutoAttack for the top WRN-70-16 model of Gowal et al. 2020), while others lose most of their claimed robustness, down to a robust accuracy of 1.35 for Jin and Rinard (2020) against 71.22 reported. A small gap means the original evaluation was already strong; a large gap points to gradient masking or a weak attack in the original paper. The README notes that the CIFAR-10 and CIFAR-100 lists are no longer maintained and that RobustBench carries the current leaderboards, so use that table as a worked illustration, not as a current ranking.
When you run the attacks individually, compare them. If Square Attack, the black-box member, finds more adversarial examples than the white-box gradient attacks, the project documentation treats that as a hint that the robustness is being overestimated, as discussed in the next section.
Troubleshooting AutoAttack
Since September 2021 AutoAttack runs automatic checks and raises a flag when the classifier behaves in a way typical of non-standard models. The project’s flags_doc.md describes each flag, why it appears and what to do about it.
| Flag | Raised when | Suggested remedy in the documentation |
|---|---|---|
| Randomized defenses | Clean accuracy on a batch, or the logits, vary across multiple runs | Use version='rand', which combines APGD with EoT |
| Softmax output is given | The model outputs a probability distribution | Return logits instead; softmax output can cause numerical instability that stops gradient-based attacks working well |
| Zero gradient | The gradient at the random starting point of APGD is zero for any image with the DLR loss | Use margin-based losses; for components such as input quantization use BPDA (Athalye et al., 2018) or black-box attacks like Square Attack |
| Square Attack improves the robustness evaluation | Square Attack lowers the robust accuracy found by the white-box attacks | Run Square Attack with a higher budget (more queries, random restarts) or design adaptive attacks |
| Optimization at inference time (PyTorch only) | Standard PyTorch gradient functions are called while running inference | Design adaptive attacks; AutoAttack gives only a first estimate for dynamic defenses |
The same document is explicit about a wider limit: in its standard version AutoAttack is not suitable for randomized, non-differentiable or dynamic defenses, which rely on different principles from commonly used robust models. For those, specific modifications or adaptive attacks (Tramèr et al., 2020, arXiv 2002.08347) may be needed. A low attack success rate on such a model is therefore not evidence of robustness.
Free alternatives to AutoAttack
Several open-source libraries cover attack evaluation. The table lists what each project’s own repository states; none of them replaces the fixed, tuning-free protocol that makes AutoAttack results comparable between papers.
| Tool | What its repository says it is | Relation to AutoAttack |
|---|---|---|
| torchattacks | A PyTorch library of adversarial attacks, MIT licensed, installed with pip install torchattacks |
Its attack list includes an AutoAttack entry (Linf, L2) described as APGD + APGDT + FAB + Square, as well as APGD and APGDT on their own |
| Adversarial Robustness Toolbox (ART) | A Python library for machine learning security hosted by the Linux Foundation AI and Data Foundation, covering evasion, poisoning, extraction and inference threats across TensorFlow, Keras, PyTorch, scikit-learn and other frameworks | A broader security toolkit rather than a single robustness protocol |
| Foolbox | A Python library for running adversarial attacks to benchmark model robustness, natively in PyTorch, TensorFlow and JAX, with gradient-based and decision-based attacks | A general attack collection for benchmarking |
| RobustBench | A benchmark with leaderboards and a Model Zoo of robust classifiers; it adopted AutoAttack as its standard evaluation in October 2020 | Use it to compare your numbers with current published results |
Check each project’s current documentation before depending on a particular attack class, since library contents change between releases.
Reporting results from an AutoAttack run
A reproducible report names the dataset and test-set size, the model architecture and training recipe, the norm, the epsilon, the AutoAttack version (standard, plus, rand or a custom attack list), the random seed, the library commit you installed from, and the clean and robust accuracy. If you changed the attack list or restart counts, say so, because the standard version is what the published tables and RobustBench use. If any of the automatic flags above was raised, state it and what you did about it. Citing the method means citing Croce and Hein (ICML 2020), and the README also lists the 2021 ICML paper on l1-APGD for sparse attacks for the L1 version.
Sources
- fra31/auto-attack: AutoAttack repository and README (Croce and Hein), read 2026-11-17
- AutoAttack flags_doc.md: On the usage of AutoAttack, read 2026-11-17
- Croce and Hein (2020), Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks, arXiv 2003.01690, read 2026-11-17
- torchattacks (Harry24k/adversarial-attacks-pytorch) README, read 2026-11-17
- Adversarial Robustness Toolbox README, read 2026-11-17
- Foolbox README, read 2026-11-17
- RobustBench leaderboards
Data: Reported versus AutoAttack robust accuracy of selected CIFAR-10 Linf defenses (eps = 8/255)
Selected rows from the AutoAttack repository's CIFAR-10 Linf results table (eps = 8/255): clean accuracy, the robust accuracy reported by each defense's authors, and the robust accuracy under the standard AutoAttack evaluation on the full test set. The last column is the difference, computed as reported minus AutoAttack; it shows how far claimed robustness was overestimated.
Period covered: 2018/2021.
| Defense (paper) | Architecture | Clean accuracy (%) | Reported robust accuracy (%) | AutoAttack robust accuracy (%) | Overestimate (reported minus AutoAttack) (percentage points) |
|---|---|---|---|---|---|
| Gowal et al. 2020 (extra data) | WRN-70-16 | 91.1 | 65.87 | 65.88 | -0.01 |
| Carmon et al. 2019 (extra data) | WRN-28-10 | 89.69 | 62.5 | 59.53 | 2.97 |
| Wang et al. 2020 (extra data) | WRN-28-10 | 87.5 | 65.04 | 56.29 | 8.75 |
| Hendrycks et al. 2019 (extra data) | WRN-28-10 | 87.11 | 57.4 | 54.92 | 2.48 |
| Rice et al. 2020 | WRN-34-20 | 85.34 | 58 | 53.42 | 4.58 |
| Engstrom et al. 2019 | RN-50 | 87.03 | 53.29 | 49.25 | 4.04 |
| Madry et al. 2018 | WRN-34-10 | 87.14 | 47.04 | 44.04 | 3 |
| Pang et al. 2020a | RN-32 | 80.89 | 55 | 43.48 | 11.52 |
| Zhang and Wang 2019 | WRN-28-10 | 89.98 | 60.6 | 36.64 | 23.96 |
| Zhang and Xu 2020 | WRN-28-10 | 90.25 | 68.7 | 36.45 | 32.25 |
| Wang and Zhang 2019 | WRN-28-10 | 92.8 | 58.6 | 29.35 | 29.25 |
| Jin and Rinard 2020 | RN-18 | 90.84 | 71.22 | 1.35 | 69.87 |
| Chan et al. 2020 | WRN-34-10 | 93.79 | 15.5 | 0.26 | 15.24 |
Sources
- fra31/auto-attack README: Adversarial Defenses Evaluation, CIFAR-10 Linf (accessed 2026-11-17)
- Croce and Hein (2020), arXiv 2003.01690 (accessed 2026-11-17)
Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.







