Garak: LLM Vulnerability Scanner and Probing Guide

Garak (the Generative AI Red-teaming & Assessment Kit) is an open-source vulnerability scanner and adversarial testing framework for Large Language Models (LLMs) and generative artificial intelligence systems. Often characterized as an “nmap for LLMs,” garak systematically probes models to detect security vulnerabilities, jailbreak risks, training data leakage, toxic outputs, and policy failures before systems enter production.

While functional benchmarks such as the SWE-bench software engineering benchmark and Humanity’s Last Exam frontier evaluation measure task capability and reasoning boundaries, garak evaluates model robustness under deliberate adversarial stress. Its automated probe execution, prompt mutation buffers, and output evaluation classifiers support repeatable security audits that integrate directly into rigorous AI output verification workflows.

What It Is

Formally documented in garak: A Framework for Security Probing Large Language Models (Derczynski et al., 2024), garak is a Python-based command-line security tool that automates black-box and grey-box security assessments across commercial APIs and open-weight models. The framework structures testing into four modular plugin layers:

• Generators: Connection interfaces that communicate with the target model, managing API authentication, request schemas, temperature parameters, and rate limits across cloud services and local runtimes.
• Probes: Attack modules that construct adversarial prompts targeting specific vulnerabilities, such as system prompt extraction, jailbreaks, data replay, or malicious payload generation.
• Buffs: Transformation plugins that mutate probe prompts prior to submission (e.g., applying Base64 encoding or character substitutions) to test whether simple obfuscation bypasses defensive filters.
• Detectors: Evaluation modules that classify the model’s output to determine whether an attack succeeded (a “hit”) or was resisted with an appropriate refusal.

Who Runs It and What It Covers

Garak was founded by AI security researchers Leon Derczynski, Erick Galinkin, Jeffrey Martin, Subho Majumdar, and Nanna Inie. The project is maintained as an open-source codebase under the NVIDIA open-source organization on GitHub (NVIDIA/garak), with active participation from academic laboratories and the broader AI safety community.

The scanner covers dozens of vulnerability families, mapping directly to recognized industry security taxonomies:

• Prompt Injection & Jailbreaks (OWASP LLM01): Probes such as promptinject and dan test whether models ignore system prompts or adopt unrestricted adversarial personas.
• Sensitive Information Disclosure (OWASP LLM02): The leakreplay suite tests whether models regurgitate pre-training data, API keys, or personal identifiers.
• Improper Output Handling (OWASP LLM05): The xss probe examines whether models generate malicious scripts or unescaped HTML that could compromise client applications.
• Excessive Agency & Misuse (OWASP LLM06): The malwaregen probe tests whether safety guardrails block requests to author malicious exploits or attack scripts.
• Taxonomy Alignment: Probes are cross-tagged with MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) tactics and AI Vulnerability Database (AVID) identifiers.

How to Search or Use It

Garak is executed directly from the terminal. The following sequence demonstrates how to run structured vulnerability assessments using official CLI commands:

1. Install the Scanner: Install the package in an isolated virtual environment using pip:

   python -m pip install -U garak

2. Configure Target Credentials: Export necessary API keys for commercial endpoints:

   export OPENAI_API_KEY="sk-..."

3. List Available Modules: Inspect installed generators, probes, and detectors to select audit targets:

   python -m garak --list_generators
   python -m garak --list_probes
   python -m garak --list_detectors

4. Execute Targeted Probing: Run focused scans against a model by specifying the generator, model identifier, and probe modules:

   python -m garak --model_type openai --model_name gpt-4o-mini --probes dan,promptinject

5. Test Obfuscated Attacks with Buffs: Apply prompt mutation buffers to evaluate resilience against encoded inputs:

   python -m garak --model_type openai --model_name gpt-4o-mini --probes promptinject --buffs encoding

6. Inspect Execution Logs: Review the run artifacts generated in the working directory, including the full JSON Lines report (garak.<uuid>.report.jsonl) and the hit log recording successful exploit attempts.
7. Generate Taxonomy Digests: Structure scan findings into an OWASP-aligned HTML summary using garak’s analysis utility:

   python -m garak.analyze.report_digest garak.<run_id>.report.jsonl owasp > report.html

Strengths and Limits

Garak provides several core strengths for computational research and security evaluation:

• Deterministic Audit Trails: Every prompt attempt, response string, and classification decision is logged in JSON Lines format, ensuring verifiable experimental reproducibility.
• Extensible Plugin Architecture: Research groups can create custom probes and detectors by subclassing standard base classes without modifying the scanning engine.
• Broad Model Support: Out-of-the-box generators connect to commercial APIs (OpenAI, Replicate), local Hugging Face models, and inference runtimes like Ollama and vLLM.
• Formal Taxonomy Mapping: Built-in tagging against OWASP LLM Top 10 and MITRE ATLAS provides clear traceability for compliance and security reviews.

The platform also has defined operational boundaries:

• Black-Box Scope: Probing operates at the input-output boundary and does not evaluate internal model weights, attention maps, or activation states.
• Detector Heuristics: Automated classifiers rely on regex patterns or secondary models, which can produce false positives on benign edge cases or false negatives on novel exploits.
• API Cost and Rate Limits: Exhaustive scans generate thousands of queries, requiring careful batch management to avoid API throttling and high service charges.
• Need for Human Oversight: Automated probing cannot replace comprehensive red-teaming for nuanced institutional threats or broader research security protocols. Teams should evaluate findings alongside complementary tools, such as our AI detector comparison.

Alternatives

Research teams selecting an evaluation framework must balance automated vulnerability scanning against multi-turn red-teaming and prompt engineering tests. The following table contrasts garak with primary alternatives:

Tool	Primary Focus	Testing Architecture	Key Output	Best Fit in Research
garak	Automated LLM vulnerability & safety scanning	Generators, Probes, Buffs, Detectors	JSONL audit trace, hit log, HTML digest	Automated vulnerability baselines, regression testing, and OWASP mapping
PyRIT (Microsoft)	Dynamic multi-turn adversarial red-teaming	Orchestrators, Converters, Targets, Scorers	Memory database & structured JSON logs	Multi-turn conversational jailbreaks and automated attack strategy research
Promptfoo	Application testing, prompt evaluation & CI/CD	YAML test assertions, LLM-as-a-judge	Interactive web UI, JSON, CSV	CI/CD regression testing for application prompts and RAG pipelines
Giskard	ML/LLM quality, bias, and performance scanning	Automated tests, tabular & text inspectors	HTML interactive inspection reports	Compliance scanning, tabular ML validation, and bias detection
Inspect (UK AISI)	Frontier AI capabilities and safety evaluation	Tasks, Solvers, Scorers, Dataset loaders	Evaluation logs, terminal visualizer	Government and academic evaluations of frontier reasoning and cyber risks

Frequently Asked Questions

How does garak decide whether a test attempt is a vulnerability hit?

Each probe generates an Attempt containing adversarial prompts. When the generator retrieves the target model’s response, it submits the text to configured detectors. Detectors evaluate whether the model complied with the probe’s adversarial request (e.g., generating exploit code or confirming restricted instructions) rather than returning a refusal. When compliance is detected, garak marks the attempt as a hit and records the event in the hit log.

What is the difference between garak and a functional benchmark like SWE-bench?

SWE-bench evaluates functional programming ability by testing whether models can solve real-world software engineering issues. In contrast, garak evaluates safety resilience and exploitability by testing whether models succumb to adversarial manipulation. High performance on functional benchmarks does not guarantee resilience against the prompt injections or data leakage attacks evaluated by garak.

Can garak scan self-hosted or private institutional models?

Yes. Garak supports local Hugging Face pipelines (--model_type huggingface) and provides a configurable rest generator (--model_type rest) that connects to custom HTTP endpoints. Research laboratories can evaluate self-hosted models on vLLM or internal servers without routing traffic through third-party services.

Does running garak guarantee complete safety compliance?

No. While garak systematically tests models against hundreds of known attack vectors, automated scanning cannot guarantee the absence of all vulnerabilities. Adversaries frequently formulate novel jailbreaks and context manipulation techniques. Automated testing should serve as one component within a defense-in-depth security strategy alongside runtime guardrails and manual red-teaming.

Sources

• Derczynski, L., Galinkin, E., Martin, J., Majumdar, S., & Inie, N. (2024). garak: A Framework for Security Probing Large Language Models. arXiv:2406.11036. https://arxiv.org/abs/2406.11036 (Checked 2026-11-07).
• Garak Project Documentation. Generative AI Red-teaming & Assessment Kit. Read the Docs. https://garak.readthedocs.io/ (Checked 2026-11-07).
• NVIDIA. garak: LLM vulnerability scanner. GitHub Repository. https://github.com/NVIDIA/garak (Checked 2026-11-07).
• OWASP Foundation. OWASP Top 10 for Large Language Model Applications. https://owasp.org/www-project-top-10-for-large-language-model-applications/ (Checked 2026-11-07).