PLDI: Research Software, Artifact Evaluation, and Execution Guide

The ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) is the premier international research forum and publication venue for innovations in programming language design, optimizing compilers, runtime systems, program analysis, and formal verification. Published under the Gold Open Access model in the journal Proceedings of the ACM on Programming Languages (PACMPL), PLDI mandates rigorous scientific peer review coupled with an independent Artifact Evaluation process that inspects, validates, and badges computational software packages, benchmark suites, and machine-checked proofs.

What it is used for in research

In computer science and software engineering, PLDI serves as the foundational venue where theoretical advances in programming semantics meet practical, high-performance systems implementation. Researchers, computer systems architects, and verification engineers look to PLDI for breakthroughs in compiler optimizations, static and dynamic analysis frameworks, memory safety guarantees, automated program synthesis, and hardware-software co-design. Work published at PLDI directly shapes mainstream industrial development environments, modern language runtimes such as the Java Virtual Machine and JavaScript V8 engines, and high-assurance verification toolchains.

Research presented at PLDI spans several interconnected technical domains:

• Compiler Construction and Optimization: Intermediate representations (IR), polyhedral loop transformations, just-in-time (JIT) compilation, automatic vectorization, and domain-specific compilation pipelines for artificial intelligence and machine learning accelerators.
• Formal Methods and Verification: Rigorous deductive verification, interactive theorem proving, automated model checking, and satisfiability modulo theories (SMT) solving applied to critical infrastructure. For in-depth exploration of proof methodologies, researchers consult our companion guides on formal verification proofs and modern formal methods research.
• Program Analysis and Type Systems: Sound static analysis, abstract interpretation, information flow tracking, gradual typing, affine type systems, and concurrency defect detection (including data races, deadlocks, and atomicity violations).
• Runtime Systems and Memory Management: Real-time garbage collection algorithms, memory allocation architectures, persistent memory abstractions, and cache-coherent multicore execution models.

Historically established in 1988 as the successor to the ACM SIGPLAN Symposium on Compiler Construction, PLDI transitioned in 2017 to an integrated journal-conference publishing structure. Papers accepted to PLDI appear directly in the journal Proceedings of the ACM on Programming Languages (PACMPL). This publishing model guarantees permanent Gold Open Access under Creative Commons Attribution 4.0 (CC BY 4.0) licenses, fully sponsored by ACM SIGPLAN without author-facing Article Processing Charges (APCs). The evaluation incorporates rigorous double-anonymized peer review across multiple submission rounds, ensuring equitable assessment based solely on technical merit.

A distinctive hallmark of PLDI is its pioneer role in software artifact evaluation. Because computer systems research depends heavily on complex implementations, PLDI was among the earliest computing conferences to establish a dedicated Artifact Evaluation Committee (AEC). The AEC independently tests software artifacts—including source code, virtual machines, benchmark inputs, and automated evaluation scripts—to verify that experimental claims match executable reality. This process protects the scientific literature against reproducibility failures and provides the community with durable, extensible software components.

Install or access (official source only)

Accessing PLDI research and executing its associated software artifacts requires utilizing official, authenticated repositories and conforming to ACM reproducibility standards. Researchers interact with three primary layers: publication access, persistent artifact retrieval, and execution container configuration.

All published PLDI articles and archival proceedings are permanently accessible through the official ACM Digital Library within the PACMPL collection (dl.acm.org/journal/pacmpl) and through the annual SIGPLAN conference web portals (such as pldi24.sigplan.org and pldi25.sigplan.org). Each publication is assigned a permanent Crossref Digital Object Identifier (DOI), which resolves directly to the full-text PDF, metadata, and linked computational artifacts.

To inspect and execute software artifacts associated with PLDI papers, researchers retrieve packages preserved in certified long-term digital archives, primarily Zenodo and Software Heritage. In accordance with the ACM Artifact Review and Badging policy, artifacts awarded availability badges must reside in non-volatile repositories that guarantee persistent DOIs, rather than unstable personal websites or unarchived repositories.

Executing research software artifacts requires establishing a clean, isolated host environment. Modern PLDI artifacts standardize on containerized deployments to avoid dependency conflicts across host operating systems. The standard toolchain requirements include:

• Container Engine: Docker Engine version 20.10 or higher, Podman, or Singularity/Apptainer for high-performance computing (HPC) clusters. For architectural comparisons across container technologies, see our dictionary overview of container images.
• Virtualization and Hardware Flags: x86_64 host architecture with hardware virtualization enabled (Intel VT-x or AMD-V). Many compiler and runtime artifacts rely on specific CPU instruction extensions, such as AVX-512, Intel TSX, or hardware performance counters.
• Formal Proof Assistants: Interactive proof environments such as Coq (version 8.16+), Isabelle/HOL, Lean, or Agda, often managed via language package managers such as OPAM.
• Automated Theorem Solvers: SMT solvers such as Z3 (version 4.8.12+) or CVC5, typically linked dynamically or packaged directly within the artifact container.

Researchers can pull verified PLDI artifact containers directly from public registries or build them locally from archived Dockerfiles using standard commands:

# Pull a published PLDI artifact container from an archival registry
docker pull ghcr.io/sigplan-artifacts/pldi-evaluation:latest

# Verify local container runtime capabilities
docker run --rm ghcr.io/sigplan-artifacts/pldi-evaluation:latest uname -m

Step-by-step task

Reproducing an experimental evaluation from a PLDI publication requires executing the artifact under conditions that mirror the original paper review. The following numbered walkthrough demonstrates how a researcher downloads, validates, and runs a containerized PLDI research software artifact to replicate empirical benchmark measurements and formal verification claims.

1. Locate and Download the Certified Archival Package: Retrieve the artifact archive using its registered Zenodo or ACM Digital Library DOI. Extract the archive into a dedicated working directory on a system with sufficient disk storage (typically 20 GB to 50 GB for compiler benchmarks and proof libraries):

   mkdir -p ~/pldi-reproduce
   cd ~/pldi-reproduce
   curl -L -o pldi-artifact.tar.gz https://zenodo.org/record/10892345/files/pldi-artifact.tar.gz
   tar -xzf pldi-artifact.tar.gz
   cd pldi-artifact

2. Inspect Artifact Documentation and Requirements: Open the mandatory README.md or INSTALL.md file. Check the documented hardware constraints, memory prerequisites, estimated execution times for test suites, and the mapping between paper sections and artifact scripts.
3. Build or Import the Container Image: If the archive supplies a pre-built Docker image tarball, load it directly into Docker to ensure exact bit-for-bit library matching. Alternatively, build the container using the provided Dockerfile:

   # Load a pre-packaged Docker image tarball
   docker load -i pldi-artifact-image.tar.gz
   
   # Alternatively, build locally if Dockerfile is provided
   docker build -t pldi-artifact:local .

4. Execute the Smoke Test (“Kick-the-Tires” Phase): In the ACM artifact evaluation protocol, reviewers first run a lightweight sanity check designed to terminate within 10 to 15 minutes. This confirms that all binaries compile, test harnesses connect, and basic solver queries resolve:

   docker run --rm -v $(pwd)/results:/workspace/results pldi-artifact:local ./kick_the_tires.sh

   Inspect the console output to confirm that all test assertions report success (e.g., [PASS] Smoke test completed: 12/12 unit tests succeeded).

5. Run the Full Experimental Benchmark Suite: Execute the comprehensive evaluation harness that reproduces the main figures and tables of the published paper. Use dedicated core pinning where execution timing is measured:

   docker run --rm --cpus='4.0' --memory='16g' -v $(pwd)/results:/workspace/results pldi-artifact:local ./run_full_experiments.sh --all

   This script automatically compiles the target benchmark suites (e.g., DaCapo, SPEC CPU, or custom microbenchmarks), runs the experimental and baseline configurations across multiple iterations, and writes raw execution logs to the mounted results volume.

6. Generate Output Tables and Figures: Process the raw benchmark logs and proof verification metrics into formatted tables matching the paper:

   python3 scripts/generate_tables.py --input-dir /workspace/results --output-format markdown

   Compare the generated Markdown or CSV output against Table 1 and Table 2 in the published paper, noting whether execution times and speedup ratios fall within expected variance boundaries.

Output and how to read it

Interpreting outputs from a PLDI research software artifact involves evaluating both the empirical measurement logs generated by the tools and the official ACM badges awarded to the publication. Researchers must understand how raw metrics translate into valid scientific claims.

When the evaluation harness finishes, it generates structured output files (typically in JSON, CSV, or formatted ASCII tables) located in the output directory. A typical benchmark reproduction log reports several core metrics:

• Execution Time and Speedup: Reported as geometric mean speedup relative to a defined baseline compiler (such as Clang -O3 or standard OpenJDK). Speedup is calculated as Baseline_Time / Optimized_Time. A value greater than 1.0 indicates performance improvement.
• Variance and Confidence Intervals: In accordance with ACM SIGPLAN Empirical Evaluation Guidelines, robust benchmarks present mean runtimes across at least 10 to 30 warm-up and measurement iterations, accompanied by 95% confidence intervals or standard deviations. Single-run timings are invalid evidence of performance.
• Memory Footprint and Peak RSS: Peak Resident Set Size measured in megabytes, indicating memory overhead introduced by custom runtime instrumentation, garbage collection barriers, or static analysis state trackers.
• Verification Solver Metrics: For formal verification tools, outputs indicate whether SMT queries returned sat, unsat, or timeout, along with the total solver CPU seconds and number of proof obligations discharged.

At the publication level, papers in PACMPL (PLDI) display distinct ACM Artifact Badges on their title page. Understanding these badges clarifies what level of vetting the software has undergone:

• Artifacts Available: Confirms that the code and data have been deposited in a persistent, citable public repository with a registered DOI. It does not certify that the code executes or reproduces the results.
• Artifacts Evaluated – Functional: Certifies that the Artifact Evaluation Committee successfully deployed the artifact, confirmed the documentation was sufficient, executed the core functionality, and verified that the software behaves as described.
• Artifacts Evaluated – Reusable: Indicates an exceptional level of software engineering quality. The artifact features clean modularity, extensive documentation, clear APIs, and instructions that enable other researchers to adapt and extend the software for new research tasks.
• Results Reproduced: The highest standard of experimental rigor, certifying that the independent committee executed the artifact on appropriate hardware and obtained empirical results that quantitatively match the central claims in the paper within acceptable variance.

Troubleshooting

When deploying and replicating complex PLDI research artifacts, researchers commonly encounter configuration hurdles arising from microarchitectural discrepancies, dependency bit-rot, and solver timeouts. The table below and accompanying guidance outline standard remediation steps.

Issue	Probable Cause	Diagnostic Command	Remediation Step
Architecture emulation failure	x86_64 container executed on ARM64 host (Apple Silicon or AWS Graviton)	uname -m && docker inspect --format '{{.Architecture}}' container_image	Pass --platform linux/amd64 to Docker or run on native x86_64 hardware for timing-sensitive benchmarks.
Out-of-memory (OOM) killed process	Memory limits exceeded during whole-program static analysis or model checking	dmesg -T | grep -i oom	Increase Docker host memory limit using --memory='32g' --shm-size='4g' in the execution command.
SMT solver syntax or version mismatch	Local solver (Z3 or CVC5) updated past the syntax supported by the artifact	z3 --version	Pin the exact solver version specified in the artifact documentation inside the container environment.
Dependency repository 404 error	Bit-rot in external package repositories (expired apt PPAs, deprecated Ruby or Python gems)	apt-get update && cargo check	Switch package repository URLs to official archival mirrors (such as Ubuntu Old Releases or Debian Archive).
Severe timing variance	Host CPU frequency scaling, hyperthreading contention, or background OS tasks	cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor	Set CPU governor to performance mode, disable Turbo Boost, and isolate cores using taskset -c 2,3.

A frequent point of failure in older PLDI artifacts is the reliance on dynamic web fetching during the container build process. When upstream package repositories retire older compiler releases or third-party libraries, container builds fail. To mitigate this, researchers should always search for pre-built container tarballs archived directly alongside the Zenodo record, rather than attempting to rebuild legacy Dockerfiles from scratch.

For formal verification pipelines, non-deterministic solver runtimes can cause proofs to exceed the default timeout thresholds. If an artifact script fails with an unexpected timeout on proof discharge, inspect the solver flags in the harness configuration. Adjusting the per-query timeout parameter (for example, increasing -t:10000 to -t:30000 milliseconds) often allows proofs to complete successfully on slower host processors without compromising soundness.

Free alternatives

Researchers investigating programming languages, compilers, and formal verification have access to several premier, peer-reviewed academic venues and open reproducibility platforms. While PLDI represents the gold standard for combined language design and implementation performance, companion conferences specialize in adjacent theoretical or architectural niches.

The structured comparison below outlines the primary alternative publication venues and artifact evaluation ecosystems across computer systems and programming language research:

Venue / Ecosystem	Primary Research Focus	Review Model	Artifact Evaluation	Open Access Status
POPL (Principles of Programming Languages)	Theoretical foundations, formal semantics, logic, and proof systems	Dual-round, double-anonymous (PACMPL)	Mandatory AEC review option; ACM badging	Gold Open Access (PACMPL, CC BY 4.0)
OOPSLA (Object-Oriented Programming, Systems, Languages and Applications)	Software engineering, language design, modularity, and practical systems	Dual-deadline, double-anonymous (PACMPL)	Active AEC review; ACM badging	Gold Open Access (PACMPL, CC BY 4.0)
ICFP (International Conference on Functional Programming)	Functional programming languages, type theory, and mathematical models	Dual-round, double-anonymous (PACMPL)	Active AEC review; ACM badging	Gold Open Access (PACMPL, CC BY 4.0)
ASPLOS (Architectural Support for Programming Languages and OS)	Hardware-software interface, architecture, operating systems, and compilers	Multi-deadline rolling review (ACM)	Rigorous AEC track; ACM badging	ACM Open / Open Access
CAV (Computer Aided Verification)	Formal verification algorithms, model checking, SAT/SMT solving	Single-round, double-anonymous (Springer LNCS)	Dedicated AEC track; reproducibility badges	Hybrid / Green Open Access
TACAS (Tools and Algorithms for the Construction and Analysis of Systems)	Tool architectures, algorithm design, software verification benchmarks	Single-round with tool paper track (Springer LNCS)	Mandatory artifact submission for tool papers	Hybrid / Green Open Access

In addition to formal conference publication venues, researchers utilize open-access reproducibility repositories such as Papers with Code, Zenodo, and the Software Heritage Archive to publish and index research software independently of conference cycles. For broader methodological principles regarding research software packaging, readers can explore our guide on software engineering in research.

The data table below compares key operational specifications, evaluation deadlines, and artifact badge adoption across these major computer science venues.

Frequently asked questions

What is the primary difference between PLDI and POPL?

While both PLDI and POPL are premier ACM SIGPLAN conferences published in PACMPL, they focus on different aspects of programming language research. POPL (Principles of Programming Languages) focuses primarily on theoretical foundations, formal semantics, type theory, and mathematical proofs. PLDI (Programming Language Design and Implementation) emphasizes practical implementation, compiler optimization algorithms, execution runtime performance, and experimental systems evaluation.

Are papers published at PLDI open access?

Yes. Since 2017, all research papers accepted to PLDI are published in the journal Proceedings of the ACM on Programming Languages (PACMPL). PACMPL is a Gold Open Access journal published under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. ACM SIGPLAN covers all publication costs, ensuring that neither authors nor readers pay access fees or mandatory Article Processing Charges.

What is the role of the PLDI Artifact Evaluation Committee?

The Artifact Evaluation Committee (AEC) is an independent body of computer science researchers, postdocs, and senior graduate students who review the software, datasets, and virtual machines accompanying accepted papers. The committee tests whether artifacts are executable, well-documented, complete, and capable of reproducing the quantitative and qualitative claims presented in the paper, subsequently awarding formal ACM badges.

Is artifact evaluation mandatory for PLDI authors?

Artifact evaluation is typically optional but strongly encouraged for authors of accepted PLDI papers. A paper’s acceptance into the conference program is determined by the peer review committee prior to artifact evaluation. Submitting an artifact allows authors to earn official ACM reproducibility badges that appear directly on the published article in the ACM Digital Library.

What are the criteria for the Artifacts Evaluated – Reusable badge?

To receive the Reusable badge, an artifact must exceed basic operational functionality. It must feature clean software architecture, well-documented source code, clear API definitions, comprehensive testing suites, and modular packaging that allows other researchers to easily modify, extend, or integrate the software into new research projects.

Can an artifact evaluation failure revoke a paper’s acceptance?

No. The artifact evaluation process is intentionally decoupled from paper acceptance decisions. A paper that fails artifact evaluation or encounters severe software execution errors will still be published in PACMPL. However, the publication will not be awarded the corresponding ACM artifact badges, and the evaluation report will remain on record with the committee.

Sources

• ACM SIGPLAN PLDI Official Portal and Steering Committee Policies — https://www.sigplan.org/Conferences/PLDI/ (accessed 2026-11-14)
• Proceedings of the ACM on Programming Languages (PACMPL) — https://dl.acm.org/journal/pacmpl (accessed 2026-11-14)
• ACM Artifact Review and Badging Current Policy — https://www.acm.org/publications/policies/artifact-review-and-badging-current (accessed 2026-11-14)
• ACM SIGPLAN Artifact Evaluation Committee Guidelines — https://www.sigplan.org/Artifacts/ (accessed 2026-11-14)
• ACM SIGPLAN Empirical Evaluation Guidelines — https://www.sigplan.org/Resources/EmpiricalEvaluation/ (accessed 2026-11-14)