Researchers from Figma and Stanford University introduced MotionGraphicsBench on November 10, 2026, establishing an evaluation benchmark that assesses how effectively large language models animate static vector graphics using deterministic, programmatic verification rather than subjective human scoring or vision-language model judges. This benchmark isolates motion generation from graphic design across 100 expert-curated tasks, providing university laboratories and multimodal artificial intelligence researchers with reproducible, fine-grained evaluation code.
Isolating motion generation through a designer-animator workflow
Evaluating automated animation systems has historically suffered from a significant methodological bottleneck: confounding static asset creation with temporal motion synthesis. In standard text-to-video or text-to-animation benchmarks, such as SVG-Bench or VGBench, models are tasked with generating visual assets from scratch alongside their animations. When a generation fails, evaluators cannot readily determine whether the model lacked visual design capabilities or failed to coordinate the spatio-temporal physics of the requested movement.
MotionGraphicsBench resolves this ambiguity by mirroring a professional production workflow where a designer supplies completed visual assets and an animator adds motion while preserving the original design. Derived from 100 real-world motion designs in the LottieAnimation-660K corpus, each benchmark task provides models with two inputs: a complete input static Scalable Vector Graphics (SVG) file containing all necessary geometry—including layers that may initially be invisible—and a natural-language motion edit prompt. Evaluated models are instructed to animate the supplied visual elements without generating missing shapes or altering fundamental design assets.
The benchmark prompts emphasize relative spatial and temporal relationships rather than hardcoded coordinates or frame timestamps, reflecting how human designers communicate creative intent. Across the 100 tasks, the benchmark defines a taxonomy of 17 animation requirement types spanning spatial structure, temporal sequencing, and appearance or visibility modifications. Individual prompts combine between 7 and 14 distinct requirement types, challenging models to coordinate multi-part kinematic chains, such as opposing scissor blades, oscillating pendulums, or synchronized status indicators.
Deterministic programmatic verification versus VLM judges
A central innovation of MotionGraphicsBench is its rejection of both single-reference pixel comparisons and vision-language model (VLM) judges in favor of deterministic, executable Python programs. Rigid pixel-to-pixel comparison against a reference animation penalizes valid stylistic variations in timing, easing, or trajectory that nonetheless satisfy the user’s prompt. Conversely, relying on frontier VLMs as evaluators introduces stochastic scoring variance, high inference costs, and opaque reasoning that obscures which specific physical constraints failed.
To establish objective ground truth, each of the 100 tasks is paired with a standalone Python verification program running locally on CPU. Evaluated models generate JavaScript animation scripts using the standard GreenSock Animation Platform (GSAP) library, which are rendered via a headless browser into sequential SVG frames. The task-specific verification script ingests these frames and executes a series of human-readable geometric and temporal assertions, measuring element transformations, relative distances, bounding-box overlaps, and visibility transitions over time. An animation is certified as successful only if every programmatic check passes.
To construct these verification programs without human authoring bottlenecks, the researchers deployed a two-agent multi-agent pipeline using GPT-6 Astra. One agent authored candidate verification checks while a synthesizer agent generated targeted positive and negative edge cases from 1,585 human-labeled animations. On a balanced held-out test suite of 180 animations across Gemini 3.1 Pro, Gemma 4 31B IT, GPT-5.5, and Claude Sonnet 5, the programmatic verification suite achieved 93.3% raw agreement with human consensus judgments, substantially outperforming a GPT-6 Astra VLM baseline judge that achieved 83.3% agreement.
Evaluation findings and common failure modes across 32 models
The initial release includes evaluations of 32 proprietary and open-weight models across 47 configurations and reasoning-effort tiers from 12 providers. At their highest available reasoning-effort settings, only four models exceeded a 50% pass@1 success rate, with an overall mean pass@1 of 30.3% across all configurations. GPT-6 Astra at maximum reasoning effort achieved the top pass@1 score at 67.7%. Open-weight architectures demonstrated competitive reasoning capabilities, led by GLM 5.3 at 39.8%, Qwen 3.8 at 39.4%, and Qwen 3.8 Flash-Next at 34.0%, outperforming several proprietary alternatives.
When evaluated across 32 independent attempts per task (pass@32) on a subset of 15 models, mean success increased from 36.7% to 73.7%, with pass@32 scores ranging from 38.0% to 95.0%. Programmatic inspection of failed attempts revealed three primary failure categories:
- Reference-pose mismatches: Elements failed to reach designated terminal or intermediate geometric states, such as scissor blades stopping short of an overlapping closed position.
- Motion trajectory mismatches: Elements followed incorrect directional vectors or inverted movement axes, such as earbuds translating downward rather than ascending outward along a mirrored trajectory.
- Event coordination mismatches: Dependent multi-element sequences executed out of order or failed to maintain synchronous timing, such as status lights remaining illuminated after gauge needles departed trigger ranges.
Relevance for academic research offices and multimodal AI evaluation
For university research offices, computer science departments, and AI safety institutes, MotionGraphicsBench represents a vital paradigm shift toward programmatic evaluation in generative artificial intelligence. The prevalence of subjective evaluation metrics and opaque LLM-as-a-judge scoring has created significant challenges for scientific reproducibility across multimodal benchmarks. By substituting probabilistic scoring with executable code assertions, the benchmark demonstrates how academic laboratories can apply rigorous principles of code verification to complex generative outputs.
This deterministic approach directly supports university governance frameworks for conducting reproducible AI experiments and developing standardized synthetic benchmarks. Researchers can evaluate model checkpoints locally on commodity CPU infrastructure without incurring unpredictable API fees or dealing with non-deterministic evaluation drift over time. Furthermore, the benchmark’s open release aligns with community standards embodied by the UK AI Safety Institute’s Inspect evaluation framework and reinforces the FAIR4RS principles for research software by ensuring all task data, prompts, and verification scripts are fully accessible, inspectable, and reusable.
Sources
- Hugging Face Blog, “MotionGraphicsBench: Evaluating Motion Graphics Generation with Programmatic Verification,” published November 10, 2026. https://huggingface.co/blog/figma-ai/motion-graphics-bench
- Ma, J., Jindal, A. K., Wang, C., and Bhakthavatsalam, S., “MotionGraphicsBench: Evaluating Motion Graphics Generation with Programmatic Verification,” arXiv:2611.10592 [cs.CV], submitted November 7, 2026. https://arxiv.org/abs/2611.10592
- Figma AI, “MotionGraphicsBench Dataset and Verification Suite,” Hugging Face Datasets repository, accessed November 14, 2026. https://huggingface.co/datasets/figma-ai/motion-graphics-bench
- MotionGraphicsBench Project Team, “MotionGraphicsBench Tasks and Interactive Viewer,” project documentation, accessed November 14, 2026. https://motion-graphics-bench.github.io/
