Skip to main content
v2026.11,986 entries · CC-BY 4.0
CASRAIAsk CASRAI · AI assistant for researchAsk a research question. Answers come with their sources.Funder rules, compliance, methods and publishing. Answers link the sources they rest on.

Editorial · CASRAI · AI and ML research outputs

Stanford Researchers Open-Source Memorilla Latent Memory Module for Frozen LLMs

Stanford researchers open-sourced Memorilla, a latent memory module compressing large document collections into 16 tokens for frozen language models.

Published 16 Nov 2026· 5 minute read

Ask CASRAI · free to try

Ask about this story

2 free questions a day, no account.

An AI assistant for research: funder rules, compliance, methods and publishing. It links the sources behind its answers and labels answers that come from a web search.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays open to read, with no account and no card.

CASRAI is the reference for research administration — bookmark it for the next question.

On November 14, 2026, researchers from Stanford University’s SNAP group, in collaboration with computer scientists from MIT, the University of Pennsylvania, and the University of Toronto, open-sourced Memorilla, an architecture that provides frozen large language models with persistent, query-conditioned access to extensive document collections. By translating document corpora into compact latent vectors rather than packing raw text into prompts, Memorilla compresses context by over a hundredfold while preserving critical facts and inter-document relationships across long-horizon research and agent workflows.

Open-Source Release and Core Technical Architecture

The Memorilla system, published under the permissive Apache 2.0 license, was developed by doctoral researchers Adibvafa Fallahpour, Parsa Idehpour, Vignesh Kothapalli, Nikita Mounier, Shirley Wu, and Shayan Pardis alongside Professor Jure Leskovec at the Stanford Network Analysis Project (SNAP). The codebase is hosted on GitHub at https://github.com/snap-stanford/memorilla, with pre-trained checkpoints on Hugging Face at https://huggingface.co/memorilla and interactive documentation at https://memory.stanford.edu/. Theoretical foundations and benchmark validations appear in the preprint Memorilla: Latent Semantic Memory for LLMs on arXiv (arXiv:2610.21375).

Rather than requiring model parameter fine-tuning or expanding active context windows, Memorilla acts as a modular bridge connecting frozen document encoders to frozen text decoders. The framework meets established open-source model criteria, distributing complete training recipes, attention profiling harnesses, and evaluation suites.

The operational pipeline consists of three computational stages:

  1. Query-Agnostic Pre-Encoding: Large document collections, interaction histories, or research repositories are embedded once using a frozen encoder model, Qwen3-Embedding-4B. Pre-encoding decouples corpus ingestion from query time, amortizing embedding overhead across subsequent user sessions.
  2. Query-Conditioned Latent Compression: Upon receiving an inference query, a lightweight trainable cross-attention memory module (~194 million parameters) processes the stored embeddings. The module extracts query-relevant context and compresses the entire collection into exactly 16 latent memory tokens.
  3. Frozen Decoder Generation: The 16 memory vectors are projected into the input space of a frozen foundation model decoder (such as Qwen3-8B). The decoder processes these soft tokens alongside the user prompt, producing grounded responses without loading raw source documents into the context window.

Empirical Benchmarks and Latency Profiling

The authors evaluated Memorilla across knowledge-intensive question answering, long-horizon conversational tracking, narrative synthesis, and agentic reinforcement learning. Across these tasks, latent compression matched or exceeded the accuracy of full-context prompting while cutting token footprints by orders of magnitude.

Evaluation Metric / Benchmark Standard Baseline Method Memorilla Performance Measured Efficiency Gain
FactKG Verification Accuracy 80.43% (Full-context prompt) 84.33% accuracy 108× fewer input tokens
PersonaMem-v2 Long Tracking Standard RAG / Mem0 baseline State-of-the-art multi-session recall ~2,000× token compression (107 chats)
NarrativeQA Long Context Truncated / sliding window context Full narrative synthesis Over 4,000× context reduction
TextWorld RL Success Rate 18.6% task completion 24.2% task completion Continuous trajectory learning
Inference Latency (512 docs / 128k tokens) 19.9 seconds (27× slowdown) 0.74 seconds latency Constant decoder overhead

On the FactKG factual reasoning benchmark, Memorilla achieved an accuracy of 84.33%, outperforming full-document prompting (80.43%) while consuming 108 times fewer document tokens. In multi-session tracking on PersonaMem-v2, where an agent recalls user context across 107 historical conversations (~32,000 tokens), Memorilla synthesized biographical clues into 16 latent tokens, avoiding generic responses. In TextWorld reinforcement learning environments, providing compressed trajectory history elevated average task completion from 18.6% to 24.2%. Independent analysis by MarkTechPost corroborated these findings, noting stable sub-second latency across expanded corpora.

Memory Caching and Compute Scaling

Hardware profiling on NVIDIA H100 and B200 GPUs highlighted dramatic efficiency advantages. When scaling a corpus from 1 to 512 documents (~128,000 tokens), raw prompt stuffing increased latency from 0.75 seconds to 19.9 seconds—a 27-fold slowdown. Memorilla maintained a response latency of 0.74 seconds on an H100 GPU, since the frozen decoder receives exactly 16 tokens regardless of corpus size.

To support massive enterprise collections, the developers introduced an optional key-value memory cache. During ingestion, the system precomputes and caches cross-attention keys and values. Incoming queries attend over these stored tensors rather than re-evaluating raw embeddings. On an NVIDIA B200 GPU, querying one million documents took 8.1 milliseconds (5.0 milliseconds with torch.compile). Querying four million documents (~one billion tokens) required 33 milliseconds and 80 GiB of VRAM, fitting within a single accelerator. These benchmarks establish clear baselines for institutional compute (FLOPs estimate) and memory capacity budgeting.

Context Window Trade-Offs and Retrieval Bottlenecks

Memorilla addresses foundational constraints affecting retrieval-augmented generation (RAG) and long-context models. While modern foundation models offer context windows reaching 128,000 to one million tokens, feeding voluminous source literature directly into prompts incurs steep self-attention computational overhead, high inference latency, and attentional degradation (the ‘lost-in-the-middle’ effect).

Conventional chunk retrieval pipelines rely on sparse keyword matching or dense vector similarity to fetch top-k passages, frequently fragmenting multi-document evidence. Text-based memory systems like Mem0 and MemoRAG rely on dynamic knowledge graphs or generated summaries, but text abstraction risks dropping critical nuances and requires multi-turn prompt chaining. By operating within continuous latent embedding space, Memorilla preserves diffuse contextual signals without intermediate text distillation, supporting computational reproducibility through deterministic latent representations.

Significance for University Computing and Research Teams

The release provides several operational benefits for academic institutions, libraries, and high-performance computing centers:

  • Democratized Large-Corpus Synthesis: Academic research teams facing GPU budget constraints can query corpora spanning one billion tokens on a single 80 GB accelerator, avoiding the multi-node infrastructure required by monolithic long-context decoders.
  • Enhanced Data Custody and Compliance: Institutional governance protocols frequently restrict sharing sensitive research outputs, patient records, or proprietary disclosures with external cloud APIs. Local Memorilla deployments keep all corpus indexing and inference behind institutional firewalls.
  • Standardized Experimental Frameworks: Open-source weights and reproducible pipelines support reproducible AI experiment methodologies, allowing researchers to study memory consolidation without exposure to proprietary API version drift.

Implementation Requirements and Future Directions

Memorilla requires Linux, Python 3.11, PyTorch, CUDA, and FlashAttention-2, with inference also supported via vLLM. The three-stage training pipeline (Stage 1 English Wikipedia, Stage 2 Mixture, Stage 3 Multitask) completes in roughly 19 hours on four NVIDIA H100 80GB GPUs using 310 GiB of precomputed embeddings. Future milestones include scaling latent memory modules to 70B+ parameter decoders and integrating multimodal encoders for scientific figures and structured tabular data.

Sources

  • Stanford SNAP Group, “Memorilla: Latent Semantic Memory for LLMs (Source Code Repository),” GitHub, 2026-11-15, https://github.com/snap-stanford/memorilla
  • Adibvafa Fallahpour, Parsa Idehpour, Vignesh Kothapalli, Nikita Mounier, Shirley Wu, Shayan Pardis, and Jure Leskovec, “Memorilla: Latent Semantic Memory for LLMs,” arXiv preprint arXiv:2610.21375, 2026-11-15, https://arxiv.org/abs/2610.21375
  • Stanford University Network Analysis Project (SNAP), “Memorilla Project Portal and Documentation,” Stanford University, 2026-11-15, https://memory.stanford.edu/
  • Hugging Face Community, “Memorilla Model Collection and Embeddings,” Hugging Face Repository, 2026-11-15, https://huggingface.co/memorilla
  • MarkTechPost Media, “Stanford Researchers Open-Source Memorilla: A 194M Latent Semantic Memory Module for Frozen LLMs,” MarkTechPost Technical Review, 2026-11-15, https://www.marktechpost.com

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →