On November 14, 2026, researchers from Stanford University’s SNAP group, in collaboration with computer scientists from MIT, the University of Pennsylvania, and the University of Toronto, open-sourced Memorilla, an architecture that provides frozen large language models with persistent, query-conditioned access to extensive document collections. By translating document corpora into compact latent vectors rather than packing raw text into prompts, Memorilla compresses context by over a hundredfold while preserving critical facts and inter-document relationships across long-horizon research and agent workflows.
Open-Source Release and Core Technical Architecture
The Memorilla system, published under the permissive Apache 2.0 license, was developed by doctoral researchers Adibvafa Fallahpour, Parsa Idehpour, Vignesh Kothapalli, Nikita Mounier, Shirley Wu, and Shayan Pardis alongside Professor Jure Leskovec at the Stanford Network Analysis Project (SNAP). The codebase is hosted on GitHub at https://github.com/snap-stanford/memorilla, with pre-trained checkpoints on Hugging Face at https://huggingface.co/memorilla and interactive documentation at https://memory.stanford.edu/. Theoretical foundations and benchmark validations appear in the preprint Memorilla: Latent Semantic Memory for LLMs on arXiv (arXiv:2610.21375).
Rather than requiring model parameter fine-tuning or expanding active context windows, Memorilla acts as a modular bridge connecting frozen document encoders to frozen text decoders. The framework meets established open-source model criteria, distributing complete training recipes, attention profiling harnesses, and evaluation suites.
The operational pipeline consists of three computational stages:
- Query-Agnostic Pre-Encoding: Large document collections, interaction histories, or research repositories are embedded once using a frozen encoder model, Qwen3-Embedding-4B. Pre-encoding decouples corpus ingestion from query time, amortizing embedding overhead across subsequent user sessions.
- Query-Conditioned Latent Compression: Upon receiving an inference query, a lightweight trainable cross-attention memory module (~194 million parameters) processes the stored embeddings. The module extracts query-relevant context and compresses the entire collection into exactly 16 latent memory tokens.
- Frozen Decoder Generation: The 16 memory vectors are projected into the input space of a frozen foundation model decoder (such as Qwen3-8B). The decoder processes these soft tokens alongside the user prompt, producing grounded responses without loading raw source documents into the context window.
Empirical Benchmarks and Latency Profiling
The authors evaluated Memorilla across knowledge-intensive question answering, long-horizon conversational tracking, narrative synthesis, and agentic reinforcement learning. Across these tasks, latent compression matched or exceeded the accuracy of full-context prompting while cutting token footprints by orders of magnitude.
| Evaluation Metric / Benchmark | Standard Baseline Method | Memorilla Performance | Measured Efficiency Gain |
|---|---|---|---|
| FactKG Verification Accuracy | 80.43% (Full-context prompt) | 84.33% accuracy | 108× fewer input tokens |
| PersonaMem-v2 Long Tracking | Standard RAG / Mem0 baseline | State-of-the-art multi-session recall | ~2,000× token compression (107 chats) |
| NarrativeQA Long Context | Truncated / sliding window context | Full narrative synthesis | Over 4,000× context reduction |
| TextWorld RL Success Rate | 18.6% task completion | 24.2% task completion | Continuous trajectory learning |
| Inference Latency (512 docs / 128k tokens) | 19.9 seconds (27× slowdown) | 0.74 seconds latency | Constant decoder overhead |
On the FactKG factual reasoning benchmark, Memorilla achieved an accuracy of 84.33%, outperforming full-document prompting (80.43%) while consuming 108 times fewer document tokens. In multi-session tracking on PersonaMem-v2, where an agent recalls user context across 107 historical conversations (~32,000 tokens), Memorilla synthesized biographical clues into 16 latent tokens, avoiding generic responses. In TextWorld reinforcement learning environments, providing compressed trajectory history elevated average task completion from 18.6% to 24.2%. Independent analysis by MarkTechPost corroborated these findings, noting stable sub-second latency across expanded corpora.
Memory Caching and Compute Scaling
Hardware profiling on NVIDIA H100 and B200 GPUs highlighted dramatic efficiency advantages. When scaling a corpus from 1 to 512 documents (~128,000 tokens), raw prompt stuffing increased latency from 0.75 seconds to 19.9 seconds—a 27-fold slowdown. Memorilla maintained a response latency of 0.74 seconds on an H100 GPU, since the frozen decoder receives exactly 16 tokens regardless of corpus size.
To support massive enterprise collections, the developers introduced an optional key-value memory cache. During ingestion, the system precomputes and caches cross-attention keys and values. Incoming queries attend over these stored tensors rather than re-evaluating raw embeddings. On an NVIDIA B200 GPU, querying one million documents took 8.1 milliseconds (5.0 milliseconds with torch.compile). Querying four million documents (~one billion tokens) required 33 milliseconds and 80 GiB of VRAM, fitting within a single accelerator. These benchmarks establish clear baselines for institutional compute (FLOPs estimate) and memory capacity budgeting.
Context Window Trade-Offs and Retrieval Bottlenecks
Memorilla addresses foundational constraints affecting retrieval-augmented generation (RAG) and long-context models. While modern foundation models offer context windows reaching 128,000 to one million tokens, feeding voluminous source literature directly into prompts incurs steep self-attention computational overhead, high inference latency, and attentional degradation (the ‘lost-in-the-middle’ effect).
Conventional chunk retrieval pipelines rely on sparse keyword matching or dense vector similarity to fetch top-k passages, frequently fragmenting multi-document evidence. Text-based memory systems like Mem0 and MemoRAG rely on dynamic knowledge graphs or generated summaries, but text abstraction risks dropping critical nuances and requires multi-turn prompt chaining. By operating within continuous latent embedding space, Memorilla preserves diffuse contextual signals without intermediate text distillation, supporting computational reproducibility through deterministic latent representations.
Significance for University Computing and Research Teams
The release provides several operational benefits for academic institutions, libraries, and high-performance computing centers:
- Democratized Large-Corpus Synthesis: Academic research teams facing GPU budget constraints can query corpora spanning one billion tokens on a single 80 GB accelerator, avoiding the multi-node infrastructure required by monolithic long-context decoders.
- Enhanced Data Custody and Compliance: Institutional governance protocols frequently restrict sharing sensitive research outputs, patient records, or proprietary disclosures with external cloud APIs. Local Memorilla deployments keep all corpus indexing and inference behind institutional firewalls.
- Standardized Experimental Frameworks: Open-source weights and reproducible pipelines support reproducible AI experiment methodologies, allowing researchers to study memory consolidation without exposure to proprietary API version drift.
Implementation Requirements and Future Directions
Memorilla requires Linux, Python 3.11, PyTorch, CUDA, and FlashAttention-2, with inference also supported via vLLM. The three-stage training pipeline (Stage 1 English Wikipedia, Stage 2 Mixture, Stage 3 Multitask) completes in roughly 19 hours on four NVIDIA H100 80GB GPUs using 310 GiB of precomputed embeddings. Future milestones include scaling latent memory modules to 70B+ parameter decoders and integrating multimodal encoders for scientific figures and structured tabular data.
Sources
- Stanford SNAP Group, “Memorilla: Latent Semantic Memory for LLMs (Source Code Repository),” GitHub, 2026-11-15, https://github.com/snap-stanford/memorilla
- Adibvafa Fallahpour, Parsa Idehpour, Vignesh Kothapalli, Nikita Mounier, Shirley Wu, Shayan Pardis, and Jure Leskovec, “Memorilla: Latent Semantic Memory for LLMs,” arXiv preprint arXiv:2610.21375, 2026-11-15, https://arxiv.org/abs/2610.21375
- Stanford University Network Analysis Project (SNAP), “Memorilla Project Portal and Documentation,” Stanford University, 2026-11-15, https://memory.stanford.edu/
- Hugging Face Community, “Memorilla Model Collection and Embeddings,” Hugging Face Repository, 2026-11-15, https://huggingface.co/memorilla
- MarkTechPost Media, “Stanford Researchers Open-Source Memorilla: A 194M Latent Semantic Memory Module for Frozen LLMs,” MarkTechPost Technical Review, 2026-11-15, https://www.marktechpost.com







