Researchers Release ArchSpace Preprint for Tracking and Evaluating LLM Architecture Design

On November 12, 2026, researchers from Shanghai AI Laboratory, East China Normal University, Fudan University, and Shanghai Jiao Tong University released the ArchSpace preprint on arXiv, introducing an open-source evaluation framework and collaborative registry designed to systematically record, review, and reference large language model architectural designs.

Addressing the Industry-Academia Divide in LLM Architecture Research

The design of a large language model architecture directly dictates its training throughput, inference efficiency, and downstream task performance. In an arXiv preprint titled ‘ArchSpace: Recording, Reviewing, and Referencing LLM Architecture Research’ (arXiv:2611.17332), lead author Jie Wang and colleagues highlight a widening imbalance in how architectural advances are documented and disseminated across the machine learning community.

While leading industrial laboratories—including teams behind open-weight model families such as DeepSeek, Qwen, and Kimi—frequently incorporate novel architectural components into production models, public technical reports rarely document the iterative search process, the architectural alternatives explored, or the specific design choices that failed. Conversely, academic research groups face acute computational limitations that often restrict architectural experiments to small model scales and disparate training baselines. According to an empirical literature survey conducted by the authors, academic publications represent approximately 33% of LLM architecture papers but receive less than 10% of total citations, reflecting significant difficulty in demonstrating that small-scale academic findings generalize to large-scale deployment.

ArchSpace proposes to bridge this gap by establishing a shared public repository where architectural hypotheses, full training configurations, and empirical validation metrics are openly documented and subjected to community review.

The Two-Phase ArchSpace Validation Framework

To establish credible comparisons without requiring prohibitive computational budgets, the preprint outlines a standardized two-phase evaluation framework designed to separate architectural effects from training confounds.

Phase I: Scaling Ladders and Compute-Optimal Analysis

In Phase I, proposed architectural modifications are benchmarked against standard baselines across a calibrated scaling ladder comprising five distinct training compute budgets: 1 × 10¹⁸, 3 × 10¹⁸, 1 × 10¹⁹, 3 × 10¹⁹, and 1 × 10²⁰ FLOPs. Rather than relying on single-point comparisons at arbitrary parameter sizes, the framework fits empirical scaling laws across these budgets to model the relationship between compute, parameter count, dataset size, and validation loss.

The framework supports three standardized search protocols reflecting different computational resource levels:

• Setting 1 (Baseline-Optimal): Evaluates the new architecture using hyperparameters adopted from the reference baseline, minimizing computational overhead.
• Setting 2 (IsoFLOP Sweeps): Sweeps model parameters and training token counts at each fixed compute budget to determine the architecture-specific optimal allocation.
• Setting 3 (Tuned IsoFLOP): Performs localized learning rate and batch size tuning alongside parameter-data sweeps to ensure fair optimization parity.

Under this protocol, an architectural variation is considered an empirical improvement only if its fitted compute-loss curve establishes a new Pareto frontier relative to reference baselines.

Phase II: Extended Pretraining and Long-Context Mid-Training

Phase II verifies whether architectural advantages identified on the scaling ladder persist through sustained pretraining and specialized training regimes. The framework standardizes pretraining of 1-billion-parameter models on 150 billion tokens sampled from the Dolma 3 Pool, followed by 10 billion tokens of long-context mid-training drawn from the Dolma 3 Longmino Mix.

To maintain rigorous experimental isolation, all comparisons utilize the standardized OLMo 3 tokenizer (allenai/dolma2-tokenizer) and identical validation distributions across eleven distinct evaluation domains. The authors supply complete baseline recipes and fitted scaling parameters for both Qwen3.5 and OLMo3 architectures as community reference points.

Standardizing Architectural Registries for ML Reproducibility

Beyond benchmarking metrics, ArchSpace introduces a structured workflow to record variations across key neural network dimensions, including attention mechanisms (such as grouped-query attention and multi-head attention), normalization layers (RMSNorm and LayerNorm), positional embeddings (rotary position embeddings), and feed-forward configurations (SwiGLU and mixture-of-experts).

The proposed workflow relies on structured GitHub Architecture Proposal issues paired with pull requests that link executable code, configuration files, model checkpoints, training logs, and step-by-step reproduction instructions. Critically, the framework mandates recording negative results and abandoned trials, capturing why specific modifications degraded training stability or failed to improve compute efficiency.

This systematic documentation addresses a pervasive bottleneck in computational reproducibility across machine learning research. By enforcing uniform data mixtures and tokenization, ArchSpace ensures that downstream performance differences reflect genuine architectural improvements rather than unstated advantages in training data provenance or hyperparameter tuning.

Significance for Research Institutions and University Labs

For academic researchers, university research offices, and research computing administrators, the release of the ArchSpace preprint points toward practical methodologies for conducting high-impact machine learning research under finite computational resources.

By enabling researchers to project large-scale performance from low-FLOP scaling ladders, the framework reduces the financial and environmental costs of training unviable architectures at scale. For research institutions allocating institutional high-performance computing (HPC) clusters, adopting standardized validation frameworks offers a transparent mechanism to verify the methodology of reproducible AI experiments before committing institutional compute grants.

Furthermore, structured registries of validated and refuted model architectures support automated literature review and systematic evidence synthesis, enabling research teams to identify viable design directions without repeating undocumented exploratory failures. As preprints of this nature stimulate community evaluation, they provide foundational scaffolding for open, auditable AI and ML research outputs across academia and industry alike.

Sources

• Wang, J., Zhu, T., Kuang, J., Xu, Z., Chai, Y., Zheng, D., Cheng, Y., Ji, T., Lin, Z., Zhang, Q., and Zhou, B. (2026). ArchSpace: Recording, Reviewing, and Referencing LLM Architecture Research. arXiv preprint arXiv:2611.17332 [cs.CL]. Submitted November 12, 2026. Checked 2026-11-14. URL: https://arxiv.org/abs/2611.17332
• arXiv e-print HTML: ArchSpace: Recording, Reviewing, and Referencing LLM Architecture Research (arXiv:2611.17332v1). Published November 12, 2026. Checked 2026-11-14. URL: https://arxiv.org/html/2611.17332v1