ArXiv Sanity

ArXiv Sanity (and its modern rewrite, ArXiv Sanity Lite) is an open-source literature discovery and curation platform designed to help researchers track, filter, and rank preprint publications on arXiv. Conceived and developed by computer scientist Andrej Karpathy, the tool provides automated preprint ingestion, personalized tagging, and machine learning–driven recommendations based on abstract text similarity.

Preprint volume in artificial intelligence and machine learning has expanded dramatically, overwhelming conventional chronological RSS feeds and broad subject alerts. Within contemporary workflows in machine learning and related artificial intelligence disciplines, tools like ArXiv Sanity allow scholars to establish automated relevance filters rather than manually reviewing hundreds of daily submissions.

What ArXiv Sanity Is

ArXiv Sanity is a specialized web interface and backend pipeline built to tame information overload on the arXiv repository. The platform periodically ingests new preprint metadata directly from the official arXiv API, extracts text features from titles and abstracts, and lets researchers curate personal libraries using custom topic tags.

Rather than relying on proprietary citation indices or collaborative filtering from commercial networks, ArXiv Sanity operates on a transparent content-based recommendation paradigm. When an investigator tags papers reflecting their specific research focus, the system fits a linear Support Vector Machine (SVM) against term frequency–inverse document frequency (TF-IDF) feature vectors of paper abstracts. This model then calculates similarity scores across unread preprints to surface the most relevant newly submitted manuscripts.

Who Runs It and What It Covers

ArXiv Sanity was originally created in 2016 by Andrej Karpathy while completing his doctoral studies at Stanford University and subsequently working at OpenAI and Tesla. The original system, known as ArXiv Sanity Preserver, indexed over 25,000 papers across core computer science and statistics categories, specifically focusing on:

• Computer Vision (cs.CV)
• Artificial Intelligence (cs.AI)
• Computation and Language (cs.CL)
• Machine Learning (cs.LG)
• Neural and Evolutionary Computing (cs.NE)
• Machine Learning in Statistics (stat.ML)

In November 2021, Karpathy released arxiv-sanity-lite, an open-source, from-scratch redesign. While the initial platforms were hosted publicly on community domains (arxiv-sanity.com and arxiv-sanity-lite.com), active centralized public hosting has transitioned to an open-source self-hosting model. Today, the repository is maintained openly on GitHub under the MIT license, where researchers, laboratory groups, and academic institutions clone and run local instances customized to their own disciplinary subsets.

How to Search and Use ArXiv Sanity

Researchers deploying and using ArXiv Sanity Lite locally follow a straightforward, modular execution pipeline governed by three Python components:

1. Database Ingestion via the arXiv API (arxiv_daemon.py): The ingestion daemon queries the arXiv API at scheduled intervals (typically via a cron job). To maintain strict compliance with arXiv API usage guidelines, requests are paced with rate-limiting pauses, retrieving preprint IDs, titles, abstracts, authors, and submission timestamps.
2. Feature Extraction and Model Computation (compute.py): Once new records are detected, the computation script tokenizes paper abstracts and builds a global TF-IDF matrix using scikit-learn. For every user account and tag category, the script trains a regularized linear Support Vector Machine using tagged papers as positive instances and randomly sampled untagged manuscripts as background negative instances.
3. Local Interface Launch (serve.py): The web application is served through a lightweight Flask server. Researchers launch the server locally via standard terminal commands (export FLASK_APP=serve.py; flask run), connecting via their web browser to access the local dashboard.
4. Browsing and Sorting: Within the user interface, investigators can browse newly polled manuscripts, sort preprints by submission date, or sort results by SVM similarity scores relative to specific active tags.
5. Tagging Papers: Clicking the tag icon on a paper immediately adds the manuscript to the researcher’s local library and marks it for inclusion in subsequent SVM retraining cycles.
6. Automated Email Summaries: Researchers configuring SMTP credentials can enable automated daily email digests summarizing top-scoring unread preprints for their active tags.

Strengths and Limits

Understanding where ArXiv Sanity excels and where complementary bibliographic tools are required helps research groups construct efficient literature surveillance pipelines.

Key Strengths

• Full Algorithmic Transparency: Unlike black-box recommendation engines deployed on commercial platforms, ArXiv Sanity uses a clearly documented TF-IDF and SVM pipeline where recommendations directly trace to explicit user tags and abstract vocabularies.
• Zero Tracking and Commercial Ad Overhead: Running locally guarantees complete data privacy for active grant proposals and ongoing investigations, preventing commercial tracking of literature interests.
• Lightweight Codebase: The entire ArXiv Sanity Lite software suite consists of fewer than 1,000 lines of standard Python, making it accessible for researchers to audit, modify, and extend.
• Custom Category Ingestion: By adjusting query parameters in arxiv_daemon.py, laboratories can repurpose the tool beyond machine learning to track physics, mathematics, quantitative biology, or economics.

Operational Limits

• Abstract-Only Indexing: The recommendation algorithm evaluates titles and abstracts; it does not parse full-text PDFs, experimental tables, or algorithmic code repositories.
• Self-Hosting Requirement: Because centralized public hosting is deprecated, users must possess basic familiarity with Python environments, virtual environments, and local cron orchestration.
• Absence of Citation Graphs: ArXiv Sanity focuses purely on incoming preprint stream filtering rather than multi-generational citation tracking, bibliographic co-citation analysis, or journal metrics.

Alternatives to ArXiv Sanity

Researchers in AI and computer science utilize a spectrum of literature discovery and curation systems. The table below compares ArXiv Sanity Lite with major academic literature platforms across key research criteria:

Platform	Primary Focus	Recommendation Mechanism	Hosting Model	Community Features
ArXiv Sanity Lite	arXiv preprint filtering and triage	TF-IDF text features with linear SVM per tag	Self-hosted (local Python/Flask)	Single-user / local tags
Hugging Face Daily Papers	Daily AI preprint curation and debate	Community upvotes, author claims, and editorial review	Centralized cloud platform	Public comment threads, artifact linking
AlphaXiv	Interactive arXiv manuscript discussion	Subject category browsing and trending activity	Centralized cloud platform	Inline document annotations and peer comments
Semantic Scholar	Cross-disciplinary bibliographic search	Neural embedding and citation graph modeling	Centralized cloud platform	Citation velocity, influence metrics, library feeds
Connected Papers	Visual bibliographic co-citation analysis	Co-citation and bibliographic coupling similarity	Centralized cloud platform	Visual 2D graph mapping of paper neighborhoods

When selecting literature discovery infrastructure, research teams often evaluate trade-offs between disciplinary scope, such as differentiating machine learning vs artificial intelligence paradigms, and organizational compliance benchmarks like the NIST AI RMF vs ISO 42001 frameworks. Preprints discovered through ArXiv Sanity can subsequently be evaluated against institutional publishing standards, including the Elsevier generative AI authorship policy.

Frequently Asked Questions

Is the original ArXiv Sanity Preserver still operating online?

The original public domain (arxiv-sanity.com) and its subsequent test instance (arxiv-sanity-lite.com) are no longer maintained as active public web services. Researchers wishing to use ArXiv Sanity run the open-source arxiv-sanity-lite codebase on their local machines or institutional servers.

How does ArXiv Sanity rank preprints for a user?

ArXiv Sanity Lite uses an scikit-learn Support Vector Machine (linear SVM) trained on TF-IDF features extracted from paper abstracts. Papers explicitly tagged by the user serve as positive training examples, while randomly selected background preprints serve as negative examples. The model outputs a continuous similarity score used to sort unread submissions.

Does ArXiv Sanity download full-text PDFs?

The original 2016 ArXiv Sanity Preserver downloaded PDFs to extract text and render first-page thumbnail images. In contrast, the modern arxiv-sanity-lite architecture operates exclusively on preprint metadata and abstracts retrieved via the arXiv API, significantly reducing local storage and bandwidth requirements.

Can ArXiv Sanity monitor subjects outside machine learning?

Yes. Although pre-configured with arXiv computer science and statistics categories (such as cs.CV, cs.AI, cs.CL, and stat.ML), researchers can modify the category strings in arxiv_daemon.py to poll any category supported by arXiv, including astrophysics, quantum physics, or quantitative biology.

Sources

• Karpathy, A. (2021). arxiv-sanity-lite. GitHub repository: https://github.com/karpathy/arxiv-sanity-lite (Accessed: 2026-11-07).
• Karpathy, A. (2016). arxiv-sanity-preserver. GitHub repository: https://github.com/karpathy/arxiv-sanity-preserver (Accessed: 2026-11-07).
• arXiv. (2026). arXiv API User’s Manual. arXiv.org: https://info.arxiv.org/help/api/index.html (Accessed: 2026-11-07).