CASRAI Dictionary

Tag: bibliometrics tools

  • Dimensions Altmetrics, Scopus & Web of Science: A DORA-Aligned Comparison

    Dimensions altmetrics, Scopus CiteScore, and Web of Science’s Impact Factor answer different questions about the same paper: how much online attention it attracted, how its journal’s four-year citation average compares, and how its two-year citation count compares against a curated index. No single number from any one database satisfies the San Francisco Declaration on Research Assessment (DORA)’s call for multi-indicator, qualitative-plus-quantitative evaluation — which is why research offices increasingly triangulate across all three.

    A citation database is a structured index of scholarly publications and their citation links, used to measure research coverage, impact, and attention across disciplines. Dimensions, Scopus, and Web of Science each build that index differently, and the differences matter directly for institutions trying to run dimensions altmetrics-aware, DORA-compliant assessment rather than single-metric ranking.

    How does coverage differ across Dimensions, Scopus and Web of Science?

    Coverage breadth is the single biggest structural difference between the three databases, and it is measurable rather than a matter of opinion. A 2021 Scientometrics study by Singh, Singh, Karmakar, Leta and Mayr found that Dimensions indexes 82.22% more journals than Web of Science and 48.17% more journals than Scopus, largely because Dimensions ingests preprints, grants, patents, clinical trials, and policy documents alongside conventional journal articles.

    A separate large-scale comparison published in Quantitative Science Studies (Visser, van Eck and Waltman, 2021, MIT Press) benchmarked Scopus, Web of Science, Dimensions, Crossref and Microsoft Academic together and found that Dimensions and Crossref offer the broadest raw coverage, while Scopus and Web of Science retain more curated, higher-quality affiliation and subject metadata. Web of Science’s Core Collection remains the most selective of the three, with editorial evaluation criteria dating to Eugene Garfield’s 1960 Science Citation Index; Scopus, launched by Elsevier in 2004, applies a comparatively more inclusive Content Selection and Advisory Board process.

    The practical implication: a citation count pulled from only one database will systematically undercount or overcount depending on discipline, document type, and region. A 2020 comparison from the German Kompetenznetzwerk Bibliometrie (Stahlschmidt and Hinze) reached the same conclusion — the three sources are not interchangeable, and cross-checking is a foundational bibliometric hygiene step, not an optional extra.

    What metrics does each database produce?

    Each platform has developed its own headline indicator, and none of the three is a like-for-like substitute for the others.

    Database Owner Headline metric Citation window Altmetrics integration
    Dimensions Digital Science Citation counts + linked Altmetric Attention Score No fixed window; article-level Native — shares parent company with Altmetric
    Scopus Elsevier CiteScore; Field-Weighted Citation Impact (FWCI) via SciVal 4-year rolling window PlumX Metrics
    Web of Science Clarivate Journal Impact Factor (JCR) 2-year window (5-year variant available) Article-level usage counts; expanding via Research Intelligence tools

    CiteScore, introduced by Elsevier in 2016, divides all citations a journal receives in a given year by all documents (not only “citable items”) published in the preceding four years, and is published free of charge — a deliberate contrast with the subscription-gated Journal Impact Factor. Field-Weighted Citation Impact normalises a paper’s citations against the world average for its subject, publication year, and document type, where a score of 1.0 represents parity with the global average; this makes FWCI more field-comparable than a raw citation count. The Altmetric Attention Score, meanwhile, is not a citation metric at all — it is a weighted count of online attention (news coverage, policy documents, X/social posts, Wikipedia references, blogs) that Dimensions surfaces natively because Dimensions and Altmetric are both Digital Science products.

    Which database best supports DORA-compliant, multi-indicator assessment?

    DORA, published in 2012 and now signed by thousands of organisations worldwide, asks institutions to stop using journal-based metrics such as the Impact Factor as a proxy for the quality of an individual researcher’s contributions, and instead to consider the value and impact of all research outputs alongside qualitative peer judgement. The 2015 Leiden Manifesto (Hicks, Wouters, de Rijcke and Rafols, published in Nature) added ten operating principles for responsible metrics use, including that quantitative evaluation should support, not replace, qualitative expert assessment.

    All three database vendors now publicly reference these frameworks, but their practical alignment differs. Digital Science, Dimensions’ parent company, is listed on DORA’s public signatory register, and Dimensions’ native pairing with Altmetric gives assessors an attention-based indicator alongside citations without needing a separate subscription. Elsevier has endorsed the Leiden Manifesto and built CiteScore’s open methodology partly in response to its principles. Clarivate likewise cites the Leiden Manifesto in its own responsible-metrics guidance and has begun layering a “Societal Impact Framework” onto Web of Science Research Intelligence to capture impact beyond citation counts.

    None of the three databases is independently DORA-compliant by design — compliance is a property of how an institution uses the data, not of the database itself. A single Impact Factor, CiteScore, or Altmetric Attention Score used alone to rank individuals contradicts DORA regardless of source. Multi-indicator assessment requires combining citation-based indicators from at least one curated database with attention-based indicators and qualitative peer review — which is precisely why UK funders and the Research Excellence Framework have explicitly excluded journal impact factors from submission guidance since 2014, requiring panel-level qualitative judgement instead.

    Where does OpenAlex fit as an open alternative?

    OpenAlex, launched in 2022 by the non-profit OurResearch as a fully open successor to the discontinued Microsoft Academic Graph, has emerged as the fourth reference point in this comparison. Unlike Dimensions, Scopus, and Web of Science, OpenAlex publishes its entire dataset and API without subscription cost, drawing on Crossref, ORCID, and ROR identifiers for disambiguation rather than proprietary matching.

    OpenAlex does not yet match the curated metadata quality or the established institutional trust of Scopus or Web of Science, and it carries no equivalent to the Altmetric Attention Score. But for institutions constrained by licensing budgets, or for bibliometrics tools built on reproducible, auditable pipelines, OpenAlex is increasingly used as a free cross-check against the commercial databases rather than a replacement for them.

    Answer-first questions

    What is Altmetric a measure of?

    Altmetric measures online attention, not citation impact. It tracks mentions of a research output across news media, policy documents, social platforms, blogs, and Wikipedia, then produces a weighted Attention Score. Because it captures engagement that predates or bypasses formal citation, it is treated as complementary to citation-based indicators, not a replacement for them.

    What counts as a good Altmetric score?

    There is no universal threshold, because Attention Scores vary enormously by field, output type, and publication date. As a rough benchmark, Altmetric itself notes that a score above roughly 20 typically outperforms most tracked outputs, but comparisons are only meaningful against similar papers in the same journal and timeframe, never as an absolute cutoff.

    Is Scopus or Web of Science better for research assessment?

    Neither is unconditionally “better” — Scopus offers broader, more geographically diverse journal coverage with a transparent four-year CiteScore, while Web of Science offers deeper historical coverage back to 1900 and the still-widely-recognised Impact Factor. DORA-aligned assessment favours using both alongside non-citation indicators rather than choosing one as authoritative.

    Implications for research offices

    Research administrators selecting or combining these tools should treat the choice as an assessment-design decision, not a procurement afterthought. Three practical consequences follow directly from the coverage and metric differences above:

    • A researcher’s citation count and h-index will differ meaningfully between Dimensions, Scopus and Web of Science — institutions must specify and disclose which source underlies any reported figure.
    • Attention-based data (Altmetric, PlumX) captures policy and public engagement that citation-only databases miss entirely, which matters for funders assessing societal impact pathways.
    • Free, open sources such as OpenAlex are viable supplementary cross-checks, particularly where licensing cost restricts access to all three commercial platforms.

    Conclusion

    The three databases are converging on responsible-metrics language while remaining structurally distinct in coverage, indicator design, and cost. Institutions that want genuinely DORA-compliant, multi-indicator assessment should treat Dimensions, Scopus and Web of Science as complementary evidence sources — pairing at least one citation database with an attention-based indicator and qualitative peer review — rather than defaulting to whichever single number is easiest to pull from a subscription dashboard.

  • What Is Bibliometrics? A Research Office Primer

    Bibliometrics is the quantitative analysis of scholarly publications and the citations between them, used to measure research output, impact and collaboration patterns. For a research office, the practical challenge is rarely gathering these numbers — library systems, funders and university dashboards supply them constantly — but recognising which of the three main types of bibliometrics a given report represents, and what it can and cannot responsibly tell you.

    In its simplest form, bibliometrics is the statistical analysis of books, articles and other publications, most often using citation counts to describe patterns in scholarly communication. That one-line definition, drawn from the OECD’s usage and echoed by university library guides, is the starting point for everything that follows.

    What is bibliometrics?

    Bibliometrics applies statistical methods to bibliographic data — publication counts, citation counts, co-authorship networks and, increasingly, download and mention data — to describe and evaluate scholarly activity. It sits alongside scientometrics, a closely related field that extends the same statistical logic to science and technology output more broadly; in practice research offices treat the two terms as near-synonyms.

    Eugene Garfield, founder of the Institute for Scientific Information and creator of the Science Citation Index in 1964, is widely credited as a founding figure of modern bibliometrics. His citation-indexing work established the infrastructure — later commercialised as Web of Science — that most present-day bibliometric reporting still depends on.

    A metrics report a research office receives is rarely a single “bibliometric score.” It is usually a blend of three distinct analytical modes, and conflating them is the single most common source of misread reports.

    What are the three types of bibliometrics?

    Library and information science distinguishes descriptive, evaluative and relational bibliometrics. Each answers a different question, and each carries a different risk of misinterpretation when applied outside its proper scope.

    Type Core question it answers Typical output Main risk if misread
    Descriptive How much has been published, by whom, where? Publication counts, output by year, discipline or department Treated as a quality signal when it only measures volume
    Evaluative How much impact or influence has that output had? Citation counts, h-index, Journal Impact Factor, Field-Weighted Citation Impact Used to rank individuals directly, ignoring field and career-stage differences
    Relational How are researchers, topics or institutions connected? Co-authorship networks, co-citation maps, research-front clustering Read as a measure of quality rather than of structure or collaboration

    Descriptive bibliometrics is the safest category for research offices to report externally, because it counts rather than judges. Evaluative bibliometrics is the category most prone to misuse — a single h-index or Journal Impact Factor figure says nothing about an individual paper’s quality. Relational bibliometrics is the least familiar to non-specialists but the most useful for identifying emerging collaboration opportunities or research strengths across a department.

    What bibliometric indicators will appear in a metrics report?

    Most institutional metrics reports combine a handful of recurring indicators. Knowing which category each one belongs to prevents a descriptive count being read as an evaluative judgement.

    • Citation count — the raw number of times a work has been cited; evaluative, but highly field- and age-dependent.
    • h-index — an author-level figure meaning a researcher has h publications each cited at least h times; evaluative, and known to disadvantage early-career researchers and those in low-citation-rate fields.
    • Journal Impact Factor (JIF) — the average citations per article in a journal over the preceding two years; a journal-level, not an article-level, indicator.
    • Field-Weighted Citation Impact (FWCI) — a normalised indicator comparing a publication’s citations against the global average for its subject, document type and publication year; a value above 1 indicates above-average performance for that field.
    • Altmetrics — non-citation signals such as policy-document mentions, news coverage, social media activity and downloads, which supplement rather than replace citation-based evaluation.

    These indicators are drawn from different underlying databases, and coverage varies. Web of Science and Scopus apply curated, subscription-based indexing; Google Scholar offers broad, free coverage with less curation; Dimensions links publications to grants and clinical trials on a freemium basis. A report’s headline number can shift depending on which source supplied it.

    How should research offices use bibliometrics responsibly?

    Bibliometrics should inform, not replace, expert judgement. Three widely referenced frameworks set out how research offices can operationalise that principle rather than treat it as an aspiration.

    The San Francisco Declaration on Research Assessment (DORA), launched in 2012, commits signatory institutions to avoid using journal-based metrics such as the Journal Impact Factor in hiring, promotion or funding decisions. Imperial College London, for example, states it has applied this commitment since becoming a DORA signatory in 2017.

    The UK’s Metric Tide review, commissioned by the then Higher Education Funding Council for England (now part of UK Research and Innovation) and published in 2015, set out five principles for responsible metrics: robustness, humility, transparency, diversity and reflexivity. Those five principles remain the reference point most UK research offices cite when drafting internal metrics policies.

    INORMS’ Research Evaluation Working Group publishes the SCOPE framework — Start, Context, Options, Probe, Evaluate — a five-step method research administrators can apply before commissioning or interpreting any metrics report, rather than defaulting to whichever indicator a database happens to surface first.

    • Start by clarifying the purpose of the evaluation before selecting any indicator.
    • Establish the context: discipline, career stage, output type and comparator group.
    • Identify the options available, including qualitative alternatives such as peer review or narrative CVs.
    • Probe the suitability and limitations of each proposed indicator.
    • Evaluate the process itself once the assessment is complete, and refine it for next time.

    Momentum toward narrative-based assessment has also grown outside the UK: the 2022 Coalition for Advancing Research Assessment (CoARA), joined by many Horizon Europe-affiliated funders and institutions, commits signatories to reduce reliance on journal-based and output-count metrics in funding and hiring decisions.

    It is worth distinguishing bibliometrics from contributor-level attribution. Bibliometrics counts citations and outputs; it does not record who did what on a given paper. CASRAI originated the CRediT contributor role taxonomy in 2014 for that separate purpose, and the standard is now stewarded by NISO as ANSI/NISO Z39.104-2022. A research office reconciling a bibliometrics report with authorship disputes should reach for CRediT roles, not citation counts.

    Common questions about bibliometrics

    What is the meaning of bibliometric?

    “Bibliometric” describes any measure derived from the statistical analysis of published scholarly output — most commonly publication counts and citation counts. The term covers the underlying data point (a bibliometric) and the wider field that studies it (bibliometrics), and applies equally to authors, journals, institutions and individual articles.

    What is an example of bibliometrics?

    The h-index is the most commonly cited example: an author with an h-index of 20 has 20 publications that have each received at least 20 citations. Other everyday examples include a journal’s Impact Factor, a department’s annual publication count, and a co-authorship map showing collaboration between institutions.

    What is bibliometrics in simple terms?

    In simple terms, bibliometrics counts and analyses publications and citations to show how much research is being produced and how much attention it receives. It turns scattered publication records into structured evidence — useful for funding reports and CVs, but never a full substitute for reading the work itself.

    Who is the father of bibliometrics?

    Eugene Garfield (1925–2017) is widely regarded as the founding figure of bibliometrics and scientometrics. As founder of the Institute for Scientific Information, he created the Science Citation Index in 1964, establishing the citation-indexing infrastructure that underpins most bibliometric analysis conducted today.

    What this means for research offices

    A metrics report that blends descriptive, evaluative and relational bibliometrics without labelling which is which will inevitably be misread by whoever receives it next — a promotion panel, a funder, or a departmental head. Labelling each figure by type, and pairing evaluative indicators with the field-normalisation context they need, is a low-cost fix most research offices can apply immediately.

    As narrative-assessment frameworks such as CoARA and DORA gain signatories, research offices should expect bibliometric reports to sit alongside, not instead of, qualitative evidence in funding and promotion decisions. Building that dual capability now — clear metrics literacy plus a credible narrative-CV process — will matter more with each assessment cycle, not less.