Skip to main content
v2026.11,986 entries · CC-BY 4.0

CountVectorizer in scikit-learn: Syntax, Example and Output

How to use sklearn CountVectorizer on research text: install, fit_transform, reading the count matrix, key parameters and fixes for common errors.

Written and maintained by CASRAI Editorial Board

Published · Last updated

On this page

CountVectorizer is the scikit-learn class that converts a collection of text documents into a matrix of token counts: one row per document, one column per distinct token, and each cell holds how many times that token appears in that document. You create it, call fit_transform on a list of strings, and read the vocabulary with get_feature_names_out(). The result is a sparse matrix that downstream models, clustering and similarity code can use.

What CountVectorizer is used for in research

Researchers use it as the first numeric step in text analysis: turning abstracts, free-text survey answers, interview transcripts, trial narratives or grant summaries into features. The scikit-learn user guide calls this vectorization and describes the combined steps of tokenization, counting and normalization as a ‘Bag of Words’ or ‘Bag of n-grams’ representation, which ignores word position (scikit-learn user guide, text feature extraction). The documentation lists the output as a sparse matrix of token counts (CountVectorizer API reference).

Typical uses are a count table you can inspect directly, input to a classifier, and the counting stage before weighting. The same documentation notes that a count matrix can be followed by TfidfTransformer when you want weighted rather than raw counts. If you work with feature tables more generally, the guides on scikit-learn and splitting data into training and test sets cover the surrounding workflow, and the guides on OneHotEncoder and MinMaxScaler cover the equivalent steps for categorical and numeric columns.

How to install scikit-learn

CountVectorizer ships inside scikit-learn, so there is nothing separate to install. The official installation page gives pip install -U scikit-learn and recommends doing it inside a virtual environment: python -m venv sklearn-env, then source sklearn-env/bin/activate on Linux and macOS or sklearn-env\Scripts\activate on Windows. To confirm what is installed, run python -m pip show scikit-learn (scikit-learn installation instructions). Check the same page for the Python and dependency versions your release requires.

How to use CountVectorizer step by step

The example below uses four short sentences written for this page, in the style of methods-section text. They are an illustrative corpus, not data from a real study.

  1. Import the class: from sklearn.feature_extraction.text import CountVectorizer.
  2. Put your documents in a list of strings, for example docs = ['Sample size was justified with a power analysis.', 'The power analysis assumed a medium effect size.', 'Data were analysed with a mixed model.', 'A mixed model handled missing data.'].
  3. Create the vectorizer: vectorizer = CountVectorizer(). All arguments are optional and keyword-only.
  4. Learn the vocabulary and count in one call: X = vectorizer.fit_transform(docs).
  5. Read the column labels: vectorizer.get_feature_names_out().
  6. View the counts as an ordinary table: X.toarray(). For large corpora keep X sparse and avoid toarray().

Running these steps on scikit-learn 1.9.1 gives a 4 by 18 matrix with 25 non-zero cells, and the following vocabulary in alphabetical order: analysed, analysis, assumed, data, effect, handled, justified, medium, missing, mixed, model, power, sample, size, the, was, were, with.

How to read the output

fit_transform returns a SciPy sparse matrix. SciPy describes the CSR format as efficient for row slicing and matrix-vector products and slow for column slicing (SciPy csr_matrix reference). The scikit-learn user guide adds that such a matrix is typically more than 99% zeros on real corpora, which is why sparse storage is used.

Rows follow the order of your input list; columns follow the alphabetical order of get_feature_names_out(). For the example corpus, X.toarray() prints:

Document analysed analysis assumed data effect handled justified medium missing mixed model power sample size the was were with
1 0 1 0 0 0 0 1 0 0 0 0 1 1 1 0 1 0 1
2 0 1 1 0 1 0 0 1 0 0 0 1 0 1 1 0 0 0
3 1 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0 1 1
4 0 0 0 1 0 1 0 0 1 1 1 0 0 0 0 0 0 0

Summing each column, X.sum(axis=0), gives corpus-level term frequencies: analysis, data, mixed, model, power, size and with each occur twice, and every other token once. Two further behaviours are worth knowing. vectorizer.vocabulary_['power'] returns 11, the column index of ‘power’. And vectorizer.transform(['Completely novel words']) returns a row of all zeros, because a fitted vectorizer ignores words that were not in the vocabulary it learned. The official guide shows the same behaviour with unseen words.

Which parameters change the counts

The constructor defaults are lowercase=True, stop_words=None, token_pattern=r"(?u)bww+b", ngram_range=(1, 1), analyzer='word', max_df=1.0, min_df=1, max_features=None, binary=False and dtype=numpy.int64 (CountVectorizer API reference). The data table below records what each change does to the number of features on the four-sentence example corpus; every figure was produced by running the code on scikit-learn 1.9.1.

Token pattern and single-letter words

The default tokenizer extracts words of at least two letters, so the article ‘a’ is dropped. The user guide recommends token_pattern=r'bw+b' to keep single characters. On the example corpus this raises the vocabulary from 18 to 19 features by adding ‘a’. In research text this matters for terms such as ‘p’ or ‘n’ only if your corpus uses them as separate tokens.

Stop words, min_df and max_df

stop_words='english' removed ‘the’, ‘was’, ‘were’, ‘with’ (and nothing else) on this corpus, leaving 14 features. The documentation warns that the built-in English list has known issues, is not a one-size-fits-all solution, and that popular lists can include informative words, so check it against your domain. A stop list must also be consistent with the tokenizer: the guide gives the example that ‘we’ve’ is split into ‘we’ and ‘ve’. min_df and max_df accept an integer count or a float proportion and are ignored when you pass a fixed vocabulary. With min_df=2 only the seven tokens that appear in at least two documents remain: analysis, data, mixed, model, power, size, with.

N-grams and binary counts

ngram_range=(1, 2) counts single words and adjacent word pairs, producing 37 features here, and (2, 2) gives 19 pairs only, including ‘power analysis’, ‘effect size’, ‘mixed model’ and ‘missing data’. That is the main way to capture short multiword research terms, since the guide notes that unigrams cannot capture phrases or word order. binary=True sets every non-zero count to 1, recording presence rather than frequency; the feature count is unchanged at 18. max_features keeps only the highest-frequency terms across the corpus, so when several tokens tie on frequency, check which are kept.

Troubleshooting CountVectorizer

  • A short word is missing from the vocabulary. The default token pattern needs two or more word characters. Pass token_pattern=r'bw+b'.
  • Everything in the new text is zero. transform counts only words in the learned vocabulary. If the new document shares no words with the fitting corpus, the row is all zeros. This is expected, not a bug.
  • Test-set information leaks into the vocabulary. Call fit_transform on the training documents only and transform on the test documents, as in the guide on splitting data in Python. Fitting on the full corpus lets the vocabulary include words that appear only in held-out text.
  • Memory grows with the corpus. The user guide states that the vocabulary is held in memory, grows with the corpus, and needs a full pass over the data. It names HashingVectorizer as the stateless alternative.
  • Stop words behave oddly. Stop-word lists must match the tokenizer; see the contractions example above.
  • Calling toarray() exhausts memory. The dense array allocates every zero. Keep the matrix sparse for large corpora.
  • Stop-word or tokenizer arguments seem ignored. The documentation says stop_words and tokenizer apply only when analyzer='word', and a custom preprocessor replaces strip_accents and lowercase.

Free alternatives to CountVectorizer

Tool What the official documentation says Trade-off for research use
HashingVectorizer (scikit-learn) Converts a collection of text documents to a matrix of token occurrences, as a scipy.sparse matrix; default n_features is 220 (1,048,576). Stateless, so no vocabulary is stored; the user guide lists no inverse_transform and no IDF weighting as the costs.
TfidfTransformer (scikit-learn user guide) Can be appended after counting in a pipeline to weight the counts. Adds weighting; it does not replace counting.
quanteda dfm() (R) Constructs a sparse document-feature matrix from a tokens or dfm object. R equivalent for analysts who work in R; input is a tokens object rather than raw strings.

Frequently asked questions

What does CountVectorizer return?

It returns a sparse matrix with one row per document and one column per vocabulary token, each cell holding that token’s count in that document. The vocabulary itself is in vocabulary_ and, as an ordered array, in get_feature_names_out().

How do I get the feature names?

Call vectorizer.get_feature_names_out() after fitting. The names are in the same order as the matrix columns.

How do I count word pairs instead of single words?

Set ngram_range=(2, 2) for pairs only, or (1, 2) for single words and pairs. Both are shown with real feature counts above.

Does CountVectorizer remove stop words by default?

No. The default is stop_words=None. You can pass 'english' or your own list, but the documentation notes known issues with the built-in English list.

Should I fit CountVectorizer on the whole dataset?

For prediction tasks, fit on the training documents only and use transform on test documents, so the learned vocabulary does not include held-out text.

Sources

Data: Effect of CountVectorizer settings on feature count (four-sentence example corpus)

Number of vocabulary features CountVectorizer produces on a four-sentence illustrative corpus under different settings. Compiled by running scikit-learn 1.9.1 on the corpus shown in the guide; setting semantics come from the scikit-learn CountVectorizer API reference and user guide.

Period covered: scikit-learn 1.9.1, run 2026-11-16.

Effect of CountVectorizer settings on feature count (four-sentence example corpus)
SettingFeatures (count)What changed
Defaults18Baseline: 2+ letter tokens, lowercase, unigrams
token_pattern=r'\b\w+\b'19Adds the single-letter token 'a'
stop_words='english'14Removes the, was, were, with
ngram_range=(1, 2)37Unigrams plus adjacent word pairs
ngram_range=(2, 2)19Adjacent word pairs only
min_df=27Keeps tokens in at least two documents
binary=True18Same vocabulary; counts capped at 1

Sources

Compiled by CASRAI from the sources above. Free to reuse under CC BY 4.0 with attribution.

Follow CASRAI

Research-administration guidance, standards updates and independent tool reviews.

Ask CASRAI · free to try

Ask about CountVectorizer in scikit-learn: Syntax, Example and Output

2 free questions a day, no account.

An AI assistant for research: funder rules, compliance, methods and publishing. It links the sources behind its answers and labels answers that come from a web search.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays open to read, with no account and no card.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →