What Is Sentiment Analysis? Methods, Python and R Steps, and Limits

Sentiment analysis is the computational classification of text by the attitude it expresses, usually positive, negative or neutral. It is also called opinion mining. In research it turns open-text material such as survey comments, interview excerpts, posts and reviews into variables you can count, compare and model. This guide gives the definition, then runnable Python and R steps taken from the official documentation of each tool, how to read the output, and the validity limits you should report alongside any result.

What sentiment analysis is used for in research

Sentiment analysis sits inside natural language processing and is usually built with machine learning or with word lists. IBM’s overview describes rule-based methods, machine-learning methods (it lists algorithms such as naive Bayes, support vector machines and deep learning), and hybrids that combine the two. It also describes variants: fine-grained scoring on a graded scale, aspect-based analysis that attaches a sentiment to a named feature, and emotion detection.

Researchers use it where a large body of text needs a first-pass quantitative description: tracking how public comment on a policy shifts over time, screening free-text responses before closer reading, or measuring the tone of documents in marketing research. It supplements, and does not replace, careful qualitative reading of the same material, because automated scores inherit the blind spots described under troubleshooting below.

Install or access

Three official routes cover most research workflows. Try any of them on a few sentences before you commit a corpus to it.

Python: NLTK (VADER)

NLTK’s sentiment module includes VADER, a lexicon-and-rule analyser. Install NLTK first, then fetch its data through the NLTK Downloader, which opens when you call nltk.download() (the NLTK data page also documents python -m nltk.downloader all on the command line). VADER loads its word list from the NLTK data package sentiment/vader_lexicon.zip, so that data must be present on the machine.

pip install nltk
python -c "import nltk; nltk.download()"

Python: Hugging Face transformers

The Hugging Face installation page says the library works with PyTorch and has been tested on Python 3.11+ and PyTorch 2.6+. Install with uv pip install "transformers[torch]", dropping uv to use plain pip, or with conda: conda install conda-forge::transformers. For a CPU-only machine, install PyTorch from the CPU index first (uv pip install torch --index-url https://download.pytorch.org/whl/cpu) and then install transformers. Its own smoke test is the second line below; the documented output is a label of POSITIVE with a score close to 1.

pip install "transformers[torch]"
python -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('hugging face is the best'))"

R: tidytext

The tidytext package (Silge and Robinson, Text Mining with R) provides the sentiment lexicons used below. Install it from CRAN with the standard install.packages("tidytext") command, then load it with library(tidytext).

Step-by-step: how to do sentiment analysis in Python

Python offers two approaches that behave very differently: a lexicon scorer (VADER) and a trained model through the transformers pipeline.

Option 1: VADER in NLTK

1. Install NLTK and download its data (see above).
2. Import the analyser with from nltk.sentiment.vader import SentimentIntensityAnalyzer and create it with SentimentIntensityAnalyzer().
3. Call polarity_scores(sentence) on each text. It returns a dictionary with the keys neg, neu, pos and compound.
4. Store the four values per document in a table and carry them forward to your analysis.

from nltk.sentiment.vader import SentimentIntensityAnalyzer
sid = SentimentIntensityAnalyzer()
ss = sid.polarity_scores("VADER is smart, handsome, and funny.")
print(ss)

The NLTK documentation example scores “VADER is smart, handsome, and funny.” and reports a compound value of 0.8316.

Option 2: the transformers pipeline

1. Install transformers with PyTorch (see above).
2. Create the pipeline with pipeline("text-classification"). The official pipelines documentation states that sentiment-analysis is an alias of the text-classification task.
3. Pass one string, or a list of strings to classify several texts at once. The result is a list of dictionaries, each with a label (string) and a score (float).
4. To control which model is used, name it explicitly with pipeline(model="...") and report the model name in your methods. The documentation shows this with FacebookAI/roberta-large-mnli, whose labels include NEUTRAL.

from transformers import pipeline
pipe = pipeline("text-classification")
pipe(["This restaurant is awesome", "The wait was far too long"])

The documentation’s example, pipe("This restaurant is awesome"), returns a POSITIVE label with a score of about 0.9999.

Step-by-step: how to do sentiment analysis in R

In R the standard route is lexicon matching with tidytext. The steps follow the workflow in the Text Mining with R sentiment chapter; the code is adapted to a generic data frame of documents.

1. Load tidytext, dplyr and tidyr and inspect a lexicon with get_sentiments(). Three are offered: "afinn", "bing" and "nrc".
2. Split text into one word per row with unnest_tokens(word, text).
3. Join the tokens to a lexicon with inner_join(get_sentiments("bing")), which keeps only words the lexicon knows.
4. Count positive and negative words per document (or per fixed block of lines in a long text), reshape with pivot_wider(names_from = sentiment, values_from = n, values_fill = 0), and compute positive - negative as the net score.

install.packages("tidytext")
library(dplyr)
library(tidyr)
library(tidytext)
get_sentiments("bing")

# docs: a data frame with one row per document and columns doc_id and text
docs_sentiment <- docs %>%
  unnest_tokens(word, text) %>%
  inner_join(get_sentiments("bing"), by = "word") %>%
  count(doc_id, sentiment) %>%
  pivot_wider(names_from = sentiment, values_from = n, values_fill = 0) %>%
  mutate(sentiment = positive - negative)

For emotion categories, filter the NRC lexicon to one emotion and count the matching words. The book’s own example filters to joy:

nrc_joy <- get_sentiments("nrc") %>%
  filter(sentiment == "joy")
docs %>%
  unnest_tokens(word, text) %>%
  inner_join(nrc_joy, by = "word") %>%
  count(word, sort = TRUE)

Output and how to read it

• VADER (NLTK): neg, neu and pos are the negative, neutral and positive components of the result, and compound is a single combined score. The NLTK page gives example values but states no cut-off thresholds, so choose and report your own rule for calling a document positive or negative rather than citing one as official.
• transformers pipeline: label is the predicted class and score is the model’s confidence probability for that label. A score of 0.99 means the model is confident in the label, not that the writer felt strongly.
• tidytext lexicons: AFINN assigns each word an integer from -5 to 5. Bing labels each word positive or negative. NRC tags words yes or no for positive, negative, anger, anticipation, disgust, fear, joy, sadness, surprise and trust. A net score above zero marks a net positive section and below zero a net negative one.

Whichever tool you use, state the tool, the lexicon or model name, and the version in your methods section. That is the minimum for computational reproducibility, because different lexicons and models can score the same text differently.

Troubleshooting

• Negation is missed by the lexicons. The tidytext chapter notes that all three lexicons are unigram-based and do not account for negation such as “not true”. A sentence like “not good” can be scored on “good” alone. Spot-check a sample of scored texts by hand.
• Lexicon licences differ. The same chapter warns that the lexicons have different licences, and some require you to agree to a licence to download. Check the licence before using a lexicon in a funded or published project.
• VADER cannot find its word list. VADER loads sentiment/vader_lexicon.zip from the NLTK data directory. If NLTK reports that resource missing, open the NLTK Downloader with nltk.download() and fetch it.
• Context, irony and idiom. IBM names lack of context, irony and sarcasm, negation and idiomatic language as common drawbacks of automated sentiment analysis. Treat scores on such text as unreliable.
• Which model did the pipeline use? If you call the pipeline without naming a model, record which one was loaded; naming the model explicitly with pipeline(model="...") avoids ambiguity.

Alternative tools compared

Tool	Language	Approach	What you get back
NLTK VADER	Python	Lexicon and rules	neg, neu, pos and compound per text
Hugging Face transformers pipeline	Python (PyTorch)	Pretrained model	A label and a confidence score per text
tidytext with AFINN	R	Word list	An integer from -5 to 5 per matched word
tidytext with Bing	R	Word list	Positive or negative per matched word
tidytext with NRC	R	Word list	Yes/no tags for positive, negative and eight emotions

FAQ

How to do sentiment analysis in R?

Use the tidytext package: split your text into words with unnest_tokens(), join the words to a lexicon from get_sentiments() (AFINN, Bing or NRC), count positive and negative words per document, and subtract negative from positive to get a net score. The step-by-step section above gives the code.

How to do sentiment analysis in Python?

Either score text with VADER in NLTK using SentimentIntensityAnalyzer().polarity_scores(text), or classify it with pipeline("text-classification") from Hugging Face transformers, where sentiment-analysis is an alias for the same task. VADER gives four scores per text; the pipeline gives a label and a confidence score.

Is sentiment analysis the same as opinion mining?

Yes. IBM’s overview uses the two terms for the same task: classifying text as positive, negative or neutral.

Sources

Links checked on 2026-10-29.

• Hugging Face Transformers documentation, Pipelines and Installation (checked 2026-10-29).
• NLTK documentation, Sentiment Analysis howto and Installing NLTK Data (checked 2026-10-29); NLTK source for nltk.sentiment.vader (checked 2026-10-29).
• Julia Silge and David Robinson, Text Mining with R, chapter 2: Sentiment analysis with tidy data (checked 2026-10-29).
• IBM, What is sentiment analysis? (checked 2026-10-29).