What Is Web Scraping? A Researcher’s Guide

Web scraping is the automated extraction of data from web pages: a program requests a page, reads its HTML, and pulls out the pieces you want (text, links, table cells) into a structured form such as a table. No standard defines the term; it is used for any script that does this. For researchers, the practical rule is to scrape only when the data you need has no official API or bulk download, and to read the site’s terms and your ethics requirements first.

What web scraping is used for in research

Scraping turns information that is published for people to read into data you can analyse. A researcher might collect the entries of a public listing, the rows of an HTML table, or the text of a set of pages for later analysis. It is a last resort rather than a first choice: many scholarly sources provide a documented interface that returns structured data directly, and that route is more stable and easier to cite. Check these first:

• The Crossref REST API and the Crossref guide for publication metadata.
• The PubMed E-utilities for biomedical records.
• The Semantic Scholar API for paper and citation data.

If an API or bulk download covers your question, use it. If not, scraping may be the only route, and the rest of this guide shows how to do it in R and in Python.

Rules to check before you scrape

robots.txt is not permission

Sites publish crawling preferences in a robots.txt file, defined by the Robots Exclusion Protocol (RFC 9309). The RFC states that these rules are "not a form of access authorization." A robots.txt file tells automated clients what the site operator asks them not to fetch; it does not grant you permission, and it does not answer any legal question. When rules overlap, RFC 9309 says the most specific match found must be used.

Terms of service, law and ethics

Terms of service and the law vary by site and by country, and this guide gives no legal verdict. Read the site’s terms, and if your project involves people’s data, check your institution’s requirements for ethical approval before collecting anything.

Install or access (official sources only)

R: rvest

The rvest package "helps you scrape (or harvest) data from web pages." Install it from CRAN, either on its own or as part of the tidyverse:

install.packages("rvest") # or install.packages("tidyverse")

Documentation: rvest.tidyverse.org.

Python: Requests and Beautiful Soup

Beautiful Soup is "a Python library for pulling data out of HTML and XML files." The package name is beautifulsoup4, not BeautifulSoup:

pip install beautifulsoup4

Beautiful Soup parses HTML but does not download it. The Requests library fetches the page; install it by following its own documentation at requests.readthedocs.io.

How to do web scraping in R?

Use rvest: read the page with read_html(), select elements with html_elements(), and extract text or tables. The examples below follow the rvest documentation.

1. Load the page: starwars <- read_html("https://rvest.tidyverse.org/articles/starwars.html")
2. Select every element of one kind: films <- starwars |> html_elements("section")
3. Pull a value out of each: title <- films |> html_element("h2") |> html_text2()
4. For an HTML table, select the element by its CSS class and convert it: html |> html_element(".tracklist") |> html_table()

Use html_elements() (plural) when you want every match and html_element() (singular) when you want one match per parent element. Replace the example URL and selectors with those of your own page; you find the right selector by inspecting the page’s HTML in your browser.

How to do web scraping in Python?

Use Requests to download the page and Beautiful Soup to read it. The calls below come from the two libraries’ documentation; the URL is a placeholder for your own page.

1. Fetch the page with an identifying user agent and a timeout: r = requests.get('https://example.org/page', headers={'user-agent': 'my-app/0.0.1'}, timeout=10)
2. Stop on an error response: r.raise_for_status() raises an HTTPError for 4XX and 5XX responses.
3. Parse the HTML: soup = BeautifulSoup(r.text, 'html.parser')
4. Find what you need: soup.find_all(...) "looks through a tag’s descendants and retrieves all descendants that match your filters."
5. Extract the text: get_text() returns "the human-readable text inside a document or tag."

The Requests documentation is explicit about timeouts: "Nearly all production code should use this parameter in nearly all requests. Failure to do so can cause your program to hang indefinitely," and if you do not set one, requests do not time out. Use a descriptive user agent that names your project or institution rather than one that imitates a browser.

Output and how to read it

• R: html_text2() returns the text of each selected element as readable text, and html_table() converts an HTML table into a table you can analyse.
• Python: r.status_code gives the HTTP status, r.text the page as text, and r.json() decodes a JSON response. Beautiful Soup returns tag objects; get_text() turns them into plain text.

Record the URL and the date you collected each page, and keep the raw responses you downloaded, so that others can see exactly what your analysis started from. This supports the computational reproducibility of the study; a scripted environment such as the one in the reproducible conda environment guide helps with the rest.

Troubleshooting

• Error status. Call r.raise_for_status() so a 4XX or 5XX response stops the script instead of being parsed as if it were data.
• Script hangs. Add the timeout parameter to every request; without it, requests never time out.
• Garbled characters. Check r.encoding, and use r.content (the raw bytes) if the encoding is wrong.
• Unexpected page. Response.history lists the redirects your request followed.

Free alternatives

Option	Language	What it does
rvest	R	Scrapes (harvests) data from web pages; reads HTML, selects elements, extracts text and tables.
Beautiful Soup	Python	Pulls data out of HTML and XML files.
Requests	Python	Fetches pages and API responses, with status checks, headers and timeouts.
An official API or bulk download	Any	Returns structured data the source intends you to use; check for one before scraping.

Frequently asked questions

How do I do web scraping in R?

Install rvest with install.packages("rvest"), load a page with read_html(), select elements with html_elements(), and extract text with html_text2() or tables with html_table().

How do I do web scraping in Python?

Fetch the page with Requests (requests.get, with a timeout and raise_for_status()), then parse r.text with Beautiful Soup (pip install beautifulsoup4) and extract data with find_all() and get_text().

Does robots.txt mean I am allowed to scrape a site?

No. RFC 9309 says its rules are "not a form of access authorization." Check the site’s terms and your institution’s requirements separately.

Sources

• RFC 9309, Robots Exclusion Protocol: https://www.rfc-editor.org/rfc/rfc9309 (checked 2026-10-29).
• rvest documentation: https://rvest.tidyverse.org/ (checked 2026-10-29).
• Beautiful Soup documentation: https://www.crummy.com/software/BeautifulSoup/bs4/doc/ (checked 2026-10-29).
• Requests quickstart: https://requests.readthedocs.io/en/latest/user/quickstart/ (checked 2026-10-29).