Our pick on cost per GB · Verified 18 August 2026
Thordata — the best price-per-gigabyte at research-project volumes
Residential from $0.65/GB at volume
Academic data collection is bursty and mid-volume — a few hundred gigabytes over a fieldwork window, not a sustained commercial pipeline — and that is exactly the band where Thordata prices well: $1.50/GB at 50 GB and $1.00/GB at 150 GB, against a pool of 100M+ residential IPs across 190+ countries. It also sells SERP and scraper APIs by the thousand responses, which suits a study that needs search-results data without building and maintaining a parser.
See Thordata pricing Opens on the vendor’s site · CASRAI referral link
Read the ethics section before you buy anything → — The technical decision is easy. The IRB, terms-of-service and data-protection questions are the ones that stop projects, and they are best answered before procurement rather than after.
Editorial disclosure: CASRAI has commercial referral arrangements with some of the vendors named on this page, and may earn a commission if you subscribe to them. We name them here regardless of whether a link is present. We only recommend tools our editorial team has independently researched. Read our full disclosure policy.
In summary
- Proxies route your requests through other IP addresses, so a target site sees many ordinary-looking visitors rather than one server making thousands of requests.
- Residential proxies use real consumer IPs and are hardest to block; datacentre proxies are far cheaper and fine for sites that do not fight back.
- Thordata residential pricing: $2.00 for 1 GB, $1.50/GB at 50 GB, $1.00/GB at 150 GB, down to $0.65–0.73/GB above 1,000 GB. Datacentre and static ISP $0.75/IP. Verified 18 August 2026.
- Estimate volume before buying: a text-only page is often 50–200 KB, so 100,000 pages is roughly 5–20 GB. Rendering JavaScript multiplies that severalfold.
- Get the ethics and terms-of-service position settled first. A dataset you cannot publish or share is worse than no dataset.
Proxy types compared
Thordata pricing verified from the vendor pricing page, 18 August 2026
| Dimension | Residential | Datacentre | Static ISP | Mobile |
|---|---|---|---|---|
| What the IP looks like | A real home broadband subscriber | Obviously a server | A residential ISP block, but static | A mobile carrier subscriber |
| Thordata price | From $0.65/GB at volume; $2.00 for 1 GB | $0.75 per IP | $0.75 per IP | $2.20/GB |
| Billing model | Per gigabyte | Per IP | Per IP, unlimited bandwidth | Per gigabyte |
| Block resistance | High | Low on defended sites | High | Highest |
| Speed | Variable — real consumer connections | Fastest | Fast and consistent | Slowest |
| Use it for | Geographically-specific collection from sites that personalise or rate-limit | High-volume collection from open or tolerant sources | Long sessions needing a stable identity | Mobile-only content and apps |
Per-IP products bill for the address rather than the traffic, so they suit sustained collection; per-GB products suit bursty work where you would otherwise pay for idle IPs.
Why a research project ends up needing proxies
Two distinct problems drive this, and they call for different configurations — which is why buying before diagnosing usually wastes money.
Geographic personalisation. A great deal of the modern web serves different content by location. Prices differ by country. Search results differ by country and often by city. Marketplace listings, availability, advertising and recommendation feeds all vary. For a comparative study this is not an obstacle to route around — it is the phenomenon under investigation, and observing it requires requests that genuinely originate in each market. A single server in one country cannot see what a user in another country sees, and asking a collaborator to take screenshots does not scale or replicate.
Rate limiting and blocking. Sites reasonably defend themselves against automated traffic. A research script making a few thousand requests from one address will be throttled, served a challenge, or blocked outright — often partway through collection, which is worse than being blocked at the start because it silently biases the dataset. Distributing requests across many addresses, at a considerate rate, keeps collection viable.
If your problem is only the second one, the honest first move is to slow down rather than to buy proxies. Politeness delays, respecting robots.txt, off-peak scheduling and caching aggressively so you never fetch the same page twice will get a surprising number of projects home without any proxy spend at all.
The ethics and legal position
This is the section proxy vendors do not write, and it is the one that determines whether your project finishes.
Ethics review. Many researchers assume publicly-accessible means exempt. Review boards increasingly disagree, particularly where the data concerns identifiable individuals, where aggregation makes re-identification possible, or where users had a reasonable expectation their posts would not be compiled into a dataset. Go to your IRB or ethics committee before collection, with a written plan covering what you will collect, how you will store and de-identify it, what you will publish, and what you will do if you incidentally capture something sensitive. Retrospective approval is not a thing, and a committee’s most common objection — that the design collects more than the question requires — is much cheaper to fix at the design stage.
Terms of service. Most large platforms prohibit automated collection in their terms. Breaching them is generally a contractual matter rather than a criminal one in most jurisdictions, but it is not nothing: accounts get terminated mid-study, and some journals and funders now ask directly how data was obtained. The legal picture around scraping publicly-available data has moved in researchers’ favour in several jurisdictions in recent years, but it remains genuinely unsettled and varies by country. Involve your institution’s legal office for anything beyond small-scale public-page collection rather than relying on a summary — including this one.
Data protection. If you are collecting personal data from EU or UK subjects, GDPR applies to your collection regardless of whether the data was public. You need a lawful basis, and for research this is usually — but not automatically — legitimate interests or a public-task basis with appropriate safeguards. Article 14 transparency obligations apply when you collect data about people from a source other than the people themselves, and the research exemption from notifying every subject has conditions rather than being a blanket pass. Your DPO should see the plan.
Reproducibility. Web data is not stable — pages change, disappear and are personalised. Log what you collected, when, from which country, and with what configuration, and archive raw responses rather than only parsed output. A reviewer asking “could someone reproduce this?” needs a better answer than a script that no longer returns the same thing.
Estimating how much bandwidth you actually need
Residential proxies bill per gigabyte, so the budget line depends entirely on how much data you pull — and researchers consistently underestimate this by an order of magnitude.
Rough working figures: a text-heavy HTML page is often 50–200 KB. A hundred thousand such pages is therefore somewhere around 5–20 GB, which at $1.00–1.50/GB is a modest cost. But that assumes you fetch only the HTML. If you render pages in a headless browser — necessary for JavaScript-heavy sites — you also pull CSS, fonts, images, trackers and analytics scripts, and per-page transfer can rise several-fold. Rendering a hundred thousand modern pages can comfortably exceed 100 GB.
Three practical savings, in order of effect. Block asset loading in your headless browser — images, media and fonts are usually irrelevant to your research question and are most of the bytes. Cache aggressively so a re-run does not re-fetch what you already have; a resumable crawler that skips completed URLs pays for itself the first time a job fails at 80%. Use the cheapest proxy type that works for each source: run a small pilot against datacentre IPs first, and only move a source to residential when it demonstrably blocks you.
Always pilot before committing. Collect a few hundred pages, measure actual bytes per page, and multiply. Buying a large bandwidth package on an unpiloted estimate is the most common way this line item goes wrong.
When an API beats raw proxies
Raw proxies give you IP addresses; you still write and maintain the collection code, handle challenges, parse markup, and repair everything when a target site redesigns. For some common tasks that maintenance burden is the real cost, and a managed API is cheaper in researcher-hours even when it looks more expensive per request.
Thordata prices a SERP API at $0.70–0.80 per 1,000 responses, a Web Scraper API at $0.50–1.00 per 1,000 results, a Web Unlocker at $1.00 per 1,000 and a Scraping Browser at $2.50/GB (verified 18 August 2026). For a study needing search-results data across many countries — a very common design in computational social science and in market and information-access research — a SERP API removes the entire problem of parsing a results page that changes layout every few months.
The rule of thumb: if what you need is a well-defined, structured dataset that many people want (search results, product listings), buy the API. If your target is idiosyncratic — a specific set of institutional sites, a niche forum, a government register — you will be writing custom collection code anyway, and raw proxies are the cheaper input.
Collecting considerately
Beyond the legal position there is a professional one. Research collection that degrades a service for its actual users is bad practice, and it is the behaviour that has hardened platforms against academic access generally.
- Read
robots.txtand honour it unless you have a specific, documented, ethics-approved reason not to. - Rate-limit yourself well below what the target can handle. There is rarely a research reason to collect as fast as technically possible.
- Identify yourself in the user-agent string with a project name and contact address where you are not deliberately studying differential treatment. Site operators who can reach you will often grant access rather than block you.
- Ask first for anything substantial. Many organisations will provide a data dump or API access to a named academic project. It is faster, cleaner and more citable than scraping.
- Collect only what your question needs. This is both the ethical position and the one that keeps the bandwidth bill down.
- Store raw responses. Reparsing an archive is free; recollecting a web that has moved on is impossible.
Pilot before you commit to a bandwidth package
Buy the smallest package, collect a few hundred pages, and measure real bytes per page against your target sources. The extrapolated figure is what belongs in the grant — not an estimate from a vendor calculator.
Residential from $0.65/GB at volume
See Thordata pricing Opens on the vendor’s site · CASRAI referral link
Frequently asked questions
What is a web scraping proxy?
An intermediary server that forwards your requests so the target site sees the proxy’s IP address rather than yours. Using a pool of them spreads requests across many addresses, which lets collection continue without triggering rate limits and lets requests originate in a chosen country.
Do I need residential proxies, or will datacentre proxies do?
Start with datacentre — they are far cheaper and adequate for open or tolerant sources. Move a source to residential only when it demonstrably blocks or challenges datacentre traffic, or when you need requests to genuinely originate with a consumer ISP in a specific country. Test per source rather than buying residential for everything.
How much bandwidth does a scraping project use?
A text-heavy HTML page is often 50–200 KB, so 100,000 pages is roughly 5–20 GB. Rendering JavaScript in a headless browser pulls CSS, fonts, images and scripts too, and can multiply that several-fold. Always pilot a few hundred pages and measure before buying a package.
Is web scraping legal for academic research?
It depends on jurisdiction, on what you collect, and on the source’s terms. The position on publicly-available data has moved in researchers’ favour in several jurisdictions but remains genuinely unsettled. Separately, data protection law applies to personal data regardless of whether it was public. Get your institution’s legal office and DPO involved for anything beyond small-scale public-page collection.
Do I need ethics approval to scrape public web data?
Frequently yes, and you should ask rather than assume. Review boards increasingly hold that publicly-accessible does not mean exempt, particularly where data concerns identifiable individuals or where aggregation enables re-identification. Approach them before collection with a written plan — retrospective approval does not exist.
How much do proxies cost?
Thordata residential proxies run $2.00 for 1 GB, $1.50/GB at 50 GB and $1.00/GB at 150 GB, falling to $0.65–0.73/GB above 1,000 GB. Datacentre and static ISP proxies are $0.75 per IP, and mobile is $2.20/GB. Verified 18 August 2026.
Should I use a scraping API instead of raw proxies?
If you need a well-defined structured dataset that many people want — search results, product listings — an API is usually cheaper in researcher-hours because you avoid writing and maintaining parsers against sites that redesign. If your targets are idiosyncratic, you will be writing custom code anyway and raw proxies are the cheaper input.
