The web is the world's largest database of structured information — product prices updated daily, travel rates revised by the hour, social engagement counted in real time, supplier catalogues and company filings refreshed continuously. Web scraping is the discipline that turns that raw public data into queryable datasets. This guide maps the major use cases by industry, explains what each requires technically, and links to the deep-dive guides with ready-to-run code for each target.
E-commerce: Price Intelligence and Competitive Monitoring
Price intelligence is the canonical web scraping use case and the one with the clearest ROI story: a retailer who knows that a competitor repriced a product four hours ago can respond immediately rather than discovering it at the weekly pricing review.
The data available on a public e-commerce listing page is richer than it looks. Most major platforms embed a schema.org Product JSON-LD block in every item page — name, price, currency, stock status, seller, aggregate review score — that is far more stable than CSS selectors against the surrounding markup. Amazon's Sponsored Display data, Walmart's pick-up-in-store availability, and eBay's auction status all layer on top of that core structured data.
Common use cases in this vertical:
- Competitor price monitoring (daily or intraday sweeps of SKU lists)
- Stock availability alerts for high-demand products
- Review sentiment aggregation for product improvement
- Marketplace seller landscape mapping (who is selling what, at what price)
- Search rank tracking (where does a product appear in organic and sponsored results)
See our E-commerce Scraping overview for architecture guidance, and the per-platform guides for Amazon, Walmart, eBay, and Alibaba for working code in PHP, Node.js, and Rust.
Travel: Rate Monitoring and Price Aggregation
Travel pricing is uniquely volatile: hotel nightly rates shift in response to demand signals and local event calendars; airfares can move by the minute around a flight booking event. That volatility is exactly what makes travel data commercially valuable — to OTAs benchmarking their own inventory, to revenue managers at independent hotels, to airlines tracking competitor fare strategies, and to analysts modelling city-level supply and occupancy.
The technical challenge in travel scraping is that the most valuable data — nightly rates, room-type breakdowns, per-itinerary fare detail — lives inside JavaScript-rendered page state rather than server-rendered HTML. A plain HTTP client retrieves an empty shell; the pricing data appears only after full client-side hydration. Airbnb, Booking.com, and Kayak also all operate bot-management layers that reject datacenter IP ranges at the CDN edge.
Common use cases in this vertical:
- Hotel rate parity monitoring (are OTAs undercutting direct-booking rates?)
- Vacation rental market analysis (average nightly price by neighbourhood, seasonality)
- Flight price alerting (track a route and alert when fares drop below a threshold)
- Availability-based demand inference (sold-out inventory reveals high-demand dates)
See Travel Data Scraping: Hotels, Flights & Rentals for the full landscape, then Airbnb, Booking.com, and Kayak for the platform-specific parsing.
Social Media: Sentiment and Brand Monitoring
Social scraping of public data — profiles, posts, hashtags, engagement counts — powers brand safety auditing, influencer discovery, trend monitoring, and political research. The signal value is in the trajectory: a brand-mention post gaining 50,000 likes in two hours reads very differently from one gaining 50,000 over a month.
Public social profile pages are intentionally indexable: they emit Open Graph meta tags (og:title, og:description, og:image) and, on some platforms, JSON-LD ProfilePage objects that are designed to survive layout changes. Post text and engagement counts require more fragile parsing — and on TikTok and Facebook, they require full JavaScript rendering before they appear.
Common use cases in this vertical:
- Brand mention tracking (volume and sentiment of posts naming your brand)
- Influencer discovery (follower counts, engagement rate, niche by hashtag)
- Competitor content performance benchmarking
- Trend detection (which hashtags are accelerating; which are declining)
- Crisis monitoring (sudden spike in negative brand mentions)
See Social Media Scraping: Profiles, Posts & Sentiment for the overview, then Instagram, TikTok, and Facebook for working code samples.
B2B Research: Lead Generation and Supplier Discovery
B2B scraping covers a wide range of targets: company directories (LinkedIn company pages, Crunchbase, local business listings), supplier catalogues on Alibaba or Global Sources, job postings as a proxy for hiring intent, and public company filings as a source of executive contact information and financial data.
The data formats here are more varied than in e-commerce. Some targets expose clean JSON-LD Organization or LocalBusiness blocks; others require CSS-selector-based extraction against rendered HTML; still others structure their data inside embedded JavaScript objects. Unlike social media, many B2B research targets are less aggressively protected — but the most valuable ones (LinkedIn, Glassdoor, Crunchbase) run the same residential-IP-required bot management as the top consumer platforms.
Common use cases in this vertical:
- Lead list building (company name, size, industry, decision-maker contact)
- Supplier price and MOQ comparison across B2B marketplaces
- Job posting analysis (competitor hiring signals, salary benchmarking)
- Company funding and growth trajectory monitoring
- Review aggregation across G2, Capterra, and similar platforms
See How to Scrape Alibaba for a worked example, and How to Avoid Getting Blocked for the anti-bot patterns that apply across this vertical.
Market Research: Search Engines, News, and Public Records
Search engine result pages (SERPs), news archives, patent databases, and public regulatory filings form a distinct scraping category. The use cases are more analytical than operational — you are not building a live price feed but reconstructing a landscape over time.
SERP scraping tracks organic keyword rankings, identifies featured snippets and SERP features, and monitors paid ad copy for competitor intelligence. News scraping fuels financial NLP models (identifying company mentions in earnings reports or regulatory filings), tracks brand coverage, and populates content intelligence databases. Public records scraping extracts court filings, property records, and corporate registration data for due-diligence workflows.
Common use cases in this vertical:
- Organic rank tracking across target keywords
- Competitor ad copy and landing page surveillance
- News sentiment analysis for financial modelling
- Patent landscape mapping for R&D teams
- Public company filing extraction for ESG and due-diligence workflows
See our Google Search Scraping guide for SERP-specific code and the Web Scraping Without Getting Blocked guide for the defences common to high-value research targets.
Technical Prerequisites Across Use Cases
Despite the diversity of targets, the same technical foundation underpins all production scraping:
Proxy infrastructure. Any target with meaningful anti-bot investment blocks datacenter IPs at the CDN layer. Residential or ISP proxies with realistic TLS fingerprints are the entry requirement for e-commerce, travel, and social targets. For simpler targets — news sites, public records, SERP datasets — rotating datacenter proxies are often sufficient. See Proxy Types Explained for a decision framework.
JavaScript rendering. React and Next.js SPAs serve empty shells on initial load. Unless the proxy layer handles rendering for you, your scraper needs a headless browser or will return empty content on the most valuable targets.
Structured data parsing. JSON-LD schema.org blocks and Open Graph meta tags are the most reliable parsing targets across use cases — they are designed to be machine-readable and survive markup experiments better than CSS selectors. When they are present, always prefer them over fragile DOM selectors.
Rate control and retry logic. All anti-bot systems operate on request rate as one of their primary signals. Scrape at a cadence that is invisible in aggregate traffic; implement exponential back-off on 429 and 503 responses.
A Generic Structured-Data Extractor
This pattern — fetch via proxy, extract JSON-LD and Open Graph, return structured output — applies as a starting point to any web scraping use case. The per-platform guides linked throughout this article layer on the site-specific parsing needed for richer fields.
export PROXY_URL="http://brd-customer-<id>-zone-<unblocker_zone>:<password>@brd.superproxy.io:22225"
Need proxy infrastructure? Explore Bright Data's data collection tools →
PHP
<?php
// Run: php extract.php "https://example.com/any-page"
$proxy = getenv('PROXY_URL');
$url = $argv[1] ?? 'https://example.com/';
$ch = curl_init($url);
curl_setopt_array($ch, [
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_PROXY => $proxy,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_TIMEOUT => 60,
CURLOPT_HTTPHEADER => [
'Accept-Language: en-US,en;q=0.9',
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
],
]);
$html = curl_exec($ch);
curl_close($ch);
$doc = new DOMDocument();
@$doc->loadHTML($html);
$xp = new DOMXPath($doc);
// Extract all JSON-LD blocks
$jsonld = [];
foreach ($xp->query('//script[@type="application/ld+json"]') as $node) {
$decoded = json_decode($node->textContent, true);
if ($decoded) $jsonld[] = $decoded;
}
// Extract Open Graph metadata
$og = [];
foreach ($xp->query('//meta[starts-with(@property,"og:")]') as $node) {
$key = str_replace('og:', '', $node->getAttribute('property'));
$og[$key] = $node->getAttribute('content');
}
echo json_encode(['url' => $url, 'jsonld' => $jsonld, 'og' => $og],
JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE), PHP_EOL;
Node.js
// extract.mjs — node extract.mjs "https://example.com/any-page"
// Install: npm i axios https-proxy-agent cheerio
import axios from 'axios';
import { HttpsProxyAgent } from 'https-proxy-agent';
import * as cheerio from 'cheerio';
const agent = new HttpsProxyAgent(process.env.PROXY_URL);
const url = process.argv[2] ?? 'https://example.com/';
const { data: html } = await axios.get(url, {
httpsAgent: agent, proxy: false, timeout: 60_000,
headers: {
'Accept-Language': 'en-US,en;q=0.9',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
},
});
const $ = cheerio.load(html);
// Collect all JSON-LD blocks
const jsonld = [];
$('script[type="application/ld+json"]').each((_, el) => {
try { jsonld.push(JSON.parse($(el).text())); } catch { /* skip malformed */ }
});
// Collect Open Graph metadata
const og = {};
$('meta[property^="og:"]').each((_, el) => {
const key = $(el).attr('property').replace('og:', '');
og[key] = $(el).attr('content');
});
console.log(JSON.stringify({ url, jsonld, og }, null, 2));
Rust
// Cargo.toml:
// reqwest = { version = "0.12", features = ["blocking"] }
// scraper = "0.20"
// serde_json = "1"
use scraper::{Html, Selector};
use serde_json::{json, Map, Value};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let url = std::env::args().nth(1)
.unwrap_or_else(|| "https://example.com/".into());
let client = reqwest::blocking::Client::builder()
.proxy(reqwest::Proxy::all(std::env::var("PROXY_URL")?)?)
.danger_accept_invalid_certs(true)
.build()?;
let html = client
.get(&url)
.header("Accept-Language", "en-US,en;q=0.9")
.header("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
.send()?
.text()?;
let doc = Html::parse_document(&html);
let ld_sel = Selector::parse(r#"script[type="application/ld+json"]"#).unwrap();
let og_sel = Selector::parse(r#"meta[property]"#).unwrap();
let jsonld: Vec<Value> = doc.select(&ld_sel)
.filter_map(|el| serde_json::from_str(&el.text().collect::<String>()).ok())
.collect();
let mut og = Map::new();
for el in doc.select(&og_sel) {
let prop = el.value().attr("property").unwrap_or("");
if prop.starts_with("og:") {
let key = prop.trim_start_matches("og:");
let val = el.value().attr("content").unwrap_or("");
og.insert(key.to_string(), Value::String(val.to_string()));
}
}
println!("{}", serde_json::to_string_pretty(&json!({
"url": url,
"jsonld": jsonld,
"og": og,
}))?);
Ok(())
}
Choosing the Right Infrastructure
Matching your proxy layer to the target is the single most consequential infrastructure decision in any scraping project. Our proxy provider comparison guides cover the leading options side by side. For most high-value targets — major e-commerce platforms, travel sites, and social networks — you need a provider that combines residential IP rotation with JavaScript rendering, since both are required simultaneously.
Bright Data's Web Unlocker and data collection tools handle residential IP rotation, JavaScript rendering, TLS fingerprint normalisation, and CAPTCHA solving behind a single proxy endpoint — the same infrastructure reflected in every code sample across this guide. For large historical datasets, pre-built datasets are often faster to acquire than bootstrapping a full scraper from scratch.
Related: Bright Data Web Unlocker review, How to Avoid Getting Blocked, Proxy Types Explained, E-commerce Scraping, Travel Data Scraping, and Social Media Scraping.
Start collecting structured web data at scale →
FAQ
What are the most common web scraping use cases?
Price intelligence and competitor monitoring in e-commerce, hotel and flight rate tracking in travel, brand sentiment and influencer discovery in social media, and lead generation or supplier research in B2B are the dominant production use cases. Each has its own target sites, data formats, and anti-bot environment.
Do I need residential proxies for all scraping use cases?
Not always. Many simpler targets — news sites, academic databases, public company filings — serve content reliably through datacenter proxies. Residential or ISP proxies become necessary when the target runs bot-management middleware (Cloudflare, DataDome, Akamai) or blocks known cloud IP ranges wholesale, as most major e-commerce, travel, and social platforms do.
How do I handle sites that require JavaScript rendering?
Sites built as React or Next.js SPAs serve an empty HTML shell on the initial request; useful data appears only after client-side hydration. You need either a headless browser (Playwright, Puppeteer) or a proxy service that handles rendering for you. The latter is usually more efficient at scale because it offloads browser fleet management.
Is it legal to scrape public web data?
Whether scraping publicly visible data is lawful depends on jurisdiction, the nature of the data, and how it is used. Many courts have treated scraping of publicly accessible data as permissible, but platform Terms of Service and data-protection laws (GDPR, CCPA) add separate layers of constraint. Get jurisdiction-specific legal advice before deploying a production scraper.