How to Solve CAPTCHAs When Web Scraping: A Practical Guide

2026-06-12T00:00:00+00:00

CAPTCHAs are the single most common wall between a scraper and the data it needs. The moment a site suspects automation, it serves a challenge — and your pipeline stalls. This guide covers the CAPTCHA types you'll hit, why they trigger, and the three practical ways to get past them, from prevention to fully automated solving.</p>
Why CAPTCHAs trigger during scraping</h2>
A CAPTCHA isn't random — it's a response to signals that say "bot." The usual triggers:</p>

Datacenter IPs</strong> with no residential reputation.</li>
Too many requests</strong> from one IP in a short window.</li>
Missing or inconsistent browser fingerprints</strong> (no JavaScript execution, headless signatures, mismatched headers).</li>
Behavioral flags</strong> — no mouse movement, inst. navigation, perfect timing.</li> </ul> The first lesson: the best CAPTCHA is the one you never see. Much of CAPTCHA handling is really about not triggering them</em> in the first place — see How to Avoid Getting Blocked</a> for the full prevention playbook.</p>
The CAPTCHA types you'll encounter</h2>
Different sites deploy different challenge systems, and each behaves differently:</p>

reCAPTCHA (Google)</strong> — v2 "I'm not a robot" checkbox + image grids, and invisible v3 scoring.</li>
hCaptcha</strong> — image-selection challenges, common on Cloudflare-fronted sites.</li>
Cloudflare Turnstile</strong> — a lightweight, often invisible challenge replacing classic CAPTCHAs.</li>
AWS WAF CAPTCHA</strong> — puzzle challenges on AWS-protected endpoints.</li>
PerimeterX / HUMAN, DataDome, Akamai</strong> — anti-bot platforms that escalate to CAPTCHAs when suspicious.</li>
FunCaptcha (Arkose), GeeTest, KeyCAPTCHA, Yandex</strong> — slider, rotation, and puzzle variants.</li>
Image / text / click CAPTCHAs</strong> — the classic distorted-text and "click all the X" challenges.</li> </ul>
Knowing which one you face matters: a Turnstile pass is cheap and fast; a reCAPTCHA v2 image grid is expensive and slow to solve manually.</p>
Approach 1: Prevent them (cheapest)</h2>
Stop the challenge before it appears:</p>

Route through residential proxies</a></strong> so traffic looks like a real consumer ISP.</li>
Send complete, consistent headers</strong> and a realistic User-Agent.</li>
Throttle and randomize</strong> request timing.</li>
Render JavaScript</strong> with a real browser engine when the site expects it (see Playwright scraping</a>).</li> </ul>
For low-to-medium volume on moderately protected sites, prevention alone often keeps you CAPTCHA-free.</p>
Approach 2: Solving services (manual integration)</h2>
Standalone solving APIs accept a CAPTCHA's site-key and return a solution token you submit with your request. They work, but you own the integration: detecting the challenge, extracting parameters, calling the solver, injecting the token, and retrying. Per-site engineering, and it breaks when the challenge changes.</p>
Approach 3: Automatic solving via a Web Unlocker (least effort)</h2>
The lowest-maintenance path bundles detection, solving, proxies, and rendering behind one endpoint. You send a URL; you get back unblocked HTML — CAPTCHA handled transparently.</p>
Bright Data's CAPTCHA solver</a>, part of its Web Unlocker, automatically detects and solves CAPTCHAs by default</strong> — no site-keys, no token injection. It covers the full range above (reCAPTCHA, hCaptcha, Turnstile, PerimeterX, FunCaptcha, GeeTest, AWS WAF, KeyCAPTCHA, Yandex, and image/text/click variants), submits any associated forms after solving, and returns the result as HTML, JSON, Markdown, or a screenshot. Auto-solving can be toggled off per request or per CAPTCHA type when you want manual control.</p>
A request goes through a single endpoint — no per-CAPTCHA plumbing:</p>
curl -X POST "https://api.brightdata.com/request" \ -H "Authorization: Bearer $BRIGHTDATA_TOKEN" \ -H "Content-Type: application/json" \ -d '{"zone":"web_unlocker","url":"https://example.com/protected","format":"raw"}' </code></pre> # pip install requests import os, requests resp = requests.post( "https://api.brightdata.com/request", headers={"Authorization": f"Bearer {os.environ['BRIGHTDATA_TOKEN']}"}, json={"zone": "web_unlocker", "url": "https://example.com/protected", "format": "raw"}, timeout=60, ) print(resp.text) # unblocked HTML, CAPTCHA already solved </code></pre> // node captcha.mjs const resp = await fetch("https://api.brightdata.com/request", { method: "POST", headers: { Authorization: `Bearer ${process.env.BRIGHTDATA_TOKEN}`, "Content-Type": "application/json", }, body: JSON.stringify({ zone: "web_unlocker", url: "https://example.com/protected", format: "raw" }), }); console.log(await resp.text()); </code></pre> Because solving, IP rotation, and rendering happen server-side, your code stays the same whether the target throws a Turnstile, a reCAPTCHA grid, or nothing at all.</p> DIY vs. managed: how to choose</h2> </th> DIY solving service</th> Automatic Web Unlocker</th></tr></thead> Integration effort</td> High (per-site)</td> Low (one endpoint)</td></tr> Handles new CAPTCHA types</td> You adapt</td> Vendor adapts</td></tr> Proxies + rendering included</td> No</td> Yes</td></tr> Best for</td> One known target, full control</td> Many targets, low maintenance</td></tr> </tbody></table> For a single, stable target you fully control, a solving service is fine. For scraping many protected sites without babysitting each challenge, an automatic unlocker is the pragmatic choice.</p> The bottom line</h2> Handle CAPTCHAs in this order: avoid triggering them</strong> (proxies, fingerprints, pacing), then solve what's left</strong>. For a handful of easy targets, prevention plus a solving service works. For protected sites at scale, an automatic solver that bundles unblocking infrastructure removes the maintenance entirely.</p> Related: Bright Data Web Unlocker review</a>, How to Avoid Getting Blocked</a>, Bright Data Scraping Browser</a>, and proxy types explained</a>.</em></p> Solve CAPTCHAs automatically with Bright Data →</a></strong></p> eBay Product Tracking: Scraper Code Samples 2026-06-12T00:00:00+00:00 eBay is a prime target for price tracking, competitor monitoring, and deal sourcing — but its listing pages are heavily A/B-tested, so scraping rendered HTML with CSS selectors breaks constantly. The reliable source is the JSON-LD</strong> block eBay embeds in every item page: a <script type="application/ld+json"></code> tag containing a clean Product</code> object with name, price, currency, and availability. The samples below fetch the page through the Bright Data Web Unlocker</a> and read straight from that structured data.</p> Prerequisites</h2> export PROXY_URL="http://brd-customer-<id>-zone-<unblocker_zone>:<password>@brd.superproxy.io:22225" </code></pre> No Bright Data account yet?</strong> Get started with the eBay collector →</a></p> </blockquote> Items are identified by their numeric item ID</strong> from the URL: https://www.ebay.com/itm/<id></code>.</p> PHP</h2> <?php // Run: php ebay.php 167890123456 $proxy = getenv('PROXY_URL'); $itemId = $argv[1] ?? '167890123456'; $ch = curl_init("https://www.ebay.com/itm/$itemId"); curl_setopt_array($ch, [ CURLOPT_RETURNTRANSFER => true, CURLOPT_FOLLOWLOCATION => true, CURLOPT_PROXY => $proxy, CURLOPT_SSL_VERIFYPEER => false, CURLOPT_TIMEOUT => 60, CURLOPT_HTTPHEADER => ['Accept-Language: en-US,en;q=0.9'], ]); $html = curl_exec($ch); curl_close($ch); // eBay ships product data as JSON-LD. Find the block whose @type is "Product". $doc = new DOMDocument(); @$doc->loadHTML($html); $xp = new DOMXPath($doc); $product = null; foreach ($xp->query('//script[@type="application/ld+json"]') as $node) { $ld = json_decode($node->textContent, true); if (($ld['@type'] ?? '') === 'Product') { $product = $ld; break; } } $offer = $product['offers'] ?? []; echo json_encode([ 'id' => $itemId, 'name' => $product['name'] ?? null, 'price' => $offer['price'] ?? null, 'currency' => $offer['priceCurrency'] ?? null, 'availability' => $offer['availability'] ?? null, ], JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES), PHP_EOL; </code></pre> Node.js</h2> // ebay.mjs — node ebay.mjs 167890123456 // Install: npm i axios https-proxy-agent cheerio import axios from 'axios'; import { HttpsProxyAgent } from 'https-proxy-agent'; import * as cheerio from 'cheerio'; const agent = new HttpsProxyAgent(process.env.PROXY_URL); const itemId = process.argv[2] ?? '167890123456'; const { data: html } = await axios.get(`https://www.ebay.com/itm/${itemId}`, { httpsAgent: agent, proxy: false, timeout: 60_000, headers: { 'Accept-Language': 'en-US,en;q=0.9' }, }); const $ = cheerio.load(html); let product = {}; $('script[type="application/ld+json"]').each((_, el) => { try { const ld = JSON.parse($(el).text()); if (ld['@type'] === 'Product') product = ld; } catch { /* skip malformed blocks */ } }); const offer = product.offers ?? {}; console.log(JSON.stringify({ id: itemId, name: product.name ?? null, price: offer.price ?? null, currency: offer.priceCurrency ?? null, availability: offer.availability ?? null, }, null, 2)); </code></pre> Rust</h2> // Cargo.toml: // reqwest = { version = "0.12", features = ["blocking"] } // scraper = "0.20" // serde_json = "1" use scraper::{Html, Selector}; use serde_json::Value; fn main() -> Result<(), Box<dyn std::error::Error>> { let item_id = std::env::args().nth(1).unwrap_or_else(|| "167890123456".into()); let client = reqwest::blocking::Client::builder() .proxy(reqwest::Proxy::all(std::env::var("PROXY_URL")?)?) .danger_accept_invalid_certs(true) .build()?; let html = client .get(format!("https://www.ebay.com/itm/{item_id}")) .header("Accept-Language", "en-US,en;q=0.9") .send()? .text()?; let doc = Html::parse_document(&html); let sel = Selector::parse(r#"script[type="application/ld+json"]"#).unwrap(); let mut product = Value::Null; for el in doc.select(&sel) { let raw = el.text().collect::<String>(); if let Ok(ld) = serde_json::from_str::<Value>(&raw) { if ld["@type"] == "Product" { product = ld; break; } } } let offer = &product["offers"]; let out = serde_json::json!({ "id": item_id, "name": product["name"], "price": offer["price"], "currency": offer["priceCurrency"], "availability": offer["availability"], }); println!("{}", serde_json::to_string_pretty(&out)?); Ok(()) } </code></pre> Notes</h2> JSON-LD survives eBay's frequent layout experiments far better than CSS selectors — the same block also carries image</code>, brand</code>, sku</code>, and aggregateRating</code>.</li> availability</code> is a schema.org URL (e.g. https://schema.org/InStock</code>); strip the prefix if you only want the status word.</li> Auction listings differ from fixed-price ones — for live bids you may need the bidding section in the DOM rather than JSON-LD.</li> For a tracker, persist {id, price, timestamp}</code> per run and schedule with cron — see the Amazon tracker</a> for the full alerting pattern.</li> </ul> Scaling beyond a few items</h2> Hitting thousands of eBay listings on a schedule means rotating IPs, handling blocks, and absorbing layout changes — maintenance that adds up fast. Bright Data's eBay data collector</a> returns structured listing data without you managing any of that infrastructure, and pre-built datasets</a> cover bulk historical pulls.</p> See our E-commerce Web Scraping Solutions</a> overview, the eBay Product Search Scraping</a> guide for market-level pricing, the Bright Data Web Unlocker review</a>, and How to Avoid Getting Blocked</a>.</em></p> Scrape eBay at scale with Bright Data →</a></strong></p> Scrape Public Instagram Data: Profiles, Posts, Hashtags 2026-06-12T00:00:00+00:00 Instagram is among the richest sources of public social data on the web — brand sentiment, influencer reach, competitor engagement, and trending content all surface through its public-facing pages. This guide covers what you can realistically collect from Instagram without authentication</strong>, why naive scrapers fail within minutes, and working code that extracts structured profile data from public pages.</p> Public data vs. authenticated data</h2> The boundary matters both technically and legally:</p> Public data</strong> — profiles, posts, and hashtag pages visible to a logged-out visitor in a browser. This is the only safe target for automated collection.</li> Authenticated data</strong> — feeds, Stories, DMs, or anything that requires a login. Instagram's Terms of Use explicitly prohibit automated access to the logged-in experience, and doing so risks account bans and legal exposure.</li> </ul> Everything below targets logged-out public pages only</strong>. Instagram's ToS also restricts automated access to public pages; weigh that against your use case and, for commercial applications, review the relevant platform terms and seek counsel. This guide is technical documentation, not legal advice.</p> What public Instagram pages expose</h2> For a public profile (https://www.instagram.com/<username>/</code>), Instagram embeds a ProfilePage</code> JSON-LD block containing a Person</code> entity with the account's display name, username, bio, and profile image URL. Post pages (/p/<shortcode>/</code>) carry a CreativeWork</code> block with the caption and publish timestamp. Hashtag pages (/explore/tags/<tag>/</code>) are the most JavaScript-heavy and effectively require a fully rendered DOM.</p> JSON-LD is the most stable parsing target across all these pages — far more durable than CSS selectors, which break on every layout experiment Instagram runs.</p> Why DIY scraping breaks fast</h2> Instagram is one of the most aggressively defended scraping targets on the web:</p> Datacenter IPs are blocked immediately.</strong> You need residential proxies</a> that present as genuine browser sessions from real consumer ISPs.</li> Heavy JavaScript rendering.</strong> Public pages ship minimal HTML to unrecognized clients. The full content only appears after client-side hydration, so a plain HTTP request often returns a login wall or empty shell.</li> Fingerprinting and rate limits.</strong> Browser fingerprint checks, TLS fingerprinting, and per-IP rate limits combine to make rotating bare proxies insufficient at any meaningful volume.</li> Frequent markup changes.</strong> Even when you reach the page, CSS-selector-based parsers break on every redesign — JSON-LD stays stable through them.</li> </ul> Prerequisites</h2> export PROXY_URL="http://brd-customer-<id>-zone-<unblocker_zone>:<password>@brd.superproxy.io:22225" </code></pre> The samples below route all requests through the Bright Data Web Unlocker</a>, which handles JavaScript rendering, browser fingerprinting, CAPTCHA solving, and IP rotation automatically. Supply your zone credentials in PROXY_URL</code>.</p> No account yet?</strong> Explore the Instagram data collector →</a></p> </blockquote> Profiles are identified by username (natgeo</code>, nasa</code>, etc.).</p> PHP</h2> <?php // Run: php instagram.php natgeo $proxy = getenv('PROXY_URL'); $username = $argv[1] ?? 'natgeo'; $ch = curl_init("https://www.instagram.com/{$username}/"); curl_setopt_array($ch, [ CURLOPT_RETURNTRANSFER => true, CURLOPT_FOLLOWLOCATION => true, CURLOPT_PROXY => $proxy, CURLOPT_SSL_VERIFYPEER => false, CURLOPT_TIMEOUT => 60, CURLOPT_HTTPHEADER => [ 'Accept-Language: en-US,en;q=0.9', 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', ], ]); $html = curl_exec($ch); curl_close($ch); $doc = new DOMDocument(); @$doc->loadHTML($html); $xp = new DOMXPath($doc); $person = null; foreach ($xp->query('//script[@type="application/ld+json"]') as $node) { $ld = json_decode($node->textContent, true); $type = $ld['@type'] ?? ''; if ($type === 'ProfilePage' && isset($ld['mainEntity'])) { $person = $ld['mainEntity']; break; } if ($type === 'Person') { $person = $ld; break; } } $img = $person['image']['url'] ?? ($person['image'] ?? null); echo json_encode([ 'username' => $username, 'name' => $person['name'] ?? null, 'handle' => $person['alternateName'] ?? null, 'description' => $person['description'] ?? null, 'image' => $img, ], JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES), PHP_EOL; </code></pre> Node.js</h2> // instagram.mjs — node instagram.mjs natgeo // Install: npm i axios https-proxy-agent cheerio import axios from 'axios'; import { HttpsProxyAgent } from 'https-proxy-agent'; import * as cheerio from 'cheerio'; const agent = new HttpsProxyAgent(process.env.PROXY_URL); const username = process.argv[2] ?? 'natgeo'; const { data: html } = await axios.get(`https://www.instagram.com/${username}/`, { httpsAgent: agent, proxy: false, timeout: 60_000, headers: { 'Accept-Language': 'en-US,en;q=0.9', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', }, }); const $ = cheerio.load(html); let person = null; $('script[type="application/ld+json"]').each((_, el) => { try { const ld = JSON.parse($(el).text()); if (ld['@type'] === 'ProfilePage' && ld.mainEntity) person = ld.mainEntity; else if (ld['@type'] === 'Person') person = ld; } catch { /* skip malformed blocks */ } }); console.log(JSON.stringify({ username, name: person?.name ?? null, handle: person?.alternateName ?? null, description: person?.description ?? null, image: person?.image?.url ?? person?.image ?? null, }, null, 2)); </code></pre> Rust</h2> // Cargo.toml: // reqwest = { version = "0.12", features = ["blocking"] } // scraper = "0.20" // serde_json = "1" use scraper::{Html, Selector}; use serde_json::Value; fn main() -> Result<(), Box<dyn std::error::Error>> { let username = std::env::args().nth(1).unwrap_or_else(|| "natgeo".into()); let client = reqwest::blocking::Client::builder() .proxy(reqwest::Proxy::all(std::env::var("PROXY_URL")?)?) .danger_accept_invalid_certs(true) .build()?; let html = client .get(format!("https://www.instagram.com/{username}/")) .header("Accept-Language", "en-US,en;q=0.9") .header("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8") .send()? .text()?; let doc = Html::parse_document(&html); let sel = Selector::parse(r#"script[type="application/ld+json"]"#).unwrap(); let mut person = Value::Null; for el in doc.select(&sel) { let raw = el.text().collect::<String>(); if let Ok(ld) = serde_json::from_str::<Value>(&raw) { match ld["@type"].as_str() { Some("ProfilePage") if ld["mainEntity"].is_object() => { person = ld["mainEntity"].clone(); break; } Some("Person") => { person = ld; break; } _ => {} } } } let image_url = person["image"]["url"] .as_str() .or_else(|| person["image"].as_str()) .map(|s| Value::String(s.to_owned())) .unwrap_or(Value::Null); println!("{}", serde_json::to_string_pretty(&serde_json::json!({ "username": username, "name": person["name"], "handle": person["alternateName"], "description": person["description"], "image": image_url, }))?); Ok(()) } </code></pre> Adapting to posts and hashtags</h2> Swap the URL and the @type</code> check to hit other public endpoints:</p> Target</th> URL pattern</th> JSON-LD @type</code></th> Key fields</th></tr></thead> Post</td> /p/<shortcode>/</code></td> CreativeWork</code></td> caption</code>, datePublished</code>, author</code></td></tr> Hashtag</td> /explore/tags/<tag>/</code></td> — (JS-rendered grid)</td> Use the collector API</td></tr> </tbody></table> Hashtag pages and post grids require full browser rendering. The Web Unlocker handles this transparently, but expect higher latency than static-HTML targets.</p> Notes</h2> alternateName</code> is the @username</code> handle; name</code> is the display name shown on the profile.</li> Like counts and follower counts are not exposed in JSON-LD — they appear in the JS-rendered DOM. Accessing them via the collector API is more reliable than parsing the DOM directly.</li> For GDPR/CCPA compliance, treat bio text and profile images as personal data even when publicly posted.</li> Instagram's public JSON-LD targets desktop rendering; the Accept</code> header above steers the response away from mobile fallback pages.</li> </ul> Scaling beyond individual profiles</h2> Monitoring hundreds of accounts or tracking hashtag volume requires IP rotation, rate-limit awareness, rendering infrastructure, and ongoing schema maintenance as Instagram updates its pages. Bright Data's Instagram data collector</a> abstracts all of that — specify a list of usernames, hashtags, or post URLs and receive clean structured JSON on a schedule. For bulk historical analysis, pre-built datasets</a> are often faster to acquire than running a scraper from scratch.</p> Related: Bright Data Web Unlocker review</a>, How to Avoid Getting Blocked</a>, proxy types explained</a>, and the LinkedIn scraping guide</a>.</em></p> Collect public Instagram data at scale →</a></strong></p> How to Scrape LinkedIn: Public Profiles, Companies, Jobs 2026-06-12T00:00:00+00:00 LinkedIn is the richest source of professional data on the web — profiles, companies, job postings, and hiring signals that power recruiting tools, lead generation, and market research. It's also one of the hardest and most legally sensitive</strong> targets to scrape. This guide covers what you can realistically collect, where the limits are, and the approaches that actually hold up at scale.</p> What "scraping LinkedIn" actually means</h2> There's a critical distinction that determines both feasibility and legality:</p> Public data</strong> — pages visible to a logged-out visitor (public profiles, company pages, public job listings). The landmark hiQ Labs v. LinkedIn</em> litigation established that scraping publicly accessible</strong> data is generally not a Computer Fraud and Abuse Act violation.</li> Authenticated data</strong> — anything behind a login. Accessing it requires accepting LinkedIn's User Agreement, which prohibits scraping</strong>. Automating logged-in accounts risks bans and legal exposure.</li> </ul> The practical rule: stick to public, logged-out data, and never automate logged-in accounts.</strong> Everything below assumes public data only. Consult counsel for your specific use case — this is guidance, not legal advice.</p> Why DIY LinkedIn scraping breaks fast</h2> Even on public pages, LinkedIn is aggressive:</p> Heavy bot detection.</strong> Datacenter IPs are blocked almost immediately; you need residential proxies</a> that present as real users.</li> Rate limits and challenges.</strong> Volume triggers CAPTCHAs and soft blocks within a handful of requests per IP.</li> Constant markup churn.</strong> Profile and company DOM structures change often, breaking selector-based scrapers.</li> JavaScript rendering.</strong> Much of the page hydrates client-side, so plain HTTP requests miss data unless you render or hit the underlying endpoints.</li> </ul> A minimal public-company fetch through an unblocking proxy looks like this — useful for low volume, but fragile:</p> // linkedin-company.mjs — node linkedin-company.mjs stripe // Install: npm i axios https-proxy-agent cheerio import axios from 'axios'; import { HttpsProxyAgent } from 'https-proxy-agent'; import * as cheerio from 'cheerio'; const agent = new HttpsProxyAgent(process.env.PROXY_URL); const slug = process.argv[2] ?? 'stripe'; const { data: html } = await axios.get(`https://www.linkedin.com/company/${slug}`, { httpsAgent: agent, proxy: false, timeout: 60_000, headers: { 'Accept-Language': 'en-US,en;q=0.9' }, }); // Public company pages expose an Organization JSON-LD block. const $ = cheerio.load(html); let org = {}; $('script[type="application/ld+json"]').each((_, el) => { try { const ld = JSON.parse($(el).text()); if (ld['@type'] === 'Organization') org = ld; } catch { /* skip */ } }); console.log(JSON.stringify({ name: org.name ?? null, url: org.url ?? null, employees: org.numberOfEmployees?.value ?? null, description: org.description ?? null, }, null, 2)); </code></pre> This works for a few requests. At hundreds or thousands, you'll spend most of your time fighting blocks and patching selectors instead of using the data.</p> The scalable approach: managed collection</h2> For production LinkedIn data, the economics favor a managed collector that handles proxies, rendering, anti-bot, and schema maintenance for you. Bright Data's LinkedIn data collector</a> returns structured public profile, company, and job records via API or scheduled delivery — you specify inputs and receive clean JSON, with the unblocking infrastructure abstracted away.</p> For analysis that doesn't need real-time freshness, pre-built datasets</a> are often the better buy: large, ready-to-query snapshots of public LinkedIn data without running any scraper at all. See our datasets vs. web scraping</a> breakdown for when buying beats building.</p> Common use cases</h2> Recruiting & sourcing</strong> — public profiles matching role, skills, and location.</li> Lead generation</strong> — company size, industry, and decision-maker signals.</li> Job market intelligence</strong> — posting volume, titles, and hiring trends by company or sector.</li> Competitive monitoring</strong> — headcount growth and org changes over time.</li> </ul> Staying compliant and unblocked</h2> Public data only; never automate authenticated sessions.</li> Respect rate limits — pace requests and rotate residential IPs</a>.</li> Collect only what you need, and review data-protection rules (GDPR/CCPA) for personal data.</li> For the full anti-bot playbook, see How to Avoid Getting Blocked</a>.</li> </ul> Related: Web Scraping with Python</a>, the Bright Data review</a>, and our proxy types explained</a> guide.</em></p> Collect public LinkedIn data at scale with Bright Data →</a></strong></p> How to Handle Pagination in Web Scraping: The Complete Guide 2026-06-08T00:00:00+00:00 Pagination is one of the first obstacles every web scraper hits. Product listings, search results, news archives — any site with more items than fit on one page uses some form of it. Miss it, and your scraper silently collects a fraction of the data you need without any error to alert you.</p> This guide covers every pagination pattern you'll encounter in the wild and shows you exactly how to handle each one in Python.</p> Why Pagination Matters</h2> A single product category on a large e-commerce site can span hundreds of pages. A news site's archive may run thousands. If your scraper stops at page one, you might capture 1% of the data you actually need — and you won't even know it's missing.</p> The goal: build a loop that keeps following pages until there are no more.</p> Pattern 1: Query String Pagination (?page=N</code>)</h2> The simplest and most common pattern. Each page is a distinct URL with a numeric parameter:</p> https://example.com/products?page=1 https://example.com/products?page=2 https://example.com/products?page=3 </code></pre> How to detect it:</strong> Click "Next" or a page number and watch the URL. If a page=</code>, p=</code>, or start=</code> parameter appears or increments, you're dealing with query string pagination.</p> How to scrape it:</strong></p> import requests from bs4 import BeautifulSoup BASE_URL = "https://example.com/products" all_items = [] page = 1 while True: response = requests.get(BASE_URL, params={"page": page}) soup = BeautifulSoup(response.text, "html.parser") items = soup.select(".product-card") if not items: break # no more results all_items.extend(items) page += 1 </code></pre> The key is a termination condition</strong> — here we stop when the page returns no items. Alternatives include checking for a disabled "Next" button or comparing the current page number against a total page count you extract from the HTML.</p> Pattern 2: Offset / Cursor Pagination</h2> Many sites (and especially JSON APIs) use an offset</code> rather than a page number:</p> https://example.com/products?offset=0&limit=24 https://example.com/products?offset=24&limit=24 https://example.com/products?offset=48&limit=24 </code></pre> Handle this by incrementing offset</code> by the page size each iteration:</p> LIMIT = 24 offset = 0 while True: response = requests.get(BASE_URL, params={"limit": LIMIT, "offset": offset}) data = response.json() if not data["items"]: break process(data["items"]) offset += LIMIT </code></pre> Some APIs return a next_cursor</code> token instead of a numeric offset. Pass cursor=<token></code> in the next request and stop when the response contains no cursor field.</p> Pattern 3: "Next" Button Crawling</h2> Some sites don't expose the total page count — they just render a "Next" link. Your scraper needs to find and follow it:</p> from urllib.parse import urljoin import requests from bs4 import BeautifulSoup url = "https://example.com/products" while url: response = requests.get(url) soup = BeautifulSoup(response.text, "html.parser") items = soup.select(".product-card") process(items) next_link = soup.select_one("a.pagination__next") if next_link and next_link.get("href"): url = urljoin(url, next_link["href"]) else: url = None # no more pages </code></pre> urljoin</code> handles relative URLs gracefully — many sites link to /products?page=2</code> rather than a full absolute URL.</p> Tip:</strong> If you're getting blocked mid-pagination, rotating proxies will help. Distributing requests across different IP addresses prevents any single IP from accumulating a suspicious request count. See Residential vs. Datacenter vs. Mobile Proxies</a> for guidance on which type to use, or check out Bright Data's residential proxy network</a> which offers 400M+ IPs with automatic rotation.</p> </blockquote> Pattern 4: Infinite Scroll (AJAX / XHR)</h2> The most deceptive pattern. The page appears to have no pagination — content loads as you scroll. Under the hood, the browser fires XHR requests to load batches of results.</p> How to detect it:</strong> Open DevTools → Network → XHR/Fetch. Scroll the page and watch for new requests. You'll usually see a JSON endpoint being called with page</code> or offset</code> parameters.</p> Option A: Hit the underlying API directly.</strong> This is faster and cleaner than automating a real browser. Copy the XHR URL from DevTools and replicate the request:</p> # The underlying API endpoint discovered via DevTools API_URL = "https://example.com/api/products" headers = { "X-Requested-With": "XMLHttpRequest", "Accept": "application/json", } page = 1 while True: response = requests.get(API_URL, params={"page": page}, headers=headers) data = response.json() if not data.get("results"): break process(data["results"]) page += 1 </code></pre> Option B: Use Playwright to simulate scrolling.</strong> If the underlying API is obfuscated or requires complex auth tokens, drive a real browser:</p> from playwright.sync_api import sync_playwright with sync_playwright() as p: browser = p.chromium.launch(headless=True) page = browser.new_page() page.goto("https://example.com/products") prev_height = 0 while True: page.evaluate("window.scrollTo(0, document.body.scrollHeight)") page.wait_for_timeout(2000) # wait for new content to load new_height = page.evaluate("document.body.scrollHeight") if new_height == prev_height: break # reached the bottom prev_height = new_height html = page.content() browser.close() </code></pre> For JavaScript-heavy pagination on tough sites</strong>, a managed scraping browser handles rendering, fingerprinting, and proxy rotation for you — no Playwright configuration required. Bright Data's Scraping Browser</a> is purpose-built for exactly this use case.</p> </blockquote> Pattern 5: Directory + Detail Pages (Two-Pass Crawl)</h2> For directory-style sites — real estate listings, job boards, business directories — you typically need two loops: one for the paginated index, one for each detail page linked from it.</p> from urllib.parse import urljoin import requests from bs4 import BeautifulSoup INDEX_URL = "https://example.com/listings?page={page}" detail_urls = [] # Pass 1: collect all detail-page URLs from the paginated index page = 1 while True: soup = BeautifulSoup( requests.get(INDEX_URL.format(page=page)).text, "html.parser" ) links = [urljoin(INDEX_URL, a["href"]) for a in soup.select("a.listing-link")] if not links: break detail_urls.extend(links) page += 1 # Pass 2: scrape each detail page for url in detail_urls: detail = BeautifulSoup(requests.get(url).text, "html.parser") process(detail) </code></pre> Handling Duplicates and Resumability</h2> Long pagination runs can fail halfway through — a network error, a block, or a timeout. Protect yourself with a seen-set so you don't reprocess items, and save progress to disk so you can resume:</p> import json, os STATE_FILE = "scrape_state.json" seen = set(json.load(open(STATE_FILE)) if os.path.exists(STATE_FILE) else []) for item in scraped_items: if item["id"] not in seen: save(item) seen.add(item["id"]) # checkpoint regularly so a crash doesn't lose everything json.dump(list(seen), open(STATE_FILE, "w")) </code></pre> Rate Limiting and Politeness</h2> Pagination scrapers fire many requests in a tight loop — exactly the pattern anti-bot systems watch for. Always add a randomized delay between page requests:</p> import time, random time.sleep(random.uniform(1.5, 4.0)) </code></pre> For large crawls, combine this with proxy rotation so no single IP bears your full request volume. The full anti-blocking playbook is here</a>.</p> If you'd rather skip the infrastructure work entirely, managed scraping APIs handle proxy rotation, retries, and JavaScript rendering behind a single endpoint. ScraperAPI</a> and ZenRows</a> are popular options; Oxylabs</a> covers enterprise-scale crawls. Compare them head-to-head in our scraping API reviews</a> and Bright Data vs. Oxylabs comparison</a>.</p> Quick Reference</h2> Pagination Pattern</th> How to Detect</th> Scraping Approach</th></tr></thead> Query string (?page=N</code>)</td> URL parameter increments on "Next"</td> Loop incrementing page</code> param</td></tr> Offset (?offset=N</code>)</td> URL offset increases by page size</td> Loop incrementing offset</code> by page size</td></tr> Next-button crawl</td> "Next" link present, no page count</td> Follow href</code> of "Next" link</td></tr> Infinite scroll — JSON API</td> XHR requests visible in DevTools</td> Call the XHR endpoint directly</td></tr> Infinite scroll — rendered</td> No clean XHR endpoint found</td> Playwright scroll-to-bottom loop</td></tr> Directory + detail pages</td> Index lists links, detail pages hold data</td> Two-pass loop</td></tr> </tbody></table> The Bottom Line</h2> Most pagination boils down to a loop: request a page, collect what you need, find the next URL, repeat until done. The tricky cases — infinite scroll, obfuscated APIs, aggressive anti-bot systems — are solved either by reverse-engineering the underlying network request or by driving a real browser.</p> Build in a termination condition, add politeness delays, and rotate your IPs on any large run. Those three habits transform a fragile one-off script into a reliable data pipeline.</p> Get started with Bright Data →</a></strong></p> New to scraping? Start with our Web Scraping with Python guide</a> or read How to Avoid Getting Blocked While Scraping</a>.</em></p> Web Scraping with Playwright in Python: Full Guide 2026-06-08T00:00:00+00:00 Plain HTTP requests fail on the modern web. A growing share of sites — e-commerce storefronts, travel search engines, social platforms — render their content entirely in JavaScript. Send a requests.get()</code> to one of those URLs and you'll get an empty shell. That's where Playwright comes in.</p> Playwright is a browser automation library from Microsoft that drives a real Chromium, Firefox, or WebKit engine from Python code. Because it runs a real browser, JavaScript executes, SPAs render, and lazy-loaded content appears — exactly as it would for a human visitor. This guide covers everything you need to scrape dynamic websites with Playwright in Python, from initial setup to production-ready techniques.</p> Installation and Setup</h2> Install the library and download the browser binaries:</p> pip install playwright playwright install chromium </code></pre> Playwright manages the browser installation for you — no separate ChromeDriver or binary to keep in sync.</p> Basic Scraping: Your First Playwright Script</h2> A minimal scraper looks like this:</p> from playwright.sync_api import sync_playwright with sync_playwright() as p: browser = p.chromium.launch(headless=True) page = browser.new_page() page.goto("https://example.com") title = page.title() html = page.content() # full rendered HTML after JS executes print(title) browser.close() </code></pre> headless=True</code> runs the browser invisibly. Switch to headless=False</code> during development to watch what's happening in real time.</p> Waiting for Dynamic Content</h2> The most common mistake is reading the page before JavaScript has finished rendering it. Playwright offers several strategies:</p> # Wait for a specific element to appear in the DOM page.wait_for_selector(".product-list") # Wait until network activity has settled page.goto("https://example.com", wait_until="networkidle") # Wait for a URL pattern (useful after form submissions or redirects) page.wait_for_url("**/results**") </code></pre> For infinite-scroll pages, trigger scrolls programmatically and pause for content to load:</p> for _ in range(5): page.evaluate("window.scrollTo(0, document.body.scrollHeight)") page.wait_for_timeout(1500) </code></pre> Extracting Data from Rendered Pages</h2> Once the page is rendered, use Playwright's locators or hand the HTML to BeautifulSoup for parsing:</p> from bs4 import BeautifulSoup # Direct extraction with Playwright locators titles = page.locator(".product-title").all_inner_texts() price = page.locator("span.price").first.inner_text() href = page.locator("a.product-link").first.get_attribute("href") # Alternatively, pass rendered HTML to BeautifulSoup soup = BeautifulSoup(page.content(), "html.parser") cards = soup.select(".product-card") for card in cards: print(card.select_one("h2").get_text(strip=True)) </code></pre> Handling Login-Gated Content</h2> Playwright handles authentication naturally by filling forms just as a user would:</p> page.goto("https://example.com/login") page.fill("#email", "user@example.com") page.fill("#password", "s3cr3t") page.click("button[type='submit']") page.wait_for_url("**/dashboard") # Save session state so you don't log in on every run page.context.storage_state(path="session.json") </code></pre> On subsequent runs, restore the saved session:</p> context = browser.new_context(storage_state="session.json") page = context.new_page() </code></pre> Scraping Across Multiple Pages</h2> Most real scrapers need to walk through pagination. Here's a clean pattern:</p> results = [] page_num = 1 while True: page.goto(f"https://example.com/products?page={page_num}") page.wait_for_selector(".product-card") items = page.locator(".product-card").all_inner_texts() if not items: break results.extend(items) # Stop when there is no "Next" button next_btn = page.locator("a.next-page") if not next_btn.is_visible(): break page_num += 1 page.wait_for_timeout(2000) # polite delay between pages print(f"Scraped {len(results)} items across {page_num} pages") </code></pre> For a broader look at avoiding rate limits and bans during multi-page runs, see How to Avoid Getting Blocked While Web Scraping</a>.</p> </blockquote> Reducing Your Fingerprint: Stealth Settings</h2> Headless Playwright leaks navigator.webdriver = true</code> and a handful of other signals that anti-bot systems watch for. A few settings reduce that exposure significantly:</p> context = browser.new_context( viewport={"width": 1920, "height": 1080}, locale="en-US", timezone_id="America/New_York", user_agent=( "Mozilla/5.0 (Windows NT 10.0; Win64; x64) " "AppleWebKit/537.36 (KHTML, like Gecko) " "Chrome/124.0.0.0 Safari/537.36" ), ) page = context.new_page() # Mask the webdriver flag before any page load page.add_init_script("delete Object.getPrototypeOf(navigator).webdriver") </code></pre> For a more comprehensive set of stealth patches, install playwright-stealth</code>:</p> pip install playwright-stealth </code></pre> from playwright_stealth import stealth_sync stealth_sync(page) page.goto("https://example.com") </code></pre> Rotating Proxies with Playwright</h2> Distributing requests across many IP addresses is essential once you move beyond toy projects. Playwright supports proxies natively at the browser or context level:</p> context = browser.new_context( proxy={ "server": "http://proxy.example.com:8080", "username": "user", "password": "pass", } ) page = context.new_page() </code></pre> For serious scraping, residential proxies are far less likely to be blocked than datacenter IPs — they route through real consumer devices and look indistinguishable from normal traffic. See our proxy types guide</a> for a full breakdown of your options.</p> Recommended proxies for Playwright:</strong> Bright Data</a> offers a 400M+ residential IP pool with automatic rotation. Budget-friendly alternatives include IPRoyal</a>, DataImpulse</a>, and HydraProxy</a>. Browse all head-to-head comparisons in our proxy and scraper reviews</a>.</p> </blockquote> Async Playwright for Parallel Scraping</h2> The sync API is convenient for scripts, but for scraping many URLs at once, the async API lets you run browser contexts concurrently:</p> import asyncio from playwright.async_api import async_playwright async def scrape_url(url: str) -> str: async with async_playwright() as p: browser = await p.chromium.launch(headless=True) page = await browser.new_page() await page.goto(url) content = await page.content() await browser.close() return content async def main(): urls = ["https://example.com/page/1", "https://example.com/page/2"] results = await asyncio.gather(*[scrape_url(u) for u in urls]) for html in results: print(len(html), "bytes") asyncio.run(main()) </code></pre> For production crawls, consider managing a shared browser instance and creating a new context per task rather than a new browser process — it's significantly cheaper on resources.</p> When Self-Hosted Playwright Isn't Enough</h2> Even with stealth settings and good proxies, the most aggressively protected targets (Cloudflare Turnstile, Akamai Bot Manager, PerimeterX) can still block a self-managed browser. At that point, the economics shift in favor of a managed solution.</p> The Bright Data Scraping Browser</a> connects your existing Playwright code to a hosted browser that adds built-in CAPTCHA solving and residential IP rotation on every request — your scripts stay almost identical, but Bright Data handles staying unblocked. For simpler HTTP-based needs, ZenRows</a> and ScraperAPI</a> offer one-endpoint Web Unlocker APIs that handle rendering and unblocking without browser automation.</p> Compare providers in our scraping service comparisons</a>.</p> Quick Reference</h2> Task</th> Playwright API</th></tr></thead> Wait for element</td> page.wait_for_selector(".class")</code></td></tr> Wait for network quiet</td> goto(..., wait_until="networkidle")</code></td></tr> Scroll to load more</td> page.evaluate("window.scrollTo(...)")</code></td></tr> Extract text list</td> page.locator(".item").all_inner_texts()</code></td></tr> Get attribute</td> .get_attribute("href")</code></td></tr> Save/restore session</td> storage_state(path=...)</code></td></tr> Use proxy</td> new_context(proxy={...})</code></td></tr> </tbody></table> The Bottom Line</h2> Playwright is the most capable self-hosted option for scraping JavaScript-heavy sites. Its Python API is clean, its waiting primitives are reliable, and it handles everything from simple HTML extraction to authenticated multi-page crawls. Pair it with stealth patches and rotating residential proxies, and you have a scraper that handles the vast majority of real-world targets without a managed service.</p> Get rotating residential proxies for your Playwright scraper →</a></strong></p> New to scraping? Start with our Web Scraping with Python guide</a>, or read How to Avoid Getting Blocked</a> for the full anti-bot playbook.</em></p> eBay Product Search Scraping: Code Samples 2026-06-08T00:00:00+00:00 Scraping eBay search results gives you market-level pricing data that a single product page can't: you see dozens of competing listings at once, complete with condition grades, shipping costs, and seller ratings. That makes eBay one of the best targets for price benchmarking, resale arbitrage research, and competitive inventory monitoring.</p> The catch is that eBay's results pages are JavaScript-rendered and protected by bot detection, so plain HTTP requests usually return thin markup or a challenge page. The samples below route requests through ZenRows</a> — a managed scraping browser that handles rendering and fingerprint bypasses on your behalf — and then parse the returned HTML with standard libraries.</p> Prerequisites</h2> Sign up for a ZenRows</a> account and set your API key:</p> export ZENROWS_API_KEY="your_api_key_here" </code></pre> Each sample accepts a search query as a command-line argument and outputs a JSON array of listings. Each object contains the title, price string, condition label, shipping cost, and canonical listing URL (tracking parameters stripped).</p> Understanding eBay Search HTML</h2> eBay renders each result as an <li class="s-item"></code> inside <ul class="srp-results"></code>. The useful sub-elements are:</p> Field</th> CSS Selector</th></tr></thead> Title</td> .s-item__title</code></td></tr> Price</td> .s-item__price</code></td></tr> Condition</td> .SECONDARY_INFO</code></td></tr> Shipping</td> .s-item__shipping</code></td></tr> Listing URL</td> a.s-item__link</code> (href attribute)</td></tr> </tbody></table> The first s-item</code> in every results page is always a ghost "Shop on eBay" placeholder row that eBay injects — the samples below filter it out.</p> PHP</h2> <?php // Run: php ebay.php "mechanical keyboard" $apiKey = getenv('ZENROWS_API_KEY'); $query = $argv[1] ?? 'mechanical keyboard'; $targetUrl = 'https://www.ebay.com/sch/i.html?' . http_build_query([ '_nkw' => $query, '_sop' => 12, // sort by best match ]); $apiUrl = 'https://api.zenrows.com/v1/?' . http_build_query([ 'apikey' => $apiKey, 'url' => $targetUrl, 'js_render' => 'true', ]); $ch = curl_init($apiUrl); curl_setopt_array($ch, [ CURLOPT_RETURNTRANSFER => true, CURLOPT_TIMEOUT => 60, ]); $html = curl_exec($ch); if ($html === false) { fwrite(STDERR, 'Request failed: ' . curl_error($ch) . PHP_EOL); exit(1); } curl_close($ch); $doc = new DOMDocument(); @$doc->loadHTML($html); $xp = new DOMXPath($doc); $text = fn(DOMNode $ctx, string $q): string => trim($xp->query($q, $ctx)->item(0)?->textContent ?? ''); $items = []; foreach ($xp->query('//li[contains(@class,"s-item")]') as $li) { $title = $text($li, './/*[contains(@class,"s-item__title")]'); if (!$title || str_starts_with($title, 'Shop on eBay')) continue; $href = $xp->query('.//a[contains(@class,"s-item__link")]', $li) ->item(0)?->getAttribute('href') ?? ''; $items[] = [ 'title' => $title, 'price' => $text($li, './/*[contains(@class,"s-item__price")]'), 'condition' => $text($li, './/*[contains(@class,"SECONDARY_INFO")]'), 'shipping' => $text($li, './/*[contains(@class,"s-item__shipping")]'), 'url' => strtok($href, '?'), ]; } echo json_encode($items, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES), PHP_EOL; </code></pre> Node.js</h2> // ebay.mjs — node ebay.mjs "mechanical keyboard" // Install: npm i axios cheerio import axios from 'axios'; import * as cheerio from 'cheerio'; const apiKey = process.env.ZENROWS_API_KEY; const query = process.argv[2] ?? 'mechanical keyboard'; const targetUrl = `https://www.ebay.com/sch/i.html?_nkw=${encodeURIComponent(query)}&_sop=12`; const { data: html } = await axios.get('https://api.zenrows.com/v1/', { params: { apikey: apiKey, url: targetUrl, js_render: 'true' }, timeout: 60_000, }); const $ = cheerio.load(html); const items = []; $('li.s-item').each((_, el) => { const title = $(el).find('.s-item__title').text().trim(); if (!title || title.startsWith('Shop on eBay')) return; items.push({ title, price: $(el).find('.s-item__price').first().text().trim(), condition: $(el).find('.SECONDARY_INFO').text().trim(), shipping: $(el).find('.s-item__shipping').text().trim(), url: $(el).find('a.s-item__link').attr('href')?.split('?')[0] ?? '', }); }); console.log(JSON.stringify(items, null, 2)); </code></pre> Rust</h2> // Cargo.toml: // reqwest = { version = "0.12", features = ["blocking"] } // scraper = "0.20" // serde_json = "1" // urlencoding = "2" use scraper::{Html, Selector}; use serde_json::{json, Value}; fn sel(s: &str) -> Selector { Selector::parse(s).unwrap() } fn pick_text(root: &scraper::ElementRef, selector: &Selector) -> String { root.select(selector) .next() .map(|e| e.text().collect::<String>().trim().to_string()) .unwrap_or_default() } fn main() -> Result<(), Box<dyn std::error::Error>> { let api_key = std::env::var("ZENROWS_API_KEY")?; let query = std::env::args().nth(1).unwrap_or_else(|| "mechanical keyboard".into()); let target = format!( "https://www.ebay.com/sch/i.html?_nkw={}&_sop=12", urlencoding::encode(&query) ); let html = reqwest::blocking::Client::new() .get("https://api.zenrows.com/v1/") .query(&[ ("apikey", api_key.as_str()), ("url", target.as_str()), ("js_render", "true"), ]) .timeout(std::time::Duration::from_secs(60)) .send()? .text()?; let doc = Html::parse_document(&html); let item_sel = sel("li.s-item"); let title_sel = sel(".s-item__title"); let price_sel = sel(".s-item__price"); let cond_sel = sel(".SECONDARY_INFO"); let ship_sel = sel(".s-item__shipping"); let link_sel = sel("a.s-item__link"); let mut items: Vec<Value> = Vec::new(); for el in doc.select(&item_sel) { let title = pick_text(&el, &title_sel); if title.is_empty() || title.starts_with("Shop on eBay") { continue; } let url = el.select(&link_sel) .next() .and_then(|a| a.value().attr("href")) .map(|h| h.split('?').next().unwrap_or(h).to_string()) .unwrap_or_default(); items.push(json!({ "title": title, "price": pick_text(&el, &price_sel), "condition": pick_text(&el, &cond_sel), "shipping": pick_text(&el, &ship_sel), "url": url, })); } println!("{}", serde_json::to_string_pretty(&items)?); Ok(()) } </code></pre> Paginating Through Results</h2> eBay shows up to 240 results per page and uses the _pgn</code> parameter for pagination. To walk multiple pages, increment _pgn</code> from 1 upward and stop when the results list is empty:</p> https://www.ebay.com/sch/i.html?_nkw=mechanical+keyboard&_sop=12&_pgn=2 </code></pre> Add a short pause between requests (one to two seconds) to stay within polite crawl rates.</p> Targeting Completed Sales for Price Benchmarks</h2> To get sold prices instead of active asking prices — useful for understanding true market value — add the completed-listings parameters:</p> &LH_Complete=1&LH_Sold=1 </code></pre> Sold listings are a much more reliable baseline for pricing decisions than active listings, since active prices reflect what sellers want</em> rather than what buyers actually pay.</p> Building a Price Monitor</h2> Turn this scraper into a lightweight price monitor:</p> Run on a schedule</strong> — use a cron job or task queue. Daily is fine for slow-moving categories; hourly suits fast-moving ones like consumer electronics.</li> Store price history</strong> — write {query, title, price, timestamp}</code> to a database or CSV on each run.</li> Alert on drops</strong> — compare the latest price against a rolling minimum and fire a notification when a new low appears.</li> </ol> The Amazon Product Tracking</a> and Walmart Product Tracking</a> guides use the same store-and-compare pattern with different parsers. To track a single eBay listing by item ID (rather than search results), see eBay Product Tracking</a>.</p> Notes</h2> Selector drift</strong> — eBay's CSS classes are reasonably stable, but they shift with major redesigns. If results come back empty, open the live page in DevTools, inspect a listing card, and update the selectors.</li> Price strings</strong> — prices can be ranges ($20.00 to $60.00</code> for lot sales) or include text like "or Best Offer". Store the raw string and parse it downstream, or use a regex to extract the lower bound.</li> Alternative unblocking services</strong> — if you need higher throughput or different billing models, Bright Data's Web Unlocker</a> and Oxylabs</a> are strong alternatives. See our ZenRows vs. ScraperAPI comparison</a> and the broader proxy and scraping tool comparisons</a> for a side-by-side view.</li> </ul> See also: E-commerce Web Scraping Solutions</a>, ZenRows review</a>, and How to Avoid Getting Blocked</a>.</em></p> Get started with ZenRows →</a></strong></p> Bright Data vs ZenRows: Proxy Platform or Anti-Bot API? 2026-06-03T00:00:00+00:00 Bright Data</a> and ZenRows</a> both get you unblocked data at scale, but they come at the problem from different angles. Bright Data is a full proxy platform with the industry's largest network and a suite of unblocking tools. ZenRows is an anti-bot-first scraping API that wraps everything into a single endpoint. Here's how to choose.</p> Approach</h2> Bright Data:</strong> A complete proxy platform — pick proxy types, configure rotation, and layer on tools like the Web Unlocker</a> and Scraping Browser</a>. Maximum control.</li> ZenRows:</strong> A scraping API focused on anti-bot bypass — send a URL, get clean HTML/JSON/Markdown back. Maximum simplicity.</li> </ul> Proxy Network</h2> Bright Data:</strong> Over 400 million residential IPs</strong> across 195 countries, plus datacenter, ISP, and mobile proxies — exposed for direct use.</li> ZenRows:</strong> A large residential pool across 190+ countries, abstracted behind the API via premium_proxy=true</code>.</li> </ul> Bright Data gives you the raw network; ZenRows manages proxy selection for you.</p> Anti-Bot & Features</h2> Bright Data:</strong> Web Unlocker and Scraping Browser handle CAPTCHAs, fingerprinting, and retries; plus SERP API</a> and ready-made datasets</a>.</li> ZenRows:</strong> Anti-bot bypass (Cloudflare, DataDome, PerimeterX, Akamai) is the core product, with JS rendering, a Scraping Browser, and HTML/JSON/Markdown output for AI pipelines.</li> </ul> Pricing</h2> Bright Data:</strong> Pay-as-you-go, bandwidth-based pricing with volume discounts.</li> ZenRows:</strong> Credit/request-based pricing with a free trial; cost scales with JS rendering and premium proxies.</li> </ul> Bandwidth-based vs. request-based billing is a key difference: heavy-HTML pages can favor a request model, while light high-volume scraping can favor bandwidth pricing.</p> Quick Comparison</h2> </th> Bright Data</th> ZenRows</th></tr></thead> Model</td> Proxy network + tools</td> Anti-bot scraping API</td></tr> Control</td> High</td> Low (handled for you)</td></tr> Network</td> 400M+ residential</td> Large residential pool</td></tr> Billing</td> Per GB</td> Per credit/request</td></tr> Best for</td> Custom, large-scale pipelines</td> Fast anti-bot bypass</td></tr> </tbody></table> Conclusion</h2> Use Bright Data if:</strong> You want control, the largest network, and a full unblocking toolkit for custom pipelines. Read the full Bright Data review</a>.</li> Use ZenRows if:</strong> You want the simplest path past aggressive anti-bot systems, with clean HTML/Markdown output. Read the full ZenRows review</a>.</li> </ul> Also weighing scraping APIs? See ZenRows vs ScraperAPI</a>.</em></p> ZenRows vs ScraperAPI: Which Web Scraping API Wins? 2026-06-03T00:00:00+00:00 ZenRows</a> and ScraperAPI</a> are two of the most popular web scraping APIs — both turn "managing proxies and browsers" into a single API call. They overlap heavily, but they lean in different directions: ZenRows is anti-bot-first, while ScraperAPI is simplicity-and-scale-first. Here's how they compare.</p> Approach</h2> ZenRows:</strong> Built to defeat aggressive anti-bot systems (Cloudflare, DataDome, PerimeterX, Akamai), with a Scraping Browser and multiple output formats.</li> ScraperAPI:</strong> Built for easy, high-volume scraping with structured data endpoints and a generous free tier.</li> </ul> Anti-Bot & Proxies</h2> ZenRows:</strong> Anti-bot bypass is the headline feature; residential proxies across 190+ countries via premium_proxy=true</code>, plus fingerprint/CAPTCHA handling tuned for protected targets.</li> ScraperAPI:</strong> Datacenter, residential, and mobile proxies with automatic rotation; premium tiers (premium=true</code> / ultra_premium=true</code>) raise success on harder sites.</li> </ul> ZenRows tends to have the edge on the most heavily protected targets; ScraperAPI is very capable on mainstream sites.</p> Features</h2> ZenRows:</strong> Universal Scraper API, Scraping Browser, JS rendering with click/scroll instructions, and output as HTML, JSON (auto-parse), or Markdown</strong> — great for AI/LLM pipelines.</li> ScraperAPI:</strong> Single-endpoint API, JS rendering, async jobs/DataPipeline, and pre-built structured data endpoints</strong> for Amazon, Google Search, and Google Shopping.</li> </ul> Pricing</h2> Both use credit-based, pay-as-you-grow</strong> pricing where JS rendering and premium proxies cost more credits:</p> ZenRows:</strong> Free trial available; paid plans from an entry developer tier up to enterprise.</li> ScraperAPI:</strong> Free monthly credits plus a trial; entry tier scaling to high-volume plans.</li> </ul> Check current pricing for ZenRows</a> and ScraperAPI</a> — plans change periodically.</p> </blockquote> Quick Comparison</h2> </th> ZenRows</th> ScraperAPI</th></tr></thead> Strength</td> Anti-bot bypass</td> Simplicity & scale</td></tr> Output formats</td> HTML, JSON, Markdown</td> HTML, JSON (structured endpoints)</td></tr> Scraping browser</td> Yes</td> Via render mode</td></tr> Free tier</td> Trial credits</td> Free monthly credits + trial</td></tr> Best for</td> Heavily protected targets</td> Mainstream & e-commerce/SERP</td></tr> </tbody></table> Conclusion</h2> Use ZenRows if:</strong> Your targets are protected by Cloudflare/DataDome and anti-bot bypass is your priority, or you want Markdown output for AI pipelines. Read the full ZenRows review</a>.</li> Use ScraperAPI if:</strong> You want the simplest integration, a generous free tier, and ready-made structured endpoints for e-commerce and search. Read the full ScraperAPI review</a>.</li> </ul> Need a full proxy network with maximum control instead? See Bright Data</a>.</em></p> ZenRows Review: Anti-Bot Web Scraping API That Just Works 2026-06-03T00:00:00+00:00 ZenRows is a web scraping toolkit built around one hard problem: getting past modern anti-bot systems. With a single API call it bypasses Cloudflare, DataDome, PerimeterX, and Akamai, renders JavaScript, rotates premium residential proxies, and hands you clean HTML, JSON, or Markdown. For developers who keep hitting blocks on tough targets, it's one of the most capable plug-and-play options available.</p> How It Works</h2> Instead of running your own headless browsers and proxy rotation, you send a target URL to ZenRows' Universal Scraper API with your API key. ZenRows picks the right proxy, solves anti-bot challenges, optionally renders JavaScript, and returns the result:</p> curl "https://api.zenrows.com/v1/?apikey=YOUR_KEY&url=https://example.com&js_render=true&premium_proxy=true" </code></pre> That one request transparently handles fingerprinting, CAPTCHAs, retries, and rotation — no infrastructure on your side.</p> Key Features</h2> Universal Scraper API:</strong> Scrape any page with a single GET request — works in any language.</li> Advanced anti-bot bypass:</strong> Designed to defeat Cloudflare, DataDome, PerimeterX, and Akamai automatically.</li> Scraping Browser:</strong> A hosted, headless browser endpoint for Playwright/Puppeteer/Selenium when you need full interaction.</li> Residential proxies:</strong> A large pool of residential IPs across 190+ countries with city/country geotargeting (premium_proxy=true</code>).</li> JavaScript rendering:</strong> Add js_render=true</code> for dynamic, JS-heavy pages, with support for click/scroll/wait JS instructions.</li> AI-powered auto-parsing & output formats:</strong> Return raw HTML, structured JSON via auto-parse, or clean Markdown — handy for LLM/RAG pipelines.</li> High concurrency:</strong> Run many requests in parallel, scaling with your plan tier.</li> </ul> Best For</h2> Developers scraping heavily protected targets</strong> that block typical requests and basic scraper APIs</li> Teams that want anti-bot bypass and proxies behind one simple endpoint</strong></li> Feeding clean HTML/Markdown into AI and LLM data pipelines</strong></li> </ul> Pricing</h2> ZenRows uses a request/credit-based, pay-as-you-grow model</strong>. Plans scale by API credits per month, with higher tiers unlocking more concurrency and premium residential proxies. Requests that need JS rendering or premium proxies consume more credits than a basic request.</p> Free trial:</strong> Free test credits to start.</li> Paid plans:</strong> Start at an entry-level monthly developer tier and scale up to business and enterprise.</li> </ul> Credit costs vary by request type. Check the current pricing</a> for exact figures, as plans are updated periodically.</p> </blockquote> Performance</h2> ZenRows' strength is success rate on hard targets</strong>. On sites guarded by Cloudflare or DataDome — where plain requests and lighter scraper APIs fail — enabling premium_proxy=true</code> and js_render=true</code> delivers consistently high success rates. For simple, unprotected pages a basic plan or a plain proxy may be more cost-effective, but for the tough cases ZenRows is built exactly for that fight.</p> Pros & Cons</h2> Pros</strong></p> Best-in-class anti-bot bypass behind a single API call</li> Multiple output formats (HTML, JSON, Markdown) — great for AI pipelines</li> Scraping Browser option for full interaction when you need it</li> Free trial available</li> </ul> Cons</strong></p> Premium proxies and JS rendering consume credits faster</li> Less granular raw-proxy control than a dedicated network like Bright Data</a></li> </ul> Conclusion</h2> ZenRows is an excellent choice when getting blocked is your main problem</strong>. Its anti-bot bypass, residential proxies, and flexible output formats remove nearly all the friction of scraping protected sites — without managing any infrastructure. If you only scrape simple pages, a basic proxy may be cheaper, but for tough targets ZenRows is one of the most reliable APIs on the market.</p> Start scraping with ZenRows →</a></strong></p> Comparing options? See ZenRows vs ScraperAPI</a>, Bright Data vs ZenRows</a>, and our Bright Data review</a>.</em></p> Looking at Other Options?</h2> If ZenRows doesn't fit your workload, we've rounded up the best ZenRows alternatives</a> — including how they differ on pricing model, proxy quality, and API features.</p> Bright Data vs IPRoyal: Premium Power or Budget Flexibility? 2026-06-02T00:00:00+00:00 Bright Data</a> and IPRoyal</a> sit at two ends of the proxy market. Bright Data is the enterprise gold standard with the industry's largest network and a full suite of unblocking tools. IPRoyal is the flexible, affordable favorite for developers who want dependable proxies without a big commitment. Which one fits your project?</p> Proxy Network</h2> Bright Data:</strong> Over 400 million residential IPs</strong> across 195 countries, plus datacenter, ISP, and 7M+ mobile IPs.</li> IPRoyal:</strong> Millions of ethically sourced residential IPs with country, state, and city targeting, plus ISP, datacenter, mobile, and sneaker proxies.</li> </ul> Bright Data's network is far larger and more granular, which matters most when you need high concurrency or hard-to-reach geographies.</p> Features</h2> Bright Data:</strong> A complete unblocking platform — Web Unlocker</a>, Scraping Browser</a>, SERP API</a>, and ready-made datasets</a>. Built-in CAPTCHA solving and anti-bot evasion.</li> IPRoyal:</strong> Focused, no-frills proxies with one killer perk — residential traffic that never expires</strong> — and simple pay-as-you-go billing.</li> </ul> Bright Data is a platform; IPRoyal is a clean, affordable proxy service.</p> Pricing</h2> Bright Data:</strong> Pay-as-you-go, bandwidth-based pricing with volume discounts on monthly and yearly plans.</li> IPRoyal:</strong> Competitively priced per GB with non-expiring traffic and no monthly commitment — generally cheaper for small to mid-size projects.</li> </ul> Performance</h2> In testing, Bright Data's residential proxies hit 99.5%+ success rates</strong> on tough targets like Amazon, Walmart, and Google, with sub-2-second average response times. IPRoyal delivers solid, dependable performance on mainstream targets at a lower price, though premium providers keep an edge on the most aggressively defended sites.</p> Quick Comparison</h2> </th> Bright Data</th> IPRoyal</th></tr></thead> Network size</td> 400M+ residential</td> Millions of IPs</td></tr> Tooling</td> Full unblocking suite</td> Proxies only</td></tr> Pricing</td> Premium</td> Budget-friendly</td></tr> Free trial</td> Available</td> Pay-as-you-go</td></tr> Best for</td> Enterprise / tough targets</td> Cost-conscious developers</td></tr> </tbody></table> Conclusion</h2> Use Bright Data if:</strong> You need maximum success rates, the largest network, or unblocking tools like the Web Unlocker and Scraping Browser. Read the full Bright Data review</a>.</li> Use IPRoyal if:</strong> You want reliable, affordable proxies for mainstream targets and value non-expiring traffic. Read the full IPRoyal review</a>.</li> </ul> Get started with Bright Data →</a></strong></p> Bright Data vs ScraperAPI: Proxy Network or Scraping API? 2026-06-02T00:00:00+00:00 Bright Data</a> and ScraperAPI</a> solve the same problem — getting unblocked data at scale — but with different philosophies. Bright Data gives you a vast proxy network plus a toolkit of unblocking products. ScraperAPI wraps everything into a single, dead-simple API endpoint. Here's how to choose.</p> Approach</h2> Bright Data:</strong> A full proxy platform. You pick proxy types, configure rotation, and optionally layer on tools like the Web Unlocker</a> and Scraping Browser</a>. Maximum control.</li> ScraperAPI:</strong> Send a URL to one API endpoint and get clean HTML back. Proxy rotation, retries, and CAPTCHA handling are all hidden behind the call. Maximum simplicity.</li> </ul> Proxy Network</h2> Bright Data:</strong> Over 400 million residential IPs</strong> across 195 countries, plus datacenter, ISP, and mobile.</li> ScraperAPI:</strong> Automatic rotation across datacenter, residential, and mobile proxies (millions of IPs), selected for you based on the request.</li> </ul> Bright Data exposes the raw network; ScraperAPI abstracts it away.</p> Features</h2> Bright Data:</strong> Web Unlocker, Scraping Browser, SERP API</a>, ready-made datasets</a>, built-in CAPTCHA solving and fingerprinting.</li> ScraperAPI:</strong> JavaScript rendering, automatic proxy/CAPTCHA handling, structured data endpoints, and premium proxy modes (premium=true</code> / ultra_premium=true</code>) for tougher targets.</li> </ul> Pricing</h2> Bright Data:</strong> Pay-as-you-go, bandwidth-based pricing with volume discounts. You pay for bandwidth.</li> ScraperAPI:</strong> A credit-based, pay-as-you-grow</strong> model — plans scale by monthly API credits, with a generous free tier and a free trial. You pay per request (JS rendering and premium proxies cost more credits).</li> </ul> Bandwidth-based vs. request-based billing is the key difference: heavy-HTML pages favor a credit model, while light high-volume scraping can favor bandwidth pricing.</p> Quick Comparison</h2> </th> Bright Data</th> ScraperAPI</th></tr></thead> Model</td> Proxy network + tools</td> All-in-one scraping API</td></tr> Control</td> High</td> Low (handled for you)</td></tr> Billing</td> Per GB</td> Per credit/request</td></tr> Setup effort</td> Moderate</td> Minimal</td></tr> Best for</td> Custom, large-scale pipelines</td> Fast, simple integration</td></tr> </tbody></table> Conclusion</h2> Use Bright Data if:</strong> You want control, the largest network, and a full unblocking toolkit for custom pipelines. Read the full Bright Data review</a>.</li> Use ScraperAPI if:</strong> You want the simplest possible integration — one endpoint, no proxy management. Read the full ScraperAPI review</a>.</li> </ul> Not sure which proxy type you even need? See our guide to residential vs. datacenter vs. mobile proxies</a>.</em></p> HydraProxy vs IPRoyal: Best Flexible, Low-Budget Proxies? 2026-06-02T00:00:00+00:00 HydraProxy</a> and IPRoyal</a> both target developers who want flexible, affordable proxies without locking into a big monthly plan — and both offer traffic that doesn't expire. They're especially popular with solo developers and social media managers. So which one should you pick?</p> Proxy Network</h2> HydraProxy:</strong> Rotating residential proxies plus real 4G/5G mobile IPs</strong>, with a focus on flexibility and small budgets.</li> IPRoyal:</strong> A broader lineup — residential, static residential (ISP), datacenter, mobile, and sneaker proxies — with country, state, and city-level targeting.</li> </ul> IPRoyal offers more proxy types and finer geo-targeting; HydraProxy keeps things lean and is particularly handy for mobile-IP use cases.</p> Standout Features</h2> HydraProxy:</strong> Very low minimum top-ups</strong> and pay-as-you-go billing make it easy to start tiny. Strong mobile-proxy option for social and app targets.</li> IPRoyal:</strong> Non-expiring residential traffic</strong>, broad proxy-type coverage, and city-level targeting.</li> </ul> Both offer non-expiring traffic — a big plus for occasional use.</p> Pricing</h2> Both are low-cost, pay-as-you-go, with no monthly commitment:</p> HydraProxy:</strong> Micro-budget friendly with small minimum top-ups — ideal when you only need a little traffic.</li> IPRoyal:</strong> Competitively priced per GB with non-expiring traffic and more proxy types to choose from.</li> </ul> Promotional rates change often — check current pricing for HydraProxy</a> and IPRoyal</a> directly.</p> </blockquote> Performance</h2> Both perform well on mainstream targets. HydraProxy shines for mobile-IP tasks like managing social accounts, while IPRoyal is a dependable all-rounder across residential and ISP networks. For the most aggressive enterprise anti-bot systems, a premium provider like Bright Data</a> remains stronger.</p> Quick Comparison</h2> </th> HydraProxy</th> IPRoyal</th></tr></thead> Minimum top-up</td> Very low</td> Low</td></tr> Proxy types</td> Residential, mobile</td> Residential, ISP, DC, mobile, sneaker</td></tr> Traffic expiry</td> Non-expiring</td> Non-expiring</td></tr> Standout</td> Cheapest entry + mobile</td> Most proxy types + geo-targeting</td></tr> Best for</td> Social tasks, tiny budgets</td> All-round flexible scraping</td></tr> </tbody></table> Conclusion</h2> Use HydraProxy if:</strong> You want the lowest possible entry point and need mobile IPs for social or app tasks. Read the full HydraProxy review</a>.</li> Use IPRoyal if:</strong> You want more proxy types, finer geo-targeting, and a dependable all-rounder. Read the full IPRoyal review</a>.</li> </ul> For the toughest targets, step up to Bright Data</a>.</em></p> IPRoyal vs DataImpulse: Which Budget Proxy Provider Wins? 2026-06-02T00:00:00+00:00 If you want reliable residential proxies without enterprise-level bills, IPRoyal</a> and DataImpulse</a> are two of the most popular budget-friendly, pay-as-you-go options. Both skip mandatory subscriptions and let you pay only for the traffic you use — but they have different strengths. Here's how they compare.</p> Proxy Network</h2> IPRoyal:</strong> Millions of ethically sourced residential IPs with country, state, and city-level targeting, plus static residential (ISP), datacenter, mobile, and sneaker proxies.</li> DataImpulse:</strong> Millions of ethically sourced residential IPs across virtually every country, plus mobile, datacenter, and ISP proxies from a single dashboard.</li> </ul> Both cover the major proxy types. IPRoyal edges ahead on niche options (notably dedicated sneaker proxies), while DataImpulse keeps everything simple in one unified dashboard.</p> Standout Features</h2> IPRoyal:</strong> Its headline perk is non-expiring residential traffic</strong> — GBs you buy never expire, which is genuinely useful for occasional or seasonal scraping.</li> DataImpulse:</strong> Built around rock-bottom pricing</strong> and an easy top-up balance model that makes it trivial to start small and scale up.</li> </ul> Pricing</h2> Both use affordable pay-as-you-go pricing with no forced monthly commitment:</p> IPRoyal:</strong> Competitively priced per GB, with the standout benefit that purchased residential traffic doesn't expire.</li> DataImpulse:</strong> Among the most affordable residential proxies on the market, billed against a top-up balance.</li> </ul> Promotional rates change often — check current per-GB pricing for IPRoyal</a> and DataImpulse</a> directly.</p> </blockquote> Performance</h2> Both deliver reliable success rates on mainstream targets across their residential and mobile networks. For the most aggressive enterprise-grade anti-bot systems, a premium provider like Bright Data</a> still has the edge — but for everyday scraping, both offer excellent value.</p> Quick Comparison</h2> </th> IPRoyal</th> DataImpulse</th></tr></thead> Pricing model</td> Pay-as-you-go</td> Pay-as-you-go (balance)</td></tr> Standout perk</td> Non-expiring traffic</td> Lowest-cost residential</td></tr> Proxy types</td> Residential, ISP, DC, mobile, sneaker</td> Residential, mobile, DC, ISP</td></tr> Best for</td> Occasional / seasonal scraping</td> Indie devs on a tight budget</td></tr> </tbody></table> Conclusion</h2> Use IPRoyal if:</strong> You scrape occasionally or seasonally and want traffic that never expires, or you need sneaker proxies. Read the full IPRoyal review</a>.</li> Use DataImpulse if:</strong> Absolute lowest cost is your priority and you want a simple top-up model. Read the full DataImpulse review</a>.</li> </ul> Need maximum success rates on heavily protected sites instead? See Bright Data</a>.</em></p> Bright Data Scraping Browser: CAPTCHA-Solving Cloud Browser 2026-06-02T00:00:00+00:00 If you've ever built a scraper that worked perfectly in testing only to get blocked, fingerprinted, or buried under CAPTCHAs the moment it hit production, the Bright Data Scraping Browser</a> is built to solve exactly that problem. It's a fully hosted, cloud-based browser that combines real browser automation with Bright Data's industry-leading unblocking infrastructure — so you can run Playwright, Puppeteer, or Selenium scripts at scale without managing proxies, headless browser farms, or anti-bot countermeasures yourself.</p> Want to try it?</strong> Start scraping →</a></p> </blockquote> What Is the Scraping Browser?</h2> The Scraping Browser is a remote, GUI-style browser hosted on Bright Data's infrastructure. Instead of spinning up Chromium on your own servers and bolting on proxy rotation and CAPTCHA handling, you connect your existing automation framework to Bright Data's browser over the Chrome DevTools Protocol (CDP) with a single line of code</strong>. Every session automatically routes through Bright Data's residential proxy network and unlocking engine.</p> In short: you write normal browser-automation code, and Bright Data handles the part that's hard to keep working — staying unblocked.</p> Key Features</h2> Fully hosted with unlimited concurrent sessions.</strong> Scale from one browser to thousands without provisioning any infrastructure of your own.</li> Massive residential IP pool.</strong> Access to 400M+ residential IPs across 195 countries, so your traffic looks like a real user from virtually anywhere.</li> Built-in CAPTCHA solving.</strong> CAPTCHAs are detected and solved automatically as part of the unlocking flow.</li> Automatic anti-bot evasion.</strong> Browser fingerprint management, cookie handling, user-agent and referral header configuration, and automatic retries with IP rotation are all handled for you.</li> Full JavaScript rendering.</strong> Dynamic, JS-heavy sites render exactly as they would in a real browser, with data-integrity validation built in.</li> Native framework support.</strong> Works out of the box with Playwright, Puppeteer, and Selenium</strong> via a CDP/WebSocket endpoint.</li> Built-in debugger.</strong> A Chrome DevTools-compatible debugger lets you inspect and troubleshoot live sessions.</li> </ul> Why Use It Instead of a Self-Hosted Headless Browser?</h2> Running headless Chrome yourself means you're also responsible for proxy rotation, CAPTCHA services, fingerprint randomization, and constant maintenance as targets update their defenses. The Scraping Browser bundles all of that into a managed service:</p> Self-hosted headless browser</th> Bright Data Scraping Browser</th></tr></thead> You manage proxy rotation</td> Residential rotation built in</td></tr> You integrate CAPTCHA solvers</td> CAPTCHA solving automatic</td></tr> You handle fingerprinting & cookies</td> Handled automatically</td></tr> You scale and maintain browser servers</td> Unlimited hosted concurrency</td></tr> You patch for new anti-bot updates</td> Bright Data maintains the unlocker</td></tr> </tbody></table> Getting Started: Connecting Your Code</h2> Because the Scraping Browser speaks the Chrome DevTools Protocol, integration is a one-liner — you just point your automation library at Bright Data's WebSocket endpoint instead of a local browser.</p> Puppeteer (Node.js)</h3> const puppeteer = require('puppeteer-core'); // Your Bright Data Scraping Browser endpoint const BROWSER_WS = 'wss://USERNAME:PASSWORD@brd.superproxy.io:9222'; (async () => { const browser = await puppeteer.connect({ browserWSEndpoint: BROWSER_WS }); const page = await browser.newPage(); await page.goto('https://example.com', { waitUntil: 'networkidle2' }); console.log(await page.title()); await browser.close(); })(); </code></pre> Playwright (Node.js)</h3> const { chromium } = require('playwright'); const BROWSER_WS = 'wss://USERNAME:PASSWORD@brd.superproxy.io:9222'; (async () => { const browser = await chromium.connectOverCDP(BROWSER_WS); const page = await browser.newPage(); await page.goto('https://example.com'); console.log(await page.title()); await browser.close(); })(); </code></pre> Selenium connections work the same way through the remote WebDriver endpoint — your existing scripts stay almost entirely unchanged.</p> Best Use Cases</h2> Scraping heavily protected sites</strong> that block datacenter IPs or aggressively serve CAPTCHAs.</li> Large-scale data collection</strong> where you need many parallel browser sessions without managing servers.</li> Dynamic, JavaScript-rendered pages</strong> (infinite scroll, SPAs, lazy-loaded content) that require a real browser.</li> Price monitoring, SERP tracking, and market research</strong> across many geographies thanks to the global residential network.</li> </ul> Pricing</h2> Pay-as-you-go from $5/GB</strong>, with custom plans available for higher volume.</li> Free trial available.</strong></li> Up to 37% savings</strong> on long-term/annual plans.</li> Available through the AWS Marketplace</strong> for consolidated billing, plus 24/7 support.</li> </ul> The Bottom Line</h2> If your scraping projects keep running into blocks, the Bright Data Scraping Browser removes the most painful parts of browser automation — proxies, CAPTCHAs, and fingerprinting — while letting you keep the Playwright, Puppeteer, or Selenium code you already know. For teams that need reliable, scalable access to tough targets, it's one of the most capable options on the market.</p> Get started with the Bright Data Scraping Browser →</a></strong></p> Want the full picture on Bright Data's tools and pricing? Read our complete Bright Data review</a>.</em></p> How to Avoid Getting Blocked While Web Scraping 2026-06-02T00:00:00+00:00 The single biggest reason web scrapers fail isn't bad code — it's getting blocked. Modern websites deploy sophisticated anti-bot systems (Cloudflare, DataDome, PerimeterX, Akamai) that detect and ban automated traffic within seconds. The good news: with the right techniques, you can scrape reliably at scale. This guide walks through the ten most effective methods, from quick wins to production-grade infrastructure.</p> 1. Rotate Your IP Address with Proxies</h2> If hundreds of requests hit a site from a single IP in a short window, that IP gets banned — fast. Rotating proxies distribute your requests across a large pool of IP addresses so no single address looks suspicious.</p> This is the foundation of reliable scraping. Datacenter proxies are cheap and fast but easier to detect; residential and mobile proxies route through real consumer devices and are far harder to block. See our full breakdown in Residential vs. Datacenter vs. Mobile Proxies</a>.</p> Recommended:</strong> Bright Data</a> offers a 400M+ residential IP pool with automatic rotation. Budget-friendly alternatives include IPRoyal</a> and DataImpulse</a>.</p> </blockquote> 2. Set a Realistic User-Agent</h2> The default User-Agent</code> of libraries like Python's requests</code> (e.g. python-requests/2.31.0</code>) is an instant giveaway. Always send a real browser User-Agent, and rotate between several.</p> import requests headers = { "User-Agent": ( "Mozilla/5.0 (Windows NT 10.0; Win64; x64) " "AppleWebKit/537.36 (KHTML, like Gecko) " "Chrome/124.0.0.0 Safari/537.36" ) } response = requests.get("https://example.com", headers=headers) </code></pre> 3. Send a Complete Set of Headers</h2> Real browsers send far more than just a User-Agent. Anti-bot systems check for the presence and consistency of headers like Accept</code>, Accept-Language</code>, Accept-Encoding</code>, and Referer</code>. A request missing these looks robotic.</p> headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8", "Accept-Language": "en-US,en;q=0.9", "Accept-Encoding": "gzip, deflate, br", "Referer": "https://www.google.com/", "Connection": "keep-alive", } </code></pre> 4. Throttle Your Request Rate</h2> Humans don't load 50 pages per second. Aggressive request rates are one of the easiest patterns to detect. Add delays between requests — and randomize them so the timing doesn't look mechanical.</p> import time import random for url in urls: scrape(url) time.sleep(random.uniform(2, 6)) # random 2–6 second delay </code></pre> 5. Respect robots.txt and Rate Limits</h2> Before scraping, check the site's robots.txt</code> and any published rate limits. Honoring them keeps you ethical, reduces your footprint, and lowers your chances of being flagged. Scraping publicly available data is generally permissible, but always avoid logged-in/private areas and personal data.</p> 6. Handle CAPTCHAs</h2> Once you trigger a CAPTCHA, basic scrapers are stuck. You have three options: avoid triggering them in the first place (the techniques in this guide), use a CAPTCHA-solving service, or use a tool that solves them automatically. Managed solutions like the Bright Data Scraping Browser</a> detect and solve CAPTCHAs as part of their unlocking flow, with no extra integration.</p> 7. Use a Headless Browser for JavaScript-Heavy Sites</h2> Many modern sites render content with JavaScript, so a plain HTTP request returns an empty shell. Tools like Playwright</strong>, Puppeteer</strong>, and Selenium</strong> drive a real browser engine that executes JavaScript just like a user's browser would.</p> from playwright.sync_api import sync_playwright with sync_playwright() as p: browser = p.chromium.launch(headless=True) page = browser.new_page() page.goto("https://example.com") print(page.title()) browser.close() </code></pre> 8. Manage Your Browser Fingerprint</h2> Anti-bot systems build a "fingerprint" from dozens of signals — screen resolution, installed fonts, WebGL renderer, timezone, and navigator properties. Headless browsers leak tell-tale values (like navigator.webdriver = true</code>). Use stealth plugins (e.g. playwright-stealth</code>, puppeteer-extra-plugin-stealth</code>) or a managed browser that handles fingerprinting for you.</p> 9. Maintain Cookies and Sessions</h2> Real users carry cookies across requests. A scraper that drops cookies on every request looks anonymous and suspicious. Use a session object to persist cookies naturally.</p> import requests session = requests.Session() session.headers.update(headers) session.get("https://example.com") # picks up cookies session.get("https://example.com/page/2") # reuses them </code></pre> 10. Use a Web Unlocker or Scraping API for Tough Targets</h2> When you're up against the most aggressive anti-bot systems, managing all of the above yourself becomes a full-time job. Web Unlockers and scraping APIs bundle proxy rotation, header management, fingerprinting, CAPTCHA solving, and automatic retries into a single endpoint — you send a URL and get clean HTML back.</p> For tough targets:</strong> The Bright Data Web Unlocker</a> handles blocks automatically. Start scraping →</a></p> For a simpler pay-as-you-go API, ScraperAPI</a> is also worth a look.</p> </blockquote> Quick Reference</h2> Technique</th> Difficulty</th> Impact</th></tr></thead> Rotate IPs with proxies</td> Medium</td> Very High</td></tr> Realistic User-Agent</td> Easy</td> High</td></tr> Complete headers</td> Easy</td> Medium</td></tr> Throttle request rate</td> Easy</td> High</td></tr> Handle CAPTCHAs</td> Hard</td> High</td></tr> Headless browser</td> Medium</td> High</td></tr> Manage fingerprint</td> Hard</td> High</td></tr> Maintain sessions</td> Easy</td> Medium</td></tr> Web Unlocker / API</td> Easy</td> Very High</td></tr> </tbody></table> The Bottom Line</h2> Avoiding blocks is about looking like a real user: rotating IPs, sending believable headers, pacing your requests, and rendering JavaScript when needed. For small projects, the manual techniques here go a long way. For scraping protected sites at scale, a managed solution that bundles unblocking infrastructure will save you enormous time.</p> Get started with Bright Data →</a></strong></p> New to scraping? Start with our Web Scraping with Python guide</a>, or compare providers in our proxy and scraper reviews</a>.</em></p> Residential vs Datacenter vs Mobile Proxies: Which to Use? 2026-06-02T00:00:00+00:00 Proxies are the backbone of any serious web scraping operation — they're how you rotate IP addresses, avoid blocks, and access geo-restricted content. But not all proxies are created equal. Choosing the wrong type can mean wasted money, constant bans, or painfully slow scrapes. This guide explains the four main proxy types, when to use each, and how to pick the right one for your project.</p> Why Proxies Matter for Scraping</h2> When you scrape a website, every request carries your IP address. Send too many requests from one IP and you'll be rate-limited or banned. Proxies route your traffic through other IP addresses, letting you distribute requests across a large pool so your scraper looks like many different visitors instead of one aggressive bot. (For the full anti-blocking playbook, see How to Avoid Getting Blocked While Web Scraping</a>.)</p> The four proxy types below differ mainly in where the IP comes from</strong> — and that origin determines how trustworthy the IP looks to anti-bot systems.</p> Datacenter Proxies</h2> Datacenter proxies come from servers in data centers, not from internet service providers (ISPs). They're created in bulk and aren't tied to a physical home or device.</p> Pros:</strong> Cheapest option, extremely fast, available in huge quantities.</li> Cons:</strong> Easiest to detect and block. Because many share the same subnet ("IP neighborhood"), one flagged IP can taint others.</li> Best for:</strong> High-volume scraping of sites with weak or no anti-bot protection, internal tools, and speed-sensitive tasks where cost matters.</li> </ul> Learn more in our Bright Data Datacenter Proxies review</a>.</p> </blockquote> Residential Proxies</h2> Residential proxies route through real consumer devices using IPs assigned by ISPs to actual homes. To a target website, your traffic looks like a genuine person browsing from their living room.</p> Pros:</strong> Very hard to detect and block; high trust; excellent for geo-targeting at the city/country level.</li> Cons:</strong> More expensive than datacenter proxies; typically slower; usually billed by bandwidth (per GB).</li> Best for:</strong> Scraping protected sites (e-commerce, travel, social media, sneaker/ticket sites), ad verification, and any target that blocks datacenter IPs.</li> </ul> See the Bright Data Residential Proxies review</a>. Budget options worth comparing: IPRoyal</a>, DataImpulse</a>, and HydraProxy</a>.</p> </blockquote> Mobile Proxies</h2> Mobile proxies use IP addresses assigned by mobile carriers to 3G/4G/5G devices. They're the hardest of all to block — because carriers share a small number of IPs among many real users (via CGNAT), banning a mobile IP risks blocking thousands of legitimate customers.</p> Pros:</strong> Highest trust level and lowest block rate; ideal for the most aggressive anti-bot targets.</li> Cons:</strong> The most expensive option; can be slower and have variable reliability.</li> Best for:</strong> Scraping social media platforms, mobile-only content, and the toughest targets where nothing else gets through.</li> </ul> Details in the Bright Data Mobile Proxies review</a>.</p> </blockquote> ISP Proxies (The Hybrid)</h2> ISP proxies are a middle ground: they're hosted in data centers (so they're fast) but registered under real ISPs (so they carry residential-level trust). You get datacenter speed with much of residential legitimacy.</p> Pros:</strong> Fast and stable like datacenter, trusted like residential; often sold as static (sticky) IPs.</li> Cons:</strong> Pricier than datacenter; smaller pools than residential networks.</li> Best for:</strong> Tasks needing both speed and trust — account management, sustained sessions, and sneaker/retail scraping.</li> </ul> Read the Bright Data ISP Proxies review</a>.</p> </blockquote> Side-by-Side Comparison</h2> Proxy Type</th> Detection Risk</th> Speed</th> Cost</th> Best Use Case</th></tr></thead> Datacenter</strong></td> High</td> Fastest</td> $</td> Unprotected, high-volume targets</td></tr> Residential</strong></td> Low</td> Medium</td> $$$</td> Protected sites, geo-targeting</td></tr> Mobile</strong></td> Lowest</td> Variable</td> $$$$</td> Social media, toughest targets</td></tr> ISP</strong></td> Low</td> Fast</td> $$</td> Speed + trust, static sessions</td></tr> </tbody></table> Static vs. Rotating Proxies</h2> Independent of type, proxies are delivered in two modes:</p> Rotating proxies</strong> assign a new IP from the pool on each request (or at set intervals) — ideal for large-scale scraping where you want to spread requests across many IPs.</li> Static (sticky) proxies</strong> keep the same IP for the duration of a session — ideal for tasks that need a consistent identity, like staying logged in.</li> </ul> How to Choose</h2> Start cheap.</strong> If your target has little protection, datacenter proxies will do — don't overpay.</li> Hitting blocks?</strong> Move up to residential proxies. This solves the vast majority of blocking problems.</li> Still blocked, or scraping mobile-first platforms?</strong> Step up to mobile proxies.</li> Need speed and</em> trust for logged-in sessions?</strong> Choose ISP proxies.</li> Don't want to manage proxies at all?</strong> Use a managed unblocking solution like the Bright Data Scraping Browser</a> or Web Unlocker</a>, which handle proxy selection and rotation automatically.</li> </ol> The Bottom Line</h2> There's no single "best" proxy type — only the best fit for your target and budget. Datacenter proxies win on cost and speed, residential proxies win on stealth, mobile proxies win against the toughest defenses, and ISP proxies balance speed with trust. For most scrapers, rotating residential proxies are the sweet spot.</p> Get started:</strong> Bright Data</a> offers all four proxy types. Explore the network →</a></p> </blockquote> Want to compare providers head-to-head? Browse our proxy and scraper reviews</a> or read Bright Data vs. Oxylabs</a>.</em></p> Best Proxy & Scraping Services, Ranked (2026) 2026-06-02T00:00:00+00:00 We've tested and reviewed the leading proxy and web scraping providers on real-world targets. Below is our independent ranking for 2026</strong> — based on success rates, network size, features, pricing, and value. Bright Data</a> takes the top spot as the industry gold standard, but every provider here is one we actively recommend.</p>

Datasets vs Web Scraping: When to Buy Data Instead

How to Solve CAPTCHAs When Web Scraping: A Practical Guide

eBay Product Tracking: Scraper Code Samples

Scrape Public Instagram Data: Profiles, Posts, Hashtags

How to Scrape LinkedIn: Public Profiles, Companies, Jobs

How to Handle Pagination in Web Scraping: The Complete Guide

Web Scraping with Playwright in Python: Full Guide

eBay Product Search Scraping: Code Samples

Bright Data vs ZenRows: Proxy Platform or Anti-Bot API?

ZenRows vs ScraperAPI: Which Web Scraping API Wins?

ZenRows Review: Anti-Bot Web Scraping API That Just Works

Looking at Other Options?</h2>
If ZenRows doesn't fit your workload, we've rounded up the best ZenRows alternatives</a> — including how they differ on pricing model, proxy quality, and API features.</p>

Bright Data vs IPRoyal: Premium Power or Budget Flexibility?

Bright Data vs ScraperAPI: Proxy Network or Scraping API?

HydraProxy vs IPRoyal: Best Flexible, Low-Budget Proxies?

IPRoyal vs DataImpulse: Which Budget Proxy Provider Wins?

Bright Data Scraping Browser: CAPTCHA-Solving Cloud Browser

Getting Started: Connecting Your Code</h2>
Because the Scraping Browser speaks the Chrome DevTools Protocol, integration is a one-liner — you just point your automation library at Bright Data's WebSocket endpoint instead of a local browser.</p>

How to Avoid Getting Blocked While Web Scraping

Residential vs Datacenter vs Mobile Proxies: Which to Use?

Best Proxy & Scraping Services, Ranked (2026)

Web-Scrapers.com

ZenRows Alternatives: 5 Best Web Scraping APIs (2026)

Is there a ZenRows alternative with a free tier?</h3>
Yes — ScraperAPI</a> offers free API credits every month, not just a one-time trial, so you can evaluate it and keep running small jobs at no cost. ZenRows itself provides free trial credits to start, but not an ongoing free tier.</p>

How to Scrape Alibaba: B2B Product and Supplier Data