Web Scraping with Playwright in Python: Full Guide

Plain HTTP requests fail on the modern web. A growing share of sites — e-commerce storefronts, travel search engines, social platforms — render their content entirely in JavaScript. Send a requests.get() to one of those URLs and you'll get an empty shell. That's where Playwright comes in.

Playwright is a browser automation library from Microsoft that drives a real Chromium, Firefox, or WebKit engine from Python code. Because it runs a real browser, JavaScript executes, SPAs render, and lazy-loaded content appears — exactly as it would for a human visitor. This guide covers everything you need to scrape dynamic websites with Playwright in Python, from initial setup to production-ready techniques.

Installation and Setup

Install the library and download the browser binaries:

pip install playwright
playwright install chromium

Playwright manages the browser installation for you — no separate ChromeDriver or binary to keep in sync. The examples here were run against Playwright 1.64 for Python in October 2026.

Basic Scraping: Your First Playwright Script

A minimal scraper looks like this:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com")

    title = page.title()
    html = page.content()   # full rendered HTML after JS executes

    print(title)
    browser.close()

headless=True runs the browser invisibly. Switch to headless=False during development to watch what's happening in real time.

Waiting for Dynamic Content

The most common mistake is reading the page before JavaScript has finished rendering it. Playwright offers several strategies:

# Wait for a specific element to appear in the DOM
page.wait_for_selector(".product-list")

# Or wait on a locator, which also auto-waits before every action
page.locator(".product-list").wait_for()

# Wait for a URL pattern (useful after form submissions or redirects)
page.wait_for_url("**/results**")

You'll see wait_until="networkidle" in a lot of older tutorials. Playwright's own API docs now mark it as discouraged: pages with analytics, ads or polling may never go quiet, so the wait can run until it times out. Wait for the element you actually need instead.

For infinite-scroll pages, trigger scrolls programmatically and pause for content to load:

for _ in range(5):
    page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
    page.wait_for_timeout(1500)

Extracting Data from Rendered Pages

Once the page is rendered, use Playwright's locators or hand the HTML to BeautifulSoup for parsing:

from bs4 import BeautifulSoup

# Direct extraction with Playwright locators
titles = page.locator(".product-title").all_inner_texts()
price  = page.locator("span.price").first.inner_text()
href   = page.locator("a.product-link").first.get_attribute("href")

# Alternatively, pass rendered HTML to BeautifulSoup
soup  = BeautifulSoup(page.content(), "html.parser")
cards = soup.select(".product-card")
for card in cards:
    print(card.select_one("h2").get_text(strip=True))

Handling Login-Gated Content

Playwright handles authentication naturally by filling forms just as a user would:

page.goto("https://example.com/login")
page.locator("#email").fill("[email protected]")
page.locator("#password").fill("s3cr3t")
page.locator("button[type='submit']").click()
page.wait_for_url("**/dashboard")

# Save session state so you don't log in on every run
page.context.storage_state(path="session.json")

On subsequent runs, restore the saved session:

context = browser.new_context(storage_state="session.json")
page    = context.new_page()

Scraping Across Multiple Pages

Most real scrapers need to walk through pagination. Here's a clean pattern:

results  = []
page_num = 1

while True:
    page.goto(f"https://example.com/products?page={page_num}")
    page.wait_for_selector(".product-card")

    items = page.locator(".product-card").all_inner_texts()
    if not items:
        break

    results.extend(items)

    # Stop when there is no "Next" button
    next_btn = page.locator("a.next-page")
    if not next_btn.is_visible():
        break

    page_num += 1
    page.wait_for_timeout(2000)   # polite delay between pages

print(f"Scraped {len(results)} items across {page_num} pages")

For a broader look at avoiding rate limits and bans during multi-page runs, see How to Avoid Getting Blocked While Web Scraping.

Reducing Your Fingerprint: Stealth Settings

Headless Playwright leaks navigator.webdriver = true and a handful of other signals that anti-bot systems watch for. Its user agent also says HeadlessChrome. A few settings reduce that exposure:

# Read the real user agent and drop the "Headless" giveaway, so the
# version always matches the browser engine you're running
probe = browser.new_page()
user_agent = probe.evaluate("navigator.userAgent").replace("HeadlessChrome", "Chrome")
probe.close()

context = browser.new_context(
    viewport={"width": 1920, "height": 1080},
    locale="en-US",
    timezone_id="America/New_York",
    user_agent=user_agent,
)

page = context.new_page()
# Mask the webdriver flag before any page load
page.add_init_script("delete Object.getPrototypeOf(navigator).webdriver")

Avoid hard-coding a user agent string copied from a tutorial. An old Chrome version in the header, next to a current browser engine, is itself a mismatch that detection can spot.

For a more comprehensive set of stealth patches, install playwright-stealth:

pip install playwright-stealth

Version 2 changed the API: the old stealth_sync(page) function is gone, replaced by a Stealth class that patches every page you open:

from playwright.sync_api import sync_playwright
from playwright_stealth import Stealth

with Stealth().use_sync(sync_playwright()) as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com")
    print(page.evaluate("navigator.webdriver"))   # False

Keep expectations modest. The package's own README says not to expect it to bypass anything but the simplest bot detection. It hides the obvious tells; it doesn't defeat commercial anti-bot systems.

Rotating Proxies with Playwright

Distributing requests across many IP addresses is essential once you move beyond toy projects. Playwright supports proxies natively at the browser or context level:

context = browser.new_context(
    proxy={
        "server":   "http://proxy.example.com:8080",
        "username": "user",
        "password": "pass",
    }
)
page = context.new_page()

For serious scraping, residential proxies are far less likely to be blocked than datacenter IPs — they route through real consumer devices and look indistinguishable from normal traffic. See our proxy types guide for a full breakdown, residential vs datacenter proxies for when the cheaper option is enough, and rotating proxies for rotation and sticky sessions.

Recommended proxies for Playwright: Bright Data offers a 400M+ residential IP pool with automatic rotation. Budget-friendly alternatives include IPRoyal, DataImpulse, and HydraProxy. Browse all head-to-head comparisons in our proxy and scraper reviews.

Async Playwright for Parallel Scraping

The sync API is convenient for scripts, but for scraping many URLs at once, the async API lets you run browser contexts concurrently. Launch one browser, give each URL its own context, and cap how many run at once:

import asyncio
from playwright.async_api import async_playwright

MAX_CONCURRENT = 5

async def scrape_url(browser, sem: asyncio.Semaphore, url: str) -> str:
    async with sem:                                  # at most MAX_CONCURRENT at once
        context = await browser.new_context()        # isolated cookies per task
        page = await context.new_page()
        try:
            await page.goto(url, wait_until="domcontentloaded")
            return await page.content()
        finally:
            await context.close()

async def main():
    urls = ["https://example.com/page/1", "https://example.com/page/2"]
    sem = asyncio.Semaphore(MAX_CONCURRENT)
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        results = await asyncio.gather(*[scrape_url(browser, sem, u) for u in urls])
        await browser.close()
    for html in results:
        print(len(html), "bytes")

asyncio.run(main())

A new context costs a fraction of a new browser process, and the semaphore keeps you from opening hundreds of tabs, and hitting the target with hundreds of parallel requests, when the URL list is long.

When Self-Hosted Playwright Isn't Enough

Even with stealth settings and good proxies, the most aggressively protected targets (Cloudflare Turnstile, Akamai Bot Manager, PerimeterX) can still block a self-managed browser. At that point, the economics shift in favor of a managed solution.

The Bright Data Scraping Browser (now called Browser API) connects your existing Playwright code to a hosted browser that adds built-in CAPTCHA solving and residential IP rotation on every request — your scripts stay almost identical, but Bright Data handles staying unblocked. For simpler HTTP-based needs, ZenRows and ScraperAPI offer one-endpoint Web Unlocker APIs that handle rendering and unblocking without browser automation.

Compare providers in our scraping service comparisons.

Quick Reference

The Bottom Line

Playwright is the most capable self-hosted option for scraping JavaScript-heavy sites. Its Python API is clean, its waiting primitives are reliable, and it handles everything from simple HTML extraction to authenticated multi-page crawls. Pair it with stealth patches and rotating residential proxies, and you have a scraper that handles the vast majority of real-world targets without a managed service.

Get rotating residential proxies for your Playwright scraper →

New to scraping? Start with our Web Scraping with Python guide, or read How to Avoid Getting Blocked for the full anti-bot playbook.

FAQ

Should I use wait_until="networkidle" in Playwright?

Not as your main wait. Playwright's own docs mark networkidle as discouraged, because pages with analytics, ads or polling may never go quiet. Wait for the element you need with a locator or wait_for_selector instead.

Does playwright-stealth still use stealth_sync?

No. Version 2 of playwright-stealth replaced stealth_sync with a Stealth class. Wrap sync_playwright() in Stealth().use_sync(...) and every page gets the patches. The package's own README says not to expect it to bypass anything but the simplest bot detection.

Which user agent should I set in Playwright?

One that matches the browser you actually run. Headless Chromium announces itself as HeadlessChrome, so read the real user agent from a page and replace that token, rather than hard-coding an old Chrome version that no longer matches the engine's other signals.

How do I scrape many URLs in parallel with Playwright?

Launch one browser, give each URL its own context, and cap concurrency with an asyncio.Semaphore. A new context per task is far cheaper than a new browser process, and the semaphore stops you from opening hundreds of tabs at once.