Tech With Galvan
Back to Articles
DevelopmentApr 19, 2026 • 5 min read

Web Scraping with Playwright (Modern Python, Beats BeautifulSoup)

Scrape modern JavaScript-rendered websites with Playwright for Python — async, headless, with screenshots and auto-wait. A modern upgrade to BeautifulSoup.

Galvan
Galvan

Founder & Creator

Introduction

BeautifulSoup is great for static HTML. But in 2026, most sites are JavaScript-rendered SPAs — BeautifulSoup sees an empty <div id="root"></div> and gives up. Playwright fixes that: it drives a real browser, waits for content to render, and lets you scrape like a user.

This post upgrades your scraping toolkit with Playwright for Python.

Why Playwright over BeautifulSoup/Selenium?

  • Auto-wait — no time.sleep(); Playwright waits for elements to appear.
  • Multiple browsers — Chromium, Firefox, WebKit from one API.
  • Async + sync APIs — scrape fast with asyncio.
  • Screenshots & PDFs — capture pages for debugging or archiving.
  • Network interception — block images/CSS for 10x speed.

Setup

pip install playwright
playwright install chromium   # downloads browser (~150MB)

Step 1: Basic scrape (sync)

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://news.ycombinator.com")
    page.wait_for_selector(".titleline")
    titles = page.locator(".titleline > a").all_text_contents()
    for t in titles[:10]:
        print(t)
    browser.close()

Notice: no requests.get, no BeautifulSoup, no manual parsing. Playwright renders the page fully (including JS) and gives you a jQuery-like locator API.

Step 2: Async for speed

import asyncio
from playwright.async_api import async_playwright

async def scrape(url):
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url)
        await page.wait_for_selector("h1")
        title = await page.locator("h1").first.text_content()
        await browser.close()
        return title

async def main():
    urls = ["https://example.com", "https://example.org"]
    results = await asyncio.gather(*[scrape(u) for u in urls])
    print(results)

asyncio.run(main())

asyncio.gather runs scrapes in parallel — 10x faster than sequential.

Step 3: Extract structured data

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    page = p.chromium.launch(headless=True).new_page()
    page.goto("https://example.com/products")
    page.wait_for_selector(".product-card")

    products = []
    for card in page.locator(".product-card").all():
        products.append({
            "name": card.locator(".name").text_content(),
            "price": card.locator(".price").text_content(),
            "link": card.locator("a").get_attribute("href"),
        })
    print(products)

Locators are lazy and auto-retry — if .name isn’t there yet, Playwright waits (up to 30s by default).

Step 4: Screenshots & debugging

page.screenshot(path="page.png", full_page=True)
page.pdf(path="page.pdf")   # Chromium only

When a scrape breaks, a screenshot tells you exactly what the browser saw. Add headless=False during development to watch it live.

Step 5: Block resources for speed

page.route("**/*.{png,jpg,jpeg,gif,css,woff}", lambda route: route.abort())
page.goto(url)   # 3-5x faster

Block images, fonts, and CSS when you only need text. Massive speedup on heavy sites.

Step 6: Handle pagination

all_items = []
page.goto("https://example.com/list")
while True:
    page.wait_for_selector(".item")
    all_items.extend(page.locator(".item").all_text_contents())
    next_btn = page.locator("a.next")
    if not next_btn.is_visible():
        break
    next_btn.click()
    page.wait_for_load_state("networkidle")
print(len(all_items))

Playwright vs BeautifulSoup (when to use which)

BeautifulSoup Playwright
Static HTML ✅ Fast, light Overkill
JS-rendered (React/Vue) ❌ Empty page ✅ Renders fully
Speed Very fast Slower (browser overhead)
Memory Tiny ~150MB browser
Login/cookies Manual Built-in (persistent context)
Anti-bot Easy to block Harder to detect

Rule of thumb: try BeautifulSoup first. If you get empty results or None, switch to Playwright.

Common mistakes

  • Forgetting playwright install — the Python package alone isn’t enough; you need the browser binaries.
  • time.sleep() instead of wait_for_selector — flaky. Always wait for a specific element.
  • Not closing the browser — leaks memory. Use with/context managers or try/finally.
  • Scraping too fast — add page.wait_for_timeout(1000) or respect robots.txt. Be polite.
  • Ignoring robots.txt — check it before scraping; some sites explicitly forbid automated access.

Key Concepts

  • Playwright = real browser automation, not HTML parsing
  • Locators = lazy, auto-retrying element queries
  • Auto-wait = no manual sleeps
  • Async API = parallel scraping
  • Resource blocking = 3-5x speedup

What to Try Next

FAQ

Playwright vs Selenium in 2026?

Playwright is the modern default: faster, more reliable auto-wait, better async support, and multi-browser from one API. Selenium is still fine for legacy/enterprise, but new projects should use Playwright.

Scraping public data is generally legal, but always check robots.txt, Terms of Service, and never scrape personal data without consent. When in doubt, use the site’s official API.

Can Playwright bypass Cloudflare/bot protection?

Sometimes, but it’s an arms race. For protected sites, consider: residential proxies, stealth plugins, or — better — the official API. Don’t build a business on scraping a site that doesn’t want to be scraped.