Introduction
BeautifulSoup is great for static HTML. But in 2026, most sites are JavaScript-rendered SPAs — BeautifulSoup sees an empty <div id="root"></div> and gives up. Playwright fixes that: it drives a real browser, waits for content to render, and lets you scrape like a user.
This post upgrades your scraping toolkit with Playwright for Python.
Why Playwright over BeautifulSoup/Selenium?
- Auto-wait — no
time.sleep(); Playwright waits for elements to appear. - Multiple browsers — Chromium, Firefox, WebKit from one API.
- Async + sync APIs — scrape fast with
asyncio. - Screenshots & PDFs — capture pages for debugging or archiving.
- Network interception — block images/CSS for 10x speed.
Setup
pip install playwright
playwright install chromium # downloads browser (~150MB)
Step 1: Basic scrape (sync)
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto("https://news.ycombinator.com")
page.wait_for_selector(".titleline")
titles = page.locator(".titleline > a").all_text_contents()
for t in titles[:10]:
print(t)
browser.close()
Notice: no requests.get, no BeautifulSoup, no manual parsing. Playwright renders the page fully (including JS) and gives you a jQuery-like locator API.
Step 2: Async for speed
import asyncio
from playwright.async_api import async_playwright
async def scrape(url):
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
page = await browser.new_page()
await page.goto(url)
await page.wait_for_selector("h1")
title = await page.locator("h1").first.text_content()
await browser.close()
return title
async def main():
urls = ["https://example.com", "https://example.org"]
results = await asyncio.gather(*[scrape(u) for u in urls])
print(results)
asyncio.run(main())
asyncio.gather runs scrapes in parallel — 10x faster than sequential.
Step 3: Extract structured data
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
page = p.chromium.launch(headless=True).new_page()
page.goto("https://example.com/products")
page.wait_for_selector(".product-card")
products = []
for card in page.locator(".product-card").all():
products.append({
"name": card.locator(".name").text_content(),
"price": card.locator(".price").text_content(),
"link": card.locator("a").get_attribute("href"),
})
print(products)
Locators are lazy and auto-retry — if .name isn’t there yet, Playwright waits (up to 30s by default).
Step 4: Screenshots & debugging
page.screenshot(path="page.png", full_page=True)
page.pdf(path="page.pdf") # Chromium only
When a scrape breaks, a screenshot tells you exactly what the browser saw. Add headless=False during development to watch it live.
Step 5: Block resources for speed
page.route("**/*.{png,jpg,jpeg,gif,css,woff}", lambda route: route.abort())
page.goto(url) # 3-5x faster
Block images, fonts, and CSS when you only need text. Massive speedup on heavy sites.
Step 6: Handle pagination
all_items = []
page.goto("https://example.com/list")
while True:
page.wait_for_selector(".item")
all_items.extend(page.locator(".item").all_text_contents())
next_btn = page.locator("a.next")
if not next_btn.is_visible():
break
next_btn.click()
page.wait_for_load_state("networkidle")
print(len(all_items))
Playwright vs BeautifulSoup (when to use which)
| BeautifulSoup | Playwright | |
|---|---|---|
| Static HTML | ✅ Fast, light | Overkill |
| JS-rendered (React/Vue) | ❌ Empty page | ✅ Renders fully |
| Speed | Very fast | Slower (browser overhead) |
| Memory | Tiny | ~150MB browser |
| Login/cookies | Manual | Built-in (persistent context) |
| Anti-bot | Easy to block | Harder to detect |
Rule of thumb: try BeautifulSoup first. If you get empty results or None, switch to Playwright.
Common mistakes
- Forgetting
playwright install— the Python package alone isn’t enough; you need the browser binaries. time.sleep()instead ofwait_for_selector— flaky. Always wait for a specific element.- Not closing the browser — leaks memory. Use
with/context managers ortry/finally. - Scraping too fast — add
page.wait_for_timeout(1000)or respectrobots.txt. Be polite. - Ignoring
robots.txt— check it before scraping; some sites explicitly forbid automated access.
Key Concepts
- Playwright = real browser automation, not HTML parsing
- Locators = lazy, auto-retrying element queries
- Auto-wait = no manual sleeps
- Async API = parallel scraping
- Resource blocking = 3-5x speedup
What to Try Next
- Scrape a Streamlit app gallery page and build a searchable index.
- Combine with your PDF merger — scrape articles, save as PDF.
- Add Resend email alerts when a price drops.
- Build a Streamlit dashboard showing scraped data, refreshed daily.
FAQ
Playwright vs Selenium in 2026?
Playwright is the modern default: faster, more reliable auto-wait, better async support, and multi-browser from one API. Selenium is still fine for legacy/enterprise, but new projects should use Playwright.
Is scraping legal?
Scraping public data is generally legal, but always check robots.txt, Terms of Service, and never scrape personal data without consent. When in doubt, use the site’s official API.
Can Playwright bypass Cloudflare/bot protection?
Sometimes, but it’s an arms race. For protected sites, consider: residential proxies, stealth plugins, or — better — the official API. Don’t build a business on scraping a site that doesn’t want to be scraped.