Introduction
BeautifulSoup and Playwright are great for one-off scrapes. But for production scraping — thousands of pages, retries, deduplication, data pipelines — you need Scrapy. It’s the industry-standard scraping framework: async, built-in, and extensible.
Scrapy vs the alternatives
| BeautifulSoup | Playwright | Scrapy | |
|---|---|---|---|
| Learning curve | Low | Medium | Medium-High |
| Speed | Fast (sync) | Slow (browser) | Very fast (async) |
| JS rendering | No | Yes | No (add scrapy-playwright) |
| Retries/dedup | Manual | Manual | Built-in |
| Pipelines | Manual | Manual | Built-in |
| Scale | Small | Small | Large |
| Best for | Quick scripts | JS sites | Production crawlers |
Rule: BeautifulSoup for quick scripts, Playwright for JS sites, Scrapy for production crawlers.
Setup
pip install scrapy
scrapy startproject myscraper
cd myscraper
Creates:
myscraper/
├── scrapy.cfg
└── myscraper/
├── __init__.py
├── items.py
├── middlewares.py
├── pipelines.py
├── settings.py
└── spiders/
└── __init__.py
Step 1: Create a spider
scrapy genspider quotes quotes.toscrape.com
Creates spiders/quotes.py:
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
allowed_domains = ["quotes.toscrape.com"]
start_urls = ["https://quotes.toscrape.com/"]
def parse(self, response):
for quote in response.css("div.quote"):
yield {
"text": quote.css("span.text::text").get(),
"author": quote.css("small.author::text").get(),
"tags": quote.css("div.tags a.tag::text").getall(),
}
# Follow pagination
next_page = response.css("li.next a::attr(href)").get()
if next_page:
yield response.follow(next_page, self.parse)
Run it:
scrapy crawl quotes -o quotes.json
Scrapy handles concurrency, retries, deduplication, and rate limiting automatically.
Step 2: Items (typed data)
# items.py
import scrapy
class QuoteItem(scrapy.Item):
text = scrapy.Field()
author = scrapy.Field()
tags = scrapy.Field()
Use in spider:
from myscraper.items import QuoteItem
def parse(self, response):
for quote in response.css("div.quote"):
item = QuoteItem()
item["text"] = quote.css("span.text::text").get()
item["author"] = quote.css("small.author::text").get()
item["tags"] = quote.css("div.tags a.tag::text").getall()
yield item
Items are dicts with validation (optional, via itemloaders).
Step 3: Item pipelines (process data)
# pipelines.py
class CleanPipeline:
def process_item(self, item, spider):
item["text"] = item["text"].strip().strip('"')
item["tags"] = [t.lower() for t in item["tags"]]
return item
class DuplicatesPipeline:
def __init__(self):
self.seen = set()
def process_item(self, item, spider):
key = item["text"]
if key in self.seen:
raise scrapy.exceptions.DropItem(f"Duplicate: {key[:50]}")
self.seen.add(key)
return item
Enable in settings.py:
ITEM_PIPELINES = {
"myscraper.pipelines.CleanPipeline": 100,
"myscraper.pipelines.DuplicatesPipeline": 200,
}
Lower numbers run first. Pipelines can clean, validate, dedup, store to DB, etc.
Step 4: Settings (tuning)
# settings.py
BOT_NAME = "myscraper"
ROBOTSTXT_OBEY = True
CONCURRENT_REQUESTS = 16
CONCURRENT_REQUESTS_PER_DOMAIN = 8
DOWNLOAD_DELAY = 0.5 # seconds between requests
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY = 10
RETRY_TIMES = 3
RETRY_HTTP_CODES = [500, 502, 503, 504, 408, 429]
AutoThrottle automatically adjusts request rate based on server response times. Be polite.
Step 5: Middleware (custom headers, proxies)
# middlewares.py
class CustomHeadersMiddleware:
def process_request(self, request, spider):
request.headers["User-Agent"] = "MyBot/1.0"
request.headers["Accept-Language"] = "en-US,en;q=0.9"
Enable:
DOWNLOADER_MIDDLEWARES = {
"myscraper.middlewares.CustomHeadersMiddleware": 543,
}
Step 6: Store to database
# pipelines.py
import sqlite3
class SQLitePipeline:
def open_spider(self, spider):
self.conn = sqlite3.connect("quotes.db")
self.conn.execute("CREATE TABLE IF NOT EXISTS quotes (text TEXT, author TEXT, tags TEXT)")
def close_spider(self, spider):
self.conn.close()
def process_item(self, item, spider):
self.conn.execute(
"INSERT INTO quotes (text, author, tags) VALUES (?, ?, ?)",
(item["text"], item["author"], ",".join(item["tags"]))
)
self.conn.commit()
return item
Step 7: Scrape JS-rendered sites (scrapy-playwright)
pip install scrapy-playwright
# settings.py
DOWNLOAD_HANDLERS = {
"http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
"https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
PLAYWRIGHT_BROWSER_TYPE = "chromium"
In spider:
def start_requests(self):
yield scrapy.Request(
url="https://js-site.com",
meta={"playwright": True},
)
Now Scrapy renders JS pages with Playwright. Best of both worlds.
Step 8: Export formats
scrapy crawl quotes -o quotes.json # JSON
scrapy crawl quotes -o quotes.csv # CSV
scrapy crawl quotes -o quotes.jsonl # JSON Lines (streaming)
scrapy crawl quotes -o quotes.xml # XML
JSONL is best for large datasets (streaming, low memory).
Common mistakes
- Ignoring
robots.txt—ROBOTSTXT_OBEY = Trueby default. Respect it. - Too many concurrent requests — start with 8-16, increase carefully.
- No delay —
DOWNLOAD_DELAYprevents IP bans. Always set it. - Not handling pagination — follow
nextlinks recursively. - Scraping without permission — check ToS. Some sites explicitly forbid scraping.
- Not using
allowed_domains— without it, Scrapy follows links to other sites.
Key Concepts
- Spider = class that defines how to scrape a site
- Item = structured data container
- Pipeline = process/validate/store items
- Middleware = modify requests/responses
- AutoThrottle = automatic rate adjustment
response.follow()= follow links and parse
What to Try Next
- Scrape a Streamlit app gallery and build an index.
- Store scraped data in Supabase.
- Combine with aiohttp for hybrid approaches.
- Add Resend email alerts when new items appear.
FAQ
Scrapy vs BeautifulSoup — when to use which?
BeautifulSoup = quick scripts, static HTML, small scale. Scrapy = production crawlers, large scale, built-in retries/pipelines. Start with BeautifulSoup; move to Scrapy when you need reliability.
Can Scrapy handle JavaScript?
Not natively. Use scrapy-playwright (recommended) or scrapy-splash. For simple JS, scrapy-playwright is the modern choice.
Is scraping legal?
Scraping public data is generally legal, but always check robots.txt and Terms of Service. Never scrape personal data without consent. When in doubt, use the site’s API. Some sites (LinkedIn, Facebook) aggressively block and may take legal action. Be respectful.