Tech With Galvan
Back to Articles
DevelopmentJan 10, 2027 • 6 min read

Web Scraping with Scrapy: The Production Framework

Build production-grade web scrapers with Scrapy — spiders, items, pipelines, and middleware. When Scrapy beats BeautifulSoup and Playwright.

Galvan
Galvan

Founder & Creator

Introduction

BeautifulSoup and Playwright are great for one-off scrapes. But for production scraping — thousands of pages, retries, deduplication, data pipelines — you need Scrapy. It’s the industry-standard scraping framework: async, built-in, and extensible.

Scrapy vs the alternatives

BeautifulSoup Playwright Scrapy
Learning curve Low Medium Medium-High
Speed Fast (sync) Slow (browser) Very fast (async)
JS rendering No Yes No (add scrapy-playwright)
Retries/dedup Manual Manual Built-in
Pipelines Manual Manual Built-in
Scale Small Small Large
Best for Quick scripts JS sites Production crawlers

Rule: BeautifulSoup for quick scripts, Playwright for JS sites, Scrapy for production crawlers.

Setup

pip install scrapy
scrapy startproject myscraper
cd myscraper

Creates:

myscraper/
├── scrapy.cfg
└── myscraper/
    ├── __init__.py
    ├── items.py
    ├── middlewares.py
    ├── pipelines.py
    ├── settings.py
    └── spiders/
        └── __init__.py

Step 1: Create a spider

scrapy genspider quotes quotes.toscrape.com

Creates spiders/quotes.py:

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    allowed_domains = ["quotes.toscrape.com"]
    start_urls = ["https://quotes.toscrape.com/"]

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small.author::text").get(),
                "tags": quote.css("div.tags a.tag::text").getall(),
            }
        # Follow pagination
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

Run it:

scrapy crawl quotes -o quotes.json

Scrapy handles concurrency, retries, deduplication, and rate limiting automatically.

Step 2: Items (typed data)

# items.py
import scrapy

class QuoteItem(scrapy.Item):
    text = scrapy.Field()
    author = scrapy.Field()
    tags = scrapy.Field()

Use in spider:

from myscraper.items import QuoteItem

def parse(self, response):
    for quote in response.css("div.quote"):
        item = QuoteItem()
        item["text"] = quote.css("span.text::text").get()
        item["author"] = quote.css("small.author::text").get()
        item["tags"] = quote.css("div.tags a.tag::text").getall()
        yield item

Items are dicts with validation (optional, via itemloaders).

Step 3: Item pipelines (process data)

# pipelines.py
class CleanPipeline:
    def process_item(self, item, spider):
        item["text"] = item["text"].strip().strip('"')
        item["tags"] = [t.lower() for t in item["tags"]]
        return item

class DuplicatesPipeline:
    def __init__(self):
        self.seen = set()

    def process_item(self, item, spider):
        key = item["text"]
        if key in self.seen:
            raise scrapy.exceptions.DropItem(f"Duplicate: {key[:50]}")
        self.seen.add(key)
        return item

Enable in settings.py:

ITEM_PIPELINES = {
    "myscraper.pipelines.CleanPipeline": 100,
    "myscraper.pipelines.DuplicatesPipeline": 200,
}

Lower numbers run first. Pipelines can clean, validate, dedup, store to DB, etc.

Step 4: Settings (tuning)

# settings.py
BOT_NAME = "myscraper"
ROBOTSTXT_OBEY = True
CONCURRENT_REQUESTS = 16
CONCURRENT_REQUESTS_PER_DOMAIN = 8
DOWNLOAD_DELAY = 0.5   # seconds between requests
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY = 10
RETRY_TIMES = 3
RETRY_HTTP_CODES = [500, 502, 503, 504, 408, 429]

AutoThrottle automatically adjusts request rate based on server response times. Be polite.

Step 5: Middleware (custom headers, proxies)

# middlewares.py
class CustomHeadersMiddleware:
    def process_request(self, request, spider):
        request.headers["User-Agent"] = "MyBot/1.0"
        request.headers["Accept-Language"] = "en-US,en;q=0.9"

Enable:

DOWNLOADER_MIDDLEWARES = {
    "myscraper.middlewares.CustomHeadersMiddleware": 543,
}

Step 6: Store to database

# pipelines.py
import sqlite3

class SQLitePipeline:
    def open_spider(self, spider):
        self.conn = sqlite3.connect("quotes.db")
        self.conn.execute("CREATE TABLE IF NOT EXISTS quotes (text TEXT, author TEXT, tags TEXT)")

    def close_spider(self, spider):
        self.conn.close()

    def process_item(self, item, spider):
        self.conn.execute(
            "INSERT INTO quotes (text, author, tags) VALUES (?, ?, ?)",
            (item["text"], item["author"], ",".join(item["tags"]))
        )
        self.conn.commit()
        return item

Step 7: Scrape JS-rendered sites (scrapy-playwright)

pip install scrapy-playwright
# settings.py
DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
PLAYWRIGHT_BROWSER_TYPE = "chromium"

In spider:

def start_requests(self):
    yield scrapy.Request(
        url="https://js-site.com",
        meta={"playwright": True},
    )

Now Scrapy renders JS pages with Playwright. Best of both worlds.

Step 8: Export formats

scrapy crawl quotes -o quotes.json       # JSON
scrapy crawl quotes -o quotes.csv        # CSV
scrapy crawl quotes -o quotes.jsonl      # JSON Lines (streaming)
scrapy crawl quotes -o quotes.xml        # XML

JSONL is best for large datasets (streaming, low memory).

Common mistakes

  • Ignoring robots.txt — ROBOTSTXT_OBEY = True by default. Respect it.
  • Too many concurrent requests — start with 8-16, increase carefully.
  • No delay — DOWNLOAD_DELAY prevents IP bans. Always set it.
  • Not handling pagination — follow next links recursively.
  • Scraping without permission — check ToS. Some sites explicitly forbid scraping.
  • Not using allowed_domains — without it, Scrapy follows links to other sites.

Key Concepts

  • Spider = class that defines how to scrape a site
  • Item = structured data container
  • Pipeline = process/validate/store items
  • Middleware = modify requests/responses
  • AutoThrottle = automatic rate adjustment
  • response.follow() = follow links and parse

What to Try Next

FAQ

Scrapy vs BeautifulSoup — when to use which?

BeautifulSoup = quick scripts, static HTML, small scale. Scrapy = production crawlers, large scale, built-in retries/pipelines. Start with BeautifulSoup; move to Scrapy when you need reliability.

Can Scrapy handle JavaScript?

Not natively. Use scrapy-playwright (recommended) or scrapy-splash. For simple JS, scrapy-playwright is the modern choice.

Scraping public data is generally legal, but always check robots.txt and Terms of Service. Never scrape personal data without consent. When in doubt, use the site’s API. Some sites (LinkedIn, Facebook) aggressively block and may take legal action. Be respectful.