Web Scraper using Python and BeautifulSoup

Introduction

Web scraping is the skill that turns the internet into your database. This tutorial builds a polite, working scraper with BeautifulSoup: fetch a page, parse the HTML, extract structured data (headlines, links, prices), follow pagination, and export everything to CSV. The same request-and-parse loop powers the weather app — except there the API returned JSON, and here you parse HTML yourself.

The polite part matters: respecting robots.txt, rate limiting, and identifying your scraper are what separate a tool from a nuisance.

Features

CSS selector extraction — target any element precisely.
Pagination support — follow ?page=N links automatically.
Polite crawling — 1-second delays, custom User-Agent, robots.txt awareness.
Structured output — results as a list of dicts, saved to CSV.
Error resilience — one bad page doesn’t kill the run.

Prerequisites

Python 3.8+ — from python.org.
Dependencies:

pip install requests beautifulsoup4 pandas

Step 1: Create the Script

Save as scraper.py — this example scrapes quote authors and texts from quotes.toscrape.com (a site built for scraping practice):

import requests
from bs4 import BeautifulSoup
import pandas as pd
import time

HEADERS = {"User-Agent": "MyLearningScraper/1.0 (contact: [email protected])"}


def scrape_page(url):
    response = requests.get(url, headers=HEADERS, timeout=10)
    response.raise_for_status()
    soup = BeautifulSoup(response.text, "html.parser")

    results = []
    for quote in soup.select("div.quote"):
        results.append({
            "text": quote.select_one("span.text").get_text(strip=True),
            "author": quote.select_one("small.author").get_text(strip=True),
            "tags": ",".join(t.get_text() for t in quote.select("a.tag")),
        })

    next_link = soup.select_one("li.next a")
    next_url = "https://quotes.toscrape.com" + next_link["href"] if next_link else None
    return results, next_url


def scrape_all(start_url, max_pages=5):
    all_rows, url, page = [], start_url, 1
    while url and page <= max_pages:
        print(f"Scraping page {page}: {url}")
        try:
            rows, url = scrape_page(url)
            all_rows.extend(rows)
        except requests.RequestException as e:
            print(f"  Skipping ({e})")
            break
        page += 1
        time.sleep(1)  # be polite
    return all_rows


if __name__ == "__main__":
    data = scrape_all("https://quotes.toscrape.com/")
    df = pd.DataFrame(data)
    df.to_csv("quotes.csv", index=False)
    print(f"Saved {len(df)} rows to quotes.csv")
    print(df.head())

Step 2: Run the Scraper

python scraper.py

Watch it page through the site, then open quotes.csv — structured data extracted from raw HTML.

How It Works

BeautifulSoup turns raw HTML into a navigable tree, and soup.select() queries it with CSS selectors — the same syntax you use in stylesheets. div.quote finds containers; span.text finds the headline inside each container. The two-level pattern (select containers, then extract fields per container) is the fundamental scraping loop, and it maps directly to how you’d read the page in browser DevTools: right-click an element, Copy selector, adapt.

Pagination is recursion-lite: each page’s HTML contains a li.next a link to the following page. Following it until None walks the whole site — with max_pages as a safety brake, because real sites have surprising link structures.

The politeness layer has three parts: a custom User-Agent that identifies the scraper (many sites block default python-requests), time.sleep(1) between requests so you don’t hammer the server, and try/except so a timeout on page 14 doesn’t discard pages 1–13.

Common Errors & Fixes

AttributeError: 'NoneType' object has no attribute 'get_text' — your selector matched nothing on some page; guard each extraction or use select_one with a check, as the code does per-container.
403 Forbidden — the site blocks your User-Agent; send a descriptive one (never scrape LinkedIn or similar — they actively sue).
Empty results on JS-heavy sites — BeautifulSoup parses the initial HTML; if content loads via JavaScript, you need Selenium or Playwright to render first.
Encoding garbage — pass response.content (bytes) to BeautifulSoup and let it detect encoding, rather than response.text.

Key Concepts

CSS selectors — precise element targeting without regex.
Container loop — select containers, extract fields within.
Pagination as a linked list — follow next-links until they vanish.
Rate limiting — one second per page keeps you welcome.

What to Try Next

Scrape a product page — price, rating, and availability with the same loop.
Add robots.txt checking with urllib.robotparser before each domain.
Schedule the scraper with cron on your home server and chart price history with the dashboard.
Add a Streamlit front-end — paste a URL, see the extracted table, download CSV.

FAQ

Is web scraping legal?

Scraping public data is generally legal in many jurisdictions, but terms of service, copyright, and personal-data laws (GDPR) create real exceptions. Check robots.txt, avoid personal data, and prefer official APIs whenever one exists.

When should I use an API instead?

Always when one exists — APIs are stable, documented, and sanctioned. Scraping is for when the data exists only in HTML.

Use requests.Session() to maintain cookies through the login POST, then scrape authenticated pages. Only do this where the terms permit it.