Tech With Galvan
Back to Articles
DevelopmentOct 31, 2026 • 5 min read

Async Web Scraping with aiohttp (10x Faster Than requests)

Scrape websites 10x faster with aiohttp and asyncio — concurrent requests, rate limiting, and error handling. A modern upgrade to requests + BeautifulSoup.

Galvan
Galvan

Founder & Creator

Introduction

requests is synchronous — one request at a time. Scraping 100 pages takes 100 × 200ms = 20 seconds. aiohttp + asyncio runs them concurrently: 100 pages in ~2 seconds. That’s a 10x speedup with the same code structure.

This post upgrades your scraper to async.

Setup

pip install aiohttp beautifulsoup4

Step 1: Basic async fetch

import asyncio, aiohttp

async def fetch(session, url):
    async with session.get(url) as resp:
        return await resp.text()

async def main():
    async with aiohttp.ClientSession() as session:
        html = await fetch(session, "https://example.com")
        print(html[:200])

asyncio.run(main())

ClientSession is reusable — create once, make many requests. async with ensures cleanup.

Step 2: Concurrent scraping (the 10x win)

import asyncio, aiohttp
from bs4 import BeautifulSoup

async def fetch(session, url):
    async with session.get(url) as resp:
        return await resp.text()

async def scrape_one(session, url):
    html = await fetch(session, url)
    soup = BeautifulSoup(html, "html.parser")
    return soup.title.string if soup.title else "No title"

async def main(urls):
    async with aiohttp.ClientSession() as session:
        tasks = [scrape_one(session, u) for u in urls]
        return await asyncio.gather(*tasks)

urls = [f"https://example.com/page/{i}" for i in range(100)]
results = asyncio.run(main(urls))
print(f"Scraped {len(results)} pages")

100 pages in ~2s instead of 20s. asyncio.gather runs all fetches concurrently.

Step 3: Rate limiting with semaphores

Don’t hammer the server — limit concurrency:

sem = asyncio.Semaphore(10)   # max 10 concurrent

async def fetch_limited(session, url):
    async with sem:
        async with session.get(url) as resp:
            await asyncio.sleep(0.1)   # be polite
            return await resp.text()

10 concurrent requests with a 100ms delay between batches. Respectful and fast.

Step 4: Error handling

async def fetch_safe(session, url):
    try:
        async with session.get(url, timeout=aiohttp.ClientTimeout(total=10)) as resp:
            if resp.status == 200:
                return await resp.text()
            return None
    except asyncio.TimeoutError:
        print(f"Timeout: {url}")
        return None
    except Exception as e:
        print(f"Error {url}: {e}")
        return None

Always set timeouts. Network requests can hang forever.

Step 5: Retries with backoff

async def fetch_with_retry(session, url, max_retries=3):
    for attempt in range(max_retries):
        try:
            async with session.get(url) as resp:
                if resp.status == 200:
                    return await resp.text()
                if resp.status == 429:   # rate limited
                    await asyncio.sleep(2 ** attempt)
                    continue
        except Exception:
            await asyncio.sleep(2 ** attempt)
    return None

Exponential backoff on errors. Handle 429 (rate limit) by waiting longer.

Step 6: Parse and extract data

async def scrape_products(session, url):
    html = await fetch(session, url)
    soup = BeautifulSoup(html, "html.parser")
    products = []
    for card in soup.select(".product-card"):
        products.append({
            "name": card.select_one(".name").get_text(strip=True),
            "price": card.select_one(".price").get_text(strip=True),
            "link": card.select_one("a")["href"],
        })
    return products

async def main():
    async with aiohttp.ClientSession() as session:
        urls = [f"https://shop.example.com/products?page={i}" for i in range(1, 11)]
        tasks = [scrape_products(session, u) for u in urls]
        results = await asyncio.gather(*tasks)
        all_products = [p for page in results for p in page]
        print(f"Found {len(all_products)} products")

Step 7: Save results

import json

async def main():
    ...
    with open("products.json", "w") as f:
        json.dump(all_products, f, indent=2)
    print(f"Saved {len(all_products)} products")

Or use aiofiles for async file I/O.

Step 8: Progress tracking

import asyncio

async def main(urls):
    async with aiohttp.ClientSession() as session:
        tasks = [scrape_one(session, u) for u in urls]
        results = []
        for coro in asyncio.as_completed(tasks):
            result = await coro
            results.append(result)
            print(f"Progress: {len(results)}/{len(urls)}")
        return results

as_completed yields results as they finish (not in order) — great for progress bars.

aiohttp vs requests vs Playwright

requests aiohttp Playwright
Speed Slow (sync) Fast (async) Slowest (browser)
JS rendering No No Yes
Concurrency No Yes (asyncio) Limited
Memory Low Low High (~150MB)
Use case Simple scrapes Bulk static scrapes JS-rendered sites

Rule: try requests first. If slow, use aiohttp. If the page is empty/JS-rendered, use Playwright.

Common mistakes

  • Creating a session per request — ClientSession is reusable; create once.
  • No timeout — requests can hang. Always set ClientTimeout.
  • No rate limiting — semaphores + delays. Be polite.
  • Ignoring robots.txt — check before scraping.
  • Not closing the session — use async with to ensure cleanup.
  • Mixing sync and async — requests.get inside async code blocks the event loop. Use aiohttp or asyncio.to_thread.

Key Concepts

  • aiohttp = async HTTP client for Python
  • ClientSession = reusable connection pool
  • asyncio.gather = run many coroutines concurrently
  • Semaphore = limit concurrency
  • as_completed = process results as they arrive

What to Try Next

FAQ

aiohttp vs httpx?

Both are async HTTP clients. httpx has a sync API too (drop-in requests replacement) and supports HTTP/2. aiohttp is more mature and slightly faster. For async-only, either works. For sync+async, use httpx.

Can I use Playwright asynchronously?

Yes — Playwright has an async API (playwright.async_api). Combine with aiohttp for the best of both: fast async fetching + browser rendering when needed.

How many concurrent requests is too many?

Depends on the target server. Start with 10-20 concurrent, add delays, and monitor for 429/503 responses. Be respectful — scraping is a privilege, not a right.