Introduction
requests is synchronous — one request at a time. Scraping 100 pages takes 100 × 200ms = 20 seconds. aiohttp + asyncio runs them concurrently: 100 pages in ~2 seconds. That’s a 10x speedup with the same code structure.
This post upgrades your scraper to async.
Setup
pip install aiohttp beautifulsoup4
Step 1: Basic async fetch
import asyncio, aiohttp
async def fetch(session, url):
async with session.get(url) as resp:
return await resp.text()
async def main():
async with aiohttp.ClientSession() as session:
html = await fetch(session, "https://example.com")
print(html[:200])
asyncio.run(main())
ClientSession is reusable — create once, make many requests. async with ensures cleanup.
Step 2: Concurrent scraping (the 10x win)
import asyncio, aiohttp
from bs4 import BeautifulSoup
async def fetch(session, url):
async with session.get(url) as resp:
return await resp.text()
async def scrape_one(session, url):
html = await fetch(session, url)
soup = BeautifulSoup(html, "html.parser")
return soup.title.string if soup.title else "No title"
async def main(urls):
async with aiohttp.ClientSession() as session:
tasks = [scrape_one(session, u) for u in urls]
return await asyncio.gather(*tasks)
urls = [f"https://example.com/page/{i}" for i in range(100)]
results = asyncio.run(main(urls))
print(f"Scraped {len(results)} pages")
100 pages in ~2s instead of 20s. asyncio.gather runs all fetches concurrently.
Step 3: Rate limiting with semaphores
Don’t hammer the server — limit concurrency:
sem = asyncio.Semaphore(10) # max 10 concurrent
async def fetch_limited(session, url):
async with sem:
async with session.get(url) as resp:
await asyncio.sleep(0.1) # be polite
return await resp.text()
10 concurrent requests with a 100ms delay between batches. Respectful and fast.
Step 4: Error handling
async def fetch_safe(session, url):
try:
async with session.get(url, timeout=aiohttp.ClientTimeout(total=10)) as resp:
if resp.status == 200:
return await resp.text()
return None
except asyncio.TimeoutError:
print(f"Timeout: {url}")
return None
except Exception as e:
print(f"Error {url}: {e}")
return None
Always set timeouts. Network requests can hang forever.
Step 5: Retries with backoff
async def fetch_with_retry(session, url, max_retries=3):
for attempt in range(max_retries):
try:
async with session.get(url) as resp:
if resp.status == 200:
return await resp.text()
if resp.status == 429: # rate limited
await asyncio.sleep(2 ** attempt)
continue
except Exception:
await asyncio.sleep(2 ** attempt)
return None
Exponential backoff on errors. Handle 429 (rate limit) by waiting longer.
Step 6: Parse and extract data
async def scrape_products(session, url):
html = await fetch(session, url)
soup = BeautifulSoup(html, "html.parser")
products = []
for card in soup.select(".product-card"):
products.append({
"name": card.select_one(".name").get_text(strip=True),
"price": card.select_one(".price").get_text(strip=True),
"link": card.select_one("a")["href"],
})
return products
async def main():
async with aiohttp.ClientSession() as session:
urls = [f"https://shop.example.com/products?page={i}" for i in range(1, 11)]
tasks = [scrape_products(session, u) for u in urls]
results = await asyncio.gather(*tasks)
all_products = [p for page in results for p in page]
print(f"Found {len(all_products)} products")
Step 7: Save results
import json
async def main():
...
with open("products.json", "w") as f:
json.dump(all_products, f, indent=2)
print(f"Saved {len(all_products)} products")
Or use aiofiles for async file I/O.
Step 8: Progress tracking
import asyncio
async def main(urls):
async with aiohttp.ClientSession() as session:
tasks = [scrape_one(session, u) for u in urls]
results = []
for coro in asyncio.as_completed(tasks):
result = await coro
results.append(result)
print(f"Progress: {len(results)}/{len(urls)}")
return results
as_completed yields results as they finish (not in order) — great for progress bars.
aiohttp vs requests vs Playwright
| requests | aiohttp | Playwright | |
|---|---|---|---|
| Speed | Slow (sync) | Fast (async) | Slowest (browser) |
| JS rendering | No | No | Yes |
| Concurrency | No | Yes (asyncio) | Limited |
| Memory | Low | Low | High (~150MB) |
| Use case | Simple scrapes | Bulk static scrapes | JS-rendered sites |
Rule: try requests first. If slow, use aiohttp. If the page is empty/JS-rendered, use Playwright.
Common mistakes
- Creating a session per request —
ClientSessionis reusable; create once. - No timeout — requests can hang. Always set
ClientTimeout. - No rate limiting — semaphores + delays. Be polite.
- Ignoring
robots.txt— check before scraping. - Not closing the session — use
async withto ensure cleanup. - Mixing sync and async —
requests.getinside async code blocks the event loop. Useaiohttporasyncio.to_thread.
Key Concepts
- aiohttp = async HTTP client for Python
ClientSession= reusable connection poolasyncio.gather= run many coroutines concurrently- Semaphore = limit concurrency
as_completed= process results as they arrive
What to Try Next
- Scrape a Streamlit app gallery and build a searchable index.
- Combine with BeautifulSoup for parsing.
- Add Resend email alerts when prices drop.
- Store scraped data in Supabase.
FAQ
aiohttp vs httpx?
Both are async HTTP clients. httpx has a sync API too (drop-in requests replacement) and supports HTTP/2. aiohttp is more mature and slightly faster. For async-only, either works. For sync+async, use httpx.
Can I use Playwright asynchronously?
Yes — Playwright has an async API (playwright.async_api). Combine with aiohttp for the best of both: fast async fetching + browser rendering when needed.
How many concurrent requests is too many?
Depends on the target server. Start with 10-20 concurrent, add delays, and monitor for 429/503 responses. Be respectful — scraping is a privilege, not a right.