Introduction
Knowing your competitors’ prices is critical for pricing strategy. But checking manually is tedious and inconsistent. Automated price monitoring checks competitor prices daily and alerts you when they change.
This post builds a complete competitor price monitoring system.
What you’ll build
- A Python scraper that checks competitor prices daily
- Price history tracking (SQLite database)
- Alert system (email/WhatsApp when prices change)
- A Streamlit dashboard for price comparison
Step 1: Identify competitors and products
Create a config file with competitors and their product URLs:
// competitors.json
[
{
"name": "Competitor A",
"products": [
{"name": "Product 1", "url": "https://competitor-a.com/product1", "selector": ".price"},
{"name": "Product 2", "url": "https://competitor-a.com/product2", "selector": ".price"}
]
},
{
"name": "Competitor B",
"products": [
{"name": "Product 1", "url": "https://competitor-b.com/product1", "selector": ".product-price"},
]
}
]
Step 2: Scrape prices
import requests
from bs4 import BeautifulSoup
import re, json
def scrape_price(url, selector):
"""Scrape a price from a URL using a CSS selector."""
try:
r = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=10)
soup = BeautifulSoup(r.text, "html.parser")
element = soup.select_one(selector)
if element:
text = element.text.strip()
# Extract numeric price
price = re.search(r"[\d,]+\.?\d*", text.replace(",", ""))
return float(price.group()) if price else None
return None
except Exception as e:
print(f"Error scraping {url}: {e}")
return None
def check_all_prices():
"""Check prices for all competitors and products."""
with open("competitors.json") as f:
competitors = json.load(f)
results = []
for comp in competitors:
for product in comp["products"]:
price = scrape_price(product["url"], product["selector"])
results.append({
"competitor": comp["name"],
"product": product["name"],
"price": price,
"url": product["url"],
"checked_at": datetime.now().isoformat(),
})
print(f"{comp['name']} | {product['name']}: Rs. {price}")
time.sleep(1) # be polite
return results
Step 3: Store price history
import sqlite3
from datetime import datetime
def init_db():
conn = sqlite3.connect("prices.db")
conn.execute("""
CREATE TABLE IF NOT EXISTS prices (
id INTEGER PRIMARY KEY AUTOINCREMENT,
competitor TEXT,
product TEXT,
price REAL,
checked_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
""")
conn.commit()
conn.close()
def save_prices(results):
conn = sqlite3.connect("prices.db")
for r in results:
conn.execute(
"INSERT INTO prices (competitor, product, price, checked_at) VALUES (?, ?, ?, ?)",
(r["competitor"], r["product"], r["price"], r["checked_at"])
)
conn.commit()
conn.close()
def get_price_history(competitor, product, days=30):
conn = sqlite3.connect("prices.db")
rows = conn.execute("""
SELECT price, checked_at FROM prices
WHERE competitor = ? AND product = ?
AND checked_at >= datetime('now', ?)
ORDER BY checked_at
""", (competitor, product, f"-{days} days")).fetchall()
conn.close()
return rows
Step 4: Detect price changes
def detect_changes(results):
"""Detect price changes since last check."""
conn = sqlite3.connect("prices.db")
changes = []
for r in results:
# Get previous price
prev = conn.execute("""
SELECT price FROM prices
WHERE competitor = ? AND product = ?
ORDER BY checked_at DESC LIMIT 1 OFFSET 1
""", (r["competitor"], r["product"])).fetchone()
if prev and prev[0] != r["price"]:
change = r["price"] - prev[0]
pct = (change / prev[0]) * 100
changes.append({
"competitor": r["competitor"],
"product": r["product"],
"old_price": prev[0],
"new_price": r["price"],
"change": change,
"pct_change": pct,
})
conn.close()
return changes
Step 5: Send alerts
import resend, os
resend.api_key = os.environ["RESEND_API_KEY"]
def send_price_alert(changes):
"""Send email alert for price changes."""
if not changes:
return
body = "<h3>Competitor Price Changes Detected</h3><ul>"
for c in changes:
direction = "📈 increased" if c["change"] > 0 else "📉 decreased"
body += f"<li><strong>{c['competitor']} - {c['product']}:</strong> Rs. {c['old_price']:,.0f} → Rs. {c['new_price']:,.0f} ({direction} {abs(c['pct_change']):.1f}%)</li>"
body += "</ul>"
resend.Emails.send({
"from": "Price Alerts <[email protected]>",
"to": [os.environ["YOUR_EMAIL"]],
"subject": f"Competitor Price Alert: {len(changes)} changes detected",
"html": body,
})
print(f"Alert sent for {len(changes)} price changes")
Step 6: Automate with cron
# Check prices daily at 8 AM
0 8 * * * cd /path/to/project && python price_monitor.py
# price_monitor.py
import json, time
from datetime import datetime
init_db()
results = check_all_prices()
save_prices(results)
changes = detect_changes(results)
send_price_alert(changes)
print(f"\nChecked {len(results)} products. {len(changes)} price changes detected.")
Step 7: Streamlit dashboard
import streamlit as st
import pandas as pd
import plotly.express as px
import sqlite3
st.title("📊 Competitor Price Monitor")
conn = sqlite3.connect("prices.db")
df = pd.read_sql("SELECT * FROM prices ORDER BY checked_at DESC", conn)
# Latest prices
st.subheader("Latest Prices")
latest = df.drop_duplicates(subset=["competitor", "product"], keep="first")
st.dataframe(latest[["competitor", "product", "price", "checked_at"]], use_container_width=True)
# Price history chart
st.subheader("Price History")
selected_product = st.selectbox("Product", df["product"].unique())
filtered = df[df["product"] == selected_product]
fig = px.line(filtered, x="checked_at", y="price", color="competitor",
title=f"Price History: {selected_product}")
st.plotly_chart(fig, use_container_width=True)
# Price comparison
st.subheader("Price Comparison")
comparison = latest.pivot(index="product", columns="competitor", values="price")
st.dataframe(comparison, use_container_width=True)
Best practices
- Be polite — add delays between requests (1-2 seconds)
- Respect robots.txt — check before scraping
- Handle errors gracefully — sites change their HTML; your scraper will break
- Don’t scrape too frequently — daily is enough for most businesses
- Use official APIs when available — many e-commerce platforms have price APIs
- Store historical data — price trends are more valuable than single snapshots
Common mistakes
- No rate limiting — scraping too fast gets your IP banned
- Hardcoded selectors — when the site changes HTML, your scraper breaks. Use multiple selectors.
- No error handling — one failed scrape shouldn’t crash the whole script
- Ignoring price drops — a competitor lowering prices is more important than raising them
- Not tracking your own prices — include your own prices in the comparison
Key Concepts
- Price monitoring = tracking competitor prices over time
- Web scraping = extracting data from websites
- Price history = historical price data for trend analysis
- Alert = notification when a price changes
- CSS selector = HTML element identifier for scraping
What to Try Next
- Scrape business leads from Google Maps
- Automate SEO reports for clients
- Track Google rankings automatically
- Build a monitoring dashboard in Streamlit
FAQ
Is price monitoring legal?
Monitoring publicly available prices is generally legal. However, scraping may violate a site’s Terms of Service. Use official APIs when available. For e-commerce, many platforms (Amazon, Shopify) have price APIs or allow scraping with rate limits.
How often should I check competitor prices?
Daily for most businesses. Hourly for highly competitive markets (e-commerce, travel). Weekly for B2B services. Match the frequency to how often competitors change prices.
What if a competitor blocks my scraper?
Use: (1) rotating user agents, (2) proxies, (3) longer delays between requests, (4) official APIs instead of scraping. If all else fails, use a price monitoring service (Prisync, Competera, etc.).
Can I monitor prices on Amazon?
Amazon has strict anti-scraping measures. Use the Amazon Product Advertising API or a specialized service (Keepa, CamelCamelCamel). Don’t scrape Amazon directly — you’ll get banned. See Web Scraping with Playwright.