Tech With Galvan
Back to Articles
AutomationSep 8, 2029 • 5 min read

How to Monitor Competitor Prices Automatically

Monitor competitor prices automatically — Python scraper with alerts, price tracking, and a dashboard. For e-commerce and service businesses.

Galvan
Galvan

Founder & Creator

Introduction

Knowing your competitors’ prices is critical for pricing strategy. But checking manually is tedious and inconsistent. Automated price monitoring checks competitor prices daily and alerts you when they change.

This post builds a complete competitor price monitoring system.

What you’ll build

  • A Python scraper that checks competitor prices daily
  • Price history tracking (SQLite database)
  • Alert system (email/WhatsApp when prices change)
  • A Streamlit dashboard for price comparison

Step 1: Identify competitors and products

Create a config file with competitors and their product URLs:

// competitors.json
[
  {
    "name": "Competitor A",
    "products": [
      {"name": "Product 1", "url": "https://competitor-a.com/product1", "selector": ".price"},
      {"name": "Product 2", "url": "https://competitor-a.com/product2", "selector": ".price"}
    ]
  },
  {
    "name": "Competitor B",
    "products": [
      {"name": "Product 1", "url": "https://competitor-b.com/product1", "selector": ".product-price"},
    ]
  }
]

Step 2: Scrape prices

import requests
from bs4 import BeautifulSoup
import re, json

def scrape_price(url, selector):
    """Scrape a price from a URL using a CSS selector."""
    try:
        r = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=10)
        soup = BeautifulSoup(r.text, "html.parser")
        element = soup.select_one(selector)
        if element:
            text = element.text.strip()
            # Extract numeric price
            price = re.search(r"[\d,]+\.?\d*", text.replace(",", ""))
            return float(price.group()) if price else None
        return None
    except Exception as e:
        print(f"Error scraping {url}: {e}")
        return None

def check_all_prices():
    """Check prices for all competitors and products."""
    with open("competitors.json") as f:
        competitors = json.load(f)
    
    results = []
    for comp in competitors:
        for product in comp["products"]:
            price = scrape_price(product["url"], product["selector"])
            results.append({
                "competitor": comp["name"],
                "product": product["name"],
                "price": price,
                "url": product["url"],
                "checked_at": datetime.now().isoformat(),
            })
            print(f"{comp['name']} | {product['name']}: Rs. {price}")
            time.sleep(1)  # be polite
    
    return results

Step 3: Store price history

import sqlite3
from datetime import datetime

def init_db():
    conn = sqlite3.connect("prices.db")
    conn.execute("""
        CREATE TABLE IF NOT EXISTS prices (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            competitor TEXT,
            product TEXT,
            price REAL,
            checked_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        )
    """)
    conn.commit()
    conn.close()

def save_prices(results):
    conn = sqlite3.connect("prices.db")
    for r in results:
        conn.execute(
            "INSERT INTO prices (competitor, product, price, checked_at) VALUES (?, ?, ?, ?)",
            (r["competitor"], r["product"], r["price"], r["checked_at"])
        )
    conn.commit()
    conn.close()

def get_price_history(competitor, product, days=30):
    conn = sqlite3.connect("prices.db")
    rows = conn.execute("""
        SELECT price, checked_at FROM prices 
        WHERE competitor = ? AND product = ? 
        AND checked_at >= datetime('now', ?)
        ORDER BY checked_at
    """, (competitor, product, f"-{days} days")).fetchall()
    conn.close()
    return rows

Step 4: Detect price changes

def detect_changes(results):
    """Detect price changes since last check."""
    conn = sqlite3.connect("prices.db")
    changes = []
    
    for r in results:
        # Get previous price
        prev = conn.execute("""
            SELECT price FROM prices 
            WHERE competitor = ? AND product = ? 
            ORDER BY checked_at DESC LIMIT 1 OFFSET 1
        """, (r["competitor"], r["product"])).fetchone()
        
        if prev and prev[0] != r["price"]:
            change = r["price"] - prev[0]
            pct = (change / prev[0]) * 100
            changes.append({
                "competitor": r["competitor"],
                "product": r["product"],
                "old_price": prev[0],
                "new_price": r["price"],
                "change": change,
                "pct_change": pct,
            })
    
    conn.close()
    return changes

Step 5: Send alerts

import resend, os

resend.api_key = os.environ["RESEND_API_KEY"]

def send_price_alert(changes):
    """Send email alert for price changes."""
    if not changes:
        return
    
    body = "<h3>Competitor Price Changes Detected</h3><ul>"
    for c in changes:
        direction = "📈 increased" if c["change"] > 0 else "📉 decreased"
        body += f"<li><strong>{c['competitor']} - {c['product']}:</strong> Rs. {c['old_price']:,.0f} → Rs. {c['new_price']:,.0f} ({direction} {abs(c['pct_change']):.1f}%)</li>"
    body += "</ul>"
    
    resend.Emails.send({
        "from": "Price Alerts <[email protected]>",
        "to": [os.environ["YOUR_EMAIL"]],
        "subject": f"Competitor Price Alert: {len(changes)} changes detected",
        "html": body,
    })
    print(f"Alert sent for {len(changes)} price changes")

Step 6: Automate with cron

# Check prices daily at 8 AM
0 8 * * * cd /path/to/project && python price_monitor.py
# price_monitor.py
import json, time
from datetime import datetime

init_db()
results = check_all_prices()
save_prices(results)
changes = detect_changes(results)
send_price_alert(changes)
print(f"\nChecked {len(results)} products. {len(changes)} price changes detected.")

Step 7: Streamlit dashboard

import streamlit as st
import pandas as pd
import plotly.express as px
import sqlite3

st.title("📊 Competitor Price Monitor")

conn = sqlite3.connect("prices.db")
df = pd.read_sql("SELECT * FROM prices ORDER BY checked_at DESC", conn)

# Latest prices
st.subheader("Latest Prices")
latest = df.drop_duplicates(subset=["competitor", "product"], keep="first")
st.dataframe(latest[["competitor", "product", "price", "checked_at"]], use_container_width=True)

# Price history chart
st.subheader("Price History")
selected_product = st.selectbox("Product", df["product"].unique())
filtered = df[df["product"] == selected_product]
fig = px.line(filtered, x="checked_at", y="price", color="competitor",
              title=f"Price History: {selected_product}")
st.plotly_chart(fig, use_container_width=True)

# Price comparison
st.subheader("Price Comparison")
comparison = latest.pivot(index="product", columns="competitor", values="price")
st.dataframe(comparison, use_container_width=True)

Best practices

  1. Be polite — add delays between requests (1-2 seconds)
  2. Respect robots.txt — check before scraping
  3. Handle errors gracefully — sites change their HTML; your scraper will break
  4. Don’t scrape too frequently — daily is enough for most businesses
  5. Use official APIs when available — many e-commerce platforms have price APIs
  6. Store historical data — price trends are more valuable than single snapshots

Common mistakes

  • No rate limiting — scraping too fast gets your IP banned
  • Hardcoded selectors — when the site changes HTML, your scraper breaks. Use multiple selectors.
  • No error handling — one failed scrape shouldn’t crash the whole script
  • Ignoring price drops — a competitor lowering prices is more important than raising them
  • Not tracking your own prices — include your own prices in the comparison

Key Concepts

  • Price monitoring = tracking competitor prices over time
  • Web scraping = extracting data from websites
  • Price history = historical price data for trend analysis
  • Alert = notification when a price changes
  • CSS selector = HTML element identifier for scraping

What to Try Next

FAQ

Monitoring publicly available prices is generally legal. However, scraping may violate a site’s Terms of Service. Use official APIs when available. For e-commerce, many platforms (Amazon, Shopify) have price APIs or allow scraping with rate limits.

How often should I check competitor prices?

Daily for most businesses. Hourly for highly competitive markets (e-commerce, travel). Weekly for B2B services. Match the frequency to how often competitors change prices.

What if a competitor blocks my scraper?

Use: (1) rotating user agents, (2) proxies, (3) longer delays between requests, (4) official APIs instead of scraping. If all else fails, use a price monitoring service (Prisync, Competera, etc.).

Can I monitor prices on Amazon?

Amazon has strict anti-scraping measures. Use the Amazon Product Advertising API or a specialized service (Keepa, CamelCamelCamel). Don’t scrape Amazon directly — you’ll get banned. See Web Scraping with Playwright.