Introduction
Manual data entry from PDFs and receipts is tedious and error-prone. Automated data extraction reads PDFs, extracts key fields (amounts, dates, vendor names), and saves them to a spreadsheet. This post builds a complete system.
Approach 1: Text-based PDFs (no OCR needed)
For PDFs with selectable text (most digital invoices):
import pdfplumber
import re
def extract_from_pdf(pdf_path):
"""Extract text and key fields from a text-based PDF."""
with pdfplumber.open(pdf_path) as pdf:
text = ""
for page in pdf.pages:
text += page.extract_text() or ""
# Extract fields with regex
data = {}
# Amount
amt_match = re.search(r"(?:Total|Amount|Grand Total)[:\s]*Rs\.?\s*([\d,]+\.?\d*)", text, re.IGNORECASE)
data["amount"] = float(amt_match.group(1).replace(",", "")) if amt_match else None
# Date
date_match = re.search(r"(\d{2}[-/]\d{2}[-/]\d{2,4})", text)
data["date"] = date_match.group(1) if date_match else None
# Invoice number
inv_match = re.search(r"(?:Invoice|Bill|Receipt)\s*(?:No|#|Number)[:\s]*([A-Z0-9-]+)", text, re.IGNORECASE)
data["invoice_number"] = inv_match.group(1) if inv_match else None
# Vendor (usually at the top)
lines = text.strip().split("\n")
data["vendor"] = lines[0].strip() if lines else None
data["raw_text"] = text
return data
# Usage
data = extract_from_pdf("invoice.pdf")
print(data)
# {'amount': 5000.0, 'date': '01-03-26', 'invoice_number': 'INV-001', 'vendor': 'Acme Corp', ...}
Approach 2: Scanned PDFs / images (OCR needed)
For scanned documents or images, use OCR:
pip install pytesseract pillow pdf2image
# Also install Tesseract OCR
# macOS: brew install tesseract
# Linux: apt install tesseract-ocr
import pytesseract
from PIL import Image
from pdf2image import convert_from_path
import re
def ocr_pdf(pdf_path):
"""OCR a scanned PDF and extract text."""
images = convert_from_path(pdf_path, dpi=300)
text = ""
for img in images:
text += pytesseract.image_to_string(img) + "\n"
return text
def ocr_image(image_path):
"""OCR an image (receipt photo)."""
img = Image.open(image_path)
return pytesseract.image_to_string(img)
# Usage
text = ocr_pdf("scanned_invoice.pdf")
# Then use the same regex extraction as above
Approach 3: AI-powered extraction (most accurate)
Use an LLM to extract structured data from any document:
import openai, os, json
client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])
def ai_extract(text):
"""Use AI to extract structured data from document text."""
prompt = f"""Extract the following fields from this document text.
Return ONLY a JSON object with these fields:
- vendor (string)
- date (string, YYYY-MM-DD)
- invoice_number (string)
- amount (number)
- tax (number or null)
- currency (string)
- description (string)
Document text:
{text[:3000]}"""
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
response_format={"type": "json_object"},
)
return json.loads(response.choices[0].message.content)
Why AI is better: Handles varied formats, understands context, and doesn’t break when the layout changes. Cost: ~$0.001 per document.
Step 2: Process multiple files
import os, glob
import pandas as pd
def process_folder(folder_path):
"""Process all PDFs in a folder and save to CSV."""
results = []
for pdf_path in glob.glob(os.path.join(folder_path, "*.pdf")):
print(f"Processing: {pdf_path}")
try:
text = extract_text(pdf_path) # Use pdfplumber or OCR
data = ai_extract(text) # Use AI or regex
data["file"] = os.path.basename(pdf_path)
results.append(data)
except Exception as e:
print(f"Error processing {pdf_path}: {e}")
results.append({"file": os.path.basename(pdf_path), "error": str(e)})
df = pd.DataFrame(results)
df.to_csv("extracted_data.csv", index=False)
print(f"\nProcessed {len(results)} files. Saved to extracted_data.csv")
return df
process_folder("invoices/")
Step 3: Save to Google Sheets
import gspread
from google.oauth2.service_account import Credentials
def save_to_sheets(data):
creds = Credentials.from_service_account_file("service_account.json",
scopes=["https://www.googleapis.com/auth/spreadsheets"])
gc = gspread.authorize(creds)
sheet = gc.open("Expense Tracker").sheet1
for item in data:
sheet.append_row([
item.get("date", ""),
item.get("vendor", ""),
item.get("amount", 0),
item.get("tax", 0),
item.get("description", ""),
item.get("file", ""),
])
print(f"Saved {len(data)} rows to Google Sheets")
Step 4: Automate with a watch folder
import time, os
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
class PDFHandler(FileSystemEventHandler):
def on_created(self, event):
if event.src_path.endswith(".pdf"):
print(f"New PDF detected: {event.src_path}")
data = process_pdf(event.src_path)
save_to_sheets([data])
print(f"Processed and saved: {data}")
observer = Observer()
observer.schedule(PDFHandler(), path="./inbox", recursive=False)
observer.start()
print("Watching for new PDFs in ./inbox/")
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
observer.stop()
observer.join()
Drop a PDF in the ./inbox/ folder and it’s automatically processed and saved.
Common mistakes
- Low-quality scans — OCR accuracy drops significantly on low-res or blurry images. Use 300 DPI minimum.
- Complex layouts — tables, multi-column layouts confuse OCR. Use AI extraction for complex documents.
- Not validating extracted data — always review AI-extracted data before saving. AI can hallucinate.
- No error handling — if one PDF fails, the script should continue to the next. Don’t let one bad file stop the batch.
- Ignoring duplicates — if you process the same PDF twice, you’ll get duplicate entries. Use invoice numbers to dedupe.
Key Concepts
- OCR = Optical Character Recognition (image → text)
- pdfplumber = Python library for text-based PDFs
- pytesseract = Python wrapper for Tesseract OCR
- AI extraction = using LLMs to extract structured data
- Watchdog = Python library for file system monitoring
- Deduplication = preventing duplicate entries
What to Try Next
- Automate expense tracking from bank SMS for financial data
- Track freelance income and expenses for accounting
- Build an expense dashboard in Streamlit
- Automate invoices for the other direction
FAQ
OCR vs AI extraction — which is better?
OCR = free, fast, but struggles with complex layouts and low-quality scans. AI extraction = more accurate, handles any format, but costs ~$0.001/document. Use OCR for clean, simple documents. Use AI for complex or varied formats.
Can this handle handwritten receipts?
OCR struggles with handwriting. AI extraction (GPT-4o with vision) can read handwriting reasonably well. For critical documents, always have a human review.
What’s the accuracy rate?
OCR: 85-95% on clean digital PDFs, 60-80% on scanned documents. AI extraction: 90-98% on most documents. Always validate extracted data before saving.
Can I process 1000+ documents?
Yes — the script processes files in a loop. For large batches (1000+), use multiprocessing or a queue (Celery, RQ). Consider cloud-based OCR (Google Cloud Vision, AWS Textract) for better accuracy at scale. See Bulk PDF Invoices.