Introduction
Chat with PDF used a cloud API. That’s fine for demos — but what if your documents are confidential? Client contracts, medical records, internal wikis. Sending them to a cloud LLM is a non-starter.
RAG with local LLMs fixes that: chunk your docs, embed them locally with Ollama, store in Chroma, and answer questions with a local model. Nothing leaves your machine. Zero API cost.
What you’ll build
- A Streamlit app that ingests PDFs/TXT
- Local embeddings via Ollama (
nomic-embed-text) - Local vector store (Chroma, persisted to disk)
- Local answer generation via
llama3.2
Prerequisites
# Ollama
curl -fsSL https://ollama.com/install.sh | sh
ollama pull nomic-embed-text
ollama pull llama3.2
pip install streamlit chromadb ollama pypdf
Step 1: Chunking
from pypdf import PdfReader
def load_text(path):
reader = PdfReader(path)
return "\n".join(p.extract_text() or "" for p in reader.pages)
def chunk(text, size=500, overlap=50):
words = text.split()
out = []
i = 0
while i < len(words):
out.append(" ".join(words[i:i+size]))
i += size - overlap
return out
Chunking is the most underrated step. Too big → the model gets lost; too small → no context. 500 words with 50 overlap is a sane default.
Step 2: Embed and store in Chroma
import chromadb, ollama
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection("docs")
def ingest(chunks):
for i, c in enumerate(chunks):
emb = ollama.embeddings(model="nomic-embed-text", prompt=c)["embedding"]
collection.add(ids=[f"chunk-{i}"], documents=[c], embeddings=[emb])
PersistentClient writes to disk, so you ingest once and query forever.
Step 3: Retrieve + answer
def retrieve(question, k=4):
q_emb = ollama.embeddings(model="nomic-embed-text", prompt=question)["embedding"]
res = collection.query(query_embeddings=[q_emb], n_results=k)
return res["documents"][0]
def answer(question):
ctx = "\n\n---\n\n".join(retrieve(question))
prompt = f"""Answer using ONLY the context below. If unsure, say "I don't know."
Context:
{ctx}
Question: {question}"""
return ollama.chat(model="llama3.2", messages=[{"role": "user", "content": prompt}])["message"]["content"]
Step 4: Streamlit UI
import streamlit as st
st.title("Private RAG — 100% local")
uploaded = st.file_uploader("Upload PDF", type="pdf")
if uploaded and "ingested" not in st.session_state:
text = load_text(uploaded)
chunks = chunk(text)
ingest(chunks)
st.session_state.ingested = True
st.success(f"Ingested {len(chunks)} chunks")
q = st.text_input("Ask a question")
if q and "ingested" in st.session_state:
with st.spinner("Thinking locally..."):
st.write(answer(q))
Why this beats cloud RAG
| Cloud (Mistral/OpenAI) | Local (Ollama) | |
|---|---|---|
| Privacy | Data leaves machine | 100% local |
| Cost | Per-token billing | Free |
| Speed | Network latency | Depends on hardware |
| Quality | Top-tier models | Good for 8B+ models |
| Offline | No | Yes |
Common mistakes
- Chunking by characters, not words — breaks mid-word. Split on words, then join.
- No overlap — answers spanning two chunks get lost. Always overlap.
- Re-ingesting every run — use
PersistentClient+ a session flag. - Tiny embedding model on big docs —
nomic-embed-text(137M) is fine; for huge corpora usemxbai-embed-large. - Forgetting to pull models —
ollama pullboth embed and chat models before running.
Key Concepts
- RAG = Retrieve relevant chunks → Augment prompt → Generate answer
- Embeddings = vector representations of text for similarity search
- Chunking + overlap = the secret to good retrieval
- Persistent vector store = ingest once, query forever
- Local LLM = privacy + zero cost, trades some quality
What to Try Next
- Add source citations — store page numbers in Chroma metadata and show them.
- Support multiple documents with per-doc filtering.
- Swap
llama3.2formistralorphi3and compare answer quality. - Combine with prompt engineering techniques to tighten the answer prompt.
- Deploy as a Cloudflare Pages app (embeddings run client-side? No — host the Ollama backend on a GPU box).
FAQ
How much RAM do I need?
llama3.2 (3B) runs on 8GB. llama3.1:8b needs 16GB. Embeddings are tiny. A modern laptop handles this fine.
Is local RAG as good as ChatGPT on my docs?
For factual Q&A over your own docs, often yes — retrieval grounds the answer. For reasoning-heavy tasks, cloud models still win. Hybrid: retrieve locally, generate with a cloud model if privacy allows.
Can I use this with the PDF chat tutorial?
Yes — this post is the privacy-first sibling. Same UX, different backend. Pick based on your data sensitivity.