Tech With Galvan
Back to Articles
AIApr 5, 2026 • 6 min read

RAG with Local LLMs: Ollama + Chroma + Streamlit (Private, Free)

Build a private RAG app with local LLMs — chunk documents, embed with Ollama, store in Chroma, and query from Streamlit. No API keys, no data leaves your machine.

Galvan
Galvan

Founder & Creator

Introduction

Chat with PDF used a cloud API. That’s fine for demos — but what if your documents are confidential? Client contracts, medical records, internal wikis. Sending them to a cloud LLM is a non-starter.

RAG with local LLMs fixes that: chunk your docs, embed them locally with Ollama, store in Chroma, and answer questions with a local model. Nothing leaves your machine. Zero API cost.

What you’ll build

  • A Streamlit app that ingests PDFs/TXT
  • Local embeddings via Ollama (nomic-embed-text)
  • Local vector store (Chroma, persisted to disk)
  • Local answer generation via llama3.2

Prerequisites

# Ollama
curl -fsSL https://ollama.com/install.sh | sh
ollama pull nomic-embed-text
ollama pull llama3.2
pip install streamlit chromadb ollama pypdf

Step 1: Chunking

from pypdf import PdfReader

def load_text(path):
    reader = PdfReader(path)
    return "\n".join(p.extract_text() or "" for p in reader.pages)

def chunk(text, size=500, overlap=50):
    words = text.split()
    out = []
    i = 0
    while i < len(words):
        out.append(" ".join(words[i:i+size]))
        i += size - overlap
    return out

Chunking is the most underrated step. Too big → the model gets lost; too small → no context. 500 words with 50 overlap is a sane default.

Step 2: Embed and store in Chroma

import chromadb, ollama

client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection("docs")

def ingest(chunks):
    for i, c in enumerate(chunks):
        emb = ollama.embeddings(model="nomic-embed-text", prompt=c)["embedding"]
        collection.add(ids=[f"chunk-{i}"], documents=[c], embeddings=[emb])

PersistentClient writes to disk, so you ingest once and query forever.

Step 3: Retrieve + answer

def retrieve(question, k=4):
    q_emb = ollama.embeddings(model="nomic-embed-text", prompt=question)["embedding"]
    res = collection.query(query_embeddings=[q_emb], n_results=k)
    return res["documents"][0]

def answer(question):
    ctx = "\n\n---\n\n".join(retrieve(question))
    prompt = f"""Answer using ONLY the context below. If unsure, say "I don't know."

Context:
{ctx}

Question: {question}"""
    return ollama.chat(model="llama3.2", messages=[{"role": "user", "content": prompt}])["message"]["content"]

Step 4: Streamlit UI

import streamlit as st

st.title("Private RAG — 100% local")
uploaded = st.file_uploader("Upload PDF", type="pdf")

if uploaded and "ingested" not in st.session_state:
    text = load_text(uploaded)
    chunks = chunk(text)
    ingest(chunks)
    st.session_state.ingested = True
    st.success(f"Ingested {len(chunks)} chunks")

q = st.text_input("Ask a question")
if q and "ingested" in st.session_state:
    with st.spinner("Thinking locally..."):
        st.write(answer(q))

Why this beats cloud RAG

Cloud (Mistral/OpenAI) Local (Ollama)
Privacy Data leaves machine 100% local
Cost Per-token billing Free
Speed Network latency Depends on hardware
Quality Top-tier models Good for 8B+ models
Offline No Yes

Common mistakes

  • Chunking by characters, not words — breaks mid-word. Split on words, then join.
  • No overlap — answers spanning two chunks get lost. Always overlap.
  • Re-ingesting every run — use PersistentClient + a session flag.
  • Tiny embedding model on big docs — nomic-embed-text (137M) is fine; for huge corpora use mxbai-embed-large.
  • Forgetting to pull models — ollama pull both embed and chat models before running.

Key Concepts

  • RAG = Retrieve relevant chunks → Augment prompt → Generate answer
  • Embeddings = vector representations of text for similarity search
  • Chunking + overlap = the secret to good retrieval
  • Persistent vector store = ingest once, query forever
  • Local LLM = privacy + zero cost, trades some quality

What to Try Next

  • Add source citations — store page numbers in Chroma metadata and show them.
  • Support multiple documents with per-doc filtering.
  • Swap llama3.2 for mistral or phi3 and compare answer quality.
  • Combine with prompt engineering techniques to tighten the answer prompt.
  • Deploy as a Cloudflare Pages app (embeddings run client-side? No — host the Ollama backend on a GPU box).

FAQ

How much RAM do I need?

llama3.2 (3B) runs on 8GB. llama3.1:8b needs 16GB. Embeddings are tiny. A modern laptop handles this fine.

Is local RAG as good as ChatGPT on my docs?

For factual Q&A over your own docs, often yes — retrieval grounds the answer. For reasoning-heavy tasks, cloud models still win. Hybrid: retrieve locally, generate with a cloud model if privacy allows.

Can I use this with the PDF chat tutorial?

Yes — this post is the privacy-first sibling. Same UX, different backend. Pick based on your data sensitivity.