Introduction
Chat with PDF used a cloud API with manual chunking. LangChain makes RAG easier: document loaders, text splitters, vector stores, and retrieval chains are all built in. This post builds a RAG chatbot in Streamlit with LangChain — load docs, embed, retrieve, and chat, with streaming and source citations.
What you’ll build
- Upload PDFs/TXT/MD
- Auto-chunk and embed with OpenAI (or local Ollama)
- Chat with your documents
- Stream responses with source citations
Setup
pip install streamlit langchain langchain-openai langchain-community chromadb pypdf
Set OPENAI_API_KEY in env or .streamlit/secrets.toml.
Step 1: Load and chunk documents
from langchain_community.document_loaders import PyPDFLoader, TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
def load_and_split(uploaded_file):
# Save to temp file
import tempfile, os
with tempfile.NamedTemporaryFile(delete=False, suffix=".pdf") as f:
f.write(uploaded_file.read())
path = f.name
loader = PyPDFLoader(path)
docs = loader.load()
splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200,
)
return splitter.split_documents(docs)
RecursiveCharacterTextSplitter tries to split on paragraphs, then sentences, then words — keeping chunks coherent.
Step 2: Create vector store
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
def create_vectorstore(chunks):
embeddings = OpenAIEmbeddings()
return Chroma.from_documents(chunks, embeddings)
For local/private (no API): use OllamaEmbeddings + local Chroma. See RAG with local LLMs.
Step 3: Create retrieval chain
from langchain_openai import ChatOpenAI
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.prompts import ChatPromptTemplate
def create_chain(vectorstore):
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})
prompt = ChatPromptTemplate.from_template("""
Answer the question based only on the following context.
If you don't know, say "I don't know based on the provided documents."
Context: {context}
Question: {input}
""")
document_chain = create_stuff_documents_chain(llm, prompt)
return create_retrieval_chain(retriever, document_chain)
Step 4: Streamlit UI with streaming
import streamlit as st
st.title("📚 Chat with Your Docs")
uploaded = st.file_uploader("Upload PDF", type="pdf")
if uploaded and "chain" not in st.session_state:
with st.spinner("Processing document..."):
chunks = load_and_split(uploaded)
vs = create_vectorstore(chunks)
st.session_state.chain = create_chain(vs)
st.session_state.messages = []
st.success(f"Processed {len(chunks)} chunks")
if "messages" not in st.session_state:
st.session_state.messages = []
for msg in st.session_state.messages:
with st.chat_message(msg["role"]):
st.write(msg["content"])
if msg.get("sources"):
with st.expander("Sources"):
for s in msg["sources"]:
st.write(s)
if prompt := st.chat_input("Ask a question"):
st.session_state.messages.append({"role": "user", "content": prompt})
with st.chat_message("user"):
st.write(prompt)
with st.chat_message("assistant"):
# Stream the response
response = st.session_state.chain.invoke({"input": prompt})
answer = response["answer"]
sources = [d.page_content[:200] for d in response["context"]]
st.write(answer)
if sources:
with st.expander("Sources"):
for s in sources:
st.write(s)
st.session_state.messages.append({
"role": "assistant", "content": answer, "sources": sources
})
Step 5: Streaming responses
For token-by-token streaming, use LangChain’s stream:
def stream_answer(chain, question):
for chunk in chain.stream({"input": question}):
if "answer" in chunk:
yield chunk["answer"]
Then in Streamlit:
with st.chat_message("assistant"):
response = st.write_stream(stream_answer(chain, prompt))
Step 6: Multiple documents
uploaded_files = st.file_uploader("Upload PDFs", type="pdf", accept_multiple_files=True)
if uploaded_files:
all_chunks = []
for f in uploaded_files:
all_chunks.extend(load_and_split(f))
vs = create_vectorstore(all_chunks)
st.session_state.chain = create_chain(vs)
Step 7: Conversation memory
from langchain.memory import ConversationBufferMemory
memory = ConversationBufferMemory(
memory_key="chat_history",
return_messages=True,
output_key="answer",
)
# Add memory to the chain
chain = create_retrieval_chain(retriever, document_chain)
# Use memory in the prompt template
For production, use ConversationSummaryMemory to compress long histories.
Step 8: Local/private version (Ollama)
Swap OpenAI for local models:
from langchain_community.embeddings import OllamaEmbeddings
from langchain_community.llms import Ollama
embeddings = OllamaEmbeddings(model="nomic-embed-text")
llm = Ollama(model="llama3.2")
100% local, no API keys. Quality depends on the model.
Common mistakes
- Chunk size too big/small — 1000 chars with 200 overlap is a good default. Tune per document type.
- Not showing sources — users need to verify answers. Always show retrieved chunks.
- No memory management — long conversations overflow context. Trim or summarize.
- Ignoring retrieval quality — if retrieval is bad, the LLM can’t answer well. Test retrieval separately.
- Re-embedding on every query — embed once, store in Chroma, reuse. Don’t re-embed.
Key Concepts
- RAG = Retrieve relevant chunks → Augment prompt → Generate answer
- LangChain = framework for LLM apps (loaders, splitters, chains)
- Retriever = finds relevant chunks from vector store
- Stuff documents chain = puts retrieved docs into the prompt
- Streaming = token-by-token output for responsive UI
What to Try Next
- Add auth for per-user document stores.
- Persist vector stores with Supabase (pgvector).
- Use Hugging Face embeddings instead of OpenAI.
- Add MCP tools so the chatbot can use external data.
FAQ
LangChain vs building RAG from scratch?
LangChain = faster to build, good defaults, but abstracts away details. From scratch = full control, fewer dependencies, easier to debug. Start with LangChain; go custom when you need control.
LangChain vs LlamaIndex?
LlamaIndex is RAG-focused (better retrieval, more data connectors). LangChain is more general (agents, tools, chains). For pure RAG, LlamaIndex is often better. For agents + RAG, LangChain.
Is LangChain production-ready?
Yes, but it changes fast (breaking changes between versions). Pin your version. For production, consider LangChain’s langserve or building a custom pipeline. The concepts (load, split, embed, retrieve, generate) are stable even if the API changes.