Introduction
LangChain is great for agents and chains. But for pure RAG (retrieve + generate), LlamaIndex is often better: purpose-built for retrieval, with smarter chunking, better index types, and simpler APIs. This post builds a RAG chatbot with LlamaIndex + Streamlit.
LlamaIndex vs LangChain (for RAG)
| LlamaIndex | LangChain | |
|---|---|---|
| Focus | Retrieval + RAG | General LLM apps (agents, chains, tools) |
| Chunking | Smarter (sentence-aware, semantic) | Basic (recursive character) |
| Index types | Vector, tree, keyword, knowledge graph | Vector (via integrations) |
| Query engine | Built-in (with citations, reranking) | Manual (retriever + chain) |
| Learning curve | Lower (for RAG) | Higher |
| Ecosystem | Smaller | Larger |
Rule: Use LlamaIndex for RAG-focused apps. Use LangChain for agents and multi-step pipelines.
Setup
pip install streamlit llama-index llama-index-llms-openai llama-index-embeddings-openai
Step 1: Load and index documents
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
# Load documents from a directory
documents = SimpleDirectoryReader("./data").load_data()
# Create index (chunks + embeds + stores)
index = VectorStoreIndex.from_documents(documents)
SimpleDirectoryReader loads PDF, TXT, MD, DOCX, and more. VectorStoreIndex handles chunking, embedding, and storage in one call.
Step 2: Query the index
query_engine = index.as_query_engine()
response = query_engine.query("What is the main topic of the document?")
print(response.response)
# Source nodes (citations)
for node in response.source_nodes:
print(f"Score: {node.score:.3f} | {node.text[:100]}...")
The query engine retrieves relevant chunks and generates an answer with citations.
Step 3: Chat engine (with memory)
chat_engine = index.as_chat_engine(chat_mode="condense_question")
response = chat_engine.chat("What is RAG?")
print(response.response)
# Follow-up (uses conversation history)
response = chat_engine.chat("How does it compare to fine-tuning?")
print(response.response)
condense_question mode reformulates follow-up questions using chat history for better retrieval.
Step 4: Streamlit UI
import streamlit as st
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
st.title("📚 Chat with Your Docs (LlamaIndex)")
uploaded = st.file_uploader("Upload PDF", type="pdf")
if uploaded and "index" not in st.session_state:
with st.spinner("Indexing document..."):
import tempfile, os
with tempfile.NamedTemporaryFile(delete=False, suffix=".pdf") as f:
f.write(uploaded.read())
path = f.name
documents = SimpleDirectoryReader(input_files=[path]).load_data()
st.session_state.index = VectorStoreIndex.from_documents(documents)
st.session_state.chat_engine = st.session_state.index.as_chat_engine(
chat_mode="condense_question"
)
st.session_state.messages = []
st.success(f"Indexed {len(documents)} documents")
if "messages" not in st.session_state:
st.session_state.messages = []
for msg in st.session_state.messages:
with st.chat_message(msg["role"]):
st.write(msg["content"])
if prompt := st.chat_input("Ask a question"):
st.session_state.messages.append({"role": "user", "content": prompt})
with st.chat_message("user"):
st.write(prompt)
with st.chat_message("assistant"):
response = st.session_state.chat_engine.chat(prompt)
st.write(response.response)
with st.expander("Sources"):
for node in response.source_nodes:
st.write(f"**Score: {node.score:.3f}**")
st.write(node.text[:300])
st.session_state.messages.append({"role": "assistant", "content": response.response})
Step 5: Streaming responses
from llama_index.core.llms import ChatMessage
def stream_chat(chat_engine, message):
response = chat_engine.stream_chat(message)
for token in response.response_gen:
yield token
# In Streamlit:
with st.chat_message("assistant"):
response = st.write_stream(stream_chat(chat_engine, prompt))
Step 6: Custom chunking
from llama_index.core.node_parser import SentenceSplitter
splitter = SentenceSplitter(
chunk_size=512,
chunk_overlap=50,
separator=" ",
)
index = VectorStoreIndex.from_documents(documents, transformations=[splitter])
SentenceSplitter splits on sentence boundaries — better than character-based splitting.
Step 7: Local/private version (Ollama)
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding
llm = Ollama(model="llama3.2", request_timeout=120)
embed_model = OllamaEmbedding(model_name="nomic-embed-text")
index = VectorStoreIndex.from_documents(documents, embed_model=embed_model)
chat_engine = index.as_chat_engine(llm=llm, chat_mode="condense_question")
100% local, no API keys. See RAG with local LLMs.
Step 8: Persist the index
# Save
index.storage_context.persist(persist_dir="./storage")
# Load
from llama_index.core import StorageContext, load_index_from_storage
storage_context = StorageContext.from_defaults(persist_dir="./storage")
index = load_index_from_storage(storage_context)
Don’t re-embed on every run — persist and load.
Step 9: Advanced retrieval (reranking)
from llama_index.core.postprocessor import SentenceTransformerRerank
reranker = SentenceTransformerRerank(top_n=3, model="cross-encoder/ms-marco-MiniLM-L-6-v2")
query_engine = index.as_query_engine(
similarity_top_k=10, # retrieve 10
node_postprocessors=[reranker], # rerank to top 3
)
Retrieve more, then rerank for better precision. Cross-encoders are much more accurate than bi-encoder similarity.
LlamaIndex vs LangChain (recap)
| Feature | LlamaIndex | LangChain |
|---|---|---|
| RAG quality | Better (smarter chunking, reranking) | Good |
| Agents | Basic | Excellent |
| Tools | Limited | Extensive |
| Learning curve | Lower (for RAG) | Higher |
| Ecosystem | Smaller | Larger |
Rule: LlamaIndex for RAG. LangChain for agents. Both can be combined.
Common mistakes
- Not persisting the index — re-embedding on every run is slow and expensive.
- Default chunk size too large — 1024 tokens is often too big. Try 512.
- No reranking — bi-encoder similarity misses nuances. Add a cross-encoder reranker.
- Ignoring source nodes — always show citations so users can verify.
- Not handling empty results — if no relevant chunks found, the LLM hallucinates. Add a threshold.
Key Concepts
- LlamaIndex = RAG-focused framework (load, index, query, chat)
- VectorStoreIndex = chunks + embeds + stores in one call
- Query engine = retrieves + generates answer
- Chat engine = query engine + conversation memory
- Reranker = improves retrieval precision
- Persistence = save/load index to disk
What to Try Next
- Compare LlamaIndex vs LangChain RAG quality on your documents.
- Add auth for per-user document stores.
- Persist with Pinecone for production scale.
- Combine with MCP for external data sources.
FAQ
LlamaIndex vs LangChain — which for RAG?
LlamaIndex for pure RAG (better retrieval, simpler API). LangChain for RAG + agents + tools. If your app is just “chat with docs,” use LlamaIndex. If you need agents, use LangChain (or both).
Is LlamaIndex production-ready?
Yes — used by many companies. The API is stable (v0.10+). For production, use persistent vector stores (Pinecone, Weaviate) and add reranking.
Can I use LlamaIndex with local models?
Yes — use Ollama for LLM and OllamaEmbedding for embeddings. 100% local, no API keys. Quality depends on the model.<|end|><|start|>write to=assistant<|channel|>analysis<|message|>Created file successfully: src/content/posts/rag-chatbot-llamaindex-streamlit.md