Tech With Galvan
Back to Articles
AIMay 30, 2027 • 6 min read

RAG Chatbot with LlamaIndex + Streamlit (Better Retrieval)

Build a RAG chatbot with LlamaIndex and Streamlit — document loading, indexing, querying, and streaming. Better retrieval than LangChain for pure RAG.

Galvan
Galvan

Founder & Creator

Introduction

LangChain is great for agents and chains. But for pure RAG (retrieve + generate), LlamaIndex is often better: purpose-built for retrieval, with smarter chunking, better index types, and simpler APIs. This post builds a RAG chatbot with LlamaIndex + Streamlit.

LlamaIndex vs LangChain (for RAG)

LlamaIndex LangChain
Focus Retrieval + RAG General LLM apps (agents, chains, tools)
Chunking Smarter (sentence-aware, semantic) Basic (recursive character)
Index types Vector, tree, keyword, knowledge graph Vector (via integrations)
Query engine Built-in (with citations, reranking) Manual (retriever + chain)
Learning curve Lower (for RAG) Higher
Ecosystem Smaller Larger

Rule: Use LlamaIndex for RAG-focused apps. Use LangChain for agents and multi-step pipelines.

Setup

pip install streamlit llama-index llama-index-llms-openai llama-index-embeddings-openai

Step 1: Load and index documents

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

# Load documents from a directory
documents = SimpleDirectoryReader("./data").load_data()

# Create index (chunks + embeds + stores)
index = VectorStoreIndex.from_documents(documents)

SimpleDirectoryReader loads PDF, TXT, MD, DOCX, and more. VectorStoreIndex handles chunking, embedding, and storage in one call.

Step 2: Query the index

query_engine = index.as_query_engine()

response = query_engine.query("What is the main topic of the document?")
print(response.response)

# Source nodes (citations)
for node in response.source_nodes:
    print(f"Score: {node.score:.3f} | {node.text[:100]}...")

The query engine retrieves relevant chunks and generates an answer with citations.

Step 3: Chat engine (with memory)

chat_engine = index.as_chat_engine(chat_mode="condense_question")

response = chat_engine.chat("What is RAG?")
print(response.response)

# Follow-up (uses conversation history)
response = chat_engine.chat("How does it compare to fine-tuning?")
print(response.response)

condense_question mode reformulates follow-up questions using chat history for better retrieval.

Step 4: Streamlit UI

import streamlit as st
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

st.title("📚 Chat with Your Docs (LlamaIndex)")

uploaded = st.file_uploader("Upload PDF", type="pdf")

if uploaded and "index" not in st.session_state:
    with st.spinner("Indexing document..."):
        import tempfile, os
        with tempfile.NamedTemporaryFile(delete=False, suffix=".pdf") as f:
            f.write(uploaded.read())
            path = f.name

        documents = SimpleDirectoryReader(input_files=[path]).load_data()
        st.session_state.index = VectorStoreIndex.from_documents(documents)
        st.session_state.chat_engine = st.session_state.index.as_chat_engine(
            chat_mode="condense_question"
        )
        st.session_state.messages = []
    st.success(f"Indexed {len(documents)} documents")

if "messages" not in st.session_state:
    st.session_state.messages = []

for msg in st.session_state.messages:
    with st.chat_message(msg["role"]):
        st.write(msg["content"])

if prompt := st.chat_input("Ask a question"):
    st.session_state.messages.append({"role": "user", "content": prompt})
    with st.chat_message("user"):
        st.write(prompt)

    with st.chat_message("assistant"):
        response = st.session_state.chat_engine.chat(prompt)
        st.write(response.response)
        with st.expander("Sources"):
            for node in response.source_nodes:
                st.write(f"**Score: {node.score:.3f}**")
                st.write(node.text[:300])

    st.session_state.messages.append({"role": "assistant", "content": response.response})

Step 5: Streaming responses

from llama_index.core.llms import ChatMessage

def stream_chat(chat_engine, message):
    response = chat_engine.stream_chat(message)
    for token in response.response_gen:
        yield token

# In Streamlit:
with st.chat_message("assistant"):
    response = st.write_stream(stream_chat(chat_engine, prompt))

Step 6: Custom chunking

from llama_index.core.node_parser import SentenceSplitter

splitter = SentenceSplitter(
    chunk_size=512,
    chunk_overlap=50,
    separator=" ",
)

index = VectorStoreIndex.from_documents(documents, transformations=[splitter])

SentenceSplitter splits on sentence boundaries — better than character-based splitting.

Step 7: Local/private version (Ollama)

from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding

llm = Ollama(model="llama3.2", request_timeout=120)
embed_model = OllamaEmbedding(model_name="nomic-embed-text")

index = VectorStoreIndex.from_documents(documents, embed_model=embed_model)
chat_engine = index.as_chat_engine(llm=llm, chat_mode="condense_question")

100% local, no API keys. See RAG with local LLMs.

Step 8: Persist the index

# Save
index.storage_context.persist(persist_dir="./storage")

# Load
from llama_index.core import StorageContext, load_index_from_storage
storage_context = StorageContext.from_defaults(persist_dir="./storage")
index = load_index_from_storage(storage_context)

Don’t re-embed on every run — persist and load.

Step 9: Advanced retrieval (reranking)

from llama_index.core.postprocessor import SentenceTransformerRerank

reranker = SentenceTransformerRerank(top_n=3, model="cross-encoder/ms-marco-MiniLM-L-6-v2")

query_engine = index.as_query_engine(
    similarity_top_k=10,       # retrieve 10
    node_postprocessors=[reranker],  # rerank to top 3
)

Retrieve more, then rerank for better precision. Cross-encoders are much more accurate than bi-encoder similarity.

LlamaIndex vs LangChain (recap)

Feature LlamaIndex LangChain
RAG quality Better (smarter chunking, reranking) Good
Agents Basic Excellent
Tools Limited Extensive
Learning curve Lower (for RAG) Higher
Ecosystem Smaller Larger

Rule: LlamaIndex for RAG. LangChain for agents. Both can be combined.

Common mistakes

  • Not persisting the index — re-embedding on every run is slow and expensive.
  • Default chunk size too large — 1024 tokens is often too big. Try 512.
  • No reranking — bi-encoder similarity misses nuances. Add a cross-encoder reranker.
  • Ignoring source nodes — always show citations so users can verify.
  • Not handling empty results — if no relevant chunks found, the LLM hallucinates. Add a threshold.

Key Concepts

  • LlamaIndex = RAG-focused framework (load, index, query, chat)
  • VectorStoreIndex = chunks + embeds + stores in one call
  • Query engine = retrieves + generates answer
  • Chat engine = query engine + conversation memory
  • Reranker = improves retrieval precision
  • Persistence = save/load index to disk

What to Try Next

  • Compare LlamaIndex vs LangChain RAG quality on your documents.
  • Add auth for per-user document stores.
  • Persist with Pinecone for production scale.
  • Combine with MCP for external data sources.

FAQ

LlamaIndex vs LangChain — which for RAG?

LlamaIndex for pure RAG (better retrieval, simpler API). LangChain for RAG + agents + tools. If your app is just “chat with docs,” use LlamaIndex. If you need agents, use LangChain (or both).

Is LlamaIndex production-ready?

Yes — used by many companies. The API is stable (v0.10+). For production, use persistent vector stores (Pinecone, Weaviate) and add reranking.

Can I use LlamaIndex with local models?

Yes — use Ollama for LLM and OllamaEmbedding for embeddings. 100% local, no API keys. Quality depends on the model.<|end|><|start|>write to=assistant<|channel|>analysis<|message|>Created file successfully: src/content/posts/rag-chatbot-llamaindex-streamlit.md