Introduction
The basic RAG chatbot answers questions about documents. But a production chatbot needs more: conversation memory (remember previous turns), tool calling (search, calculate, fetch data), and streaming (token-by-token). This post builds all three with LangChain + Streamlit.
What you’ll build
- Streaming responses with
st.write_stream - Conversation memory (buffer + summary)
- Tool calling (weather, calculator, search)
- Source citations
- Clean chat UI with avatars
Setup
pip install streamlit langchain langchain-openai langchain-community
Step 1: Tools
from langchain.tools import tool
import requests, math
@tool
def get_weather(city: str) -> str:
"""Get current weather for a city."""
# Replace with real API
return f"22°C, sunny in {city}"
@tool
def calculate(expression: str) -> str:
"""Evaluate a math expression safely."""
try:
return str(eval(expression, {"__builtins__": {}}, {"math": math}))
except Exception as e:
return f"Error: {e}"
@tool
def search_web(query: str) -> str:
"""Search the web for information."""
# Replace with real search API (DuckDuckGo, Tavily, etc.)
return f"Search results for: {query}"
tools = [get_weather, calculate, search_web]
Step 2: LLM with tools
from langchain_openai import ChatOpenAI
from langchain.agents import create_tool_calling_agent, AgentExecutor
from langchain_core.prompts import ChatPromptTemplate
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0, streaming=True)
prompt = ChatPromptTemplate.from_messages([
("system", "You are a helpful assistant. Use tools when needed. Be concise."),
("placeholder", "{chat_history}"),
("human", "{input}"),
("placeholder", "{agent_scratchpad}"),
])
agent = create_tool_calling_agent(llm, tools, prompt)
executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
Step 3: Conversation memory
from langchain.memory import ConversationBufferWindowMemory
memory = ConversationBufferWindowMemory(
k=10, # keep last 10 turns
memory_key="chat_history",
return_messages=True,
output_key="output",
)
ConversationBufferWindowMemory keeps the last K turns. For long conversations, use ConversationSummaryMemory to compress older turns.
Step 4: Streaming with callbacks
from langchain.callbacks.base import BaseCallbackHandler
from queue import Queue
import threading
class StreamHandler(BaseCallbackHandler):
def __init__(self, queue):
self.queue = queue
def on_llm_new_token(self, token, **kwargs):
self.queue.put(token)
def on_llm_end(self, *args, **kwargs):
self.queue.put(None) # sentinel
def stream_response(executor, memory, user_input):
queue = Queue()
handler = StreamHandler(queue)
def run():
result = executor.invoke(
{"input": user_input, "chat_history": memory.chat_memory.messages},
config={"callbacks": [handler]},
)
memory.save_context({"input": user_input}, {"output": result["output"]})
queue.put(None)
thread = threading.Thread(target=run)
thread.start()
while True:
token = queue.get()
if token is None:
break
yield token
Step 5: Streamlit UI
import streamlit as st
st.title("🤖 LangChain Chatbot")
st.caption("Memory + Tools + Streaming")
if "messages" not in st.session_state:
st.session_state.messages = []
if "memory" not in st.session_state:
st.session_state.memory = ConversationBufferWindowMemory(
k=10, memory_key="chat_history", return_messages=True, output_key="output"
)
for msg in st.session_state.messages:
with st.chat_message(msg["role"]):
st.write(msg["content"])
if prompt := st.chat_input("Ask anything..."):
st.session_state.messages.append({"role": "user", "content": prompt})
with st.chat_message("user"):
st.write(prompt)
with st.chat_message("assistant"):
response = st.write_stream(stream_response(executor, st.session_state.memory, prompt))
st.session_state.messages.append({"role": "assistant", "content": response})
Step 6: Tool usage display
Show which tools were called:
with st.chat_message("assistant"):
with st.status("Thinking...", expanded=True) as status:
st.write("Calling tools...")
response = stream_response(executor, memory, prompt)
full = st.write_stream(response)
status.update(label="Done", state="complete", expanded=False)
st.write(full)
Step 7: Clear memory
with st.sidebar:
if st.button("Clear chat"):
st.session_state.messages = []
st.session_state.memory.clear()
st.rerun()
Step 8: Persist memory (database)
import sqlite3
def save_message(session_id, role, content):
conn = sqlite3.connect("chat.db")
conn.execute("INSERT INTO messages (session_id, role, content) VALUES (?,?,?)",
(session_id, role, content))
conn.commit()
conn.close()
def load_history(session_id):
conn = sqlite3.connect("chat.db")
rows = conn.execute("SELECT role, content FROM messages WHERE session_id=? ORDER BY id", (session_id,)).fetchall()
conn.close()
return [{"role": r[0], "content": r[1]} for r in rows]
Common mistakes
- No memory — every turn is stateless. Add
ConversationBufferWindowMemory. - Blocking streaming —
executor.invokeblocks. Use threading + queue for true streaming. - Tool descriptions too vague — the model won’t know when to use them. Write clear docstrings.
- Unbounded history — context overflow. Use window memory (k=10) or summary memory.
- No error handling — tool failures crash the agent. Wrap tools in try/except.
Key Concepts
- Agent = LLM + tools + prompt
- Tool calling = model decides when to call functions
- Memory = conversation history passed to the LLM
- Streaming = token-by-token output via callbacks
- AgentExecutor = runs the agent loop (think → tool → think → answer)
What to Try Next
- Add MCP tools for external capabilities.
- Persist memory with Supabase.
- Add RAG — combine tools with document retrieval.
- Deploy with Docker.
FAQ
LangChain agent vs OpenAI Assistants API?
LangChain = you control the loop, works with any model, more flexible. Assistants API = managed, less code, OpenAI-only. Use LangChain for control; Assistants API for speed.
How do I reduce latency?
Use gpt-4o-mini (fast), limit memory to k=5, and stream. For tool-heavy agents, the bottleneck is tool execution — parallelize independent tools.
Can I use local models?
Yes — swap ChatOpenAI for ChatOllama. Tool calling quality depends on the model; use 8B+ models for reliable tool use.