Introduction
The basic Mistral chatbot sends the full history every time and blocks while waiting. That works for demos. For a production chatbot, you need streaming, persistent memory, tool calling, and a clean UI.
This post upgrades the chatbot to production quality.
What we’re building
- Streaming responses (token-by-token)
- Persistent memory (survives page refresh)
- Tool calling (weather, search, DB queries)
- Clean chat UI with avatars and timestamps
- Error handling and retry
Step 1: Streaming with st.write_stream
import streamlit as st
from openai import OpenAI
client = OpenAI()
def stream_response(messages):
stream = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
stream=True,
)
for chunk in stream:
if chunk.choices[0].delta.content:
yield chunk.choices[0].delta.content
# UI
if prompt := st.chat_input("Ask anything"):
st.session_state.messages.append({"role": "user", "content": prompt})
with st.chat_message("user"):
st.write(prompt)
with st.chat_message("assistant"):
response = st.write_stream(stream_response(st.session_state.messages))
st.session_state.messages.append({"role": "assistant", "content": response})
st.write_stream consumes a generator and renders tokens as they arrive — the user sees the reply “typing” in real time.
Step 2: Persistent memory (survives refresh)
Streamlit session state resets on refresh. For persistence, use a database:
import sqlite3, json
from datetime import datetime
def save_message(session_id, role, content):
conn = sqlite3.connect("chat.db")
conn.execute("INSERT INTO messages (session_id, role, content, ts) VALUES (?,?,?,?)",
(session_id, role, content, datetime.now().isoformat()))
conn.commit()
conn.close()
def load_history(session_id):
conn = sqlite3.connect("chat.db")
rows = conn.execute("SELECT role, content FROM messages WHERE session_id=? ORDER BY ts", (session_id,)).fetchall()
conn.close()
return [{"role": r[0], "content": r[1]} for r in rows]
On app start, load history from DB instead of starting empty. Use a session ID from st.session_state or a cookie.
Step 3: Tool calling
import json
TOOLS = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current weather for a city",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
},
]
def get_weather(city):
# Real API call here
return f"22°C, sunny in {city}"
TOOL_MAP = {"get_weather": get_weather}
def chat_with_tools(messages):
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=messages,
tools=TOOLS,
)
msg = response.choices[0].message
if msg.tool_calls:
messages.append(msg)
for tc in msg.tool_calls:
result = TOOL_MAP[tc.function.name](**json.loads(tc.function.arguments))
messages.append({"role": "tool", "tool_call_id": tc.id, "content": result})
# Get final answer
final = client.chat.completions.create(model="gpt-4o-mini", messages=messages)
return final.choices[0].message.content
return msg.content
The model decides when to call tools; you execute them and feed results back. This is the agent loop in production form.
Step 4: Clean chat UI
import streamlit as st
st.set_page_config(page_title="Chat", page_icon="💬", layout="wide")
# Sidebar
with st.sidebar:
st.title("💬 Chat")
if st.button("New chat"):
st.session_state.messages = []
st.rerun()
st.divider()
st.caption("Model: gpt-4o-mini")
# Main
st.title("AI Assistant")
if "messages" not in st.session_state:
st.session_state.messages = []
for msg in st.session_state.messages:
with st.chat_message(msg["role"]):
st.write(msg["content"])
if prompt := st.chat_input("Type a message..."):
st.session_state.messages.append({"role": "user", "content": prompt})
with st.chat_message("user"):
st.write(prompt)
with st.chat_message("assistant"):
with st.spinner("Thinking..."):
reply = chat_with_tools(st.session_state.messages)
st.write(reply)
st.session_state.messages.append({"role": "assistant", "content": reply})
Step 5: Error handling and retry
import time
def safe_chat(messages, retries=3):
for attempt in range(retries):
try:
return chat_with_tools(messages)
except Exception as e:
if attempt < retries - 1:
time.sleep(2 ** attempt) # exponential backoff
else:
return f"Sorry, I hit an error: {e}"
Wrap every LLM call in retry logic — rate limits and transient errors are common.
Step 6: System prompt and persona
SYSTEM_PROMPT = """You are a helpful assistant for Tech With Galvan's readers.
- Keep answers concise (2-4 sentences unless asked for detail).
- Use markdown for code and lists.
- If you don't know, say so — don't hallucinate.
- When using tools, explain what you're doing briefly."""
messages = [{"role": "system", "content": SYSTEM_PROMPT}] + st.session_state.messages
The system prompt shapes the entire experience. Iterate on it based on user feedback.
Step 7: Memory management (context window)
LLMs have context limits. Trim old messages to stay within budget:
def trim_messages(messages, max_tokens=3000):
# Keep system prompt + last N messages
system = [m for m in messages if m["role"] == "system"]
others = [m for m in messages if m["role"] != "system"]
# Keep last ~10 messages (rough heuristic)
return system + others[-10:]
For production, use token counting (tiktoken) instead of message count.
Streaming + tools (combined)
For the best UX, stream the final answer after tool calls:
def stream_with_tools(messages):
# First call (may trigger tool use)
response = client.chat.completions.create(
model="gpt-4o-mini", messages=messages, tools=TOOLS, stream=False
)
msg = response.choices[0].message
if msg.tool_calls:
messages.append(msg)
for tc in msg.tool_calls:
result = TOOL_MAP[tc.function.name](**json.loads(tc.function.arguments))
messages.append({"role": "tool", "tool_call_id": tc.id, "content": result})
# Stream the final answer
stream = client.chat.completions.create(
model="gpt-4o-mini", messages=messages, stream=True
)
for chunk in stream:
if chunk.choices[0].delta.content:
yield chunk.choices[0].delta.content
else:
yield msg.content
Common mistakes
- No streaming — users stare at a spinner for 5s. Always stream.
- Unbounded history — context overflow causes errors. Trim regularly.
- No error handling — LLM APIs fail. Add retry with backoff.
- Tools without descriptions — the model won’t know when to use them. Write clear descriptions.
- Storing secrets in session state — API keys go in env vars /
st.secrets, never insession_state. - No system prompt — the model’s behavior is unpredictable without guidance.
Key Concepts
- Streaming = token-by-token rendering via
st.write_stream - Persistent memory = DB-backed message history
- Tool calling = model invokes functions, you execute them
- Context trimming = stay within token limits
- System prompt = shapes the assistant’s behavior
What to Try Next
- Add MCP tools so the chatbot can use external capabilities.
- Persist conversations with Supabase.
- Add auth for per-user chat history.
- Deploy with theming for a polished look.
FAQ
Streaming vs non-streaming — when to use which?
Always stream for user-facing chat. Non-streaming is fine for background tasks, batch processing, or when you need the full response before acting (e.g., tool calls need the complete message).
How do I handle tool calls with streaming?
Tool calls require the full response first (to get the tool call JSON). Stream only the final answer after tools are executed. See Step 7.
OpenAI vs Anthropic vs Mistral for chatbots?
OpenAI (GPT-4o): best tool calling, most reliable. Anthropic (Claude): great for long context, strong reasoning. Mistral: cheaper, open weights, good for cost-sensitive apps. All support streaming and tool calling. Pick based on budget and quality needs.