Tech With Galvan
Back to Articles
AISep 27, 2026 • 6 min read

Advanced Streamlit Chatbot: Memory, Tools, and Streaming

Build an advanced Streamlit chatbot with persistent memory, tool calling, streaming responses, and a clean chat UI — production-ready patterns.

Galvan
Galvan

Founder & Creator

Introduction

The basic Mistral chatbot sends the full history every time and blocks while waiting. That works for demos. For a production chatbot, you need streaming, persistent memory, tool calling, and a clean UI.

This post upgrades the chatbot to production quality.

What we’re building

  • Streaming responses (token-by-token)
  • Persistent memory (survives page refresh)
  • Tool calling (weather, search, DB queries)
  • Clean chat UI with avatars and timestamps
  • Error handling and retry

Step 1: Streaming with st.write_stream

import streamlit as st
from openai import OpenAI

client = OpenAI()

def stream_response(messages):
    stream = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=messages,
        stream=True,
    )
    for chunk in stream:
        if chunk.choices[0].delta.content:
            yield chunk.choices[0].delta.content

# UI
if prompt := st.chat_input("Ask anything"):
    st.session_state.messages.append({"role": "user", "content": prompt})
    with st.chat_message("user"):
        st.write(prompt)

    with st.chat_message("assistant"):
        response = st.write_stream(stream_response(st.session_state.messages))
    st.session_state.messages.append({"role": "assistant", "content": response})

st.write_stream consumes a generator and renders tokens as they arrive — the user sees the reply “typing” in real time.

Step 2: Persistent memory (survives refresh)

Streamlit session state resets on refresh. For persistence, use a database:

import sqlite3, json
from datetime import datetime

def save_message(session_id, role, content):
    conn = sqlite3.connect("chat.db")
    conn.execute("INSERT INTO messages (session_id, role, content, ts) VALUES (?,?,?,?)",
                 (session_id, role, content, datetime.now().isoformat()))
    conn.commit()
    conn.close()

def load_history(session_id):
    conn = sqlite3.connect("chat.db")
    rows = conn.execute("SELECT role, content FROM messages WHERE session_id=? ORDER BY ts", (session_id,)).fetchall()
    conn.close()
    return [{"role": r[0], "content": r[1]} for r in rows]

On app start, load history from DB instead of starting empty. Use a session ID from st.session_state or a cookie.

Step 3: Tool calling

import json

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {"city": {"type": "string"}},
                "required": ["city"],
            },
        },
    },
]

def get_weather(city):
    # Real API call here
    return f"22°C, sunny in {city}"

TOOL_MAP = {"get_weather": get_weather}

def chat_with_tools(messages):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=messages,
        tools=TOOLS,
    )
    msg = response.choices[0].message

    if msg.tool_calls:
        messages.append(msg)
        for tc in msg.tool_calls:
            result = TOOL_MAP[tc.function.name](**json.loads(tc.function.arguments))
            messages.append({"role": "tool", "tool_call_id": tc.id, "content": result})

        # Get final answer
        final = client.chat.completions.create(model="gpt-4o-mini", messages=messages)
        return final.choices[0].message.content
    return msg.content

The model decides when to call tools; you execute them and feed results back. This is the agent loop in production form.

Step 4: Clean chat UI

import streamlit as st

st.set_page_config(page_title="Chat", page_icon="💬", layout="wide")

# Sidebar
with st.sidebar:
    st.title("💬 Chat")
    if st.button("New chat"):
        st.session_state.messages = []
        st.rerun()
    st.divider()
    st.caption("Model: gpt-4o-mini")

# Main
st.title("AI Assistant")

if "messages" not in st.session_state:
    st.session_state.messages = []

for msg in st.session_state.messages:
    with st.chat_message(msg["role"]):
        st.write(msg["content"])

if prompt := st.chat_input("Type a message..."):
    st.session_state.messages.append({"role": "user", "content": prompt})
    with st.chat_message("user"):
        st.write(prompt)

    with st.chat_message("assistant"):
        with st.spinner("Thinking..."):
            reply = chat_with_tools(st.session_state.messages)
        st.write(reply)
    st.session_state.messages.append({"role": "assistant", "content": reply})

Step 5: Error handling and retry

import time

def safe_chat(messages, retries=3):
    for attempt in range(retries):
        try:
            return chat_with_tools(messages)
        except Exception as e:
            if attempt < retries - 1:
                time.sleep(2 ** attempt)   # exponential backoff
            else:
                return f"Sorry, I hit an error: {e}"

Wrap every LLM call in retry logic — rate limits and transient errors are common.

Step 6: System prompt and persona

SYSTEM_PROMPT = """You are a helpful assistant for Tech With Galvan's readers.
- Keep answers concise (2-4 sentences unless asked for detail).
- Use markdown for code and lists.
- If you don't know, say so — don't hallucinate.
- When using tools, explain what you're doing briefly."""

messages = [{"role": "system", "content": SYSTEM_PROMPT}] + st.session_state.messages

The system prompt shapes the entire experience. Iterate on it based on user feedback.

Step 7: Memory management (context window)

LLMs have context limits. Trim old messages to stay within budget:

def trim_messages(messages, max_tokens=3000):
    # Keep system prompt + last N messages
    system = [m for m in messages if m["role"] == "system"]
    others = [m for m in messages if m["role"] != "system"]
    # Keep last ~10 messages (rough heuristic)
    return system + others[-10:]

For production, use token counting (tiktoken) instead of message count.

Streaming + tools (combined)

For the best UX, stream the final answer after tool calls:

def stream_with_tools(messages):
    # First call (may trigger tool use)
    response = client.chat.completions.create(
        model="gpt-4o-mini", messages=messages, tools=TOOLS, stream=False
    )
    msg = response.choices[0].message

    if msg.tool_calls:
        messages.append(msg)
        for tc in msg.tool_calls:
            result = TOOL_MAP[tc.function.name](**json.loads(tc.function.arguments))
            messages.append({"role": "tool", "tool_call_id": tc.id, "content": result})

        # Stream the final answer
        stream = client.chat.completions.create(
            model="gpt-4o-mini", messages=messages, stream=True
        )
        for chunk in stream:
            if chunk.choices[0].delta.content:
                yield chunk.choices[0].delta.content
    else:
        yield msg.content

Common mistakes

  • No streaming — users stare at a spinner for 5s. Always stream.
  • Unbounded history — context overflow causes errors. Trim regularly.
  • No error handling — LLM APIs fail. Add retry with backoff.
  • Tools without descriptions — the model won’t know when to use them. Write clear descriptions.
  • Storing secrets in session state — API keys go in env vars / st.secrets, never in session_state.
  • No system prompt — the model’s behavior is unpredictable without guidance.

Key Concepts

  • Streaming = token-by-token rendering via st.write_stream
  • Persistent memory = DB-backed message history
  • Tool calling = model invokes functions, you execute them
  • Context trimming = stay within token limits
  • System prompt = shapes the assistant’s behavior

What to Try Next

  • Add MCP tools so the chatbot can use external capabilities.
  • Persist conversations with Supabase.
  • Add auth for per-user chat history.
  • Deploy with theming for a polished look.

FAQ

Streaming vs non-streaming — when to use which?

Always stream for user-facing chat. Non-streaming is fine for background tasks, batch processing, or when you need the full response before acting (e.g., tool calls need the complete message).

How do I handle tool calls with streaming?

Tool calls require the full response first (to get the tool call JSON). Stream only the final answer after tools are executed. See Step 7.

OpenAI vs Anthropic vs Mistral for chatbots?

OpenAI (GPT-4o): best tool calling, most reliable. Anthropic (Claude): great for long context, strong reasoning. Mistral: cheaper, open weights, good for cost-sensitive apps. All support streaming and tool calling. Pick based on budget and quality needs.