Tech With Galvan
Back to Articles
AIMay 17, 2026 • 6 min read

Build a Voice Agent: Whisper + LLM + Streamlit (Speak, Think, Reply)

Build a voice agent in Python — transcribe speech with Whisper, reason with an LLM, and reply with text-to-speech, all in a Streamlit app. Free and local.

Galvan
Galvan

Founder & Creator

Introduction

Voice is the most natural interface — and in 2026, you can build a full voice agent in Python for free. Whisper transcribes speech, an LLM reasons, and gTTS/pyttsx3 speaks the reply. All in a Streamlit app, all runnable locally.

This post builds a complete voice agent: speak → transcribe → think → reply (text + voice).

Architecture

Microphone → Whisper (STT) → LLM (reasoning) → TTS → Speaker
                ↑________________memory________________↓

Each step is a swappable component. We’ll use local Whisper + Ollama for fully offline, and show the cloud swap for better quality.

Prerequisites

pip install streamlit openai-whisper sounddevice scipy ollama gtts
# For TTS playback
pip install pygame

You also need ffmpeg: brew install ffmpeg (macOS) or apt install ffmpeg.

Step 1: Record audio

import sounddevice as sd
import numpy as np
from scipy.io.wavfile import write

def record(seconds=5, fs=16000):
    audio = sd.rec(int(seconds * fs), samplerate=fs, channels=1, dtype="float32")
    sd.wait()
    write("input.wav", fs, audio)
    return "input.wav"

16kHz mono is what Whisper expects. 5 seconds is a good default for a turn.

Step 2: Transcribe with Whisper

import whisper

model = whisper.load_model("base")   # tiny/base/small/medium/large

def transcribe(path):
    result = model.transcribe(path)
    return result["text"].strip()

base (74M) runs in ~1s on a modern laptop. small (244M) is more accurate but slower. Use medium+ only on GPU.

Step 3: Reason with an LLM

import ollama

SYSTEM = """You are a helpful voice assistant. Keep replies under 2 sentences
so they sound natural when spoken. Be conversational."""

def think(user_text, history):
    messages = [{"role": "system", "content": SYSTEM}] + history
    messages.append({"role": "user", "content": user_text})
    reply = ollama.chat(model="llama3.2", messages=messages)["message"]["content"]
    return reply

The “under 2 sentences” constraint is critical — spoken responses should be short. Long paragraphs sound robotic.

Step 4: Speak the reply (TTS)

from gtts import gTTS
import pygame, os

def speak(text):
    tts = gTTS(text=text, lang="en")
    tts.save("reply.mp3")
    pygame.mixer.init()
    pygame.mixer.music.load("reply.mp3")
    pygame.mixer.music.play()
    while pygame.mixer.music.get_busy():
        pygame.time.wait(100)

For offline TTS, use pyttsx3 (system voices, no network). For higher quality, use ElevenLabs or OpenAI TTS (cloud).

Step 5: Streamlit UI with memory

import streamlit as st

st.title("🎙️ Voice Agent")
st.caption("Speak, think, reply — fully local.")

if "history" not in st.session_state:
    st.session_state.history = []

if st.button("🎤 Speak (5s)"):
    with st.spinner("Listening..."):
        audio_path = record(5)
    with st.spinner("Transcribing..."):
        user_text = transcribe(audio_path)
    st.write(f"**You:** {user_text}")

    with st.spinner("Thinking..."):
        reply = think(user_text, st.session_state.history)
    st.write(f"**Agent:** {reply}")

    st.session_state.history.append({"role": "user", "content": user_text})
    st.session_state.history.append({"role": "assistant", "content": reply})

    speak(reply)

Step 6: Add a text fallback

Voice fails in noisy rooms. Always offer a text input:

text_input = st.text_input("Or type:")
if text_input and st.button("Send text"):
    reply = think(text_input, st.session_state.history)
    st.write(f"**Agent:** {reply}")
    st.session_state.history.append({"role": "user", "content": text_input})
    st.session_state.history.append({"role": "assistant", "content": reply})
    speak(reply)

Cloud upgrade (better quality)

Swap local components for cloud APIs when you need quality:

Step Local (free) Cloud (better)
STT Whisper (base) OpenAI Whisper API / Deepgram
LLM Ollama (llama3.2) GPT-4o / Claude
TTS gTTS / pyttsx3 ElevenLabs / OpenAI TTS

The architecture stays identical — just change the function bodies.

Common mistakes

  • Long replies — spoken text should be 1-2 sentences. Constrain the LLM in the system prompt.
  • No memory — without history, every turn is stateless. Pass the last 6-10 messages.
  • Blocking the UI — Whisper + LLM + TTS can take 5-10s. Use st.spinner and consider streaming.
  • Wrong sample rate — Whisper wants 16kHz mono. Other rates cause garbage transcription.
  • No text fallback — voice fails in noise; always offer typing.

Key Concepts

  • STT = Speech-to-Text (Whisper)
  • LLM = reasoning (Ollama / cloud)
  • TTS = Text-to-Speech (gTTS / ElevenLabs)
  • Memory = conversation history passed to the LLM
  • Constraint prompting = “under 2 sentences” for natural speech

What to Try Next

  • Add wake word detection (“Hey Galvan”) with pvporcupine.
  • Add function calling so the agent can check weather, set reminders, etc. (see MCP).
  • Stream the LLM response and speak sentence-by-sentence for lower latency.
  • Deploy as a Streamlit app on Community Cloud (use cloud STT/TTS for the deployed version).

FAQ

Can this run fully offline?

Yes — Whisper (local), Ollama (local), and pyttsx3 (system TTS) need no network. gTTS needs internet; swap for pyttsx3 for full offline.

Whisper vs OpenAI Whisper API?

Local Whisper is free and private but slower and less accurate on the base model. The API is faster and uses large-v3 quality. For production, use the API; for learning/privacy, local.

How do I reduce latency?

Use whisper-tiny (fastest), a small LLM (phi3, llama3.2:1b), and stream TTS sentence-by-sentence. Target: <3s per turn.