Introduction
Voice is the most natural interface — and in 2026, you can build a full voice agent in Python for free. Whisper transcribes speech, an LLM reasons, and gTTS/pyttsx3 speaks the reply. All in a Streamlit app, all runnable locally.
This post builds a complete voice agent: speak → transcribe → think → reply (text + voice).
Architecture
Microphone → Whisper (STT) → LLM (reasoning) → TTS → Speaker
↑________________memory________________↓
Each step is a swappable component. We’ll use local Whisper + Ollama for fully offline, and show the cloud swap for better quality.
Prerequisites
pip install streamlit openai-whisper sounddevice scipy ollama gtts
# For TTS playback
pip install pygame
You also need ffmpeg: brew install ffmpeg (macOS) or apt install ffmpeg.
Step 1: Record audio
import sounddevice as sd
import numpy as np
from scipy.io.wavfile import write
def record(seconds=5, fs=16000):
audio = sd.rec(int(seconds * fs), samplerate=fs, channels=1, dtype="float32")
sd.wait()
write("input.wav", fs, audio)
return "input.wav"
16kHz mono is what Whisper expects. 5 seconds is a good default for a turn.
Step 2: Transcribe with Whisper
import whisper
model = whisper.load_model("base") # tiny/base/small/medium/large
def transcribe(path):
result = model.transcribe(path)
return result["text"].strip()
base (74M) runs in ~1s on a modern laptop. small (244M) is more accurate but slower. Use medium+ only on GPU.
Step 3: Reason with an LLM
import ollama
SYSTEM = """You are a helpful voice assistant. Keep replies under 2 sentences
so they sound natural when spoken. Be conversational."""
def think(user_text, history):
messages = [{"role": "system", "content": SYSTEM}] + history
messages.append({"role": "user", "content": user_text})
reply = ollama.chat(model="llama3.2", messages=messages)["message"]["content"]
return reply
The “under 2 sentences” constraint is critical — spoken responses should be short. Long paragraphs sound robotic.
Step 4: Speak the reply (TTS)
from gtts import gTTS
import pygame, os
def speak(text):
tts = gTTS(text=text, lang="en")
tts.save("reply.mp3")
pygame.mixer.init()
pygame.mixer.music.load("reply.mp3")
pygame.mixer.music.play()
while pygame.mixer.music.get_busy():
pygame.time.wait(100)
For offline TTS, use pyttsx3 (system voices, no network). For higher quality, use ElevenLabs or OpenAI TTS (cloud).
Step 5: Streamlit UI with memory
import streamlit as st
st.title("🎙️ Voice Agent")
st.caption("Speak, think, reply — fully local.")
if "history" not in st.session_state:
st.session_state.history = []
if st.button("🎤 Speak (5s)"):
with st.spinner("Listening..."):
audio_path = record(5)
with st.spinner("Transcribing..."):
user_text = transcribe(audio_path)
st.write(f"**You:** {user_text}")
with st.spinner("Thinking..."):
reply = think(user_text, st.session_state.history)
st.write(f"**Agent:** {reply}")
st.session_state.history.append({"role": "user", "content": user_text})
st.session_state.history.append({"role": "assistant", "content": reply})
speak(reply)
Step 6: Add a text fallback
Voice fails in noisy rooms. Always offer a text input:
text_input = st.text_input("Or type:")
if text_input and st.button("Send text"):
reply = think(text_input, st.session_state.history)
st.write(f"**Agent:** {reply}")
st.session_state.history.append({"role": "user", "content": text_input})
st.session_state.history.append({"role": "assistant", "content": reply})
speak(reply)
Cloud upgrade (better quality)
Swap local components for cloud APIs when you need quality:
| Step | Local (free) | Cloud (better) |
|---|---|---|
| STT | Whisper (base) | OpenAI Whisper API / Deepgram |
| LLM | Ollama (llama3.2) | GPT-4o / Claude |
| TTS | gTTS / pyttsx3 | ElevenLabs / OpenAI TTS |
The architecture stays identical — just change the function bodies.
Common mistakes
- Long replies — spoken text should be 1-2 sentences. Constrain the LLM in the system prompt.
- No memory — without history, every turn is stateless. Pass the last 6-10 messages.
- Blocking the UI — Whisper + LLM + TTS can take 5-10s. Use
st.spinnerand consider streaming. - Wrong sample rate — Whisper wants 16kHz mono. Other rates cause garbage transcription.
- No text fallback — voice fails in noise; always offer typing.
Key Concepts
- STT = Speech-to-Text (Whisper)
- LLM = reasoning (Ollama / cloud)
- TTS = Text-to-Speech (gTTS / ElevenLabs)
- Memory = conversation history passed to the LLM
- Constraint prompting = “under 2 sentences” for natural speech
What to Try Next
- Add wake word detection (“Hey Galvan”) with
pvporcupine. - Add function calling so the agent can check weather, set reminders, etc. (see MCP).
- Stream the LLM response and speak sentence-by-sentence for lower latency.
- Deploy as a Streamlit app on Community Cloud (use cloud STT/TTS for the deployed version).
FAQ
Can this run fully offline?
Yes — Whisper (local), Ollama (local), and pyttsx3 (system TTS) need no network. gTTS needs internet; swap for pyttsx3 for full offline.
Whisper vs OpenAI Whisper API?
Local Whisper is free and private but slower and less accurate on the base model. The API is faster and uses large-v3 quality. For production, use the API; for learning/privacy, local.
How do I reduce latency?
Use whisper-tiny (fastest), a small LLM (phi3, llama3.2:1b), and stream TTS sentence-by-sentence. Target: <3s per turn.