Run AI Models Locally with Ollama (Free, No API Keys)

Introduction

Every AI project so far on this blog — the Mistral chatbot, the PDF chat app — depended on a cloud API: an API key, an internet connection, usage limits, and a subscription eventually. Ollama changes the equation completely: it runs full AI models on your own computer, free forever, offline, with no data leaving your machine.

The setup takes five minutes, and by the end you’ll have a Python-controlled local chatbot — the private twin of the cloud chatbot you may have already built.

What is Ollama?

Ollama is a free tool that downloads, manages, and runs open-source AI models locally. Think of it as “the app store for AI models”: one command downloads a model, another runs it, and it exposes a local web API your Python code can call.

Why run models locally instead of calling a cloud API?

Free forever — no API bills, no rate limits, no subscription
Private — your data never leaves your computer
Offline — works on a plane, in a hostel, in a village
Unlimited experiments — break it, retry, iterate without watching a meter

Prerequisites

Python 3.9+ — from python.org.
8 GB RAM minimum (16 GB comfortable) — models live in memory.
Ollama — one installer from ollama.com (Windows, macOS, Linux).
Python packages:

pip install requests streamlit

Step 1: Install Ollama and Pull a Model

Install Ollama from the website, then open a terminal:

ollama pull llama3.2        # downloads ~2GB — one time only
ollama run llama3.2         # chat with it RIGHT NOW in the terminal

That’s a working AI on your machine. Type /bye to exit. Useful commands:

ollama list                 # models you have
ollama pull gemma2:2b       # a smaller, faster model
ollama rm llama3.2          # free up disk space

Model picking, honestly: llama3.2 (3B) is the sweet default for 8 GB machines. gemma2:2b and phi3 run on less. Bigger models are smarter but slower and hungrier — start small.

Step 2: Talk to It from Python

Ollama runs a local API server at localhost:11434. Your Python code talks to it with plain requests:

import requests

def ask_ollama(prompt):
    response = requests.post(
        "http://localhost:11434/api/generate",
        json={
            "model": "llama3.2",
            "prompt": prompt,
            "stream": False,
        },
        timeout=120,
    )
    return response.json()["response"]

print(ask_ollama("Explain what an API is in one sentence for a 12-year-old"))

No API key. No billing. Just a POST to your own machine.

Step 3: Build the Local Chatbot

The chat pattern from the Mistral chatbot tutorial transfers directly — Ollama’s /api/chat endpoint accepts the same messages list:

import streamlit as st
import requests

st.set_page_config(page_title="Local AI Chat", page_icon="🦙")
st.title("🦙 Local AI Chat — 100% Offline")

if "messages" not in st.session_state:
    st.session_state.messages = []

# show history
for msg in st.session_state.messages:
    with st.chat_message(msg["role"]):
        st.write(msg["content"])

# user input
if prompt := st.chat_input("Ask anything..."):
    st.session_state.messages.append({"role": "user", "content": prompt})
    with st.chat_message("user"):
        st.write(prompt)

    with st.chat_message("assistant"):
        with st.spinner("Thinking locally..."):
            response = requests.post(
                "http://localhost:11434/api/chat",
                json={
                    "model": "llama3.2",
                    "messages": st.session_state.messages,
                    "stream": False,
                },
                timeout=120,
            )
            reply = response.json()["message"]["content"]
        st.write(reply)

    st.session_state.messages.append({"role": "assistant", "content": reply})

Run it:

streamlit run local_chat.py

Turn off your Wi-Fi and keep chatting — that’s the moment it clicks.

Common Errors & Fixes

Connection refused to localhost:11434 — the Ollama app isn’t running. Launch it (or run ollama serve), then retry.
model 'llama3.2' not found — you haven’t pulled it yet: ollama pull llama3.2.
Very slow responses — you’re on a big model with little RAM; switch to gemma2:2b, or close memory-hungry apps.
First response takes ages — the model is loading into memory (one time per session); later responses are much faster.

Key Concepts

Local inference — models run on your hardware; zero cost per message
Ollama as a local API — same request/response thinking as cloud APIs
Model sizes — parameter counts (2B, 3B, 8B) trade intelligence for speed
Portability of patterns — chat messages, streaming, and history transfer from cloud to local unchanged

What to Try Next

Add a model selector in the sidebar — switch between your installed models live.
Add streaming responses with Ollama’s "stream": true for typewriter-style output.
Point the PDF chat app’s generation step at Ollama — a fully local, fully free document assistant.
Explore vision models (llama3.2-vision) to describe images locally, like a private image classifier with words.

FAQ

Is local AI as smart as ChatGPT?

Small local models (2–3B) are noticeably weaker than frontier cloud models — best for drafting, summarizing, and chatting, not expert reasoning. Larger local models close the gap but need serious hardware. The trade is intelligence for privacy, cost, and control.

Can my laptop really run these?

If it has 8 GB RAM, the small models run fine on CPU. A gaming GPU makes everything much faster, but it’s optional for learning.

Where do I go from Ollama to real applications?

Exactly where the cloud-based tutorials on this site go — chatbots, RAG, summarizers — except your localhost:11434 replaces the paid API endpoint. The AI flashcard generator and review sentiment dashboard patterns both port over.