What Are Tokens? The Atoms of Python

What Are Tokens? The Atoms of Python

Every sentence in English is built from words. Every Python program is built from tokens — the smallest pieces the interpreter can recognize. Before you can write sentences, you need to know your words.

The five kinds of tokens

Take this single line of Python:

price = price + 10

The interpreter doesn’t see it as one blob — it splits it into tokens:

price    =    price    +    10

Five tokens, five jobs:

The five families:

1. Keywords — Python’s own reserved words (if, for, def, True…) — next lesson covers all of them
2. Identifiers — names you invent for variables, functions, classes
3. Literals — fixed values written directly: 10, "hello", 3.14, True
4. Operators — symbols that do work: + - * / = == < >
5. Delimiters — punctuation that structures code: ( ) , : . [ ]

Why should a beginner care?

Because every syntax error you’ll ever see is a token problem:

• pront("hi") → the identifier pront doesn’t exist (typo)
• print("hi" → a missing delimiter )
• if x = 5: → wrong operator; = assigns, == compares

When you read an error message, you’re really asking: “which token is wrong here?”

A peek behind the curtain

The interpreter’s first job is tokenization — chopping your file into tokens before understanding anything. You can watch it happen:

import tokenize

with open("hello.py", "rb") as f:
    for token in tokenize.tokenize(f.readline):
        print(token.string, "→", token.type)

Run that on any Python file and you’ll see the exact token stream the interpreter sees. It’s proof: your beautiful code is, to the machine, a river of tokens.

The one idea to carry forward

Python is strict about tokens. Misspell an identifier, use = where you meant ==, forget a colon — the interpreter stops and complains immediately. This feels annoying for a week. Then you realize it’s a superpower: Python catches your mistakes before they become bugs.

✅ Checkpoint

1. What are the five token families? (Keywords, identifiers, literals, operators, delimiters)
2. In age = 20, which tokens are which? (age = identifier, = = operator, 20 = literal)
3. Is "hello" a keyword, identifier, or literal? (A literal — a string value written directly)

Next: the most important token family — the 35 keywords Python reserved for itself.