Lesson 16 — What Is a Token in AI?

When we write a sentence, we see words. An AI model sees something slightly different.

Before processing text, many AI models break it into smaller pieces called tokens.

What Is a Token?

A token is a small piece of text that an AI model can process.

A token can be a complete word, part of a word, punctuation, or another small piece of text.

For example, a sentence such as “AI is useful.” can be split into several tokens.

Computers cannot process human language exactly as we see it.

Tokens provide a practical way for an AI model to convert text into information it can process mathematically.

One word does not necessarily equal one token.

A long or unusual word may be divided into several tokens, while a short word may sometimes be represented by a single token.

Tokens are important because AI models have limits on how much text they can process.

The more text you provide, the more tokens are used.

This is also why AI services sometimes describe limits using token counts rather than simply counting words.

Tokens are used not only when an AI reads your prompt, but also when it generates its answer.

The model predicts one token at a time and uses those tokens to build the final response.

Think of tokens like building blocks.

A sentence is built from many small blocks. The AI processes these blocks and uses them to understand and generate text.

Tokens help us understand how AI processes text. But there is another important limitation: how many tokens can an AI model consider at once.

In the next lesson, we will discover what a context window is.