A model does not read words. It reads numbers. Understanding the translation explains most of the strange behaviour you will run into — why it miscounts letters, why some words cost more than others, and why it seems to grasp meaning at all.
In module 01 we said a language model predicts what comes next. That was true but incomplete, because it skipped a step. Before any prediction happens, your sentence has to become something a machine can do arithmetic on.
There are two translations, and they do different jobs. The first chops your text into pieces and gives each piece a number. The second turns each number into a position in space — and that second one is where something like meaning appears.
A token is a chunk of text. Not quite a word, not quite a syllable. Common words are usually one token. Rarer words get split into pieces. Punctuation and spaces count too.
The model has a fixed vocabulary — typically somewhere around 100,000 tokens — and every piece of text you send has to be expressed using only those. Think of it as a very large set of alphabet blocks. Anything you want to say must be spelled out with the blocks that exist.
Each block is one token. The small number beneath is its ID — the actual value the model receives.
Notice what happened with the longer or stranger words. “Unbelievable” is not in the vocabulary as a single block, so it comes apart into pieces that are. This is why a model can handle a word it has never seen: it spells it out of smaller parts.
Ask a model how many R’s are in “strawberry” and it often gets it wrong. Not because it cannot count — because it never saw the letters. It saw two or three tokens. Asking it to count letters is like asking you to count the brushstrokes in a word someone spoke aloud.
It also explains cost. Providers bill per token, not per word. English runs roughly 0.75 words per token, so 1,000 tokens is about 750 words. Code, unusual names, and languages that do not use the Latin alphabet all use more tokens for the same content — sometimes several times more.
Token IDs are just labels. Token 3007 is not larger or more important than token 12. The number carries no meaning at all — it is a name, not a value.
So the model does something more interesting. It turns each token into a long list of numbers — hundreds or thousands of them — called an embedding. And you can think of that list as coordinates. A position in space.
Two dimensions are easy to picture: a point on a map. Three, a point in a room. An embedding might have 1,536 dimensions, which nobody can picture — but the principle survives. Things near each other are similar. Things far apart are not.
Flattened to two dimensions so it fits on a screen. The real thing has hundreds — but the idea is exactly this.
Nobody hand-placed those words. The model arrived at the arrangement by reading enormous amounts of text and noticing that some words keep similar company. “Doctor” and “nurse” appear in the same kinds of sentences. “Doctor” and “pavement” do not.
That is the whole trick. Meaning, to a language model, is a claim about which company a word keeps.
Embeddings are what make search-by-meaning possible. Ask a normal search engine for “how do I stop my code crashing” and it looks for those words. Ask an embedding-based search and it can return a document about “exception handling” that shares not one word with your question. You will meet this again in module 06, where it becomes the engine behind retrieval.
It is tempting to conclude the model understands. Be careful with that word.
The arrangement is real and it is genuinely useful. But it was built entirely from statistical regularities in text. The model has never seen a doctor, felt an illness, or been to a hospital. It knows the shape of how people write about those things, which is a much narrower thing than understanding — and it is why a model can produce a fluent, well-structured paragraph that is confidently wrong.
Fluency and accuracy come from different places. Tokens and embeddings buy you fluency. Accuracy has to come from somewhere else, and most of this course is about where.
Two short experiments. Both are free, using an AI assistant you may already have access to.
You do not need to know how embeddings are computed. You need to know that text becomes chunks, chunks become positions, and nearness in that space is the model’s entire notion of similarity. Everything from search to hallucination traces back to it.