Prompts, tokens, and the window

For all the exotic machinery underneath, the interface to an LLM is almost comically simple. Text goes in, text comes out.

This lesson is the anatomy of both directions, and the hard limit everything must fit inside.

What is a prompt?

The text going in is the promptConcept · lights on your mappromptEverything sent to the model for one request: instructions (“you are a helpful barista assistant”), context (the menu, the customer’s question), and the ask. Prompt quality changes output quality so much that shaping prompts is real engineering work, the closest thing to “programming” a model without training it., and it is much more than the user’s question.

A production prompt is a composed document, one that stacks instructions, rules, relevant data, conversation history, and then, finally, the ask. Here is what the coffee app sends when a user taps “ask the assistant”.

Nobody typed that document by hand. The backend assembled it in the moment between the tap and the request, wrapping seven user words in everything the model needs.

Why so much care over input text? Last lesson is the reason, since the model’s behavior cannot be edited. The prompt is the primary steering wheel, which is why “prompt engineering” is a phrase said without irony in serious buildings.

What is a token?

The model doesn’t see letters, though. Text is chopped into tokenConcept · lights on your maptoken (LLM)The unit models actually read and write, a fragment of a word rather than a whole one. Capacity, reply length, and price are all counted in fragments, and the API bills them in both directions, so every AI feature runs a tab.s, word-fragments, roughly three-quarters of an English word each.

(A familiar word is doing new work here. Module 6’s auth token and this token are unrelated homonyms, and context tells you which is meant.)

Why fragments instead of whole words? A fixed menu of fragments can spell anything ever typed, including new names, typos, and code, while a menu of whole words could never keep up.

Tokens matter because everything is denominated in them: model capacity, response length, and, critically for products, price. AI APIs bill per token, input and output both, so in the anatomy box, those 2,300 tokens of scaffolding are billed on every single tap. The token meter runs under every AI feature.

What is the context window?

And capacity has a hard edge. The context windowConcept · lights on your mapcontext windowThe ceiling on how many tokens fit into one request, counting what you send and what comes back. Whatever misses the cut simply does not exist for the model, which is why long chats lose their openings and why choosing what makes the cut is real work. is the fixed budget of tokens the model can consider at once, prompt and response together.

Sizes vary by model, from dozens of pages’ worth to many books’ worth. Every window is finite, though, and the rule at its edge is absolute. Whatever falls outside is not deprioritized but nonexistent.

The model also keeps no memory between requests, and the reason sits in last lesson. The only thing inside a model is the billions of numbers that training froze, and answering a prompt reads those numbers without changing a single one. A conversation leaves no trace behind, so every request starts from zero.

Module 1’s state must be re-sent every single time, and that is the anatomy box’s [history] block, statelessness at its most radical.

Now the familiar mystery solves itself. Long chats forget their beginnings because the conversation grows past the budget and the earliest messages no longer fit in what gets sent. The app trimmed the oldest messages to make room, and to the model, they never existed.

How the answer comes out

Output generationConcept · lights on your mapoutput generationHow the reply gets made. The model picks a plausible next fragment in light of everything already written, adds it, and goes again, thousands of rounds per reply. That loop is the typewriter streaming, the fluency, and, next lesson, the confident wrongness. happens one token at a time: pick a likely next fragment, append it, ask “now what’s likely?”, thousands of times.

That loop is why AI answers stream in like a typewriter. The streaming is real generation rather than theater, and each fragment is being chosen as you watch. Even stopping works this way, since the end of the answer is one more fragment the model can pick.

And it plants the seed next lesson harvests. Every token is chosen for likelihood, not truth, and nothing in the loop checks facts. Hold that thought.

The mental model to remember

The prompt is everything sent per request: instructions, data, history, and the ask, composed into one document. It is the primary steering wheel for behavior nobody can edit.

The token is the unit of everything: what the model reads, what it writes, and what the API bills. It is the meter under every AI feature.

The context window is the hard budget, prompt plus response. Outside it, nothing exists. The model is stateless, and its world is re-sent every request.

Output generation is one likely token at a time, fed back in thousands of times, and every pick is likelihood rather than truth.

You should now be able to read an AI feature’s request the way the anatomy box does, and hear “the window’s full” as a design problem about what to send, not a malfunction to fix.