The assistant

Monday evening, Case opens the Traversal app with a question about next year and, instead of digging through the FAQ, types it into the little help widget. Can I enter the lottery again if I finished this year?

The answer starts appearing word by word, and everything about how it happens is worth watching in slow motion. What the model actually receives is not Case’s eleven words. It is a promptConcept · lights on your mappromptEverything sent to the model for one request: instructions (“you are a race-help assistant, answer from the provided documents”), context (whatever documents the feature pastes in, the conversation so far), and the user’s actual ask. Prompt quality moves output quality so much that shaping prompts is real engineering work. assembled by Traversal’s backend: a page of instructions, a pasted excerpt of the FAQ’s most-asked answers, the chat so far, and then, at the bottom, Case’s eleven words riding on top of the scaffolding.

The meter and the window

The model does not read that prompt in words. It reads it in tokensConcept · lights on your maptoken (LLM)The unit a language model actually reads and writes: a fragment of text, roughly three-quarters of an English word. Input, output, context windows, and API price are all measured in tokens, which makes the token the meter running under every AI feature., word-fragments a few characters long, sharing nothing but a name with the credential Act I called a token. The fragments are also the billing meter. Every request Traversal sends is priced by tokens in and tokens out, so that page of scaffolding doubles as spend, billed again on every single tap.

The fragments must also fit. A model can consider only so much at once, and that budget is its context windowConcept · lights on your mapcontext windowThe maximum tokens a model can consider at once, prompt plus response together. Everything it “knows” about a request must fit inside; anything beyond may as well not exist, which is why long conversations forget their beginnings.. The window is the whole world. The model has no memory between requests, so the backend re-sends the conversation every time, and deciding what earns a place inside the budget is a real design meeting at every company shipping one of these.

One fragment at a time

And the typewriter effect on Case’s screen, no loading animation at all, is the plain truth of output generationConcept · lights on your mapoutput generationHow the answer is produced: one token at a time, each chosen by likelihood given everything so far, appended, and fed back in to choose the next. It explains the streaming, the fluency, and the confident wrongness to come.. Yes. Then a likely next fragment, then the next, each chosen by likelihood given everything before it, the answer assembling itself out of statistics at reading speed. Case gets a correct, friendly paragraph: finishers welcome, lottery opens in January, good luck.

Read that mechanism once more, though, slowly. Each fragment is chosen for being likely, not for being true. Tuesday morning, the support queue demonstrates the difference.