Typorithm · medium

Context, Cost & Latency

context-limitstransformerslatencyttftcost

Saved on this device

Full lesson · 8 learning steps Open any exercise for focused practice

Step 1 of 8 · intro card

Learn Context, Cost & Latency

Inference & Sampling

Inference has two phases — prefill processes the whole prompt in parallel (driving time-to-first-token) and decode generates tokens one by one. Cost scales with input plus output tokens, and a longer context means more KV-cache memory and higher latency.

A request runs in two phases. Prefill ingests the entire prompt at once in parallel and produces the first token; this is what time-to-first-token (TTFT) measures. Decode then generates the remaining tokens sequentially, one per step.

Total latency is roughly TTFT plus the number of output tokens times the per-token decode time. Because decode is sequential, longer responses take proportionally longer regardless of how fast prefill was.

Cost is usually billed on input tokens plus output tokens. A longer context also enlarges the KV cache, consuming more memory and adding latency, which is why prompt length and output length both matter for speed and price.

Prefill is parallel and sets TTFT; decode is sequential.

Total ≈ TTFT + outputTokens × perTokenTime.

Longer context = bigger KV cache, more memory and latency.

// prefill (parallel) then decode (sequential)
const ttft = prefillTime(promptTokens);
const total = ttft + outputTokens * perTokenTime;
// longer context = more memory + latency

Read the implementation

Connect each part of the code to the concept before you type.

1. Two phases

Inference has two phases: prefill processes the prompt in parallel, then decode runs sequentially.

// prefill (parallel) then decode (sequential)

2. Time to first token

Prefill over the prompt tokens drives time-to-first-token (TTFT).

const ttft = prefillTime(promptTokens);

3. Total latency

Total time is TTFT plus the sequential decode of each output token.

const total = ttft + outputTokens * perTokenTime;

4. Decode cost

Each output token adds a fixed per-token decode time, so latency grows with output length.

outputTokens * perTokenTime

5. Context cost

A longer context means a larger KV cache, more memory, and higher latency.

// longer context = more memory + latency

A complete learning loop

Understand

Read the concept map and follow the code breakdown.

Reconstruct

Type the implementation, repair mistakes and work through every recall exercise.

Verify

Use the final check without hints; revisit weak concepts in short practice sessions.