Output Tokens★

Output tokens (also called completion tokens) are the text fragments generated by the model and sent back to your client application.

You interact with output tokens when:

• Waiting for a chat response to finish streaming onto your screen.
• Encountering a truncated response because you reached the model's output limit.
• Sizing the latency budgets for real-time customer APIs.

Technical Details: Latency & Cost

Unlike input tokens, output tokens are the primary bottleneck in LLM applications:

1. Sequential Generation: Because the model operates on next-token prediction, it must compute each output token one by one. The model cannot compute token #2 until token #1 has been fully generated and appended back to the context.
2. GPU Memory Bandwidth Bound: For every single token generated, the GPU must read all billions of model parameters from its memory. This makes output token generation slow (high latency) and expensive.
3. Premium Cost: Output tokens are typically 3x to 5x more expensive than input tokens.
4. Limits: Models have strict output limits (typically 4,096 to 8,192 tokens) which are separate from their massive input context windows.

Field Applications & Latency Optimization

1. Fullstack Developers (Streaming Responses)

To prevent users from waiting for the entire completion to finish, fullstack developers stream output tokens to the UI as they are generated:

• Code Example (Node.js Streaming):

  const stream = await client.messages.create({
    model: "claude-3-5-sonnet",
    max_tokens: 1000,
    messages: [{ role: "user", content: "Write a long essay on code structure." }],
    stream: true
  });
  
  for await (const chunk of stream) {
    if (chunk.type === "content_block_delta") {
      process.stdout.write(chunk.delta.text); // Print tokens in real-time
    }
  }