Non-determinism★

Non-determinism means that repeating a request with the exact same inputs can yield different results. This is caused by two layers in the inference pipeline:

1. Sampling Parameters (Software): During generation, the model outputs log-odds (logits) for the vocabulary. If Temperature is set above 0, or Top-p is less than 1.0, the harness samples from these distributions probabilistically. A temperature of 0 makes the output highly deterministic by always selecting the most probable token.
2. Hardware Precision (Hardware): Even at temperature 0, enterprise model serving batches multiple requests onto massive GPU clusters. Tiny floating-point rounding errors (e.g. bfloat16 arithmetic variations) can accumulate across layers, occasionally tipping the probability distribution and changing a token choice, which alters all subsequent predictions.

Calibration Parameters in Code

When building harnesses, adjust sampling parameters depending on your task goals:

// For code generation, data extraction, and strict schemas (Deterministic)
const strictConfig = {
  temperature: 0.0,
  topP: 0.1
};

// For brainstorming, creative writing, and prompt diversification (Exploratory)
const creativeConfig = {
  temperature: 0.7,
  topP: 0.95
};