protologue

Prompt Sensitivity

Also called prompt brittleness, format sensitivity.

Prompt sensitivity is the variation in a model's performance caused by superficial changes to a prompt, such as formatting, separators, spacing, or wording, that do not change the task's meaning.

Description

Sclar et al. found accuracy differences of tens of percentage points from formatting choices alone, and argued that evaluations should report performance across a range of plausible formats.

Sources

  1. Sclar et al. (2023). Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting.

Cite this entry

Protologue. (2026). Prompt Sensitivity. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.0.0, PTL-0100). https://protologue.com/t/prompt-sensitivity/

BibTeX
@misc{protologue_prompt_sensitivity,
  title = {Prompt Sensitivity},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.0.0},
  note = {Entry PTL-0100},
  url = {https://protologue.com/t/prompt-sensitivity/}
}

Markdown JSON