The mechanism behind the errors.
WHAT THESE SYSTEMS DO
Generate text that is statistically likely given what came before.
Not retrieve facts. Not reason to conclusions. Predict plausible text.
WHY THAT PRODUCES ERRORS
Plausible and true are different properties.
Where the model lacks information, plausible text is still produced.
WHY IT SOUNDS CERTAIN
Fluency and accuracy are separate. The model optimises for the first.
Training data is mostly confident prose, so the output is confident prose.
WHAT THIS MEANS
You cannot judge reliability from how the answer sounds.
WHERE ERRORS CONCENTRATE
Specific figures Names, dates and places Citations and references Recent events Obscure subjects Arithmetic
WHERE THEY ARE RARER
Explanations of well-established concepts Answers about material you provided
Language tasks: rewriting, summarising, translating
THE PRACTICAL RESPONSE
Verify in proportion to consequence. Supply the facts yourself where they matter.
