Understanding why a system decided something.
THE DEFINITION
The ability to explain why a model produced a particular output.
THE PROBLEM
Many capable models cannot. The decision emerges from millions of values with no human-readable logic.
WHY IT MATTERS
Regulated decisions frequently require an explanation: credit, employment, insurance, medical.
"The model said so" is not an acceptable reason to decline someone a loan.
THE TRADE-OFF
Simpler models are explainable and generally less capable. Complex models are more capable and less explainable.
WHEN TO INSIST ON IT
Decisions affecting individuals materially Regulated contexts Anything you may have to justify
FOR LANGUAGE MODELS
They can produce an explanation of their reasoning, and that explanation is generated text rather than an account of the actual process.
Useful, and not the same as transparency.
RELATED TERMS
Bias, accountability, human oversight.
