Why some systems cannot explain themselves.
THE PROBLEM
Many capable models cannot say why they produced a particular output.
The decision emerges from millions of numerical values with no human-readable logic.
WHY IT MATTERS
Regulated decisions frequently require an explanation: credit, employment, insurance, medical.
"The model said so" is not an acceptable reason to decline someone a loan.
THE TRADE-OFF
Simpler models are explainable and generally less capable. Complex models are more capable and less explainable.
Which matters depends on the use.
TECHNIQUES THAT HELP
Methods that indicate which inputs most influenced a particular output.
Approximate and useful.
WHEN TO INSIST ON EXPLAINABILITY
Decisions affecting individuals materially Regulated contexts Anything you may have to justify
WHEN IT MATTERS LESS
Recommendations Internal drafting assistance Low-stakes categorisation
FOR LANGUAGE MODELS
They can produce an explanation of their reasoning, and that explanation is generated text rather than an account of the actual process.
Useful, and not the same as genuine transparency.
