Where it helps and where it does not.
WHAT IT CAN DO WELL
Draft replies for a person to review Summarise long ticket histories Suggest relevant documentation Categorise tickets Translate
WHAT IT DOES BADLY
Anything requiring certainty about your specific systems Judgement about exceptions Emotional situations
WHY CERTAINTY IS THE PROBLEM
It produces confident answers that are wrong, and customers act on them.
WHAT TO DO ABOUT IT
Keep a person between the output and the customer.
WHAT REVIEWING SHOULD CHECK
Factual accuracy Whether it addressed what was asked Tone
WHAT TO NEVER SEND UNREVIEWED
Anything about billing, security or data.
WHAT TO BE CAREFUL WITH
Sending customer data to an external service.
WHAT TO ESTABLISH
What the provider does with it, and whether you are permitted to send it.
WHAT TO REDACT
Anything identifying, where possible.
WHAT TO MEASURE
How often drafts are accepted unchanged.
WHAT A LOW RATE MEANS
It is not saving time.
WHAT A HIGH RATE MIGHT ALSO MEAN
Reviewers are not reading carefully.
WHAT TO CHECK
Quality, independently of acceptance.
WHAT TO TELL CUSTOMERS
Whether they are speaking to a person, honestly.
