Making models helpful and safe.
THE PROBLEM
A model trained purely to predict text will produce whatever the training data suggests, including unhelpful or harmful output.
WHAT ALIGNMENT MEANS
Additional training so the model follows instructions, is helpful, and declines harmful requests.
HOW IT IS DONE
Training on examples of good responses Human feedback ranking outputs, used to train the model toward preferred behaviour Explicit rules and guidelines
WHAT THIS PRODUCES
Models that answer questions rather than continuing your text Models that decline certain requests Models with a consistent tone
WHAT IT DOES NOT SOLVE
Factual accuracy. An aligned model is helpful and still hallucinates.
WHY MODELS SOMETIMES REFUSE REASONABLE REQUESTS
Alignment involves judgement calls applied broadly. Some legitimate requests resemble problematic ones.
Rephrasing with context frequently helps.
WHY THEY SOMETIMES AGREE TOO READILY
Models are trained to be helpful, which can produce agreement where disagreement would serve you better.
Ask explicitly for criticism if you want it.
