Reinforcement Learning from Human Feedback (RLHF)

Reinforcement Learning from Human Feedback (RLHF) is an AI alignment technique used to train language models to produce outputs that humans prefer. It works by collecting human feedback on model outputs, training a reward model to predict human preferences, and then using reinforcement learning to optimise the language model to score highly according to that reward model. The process typically has three steps. First, human annotators compare pairs of model outputs and indicate which is better. Second, these preference judgments are used to train a reward model — a separate neural network that learns to predict which outputs humans would prefer. Third, the language model is fine-tuned using reinforcement learning (often Proximal Policy Optimisation, or PPO) to maximise the reward model's score. RLHF was a key innovation behind ChatGPT and has been adopted by virtually all major AI labs. It addresses a fundamental challenge: pre-trained models learn to mimic the distribution of text on the internet, which includes both helpful and harmful content. RLHF steers the model towards consistently helpful, honest, and safe behaviour.

RLHF is what transforms a capable but unreliable language model into a trustworthy AI assistant suitable for business use. Models trained with RLHF are more likely to follow instructions accurately, provide helpful responses, decline harmful requests, and admit when they do not know something. For organisations evaluating AI providers, understanding RLHF helps assess model quality. The investment a provider has made in human feedback, the diversity of their annotators, and the rigour of their alignment process all influence how well the model performs in real-world business contexts. Organisations that fine-tune their own models can also benefit from RLHF principles. Collecting feedback from domain experts on model outputs and using that feedback to improve model behaviour creates a virtuous cycle of improvement. Even simpler approaches — like using preference data to guide prompt engineering — draw on the same underlying concepts.