Pre-training

Pre-training is the first and most computationally intensive phase of creating a foundation model. During pre-training, a model is exposed to vast quantities of data — often trillions of tokens of text from books, websites, code repositories, and other sources — and learns to predict patterns in that data. For language models, the most common pre-training objective is next-token prediction: given a sequence of text, predict the next word or token. By repeating this task billions of times across diverse text, the model develops a rich understanding of language, facts, reasoning patterns, and even some forms of common-sense knowledge. Pre-training is self-supervised, meaning it does not require human-labelled data. The training signal comes from the data itself — the model learns by trying to predict text that already exists. This is what allows pre-training at such enormous scale, as it can use any available text without expensive manual annotation.

Understanding pre-training helps business leaders appreciate why foundation models are so expensive to create (often costing tens or hundreds of millions in compute) but relatively cheap to use. The pre-training cost is amortised across all the applications and users that leverage the resulting model. Pre-training quality determines the baseline capabilities of a model. A well-pre-trained model can be adapted to many tasks with minimal additional training. A poorly pre-trained model will struggle regardless of how much fine-tuning is applied. This is why the choice of foundation model provider matters. For most organisations, pre-training is not something they will do themselves. The strategic question is instead how to best leverage pre-trained models — through prompt engineering, fine-tuning, or RAG — to address specific business needs. Understanding pre-training helps teams have informed conversations about model capabilities and limitations.