Continued pre-training (CPT)
Further training of a base model on raw text from one field, so it learns the field's language before it learns a task.
Category
How a model is chosen and adapted
Related terms
Continued pre-training (CPT) is further training of an existing base model on raw, unlabeled text from one field, so the model learns that field's vocabulary and phrasing before it is taught any task.
When it is worth doing
Only when a test shows the base model failing on the field's language itself: the terms of a regulation, an accounting standard, or a plant's technical documents. If the model already reads the material correctly, continued pre-training adds a training run and teaches nothing the task needs. It is a conditional step, not a fixed stage of every build.
What it does not do
It does not teach a format, a task, or a policy. Those come from supervised fine-tuning on graded examples, and rules and approval limits stay outside the model. The text it trains on must be text the customer has the right to use, and it stays with that customer.
The regulation to policy use case shows it as an optional step, run only if the base model fails legal vocabulary on the test.
Often confused with
How it differs from similar terms.
| continued pre-training (CPT) | Compared with | What's different |
|---|---|---|
| Continued pre-training | Supervised fine-tuning | Continued pre-training reads raw text to learn a field's language. Supervised fine-tuning learns from input and output pairs to do one task in one format. |
Want these terms worked through on one of your own decisions? Start with a Decision Blueprint