Is the model drifting, or is the population changing?

A monitoring report turns amber: the score distribution of this month's applicants has moved away from the one the model was built on. The tempting reading is that the model is drifting and needs rebuilding. Often it does not. Three quite different things produce the same amber light, and each one calls for a different response.

Three things that look alike

The population changed. The people coming in are different: a new sales channel, a campaign aimed at younger applicants, a partner that sends a new kind of customer, an economy that moved. The inputs shift, but the link between those inputs and repayment may still hold. A model can be shifted and still right.

The relationship changed. This is concept drift in the strict sense: the same profile now turns out differently. A rate shock changes who can carry a repayment; a collections process changes who cures; a new product term changes what a high utilization means. A review of the field describes concept drift as “unforeseeable changes in the underlying distribution of streaming data over time.” When the relationship itself has moved, the model is estimating last year's world.

The data broke. Nothing changed outside. A source system migrated, a field changed its meaning, a missing value is now filled with zero, a bureau file changed its format, a category was renamed. This is the most common cause of an amber light, and the most fixable.

What a stability index tells you, and what it does not

The population stability index (PSI) compares how the score is spread across bands in a reference period with how it is spread now. The characteristic stability index (CSI) runs the same calculation on one input at a time. A statistical study of the index describes it as “a statistic that measures how much a variable has shifted over time,” and points out that its familiar rule-of-thumb cut-offs are applied “without reference to Type I or Type II error rates.”

That makes PSI a signal, not a verdict. Its value depends on how the bands are drawn and on sample size: a small segment can look alarming on noise, and a large shift can hide inside one wide band. Above all, PSI says nothing about outcomes. It measures movement in who arrives, not whether the model still estimates their risk correctly. A model can have a calm PSI and be wrong, or a red PSI and still be right.

A triage order

When the light turns amber, we work through four steps in order, because the cheap checks rule out the expensive conclusions.

1. Check the pipeline first. Missing rates, new or vanished categories, value ranges, field definitions, and recent changes in source systems, against the data contract written at go-live. A break is fixed in the data, and the affected cases are re-scored if a decision used them.
2. Find where the shift lives. Run CSI per input and cut the score by channel, product, region, and partner. Then ask the business what changed: a campaign, a cut-off, a partner, a process. Most shifts have a name inside the institution before they have one on the dashboard.
3. Read the outcomes as they mature. Calibration by band and ranking power on recent cohorts, with early indicators such as first-payment default or early arrears while the full performance window is still open.
4. Decide, and write the decision down. What was found, what was done, and who approved it.

What each finding calls for

• A pipeline break: fix the data, not the model.
• A population change, with the relationship holding: keep watching, check calibration in the new segment, and re-simulate the cut-offs, because the same thresholds now approve a different volume. If the model is being used on a segment it was never built for, that is a new use and needs its own analysis.
• The ranking holds, the probabilities do not: recalibrate.
• The ranking has dropped: redevelop, on data that includes the new period.

This mirrors the supervisory view. The revised US model risk guidance (SR 26-2, April 2026) says that “a model that no longer performs as expected may warrant overlays, adjustment, or redevelopment of the model,” depending on the institution's own policy. The institution approves each of these steps; none of them happens quietly.

Monitoring after go-live

The monitoring plan is written before go-live, not after the first alert. It names the reference period, the indices for the score and each main input, the segment cuts, the outcome reads with their maturity windows, the alert levels and who receives each alert, and the route to recalibration or redevelopment. SR 26-2 frames ongoing monitoring as evaluating “the extent to which a model is performing as expected given potential changes in products, exposures, activities, clients, data relevance, or market conditions.” Each item in that list is a reason the light can turn amber.

The plan also watches the actions, not only the model. A score feeds bands and an action matrix; if approvals in one band climb or a review queue overflows, the strategy needs attention even when the model is fine. The results of each action are compared with a control group, so a change in outcomes can be traced to the model, the population, or the strategy.

The question on the dashboard is never only “is the index red?” It is what changed, where, and whether the model still estimates it correctly. The answer to that decides whether the next step is a data fix, a recalibration, a rebuild, or simply another month of watching.