Vector stores in production: plan for relevance decay

A vector store is fast on day one, expensive on day ninety, and silently wrong on day two-hundred. The expensive part is well-understood. The silently-wrong part is what kills production deployments, and it has a name: relevance decay.

What you already know

A vector store indexes text (or other modal data) as embeddings, then serves nearest-neighbor queries. The mechanics are not the hard part. Pinecone, Weaviate, Qdrant, pgvector — the vendors are competent and the operational story is broadly the same.

The hard parts are not the infrastructure. The hard parts are what happens to the index over time, what the index does under load, and what the index does when the embedding model changes underneath it.

Day one — fast

Day one, the index is small, the queries are tight, and the relevance is high. This is the demo. Every vector store ships a demo that looks like this.

The trap is to ship to production without instrumenting day-ninety and day-two-hundred metrics. Day one is not the steady state; the steady state is six months later, and the steady state is what you live with.

Day ninety — expensive

By day ninety, the index has grown. Either the customer's data is being added (a knowledge index, a banking history) or the application is generating data (interaction logs, call summaries). The query latency starts to drift, the cost per query climbs, and someone in the customer's operations team asks why their bill is up.

Operating notes for day ninety:

• Eviction. Old vectors that are no longer relevant — superseded runbook versions, expired support tickets, deprecated catalog items — should leave the index. Most vector stores do not do this for you. The eviction policy is part of the deployment.
• Index size budgets. Set a hard cap on index size per customer, per product. When the cap is approached, eviction kicks in, or new indexing pauses, or the customer is alerted. Silent unbounded growth is a billing surprise that the customer will remember.
• Query budgets. Tag every query with the agent that originated it and the deployment it serves. An over-budget vector-store bill usually traces to a single agent running unbounded queries, and an untagged index gives you no way to find which one.

Day two-hundred — silently wrong

This is the failure mode the literature does not warn about. Relevance decays.

It decays for three reasons. Embedding model drift: the customer or the vendor updated the embedding model, and old vectors were embedded with the previous version. New queries are embedded with the new version. The two are not commensurable; the nearest-neighbor results are technically near in vector space and semantically far. The system returns plausible-looking answers that are quietly worse.

Source drift: the underlying documents were updated, but the index was not re-indexed. Old vectors point to text that no longer says what the agent is summarizing. The agent reads the chunk, summarizes correctly, and produces an answer that is correct for last quarter and wrong for this quarter.

Query distribution drift: the queries have changed but the index has not. The customer started using the agent for a new use case six weeks ago; the index was tuned for the old use case; the relevance scores look normal because the system is doing exactly what it was tuned to do, which is no longer the right thing.

How to catch it

Three metrics, watched continuously:

• Embedding model version per chunk. If your index has more than one embedding model version represented, you are decaying. Either rebuild on the new model or freeze on the old one. Do not run both.
• Source freshness. Tag each chunk with the source's last meaningful edit; a knowledge-base reader should carry that through into the index. If average freshness drops below the tolerance the work actually has — operational documents age fast — re-index.
• Relevance feedback. When the customer's operators flag an agent answer as wrong, route the flag back to the chunks that produced it. A chunk that has produced multiple flagged answers is a chunk that should be evicted, demoted, or re-embedded.

The third metric is the one that turns the vector store into a system that ages well. Without operator feedback, the index will decay; with it, the index improves over time.

What we do not promise

Retrieval is not the right architecture for every use case, and a vector store is not the right retrieval backend for every retrieval architecture. For some workloads, a keyword index plus a re-ranker beats a vector index; for some, a graph database does. The vector store is one tool. It is the most-used tool, and it deserves the operational care that most deployments do not give it.

That operational care is the difference between an index that decays and one that improves. It is also the boring half of retrieval — and the boring half is what ships.