Cross-Session Inconsistency
Same question. Same document. Different answer. No notification.
This page covers one specific way AI gets things wrong at work, and what to do about it.
It runs in order. What goes wrong, why it happens, where you'd notice it on an ordinary day, who takes the blame, roughly what it costs, and the check that catches it. Then one thing to try this week.
The dollar figures are estimates, not measurements. The assumptions behind each one are printed right there, so you can swap in numbers that fit your job. Anything actually measured carries an OBSERVED tag.
What is cross-session inconsistency?
Two independent sources of variance:
1. Sampling. Generation involves stochastic selection. Even at low-temperature settings, outputs on complex tasks vary between runs — sometimes cosmetically, sometimes substantively.
2. Silent model updates. The model behind a product endpoint changes. Vendors update, deprecate, and re-tune, and behavior can shift without a version bump visible to you. A process validated in March may behave differently in June with no change on your side.
The second is the one that breaks governance. Any control that was tested once and assumed stable is resting on an assumption the vendor never agreed to.
What do people assume?
That identical inputs produce identical outputs — the property every other business system has. That if the analysis was right on Tuesday, running it again Thursday confirms it.
Outputs vary across runs. They also vary when the underlying model changes, which happens without announcement.
Where does it show up at work?
A compliance team validates an AI classification workflow, documents the validation, and puts it into production. Six months later a routine audit re-runs the validation set.
Agreement has drifted. Nothing in the organization changed. Nobody was notified. The control has been reporting green against a model that no longer behaves the way the validation documented.
Who carries the downside?
Vendor: none — terms permit model changes. Executive: attests to control effectiveness. Manager: owns the control. You: performed the original validation, which is now the document that describes a system that no longer exists.
What does it cost?
[MODELED — not reported]
ASSUMPTIONS Validated AI workflows: 4 Probability of material drift per workflow per year: 20% Expected drift events: ~0.8 / year Cost per event: $20,000 – $150,000 (revalidation, remediation of decisions made during drift, control deficiency finding)
Annualized exposure: ~$16,000 – $120,000
How do you control for it?
Scheduled revalidation against a fixed regression set — a held-out set of inputs with known-correct outputs, re-run on a defined cadence, with results logged and compared over time.
Plus: capture and log the model/version identifier on every consequential output, so drift can be correlated to a change rather than guessed at.
CONTROL COST Quarterly revalidation: 4 / year Time per cycle: 10 hours Annual: 40 hours Fully loaded rate: $85 / hour
Annualized control cost: $3,400
What should you do this week?
RECOMMENDATION
Build a regression set for one AI-assisted process you own: twenty inputs, known-correct outputs, stored. Run it quarterly. Log the results and the model version.
This is the most technically credible artifact in the entire book, it takes half a day to build, and it puts you in the small population of people who can answer "how do you know it still works?" with evidence rather than assertion.
Evidence
ANALYSIS Sampling variance is an architectural property — state as design fact.
REPORTED Documented behavioral drift in production model endpoints between versions.
This is one of 24 failure modes. The book gives you all of them — plus the controls that catch each one and a 90-day plan to prove you ran them.
Preorder the Book