Most AI teams treat evals as a launch gate. A big rubric, a hand-curated test set, two engineers spending three weeks scoring outputs, a green light, and then nothing for the next quarter. That is the wrong shape for the work. Models drift. Prompts drift. The world the agent operates in drifts. A launch eval proves the launch was fine. It does not catch the regression in week six.
We invert the priority. Cadence first, coverage second. A small eval that runs every week beats a comprehensive eval that runs every quarter. The weekly eval does not have to be deep. It has to be honest, automated, and on the calendar. The first version we ship is usually a 50-example human-graded sample with three metrics: task success, ICP fit, and tone.
What the cadence catches that coverage misses. Silent prompt regressions after a model update. Drift in the ICP because the company moved upmarket. A scraper change that quietly poisoned the input data. A new edge case showing up in production three times a week. None of these were in the launch test set. All of them show up in the weekly eval if you run one.
The mechanics matter. The eval has to be cheap enough to run weekly, which means automated where possible, human-graded where it is not. The sample has to be drawn from the actual production traffic, not a curated set, otherwise it stops resembling what the system is doing. The metric has to be defended by someone in the room. If no one will own a number, the number does not belong on the eval.
There is a second-order effect we did not expect. Teams that ship a weekly eval ship faster, not slower. They are willing to push prompt changes on a Tuesday because the eval will catch the regression on Friday. The cadence is permission to move. Without it, every change is high-risk because nothing tells you whether it worked.
If you have one week to invest in evals before shipping an agent to production, do not spend it on coverage. Spend it on the smallest eval you can run every Friday, forever, automatically. The coverage compounds later. The cadence is the load-bearing piece.