What it takes to trust an agent doing the same work every day
Trusting an agent once is a demo. Trusting it on day two hundred, through drift, quiet regressions, and failures that do not announce themselves, is the actual problem. This note collects what we watch for and what we still cannot see.
A single good run tells you the agent can do the work. It tells you almost nothing about whether you can rely on it. Reliability is a property of the hundredth run and the thousandth, of the days when the input drifts a little, the day a dependency changes underneath it, the day it fails in a way that produces plausible output instead of an error.
The failures that worry us most are the silent ones. A crash is honest; it stops and asks for attention. A confident wrong answer does neither, and by the time it is noticed it has already been trusted downstream. So the work here is less about peak capability and more about knowing, continuously, whether the thing is still doing what it did yesterday.
What we are measuring
- How would we notice drift in an agent's behaviour before a person downstream noticed it as a problem?
- What does a silent failure look like from the outside, and can it be made loud without drowning everything in false alarms?
- When behaviour regresses, how do we tell a real regression from ordinary variation in a system that is not deterministic?
- What would we need to observe day over day to say, honestly, that trust is still warranted?
Open questions
- Can trust be maintained continuously, or does it always have to be re-earned after a change?
- What is the smallest set of signals that would let a team sleep at night with an agent in the loop?