[01] BLOG
What we learned replaying 40 million traces
Replay turned out to be less about debugging and more about trust. Here’s what the data says about how teams actually use it.
AUTHOR
Lena Park
CATEGORY
Engineering
DATE
When we shipped replay, we expected teams to use it after incidents. They do, but that’s a third of replays. Most happen before a deploy, when someone wants to see a new prompt run against last week’s real traffic.
We looked at 40 million replays across 1,200 teams over six months. Here is what stood out, and what it changed about how we build.
Replays cluster around releases
Across every plan, replay volume spikes in the two hours before a deploy and drops sharply after. Teams aren’t debugging; they’re rehearsing. A replay is the closest thing an agent has to a staging environment, because it runs the new version against inputs you know happened.
The pre-deploy ritual
The most common pattern looks the same everywhere:
Pick twenty to fifty traces from the last week, weighted towards the weird ones.
Replay them against the candidate prompt or model.
Diff the tool calls, not just the final answer.
Ship if nothing surprising moved.
Step three matters more than we expected. Final answers often look fine while the path to them gets worse: an extra search, a retried tool, a slower model call. Those regressions only show up when you compare step by step.
Trust is the real product
“We didn’t buy replay to fix bugs. We bought it so legal would stop asking what the agent might do.” Head of Platform, logistics customer
Teams in regulated industries replay three times more often than everyone else, and they keep replays four times longer. For them, a stored, reproducible trace is evidence. It answers the question every reviewer eventually asks: show me exactly what happened.
What changed on our side
Replays are now first-class objects with their own retention, separate from traces.
The diff view leads with tool calls and cost, and puts the final answer last.
You can pin a replay set to a branch, so every pull request runs the same rehearsal automatically.
What’s next
We are working on replay sampling: picking the traces most likely to break a change, instead of the most recent. Early results catch twice as many regressions with half the replays. If you want early access, reply to the changelog email.
MORE POSTS