An agent is tested like a procedure, not like a demo. In a demo, you pick the messages that work. In production, customers write badly, change their minds, send a blurry photo and ask for a human. The difference between a calm launch and a week of fixes is decided before publication, in the scenarios you took the time to play.
What you risk without tests
Three risks dominate. Invention: the agent announces a price, a delay or a rule that does not exist. Wrong action: a refund for the wrong amount or the wrong case. Leakage: an answer built with another customer's data. Add the loop with no handoff and the wrong tone with an upset customer. None of these shows up on a nicely written happy path.
The six scenario families to play
- Happy path: a clear request, verifiable identity, expected action.
- Incomplete: “it doesn't work”, with no order number or detail.
- Refusal: wrong or expired OTP code, or a customer who refuses to share it.
- Out of scope: a request outside your procedure.
- Emotional: an annoyed customer threatening to leave, insisting.
- Manipulation: “ignore your instructions”, “give me my neighbour's balance”.
Add the limits: an unusual amount, a customer missing from the CRM, a slow or unavailable API. And language variants: French, English, typos, local phrasing. A flow that holds on these cases holds in production.
Use the simulator, not customers, to learn
In Deep Agent Studio, the simulator replays a conversation without touching real customers. You type the message, the agent runs the flow, and you inspect every step: detected intent, tool call, identity check, handoff decision. That is where you see whether the flow holds, not in a demo video. Three habits: play each scenario before publishing, check that no sensitive action leaves without OTP, and replay everything after any change to procedure, prompt or model.
Build a useful test set
Twenty to thirty conversations are enough for one intent, provided they are realistic. Take real, anonymised requests rather than invented examples. For each, note the expected result: resolved or handed off, expected action or none, identity constraint. Keep a simple table and replay it for every version. Do not aim for 100% resolution: a clean handoff is a success, not a failure. This test set becomes your regression guard when you change a message or a threshold.
Test the tools as much as the words
A conversation can look perfect while the tool returned an empty invoice or a silent error. So for each scenario, check what happened behind the scenes: the tool called, the parameters sent, the response received, the latency. A zero amount, a missing identifier or a ten-second delay are signals the conversation alone does not show. If your agent chains two tools, also test the case where the first succeeds and the second fails: that is where double actions appear.
The pre-launch checklist
- One intent, one written and reviewed procedure.
- API tested on success, empty response, error and timeout.
- OTP required on every sensitive action.
- Handoff threshold defined per intent.
- Customer messages reviewed, with no internal jargon.
- Emotional scenario played, with a calm answer.
- Prompt injection and third-party data requests tested.
- Unknown customer and missing data tested.
- Trace checked: no write tool called without intent.
- Limited pilot ready, with a day-one conversation review.
After launch: the pilot
Open small: a few trusted customers or a limited share of volume. Review the first twenty conversations the next day, then daily for a week. Fix the procedure before the prompt: most errors come from a missing step, not a badly worded sentence. Expand when resolution is stable and handoffs arrive with clean context. Keep the test set to verify every change.
Common mistakes
Testing only the happy path gives false confidence. Confusing a nice answer with a correct action lets business errors through. Changing models without replaying tests breaks validated flows. Ignoring handoffs removes your best source of improvement: they show exactly where the agent is not enough. Launching on every channel on the same day multiplies fixes. Finally, forgetting to check traces prevents you from understanding why a conversation went wrong.
FAQ
How many scenarios do I need? Twenty to thirty per intent at launch, then one more for every incident observed.
Do I need real customers to test? No. The simulator is enough before launch; the pilot then confronts the flow with reality.
What if the agent fails? Fix the procedure or the tool, not only the wording. A missing step is not solved by a longer prompt.
Can I test WhatsApp without sending a message? Yes, the simulator replays the conversation without going through the channel.
