How Do You Measure Whether an AI Phone Agent Is Working?

Short answer: measure what happened on the calls, not how many were placed. For outbound calls, watch the connect rate (calls that reached a person) and why the rest failed. For every call, track how many were resolved by the agent, how many were transferred and why, which questions it could not answer, and what callers did next (booked, paid, called back). Then read a sample of transcripts every week. Numbers show where the problems are; transcripts show what to change.

Why "calls handled" is not a useful number

An agent can answer a thousand calls and still fail: by talking to voicemail, giving wrong answers confidently, or passing everything to your team. The useful measures describe outcomes for people.

The core metrics

How do you define a "resolved" call?

Decide before you start counting. A practical definition: the caller's reason for calling was dealt with on the call, by an answer that was correct, an action that was completed (a booking, a change, a lead recorded) or a message with everything your team needs to call back. A call that ended because the caller gave up is not resolved, however short it was.

A weekly transcript review in 30 minutes

1. Pick a sample. A handful of ordinary calls, every transferred call you have time for, and the shortest and longest calls of the week.
2. Mark each one. Correct and complete, correct but clumsy, wrong answer, should have transferred, transferred unnecessarily.
3. Find the cause. Missing or outdated knowledge, unclear instructions, a hand-over rule that is too loose or too tight, or a speech problem (a name or number misheard).
4. Change one thing at a time. Then check next week whether that type of mistake went down.

Signals that something is wrong

• Lots of hang-ups in the first few seconds. Check the greeting: too long, unclear who is calling, or cut off by callers saying "hello".
• Callers repeating themselves. The agent may be interrupting them, or mishearing in noisy conditions; look at turn-taking settings.
• A transfer rate that keeps climbing. The knowledge base is missing common answers, or the hand-over rules are too broad.
• Talk time rising on outbound with no rise in outcomes. The agent may be over-explaining. Shorten replies in the instructions.

What should you expect in the first month?

Expect the first two weeks to be the noisiest. Most early problems are missing facts and hand-over rules that are too broad or too narrow, and both are quick to fix once you see them in transcripts. Judge the agent on the trend, not on day one: are unanswered questions falling, are unnecessary transfers falling, and are callers finishing what they rang for? If those three move the right way, the agent is working.

Test changes properly

When you change an outbound opening line, test the old and new versions on two halves of the same list at the same time, rather than week against week, so the time of year and the list don't distort the result. See A/B testing an outbound opening line.

What ThinnestAI shows you

• Call outcomes for campaigns: connect rate, average talk time and why calls failed (busy, declined, unreachable, voicemail), with voicemails kept out of the connect rate and talk time.
• A full call log: transcript and outcome for every call, recording playback when recording is switched on, and CSV export. See transcripts and logs.
• Agent analytics, including the questions the agent could not answer, so you know what to add to the knowledge base.
• Leads and escalations captured during calls, with links back to the conversation.
• Signed webhooks when a call finishes and when it has been analyzed (with the transcript), so outcomes can land in your CRM next to bookings and sales.

Next step: the voice campaigns page for outbound results, or the voice AI agent page for inbound.