Use case
Customer support automation
An agent answers routine questions from your own policies and account data, takes a small set of allowed actions, and hands everything else to a person with the case already summarised.
The problem
How the work goes today
Most of the queue is the same few questions: where is my order, how do I return this, why was I charged twice, how do I reset my login. An agent opens the ticket, looks the customer up in two or three systems, finds the policy, and types an answer they have typed a hundred times before.
The knowledge base is partly out of date, so experienced agents answer from memory and new ones ask in the team chat. Answers drift between people and shifts. Refunds and exceptions depend on who picked up the ticket.
When a case does need a specialist, it is passed on with a one-line note. The customer gets asked the same questions again, and the specialist spends the first five minutes rebuilding the context.
15%
Average rise in issues resolved per hour when 5,172 support agents at a large software firm got an AI assistant. This is the published journal figure. The earlier working paper reported 14% on average and 34% for the newest and least experienced agents, with little effect on the most experienced.[1]
14%
Share of customer service issues that customers said were fully resolved in self-service, and 36% even for very simple issues. Survey of 5,728 customers in Dec 2023, so it predates current AI agents. Reported by trade press.[2]
2.3 million
Conversations Klarna's assistant handled in its first month, about two-thirds of its service chats, according to the company. Read it alongside the next figure.[3]
May 2025
Klarna's CEO said cost had been too dominant a factor and quality had dropped. The company committed that customers could always reach a person and began hiring human agents again. Reported second-hand, citing Bloomberg.[4]
How it works
Step by step
Highlighted steps are where a person decides. Everything else runs on its own and is logged.
Systems involved
- Helpdesk (Zendesk, Intercom, Freshdesk, Salesforce Service Cloud)
- Knowledge base (Zendesk Guide, Confluence, Notion)
- Order management or billing (Shopify, Stripe, Chargebee)
- CRM (Salesforce, HubSpot)
- Contact centre and voice (Genesys, NICE, Amazon Connect)
- Messaging channels (WhatsApp Business, web chat)
- 01System
Message arrives
A chat, email or web form lands in the helpdesk (Zendesk, Intercom or Freshdesk) and the customer is identified.
- 02Agent
Classify
The agent tags intent, language, customer tier and risk flags such as a legal threat or a vulnerable customer, and writes them to the ticket.
- 03Agent
Look up context
It reads the order, subscription or account record from the OMS, billing system or CRM with read-only access.
- 04Agent
Retrieve policy
It pulls the relevant, versioned policy and help articles from the knowledge base and escalates if nothing relevant is found.
- 05Agent
Draft the reply
It writes an answer that cites the policy it used, then runs checks for personal data, unauthorised promises and tone.
- 06System
Take an allowed action
For actions on an allow-list, such as resending an invoice or starting a return under a set value, the OMS or billing API is called with parameters validated server-side.
- 07Person
Approve exceptions
Refunds above the threshold, policy exceptions, account closures and any new type of request go to a support lead in the helpdesk to approve or take over.
- 08Agent
Hand off with a summary
When a person takes the case, the helpdesk ticket carries a summary, the customer's details and the steps already tried.
- 09System
Log and sample
Every conversation is logged with its sources and actions, and a weekly sample is reviewed for wrong facts and unauthorised commitments.
Measurement
What we measure
Agreed before anything is built, with a baseline taken from how the work runs today.
- Containment rate
- Share of conversations closed by the agent with no hand-off to a person and no repeat contact on the same issue within 7 days. Conversations the customer abandons are counted as not contained.
- Repeat-contact rate
- Share of customers who get in touch again about the same issue within 7 days, reported separately for agent-handled and person-handled cases. This is the check against containment that only looks good on paper.
- CSAT by channel and intent
- Post-conversation satisfaction score for agent-handled and person-handled cases, compared within the same intent so easy questions don't flatter the agent.
- Hand-off quality
- Share of hand-offs where the person taking over did not need to ask the customer for information the agent already had.
- Policy-violation rate
- Share of answers in the weekly QA sample that state a wrong fact or make a commitment outside policy, reviewed by a support lead.
- Cost per resolution
- Model, software and staff cost for the period divided by the number of issues resolved, compared with the pre-launch baseline.
Pitfalls
What usually goes wrong
- Containment that hides unhappy customers
- Report repeat contact and CSAT next to containment, and keep a small holdout group that always goes to a person so the comparison stays honest.
- The agent invents a policy
- Answer only from retrieved, versioned policy text, escalate when retrieval finds nothing relevant, and keep a set of tricky questions that is re-run before every change.
- Customers can't reach a person
- Offer a route to a person in every conversation, and send vulnerable customers, complaints and legal issues straight to the team.
- Actions nobody meant to allow
- Keep an allow-list of actions with value limits, validate every parameter on the server, and use idempotent APIs so a retry can't refund twice.
- Instructions hidden in customer messages
- Treat message text as data, give the agent only the permissions its allowed actions need, and never let a message change those permissions.
Build or buy
When a ready-made tool is enough
Custom work isn't always the answer. Here's how we'd decide.
- Helpdesk-native and specialist agents (Zendesk AI agents, Intercom Fin, Salesforce Agentforce, Ada, Sierra, Decagon) are enough when your knowledge base is in good shape, you run a standard helpdesk and the actions are simple.
- Ask any vendor how it defines a resolution before comparing their figures. The definitions differ and the published rates are their own measurements.
- Custom work pays off when actions reach into several internal systems such as a legacy OMS, core banking or a hotel PMS, or when data residency rules out a hosted product.
- It also makes sense when per-resolution pricing gets expensive at your volume, or when support is one agent in a wider workflow that needs its own evaluation and audit trail.
Industries
Where this applies
Sources
- 1.Brynjolfsson, Li and Raymond, Generative AI at Work (QJE 2025; preprint Apr 2023)
- 2.CX Today, Gartner study on self-service resolution (Aug 2024)
- 3.Klarna, AI assistant handles two-thirds of customer service chats in its first month (Feb 2024)
- 4.Entrepreneur, Klarna CEO reverses course by hiring more humans (May 2025)
Figures are as reported by each source on its publication date. Examples are public industry cases, not AI Sewa clients.
Next step