A voice AI agent is a customer service agent that talks: it hears the customer, understands what they need, accesses sources and systems and answers by voice. It answers inbound calls, places outbound calls for reminders and updates, and fills in forms by voice. The real difficulty is not the language model but the details: Hebrew speech recognition of numbers, ID numbers and names, short response times, stopping when the customer interrupts, identification over the phone, handover to an agent with context, and the privacy of recordings.
What is a voice AI agent, and how does it work?
A voice agent has three layers: speech recognition (Speech-to-Text), the agent itself, which understands and decides, and speech synthesis (Text-to-Speech). Services such as Azure AI Speech and Voice Live connect the layers into a single real-time flow, which shortens response time.
The important part: the voice agent is not a separate system. It runs on the same secured knowledge, the same identification and the same permissions as the chat. Whatever the bot can answer in writing, it can answer by voice, and whatever it is not allowed to do in writing, it is not allowed to do on the phone either. Answers come from approved sources using RAG (guide: what is RAG), and for the service agent itself there is a guide to AI customer service agents.
There are three common places for voice: voice conversation on the website, a phone bot, and WhatsApp voice messages that are transcribed and answered like a written message.
Inbound and outbound calls: what is the difference?
| Inbound calls | Outbound calls | |
|---|---|---|
| Who initiates | The customer calls the contact center | The agent calls the customer |
| Common uses | Answering questions, case status, opening a case and routing to a department | Reminders, updates on a closed case, callbacks and surveys |
| What to prepare | Approved sources, an identification flow and rules for handover to an agent | A recipient list, the purpose of the call, and what to do when nobody answers or someone else picks up |
| The main risk | A customer stuck without an answer | Giving information to the wrong person |
| End of the call | A resolution, an open case or a handover to an agent | Logging the outcome in the CRM |
Why is Hebrew speech recognition harder than it looks?
Hebrew speech recognition has improved a lot, but a service call is full of exactly the things that are hard to recognize:
- Numbers. A phone number, an order number or an amount is said in many ways: digit by digit, in pairs or as a whole number.
- ID numbers. Nine digits, and one mistake makes the whole request wrong. Check the check digit and read the number back for confirmation.
- Names and streets. First names, last names, streets and products are not always familiar to the model. A term list tailored to the organization helps recognize them.
- Accents and mixed languages. Customers switch between Hebrew and English, Russian or Arabic mid-sentence. Automatic language detection helps, and you still need to test on real recordings.
- Noise and weak lines. A call from a car, a work site or a busy street sounds different from a recording in the office.
The solution is not just a better model but conversation design: ask for one detail at a time, confirm out loud what was captured, allow numbers to be typed on the phone keypad, and validate each field before moving on.
Response time and barge-in: what makes a call feel natural?
In chat, a short wait goes unnoticed. On the phone, every silence feels like a dropped call. So:
- Real-time streaming. The agent starts speaking before the full answer is ready.
- Short answers. A sentence or two and a follow-up question, without reading out long lists. On the website, the screen shows the results while the bot speaks.
- A holding phrase. When the agent queries a system, it says "one moment, I'm checking" instead of going silent.
- Barge-in (Barge-in). When the customer starts talking, the agent stops and listens. Without it, the customer has to wait for the end of every sentence.
- End-of-speech detection. The agent needs to tell a breath from the end of a sentence, so it does not cut off a customer who is reading out a number.
How do you identify the caller on the phone?
The number the customer calls from is a hint, not identification. You can use it to look up the likely customer in the CRM and shorten the call, but before any personal data you need an additional identification step, such as a one-time code, according to the organization's policy.
The rule is the same as in chat: general information is open to everyone, personal information only after identification, and actions only within permissions. On outbound calls, first confirm that the right person answered, and only then share details.
How does a voice agent connect to the contact center and the CRM?
- Telephony. A connection to the contact center's phone system or telephony provider, so the agent receives calls on the same number and transfers them to the existing queues.
- CRM. The agent retrieves the customer's details and cases, opens a new case, and every call is logged with a transcript and a summary.
- Filling in forms by voice. The agent collects field after field, validates the ID number, phone, email and date, and reads back the complete request for confirmation before submitting.
- Handover to an agent. The customer can ask for an agent at any point. The agent receives the call with a summary, a transcript and what was already checked, and does not ask the same questions again.
- Translation for the agent. When the customer speaks another language, automatic language detection and real-time translation let the agent join the conversation.
Recording, transcription and privacy: what do you need to plan?
A voice call is personal data: the voice itself, the transcript and the details the customer shared. So decide in advance:
- Whether to record, or to keep only a transcript and a summary.
- Where the data is stored, and for how long.
- What is sent to the model and what is not. In KESHER, for example, identifying details are not sent for translation.
- Who is allowed to listen to a recording or read a transcript.
- How the customer knows they are talking to an automated system and that the call is recorded. In KESHER, the bot introduces itself as an automated system and announces the recording.
A voice agent needs a design that complies with Amendment 13 to Israel's Privacy Protection Law: where the data is stored, what is sent to the model and who sees what. The legal interpretation for your organization is best settled with your privacy protection officer or legal counsel.
What should you test before a voice agent goes live?
- Real customer recordings, with accents, noise and weak lines, not only recordings made by the team.
- Numbers, ID numbers, names and streets, including read-back and correction.
- Interruptions mid-sentence, long silences, and a customer who asks to hear something again.
- Questions with no source: the agent says so and does not guess.
- Failed identification, and what the customer gets next.
- Handover to an agent: the context arrives, and there is a response even when all agents are busy.
- On outbound calls: no answer, voicemail, and someone else picking up.
- The transparency notice, and what is actually saved in the CRM.
- An alternative for people who cannot or do not want to talk to a bot: an agent, keypad input or a written channel.
After go-live, measure the same metrics as for a written service agent: containment, handover to an agent and its reasons, and satisfaction. For voice, add calls dropped mid-way and fields the agent had to ask for again. For the cost of development and operation, see how much an AI agent costs an organization.
Examples from our projects
KESHER · voice on the website: talk to the bot in Hebrew, English or Russian, including filling in forms by voice, while the screen shows results in parallel.
KESHER · phone bot: voice answering for inbound calls to the contact center, and outbound calls for reminders, updates on a closed case, callbacks and surveys.
Ashtrom Group · WhatsApp voice messages: the customer asks where the concrete is in a voice message, the message is transcribed, and the agent replies with the arrival time and a location pin. For connecting WhatsApp to your systems, see the guide to WhatsApp and CRM integration.
Israel Aerospace Industries · AICare: an assistant in the iCare customer portal that answers from the technical literature in writing and by voice, in any language, with translation for the agent when they join the conversation.
Frequently asked questions
Does a voice AI agent understand Hebrew?
Yes. Speech recognition and speech synthesis work in Hebrew, as well as in English, Russian and other languages. Numbers, ID numbers and names need careful design: one detail at a time, read-back for confirmation and validation.
Can the bot call customers on its own initiative?
Yes. Outbound calls are used for reminders, updates on a closed case, callbacks and surveys. Before the bot shares any information, it confirms that the right person answered.
How does the bot identify the caller?
The phone number is used only as a hint to look up the customer in the CRM. Personal data requires identification first, for example a one-time code. General information is open to everyone.
What happens when the customer asks for an agent?
The call is transferred to an agent at any point, with a summary, a transcript and what was already checked, so the customer does not have to repeat themselves. When the customer speaks another language, real-time translation helps the agent join.
How are call recording and transcription handled?
Decide in advance what is stored, where, for how long and who can see it, and the bot introduces itself as an automated system and announces that the call is recorded. The design needs to comply with Amendment 13 to Israel's Privacy Protection Law, and the legal interpretation is best settled with your privacy protection officer.
CTO at Zooz-Tech. Leads development and architecture of AI agents, Generative UI and Azure infrastructure.