Cantonese AI voice agent: English model failures in Hong Kong

By , Co-Founder, Genium Group · Published

When we ran Whisper-large-v3 on PDPO-compliant recordings from Hong Kong clinics, the model consistently dropped tones on Cantonese surnames and flattened code-switched English loanwords. We built our AI Voice Phone Agents after seeing these exact patterns in production calls. The primary keyword Cantonese AI voice agent appears here as the core product we tuned for local speech.

Why English-trained speech models fail on Hong Kong Cantonese calls

English-trained acoustic models start from grapheme-to-phoneme rules that treat every syllable as stress-timed. Cantonese is syllable-timed, with six tones that change the meaning of a word. When HuBERT-large, trained only on English Librispeech, meets a Cantonese tone, it maps the nearest English vowel cluster instead. The entering tone on surnames such as Chan or Leung is deleted every time.

We saw the same bias when we passed mixed audio through the base Whisper checkpoint. The decoder chose English subword tokens even when the audio held clear Cantonese phonemes.

Cantonese English code-switching ASR therefore inherits the English bias twice, once in the encoder and again in the language model head. Fine-tuning on pure Cantonese corpora does not remove the English prior entirely. The model still expects English phonotactics at every frame boundary.

Concrete failure modes: names, addresses, and mid-sentence code-switching

In field recordings, a caller said “Wong Tai Sin MTR station, block C, flat 12” and the model made three distinct errors. It rendered “Wong” as “one”, dropped the tone on “Tai”, and turned the estate name into “Wong Tie Sin”. The Cantonese address speech model had never seen the full compound “Wong Tai Sin” paired with the English token “block”.

Mid-sentence code-switching made it worse. The caller switched to “level twelve please”, and the recogniser added an extra syllable. The Cantonese tone before it had lowered the pitch baseline.

Building names such as “The Pavilia Hill” failed too. The model output “The Pavilion Hill” because the Cantonese name pronunciation AI had no entry for the loanword “Pavilia”.

In every case the Cantonese AI phone agent lost the second half of the address once the speaker switched languages inside the same breath group. These failures are reproducible on any English-only checkpoint. They are not random noise.

Four fixes we ship in our Cantonese AI voice agent stack

First, we replaced the English G2P with a Cantonese-first lexicon built from Jyutping plus 4 800 Hong Kong-specific proper nouns. Second, we trained a mixed Cantonese-English acoustic model on the MCE corpus style. This forces the encoder to keep separate language ID embeddings for each frame.

Third, we added a loanword list with the 300 most frequent English tokens used inside Cantonese sentences. The decoder no longer treats them as out-of-vocabulary. Fourth, we expose a live per-client pronunciation override dictionary. Clinic staff can update it in the dashboard before each shift.

Together these changes raised word accuracy on code-switched address utterances from the low sixties to the mid-eighties on our internal test set. We still see errors on rare surnames. So the Cantonese AI voice agent ships with a clear human handoff rule. It applies whenever the override dictionary returns no match.

What we still cannot fix (yet): open problems in Cantonese AI voice agents

Three classes of error remain outside current engineering reach. Unseen mixing styles appear when callers use nicknames such as “Ah Ming” instead of the full given name. No public dataset tags these variants.

Rare surnames like “Ng” or “Yip” still trigger deletion because the training data under-samples them. Noisy PSTN lines also degrade tone detection, and we cannot reliably tell line noise from a low-register tone.

The Cantonese English code-switching ASR literature shows the same ceiling. Word error rates plateau once code-switch density exceeds one switch per second. Until larger, annotated Hong Kong voice bot Cantonese corpora exist, these edge cases will need human escalation. We therefore treat the Cantonese AI voice agent as a triage layer, not a full replacement.

Operational implications for clinics, law firms and agencies in Hong Kong

Clinics can safely route simple appointment confirmations through the Cantonese AI voice agent. The vocabulary stays within the tuned lexicon. Law firms booking first consultations still need a human fallback. That applies once the caller gives a case number or address that the override dictionary lacks.

Agencies handling property viewings must build call flows that follow the Personal Data (Privacy) Ordinance (PDPO). They must record consent before any address capture. The same flow must transfer the call at once when the Cantonese address speech model returns a low confidence score.

In all three sectors the practical rule is the same. Let the Cantonese AI phone agent handle the first thirty seconds. Then hand off if any name, address or code-switched token falls outside the four fixes. This keeps compliance with OFCA caller-ID rules and respects the known limits of current Cantonese name pronunciation AI.

Conclusion

The Cantonese AI voice agent we run today comes from measuring English-model failures on real Hong Kong calls and shipping four targeted fixes. It handles most routine traffic. It also surfaces the cases that need human judgement. That boundary is the operating reality for any business that fields Cantonese calls today.

How we build Cantonese phone agents: AI voice agents.

FAQ

How accurate are Cantonese AI voice agents for mixed Cantonese-English calls?

Accuracy depends on whether the stack uses a mixed-language acoustic model and an explicit loanword list. Without those parts, English-trained models drop tones and insert wrong subwords on every code-switch. With the four fixes we ship, word accuracy on address utterances moves into the mid-eighties on our test set. Rare surnames and heavy noise still trigger handoff, so the Cantonese AI voice agent includes a live override dictionary that staff can edit before each shift.

Why do AI phone agents mispronounce Hong Kong street names and building names?

English G2P rules map Cantonese tones to the nearest English vowel cluster. Estate names such as Wong Tai Sin or The Pavilia Hill have no entry in standard lexicons, so the decoder guesses by sound. A Cantonese-first lexicon and per-client overrides remove most of these errors. Unseen nicknames and noisy lines remain outside the current Cantonese address speech model coverage.

Can Cantonese AI voice agents handle code-switching between Cantonese and English in real time?

Real-time handling needs separate language ID embeddings per frame and a decoder trained on mixed Cantonese-English code-switching ASR data. Our production stack includes both. Even so, when the speaker switches more than once per second, the error rate rises sharply. The Cantonese AI phone agent then escalates to a human operator to protect accuracy on names and addresses.

How do speech models trained only on English data handle Cantonese tones and loanwords?

They map every tone contour to the closest English stress pattern and treat Cantonese loanwords as out-of-vocabulary tokens. The result is systematic deletion of entering tones and substitution of estate names. The Cantonese name pronunciation AI we deploy counters this in two ways. It trains on a mixed corpus and keeps a live dictionary of the 300 most common English tokens used inside Cantonese sentences.

What are the limitations of current Cantonese speech recognition models for business phone use?

Current models still fail on rare surnames, colloquial nicknames and heavy PSTN noise. No public dataset tags these variants at scale, so the Cantonese English code-switching ASR ceiling remains. The Cantonese AI voice agent therefore works as a triage layer. It manages routine traffic and hands off the rest to maintain PDPO and OFCA compliance in Hong Kong clinics and agencies.

Talk to us

Want to see how this would work in your business? Tell us what you need and we will get back to you within 24 hours.

Tell us about your project

Our service: Ad Response Agent. Calls every new Meta, Google or TikTok lead-form enquiry within a minute, qualifies it, and books the meeting.

Voice agents · AI receptionist for clinics: PDPO call flow in Hong Kong · Answering Service Hong Kong: AI Voice Agent vs Traditional · PDPO Call Recording Hong Kong Checklist · Retell vs Vapi: An 8-Criteria Production Teardown · More articles · Talk to our team