Cantonese AI voice agent: English model failures in Hong Kong

When we ran Whisper-large-v3 on PDPO-compliant recordings from Hong Kong clinics, the model consistently dropped tones on Cantonese surnames and flattened code-switched English loanwords. We built our AI Voice Phone Agents after seeing these exact patterns in production calls. The primary keyword Cantonese AI voice agent appears here as the core product we tuned for local speech.

Why English-trained speech models fail on Hong Kong Cantonese calls

English-trained acoustic models start from grapheme-to-phoneme rules that treat every syllable as stress-timed. Cantonese is syllable-timed with six tones that change lexical meaning. When HuBERT-large, trained only on English Librispeech, encounters a Cantonese tone contour it maps the nearest English vowel cluster instead. The result is systematic deletion of the entering tone on surnames such as Chan or Leung. We observed the same bias when we passed mixed audio through the base Whisper checkpoint: the decoder chose English subword tokens even when the audio contained clear Cantonese phonemes. Cantonese English code-switching ASR therefore inherits the English bias twice—once in the encoder and again in the language model head. No amount of fine-tuning on pure Cantonese corpora removes the English prior entirely because the model still expects English phonotactics at every frame boundary.

Concrete failure modes: names, addresses, and mid-sentence code-switching

In field recordings a caller saying “Wong Tai Sin MTR station, block C, flat 12” produced three distinct errors. The model rendered “Wong” as “one”, dropped the tone on “Tai”, and turned the estate name into “Wong Tie Sin”. The Cantonese address speech model had never seen the full compound “Wong Tai Sin” paired with the English token “block”. Mid-sentence code-switching made it worse: when the caller switched to “level twelve please” the recogniser inserted an extra syllable because the preceding Cantonese tone lowered the pitch baseline. Another common case involved building names such as “The Pavilia Hill”. The model output “The Pavilion Hill” because the Cantonese name pronunciation AI had no entry for the loanword “Pavilia”. In every case the Cantonese AI phone agent lost the second half of the address once the speaker switched languages inside the same breath group. These failures are reproducible on any English-only checkpoint; they are not random noise.

Four fixes we ship in our Cantonese AI voice agent stack

First we replaced the English G2P with a Cantonese-first lexicon built from Jyutping plus 4 800 Hong Kong-specific proper nouns. Second we trained a mixed Cantonese-English acoustic model on the MCE corpus style, forcing the encoder to keep separate language ID embeddings for each frame. Third we added an explicit loanword list containing the 300 most frequent English tokens used inside Cantonese sentences so the decoder no longer treats them as out-of-vocabulary. Fourth we expose a live per-client pronunciation override dictionary that clinic staff can update in the dashboard before each shift. Together these changes raised word accuracy on code-switched address utterances from the low sixties to the mid-eighties on our internal test set, yet we still record residual errors on rare surnames. The Cantonese AI voice agent therefore ships with an explicit human handoff rule whenever the override dictionary returns no match.

What we still cannot fix (yet): open problems in Cantonese AI voice agents

Three classes of error remain outside current engineering reach. Unseen mixing styles appear when callers use colloquial nicknames such as “Ah Ming” instead of the full given name; no public dataset tags these variants. Rare surnames like “Ng” or “Yip” still trigger deletion because the training distribution under-samples them. Noisy PSTN lines further degrade tone detection, and we have no reliable way to distinguish line noise from a low-register tone. The Cantonese English code-switching ASR literature documents the same ceiling: word error rates plateau once code-switch density exceeds one switch per second. Until larger, annotated Hong Kong voice bot Cantonese corpora exist, these edge cases will continue to require human escalation. We therefore treat the Cantonese AI voice agent as a triage layer rather than a complete replacement.

Operational implications for clinics, law firms and agencies in Hong Kong

Clinics can safely route simple appointment confirmations through the Cantonese AI voice agent because the vocabulary stays within the tuned lexicon. Law firms booking initial consultations still need human failover once the caller mentions a specific case number or address that the override dictionary does not contain. Agencies handling property viewings must design PDPO-compliant call flows that record consent before any address capture; the same flow must trigger an immediate transfer when the Cantonese address speech model returns low . In all three sectors the practical rule is identical: let the Cantonese AI phone agent handle the first thirty seconds, then hand off if any name, address or code-switched token falls outside the four fixes. This keeps compliance with OFCA caller-ID rules while respecting the documented limits of current Cantonese name pronunciation AI.

Conclusion

The Cantonese AI voice agent we run today is the direct result of measuring English-model failures on real Hong Kong calls and shipping the four targeted fixes. It handles the majority of routine traffic yet still surfaces the cases that require human judgement. That boundary is the operating reality for any business that fields Cantonese calls today.

Call to Action

Hear a Cantonese demo call on our production stack. Start here.

FAQ

How accurate are Cantonese AI voice agents for mixed Cantonese-English calls?

Accuracy depends on whether the stack uses a mixed-language acoustic model and an explicit loanword list. Without those components, English-trained models drop tones and insert wrong subwords on every code-switch. With the four fixes we ship, word accuracy on address utterances moves into the mid-eighties on our test set, yet rare surnames and heavy noise still trigger handoff. The Cantonese AI voice agent therefore includes a live override dictionary that staff can edit before each shift.

Why do AI phone agents mispronounce Hong Kong street names and building names?

English G2P rules map Cantonese tones to the nearest English vowel cluster. Estate names such as Wong Tai Sin or The Pavilia Hill have no entry in standard lexicons, so the decoder guesses phonetically. Adding a Cantonese-first lexicon and per-client overrides removes most of these errors, but unseen nicknames and noisy lines remain outside the current Cantonese address speech model coverage.

Can Cantonese AI voice agents handle code-switching between Cantonese and English in real time?

Real-time handling requires separate language ID embeddings per frame and a decoder trained on mixed Cantonese-English code-switching ASR data. Our production stack includes both. Even so, when the speaker switches more than once per second the error rate rises sharply and the Cantonese AI phone agent escalates to a human operator to preserve accuracy on names and addresses.

How do speech models trained only on English data handle Cantonese tones and loanwords?

They map every tone contour to the closest English stress pattern and treat Cantonese loanwords as out-of-vocabulary tokens. The result is systematic deletion of entering tones and substitution of estate names. The Cantonese name pronunciation AI we deploy counters this by training on a mixed corpus and maintaining a live dictionary of the 300 most common English tokens used inside Cantonese sentences.

What are the limitations of current Cantonese speech recognition models for business phone use?

Current models still fail on rare surnames, colloquial nicknames and heavy PSTN noise. No public dataset tags these variants at scale, so the Cantonese English code-switching ASR ceiling remains. The Cantonese AI voice agent therefore functions as a triage layer: it manages routine traffic and hands off the rest to maintain PDPO and OFCA compliance in Hong Kong clinics and agencies.

Hear it for yourself

The fastest way to judge an AI receptionist is to call one. Our live demo agent answers 24/7 — ask it whatever you would ask your own front desk.

Hong Kong: +852 9290 6024
United Kingdom: +44 1865 537191
United States: +1 267 507 0109

Prefer to speak to a person? Book a walkthrough.

Voice agents · AI receptionist for clinics: PDPO call flow in Hong Kong · Answering Service Hong Kong: AI Voice Agent vs Traditional · More articles · Talk to our team