Apple researchers have developed a lightweight way to detect code-switching—the use of two languages within one spoken utterance—directly from audio. The method could help teams find underrepresented multilingual recordings for training without first producing accurate transcripts in every language.
The researchers inspect frame-level hidden states inside a pretrained speech encoder. They identify dimensions that respond selectively to each language using monolingual recordings, then aggregate those activations across an utterance to decide whether both languages are present. Only a small code-switching dataset is needed to tune the final decision rule.
Using the Whisper-small encoder, the approach achieved a mean F1 score of 0.798 across 14 language pairs in the CS-FLEURS benchmark. A convolutional neural network baseline reached 0.776, while a linear system built on HuBERT features and trained for monolingual language identification scored 0.751. F1 combines precision and recall, so it rewards finding mixed-language speech without producing too many false positives.
The result suggests that general speech encoders already contain useful language-location signals even when they were not specifically trained for this task. The study evaluates identification of mixed-language utterances, not transcription quality or real-time product performance. Its practical value is mainly in assembling more representative training corpora for multilingual speech systems.