A Speech Corpus for Mizo Automatic Speech Recognition: Whisper and SraVaani 1.0 Fine-Tuning with Morphology-Aware Evaluation

arXiv:2608.193612026-08-21

Building speech recognition for Mizo, a low-resource Indian language, by fine-tuning Whisper and SraVaani on 17.62 hours of new data

Researchers collected and cleaned 17.62 hours of Mizo speech from 200 speakers in Mizoram, India, then fine-tuned Whisper models (which don't natively support Mizo) and SraVaani 1.0 (which does) on this data. Whisper-large-v3 reached the lowest conventional word error rate of 18.08%, dropping to 7.22% under a new morphology-aware metric designed for Mizo's flexible spacing. SraVaani 1.0 improved dramatically after fine-tuning, from 58.27% to 29.45% word error rate.

METAL MEDIA explanatory visual

Building speech recognition for Mizo, a low-resource Indian language, by fine-tuning Whisper and SraVaani on 17.62 hours of new data

01200 speakers read about 8,000 Mizo sentences drawn from newspapers and court judgments via a web interface, yielding a curated corpus of 8,274 sentence-level recordings totaling 17.62 hours.
02Three Whisper model sizes (small, medium, large-v3) and the Indic multilingual model SraVaani 1.0, which already supports Mizo, were each fine-tuned and evaluated on speaker-independent train/validation/test splits.
03Because Mizo morphological boundaries can be written either as separate words or joined together, standard WER inflates errors; the authors introduced a morphology-aware WER that allows merging up to four adjacent words before comparing sequences.
04Whisper-large-v3 achieved the best results at 18.08% conventional WER and 7.22% morphology-aware WER; SraVaani 1.0 started at 58.27% WER in zero-shot testing but improved to 29.45% WER and 17.93% morphology-aware WER after Mizo-specific fine-tuning.
05Error analysis showed the untuned SraVaani 1.0 sometimes output entirely wrong scripts (Meitei-mayek or Devanagari instead of Mizo), and made many name-recognition and glottal-stop errors, most of which decreased substantially after fine-tuning.

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

200 speakers read about 8,000 Mizo sentences drawn from newspapers and court judgments via a web interface, yielding a curated corpus of 8,274 sentence-level recordings totaling 17.62 hours.
Three Whisper model sizes (small, medium, large-v3) and the Indic multilingual model SraVaani 1.0, which already supports Mizo, were each fine-tuned and evaluated on speaker-independent train/validation/test splits.
Because Mizo morphological boundaries can be written either as separate words or joined together, standard WER inflates errors; the authors introduced a morphology-aware WER that allows merging up to four adjacent words before comparing sequences.
Whisper-large-v3 achieved the best results at 18.08% conventional WER and 7.22% morphology-aware WER; SraVaani 1.0 started at 58.27% WER in zero-shot testing but improved to 29.45% WER and 17.93% morphology-aware WER after Mizo-specific fine-tuning.
Error analysis showed the untuned SraVaani 1.0 sometimes output entirely wrong scripts (Meitei-mayek or Devanagari instead of Mizo), and made many name-recognition and glottal-stop errors, most of which decreased substantially after fine-tuning.

Figure 1: Histogram of duration of speech files in the corpus.

Table 1: Data split for training, validation and testing.
	Speakers	Sentences	Hours
Training	184	7656	16.18
Validation	11	426	1.02
Testing	05	192	0.42

Figure 2: Schematic diagram showing the flow of the experimental protocol.

Table 2: Durational characteristics of the speech database
Sentences	8274
Total duration	17.62 hours
Minimum duration	0.63 seconds
Maximum duration	41.22 seconds
Mean duration	7.67 seconds
Median duration	6.94 seconds

Table 3: Characteristics of the multilingual ASR models evaluated in this study.
Model	Architecture	Parameters	Pretraining data	Mel bins
Whisper-small	Encoder–Decoder Transformer	244 M	680k h	80
Whisper-medium	Encoder–Decoder Transformer	769 M	680k h	80
Whisper-large-v3	Encoder–Decoder Transformer	1,550 M	∼5M h	128
SraVaani 1.0	FastConformer Hybrid RNNT/CTC	∼430 M	∼31k h	–

Table 4: Fine-tuning configurations used for the ASR models.
Model	Optimizer	LR	Effective batch size	Epochs	Selection
Whisper-small	Adafactor	5×10−6	16	20	Best val. WER
Whisper-medium	Adafactor	5×10−6	16	20	Best val. WER
Whisper-large-v3	Adafactor	5×10−6	16	20	Best val. WER
SraVaani 1.0	AdamW	1×10−4	16	20 + extended	Best val. WER

Table 5: Best epochs and corresponding validation WER of four fine-tuned models.
Model	Best epoch	Validation WER
Whisper-small Mizo–FT	15	28.99
Whisper-medium Mizo–FT	13	26.51
Whisper-large-v3 Mizo–FT	13	23.00
SraVaani 1.0 Mizo–FT	18	33.81

Table 6: Results of evaluation on five models.
Model	CER (%)	WER (%)	MA-WER (%)
Whisper-small Mizo–FT	04.83	24.00	11.49
Whisper-medium Mizo–FT	04.02	21.69	08.87
Whisper-large-v3 Mizo–FT	03.26	18.08	07.22
SraVaani 1.0	17.71	58.27	36.27
SraVaani 1.0 Mizo-FT	06.90	29.45	17.93

Table 7: Error distribution by model. Foreign-script outputs are reported as the number of utterances; all other entries indicate the number of error occurrences.
Model	Foreign script	Names	Glottal stops	Numeral transcripts	Code-mix error	< t Ω >
Whisper-small Mizo–FT	NIL	17	7	2	9	0
Whisper-medium Mizo–FT	NIL	18	6	5	3	0
Whisper-large-v3 Mizo–FT	4 sentences	8	4	3	2	0
SraVaani 1.0	21 sentences	44	9	19	49	31
SraVaani 1.0 Mizo–FT	NIL	24	6	2	19	4

Why it matters

This work delivers a publicly released speech corpus, fine-tuned ASR models, and a language-appropriate evaluation metric for Mizo, giving future developers of Mizo speech technology a concrete starting point. It also demonstrates that standard WER can overstate errors for languages with flexible morphological spacing, a lesson relevant to other Tibeto-Burman languages using Roman script.

Terms in this paper

WER (Word Error Rate) · a standard metric counting word-level substitutions, deletions, and insertions between recognized and reference text
Morphology-aware WER (MA-WER) · a new metric that allows merging up to four adjacent words before comparison, designed to tolerate Mizo's variable morpheme spacing
CER (Character Error Rate) · the same kind of error measurement as WER but computed at the character level instead of word level
Zero-shot evaluation · testing a pretrained model on a language without any additional fine-tuning on that language's data
Whisper / SraVaani 1.0 · Whisper is a multilingual transformer-based speech recognition and translation model; SraVaani 1.0 is a multilingual model covering several Indian languages

Original abstract (English)

This study reports the development of an Automatic Speech Recognition (ASR) system in Mizo, a low-resource language. The development included collecting 17.62 hours of speech data, curating it, and fine-tuning the Mizo ASR system with three Whisper multilingual models and with the SraVaani 1.0 Indic multilingual model. Whisper-large-v3 achieved the lowest conventional WER (18.08%), while morphology-aware evaluation yielded a WER of 7.22%. Zero-shot evaluation of the SraVaani 1.0 Indic multilingual model yielded a WER of 58.27%, while Mizo-specific fine-tuning reduced the conventional WER to 29.45% and the morphology-aware WER to 17.93%. The results demonstrate that the Whisper model can achieve a substantially low WER, even when adapted to an unseen language. In contrast, SraVaani 1.0 supports the Mizo language in its multilingual model; however, fine-tuning with carefully curated Mizo speech data substantially improves its performance.

Authors · Priyankoo Sarmah, Sanasam Ranbir Singh, Lalhmingmawia

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

NCT 127 drop BLINGY, their first full album in two years

Figures: Priyankoo Sarmah et al., arXiv:2608.19361, cc-by-nc-nd-4.0