Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox

METAL MEDIA

NE-BERT: A Multilingual Language Model for Nine Northeast Indian Languages

arXiv:2608.180942026-08-20

A from-scratch language model for 9 minority languages of Northeast India beats big multilingual models on understanding them

A researcher affiliated with MWire Labs trained a language model called NE-BERT on about 8.3 million sentences covering 9 indigenous Northeast Indian languages plus Hindi and English. To handle extremely data-poor languages like Pnar, which has only 1,002 sentences, the team used weighted sampling and built a custom tokenizer, the tool that splits text into pieces a model can process. The resulting model predicted text far better than existing Indic-focused models IndicBERT-V2 and MuRIL, by 15.97x and 7.64x respectively on average perplexity, a measure of how well a model predicts text.

METAL MEDIA explanatory visual

A from-scratch language model for 9 minority languages of Northeast India beats big multilingual models on understanding them

  1. 01Northeast India has over 200 distinct languages, yet existing multilingual language models barely cover them
  2. 02The team gathered about 8.3 million sentences across Assamese, Garo, Khasi, Meitei, Mizo, Naga, Nyishi, Pnar, and Kokborok plus Hindi and English, and upsampled ultra-scarce languages like Pnar (1,002 sentences) and Kokborok (2,463 sentences) by 100x during tokenizer training to prevent their vocabulary from being fragmented into useless character-level pieces
  3. 03Instead of the common Byte-Pair Encoding method, they chose SentencePiece Unigram to build a custom 50,368-token tokenizer, which split words 1.50x more efficiently than mBERT's tokenizer
  4. 04On perplexity (lower is better), NE-BERT averaged 2.21 across the 9 Northeast Indian languages, far ahead of IndicBERT-V2's 35.29 and MuRIL's 16.88, with especially large gaps on ultra-low-resource languages like Pnar and Nyishi where the baseline models nearly broke down
  5. 05On a real downstream task, part-of-speech tagging, NE-BERT averaged 82.4% accuracy across Khasi, Mizo, and Nagamese, beating mBERT by 9.1 percentage points and IndicBERT-V2 by 23.2 points, while training the whole model cost just $7.31 on a single A40 GPU over about 17 hours
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

  1. Northeast India has over 200 distinct languages, yet existing multilingual language models barely cover them
  2. The team gathered about 8.3 million sentences across Assamese, Garo, Khasi, Meitei, Mizo, Naga, Nyishi, Pnar, and Kokborok plus Hindi and English, and upsampled ultra-scarce languages like Pnar (1,002 sentences) and Kokborok (2,463 sentences) by 100x during tokenizer training to prevent their vocabulary from being fragmented into useless character-level pieces
  3. Instead of the common Byte-Pair Encoding method, they chose SentencePiece Unigram to build a custom 50,368-token tokenizer, which split words 1.50x more efficiently than mBERT's tokenizer
  4. On perplexity (lower is better), NE-BERT averaged 2.21 across the 9 Northeast Indian languages, far ahead of IndicBERT-V2's 35.29 and MuRIL's 16.88, with especially large gaps on ultra-low-resource languages like Pnar and Nyishi where the baseline models nearly broke down
  5. On a real downstream task, part-of-speech tagging, NE-BERT averaged 82.4% accuracy across Khasi, Mizo, and Nagamese, beating mBERT by 9.1 percentage points and IndicBERT-V2 by 23.2 points, while training the whole model cost just $7.31 on a single A40 GPU over about 17 hours
Figure 2: Representative sentences from each Northeast Indian language in our corpus. For Bengali-Assamese script languages (Assamese, Meitei), both the original script and Latin transliterations (in italics) are shown.
Figure 2: Representative sentences from each Northeast Indian language in our corpus. For Bengali-Assamese script languages (Assamese, Meitei), both the original script and Latin transliterations (in italics) are shown.
Table 1: Corpus statistics showing sentence counts, token counts for NE languages, virtual counts after weighted sampling for tokenizer training, language families, and data sources.
LanguageISOSentencesTokensVirtual CountWeightFamilySource
Anchor Languages
Hindihin3,404,007—170,2000.05×Indo-AryanHF Datasets
Englisheng500,000—100,0000.2×GermanicHF Datasets
Northeast Indian Languages
Meiteimni1,354,32342,504,1811,354,3231.0×Sino-TibetanCurated
Assameseasm1,000,00038,652,3911,000,0001.0×Indo-AryanCurated
Khasikha1,000,00017,472,6061,000,0001.0×AustroasiaticCurated
Mizolus1,000,00026,774,1641,000,0001.0×Sino-TibetanCurated
Nyishinjz55,870560,3741,117,40020.0×Sino-TibetanWMT 2025
Naganag13,918508,980278,36020.0×Sino-TibetanCurated
Garogrt10,817243,251216,34020.0×Sino-TibetanCurated
Kokboroktrp2,46389,851246,300100.0×Sino-TibetanWMT 2025
Pnarpbv1,00252,144100,200100.0×AustroasiaticCurated
NE Total4,438,393126,857,942
Overall Total8,342,400—6,583,123
Table 2: Model architecture comparison showing parameter counts, layer depth, hidden dimension, and attention heads.
ModelParamsLayersHiddenHeads
NE-BERT149M2276812
mBERT110M1276812
IndicBERT-V2237M12102416
MuRIL236M24102416
Table 3: Perplexity on 500-sentence test sets. NE-BERT achieves lowest average across all 9 NE languages.
LanguageNE-BERTIB-V2MuRILmBERT
Assamese1.769.015.621.65
Meitei1.893.773.221.44
Khasi1.272.401.931.36
Mizo1.908.957.002.44
Garo2.6426.3715.883.64
Kokborok1.729.155.412.23
Pnar2.9266.9239.655.30
Naga1.493.833.391.81
Nyishi4.33187.2075.806.05
English1.5514.815.572.64
Hindi1.4310.085.751.79
NE Avg.2.2135.2916.882.77
Overall2.1731.1415.382.76
Table 4: Tokenization fertility (tokens per word) across all 9 NE languages. Lower values indicate more efficient tokenization. NE-BERT achieves lowest average fertility across NE languages.
LanguageNE-BIB-V2MuRILmB
Assamese1.631.611.723.80
Meitei1.522.773.014.17
Khasi1.311.771.781.78
Mizo1.361.731.791.78
Garo2.372.842.802.89
Kokborok2.072.072.222.19
Pnar1.611.691.691.67
Naga1.561.931.771.97
Nyishi1.672.342.442.36
NE Avg.1.682.082.142.51
Table 5: Bits per character (BPC) across all 9 NE languages. Lower is better. NE-BERT achieves lowest average BPC.
LanguageNE-BIB-V2MuRmB
Assamese0.2300.8800.7310.442
Meitei0.2200.8150.7780.331
Khasi0.0920.4450.3360.158
Mizo0.2721.1631.0630.486
Garo0.5031.9971.6630.800
Kokborok0.2771.1360.9260.433
Pnar0.6802.7952.4461.092
Naga0.1710.7090.5940.321
Nyishi0.7893.7633.2311.306
English0.1900.8620.5580.328
Hindi0.1920.8630.6590.345
NE Avg.0.3591.5451.3070.597
Overall0.3471.4971.2710.590
Table 6: POS tagging accuracy (%) on test sets. NE-BERT outperforms all baselines across all languages.
LanguageNE-BERTmBERTIB-V2MuRIL
Khasi87.782.380.161.2
Mizo73.266.355.743.1
Nagamese86.471.341.744.8
Average82.473.359.249.7

Why it matters

This work shows that for languages with almost no digital data, careful data balancing and tokenizer design can matter more than simply scaling up model size, making it possible to build effective language models cheaply. By releasing the model, training code, and datasets under CC-BY-4.0, it gives underserved language communities a concrete starting point to build their own NLP tools.

Terms in this paper

  • Perplexity · a score measuring how well a language model predicts text; lower means better prediction
  • Tokenization fertility · the average number of pieces (tokens) a tokenizer splits each word into; lower means more efficient
  • SentencePiece Unigram · a method for splitting text into subword pieces using probabilities, which preserves complex word structures better than greedy methods
  • Weighted sampling · artificially increasing the representation of data-scarce languages during training to fix imbalance
  • Masked language modeling (MLM) · a training method where parts of a sentence are hidden and the model learns to guess the missing words

Original abstract (English)

Large pretrained language models have demonstrated remarkable capabilities across diverse languages, yet critically underrepresented low-resource languages remain marginalized. We present NE-BERT, a domain-specific multilingual encoder model trained on approximately 8.3 million sentences spanning 9 Northeast Indian languages and 2 anchor languages (Hindi, English), a linguistically diverse region with minimal representation in existing multilingual models. By employing weighted data sampling and a custom SentencePiece Unigram tokenizer, NE-BERT outperforms IndicBERT-V2 and MuRIL across all 9 Northeast Indian languages, achieving 15.97X and 7.64X lower average perplexity respectively, with 1.50X better tokenization fertility than mBERT. We address critical vocabulary fragmentation issues in extremely low-resource languages such as Pnar (1,002 sentences) and Kokborok (2,463 sentences) through aggressive upsampling strategies. Downstream evaluation on part-of-speech tagging validates practical utility on three Northeast Indian languages. We release NE-BERT, test sets, and training corpus under CC-BY-4.0 to support NLP research and digital inclusion for Northeast Indian communities.

Authors · Badal Nyalang

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Badal Nyalang et al., arXiv:2608.18094, CC BY 4.0