Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox›
Tangut Word Segmentation under Extreme Resource Scarcity: Integrating Traditional Lexicons and Unlabeled Text
arXiv:2608.184372026-08-20
Teaching AI to find word boundaries in Tangut, an extinct script with no spaces and no native speakers
Tangut script never marks where one word ends and the next begins, and since the language is extinct with no native speakers, figuring out word boundaries is unusually hard. Researchers combined 2,750 expert-segmented passages (31,893 tokens), a traditional dictionary, and unlabeled Tangut text to tackle this for the first time systematically. A CRF model using lexicon and statistical features reached about 0.91 F1, and their lightweight TangutEncoder model reached the best mean F1 of 0.911 while also being better at recognizing words never seen in training.
METAL MEDIA explanatory visual
Teaching AI to find word boundaries in Tangut, an extinct script with no spaces and no native speakers
01Tangut specialists at the Chinese Academy of Social Sciences manually segmented a Buddhist scripture (Mahāratnakūṭa-sūtra) and a secular encyclopedia (Leilin) into 2,750 annotated segments totaling 31,893 word tokens to serve as ground truth.
02Instead of picking just one dictionary match when candidates overlap, the team built a 'lexicon-lattice' representation that keeps all overlapping dictionary candidates and calibrates a reliability score for each based on how well it matches actual gold-standard boundaries.
03From about 318,000 characters of unlabeled Tangut text, they extracted statistics like bigram frequency, character association, and neighbor entropy, and also pretrained a compact character-level encoder called TangutEncoder (about 2.5 million parameters) using masked-language-model pretraining.
04Under five-fold cross-validation, the CRF with lexicon and statistical features reached roughly 0.91 F1, while the full TangutEncoder system achieved the highest mean F1 (0.911) and better recall on out-of-vocabulary (OOV) words not seen during training.
05The corpus is limited to just two works and skewed toward secular text, so while it generalizes well to held-out passages from the same sources, transfer to entirely new documents remains untested.
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.
What they did
Tangut specialists at the Chinese Academy of Social Sciences manually segmented a Buddhist scripture (Mahāratnakūṭa-sūtra) and a secular encyclopedia (Leilin) into 2,750 annotated segments totaling 31,893 word tokens to serve as ground truth.
Instead of picking just one dictionary match when candidates overlap, the team built a 'lexicon-lattice' representation that keeps all overlapping dictionary candidates and calibrates a reliability score for each based on how well it matches actual gold-standard boundaries.
From about 318,000 characters of unlabeled Tangut text, they extracted statistics like bigram frequency, character association, and neighbor entropy, and also pretrained a compact character-level encoder called TangutEncoder (about 2.5 million parameters) using masked-language-model pretraining.
Under five-fold cross-validation, the CRF with lexicon and statistical features reached roughly 0.91 F1, while the full TangutEncoder system achieved the highest mean F1 (0.911) and better recall on out-of-vocabulary (OOV) words not seen during training.
The corpus is limited to just two works and skewed toward secular text, so while it generalizes well to held-out passages from the same sources, transfer to entirely new documents remains untested.
Figure 1: Surviving fragment of Mahāratnakūṭa-sūtra, Scroll 68.
Table 1: Statistics of the expert-annotated Tangut corpus.
Category
Segments
Tokens
Types
Buddhist scriptures
234
3,717
769
Secular documents
2,516
28,176
4,046
Total
2,750
31,893
4,433
Figure 2: Annotation examples from the religious and secular portions of the corpus. Vertical bars indicate expert-annotated word boundaries.
Table 2: Composition of the 20-dimensional lexicon representation.
Symbol
Feature group
Dim.
BIE
Lexicon-lattice indicators
11
R
Reliability of observed entries
3
P
Prior for unseen entries
3
M
Lexicographic metadata
3
Dictall
Full lexicon representation
20
Table 3: Explicit distributional features extracted from unlabeled text.
Symbol
Feature group
Dim.
Freq
Bigram frequency
2
Coo
Character association
2
Ent
Neighbor entropy
4
Distall
Full distributional representation
8
Table 4: Overall segmentation results under five-fold cross-validation.
Model
P
R
F1
OOV-R
IV-R
Supervised baselines (RQ1)
Dict-corpus
0.845±0.005
0.890±0.003
0.867±0.004
0.219±0.014
0.954±0.002
CRF
0.880±0.004
0.888±0.003
0.884±0.003
0.415±0.026
0.933±0.003
BiLSTM–CRF
0.862±0.008
0.873±0.008
0.868±0.008
0.472±0.014
0.912±0.006
Transformer-Random
0.848±0.008
0.863±0.010
0.855±0.009
0.519±0.020
0.896±0.010
Lexicon-lattice ablation (RQ2)
CRF+BIE
0.898±0.004
0.897±0.002
0.897±0.003
0.489±0.015
0.936±0.002
+ BIE + R
0.902±0.005
0.898±0.003
0.900±0.004
0.473±0.021
0.939±0.002
+ BIE + R + P
0.900±0.003
0.900±0.005
0.900±0.004
0.463±0.020
0.941±0.003
+ Dictall
0.903±0.005
0.902±0.003
0.902±0.004
0.467±0.021
0.944±0.002
Explicit distributional features (RQ3)
CRF+ Dictall + Freq
0.905±0.003
0.904±0.003
0.904±0.003
0.483±0.011
0.944±0.002
+ Dictall + Freq + Coo
0.906±0.003
0.905±0.003
0.905±0.003
0.486±0.009
0.945±0.001
+ Dictall + Distall
0.907±0.004
0.906±0.004
0.907±0.004
0.491±0.016
0.947±0.002
Representation learning and fusion (RQ4–RQ5)
Transformer-Char2Vec
0.844±0.005
0.860±0.012
0.852±0.008
0.491±0.010
0.895±0.012
TangutEncoder
0.881±0.004
0.885±0.002
0.883±0.003
0.570±0.011
0.915±0.004
TangutEncoder + Dictall
0.902±0.003
0.913±0.006
0.907±0.004
0.606±0.014
0.942±0.005
TangutEncoder + Dictall + Distall
0.905±0.003
0.916±0.004
0.911±0.003
0.608±0.014
0.946±0.003
Table 5: Mean performance by document genre.
Secular
Religious
Model
F1
OOV-R
IV-R
F1
OOV-R
IV-R
Dict-corpus
0.869±0.004
0.229±0.017
0.956±0.003
0.855±0.019
0.132±0.028
0.936±0.010
CRF
0.887±0.010
0.430±0.035
0.934±0.007
0.839±0.011
0.262±0.067
0.909±0.009
BiLSTM–CRF
0.875±0.007
0.498±0.016
0.917±0.006
0.810±0.021
0.244±0.028
0.875±0.021
Transformer-Random
0.868±0.008
0.545±0.018
0.906±0.010
0.760±0.014
0.280±0.060
0.816±0.017
CRF+Dictall
0.904±0.005
0.481±0.024
0.942±0.004
0.872±0.004
0.301±0.058
0.938±0.008
CRF+Dictall + Distall
0.910±0.005
0.506±0.022
0.947±0.002
0.881±0.005
0.355±0.060
0.939±0.008
Transformer-Char2Vec
0.866±0.008
0.514±0.011
0.908±0.012
0.749±0.016
0.284±0.033
0.800±0.022
TEnc
0.892±0.002
0.587±0.016
0.922±0.004
0.817±0.016
0.422±0.039
0.863±0.015
TEnc + Dictall
0.915±0.004
0.633±0.023
0.946±0.006
0.849±0.018
0.362±0.065
0.908±0.007
TEnc + Dictall + Distall
0.917±0.003
0.634±0.022
0.949±0.004
0.863±0.015
0.385±0.058
0.920±0.010
Table 6: Normalized overlap between annotated and unlabeled text.
Audit unit
All
Religious
Secular
Exact segment (≥3 chars)
0
0
0
5-gram overlap (%)
0.37
2.66
0.02
10-gram overlap (%)
0.01
0.08
0.00
Table 7: Normalized linguistic-label inventory.
Group
Labels
Core lexical categories
a, c, d, m, n, p, q, r, u, v
Fine-grained lexical labels
b, l, t, nb, nc, nh, nl, no, ns, mc, mo, rd, ri, rp
Table 8: Hyperparameters of the linear CRF. All reported CRF results use a fixed budget of 200 L-BFGS iterations.
Parameter
Value
L1 coefficient (c1)
1.0
L2 coefficient (c2)
10−3
Maximum L-BFGS iterations
200
Validation-based stopping
Not used
All possible transitions
Enabled
Character context window
±2 characters
Inner folds for OOF reliability
5
Reliability smoothing (κ)
5
Table 9: Preliminary hyperparameter search for the supervised BiLSTM–CRF. Emb. denotes the character-embedding dimension, and Hidden denotes the concatenated output dimension of the two directions. The best segmentation F1 is highlighted.
Layers
Emb.
Hidden
Batch
Dropout
LR
F1
2
100
256
512
0.3
5×10−4
0.8636
2
100
128
512
0.3
5×10−4
0.8595
2
100
64
512
0.3
𝟓×𝟏𝟎−𝟒
0.8905
2
100
32
512
0.3
5×10−4
0.8873
2
100
16
512
0.3
5×10−4
0.8423
1
100
64
512
0.3
5×10−4
0.8638
1
100
128
512
0.3
5×10−4
0.8569
3
100
64
512
0.3
5×10−4
0.8659
2
100
64
1024
0.3
5×10−4
0.8829
2
100
64
256
0.3
5×10−4
0.8822
2
100
64
128
0.3
5×10−4
0.8861
2
100
64
64
0.3
5×10−4
0.8870
2
100
64
32
0.3
5×10−4
0.8900
2
100
64
16
0.3
5×10−4
0.8854
2
100
64
32
0.3
1×10−4
0.8847
2
32
64
512
0.3
5×10−4
0.8382
2
64
64
512
0.3
5×10−4
0.8496
2
128
64
512
0.3
5×10−4
0.8814
2
256
64
512
0.3
5×10−4
0.8755
2
100
64
512
0.1
5×10−4
0.8824
2
100
64
512
0.5
5×10−4
0.8870
Table 10: Selected architectural and training parameters of the BiLSTM–CRF. The maximum number of epochs is only an upper bound; the checkpoint with the lowest development loss is restored after early stopping.
Parameter
Selected value
Character embedding size
100
BiLSTM layers
2
Hidden size
32 per direction
BiLSTM output size
64
Emission size
4
Model dropout
0.3
Dictionary dropout
0.2
Parameter initialization
Xavier uniform
Optimizer
Adam
Learning rate
5×10−4
Batch size
512 sentences
Maximum epochs
10,000
Early-stopping patience
3 epochs
Selection criterion
Development NLL
Gradient clipping
5.0
Table 11: Shared architecture of the Transformer–CRF variants. The two external-feature projections are included only in the corresponding fusion models.
Parameter
Value
Character embedding size
192
Maximum sequence length
128
Transformer layers
3
Attention heads
4
Dimension per head
48
Feed-forward size
768
Activation
GELU
Normalization
Pre-LN
Encoder dropout
0.15
Task-head dropout
0.20
Position embeddings
Learned
Dictionary projection
20→32
Distributional projection
8→16
Output layer
Linear + CRF
Table 12: Training parameters of the static Char2Vec initialization.
Char2Vec parameter
Value
Training objective
Skip-gram
Vector size
192
Context window
5
Negative samples
10
Minimum frequency
1
Training epochs
30
Workers
1
Random seed
42
Minimum sequence length
2 characters
Table 13: Masked-language-model pretraining parameters of TangutEncoder. Pretraining completed the full 5,000 steps, and the checkpoint with the lowest validation loss was retained.
MLM parameter
Value
Masking ratio
0.15
Span-masking probability
0.50
Span length
2–4 characters
Replacement strategy
80/10/10
Optimizer
AdamW
Learning rate
3×10−4
Weight decay
0.01
Batch size
32 sequences
Maximum steps
5,000
Warm-up steps
500
Learning-rate schedule
Warm-up then constant
Evaluation interval
200 steps
Early-stopping patience
5 evaluations
Gradient clipping
1.0
Validation split
5 held-out UUIDs
Table 14: Downstream training parameters shared by Transformer–Random, Transformer–Char2Vec and TangutEncoder.
Fine-tuning parameter
Value
Frozen-encoder epochs
3
Optimizer
AdamW
Encoder learning rate
5×10−5
Task-head learning rate
5×10−4
Weight decay
0.01
Batch size
32 sentences
Maximum fine-tuning epochs
10,000
LR-reduction patience
5 epochs
LR-reduction factor
0.3
Early-stopping patience
10 epochs
Selection criterion
Development NLL
Gradient clipping
1.0
Table 15: Comparison of association measures under five-fold cross-validation.
Association features
P
R
F1
OOV-R
IV-R
Freq + dPMI
0.906±0.003
0.905±0.003
0.905±0.003
0.486±0.009
0.945±0.001
Freq + dPMI + Ent
0.907±0.004
0.906±0.004
0.907±0.004
0.491±0.016
0.947±0.002
Freq + Dice
0.905±0.003
0.905±0.004
0.905±0.003
0.485±0.010
0.945±0.002
Freq + Dice + Ent
0.907±0.004
0.906±0.004
0.906±0.004
0.489±0.016
0.946±0.002
Freq + t-score
0.905±0.004
0.904±0.004
0.904±0.004
0.483±0.014
0.944±0.003
Freq + t-score + Ent
0.907±0.003
0.906±0.004
0.907±0.004
0.490±0.013
0.946±0.002
Table 16: Bidirectional maximum matching with different dictionary sources. Values are five-fold means.
Model
P
R
F1
OOV-R
IV-R
Dict-corpus
.845
.890
.867
.219
.954
Dict-dictionary
.787
.728
.756
.666
.733
Dict-all
.821
.740
.779
.665
.748
Table 17: BiLSTM–CRF variants with progressively richer external features.
Model
External features
P
R
F1
OOV-R
IV-R
BiLSTM
0
0.862±0.008
0.873±0.008
0.868±0.008
0.472±0.014
0.912±0.006
BiLSTM+BIE
11
0.885±0.005
0.894±0.004
0.889±0.004
0.546±0.011
0.927±0.004
BiLSTM+BIE+R
14
0.891±0.003
0.898±0.004
0.894±0.004
0.546±0.009
0.931±0.003
BiLSTM+Dictall
17
0.893±0.003
0.900±0.003
0.897±0.003
0.551±0.011
0.933±0.003
+ Dictall + IL
28
0.896±0.003
0.900±0.002
0.898±0.002
0.557±0.009
0.933±0.002
+ Dictall + IL + DD
30
0.897±0.004
0.904±0.003
0.900±0.003
0.554±0.006
0.938±0.002
+ Dictall + IL + DD + Freq
30+2
0.902±0.004
0.909±0.005
0.906±0.004
0.560±0.008
0.943±0.005
+ Dictall + IL + DD + Freq + Coo
30+4
0.902±0.004
0.909±0.004
0.905±0.003
0.563±0.016
0.941±0.004
+ Dictall + IL + DD + Distall
30+8
0.902±0.003
0.909±0.004
0.905±0.003
0.562±0.014
0.942±0.004
Table 18: Genre-level results for the BiLSTM–CRF feature variants.
Secular
Religious
Model
F1
OOV-R
F1
OOV-R
BiLSTM
0.875±0.007
0.498±0.016
0.810±0.021
0.244±0.028
BiLSTM+BIE
0.897±0.005
0.573±0.015
0.830±0.018
0.305±0.043
BiLSTM+BIE+R
0.903±0.004
0.574±0.014
0.829±0.013
0.286±0.053
BiLSTM+Dictall
0.905±0.003
0.579±0.014
0.835±0.016
0.298±0.037
+ Dictall + IL
0.905±0.002
0.586±0.011
0.842±0.018
0.300±0.043
+ Dictall + IL + DD
0.907±0.001
0.586±0.008
0.850±0.021
0.270±0.041
+ Dictall + IL + DD + Freq
0.911±0.003
0.588±0.012
0.862±0.019
0.316±0.056
+ Dictall + IL + DD + Freq + Coo
0.911±0.002
0.591±0.023
0.859±0.026
0.322±0.076
+ Dictall + IL + DD + Distall
0.911±0.001
0.589±0.020
0.860±0.021
0.326±0.051
Table 19: Downstream results of lexicon-aware continued pretraining. Both encoders are evaluated with the same dictionary and distributional features.
Pretraining
F1
OOV-R
IV-R
MLM
0.911±0.003
0.608±0.014
0.946±0.003
MLM + WordRank
0.912±0.003
0.617±0.012
0.946±0.003
Why it matters
Because Tangut has no native speakers and only scarce digitized resources, this work shows a practical way to build word segmentation by combining traditional dictionaries, unlabeled text, and a small amount of expert annotation. Word segmentation is a prerequisite for nearly everything else in Tangut digital research—search, part-of-speech tagging, alignment, translation—so this approach offers a template that could also help other severely under-resourced historical languages.
Terms in this paper
BIES tagging · Labeling each character as the Beginning, Inside, or End of a multi-character word, or a Single-character word, to mark boundaries
CRF (Conditional Random Field) · A statistical model that picks the most likely sequence of labels by considering neighboring characters
lexicon-lattice · A representation that keeps all overlapping dictionary word-candidates at a position instead of forcing a single choice
MLM (masked language model) pretraining · Training a model to predict hidden characters from context, which teaches it contextual understanding of text
OOV (out-of-vocabulary) · A word that never appeared in the training data, so the model has to guess it purely from context or outside knowledge
Original abstract (English)
Tangut is an extinct language whose script does not explicitly mark word boundaries. We present the first systematic study of Tangut word segmentation using 2,750 expert-annotated segments(31,893 tokens), traditional lexicons, and unlabeled text. Our framework combines a reliability-calibrated lexicon-lattice representation, explicit distributional statistics, and a lightweight character encoder pretrained with MLM. Segment-level five-fold cross-validation shows that lexical and statistical features raise CRF F1 to approximately 0.91. The full TangutEncoder reaches the highest mean F1 (0.911) and improves recall beyond the labeled training vocabulary. These results demonstrate generalization beyond the limited supervised vocabulary across thematically diverse held-out passages, while document-level transfer remains to be evaluated.
