Efficient Adaptation of LLMs for Hate Speech Detection in Low-Resource Languages: A Comparative Study on Roman Urdu
For catching hate speech in 'Roman Urdu' (Urdu written in Latin letters), fine-tuning a tiny fraction of an AI model beats just asking it directly
Roman Urdu, the informally spelled version of Urdu used across South Asian social media, has no fixed spelling rules, making it hard for AI to detect hate speech in it. Researchers tested six language models (Mistral, LLaMA, Falcon, Gemma, DeepSeek, multilingual BERT) both as-is (zero-shot) and after fine-tuning them with LoRA, a method that trains only a small add-on layer instead of the whole model. Zero-shot models scored a middling F1 of about 0.56 at best, while LoRA fine-tuning pushed the best model, Mistral-7B, to an F1 above 0.93.
METAL MEDIA explanatory visual
For catching hate speech in 'Roman Urdu' (Urdu written in Latin letters), fine-tuning a tiny fraction of an AI model beats just asking it directly
- 01Tested on PURUTT, a dataset of over 72,000 Roman Urdu social media comments (about 18% labeled toxic, 82% non-toxic)
- 02Compared six transformer models under two conditions: zero-shot (no training) and LoRA-based fine-tuning that updates only around 7 million parameters (a small slice of the full model) using 4-bit quantization to save memory
- 03Zero-shot models struggled, often over-flagging comments as toxic, with F1 scores ranging from 0.15 to 0.56 depending on the model
- 04After LoRA fine-tuning, F1 scores jumped to between 0.75 and 0.94 across all models, with Mistral-7B-v0.3 topping the list at 0.9387
- 05Class weighting was used to correct for the dataset having far more non-toxic than toxic examples
What they did
- Tested on PURUTT, a dataset of over 72,000 Roman Urdu social media comments (about 18% labeled toxic, 82% non-toxic)
- Compared six transformer models under two conditions: zero-shot (no training) and LoRA-based fine-tuning that updates only around 7 million parameters (a small slice of the full model) using 4-bit quantization to save memory
- Zero-shot models struggled, often over-flagging comments as toxic, with F1 scores ranging from 0.15 to 0.56 depending on the model
- After LoRA fine-tuning, F1 scores jumped to between 0.75 and 0.94 across all models, with Mistral-7B-v0.3 topping the list at 0.9387
- Class weighting was used to correct for the dataset having far more non-toxic than toxic examples
Why it matters
This shows a practical path for building hate-speech filters in languages that lack large labeled datasets and where organizations can't afford to retrain an entire large model. It suggests that lightweight fine-tuning, rather than massive compute, may be the more realistic route to real-world content moderation in low-resource languages beyond Roman Urdu.
Terms in this paper
- Low-Resource Language (LRL) · A language with little annotated data or NLP tooling available, making it harder for AI models to handle well
- Zero-shot inference · Using a pretrained model to make predictions on a new task without any additional training on that task
- LoRA (Low-Rank Adaptation) · A technique that fine-tunes a model by adding small trainable matrices to its layers while freezing the original weights, cutting training cost
- PEFT (Parameter-Efficient Fine-Tuning) · A family of methods that adapt large models by updating only a small subset of parameters instead of the entire model
- F1-score · A single accuracy metric combining precision and recall, useful when classes are imbalanced
- 4-bit NF4 quantization · Compressing a model's numbers into a smaller 4-bit format to reduce memory use during training/inference
Original abstract (English)
It is challenging to detect hate speech in Low Resource Languages (LRLs) because of the absence of annotated data, the informality of its language structure, and the lack of standardized grammar. A good example of such a challenge is Roman Urdu which is broadly used by South Asians on social media and has a high variation while lacking contextually consistent spellings. The objective of this paper is to conduct a comprehensive assessment of Large Language Models (LLMs) for Hate Speech Detection (HSD) in Roman Urdu script and fine-tune these models using the Parameter-Efficient Fine-Tuning (PEFT) method called Low-Rank Adaptation (LoRA). To evaluate zero-shot inference, we benchmarked it against PEFT on different transformer models, including Mistral, LLaMA, Falcon, and multilingual BERT. Experiments are conducted on the PURUTT (Parallel Urdu and Roman Urdu Corpus for Toxic Comments and Transliteration) dataset with over 72,000 annotated comments. The results suggest that zero shot models perform moderately (F1 = 0.56), but updating a small fraction of the model trainable parameters improves the classification performance significantly (F1 > 0.93). Our results have shown that PEFT delivers outstanding performance alongside excellent computational efficiency, making it highly suitable for low-resource language processing tasks.
Read on arXivLatest papers
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the time
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papers
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad action
- EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise it
- EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknesses
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversight
- PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappears
- Automated Summarization of Financial News Using Large Language Models and Retrieval-Augmented Generation: An Early Empirical Study (Fall 2023)Testing AI summaries of stock news, the simple approach beat the trendy retrieval-based one