Transformer Models for Text Summarization: A Comparative Study of BART, BERT, and RoBERTa
Head-to-head test of three summarization AIs shows BART, which rewrites text from scratch, beats models that just pick existing sentences
Researchers compared three transformer-based summarization models, BERT, RoBERTa, and BART, on the CNN/DailyMail news dataset. BERT and RoBERTa were fine-tuned to pick out important sentences directly from articles (extractive summarization), while BART was used as a ready-made, already summarization-tuned model that writes new sentences (abstractive summarization). Scored with the ROUGE and BLEU metrics, BART came out clearly ahead of both, and RoBERTa outperformed BERT.
METAL MEDIA explanatory visual
Head-to-head test of three summarization AIs shows BART, which rewrites text from scratch, beats models that just pick existing sentences
- 01BERT and RoBERTa, which select sentences one by one ('extractive summarization'), were fine-tuned for 16 epochs on 15,000 training articles from CNN/DailyMail, while BART used an already summarization-pretrained checkpoint (BART-large-CNN) without any additional fine-tuning.
- 02All experiments ran on a CPU-only machine with no GPU, forcing smaller batch sizes and fewer training epochs than a typical setup.
- 03On the ROUGE-1 score, BART scored about 0.4117 versus RoBERTa's 0.1820 and BERT's 0.1485, a gap that held up similarly across BLEU and other ROUGE variants.
- 04RoBERTa also reached a lower final training loss (1.47) than BERT (2.11), a rough sign it fit the training data better under the same setup.
- 05Because BART generates new wording, its summaries read more fluently, but manual checks found occasional factual slips like altered numbers or invented details (hallucination); BERT and RoBERTa, which just copy original sentences, stayed more factually accurate but produced less polished summaries.
What they did
- BERT and RoBERTa, which select sentences one by one ('extractive summarization'), were fine-tuned for 16 epochs on 15,000 training articles from CNN/DailyMail, while BART used an already summarization-pretrained checkpoint (BART-large-CNN) without any additional fine-tuning.
- All experiments ran on a CPU-only machine with no GPU, forcing smaller batch sizes and fewer training epochs than a typical setup.
- On the ROUGE-1 score, BART scored about 0.4117 versus RoBERTa's 0.1820 and BERT's 0.1485, a gap that held up similarly across BLEU and other ROUGE variants.
- RoBERTa also reached a lower final training loss (1.47) than BERT (2.11), a rough sign it fit the training data better under the same setup.
- Because BART generates new wording, its summaries read more fluently, but manual checks found occasional factual slips like altered numbers or invented details (hallucination); BERT and RoBERTa, which just copy original sentences, stayed more factually accurate but produced less polished summaries.
Why it matters
This gives practical guidance for choosing a summarization model for real tasks like condensing news or reports. But since BART was compared in its off-the-shelf form while BERT and RoBERTa were specifically fine-tuned, the size of BART's advantage should be read with that mismatch in mind.
Terms in this paper
- BERT · An encoder-only language model pretrained to understand text bidirectionally, reading context from both directions
- RoBERTa · A version of BERT trained with improved methods and more data for better performance
- BART · A model with both an understanding encoder and a text-generating decoder, well suited to writing new summary text
- Extractive summarization · Building a summary by copying existing sentences straight from the source text
- Abstractive summarization · Building a summary by generating new sentences that paraphrase the source
- ROUGE / BLEU · Automated scoring metrics that measure overlap between an AI-generated summary and a human-written reference summary
- Hallucination / factual inconsistency · When an AI confidently produces details or numbers not actually present in the source text
Original abstract (English)
Text summarization refers to the task of condensing a document into a shorter version while preserving its key information. Automatic text summarization (ATS), driven by advancements in natural language processing (NLP), has developed rapidly in recent years. ATS methods are commonly categorized by input type (such as single-document or multi-document summarization) and by output type (extractive, abstractive, and hybrid). This article presents a focused review of modern summarization techniques with an emphasis on transformer based models and large language models (LLMs), specifically BERT, RoBERTa and BART. It examines their architectures, pretraining strategies, and their suitability for extractive and abstractive summarization tasks.
Read on arXivLatest papers
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the time
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papers
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad action
- EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise it
- EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknesses
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversight
- PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappears
- Automated Summarization of Financial News Using Large Language Models and Retrieval-Augmented Generation: An Early Empirical Study (Fall 2023)Testing AI summaries of stock news, the simple approach beat the trendy retrieval-based one