K-文化的一切——从回归到 K-美妆，发送到您的邮箱订阅邮件

METAL MEDIA

FrenchNews-7: Benchmarking Cross-Publisher French News Editorial Desk Classification

arXiv:2608.180972026-08-20

一个把法国新闻文章自动分到7个编辑版块的基准测试,换家媒体也照样管用

研究者从13家法国媒体收集了87,637篇文章,直接分析文章网址结构,归纳出社会、文化休闲、国际、政治、体育、经济、科技这7个编辑版块分类体系。用这些数据微调的CamemBERT模型在完全没见过的4家媒体的文章上达到宏观F1值0.799,超过了零样本运行的GPT-OSS-120B、Mistral Small 3.2和Llama-3.3-70B。经济和社会这两类最难分,连人类标注者意见都不统一,说明这更多是媒体编辑习惯不同造成的,而非模型能力问题。

METAL MEDIA 解读图

一个把法国新闻文章自动分到7个编辑版块的基准测试,换家媒体也照样管用

  1. 01研究者分析了13家法国媒体(如Le Monde、Le Figaro)共87,637篇文章的网址路径片段,据此归纳出扎根于新闻编辑室实际分类习惯的7类版块体系。
  2. 02对于网址片段含义不明确的27.8%文章,用大语言模型GPT-OSS-120B打标签,再由2名人类和2名大语言模型对整个流程进行核查。
  3. 03基于法语专门预训练的CamemBERT模型经微调后,在完全未参与训练的4家媒体的2,100篇文章上达到宏观F1值0.799,超过零样本方式的GPT-OSS-120B(0.758)、Mistral Small 3.2(0.775)和Llama-3.3-70B(0.771)。
  4. 04经济类别表现最弱,召回率只有0.517,这和蒙眼人类标注者在同一批文章上仅55%的一致率相近,说明难点在于不同媒体编辑归类习惯本身就不统一,而不是模型的局限。
  5. 05研究团队已将微调好的模型和标注数据集发布到Hugging Face,方便其他研究者用于跨媒体多样性分析和议程设置等研究。
这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

  1. 研究者分析了13家法国媒体(如Le Monde、Le Figaro)共87,637篇文章的网址路径片段,据此归纳出扎根于新闻编辑室实际分类习惯的7类版块体系。
  2. 对于网址片段含义不明确的27.8%文章,用大语言模型GPT-OSS-120B打标签,再由2名人类和2名大语言模型对整个流程进行核查。
  3. 基于法语专门预训练的CamemBERT模型经微调后,在完全未参与训练的4家媒体的2,100篇文章上达到宏观F1值0.799,超过零样本方式的GPT-OSS-120B(0.758)、Mistral Small 3.2(0.775)和Llama-3.3-70B(0.771)。
  4. 经济类别表现最弱,召回率只有0.517,这和蒙眼人类标注者在同一批文章上仅55%的一致率相近,说明难点在于不同媒体编辑归类习惯本身就不统一,而不是模型的局限。
  5. 研究团队已将微调好的模型和标注数据集发布到Hugging Face,方便其他研究者用于跨媒体多样性分析和议程设置等研究。
Figure 1: Row-normalized confusion matrix for CamemBERT-base on the in-distribution test set (n=13,146).
Figure 1: Row-normalized confusion matrix for CamemBERT-base on the in-distribution test set (n=13,146).
Table 1: Candidate URL families ranked by volume. The rule marks the seven-family break; lower families are absorbed into semantic parents or deferred to Bucket B. The 2,890 articles with absent or unparseable URL paths are labeled via Bucket B.
FamilyArticles% corpusPublishers
International16,09018.0%13/13
Culture/Loisirs15,63717.5%13/13
Politique11,51312.9%13/13
Société11,23412.6%13/13
Économie8,2339.2%13/13
Sport7,1788.0%13/13
Sciences & Tech.3,1443.5%10/13
Environnement1,9762.2%10/13
Santé1,0301.2%9/13
Médias9031.0%11/13
Opinion/Édito8731.0%6/13
Régional1,3661.5%5/13
Format/Misc5,5706.2%11/13
Figure 2: Per-class precision, recall, and F1 on the pooled mixed-class cross-publisher test set (n=2,100, 300 per category).
Figure 2: Per-class precision, recall, and F1 on the pooled mixed-class cross-publisher test set (n=2,100, 300 per category).
Table 2: Corpus composition by publisher. Type: NP = national press, RP = regional press, DN = digital-native, BR = broadcast web. Bucket A = deterministic slug-label share. ∗Post-deduplication; 132 duplicates removed.
PublisherArticles%TypeBucket A
Le Monde33,09037.8NP63.2
L’Express9,09310.4NP82.6
L’Humanité7,9759.1NP68.6
Le Point7,8779.0NP87.6
Le Figaro7,2708.3NP63.0
JDD7,2698.3NP100.0
La Croix1,9102.2NP52.7
Le HuffPost2,4732.8DN84.0
TF1 INFO2,0442.3BR97.4
Le Parisien2,3752.7NP82.9
Slate.fr2,1052.4DN34.3
20 Minutes2,2912.6DN94.9
Ouest-France1,8652.1RP39.0
Total87,637∗10013
Figure 3: Editorial topic fingerprints (2022–2025). Predicted topic shares for 13 outlets across 7 categories, aggregated from the unlabelled longitudinal corpus. Each cell shows the percentage of articles assigned to a given topic; rows sum to 100%.
Figure 3: Editorial topic fingerprints (2022–2025). Predicted topic shares for 13 outlets across 7 categories, aggregated from the unlabelled longitudinal corpus. Each cell shows the percentage of articles assigned to a given topic; rows sum to 100%.
Table 3: Dataset distribution by label source. Bucket A = deterministic slug rules; Bucket B = LLM-labeled ambiguous slugs.
CategoryBucket ABucket BTotal%
Société12,6666,65819,32422.1%
International15,5103,13218,64221.3%
Culture & Loisirs13,3055,04518,35020.9%
Politique7,4063,14010,54612.0%
Sport6,4781,9268,4049.6%
Économie5,9552,2198,1749.3%
Sciences & Technologies1,9822,2154,1974.8%
Total63,30224,33587,637100%
SHA-256 corpus fingerprint: c14a8ae609d009c9289393e6f0be761dda9f015aa4c17f515b09388742a2c218
Figure 4: Pairwise Jensen–Shannon divergence of outlet topic distributions over 48 months (2022–2025). Higher values indicate greater topical differentiation (see text for statistics). All values are computed from classifier predictions, not gold labels.
Figure 4: Pairwise Jensen–Shannon divergence of outlet topic distributions over 48 months (2022–2025). Higher values indicate greater topical differentiation (see text for statistics). All values are computed from classifier predictions, not gold labels.
Table 12: Comparison of CamemBERT-base (512 tokens) with CamemBERTav2 at 512-token truncation and its native 1,024-token context. In-distribution metrics on the 13,146-article test split; cross-publisher macro-F1 on the 2,100-article held-out outlet pool.
ModelContextIn-dist. AccIn-dist. F1Cross-pub F1
CamemBERT-base512 tok0.8600.8470.799
CamemBERTav2512 tok0.8560.843—
CamemBERTav21,024 tok0.8610.8470.798
Table 13: In-distribution pairwise significance (13,146 test set, seed 42). Paired bootstrap (10,000 replicates) on macro-F1; McNemar’s test on per-article correctness. Holm correction over all 12 raw p-values. Δ = macro-F1(A) − macro-F1(B). p∗<0.05 after correction.
Model AModel BΔ F1BootstrapMcNemar
p (Holm)95% CISig.χ2p (Holm)Sig.
CamemBERT-v1CamemBERTav2+0.0040.086[−0.002, 0.009]2.110.147
CamemBERT-v1mBERT+0.014<0.001[0.008, 0.020]∗31.26<0.001∗
CamemBERT-v1TF-IDF+0.035<0.001[0.029, 0.042]∗152.34<0.001∗
CamemBERTav2mBERT+0.0100.001[0.005, 0.016]∗18.91<0.001∗
CamemBERTav2TF-IDF+0.032<0.001[0.025, 0.038]∗128.90<0.001∗
mBERTTF-IDF+0.021<0.001[0.015, 0.028]∗56.70<0.001∗

为什么重要

此前没有专门针对跨媒体场景的法语新闻分类基准,这项工作填补了这一空白,为比较不同法国媒体报道倾向的研究提供了实用工具。它也证明了一个体量更小、经过针对性微调的模型,在具体实际任务上可以胜过体量大得多的通用大语言模型。

本文术语

  • CamemBERT · 基于约138GB法语文本预训练的RoBERTa结构语言模型
  • 网址片段(slug) · 新闻文章网址路径中表示版块名称的部分,例如lemonde.fr/politique/
  • 宏观F1(macro-F1) · 对每个类别的表现取相同权重平均后得到的整体准确度指标
  • 零样本(zero-shot) · 不给模型任何示例,只凭说明直接让它分类
  • 留出集(held-out) · 完全不参与训练、只用来测试模型泛化能力的数据

无法转载的图表

  • Figure 5: Top-10 tokens per class by aggregated Integrated Gradients attribution score (199-article stratified sample, CamemBERT-base full-text model). All leading tokens are topically interpretable; no publisher-identifying surface marker appears in any top-10 list.
在原文中查看图表 →

论文原文摘要（英文）

We present FrenchNews-7, a cross-publisher France-based French-language news editorial desk classification benchmark combining a large multi-outlet corpus, a URL-derived seven-class taxonomy, and a fine-tuned CamemBERT classifier. Labels are assigned via a hybrid pipeline combining publisher URL slugs with LLM annotation for structurally ambiguous cases, audited through an inter-rater study (2 humans + 2 LLMs; pairwise $\kappa \geq 0.766$, human--human $\kappa = 0.806$). We evaluate lexical, multilingual, and French-specific trained classifiers under both in-distribution and held-out-publisher settings, with additional comparison against zero-shot LLM baselines (GPT-OSS-120B, Mistral Small 3.2, Llama-3.3-70B) on the held-out pool. The strongest model, CamemBERT-base on full article text, outperforms headline-only input, generalizes to unseen outlets, and exceeds all three zero-shot LLM baselines on overall recall (0.799), with the gap concentrated in the ambiguous editorial-boundary categories Economie and Societe. Cross-publisher evaluation reveals uneven boundary stability: Sport, Culture & Loisirs, and International transfer cleanly, while Economie (recall = 0.517) is close to blinded human agreement (0.55), and Societe (precision = 0.577) absorbs boundary ambiguity, both suggesting editorial conventions rather than recoverable classifier headroom. The fine-tuned CamemBERT-base model, labeled manifest, reference collection scripts, and a reliability-tier guidance table are available at https://huggingface.co/LeFrenchNewsLab/camembert-base-frenchnews7 (model) and https://huggingface.co/datasets/LeFrenchNewsLab/frenchnews-7 (dataset).

作者 · Amr Sobhy

在 arXiv 阅读

最新论文

全部论文 →

METAL MEDIA 最新报道

图片来源： Amr Sobhy et al., arXiv:2608.18097, CC BY 4.0