FrenchNews-7: Benchmarking Cross-Publisher French News Editorial Desk Classification

arXiv:2608.180972026-08-20

A benchmark that sorts French news articles into 7 editorial desks and holds up even on outlets it never saw

Researchers built FrenchNews-7 from 87,637 articles across 13 France-based outlets, deriving a seven-category editorial desk taxonomy (Société, Culture & Loisirs, International, Politique, Sport, Économie, Sciences & Technologies) directly from article URL structure. A fine-tuned CamemBERT model reached macro-F1 0.799 on four completely unseen outlets, beating zero-shot GPT-OSS-120B, Mistral Small 3.2, and Llama-3.3-70B. The economy and society categories remained the hardest, matching how much even humans disagree on where those articles belong editorially.

METAL MEDIA explanatory visual

A benchmark that sorts French news articles into 7 editorial desks and holds up even on outlets it never saw

01The authors analyzed URL path segments (slugs) from 87,637 articles across 13 French outlets like Le Monde and Le Figaro to build a 7-class editorial taxonomy grounded in how newsrooms actually organize their sections.
02For the 27.8% of articles whose URL slug was ambiguous, an LLM (GPT-OSS-120B) assigned labels, and the whole pipeline was audited with 2 human raters and 2 LLM raters for reliability.
03A CamemBERT model (a RoBERTa-style encoder pretrained specifically on French text) was fine-tuned on this data and achieved macro-F1 0.799 on 2,100 articles from four outlets never used in training, outperforming zero-shot LLM baselines GPT-OSS-120B (0.758), Mistral Small 3.2 (0.775), and Llama-3.3-70B (0.771).
04The Économie category was weakest, with recall of only 0.517 — close to the 55% agreement rate blinded human annotators showed on the same articles, suggesting the difficulty reflects genuinely inconsistent editorial conventions across publishers rather than a model shortcoming.
05The fine-tuned model and labeled dataset were released on Hugging Face so other researchers can use them for cross-outlet media diversity studies and agenda-setting research.

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

The authors analyzed URL path segments (slugs) from 87,637 articles across 13 French outlets like Le Monde and Le Figaro to build a 7-class editorial taxonomy grounded in how newsrooms actually organize their sections.
For the 27.8% of articles whose URL slug was ambiguous, an LLM (GPT-OSS-120B) assigned labels, and the whole pipeline was audited with 2 human raters and 2 LLM raters for reliability.
A CamemBERT model (a RoBERTa-style encoder pretrained specifically on French text) was fine-tuned on this data and achieved macro-F1 0.799 on 2,100 articles from four outlets never used in training, outperforming zero-shot LLM baselines GPT-OSS-120B (0.758), Mistral Small 3.2 (0.775), and Llama-3.3-70B (0.771).
The Économie category was weakest, with recall of only 0.517 — close to the 55% agreement rate blinded human annotators showed on the same articles, suggesting the difficulty reflects genuinely inconsistent editorial conventions across publishers rather than a model shortcoming.
The fine-tuned model and labeled dataset were released on Hugging Face so other researchers can use them for cross-outlet media diversity studies and agenda-setting research.

Figure 1: Row-normalized confusion matrix for CamemBERT-base on the in-distribution test set (n=13,146).

Table 1: Candidate URL families ranked by volume. The rule marks the seven-family break; lower families are absorbed into semantic parents or deferred to Bucket B. The 2,890 articles with absent or unparseable URL paths are labeled via Bucket B.
Family	Articles	% corpus	Publishers
International	16,090	18.0%	13/13
Culture/Loisirs	15,637	17.5%	13/13
Politique	11,513	12.9%	13/13
Société	11,234	12.6%	13/13
Économie	8,233	9.2%	13/13
Sport	7,178	8.0%	13/13
Sciences & Tech.	3,144	3.5%	10/13
Environnement	1,976	2.2%	10/13
Santé	1,030	1.2%	9/13
Médias	903	1.0%	11/13
Opinion/Édito	873	1.0%	6/13
Régional	1,366	1.5%	5/13
Format/Misc	5,570	6.2%	11/13

Figure 2: Per-class precision, recall, and F1 on the pooled mixed-class cross-publisher test set (n=2,100, 300 per category).

Table 2: Corpus composition by publisher. Type: NP = national press, RP = regional press, DN = digital-native, BR = broadcast web. Bucket A = deterministic slug-label share. ∗Post-deduplication; 132 duplicates removed.
Publisher	Articles	%	Type	Bucket A
Le Monde	33,090	37.8	NP	63.2
L’Express	9,093	10.4	NP	82.6
L’Humanité	7,975	9.1	NP	68.6
Le Point	7,877	9.0	NP	87.6
Le Figaro	7,270	8.3	NP	63.0
JDD	7,269	8.3	NP	100.0
La Croix	1,910	2.2	NP	52.7
Le HuffPost	2,473	2.8	DN	84.0
TF1 INFO	2,044	2.3	BR	97.4
Le Parisien	2,375	2.7	NP	82.9
Slate.fr	2,105	2.4	DN	34.3
20 Minutes	2,291	2.6	DN	94.9
Ouest-France	1,865	2.1	RP	39.0
Total	87,637∗	100	13

Figure 3: Editorial topic fingerprints (2022–2025). Predicted topic shares for 13 outlets across 7 categories, aggregated from the unlabelled longitudinal corpus. Each cell shows the percentage of articles assigned to a given topic; rows sum to 100%.

Table 3: Dataset distribution by label source. Bucket A = deterministic slug rules; Bucket B = LLM-labeled ambiguous slugs.
Category	Bucket A	Bucket B	Total	%
Société	12,666	6,658	19,324	22.1%
International	15,510	3,132	18,642	21.3%
Culture & Loisirs	13,305	5,045	18,350	20.9%
Politique	7,406	3,140	10,546	12.0%
Sport	6,478	1,926	8,404	9.6%
Économie	5,955	2,219	8,174	9.3%
Sciences & Technologies	1,982	2,215	4,197	4.8%
Total	63,302	24,335	87,637	100%
SHA-256 corpus fingerprint: c14a8ae609d009c9289393e6f0be761dda9f015aa4c17f515b09388742a2c218

Figure 4: Pairwise Jensen–Shannon divergence of outlet topic distributions over 48 months (2022–2025). Higher values indicate greater topical differentiation (see text for statistics). All values are computed from classifier predictions, not gold labels.

Table 12: Comparison of CamemBERT-base (512 tokens) with CamemBERTav2 at 512-token truncation and its native 1,024-token context. In-distribution metrics on the 13,146-article test split; cross-publisher macro-F1 on the 2,100-article held-out outlet pool.
Model	Context	In-dist. Acc	In-dist. F1	Cross-pub F1
CamemBERT-base	512 tok	0.860	0.847	0.799
CamemBERTav2	512 tok	0.856	0.843	—
CamemBERTav2	1,024 tok	0.861	0.847	0.798

Table 13: In-distribution pairwise significance (13,146 test set, seed 42). Paired bootstrap (10,000 replicates) on macro-F1; McNemar’s test on per-article correctness. Holm correction over all 12 raw p-values. Δ = macro-F1(A) − macro-F1(B). p∗<0.05 after correction.
Model A	Model B	Δ F1	Bootstrap		McNemar
			p (Holm)	95% CI	Sig.	χ2	p (Holm)	Sig.
CamemBERT-v1	CamemBERTav2	+0.004	0.086	[−0.002, 0.009]		2.11	0.147
CamemBERT-v1	mBERT	+0.014	<0.001	[0.008, 0.020]	∗	31.26	<0.001	∗
CamemBERT-v1	TF-IDF	+0.035	<0.001	[0.029, 0.042]	∗	152.34	<0.001	∗
CamemBERTav2	mBERT	+0.010	0.001	[0.005, 0.016]	∗	18.91	<0.001	∗
CamemBERTav2	TF-IDF	+0.032	<0.001	[0.025, 0.038]	∗	128.90	<0.001	∗
mBERT	TF-IDF	+0.021	<0.001	[0.015, 0.028]	∗	56.70	<0.001	∗

Why it matters

This fills a gap for French-language media research by providing a classifier that reliably generalizes across publishers, something no prior French benchmark directly targeted. It also shows a smaller, specialized fine-tuned model can beat much larger general-purpose LLMs on a focused real-world classification task.

Terms in this paper

CamemBERT · A RoBERTa-based language model pretrained on roughly 138GB of French text
slug · The section-name segment embedded in a news article's URL path, e.g. lemonde.fr/politique/
macro-F1 · An overall accuracy score that averages performance equally across all categories
zero-shot · Asking a model to classify without giving it any labeled examples first
held-out · Data kept completely separate from training, used only to test generalization

Figures we cannot republish

Figure 5: Top-10 tokens per class by aggregated Integrated Gradients attribution score (199-article stratified sample, CamemBERT-base full-text model). All leading tokens are topically interpretable; no publisher-identifying surface marker appears in any top-10 list.

See the figures in the original paper →

Original abstract (English)

We present FrenchNews-7, a cross-publisher France-based French-language news editorial desk classification benchmark combining a large multi-outlet corpus, a URL-derived seven-class taxonomy, and a fine-tuned CamemBERT classifier. Labels are assigned via a hybrid pipeline combining publisher URL slugs with LLM annotation for structurally ambiguous cases, audited through an inter-rater study (2 humans + 2 LLMs; pairwise $\kappa \geq 0.766$, human--human $\kappa = 0.806$). We evaluate lexical, multilingual, and French-specific trained classifiers under both in-distribution and held-out-publisher settings, with additional comparison against zero-shot LLM baselines (GPT-OSS-120B, Mistral Small 3.2, Llama-3.3-70B) on the held-out pool. The strongest model, CamemBERT-base on full article text, outperforms headline-only input, generalizes to unseen outlets, and exceeds all three zero-shot LLM baselines on overall recall (0.799), with the gap concentrated in the ambiguous editorial-boundary categories Economie and Societe. Cross-publisher evaluation reveals uneven boundary stability: Sport, Culture & Loisirs, and International transfer cleanly, while Economie (recall = 0.517) is close to blinded human agreement (0.55), and Societe (precision = 0.577) absorbs boundary ambiguity, both suggesting editorial conventions rather than recoverable classifier headroom. The fine-tuned CamemBERT-base model, labeled manifest, reference collection scripts, and a reliability-tier guidance table are available at https://huggingface.co/LeFrenchNewsLab/camembert-base-frenchnews7 (model) and https://huggingface.co/datasets/LeFrenchNewsLab/frenchnews-7 (dataset).

Authors · Amr Sobhy

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Amr Sobhy et al., arXiv:2608.18097, CC BY 4.0