Table 1: Comparison of value frameworks used for AI evaluation. ✓ = full support, ⚫ = partial, ✗ = not supported. Macro. (Macro-societal): targets societal-level values rather than individual ethics or safety. Multi. (Multimodal): operationalized on image–text content. Quant. (Quantitative): yields ordinal or continuous scores rather than categorical labels.
Framework
Macro.
Multi.
Quant.
ETHICS Hendrycks et al. (2020)
✗
✗
✓
BBQ Parrish et al. (2022)
✗
✗
✓
Constitutional AI Bai et al. (2022)
✗
✗
✗
Ch3Ef Shi et al. (2024)
✗
✓
✓
VIVA Hu et al. (2024)
✗
✓
✗
ValueBench Ren et al. (2024)
✗
✗
✓
CLAVE Yao et al. (2024)
✗
✗
✓
SPA-VL Zhang et al. (2025)
✗
✓
✗
Safe RLHF-V Ji et al. (2025)
✗
✓
✗
MORALISE Lin et al. (2025)
✗
✓
✗
HumaniBench Raza et al. (2025)
⚫
✓
✗
ValueCompass Shen et al. (2025)
✗
✗
✓
MAVEN (ours)
✓
✓
✓
Table 2: Theoretical foundations of Maven’s six primary dimensions. UDHR/ICCPR/ICESCR numbers refer to article numbers. Pre. denotes Preamble. Schwartz value abbreviations: Univ. = Universalism, Benev. = Benevolence, Conf. = Conformity, Self-dir. = Self-direction, Stim. = Stimulation.
Dimension
Instruments
Schwartz
Peace
UN Charter; UDHR Pre.
Security
Development
UDHR 22, 25; ICESCR
Universalism
Equity
UDHR 1, 7; ICESCR 2
Univ., Benev.
Justice
UDHR 6–11; ICCPR 14
Univ., Conf.
Democracy
UDHR 21; ICCPR 25
Self-dir.
Freedom
UDHR 18–20; ICCPR 18–22
Self-dir., Stim.
Table 3: MacroValue-Bench data sources.
Source
Count
%
Weibo
573
49.5
Global Times Online
245
21.2
Ch3Ef-harmless Shi et al. (2024)
173
15.0
People’s Daily Online
166
14.3
Total
1,157
100.0
Table 4: Performance on MacroValue-Bench. All metrics are higher-is-better (↑); bold indicates the best overall; green percentages report the relative gain over the preceding row. Doubao-Seed-1.6-V leads across all six metrics among commercial models, while our compact 2B evaluator matches Qwen3-VL-8B in the same family and approaches frontier commercial VLMs at a 4× smaller parameter count. SA-MDPO alone delivers the bulk of the gain (+52.4% QWK over the baseline), and MRC adds a further +4.2% QWK to reach 0.624.
Model
QWK ↑
Acc. ↑
F1macro ↑
VSMS ↑
Recall ↑
Precision ↑
Commercial Models
GPT-4o Hurst et al. (2024)
0.563
0.471
0.346
0.919
0.919
0.919
GPT-5 Singh et al. (2025)
0.637
0.724
0.520
0.962
0.962
0.962
Gemini-2.5-Pro Comanici et al. (2025)
0.651
0.646
0.471
0.950
0.950
0.950
Moonshot-V1-Vision Team et al. (2025)
0.436
0.673
0.321
0.846
0.846
0.846
Hunyuan-T1-Vision Team (2025b)
0.588
0.689
0.441
0.933
0.933
0.933
Qwen3-VL-Plus Bai et al. (2025a)
0.638
0.653
0.469
0.935
0.935
0.935
Doubao-Seed-1.6-V ByteDance (2025)
0.702
0.756
0.524
0.966
0.966
0.966
Open-source Models
GLM-4.5V Hong et al. (2025)
0.686
0.739
0.500
0.957
0.957
0.957
Qwen2.5-VL-7B Bai et al. (2025b)
0.245
0.617
0.225
0.921
0.913
0.945
Qwen2.5-VL-3B Bai et al. (2025b)
0.238
0.630
0.230
0.951
0.944
0.958
Qwen3-VL-8B Bai et al. (2025a)
0.546
0.691
0.380
0.961
0.961
0.961
Qwen3-VL-4B Bai et al. (2025a)
0.442
0.663
0.322
0.954
0.954
0.954
Qwen3-VL-2B Bai et al. (2025a)
0.393
0.497
0.300
0.798
0.798
0.798
Ours
Qwen3-VL-2B + SA-MDPO
0.599 (+52.4%)
0.706 (+42.1%)
0.482 (+60.7%)
0.954 (+19.5%)
0.954 (+19.5%)
0.954 (+19.5%)
Qwen3-VL-2B + SA-MDPO + MRC
0.624 (+4.2%)
0.719 (+1.8%)
0.497 (+3.1%)
0.959 (+0.5%)
0.959 (+0.5%)
0.959 (+0.5%)
Table 5: Ablation study comparing DPO, MDPO, and SA-MDPO under different training dataset.
Method
Acc.
QWK
VSMS
Baseline
0.497
0.393
0.798
DPO (unbalanced)
0.190
0.153
0.517
DPO (balanced)
0.654
0.267
0.946
MDPO (unbalanced)
0.656
0.503
0.920
MDPO (balanced)
0.670
0.445
0.964
SA-MDPO (unbalanced)
0.666
0.518
0.934
SA-MDPO (balanced)
0.706
0.599
0.954
Table 6: Effect of Multi-Role Consensus (MRC) on Qwen3-VL-Instruct models.
Model
Acc.
QWK
VSMS
Qwen3-VL-2B (×14 runs)
0.516
0.405
0.808
Qwen3-VL-2B (MRC)
0.527
0.432
0.815
Qwen3-VL-4B (×14 runs)
0.673
0.404
0.958
Qwen3-VL-4B (MRC)
0.662
0.424
0.957
Qwen3-VL-8B (×14 runs)
0.689
0.526
0.964
Qwen3-VL-8B (MRC)
0.706
0.540
0.968
Table 7: Secondary indicators of Peace.
Indicator
Definition
Peace. Non-violence and harmonious coexistence across individual, social, national, and global levels.
World Peace
Absence of war or conflict among nations, with disputes resolved through diplomacy, cooperation, and rule of law.
Regional Peace
Stable and friendly relations among countries within a geographic region through regional cooperation mechanisms.
National Peace
Absence of large-scale violence or civil war within a country, maintained by rule of law and stable governance.
Social Peace
Elimination of violent confrontation among communities or social groups via inclusive policies and exchange.
Political Peace
Resolving political differences through democratic consultation, elections, or negotiation rather than repression.
Economic Peace
Reducing conflict caused by poverty or inequality through fair distribution and equal economic opportunities.
Cultural Peace
Mutual respect and inclusion among cultural traditions, value systems, and belief frameworks.
Cyber Peace
Security and cooperation in digital space, curbing cyberattacks, disinformation, and digital surveillance.
Ecological Peace
Peaceful resolution of environmental resource disputes and cooperation on transnational ecological issues.
Community Peace
Harmonious relationships within neighborhoods or professional groups through community governance and mediation.
Family Peace
Absence of violence or oppression among family members, achieved through equal communication and shared responsibilities.
Inner Peace
Individual psychological tranquility and balance, cultivated through self-awareness and emotion regulation.
Table 8: Secondary indicators of Development.
Indicator
Definition
Development. Continuous progress and capacity enhancement of individuals, societies, and natural systems.
Economic Development
Material wealth growth and structural optimization, including industrial upgrading and inclusive prosperity.
Social Development
Improving livelihoods through public services in education, healthcare, and housing, narrowing wealth gaps.
Personal Development
Realizing individual potential through education, skill training, and mental health support.
Technological Development
Scientific breakthroughs driving productivity, with ethical constraints and equitable benefit distribution.
Institutional Development
Improvement of laws, policies, and governance to enhance adaptability, transparency, and efficiency.
Educational Development
Equitable distribution and quality improvement of educational resources, cultivating critical thinking.
Cultural Development
Protection and innovation of cultural diversity, including artistic flourishing and heritage preservation.
Environmental Development
Coordination of ecological protection and economic growth via green technologies and circular economy.
Health Development
Strengthening public health systems and achieving universal health coverage.
Urban-Rural Development
Eliminating the urban-rural dual structure through interconnected infrastructure and equitable services.
Regional Development
Reducing regional disparities through transportation networks, industrial policies, and complementary specialization.
Political Development
Modernization and democratization of political systems, expanding citizen participation and accountability.
Table 9: Secondary indicators of Equity.
Indicator
Definition
Equity. Reasonable distribution of resources, opportunities, rights, and responsibilities among members of society.
Opportunity Equity
Equal rights in education, employment, and entrepreneurship regardless of background, gender, or location.
Procedural Equity
Fair rule enforcement: consistent legal procedures, transparent standards, and competition free from privilege.
Educational Equity
Balanced resource allocation, access, and quality in education across socioeconomic backgrounds.
Employment Equity
Hiring, promotion, and compensation based solely on ability, prohibiting discrimination by gender, age, or race.
Healthcare Equity
Universal access to high-quality medical services regardless of payment ability or location.
Judicial Equity
Equality before the law, independent and unbiased trials, and accessible legal aid for vulnerable groups.
Market Equity
Healthy competition order through anti-monopoly and anti-unfair-competition measures protecting SMEs and consumers.
Tax Equity
Ability-to-pay taxation that prevents evasion and unjustified privileges, regulating wealth gaps.
Welfare Equity
Universal social security with need-based distribution, especially protecting elderly and disabled.
Political Equity
Equal citizen participation in voting, eligibility for office, and policy influence, free from wealth-based monopoly.
Information Equity
Equal access to knowledge in the digital age, narrowing the digital divide and protecting data privacy.
Intergenerational Equity
Balancing present and future interests, avoiding overuse of resources or unsustainable debt.
Table 10: Secondary indicators of Justice.
Indicator
Definition
Justice. Institutions and actions aligned with moral and legal principles, protecting rights and balancing interests.
Legal Justice
Legislation and judiciary conform to moral reasoning, with impartial laws and proportionate punishment.
Procedural Justice
Normative and neutral decision-making with rights to hearing, recusal, and reasoned explanations.
Rights-Based Justice
Guarantee of inalienable basic rights via constitutions, especially protecting minorities from majority tyranny.
Distributive Justice
Fair sharing of social resources and wealth, opposing extreme inequality through tax and welfare policies.
Social Justice
Equal and inclusive social structure, eliminating class, racial, or gender oppression.
Economic Justice
Critique of exploitative economic relations, advocating balanced labor-capital ties and fair trade rules.
Gender Justice
Opposition to gender-based discrimination and violence, pursuing gender equality across spheres.
Environmental Justice
Fair distribution of ecological risks and benefits, opposing pollution transfer to vulnerable communities.
Historical Justice
Acknowledgment and repair of past wrongs (e.g., colonialism, genocide) through apologies and reparations.
Intergenerational Justice
Current generations’ responsibility for the future, avoiding irreversible environmental or fiscal damage.
International Justice
Challenge to hegemony and unequal global orders, advocating sovereignty equality and democratized rules.
Cultural Justice
Respect for cultural identity and expression, opposing cultural hegemony or assimilationist oppression.
Table 11: Secondary indicators of Democracy.
Indicator
Definition
Democracy. Institutionalized public participation in decision-making and oversight of power.
Popular Sovereignty
The people as the ultimate source of state power, with constitutional principles establishing this status.
Electoral Democracy
Selection of representatives or leaders through regular, competitive, and free elections.
Representative Democracy
Citizens elect representatives who must faithfully reflect public opinion and remain accountable.
Rule-of-Law Democracy
Democratic operations bound by constitutional and legal frameworks, with separation of powers and rights protection.
Constitutional Democracy
A written constitution defining government powers and citizen rights, safeguarded by judicial review.
Grassroots Democracy
Direct participation at micro-levels such as residents’ self-governance and participatory budgeting.
Participatory Democracy
Active citizen engagement across political, economic, and cultural domains via multiple channels.
Deliberative Democracy
Consensus through rational dialogue and deliberation rather than simple voting.
Procedural Democracy
Normative and predictable democratic processes with transparent rules and open information.
Supervisory Democracy
Independent media, civil society, and anti-corruption bodies constraining power and exposing misconduct.
Local Democracy
Local government autonomy and decentralization, tailoring policies to local needs.
Digital Democracy
Use of information technology (e-voting, online consultation, open data) to expand participation.
Table 12: Secondary indicators of Freedom.
Indicator
Definition
Freedom. Individual autonomy in thought, expression, action, and choice, free from oppression and coercion.
Freedom of Thought
Right to independently form opinions, beliefs, and values without coercion or indoctrination.
Freedom of Belief
Right to choose, change, and practice religion or worldview, with state neutrality and protection of non-belief.
Freedom of Speech
Protection of expression without prior censorship or retaliation, balanced against public order and security.
Academic Freedom
Protection of scholars’ research, teaching, and publication from political or economic interference.
Creative Freedom
Autonomous expression for artists and cultural workers, bearing responsibility against incitement of violence.
Personal Liberty
Protection from unlawful arrest, arbitrary detention, or torture, with strict judicial procedures.
Privacy Freedom
Protection of personal life, data, and communications from illegal collection or disclosure.
Freedom of Association
Right to form groups (NGOs, unions, parties) for common purposes, with transparent registration.
Freedom of Assembly
Right to peaceful gatherings and demonstrations under prior notification rather than approval.
Freedom of Movement
Right to choose residence, travel, and lawfully enter or exit a country.
Economic Freedom
Property rights, entrepreneurship, contract autonomy, and employment choice with oversight against abuse.
Freedom of Choice
Autonomous decision-making across life domains free from social or familial coercion.
Table 13: Intra-VLM consistency. Each VLM is queried three times on every item; agreement is measured across the three runs. All three VLMs exhibit substantial self-consistency, with an average QWK of 0.71 across the three VLMs.
Dimension
Fleiss κ
Avg. QWK
Exact
GLM-4.5V (3 runs)
Peace
0.559
0.764
0.618
Development
0.599
0.781
0.655
Equity
0.487
0.633
0.679
Justice
0.486
0.655
0.638
Democracy
0.337
0.432
0.805
Freedom
0.481
0.590
0.692
Overall
0.492
0.643
0.681
Doubao-Seed-1.6-V (3 runs)
Peace
0.655
0.785
0.694
Development
0.666
0.802
0.700
Equity
0.581
0.689
0.724
Justice
0.571
0.721
0.622
Democracy
0.429
0.534
0.829
Freedom
0.530
0.666
0.717
Overall
0.572
0.699
0.714
Gemini-2.5-Pro (3 runs)
Peace
0.676
0.861
0.641
Development
0.607
0.858
0.587
Equity
0.587
0.741
0.588
Justice
0.665
0.798
0.646
Democracy
0.584
0.686
0.741
Freedom
0.511
0.725
0.497
Overall
0.605
0.778
0.617
Table 14: Inter-VLM agreement across GLM-4.5V, Doubao-Seed-1.6-V, and Gemini-2.5-Pro. Exact is the primary-dimension exact-agreement rate; Indicator is the exact-agreement rate on the corresponding 12 secondary indicators per dimension.
Dimension
Fleiss κ
Avg. QWK
Exact
Indicator
Peace
0.410
0.650
0.450
0.818
Development
0.242
0.615
0.256
0.765
Equity
0.305
0.518
0.487
0.871
Justice
0.339
0.557
0.464
0.837
Democracy
0.231
0.352
0.750
0.920
Freedom
0.204
0.447
0.400
0.848
Overall
0.288
0.523
0.468
0.843
Table 15: Human modification rate: the fraction of primary-dimension labels that human verifiers modified after 3-VLM majority voting.
Dimension
Rate (%)
Peace
25.11
Justice
19.02
Freedom
17.09
Development
15.60
Equity
12.78
Democracy
03.86
Overall
15.58
Table 16: The 14 personas used in Multi-Role Consensus (MRC), each emphasizing a complementary set of MAVEN secondary indicators (italicized). Persona 1 serves as an impartial baseline; personas 2–14 each bring a stakeholder-specific perspective. The full system prompts are released with the dataset.
#
Persona
Primary Evaluative Lens
1
Value Alignment Analyst
Impartial baseline; weighs all six primary dimensions equally based on textual evidence.
2
Educator
Educational Development, Opportunity Equity, Freedom of Thought; sensitive to barriers to learning.
3
Public Health Physician
Health Development, Healthcare Equity, Personal Liberty (bodily autonomy), Inner Peace.
Figure 1: Overview of the MAVEN hierarchical framework and evaluator optimization. (I) MAVEN Framework: Macro-societal value taxonomy with 6 primary dimensions and 72 secondary indicators. Secondary indicators are constructed through an iterative human–AI collaborative pipeline to ensure comprehensive coverage and conceptual clarity. (II) Evaluator Optimization: A compact student VLM is distilled from frontier teachers via our proposed SA-MDPO. Then multi-role consensus aggregates different stakeholder perspectives at inference time.
Figure 2: Overview of MacroValue-Bench construction and the evaluation pipeline. (Part A) We collect ∼30K multimodal items from four sources, sample 1,157 image–text pairs, annotate each pair via three frontier VLMs, then verify with human annotators. Each item carries a structured dimensional label. (Part B) We evaluate VLMs’ predictions against ground-truth annotations using Value-Aware Soft Match (Recall direction shown; Precision is symmetric), which assigns partial credit to semantically close cross-dimension predictions.
Figure 3: Cosine similarity between Qwen3-Embedding-8B representations of MAVEN’s 72 secondary indicators. (Left) Similarity matrix showing block-diagonal within-dimension coupling and off-block-diagonal cross-dimension neighbors. (Right) Distribution of pairwise similarities, with a dense bulk near 0.40 and a sparse right tail beyond 0.65.
Figure 4: Case study. Qwen: Qwen3-VL-8B-Instruct; Gemini: Gemini-2.5-Pro; Ours: Qwen3-VL-2B trained under SA-MDPO with MRC. Top: predicted primary scores. Qwen underactivates all dimensions; Gemini over-activates Peace and Development (false positives); our distilled 2B model with MRC correctly identifies Equity, Justice, and Freedom with no false positives. Bottom: per-indicator predictions. Above the divider are the 7 ground-truth non-zero indicators; below are indicators that models hallucinated. Our compact 2B model matches Gemini-2.5-Pro’s indicator recall (6/7) while producing zero false positives.
Figure 5: Per-dimension comparison across closed-source baselines on three metrics (Accuracy, QWK, F1macro).
Figure 6: Per-dimension QWK of the Qwen3-VL family (2B/4B/8B) and our SA-MDPO distilled evaluator. Our compact 2B model matches or exceeds Qwen3-VL-8B on all six dimensions.
Assessing whether multimodal content aligns with macro-societal values, such as peace, justice, and freedom, has become an increasingly urgent challenge. Existing frameworks are largely confined to safety-oriented taxonomies, text-only psychometric probes, or single-label classification. Therefore, we propose MAVEN, a hierarchical framework for macro-societal value evaluation of multimodal content, grounded in international human-rights instruments and cultural value theory. MAVEN organizes values into 6 primary dimensions and 72 secondary indicators, supporting multi-level quantitative scoring. Building on MAVEN, we construct a human-verified multimodal benchmark and a soft-match metric to evaluate VLMs' assessments across value dimensions. For evaluator optimization, we propose a span-adaptive variant of multi-level preference optimization for evaluator distillation, together with a training-free multi-role consensus strategy at inference time. We evaluate existing open- and closed-source VLMs on our benchmark, revealing shared tendencies and clear differences in macro-societal value judgments. Experiments show that our compact 2B evaluator matches its 8B counterpart in the same family and approaches frontier closed-source VLMs, offering a practical path toward scalable macro-societal value evaluation. Our SA-MDPO implementation and MacroValue-Bench are available at https://github.com/zzzzzzzzjj/MAVEN.
作者 · Zijuan Zhao, Zheren Fu, Hou Xia, Licheng Zhang, Yi Liu, Zhendong Mao
