컴백부터 K-뷰티까지 — K-컬쳐의 모든 것을 메일로 받아보세요메일로 받아보기

이미지와 글이 평화·정의·자유 같은 사회적 가치에 맞는지, 작은 AI 모델로도 대형 모델만큼 판단할 수 있게 만들었다

arXiv:2608.180962026-08-20

MAVEN: A Macro-Societal Value Evaluation Framework of Multimodal Content with Compact Aligned Evaluators

이미지와 글이 평화·정의·자유 같은 사회적 가치에 맞는지, 작은 AI 모델로도 대형 모델만큼 판단할 수 있게 만들었다

연구팀은 평화, 발전, 형평, 정의, 민주주의, 자유라는 6개 큰 가치와 이를 세분화한 72개 지표로 이루어진 MAVEN이라는 평가 틀을 만들고, 이를 기반으로 이미지와 글이 섞인 실제 콘텐츠 1,157개로 구성된 MacroValue-Bench라는 시험지를 함께 내놓았다. 이어서 대형 AI 모델(교사)의 판단 능력을 20억 개 파라미터짜리 작은 시각언어모델(학생)에 옮겨 담는 SA-MDPO라는 학습 방법과, 하나의 모델에게 판사·외교관 등 14개의 서로 다른 역할을 맡겨 답을 모으는 MRC라는 추론 기법을 제안했다. 그 결과 20억 개짜리 작은 모델이 같은 계열의 80억 개짜리 모델을 능가하고 유명 상용 모델들에 근접하는 성능을 보였다.

METAL MEDIA 해설 도표

이미지와 글이 평화·정의·자유 같은 사회적 가치에 맞는지, 작은 AI 모델로도 대형 모델만큼 판단할 수 있게 만들었다

01기존의 AI 가치 평가는 안전성(욕설, 혐오 등) 위주이거나 텍스트로만 질문하는 방식이라 실제 이미지+텍스트 콘텐츠의 '사회적 가치' 정렬 여부를 재기 어려웠다
02유엔인권선언 등 국제 인권 문서와 문화 가치 이론에 근거해 평화·발전·형평·정의·민주주의·자유 6개 대분류와 72개 세부 지표로 이루어진 2단계 평가 체계 MAVEN을 만들고, 뉴스·소셜미디어 등에서 수집한 1,157개 이미지-텍스트 쌍에 3개의 최상위 AI가 라벨을 달고 8명의 사람 검증자가 최종 확인한 MacroValue-Bench를 구축했다
03지표 간 의미가 겹치는 문제(예: '세대 간 형평'과 '세대 간 정의'가 비슷함)를 반영해 완전히 틀린 답과 애매하게 가까운 답을 구분해서 채점하는 VSMS라는 채점 방식을 새로 제안했다
04Qwen3-VL이라는 20억 개 파라미터의 작은 모델에 SA-MDPO라는 선호도 학습 방법을 적용해 대형 교사 모델들의 판단을 흡수시켰고, 등급 격차에 따라 학습 강도를 다르게 주는 방식으로 기존 방법(MDPO)보다 세밀한 신호를 살렸다
05추론 시에는 하나의 모델에게 판사, 교육자, 외교관, 노동권 옹호자 등 14가지 역할을 부여해 답을 모아 다수결로 합치는 MRC를 적용했고, 그 결과 QWK라는 순위 일치도 지표가 기준 모델 0.39에서 0.60으로, MRC까지 더하면 0.624로 올라 80억 개 모델을 앞질렀다

METAL MEDIA이 원문을 바탕으로 재구성한 해설 도표이며, 논문 저자의 원문 figure가 아닙니다.

무엇을 했나

기존의 AI 가치 평가는 안전성(욕설, 혐오 등) 위주이거나 텍스트로만 질문하는 방식이라 실제 이미지+텍스트 콘텐츠의 '사회적 가치' 정렬 여부를 재기 어려웠다
유엔인권선언 등 국제 인권 문서와 문화 가치 이론에 근거해 평화·발전·형평·정의·민주주의·자유 6개 대분류와 72개 세부 지표로 이루어진 2단계 평가 체계 MAVEN을 만들고, 뉴스·소셜미디어 등에서 수집한 1,157개 이미지-텍스트 쌍에 3개의 최상위 AI가 라벨을 달고 8명의 사람 검증자가 최종 확인한 MacroValue-Bench를 구축했다
지표 간 의미가 겹치는 문제(예: '세대 간 형평'과 '세대 간 정의'가 비슷함)를 반영해 완전히 틀린 답과 애매하게 가까운 답을 구분해서 채점하는 VSMS라는 채점 방식을 새로 제안했다
Qwen3-VL이라는 20억 개 파라미터의 작은 모델에 SA-MDPO라는 선호도 학습 방법을 적용해 대형 교사 모델들의 판단을 흡수시켰고, 등급 격차에 따라 학습 강도를 다르게 주는 방식으로 기존 방법(MDPO)보다 세밀한 신호를 살렸다
추론 시에는 하나의 모델에게 판사, 교육자, 외교관, 노동권 옹호자 등 14가지 역할을 부여해 답을 모아 다수결로 합치는 MRC를 적용했고, 그 결과 QWK라는 순위 일치도 지표가 기준 모델 0.39에서 0.60으로, MRC까지 더하면 0.624로 올라 80억 개 모델을 앞질렀다

Table 1: Comparison of value frameworks used for AI evaluation. ✓ = full support, ⚫ = partial, ✗ = not supported. Macro. (Macro-societal): targets societal-level values rather than individual ethics or safety. Multi. (Multimodal): operationalized on image–text content. Quant. (Quantitative): yields ordinal or continuous scores rather than categorical labels.
Framework	Macro.	Multi.	Quant.
ETHICS Hendrycks et al. (2020)	✗	✗	✓
BBQ Parrish et al. (2022)	✗	✗	✓
Constitutional AI Bai et al. (2022)	✗	✗	✗
Ch3Ef Shi et al. (2024)	✗	✓	✓
VIVA Hu et al. (2024)	✗	✓	✗
ValueBench Ren et al. (2024)	✗	✗	✓
CLAVE Yao et al. (2024)	✗	✗	✓
SPA-VL Zhang et al. (2025)	✗	✓	✗
Safe RLHF-V Ji et al. (2025)	✗	✓	✗
MORALISE Lin et al. (2025)	✗	✓	✗
HumaniBench Raza et al. (2025)	⚫	✓	✗
ValueCompass Shen et al. (2025)	✗	✗	✓
MAVEN (ours)	✓	✓	✓

Table 2: Theoretical foundations of Maven’s six primary dimensions. UDHR/ICCPR/ICESCR numbers refer to article numbers. Pre. denotes Preamble. Schwartz value abbreviations: Univ. = Universalism, Benev. = Benevolence, Conf. = Conformity, Self-dir. = Self-direction, Stim. = Stimulation.
Dimension	Instruments	Schwartz
Peace	UN Charter; UDHR Pre.	Security
Development	UDHR 22, 25; ICESCR	Universalism
Equity	UDHR 1, 7; ICESCR 2	Univ., Benev.
Justice	UDHR 6–11; ICCPR 14	Univ., Conf.
Democracy	UDHR 21; ICCPR 25	Self-dir.
Freedom	UDHR 18–20; ICCPR 18–22	Self-dir., Stim.

Table 3: MacroValue-Bench data sources.
Source	Count	%
Weibo	573	49.5
Global Times Online	245	21.2
Ch3Ef-harmless Shi et al. (2024)	173	15.0
People’s Daily Online	166	14.3
Total	1,157	100.0

Table 4: Performance on MacroValue-Bench. All metrics are higher-is-better (↑); bold indicates the best overall; green percentages report the relative gain over the preceding row. Doubao-Seed-1.6-V leads across all six metrics among commercial models, while our compact 2B evaluator matches Qwen3-VL-8B in the same family and approaches frontier commercial VLMs at a 4× smaller parameter count. SA-MDPO alone delivers the bulk of the gain (+52.4% QWK over the baseline), and MRC adds a further +4.2% QWK to reach 0.624.
Model	QWK ↑	Acc. ↑	F1macro ↑	VSMS ↑	Recall ↑	Precision ↑
Commercial Models
GPT-4o Hurst et al. (2024)	0.563	0.471	0.346	0.919	0.919	0.919
GPT-5 Singh et al. (2025)	0.637	0.724	0.520	0.962	0.962	0.962
Gemini-2.5-Pro Comanici et al. (2025)	0.651	0.646	0.471	0.950	0.950	0.950
Moonshot-V1-Vision Team et al. (2025)	0.436	0.673	0.321	0.846	0.846	0.846
Hunyuan-T1-Vision Team (2025b)	0.588	0.689	0.441	0.933	0.933	0.933
Qwen3-VL-Plus Bai et al. (2025a)	0.638	0.653	0.469	0.935	0.935	0.935
Doubao-Seed-1.6-V ByteDance (2025)	0.702	0.756	0.524	0.966	0.966	0.966
Open-source Models
GLM-4.5V Hong et al. (2025)	0.686	0.739	0.500	0.957	0.957	0.957
Qwen2.5-VL-7B Bai et al. (2025b)	0.245	0.617	0.225	0.921	0.913	0.945
Qwen2.5-VL-3B Bai et al. (2025b)	0.238	0.630	0.230	0.951	0.944	0.958
Qwen3-VL-8B Bai et al. (2025a)	0.546	0.691	0.380	0.961	0.961	0.961
Qwen3-VL-4B Bai et al. (2025a)	0.442	0.663	0.322	0.954	0.954	0.954
Qwen3-VL-2B Bai et al. (2025a)	0.393	0.497	0.300	0.798	0.798	0.798
Ours
Qwen3-VL-2B + SA-MDPO	0.599 (+52.4%)	0.706 (+42.1%)	0.482 (+60.7%)	0.954 (+19.5%)	0.954 (+19.5%)	0.954 (+19.5%)
Qwen3-VL-2B + SA-MDPO + MRC	0.624 (+4.2%)	0.719 (+1.8%)	0.497 (+3.1%)	0.959 (+0.5%)	0.959 (+0.5%)	0.959 (+0.5%)

Table 5: Ablation study comparing DPO, MDPO, and SA-MDPO under different training dataset.
Method	Acc.	QWK	VSMS
Baseline	0.497	0.393	0.798
DPO (unbalanced)	0.190	0.153	0.517
DPO (balanced)	0.654	0.267	0.946
MDPO (unbalanced)	0.656	0.503	0.920
MDPO (balanced)	0.670	0.445	0.964
SA-MDPO (unbalanced)	0.666	0.518	0.934
SA-MDPO (balanced)	0.706	0.599	0.954

Table 6: Effect of Multi-Role Consensus (MRC) on Qwen3-VL-Instruct models.
Model	Acc.	QWK	VSMS
Qwen3-VL-2B (×14 runs)	0.516	0.405	0.808
Qwen3-VL-2B (MRC)	0.527	0.432	0.815
Qwen3-VL-4B (×14 runs)	0.673	0.404	0.958
Qwen3-VL-4B (MRC)	0.662	0.424	0.957
Qwen3-VL-8B (×14 runs)	0.689	0.526	0.964
Qwen3-VL-8B (MRC)	0.706	0.540	0.968

Table 7: Secondary indicators of Peace.
Indicator	Definition
Peace. Non-violence and harmonious coexistence across individual, social, national, and global levels.
World Peace	Absence of war or conflict among nations, with disputes resolved through diplomacy, cooperation, and rule of law.
Regional Peace	Stable and friendly relations among countries within a geographic region through regional cooperation mechanisms.
National Peace	Absence of large-scale violence or civil war within a country, maintained by rule of law and stable governance.
Social Peace	Elimination of violent confrontation among communities or social groups via inclusive policies and exchange.
Political Peace	Resolving political differences through democratic consultation, elections, or negotiation rather than repression.
Economic Peace	Reducing conflict caused by poverty or inequality through fair distribution and equal economic opportunities.
Cultural Peace	Mutual respect and inclusion among cultural traditions, value systems, and belief frameworks.
Cyber Peace	Security and cooperation in digital space, curbing cyberattacks, disinformation, and digital surveillance.
Ecological Peace	Peaceful resolution of environmental resource disputes and cooperation on transnational ecological issues.
Community Peace	Harmonious relationships within neighborhoods or professional groups through community governance and mediation.
Family Peace	Absence of violence or oppression among family members, achieved through equal communication and shared responsibilities.
Inner Peace	Individual psychological tranquility and balance, cultivated through self-awareness and emotion regulation.

Table 8: Secondary indicators of Development.
Indicator	Definition
Development. Continuous progress and capacity enhancement of individuals, societies, and natural systems.
Economic Development	Material wealth growth and structural optimization, including industrial upgrading and inclusive prosperity.
Social Development	Improving livelihoods through public services in education, healthcare, and housing, narrowing wealth gaps.
Personal Development	Realizing individual potential through education, skill training, and mental health support.
Technological Development	Scientific breakthroughs driving productivity, with ethical constraints and equitable benefit distribution.
Institutional Development	Improvement of laws, policies, and governance to enhance adaptability, transparency, and efficiency.
Educational Development	Equitable distribution and quality improvement of educational resources, cultivating critical thinking.
Cultural Development	Protection and innovation of cultural diversity, including artistic flourishing and heritage preservation.
Environmental Development	Coordination of ecological protection and economic growth via green technologies and circular economy.
Health Development	Strengthening public health systems and achieving universal health coverage.
Urban-Rural Development	Eliminating the urban-rural dual structure through interconnected infrastructure and equitable services.
Regional Development	Reducing regional disparities through transportation networks, industrial policies, and complementary specialization.
Political Development	Modernization and democratization of political systems, expanding citizen participation and accountability.

Table 9: Secondary indicators of Equity.
Indicator	Definition
Equity. Reasonable distribution of resources, opportunities, rights, and responsibilities among members of society.
Opportunity Equity	Equal rights in education, employment, and entrepreneurship regardless of background, gender, or location.
Procedural Equity	Fair rule enforcement: consistent legal procedures, transparent standards, and competition free from privilege.
Educational Equity	Balanced resource allocation, access, and quality in education across socioeconomic backgrounds.
Employment Equity	Hiring, promotion, and compensation based solely on ability, prohibiting discrimination by gender, age, or race.
Healthcare Equity	Universal access to high-quality medical services regardless of payment ability or location.
Judicial Equity	Equality before the law, independent and unbiased trials, and accessible legal aid for vulnerable groups.
Market Equity	Healthy competition order through anti-monopoly and anti-unfair-competition measures protecting SMEs and consumers.
Tax Equity	Ability-to-pay taxation that prevents evasion and unjustified privileges, regulating wealth gaps.
Welfare Equity	Universal social security with need-based distribution, especially protecting elderly and disabled.
Political Equity	Equal citizen participation in voting, eligibility for office, and policy influence, free from wealth-based monopoly.
Information Equity	Equal access to knowledge in the digital age, narrowing the digital divide and protecting data privacy.
Intergenerational Equity	Balancing present and future interests, avoiding overuse of resources or unsustainable debt.

Table 10: Secondary indicators of Justice.
Indicator	Definition
Justice. Institutions and actions aligned with moral and legal principles, protecting rights and balancing interests.
Legal Justice	Legislation and judiciary conform to moral reasoning, with impartial laws and proportionate punishment.
Procedural Justice	Normative and neutral decision-making with rights to hearing, recusal, and reasoned explanations.
Rights-Based Justice	Guarantee of inalienable basic rights via constitutions, especially protecting minorities from majority tyranny.
Distributive Justice	Fair sharing of social resources and wealth, opposing extreme inequality through tax and welfare policies.
Social Justice	Equal and inclusive social structure, eliminating class, racial, or gender oppression.
Economic Justice	Critique of exploitative economic relations, advocating balanced labor-capital ties and fair trade rules.
Gender Justice	Opposition to gender-based discrimination and violence, pursuing gender equality across spheres.
Environmental Justice	Fair distribution of ecological risks and benefits, opposing pollution transfer to vulnerable communities.
Historical Justice	Acknowledgment and repair of past wrongs (e.g., colonialism, genocide) through apologies and reparations.
Intergenerational Justice	Current generations’ responsibility for the future, avoiding irreversible environmental or fiscal damage.
International Justice	Challenge to hegemony and unequal global orders, advocating sovereignty equality and democratized rules.
Cultural Justice	Respect for cultural identity and expression, opposing cultural hegemony or assimilationist oppression.

Table 11: Secondary indicators of Democracy.
Indicator	Definition
Democracy. Institutionalized public participation in decision-making and oversight of power.
Popular Sovereignty	The people as the ultimate source of state power, with constitutional principles establishing this status.
Electoral Democracy	Selection of representatives or leaders through regular, competitive, and free elections.
Representative Democracy	Citizens elect representatives who must faithfully reflect public opinion and remain accountable.
Rule-of-Law Democracy	Democratic operations bound by constitutional and legal frameworks, with separation of powers and rights protection.
Constitutional Democracy	A written constitution defining government powers and citizen rights, safeguarded by judicial review.
Grassroots Democracy	Direct participation at micro-levels such as residents’ self-governance and participatory budgeting.
Participatory Democracy	Active citizen engagement across political, economic, and cultural domains via multiple channels.
Deliberative Democracy	Consensus through rational dialogue and deliberation rather than simple voting.
Procedural Democracy	Normative and predictable democratic processes with transparent rules and open information.
Supervisory Democracy	Independent media, civil society, and anti-corruption bodies constraining power and exposing misconduct.
Local Democracy	Local government autonomy and decentralization, tailoring policies to local needs.
Digital Democracy	Use of information technology (e-voting, online consultation, open data) to expand participation.

Table 12: Secondary indicators of Freedom.
Indicator	Definition
Freedom. Individual autonomy in thought, expression, action, and choice, free from oppression and coercion.
Freedom of Thought	Right to independently form opinions, beliefs, and values without coercion or indoctrination.
Freedom of Belief	Right to choose, change, and practice religion or worldview, with state neutrality and protection of non-belief.
Freedom of Speech	Protection of expression without prior censorship or retaliation, balanced against public order and security.
Academic Freedom	Protection of scholars’ research, teaching, and publication from political or economic interference.
Creative Freedom	Autonomous expression for artists and cultural workers, bearing responsibility against incitement of violence.
Personal Liberty	Protection from unlawful arrest, arbitrary detention, or torture, with strict judicial procedures.
Privacy Freedom	Protection of personal life, data, and communications from illegal collection or disclosure.
Freedom of Association	Right to form groups (NGOs, unions, parties) for common purposes, with transparent registration.
Freedom of Assembly	Right to peaceful gatherings and demonstrations under prior notification rather than approval.
Freedom of Movement	Right to choose residence, travel, and lawfully enter or exit a country.
Economic Freedom	Property rights, entrepreneurship, contract autonomy, and employment choice with oversight against abuse.
Freedom of Choice	Autonomous decision-making across life domains free from social or familial coercion.

Table 13: Intra-VLM consistency. Each VLM is queried three times on every item; agreement is measured across the three runs. All three VLMs exhibit substantial self-consistency, with an average QWK of 0.71 across the three VLMs.
Dimension	Fleiss κ	Avg. QWK	Exact
GLM-4.5V (3 runs)
Peace	0.559	0.764	0.618
Development	0.599	0.781	0.655
Equity	0.487	0.633	0.679
Justice	0.486	0.655	0.638
Democracy	0.337	0.432	0.805
Freedom	0.481	0.590	0.692
Overall	0.492	0.643	0.681
Doubao-Seed-1.6-V (3 runs)
Peace	0.655	0.785	0.694
Development	0.666	0.802	0.700
Equity	0.581	0.689	0.724
Justice	0.571	0.721	0.622
Democracy	0.429	0.534	0.829
Freedom	0.530	0.666	0.717
Overall	0.572	0.699	0.714
Gemini-2.5-Pro (3 runs)
Peace	0.676	0.861	0.641
Development	0.607	0.858	0.587
Equity	0.587	0.741	0.588
Justice	0.665	0.798	0.646
Democracy	0.584	0.686	0.741
Freedom	0.511	0.725	0.497
Overall	0.605	0.778	0.617

Table 14: Inter-VLM agreement across GLM-4.5V, Doubao-Seed-1.6-V, and Gemini-2.5-Pro. Exact is the primary-dimension exact-agreement rate; Indicator is the exact-agreement rate on the corresponding 12 secondary indicators per dimension.
Dimension	Fleiss κ	Avg. QWK	Exact	Indicator
Peace	0.410	0.650	0.450	0.818
Development	0.242	0.615	0.256	0.765
Equity	0.305	0.518	0.487	0.871
Justice	0.339	0.557	0.464	0.837
Democracy	0.231	0.352	0.750	0.920
Freedom	0.204	0.447	0.400	0.848
Overall	0.288	0.523	0.468	0.843

Table 15: Human modification rate: the fraction of primary-dimension labels that human verifiers modified after 3-VLM majority voting.
Dimension	Rate (%)
Peace	25.11
Justice	19.02
Freedom	17.09
Development	15.60
Equity	12.78
Democracy	03.86
Overall	15.58

Table 16: The 14 personas used in Multi-Role Consensus (MRC), each emphasizing a complementary set of MAVEN secondary indicators (italicized). Persona 1 serves as an impartial baseline; personas 2–14 each bring a stakeholder-specific perspective. The full system prompts are released with the dataset.
#	Persona	Primary Evaluative Lens
1	Value Alignment Analyst	Impartial baseline; weighs all six primary dimensions equally based on textual evidence.
2	Educator	Educational Development, Opportunity Equity, Freedom of Thought; sensitive to barriers to learning.
3	Public Health Physician	Health Development, Healthcare Equity, Personal Liberty (bodily autonomy), Inner Peace.
4	Financial Analyst	Economic Development, Market Equity, Economic Peace, Economic Freedom.
5	Judge	Procedural Justice, Rule-of-Law Democracy, Judicial Equity, Rights-Based Justice.
6	Environmental Scientist	Ecological Peace, Environmental Development, Intergenerational Equity, Environmental Justice.
7	Socially-Conscious Artist	Cultural Peace, Cultural Development, Creative Freedom, Freedom of Speech.
8	Social Worker	Social Peace, Family/Community Peace, Welfare Equity, Social Justice.
9	Investigative Journalist	Information Equity, Supervisory Democracy, Freedom of Speech, Political Peace.
10	Technologist	Technological Development, Cyber Peace, Digital Democracy, Privacy Freedom.
11	Diplomat	World/Regional Peace, International Justice, Popular Sovereignty, Cultural Peace.
12	Political Scientist	Institutional Development, Electoral Democracy, Constitutional Democracy, Political Equity.
13	Historian & Sociologist	Historical Justice, Gender Justice, Freedom of Belief, Academic Freedom.
14	Labor Rights Advocate	Employment Equity, Distributive Justice, Tax Equity, Freedom of Movement/Association.

왜 중요한가

이미지와 텍스트가 뒤섞인 뉴스·SNS 콘텐츠가 사회적으로 어떤 가치를 담고 있는지 자동으로, 그것도 저렴한 소형 모델로 가려낼 수 있다는 것은 콘텐츠 검수나 미디어 분석 도구를 만드는 실무자에게 실질적인 선택지가 늘어난다는 뜻이다. 다만 저자들 스스로 이 틀이 특정 문화권(중국어권 소스, 동아시아 검증자)에 치우쳐 있고 검열 도구로 오용될 위험을 경고하고 있어, 그대로 가져다 쓰기보다는 사용 맥락에 맞게 검증해야 한다.

이 논문의 용어

VLM(시각언어모델) · 이미지와 텍스트를 함께 이해하고 답을 생성하는 AI 모델
QWK(Quadratic Weighted Kappa) · 순서가 있는 등급(예: -2~+2점) 사이에서 사람과 모델의 판단이 얼마나 일치하는지, 크게 틀릴수록 더 감점하며 재는 지표
DPO/MDPO/SA-MDPO · 여러 개의 답 중 더 나은 답을 선호하도록 모델을 학습시키는 방법들로, SA-MDPO는 등급 격차가 클수록 학습 강도를 더 세게 주도록 개선한 버전
지식 증류(distillation) · 크고 성능 좋은 교사 모델의 판단 능력을 작고 가벼운 학생 모델에 옮겨 담는 학습 방식
VSMS(Value-Aware Soft Match Score) · 정답과 예측 지표가 완전히 다른 것보다 의미상 가까운 지표일 때 부분 점수를 주는 채점 방식

본문에 싣지 못한 그림

Figure 1: Overview of the MAVEN hierarchical framework and evaluator optimization. (I) MAVEN Framework: Macro-societal value taxonomy with 6 primary dimensions and 72 secondary indicators. Secondary indicators are constructed through an iterative human–AI collaborative pipeline to ensure comprehensive coverage and conceptual clarity. (II) Evaluator Optimization: A compact student VLM is distilled from frontier teachers via our proposed SA-MDPO. Then multi-role consensus aggregates different stakeholder perspectives at inference time.
Figure 2: Overview of MacroValue-Bench construction and the evaluation pipeline. (Part A) We collect ∼30K multimodal items from four sources, sample 1,157 image–text pairs, annotate each pair via three frontier VLMs, then verify with human annotators. Each item carries a structured dimensional label. (Part B) We evaluate VLMs’ predictions against ground-truth annotations using Value-Aware Soft Match (Recall direction shown; Precision is symmetric), which assigns partial credit to semantically close cross-dimension predictions.
Figure 3: Cosine similarity between Qwen3-Embedding-8B representations of MAVEN’s 72 secondary indicators. (Left) Similarity matrix showing block-diagonal within-dimension coupling and off-block-diagonal cross-dimension neighbors. (Right) Distribution of pairwise similarities, with a dense bulk near 0.40 and a sparse right tail beyond 0.65.
Figure 4: Case study. Qwen: Qwen3-VL-8B-Instruct; Gemini: Gemini-2.5-Pro; Ours: Qwen3-VL-2B trained under SA-MDPO with MRC. Top: predicted primary scores. Qwen underactivates all dimensions; Gemini over-activates Peace and Development (false positives); our distilled 2B model with MRC correctly identifies Equity, Justice, and Freedom with no false positives. Bottom: per-indicator predictions. Above the divider are the 7 ground-truth non-zero indicators; below are indicators that models hallucinated. Our compact 2B model matches Gemini-2.5-Pro’s indicator recall (6/7) while producing zero false positives.
Figure 5: Per-dimension comparison across closed-source baselines on three metrics (Accuracy, QWK, F1macro).
Figure 6: Per-dimension QWK of the Qwen3-VL family (2B/4B/8B) and our SA-MDPO distilled evaluator. Our compact 2B model matches or exceeds Qwen3-VL-8B on all six dimensions.

원문에서 그림 보기 →

저자 · Zijuan Zhao, Zheren Fu, Hou Xia, Licheng Zhang, Yi Liu, Zhendong Mao

arXiv에서 원문 보기

최신 논문

논문 전체 보기 →

METAL MEDIA 최신 기사