Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox›
What to Edit Next: Visually Aligned Image-Editing Follow-Up Suggestions in Conversational Systems
arXiv:2608.075652026-08-02
A three-stage training pipeline teaches an AI to suggest the next image edit that users actually click and that matches what's really in the picture
In conversational image-editing apps, follow-up edit suggestions can ask for something already true of the image or something that isn't there at all. The authors first fine-tune a model on human-reviewed appropriate follow-up intents, then use real user clicks to train a reward model and reinforcement learning, and finally add an image-checking verifier as an extra reward to cut these visual mismatches. The full system was deployed on Qwen App and tested in a large-scale randomized online experiment.
METAL MEDIA explanatory visual
Three-stage training pipeline
Evidence statusMeasured results reported
Stage 1: Rule-guided SFTCombines real image/query/intent inputs with a human-reviewed table of 61 follow-up intents to build six-suggestion training targets and fine-tune the multimodal policy
Stage 2: Click-based RLAn 8B reward model trained on position-aware click pairs plus four quality signals (validity, SFT-closeness, length, diversity) optimized via GRPO; visual inconsistency rose from 3.0% to 3.7%
Stage 3: Visual verifier rewardA separate image-first verifier checks each suggestion's required source and target state, adding a sixth grounding reward that cuts inconsistency to 0.9%
Deployment & A/B testOnly the single 8B policy is served (verifier used only in training); a 14-day randomized test with millions of users measured CTR, take-away rate, and conversation turns
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.
What they did
Auditing 100,000 real multi-turn image-editing conversations from Qwen App, the authors found 80.1% of follow-up edit requests depend on the current image and cannot be inferred from the prior text alone.
Stage 1 (SFT) fine-tunes a multimodal policy on six-suggestion sets built from real image-query-intent inputs combined with a human-reviewed table of 61 valid follow-up editing intents.
Stage 2 trains an 8B vision-language reward model on position-aware click preference pairs from real users, then uses multi-objective GRPO to optimize five signals: click preference, output validity, closeness to the SFT policy, content-aware length, and within-slate diversity.
Stage 2 alone improved expert-rated quality but increased visual inconsistency (suggestions requiring an absent object or requesting an already-satisfied state) from 3.0% after SFT to 3.7%.
Stage 3 adds a separate image-first structured verifier that checks required sources and target states for each suggestion, feeding a sixth 'grounding' reward into GRPO to fix this regression.
Figure 1: Illustrative multi-turn policy comparison. A click-supervised policy without visual-consistency supervision (left) proposes a slate containing a missing-source edit and an already-satisfied target; the user selects the executable lighting edit. On the updated image, the full framework (right) produces executable, image-consistent, and non-redundant follow-up edits.
Figure 2: Three-stage training framework. Stage 1 constructs validated SFT slates from real contexts. Stage 2 trains a click reward model and optimizes five non-grounding rewards with GRPO. Stage 3 derives an image-first source–target grounding reward and supplies it as a sixth signal to a separate GRPO run, initialized from SFT and otherwise using the Stage 2 pipeline.
Table 2: Cumulative ablation of RL components. All rows start from the SFT policy.
RL configuration
GSB vs base ↑
Ground. ↓
Redund. ↓
SFT initialization
+332
3.0
17.2
+ core RL rewards
+374
4.4
28.1
+ max-pair diversity
+383
4.0
19.1
+ norm./dynamic weights
+405
3.7
11.9
+ grounding reward
+𝟒𝟒𝟔
0.9
8.8
Figure 3: SFT data-construction pipeline (general track). Pale blocks are the data surviving each stage; grey ribbons show removed records and their filtering reasons.
Table 4: Verifier evaluation on the visual-consistency calibration set.
Visual inconsistency recall ↑
False
Verifier
Missing src.
Already sat.
Union
rejection ↓
Single-pass VLM
61.6%
43.4%
47.5%
22.2%
Source–target
92.9%
74.5%
78.7%
0.6%
Table 5: Stage-wise visual inconsistency by error type. Union denotes the rate of suggestions with either error.
Policy
Missing source ↓
Already-satisfied target ↓
Visual inconsistency (union) ↓
SFT (Stage 1)
0.74%
2.26%
3.0%
SFT + RL (Stage 2)
0.70%
3.00%
3.7%
Full framework (Stage 3)
0.42%
0.48%
0.9%
Table 6: Preference-data funnel. Counts in the first three rows are impression-level unless marked as suggestion-level.
Stage
Filtering operation
Output
Source
Three displayed suggestions and a non-empty click
663,926 requests
Record filtering
Remove click not in display (1,861), invalid intent (45,022), invalid clicked type/sub-tag (120,344), fewer than two usable continuation suggestions (15,344), and repeated sessions (249,062)
232,293 impressions
Image transfer
Retain records with a successfully transferred image
232,281 impressions
Pair eligibility
Remove unusable query (944), query-copy click (66), missing final intent (30), and no usable suggestion above the click (102,491)
128,750 impressions
Suggestion pruning
Drop generic refresh-like (39,829), non-continuation (45,420), and below-click options (204,173); these are suggestion-level counts
173,071 pairs
Request split
Group by request ID
164,401 train / 8,670 validation pairs
Table 7: Per-candidate dimensions emitted by one verifier call.
Dimension
Judged from
Used for
missing source
source existence bits with evidence
RL grounding g^
already satisfied
target vs. current state
RL grounding g^
plausibility
fit to image and scene
monitoring only
effective change
non-trivial visible change
monitoring only
content density
global appearance vs. object/region edit
length budget class
evidence, states
pointing phrases
audits and error analysis
Table 8: Source and target semantics by edit type.
Operation
Required source
Target check
Add
anchor/context entity
requested addition not satisfied
Remove
object to remove
object not already absent
Modify
object and current state
new state not already satisfied
Relocate
object and reference region
requested relation not satisfied
Global style
image
style change checkable or uncertain
Table 9: Diversity signal and aggregation on the semantic-redundancy probe.
Variant
Adapt. λ
Redund. ↓
Near-dup ↓
SFT
×
14.8%
1.4%
Jaccard
×
28.0%
2.2%
Embed., mean
×
30.0%
2.2%
Embed., max-pair
×
20.8%
0.8%
Embed., max-pair
✓
9.6%
0.4%
Table 10: RM-only genericization stress test. Cross-image reuse and within-list redundancy are measured at the final checkpoint.
Steps
Δ click-RM
Reuse ↓
Redund. ↓
30
+0.31
–
–
80
+1.25
–
–
160
+2.46
99.9%
0.1%
Findings
In offline evaluation, the full three-stage framework reduced visual inconsistency from 3.7% (Stage 2) to 0.9%, relative to Stage 2, while preserving expert-rated suggestion quality.
In a 14-day user-randomized A/B test on Qwen App involving millions of users, the full framework improved recommendation CTR by 32.70%, image take-away rate by 16.32%, and average conversation turns per user by 39.90% relative to the previously deployed prompt-engineered policy, all statistically significant (p<0.05).
Training with the click reward model alone caused the policy to collapse into repeating one generic, source-free edit across almost every image (99.9% cross-image reuse after 160 steps), showing that click preference alone is exploitable without the PPL, diversity, and grounding safeguards.
With the full reward set, the deployed policy avoided this collapse, showing 54.3% cross-image reuse versus 57.3% for the SFT baseline, with similarly small length variation.
Without the content-aware length reward, the click reward model's bias toward longer text pushed the mean suggestion length from 11.9 to 17.0 characters via non-essential wording rather than added content.
Where it can be used
Designing follow-up suggestion features for conversational image generation or editing tools
Building click-based reward models for recommendation systems while mitigating display-position bias through position-aware preference pairs
Using a separate verifier model as a training-time reward signal to enforce factual/visual grounding without adding serving latency
Balancing multiple competing objectives (preference, diversity, length, format validity) within a single multi-objective RL training pipeline
Limits and open work
The study is specific to Qwen App and the Qwen3-VL model family (8B policy, 30B verifier); transfer to other products or model families is not demonstrated.
The visual verifier is used only during training and checkpoint selection, not at serving time, so the deployed policy's visual errors are reduced but not directly checked at inference.
The verifier fails open on subjective or unclear target states, meaning some real visual inconsistencies may go undetected.
The online A/B test ran for 14 days on a fixed 5% traffic allocation; longer-term effects or behavior under different traffic conditions are not reported.
The authors note that position-aware click pairing reduces but does not fully remove display-position bias in the preference data.
Why it matters
As conversational assistants increasingly help users iterate on images, follow-up suggestions that ignore the actual image content waste user effort and erode trust; prior recommendation research mostly targeted text-only dialogue. This work shows, with a live deployment and millions of users, that combining human-defined intents, real click behavior, and explicit visual verification can measurably improve engagement while reducing visually invalid suggestions.
Terms in this paper
SFT (supervised fine-tuning) · Training a model further on curated example inputs and target outputs
GRPO · A reinforcement learning method that compares multiple sampled outputs against each other to push the policy toward higher-reward ones
reward model (RM) · A model trained on signals like user clicks to score how preferable a suggestion is
visual inconsistency · When a suggested edit requires an object/state missing from the image, or requests a state that already holds
GSB (Good-Same-Bad) · A score representing how much better or worse expert raters judged a policy compared to the reference policy
Figures we cannot republish
Figure 4: Source–target visual consistency checks. A suggestion is invalid when it presupposes an absent source or requests a visually checkable target state that already holds.
Conversational assistants increasingly recommend follow-up edits to help users continue a task. Existing systems primarily target text-only interactions, leaving image-creation conversations underexplored. In image-creation tasks, useful follow-up edit suggestions must reflect user preferences, offer diverse directions, and remain executable on the current image. We collected 100,000 real multi-turn image-creation conversation samples from Qwen App and found that 80.1% are image-dependent, underscoring the need for multimodal recommendation. We address this setting with a three-stage framework. In Stage 1, we use real online data to build a human-reviewed table of appropriate follow-up editing intents, then create SFT targets and fine-tune a multimodal policy. In Stage 2, to align rule-guided SFT suggestions with actual user choices, we use user click feedback to optimize the policy through multi-objective reinforcement learning. In Stage 3, to reduce visual inconsistencies between suggested edits and the current image, we introduce a visual verifier as additional training supervision. Extensive experiments demonstrate that our framework significantly outperforms baselines on both automatic and human evaluations. In a live user-randomized A/B test with millions of users, our final framework reduces visual inconsistency from 3.7% to 0.9%. Furthermore, it significantly improves recommendation CTR by 32.70%, image take-away rate by 16.32%, and average conversation turns per user by 39.90% (all p<0.05).
