Text-to-Image · Diffusion Models · Reinforcement Learning
Improving Text-to-Image Without Retraining: Z-Reward vs InterleaveThinker
Z-Reward replaces the scalar image reward with a score distribution (9B student at 88.6% accuracy, one token); InterleaveThinker wraps a frozen generator in a planner-critic loop (UEval 18.2 to 66.3).
Two levers, one frozen generator
Both papers in this comparison start from the same observation: the base text-to-image model is no longer the cheapest place to spend your budget. Retraining FLUX-class generators is expensive, so the question shifts to what you can bolt on around a frozen one. The two answers here attack opposite sides of the pipeline.
Z-Reward works on the reward side. It is a reward model for T2I post-training that predicts a full distribution over rubric scores instead of one scalar, trained with Group-wise Direct Score Optimization (GDSO) and distilled with Reasoning-Internalized Score Distillation (RISD). You use it inside an RL or preference loop that tunes the generator.
InterleaveThinker works on the inference side. It is a planner-critic agent loop wrapped around a frozen image generator, built for interleaved text-image sequences (tutorials, visual stories, step-by-step procedures) where each image must stay consistent with the ones before it. Nothing about the generator changes; the wrapper plans the sequence up front and critiques every image after it is generated.
The distinction matters because the failure modes are different. A weak reward corrupts everything downstream of post-training. A missing planner-critic corrupts long sequences one compounding step at a time.
Z-Reward: keep the judge’s uncertainty, drop the judge’s cost
The starting complaint is that a scalar reward throws away information. Two images rated 4.0 and 4.2 carry a tiny, noisy gap, but a scalar head treats it like the gap between 2.0 and 4.5. Annotator disagreement disappears entirely. Z-Reward instead predicts a probability distribution over a 9-level half-point scale (1.0 to 5.0) across four rubric dimensions: text-image alignment, realism, aesthetics, and physical plausibility. The distribution keeps uncertainty and gap size; its expectation still gives a smooth scalar for the optimizer.
GDSO, the teacher objective, stacks three signals: a policy-gradient term computed from the expectation of the predicted distribution, a pointwise cross-entropy term that pins score bins to human labels, and a pairwise term that forces predicted gaps between two images to match annotated gaps. The last term is what keeps ranking correct even when absolute calibration drifts.
The teacher is a 27B vision-language model. On the authors’ internal annotated test set it reaches 89.6% human-preference accuracy (HPA) with PLCC 0.7620 and SRCC 0.7132.
Reward accuracy, same setup
| Reward | HPA | Inference cost | Source |
|---|---|---|---|
| SFT baseline | 81.35% | standard | Z-Reward paper, internal test set |
| RewardDance | 84.25% | standard | same setup |
| GRPO-style reward | 86.04% | standard | same setup |
| Z-Reward teacher (27B, GDSO) | 89.6% | reasoning chain | same setup |
| Z-Reward student (9B, RISD) | 88.6% | 1 output token | same setup |
| On-policy distillation ablation | 83.11% | ~750 output tokens | same student size, same teacher |
The student row is the deployable result. RISD distills the teacher’s reasoning-conditioned distribution through a KL term rather than copying the reasoning text, so the 9B student emits a single token at inference and lands at 88.6% HPA, within one point of its 27B teacher. The ablation is the cleanest evidence that the value lives in the distribution, not the chain: on-policy distillation, which keeps explicit reasoning, reaches 83.11% while spending about 750 tokens per score.
Downstream, the authors report a 41.3% net Good-Same-Bad (GSB) human-preference improvement over the SFT generator baseline when this reward drives post-training. Read that number with its guardrail: GSB is a human side-by-side preference against the no-reward starting point, not a win over a generator tuned with a strong scalar reward, and there is no FID or GenEval cross-check. The honest claim is “a better optimization target than nothing,” with the reward-accuracy table supplying the rest.
InterleaveThinker: planning and critique around a frozen generator
InterleaveThinker targets a different failure: interleaved generation, where a tutorial or visual story needs a sequence of images that stay consistent with each other. Errors compound across steps: a wrong object in image two poisons image five, and a single-call generator has no mechanism to catch that.
The system splits the job. A Planner writes the full sequence of image instructions up front, before any image exists, deliberately preventing the plan from being dragged around by early visual mistakes. A Critic checks each generated image, decides whether the step succeeded, and writes a refinement prompt when it fails. Both start from Qwen3-VL-8B-Instruct, trained on Planner-SFT-80k, Critic-SFT-112k, and Critic-RL-13k, where the RL stage is GRPO with an accuracy reward for judging correctly and a step-wise reward for improving the next iteration. The reported training cost is about 50 hours on eight H800 GPUs, and inference can spend up to five refinement iterations per step.
UEval, same harness
| System | UEval average | Notes |
|---|---|---|
| FLUX.2-klein raw | 18.2 | ablation, no wrapper |
| + Qwen3-VL-8B wrapper | 48.1 | ablation baseline |
| + Planner-SFT | 60.5 | ablation |
| Full-SFT (planner + critic) | 64.5 | ablation |
| Full-RL | 66.3 | FLUX.2-klein-9B |
| One-agent variant | 54.5 | merging planner and critic |
| Qwen-Image-Edit integration | 67.2 | different frozen generator |
| Nano Banana (proprietary) | 66.0 | reported comparison |
| Nano Banana Pro (proprietary) | 76.1 | reported comparison |
Two readings of this table. First, the ablation ladder (18.2 → 48.1 → 60.5 → 64.5 → 66.3) is strong mechanism evidence that planning and critiquing are separable skills: the one-agent variant that merges them drops to 54.5. Second, the ceiling is real: Nano Banana Pro’s 76.1 is ten points above the best wrapper configuration, so an agent loop around an open 9B generator is competitive with a mid proprietary model, not with the frontier one.
On sequence-level metrics the pattern repeats. WISE moves from 0.47 to 0.73 on FLUX.2-klein; RISE moves from 13.3 to 28.9 overall, with the temporal category jumping from 7.1 to 36.5; CoMM reports 9.3/9.6 style consistency and 9.2/9.6 entity consistency across its two tasks. The temporal RISE jump supports the paper’s thesis: the critic specifically repairs the across-step drift that single-image metrics never see.
Key numbers
| Measurement | Z-Reward | InterleaveThinker | Setting and source | Same harness? |
|---|---|---|---|---|
| Headline metric | 88.6% HPA (9B student, 1 token) | 66.3 UEval (FLUX.2-klein-9B) | Each paper’s own benchmark | No, different tasks |
| vs strongest baseline | 89.6% teacher vs 86.04% GRPO-style reward | 67.2 vs Nano Banana Pro 76.1 | Z-Reward internal set / UEval | Yes, within each paper |
| Biggest ablation gap | RISD 88.64% at 1 token vs OPD 83.11% at ~750 tokens | One-agent variant 54.5 vs 66.3 full system | Same student size / same frozen generator | Yes, within each paper |
| Downstream effect | 41.3% net GSB over SFT baseline | WISE 0.47 → 0.73; RISE temporal 7.1 → 36.5 | Human side-by-side / sequence metrics | Within each paper |
| Serving cost | 1 output token per score | Up to 5 generator calls per step | Reported designs | n/a |
When to use which
| Question | Z-Reward | InterleaveThinker |
|---|---|---|
| What does it change? | The reward signal used to post-train the generator | The prompts and retries the generator receives at inference |
| Does the generator get retrained? | Yes, by the RL loop the reward feeds | No, the generator stays frozen |
| What failure does it fix? | Scalar rewards losing calibration and gap information | Compounding inconsistency across image sequences |
| Headline number | 9B student at 88.6% HPA, 1 token | UEval 18.2 → 66.3 on FLUX.2-klein |
| Main cost | Training a 27B teacher, then distilling; rubric labels at scale | Inference latency, up to 5 refinement calls per step |
| Quiet dependency | An internal rubric and annotation pipeline | Judge quality: the Critic’s RL data is built with Gemini 2.5 Pro scores |
The rows are nearly complementary. Z-Reward assumes you control the training loop and have rubric-style labels; InterleaveThinker assumes the model is frozen and your problem is sequence consistency, not single-image quality. A team building a product around interleaved visual content could plausibly run both: post-train the generator with a distributional reward, then wrap the result in a planner-critic loop. Nothing in either paper tests that stack, so treat it as an inference from the two mechanisms, not a measured result.
Limits and open questions
Both papers grade their headline numbers on their own setups, and the guardrails matter. Z-Reward’s HPA and GSB figures come from an internally annotated test set with an in-house four-dimension rubric: there is no public preference benchmark, no FID, CLIPScore, or GenEval cross-check, and the GSB win is against an SFT baseline rather than a generator tuned with a strong scalar reward. InterleaveThinker’s UEval, WISE, RISE, and CoMM numbers compress human visual expectations into scores; users will notice layout taste and narrative pacing the metrics miss, and the Critic inherits Gemini 2.5 Pro’s preferences because that model built its training data.
Cost is the second shared caveat. Z-Reward’s student is cheap at serving time, but the two-stage teacher-plus-distillation pipeline needs distributional labels at scale, and the paper does not test whether a smaller teacher degrades the gain. InterleaveThinker shifts its cost to inference, where a high-value visual sequence may justify many generator calls but a latency-sensitive product will not.
FAQ
Can either method improve image generation without retraining the base model?
InterleaveThinker can: it wraps a frozen generator in a planner-critic loop and lifts FLUX.2-klein from 18.2 to 66.3 average on UEval. Z-Reward cannot by itself, since it is a reward model, but its 9B student serves at one output token, so the reward side stays cheap once trained.
What is the strongest evidence that planner and critic should be separate agents?
InterleaveThinker’s ablation: a one-agent variant that merges planning and critique drops from 66.3 to 54.5 UEval on the same frozen generator, while the staged SFT ladder (planner alone 60.5, full SFT 64.5, full RL 66.3) shows each stage adding real value.
Does Z-Reward beat a scalar reward for text-to-image post-training?
On reward accuracy, yes, on the authors’ internal setup: the 27B teacher reaches 89.6% HPA versus 86.04% for a GRPO-style scalar reward and 84.25% for RewardDance, with the 9B student at 88.6%. On generated-image quality the evidence is thinner: the 41.3% net GSB improvement is measured against an SFT baseline, not against a generator tuned with a strong scalar reward.
Is InterleaveThinker better than proprietary models on interleaved generation?
No. The best wrapper configuration reaches 67.2 UEval average with Qwen-Image-Edit, above Nano Banana at 66.0 but well below Nano Banana Pro at 76.1. It is competitive with mid-tier proprietary systems around an open generator.
How much do InterleaveThinker’s sequence metrics improve beyond UEval?
WISE moves from 0.47 to 0.73 and RISE from 13.3 to 28.9 overall on FLUX.2-klein, with the temporal category jumping from 7.1 to 36.5; the cross-step drift a single-image generator cannot self-correct is exactly what the critic loop repairs.
One line: Z-Reward makes the reward worth optimizing, InterleaveThinker makes a frozen generator worth wrapping. Pick by whether your bottleneck is training signal or sequence consistency. Read the Z-Reward paper and the InterleaveThinker paper for the full setup behind each number.