Text-to-Image · Video Generation · Self-Supervised Learning
Bootstrap Your Generator: Unpaired Editing That Beats Supervised Models
ByG trains image and video editing models with no paired data and no reward model, winning 75.3% of head-to-head video votes against Ditto, which was supervised on millions of pairs.
Quick answer
ByG (Bootstrap Your Generator) trains a flow-matching editor without a single paired before/after example. It learns only from a frozen base generator’s own knowledge, combining instruction-following cues pulled from that model with a cycle-consistency loss that protects structure. On video editing it wins 75.3% of head-to-head human votes against Ditto, a model supervised on millions of synthetic pairs, and 85.0% on out-of-distribution 3D-CGI conversions where Ditto wins only 15.0%. The image model trains from roughly 330 unlabeled image tuples and the video model from 328 generated clips, using FLUX.1-dev and Wan2.2 as the frozen bases. The headline is that base-model semantics can stand in for both paired data and an external reward model.
Why editing usually needs paired data, and why video can’t get it
Instruction-based editors normally learn from pairs: a source image, an edit instruction, and the ground-truth result. Building millions of those pairs is expensive for images and close to impossible for video, where you would need the same clip rendered twice with one controlled change. ByG drops the requirement entirely. It asks the frozen base model what an edited frame should look like (the instruction cue), then enforces that running the edit forward and inverting it returns the original (cycle-consistency for structure). No human labels, no synthetic edit pairs, no separate reward model scoring the output. The supervision is the base model disagreeing with itself until the edit both follows the instruction and preserves identity.
What “gradient routing over clean predictions” actually fixes
Flow-matching models train on noisy states but the losses ByG cares about (does the edit follow the instruction, is structure preserved) only make sense on a clean image. The naive fix, computing those losses on a one-step clean guess, produces a blurry estimate that gives bad gradients. The naive alternative, denoising fully to a sharp image, breaks backpropagation because the multi-step sampler is not differentiable in a usable way. ByG splits the two. The forward pass conditions the losses on a clean multi-step estimate (sharp, so the loss is meaningful), but during backpropagation the gradient bypasses that estimate and flows through the one-step prediction instead. It is straight-through estimation adapted to flow matching: evaluate on the realistic clean image, learn through the differentiable noisy state. This bridges the train-inference gap, since training now optimizes the same clean output the sampler produces at inference.
This builds on the flow-matching formulation popularized by Stable Diffusion 3, and the train-versus-inference distribution gap it targets is the same one that on-policy methods like Flow-OPD attack from the distillation side.
Which piece carries the result
The ablation (a 0-10 scale for edit success and source preservation) is blunt about what matters. Remove the directional regularization loss and the model collapses: edit success drops to 0.633 while source preservation climbs to 9.767, meaning it learns to copy the input and ignore the instruction. That single loss is what stops identity collapse. Remove bootstrapping and edit success falls to 5.517, so the iterative self-improvement is the second load-bearing piece. The gradient routing and cycle loss matter less on their own (edit success 8.917 and 8.983 versus the full 8.317), which is a useful signal: the headline trick is real but the regularizer and the bootstrapping loop are what keep training from degenerating. A builder copying only the “clever” gradient routing would likely watch the model collapse.
Key results
- Video user study: ByG wins 75.3% (±2.2) of pairwise human votes overall, 70.0% on cartoon targets and 80.5% on photorealistic targets, against Ditto. Inter-rater agreement Fleiss κ = 0.44, binomial test p under 3e-15.
- Out-of-distribution video: on 3D-CGI conversions (an unseen domain), ByG wins 85.0% vs Ditto’s 15.0%, the largest gap and the clearest generalization claim.
- Video quantitative (vs Ditto): DINO structure similarity 0.718 vs 0.536, CLIP directional 0.104 vs 0.091, motion fidelity 0.715 vs 0.616.
- Long-tail style editing: style-to-photoreal scores 8.30 vs FLUX-Kontext 7.85; photoreal-to-style 6.33 vs 6.00, across six rare styles (GTA V, Minecraft, comic, low-poly, voxel, Lego).
- General image editing (GEdit-Bench): competitive overall, with large margins on hard categories: motion 6.88 vs Kontext 4.53, human edits 6.59 vs 3.94, style 6.95 vs 5.90.
- Training cost: image model trains in 30,000 steps on 8 H100s from ~330 unlabeled tuples, about 3x the per-step cost of supervised training (~2.9s vs 0.97s); video model trains in 750 steps.
How to read the headline
“Beats supervised models trained on millions of samples” is the strongest claim and the one to read carefully. The clean part is the quantitative video table (DINO 0.718 vs 0.536) and the long-tail style scores, which are automatic metrics. The user-study numbers are more subjective: 8 raters, 238 votes, and for video the authors sampled 4 outputs per method and manually picked the best one before showing it to raters, which favors methods with higher variance. The agreement (κ = 0.44) is moderate, not strong. So “wins 75.3%” means a small panel preferred a hand-selected best ByG clip over a hand-selected best Ditto clip. The structural-similarity and out-of-distribution gaps are the more durable evidence; the win-rate is supporting, not load-bearing. The general image story is also honest about being competitive rather than dominant on standard GEdit-Bench, with the gains concentrated in motion, human, and style edits.
Unpaired editing of this kind sits next to the broader push to make video editing controllable without per-task supervision, the same territory as LoomVideo and the instruction benchmarks in CoVE-Bench.
Limits and open questions
Training is roughly 3x slower per step than supervised editing, so “no data” trades data cost for compute and a more fragile optimization. The whole result depends on the frozen base model already knowing the target concept; ByG extracts cues, it does not teach the base anything new, so edits requiring knowledge the base lacks are out of reach. The benchmarks are small and partly self-built: the video set is 119 tasks the authors assembled, and the user study is 8 raters with manual best-of-4 selection, which is a generous protocol. There is no FID or large-scale automatic eval on the supervised baselines’ home turf, so the “beats supervised” claim rests on style/structure metrics plus the panel. The regularization loss that prevents collapse is clearly delicate (the ablation goes to near-zero edit success without it), suggesting the training is sensitive to that hyperparameter.
FAQ
Does ByG use any paired training data or a reward model?
No. ByG trains with zero paired before/after edits and no external reward model. Its only supervision comes from the frozen base generator: instruction-following cues extracted from that model, plus a cycle-consistency loss for structure. The image model learns from ~330 unlabeled image tuples and the video model from 328 generated clips, with FLUX.1-dev and Wan2.2 as the frozen bases.
What does gradient routing over clean predictions fix in ByG?
It bridges the train-inference gap. ByG’s losses (edit success, structure) only make sense on a sharp image, but the sampler that produces sharp images is not cleanly differentiable, and a one-step clean guess is too blurry to score. ByG conditions the loss on a multi-step clean estimate during the forward pass, then routes the backward gradient through the one-step prediction instead, a straight-through estimator for flow matching that lets training optimize the same clean output inference produces.
In the ByG ablation, which component drives the result?
The directional regularization loss is the most load-bearing: removing it drops edit success to 0.633 (out of 10) while source preservation rises to 9.767, meaning the model just copies the input. Bootstrapping is second, with edit success falling to 5.517 when removed. Gradient routing and the cycle loss matter less individually (8.917 and 8.983 vs the full model’s 8.317), so the regularizer and the bootstrapping loop are what keep training stable.
How does ByG compare to Ditto on out-of-distribution video?
On 3D-CGI conversions, an unseen domain, ByG wins 85.0% of human votes against Ditto’s 15.0%, its largest margin. Overall video win rate is 75.3% (±2.2). Quantitatively ByG holds structure far better, with DINO similarity 0.718 vs 0.536. The caveat: raters saw a manually chosen best-of-4 clip per method, so the win rate is a generous protocol; the DINO and out-of-distribution gaps are the firmer evidence.
One line: ByG shows a frozen generator already contains enough signal to train an editor with no paired data and no reward model, trading data cost for ~3x training compute and a delicate regularizer. Read the original paper on arXiv.