WorldCraft: Object Manipulation for Camera-Controlled Video World Models

Quick answer

WorldCraft takes a video world model that already obeys camera commands and bolts on object manipulation: you click an object, sketch where it should go, and the model moves that object along your path while the camera keeps navigating the scene. On the trajectory-accuracy set it reaches 38.90px trajectory error against DragAnything’s 39.86 and Wan-Move’s 44.08, with PSNR 17.23 (baselines 15.97-16.42) and LPIPS 0.363 (baselines 0.375-0.468). Camera fidelity barely moves under camera-only evaluation: RPE rotation 0.131 / translation 0.0161 at 61 frames. The lift is owned by three additions working together, not by one clever loss, and the ablations make clear which one matters where.

The gap: camera control was solved, object control was not

Interactive video world models, including the camera-controlled multi-shot work in OmniDirector as well as Yume, Matrix-Game 2.0, and GameCraft, let you fly a camera through a generated scene. What they could not do was let you grab an object and move it. The naive fix is to draw a drag path on the screen and condition on those 2D pixels, which is what DragAnything-style methods do. That breaks the moment the camera moves: a path drawn in pixel space is meaningless once the viewpoint rotates, because the same world motion now lands on different pixels. WorldCraft’s premise is that object control has to live in the world, not on the screen.

How object control is added without wrecking the camera

Three pieces stack on a frozen camera-controlled backbone.

Normalized World Trajectory (NWT) converts your on-screen drag into camera-invariant world coordinates using monocular depth, then re-projects it onto each generated frame from that frame’s current camera pose. So the path is anchored to the world; if the camera orbits, the object still goes where you meant. The ablation credits this directly: world-space plus iterative depth refinement reaches 30.82 trajectory error on small rotations versus 35.82 for a pixel-space baseline.

Spatial-Pathway LoRA (SP-LoRA) injects the object-control signal through a low-rank adapter of roughly 50M parameters while leaving the pretrained camera controller untouched. This is the load-bearing design choice. Table 5a shows that full fine-tuning to add object control wrecks the camera, blowing RPE rotation from 0.131 up to 0.237. The LoRA buys object control almost for free in camera-fidelity terms, which is the whole point of not retraining the backbone.

Trajectory-Anchored State Persistence (TASP) handles the autoregressive failure case: an object you moved leaves the frame, the camera pans back, and a naive model regenerates it in the wrong place or forgets it. TASP keeps the object’s world state across the rollout so it reappears correctly. This is what lets WorldCraft hold up over 253-frame (~10.5s) generations with off-camera excursions, the same long-horizon consistency problem that memory-augmented world models like Echo attack from the persistence-of-state angle.

Read the camera-fidelity numbers carefully

The easy misread is “WorldCraft adds object control for free, camera quality is unchanged.” It is close, but the comparison is camera-only: RPE is measured on clips where nothing is being dragged, so it tells you the camera controller survived the LoRA, not that object manipulation never costs anything visually. The honest claim is narrower than “free”: adding object control through a small adapter keeps camera tracking intact (0.131 vs 0.237 under full fine-tuning), and at the long horizon WorldCraft’s 0.123 rotation / 0.0233 combined RPE at 253 frames actually beats the WorldPlay baseline. That long-horizon edge is the more interesting result than the headline trajectory number.

Key results

• Trajectory control (static camera): trajectory error 38.90px vs DragAnything 39.86 and Wan-Move 44.08; PSNR 17.23 vs 15.97-16.42; SSIM 0.616 vs 0.592-0.600; LPIPS 0.363 vs 0.375-0.468. A modest but consistent win across all four metrics, not a blowout.
• Camera fidelity, short horizon (61 frames): RPE rotation 0.131, translation 0.0161, combined 0.0170, with the LoRA in place.
• Camera fidelity, long horizon (253 frames): RPE rotation 0.123, translation 0.0225, combined 0.0233, beating WorldPlay at the extended horizon, which is where most autoregressive world models drift.
• NWT ablation: world-space trajectory with iterative depth refinement gives 30.82 trajectory error on small rotations vs 35.82 for the pixel-space variant, isolating the world-coordinate idea as the source of the accuracy gain.
• SP-LoRA ablation: ~50M parameters is the sweet spot; full fine-tuning to add the same control degrades camera RPE from 0.131 to 0.237, so the adapter, not the control signal, is what preserves the camera.
• Curriculum ablation: a Static-BI then Dynamic-AR training schedule prevents catastrophic interference between the static-trajectory and autoregressive stages.

What it was trained on, and the baselines

Training uses 27,027 filtered clips drawn from WISA-80K (8.5K), SpatialVID-HQ (24K), and OmniWorld (2K). Evaluation splits into a 50-clip trajectory-accuracy set, a 50-clip camera-fidelity set, and a 45-clip composable set stratified by rotation magnitude. The trajectory baselines (DragAnything, Wan-Move) are static-camera drag methods, so beating them on a static set is the fair comparison; the camera baselines (Yume, Matrix-Game 2.0, GameCraft, WorldPlay) cannot do object control at all, so the comparison there is about not regressing camera quality rather than head-to-head object accuracy. Worth keeping in mind: no prior method does both jobs, so some of WorldCraft’s “wins” are really “is the only one in the room that does X.”

Limits and open questions

The paper is candid about three failure modes. Monocular depth estimation degrades at large rotations, which is exactly when world-space re-projection needs depth to be right, so the NWT advantage narrows as the camera swings hard. Trajectory control runs at latent-token granularity (16x16 pixels), so small objects fall below the grid and cannot be moved precisely. And off-camera dynamics are limited to the trajectory you specified: TASP persists what you told it to move, it does not simulate independent world dynamics for things you are not looking at. There is also no user study reported, so “controllable” is measured by trajectory error and visual metrics rather than whether people can actually steer it to intent. For builders this is a strong recipe for adding draggable objects to an existing camera world model without retraining it, with the caveat that you inherit your depth estimator’s failure modes.

FAQ

What does WorldCraft add over a camera-only video world model?

Click-and-drag object manipulation. Existing interactive world models (Yume, Matrix-Game 2.0, GameCraft) let you move a camera through a generated scene but cannot move objects in it. WorldCraft adds a path: you click an object and sketch its trajectory, and the model animates that object along your path in world coordinates while the camera continues to navigate. It does this with a ~50M-parameter adapter on a frozen camera backbone, so the camera controller keeps working.

How does WorldCraft keep object control accurate when the camera moves?

Through Normalized World Trajectory. A drag drawn in screen pixels becomes meaningless once the viewpoint rotates, because the same world motion maps to different pixels. WorldCraft lifts your 2D drag into camera-invariant world coordinates using monocular depth, then re-projects it per frame from that frame’s camera pose. The ablation shows the world-space version reaches 30.82 trajectory error on small rotations versus 35.82 for a pixel-space baseline.

Does adding object manipulation to WorldCraft hurt camera fidelity?

Barely, and that is the design intent. Object control goes in through Spatial-Pathway LoRA (~50M params) rather than retraining the backbone. Under camera-only evaluation RPE rotation stays at 0.131; the ablation shows that doing the same thing with full fine-tuning blows RPE up to 0.237. At the 253-frame long horizon WorldCraft’s combined RPE of 0.0233 even beats the WorldPlay baseline. The measurement is camera-only, though, so this means the camera controller survives, not that object motion is visually cost-free.

Why can’t WorldCraft move small objects precisely?

Because trajectory control operates at latent-token granularity, roughly 16x16 pixels per token. An object smaller than that grid cannot be addressed precisely, so fine or small-object manipulation is out of reach. This is a resolution limit of the latent space, separate from the depth-estimation errors that hit the system at large camera rotations.

One line: WorldCraft adds click-and-drag object control to a camera-driven video world model by moving the drag into world coordinates and injecting it through a small LoRA, so the camera controller stays intact (0.131 RPE) while trajectory error beats DragAnything (38.90 vs 39.86). Read the original paper on arXiv.