VLM3 vs ZipSplat vs Function2Scene: Which Turns 2D Into 3D?

The actual question

“Which 3D model is best” is the wrong query, because these systems do different jobs that happen to share an input: ordinary 2D material. VLM3 answers questions about geometry: how far away is that point, where was the camera, do these two pixels correspond. ZipSplat produces geometry, a 3D Gaussian scene you can render from new viewpoints. Function2Scene produces arrangement, an indoor layout driven by what people will do in the room rather than by what objects look like. And DIRECT edits geometry into an existing image, giving you explicit control over the inserted object’s 3D pose, which diffusion inpainting never had. Pick the job first; the table below is organized that way.

Key numbers

You start from	You want	Model	Headline numbers, as reported	Where it stops being true
One or more photos	Depth, camera pose, pixel correspondence, object 3D, returned as answers	VLM3 (4B VLM)	0.904 avg δ₁ on metric depth across 8 datasets; 94.0% AUC@30° on camera pose (ETH3D + ScanNet++); 91.35% qualitative / 58.51% quantitative on object-level 3D; 15.37 px correspondence EPE	Correspondence still loses to expert matcher UFM’s 7.89 EPE, about half the error; the [0, 2000) text coordinates cap spatial precision
A handful of unposed photos	A renderable 3D scene	ZipSplat (feed-forward splatting)	24.14 PSNR on DL3DV at 24 views and only 249K Gaussians, pose-free, vs YoNoSplat 22.01 at 1.2M and DA3 21.69 at 6.1M; 26.20 PSNR on RealEstate10K at 6 views with 62K Gaussians; 401 FPS rendering at 192 views	Trails per-pixel splatting on LPIPS at sparse views (texture sharpness); k-means allocation starves flat surfaces; degrades on moving objects
A written design brief (who uses the room, for what)	A 3D indoor layout that supports those activities	Function2Scene (LLM + VLM check-and-repair loop)	Preferred over LLM scene-synthesis baselines in 94.3% of pairwise comparisons on 30 professional interior-design cases; 17 functional design criteria in the taxonomy	Human pairwise preference on 30 in-house cases, not a public benchmark; baselines unnamed in the abstract; layout quality, not photorealism
A photo plus a reference object, at a pose you choose	The object composited into the scene, pose-controllable	DIRECT (decomposed injection, visual proxy)	Abstract claims better geometric controllability and visual quality than prior insertion methods; no public metric values to quote	No numbers available to verify against; 2D compositing, not full-scene 3D output

Read the third column as four different products, not four candidates for one. The numbers also come from four different protocols, so none of the rows are directly comparable to each other; the table marks what each measured instead of pretending otherwise.

VLM3: 3D understanding without a 3D architecture

VLM3’s claim is the most radical of the four: a standard 4B vision-language model, with no cost-volume head, no depth decoder and no geometric loss, matches expert models on most 3D understanding tasks. The trick is not architecture but preparation. Focal length unification resizes every image to a 1000-pixel effective focal length, removing contradictory supervision from mixed cameras. Text-based pixel reference names pixels as integer coordinates in a [0, 2000) range instead of overlaying visual markers, so correspondence and pose tasks become plain text generation. The authors say data mixture weighting becomes almost the most important thing once those two fixes are in place.

The reported results back the strong version of the claim for three of four tasks. Depth: 0.904 average δ₁ across 8 datasets, matching UnidepthV2 and MoGe-2, and lifting DepthLM-7B’s 0.84 to 0.90 with fewer parameters. Camera pose: 94.0% AUC@30° on ETH3D and ScanNet++, statistically tying DA3-Giant’s 94.7% and beating VGGT. Object-level 3D: 91.35% qualitative and 58.51% quantitative accuracy, ahead of SpatialRGPT-8B on both. The honest exception is dense correspondence, where VLM3’s 15.37 EPE beats DKM and RoMa but is roughly twice the error of the specialist UFM at 7.89. If your task is to look at photos and answer geometry questions, VLM3 is the paper to start from, and its own limits section tells you exactly where a dedicated model still wins.

ZipSplat: reconstruction where the Gaussian budget follows the scene

ZipSplat answers a different question: given a few photos of a scene, give me something I can render. Its argument is that per-pixel feed-forward splatting wastes primitives. One Gaussian per input pixel means a blank wall and a leafy tree get the same budget, and the count explodes with view count. ZipSplat clusters dense backbone tokens with k-means into a small set of scene tokens and decodes each token into 32 Gaussians with free 3D positions, so flat regions collapse to a few tokens and detailed regions keep many. The compression ratio is an inference-time dial: one trained model spans the whole quality-versus-count curve.

The numbers are strong precisely because they hold the count down. On DL3DV with 24 input views, pose-free, it reports 24.14 PSNR / 0.768 SSIM / 0.198 LPIPS at 249K Gaussians, against YoNoSplat’s 22.01 at 1.2M and DA3’s 21.69 at 6.1M, better on every metric with about 5x fewer primitives. Swapping ZipSplat’s decoder for a per-pixel decoder on the identical backbone gives 1.5 to 2.7 dB higher PSNR with 13-25x fewer Gaussians, which isolates the decoder as the source of the gain. On RealEstate10K with only 6 views it reports 26.20 PSNR at 62K Gaussians, even edging DepthSplat’s 24.16 which was allowed ground-truth poses. The cost shows up in texture: at sparse views it trails per-pixel methods on LPIPS, and its k-means allocation can starve flat surfaces while over-serving vegetation. Note the relationship to VLM3. ZipSplat can run on VGGT or DA3-G backbones, the very expert models VLM3 argues a plain VLM can replace. The two papers quietly disagree about how much specialized 3D machinery you need, and both have data on their side.

Function2Scene: layouts driven by what the room is for

Function2Scene changes the input entirely. Instead of images, you write a functional brief. A family with two kids cooks together on weekends, say. The system generates a 3D indoor layout judged by how well it supports those occupants. It parses the brief into occupant personas and activities, derives constraints from a taxonomy of 17 functional design criteria spanning spatial, ergonomic, activity and environmental considerations, and then generates through an iterative check-and-repair loop that combines geometric measurement, LLM contextual reasoning and VLM visual assessment, rather than asking an LLM to emit a final scene in one shot.

The reported evaluation is small but concrete: on 30 professionally written interior-design cases, outputs were preferred over recent LLM-based scene synthesis baselines in 94.3% of pairwise comparisons. Treat that number for what it is, human preference on an in-house set with unnamed baselines, and the contribution reads as the task framing and the check-and-repair architecture rather than a leaderboard win. For simulation, robotics or game teams that need believable interiors driven by usage rather than appearance, it is the only one of the four aimed at that job.

DIRECT: the editing special case

DIRECT rounds out the picture for a narrower job: inserting a reference object into a background photo at a 3D pose you control interactively. Prior diffusion-based insertion is 2D inpainting with no explicit control over object pose. DIRECT decomposes the injection into appearance guidance from the reference, geometry guidance from a user-adjusted 3D proxy, and context guidance from the background, fed through separate pathways to avoid feature entanglement, plus an automated data-construction pipeline. The abstract states it outperforms previous methods in geometric controllability and visual quality, but gives no metric values, so there is nothing quotable here. Include it in this comparison for the job it defines, not for evidence of winning it.

When to use which

• Measure or query geometry in existing photos (depth maps, camera pose, point correspondences, object dimensions): VLM3. It needs no camera intrinsics thanks to focal unification, and a 4B model is cheap to run. Skip it for dense matching; UFM’s 7.89 EPE versus VLM3’s 15.37 is the one place a specialist clearly still wins.
• Rebuild a scene you can render, from a handful of photos, no poses available: ZipSplat. Pose-free is its headline, not a fallback. Feeding ground-truth poses adds only about 0.1 dB. Turn the compression ratio down when you have few views; the floor is roughly r = 0.5·sqrt(2/N) for N views.
• Generate an interior layout from a written brief: Function2Scene. Its 94.3% pairwise preference is on 30 in-house cases, so validate on your own floor plans before trusting it, but the functional-criteria framing is the right starting point for occupant-driven design.
• Composite one object into a photo at a chosen pose: DIRECT is the only one of the four designed for it, but with no public numbers, evaluate it on your own insertion cases before adopting.

Limits and open questions

These four papers do not share a benchmark, a task or an output format, so no head-to-head number exists between any pair of them; the table’s fourth column is the honest substitute. VLM3’s numbers come from benchmarks where expert models were tuned, and its correspondence caveat shows the “no 3D architecture” story is not complete. ZipSplat’s comparisons mix pose-free and pose-required baselines, and the paper says little about training data scale, which matters for reproduction. Function2Scene’s evaluation is 30 professional cases with unnamed baselines, the smallest evidence base here. And DIRECT contributes a task definition with no quantified results at all. The genuinely interesting open question cuts across all four: VLM3 argues generic transformers absorb 3D from data alone, while ZipSplat and DIRECT still build specialized geometric machinery on top. Which position the field lands on in 2027 will decide whether “3D models” remain a separate category.

FAQ

Is VLM3 better than expert 3D models?

On the tasks VLM3 reports, mostly yes for depth, camera pose and object-level 3D: 0.904 average depth δ₁ across 8 datasets, 94.0% pose AUC@30° statistically tying DA3-Giant’s 94.7%, and 91.35% qualitative accuracy on object-level 3D, all from a 4B general VLM. The exception is dense pixel correspondence, where the specialist matcher UFM reaches 7.89 EPE versus VLM3’s 15.37, about half the error.

Does ZipSplat need camera poses?

No. ZipSplat reconstructs and renders pose-free on both DL3DV and RealEstate10K, and feeding ground-truth poses as optional camera priors adds only about 0.1 dB. That is why its 24.14 PSNR on DL3DV at 24 views with 249K Gaussians, against DA3’s 21.69 at 6.1M Gaussians, is the headline comparison.

What is the difference between VLM3 and ZipSplat?

They produce different things from photos. VLM3 outputs answers about geometry, depth values, camera poses and correspondences, through text, with no scene representation. ZipSplat outputs an actual 3D Gaussian scene you can render from new viewpoints. If you want to query existing photos, use VLM3; if you want to walk around inside them, use ZipSplat.

Is Function2Scene’s 94.3% a benchmark result?

Not in the public-benchmark sense. It is the rate at which human judges preferred Function2Scene’s layouts over recent LLM-based scene synthesis baselines, across 30 professionally written interior-design cases. The baselines are not named in the abstract, so read it as evidence the check-and-repair loop beats direct LLM scene generation in that setting, not as a settled ranking.

Which one should I use for image editing with 3D control?

DIRECT. It is the only one of the four built for compositing a reference object into a photo at a user-chosen 3D pose, by injecting appearance, geometry and context guidance through separate pathways. Be aware its public abstract reports no metric values, so run your own evaluation before adopting it.