GS-Voxel: Fitting-Free Structured Latents for Large-Scale 3DGS Generation

arXiv:2608.179882026-08-17

A rule-based way to turn millions of scattered 3D points into tidy grid data, without re-optimizing each scene

Aerial 3D scenes captured with 3D Gaussian Splatting (3DGS) contain millions of unordered points whose count varies wildly, which is a bad fit for generative AI models that expect grid-shaped data. GS-Voxel deterministically reorganizes an already-optimized 3DGS scene into a sparse voxel grid without any extra per-scene optimization, and uses this representation to power an image-conditioned pipeline that generates large aerial 3DGS scenes. The sole author, Ming Qian, shows generation from 200m x 200m tiles up to areas as large as 1,400m x 800m.

METAL MEDIA explanatory visual

A rule-based way to turn millions of scattered 3D points into tidy grid data, without re-optimizing each scene

01Problem: pre-optimized 3DGS scenes are unordered point sets with wildly varying counts (over 3 million primitives in a 200m x 200m tile), which cannot be fed directly into sparse latent generative models expecting fixed-channel spatial grids.
02Solution: GS-Voxel splits the scene into fixed-size voxels, keeps only the top-16 highest-opacity points per voxel, and quantizes their position and appearance attributes into 8-bit values -- all without re-optimizing any Gaussian or fitting the whole scene to one global template, hence 'fitting-free'.
03Architecture: a Geometry VAE encodes which voxels are occupied (hierarchical subdivision), while a separate Local Attribute VAE encodes the Gaussian appearance attributes inside occupied voxels, producing two compact structured latents.
04Generation: a three-stage flow-matching pipeline conditioned on a satellite/aerial image progressively predicts coarse structure, then fine geometry, then attributes; overlap-aware tiled inference stitches tiles together to synthesize areas far larger than the 200m x 200m training crop.
05Results: on 10 validation tiles, direct GS-Voxel conversion preserved rendering quality close to the original 3DGS; the full generative pipeline achieved FID 28.0 and KID 0.020 under the authors' own evaluation protocol (not directly comparable to numbers from other papers, which use different datasets).

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

Problem: pre-optimized 3DGS scenes are unordered point sets with wildly varying counts (over 3 million primitives in a 200m x 200m tile), which cannot be fed directly into sparse latent generative models expecting fixed-channel spatial grids.
Solution: GS-Voxel splits the scene into fixed-size voxels, keeps only the top-16 highest-opacity points per voxel, and quantizes their position and appearance attributes into 8-bit values -- all without re-optimizing any Gaussian or fitting the whole scene to one global template, hence 'fitting-free'.
Architecture: a Geometry VAE encodes which voxels are occupied (hierarchical subdivision), while a separate Local Attribute VAE encodes the Gaussian appearance attributes inside occupied voxels, producing two compact structured latents.
Generation: a three-stage flow-matching pipeline conditioned on a satellite/aerial image progressively predicts coarse structure, then fine geometry, then attributes; overlap-aware tiled inference stitches tiles together to synthesize areas far larger than the 200m x 200m training crop.
Results: on 10 validation tiles, direct GS-Voxel conversion preserved rendering quality close to the original 3DGS; the full generative pipeline achieved FID 28.0 and KID 0.020 under the authors' own evaluation protocol (not directly comparable to numbers from other papers, which use different datasets).

Figure 1: Overview of our structured-latent pipeline. We first convert a compatible pre-optimized 3DGS reconstruction into GS-Voxel, a sparse representation of the retained local Gaussian parameters, and then encode it with a factorized VAE. The Geometry VAE models hierarchical subdivision and occupancy, while the Local Attribute VAE models Gaussian attributes on the decoded support.

Table 1: Taxonomic comparison of representative outdoor 3D scene creation systems.
Method	Training Data	Condition	Beyond Buildings	Large-Scale	Core Paradigm
Sat2Scene 13	Images & Height Map	Layout	✗	✗	Geo. Coloring
Sat2City 7	Mesh	Height Map	✗	✓	Building Object Gen.
Sat2Density++ 24	Images	Single Satellite	✓	✗	Feedforward
Sat3DGen 25	Images	Single Satellite	✓	✗	Feedforward
UrbanWorld 29	Mesh + UV	OSM or Layout	✓	✓	Multi-Stage
SynCity 3	N/A (training-free)	Text Prompt	✓	✓	Multi-Stage
SkyFall-GS 11	Images	Multi-view Satellite	✓	✓	Iterative Optimization
Orbit2Ground 40	Images	Multi-view Satellite	✓	✓	Iterative Optimization
XCube 27	Mesh	LiDAR Scan	✓	✓	3D Generative
EarthCrafter 17	Mesh	Satellite + Depth	✓	✓	3D Generative
Ours	3DGS	Single Satellite-View Image	✓	✓	3D Generative

Figure 2: Deterministic conversion from a compatible pre-optimized 3DGS reconstruction to GS-Voxel.

Table 2: Per-layer configuration of the virtual aerial camera rig. Pitch is measured as the deviation from the nadir direction. For pitch=0∘ a single yaw is used (top-down view); for pitch>0∘ four compass yaws {0∘,90∘,180∘,270∘} are sampled. Views/cell counts cameras per xy grid cell; Views/window assumes a 10×10 grid.
Layer	z	FOV	Pitches (deg)	Views/cell	Views/window
0	0.0	26∘	{0,30,45,60}	13	1,300
1	0.3	18∘	{15,30,45}	12	1,200
2	0.6	14∘	{0,15,30}	9	900
3	0.9	14∘	{15}	4	400
4	1.2	14∘	{0}	1	100
Total	39	3,900

Figure 4: Examples of retained and rejected aerial renders. Filtering removes unreliable or severely degraded supervision; only the retained views supervise the Local Attribute VAE.

Table 3: Direct GS-Voxel conversion before VAE encoding, evaluated on 10 validation tiles. We report average retained-primitives and active-voxel counts rounded to the nearest thousand (K), together with rendered reconstruction quality. Bold settings denote our default configuration.
R	Kin	Avg. Retained Primitives	Avg. Active Voxels	SSIM ↑	PSNR ↑	LPIPS ↓
256	1	377K	377K	0.55	18.52	0.425
256	4	971K	377K	0.85	26.82	0.166
256	8	1,340K	377K	0.95	33.26	0.058
256	16	1,602K	377K	0.98	40.04	0.023
256	32	1,737K	377K	0.98	40.94	0.021
512	1	980K	980K	0.87	27.38	0.156
512	4	1,501K	980K	0.98	39.20	0.025

Figure 5: Tile-level 3DGS generations conditioned on rendered satellite-view images.

Table 4: Choice of Kout for the Local Attribute VAE, evaluated using rendered-image reconstruction quality.
Method / Setting	PSNR ↑	SSIM ↑	LPIPS ↓
Kout=1	22.81	0.61	0.350
Kout=4 (Ours)	23.09	0.62	0.331
Kout=8	22.12	0.58	0.354

Figure 6: Large-area 3DGS generation (1/2) from constructed satellite-view conditions.

Table 5: Separate evaluation of the Geometry VAE and Local Attribute VAE. Shape IoU measures support reconstruction; PSNR, SSIM, and LPIPS measure attribute reconstruction on the target support.
Model	Shape IoU ↑	PSNR ↑	SSIM ↑	LPIPS ↓
Single-stage VAE	0.76	21.01	0.54	0.418
Geometry VAE	0.99	–	–	–
Local Attribute VAE	–	23.09	0.62	0.331

Figure 7: Large-area 3DGS generation (2/2). Overlap-aware tiled inference produces a large-area 3DGS primitive set from the satellite-view condition.

Table 6: Cross-paper reference for image-level FID/KID. Baselines use different ground-truth sets and camera protocols; values are not directly comparable and no ranking is implied.
Method	FID	KID
CityDreamer 37	97.3	0.096
GaussianCity 38	86.9	0.090
EarthCrafter 17	69.5	0.061
Ours	28.0	0.020

Why it matters

This matters for applications like drone mapping, urban simulation, or disaster-response analysis that need large, photorealistic 3D scenes generated quickly, because it lets real, already-reconstructed 3DGS data be used to train generative models without per-scene optimization or a fixed global point budget. The authors note this is still an early-stage method limited to simple view-independent (SH0) color aerial scenes.

Terms in this paper

3D Gaussian Splatting (3DGS) · a rendering technique that represents a scene as many semi-transparent 3D ellipsoid 'blobs' (Gaussians) instead of a mesh
Voxel · a single cell in a 3D grid, the 3D equivalent of a 2D pixel
VAE (Variational Autoencoder) · a neural network trained to compress data into a compact latent representation and reconstruct it back
Flow matching · a generative modeling technique that learns to gradually transform noise into target data, similar in spirit to diffusion models
Fitting-free · converting data using fixed deterministic rules only, with no per-scene optimization or template fitting

Figures we cannot republish

Figure 3: Conditional generation pipeline. Given a satellite-view image, a coarse sparse-structure stage predicts scene support; GS-Voxel-specific geometry and attribute stages then generate fine support and Gaussian parameters, yielding a standard 3DGS primitive set.

See the figures in the original paper →

Original abstract (English)

Many scalable latent 3D generators operate on structured tensors, whereas pre-optimized 3D Gaussian Splatting (3DGS) reconstructions are unordered, spatially irregular, and vary widely in primitive count. We present GS-Voxel, a fitting-free structured latent framework, and evaluate it for large-scale aerial 3D Gaussian scene generation. GS-Voxel deterministically converts a compatible pre-optimized 3DGS reconstruction into sparse active voxels without additional per-scene optimization, retaining the sub-voxel positions and rendering attributes of the selected primitives. A GS-specific factorized VAE then separately encodes voxel geometry and local Gaussian attributes into sparse 3D latents whose size grows with the number of occupied voxels rather than being limited by a fixed scene-wide primitive count. We train image-conditioned flow models in the GS-Voxel latent space to generate aerial 3DGS scenes. A key application enabled by GS-Voxel is large-area scene generation: overlap-aware tiled inference extends synthesis beyond a single training crop conditioned on satellite-view images. Our results show that GS-Voxel provides structured latents for pre-optimized aerial 3DGS reconstructions, with latent capacity that grows with the number of occupied voxels.

Authors · Ming Qian

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Ming Qian et al., arXiv:2608.17988, CC BY 4.0