K-文化的一切——从回归到 K-美妆，发送到您的邮箱订阅邮件

METAL MEDIA

GS-Voxel: Fitting-Free Structured Latents for Large-Scale 3DGS Generation

arXiv:2608.179882026-08-17

无需为每个场景重新优化,就能把数百万个散乱3D点整理成规整网格数据的方法

用3D Gaussian Splatting(3DGS)重建的航拍3D场景包含数百万个杂乱无章、数量各不相同的点,这种格式很难直接喂给期望规整网格输入的生成式AI模型。GS-Voxel通过确定性规则,把已经优化好的3DGS场景重新组织成稀疏体素网格,整个过程不需要针对每个场景再做优化,并以此作为表征,支撑一个根据图像条件生成大范围航拍3DGS场景的流水线。论文作者为Ming Qian一人,实验展示了从200米×200米小图块到最大1400米×800米大范围场景的生成效果。

METAL MEDIA 解读图

无需为每个场景重新优化,就能把数百万个散乱3D点整理成规整网格数据的方法

  1. 01问题:预先优化好的3DGS场景是杂乱无序的点集,数量差异极大(单个200米×200米图块可超过300万个基元),无法直接输入需要固定通道网格结构的稀疏生成模型。
  2. 02方法:GS-Voxel把场景划分为固定大小的体素(3D网格单元),每个体素内只保留不透明度最高的前16个点,并将位置、颜色等属性量化为8比特数值存储,整个过程不重新优化任何高斯点、也不把整场景硬套进单一固定模板,因此称为'免拟合(fitting-free)'。
  3. 03结构:用Geometry VAE专门编码体素的占据形状(分层细分结构),用另一个Local Attribute VAE专门编码被占据体素内高斯点的外观属性,分别压缩成两个结构化潜变量。
  4. 04生成:以卫星/航拍图像为条件的三阶段流匹配(flow matching)模型依次预测粗略结构、精细几何形状、最终属性;再通过重叠感知的分块推理把多个图块拼接起来,合成远超训练时200米×200米图块范围的大场景。
  5. 05效果:在10个验证图块上,仅做GS-Voxel转换就能较好保留原始3DGS的渲染质量;完整生成流水线在作者自己的评测协议下取得FID 28.0、KID 0.020(与其他论文数值因数据集不同不可直接比较)。
这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

  1. 问题:预先优化好的3DGS场景是杂乱无序的点集,数量差异极大(单个200米×200米图块可超过300万个基元),无法直接输入需要固定通道网格结构的稀疏生成模型。
  2. 方法:GS-Voxel把场景划分为固定大小的体素(3D网格单元),每个体素内只保留不透明度最高的前16个点,并将位置、颜色等属性量化为8比特数值存储,整个过程不重新优化任何高斯点、也不把整场景硬套进单一固定模板,因此称为'免拟合(fitting-free)'。
  3. 结构:用Geometry VAE专门编码体素的占据形状(分层细分结构),用另一个Local Attribute VAE专门编码被占据体素内高斯点的外观属性,分别压缩成两个结构化潜变量。
  4. 生成:以卫星/航拍图像为条件的三阶段流匹配(flow matching)模型依次预测粗略结构、精细几何形状、最终属性;再通过重叠感知的分块推理把多个图块拼接起来,合成远超训练时200米×200米图块范围的大场景。
  5. 效果:在10个验证图块上,仅做GS-Voxel转换就能较好保留原始3DGS的渲染质量;完整生成流水线在作者自己的评测协议下取得FID 28.0、KID 0.020(与其他论文数值因数据集不同不可直接比较)。
Figure 1: Overview of our structured-latent pipeline. We first convert a compatible pre-optimized 3DGS reconstruction into GS-Voxel, a sparse representation of the retained local Gaussian parameters, and then encode it with a factorized VAE. The Geometry VAE models hierarchical subdivision and occupancy, while the Local Attribute VAE models Gaussian attributes on the decoded support.
Figure 1: Overview of our structured-latent pipeline. We first convert a compatible pre-optimized 3DGS reconstruction into GS-Voxel, a sparse representation of the retained local Gaussian parameters, and then encode it with a factorized VAE. The Geometry VAE models hierarchical subdivision and occupancy, while the Local Attribute VAE models Gaussian attributes on the decoded support.
Table 1: Taxonomic comparison of representative outdoor 3D scene creation systems.
MethodTraining DataConditionBeyond BuildingsLarge-ScaleCore Paradigm
Sat2Scene 13Images & Height MapLayout✗✗Geo. Coloring
Sat2City 7MeshHeight Map✗✓Building Object Gen.
Sat2Density++ 24ImagesSingle Satellite✓✗Feedforward
Sat3DGen 25ImagesSingle Satellite✓✗Feedforward
UrbanWorld 29Mesh + UVOSM or Layout✓✓Multi-Stage
SynCity 3N/A (training-free)Text Prompt✓✓Multi-Stage
SkyFall-GS 11ImagesMulti-view Satellite✓✓Iterative Optimization
Orbit2Ground 40ImagesMulti-view Satellite✓✓Iterative Optimization
XCube 27MeshLiDAR Scan✓✓3D Generative
EarthCrafter 17MeshSatellite + Depth✓✓3D Generative
Ours3DGSSingle Satellite-View Image✓✓3D Generative
Figure 2: Deterministic conversion from a compatible pre-optimized 3DGS reconstruction to GS-Voxel.
Figure 2: Deterministic conversion from a compatible pre-optimized 3DGS reconstruction to GS-Voxel.
Table 2: Per-layer configuration of the virtual aerial camera rig. Pitch is measured as the deviation from the nadir direction. For pitch=0∘ a single yaw is used (top-down view); for pitch>0∘ four compass yaws {0∘,90∘,180∘,270∘} are sampled. Views/cell counts cameras per x​y grid cell; Views/window assumes a 10×10 grid.
LayerzFOVPitches (deg)Views/cellViews/window
00.026∘{0,30,45,60}131,300
10.318∘{15,30,45}121,200
20.614∘{0,15,30}9900
30.914∘{15}4400
41.214∘{0}1100
Total393,900
Figure 4: Examples of retained and rejected aerial renders. Filtering removes unreliable or severely degraded supervision; only the retained views supervise the Local Attribute VAE.
Figure 4: Examples of retained and rejected aerial renders. Filtering removes unreliable or severely degraded supervision; only the retained views supervise the Local Attribute VAE.
Table 3: Direct GS-Voxel conversion before VAE encoding, evaluated on 10 validation tiles. We report average retained-primitives and active-voxel counts rounded to the nearest thousand (K), together with rendered reconstruction quality. Bold settings denote our default configuration.
RKinAvg. Retained PrimitivesAvg. Active VoxelsSSIM ↑PSNR ↑LPIPS ↓
2561377K377K0.5518.520.425
2564971K377K0.8526.820.166
25681,340K377K0.9533.260.058
256161,602K377K0.9840.040.023
256321,737K377K0.9840.940.021
5121980K980K0.8727.380.156
51241,501K980K0.9839.200.025
Figure 5: Tile-level 3DGS generations conditioned on rendered satellite-view images.
Figure 5: Tile-level 3DGS generations conditioned on rendered satellite-view images.
Table 4: Choice of Kout for the Local Attribute VAE, evaluated using rendered-image reconstruction quality.
Method / SettingPSNR ↑SSIM ↑LPIPS ↓
Kout=122.810.610.350
Kout=4 (Ours)23.090.620.331
Kout=822.120.580.354
Figure 6: Large-area 3DGS generation (1/2) from constructed satellite-view conditions.
Figure 6: Large-area 3DGS generation (1/2) from constructed satellite-view conditions.
Table 5: Separate evaluation of the Geometry VAE and Local Attribute VAE. Shape IoU measures support reconstruction; PSNR, SSIM, and LPIPS measure attribute reconstruction on the target support.
ModelShape IoU ↑PSNR ↑SSIM ↑LPIPS ↓
Single-stage VAE0.7621.010.540.418
Geometry VAE0.99–––
Local Attribute VAE–23.090.620.331
Figure 7: Large-area 3DGS generation (2/2). Overlap-aware tiled inference produces a large-area 3DGS primitive set from the satellite-view condition.
Figure 7: Large-area 3DGS generation (2/2). Overlap-aware tiled inference produces a large-area 3DGS primitive set from the satellite-view condition.
Table 6: Cross-paper reference for image-level FID/KID. Baselines use different ground-truth sets and camera protocols; values are not directly comparable and no ranking is implied.
MethodFIDKID
CityDreamer 3797.30.096
GaussianCity 3886.90.090
EarthCrafter 1769.50.061
Ours28.00.020

为什么重要

对无人机测绘、城市仿真、灾害应对分析等需要快速生成大范围逼真3D场景的应用而言,这项工作让真实的、已重建好的3DGS数据可以直接用于训练生成模型,而不必为每个场景单独优化或限定固定的点数预算。作者也说明这仍是早期研究,目前仅限于简单的视角无关颜色(SH0)航拍场景。

本文术语

  • 3D Gaussian Splatting(3DGS) · 用大量半透明3D椭球形'点'(高斯)而非网格来表示逼真场景的渲染技术
  • 体素(Voxel) · 3D网格中的一个单元格,相当于2D图像中像素的三维版本
  • VAE(变分自编码器) · 一种训练用于把数据压缩成紧凑潜变量、再重建还原的神经网络
  • 流匹配(Flow matching) · 一种学习从噪声逐步变换为目标数据的生成模型方法,与扩散模型思路类似
  • 免拟合(Fitting-free) · 仅用固定的确定性规则进行转换,不需要针对每个场景做优化或套用模板

无法转载的图表

  • Figure 3: Conditional generation pipeline. Given a satellite-view image, a coarse sparse-structure stage predicts scene support; GS-Voxel-specific geometry and attribute stages then generate fine support and Gaussian parameters, yielding a standard 3DGS primitive set.
在原文中查看图表 →

论文原文摘要（英文）

Many scalable latent 3D generators operate on structured tensors, whereas pre-optimized 3D Gaussian Splatting (3DGS) reconstructions are unordered, spatially irregular, and vary widely in primitive count. We present GS-Voxel, a fitting-free structured latent framework, and evaluate it for large-scale aerial 3D Gaussian scene generation. GS-Voxel deterministically converts a compatible pre-optimized 3DGS reconstruction into sparse active voxels without additional per-scene optimization, retaining the sub-voxel positions and rendering attributes of the selected primitives. A GS-specific factorized VAE then separately encodes voxel geometry and local Gaussian attributes into sparse 3D latents whose size grows with the number of occupied voxels rather than being limited by a fixed scene-wide primitive count. We train image-conditioned flow models in the GS-Voxel latent space to generate aerial 3DGS scenes. A key application enabled by GS-Voxel is large-area scene generation: overlap-aware tiled inference extends synthesis beyond a single training crop conditioned on satellite-view images. Our results show that GS-Voxel provides structured latents for pre-optimized aerial 3DGS reconstructions, with latent capacity that grows with the number of occupied voxels.

作者 · Ming Qian

在 arXiv 阅读

最新论文

全部论文 →

METAL MEDIA 最新报道

图片来源： Ming Qian et al., arXiv:2608.17988, CC BY 4.0