A 3D-Driven Framework for Moving-Camera Perspective-to-360° Video Generation

Abstract

Turning a moving-camera perspective video into a 360° video requires more than just generating content outside the input frame: observations revealed over time must be preserved, unseen regions completed, and the resulting scene kept consistent as the camera viewpoint changes. Existing methods assign these different requirements to a single conditional video model, requiring them to be jointly learned from panoramic video data. We instead present a modular, 3D-driven framework that distributes these three responsibilities across specialized components, all connected through a shared scene representation. Input observations are reconstructed explicitly in 3D, while a pretrained panoramic image model completes unobserved content from a selected panoramic render. This generated content is lifted into the shared 3D scene, where it remains spatially anchored across camera viewpoints. Without requiring any additional panoramic-video training, our framework achieves state-of-the-art input-view fidelity, perceptual quality, and geometric consistency on a static-scene subset of the Argus evaluation set. These results support our decomposed, 3D-driven formulation as a promising alternative to direct video generation in this setting.

Gallery

Interactive 360° viewing requires a local server. From this folder, run python3 -m http.server 8000, then open http://localhost:8000.

Use the arrows to move between scenes. Click any panorama to explore it as an interactive 360° viewport.

Ablation Example

Click any panorama to explore it as an interactive 360° viewport.

BibTeX

@inproceedings{wakeham2027pano3d,
  title={A 3D-Driven Framework for Moving-Camera Perspective-to-360° Video Generation},
  author={Wakeham, Michael and Das, Mrinmoy Kumar and Wang, Wenqi and Tewari, Ayush and Gong, Boqing and Ghadiyaram, Deepti},
  booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
  year={2027}
}