1 of 1

Optimization Objective

Task

​

Lift4D: Harmonizing Single-View 3D Estimation for 4D Reconstruction in-the-Wild

Yehonathan Litman, Xiaoxuan Ma, Manan Shah, Nicolas Ugrinovic, Kris Kitani, Fernando De La Torre, Shubham Tulsiani

MSCV Capstone Project

​

​

​

​

​

​

​

​

Qualitative Results

​

Challenge 1: Causal Single-view Reconstruction

Motivation

Conclusion and Future Work

​

​

Challenge 3: Occlusion-aware Rendering Supervision and Reconstruction

Task: 4D Reconstruction from Monocular In-the-Wild Video

Challenges:

  • Deformable objects
  • Occluded objects
  • Generalize to in-the-wild objects

Applications:

  • Graphics & Gaming: Generating Assets
  • AR/VR: Dynamic Environments
  • Embodied AI: Real2Sim Pipelines

Causal latent conditioning: Given a video input, we obtain per-frame 3D reconstructions with SAM3D using causal latent conditioning to enforce the temporal consistency across frames.

  • We introduced Lift4D, a test-time optimization framework that successfully recovers complete 4D dynamic objects from monocular video by harmonizing image-to-3D reconstructions as priors for further optimization.
  • The final reconstructions depend on the choice of hyperparameters and are prone to the failure modes of SAM3D.
  • We aim to improve the initial geometry prediction stage which currently uses SAM3D and aspire to obtain temporally consistent predictions natively, given a video.

Challenge 2: Deformable 3D Representation and Optimization

​

​

Deformable 3D Representation and Optimization: We factorize the 4D representation into canonical 3D gaussians and per-frame deformation fields parameterized by a sparse set of control nodes similar to SC-GS. Each node is associated with a time-varying learnable transformation . The per-frame initialized 3D gaussians obtained from SAM3D then guide the joint optimization of the canonical gaussians and the deformation field.

Dynamic Reconstruction and Tracking from Videos

Shape of Motion (ICCV 2025)

Generative 4D Novel View Synthesis

TrajectoryCrafter (ICCV 2025)

Feedforward Generative 4D Reconstruction

L4GM (NIPS 2024)

Prior-aided 4D Reconstruction

PAD3R (SIGGRAPH Asia 2025)

Encourages the deformed gaussians to align with the per-frame reconstruction from SAM3D

Enforces geometric and multi-view rendering consistency between the deformed gaussians and the per-frame reconstruction

T is a learnable transform that absorbs position inconsistency. The loss encourages structure of the deformed gaussians to match per frame initialization

Enforces cross-view consistency of rendered color and depth between the deformed gaussians and the per frame reconstruction

Regularizes the deformation field to encourage structured and temporally smooth motion

,

As-Rigid-As-Possible losses encourage sparse control nodes to move in a structured manner while preserving local rigidity

Penalizes rapid changes in control node motion across frames

,