Low Rank Factorization-based for Deformable 3D Reconstruction
Discover how our proposed approach to 3D reconstruction tackles the challenges of non-rigid scenes, providing fast and robust results using factorization and neural radiance.
1
Task Force
ASRAR ALRUWAYQI
SHUBHAM TULSIANI
Adviser
2
Reconstructing Non-Rigid Scenes in 3D
Non-rigid reconstruction is a complex task of creating accurate depictions of objects that exhibit shape variability, such as dancing. Despite its challenges, scientists strive for the results to be as close to the original as possible. However, there are remaining problems that require further investigation.
3
Our drive is from:
NON-RIGID SCENES REFLECT THE TRUE NATURE OF OUR ENVIRONMENT.
FACTORIZATION ALLOWS US TO DECONSTRUCT COMPLEX DEFORMATIONS INTO MANAGEABLE COMPONENTS.
4
Problem Definition
Our objective is to develop an effective deep learning model that can synthesize new views at an arbitrary time and explicitly encode a dynamic scene that is being captured by a monocular camera and moving non-rigidly from a sparse set of images.
DYNAMIC SCENE FROM MONOCULAR CAMERA
GENRATE NEW VIEWS
5
Our Objectives
Decompose tensor and use one mode to reconstruct 3d scene.
integrate between explicitly and implicitly representation.
6
REF: NERF
Tensor Decompostion
Parallel Factors (PARAFAC), Canonical Decomposition (CANDECOMP), aka CP
7
THE CP DECOMPOSITION FACTORIZES A TENSOR INTO A SUM OF COMPONENT RANK-ONE TENSORS.
RER:BRETT W. BADER, TENSOR DECOMPOSITIONS AND APPLICATIONS∗
Tensor Decomposition Existing works:
TensoRF ECCV 2022 (static)
Left: CP decomposition, which factorizes a tensor as a sum of vector outer products.
Right: vector-matrix decomposition.
8
RER: TESNORF
Tensor Decomposition Existing works:
TensoRF ECCV 2022
9
RER: TESNORF
Tensor Decompostion
10
TENSORF
OUR EXPERMINT :
ONE MODE
RER: TESNORF
TensoRF Training Process
11
RER: TESNORF
Our Training Process
12
13
Tensor Decomposition Existing works:
Hex-Plane CVPR2023 (dynamic)
14
Tensor Decomposition Existing works:
Hex-Plane CVPR2023 (dynamic)
15
Tensor Decomposition Existing works:
K-Plane CVPR 2023 (dynamic)
16
17
low-rank performans
18
Failed Cases
19
Dealing with Deformable Scene Existing works
Nerfies ICCV 2021
20
Dealing with Deformable Scene Existing works
Nerfies ICCV 2021
21
Dealing with Deformable Scene Existing works
D-NerF CVPR 2020
22
Dealing with Deformable Scene Existing works
D-NerF CVPR 2020
23
Next Experiment
24
Timeline
LAUNCH
END
AUG 5
Brainstorming
SEPT 12
formulize solution
SEP 17
integrate bawneen hexplane and Nerfies
OCT 4
one mode experiments
NOV 1
factorize deformable felid
DEC 5
Optimization
25
Q/A