Implicit Neural Representation Tutorial

Zhenyu Jiang

ICRA 2021, Xi’an, China

Signal Representations

How do we represent signals?

​

1. Image -> Discrete pixels�
2. 3D shape -> Voxels, point clouds, meshes�
3. Sound wave -> Discrete samples of intensity

​

​

Lose details when representing signals in dis crete manner!!

Sitzmann, Vincent, et al. "Implicit neural representations with periodic activation functions." Neurips 2020

Implicit Neural Representation

Instead of representing signals in a discrete manner, new approach has been studied called implicit neural representation.

Explicit representation

​

Spatial coordinate (x, y, z) in N

->Occupancy

​

​

1. Explicit way tends to lose details

2. Memory expensive (scales with resolution)

Implicit representation

​

Spatial coordinate (x, y, z) in R

->Occupancy

​

Applications

Implicit Neural representation is applicable to a variety of scientific fields:

​

1. Image/video/audio processing�
2. 3D data processing�
3. Solving boundary value problems

Overview

1. 3D Shape Representation

���

2. Structured Implicit Functions

���

3. Neural Rendering

Implicit Functions for 3D Reconstruction

3D Representation

Mescheder, Lars, et al. "Occupancy networks: Learning 3d reconstruction in function space." CVPR 2019.

3D Representation

3D Representation

Point Cloud

Pros

• Flexible and memory efficient�

Cons

• No topology
• Limited number of points

3D Representation

Mesh

Pros

• Flexible and memory efficient
• Well-defined topology�

Cons

• Need template or primitive
• Restricted to specific domain
• Hard to process with neural networks

3D Representation

What we want

• Can represent meshes of arbitrary topology and at arbitrary resolution,
• Is not restricted to a Manhattan world
• Is not limited by excessive memory requirements,
• Preserves connectivity information,
• Is not restricted to a specific domain (e.g. object class), and
• Blends well with deep learning techniques?

​

Implicit occupancy field

Occupancy Networks: Training objective

Encoder

Marching Cube

1. Divide the input volume into a discrete set of cubes. �
2. For each cube containing a section of the iso-surface, a triangular mesh that approximates the behavior of the trilinear interpolant in the interior cube is generated.

Occupancy Networks: Mesh Extraction

Multiresolution Iso-surface Extraction�

1. Building octree by incrementally querying the occupancy network. �
2. Extract triangular mesh using marching cube.�
3. Refine via 1st and 2nd order gradient information.

​

Memory Complexity

Single Image 3D Reconstruction

Testing on Real Data

DeepSDF

Occupancy

Whether a point is occupied by the object

Sign Distance Function

Signed distance to nearest surface

Park, Jeong Joon, et al. "Deepsdf: Learning continuous signed distance functions for shape representation.” CVPR 2019.

Single Shape DeepSDF

SDF

Coded Shape DeepSDF

SDF

feature

Auto-decoder-based DeepSDF

Auto-decoder

Each latent code z is paired with a training shape X.

​

Posterior over z

​

​

SDF likelihood

​

​

Optimization

​

​

Auto-decoder-based DeepSDF

Auto-decoder

​

Training

• A random initialized latent vector is assigned to each training shape.�
• Both the latent code and the decoder weights are optimized.

​

Inference

• The decoder weights are fixed, and an optimal latent vector is estimated.

Shape Reconstruction

Shape Completion

ONet VS DeepSDF

Similarity�

• Implicit Representations�
• Decoder architecture�
• Mesh extraction

​

Difference

​

Training data sampling

• ONet: uniform sample�
• DeepSDF: uniform + near-surface sample

​

Condition on input

• ONet: Encoder. Can handle any input data but need training for each.�
• DeepSDF: Auto-decoder. Can only handle input data that provide SDF-related information but no need to re-train the decoder

Structured Implicit Functions

Limitation of ONet

Peng, Songyou, et al. "Convolutional occupancy networks.” CVPR 2019.

ConvONets

ONet

​

• Encode the input into a global feature.�
• Limited expressiveness

​

ConvONets

​

• Encode the input into a structured feature grid.�
• Exploit convolutions to process feature.�
• Query local features for occupancy prediction.

​

Encoders

1. 3D input: point clouds or voxels�
2. Convert inputs in to point features�
3. Project point features onto one or multiple plane grids (Plane Encoder) or into a volume grid (Volume Encoder) using average pooling.�
4. Average the features that fall into the same pixel/voxel cell and use it as the pixel/voxel feature.

​

​

​

Decoders

1. Process structured feature grids with 2D/3D U-Nets.�
2. Query local features for input x at point p (bilinear/trilinear sampling)��
3. Predict occupancy conditioning on the local feature�

Quantitative Results

1. Better IoU than ONet with less GPU memory usage.�
2. Three 2D planes work better than 3D volumes with less GPU memory usage.�

Object Reconstruction

Voxel Super-resolution

Scene Reconstruction

Patch-based Scene Reconstruction

1. Structure feature grid representation have translation invariance
2. Crop the large input into different patch and process separately

PIFu: Pixel-Aligned Implicit Function for

High-Resolution Clothed Human Digitization

Saito, Shunsuke, et al. "Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization." ICCV 2019.

Surface reconstruction from image

Texture reconstruction from image

Inference

Multi-view PIFu

Multi-view Results

Implicit Function for Neural Rendering

Neural Radiance Field

Mildenhall, Ben, et al. "Nerf: Representing scenes as neural radiance fields for view synthesis." ECCV 2020.

Problem and Approach

Novel view synthesis: based on multiple/single view image(s) of a scene, synthesize novel view images.

​

NeRF first reconstruct neural radiance field of the scene then render the reconstructed scene to get novel view images.

Neural radiance field with implicit functions

Volume Rendering

1. For each pixel on the image, there is a corresponding ray.�
2. Sample points inside the volume along the ray.�
3. Accumulate the color and density of the sampled points to get color of the pixel.

Volume Rendering in NeRF

1. Query the color and density of the sampled points with the learned implicit functions.�
2. Color is view-dependent while density is not.�
3. Use pixel-level loss between the rendered view and the ground truth to optimize the implicit functions.

Differentiable Volume Rendering - Formulation

Differentiable Volume Rendering - Implementation

Technique – Positional encoding

1. Directly pass coordinates and angles into the implicit function(neural network) performs poorly at representing high-frequency variation in color and geometry.�
2. First map the input into a high dimensional cosine space.

Technique – Hierarchical volume rendering

1. First sample a set of points along the ray using stratified sampling.�
2. Given the output, compute the weight of each point:��
3. Normalize the weights and produce a piecewise-constant PDF along the ray.�
4. Samples from this distribution are biased towards the relevant parts of the volume.

Optimization

1. Loss: total squared error between the rendered and true pixel colors for both the coarse and fine renderings.���
2. Need over 100 images for training�
3. Convergence: a single scene takes 1-2 days on a single Nvidia V100.

View dependent radiance

Synthetic scenes

Real world scenes

Summary

Implicit Neural Representations:�

1. Effective output representation for shape, appearance, material�
2. No discretization, model arbitrary topology�
3. Structured implicit function improves the accuracy�
4. Can be learned from images via differentiable rendering�
5. Many applications: 3D reconstruction, view synthesis, etc.

Discussion

1. Geometry must be extracted in post-processing step, which is slow and produces artifacts. (hybrid representation, Neural Marching Cube…)���

​

• What’s the application of implicit neural representations in robotics? (Represent affordance, for local ad-hoc query …)

More on implicit neural representations

• 3D reconstruction
• IM-Net https://arxiv.org/abs/1812.02822�
• Structured implicit function
• NSVF https://arxiv.org/pdf/2007.11571
• Local Implicit Grid Representations for 3D Scenes�https://arxiv.org/abs/2003.08981�
• Neural rendering
• Scene Representation Networks https://arxiv.org/pdf/1906.01618
• Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision �https://arxiv.org/abs/1912.07372

​

More on implicit neural representations

• Affordance representation
• GIGA https://arxiv.org/abs/2104.01542
• Grasping Field: Learning Implicit Representations for Human Grasps �https://arxiv.org/abs/2008.04451

​

• Hybrid Representation
• BSP-Net https://arxiv.org/abs/1911.06971�
• NeRF Followups
• D-NeRF: https://arxiv.org/abs/2011.13961
• Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes�https://arxiv.org/abs/2011.13084
• i-NeRF: https://arxiv.org/pdf/2012.05877.pdf

​

​

​