1 of 35

OpenLive3D Update

Wei Chen – 2023-09-30

2 of 35

OpenLive3D.Core

ML

Core Management

Convertor

OpenLive3D.Module

Media Manager

Face-Key

Model Manager

Config Manager

Controller

GUI

OpenLive3D-v2 Current State

Hand-Key

Health Manager

Interpolator

Main Controller

Entry Manager

Multi-

language

GUI

Control

Pose-Key

Face-mesh + Worker

Holistic + Worker

Log Manager

Meta-Key Manager

Core

Manager

Arm Magic

Landmark Manager

POI Manager

Website / App

HTML

Interface Manager

3 of 35

OpenLive3D Update

Wei Chen – 2023-09-22

4 of 35

Agenda – 2023-09-22

  • What was the Plan again?
  • What had been achieved?
  • What are the TO-DOs left?
  • Other TO-DO ideas
  • Possible Timeline
  • Make more progress today!

5 of 35

Features – 2023-09-11

  • Separation of Core ML with UI
  • Separate Electron UI from the Web
  • Single Thread Option
  • Full-Body Capture
  • Validate the integration with 3PD BlendShape API
  • Shaders (Visual Effect)
  • Mobile Support
    • If Single Thread is Working on Mobile
    • Direct Video Recording
    • Integrated as BlendShape API
  • Better GUI

6 of 35

Timeline

  • Week 1
    • 2023-09-12~2023-09-14: Separation of Core ML with UI
    • 2023-09-15: Separate Electron UI from the Web
    • 2023-09-15: Single Thread Option

​

  • Week 2
    • 2023-09-18: Hand-Tracking Icon Change
    • 2023-09-19: Study the integration with 3PD BlendShape API
    • 2023-09-20: Shaders (Visual Effect) (1 Shader Type Left)
    • 2023-09-21: Mobile Support (1 Task Left)
    • 2023-09-22: Better GUI

7 of 35

TO-DOs

Not Finished Items:

  • Full-Body Capture > New Mediapipe Libraries (Pose-Detection) [5]
  • Mobile Support > Video Auto-play (with Reference Already) [2]
  • More Effects > Need Particle System [2]

​

Other Ideas:

  • Background to Visual Effect [0]
  • Modularize [1]
  • 52 Blend-Shapes > New Mediapipe Libraries (Facial BlendShape) [1]
  • Recording Function [3]
  • Tutorials [3]
  • Quick Effects UI [4]

[Finished]

[Today]

[Future]

8 of 35

OpenLive3D Update

Wei Chen – 2023-09-21

9 of 35

Mobile Support

  • Entry Check Modification
  • Create Test Mobile Entry Page
  • Production Validation
  • Logging Addition & Evaluation (Mobile Log Window)
    • What is working / What is not?
  • Get the Code to Work Somehow
    • Might fail

​

10 of 35

Problems

  1. Mobile is blocked
  2. Safari is blocked
  3. Safari Worker initialization
  4. Safari Video auto-play failed (TODO)
  5. Safari ML result not passed to VRM (VRM disappear)

11 of 35

OpenLive3D Update

Wei Chen – 2023-09-18

2023-09-19

2023-09-20

12 of 35

Timeline

  • Week 1
    • 2023-09-12~2023-09-14: Separation of Core ML with UI
    • 2023-09-15: Separate Electron UI from the Web
    • 2023-09-15: Single Thread Option

​

  • Week 2
    • 2023-09-18: Hand-Tracking Icon Change
    • 2023-09-19: Study the integration with 3PD BlendShape API
    • 2023-09-20: Shaders (Visual Effect)
    • 2023-09-21: Mobile Support
    • 2023-09-22: Better GUI
    • Future Work: Full-Body Capture, 52 Blend-Shapes, More Effects, …

​

13 of 35

Shaders (Visual Effects)

Target

  • Build the foundation for visual effects
  • Develop 1 to 2 sample effects

​

Process

  • Inventory possible effects
  • Code structure design + sample selection
  • Code structure + sample development
  • Test

Sample Effects from Shoost v0.8.2

14 of 35

Inventory Possible Effects

  • Screen Modification
    • Screen Effects
    • Color Correction
    • Blur Sharpen (Sample Done)
  • 3D Object Changes
    • Flare / Lighting (Sample Done)
    • Camera
  • Background Modification
    • LED Panel
  • Foreground Modification
    • Weather Effects
    • Particle Effects

Sample Effects from Shoost v0.8.2

15 of 35

OpenLive3D Update

Wei Chen – 2023-09-15

16 of 35

OpenLive3D.Core

ML

Core Management

Convertor

Website / App

Media Manager

Face-Key

Model Manager

VRM Models

Config Manager

Configs

Controller

Live3D GUI

OpenLive3D-v2 State

Hand-Key

Health Manager

Interpolator

Main Controller

Entry Manager

Multi-language

GUI

Control

Pose-Key

Face-mesh + Worker

Holistic + Worker

Log + POI

Meta

Manager

Arm Magic

Landmark Manager

17 of 35

2023-09-15 Target x 2

  • Electron UI Separation
    • Inventory the difference between current electron version vs original version
    • Extract the electron version specific functions
    • Add OpenLive3D.core submodule
    • Copy and paste latest OpenLive3D-v2
    • Modify to the electron version

​

  • Thread Manager
    • Code check to compare worker vs single thread
    • Call ML within single thread
    • Thread control

18 of 35

Electron UI Difference

19 of 35

OpenLive3D Update

Wei Chen – 2023-09-12

& 2023-09-13

& 2023-09-14

20 of 35

Core Separation Plan Current Status

  • Define Controller Interface
    • Arm-Magic + Interpolator
    • Camera + ML
  • Git Submodule Establishment
  • Copy and Paste
    • Holistic + Face-mesh
  • Test Initial Separation
  • Detail Code Check
  • Adjust the Code
  • Final Test

​

21 of 35

Core Separation Plan - Core Function

The core will have these modules:

- Camera Control

- ML Module

- Motion Convertor

- Interpolator

- Thread Control

OL3DCore

  • init(config)
  • update(config)
  • getVRMMovement()
  • getLandMark()
  • getVideo()

22 of 35

Timeline

  • Week 1
    • 2023-09-12~2023-09-14: Separation of Core ML with UI
    • 2023-09-15: Separate Electron UI from the Web
    • 2023-09-15: Single Thread Option

​

  • Week 2
    • Full-Body Capture
    • Can have integration with 3PD BlendShape API
    • Shaders (Visual Effect)
    • Mobile Support
    • Better GUI

23 of 35

OpenLive3D Design

Wei Chen – 2023-09-11

24 of 35

APP

ML Landmark Recognizer

Media Source Manager

Face-Key

Model Manager

VRM Models

Config Manager

Configs

Controller

Live3D GUI

Current State

Hand-Key

Arm Magic

Interpolator

Main Controller

Entry Manager

Multi-language

GUI

Control

Pose-Key

Main-Landmark-Code

Face-mesh

Holistic

Face-mesh Worker

Holistic

Worker

25 of 35

Features

  • Separation of Core ML with UI
  • Separate Electron UI from the Web
  • Single Thread Option
  • Full-Body Capture
  • Can have integration with 3PD BlendShape API
  • Shaders (Visual Effect)
  • Mobile Support
    • If Single Thread is Working on Mobile
    • Direct Video Recording
    • Integrated as BlendShape API
  • Better GUI

26 of 35

Timeline (Deprecated)

  • Week 1
    • 2023-09-12~2023-09-13: Separation of Core ML with UI
    • 2023-09-14: Separate Electron UI from the Web
    • 2023-09-15: Single Thread Option

​

  • Week 2
    • Full-Body Capture
    • Can have integration with 3PD BlendShape API
    • Shaders (Visual Effect)
    • Mobile Support
    • Better GUI

27 of 35

Reference

​

​

28 of 35

OpenLive3D Design

Wei Chen – 2021-08-25

29 of 35

Camera

APP

Media Source Manager

Facial Landmark Recognizer

Model Manager

VRM Models

Config Manager

Configs

Controller

Live3D GUI

Effect Manager

State Manager

30 of 35

Live3D UI Functions

Running State

  • Interaction
    • Zoom In / Out Canvas
    • Rotate Canvas
    • Grab Move Canvas Position
    • Select Camera / Config
    • Select Model / Mood
    • Apply Effects
    • Open Setting
  • Display
    • Head Position, Rotation Track
    • Eye Open, Iris Position Track
    • Mouth Open, Mouth Shape Track

Setting State

  • Select Config
  • Create / Remove / Update Config
  • Modify
    • Select Model / Mood
    • Head Position Rotation Boundary
    • Left Right Eye Blink Threshold
    • Left Right Iris Boundary
    • Mouth Open Threshold
    • Mouth Shape Feature Mapping
  • Close Setting

31 of 35

Controller Functions

Initialize

  • List / Start Default Camera
  • Load Facial Landmark ML Model
  • Initialize Default State

Event Trigger

  • Select and Load Camera / Config
  • Create / Remove / Update Config

Loop

  • Screen Capture
  • Extract Facial Landmark
  • Update State
  • Generate Canvas
  • Apply Effects

32 of 35

Other Functions

Effect Manager

  • List Effects
  • Return Canvas with Applied Effects

Model Manager

  • List / Load VRM Models

State Manager

  • Camera-ID / Config-ID
  • Canvas Zoom / Rotate / Position
  • Current Effect List

Media Source Manager

  • List / Load Cameras
  • Capture Camera Screen

Facial Landmark Recognizer

  • Load ML Model
  • Return Landmark from Images

Config Manager

  • List Configs for a VRM Model
  • Load Config
  • Create / Remove / Update Config

33 of 35

Setting State Design

Model

Wei

Config

Wei

Min / Max

-20.0

-20.0

Head Roll

-20.0

Head Pitch

-20.0

Head Yaw

-20.0

Mood

Normal

Left Eye

-20.0

Right Eye

-20.0

Left Iris

-20.0

Right Iris

-20.0

Mouth Open

-20.0

Mouth Width

-20.0

Min / Max

-20.0

-20.0

Min / Max

-20.0

-20.0

Threshold

-20.0

Threshold

-20.0

Min / Max

-20.0

-20.0

Min / Max

-20.0

-20.0

Threshold

-20.0

O / A

-20.0

-20.0

Camera

Default

FPS: 30

34 of 35

Running State Design

Model

Wei

Config

Wei

Head Roll

-20.0

Head Pitch

-20.0

Head Yaw

-20.0

Mood

Normal

Left Eye

-20.0

Right Eye

-20.0

Left Iris

-20.0

Right Iris

-20.0

Mouth Open

-20.0

Mouth Width

-20.0

Camera

Default

FPS: 30

35 of 35

Streaming State Design

Flip

Normal