General Protocol for Face Scanning applied to Virtual Planning in Orthognathic Surgery  and Rhinoplasty   - Comparison between Tools

Cícero Moraes

3D Designer

Dr. Everton da Rosa

Dentist

This work aimed to address the use of photogrammetry in obtaining the frontal region of the face for use in digital 3D surgical planning.

We encourage you to translate this material into other languages. We ask you only to consult us beforehand to organize the publication

 (http://fb.me/ciceromoraes3d).

Objectives of the study

Getting photos

For the scanning of the faces  26 photos were taken in arches with two differents levels.

The camera pointed to the center of the face and the distance between the photo takings was approximately 7 ° 30 'or 7.5 ° (decimal) until 90 degrees of total rotation.

The difference in height was about 35%, one of which is above the horizon line and the other is below looking to photograph the lower part of the nose.

P.S.: It is not necessary to follow the distance and the degrees exposed in the didactic material, what is worth is to approach the desired areas.

Photography and data collection

1st Taking:

2nd Taking

 One of the individuals that accepted to participate in the first shot was photographed in 5 different environments:

In almost all environments (except for 1 and 2) the face was photographed with the following settings:

3rd Taking: One of the faces was photographed again in the same place as the first shot, however this time without makeup (present in the first shot).

Device data

Camera

Galaxy J7 Metal (SM-J710MN)

Camera Brand: Samsung

Camera Model: SM-J710MN

Flash not enabled

Exposure Program: Normal Program

Focal length: 3.7

Computer

CPU: Intel (R) Core (TM) i7-3770 CPU @ 3.40GHz

Memory RAM: 11.70GB Physical Memory 0B Virtual Memory

OS: Xubuntu Minimal 17.10, Linux 4.10.0-42-generic (x86_64)

GPU: NVIDIA Corporation GF116 [GeForce GTS 450 Rev. 2]

Board Name: P8H61-M LE / BR (ASUSTeK COMPUTER INC)

HD Model: WDC WD10EZEX-00W

Area of interest, alignment and resizing

All scans have been rotated, resized and sectioned using the features of the OrtogOnBlender addon.

When imported into Blender 3D software, the faces were aligned and resized using the intercantal distance as  a parameter.

Sample video of alignment and resizing:https://youtu.be/MTfQLnKjK0o

The defined  area of interest is the frontal portion of the face, since the simulations of orthognathic surgery and rhinoplasty usually focus on this region.

In addition to the area of interest limiting the anterior area of the face , it is essential for this type of model the use of a few faces, in order to facilitate the real-time manipulations required in the proposed procedures.

Sample  Video of the selection area of interest:

 https://youtu.be/h-bFvhLp-8g

Programs for photogrammetry

All the following tools were configured to perform the photogrammetry process in less than 20 minutes, a limit that was exceeded in isolated moments, but remained within the proposed one when generated global averages.

MVE/SMVS + Meshlab + MVS Texturing

  1. Camera Alignment (SMVS)
  2. Dense Point Cloud (SMVS)
  3. Reconstruction of 3D mesh (Meshlabserver script)
  4. 3D Mesh Texturing (MVS Texturing)

MVE/SMVS + MVS Texturing

  1. Camera Alignment
  2. Dense cloud of dots
  3. 3D Mesh Reconstruction
  4. Cleaning the 3D mesh
  5. 3D mesh texturing

OpenMVG + OpenMVS

  1. Camera alignment and sparse cloud of points (OpenMVG)
  2. Dense Dot Cloud (OpenMVS)
  3. 3D Mesh Reconstruction (OpenMVS)
  4. 3D Mesh Texturization (OpenMVS)
  5. Export to OBJ (Meshlabserver script)

Photoscan

        This software has its own workflow where it performs all the steps of the photogrammetry process.

1st Photo Shoot Results

Code: 000

Sex: Female

Age: 66

Ancestry: European

Total shooting time of 26 photos: 29 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) MVE/SMVS+MVS Texturing

3) OpenMVG+OpenMVS

4) Photoscan

Scan time MVE/SMVS+Meshlab+Mvs Texturing: 13 minutes

Scan time MVE/SMVS+Mvs Texturing: 16 minutes

Scan time OpenMVG+OpenMVS: 10 minutes

Photoscan Scan Time: 24 minutes

- Missing region on bottom of the face (1 and 2).

- Slight irregularity  in the upper left part of the face (4).

-  Missing region on the lower and lateral sides of the face (1 and 2).

- Irregularity on the lateral surface and bottom of the face (4).

- Images of the missing region on the bottom of the face (1 and 2).

- Irregularity in the lower part of the face (4).

- Irregularity in the lower part of the nose (2).

Code: 001

Sex: Male

Idade: 42

Ancestry: European

Total shooting time of 26 photos: 28 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) MVE/SMVS+MVS Texturing

3) OpenMVG+OpenMVS

4) Photoscan

Scan time MVE / SMVS + Meshlab + Mvs Texturing: 15 minutes

Scan time  MVE/SMVS+Mvs Texturing: 16 minutes

Scan time OpenMVG+OpenMVS: 11 minutes

Scan time Photoscan: 18 minutes

- Problems with scanning at the tip of the nose due to the brightness (1 and 2).

- Irregular generation on the right side of the face (3).

- Irregular generation of the left side of the face (4).

- The problem with the brightness at the tip of the nose is evidential (1 and 2).

- Difference in the base of the nose in relation to other models (3).

- Difficulty of visualization of the details due to the irregularity of the mesh (4).

- Irregularity evinced at the nose tip  (1 and 2).

- Irregularity in the nose region evinced on the left side of the face (4).

- Nasal asymmetry evinced in image  3.

- Irregularities at the tip of the nose (1 and 2).

- Irregularity at the base of the nose (3).

- Irregularity in the structure of the nose and difficulty in visualizing the details of the face (4).

- Irregularity at the nose tip evinced (1 and 2).

- Irregularity at the base of the nose (3).

- Irregularity along the face, with less intensity than on the left side (4).

Code: 002

Sex: Female

Age: 62

Ancestry: European

Total shooting time of 26 photos: 27 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) MVE/SMVS+MVS Texturing

3) OpenMVG+OpenMVS

4) Photoscan

Scan time MVE / SMVS + Meshlab + Mvs Texturing: 13 minutes

Scan time MVE/SMVS+Mvs Texturing: 16 minutes

Scan time  OpenMVG+OpenMVS: 10 minutes

Scan time  Photoscan: 21 minutes

- Irregularity in the upper face (4).

- Missing region on the right side of the face (1 and 2).

- Missing surface in the lower region (1 and 2).

- Missing region on the lower and lateral surface of the face (1 and 2).

- Irregularity in the upper, lateral and nasal regions (4).

Code: 003

Sex: Female

Age: 29

Ancestry: Mixed-race

Total shooting time of 26 photos: 31 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) MVE/SMVS+MVS Texturing

3) OpenMVG+OpenMVS

4) Photoscan

Scan time MVE/SMVS+Meshlab+Mvs Texturing: 13 minutes

Scan time MVE/SMVS+Mvs Texturing: 16 minutes

Scan time OpenMVG+OpenMVS: 9 minutes

Scan time Photoscan: 15 minutes

- Irregularity in the right surface of the face  and the eyes area(3).

- Irregularity in the lower part (4).

 Demonstration of irregularity on the right side of the face (3).


- Enlarged nasal asymmetry (3).

- Missing lower region (1 and 2).

- Lower nasal region missing (2).

- Generalized irregularity in the right part (3).

- Irregularity in the lower part (4).

- irregularity evinced at the tip of the nose (1 and 2).

- irregularity evinced in the lower part of the face (4).

- Missing region at the bottom of the face (1 and 2).

- Small missing region on the lower part of the nose (2).

- Demonstration of the irregularity in the right part of the face (3).

-  Irregularity evinced in the lower part of the face (4).

Code: 004

Sex: Male

Age: 44

Ancestry: European

Total shooting time of 26 photos: 33 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) MVE/SMVS+MVS Texturing

3) Photoscan

Scan time MVE/SMVS+Meshlab+Mvs Texturing: 14 minutes

Scan time  MVE/SMVS+Mvs Texturing: 15 minutes

Scan time  OpenMVG+OpenMVS: ERROR (did not generate 3D)

Scan time  Photoscan: 21 minutes

- Missing region at the base of the nose and face (2).

- Missing region in the lower part of the nose (2).

- Slight irregularity on the right side of the face, close to the eye (3).

Code: 005

Sex: Male

Age: 69

Ancestry: European

Total shooting time of 26 photos: 29 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) MVE/SMVS+MVS Texturing

3) Photoscan

Scan time MVE/SMVS+Meshlab+Mvs Texturing: 15 minutes

Scan time MVE/SMVS+Mvs Texturing: 17 minutes

Scan time OpenMVG+OpenMVS: ERROR (did not generate 3D)

Scan time Photoscan: 18 minutes

- Missing region  in the hair (1 and 2).

- Irregularity in the nose region (3).

- Region missing in the hair (1 and 2).

- Missing region on the right side of face (2).

- Irregularity in the nose region.

- Missing region on the left side of the nose.

Code: 006

Sex: Female

Age: 22

Ancestry: Mixed-race

Total shooting time of 26 photos: 26 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) MVE/SMVS+MVS Texturing

3) OpenMVG+OpenMVS

4) Photoscan

Scan time  MVE/SMVS+Meshlab+Mvs Texturing: 13 minutes

Scan time MVE/SMVS+Mvs Texturing: 15 minutes

Scan time OpenMVG+OpenMVS: 10 minutes

Scan time Photoscan: 15 minutes

- Irregularity at the tip of the nose (1 and 2).

- Lack of coverage in face scanning, mainly in the right part (3).

- Irregularity below the eyes, on the right side, at the border with the nose (4).

- Demonstration of irregularity in the tip of the nose (1 and 2).

- Missing region on the side of the face (2).

- Missing region at the bottom of the face (1 and 2).

Demonstration of the irregularity in the tip of the nose (1 and 2).

Code: 007

Sex: Female

Age: 29

Ancestry: African

Total shooting time of 26 photos: 27 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) MVE/SMVS+MVS Texturing

3) OpenMVG+OpenMVS

4) Photoscan

Scan time MVE/SMVS+Meshlab+Mvs Texturing: 13 minutes

Scan time  MVE/SMVS+Mvs Texturing: 14 minutes

Scan time OpenMVG+OpenMVS: 9 minutes

Scan time  Photoscan: 13 minutes

- Irregularity at the tip of the nose (1 and 2).

- Irregularities along the face (3).

- Irregularities in the region of the nose and lower face.

- Irregularity in the tip of the nose (1 and 2).

- Irregularities along the face (3 and 4).

- Small missing region at the base of the nose (2).

Code: 008

Sex: Female

Age: 9

Ancestry: European

Total shooting time of 26 photos: 28 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) MVE/SMVS+MVS Texturing

3) OpenMVG+OpenMVS

4) Photoscan

Scan time  MVE/SMVS+Meshlab+Mvs Texturing: 12 minutes

Scan timeMVE/SMVS+Mvs Texturing: 14 minutes

Scan time OpenMVG+OpenMVS: 8 minutes

Scan time Photoscan: 12 minutes

- Irregularities at the base of the face (2).

- Generalized irregularity and severe impairment of face shape (3).

- Irregularities along the face and lack of sharpness (4).

- Irregularity at the tip of the nose (1 and 2).

- Images of irregularities along the face (2 and 3).

Code: 009

Sex: Female

Age: 45

Ancestry: European

Total shooting time of 26 photos: 28 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) MVE/SMVS+MVS Texturing

3) OpenMVG+OpenMVS

4) Photoscan

Scan time  MVE/SMVS+Meshlab+Mvs Texturing: 14 minutes

Scan time  MVE/SMVS+Mvs Texturing: 16 minutes

Scan time  OpenMVG+OpenMVS: 9 minutes

Scan time  Photoscan: 21 minutes

- Generalized irregularity in the upper part of the face (1 and 2).

- Generalized irregularity and severe impairment of face shape (3).

- Irregularity in the upper left side of the face (4).

- Missing region at the bottom of the face (1 and 2).

- Images of irregularities (1, 2, 3 and 4).

Code: 010

Sex: Female

Age: 11

Ancestry: Mixed-race

Total shooting time of 26 photos: 30 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) MVE/SMVS+MVS Texturing

3) OpenMVG+OpenMVS

4) Photoscan

Scan time  MVE/SMVS+Meshlab+Mvs Texturing: 13 minutes

Scan time  MVE/SMVS+Mvs Texturing: 15 minutes

Scan time  OpenMVG+OpenMVS: 9 minutes

Photoscan scanning time: 15 minutes

- Generalized irregularity and severe impairment of face shape (3).

- Lack of sharpness (4).

- Missing region at the base of face (1 and 2).

- Missing region on the lips (2).

- Irregularity at the tip of the nose (1 and 2).

Code: 011

Sex: Female

Age: 37

Ancestry: Mixed-race

Total shooting time of 26 photos: 24 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) Photoscan

Scan time  MVE/SMVS+Meshlab+Mvs Texturing: 13 minutes

Scan time  MVE/SMVS+Mvs Texturing: ERROR (did not generate 3D)

Scan time  OpenMVG+OpenMVS: ERROR (did not generate 3D)

Scan time  Photoscan: 15 minutes

- Irregularity at the tip of the nose (1).

- Slight lack of sharpness (2).

- Missing region at the base of the face (1).

- Irregularity on the right side of the face (2).

- Irregularity in the tip of the nose (1).

Code: 012

Sex: Masculino

Age: 20

Ancestry: European

Total shooting time of 26 photos: 25 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) MVE/SMVS+MVS Texturing

3) OpenMVG+OpenMVS

4) Photoscan

Scan time MVE/SMVS+Meshlab+Mvs Texturing: 12 minutes

Scan time MVE/SMVS+Mvs Texturing: 14 minutes

Scan time OpenMVG+OpenMVS: 9 minutes

Scan time Photoscan: 15 minutes

- Irregularities on the right side of the face (2).

- Nose tip bulging (1).

- Slight irregularity in the tip of the nose (4).

Results Table

ID

Age

Ancestry

Time photos

MVE/SMVS+Meshlab

MVE/SMVS+MVS Texturing

OpenMVG+OpenMVS

Photoscan

000

66

EURO

29

13

16

10

24

001

42

EURO

28

15

16

11

18

002

62

EURO

27

13

16

10

21

003

29

MIX

31

13

16

9

15

004

44

EURO

33

14

15

ERROR

21

005

69

EURO

29

15

17

ERROR

18

006

22

MIX

26

13

15

10

15

007

29

AFR

27

13

14

9

13

008

9

EURO

28

12

14

8

12

009

45

EURO

28

14

16

9

21

010

11

MIX

30

13

15

9

15

011

37

MIX

24

12

ERROR

ERROR

17

012

20

EURO

25

12

14

9

15

Average

37

28

13

15

9

17

Results of the 2nd photo shoot

Description: External environment in the shade - natural

Total shooting time of 26 photos: 27 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) OpenMVG+OpenMVS

3) Photoscan

Scan time  MVE/SMVS+Meshlab+Mvs Texturing: 12 minutes

Scan time OpenMVG+OpenMVS: 6 minutes

Scan time Photoscan: 16 minutes

- Irregularity in the texture present in the upper left, at the base of the face and nose (1).

- Generalized irregularity in the mesh (2 and 3).

- Standardized visualization for surgical simulation (texture solid + ambient occlusion): Excellent (1), Regular (2), Good (3).

- Small missing region on lateral surface of the face (1).

- Light unevenness along the face (2).

- Irregularity on the right side of the face.

Description: External environment in the shade - dots in the nose

Total shooting time of 26 photos: 30 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) OpenMVG+OpenMVS

3) Photoscan

Scan time MVE/SMVS+Meshlab+Mvs Texturing: 11 minutes

Scan time OpenMVG+OpenMVS: 7 minutes

Scan time Photoscan: 19 minutes

The use of dots (skin markers) in the nose helps the photogrammetry algorithm to find references to its calculations, in addition it is the simplest elements to be removed in Blender Texture Mode, or in a post production process in the Gimp with the aid of the plug-in Heal Selection.

Example of using Heal Selection:

 https://www.facebook.com/ciceromoraes3d/videos/418460871657551/

Description: Indoor environment largely illuminated by external light - natural

Total shooting time of 26 photos: 24 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) OpenMVG+OpenMVS

3) Photoscan

Scan time MVE/SMVS+Meshlab+Mvs Texturing: 12 minutes

Scan time OpenMVG+OpenMVS: 7 minutes

Scan time Photoscan: 16 minutes

- Texture problems in the upper right part, the tip of the nose and the extreme left (1).

- Irregularity on the lateral surface of the face (3).

- Small missing region on the lateral surface of the face (1).

Description: Indoor environment largely illuminated by external light - dots on the nose

Total shooting time of 26 photos: 22 seconds

Order (from left ):

1) MVE/SMVS+Meshlab+MVS Texturing

2) OpenMVG+OpenMVS

3) Photoscan

Scan time MVE/SMVS+Meshlab+Mvs Texturing: 12 minutes

Scan time  OpenMVG+OpenMVS: 7 minutes

Scan time  Photoscan: 16 minutes

- Texture problems in the upper right part, the tip of the nose and the extreme left (1).

- Small missing area on the lateral surface of the face (1).

- Small irregularity on the lateral surface of the face (3).

- Demonstration of the missing region on the lateral surface of the face (1).

Description: Outside environment illuminated by the right - natural

Total shooting time of 26 photos: 22 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) OpenMVG+OpenMVS

3) Photoscan

Scan time MVE/SMVS+Meshlab+Mvs Texturing: 12 minutes

Scan time OpenMVG+OpenMVS: 7 minutes

Scan time  Photoscan: 21 minutes

- Texture problem along the face (1).

- Compromised 3D reconstruction (2).

- Small irregularities on the right side of the face (3).

- Irregularity at the tip of the nose (1).

-  Nasal asymmetry (1).

- Evidence of poor surface clarity (3).

- Small irregularities on the lateral surface of the face (3).

- Small missing region on the lateral surface of the face (1).

Description: Internal environment externally illuminated by the right - dots on the nose

Total shooting time of 26 photos: 29 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) OpenMVG+OpenMVS

3) Photoscan

Scan time MVE/SMVS+Meshlab+Mvs Texturing: 13 minutes

Scan time OpenMVG+OpenMVS: 9 minutes

Scan time  Photoscan: 21 minutes

- Contrast problems in texture (1).

- Severe structural defect of the face (2).

- Slight lack of sharpness in the structure (3).

- Small missing regions on the lateral surface of the face and nose (1).

Description: Internal environment externally illuminated by the right - nose risk markers

Total shooting time of 26 photos: 23 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) OpenMVG+OpenMVS

3) Photoscan

Scan time MVE/SMVS+Meshlab+Mvs Texturing: 11 minutes

Scan time OpenMVG+OpenMVS: 7 minutes

Scan time Photoscan: 14 minutes

Making dots/risks over an object surface that exhibits homogeneity in color helps the photogrammetry algorithm to find references to its calculations. If on the one hand this trick helps in obtaining a more coherent model with reality structurally speaking, on the other hand it can entail a great work if it is necessary to later remove the texture. This technique, however, is very useful for patients who have suffered loss of part of the face  or even limb. The 3D digitization can serve as a basis for making prostheses. Another current use is for the comparison of pre and post surgical cases, since the purpose of such an activity is linked more the anatomical structure than to the texture of the facial surface.

- Contrast problems in texture (1).

- Irregularities on the left side of the face (2 and 3).

- Irregularities along the left side of the face (2 and 3).

- Small missing region on the lateral surface of the face (1).

Description: Internal environment externally lit by the right - make-up

Total shooting time of 26 photos: 37 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) OpenMVG+OpenMVS

3) Photoscan

Scan time MVE/SMVS+Meshlab+Mvs Texturing: 12 minutes

Scan time OpenMVG+OpenMVS: 4 minutes

Scan time Photoscan: 16 minutes

- Contrast problem in texture generation (1).

- Projection problem due to region not matching with the original surface (2).

- Irregularity on the left side of the face (2 and 3).

- Missing regions on the lateral surface of the face (1).

- Structural impairment (2).

- Small missing regions on the lateral surface of the face (1).

- Nasal asymmetry (1).

- Low definition of facial surface details (3).

- Demonstration of lateral irregularities (2 and 3).

Missing regions at lateral surface of the face.

Description: Internal environment externally illuminated by the right - make-up and dots

Total shooting time of 26 photos: 34 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) OpenMVG+OpenMVS

3) Photoscan

Scan time MVE/SMVS+Meshlab+Mvs Texturing: 13 minutes

Scan time  OpenMVG+OpenMVS: 7 minutes

Scan time  Photoscan: 18 minutes

- Contrast problem in texture generation (1).

- Missing region on the lateral surface  of the face (1).

- Irregularities in the upper part with more intensity and other regions with less intensity (2).

- Irregularity on the lateral surface of the face (3).

- Missing region on the lateral surface of the face (1).

- Irregularity on the lateral surface of the face (3).

Description: Illuminated indoor environment - natural

Total shooting time of 26 photos: 29 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) OpenMVG+OpenMVS

3) Photoscan

Scan time  MVE/SMVS+Meshlab+Mvs Texturing: 10 minutes

Scan time OpenMVG+OpenMVS: ERROR (did not generate 3D)

Scan timePhotoscan: 12 minutes

- Contrast problem in texture generation in the lower and left side of the face (1).

- Irregularity at the tip of the nose (1).

- Lack of sharpness along the face (2).

- Irregularity in the tip of the nose (1).

- Slight irregularity in the lower part of the nose (2).

Description: Illuminated indoor environment - Dots on the nose

Total shooting time of 26 photos: 28 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) OpenMVG+OpenMVS

3) Photoscan

Scan time  MVE/SMVS+Meshlab+Mvs Texturing: 12 minutes

Scan time OpenMVG+OpenMVS: 4 minutes

Scan time Photoscan: 13 minutes

- Contrast problem in texture generation on the lateral surface of the face (1).

- Irregularity in the lower part of the face (2).

- Generalized irregularity on the surface of the mesh (3).

- Small irregularity at the tip of the nose (1).

- Small missing region on the lateral surface of the  face (1).

- Irregularities on the lateral surface of the face (2 and 3).

- Small irregularity in the base of the nose (3).

Description: Illuminated indoor environment - risks on the nose

Total shooting time of 26 photos: 25 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) OpenMVG+OpenMVS

3) Photoscan

Scan time MVE/SMVS+Meshlab+Mvs Texturing: 11 minutes

Scan time OpenMVG+OpenMVS: 6 minutes

Scan time Photoscan: 10 minutes

- Contrast problems in texture generation (1).

- Irregularities in the lower part of the face (2).

- Missing region on the lateral surface  of the face (3).

- Evidence of irregular projection on the face (2).

- Small missing region on the lateral surface of the  face (1).

- Evidence of irregularities on the lateral surface of the face (2 and 3).

Description: Enlightened inner environment - makeup

Total shooting time of 26 photos: 37 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) OpenMVG+OpenMVS

3) Photoscan

Scan time MVE/SMVS+Meshlab+Mvs Texturing: 12 minutes

Scan time OpenMVG+OpenMVS: 3 minutes

Scan time Photoscan: 10 minutes

- Slight problem with the texture on the lateral ends (1).

- Irregularity at the tip of the nose (1).

- Structural impairment (2).

- Low detail sharpness (3).

- Irregularity at the tip of the nose.

- Irregularities on the lateral  surface of the face (3).

Description: Enlightened inner environment - makeup and dots

Total shooting time of 26 photos: 30 seconds

Order (from left to right:

1) MVE/SMVS+Meshlab+MVS Texturing

2) OpenMVG+OpenMVS

3) Photoscan

Scan time  MVE/SMVS+Meshlab+Mvs Texturing: 11 minutes

Scan time  OpenMVG+OpenMVS: 4 minutes

Scan time  Photoscan: 11 minutes

- Small deficience in the texture, located in the lower right part of the face (1).

 Generalized irregularity in the right side of the face (2 and 3).

Description: Indoor environment dimly lit - natural

Total shooting time of 26 photos: 29 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) OpenMVG+OpenMVS

3) Photoscan

Scan Time  MVE/SMVS+Meshlab+Mvs Texturing: 11 minutes

Scan Time OpenMVG+OpenMVS: ERROR (did not generate 3D)

Scan Time Photoscan: 11 minutes

- Light irregularity on the median right side of the face (1).

- Irregularity at the tip of the nose and the lower part of the face (1).

- Irregularity generalized with greater and lesser degree along the face, little sharpness (2).

Description: Indoor environment dimly lit - Dots on the nose

Total shooting time of 26 photos: 31 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) OpenMVG+OpenMVS

3) Photoscan

Scan time MVE/SMVS+Meshlab+Mvs Texturing: 11 minutes

Scan time OpenMVG+OpenMVS: 5 minutes

Scan time Photoscan: 9 minutes

- Slight irregularity in the tip of the nose (1 and 3).

- Facial structure compromised (2).

- Irregularity between the eyes and the lower part of the face (1).

- Irregularities along the face (3).

-Demonstration of the missing region on the lateral side of the face (1).

- Evidence of irregularity at the base of the face (3).

Description: Indoor environment dimly lit - nose risks

Total shooting time of 26 photos: 27 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) OpenMVG+OpenMVS

3) Photoscan

Scan time MVE/SMVS+Meshlab+Mvs Texturing: 11 minutes

Scan time OpenMVG+OpenMVS: 4 minutes

Scan time Photoscan: 8 minutes

- Problems with the contrast in texture (1).

- Irregularities along the face (2).

- Lack of sharpness in the marginal regions of the face (3).

- Irregularity at the tip of the nose (1).

- Small missing region on side of face (1).

-Description: Indoor environment dimly lit - makeup

Total shooting time of 26 photos: 30 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) OpenMVG+OpenMVS

3) Photoscan

Scan time  MVE/SMVS+Meshlab+Mvs Texturing: 11 minutes

Scan time OpenMVG+OpenMVS: ERROR (Did not generate 3D)

Scan time Photoscan: 6 minutes

- Contrast problem in texture (1).

- Irregularity at the tip of the nose (1).

- Irregularities along the face and lack of sharpness (2).

Description: Indoor environment dimly lit - makeup and dots

Total shooting time of 26 photos: 30 seconds

Order (from left to right):

1) MVE/SMVS+Meshlab+MVS Texturing

2) OpenMVG+OpenMVS

3) Photoscan

Scan time  MVE/SMVS+Meshlab+Mvs Texturing: 11 minutes

Scan time OpenMVG+OpenMVS: 5 minutes

Scan time Photoscan: 8 minutes

- Small irregularity of the texture in the upper part of the face (1).

- Commitment of the structure (2).

- Irregularities along the marginal regions of the face (3).

Table of results

PLACE

Time 26 photos

MVE/SMVS+Meshlab

OpenMVG+OpenMVS

Photoscan

External shade - natural

27

12

6

16

External shade - nose dots

30

11

7

19

Internal environment largely illuminated by external light - natural

24

12

7

19

Internal environment widely illuminated by external light - nose dots

22

12

7

16

Internal environment externally illuminated by the right side - natural

22

12

7

21

Internal environment externally illuminated by the right side -  nose dots

29

13

9

21

Internal environment externally illuminated by the right side- nose risks

23

11

7

14

Internal environment externally Illuminated by the right side- makeup

37

12

4

16

Internal environment externally Illuminated by the right side- make-up and nose dots

34

13

7

18

Illuminated indoor environment - natural

29

10

ERROR

12

Illuminated internal environment - nose dots

28

12

4

13

Illuminated internal environment - nose risks

25

11

6

10

Illuminated inner environment - makeup

37

12

3

10

Illuminated inner environment - makeup and nose dots

30

11

4

11

Poorly illuminated internal environment - natural

29

11

ERROR

11

Poorly illuminated indoor environment - nose dots

31

11

5

9

Poorly illuminated indoor environment - nose risks

27

11

4

8

Poorly illuminated indoor environment - makeup

30

11

ERROR

6

Poorly illuminated indoor environment - makeup and nose dots

30

11

5

8

AVERAGE

29

12

6

14

Results of the 3rd photo shoot

In the first phase of the project, the volunteer code 003 (female, 45 years old, European) was photographed with facial makeup. Given the results, it was decided to make a new photo-taking without makeup in order to attest to the effectiveness of the algorithms in such a configuration. The place and time of the shot were the same as in the first phase.

Total shooting time of 26 photos: 25 seconds

Ordem (da esquerda para a direita):

1) MVE/SMVS+Meshlab+MVS Texturing

2) MVE/SMVS+MVS Texturing

3) OpenMVG+OpenMVS

4) Photoscan

Scan time MVE/SMVS+Meshlab+Mvs Texturing: 14 minutes

Scan time MVE/SMVS+Mvs Texturing: 15 minutes

Scan Time  OpenMVG+OpenMVS: 8 minutes

Scan time Photoscan: 21 minutes

Results of the 3rd photo shoot.

Results of the 1st photo shoot..

The difference in results was striking as attested in the above images. The absence of make-up has contributed to a significant increase in the quality of the scan.

Results and Analysis

Average time for taking 26 photos: 28 seconds

Average MVE / SMVS + Meshlab + MVS Texturing: 12 minutes

Average MVE / SMVS + MVS Texturing: 15 minutes

Average OpenMVG + OpenMVS: 7 minutes

Average Photoscan : 15 minutes

MVE/SMVS+Meshlab+MVS Texturing

Clean, clear and light at the same time, perfect for real-time soft tissue dynamics (ie.orthognathic surgery) and digital sculpture (ie.rhinoplasty).

Automatic aperture recognition and other camera features.

Open source software.

Optimum relation between processing time and result.

It works on command line, which allows for good integration with Blender through Python script.

It runs on Windows, Linux and MacOSX.

When the lighting is not homogeneous there is a tendency to present texture problems.

Algorithms do not deal well with glossy surfaces (nose tips).

Deletes the regions where the texture can not be projected.

It does not display many details on the created surface.

MVE/SMVS+MVS Texturing

Automatic aperture recognition and other camera features.

Open source software.

Good level of surface detail.

It works on command line, which allows for good integration with Blender through Python script.

It runs on Windows, Linux and MacOSX.

When the lighting is not homogeneous there is a tendency to present texture problems.

Algorithms do not deal well with glossy surfaces (nose tips).

Deletes the regions where the texture can not be projected.

It does not present a good relationship between processing time and result.

OpenMVG+OpenMVS

Open source software.

Fast processing.

Excellent tool for creating textures.

It works on command line, which allows for good integration with Blender through Python script.

It runs on Windows, Linux and MacOSX.

It does not have automatic recognition of focal aperture and other features of the camera, which can compromise the calculations.

It does not display many details on the created surface.

It presents irregular results and was not successful in most reconstructions, especially of young individuals.

Photoscan

Automatic aperture recognition and other camera features.

Affordable in its standard version.

Great level of surface detail.

Excellent tool for creating textures.

It runs on Windows, Linux and MacOSX.

It does not present a good relationship between processing time and result.

Despite the detailing the meshes generated sin very much in the question of sharpness.

The standard version does not allow the use of Python script which makes it difficult to communicate directly with Blender.

Problems, solutions and recommendations

Texture with contrast problem.

Manual camera setup (Pro).

Face scanned in 3D in MVE / SMVS + Meshlab with the most homogeneous texture.

Irregularities at the tip of the nose (glossy surface).

Powder for brightness reduction.

Dots (skin pencil) made just after the powder placement just in the nose region. If only the dots were painted without the makeup, the glow would still be emitted in the black regions. We can still see a slight glow in the image, even with all the care taken.

Texture Mode with dots.

Texture Mode, shortly after hiding the dots made with the help of the Smear tool. Hiding the dots in the texture can also be done with the Heal Selection filter in the Gimp image editor.

Before and after hiding the dots in the texture.

Lack of surface detail.

Missing regions or small holes in the face

Missing regions or small holes in the face

Conclusions

The results presented in this study show that a system based on open source photogrammetry and simple smartphones is quite possible.

Both the OpenMVG + OpenMVS and the MVE / SMVS based solution are already available in the OrtogOnBlender addon.

The proposal used the best of the UNIX world on the one hand, where each program does its job well, generating a good result in the end, and on the other, the convenience of commercial applications, where in a few clicks the user can perform complex tasks in a minimum of time.

The next step is to continue the studies and more than that, the use in real cases of surgical planning to adapt the technological possibilities to the needs of the professionals related to orthognathic surgery and rhinoplasty.

Acknowledgments

To Dr. Rodrigo Dornelles for the constant search for new solutions, always with the humility and patience to learn something new and generate fantastic results. To Dr. Pablo Maricevich for the vanguard in the use of open tools in rhinoplasty planning. Dr. Rodrigo Salazar for using photogrammetry in the field of oncology, restoring the brightness of the eyes and the dignity of patients who had lost them. To Dr. Graziane Olimpo for believing in the use of open tools in the field of orthodontics. To the participants of the experiment who kindly yielded their time and image to this study. To our students and the developers of all the applications used here, without you and without the help of the whole community this study would not be possible.