1 of 91

OHIF V3 Architecture

and Cornerstone 3D

​

Virtual NA-MIC Project Week 36th, Jan 20th 2022

2 of 91

Outline

  • Demo
  • Motivation for Modes
  • OHIF v3 Architecture
    • OHIF Core
    • Extensions
    • Mode Architecture
    • Mode Life Cycle
  • UI Component Library
  • Cornerstone-3D Details
  • Q&A�

​

2

3 of 91

Let’s Start with a Demo!

3

UI and design not final

4 of 91

Project Week 35th Presentation

​

  • Google Slides
    • OHIF Adoption Statistics
    • OHIF Evolution from v1 to v3
    • OHIF-v3 introduction
      • New UI
      • Mode architecture
      • Extension architecture
      • New Services

​

​

�

4

5 of 91

Why modes? One application with many workflows

Often core functionalities are being used for a specific purpose (use case)

  • Tracking has measurements
  • AI use case needs measurements
  • Other use cases might need measurements

Previously, to achieve this you had to create an extension to manage these features.

​

​

​

​

​

​

5

OHIF-v3 introduces Modes to enable building such workflows by re-using the core functionalities from the extensions.

6 of 91

New Architecture

  • Modes:
    • Configuration objects
    • Tell @ohif/viewer how to compose extensions
    • Build applications on different routes of the platform.�
  • One step closer to a “registry”
    • Shareable workflows

6

7 of 91

OHIF v3 Architecture

​

7

8 of 91

OHIF Architecture

  • How

8

9 of 91

Core Components

9

10 of 91

Core Managers

10

11 of 91

Core Services

11

12 of 91

Providers

12

13 of 91

Extensions

  • Extensions provide functionality
  • Where are the functionalities located?

13

14 of 91

Extensions

  • Extensions provide functionality
  • Where are the functionalities located?

14

Functionalities

15 of 91

Panel Module - Introduction

  • Panels (default: left and right)�
    • Stateful and reactive components�
    • Have access to managers and services�
    • Usually subscribe to events and update the state as necessary

15

16 of 91

Panel Module - Requirements

16

17 of 91

Panel Module - Requirements

​

17

18 of 91

Panel Module

18

19 of 91

Viewport Module - Introduction

  • Viewports�
    • Contains the logic for rendering DisplaySet(s)�
    • There can be more than one viewport at the same time in the Viewport Grid
      • Cornerstone Viewport to render image
      • DICOM PDF Viewport to render DICOM PDF

19

20 of 91

Viewport Module - Requirements

20

21 of 91

Viewport Module - Requirements

21

22 of 91

Viewport Module - Requirements

22

23 of 91

Toolbar Module - Introduction

  • Provides Toolbar button logic and UI
    • Tooltip
    • Split buttons
    • Dropdown tools

23

24 of 91

Toolbar Module - Requirements

​

24

25 of 91

Toolbar Module - Requirements

​

25

26 of 91

LayoutTemplate Module - Introduction

  • Defines the layout of the viewer�
    • Composes UI element
    • Add header, whiteLabelling

26

27 of 91

LayoutTemplate Module - Requirements

27

28 of 91

LayoutTemplate Module - Requirements

28

29 of 91

SOP Class Handler Module - Introduction

  • What is a DisplaySet?
    • A collection of objects to display in a viewport
    • GENERALLY, an ordered set of SOP Instance/Frame references
    • DisplaySet is the object that Hanging Protocol lays out for display�
    • For images, they are commonly split into stacks
      • Image references can be frames
      • Image references can be a transform, eg MPR
      • Examples:
        • Axial CT Series
        • Sagittal Prostate MRI Series
        • DICOM PDF report

29

30 of 91

SOP Class Handler Module - Introduction

  • Converts Series to DisplaySet(s)�
    • Example 1) A Computer Tomography Series might contain initial low-resolution localizer
      • This inconsistency is picked up and two separated DisplaySets are generated�
    • Example 2) When Ultrasound Series contain images from various orientations
      • They will be broken down into separate consistent DisplaySets

30

31 of 91

SOP Class Handler Module

31

32 of 91

SOP Class Handler Module

32

33 of 91

Hanging Protocol Module - Introduction

  • Define what layout of the viewport should the viewer starts with (2x2 layout)�
  • Define which displaySet(s) gets displayed in which position of the layout�
  • Apply certain initial viewport settings; e.g., inverting the contrast�
  • Enable certain tools based on what series are displayed: link prostate T2 and ADC MRI.

​

33

34 of 91

Hanging Protocol Module - Requirement

34

35 of 91

Hanging Protocol Module - Requirement

35

36 of 91

Hanging Protocol Module - Sample

36

37 of 91

Hanging Protocol Module - Sample

ProtocolMatchingRules

  • During matching, ALL protocols are scored based on their defined set of rules and the best one is used �
  • Can be weighted rules

37

38 of 91

Hanging Protocol Module - Sample

Stages

  • Includes one or more set of configurations for viewport grid�
  • Example:
    • Stage1: CT Axial side by side of PT Axial
    • Stage2: CT Sagittal side by side of PT Sagittal
    • Stage3: 3x3 Grid of CT (Axial, Sagittal, Coronal) with PET and the fusion viewports

38

39 of 91

Hanging Protocol Module - Sample

ViewportStructure

  • Currently only grid Structure
  • In Cornerstone-3D release version of OHIF, it will accept % of screen: any layout you want

39

40 of 91

Hanging Protocol Module - Sample

Viewports

  • Each Viewport Configuration�
  • SeriesMatchingRules:
    • Which DisplaySet to render in the viewport�
  • ViewportSettings
    • Initial WindowLevel and WindowCenter
    • Synchronizers

40

2x2 = 4 Viewports

41 of 91

Hanging Protocol Module - Sample

Viewports

  • Each Viewport Configuration�
  • SeriesMatchingRules:
    • Which DisplaySet to render in the viewport�
  • ViewportSettings
    • Initial WindowLevel and WindowCenter
    • Synchronizers

41

42 of 91

Commands Module - Introduction

  • Defines arbitrary commands (javascript functions) to a unique command name.
  • These functions are provided with all registered services, to manipulate the state of the application, as well as execute their own arbitrary code.
  • These commands can then be bound to toolbar buttons, hotkeys, and arbitrary UI defined by extensions.�
  • Examples would be:
    • Performs a serialisation of annotations and posts data somewhere.
    • Open a specific modal/dialog.
    • Set a tool active.�

​

​

42

43 of 91

Commands Module - Requirement

43

44 of 91

Commands Module - Requirement

44

45 of 91

DataSource Module - Introduction

  • Converts the study/series/instance metadata to the OHIF’s internal metadata format�
  • You can make another datasource implementation which communicates to your backend and maps to OHIF’s native format�

​

​

45

46 of 91

DataSource Module - Requirement

46

47 of 91

DataSource Module - Requirement

47

48 of 91

Custom DataSource

Any Custom DataSource should define the following API:

48

49 of 91

Mode

How does a mode use functionality from Extensions?

49

50 of 91

Mode

50

Extension ID

Module Type

Name

51 of 91

Mode

51

Extension ID

Module Type

Name

52 of 91

Mode

52

Extension ID

Module Type

Name

53 of 91

Mode

53

54 of 91

Mode

54

Extension ID

Module Type

Name

55 of 91

Mode

55

Extension ID

Module Type

Name

56 of 91

DataSource Configs

DataSource information for modes are given in the configuration of the App. Default config is located at :

platform/viewer/public/config/default.js

​

Spoiler Alert:

You can run OHIF with different configurations! See here

56

57 of 91

DataSource Configs

DataSource information for modes are given in the configuration of the App. Default config is located at :

platform/viewer/public/config/default.js

​

Spoiler Alert:

You can run OHIF with different configurations! See here

57

58 of 91

DataSource Configs

DataSource information for modes are given in the configuration of the App. Default config is located at :

platform/viewer/public/config/default.js

​

Spoiler Alert:

You can run OHIF with different configurations! See here

58

59 of 91

DataSource Configs

DataSource information for modes are given in the configuration of the App. Default config is located at :

platform/viewer/public/config/default.js

​

Spoiler Alert:

You can run OHIF with different configurations! See here

59

60 of 91

DataSource Configs

DataSource information for modes are given in the configuration of the App. Default config is located at :

platform/viewer/public/config/default.js

​

Spoiler Alert:

You can run OHIF with different configurations! See here

60

61 of 91

Mode Routes

Two routes will get registered for each mode:

  • /${mode.id}/ renders layout and uses default data source
  • /${mode.id}/${dataSourceName}? Renders the layout and used the specified data source

​

​

Example:

61

62 of 91

Looking at the Mode Architecture Again

62

63 of 91

Modes �Life Cycle

  • How

63

64 of 91

Mode Life Cycle

  • What happens from start to finish for a Mode to appear on the screen?
  • What component triggers this process?
  • What events are fired?
  • ……

64

65 of 91

Mode Life Cycle (Summary)

  • Step 0) OHIF registers all available Modes internally�
  • Step 1) Upon visit of a URL, it is broken down to find Mode and DataSource
    • https://v3-demo.ohif.org/tmtv?StudyInstanceUIDs=1.3.6.1.4.1.25403�
  • Step 2) Mode is mounted
    • DataSource is initialized
    • LayoutTemplate is fetched from the Mode config
    • DisplaySetService is initialized with fetched SOPClassHandlers
    • Registered Extensions (that are in use for the mounted mode) run their onModeEnter
    • Mounted Mode runs its onModeEnter�
  • Step 3) Route is initialized with provided StudyInstanceUIDs
    • DisplaySetService is set to create DisplaySets when Metadata is available
    • HangingProtocol is set to run after studyMetadata is available
    • Study Metadata BEGINS to fetch

​

65

66 of 91

Services

  • How

66

67 of 91

Services Subscriptions

  • Introduced Pub/Sub pattern to reduce coupling between layers�
    • Services are publishers of events and hold the set of listeners
    • Event is only published by service itself (1:M) to listeners
    • You can publish events in the services methods�

​

67

68 of 91

Services Subscriptions

  • Services are made available via ServiceManager
  • You can grab the service, and subscribe to its internal events
  • Some events examples
    • Study fetching events
    • HangingProtocol events
    • DisplaySet events
    • Measurement events
    • ….

​

​

68

69 of 91

DICOMMetadataStore

  • Central location for the metadata of studies and they are located
  • Datasources (default or custom) should communicate with DicomMetadataStore to store, and fetch Study/Series/Instance metadata�
  • Provides API for
    • add/get Study metadata
    • add/get Series metadata
    • add/get Instance metadata
    • Get StudyInstanceUIDs in the store�
  • Events emitted:
    • STUDY_ADDED
    • SERIES_ADDED
    • INSTANCES_ADDED

69

70 of 91

Custom Services

You can create your own custom services and register them with ServicesManager in the extension preRegistration

​

​

70

71 of 91

Custom Services

You can create your own custom services and register them with ServicesManager in the extension preRegistration

​

​

71

72 of 91

FAQ

What do I need to know to develop an Extension/Mode ?

Requirements:

  • JavaScript (in near future: TypeScript) and React
  • If you are developing with Cornerstone ecosystem for rendering and tools
    • CornerstoneJS
    • CornerstoneTools

Our vision is:

  • Clinicians define their workflow
  • Developers build that workflow as a Mode based on current functionalities (extensions) and new functionalities (as a new extension)

​

72

73 of 91

FAQ

How do I write a new Extension/Mode?

  • Short term approach:
    • Follow the instructions for Extension and Mode
    • In summary, you need to copy paste an existing code and change source code in OHIF�
  • In-Progress - finalizing it right now in Project Week 36:
    • We will publish a new command line tool (CLI) to handle this
    • So you will run
      • ohif-cli create-mode
      • ohif-cli create-extension

73

74 of 91

FAQ

How do I write a new Extension/Mode?

  • In-Progress - finalizing it right now in Project Week 36:
    • We will publish a new command line tool (CLI) to handle this
    • So you will run
      • ohif-cli create-mode
      • ohif-cli create-extension

74

75 of 91

FAQ

How do I add/install a new Extension/Mode that I find in Github/npm?

  • As of this presentation: You need to copy paste like previous slide�
  • Very near future (when we wrap up our project week ohif-cli project)�
    • Extensions: ohif-cli add-extension otherExtension
    • Modes: ohif-cli add-mode otherMode
      • This will install the mode and the dependant extensions for you and add them to OHIF!

​

75

76 of 91

FAQ

Do I need to clone OHIF to test Extensions and Modes?

  • Short term:
    • Yes, you need to have OHIF cloned
    • You don’t need to Fork OHIF at all�
  • In Future:
    • ohif-cli will be able to run your local mode on specific build version of OHIF without the need for you to clone it

76

77 of 91

FAQ

I want to use external libraries for my Extension, is it possible?

  • Yes, you can create your extension with any dependencies you rely on
  • For instance, you can have your ai-extension use tensorflow.js and write mode(s) to use the extension

77

78 of 91

FAQ

Can I install a Mode on the fly?

  • No you can’t right now, you need to rebuild

78

79 of 91

FAQ

Can I use vue.js or Angular for the UI in OHIF?

  • No, OHIF is built on top of react js

79

80 of 91

OHIF-v3 UI Component Library

  • https://v3-ui.ohif.org/�

80

81 of 91

Cornerstone 3D

  • Cornerstone JS re-designed for advanced image display:
    • Offscreen rendering for portability and scalability.
    • Both 2D (stack) and 3D (volume) display driven by vtk.js.
      • Drives advanced functionality like MPR/MIP/Fusion.
    • Efficient sharing of memory between viewports “under the hood”.
    • Extensible tools library very similar to the original CornerstoneTools.
  • Currently closed source, will be released open-source in April 2022.

​

​

​

​

​

81

82 of 91

Cornerstone 3D - Offscreen Rendering

82

UI with JavaScript Canvas Elements

Single WebGL Canvas “off screen”

glViewport 1

glViewport 2

glViewport 3

glViewport 4

glViewport 1

glViewport 2

glViewport 3

glViewport 4

Draw WebGL rendered data to JavaScript Canvas Elements

83 of 91

Cornerstone 3D - Volume Viewports

  • Can define one or more volumes to be added to Volume Viewports.
  • Volume Viewports view a 3D scene containing volumes.
    • This enables a suite of vtk.js features like
      • (Oblique) MPR
      • MIP
      • Image Fusion (e.g. PET/CT)

​

​

​

​

​

83

84 of 91

Cornerstone 3D - Stack Viewports

  • In legacy Cornerstonejs, stacks of images can be added to a viewport and rendered in their native acquisition direction.
  • We wanted to keep this functionality as:
    • Sometimes you only need to preview a few images quickly/want to avoid the overhead of allocating volumes.
    • Not all data is reconstructable:
      • May have missing metadata.
      • May be a “non-3D-reconstructable” modality (e.g. CR/DX/US)

​

​

​

​

​

84

85 of 91

Cornerstone 3D - Stack Viewports

  • Stack Viewports in Cornerstone3D have been upgraded to use vtk.js under the hood, using the exact same rendering pipelines as Volume Viewports, creating actors dynamically.
  • This allows us to use all of the color transfer functions/segmentation filters from vtk.js.

​

​

​

​

​

85

86 of 91

Cornerstone 3D - Memory Optimizations

  • When displaying a volume in multiple viewports, we share the volume mapper across actors so that the raw data is only stored in vRAM once.
  • Viewports clean up their resources on destruction.
  • When loading:
    • images into a stack viewport, these are pulled from volumes before the web, if available.
    • Volumes, these are pulled from the image cache before the web, if available.

​

​

​

​

​

86

87 of 91

Cornerstone 3D Tools

  • A tools layer has been built on top of Cornerstone 3D to build:
    • Manipulation tools (e.g. pan/zoom)
    • Planar annotation tools (e.g. length, ellipse)
    • Segmentation Tools (e.g. scissors)
  • Manipulation tools use viewport API to manipulate the vtk.js camera.
  • Planar annotations are natively 3D, and are rendered as SVG on top of viewports.

​

​

​

​

​

87

88 of 91

Cornerstone 3D Update to Community

Targeted Release date: April 2022

Packages to be released:

Cornerstone-3D-core and Cornerstone-3D-tools

Cornerstone-Streaming-Image-Volume

Cornerstone-3D Segmentation rendering and editing Tools

Cornerstone-3D for OHIF-V3

A Total Metabolic Tumor Volume (TMTV) mode in OHIF-v3

​

As well as documentation and migration guides for Cornerstone users.

88

89 of 91

OHIF Future Directions

  • Cornerstone 3D release
  • CZI community initiatives
    • Modes gallery
    • EOSS Diversity & Inclusion program
    • Enhanced testing suite
  • Dedicated support resourcing
  • Shift towards TypeScript
  • Open QMS project
  • Roadmap
  • Industry partnerships

89

90 of 91

Team

90

Chris Hafey

Co-founder

​

​

Trinity Urban

Co-founder

Steve Pieper, Ph.D

Advisor

​

​

James Petts, Ph.D

Developer

Danny Brown

Developer

​

Gordon J. Harris, Ph.D

Co-founder

Dan Rukas

UX/UI Design

Erik Ziegler, Ph.D

Lead Architect

​

​

Hua Li, M.D.

Developer

Supriya Somarouthu, M.D.

Subject Matter Expert

​

​

James Hanks

Project Manager

​

​

Rob Lewis

Co-founder

​

​

Matt Leary

Product Manager

Alireza Sedghi, Ph.D

Developer

91 of 91

Thank you!

​

contact@ohif.org

​

OHIF-V3 source code

​