OHIF + VTK.js

OHIF + VTK.js

Acknowledgments

• Presented By:
• James A. Petts (OHIF)�
• Contributors:
• Forrest Li (Kitware)
• Trinity Urban (MGH and OHIF)
• Steve Pieper (Isomics)
• Erik Ziegler (OHIF)
• Gordon J. Harris (MGH and OHIF)
• James A. Petts (OHIF)
• Danny Brown (OHIF)
• Stephen Aylward (Kitware)

Funding Sources

• VTK.js / Kitware
• NIH NINDS R44NS081792
• NIH NINDS R42NS086295
• NIH NIBIB and NIGMS R01EB021396
• NIH NIBIB R01EB014955
• Isomics
• NIH P41 EB015902
• Contract number 19X037Q from Leidos Biomedical Research under Task Order HHSN26100071 from NCI (NCI Imaging Data Commons)
• OHIF
• NIH U24 CA199460

Course Outline

• Introduction to VTK.js and OHIF
• Starting a VTK.js project
• VTK.js: Architecture, itk.js, widgets, and more
• OHIF + VTK.js integration
• Advanced VTK.js: WebGL and GLSL

What is OHIF?

• OHIF is the Open Health Imaging Foundation. An open source community building a suite of tools and components for web-based medical imaging workflows.
• The OHIF Viewer:
• Is a web based imaging platform which can be used to view DICOM and other images.
• Can retrieve and load images from most data sources and display these in 2D and 3D.
• Allows for the manipulation, annotation, and serialization of observations.
• Is an extensible platform for building efficient tooling for specialised clinical and research workflows.
• Supports internationalization, OpenID Connect, offline use, hotkeys, and many more features.
• Ongoing active development from a growing developer community.

What is OHIF? Live demo!

​

Where is OHIF being used?

OHIF project has collaborated with ITCR-funded projects, such as C-BIBOP, QIICR, TCIA, and Radiomics, as well as QIN tools such as ePAD. In addition to third party implementations, including:

• Institute for Systems Biology's Cancer Genomics Cloud
• NCI Imaging Data Commons
• Crowds Cure Cancer
• XNAT
• Precision Imaging Metrics
• Google Cloud Healthcare API

​

To name a few!

​

​

Extensibility

• The OHIF Viewer is designed to be flexible and extensible enough to support any workflow.
• The following core extensions are available out of the box and maintained by the OHIF community:
• Cornerstone
• Dicom-html
• Dicom-pdf
• Dicom-microscopy
• VTK

Core extensions: Cornerstone

• View 2D images and annotate them with a large array of annotation and segmentation tools.

Core extensions: dicom-microscopy

• View multi-resolution DICOM microscopy using OpenLayers technology.

Core extensions: dicom-html and dicom-pdf

• View DICOM structured reports as human readable html reports.
• View DICOM encapsulated PDFs within the viewer.

Core extensions: VTK

• View 3D images using VTK.js. We’ll go into this one in more detail.

VTK Extension Overview

Extensions are configurable through an extension object.��We’ll use the VTK extension for these examples, although the knowledge gained here is applicable to other extensions as well.

If you’d like a broader look at OHIF extensions,they are documented here.

VTK Extension Overview

getViewportModule returns the viewport component associated with the extension.

VTK Extension Overview

getToolbarModule should return an object which contains UI buttons to be put on the main OHIF toolbar.

​

VTK Extension Overview

getCommandsModule provides a list of commands which may either be bound to toolbar buttons in the toolbarModule or to hotkeys.

VTK Extension Overview

The VTK.js extension does not currently make use of it, but it is good to note that extensions can also add custom UI components to either of OHIF side panels through the panelModule. component.

VTK toolbarModule

Here we show the VTK.js toolbarModule. It is a "plain old JavaScript Object" with some key words OHIF recognises and consumes.

It constructs the UI that allows execution of commands from the commandModule.

VTK toolbarModule

The definitions object defines buttons to be placed in the toolbar. This includes the icon, label and command to be executed, as well as options to pass that command.

Two basic button types are available for extensions.

VTK toolbarModule

setToolActive - Defines the button as a toggleable tool, which lets OHIF know to untoggle other buttons of the same type to signify a change in tool. These may be interaction styles or widgets in the context of the VTK.js viewport.

VTK toolbarModule

Command - Executes the given command from the command module.

VTK toolbarModule

Custom UI components can be injected into the toolbar to change its look. Here we inject the SlabThicknessToolbarComponent to MIP controller seen above to the toolbar.

VTK toolbarModule

The toolbarModule consists of the definitions and the defaultContext. This is the context in which the defined buttons will appear in the toolbar. Here this is when the context is ACTIVE_VIEWPORT:VTK, which means these buttons will only be displayed when the active viewport is a VTK.js viewport.

VTK commandsModule

Here we show the VTK.js commandsModule. It comprises of actions and definitions and the defaultContext.

VTK commandsModule

The definitions object defines commands which can be used by the toolbarModule or bound to hotkeys.

storeContexts can be passed to commands, such that the commands may have access to data from within the redux store.

VTK commandsModule

The actions object defines the actions associated with each definition. These are functions that may change the active tool, perform an operation, or change the state in some way.

VTK commandsModule

Again, this module exposes definitions and the defaultContext to which the definitions apply.

VTK viewportModule: OHIFVTKViewport

• The OHIFVTKViewport is a light wrapper around the react-vtkjs-viewport (github.com/OHIF/react-vtkjs-viewport).
• react-vtkjs-viewport is an open source component library containing React components for easily displaying DICOM data with VTK.js.
• The OHIFVTKViewport parent component exists to hydrate the viewport from the OHIF state.

OHIFVTKViewport

• In the OHIF Viewer, volumes are represented by displaySets, which are collections of 2D frames fetched and decoded from a PACS or other backend source by cornerstoneWADOImageLoader. These frames are decompressed in parallel using web workers.
• When a displaySet is opened in a VTK viewport these 2D images are progressively loaded into a vtkDataArray, cached, and displayed in the viewport.
• During loading appropriate scaling functions are applied as provided by the DICOM headers (e.g. Hounsfield Units for CT or Standard Uptake Value for PT).

In Progress Reconstruction

react-vtkjs-viewport

The react-vtkjs-viewport contains two major elements:

• View2D and View3D React components.
• A set of maintained interactor styles and widgets.

The interactor styles and widgets are not deeply integrated with the viewport components, such that consumers of the library can choose which tools to attach, and to allow extensibility in the future.

​

​

OHIFVTKViewport

Apart from state hydration, the OHIFVTKViewport just exposes an API for interacting with the react-vtkjs-viewport state via OHIF commands.

As much functionality is pushed down to the react-vtkjs-viewport library as possible, so that it is reusable in applications other than the OHIF Viewer.

Completed Reconstruction

react-vtkjs-viewport

View2D:

• With this component we achieve both orthogonal and oblique MPR by putting the volume in the shader on the GPU and moving the camera through the volume using a small clipping range.
• We can achieve MIP by increasing the slab thickness being sampled and rendering each pixel with the maximum value in the line of sight.

View3D:

• With this component we use volume rendering to produce 3D visualizations of the volume.

Interactor Styles and Widgets

In react-vtkjs-viewport the following are exposed:

• An SVG crosshair widget.
• A wwwc interaction style for window/level.
• A rotate interaction style.
• An MPR slice interaction style.

We use these in OHIF to implement:

• 2D MPR
• Slab projection:
• Maximum Intensity Projection (MIP)
• Minimum IP
• Average IP

Developing for VTK.js in OHIF

The VTK extension is the primary injection point for extending VTK.js functionality in OHIF.

After reviewing the extension structure we can now see that a new interactor style or widget can be injected by:

• Defining a toolbarModule definition.
• Defining a command in the commandModule which changes the active interactorStyle or widget.

Check the react-vtkjs-viewport interactor styles and widgets for example implementations.

Developing for VTK.js in OHIF

Using commands is also a great place to start injecting more complex VTK functionality into OHIF.

For example you may want to have a button which opens a side panel component which provides custom UI that allows the user to define the parameters for an auto segmentation routine, or apply some filtering process to the dataset and display this in the VTK.js viewport.

Developing for VTK.js in OHIF

If you wish to make changes to how the viewport itself operates (e.g. projection method, improvements to how sample distance is calculated, etc), this is best done within react-vtkjs-viewport library, or perhaps even VTK.js itself.

Potential Future Functionality

The react-vtkjs-viewport and its integration into OHIF has opened up lots of interesting possibilities. On the list to be implemented:

• Multi volume. e.g. PET/CT fusion (Proof of Concept)
• Volume rendering within OHIF.
• Surface rendering, e.g. OBJ in DICOM
• A SVG tools framework for tools like:
• Draw a freehand line region and get the SUV.
• Interactive reference lines (Example in Vue.js)
• Segmentation display (Proof of Concept)
• Live synchronisation of segmentation data between cornerstone and VTK viewports. (Proof of Concept)
• Parametric maps.
• RTSTRUCT support.

​

​

Potential Future Functionality

At the time of reading perhaps some new features have already made it live. viewer.ohif.org for a demo of the currently available features!

​

​

https://viewer.ohif.org/