1 of 18

Neha Sharma

Doctoral Symposium

Supervisor - Mohamed Hefeeda

Enabling Wide Adoption of Hyperspectral Imaging

2 of 18

Modern Smartphone Cameras

“Photochrome” infrared LiDAR sensor RYYB sensor, Fast AI

OnePlus 8 Pro Apple iPhone 12 Pro Huawei P40 Pro

3 of 18

RGB Imaging derived from Human Vision

400nm

Visible Range

700nm

Ultraviolet

UV

1000nm

Infrared

RGB Bands

S

M

L

Sensitivity

 

4 of 18

RGB vs Hyperspectral Imaging

400nm

Visible Range

700nm

Ultraviolet

UV

1000nm

Infrared

RGB Bands

 

Reflectance

Reflectance

 

Spectral Signature

S

M

L

Sensitivity

 

5 of 18

Novel Applications on Smartphones

Avocado Ripeness/ defects

Skin cancer diagnosis

6 of 18

RGB Cameras

7 of 18

Hyperspectral Cameras

8 of 18

Challenges

    • Expensive (>30k USD)

Cost

    • Complex
    • Slow process

Hardware

    • Most bands are redundant
    • Storage
    • Transmission

Big Data

    • No video acquisition possible

Scanning

9 of 18

MobiSpectral : Turning Mobile Devices into Hyperspectral Camera

Reconstructing Spectral Bands

Modifying Mobile Cameras

Cloud Service and Mobile APIs

Supporting Diverse Illumination

10 of 18

MobiSpectral : Turning Mobile Devices into Hyperspectral Camera

Reconstructing Spectral Bands

Modifying Mobile Cameras

Cloud Service and Mobile APIs

Supporting Diverse Illumination

11 of 18

Reconstructing Spectral Bands

Image to image mapping between RGB and target bands using deep residual network (f)

Sharma, N., and Hefeeda, M. Hyperspectral Reconstruction from RGB Images for Vein Visualization. In MMSys’20

12 of 18

MobiSpectral : Turning Mobile Devices into Hyperspectral Camera

Reconstructing Spectral Bands

Modifying Mobile Cameras

Cloud Service and Mobile APIs

Supporting Diverse Illumination

13 of 18

Modifying Mobile Cameras

400 700 1000

Wavelength(nm)

​

Spectral Sensitivity

IR cut-off filter

400 700 1000

Wavelength(nm)

​

Spectral Sensitivity

Human Vision

Hyperspectral Reconstruction

Carefully selected (4-5) multispectral bands

RGB bands

Bayer Pattern

Multispectral Filter Array

14 of 18

MobiSpectral : Turning Mobile Devices into Hyperspectral Camera

Reconstructing Spectral Bands

Modifying Mobile Cameras

Cloud Service and Mobile APIs

Supporting Diverse Illumination

15 of 18

Cloud Service and Mobile APIs

Low latency

Maintain user privacy

Cost-effective

16 of 18

MobiSpectral : Turning Mobile Devices into Hyperspectral Camera

Reconstructing Spectral Bands

Modifying Mobile Cameras

Cloud Service and Mobile APIs

Supporting Diverse Illumination

17 of 18

Supporting Diverse Illumination

RGB

800nm

900nm

1000nm

Halogen

LED

CFL

Azadi, P., Sharma, N., and Hefeeda, M. Enabling Hyperspectral Imaging in Diverse Illumination Conditions for Indoor Applications. In MMSys'21

18 of 18

Conclusion

Cost-effective next-generation Hyperspectral Imaging System

Cloud Service and Mobile APIs for deep learning reconstruction model

Enabling Hyperspectral in Diverse Illumination