1 of 27

Video for Autonomous Defense Systems

Yossi Cohen

DSP-IP

2 of 27

Agenda*

  • Autonomous Defense systems
  • Video usage in Autonomous Systems
    • Live Video & WebRTC
    • Storage & Distribution Systems
    • ATR Integration
  • Current Drone systems
    • NVIDIA / QCOM / DJI
  • 2022 Systems

*for some system we will discuss civilian systems. � Challenges are mostly the same

​

​

3 of 27

Autonomous Systems

  • UAV & Drones
  • Autonomous vehicles (combat / logistics)
  • Autonomous Ships & Submarines

​

Anything else??

​

​

​

4 of 27

Teleoperation and Monitoring

  • Lowest possible latency
  • Highest video quality
    • Detect small objects Cars / Tanks / Runway etc
  • Constraint bitrate (Usually CBR)
    • Compression without using I frame (Intra-Refresh)
    • Use CBR settings variable video quality
  • Overcome communication problem / outage
    • Robustness to errors
    • Use FEC

​

​

5 of 27

Record Distribute Interrogate

  • Record
    • Video / Audio
    • Metadata
    • Synchronized
  • Distribute
    • RTP/RTSP players
    • WebRTC endpoints
    • HLS / DASH Players
  • Interrogate
    • View specific streams

​

6 of 27

ATR - Detection integration

  • Detectors for defense are built the same as in
    • Agritec - Trees / fruits
    • Logistics – Ships / trains / Containers / Barcodes
    • Inspection - Electrical insulators / Railroads
  • Defense Detectors
    • Tracks / Tanks / Ships
  • Detect in Real-Time
  • Detections on recorded stream

​

​

​

​

7 of 27

Detection

  • Real-Time detection
    • Time constraint
    • Another block in the pipe - adds to latency
  • Offline detection (on recorded stream)
    • Record detection as synchronized metadata
    • Not time constraint – could use heavy networks

​

​

8 of 27

UAV and Drones

​

UAV

Drone

Cost

X10-500

​

Optics

Liquid cooled cameras, FLIR, Expensive optics

Low-Cost optics

Communication

Satellite, BLVOS, RF

Cellular 4-5G, Wi-Fi/6

  • Require Teleoperation (mostly)
  • Mostly implemented with WebRTC

​

​

9 of 27

Drones SDK – QUALCOMM RB5 5G

  • Last month RB5 SDK by QUALCOMM
    • Includes many detectors and computer vision modules
    • +Great communication
    • -High cost

​

​

10 of 27

Drone SDKs - NVIDIA

  • No official hardware
  • NVIDIA Remote control
  • Recommendation for SDK building
    • Usually using Jetson NX and AGX for AI / Video Compression
  • + Cost, SDKs
  • without wireless communication(Wifi/6, 5G, BT)

​

​

​

11 of 27

Autonomous ships

  • Harsh optics requirements
    • Video Stabilization (With/out IMU)
    • FOG / Water drops
  • Sensor Fusion to non optics sensors
  • Fading targets by horizon
    • Requires good tracker

​

​

​

​

12 of 27

Autonomous / remote-control Vehicles

​

​

  • Teleoperation
  • Recording video/ LIDAR /other sensors
  • Detection: Person / Car / Road….
  • Weapons operation….

​

​

​

​

13 of 27

video Codecs Roadmap

14 of 27

Hardware today

  • NVIDIA & QCOM provides processing and encoding solutions for autonomous systems
  • Current SoC solutions supports:
    • Control – CPU GPIOs
    • Inference - GPU and small TPUs
    • Encoding – H264/H265 hardware encoders (up to 4K)

​

​

​

​

​

​

15 of 27

New video Codecs

  • Legacy Autonomous Systems mostly use H264
  • Upgrading to HEVC/ VVC/AV1
    • Improve video quality ~x1.3-x2 – better details
    • Enable better resolution
  • AV1 and VVC are already standardized but not available as hardware encoders

​

​

​

​

​

16 of 27

Feature Comparison H264 and HEVC/H.265

​

​

​

​

​

17 of 27

R-D Comparison H264 and HEVC/H.265

​

​

​

​

​

Here HEVC encoder is HM10,

the results taken from the paper:

“Performance Comparison of H.265/MPEG-HEVC, VP9, and H.264/MPEG-AVC Encoders” by Dan Grois et al.

18 of 27

AV1

  • AV1 is already supported by WebRTC for live video
  • Comparison

Crowd run sequence

​

​

​

​

  • Selected new features:
  • De-blur in-loop filter - non-causal Wiener filtering applied to all pixels in each 4x4 block in order to attenuate blur artifacts caused by block-based processing
  • Global and locally warped motion compensation - beneficial for panning video, rotating and fast zooming
  • Palette Mode - tailored for screen content coding

​

19 of 27

AV1 (cont.)

  • Intra Block Copy – tailored for screen content coding

​

  • InterIntra Prediction

​

​

​

​

​

​

  • Wedge Prediction - often a block is separated into two differently predicted regions not necessarily rectangular

​

​

​

​

​

20 of 27

VVC

Selected new features

Multi-type partition tree - binary and ternary split

Decoder-Side Motion Vector Refinement

Two-parameters probability estimator model in CABAC

Bi-directional optical flow

Affine Motion Model

History-based motion vector prediction

Trellis-Coded Quantization (Dependent Quantization)

Reference Picture Resampling

Intra Block Copy

​

​

​

​

​

21 of 27

VVC (cont.)

22 of 27

Comparison VVC vs. HEVC/H.265

23 of 27

2022

  • NVIDIA Release ORIN replacement to Jetson
    • AV1 encoding capabilities
    • Jetson are already part of many drones
    • Smooth upgrade of existing drone systems
    • Might replace proprietary HW codecs in UAVs

​

​

​

​

​

​

24 of 27

Small vehicles & Drones

  • Usually using NVIDIA Jetson NX / AGX (commercial use)
  • NVIDIA is launching Orin in H1 2022 upgrading jetson
  • AV1 encoding capabilities
  • Jetson are already part of many drones
  • Smooth upgrade

​

​

​

​

​

25 of 27

Autonomous vehicles

  • Platforms and SDKs
    • NVIDIA Drive
    • QCOM Ride

​

​

​

​

26 of 27

Summary

  • New 2022 platforms will provide significant boost in
    • On system processing power
    • New standards encoders – AV1
    • Easy upgrade of existing platforms
  • Suggest to test and benchmark
    • New heavy detectors with better performance
    • New Av1 codecs on existing teleoperation platforms

​

​

​

​

​

27 of 27

Yossi Cohen +972-54-5313092

yossi@dsp-ip.com

​

Thank you