HONOURS PROJECT
Multi-Modal Behavioral
Indexing for Edge-Based
Threat Detection
Jayden-Lee Van Rooyen · 4069341 · University of the Western Cape
Edge Computing
Sensor Fusion
Privacy-by-Design
Behavioral Analytics
Time-of-Flight
THE PROBLEM
02
⏱
Reactive, Not Proactive
Traditional CCTV systems serve as forensic tools for post-event analysis. Manual monitoring cannot scale to modern video volumes.
☁
Cloud Dependency
Existing AI approaches stream raw video to centralised servers, introducing latency, bandwidth costs, and significant privacy risk.
📐
No Depth Perception
Vision-only systems infer proximity from bounding-box size which is an estimate that degrades under occlusion and perspective distortion.
This project addresses all three by fusing RGB vision + ToF depth on an autonomous edge node with no cloud required.
BACKGROUND & RELATED WORK
03
01
Visual Perception
Gap: 2D only (no metric depth)
Sapkota, Zheng, Wojke, Mandalapu
02
Sensor Fusion
Gap: Assumes server-class hardware
Gimpelj, Park, Romanenko, Li & Tang
03
Motion & Pose
Gap: No simultaneous depth pairing
Wahyono, Mai & Bui, Oh et al.
04
Privacy & Behavior
Gap: No unified numeric output
Song, Al-Madani, Qamar
05
Threat Indexing
Gap: Pixel-space proximity proxy
Clarke, Pouyan, Wu et al.
06
Edge Deployment
Gap: Single-stream only (no fusion)
Pereira, Reis, Ali, Chen
No existing system unifies all six pillars into a single privacy-preserving edge deployment, this is the research gap.
THE BEHAVIORAL THREAT INDEX
04
T = (w₁ · Temporal) + (w₂ · Spatial) + (w₃ · Behavioral)
⏱ w₁ - TEMPORAL
Dwell-Time Tracking
Wahyono et al. [18], Wojke et al. [19]
📡 w₂ - SPATIAL
Depth Zone Detection
Gimpelj & Munih [5], Romanenko [15]
🦾 w₃ - BEHAVIORAL
Anomaly Scoring
Mai & Bui [7], Oh et al. [9]
Weights determined via AHP-Entropy-CRITIC framework (Wu et al. [20])
SYSTEM HARDWARE
05
Raspberry Pi 5
8GB · 2.4GHz Quad-core A76
Role: Host Logic / T Calculation
Hailo-10H NPU
AI HAT+ · 40 TOPS · PCIe 3.0
Role: Inference for w₁, w₃
Camera Module 3
12MP / HDR / PDAF · CSI-2
Role: Visual Input (x₁, x₃)
VL53L5CX ToF
8×8 Multi-zone · I2C Bus
Role: Proximity Input (x₂)
27W USB-C PSU
Raspberry Pi Official
Role: System Stability
Active Cooler
Variable Speed Fan
Role: Prevent Thermal Throttling
All inference runs locally on-device. No raw video is transmitted or stored at any point.
THREAT MODEL & SECURITY SCOPE
06
👁
Physical Tampering
Intruder covers lens or ToF matrix to induce sensor blindness and prevent detection.
🎭
Adversarial Attacks
Physical adversarial patches suppress human-class detection within the YOLO pipeline (w₃).
📶
Data Integrity
I2C bus transmission exposes ToF readings to potential spoofing, causing false negatives in w₂.
🔓
Privacy Leakage
Unauthorized physical access could allow reconstruction of retained behavioral metadata.
Privacy-by-Design: Raw frames purged from RAM immediately after keypoint extraction. Only Tracking IDs, depth coords, and T transmitted.
SYSTEM REQUIREMENTS
07
FUNCTIONAL REQUIREMENTS
FR-1
Multi-Modal Data Ingestion
1080p RGB via CSI + 8×8 depth via I2C concurrently
FR-2
Edge-Based Pose Estimation
NPU accelerator extracts skeletal keypoints in real-time
FR-3
Late-Fusion Spatial Mapping
2D bounding boxes mapped to 3D ToF coordinates
FR-4
Threat Index Calculation
Composite T computed every 200ms via weighted sum
FR-5
Automated Alert Triggering
Local notification when T exceeds safety threshold
NON-FUNCTIONAL REQUIREMENTS
⚡
Performance
< 150ms latency · ≥ 20 FPS · < 75% CPU
🔒
Security & Privacy
No raw video stored or transmitted
📈
Scalability
Additional ToF sensors via I2C multiplexer
🖥
Usability
Admin adjusts wₙ weights without coding
🔧
Maintainability
Virtual env isolation + comprehensive logging
SYSTEM PIPELINE & IMPLEMENTATION CHALLENGES
08
1
CAPTURE
RGB 1080p + 8×8 ToF
2
INFER
Hailo-10H NPU
Pose Estimation
3
FUSE
Late Fusion
Spatial Mapping
4
SCORE
Compute T
every 200ms
5
ALERT
Local notification
if T > threshold
🔒 Raw frames purged from RAM immediately after Step 2 so that only anonymized metadata persists.
Calibration
Aligning the 63° FoV of the VL53L5CX with Camera Module 3 so depth data maps accurately to visual bounding boxes. The asynchronous data rates are the central engineering challenge.
Simulation
Using mSSA (multidimensional Singular Spectrum Analysis) to test trajectory outlier detection and index weight sensitivity in a controlled virtual environment before hardware deployment.
Integration
Implementing the Hailo-10H Python pipeline to evaluate real-time inference latency and thermal stability under sustained 24/7 load on the Raspberry Pi 5.
ROADMAP (TERM 2)
09
WHAT WE PROPOSE
PHASES
🔧
PHASE 1: Hardware assembly & sensor mounting
📐
PHASE 1: ToF-to-Camera FoV calibration
💻
PHASE 2: Hailo-10H NPU pipeline implementation
🧪
PHASE 2: mSSA simulation & index weight tuning
📊
PHASE 3: Real-time latency & thermal benchmarking
Goal: Prove that sophisticated behavioral analytics can run entirely at the edge while preserving privacy without sacrificing accuracy.