HORUS EYE

Language as Dynamic Attention for Emergency Visual Analysis

Armel Yara

Vision-Language Research | Master's Project | 2026

The Problem: Information Gap in Emergency Response

Current Situation

• How many victims?

​

• Where are they located?

​

• What is their health status?

​

• What are the hazards?

The Horus Vision

• Scene understanding & detection

​

• Interactive health assessment

​

• Interactive dialogue with rescuers

​

• Extraction planning support

�

​

A Sample Multi-Task Dialogue

A Sample Multi-Task Dialogue

Scene understanding + Person detection

Health Query

Health VQA

Planning Query + Constraint input

Planning task

Planning output + Spatial grounding

Temporal VQA Query

Environmental VQA + Temporal prediction

Monitoring request + Communication relay

Continuous Monitoring Mode

A Sample Multi-Task Dialogue

What Makes This Research Novel?

What's NOT Novel

"Can language find an object in an image?"

​

• This is Visual Grounding

​

• Solved by RefCOCO benchmarks

​

• SOTA models achieve 90%+

​

​

What IS Novel

"Language as DYNAMIC attention mechanism"

​

• Iterative refinement through dialogue
• Under DEGRADED conditions (smoke, thermal)
• Real-time feedback loop
• Hallucination risks when vision fails

The Key Distinction

​

�

Standard Grounding: "Find X" → Box around X → Done

​

Our Research: "Find X" → Box → "Wrong one" → Re-adjust → "Check her breathing" → Focused analysis → Iterate...

�

​

Research Question

To what extent can natural language feedback serve as a dynamic attention mechanism to re-evaluate and refine visual analytical tasks in real-time under degraded emergency conditions?

​

HALLUCINATION RISK

Research Questions Decomposition

RQ1

How does visual grounding degrade under smoke, fog, and thermal conditions?

First benchmark

RQ2

Can iterative language feedback correct grounding errors in degraded imagery?

Dynamic refinement

RQ3

Does language-guided attention improve downstream health analysis?

End-to-end impact

RQ4

What is the hallucination risk when rich language meets degraded vision?

KEY INSIGHT

Critical Insight: The Hallucination Risk

When vision degrades, rich language can cause AI to "imagine" objects that don't exist

Minimal Query

I�Image: Heavy smoke, NO person present

​

Query: "the person"

�

Model: "No detection"

�

​

Rich Query

I

Image: Heavy smoke, NO person present

​

Query: "the injured woman waving at the west window with burns on her arms"

�

Model: "Found at [x,y]"

�

​

Why This Matters in Emergency Response

Hallucinated victim → Rescue team wastes time on empty location

Missed real victim → Someone may die

Methodology: Experimental Design

1

Grounding Under Degradation

Benchmark visual grounding accuracy under smoke, fog, thermal, low-light conditions

2

Iterative Refinement

Test if dialogue feedback ("no, the other one") can correct errors in degraded imagery

3

Language Richness Trade-off

Compare minimal vs. rich queries: accuracy gain vs. hallucination risk

4

Downstream Impact

Does better grounding improve health assessment quality?

Visual Degradation Types

Smoke

Fog + gray overlay + contrast reduction + noise

Active fire

Fog

White overlay + Gaussian blur

Low visibility

Low-light

Brightness reduction + sensor noise

Night ops

Thermal

Grayscale → infrared colormap

Thermal camera

Combined

Smoke + Low-light simultaneously

Fire at night

Datasets

Project Timeline: 8 Weeks

W1-2

Setup & Data

Environment, datasets, degradations

W3

Exp 1: ROI

Language-guided selection

W4

Exp 2: rPPG

Signal quality comparison

W5

Exp 3-4

Degradation + Dialogue

W6

Exp 5

Hallucination analysis

W7

Analysis

Complete results

W8

Writing

Paper + Presentation

→ WE ARE HERE

Key Deliverables

• Experimental code (Python & Pytorch/TensorFlow)

• Benchmark results

• Research paper

• This presentation

Research Contributions

1

First benchmark of VL grounding under emergency-relevant degradations

2

Analysis of iterative language feedback for error correction in degraded imagery

3

Quantification of the accuracy-hallucination trade-off with language richness

4

Safety analysis for deploying VL systems in emergency response

5

Design guidelines: when to use rich vs. minimal queries

Limitations & Long shots

Current Limitations

• Synthetic degradation
• RefCOCO domain ≠ emergency domain
• No real firefighter evaluation
• Single-frame (no video tracking)
• rPPG may fail at drone distances

Long shots

• Real emergency dataset collection
• Domain adaptation for thermal
• End-to-end trained system
• User study with firefighters
• Full Horus system integration

The Bigger Picture: This Is One Component of Horus

Q1: Scene Understanding | Q2: Language-Guided Analysis (THIS WORK) | Q3: Planning | Q4: Temporal | Q5: Multi-party

Summary

1

Language can serve as a dynamic attention mechanism for visual analysis

2

VL grounding degrades significantly under emergency conditions

3

Iterative dialogue can refine attention when initial grounding fails

4

Rich language improves accuracy BUT increases hallucination risk

5

This trade-off is critical for safety — our research want to quantifies it

"In emergency response, the cost of hallucination is measured in lives.

Our research helps understand when to trust AI — and when not to."

​

​

​

​

​

Thank You — Questions?

​