HorusEye

Language as Dynamic Attention for Emergency Visual Analysis

Armel Yara

Vision-Language Research | Master's Project | 2026

The Problem: Vision Fails When It Matters Most

Emergency Scenarios

🔥 Building fires with smoke

🌫️ Search & rescue in fog

🌡️ Thermal imaging at night

Drone-based victim detection

The Challenge

Current VLMs are trained on clean images

​

But emergencies have:

• Obscured visibility

• Non-standard imagery (thermal)

• Life-critical decisions

Research Question

To what extent can natural language feedback serve as a dynamic attention mechanism to re-evaluate and refine visual analytical tasks in real-time under degraded emergency conditions?

​

HALLUCINATION RISK

Research Questions

RQ1

Degradation Impact

How does VLM grounding degrade under fog, smoke, thermal?

RQ2

Language Feedback

Can iterative language feedback correct grounding errors?

RQ3

Health Analysis

Does cropping improve downstream health VQA?

RQ4

Hallucination Risk

Do VLMs hallucinate more when vision is degraded?

Methodology: Experimental Design

1

Grounding Under Degradation

Benchmark visual grounding accuracy under smoke, fog, thermal, low-light conditions

2

Iterative Refinement

Test if dialogue feedback ("no, the other one") can correct errors in degraded imagery

3

Downstream Impact

Does better grounding improve health assessment quality?

​

4

Language Richness Trade-off

​

Compare minimal vs. rich queries: accuracy gain vs. hallucination risk

How we built RefCOCO-Degraded benchmark ?

RefCOCO-Degraded benchmark

RefCOCO

Fog+Smoke+Thermal

​

RefCOCO-Degraded Benchmark Pipeline

RefCOCO

3,811 images

→

Degradation Simulation Pipeline

Fog

Depth-aware haze

Smoke

Perlin noise blend

Thermal

False-color map

RefCOCO-Degraded Benchmark

clean/

Original RGB

fog_0.5/

Fog applied

smoke_0.5/

Smoke overlay

thermal_0.5/

Thermal sim

annotations/ (bounding boxes + referring expressions)

↓

↓

Total: 15,244 images (3,811 × 4 conditions)

RefCOCO-Degraded Benchmark

RQ1: How Does Visual Grounding Degrade?

🔻

37%

Thermal Drop

The "Thermal Cliff"

Clean

0.628

Fog

0.569

Smoke

0.560

Thermal

0.397

RQ2: Can Language Feedback Correct Errors?

⬆️

+9.6%

Thermal Recovery

RQ3: Does Cropping Improve Health Analysis?

The Thermal Paradox

Cropping DESTROYS thermal context

-26%

RQ4: Hallucination Risk in Degraded Vision

Metrics Used

• Fabrication: Objects invented

• Uncertainty: "might be", "could be"

• Color mentions: Colors in thermal

Finding 1: Models Become Cautious

In degraded conditions:

• Less fabrication (0.81 → 0.46)

• More uncertainty phrases

• VLMs "know they don't know"

Finding 2: Color Hallucination Persists

Thermal images have NO natural colors

Yet models mention 2.25 colors/response

"red shirt" in purple heat map = hallucination

Model Comparison

Average Hallucination Scores:

​

Gemini 2.0 Flash: 0.31

Qwen2-VL-2B: 0.06

​

​

Project Timeline: 4 Weeks

W1-2

Setup & Data

Environment, datasets, degradations

W3

Exp 1: ROI

Language-guided selection

W5

Exp 3-4

Degradation + Dialogue

W6

Exp 5

Hallucination analysis

W7

Analysis

Complete results

W8

Writing

Paper + Presentation

→ WE ARE HERE

Key Deliverables

• Experimental code (Python & Pytorch)

• Benchmark results

• Research paper

• This presentation

Research Contributions

1

First benchmark of VL grounding under emergency-relevant degradations

2

Analysis of iterative language feedback for error correction in degraded imagery

3

Quantification of the accuracy-hallucination trade-off with language richness

4

Safety analysis for deploying VL systems in emergency response

5

Design guidelines: when to use rich vs. minimal queries

Conclusion

Degradation breaks spatial grounding (especially thermal)

​

​

Language feedback can partially recover lost performance

​

​

Cropping strategies must be condition-aware

​

​

Hallucination metrics must account for image modality

​

​

​

​

​

Thank You for your attention — Questions?

​