1 of 17

FLOWSEGMODEL: ADVANCING PERCEPTION IN AUTONOMOUS DRIVING THROUGH WEATHER-RESILIENT SEGMENTATION

AI

Dilshara Herath, Oshada Rathnayake, Thiwanka Alahakoon,

Sanjula Senadeera, Roshan Godaliyadda, and Parakrama Ekanayake

​

Multidisciplinary AI Research Centre,

University of Peradeniya, Sri Lanka

2 of 17

Motivation and Problem

    • Autonomous driving perception degrades in fog/rain/overcast due to contrast loss, occlusion, noise
    • Segmentation is safety-critical (road + vehicles)
    • Motion cues can help when appearance cues degrade

3 of 17

Optical Flow

4 of 17

Fig 1 - Architectural overview of the FlowSegModel, illustrating the integration of RGB (H × W × 3, 8-bit per channel) and optical flow (H × W × 2, 16-bit per channel) inputs into a 5-channel (H × W × 5) tensor, processed through an Encoder and Decoder architecture based on DeepLabV3+, leading to semantic segmentation output (H × W × 1) compared against RGB semantic segmentation ground truth (H × W × 1, 8-bit per channel) and label maps (H × W × 1).

​

System Architecture

    • Fuse RGB + optical flow in a modified DeepLabV3+ to improve robustness
    • Use SEA-RAFT as the flow estimator (selected via benchmarking)

​

5 of 17

Fig 2 - Encoder-Decoder architecture of DeepLabV3+, the semantic segmen tation framework used in our FlowSegModel [14]

Methodology – FlowSegModel Architecture​

​

6 of 17

Methodology – Dataset

    • A composite dataset centered on Virtual KITTI 2 (vKITTI 2) was used.
    • Dataset Features - vKITTI 2 is a synthetic benchmark that provides controllable variations in weather conditions (clone/clear weather, fog, overcast, rain, and sunset) and includes paired RGB images, optical flow ground truth, and class segmentation annotations.

7 of 17

Results – FlowSegModel Resilience

Fig 3 - Optical flow visualization for a single scene under varying weather conditions (Clone, Fog, Overcast, and Rain): (a) RGB images representing the original scene (b) Optical flow estimated using RAFT; (c) Optical flow estimated using SEA-RAFT, highlighting the differences in motion estimation across weather-induced variations.

​

​

8 of 17

TABLE 3 - SEGMENTATION PERFORMANCE METRICS FOR DIFFERENT CLASSES UNDER NORMAL AND ADVERSE WEATHER CONDITIONS, INCLUDING INTERSECTION OVER UNION(IOU), PRECISION,RECALL, AND F1-SCORE,EVALUATED FROM THE FLOW SEG MODEL

​

Results – FlowSegModel 

9 of 17

Ablation Study – Importance of RGB-FlowFusion

TABLE 5 - SEGMENTATION RESULTED ON OPTICAL-FLOW-ONLY FLOWSEGMODEL

​

​

10 of 17

Performance Metrics

IoU - Measures overlap between the predicted region and ground-truth region for class ccc, divided by their union (higher is better).

11 of 17

Fig 3 - SEGMENTATION RESULTS ACROSS VARYING WEATHER CONDITIONS (CLONE, FOG, OVERCAST, RAIN), COMPARING INPUT RGB IMAGES, GROUND TRUTH MASKS, AND OUR PREDICTED MASKS 

​

Results – Class-wise Performance (Normal vs. Adverse)​

​

12 of 17

Ablation Study – Importance of RGB-Flow Fusion

    • Study Focus - An ablation study assessed the contribution of RGB inputs by testing an optical-flow-only variant of the FlowSegModel under adverse weather.
    • Flow-Only Result - The optical-flow-only variant yielded inferior results across all classes. For instance, Vehicle IoU was 0.1431, compared to 0.8375 for the fused model.
    • Fusion Superiority - RGB-flow fusion yielded a 46.1% mean IoU improvement over the optical-flow-only variant.
    • Conclusion - Relying solely on optical flow limits semantic understanding, as flow captures motion but lacks the color and texture cues from RGB essential for distinguishing static elements like Background and Road in degraded visibility.

​

13 of 17

Conclusion and Implications

    • Key Finding - The FlowSegModel successfully demonstrates a robust semantic segmentation methodology for adverse weather conditions by fusing RGB images and optical flow.
    • Significance - This work advances robust segmentation techniques, contributing to safer and more reliable environmental perception in challenging conditions.
    • Impact - The findings hold significant implications for autonomous driving and outdoor vision systems, offering a practical approach to weather-invariant perception that may reduce real-world failure rates.

​

14 of 17

Limitations and Future Work

    • Limitations - Synthetic data (vKITTI 2) offers incomplete capture of sensor noise. Flow estimation has high computational demands, requiring optimization for edge deployment.
    • Future Work (Data) - Incorporate real datasets (e.g., BDD100K).
    • Future Work (Fusion) - Explore the use of multi-modal inputs like LiDAR.

15 of 17

THANK YOU

AI

16 of 17

Results – FlowSegModel Resilience

Normal Conditions

FlowSegModel achieved a mean IoU of 0.8406.

Adverse Conditions

Under adverse scenarios (fog, rain, overcast), the mean IoU was 0.8034.

Resilience

This represents a robust degradation of only 4.4%.

Attribution

This resilience is attributed to the optical flow's ability to provide complementary motion cues, countering visibility challenges.

TABLE 3 - COMPARISON OF OPTICAL FLOW VISUALIZATION FOR A SINGLE SCENE UNDER VARYING WEATHER CONDITIONS

​

17 of 17

Deep vs. Classical

Classical models (Farnebäck, Horn-Schunck) exhibited catastrophic performance with EPE values exceeding 512 and Fl-all (outlier rate) at 100%.

Performance Gap

Fine-tuned deep models reduced EPE by over 96% compared to classical methods.

SEA-RAFT vs. RAFT

Fine-tuning significantly enhanced both RAFT and SEA-RAFT.

SEA-RAFT Result

Fine-tuned SEA-RAFT achieved a substantially reduced EPE of 0.7758 and an Fl-all of 3.67%.

SEA-RAFT Advantage

SEA-RAFT was superior to RAFT by 56.6% lower EPE due to its weather-aware design.

Table 2 - COMPARISON OF SEA-RAFT AND RAFT MODELS WITH CLASSICAL METHODS ACROSS DIFFERENT PARAMETERS. (Detailed)