Segmentation and detection of humans in a multispectral video stream under emergency conditions

Abstract

This work focuses on improving the accuracy and robustness of human detection and segmentation in multispectral video streams during emergency situations, while maintaining performance levels sufficient for real-time operation. It is demonstrated that directly transferring detectors trained on visible-spectrum images to the infrared spectrum is ineffective due to cross-modal domain shift; the modalities differ in the physical nature of the signal, noise levels, contrast characteristics, and the set of available textural features. The study analyzes key approaches to human detection and segmentation in the infrared range, including thermal detectors and specialized datasets, domain adaptation and feature transfer from the visible spectrum, multispectral detectors with varying fusion levels, and methods for compensating for modal imbalance. A common limitation of these approaches is identified: fixed reliance on specific modalities and a uniform level of preprocessing depth across all frames. An improved method for human detection and segmentation is proposed, based on a dual-branch YOLOv8m-seg architecture that integrates a quantitative metric of modality informativeness, adaptive frame preprocessing, and cross-modal feature fusion with dynamic weighting. The informativeness of RGB and IR channels is determined based on normalized characteristics—specifically local root-mean-square contrast, luminance distribution entropy, Tenengrad gradient energy, an inverse haze density estimate, and the signal-to-noise ratio—with the thermal channel additionally accounting for target thermal range consistency. Metric values ​​fall within the [0, 1] range and are categorized into four levels: very low (I < 0.25), low (0.25 ≤ I < 0.45), medium (0.45 ≤ I < 0.70), and high (I ≥ 0.70). When I < 0.45, the intervention mode is determined by the profile of the metric components: rapid correction or neural network-based restoration is applied depending on the nature of the degradation, whereas additional preprocessing is bypassed if informativeness is sufficient. Cross-modal fusion is implemented using variable spatial weights for RGB and IR features, determined according to the local informativeness of each modality. Additionally, a training dataset generation strategy aligned with the physics of degradation is employed, alongside techniques for handling small-scale and partially occluded targets. Experimental validation was conducted using the KAIST Multispectral Pedestrian, LLVIP, and FMB open multispectral datasets, consolidated into the reproducible MS-ER protocol; materials for experiment replication are available in a public GitHub repository. The experimental dataset comprised 26,841 pairs of RGB-IR frames (totaling 53,682 images), with an additional 55,427 human masks generated for training the segmentation component. Six configurations (M1–M6) were used for a controlled comparison: RGB-only, thermal-only, static RGB-IR fusion, ICAFusion, the proposed model without adaptive preprocessing, and the full M6 model featuring an informativeness metric, adaptive preprocessing, and local modality weight control. An experimental run of 5,000 frames revealed that 46.8% of frames were processed without additional preprocessing, 31.6% underwent rapid correction, and 21.6% required deep reconstruction; the average processing time was 26.3 ms/frame, with a mean throughput of 38.0 FPS (peaking at 49.8 FPS) and peak GPU memory usage of 6.4 GB. The results confirm the viability of the proposed method and the feasibility of adaptively redistributing computational resources based on the real-time informativeness of the RGB and IR channels. Applying the most resource-intensive mode only to degraded frames maintains sufficient performance for real-time multispectral video processing while leveraging the complementary information from RGB and IR modalities for human detection and segmentation under challenging surveillance conditions.