Depth-Based Intervention Detection in the Neonatal Intensive Care Unit Using Vision Transformers

Abstract

Depth cameras can provide an effective, noncontact, and privacy-preserving means to monitor patients in the Neonatal Intensive Care Unit (NICU). Clinical interventions and routine care events can disrupt video-based patient monitoring. Automatically detecting these periods can decrease the time required for hand-annotating recordings, which is needed for system development. Moreover, the automatic detection can be used in the future for real-time or retrospective intervention event classification.

An intervention detection method based solely on depth data was developed using a vision transformer (ViT) model utilizing real-world data from patients in the NICU. Multiple design parameters were investigated, including encoding of depth data and perspective transform to account for nonoptimal camera placement. The best-performing model utilized ∼85 M trainable parameters, leveraged both perspective transform and HHA (Horizontal disparity, Height above ground, and Angle with gravity) encoding, and achieved a sensitivity of 85.6%, a precision of 89.8%, and an F1-Score of 87.6%.

Download

You can view the paper as a PDF here. The DOI is 10.3390/s24237753. The Git repository containing the code and scripts used for analysis can be found here.

Copyright Information

© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

@article{hajj-aliDepthbasedInterventionDetection2024,
title = {Depth-Based Intervention Detection in the Neonatal Intensive Care Unit Using Vision Transformers},
author = {{Hajj-Ali}, Zein and Dosso, Yasmina Souley and Greenwood, Kim and Harrold, JoAnn and Green, James R.},
year = {2024},
journal = {Sensors},
volume = {24},
number = {7753},
issn = {1424-8220},
doi = {10.3390/s24237753}}