TEMPORAL SEGMENTATION AND HAND GESTURE RECOGNITION USING MAMBA SSM ARCHITECTURE

Abstract

Despite significant advancements in assistive technology, communication barriers remain a pervasive challenge for the hearing-impaired community, particularly regarding Vietnamese Sign Language (VSL). Existing recognition systems often face a trade-off dilemma: Recurrent Neural Networks (RNNs) struggle with vanishing gradients when modeling long gesture sequences, while Transformerbased models incur quadratic computational costs that hinder real-time deployment on edge devices. To address these limitations, this thesis proposes a comprehensive end-to-end framework leveraging the Mamba State Space Model (SSM), a novel architecture capable of capturing long-range temporal dependencies with linear computational complexity (𝑂(𝑁)), thereby bridging the gap between high accuracy and operational efficiency. The core recognition framework orchestrates two specialized Mamba-based modules: a Temporal Segmenter and a Gesture Classifier. Experimental results demonstrate that the Mamba Segmenter achieves a Mean Intersection over Union (mIoU) of 55.69%, outperforming the TCN baseline by over 14%, particularly in detecting ambiguous transition states. Furthermore, the Mamba Classifier attains a remarkable mAP of 0.9937, surpassing the Bi-LSTM baseline in both stability and inference speed. These results validate the efficacy of Mamba’s Selective Scan mechanism in filtering kinematic noise while retaining crucial semantic context. Beyond theoretical modeling, this study culminates in the deployment of a fully functional real-time application using ONNX Runtime and PyQt6. The system successfully translates continuous VSL streams into natural language text with low latency on standard consumer hardware. This practical implementation proves the feasibility of Mamba SSM as a lightweight, scalable solution for sign language recognition, laying a solid foundation for future large-scale dictionary expansion.