A Real-Time Deep Learning-Based Sign Language Recognition System for Words, Alphabets, and Numbers

This paper presents a sign language recognition system based on deep learning and computer vision. It aims to support communication between deaf and hard-of-hearing individuals and the community. The proposed system translates hand gestures into textual output through real-time image and video processing. It supports three recognition modes, including number, alphabet, and word recognition. For alphabet and digit recognition, MobileNetV2 is adopted due to its lightweight architecture and suitability for real-time deployment. To improve recognition accuracy and robustness in practical scenarios, a custom-collected dataset is combined with publicly available American Sign Language (ASL) datasets. The resulting dataset covers static alphabet gestures (excluding the motion-based letters J and Z) and digit gestures corresponding to the numbers 0–9. For word recognition, the pretrained Inflated 3D ConvNet (I3D) model is adopted to process short video clips and learn both spatial and temporal gesture features. Additionally, OpenCV is employed for video processing and real-time frame capture, while DroidCam is used to capture live input. For the MobileNetV2 model, a Region of Interest (ROI) is applied before prediction. Experimental results from real-time testing demonstrate high confidence scores for alphabet and number recognition, achieving 93.43% and 92.8%, respectively, at 30 FPS. In addition, the word recognition module achieves moderate performance, with Top-1 and Top-5 accuracies of 39.2% and 68.0%, respectively.