Creating the First Bengali-Russian Sign Language Dictionary for Inclusive Multilingual Communication

Arifa Ashrafi, Yuriy Philippovich

Abstract


This paper outlines the creation of the first Bengali-Russian Sign Language Dictionary, aimed at promoting inclusive communication by a multilingual, user-friendly platform. The research involved creating a detailed dataset with static and dynamic samples for both Bengali and Russian sign languages, supported by the implementation of different Convolutional Neural Network (CNN) models for effective sign recognition. A user interface for Bengali-Russian Sign Languages has been developed to offer an accessible experience, with expert evaluations and performance efficiency confirming its usability and user-friendliness. The dictionary’s design supports future scalability, allowing for additional languages and signs to be added. By integrating advanced CNN models, the platform achieves robust accuracy, making it suitable for various educational and community settings. This work lays a foundation for further innovations in digital inclusivity, potentially transforming sign language education for hearing impaired communities especially for underprivileged individuals. This research also explores video-to-animation conversion techniques, resulting in improved feedback for users during the learning process. To understand the educational impact on diverse student groups, field studies were conducted with both street and mainstream children. The results showed that a positive, digitally inclusive learning environment greatly improves engagement and understanding. Also the findings from the research suggest that once the dictionary is launched, this platform will significantly bridge communication gaps and provide a structured educational resource for Bengali, and Russian hearing-impaired communities.

Full Text:

PDF

References


Ashrafi, A., Mokhnachev, V.S., Philippovich, Y.N., Tsilenko, L.P. (2020). Development of Image Dataset Using Hand Gesture Recognition System for Progression of Sign Language Translator. In: Silhavy, R., Silhavy, P., Prokopova, Z. (eds) Software Engineering Perspectives in Intelligent Systems. CoMeSySo 2020. Advances in Intelligent Systems and Computing, vol 1294. Springer, Cham. https://doi.org/10.1007/978-3-030-63322-6_56

Farooq, U., Rahim, M.S.M., Sabir, N. et al. Advances in machine translation for sign language: approaches, limitations, and challenges. Neural Comput & Applic 33, 14357–14399 (2021). https://doi.org/10.1007/s00521-021-06079-3

A. Ashrafi, V. S. Mokhnachev and A. E. Harlamenkov, "Improving Sign Language Recognition with Machine Learning and Artificial Intelligence," 2024 6th International Youth Conference on Radio Electronics, Electrical and Power Engineering (REEPE), Moscow, Russian Federation, 2024, pp. 1-6, doi: 10.1109/REEPE60449.2024.10479844.

A. Ashrafi, V. S. Mokhnachev, Y. N. Philippovich, and L. P. Tsilenko, “Video classification using CNN-LSTM architecture for Bengali sign language” in Fundamental and applied sciences today XXVIII, p. 140-146. Pothi.com, Bengaluru, Karnataka, India (2022).

A. Ashrafi, V. S. Mokhnachev, Y. N. Philippovich, A. Y. Harlamenkov, and L. P. Tsilenko, “Comparative Study Of Different Sign Language Recognition Methods”, Natrazvitie, St. Petersburg, 2022, pp. 28-31.

Ashrafi, A., Mokhnachev, V., Philippovich, Y., Harlamenkov, A., Chernenko, S. (2023). Russian Sign Language Recognition Using MediaPipe. In: Dolinina, O., et al. Artificial Intelligence in Models, Methods and Applications. AIES 2022. Studies in Systems, Decision and Control, vol 457. Springer, Cham. https://doi.org/10.1007/978-3-031-22938-1_21

A. R. Várkonyi-Kóczy and B. Tusor, “Human–computer interaction for smart environment applications using fuzzy hand posture and gesture models”, IEEE Trans. Instrum. Meas., 2011, 60, (5), pp. 1505– 1514.

C. C. Hsieh and D. H. Liou, “Novel Haar features for real-time hand gesture recognition using SVM”, Journal of Real-Time Image Process., 2015, 10, (2), pp. 357– 370.

M. Tang, “Recognizing hand gestures with Microsoft's Kinect”, Stanfordedu, 2011, 14, (4), pp. 303– 313

Q. Chen, N. D. Georganas, and E. M. Petriu, “Hand gesture recognition using haarlike features and a stochastic context-free grammar”, IEEE Trans. Instrum. Meas., 2008, 57, (8), pp. 1562– 1571.

G. Plouffe and A. Cretu, “Static and dynamic hand gesture recognition in depth data using dynamic time warping”, IEEE Trans. Instrum. Meas., 2016, 65, (2), pp. 305– 316.

J. Han, L. Shao, and J. Shotton, “Enhanced computer vision with Microsoft Kinect sensor: a review”, IEEE Trans. Cybern., 2013, 43, (5), pp. 1318– 1334.

C. Wang, Z. Liu, and S. Chan, “Superpixel-based hand gesture recognition with Kinect depth camera”, IEEE Trans. Multimed., 2015, 17, (1), pp. 29– 39.

Y. Yao and Y. Fu, “Contour model-based hand-gesture recognition using the Kinect sensor”, IEEE Trans. Circuits Syst. Video Technol., 2014, 24, (11), pp. 1935– 1944.

F. Jiang, C. Wang, Y Gao, S Wu, and D Zhao, “Discriminating features learning in hand gesture classification”, IET Comput. Vis., 2015, 9, (5), pp. 673– 680.

Shuxian Liu, Wei Wang, Le Deng, Huan Xu, Cnn-trans model: A parallel dual-branch network for fundus image classification, Biomedical Signal Processing and Control, Volume 96, Part B, 2024, 106621, ISSN 1746-8094, https://doi.org/10.1016/j.bspc.2024.106621.

Y. Farhan and A. Ait Madi, "Real-time Dynamic Sign Recognition using MediaPipe," 2022 IEEE 3rd International Conference on Electronics, Control, Optimization and Computer Science (ICECOCS), Fez, Morocco, 2022, pp. 1-7, doi: 10.1109/ICECOCS55148.2022.9982822.

Sign Language Detection and Recognition Using Media Pipe and Deep Learning Algorithm . (2024). International Journal of Scientific Research in Science and Technology, 11(2), 123-130. https://doi.org/10.32628/IJSRST52411223

Laveen Chandwani, Jaydeep Khilari, Kunal Gurjar et al. Gesture based Sign Language Recognition system using Mediapipe, 27 June 2023, PREPRINT (Version 1) available at Research Square [https://doi.org/10.21203/rs.3.rs-3106646/v1]

J. Zamora-Mora and M. Chacón-Rivas, "Costa Rican Sign Language Recognition using MediaPipe," 2022 International Conference on Inclusive Technologies and Education (CONTIE), Cartago, Costa Rica, 2022, pp. 1-6, doi: 10.1109/CONTIE56301.2022.10004438.

Zhang, Z., Cao, Y., Shi, X., & Liu, C. (2020). "Hand Gesture Recognition Using MediaPipe." International Journal of Advanced Computer Science and Applications, 11(6), 568-574. https://doi.org/10.14569/IJACSA.2020.0110672

Alasmari A, Alhothali A, Allinjawi A. Hybrid machine learning approach for Arabic medical web page credibility assessment. Health Informatics Journal. 2022;28(1). doi:10.1177/14604582211070998

Ashrafi, A., Mokhnachev, V.S., Philippovich, Y.N. Converting Video into Animation Using Video Tracking Technology for Developing a Multilingual Dictionary of Sign Language. IIASU'24 3rd All-Russian scientific conference "Artificial Intelligence in Automated Control Systems and Data Processing", within the framework of the congress "Russian Engineer" at Bauman Moscow State Technical University, 2024.

Ashrafi A., Mokhnachev V.S. Conversion Of Indian Sign Language Into Text Using Gesture Recognition Method. Theory and Practice of Modern Science: the View of Youth. Материалы Всероссийской научно-практической конференции на английском языке. Научное издание. Сост. Е.Н. Лашина, М.С. Липатов. Под общей редакцией В.В. Кирилловой . Санкт-Петербург, 2022. С. 204-211.

Ashrafi, A., Mokhnachev, V.S. (2023) Designing User-Friendly Interfaces For A Multilingual Sign Language Dictionary, Artificial Intelligence in Automated Control and Data Processing Systems, Collection of Articles of the 2nd All-Russian Scientific Conference: In 5 Volumes, April 27–28, Moscow.


Refbacks

  • There are currently no refbacks.


Abava  Кибербезопасность ИБП для ЦОД СНЭ

ISSN: 2307-8162