1. Home
  2. Vol 7, No 6 (2023)
  3. Haque

Recognition of Bangladeshi Sign Language (BdSL) Words using Deep Convolutional Neural Networks (DCNNs)

Aminul Haque, Rishad Amin Pulok, Md Mizanur Rahman, Sanzida Akter, Nusrat Khan, Shamsul Haque

Abstract


In a world where effective communication is fundamental, individuals who are Deaf and Dumb (D&D) often face unique challenges due to their primary mode of communication—sign language. Despite the interpreters' invaluable roles, their lack of availability causes communication difficulties for the D&D individuals. This study explores whether the field of Human-Computer Interaction (HCI) could be a potential solution. The primary objective is to assist D&D individuals with computer applications that could act as mediators to bridge the communication gap between them and the wider hearing population. To ensure their independent communication, we propose an automated system that could detect specific Bangla Sign Language (BdSL) words, addressing a critical gap in the sign language detection and recognition literature. Our approach leverages deep learning and transfer learning principles to convert webcam-captured hand gestures into textual representations in real-time. The model's development and assessment rest upon 992 images created by the authors, categorized into ten distinct classes representing various BdSL words. Our findings show the DenseNet201 and ResNet50-V2 models achieve promising training and testing accuracies of 99% and 93%, respectively.

 

Doi: 10.28991/ESJ-2023-07-06-019

Full Text: PDF


Keywords


Sign Language; BdSL; CNN; Deep Learning; ResNet50-V2; DenseNet201; MobileNet-V2; Image Processing.

References


Wazalwar, S. S., & Shrawankar, U. (2017). Interpretation of sign language into English using NLP techniques. Journal of Information and Optimization Sciences, 38(6), 895–910. doi:10.1080/02522667.2017.1372136.

BNFD (2023). The Deaf Annual Report. Bangladesh National Federation of the Deaf (BNFD), Dhaka, Bangladesh.

Perlmutter, D. M. (1991). The Language of the Deaf. New York Review of Books, 28(3), 65-72.

Perlmutter, D. M. (1993). Sonority and syllable structure in American Sign Language. Current Issues in ASL Phonology (Phonetics and Phonology), Academic Press, 227-261. doi:0.1016/B978-0-12-193270-1.50016-9.

Chandra Karmokar, B., Md. Rokibul Alam, K., & Kibria Siddiquee, M. (2012). Bangladeshi Sign Language Recognition Employing Neural Network Ensemble. International Journal of Computer Applications, 58(16), 43–46. doi:10.5120/9370-3846.

Blanco, E., & Moldovan, D. (2015). A Semantic Logic-Based Approach to Determine Textual Similarity. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(4), 683–693. doi:10.1109/taslp.2015.2403613.

Rahman, Md. M., Islam, Md. S., Rahman, Md. H., Sassi, R., Rivolta, M. W., & Aktaruzzaman, M. (2019). A New Benchmark on American Sign Language Recognition using Convolutional Neural Network. 2019 International Conference on Sustainable Technologies for Industry 4.0 (STI), Dhaka, Bangladesh. doi:10.1109/sti47673.2019.9067974.

Hossen, M. A., Govindaiah, A., Sultana, S., & Bhuiyan, A. (2018). Bengali Sign Language Recognition Using Deep Convolutional Neural Network. 2018 Joint 7th International Conference on Informatics, Electronics & Vision (ICIEV) and 2018 2nd International Conference on Imaging, Vision & Pattern Recognition (IcIVPR), Kitakyushu, Japan. doi:10.1109/iciev.2018.8640962.

Kayas, G., Hossain, S., & Hasan, M. (2016). Automatic Recognition of Bangla Sign Language Using Artificial Neural Networks (ANNS) For Deaf and Dumb to Bridge the Communication Gap. International Journal of Recent Advances in Multidisciplinary Research, 3(7), 1649-1654.

Bloshchynskyi, I., Bahrii, H., Nanivska, L., Tsviak, L., Isaieva, I., Skyba, K., ... & Mishchynska, I. (2022). Gender Characteristics of Individual’s Linguistic Behavior in the Context of Future Translators’ Professional Training. Emerging Science Journal, 6, 199-208. doi:10.28991/ESJ-2022-SIED-014.

Islam, M. S., Mousumi, S. S. S., Azad Rabby, A. K. M. S., Hossain, S. A., & Abujar, S. (2018). A potent model to recognize Bangla sign language digits using a convolutional neural network. Procedia Computer Science, 143, 611–618. doi:10.1016/j.procs.2018.10.438.

Podder, K. K., Chowdhury, M. E. H., Tahir, A. M., Mahbub, Z. Bin, Khandakar, A., Hossain, M. S., & Kadir, M. A. (2022). Bangla Sign Language (BdSL) Alphabets and Numerals Classification Using a Deep Learning Model. Sensors, 22(2), 574. doi:10.3390/s22020574.

Hasan, M. J., Nahid Hasan, S. K., & Alam, K. S. (2022). Deep Convolutional Neural Network-Based Bangla Sign Language Detection on a Novel Dataset. Machine Intelligence and Data Science Applications. Lecture Notes on Data Engineering and Communications Technologies, 132. Springer, Singapore. doi:10.1007/978-981-19-2347-0_13.

Wadhawan, A., & Kumar, P. (2020). Deep learning-based sign language recognition system for static signs. Neural Computing and Applications, 32(12), 7957–7968. doi:10.1007/s00521-019-04691-y.

Talukder, D., & Jahara, F. (2020). Real-Time Bangla Sign Language Detection with Sentence and Speech Generation. 2020 23rd International Conference on Computer and Information Technology (ICCIT), 19-21 December 2020, Dhaka, Bangladesh. doi:10.1109/iccit51783.2020.9392693.

Al Rafi, A., Hassan, R., Rabiul Islam, Md., & Nahiduzzaman, Md. (2023). Real-Time Lightweight Bangla Sign Language Recognition Model Using Pre-trained MobileNetV2 and Conditional DCGAN. Studies in Autonomic, Data-Driven and Industrial Computing, 263–276. doi:10.1007/978-981-19-7528-8_21.

Sarawate, N., Leu, M. C., & Öz, C. (2015). A real-time American Sign Language word recognition system based on neural networks and a probabilistic model. Turkish Journal of Electrical Engineering and Computer Sciences, 23, 2107–2123. doi:10.3906/elk-1303-167.

Rajam, P. S., & Balakrishnan, G. (2012). Recognition of Tamil sign language alphabet using image processing to aid deaf-dumb people. Procedia Engineering, 30, 861–868. doi:10.1016/j.proeng.2012.01.938.

Stein, D., Dreuw, P., Ney, H., Morrissey, S., & Way, A. (2007). Hand in hand: automatic sign language to English translation. TMI-07 - Proceedings of the 11th Conference on Theoretical and Methodological Issues in Machine Translation, 7-9 September, 2007, Skövde, Sweden.

Uddin, A., & Chowdhury, S. A. (2017). Hand Sign Language Recognition for Bangla Alphabet using Support Vector Machine. 2016 International Conference on Innovations in Science, Engineering and Technology, ICISET 2016, Dhaka, Bangladesh. doi:10.1109/ICISET.2016.7856479.

Rahaman, M. A., Jasim, M., Ali, M. H., & Hasanuzzaman, M. (2020). Bangla language modeling algorithm for automatic recognition of hand-sign-spelled Bangla sign language. Frontiers of Computer Science, 14(3), 143302. doi:10.1007/s11704-018-7253-3.

National Centre for Special Education (1997). Bengali Sign Language Dictionary. Ministry of Social Welfare in cooperation with the Norwegian Association of The Deaf and Bangladesh National Federation of The Deaf, Dhaka, Bangladesh.

Rafi, A. M., Nawal, N., Bayev, N. S. N., Nima, L., Shahnaz, C., & Fattah, S. A. (2019). Image-based Bengali Sign Language Alphabet Recognition for Deaf and Dumb Community. 2019 IEEE Global Humanitarian Technology Conference (GHTC), Seattle, United States. doi:10.1109/ghtc46095.2019.9033031.

Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint, arXiv:1409.1556. doi:10.48550/arXiv.1409.1556.

Shin, J., Musa Miah, A. S., Hasan, Md. A. M., Hirooka, K., Suzuki, K., Lee, H.-S., & Jang, S.-W. (2023). Korean Sign Language Recognition Using Transformer-Based Deep Neural Network. Applied Sciences, 13(5), 3029. doi:10.3390/app13053029.

Mehedi Shamrat, F. M. J., Chakraborty, S., Billah, Md. M., Kabir, M., Shadin, N. S., & Sanjana, S. (2021). Bangla numerical sign language recognition using convolutional neural networks (CNNs). Indonesian Journal of Electrical Engineering and Computer Science, 23(1), 405. doi:10.11591/ijeecs.v23.i1.pp405-413.

Akash, S. K., Chakraborty, D., Kaushik, M. M., Babu, B. S., & Zishan, Md. S. R. (2023). Action Recognition Based Real-time Bangla Sign Language Detection and Sentence Formation. 2023 3rd International Conference on Robotics, Electrical and Signal Processing Techniques (ICREST), Dhaka, Bangladesh. doi:10.1109/icrest57604.2023.10070072.

Lipi, K. A., Adrita, S. F. K., Tunny, Z. F., Munna, A. H., & Kabir, A. (2022). Static-gesture word recognition in Bangla sign language using convolutional neural network. Telkomnika (Telecommunication Computing Electronics and Control), 20(5), 1109–1116. doi:10.12928/TELKOMNIKA.v20i5.24096.

Das, S., Imtiaz, M. S., Neom, N. H., Siddique, N., & Wang, H. (2023). A hybrid approach for Bangla sign language recognition using deep transfer learning model with random forest classifier. Expert Systems with Applications, 213, 118914. doi:10.1016/j.eswa.2022.118914.

Titarmare, N., Vaidya, C., Meshram, R., Dongre, A., Jawale, P., Bambale, N., & Awachaat, O. (2023). Hand Sign Language Detection - Using Deep Neural Network. 2023 IEEE International Students’ Conference on Electrical, Electronics and Computer Science (SCEECS). doi:10.1109/sceecs57921.2023.10063060.


Full Text: PDF

DOI: 10.28991/ESJ-2023-07-06-019

Refbacks

  • There are currently no refbacks.


Copyright (c) 2023 Aminul Haque, Rishad Amin Pulok, Md Mizanur Rahman, Sanzida Akter, Nusrat Khan