1. Home
  2. Vol 8, No 3 (2024)
  3. Farooqui

Framing Assessment Questions in the Age of Artificial Intelligence: Evidence from ChatGPT 3.5

Mohammad Owais Farooqui, Mohd Imran Siddiquei, Shashank Kathpal

Abstract


With the rise of artificial intelligence (AI), higher education faces a significant challenge in learning assessment. The emergence of tools like ChatGPT raises concerns regarding the potential for cheating and the reliability of assessment outcomes. This paper aims to address these concerns by proposing a methodology for framing questions that effectively measures learning outcomes while reducing the risk of AI-enabled cheating. To achieve this objective, we employ a methodological approach that involves getting responses from ChatGPT 3.5 to various question prompts across different domains. These responses are then evaluated by faculty members specializing in management education. Through this process, we aim to identify question-framing strategies that effectively assess learning outcomes while minimizing susceptibility to AI Cheating. Our analysis reveals several key findings. Certain question Types (Decision Making, Recent Events, and Experiential Learning) demonstrate greater resilience against AI-generated responses, indicating their potential effectiveness in assessing student learning. This study offers original insights into the challenges and opportunities associated with learning assessment in the context of AI integration. The paper tries to provide valuable guidance for Policymakers, educators & students seeking to enhance the integrity and reliability of their assessment practices.

 

Doi: 10.28991/ESJ-2024-08-03-09

Full Text: PDF


Keywords


Human-Computer Interface; Improving Classroom Teaching; Teaching/Learning Strategies.

References


Gozalo-Brizuela, R., & Garrido-Merchan, E. C. (2023). ChatGPT is not all you need. A State-of-the-Art Review of large Generative AI models. arXiv preprint, arXiv:2301.04655. doi:10.48550/arXiv.2301.04655.

Susnjak, T. (2022). ChatGPT: The end of online exam integrity? arXiv preprint, arXiv:2212.09292. doi:10.48550/arXiv.2212.09292.

Lo, C. K. (2023). What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature. Education Sciences, 13(4), 410. doi:10.3390/educsci13040410.

Ouyang, F., Zheng, L., & Jiao, P. (2022). Artificial intelligence in online higher education: A systematic review of empirical research from 2011 to 2020. Education and Information Technologies, 27(6), 7893–7925. doi:10.1007/s10639-022-10925-9.

Chen, L. E., Cheng, S. Y., & Heh, J.-S. (2021). Chatbot: A Question Answering System for Student. 2021 International Conference on Advanced Learning Technologies (ICALT), Tartu, Estonia. doi:10.1109/icalt52272.2021.00110.

Qadir, J. (2023). Engineering Education in the Era of ChatGPT: Promise and Pitfalls of Generative AI for Education. 2023 IEEE Global Engineering Education Conference (EDUCON), Kuwait, Kuwait. doi:10.1109/educon54358.2023.10125121.

Mhlanga, D. (2023). Open AI in Education, the Responsible and Ethical Use of ChatGPT Towards Lifelong Learning. SSRN Electronic Journal, 1-199. doi:10.2139/ssrn.4354422.

Rudolph, J., Tan, S., & Tan, S. (2023). ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? Journal of Applied Learning and Teaching, 6(1), 342–363. doi:10.37074/jalt.2023.6.1.9.

Pavlik, J. V. (2023). Collaborating with ChatGPT: Considering the Implications of Generative Artificial Intelligence for Journalism and Media Education. Journalism & Mass Communication Educator, 78(1), 84–93. doi:10.1177/10776958221149577.

Sims, A. (2023, March 9). ChatGPT and the future of University Assessment. Times Higher Education, Washington, United States. Available online: https://www.timeshighereducation.com/campus/chatgpt-and-future-university-assessment (accessed on June 2024).

Loh, E. (2023). ChatGPT and generative AI chatbots: challenges and opportunities for science, medicine and medical leaders. BMJ Leader, 8(1), 51–54. doi:10.1136/leader-2023-000797.

Guo, B., Zhang, X., Wang, Z., Jiang, M., Nie, J., Ding, Y., Yue, J., & Wu, Y. (2023). How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection. http://arxiv.org/abs/2301.07597.

Noorbehbahani, F., Mohammadi, A., & Aminazadeh, M. (2022). A systematic review of research on cheating in online exams from 2010 to 2021. Education and Information Technologies, 27(6), 8413–8460. doi:10.1007/s10639-022-10927-7.

Arnold, I. J. M. (2016). Cheating at online formative tests: Does it pay off? Internet and Higher Education, 29, 98–106. doi:10.1016/j.iheduc.2016.02.001.

Joyner, D. A. (2023). ChatGPT in Education: Partner or Pariah? XRDS: Crossroads, The ACM Magazine for Students, 29(3), 48–51. doi:10.1145/3589651.

Moqbel, M. S. S., & Al-Kadi, A. M. T. (2023). Foreign language learning assessment in the age of ChatGPT: A theoretical account. Journal of English Studies in Arabia Felix, 2(1), 71-84. doi:10.56540/jesaf.v2i1.62.

Ifelebuegu, A. (2023). Rethinking online assessment strategies: Authenticity versus AI chatbot intervention. Journal of Applied Learning and Teaching, 6(2), 1-8.

Izzo, Z., Liu, R., & Zou, J. (2023, July). Data-driven subgroup identification for linear regression. In International Conference on Machine Learning. Proceedings of the 40th International Conference on Machine Learning, PMLR 202:14531-14552, 2023.

Goralski, M. A., & Tan, T. K. (2020). Artificial intelligence and sustainable development. The International Journal of Management Education, 18(1), 100330. doi:10.1016/j.ijme.2019.100330.

Linn, R. L., Baker, E. L., & Dunbar, S. B. (1991). Complex, Performance-Based Assessment: Expectations and Validation Criteria. Educational Researcher, 20(8), 15–21. doi:10.3102/0013189X020008015.

Natriello, G. (2013). The Impact of Evaluation Processes on Students. School and Classroom Organization, 227–246, Routledge, Milton Park, United Kingdom. doi:10.4324/9780203056950-9.

Birenbaum, M. (2007). Evaluating the Assessment: Sources of Evidence for Quality Assurance. Studies in Educational Evaluation, 33(1), 29–49. doi:10.1016/j.stueduc.2007.01.004.

Maclellan, E. (2004). How convincing is alternative assessment for use in higher education? Assessment and Evaluation in Higher Education, 29(3), 311–321. doi:10.1080/0260293042000188267.

Baartman, L. K. J., Prins, F. J., Kirschner, P. A., & van der Vleuten, C. P. M. (2007). Determining the Quality of Competence Assessment Programs: a Self-Evaluation Procedure. Studies in Educational Evaluation, 33(3–4), 258–281. doi:10.1016/j.stueduc.2007.07.004.

Roberts, T.G. (2011). What Is Experiential Learning. The Agricultural education magazine, 83, 4.

Wurdinger, S., & Allison, P. (2017). Faculty perceptions and use of experiential learning in higher education. Journal of e-learning and Knowledge Society, 13(1), 1-12.

Osoba, O., & Welser, W. (2017). An Intelligence in Our Image: The Risks of Bias and Errors in Artificial Intelligence, RAND Corporation, Santa Monica, United States. doi:10.7249/rr1744.

Banerjee, D. N., & Chanda, S. S. (2020). AI failures: A review of underlying issues. arXiv preprint, arXiv:2008.04073. doi:10.48550/arXiv.2008.04073.

Perkins, M., & Roe, J. (2023). Decoding Academic Integrity Policies: A Corpus Linguistics Investigation of AI and Other Technological Threats. Higher Education Policy. doi:10.1057/s41307-023-00323-2.

Michel-Villarreal, R., Vilalta-Perdomo, E., Salinas-Navarro, D. E., Thierry-Aguilera, R., & Gerardou, F. S. (2023). Challenges and Opportunities of Generative AI for Higher Education as Explained by ChatGPT. Education Sciences, 13(9), 856. doi:10.3390/educsci13090856.

Alafnan, M. A., Dishari, S., Jovic, M., & Lomidze, K. (2023). ChatGPT as an Educational Tool: Opportunities, Challenges, and Recommendations for Communication, Business Writing, and Composition Courses. Journal of Artificial Intelligence and Technology, 3(2), 60–68. doi:10.37965/jait.2023.0184.

Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., … Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274. doi:10.1016/j.lindif.2023.102274.

Farrokhnia, M., Banihashem, S. K., Noroozi, O., & Wals, A. (2023). A SWOT analysis of ChatGPT: Implications for educational practice and research. Innovations in Education and Teaching International, 61(3), 460–474. doi:10.1080/14703297.2023.2195846.


Full Text: PDF

DOI: 10.28991/ESJ-2024-08-03-09

Refbacks

  • There are currently no refbacks.


Copyright (c) 2024 Mohammad Owais Farooqui, Mohd Imran Siddiquei, Shashank Kathpal