Enhancing Assessment and Feedback in Game Design Programs: Leveraging Generative AI for Efficient and Meaningful Evaluation

James Hutson; Ben Fulcher; Jeremiah Ratican

doi:10.46661/ijeri.11038

Authors

James Hutson Lindenwood University https://orcid.org/0000-0002-0578-6052
Ben Fulcher Lindenwood University https://orcid.org/0009-0005-4338-6067
Jeremiah Ratican Lindenwood University

DOI:

https://doi.org/10.46661/ijeri.11038

Keywords:

Generative AI, Game Design Education, Automated Assessment, Feedback Efficiency, Student Engagement

Abstract

The integration of generative AI tools in game design education offers promising ways to streamline the grading, assessment, and feedback processes that are typically labor-intensive. In game design programs, faculty often deal with varied file formats, including 3D models, executable prototypes, videos, and complex game design documents. Traditional methods of assessment and feedback, primarily text-based, struggle to provide timely and actionable insights for students. Furthermore, only a small percentage of top students consistently review and apply feedback, leading to inefficiencies. This article explores how generative AI tools can augment these processes by automating aspects of grading, generating more personalized and meaningful feedback, and addressing the time-intensive nature of reviewing diverse file formats. Key strategies are discussed, including the use of rubrics tailored for AI-based assessment, automated prompts for narrative-driven assignments, and the application of AI in reviewing complex project builds. The objective is to create more time for faculty to engage in live mentoring and hands-on learning activities, which research shows to be more effective. Practical examples of various game design assignments, including build reviews and document evaluations, are provided to illustrate these new approaches. This shift promises to enhance student engagement and improve learning outcomes.

Downloads

Download data is not yet available.

References

Adams, E. (2014). Fundamentals of game design. Pearson Education.

Alam, A. (2023). Developing a Curriculum for Ethical and Responsible AI: A University Course on Safety, Fairness, Privacy, and Ethics to Prepare Next Generation of AI Professionals. In Intelligent Communication Technologies and Virtual Mobile Networks (pp. 879-894). Springer Nature Singapore. https://doi.org/10.1007/978-981-99-1767-9_64

Anjum, A., Li, Y., Law, N., Charity, M., & Togelius, J. (2024, May). The Ink Splotch Effect: A case study on ChatGPT as a co-creative game designer. In Proceedings of the 19th International Conference on the Foundations of Digital Games (pp. 1-15). https://doi.org/10.1145/3649921.3650010

Bennett, R. E. (2011). Formative assessment: A critical review. Assessment in education: principles, policy & practice, 18(1), 5-25. https://doi.org/10.1080/0969594X.2010.513678

Black, R.W. (2008). Adolescents and online fan fiction. Peter Lang.

Brunnberg, J. (2020). How tools shape the game authoring process. https://www.diva-portal.org/smash/record.jsf?pid=diva2%3A1449483&dswid=2939

Burke, K. (2010). From standards to rubrics in six steps: Tools for assessing student learning. Corwin Press.

Chen, Y., Clayton, E. W., Novak, L. L., Anders, S., & Malin, B. (2023). Human-centered design to address biases in artificial intelligence. Journal of medical Internet research, 25, e43251. https://doi.org/10.2196/43251

Couper, J. (2018). The struggle is real: Investigating the challenge of assigning a failing clinical grade. Nursing Education Perspectives, 39(3), 132-138. https://doi.org/10.1097/01.NEP.0000000000000295

Crompton, H., & Burke, D. (2023). Artificial intelligence in higher education: the state of the field. International Journal of Educational Technology in Higher Education, 20(1), 22. https://doi.org/10.1186/s41239-023-00392-8

Dastyar, S. (2019). The Investigation Of The Effectiveness of Participatory Learning Education On Students Motivation And Academic Achievement. International Journal of Advanced Research and Publications, 3 (8), 165-170.

Easterday, M. W., Rees Lewis, D., & Gerber, E. M. (2017). Designing crowdcritique systems for formative feedback. International Journal of Artificial Intelligence in Education, 27, 623-663. https://doi.org/10.1007/s40593-016-0125-9

Ewell, P. T., Paulson, K., & Kinzie, J. (2011). Down and in: Assessment practices at the program level. National Institute for Learning Outcomes Assessment. Retrieved from http://www.learningoutcomesassessment.org

Exterman, D. (2021). The Gamification of Everything. DevOps.com. https://devops.com/the-gamification-of-everything/

Ferrara, E. (2023). Fairness and Bias in Artificial Intelligence: A Brief Survey of Sources, Impacts, and Mitigation Strategies. Sci, 6, 3. https://doi.org/10.3390/sci6010003

Fitch, S. (2023). Art, assessment and uncertainty. Art, Design & Communication in Higher Education, 22(2), 263-275. https://doi.org/10.1386/adch_00077_1

Gui, Y., Cai, Z., Yang, Y., Kong, L., Fan, X., & Tai, R. H. (2023). Effectiveness of digital educational game and game design in STEM learning: a meta-analytic review. International Journal of STEM Education, 10(1), 36. https://doi.org/10.1186/s40594-023-00424-9

Hall, P., & Ellis, D. (2023). A systematic review of socio-technical gender bias in AI algorithms. Online Information Review, 47 (7), 1264-1279. https://doi.org/10.1108/OIR-08-2021-0452

Heil, J., & Ifenthaler, D. (2023). Online Assessment in Higher Education: A Systematic Review. Online Learning, 27(1), 187-218. https://doi.org/10.24059/olj.v27i1.3398

Hutson, J., & Plate, D. (2023). Enhancing institutional assessment and reporting through conversational technologies: exploring the potential of AI-powered tools and natural language processing. Journal of Artificial Intelligence and Robotics, 1(1), 11-22. https://doi.org/10.59232/AIR-V1I1P102

Islam, S. R., Russell, I., Eberle, W., & Dicheva, D. (2022, March). Incorporating the Concepts of Fairness and Bias into an Undergraduate Computer Science Course to Promote Fair Automated Decision Systems. In Proceedings of the 53rd ACM Technical Symposium on Computer Science Education V. 2 (pp. 1075-1075). https://doi.org/10.1145/3478432.3499043

Jacobs, M., Kurtz, C., Simon, J., & Böhmann, T. (2021). Value sensitive design and power in socio-technical ecosystems. Internet Policy Review, 10(3), 1-26. https://doi.org/10.14763/2021.3.1580

Jenkins, H. (2006). Convergence culture. University Press.

Jin, X., Jiang, Q., Xiong, W., Feng, Y., & Zhao, W. (2024). Effects of student engagement in peer feedback on writing performance in higher education. Interactive Learning Environments, 32(1), 128-143. https://doi.org/10.1080/10494820.2022.2081209

Kapp, Karl (2012). The Gamification of Learning and Instruction: Game-Based Methods and Strategies for Training and Education. Pfeiffer. https://doi.org/10.1145/2207270.2211316

Kordzadeh, N., & Ghasemaghaei, M. (2022). Algorithmic bias: review, synthesis, and future research directions. European Journal of Information Systems, 31(3), 388-409. https://doi.org/10.1080/0960085X.2021.1927212

Kramarzewski, A., & De Nucci, E. (2023). Practical Game Design: A modern and comprehensive guide to video game design. Packt Publishing Ltd.

Kritek, P. A. (2015). Strategies for effective feedback. Annals of the American Thoracic Society, 12(4), 557-560. https://doi.org/10.1513/AnnalsATS.201411-524FR

Leavy, S. (2018, May). Gender bias in artificial intelligence: The need for diversity and gender theory in machine learning. In Proceedings of the 1st international workshop on gender equality in software engineering (pp. 14-16). https://doi.org/10.1145/3195570.3195580

Levin, Douglas & Arafeh, Sousan. (2003). The Digital Disconnect: The Widening Gap Between Internet-Savvy Students and Their Schools. Internet and American Life. ii-30.

Liu, H., Choi, M., Kao, D., & Mousas, C. (2023). Synthesizing game levels for collaborative gameplay in a shared virtual environment. ACM Transactions on Interactive Intelligent Systems, 13(1), 1-36. https://doi.org/10.1145/3558773

Mashhadi, A., Zolyomi, A., & Quedado, J. (2022, April). A Case Study of Integrating Fairness Visualization Tools in Machine Learning Education. In CHI Conference on Human Factors in Computing Systems Extended Abstracts (pp. 1-7). https://doi.org/10.1145/3491101.3503568

McGinness, H. T., Caldwell, P. H., Gunasekera, H., & Scott, K. M. (2020). An educational intervention to increase student engagement in feedback. Medical Teacher, 42(11), 1289-1297. https://doi.org/10.1080/0142159X.2020.1804055

Moreno-Ger, P., Burgos, D., Martínez-Ortiz, I., Sierra, J. L., & Fernández-Manjón, B. (2008). Educational game design for online education. Computers in Human Behavior, 24(6), 2530-2540. https://doi.org/10.1016/j.chb.2008.03.012

Morton, J. K., Northcote, M., Kilgour, P., & Jackson, W. A. (2021). Sharing the construction of assessment rubrics with students: A Model for collaborative rubric construction. Journal of University Teaching & Learning Practice, 18(4). https://doi.org/10.53761/1.18.4.9

Ntoutsi, E., Fafalios, P., Gadiraju, U., Iosifidis, V., Nejdl, W., Vidal, M. E., ... & Staab, S. (2020). Bias in data‐driven artificial intelligence systems—An introductory survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10(3), e1356. https://doi.org/10.1002/widm.1356

Petrova, E. (2020). Checklists and Rubrics: Do They Really Promote Learning? In ICERI2020 Proceedings (pp. 4906-4911). IATED. https://doi.org/10.21125/iceri.2020.1069

Seering, J., Mayol, R., Harpstead, E., Chen, T., Cook, A., & Hammer, J. (2019, October). Peer feedback processes in the game industry. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play (pp. 427-438). https://doi.org/10.1145/3311350.3347176

Serrano-Laguna, Á., Manero, B., Freire, M., & Fernández-Manjón, B. (2018). A methodology for assessing the effectiveness of serious games and for inferring player learning outcomes. Multimedia Tools and applications, 77, 2849-2871. https://doi.org/10.1007/s11042-017-4467-6

Sezen, T. I. (2024). Analog prototyping for digital game design. In Encyclopedia of Computer Graphics and Games (pp. 102-104). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-031-23161-2_135

Squire, Kurt (2011). Video Games and Learning: Teaching and Participatory Culture in the Digital Age. Columbia University.

Stãnescu, I. A., Stefan, A., Kravcik, M., Lim, T., & Bidarra, R. (2013). Interoperability strategies for serious games development. Internet Learning, 2(1), 373-378. https://doi.org/10.18278/il.2.1.4

Stevens, D. D., & Levi, A. J. (2023). Introduction to rubrics: An assessment tool to save grading time, convey effective feedback, and promote student learning. Routledge.

Sudhakaran, S., González-Duque, M., Glanois, C., Freiberger, M., Najarro, E., & Risi, S. (2023, July). Prompt-guided level generation. In Proceedings of the Companion Conference on Genetic and Evolutionary Computation (pp. 179-182). https://doi.org/10.1145/3583133.3590656

Srivastava, O., Tennant, M., Grewal, P., Rubin, U., & Seamone, M. (2023). Artificial intelligence and machine learning in ophthalmology: A review. Indian Journal of Ophthalmology, 71(1), 11-17. https://doi.org/10.4103/ijo.IJO_1569_22

Tang, L. M., & Kay, J. (2014, July). Gamification: metacognitive scaffolding towards long term goals? In UMAP workshops (pp. 7-11).

Udeozor, C., Chan, P., Russo Abegão, F., & Glassey, J. (2023). Game-based assessment framework for virtual reality, augmented reality and digital game-based learning. International Journal of Educational Technology in Higher Education, 20 (36). https://doi.org/10.1186/s41239-023-00405-6

Umbrello, S. (2020). Combinatory and complementary practices of values and virtues in design: A reply to Reijers and Gordijn. Filosofia, (65), 107-121.

Wang, T., Lund, B. D., Marengo, A., Pagano, A., Mannuru, N. R., Teel, Z. A., & Pange, J. (2023). Exploring the potential impact of artificial intelligence (AI) on international students in higher education: Generative AI, chatbots, analytics, and international student success. Applied Sciences, 13(11), 6716. https://doi.org/10.3390/app13116716

Winstone, N., & Carless, D. (2019). Designing effective feedback processes in higher education: A learning-focused approach. Routledge. https://doi.org/10.4324/9781351115940

Winstone, N. E., & Boud, D. (2022). The need to disentangle assessment and feedback in higher education. Studies in higher education, 47(3), 656-667. https://doi.org/10.1080/03075079.2020.1779687

Wolf, K. D., Maya, F., & Heilmann, L. (2023). Explainable Feedback for Learning Based on Rubric-Based Multimodal Assessment Analytics with AI. https://aisop.de/EPEPLA/Feedback.pdf

Wollny, S., Schneider, J., Di Mitri, D., Weidlich, J., Rittberger, M., & Drachsler, H. (2021). Are we there yet? -A systematic literature review on chatbots in education. Frontiers in artificial intelligence, 4, https://doi.org/10.3389/frai.2021.654924

Wolz, U., Ault, C., & Nakra, T. M. (2007). Teaching game design through cross-disciplinary content and individualized student deliverables. Journal of Game Development, 2(2), 19-32.

Zarkoob, H. (2024). AI-powered methods for academic assessment: overcoming scalability challenges in large university classrooms and conference review. [Doctoral dissertation], University of British Columbia.

Zouhaier, S. (2023). The impact of Artificial intelligence on higher education: An empirical study. European Journal of Educational Sciences, 10(1), 17-33. https://doi.org/10.19044/ejes.v10no1a17

Zytko, D., J. Wisniewski, P., Guha, S., PS Baumer, E., & Lee, M. K. (2022, April). Participatory design of AI systems: opportunities and challenges across diverse users, relationships, and application domains. In CHI Conference on Human Factors in Computing Systems Extended Abstracts (pp. 1-4). https://doi.org/10.1145/3491101.3516506