The quality evaluation index system for AI-generated digital educational resources

Yanli  Chen; Chunlan  Lv

doi:10.54691/crzb0f70

Authors

Yanli Chen
Chunlan Lv

DOI:

https://doi.org/10.54691/crzb0f70

Keywords:

AI-generated content; digital educational resources; quality evaluation index; entropy weight method; Delphi method.

Abstract

With the rapid development of Generative AI (GAI) technology, AI-generated content (AIGC) has become the main mode of content creation, replacing User-Generated Content (UGC) and Professional Generated Content (PGC). This transformation has led to the adoption of a brand-new form for AI-generated digital educational resources (AIGDER). However, concerns have arisen regarding the quality of generated content. To address this issue, this paper proposes a comprehensive evaluation index system for AIGDER by integrating the Delphi method and the entropy weight method. Firstly, through a systematic review of recent literature, potential quality indicators covering content, expression, user aspects, and technical aspects were identified. Subsequently, these indicators were refined through two rounds of expert consultation using the Delphi method, and finally, a structured quality indicator system was established, including four dimensions and twenty specific indicators. After that, the weight coefficients of the quality indicators were determined using the entropy weight method, and the importance of each indicator was analyzed. The proposed system provides a framework for relevant stakeholders to select high-quality AIGDER (AI educational resources) and make AI tools conform to educational standards.

Downloads

Download data is not yet available.

References

[1] Wang, W. and K. Siau, Artificial Intelligence, Machine Learning, Automation, Robotics, Future of Work and Future of Humanity: A Review and Research Agenda. Journal of Database Management, 2019. 30(1): p. 61-79.

[2] Lin, H., et al., Comparing AIGC and traditional idea generation methods: Evaluating their impact on creativity in the product design ideation phase. Thinking Skills and Creativity, 2024. 54(000).

[3] Xu, M., et al., Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services. Ieee Communications Surveys and Tutorials, 2024. 26(2): p. 1127-1170.

[4] Cao, Y., et al., A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT arXiv. arXiv, 2023.

[5] Lambert, J. and M. Stevens, ChatGPT and Generative AI Technology: A Mixed Bag of Concerns and New Opportunities. Computers in the Schools, 2024. 41(4): p. 559-583.

[6] Wu, T.-T., et al., Promoting Self-Regulation Progress and Knowledge Construction in Blended Learning via ChatGPT-Based Learning Aid. Journal of Educational Computing Research, 2024. 61(8): p. 3-31.

[7] Kortemeyer, G., Could an artificial-intelligence agent pass an introductory physics course? Physical Review Physics Education Research, 2023. 19(1).

[8] Sun, Y., et al., AI hallucination: towards a comprehensive classification of distorted information in artificial intelligence-generated content. Humanities & Social Sciences Communications, 2024. 11(1).

[9] Watts, F.M., et al., Comparing Student and Generative Artificial Intelligence Chatbot Responses to Organic Chemistry Writing-to-Learn Assignments. Journal of Chemical Education, 2023. 100(10): p. 3806-3817.

[10] Jing, Y., et al., Bibliometric mapping techniques in educational technology research: A systematic literature review. Education and Information Technologies, 2024. 29(8): p. 9283-9311.

[11] Labadze, L., M. Grigolia, and L. Machaidze, Role of AI chatbots in education: systematic literature review. International Journal of Educational Technology in Higher Education, 2023. 20(1).

[12] Waqar, A., et al., Applications of AI in oil and gas projects towards sustainable development: a systematic literature review. Artificial Intelligence Review, 2023. 56(11): p. 12771-12798.

[13] Moher, D., et al., Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Bmj-British Medical Journal, 2009. 339.

[14] Dalkey, N. and O. Helmer, AN EXPERIMENTAL APPLICATION OF THE DELPHI METHOD TO THE USE OF EXPERTS. Management Science, 1963. 9(3): p. 458-467.

[15] Olsen, A.A., et al., How to use the Delphi method to aid in decision making and build consensus in pharmacy education. Currents in Pharmacy Teaching and Learning, 2021. 13(10): p. 1376-1385.

[16] Zhao, P., Z.M. Ali, and Y. Ahmad, Developing indicators for sustainable urban regeneration in historic urban areas: Delphi method and Analytic Hierarchy Process (AHP). Sustainable Cities and Society, 2023. 99.

[17] Khan, R.A., et al., Curriculum Viability Indicators: A Delphi Study to Determine Standards and Inhibitors of a Curriculum. Evaluation & the Health Professions, 2021. 44(3): p. 210-219.

[18] Seo, E., J. Ryu, and S. Hwang, Building key competencies into an environmental education curriculum using a modified Delphi approach in South Korea. Environmental Education Research, 2020. 26(6): p. 890-914.

[19] Guo, Y. and J. Li, Application of tender evaluation model based on entropy weight to electric power construction projects. Engineering Journal of Wuhan University, 2013. 46(2): p. 184-7.

[20] Tanaka, O.M., et al., Assessing the reliability of ChatGPT: a content analysis of self-generated and self-answered questions on clear aligners, TADs and digital imaging. Dental press journal of orthodontics, 2023. 28(5): p. e2323183-e2323183.

[21] Liu, X. and Y. Xiao, Chinese university teachers' engagement with generative AI in different stages of foreign language teaching: A qualitative enquiry through the prism of ADDIE. Education and Information Technologies, 2025. 30(1): p. 485-508.

[22] Maheshwari, G., Factors influencing students' intention to adopt and use ChatGPT in higher education: A study in the Vietnamese context. Education and Information Technologies, 2024. 29(10): p. 12167-12195.

[23] Lu, G., N.B. Hussin, and A. Sarkar, Navigating the Future: Harnessing Artificial Intelligence Generated Content (AIGC)for Enhanced Learning Experiences in Higher Education. 2024 International Conference on Advances in Modern Age Technologies for Health and Engineering Science. 2024. 1-12.