Interactive multimodal learning: towards using pedagogical agents for inclusive education

A pedagogical agent is an animated interface in an interactive online learning environment. Its role involves guiding users through instructions and participating in direct discussions. Numerous research studies underline the advantages of pedagogical agents in delivering instruction, tailoring learning experiences, and promoting inclusivity. The authors utilize machine learning techniques to develop an interactive multimodal application (combining visuals and audio) designed as a pedagogical agent. This application is intended to assist in teaching primary school students (aged 6 to 8) the recognition of colors and letters. Additionally, it incorporates voice interaction to assess their learning progress. The overarching objective of this pedagogical agent is to seamlessly integrate artificial intelligence (AI) into educational settings, thereby amplifying student engagement and motivation during the learning process. This paper will explore the journey of creating a machine learning- based application and its potential utility in supporting educators. Furthermore, the authors will investigate potential augmentations that could render the application more adept at aiding students with dyslexia. The paper will emphasize the significance of human- computer interaction (HCI), as it significantly shapes pedagogical agents' design and operational aspects.

© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.