Multimodal data lifecycle in intelligent system for synchronous lecture note-taking and semantic analysis

Abstract

The article theoretically substantiates, formalizes, and develops an end-to-end life cycle of multimodal data in an intelligent learning support system. It is determined that traditional manual note-taking and separate analysis of records create a critical cognitive barrier due to the gap between the presentation of the material and its consolidation, as well as due to the lack of a reliable factual connection between the responses of artificial intelligence and the primary lecture materials. A multi-stage model of the life cycle of multimodal data is proposed: low-level capture of the video stream of the presentation window and the audio sequence of the teacher's speech; adaptive frame filtering based on pixel-by-pixel comparison of frame images in grayscale; layout segmentation and optical text recognition using a hybrid neural network architecture; spatio-temporal synchronization of audiovisual streams with an error of less than two seconds; coordinate-temporal knowledge certification; parallel double indexing in vector and symbol spaces; synchronous streaming note-taking and organization of operational interactive polling during a lesson based on reciprocal rank merging. Unlike existing approaches, the developed methodology is focused on providing the student with the opportunity to ask questions in real time regarding the presented lecture fragment directly during the lesson and receive a verified answer. Laboratory bench verification and the obtained estimates confirm the effectiveness of the proposed solutions: the end-to-end delay of the frame processing pipeline is three hundred forty-two milliseconds, the average delay of the primary response is four hundred twenty milliseconds, and the architecture provides the target completeness of context extraction of eighty-nine percent and reduces the risk of forming statements that are not confirmed by the materials of the current lesson.