TWI840781B - Educational training test system, method and computer readable medium based on augmented reality technology - Google Patents

Abstract

The invention discloses an educational training test system, method and computer readable medium based on augmented reality technology. An augmented reality content design tool subsystem loads an augmented reality education training material, and an expert end designs an exam question, a script and an answer according to the augmented reality education training material to save to a servo device, and then loads the exam question and the script from the servo device to the augmented reality mobile device. When the augmented reality mobile device recognizes a target object associated with the exam question, the augmented reality mobile device presents the exam question associated with the target object, and records an on-site real-time picture of test process when a student end performs the exam question on the target object and transmits it to the servo device, and then the servo device compares the change of a video characteristic on a time axis between the on-site real-time picture and the answer to calculate a test result of the exam question.

Description

Educational training test system, method and computer-readable medium based on augmented reality technology

The present invention relates to an educational training test technology, and in particular to an educational training test system, method and computer-readable medium based on augmented reality (AR) technology.

In the existing augmented reality (AR) education and training or its teaching materials, in practice, experts first record teaching videos, edit the teaching videos into training materials, and then provide them to students for training in augmented reality. However, they fail to provide a method to implement testing through augmented reality to verify the learning effect of students and increase the effectiveness of training.

In a prior art, a system for virtual behavior training using an augmented reality user device is provided. The system is based on pre-made teaching materials and can dynamically change the content of the teaching materials according to the learning environment. In addition, the system verifies the operation actions of the trainee through a mobile device during the training process of the trainee.

However, this existing technology cannot synchronously design relevant test questions and answers based on augmented reality education training materials to automate the entire testing process, nor can it enable students to take tests in augmented reality after receiving education and training to evaluate learning effects, nor can it provide corresponding feedback to experts who can design augmented reality education training materials, and it cannot realize the needs of remote monitoring or real-time assistance for students through the remote collaboration function of augmented reality mobile devices.

Therefore, how to provide an innovative education and training testing technology to solve any of the above problems or provide related functions/services has become a major research topic for technical personnel in this field.

The present invention provides an innovative education and training test system, method and computer-readable medium based on augmented reality technology, which enables the expert to design related test questions and answers synchronously based on augmented reality education and training materials through electronic devices to automate the entire test process, or enables the learner to take the test in an augmented reality manner through an augmented reality mobile device after receiving education and training to evaluate the learning effect, or to provide corresponding feedback to the expert who can design augmented reality education and training materials, or to realize the needs of remote monitoring or real-time assistance for the learner through the remote collaboration function of the augmented reality mobile device.

The educational training test system based on augmented reality technology described in the present invention comprises: an electronic device at the expert end, which has an augmented reality content design tool subsystem to load an augmented reality educational training material, and then the expert end uses the augmented reality content design tool subsystem to design at least one test question, a test question script and a content design answer for the test question according to the augmented reality educational training material; a server The device stores the test questions, scripts and answers designed by the expert through the augmented reality content design tool subsystem based on the augmented reality education training materials; and at least one augmented reality mobile device of the student side loads the test questions and scripts designed by the expert through the augmented reality content design tool subsystem based on the augmented reality education training materials from the server device, wherein when the augmented reality mobile device of the student side When the augmented reality mobile device recognizes the object associated with the test question, the augmented reality mobile device on the student side presents the test question associated with the object, and the augmented reality mobile device records the real-time image of the process of the student testing the object, and the augmented reality mobile device on the student side transmits the real-time image of the process of the test question to the server device, and the server device transmits the test question to the server device. The live real-time images of the test process of the test question and the answers designed by the experts for the content of the test question are compared with each other's audio-visual characteristics on the time axis, so that the server device can calculate the test result or score of the student on the test question based on the comparison results of the audio-visual characteristics of the live real-time images of the test process of the test question and the answers designed by the experts for the content of the test question on the time axis.

The educational training test method based on augmented reality technology described in the present invention includes: an augmented reality content design tool subsystem of an electronic device at the expert end loads an augmented reality educational training material, and then the expert end uses the augmented reality content design tool subsystem to design at least one test question, a test question script, and an answer to the test question content according to the augmented reality educational training material, so as to expand the augmented reality content design tool subsystem. The reality content design tool subsystem stores the test questions, scripts and answers designed according to the augmented reality education training materials in the server device; at least one augmented reality mobile device of the learner end loads the test questions and scripts designed by the expert end through the augmented reality content design tool subsystem according to the augmented reality education training materials from the server device; and when the augmented reality mobile device of the learner end recognizes When the student finds an object related to the test question, the student's augmented reality mobile device presents the test question related to the object, and the augmented reality mobile device records the real-time image of the student's test process on the object. The student's augmented reality mobile device transmits the real-time image of the test process to the server device, and the server device transmits the test process to the server device. The live real-time images of the process and the answers designed by the experts for the content of the test questions are compared with each other's audio-visual characteristics on the time axis, so that the server device can calculate the test results or scores of the students based on the comparison results of the audio-visual characteristics of the live real-time images of the test process and the answers designed by the experts for the content of the test questions on the time axis.

The computer-readable medium of the present invention is applied to a computing device or a computer, and stores instructions to execute the above-mentioned educational training and testing method based on augmented reality technology.

In order to make the above features and advantages of the present invention more clearly understandable, the following examples are given and detailed descriptions are provided in conjunction with the attached drawings. The following description will partially explain the additional features and advantages of the present invention, and these features and advantages will be partially known from the description or can be learned through the practice of the present invention. It should be understood that both the general description above and the detailed description below are exemplary and explanatory, and are not intended to limit the scope of the present invention.

1: Educational training and testing system based on augmented reality technology

10: Electronic devices

11: First communication module

12: Browser

13: Augmented reality content design tool subsystem

131: User Interface

14: Textbook editing module

141: Exam question definition unit

20: Servo device

21: Second communication module

22: Answer recognition module

23: Storage module

30: Augmented reality mobile devices

31: Third communication module

32:Test application

33: Remote collaboration module

34: Object recognition module

35: Voice recognition module

A: Expert side

B: Server side

C: Student side

D: Expand real-life education training materials

E: Subject matter

N: Network

S11 to S15: Steps

S21 to S30: Steps

Figure 1 is a schematic diagram of the architecture of the educational training and testing system based on augmented reality technology described in the present invention.

Figure 2 is a schematic diagram of the process of the expert designing test questions, scripts and answers based on the content of the augmented reality education training materials in the education training test method based on augmented reality technology described in the present invention.

FIG3 is a schematic diagram of an embodiment of the educational training test method based on augmented reality technology described in the present invention, in which the expert side designs test questions, scripts and answers through the test question definition unit.

Figure 4 is a schematic diagram of the process of the method of conducting the test on the student side in the educational training test method based on augmented reality technology described in the present invention.

FIG5 is a schematic diagram of an implementation example of the comparison of the changes in the audio-visual features of the on-site real-time picture and the answer in the educational training test method based on augmented reality technology described in the present invention.

FIG6 is a schematic diagram of an implementation example of the comparison of changes in audio-visual features on the time axis in the educational training test system and method based on augmented reality technology described in the present invention.

The following describes the implementation of the present invention through a specific concrete implementation form. People familiar with this technology can understand other advantages and effects of the present invention from the content disclosed in this manual, and can also implement or use it through other different specific equivalent implementation forms.

FIG1 is a schematic diagram of the structure of the educational training test system 1 based on augmented reality technology described in the present invention. As shown in the figure, the educational training test system 1 based on augmented reality technology may include at least one (such as multiple) electronic devices 10 of the expert end A, a server device 20 of the server end B, and at least one (such as multiple) augmented reality mobile devices 30 of the student end C that are interconnected or communicated through a network N. The expert end A and the student end C may be located at different locations, different areas, the same location or the same area, for example, the expert end A is located at the remote end of the object E (student end C) and the student end C is located at the site of the object E, or both the expert end A and the student end C are located at the site of the object E.

In the present invention, "at least one" means more than one (such as one, two or more than three), "plurality" means more than two (such as two, three, four, five or more than ten), "connection" or "communication" means wired or wireless connection or communication, and the network N can be a wired network, a wireless network, the Internet, a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a metropolitan area network (MAN), a virtual private network (VPN), etc.

The electronic device 10 of the expert end A may have a first communication module 11, a browser 12, an augmented reality content design tool subsystem 13, and a teaching material editing module 14, etc. The augmented reality content design tool subsystem 13 may have a user interface 131, and the teaching material editing module 14 may have a test question definition unit 141. The server device 20 of the server end B may have a second communication module 21, an answer recognition module 22, and a storage module 23, etc. The augmented reality mobile device 30 of the student end C may have a third communication module 31, a test application (APP) 32, a remote collaboration module 33, an object recognition module 34 and a voice recognition module 35, etc., and the augmented reality mobile device 30 can recognize at least one or more objects E.

In one embodiment, the expert end A may be an expert (such as a professional/teacher/professor/mentor), an editor end (such as a teaching material editor/exam question editor/question setter), a production end (such as a teaching material producer), etc., the server end B may be a cloud end, a remote end, etc., and the student end C may be a student (such as a student), a test subject, a learner, etc. The electronic device 10 may be a computer, a mobile device, an editing device, etc., such as a personal computer, a tablet computer, a laptop computer, a desktop computer, a smart phone, a personal digital assistant (PDA), etc. The server device 20 may be a server or a server host, such as a cloud server, a remote server, a network server, a general server, a central server, etc. The augmented reality mobile device 30 may be an augmented reality (AR) glasses, or a smart phone, tablet computer, laptop computer, personal digital assistant (PDA), etc. with augmented reality (AR) function.

In one embodiment, the first communication module 11, the second communication module 21 or the third communication module 31 can be a communicator (chip/circuit), a communication interface, a communication software (program), etc., the browser 12 can be an Internet browser, etc., the teaching material editing module 14 can be a teaching material editor (chip/circuit), teaching material editing software (program), etc., and the test question definition unit 141 can be a test question definition software (program), etc. The answer recognition module 22 can be an answer recognition device (chip/circuit), answer recognition software (program), etc., and the storage module 23 can be a storage device, a memory, a memory card, a database, a hard disk (such as a cloud hard disk/network hard disk), etc. Various storage media. The remote collaboration module 33 may be a remote collaboration chip/circuit, remote collaboration software (program), etc., the object recognition module 34 may be an object recognizer (chip/circuit), object recognition software (program), etc., and the voice recognition module 35 may be a voice recognizer (chip/circuit), voice recognition software (program), etc. The target object E may be a target device associated with the test questions (such as test equipment/hardware equipment/operating equipment), a test object (such as a test paper), etc.

The electronic device 10 of the expert end A can be connected to the network N and use the browser 12, and the augmented reality content design tool subsystem 13 of the augmented reality education training teaching material content editing tool can load (download or obtain) the augmented reality education training teaching material D in the server device 20 of the server end B through the first communication module 11. The augmented reality content design tool subsystem 13 can be an application running on a browser 12 (such as an Internet browser), so that the expert terminal A can operate the augmented reality education training material content editing tool of the augmented reality content design tool subsystem 13 through the browser 12 of the electronic device 10, so as to edit the augmented reality education training material D through the augmented reality education training material content editing tool. The augmented reality content design tool subsystem 13 of the electronic device 10 can edit the position layout of multiple augmented reality content elements, and the augmented reality application of the augmented reality mobile device 30 (such as AR glasses) can be embedded with a web browser component (Web View) to load the edited results (design results) of the position layout of multiple augmented reality content elements from the web browser component to the augmented reality mobile device 30. The test question definition unit 141 of the teaching material editing module 14 of the electronic device 10 can provide the expert end A with the test question design function related to the augmented reality education training teaching material D, and the test question design function can include setting the subject E (such as the subject equipment) that needs to be identified in the test question, the arrangement of the relevant scripts, and the design of the corresponding answers.

The second communication module 21 of the server device 20 of the server end B can continuously obtain or receive audio and video data from the augmented reality mobile device 30 (such as AR glasses), and can also transmit the audio and video data of the augmented reality education training materials D in the server device 20 to the electronic device 10 of the expert end A (such as the editing end) and the augmented reality mobile device 30 of the student end C. The answer recognition module 22 of the server device 20 of the server end B can compare the changes of the audio-visual characteristics of the live real-time picture received from the student end C with the answer pre-defined by the expert end A in the augmented reality education training material D on the time axis, so that the answer recognition module 22 can calculate the answer status and test results (such as test scores or scores) of the student end C in real time based on the comparison results of the audio-visual characteristics of the live real-time picture and the answer on the time axis.

The augmented reality mobile device 30 (such as AR glasses) of the student end C may have a camera to capture the image of the target object E (such as the target equipment). The augmented reality mobile device 30 may also transmit data to and from the server device 20 of the server end B through the third communication module 31. The augmented reality mobile device 30 is installed with a test application 32 for the student end C to take the test. The remote collaboration module 33 may connect the augmented reality mobile device 30 of the student end C at the target object E with the electronic device 10 of the expert end A, so that the remote collaboration module 33 executes the collaboration service between the augmented reality mobile device 30 of the student end C and the electronic device 10 of the expert end A. The object recognition module 34 of the augmented reality mobile device 30 can recognize the target object E associated with the test question (such as the specified test question), so that the object recognition module 34 triggers the script or prompt associated with the test question according to the recognition result of the target object E. If the test question or answer designed by the expert terminal A through the electronic device 10 needs to be considered together with the voice feature (i.e., sound feature), the voice feature of the test question or answer can be recognized through the voice recognition module 35 of the electronic device 10.

In addition, the educational training test system 1 based on augmented reality technology described in the present invention and its method may include, for example, the following procedures [1] to [7], but are not limited thereto.

[1] Loading an augmented reality education training material D that can edit or control the content through the augmented reality content design tool subsystem 13 of the electronic device 10 of the expert terminal A.

[2] A user interface 131 is provided in the augmented reality content design tool subsystem 13, so that the expert A can design at least one (e.g., multiple) test questions, design (edit) test question scripts, and design answers for the content of each test question based on the augmented reality education training materials D through the user interface 131 of the augmented reality content design tool subsystem 13. The aforementioned script may include the trigger point (timing point) of the test question and the prompt content, etc., and the answer may be composed of at least one of the image features and the voice features (i.e., sound features).

[3] The electronic device 10 stores the test questions, scripts and answers designed by the augmented reality content design tool subsystem 13 according to the augmented reality education training materials D to the server device 20 (such as a cloud server) of the server end B (such as a cloud) through the network N.

[4] The augmented reality mobile device 30 (such as AR glasses) capable of capturing real-life images loads (downloads or obtains) the test questions and scripts designed by the expert end A through the augmented reality content design tool subsystem 13 based on the augmented reality education training materials E from the server device 20 of the server end B.

[5] When the augmented reality mobile device 30 (such as AR glasses) recognizes the target object E associated with the test question, the augmented reality mobile device 30 on the student end C presents the test question associated with the target object E.

[6] The augmented reality mobile device 30 (such as AR glasses) of the student end C records the real-time images (such as real-time live video images) of the process of the student end C taking the test on the target object E, and the augmented reality mobile device 30 of the student end C transmits the real-time images of the process of the test to the server device 20 (such as a cloud server) of the server end B (such as a cloud).

[7] The server device 20 of the server end B compares the changes in the audio-visual characteristics of the real-time images of the test process (such as real-time live audio-visual images) and the answers designed by the expert end A for the content of the test questions on the time axis, so that the server device 20 of the server end B calculates the test result or score of the student end C for the test question based on the comparison results of the changes in the audio-visual characteristics of the real-time images of the test process and the answers designed by the expert end A for the content of the test questions on the time axis.

In other words, the educational training test method based on augmented reality technology described in the present invention can be divided into the following two parts, including the method shown in Figures 2 to 3 [1] in which the expert side A designs test questions, scripts and answers based on the content of augmented reality education training materials D or through the test question definition unit 141, and the method shown in Figure 4 [2] in which the student side C conducts the test of the test questions.

FIG2 is a flowchart of the method for designing test questions, scripts and answers by the expert end A according to the content of the augmented reality education training teaching material D in the education training test method based on augmented reality technology described in the present invention. FIG3 is a flowchart of the method for designing test questions, scripts and answers by the expert end A through the test question definition unit 141 in the education training test method based on augmented reality technology described in the present invention. Please refer to FIG1 for explanation.

As shown in the flowchart of FIG. 2 , in step S11 , the expert end A uses the browser 12 of the electronic device 10 to request the server device 20 of the server end B to load the augmented reality education training material D (such as the file of the augmented reality education training material D) through the first communication module 11 . The expert end A (such as the editing end) can use the browser 12 to connect to the augmented reality education training teaching material content editing tool of the augmented reality content design tool subsystem 13, and then the augmented reality education training teaching material content editing tool requests the server device 20 of the server end B to load (download) the augmented reality education training teaching material D, and the augmented reality education training teaching material D may include teaching material videos composed of on-site real-time images (such as on-site audio and video images or on-site real-scene image images).

In step S12, the second communication module of the server device 20 of the server end B transmits the augmented reality education training material D to the electronic device 10 of the expert end A.

In step S13, the teaching material editing module 14 of the electronic device 10 of the expert end A loads the file of the augmented reality education training teaching material D transmitted by the server device 20 of the server end B.

In step S14, the expert terminal A designs at least one (e.g., multiple) test questions, scripts, and answers (see FIG. 3 ) for the extended reality education training teaching material D through the test question definition unit 141 of the teaching material editing module 14 of the electronic device 10, and the test question definition unit 141 of the teaching material editing module 14 designs the script including the trigger point (e.g., timing point) and prompt content of the test question. For example, when the object recognition module 34 of the augmented reality mobile device 30 (such as AR glasses) of the student end C recognizes real-time on-site images, if the object recognition module 34 finds an image of the target object E associated with the test question, or the image of the target object E presents a target-defined state (such as a standby state), the test application 32 of the augmented reality mobile device 30 of the student end C provides this test question. The answer to the test question can be defined by the audio-visual features included in the teaching material video of the augmented reality education training material D, and the expert terminal A can define the important step screens of the teaching material video of the augmented reality education training material D and their relationship on the time axis through the test definition unit 141 of the teaching material editing module 14 (see Figure 3), and then the test definition unit 141 of the teaching material editing module 14 will store the important step screens of the teaching material video of the augmented reality education training material D and their relationship on the time axis into a file including the test question, script and answer.

In step S15, the storage module 23 of the server device 20 of the server end B stores the test questions, scripts and answers designed by the test question definition unit 141 of the electronic device 10 of the expert end A.

As shown in the embodiment of FIG. 3, when the object recognition module 34 of the augmented reality mobile device 30 (such as AR glasses) of the learner end C recognizes the image of the target object E associated with the test question for the first time, the test application 32 of the augmented reality mobile device 30 can start downloading the test question from the server device 20 of the server end B. When the expert end A (such as the editor end) presses the "Save" button on the electronic device 10, the electronic device 10 can upload the test question, script and answer to the server device 20 (such as a cloud server) of the server end B (such as the cloud) in the form of setting data, and then the storage module 23 of the server device 20 stores the test question, script and answer designed by the test question definition unit 141.

FIG4 is a flowchart of the method for the student C to conduct a test in the educational training test method based on augmented reality technology described in the present invention. FIG5 is a schematic diagram of an embodiment of the educational training test method based on augmented reality technology described in the present invention for comparing the changes in the audio-visual features of the on-site real-time picture (such as the on-site operation picture) and the answer, and refer to FIG1 for explanation.

As shown in the flowchart of FIG4 , in step S21, the student end C is ready to take the test through the test application 32 of the augmented reality mobile device 30 (such as AR glasses), so the third communication module 31 of the augmented reality mobile device 30 of the student end C first requests the server device 20 of the server end B to load the file of the augmented reality education training material D.

In step S22, the server device 20 of the server end B transmits the file of the augmented reality education training material D including the script of the test questions to the augmented reality mobile device 30 (such as AR glasses) of the student end C through the second communication module 21.

In step S23, the test application 32 of the augmented reality mobile device 30 (such as AR glasses) of the student end C loads the file of the augmented reality education training material D including the script of the test questions from the server device 20 of the server end B.

In step S24, the object recognition module 34 of the augmented reality mobile device 30 (such as AR glasses) of the student end C recognizes the target object E (such as the target equipment) on the scene according to the design (setting) of the script, so that the test application 32 of the augmented reality mobile device 30 requests the server device 20 of the server end B to load the test questions associated with the target object E.

In step S25 to step S26, the server device 20 of the server end B transmits the test questions associated with the target object E to the augmented reality mobile device 30 (such as AR glasses) of the student end C through the second communication module 21, and then the augmented reality mobile device 30 loads the test questions through the third communication module 31.

For example, the script of the test questions can prompt the student end C about the target object E (such as the target equipment) of this test, and the student end C can align or correspond the camera (lens) of the augmented reality mobile device 30 to the target object E, so that the object recognition module 34 of the augmented reality mobile device 30 can identify the correct target object E, and then the test application 32 of the augmented reality mobile device 30 can request the server device 20 of the server end B (such as the cloud) to load (download) the test questions according to the design of the script. Next, the server device 20 of the server end B transmits the test questions to the augmented reality mobile device 30 of the student end C through the second communication module 21, so that the augmented reality mobile device 30 loads the test questions into the test application 32 through the third communication module 31, so that the test application 32 starts to display the test questions associated with the subject E.

In step S27, the student terminal C performs the test of the subject E (such as the operation of the subject equipment) according to the test questions to answer.

In step S28, the server device 20 of the server end B obtains or receives the on-site real-time picture (such as the on-site operation picture or the on-site real-time video and audio data) during the test process (such as the operation process) of the student end C from the test application 32 of the augmented reality mobile device 30 of the student end C, and then the server device 20 calls the answer recognition module 22 of the server device 20 of the server end B. That is, during the test process of the student end C, the test application 32 of the augmented reality mobile device 30 can continuously transmit the on-site real-time picture to the server device 20 of the server end B, so that the server device 20 of the server end B receives the on-site real-time picture through the second communication module 21, and then calls the answer recognition module 22 of the server device 20 for the on-site real-time picture.

In step S29, when the server device 20 of the server end B receives the on-site real-time picture (such as the on-site operation picture or the on-site real-time audio and video data), the server device 20 can call the answer recognition module 22 to compare the characteristic changes of the received on-site real-time picture and the answer, and then the answer recognition module 22 calculates the test result or score of the student end C according to the characteristic changes of the on-site real-time picture and the answer.

)之符號可代表答案識別模組22針對現場即時畫面與答案兩者之比對結果為考題已答對(如兩者匹配或測驗成功)，打叉(X)之符號可代表答案識別模組22針對現場即時畫面與答案兩者之比對結果為考題已答錯(如兩者不匹配或測驗失敗)，圓框(○)可代表有請求需為連續動作之步驟畫面。 As shown in the embodiment of FIG. 5 , the test application 32 of the augmented reality mobile device 30 can continuously transmit the on-site real-time picture (such as the on-site operation picture) of the learner end C to the server device 20 of the server end B, so that the answer recognition module 22 of the server device 20 can compare the changes of the audio and video characteristics of the on-site real-time picture and the answer on the time axis. For example, in FIG. 5 , the check mark (

) symbol may represent that the answer recognition module 22 compares the on-site real-time picture with the answer and the result is that the question has been answered correctly (such as the two match or the test is successful), the cross (X) symbol may represent that the answer recognition module 22 compares the on-site real-time picture with the answer and the result is that the question has been answered incorrectly (such as the two do not match or the test fails), and the circle (○) may represent that there is a step screen that requires a continuous action.

After the student terminal C completes all the operation steps of this test question for the target object E, if there are subsequent test questions to be completed, the test application 32 of the augmented reality mobile device 30 can prompt the student terminal C to continue the test of the subsequent test questions. On the contrary, if there are no subsequent test questions to be completed, the test application 32 of the augmented reality mobile device 30 can prompt the student terminal C that the test has been completed.

In step S30, when the test of the student end C is finished, the student end C can obtain the test result or score (such as the test score) of the test question through the augmented reality mobile device 30, and can also correct the answer to the wrong answer to the test question through the augmented reality mobile device 30. That is, when the student end C completes the test of the test question, the answer recognition module 22 of the server device 20 can provide the comparison result or score calculation result of the on-site real-time screen (such as the on-site operation screen) and the answer, so that the server device 20 automatically presents the test result or score of the student end C to the augmented reality mobile device 30, and the server device 20 can also provide the augmented reality education training materials D of the correct operation steps for the wrong answer to the test question for the student end C to refer to or watch.

Therefore, in the education and training test system 1 and method based on augmented reality technology described in the present invention, when the expert end A designs the content of the augmented reality education training material D through the electronic device 10, the test questions and answers can be designed according to the content of the augmented reality education training material D at the same time, so that after the learner end C receives the education and training related to the augmented reality education training material D, he can also take the test questions through the augmented reality method of the augmented reality mobile device 30, and the learning effect of the learner end C can also be evaluated. In addition, the present invention can deepen the impression and learning effect of the students C on the extended reality education training materials D, and can also serve as a reference for learning diagnosis or reduce training costs, and can also provide corresponding feedback to the experts A who can design the extended reality education training materials D.

In other words, the present invention can be used by the expert end A to design and expand the content, test questions and answers of the real-life education training teaching material D, and can also be used by the learner end C to conduct automated tests to verify the learning effect. The expert end A can also set the trigger point (timing point) and prompt content corresponding to the test question through the teaching material editing module 14, and the answer to the corresponding test question is the characteristic change of the important or key audio-visual data of the expanded real-life education training teaching material D on the time axis.

The answer recognition module 22 of the server device 20 can continuously and instantly compare the audio and video features of the live real-time screen of the student terminal C (such as the live test screen or real-time audio and video data) and the answer on the time axis, so that the answer recognition module 22 can calculate the test result or score of the student terminal C based on the comparison result of the live real-time screen and the answer. The server device 20 can also provide the correct answer or data (such as operation data) for the wrong answer or deducted test questions to the student terminal C for reference.

When the student C is taking the test, the augmented reality mobile device 30 is connected to the electronic device 10 of the expert A (such as a remote expert) to make the augmented reality mobile device 30 enter the monitoring mode. If the student C has questions about the test in the monitoring mode, the augmented reality mobile device 30 is connected to the electronic device 10 so that the student C can ask or reflect the questions about the test to the expert A in real time. At the same time, when the student C is taking the test (such as the operation process), the expert A can know the current answer status of the student C. If student C answers the test questions correctly, but the answer recognition module 22 misjudges or misidentifies the test results of student C, the expert A can correct the misjudgement or misidentification in real time through the electronic device 10.

For example, the embodiments described in the following procedures [1] to [8] are used to specifically illustrate the technical content or technical means of the present invention. Taking the augmented reality education training material D related to "turnout tip installation" of "railway turnout repair" as an example, the expert terminal A can design test questions and answers for this augmented reality education training material D through the electronic device 10, and then the student terminal C can take the test related to "turnout tip installation (on-line)" to verify the learning effect of the student terminal C.

[1] The expert terminal A connects to a web-based augmented reality content design tool subsystem 13 through the browser 12 of the electronic device 10, and then loads the augmented reality education training material D related to the "railway turnout repair" and "turnout tip installation" to be tested through the augmented reality content design tool subsystem 13.

[2] Designing test questions for the augmented reality education training material D related to "switch point installation". For example, the expert terminal A can design a script for the test questions related to "switch point installation" through the augmented reality content design tool subsystem 13 of the electronic device 10, and the design content of the script includes the trigger point (such as the timing point) and prompt content of the test questions. Then, the expert terminal A selects the first target E (such as a railway switch) or its state to be tested or identified through the electronic device 10 to set the trigger point (such as the timing point) for the test questions to appear. After the object recognition module 34 of the augmented reality mobile device 30 of the student end C recognizes the real-time on-site image, if the object recognition module 34 finds an image including the first target object E associated with the test question (such as an object image), or the image of the first target object E presents a target definition state (such as a standby state), the test application 32 of the augmented reality mobile device 30 of the student end C starts to provide the content of the first test question.

[3] Expert terminal A can design the content of at least one (e.g., multiple) test questions through the augmented reality content design tool subsystem 13 of the electronic device 10 as needed, and the content of the test questions can include the main text of the test questions and prompts such as completion target prompts to be displayed to the learner terminal C for viewing. Expert terminal A can confirm the event markers of the start and end of each test question, and each trigger point (e.g., timing point) will also be designed through the test question script. For example, expert terminal A can design two test questions for the augmented reality education training material D related to "switch point area installation", namely "Test question 1: Power-off operation of the first target object E (e.g., railway switch)" and "Test question 2: Installation of the second target object E (e.g., switch buckle)". When the object recognition module 34 of the augmented reality mobile device 30 of the student end C successfully recognizes the first target object E (such as a railway switch), the test application 32 of the augmented reality mobile device 30 can display the prompt content of test questions 1 and 2, so as to inform the student end C through the prompt content that the test question 1 is the power-off operation for performing the first target object E (such as a railway switch). Then, after the augmented reality mobile device 30 confirms the test completion screen of the recognized test question 1, the test application 32 of the augmented reality mobile device 30 starts to display the content of test question 2.

[4] The expert terminal A designs corresponding answers for each test question through the augmented reality content design tool subsystem 13 of the electronic device 10, and the definition of the answer can include the characteristic changes of important or key audio-visual data of the augmented reality education training material D on the time axis. The scope of the design that the expert terminal A can design includes marking of important steps on the augmented reality education training material D, continuous steps that must be completed within a specific time limit, and possible necessary matching voice steps. For example, Question 1 should include at least three important steps in sequence: "open the first object E (such as a railway switch)", "perform the power-off operation of the first object E (such as a railway switch)", and "confirm the power-off signal in the switch area". Question 2 should include at least two important steps in sequence: "check whether the internal locking block of the first object E (such as a railway switch) in the switch area has completed the lock" and "install the second object E (such as a switch lock)". The above important steps and the continuity on the timeline can be the answers, and the expert end A can complete the design of the questions and answers according to procedures [1] to [4].

[5] The student C uses the augmented reality mobile device 30 (such as wearing AR glasses) to take the computer test, and opens the test application 32 of the augmented reality mobile device 30, and then aligns or corresponds the camera (lens) of the augmented reality mobile device 30 to the test topic "first target object E (such as a railway switch)" so that the object recognition module 34 of the augmented reality mobile device 30 can identify the target object E related to the test question. Then, according to the design content of the script described in the above procedure [2], the test application 32 starts to load (download) two test questions after successfully identifying the target E. After the two test questions are loaded, the test application 32 starts to display the prompt content of the two test questions designed in the above procedure [3] to the augmented reality mobile device 30 for the student terminal C to watch.

For example, the student terminal C starts to perform the power-off operation of the first target object E (such as a railway switch) according to the prompt content (instruction) of question 1, and then continues to perform the operation of installing the second target object E (such as a switch lock) in question 2. During the test process (such as the operation process), in order to increase the accuracy of the image recognition and comparison of the answer recognition module 22 of the servo device 20 of the servo terminal B, the test application 32 of the augmented reality mobile device 30 can fix the operation perspective of the student terminal C for the first target object E or the second target object E through prompts or guidance.

[6] When the test questions begin to appear, the test application 32 of the augmented reality mobile device 30 (such as AR glasses) can transmit the real-time picture (such as the live operation picture) obtained by the camera (lens) of the augmented reality mobile device 30 to the server device 20 of the server end B. Moreover, when the server device 20 receives the real-time picture (such as the live operation picture), the answer recognition module 22 of the server device 20 can immediately compare the changes of the audio and video characteristics of the real-time picture (such as the live operation picture) and the answer designed in the above program [4] on the time axis.

For example, in the embodiment of the comparison of the changes of audio-visual features on the time axis shown in FIG6, the augmented reality mobile device 30 obtains the on-site real-time picture (such as the on-site operation picture) of the learner end C during the test process (such as the operation process) and transmits it to the server device 20 of the server end B, and then the answer recognition module 22 of the server device 20 compares the changes of the audio-visual features on the time axis between the on-site real-time picture (such as the on-site operation picture) of the learner end C during the test process (such as the operation process) and the answer of the video content of the augmented reality education training material D (such as the operation method is from "MDI" to "HANDLE").

After the test 1 is completed, the answer recognition module 22 of the server device 20 can calculate the test result, score or correctness of the test 1. If there are subsequent test questions (such as test 2...) that need to be answered by the student terminal C, the above process can be repeated [6] until all test questions are answered.

For example, when the student terminal C starts to perform the power-off operation of the first target object E (such as a railway switch) of the test question 1, the augmented reality mobile device 30 can transmit the on-site real-time screen of the student terminal C (such as the on-site operation screen) to the server device 20, so that the answer recognition module 22 of the server device 20 can confirm whether the operation steps of the student terminal C include the three important step screens of "opening the first target object E (such as a railway switch)", "performing the power-off operation of the first target object E (such as a railway switch)" and "confirming the power-off signal of the switch point area", and then the answer recognition module 22 of the server device 20 calculates the test result or score of the test question 1. The answer recognition module 22 can evaluate the correctness or incorrectness of the answer to the test question to calculate the score of the test question, and can also further evaluate the accuracy or inaccuracy of the test action (such as operation action) of the student terminal C to calculate the advanced score or final score of the test question.

[7] If the student C encounters problems during the test (such as the operation process), such as unclear test questions or other questions, the augmented reality mobile device 30 of the student C can be connected to the electronic device 10 of the expert A (such as the remote expert) at any time through the remote video collaboration function of the remote collaboration module 33, and the augmented reality mobile device 30 of the student C can be turned on or connected to the electronic device 10 through the remote monitoring function of the remote collaboration module 33 to seek real-time assistance from the expert A.

If the test process of the student end C for the test question is correct (such as correct operation), but the answer recognition module 22 of the server device 20 misjudges the recognition and comparison result of the test question as an error, then the expert end A (such as a remote expert) can also help correct the relevant errors through the electronic device 10, or the expert end A can also synchronously correct the design of the test questions or answers to make future tests smoother.

For example, during the test of question 2, if the student terminal C finds that the state of the first object E (such as a railway switch) on site has an exception and cannot continue to install the second object E (such as a switch lock), the student terminal C can communicate in real time through the augmented reality mobile device 30 or contact the electronic device 10 of the expert terminal A, so that the expert terminal A can help eliminate this exception, so that the student terminal C can perform the test or operation of the subsequent test questions after this exception is eliminated. Alternatively, the expert terminal A may also request to participate in the entire test, and at the beginning of the test, the augmented reality mobile device 30 of the student terminal C calls the electronic device 10 of the expert terminal A through the remote collaboration module 33 to enter the monitoring mode, so that the expert terminal A can monitor the test process and test results of the student terminal C remotely through the electronic device 10 in real time, and it is also helpful for the expert terminal A to review whether the test questions originally designed are suitable for the test implementation of the student terminal C.

[8] After all the test questions are completed, the test ends. The augmented reality mobile device 30 of the student end C can immediately receive the test results or scores (such as test scores) counted by the answer recognition module 22 of the server device 20, and the augmented reality mobile device 30 of the student end C can have a user interface, so that the student end C can obtain the wrong answers or correct answers of the test questions from the server device 20 through the user interface of the augmented reality mobile device 30, which is beneficial for the student end C to review the wrong answers and convenient for the student end C to review or refer to the correct answers (such as the correct operation screen). The expert end A can also allow the student end C to retest the test questions (retest) to help strengthen the student end C's impression of the test questions and learning effect.

In addition, the present invention also provides a computer-readable medium for an educational training test method based on augmented reality technology, which is applied to a computing device or computer having a processor and/or memory, and the computer-readable medium stores instructions, and the computing device or computer can execute the computer-readable medium through the processor and/or memory to execute the above content when executing the computer-readable medium. For example, the processor can be a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), etc., and the memory can be a random access memory (RAM), a memory card, a hard disk (such as a cloud/network hard disk), a database, etc., but is not limited to this.

In summary, the educational training test system, method and computer-readable medium based on augmented reality technology described in the present invention have at least the following characteristics, advantages or technical effects.

1. The present invention enables experts to use electronic devices to design relevant test questions and answers based on augmented reality education training materials, so as to automate the entire testing process, improve learning (testing) efficiency and save time.

2. The present invention enables students to take test questions through augmented reality methods of augmented reality mobile devices after receiving education and training on augmented reality education training materials, so as to evaluate the learning effect of students.

3. The present invention allows experts to use electronic devices to simultaneously design the content, test questions and answers of augmented reality education training materials, and also allows students to use augmented reality mobile devices to conduct automated tests to verify learning results.

4. The present invention can deepen the students' impression and learning effect of augmented reality education training materials, and can also serve as a reference for learning diagnosis or reduce training costs. It can also provide corresponding feedback to experts who can design augmented reality education training materials.

5. The expert end of the present invention can realize the needs of remote monitoring or real-time assistance for the student end by expanding the remote collaboration function of the real-world mobile device.

6. The present invention can detect and provide feedback on the test status (such as operation status) of the learner side to the expert side (such as the editor side), so as to serve as a reference for the expert side to adjust the content of teaching materials and test questions through electronic devices in the future.

7. The present invention enables students to have a paragraph-based learning effect through the test of augmented reality mobile devices, and also enables students to have a chapter-based impression of the key points of the overall situation, and can also improve students' learning ability and memory.

The above implementation forms are only illustrative of the principles, features and effects of the present invention, and are not intended to limit the scope of implementation of the present invention. Anyone familiar with this technology can modify and change the above implementation forms without violating the spirit and scope of the present invention. Any equivalent changes and modifications completed using the contents disclosed in the present invention should still be covered by the scope of the patent application. Therefore, the scope of protection of the present invention should be as listed in the scope of the patent application.

1: Educational training and testing system based on augmented reality technology

10: Electronic devices

11: First communication module

12: Browser

13: Augmented reality content design tool subsystem

131: User Interface

14: Textbook editing module

141: Exam question definition unit

20: Servo device

21: Second communication module

22: Answer recognition module

23: Storage module

30: Augmented reality mobile devices

31: Third communication module

32:Test application

33: Remote collaboration module

34: Object recognition module

35: Voice recognition module

A: Expert side

B: Server side

C: Student side

D: Expand real-life education training materials

E: Subject matter

N: Network

Claims (15)

An education training test system based on augmented reality technology includes: an expert-side electronic device having an augmented reality content design tool subsystem for loading an augmented reality education training material, and then the expert-side uses the augmented reality content design tool subsystem to design at least one test question, a script for the test question, and an answer to the content of the test question according to the augmented reality education training material; a server device for storing the test question, the script, and the answer designed by the expert-side through the augmented reality content design tool subsystem according to the augmented reality education training material; and at least The student-side augmented reality mobile device with a remote monitoring function loads the test questions and the script designed by the expert side through the augmented reality content design tool subsystem according to the augmented reality education training materials from the server device, wherein when the student-side augmented reality mobile device with the remote monitoring function recognizes the subject matter associated with the test questions designed by the augmented reality content design tool subsystem according to the augmented reality education training materials, the student-side augmented reality mobile device with the remote monitoring function presents the subject matter associated with the subject matter. The augmented reality mobile device with the remote monitoring function records the on-site real-time picture of the process of the student end testing the object with the augmented reality content design tool subsystem according to the augmented reality education training materials, and the augmented reality mobile device with the remote monitoring function of the student end transmits the on-site real-time picture of the process of the student end testing the object with the augmented reality content design tool subsystem according to the augmented reality education training materials to the server device, and then the augmented reality content design tool subsystem transmits the on-site real-time picture of the process of the student end testing the object with the augmented reality content design tool subsystem according to the augmented reality education training materials to the server device, and then the server device transmits the augmented reality content design tool subsystem according to the augmented reality education training materials to the server device. The live real-time picture of the test process of the test question designed according to the augmented reality education training material and the answer designed by the expert for the content of the test question are compared with each other on the time axis, so that the augmented reality content design tool subsystem can calculate the test result or score of the student for the test question based on the comparison result of the audio-visual characteristics of the live real-time picture of the test process of the test question designed according to the augmented reality education training material and the answer designed by the expert for the content of the test question on the time axis. The educational training test system as described in claim 1, wherein the augmented reality content design tool subsystem of the electronic device further edits the position layout of multiple augmented reality content elements, and the augmented reality application of the augmented reality mobile device is embedded with a web browser component, so that the web browser component loads the edited results of the position layout of the multiple augmented reality content elements to the augmented reality mobile device. The education training test system as described in claim 1, wherein the electronic device at the expert end further has a teaching material editing module, which loads the file of the augmented reality education training teaching material transmitted by the server device, so that the teaching material editing module designs the test questions, the script and the answer for the augmented reality education training teaching material, and the teaching material editing module designs the script including the trigger point and prompt content of the test question. The education training test system as described in claim 1, wherein the electronic device of the expert end further includes a teaching material editing module having a test question definition unit, so that the expert end defines the important step screens of the teaching material video of the augmented reality education training teaching material and their relationship on the time axis through the test question definition unit of the teaching material editing module, and then the test question definition unit of the teaching material editing module stores the important step screens of the teaching material video of the augmented reality education training teaching material and their relationship on the time axis into a file including the test question, the script and the answer. The education and training test system as described in claim 1, wherein the server device has an answer recognition module, which compares the changes of the audio-visual characteristics of the live real-time picture received from the student end and the answer pre-defined in the augmented reality education training teaching material on the expert end on the time axis, so that the answer recognition module can calculate the answer status and test results of the student end in real time based on the comparison result of the audio-visual characteristics of the live real-time picture and the answer on the time axis. The educational training test system as described in claim 1, wherein the augmented reality mobile device at the student end has a test application and an object recognition module, so that the test application loads the file of the augmented reality educational training teaching material including the script of the test question from the server device, and then the object recognition module recognizes the target object on the scene according to the design of the script, so that the test application requests the server device to load the test question associated with the target object. The educational training and testing system as described in claim 1, wherein the augmented reality mobile device on the student side includes a remote collaboration module having a remote video collaboration function and a remote monitoring function, so that the augmented reality mobile device on the student side can establish a remote video connection with the electronic device on the expert side through the remote video collaboration function of the remote collaboration module, and the augmented reality mobile device on the student side can connect to the electronic device through the remote monitoring function of the remote collaboration module to seek real-time assistance from the expert side. An educational training test method based on augmented reality technology includes: loading an augmented reality education training material into an augmented reality content design tool subsystem of an electronic device at an expert end, and then designing at least one test question, a script of the test question, and an answer for the content of the test question through the augmented reality content design tool subsystem according to the augmented reality education training material, so as to store the test question, the script, and the answer designed by the augmented reality content design tool subsystem according to the augmented reality education training material to a server device; At least one learner end having The augmented reality mobile device with remote monitoring function loads the test questions and the script designed by the expert end through the augmented reality content design tool subsystem according to the augmented reality education training materials from the server device; and when the augmented reality mobile device with remote monitoring function on the student end recognizes the subject matter associated with the test questions designed by the augmented reality content design tool subsystem according to the augmented reality education training materials, the augmented reality mobile device with remote monitoring function on the student end presents the test questions associated with the subject matter, and the augmented reality mobile device with remote monitoring function on the student end displays ... The augmented reality mobile device with the remote monitoring function records the on-site real-time picture of the process of the student end testing the object with the augmented reality content design tool subsystem according to the augmented reality education training materials, and the augmented reality mobile device with the remote monitoring function of the student end transmits the on-site real-time picture of the process of the test with the augmented reality content design tool subsystem according to the augmented reality education training materials to the server device, and then the augmented reality content design tool subsystem transmits the on-site real-time picture of the process of the test with the augmented reality content design tool subsystem according to the augmented reality education training materials to the server device. The live real-time picture of the test process of the test question designed by the augmented reality education training material and the answer designed by the expert for the content of the test question are compared with each other's audio-visual characteristics on the time axis, so that the augmented reality content design tool subsystem calculates the test result or score of the student for the test question based on the comparison result of the audio-visual characteristics of the live real-time picture of the test process of the test question designed by the augmented reality education training material and the answer designed by the expert for the content of the test question on the time axis. The educational training test method as described in claim 8 further includes editing the position layout of multiple augmented reality content elements by the augmented reality content design tool subsystem of the electronic device, and embedding the augmented reality application of the augmented reality mobile device into a web browser component, so that the web browser component loads the edited result of the position layout of the multiple augmented reality content elements into the augmented reality mobile device. The educational training test method as described in claim 8 further includes the material editing module of the electronic device at the expert end loading the file of the augmented reality education training material transmitted by the server device, so that the material editing module designs the test questions, the script and the answer for the augmented reality education training material, and the design of the script by the material editing module includes the trigger point and prompt content of the test question. The educational training test method as described in claim 8 further includes the expert end defining the important step screens of the teaching material video of the augmented reality education training teaching material and their relationship on the timeline through the teaching material editing module of the electronic device, so that the teaching material editing module stores the important step screens of the teaching material video of the augmented reality education training teaching material and their relationship on the timeline into a file including the test question, the script and the answer. The education and training test method as described in claim 8 further includes the answer recognition module of the server device comparing the changes of the audio-visual characteristics of the live real-time picture received from the student end and the answer pre-defined by the expert end in the augmented reality education training teaching material on the time axis, so that the answer recognition module can calculate the answer status and test results of the student end in real time based on the comparison result of the audio-visual characteristics of the live real-time picture and the answer on the time axis. The educational training test method as described in claim 8 further includes the test application of the augmented reality mobile device on the student end loading the file of the augmented reality educational training teaching material including the script of the test question from the server device, and then the object recognition module of the augmented reality mobile device on the student end recognizes the target object on the scene according to the design of the script, so that the test application requests the server device to load the test question associated with the target object. The educational training and testing method as described in claim 8 further includes the augmented reality mobile device on the student side establishing a remote video connection with the electronic device on the expert side through the remote video collaboration function of the remote collaboration module, and the augmented reality mobile device on the student side connecting to the electronic device through the remote monitoring function of the remote collaboration module to seek real-time assistance from the expert side. A computer-readable medium, used in a computing device or a computer, stores instructions for executing an educational training test method based on augmented reality technology as described in any one of claims 8 to 14.

Images