TWM672432U - Artificial intelligence assisted augmented reality system - Google Patents

Abstract

An artificial intelligence (AI) assisted augmented reality (AR) system includes an editing platform, a computer system, and an AR device. The editing platform is configured to edit expert knowledge data to be a knowledge database. The computer system is configured to generate an AI agent module according to the knowledge database and generate a solution and an application procedure from analyzing the query data by the AI agent module. The AR device is configured to obtain the application procedure from the computer system and execute the application procedure.

Description

Artificial Intelligence Assisted Augmented Reality System

The present disclosure relates to artificial intelligence and augmented reality, and more particularly to an artificial intelligence-assisted augmented reality system.

General workplace operations rely on the professional capabilities of employees to cope with complex work content. The technical training of employees includes basic training, expertise training, and on-the-job training (OJT). Generally speaking, after a period of work, people who have passed the above training should be able to control human errors to a certain proportion, but in reality, human errors often occur due to ineffective training. However, under the influence of the global declining birthrate, professional knowledge is often not successfully passed on after professional talents retire; and new employees often suffer from excessive psychological burdens due to lack of solid training, resulting in human errors often occurring in actual workplaces within companies due to human nature and lack of professionalism, making it impossible to ensure work quality and verify the correct execution of results.

The present disclosure provides an artificial intelligence (AI)-assisted augmented reality (AR) system, which includes an editing platform, a computer system, and an augmented reality device. The editing platform is used to edit expert knowledge data into a knowledge base. The computer system is used to generate an artificial intelligence agent (AI agent) module based on the knowledge base, and the artificial intelligence agent module parses query data to generate solutions and applications. The augmented reality device is used to obtain applications from the computer system and execute the applications.

In some embodiments, the AI-assisted augmented reality system further includes an Internet of Things device, which is used to provide Internet of Things data to the computer system, so that the AI agent module determines a question corresponding to the query data based on the Internet of Things data.

In other embodiments, the editing platform is further used to edit IoT data and expert knowledge data into a knowledge base.

In other embodiments, the editing platform includes a design builder, an instrument builder, a knowledge converter, a knowledge editor, a question-and-answer experience editor, and a knowledge collector.

In other embodiments, the editing platform includes a knowledge collector, a knowledge editor, and a knowledge converter.

In other embodiments, the artificial intelligence agent module includes a multimodal problem identification, definition and solution system, a knowledge application ecosystem and a human-computer collaborative scheduling system.

In other embodiments, the query data is provided by an electronic device, and the computer system is further configured to transmit the solution to the electronic device.

In other embodiments, the electronic device that provides the query data is a computer or smart glasses.

In other embodiments, the computer system is further configured to upload the result of the augmented reality device executing the application to the management platform.

In other embodiments, the augmented reality device is smart glasses, a robot, or a drone.

The following is a detailed discussion of the embodiments of the present disclosure. However, it is understood that the embodiments provide many applicable concepts that can be implemented in a variety of specific contexts. The embodiments discussed and disclosed are for illustration only and are not intended to limit the scope of the present disclosure.

FIG1 is a schematic diagram of an AI-assisted augmented reality system 100 according to some embodiments of the present disclosure. As shown in FIG1 , the AI-assisted augmented reality system 100 includes an electronic device 110, an Internet of Things device 120, an editing platform 130, a computer system 140, an augmented reality device 150, a management platform 160, and a robot/edge computing device 170.

The electronic device 110 is used to provide query data to the computer system 140. The query data can be generated in data formats such as text, error code, voice, image, video, etc. The electronic device 110 can be a computer (such as a personal computer, laptop, tablet computer, etc.), smart glasses or other suitable devices. The user can generate query data through the electronic device, for example, in the 5W1H method, that is, using question words such as who, when, what, where, why and/or how to generate query data in text or voice format, or provide query data with question categories in multimedia formats such as images and/or videos, but not limited to this.

The IoT device 120 is used to provide IoT data to the computer system 140. The IoT data may be, for example, sensing data obtained by the IoT device 120 in a specific field, mechanical parameter data, etc., but is not limited thereto. In some embodiments, the IoT device 120 is an artificial intelligence of things (AIoT) executor, which has functions such as real-time processing and data analysis. In addition, in each embodiment, the AI-assisted augmented reality system 100 may include multiple IoT devices 120.

The editing platform 130 is used to edit expert knowledge data into a knowledge base. In some embodiments, the editing platform 130 is a no-code editing platform, allowing users to edit the knowledge base through a graphical user interface and configuration. Specifically, users can edit expert knowledge bases, applications and artificial intelligence object detection models, object anchoring information, operation processes, remote monitoring equipment information input, and/or abnormal detection details in the application through a no-code development environment, and output the application to, for example, an augmented reality device, a robot, a drone, or an Internet of Things or an Artificial Intelligence of Things (AIoT) device, and apply it in different vertical fields. In addition, the editing platform 130 can be a corporate AI expert system with AR (CAESAR) Edit Platform (CEP) system editing platform, which combines the functions of an expert-level artificial intelligence system so that users do not need to have program development capabilities, but only need to focus on building knowledge in their professional fields. They can edit professional knowledge data into a dedicated knowledge base through the editing platform 130.

In some embodiments, the editing platform 130 includes a knowledge collector, a knowledge editor, and a knowledge converter, wherein the knowledge collector is used to collect data, the knowledge editor is used to edit the collected data into a knowledge base based on specific knowledge, and the knowledge converter is used to convert the data in the knowledge base into machine-readable data. In some embodiments, in addition to the knowledge collector, the knowledge editor, and the knowledge converter, the editing platform 130 also includes a design builder, an instrument builder, and a question-and-answer experience editor. The editing platform 130 may include one or more servers and databases, wherein the database stores the knowledge base, and each component of the editing platform 130 (knowledge collector, knowledge editor, and knowledge converter) may include one or more programs, libraries, or cloud services executed by the server.

The computer system 140 is coupled to the electronic device 110, the IoT device 120, and the editing platform 130, and is used to generate an artificial intelligence agent module 142 based on the knowledge base provided by the editing platform 130, and execute the artificial intelligence agent module 142, so that the artificial intelligence agent module 142 analyzes the query data to determine the corresponding problem, and generates a corresponding solution (such as a standard operating procedure) and an application 144. In some embodiments, the computer system 140 provides a multimodal input interface (including text, voice, and/or IoT sensing data, etc.) to accurately identify problems in a specific field and provide corresponding solutions, and it has knowledge update and optimization functions to ensure the continuous evolution of the knowledge base. The generated solution may be transmitted back to the electronic device 110 by the computer system 140 .

The computer system 140 may also receive the IoT data provided by the IoT device 120, so that the artificial intelligence agent module 142 defines the problem corresponding to the query data according to the IoT data. The generated artificial intelligence agent module 142 may include a multimodal problem identification, definition and solution system, a knowledge application ecosystem and/or a human-computer collaborative scheduling system, wherein the multimodal problem identification, definition and solution system provides functions such as problem classification, problem definition and solution generation, the knowledge application ecosystem provides functions such as knowledge base application, knowledge base management and knowledge base update, and the human-computer collaborative scheduling system provides functions such as task adjustment, resource allocation and real-time monitoring. The computer system 140 may include one or more servers and a database, wherein the database stores the artificial intelligence agent module 142 and the application 144, and the artificial intelligence agent module 142 may include one or more programs, libraries, or cloud services executed by the server.

The augmented reality device 150 is coupled to the computer system 140, and is used to obtain the application 144 from the computer system 140 and execute the application 144 to generate a virtual screen including objects such as text and/or icons. A person holding the augmented reality device 150 can solve problems in a specific scene according to the instructions of the virtual screen. The augmented reality device 150 can verify the execution results of the application 144 through artificial intelligence detection. The augmented reality device 150 can be implemented as, for example, smart glasses, a smart phone, a tablet computer or a monitoring system, which at least includes a screen, a processor, an inertial measurement unit, a depth sensor or a combination thereof, but is not limited thereto.

For example, when the augmented reality device 150 is a smart glasses, the smart glasses include transparent lenses that allow the user to see the real scene and display the virtual image on the transparent lenses, so that a virtual and real superimposed visual effect can be achieved. When the augmented reality device 150 is a smart phone or a tablet computer, the augmented reality device 150 also includes an image sensor for capturing the real scene, a processor (such as an image processor) for superimposing the real scene and the virtual image, and a screen for displaying the superimposed image.

The augmented reality device 150 can also calculate its position in the real space through the inertial sensing unit, and this position can also be corrected through image recognition, such as capturing a point cloud in the real space and matching this point cloud with a point cloud in the database to correct the position. When the augmented reality device 150 executes the application 144, it can provide functions such as navigation, voice prompts, text prompts, image prompts, abnormal condition recognition, object detection, etc.

The augmented reality device 150 can also be implemented as a robot or a drone, and the computer system 140 can decide to dispatch personnel, robots and/or drones to specific locations to perform tasks to solve problems, so as to achieve an optimized hybrid operation mode. In particular, in dangerous or difficult-to-reach locations, the computer system 140 can decide to dispatch robots or drones to perform tasks to solve problems and verify the results to ensure operational safety.

The management platform 160 is coupled to the computer system 140, which can receive the results of the augmented reality device 150 executing the application 144, so that the augmented reality device 150 located in a specific field can automatically store and record data during the inspection operation, and the management platform 160 can perform quality confirmation and generate a highly reliable digital inspection report based on it. The manager can use the management platform 160 to instantly grasp the real-time operation status in a specific field, and perform performance analysis and trend prediction. The generated digital inspection report also provides key performance indicators of various execution units. In terms of abnormal situation handling, the management platform 160 can quickly evaluate and recommend whether to dispatch robots or drones to specific locations for handling, so as to assist managers in making the most appropriate decisions. The management platform 160 can also continuously optimize the human-machine collaboration mode based on historical data, thereby improving overall operational efficiency.

The robot/edge computing device 170 is coupled to the computer system 140, and is used to obtain the application 144 from the computer system 140, execute the application 144 to perform calculations, and solve problems in a specific field according to the calculation results. The robot/edge computing device 170 may include one or more servers with calculation functions, and may perform corresponding actions according to the calculation results.

FIG2 is a flow chart of an AI-assisted augmented reality method 200 according to some embodiments of the present disclosure. The AI-assisted augmented reality method 200 can be performed by the AI-assisted augmented reality system 100 of FIG1 or other systems with similar architectures. In step S202, the system receives a task assignment, such as an inspection of a specific field. In step S204, the system generates an AI agent module based on a knowledge base and executes the AI agent module. In some embodiments, the system has functions such as knowledge collection, knowledge editing, and knowledge conversion, and can edit the collected data into a knowledge base and convert the data in the knowledge base into machine-readable data. In step S206, the system determines the problem corresponding to the query data. The system can receive IoT data, so that the artificial intelligence agent module determines the problem corresponding to the query data based on the IoT data. In step S208, the system generates corresponding solutions and applications based on the determined problems. The system can classify and define problems and generate corresponding solutions, and can apply, manage and update the knowledge base, and has the functions of task adjustment, resource allocation and real-time monitoring. In step S210, the system provides the application to the augmented reality device, so that when the augmented reality device executes the application, a virtual screen including objects such as text and/or icons can be generated, and/or the application is provided to the robot/edge computing device, so that the robot/edge computing device executes the application to perform calculations and solves problems in a specific scene according to the calculation results. In step S212, the system detects the execution results of the application and uploads a report.

The aforementioned AI-assisted augmented reality method 200 may be programmed into computer program instructions, which may be executed by a processor and stored in a non-transitory computer-readable medium accessible by the processor. The non-transitory computer-readable medium may be a read-only memory, a flash memory, a floppy disk, a hard disk, an optical disk, a universal serial bus (USB) flash drive, a tape, a database accessible on the Internet, or other computer-readable media that are obvious to a person skilled in the art.

According to the AI-assisted augmented reality system and method disclosed herein, users can query the problems they face in a predefined work environment through devices such as computers and smart glasses, and communicate with the system in text or voice. At the same time, the problem can be automatically identified through the sensing function of smart glasses, and information can be collected from various IoT devices to ensure the accuracy of the problem definition. In addition, the AI-assisted augmented reality system disclosed herein can establish a unified standardized communication protocol to ensure that the AI process can run smoothly on various terminal devices. Through artificial intelligence guidance and real-time artificial intelligence detection technology, we can provide real-time solutions to problems and standardize operation process design, thus realizing a one-stop development solution from knowledge introduction to practical application.

In the AI-assisted augmented reality system and method disclosed herein, the editing platform may have a code-free editing function, so that the editing platform can generate the expert knowledge content and solutions required by the AI agent module. Users can use the code-free editing environment to build an expert domain knowledge base and the system standard operating procedures and experience question and answer knowledge base corresponding to the expert domain. The code-free editing environment and standardized communication protocol provided by the editing platform can also ensure that the expert system and AI processes can run smoothly on various terminal devices. In addition, the editing platform can also integrate editing tools such as programless design builders and instrument builders to develop applications that integrate artificial intelligence and augmented reality functions, and output the applications to augmented reality devices for application in various fields. Users can troubleshoot fault events in the field based on the applications executed by the augmented reality devices to avoid errors caused by systematic human errors or lack of experience.

Although the present disclosure has been disclosed as above by way of embodiments, it is not intended to limit the present disclosure. Any person having ordinary knowledge in the relevant technical field may make some changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the definition of the attached patent application scope.

100: AI-assisted augmented reality system 110: electronic device 120: IoT device 130: editing platform 140: computer system 142: AI agent module 144: application program 150: augmented reality device 160: management platform 170: robot/edge computing device 200: AI-assisted augmented reality method S202, S204, S206, S208, S210, S212: steps

In order to have a more complete understanding of the embodiments and their advantages, reference is now made to the following description in conjunction with the accompanying drawings, wherein: FIG1 is a schematic diagram of the architecture of an artificial intelligence assisted augmented reality system according to some embodiments of the present disclosure; and FIG2 is a schematic diagram of the process of an artificial intelligence assisted augmented reality method according to some embodiments of the present disclosure.

100: Artificial intelligence-assisted augmented reality system

110: Electronic devices

120: Internet of Things devices

130:Editing platform

140: Computer system

142: Artificial Intelligence Agent Module

144: Applications

150: Augmented reality device

160: Management platform

170:Robots/edge computing devices

Claims (10)

An artificial intelligence (AI)-assisted augmented reality (AR) system includes: an editing platform for editing expert knowledge data into a knowledge base; a computer system coupled to the editing platform, the computer system is used to generate an artificial intelligence agent module based on the knowledge base, and the artificial intelligence agent module parses a query data to generate a solution and an application; and an augmented reality device coupled to the computer system, the augmented reality device is used to obtain the application from the computer system and execute the application. The artificial intelligence assisted augmented reality system as described in claim 1 further includes: an Internet of Things device coupled to the computer system, the Internet of Things device is used to provide an Internet of Things data to the computer system, so that the artificial intelligence agent module determines a question corresponding to the query data based on the Internet of Things data. An AI-assisted augmented reality system as described in claim 2, wherein the editing platform is further used to edit the IoT data and the expert knowledge data into the knowledge base. An artificial intelligence-assisted augmented reality system as described in claim 3, wherein the editing platform includes a design builder, an instrument builder, a knowledge collector, a knowledge editor, a question-and-answer experience editor, and a knowledge converter. An artificial intelligence-assisted augmented reality system as described in claim 1, wherein the editing platform includes a knowledge collector, a knowledge editor and a knowledge converter. An artificial intelligence-assisted augmented reality system as described in claim 1, wherein the artificial intelligence agent module includes a multimodal problem identification, definition and solution system, a knowledge application ecosystem and a human-machine collaborative scheduling system. An artificial intelligence assisted augmented reality system as described in claim 1, wherein the query data is provided by an electronic device, and the computer system is also used to transmit the solution to the electronic device. An artificial intelligence assisted augmented reality system as described in claim 7, wherein the electronic device providing the query data is a computer or a smart pair of glasses. An artificial intelligence assisted augmented reality system as described in claim 1, wherein the computer system is further used to upload a result of the augmented reality device executing the application to a management platform. An artificial intelligence assisted augmented reality system as described in claim 1, wherein the augmented reality device is a smart glasses, a robot or a drone.

Images