WO2022121911A1 - Virtual inspection system and visualized factory system in augmented reality environment - Google Patents

Abstract

A virtual inspection system and a visualized factory system in an augmented reality environment, at least comprising a server system (1), at least one interactive terminal and at least one control system (60). The server system (1) establishes communication connections with at least one interaction terminal and at least one control system (60), respectively, and the server system (1) at least comprises a two-dimensional module (10) for displaying a two-dimensional graphic component of a controlled object (70) and a three-dimensional module (20) for displaying a three-dimensional graphic component of the controlled object (70). A mapping relationship is established between the two-dimensional graphic component and the three-dimensional graphic component of the controlled object (70), and in response to an operation by an inspector on the two-dimensional view in the interactive terminal, the three-dimensional module (20) can locate the three-dimensional graphic component corresponding to a specified two-dimensional graphic component in the three-dimensional view.

Description

A virtual inspection system and a visual factory system in an augmented reality environment technical field

The invention relates to the technical field of virtual control, in particular to a virtual inspection system and a visual factory system in an augmented reality environment.

Background technique

Developed from technologies such as virtual manufacturing and digital prototyping, digital twins were originally proposed by the U.S. military in the study of aerospace systems, and have now been extended to many fields such as intelligent manufacturing, predicting equipment failures, and improving products. Digital twin is also used to refer to the completion of the digital model of a factory's workshop and production line before it is built. Therefore, the factory is simulated and simulated in the virtual cyberspace, and the real parameters are transmitted to the actual factory construction. After the workshop and production line are completed, the two continue to exchange information in daily operation and maintenance. However, the current virtual system of the digital twin factory only includes the equipment operation status of the virtual factory, and the inspection status of the inspectors is not specially incorporated into the digital twin virtual system, so that the inspectors can only view the system information during the inspection. The physical characteristics of the surface, and the real-time data of the equipment cannot be viewed at any time during the inspection.

For example, Chinese patent CN108628595A discloses a system and method for developing control applications for controllers used in automation systems. The controller receives sensor signals associated with the perception of the first real component during execution of the control application. Simulates the activity of a virtual part including interaction with a first real part, the virtual part being a digital twin of a second real part designed for the work environment and not present in the work environment. Virtual data is generated in response to the simulated activity of the virtual component. The control application module utilizes the sensor signals and virtual data for the first real component to determine parameters for developing the control application. During execution of the control application, an augmented reality display signal for the work environment is rendered and displayed based on the digital representation of the virtual data. However, the current defect of this invention is that when inspectors walk around the factory with wearable devices, the ordinary data request algorithm is unstable and cannot provide information accurately, thus making the system unstable.

The prior art also discloses a method for implementing inspection through an augmented reality device. Augmented reality is where virtual meets reality. AR can superimpose virtual images on real objects, providing all the necessary data for workers performing tasks. Instead of guessing what part of the equipment is faulty, staff can use augmented reality to bring that part into view to determine if it is faulty. In addition to making product production more efficient, wearables can also help workers avoid crises and keep them healthy and safe. For example, Chinese patent CN111176245A discloses a multi-terminal industrial equipment inspection and monitoring system and method based on digital twin technology, including a physical entity module, a visual interaction module, a data calculation module, and a data transmission and feedback drive module. The invention extracts all the status data in the production process of industrial equipment through PLC, and then uses the OPCUA interface to transmit the key data to the server, and uses cloud services to calculate and process the data, and cooperate with augmented reality equipment, PC and mobile terminal equipment , to realize the mapping of physical physical space to digital space, the feedback control of digital space to physical physical space, realize the visual inspection of the production status of industrial equipment, and solve the problems of poor data visualization and poor correlation between data in the production and manufacturing process. , which significantly improves the degree of data visualization and real-time feedback of data in the industrial manufacturing process, and achieves the purpose of early warning of faults and real-time visualization of equipment data. The invention obviously inspects equipment through the system, rather than viewing equipment data within a set range in a roaming manner in a three-dimensional scene.

In addition, on the one hand, there are differences in the understanding of those skilled in the art; on the other hand, because the applicant has studied a large number of documents and patents when making the present invention, but due to space limitations, all details and contents are not listed in detail, but this is by no means The present invention does not possess the features of the prior art, on the contrary, the present invention already possesses all the features of the prior art, and the applicant reserves the right to add relevant prior art to the background art.

SUMMARY OF THE INVENTION

A device or system for controlling and visualizing a controlled object using augmented reality in the prior art has the drawback that, first, in the existing device for controlling a visualized object through an augmented reality device, inspection personnel wearing the augmented reality device It can only determine its own position according to the positioning information, but cannot confirm its relative position in the factory; secondly, the inspection personnel wearing the augmented reality device cannot view the 2D view and 3D view of the controlled object at the same time, nor can they pass the The area of the 2D view is set to display the depth of field of the 3D view. In the prior art, when inspecting the controlled object in the factory, the inspector can only read the relevant information of the controlled object and its three-dimensional graphic components around him, and follow a certain route to the controlled object. The objects are inspected in sequence, and most of the accused objects will not have problems during the inspection. This inspection method will reduce the efficiency of the inspection, and cannot find the problematic accused objects in advance. How to improve the inspection efficiency of inspectors is an unsolved technical problem.

In view of the deficiencies of the prior art, the present invention provides a virtual inspection system, which at least includes a server system, at least one interactive terminal and at least one control system, and the server system establishes a communication connection with at least one interactive terminal and at least one control system respectively. , the server system includes at least a two-dimensional module for displaying a two-dimensional graphic component of the controlled object and a three-dimensional module for displaying a three-dimensional graphic component of the controlled object, and the two-dimensional graphic component of the controlled object is the same as the A mapping relationship is established between the three-dimensional graphic components, and in response to the operation of the inspector on the two-dimensional view on the interactive terminal, the three-dimensional module can locate the three-dimensional graphic corresponding to the specified two-dimensional graphic component in the three-dimensional view. part. Through the corresponding display of the two-dimensional view and the three-dimensional view of the interactive terminal, the present invention can simultaneously view the two-dimensional and three-dimensional views of the controlled object only by selecting the controlled object in one of the views.

Preferably, the attributes of the controlled object can be displayed on the interactive terminal in a two-dimensional and three-dimensional manner in parallel in response to the operation of the inspector, and the two-dimensional module and the three-dimensional module are simultaneously changed based on changes in the attributes of the controlled object. The 2D graphics part and the 3D graphics part of the controlled object. Compared with the augmented reality device in the prior art that can only display a two-dimensional view or a three-dimensional view, the parallel working mode of the two-dimensional module and the three-dimensional module of the present invention does not occupy the bandwidth of the control network, the workload is small, and the The data delay display phenomenon of the 3D model server and the 2D module makes the work efficiency of the graphics display significantly improved, so that the synchronous real-time display of the 2D graphics and the 3D graphics can be realized.

Preferably, in response to the depth of field area designated by the inspector in the two-dimensional view, the three-dimensional module sends data of the three-dimensional graphics of the device located in the depth-of-field area to the interactive terminal, so that the three-dimensional view clearly displays the data of the device located in the depth-of-field area. 3D graphics, the 3D view displays the 3D graphics of the device outside the depth of field area in a non-sharp way. This setting enables the inspectors to focus on inspection during the inspection process, without turning back due to missing the inspection target, and without causing psychological pressure due to not knowing the relative distance between themselves and the inspection target. The inspectors can know the location of the area in the factory where they are located through the two-dimensional view, and determine the information of the controlled objects around their location through the three-dimensional view.

Preferably, when the inspector's viewing range approaches or goes out of the edge of the depth-of-field area, the interactive terminal sends out reminder information to remind the inspector that the viewing range deviates from the inspection range of the designated depth-of-field area.

Preferably, the server system further includes a graphics server, which is used to store the attribute data, digital engineering data and graphic engineering data of the controlled object sent by the control system, and the graphics server stores the real-time data of the controlled object. The attribute data and the graphics data are fused to form a three-dimensional graphics component capable of three-dimensional display, and the attributes of the controlled object are displayed in a three-dimensional view. Such setting is beneficial to reduce the workload and processing volume of the 3D module, and is beneficial to the faster speed of the 3D module to display the 3D graphics components.

Preferably, in response to the interactive operation of the inspector on the interactive terminal, the two-dimensional view of the controlled object jumps to the corresponding three-dimensional view.

Preferably, the inspector can view the alarm device and its associated devices in detail on the three-dimensional view of the interactive terminal in a high-definition manner by delimiting the depth-of-field area containing the alarm device and associated devices on the two-dimensional view of the interactive terminal. 3D graphics and their attribute information.

The present invention also provides a virtual inspection method, the method includes at least: establishing a mapping relationship between a two-dimensional graphic component and a three-dimensional graphic component of a controlled object, and responding to the operation of the two-dimensional view by the inspector on the interactive terminal , the three-dimensional module can locate the three-dimensional graphic component corresponding to the specified two-dimensional graphic component in the three-dimensional view.

Preferably, the method further includes: in response to the operation of the controlled object by the inspector on the interactive terminal, the attributes of the controlled object can be displayed on the interactive terminal in a two-dimensional and three-dimensional manner in parallel, the two-dimensional module and the three-dimensional The module simultaneously changes the two-dimensional graphics components and the three-dimensional graphics components of the controlled object based on the property change of the controlled object.

Preferably, the method further includes: in response to the inspector in the depth-of-field area designated by the two-dimensional view, the interactive terminal receives and displays the three-dimensional graphics of the device located in the depth-of-field area sent by the three-dimensional module, wherein the three-dimensional view is in clear The 3D view displays the 3D graphics of the device located in the depth-of-field area, and the 3D view displays the 3D graphics of the device located outside the depth-of-field area in a non-sharp way.

Preferably, the method further includes: when the inspector's viewing range approaches or goes out of the edge of the depth-of-field area, the interactive terminal sends out reminder information to remind the inspector that the viewing range deviates from the inspection range of the designated depth-of-field area.

In the inspection method of the present invention, through the designation of the depth of field area, the three-dimensional module can reduce the amount of data transmitted to the viewing interactive terminal. The 3D module screens the high-definition data of the 3D graphics and attribute data based on the depth of field area sent by the 2D module and the device list in the depth of field area, and sends the data to the interactive terminal. That is, the 3D module can filter and deliver data in a targeted manner based on the depth of field area, thereby reducing the transmission of a large amount of unnecessary data.

The present invention also provides a visual factory system in an augmented reality environment, comprising at least an augmented reality device, a server system and several controlled objects, the server system at least includes a two-dimensional module and a three-dimensional module, the two-dimensional module is composed of The two-dimensional graphic components of the controlled object are displayed in a two-dimensional manner; the three-dimensional module displays the three-dimensional graphic components of the controlled object in a three-dimensional manner; the attribute data of the controlled object can respond to the operation instructions of the augmented reality device Displayed in parallel in the form of two-dimensional graphics and three-dimensional views, the two-dimensional module and/or the three-dimensional module can control the controlled object based on the operation instructions fed back by the augmented reality device.

Preferably, in response to an operation of delimiting a depth-of-field area of the two-dimensional view of the controlled object, the three-dimensional module transmits and displays the three-dimensional graphics component of the controlled object within the depth-of-field area to the augmented reality device.

Preferably, the 3D graphics component is formed by a graphics server that fuses graphics data and attribute data, and is sent to the 3D module in the form of a frame rate stream, and the 3D module renders the received 3D graphics component to the 3D module. way to display.

Preferably, in response to the operation information of the action feature of the augmented reality device, the three-dimensional module determines the controlled object to be viewed and/or controlled based on the inspection personnel's advancing direction and action feature.

In the visual factory system in the augmented reality environment of the present invention, the present invention takes advantage of the parallel display of the two-dimensional module and the three-dimensional module, so that the inspection personnel can combine the online inspection and the offline inspection, which not only saves physical strength, but also saves energy. It can truly inspect the controlled object. When the inspector needs to retrieve relevant information about the controlled object around him, the inspector can click on the device based on the relative position of the device in the two-dimensional view and the characteristics of the three-dimensional graphic component corresponding to the controlled object in the three-dimensional view. The controlled object and its related information are obtained without the need for the augmented reality device to determine the controlled object by collecting the information of the controlled object.

Description of drawings

1 is a schematic structural diagram of a virtual inspection system of the present invention;

Fig. 2 is another structural schematic diagram of the virtual inspection system of the present invention;

3 is a schematic diagram of one of the patrol inspections of roaming in a three-dimensional scene;

Fig. 4 is another kind of inspection schematic diagram of roaming in a three-dimensional scene;

Fig. 5 is the parallel display schematic diagram of the two-dimensional view and the three-dimensional view of a certain controlled object;

6 is a schematic diagram of one of the logical structures of the present invention;

Fig. 7 is another kind of logical structure schematic diagram of the present invention;

FIG. 8 is a two-dimensional schematic diagram of one of the controlled objects of the present invention.

List of reference signs

1: server system; 10: two-dimensional module; 20: three-dimensional module; 30: graphics server; 40: first interactive terminal; 50: second interactive terminal: 60: control system; 70: controlled object; 80: augmented reality device; 81: two-dimensional operator station; 82: three-dimensional operator station.

Detailed ways

The following detailed description is given in conjunction with the accompanying drawings.

The invention discloses a virtual inspection system and a visual factory system in an augmented reality environment.

The picture frequency stream in the present invention refers to a video stream that can form a picture. Video streaming refers to the transmission of video data that, for example, can be processed as a steady and continuous stream over a network.

In the virtual inspection system of the present invention, the inspectors can conduct virtual inspections of the digital factory through the real perspective of roaming in the three-dimensional scene. The virtual three-dimensional scene of the digital factory of the present invention is consistent with the real scene of the physical factory in the building structure, equipment arrangement, and the connection relationship of the equipment. That is, the three-dimensional graphics component of the three-dimensional scene display of the present invention can display the properties of the controlled object. When the attribute data of the controlled object changes, the attributes of the 3D graphics of the controlled object change synchronously, so that the inspectors can conduct virtual inspections by roaming in the 3D scene consistent with the real scene, check the settings Attribute data of the controlled object within the scope.

Both the two-dimensional module 10 and the three-dimensional module 20 may be one or more of an application-specific integrated chip, a CPU processor, a server, a server group, and a cloud server. The communication connection in the present invention may be a wired communication connection or a wireless communication connection. The control system 60 may be one or more of a dedicated integrated chip with a control function, a processor, a server, and a control module.

Example 1

As shown in FIG. 1 , the virtual inspection system of the present invention at least includes an interactive terminal, a server system and a control system 60 . The interactive terminal establishes a communication connection with the server system. The server system establishes a communication connection with the control system 60 . The control system 60 establishes a communication connection with a number of controlled objects 70, so as to monitor the attribute data and operation data of the controlled objects and control the controlled objects. The control system 60 includes at least a control processor and a control device, and can control the controlled object in response to the instruction of the interactive terminal. The controlled object of the present invention includes the building structure, engineering equipment, process system, electrical system, instrument control system and communication system of the factory.

The interactive terminal includes at least one interactive device. The interactive devices are, for example, operator stations, interactive mobile terminals and smart wearable devices. The interactive mobile terminal is, for example, a smart phone, a mobile computer, an ipad, and the like. In the case where the interactive terminal display screen is displayed alone, the interactive terminal can display two-dimensional graphics and/or three-dimensional graphics of the controlled object. In the case where the display screen of the interactive terminal can be displayed in a split screen, or there is more than one display screen, for example, the interactive terminal includes a first interactive terminal 40 and a second interactive terminal 50, and the first interactive terminal 40 displays the two-dimensional graphics of the controlled object , the second interactive terminal 50 displays three-dimensional graphics. Two-dimensional graphics and three-dimensional graphics are displayed in parallel.

Each component of the controlled object 70 is coded according to the predefined uniform coding rules. Preferably, the coding of each component is a unique interface for uniquely calling various components. By using the code as the calling object, the present invention does not need to perform relational retrieval, analysis, screening and other operations on the data when needed, so that the three-dimensional graphics of the controlled object can be called quickly and efficiently.

As shown in FIGS. 1 to 2 , the server system of the present invention includes at least a two-dimensional module 10 and a three-dimensional module 20 . The two-dimensional module 10 is used to display the controlled object in a two-dimensional manner, so that the inspector can control its state. In the interactive terminal, the operator can operate and display any two-dimensional graphic component of the controlled object. Among them, the two-dimensional module defines the number, function, purpose, set value, input/output data, alarm and related information of various engineering design graphics and data of the controlled object. By displaying the parameters and attributes of all variables in the control process, the two-dimensional module can notify the operator of unexpected engineering situations in the form of sound and light alarms. It can display real-time or historical trend graphs of engineering data through any combination or single-point mode, and can know the engineering Safety interlock protection status, etc. Preferably, the same controlled object in the two-dimensional module and the three-dimensional module adopts the same code, which facilitates the synchronous display and control of the same controlled object by the two-dimensional module and the three-dimensional module.

The three-dimensional module 20 is used to display the controlled object in a three-dimensional manner, so that the inspector can display the controlled object currently controlled or viewed by the inspector, as shown in FIG. 3 to FIG. 4 . Specifically, at the interactive terminal, the operator can operate and display any three-dimensional graphic component of the controlled object. Preferably, the three-dimensional graphics component of the present invention is used to display the properties of the controlled object. That is, the three-dimensional graphic component of the present invention can perform synchronous change based on the property change of the controlled object, so as to display the real-time property of the controlled object.

Preferably, the server system further includes a graphics server 30 . The graphics server is used to store the attribute data, digital engineering data and graphic engineering data of the controlled object sent by the control system. The digital engineering data in the prior art only contains pure digital data information and lacks graphic information, which leads to the need to adjust the graphics according to the engineering data when displaying the two-dimensional or three-dimensional graphics, resulting in a delay in the display or conversion of the two-dimensional or three-dimensional graphics. , even Caton. The graphic engineering data of the present invention includes graphic data, real-time attribute data and real-time personnel positioning data. The graphic data of the present invention includes various graphic data such as factory structure graphic, controlled object graphic, connection equipment graphic and so on. Specifically, the graphic data also includes flowcharts, equipment manufacturing drawings, factory building graphics, two-dimensional contour graphics of the controlled object, three-dimensional contour graphics of the controlled object, and personnel logo graphics. Factory building graphics include two-dimensional plane graphics and three-dimensional graphics that are consistent with the scale of the factory's physical building structure. The real-time personnel positioning data is the real-time position data of the personnel carrying the positioning device. Real-time engineering data, such as various instrumentation equipment, are used to collect and monitor changes in process parameters such as temperature, pressure, flow, and liquid level; valve opening or closing status; motor start or stop status; real-time physical properties of a medium data; the concentration of combustible gas monitored by the combustible gas detection probe; the concentration of toxic gas monitored by the toxic gas detection probe; the real-time video screen monitored by the on-site camera, etc. It also includes various types of actuators, such as control valves, switching valves, actuators, electrical control circuits, etc.

The graphics server fuses the real-time attribute data of the controlled object with the graphics data to form a three-dimensional graphics component capable of three-dimensional display, and displays the attributes of the controlled object in a three-dimensional view. Alternatively, the graphics server fuses the real-time attribute data of the controlled object with the graphic data to form a two-dimensional graphic component capable of two-dimensional display, and displays the attributes of the controlled object in a two-dimensional view. As shown in FIG. 3 to FIG. 5 , the inspection personnel can view the controlled object from various perspectives through the interactive terminal. Preferably, the three-dimensional graphics are lightweight processed on a graphics server or a third-party server, and the three-dimensional graphics data after the lightweight processing is fused with real-time attribute data. Such setting is beneficial to reduce the workload and processing volume of the 3D module, and is beneficial to the faster speed of the 3D module to display the 3D graphics components. Preferably, the three-dimensional module of the present invention displays the three-dimensional graphic components in the form of a video stream from a graphic server.

In response to the request of the second interactive terminal 50 for displaying the three-dimensional graphic component, the three-dimensional module displays the properties of the corresponding angle of the controlled object in a roaming manner. Preferably, the three-dimensional module displays the three-dimensional graphic components of the controlled object at a preset initial angle, and adjusts the display angle of the controlled object based on the operation of the inspector.

The attribute data of the present invention includes self-attribute data, working state attribute, control attribute, environment attribute, early warning attribute and the like. Self-attribute data is the data of the controlled object itself, such as model, size, location, replaceable reserve quantity, installation time, maintenance time and other information. The working state attribute is the working state data information of the controlled object, such as the real-time data of the temperature, pressure value, liquid level, outlet flow, and physical properties of the outlet medium in a controlled object. Control attributes are information such as control state, control operation execution time, control effect, etc., such as opening or closing of a valve, starting or stopping of a motor, and so on. Environmental attributes are data information related to the environment, such as the concentration monitored by the combustible gas detection probe; the toxic gas concentration monitored by the toxic gas detection probe; the real-time video screen monitored by the on-site camera, and so on. The early warning attribute is that the attribute data of the controlled object reaches the threshold value or operating state that needs to be alarmed.

Preferably, the two-dimensional graphic component of the controlled object is associated with the three-dimensional graphic component of the controlled object in a mapping manner, so that the inspector can locate the two-dimensional graphic component in the three-dimensional view. As shown in FIG. 8 , specifically, when the inspector clicks on a certain component on the display of the first interactive terminal 40, the two-dimensional module triggers the inspection personnel to respond to the click on the two-dimensional graphic component of the controlled object. The preset of the two-dimensional graphics component sends the display request information of the controlled object to the three-dimensional module. In response to the display request information, the three-dimensional module displays the three-dimensional graphic component corresponding to the two-dimensional graphic component of the controlled object according to a preset setting. For example, when the inspector clicks on a two-dimensional graphic component of a controlled object on the first interactive terminal 40, the display device of the second interactive terminal 50 displays the three-dimensional graphic component of the controlled object. Preferably, the three-dimensional module displays the three-dimensional graphic components of the controlled object on the second interactive terminal 50 in a rendering manner. Among them, the color of the clicked controlled object is displayed in a highlighted manner or a color-changing manner, so that the inspector can quickly determine the controlled object. Through the setting of mapping association, the present invention enables inspectors to conveniently locate two-dimensional graphics and three-dimensional graphics, so that inspectors can view two-dimensional and three-dimensional views of the controlled object at the same time.

Preferably, the first interactive terminal 40 and the second interactive terminal 50 display the controlled object in a two-dimensional and three-dimensional manner in parallel. When the attributes of the controlled object change, the two-dimensional module and the three-dimensional module simultaneously change the two dimensions of the controlled object. Properties of 3D graphics components and 3D graphics components. Compared with the serial working mode of the two-dimensional module and the three-dimensional module in the prior art, the parallel working mode of the two-dimensional module and the three-dimensional module of the present invention does not occupy the bandwidth of the control network, the workload is small, and the three-dimensional module and the two-dimensional module are reduced. The data delay display phenomenon of the module makes the work efficiency of graphic display significantly improved, so as to realize the synchronous real-time display of two-dimensional graphics and three-dimensional graphics.

Preferably, in response to the interactive operation of the inspector on the interactive terminal, the two-dimensional view of the controlled object can be jumped to the corresponding three-dimensional view. Preferably, the switching mode of the two-dimensional graphic component and the three-dimensional graphic component in the present invention at least includes: switching by clicking, switching by coding, and switching based on alarm information. The switching method of the two-dimensional graphic component and the three-dimensional graphic component of the present invention is not limited to the listed methods, and may also include other methods that can realize the switching of two-dimensional and three-dimensional graphics. Switch the coding information in the video or picture of the smart wearable device, etc. The click switching method is, for example, by clicking on a device or component on the two-dimensional screen, locating to the associated device displayed on the three-dimensional screen. The encoding switching method is, for example, directly inputting the encoding of the controlled object at the corresponding position on the two-dimensional screen to locate the device associated with the controlled object on the three-dimensional display screen. The switching method based on the alarm information is, for example, forming an alarm information list after the control system sends out the alarm information related to the controlled object. By clicking on the alarm list, locate the three-dimensional graphics component of the controlled object that has alarms.

Preferably, when the first interactive terminal 40 displays a two-dimensional view, the three-dimensional module can jump and display the three-dimensional graphics of the controlled object on the first interactive terminal in response to the switching operation of the inspector, and can also display the three-dimensional graphics of the controlled object on the second interactive terminal. The interactive terminal 50 jumps and displays the three-dimensional graphics of the controlled object. Based on the same principle, the present invention can realize jumping from the three-dimensional graphic component of the controlled object to the two-dimensional graphic component of the same controlled object. For example, when an inspector inspects a controlled object from the perspective of roaming inspection through the second interactive terminal 50, he or she clicks on the three-dimensional graphic component of the controlled object to view the properties of the corresponding component and control the corresponding component. When the inspector clicks to switch the two-dimensional graphics, the two-dimensional module displays the two-dimensional graphics component of the controlled object on the first interactive terminal 40 in response to the operation request of the inspector. Preferably, the first interactive terminal 40 can display the virtual position of the inspector in the two-dimensional view of the controlled object in response to the inspector's roaming perspective, so that the inspector can more conveniently view the roaming location. Compared with the situation in the prior art that requires staff to conduct inspections in the physical factory, the operator of the present invention can perform virtual inspections of the controlled object in the three-dimensional view at the interactive terminal, and during the inspection process. View the controlled object and real-time properties at any time and perform real-time control. For the controlled object that sends out the alarm information, the inspector of the present invention can quickly determine the abnormal controlled object and check the abnormality of attributes, and even control the abnormal controlled object through the interactive terminal. Therefore, the virtual inspection system of the present invention can save the inspection time, achieve the same working effect as the real inspection, and also help to improve the inspection efficiency.

In the prior art, the number of controlled objects in the factory is huge. Since there is no reference to a two-dimensional view, the inspectors search and view the controlled objects according to the connection relationship, and it is easy to have a blurred impression of the main object to be viewed, so that the There are more and more accused objects in inspections, which reduces the efficiency of inspections. The three-dimensional view of the present invention can display three-dimensional graphic components within a preset range including a designated controlled object, which is beneficial for inspectors to view the controlled object through the interactive terminal and conduct inspection on the controlled object. Wherein, the three-dimensional view of the present invention provides a three-dimensional view containing a controlled object and a device associated with the controlled object within the depth-of-field range according to the specified depth-of-field range request data in the two-dimensional scene. The depth of field range of the present invention refers to a display range in which a clear image can be formed on the display when a view with a limited distance is acquired. The depth of field area refers to the distance range of the three-dimensional graphic components that can clearly display the controlled object.

The area delineated on the 2D view can be a specified depth of field range. There are several controlled objects within the depth of field. When the inspector delimits the depth-of-field area, the three-dimensional view displays the three-dimensional graphics components of the controlled object according to the specified depth-of-field range. For example, the inspector delimits the depth of field area of the three-dimensional view with an unlimited geometric shape on the two-dimensional view displayed on the interactive terminal, and the two-dimensional module sends the depth of field information to the three-dimensional module. The three-dimensional module displays the depth-of-field area in three-dimensional graphics in response to the depth-of-field information. For example, the inspector draws a circle on the display screen, and the area inside the circle belongs to the depth of field area to be displayed.

When inspectors view the 3D graphics and their properties within the specified depth of field area, the picture is clear. When the operator looks at the edge of the depth of field area, the 3D graphics inside the edge is high-definition, and the definition of the 3D graphics outside the edge is reduced, which prompts the operator to view the content beyond the depth of field area, so that the operator can re-plan the viewing area. Depth of field area to avoid viewing without a clear target. In particular, when inspectors conduct real-time inspections in the factory area corresponding to the depth-of-field area, the information and images of the equipment obtained are clear. When the inspector approaches or walks out of the edge of the depth-of-field area, the interactive terminal sends out a reminder message to remind the inspector to deviate from the inspection of the designated depth-of-field area.

Preferably, for a device with abnormal properties, the two-dimensional graphic displays the two-dimensional graphic of the device that issued the alarm information and at least one other device associated with the abnormal problem of the device in a prominent color and/or logo. That is, on the two-dimensional view, the operator can clearly see the equipment that issues the alarm and the associated equipment related to the alarm. Inspectors can view the 3D graphics and attribute information of the alarm device and its associated devices in detail on the 3D view in a high-definition manner by demarcating the depth-of-field area containing the alarm device and associated devices on the 2D view. That is, by highlighting the alarm equipment and its associated equipment, the inspectors can quickly determine the equipment that needs to be focused on, avoiding the disadvantage of purposelessly checking the equipment to find out the cause of the alarm. When the inspectors inspect the alarm equipment in the factory area in real time, they can find the corresponding physical equipment based on the prominent image display in the two-dimensional view or the three-dimensional view. In particular, when the inspector uses the real-time position of the wearable device to locate the direction in the factory area, the 3D module prompts the relative position information of the device to be viewed and the inspector based on the direction of the inspector. For example, the prompt information is "XX device is on your left/front left/right/front right/rear", etc., which is convenient for inspectors to quickly lock the devices that need to be inspected.

If the delineation method of the depth-of-field area is not set, when the inspectors view the devices in the 3D view on the interactive terminal, the 3D module cannot determine the devices that the inspectors will view in the next step, and can only display all the devices in the current 3D view in high-definition. way to display. However, in practice, there are many 3D graphics and their properties of equipment, and the inspectors will not view them in detail. This causes the 3D module to send excessive unnecessary data to the interactive terminal, resulting in data delay of the interactive terminal and slow update of the 3D graphics, which affects the screen update speed of the display device. In the present invention, through the designation of the depth of field area, the three-dimensional module can reduce the amount of data transmitted to the viewing interactive terminal. The 3D module screens the high-definition data of the 3D graphics and attribute data based on the depth of field area sent by the 2D module and the device list in the depth of field area, and sends the data to the interactive terminal. That is, the 3D module can filter and deliver data in a targeted manner based on the depth of field area, thereby reducing the transmission of a large amount of unnecessary data.

Preferably, the depth of field area can be set not only in the two-dimensional view, but also in the three-dimensional view. When the inspector demarcates the depth of field area in the 3D view, the 3D module can search for data according to the depth of field area and give a list of devices in the depth of field area. Inspectors can view the information they need according to the form.

Example 2

This embodiment is a further improvement to Embodiment 1, and repeated content will not be repeated.

In this embodiment, the interactive terminal is preferably an augmented reality device. A device or system for controlling and visualizing a controlled object using augmented reality in the prior art has the drawback that, first, in the existing device for controlling a visualized object through an augmented reality device, inspection personnel wearing the augmented reality device It can only determine its own position according to the positioning information, but cannot confirm its relative position in the factory; secondly, the inspection personnel wearing the augmented reality device cannot view the 2D view and 3D view of the controlled object at the same time, nor can they pass the The area of the 2D view is set to display the depth of field of the 3D view. In the prior art, when inspecting the controlled object in the factory, the inspector can only read the relevant information of the controlled object and its three-dimensional graphic components around him, and follow a certain route to the controlled object. The objects are inspected in sequence, and most of the accused objects will not have problems during the inspection. This inspection method will reduce the efficiency of the inspection, and cannot find the problematic accused objects in advance. How to improve the inspection efficiency of inspectors is an unsolved technical problem.

Based on the deficiencies of the prior art, a system for real-time control and visualization of a digital factory in an augmented reality environment, as shown in FIG. The control system 60 establishes a communication connection with a number of controlled objects 70 to acquire attribute information and operating status of the controlled objects and to control the controlled objects. The server system 1 establishes a communication connection with the control system 60 and the augmented reality device 80 respectively for data transmission. The server system 1 includes at least a two-dimensional module 10 and a three-dimensional module 20 . The augmented reality device 80 establishes a communication connection with the two-dimensional module 10 and the three-dimensional module 20, respectively.

Preferably, the controlled objects are individually assigned in units of components. The two-dimensional module 10 in the present invention is used to display the controlled object in a two-dimensional manner, as shown in FIG. 8 , so that the inspector can control its state. Preferably, the inspection personnel can operate and display the two-dimensional graphics of any controlled object through the augmented reality device 80 .

The three-dimensional module 20 in the present invention is used to display the controlled object in a three-dimensional manner, as shown in FIGS. 4 and 5 , so that the inspector can display the controlled object currently controlled or viewed. Specifically, in at least one interaction module of the augmented reality device 80, the inspector can operate and display the three-dimensional graphics of any controlled object.

Preferably, the attributes of the controlled object can be displayed on the display module of the augmented reality device in parallel in two-dimensional and three-dimensional manners in response to the operation of the inspector. Preferably, after the attribute data of the controlled object is changed by the operator and/or the inspector, the relevant attribute parameters in the two-dimensional module and the three-dimensional module will be changed synchronously, so that the two-dimensional module and the three-dimensional module are always consistent with the attribute data real-time display of the controlled object. Compared with the augmented reality device in the prior art that can only display a two-dimensional view or a three-dimensional view, the parallel working mode of the two-dimensional module and the three-dimensional module of the present invention does not occupy the bandwidth of the control network, the workload is small, and the The data delay display phenomenon of the 3D model server and the 2D module makes the work efficiency of the graphics display significantly improved, so that the synchronous real-time display of the 2D graphics and the 3D graphics can be realized.

For example, when the inspector wears the augmented reality device 80 and walks in the factory area, the real-time data of different controlled objects is viewed. Among them, the two-dimensional view and the three-dimensional view of the factory are displayed in parallel, that is, displayed on the display module at the same time, and adaptive changes are made at the same time. For example, when the inspector views the controlled object and its attributes in the three-dimensional view, the two-dimensional view displayed by the display module can display the two-dimensional controlled object in the area where the inspector is located and the real-time action trajectory of the inspector in real time. Through the augmented reality display module with interactive functions, the inspectors can display settings for the two-dimensional view and the three-dimensional view according to their needs, such as selecting the display area, the controlled object, and the relevant information of the controlled object, etc. For example, the angular position of the two-dimensional view on the display screen of the three-dimensional view displays the two-dimensional controlled object and the track information of the inspector in parallel, or the two-dimensional view and the three-dimensional view are displayed in parallel images respectively.

Preferably, a mapping relationship is established between the two-dimensional graphic component of the controlled object and the three-dimensional graphic component of the controlled object, so that the inspector can locate the two-dimensional graphic component in the three-dimensional view, so that the inspector can select the two-dimensional graphic component in the two-dimensional view. The plant to be displayed in the 3D view. When the inspector clicks on a component on the display of the two-dimensional module, the secondary module responds to the click on the two-dimensional graphic component of the controlled object, according to the operator's preset on the two-dimensional graphic component, according to The preset is to display the three-dimensional graphic part corresponding to the two-dimensional graphic part of the controlled object. The present invention enables the operator to conveniently switch between two-dimensional graphics and three-dimensional graphics by mapping the associated settings. Preferably, the inspector determines the depth of field displayed in the three-dimensional view by clicking on the area of the two-dimensional view of the augmented reality device 80. This setting is helpful for the inspector to quickly determine the inspection area and the controlled object to be inspected, so that the The corresponding inspection method is used for inspection.

In the real factory, the factory covers a large area, and the inspectors are prone to fall into the state of aimless and confused inspection during the inspection process, that is, it is not clear which objects to be controlled need to be inspected and which areas should be prioritized. Inspection. Even if the inspection is carried out according to the factory preset inspection process, the inspection will reduce the inspection efficiency due to physical problems. If the factory has enough inspection personnel, the personnel cost will be increased accordingly, and the improved inspection efficiency will be limited. Only by increasing the number of inspection personnel cannot significantly improve the inspection efficiency.

Through the advantages of parallel display of two-dimensional modules and three-dimensional modules, the present invention enables inspection personnel to combine online inspection and offline inspection, which not only saves physical strength, but also can truly inspect the controlled object. In response to the depth-of-field area delineated based on the two-dimensional view, the three-dimensional module displays a three-dimensional graphical component of at least one plant within the depth-of-field area. For example, the inspector delimits the depth-of-field area of the three-dimensional view with an unlimited geometric shape on the displayed two-dimensional view, and the two-dimensional module sends the depth-of-field information to the three-dimensional module. The three-dimensional module displays the depth-of-field area in three-dimensional graphics in response to the depth-of-field information. The inspectors conduct online inspections of the controlled objects in the three-dimensional view in turn. Among them, the inspector can conduct a first investigation of the controlled object in the depth-of-field area, and determine at least one controlled object that needs to be viewed at a close distance.

The area delineated in the 2D view can be a specified depth of field range. There are several controlled objects within the depth of field. When the inspector delimits the depth of field area, the three-dimensional view displays the three-dimensional graphic components of the controlled object according to the specified depth of field range, and adjusts the clarity of the controlled object according to the operation of the inspector, so that the three-dimensional image of the controlled object is adjusted. The sharpness of the graphical components varies based on the needs of the operation. In this way, the inspector can inspect the accused object in a planned way.

Preferably, the three-dimensional view of the present invention requests data according to a depth of field range specified in a two-dimensional scene, and provides a three-dimensional view of a controlled object and a device associated with the controlled object within the depth of field range.

Preferably, the present invention initially displays the three-dimensional graphics of the controlled object at a preset angle. The three-dimensional module adjusts the display angle of the controlled object to a specified angle in response to the operator's click information or display angle request. Displaying in this way is beneficial for the operator to check the operation status of the controlled object at various angles and avoid the limitation of viewing angles. Preferably, the three-dimensional module 20 displays the behavior track of the task in the three-dimensional view based on the positioning information of the augmented reality device. Inspectors can view the relative positional relationship between the three-dimensional characters representing themselves and the controlled object in the three-dimensional view, so that they can quickly find the controlled objects around them that need to be inspected based on the relative positional relationship, which can quickly improve inspections. efficiency. When the 2D module and the 3D module are in parallel, the 2D view displays the position information of the inspectors synchronously in a two-dimensional manner, which is helpful for the inspectors to know the area they are inspecting. The corresponding three-dimensional view displayed by the area adjustment is more conducive to the inspection personnel to plan the scientific inspection route through the two-dimensional view.

Preferably, in the case that the inspector performs inspection on the controlled object of the specified type, the two-dimensional module displays all the positions of the controlled object of the specified type and displays at least one inspection route in response to the click operation of the inspector ; The three-dimensional module displays the inspection scene corresponding to the position information in response to the position information of the inspector, and gives reminder information when the inspector arrives within the designated range of the designated type of controlled object, so as to avoid the inspector missing or missing The controlled object to be inspected. This setting enables the inspectors to focus on inspection during the inspection process, without turning back due to missing the inspection target, and without causing psychological pressure due to not knowing the relative distance between themselves and the inspection target. The inspectors can know the location of the area in the factory where they are located through the two-dimensional view, and determine the information of the controlled objects around their location through the three-dimensional view.

In the prior art, the inspector determines the controlled object and retrieves the attributes and related information of the controlled object by using the image information, coding information, two-dimensional code information, and voice information of the controlled object collected by the augmented reality device 80 . In this way of the prior art, the inspector needs to contact the accused object first, and then check the accused object. In the case of inconvenient collection of the code and QR code of the accused object, it is difficult for the inspectors to determine the accused object and retrieve relevant information. Especially for bulky equipment, large pen-related graphics or even large printed coding information is not conducive to the collection of augmented reality devices, and small coded information needs to be collected by inspectors when they reach the vicinity of the coding plate under the huge equipment. This will undoubtedly increase the workload of the inspectors, cause unnecessary turns in the paths of the inspectors, and reduce the inspection efficiency.

In the present invention, both the two-dimensional view displayed by the two-dimensional module and the three-dimensional view displayed by the three-dimensional module can display real-time position information of the augmented reality device in real time. When the inspector needs to retrieve relevant information about the controlled object around him, the inspector can click on the device based on the relative position of the device in the two-dimensional view and the characteristics of the three-dimensional graphic component corresponding to the controlled object in the three-dimensional view. The controlled object and its related information are obtained without the need for the augmented reality device to determine the controlled object by collecting the information of the controlled object.

Preferably, the augmented reality device can determine the controlled object in the corresponding direction of its location based on the action characteristics of the inspector, and display relevant information of the controlled object based on the operation of the inspector. During the inspection process of the inspectors, the forward direction of the inspectors is determined, and the direction of the controlled objects around the inspectors' position relative to the inspectors is determined. The augmented reality device displays at least one controlled object within a certain range of the designated direction of the inspector based on the operation of tilting and/or shaking in the designated direction. For example, the augmented reality device displays the attributes and related information of the closest controlled object to the right of the inspector based on the operation of the inspector to tilt and/or shake the augmented reality device to the right. Similarly, when the inspector wears the augmented reality device to tilt and/or shake to the left, the attributes and related information of the closest controlled object to the left of the inspector are displayed. In this way, the inspector can determine the controlled object and check the running state of the controlled object without being close to the controlled object. Under normal circumstances, the running state of most of the controlled objects is normal, which can save 80% of the working time and walking path of the inspectors. When it is necessary to get close to the charged object, the inspectors will then approach the charged object for detailed equipment inspection. For example, if the inspector checks the maintenance information of the controlled object and finds that the equipment was repaired and replaced visible parts the day before, the inspector needs to approach the controlled object to check whether the repaired visible parts are consistent with the recorded information.

The action feature set by the augmented reality device of the present invention is not limited to the listed left or right tilt action, but can also be other action features, as long as the action feature can be recognized by the augmented reality device and can determine the controlled object.

Preferably, the two-dimensional module and the three-dimensional module of the present invention also store real-time data and historical data of the controlled object. Real-time data and historical data are sourced from the control system. I/O modules are arranged between the control system and the controlled object. That is, the I/O module establishes a communication connection with the controlled object. For example, a temperature transmitter is set in the controlled object. When the temperature in the device increases, the output current signal (4-20mA) of the transmitter increases, and the current signal is transmitted to the I/O module in the control room through the cable. , After analog/digital (A/D) conversion, the data is uploaded to the control system for each operation station and each server to call.

Preferably, the control system transmits real-time data to the two-dimensional module and the three-dimensional module in parallel. Preferably, the server system further includes a history server. The control system sends real-time data to the history server for dumping and storage, forming time-related historical data.

Preferably, as shown in FIG. 7 , the system for real-time control and visualization of a digital factory in an augmented reality environment of the present invention further includes a two-dimensional operator station 81 and a three-dimensional operator station 82 . Both the two-dimensional operator station 81 and the three-dimensional operator station 82 establish a communication connection with the server system 1 . The two-dimensional operator station 81 is used for displaying two-dimensional graphics of the controlled object on at least one operator station. The three-dimensional operator station 82 is used to display two-dimensional graphics of the controlled object on at least one operator station. The two-dimensional operator station 81 and the three-dimensional operator station 82 can be operated by the factory manager to display the two-dimensional view and the three-dimensional view, and view the relevant information of the controlled object. When the inspector modifies the attributes of the controlled object through the two-dimensional operator station 81 or the three-dimensional operator station 82, the two-dimensional operator station 81, the three-dimensional operator station 82, and the augmented reality device 80 display the controlled object's attributes. The synchronous change of attributes is beneficial for the inspectors to conduct inspections based on the latest information during the inspection process, reducing the error rate of inspections.

It should be noted that the above-mentioned specific embodiments are exemplary, and those skilled in the art can come up with various solutions inspired by the disclosure of the present invention, and these solutions also belong to the disclosure scope of the present invention and fall within the scope of the present invention. within the scope of protection of the invention. It should be understood by those skilled in the art that the description of the present invention and the accompanying drawings are illustrative rather than limiting to the claims. The protection scope of the present invention is defined by the claims and their equivalents. The description of the present invention contains multiple inventive concepts, such as "preferably", "according to a preferred embodiment" or "optionally" all indicate that the corresponding paragraph discloses a separate concept, and the applicant reserves the right to propose divisions according to each inventive concept the right to apply.

Claims (15)

A virtual inspection system, comprising at least a server system, at least one interactive terminal and at least one control system, wherein the server system establishes a communication connection with at least one interactive terminal and at least one control system respectively, and is characterized in that: The server system at least includes a two-dimensional module for displaying a two-dimensional graphic component of a controlled object and a three-dimensional module for displaying a three-dimensional graphic component of the controlled object, and the two-dimensional graphic component of the controlled object is the same as the three-dimensional graphic component of the controlled object. Establish a mapping relationship between graphic components, In response to the operation of the two-dimensional view by the inspector on the interactive terminal, the three-dimensional module can locate the three-dimensional graphic component corresponding to the specified two-dimensional graphic component in the three-dimensional view. The virtual inspection system according to claim 1, wherein the attributes of the controlled object can be displayed on the interactive terminal in two-dimensional and three-dimensional parallel in response to the operation of the inspector, The two-dimensional module and the three-dimensional module simultaneously change the two-dimensional graphics component and the three-dimensional graphics component of the controlled object based on the property change of the controlled object. The virtual inspection system according to claim 1 or 2, wherein, in response to the inspector in the depth of field area designated by the two-dimensional view, the three-dimensional module sends data of the three-dimensional graphics of the equipment located in the depth of field area to the interactive terminal, The three-dimensional view is made to display the three-dimensional graphics of the device located in the depth-of-field area in a clear manner, and the three-dimensional view displays the three-dimensional graphics of the device located outside the depth-of-field area in a non-clear manner. The system according to any one of claims 1 to 3, characterized in that when the range viewed by the inspection personnel approaches or goes out of the edge of the depth of field area, the interactive terminal sends out reminder information to remind the personnel that the viewing range deviates from the edge of the depth of field area. Specifies the inspection range of the depth-of-field area. The virtual inspection system according to any one of claims 1 to 4, wherein the server system further comprises a graphics server, and the graphics server is used to store the attribute data, digitized data of the controlled object sent by the control system engineering data and graphic engineering data, The graphic server fuses the real-time attribute data of the controlled object with the graphic data to form a three-dimensional graphic component capable of three-dimensional display, and displays the attributes of the controlled object in a three-dimensional view. The virtual inspection system according to any one of claims 1 to 5, characterized in that, in response to the interactive operation of the inspector on the interactive terminal, the two-dimensional view of the controlled object jumps to a corresponding three-dimensional view. The virtual inspection system according to any one of claims 1 to 6, wherein the inspection personnel can delineate a depth-of-field area containing the alarm device and associated devices on the two-dimensional view of the interactive terminal, and the On the 3D view, you can view the 3D graphics and attribute information of the alarm device and its associated devices in high-definition. A virtual inspection method, characterized in that the method at least includes: Establish a mapping relationship between the two-dimensional graphic components of the controlled object and the three-dimensional graphic components, In response to the operation of the two-dimensional view by the inspector on the interactive terminal, the three-dimensional module can locate the three-dimensional graphic component corresponding to the designated two-dimensional graphic component in the three-dimensional view. The virtual inspection method according to claim 8, wherein the method further comprises: In response to the operation of the controlled object by the inspector on the interactive terminal, the attributes of the controlled object can be displayed on the interactive terminal in two-dimensional and three-dimensional manners in parallel, The 2D module and the 3D module simultaneously change the 2D graphics part and the 3D graphics part of the controlled object based on the property change of the controlled object. The virtual inspection method according to claim 8 or 9, wherein the method further comprises: In response to the depth of field area specified by the inspector in the two-dimensional view, the interactive terminal receives and displays the three-dimensional graphics of the device located in the depth of field area sent by the three-dimensional module, wherein, The 3D view displays the 3D graphics of devices located within the depth of field area in a clear manner, and the 3D view displays the 3D graphics of devices located outside the depth of field area in a non-sharp manner. The virtual inspection method according to any one of claims 8 to 10, wherein the method further comprises: When the viewing range of the inspector approaches or goes out of the edge of the depth-of-field area, the interactive terminal sends out reminder information to remind the personnel that the viewing range deviates from the inspection range of the designated depth-of-field area. A visual factory system in an augmented reality environment, characterized in that it at least includes an augmented reality device, a server system, and a number of controlled objects, The server system includes at least a two-dimensional module and a three-dimensional module, the two-dimensional module displays the two-dimensional graphic components of the controlled object in a two-dimensional manner; the three-dimensional module displays the three-dimensional graphic components of the controlled object in a three-dimensional manner; The attribute data of the controlled object can be displayed in parallel in the form of two-dimensional graphics and three-dimensional views in response to the operation instructions of the augmented reality device, The two-dimensional module and/or the three-dimensional module can control the controlled object based on the operation instruction fed back by the augmented reality device. The visual factory system in an augmented reality environment according to claim 12, wherein, in response to a delineation operation of the depth of field area of the two-dimensional view of the controlled object, the three-dimensional module sends and sends a message to the augmented reality device. A three-dimensional graphics component of the controlled object displayed in the depth of field area. The visual factory system in an augmented reality environment according to claim 12 or 13, wherein the three-dimensional graphics component is formed by a graphics server merging graphics data and attribute data, and is sent in the form of a screen frequency stream to the 3D module, The three-dimensional module displays the received three-dimensional graphic components in a rendered manner. The visual factory system in an augmented reality environment according to any one of claims 12 to 14, wherein, in response to the operation information of the action feature of the augmented reality device, the three-dimensional module is based on the inspection personnel The forward direction and action characteristics of the determine the controlled object that needs to be viewed and/or controlled.

Images