WO2018036113A1 - Augmented reality method and system - Google Patents

Abstract

An augmented reality method and system, relating to the technical field of virtual reality. The augmented reality method comprises the following steps: S1: dividing a real image into several display areas; S2: pre-testing and recording a display area and corresponding eyeball instantaneous rotation state; S3: forming a mapping relationship between the eyeball instantaneous rotation state and the display area; S4: determining a gaze region; S5: acquiring real images to obtain associated information of target objects; S6: determining the display position on a display screen of the associated information of the target objects; and S7: superposing the associated information and the target objects in the real image and displaying same. The present method and system can accurately search the display area in which the current focal point of attention of the wearer is located, and rapidly respond to obtain the target object on which the user is focusing and the associated information thereof, combining same with a real image for real-time display on a display screen; search and display of unrelated information is reduced and the experience of interactive sensing of effective information is enhanced.

Description

 Augmented reality method and system

 Technical field

 [0001] The present invention relates to the field of virtual reality technologies, and in particular, to an augmented reality method and system.

 Background technique

 [0002] Augmented reality technology uses computer graphics technology and visualization technology to generate virtual objects that do not exist in the real environment, and accurately "embeds" virtual objects into the real environment, and integrates the virtual objects with the real environment by means of display devices. Applying virtual information to the real world, presenting the user with a new environment with realistic sensory effects to enhance the reality.

 [0003] An augmented reality system for implementing augmented reality technology requires that a large amount of positioning data and scene information be analyzed to ensure that a computer-generated virtual object can be accurately positioned in a real scene. Therefore, in the known augmented reality technology, the specific implementation process may include: acquiring real scene information; analyzing the obtained real scene information and camera position information; generating a virtual object; drawing on the visible plane according to the camera position information A virtual object, and the virtual object is displayed along with the real scene information. In the above-described technical presentation, since the overall feature of the image is used to match the recognition object, the amount of calculation in the corresponding processing is large. Moreover, there is a certain requirement on the size of the target image collected by the camera. If the target is far away from the camera, the target image captured by the camera is too small, which will result in too few features in the detected target image, and the reasonable requirements cannot be met. Matching the number of features, which in turn makes it impossible to detect the target object, making it impossible to superimpose the virtual object into the video.

In addition, hardware devices that implement augmented reality are typically tablets with on-screen displays, or similar handheld electronic devices. However, for emerging head-end products, the low-latency interactive experience and high-speed graphics rendering processing requirements are different from those for general handheld electronic products, and thus more design requirements for devices. For example, the wearer's impulsive mood on an object (commodity, building, etc.) in the real scene of the outside world may be short-lived, how to display the virtual information frame or virtual object in the head display during the short-lived interest maintenance period. Superimposed on the position of the real object displayed is a technical problem that the design needs to solve. In addition, how to create a virtual information display mechanism in the complicated external object information, only to show the real object that the wearer pays attention to at the moment, for the head-mounted device, there is no need for uninterrupted shooting prospects. , Reducing the amount of data information processing and effectively enhancing the effective information interaction experience are also urgent problems to be solved.

 technical problem

 [0005] In order to solve the above technical problem, the present invention provides an augmented reality method and system, which can reduce the calculation amount of the head-wearing device, and better obtain the information analysis of the real-time image object that the user pays attention at the moment in the headband device. , and superimpose virtual information and real world information on the display.

 Problem solution

 Technical solution

 [0006] The technical solution adopted by the present invention is: providing an augmented reality method, including the following steps,

 [0007] S1 divides the real image displayed in the display of the head mounted device into a plurality of display areas;

 [0008] S2 pre-tests and records the instantaneous rotation state of the eyeball on any display area on the display screen of the wearable device and the specific pixel display position in the area;

[0009] S3 forms a mapping relationship between the instantaneous rotation state of the eyeball pre-tested and recorded, and any display area on the display screen and a specific pixel display position in the area;

[0010] S4 determines the gaze area of the gaze on the display screen according to the instantaneous rotation state of the acquired eyeball;

[0011] S5 collects a real image displayed by the gaze area, and uploads it to a cloud database for processing, to obtain association information of each target object in the real image;

[0012] S6 determines the location of each target object, and determines the display position of the associated information of each target object on the display screen according to the location information of each target object;

[0013] S7 superimposes the association information of each target object on each screen with each target object in the real image.

 [0014] The augmented reality method of the present invention, step S5 further includes:

 [0015] S51 detects the overall/local features of the acquired real image, and identifies each target object.

 [0016] The augmented reality method of the present invention, step S51 further includes:

 [0017] S511 matches the global/local features of the real image with the overall/local features of the target objects that may be matched, and retains a reasonable match according to the geometric relative feature relationship of the global/local feature positions, and identifies according to a reasonable match. Each target object.

[0018] According to the augmented reality method of the present invention, the association information in step S5 includes direct correlation with the target object attribute. Product information.

 [0019] The present invention also provides an augmented reality system, including a head mounted device, further comprising:

[0020] an infrared camera for performing a real infrared photograph on the eyeball of the wearer of the head mounted device, and obtaining a picture of the instantaneous rotation state of the eyeball;

 [0021] an eyeball state determining unit, configured to analyze a picture of an instantaneous eyeball rotation state, and determine a gaze region of the eyeball

[0022] an eyeball recognition unit, configured to determine a realistic image that the wearer is paying attention at the moment according to the gaze region of the eyeball against the key value mapping table;

 [0023] a screen control unit, configured to control display content on the display screen according to a realistic image that the wearer is currently paying attention to

[0024] In the augmented reality system of the present invention, the key value mapping table is a mapping relationship between an instantaneous rotation state of an eyeball that is pre-tested and recorded, and any display area on the display screen and a specific pixel display position in the area. table.

 According to the augmented reality system of the present invention, the screen control unit is further configured to upload the real image that the wearer pays attention to to the cloud database for processing, and obtain association information of each target object in the real image.

 In the augmented reality system of the present invention, the screen control unit is further configured to determine a location of each target object, and determine a display position of the associated information of each target object on the display screen according to the location information of each target object.

 [0027] In the augmented reality system of the present invention, the screen control unit is further configured to superimpose and display the associated information of each target object on each of the target objects in the real image on the display screen.

 Advantageous effects of the invention

 Beneficial effect

Compared with the prior art, the augmented reality method and system provided by the present invention can map the display area where the focus is focused on the wearer at present by establishing a mapping relationship between the instantaneous rotation state of the wearer's eyeball and the display area on the display screen. Accurately search, quickly respond to the target object and related information that the user pays attention to and display it on the display screen in combination with the real image, which can reduce the search and display of irrelevant information and enhance the effective information interaction experience. At the same time, since there is no need to analyze and process all the real images displayed on the display, the head-mounted device reduces the amount of data information processing, the configuration of the device or the size of the device. Can get a big improvement.

 Brief description of the drawing

 DRAWINGS

 [0029] The present invention will be further described below in conjunction with the accompanying drawings and embodiments, in which:

 1 is a flowchart of an augmented reality method according to an embodiment of the present invention;

 FIG. 2 is a flowchart of a sub-step of step S5 in the augmented reality method according to an embodiment of the present invention.

Embodiments of the invention

 The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

 [0033] FIG. 1 is a flowchart of an augmented reality method according to an embodiment of the present invention. As shown in FIG. 1, the embodiment provides an augmented reality method, including the following steps.

 [0034] S1 divides the real image displayed in the display of the head mounted device into a plurality of display areas.

 [0035] In this step, the head mounted device may be any one of an external headset, an integrated headset, and a mobile headset. The realistic image displayed in the display can be directly displayed and displayed in the external environment by providing a transparent display on the head-mounted device. The external display environment can be additionally photographed by the external device of the head-mounted device, and then displayed through the display. The screen displays the actual image taken, and in this way, instant shooting is achieved. The display area in this embodiment is divided into four areas of upper left, upper right, lower left, and lower right according to the real image displayed by the display screen. It can be understood that the number of spatial divisions and the division form of the display area of the present invention are not limited thereto. Regardless of how the head rotates, the realistic images presented in perspective in this embodiment are all zero-delayed.

 [0036] S2 pre-tests and records the instantaneous rotation state of the blinking eye on any display area on the display screen of the wearable device and the specific pixel display position in the area.

[0037] Since the shape of each user's eyes and the range of their visual acuity are different, in order to improve the accuracy of the augmented reality, the present embodiment accurately verifies the visual field range that can be seen by the eyeball before the wearer officially uses it. And the binocular eyes gaze at any display area on the display screen and the momentary rotation state of the eyeball on the specific pixel display position in the area, and establish the position or state of the wearer's eyeball rotation and the display area on the display screen. Contacted and accurate to the pixel level, ensuring higher accuracy for post-recognition.

 [0038] S3 forms a mapping relationship between the instantaneous rotation state of the eyeball that is previously tested and recorded, and any display area on the display screen and a specific pixel display position in the area.

[0039] This step is used to establish a one-to-one relationship between the instantaneous rotation state of the wearer's eyeball and the display area on the display screen, so that the link can be quickly responded at the later inspection or identification, reducing the delay and improving the user experience.

 [0040] S4 determines the gaze area of the gaze on the display screen according to the instantaneous rotation state of the acquired eyeball.

 [0041] The infrared camera is used to take a picture of the eyeball, and the instantaneous rotation state of the eyeball can be obtained. Through the mapping relationship formed in the step S3, the specific display area that the eye is watching on the display screen can be quickly responded to.

, that is, the gaze area on the display.

[0042] S5 collects the real image displayed by the gaze area, and uploads it to the cloud database for processing, and obtains association information of each target object in the real image.

[0043] The gaze area of the wearer on the display screen, that is, the realistic scene that the wearer pays attention to at present, in this embodiment, the device only parses and processes the realistic scene that the wearer pays attention at the moment, that is, only the gaze area is presented. The real image is searched and recognized. On the one hand, the user's real focus is obtained by information retrieval and key display, solving the user's needs, reducing the invalid information display that the user does not need. On the other hand, compared with the prior art, Parsing and processing all the real images seen on the display screen greatly reduces the data processing capacity and response delay of the device, and enhances the interactive experience of effective information. The cloud database has image recognition function and information search function, which can identify the target object of interest to the user in the real image and search for the related information of the target object. The related information includes product information directly related to the attribute of the target object, such as product correlation. Specifications, product function description, company profile, price information, contact information, etc.

[0044] Preferably, as shown in FIG. 2, step S5 includes

 [0045] S51 detects the overall/local features of the acquired real image, and identifies each target object.

 [0046] This step detects the overall/local features of the real image, such as a feature matching graph in the cloud database, and then identifies the target object to find the target object.

[0047] Preferably, as shown in FIG. 2, step S51 includes

[0048] S511 enters the overall/local features of the real image with the overall/local features of the target objects that may be matched. Line matching, according to the geometric relative feature relationship of the overall/local feature position, retains a reasonable match, and identifies each target object according to a reasonable match.

 [0049] The method of this step can screen out important objects, reduce the feature matching of irrelevant objects in the real image, and achieve the effect of reducing the calculation amount and quickly obtaining the matching result.

[0050] S6 determines the location of each target object, and determines the display position of the association information of each target object on the display screen according to the location information of each target object.

[0051] The target object obtained through the processing of the cloud database will be highlighted on the display screen, and the peer device will calculate the position of each target object on the screen, and determine each target object according to the position information of each target object. The associated information is displayed on the display.

[0052] S7 superimposes the association information of each target object on each screen with each target object in the real image.

 [0053] The relationship information of each target object that the wearer pays attention to is displayed in combination with each target object of the real image directly on the display screen according to the display position of step S6, thereby realizing the fusion and seamless connection of the virtual information and the real world information.

[0054] Embodiments of the present invention further provide an augmented reality system, including a head mounted device, further comprising an infrared camera mounted on the head mounted device, an eyeball state determining unit, an eyeball identifying unit, and a screen control unit. Infrared camera for performing real-time infrared photographing on the eyeball of the wearer of the head-mounted device, and obtaining a picture of the instantaneous rotation state of the eyeball; an eyeball state determining unit for analyzing a picture of the instantaneous state of the eyeball rotation, determining the gaze area of the eyeball; The identification unit is configured to determine a realistic image that the wearer pays attention to at the moment according to the gaze area of the eyeball, and the key value mapping table is an instantaneous rotation state of the eyeball pre-tested and recorded, and any display area on the display screen and the area a mapping relationship table formed between specific pixel display positions; a screen control unit configured to control display content on the display screen according to a realistic image that the wearer is currently paying attention to. Specifically, the screen control unit includes a network transmission module and an information matching module. The network transmission module is used to upload the real image that the wearer pays attention to at the cloud database, and then download the target object and related information processed by the cloud database to the head mounted device, and the information matching module is used to display the target object and the display screen. Realistic image on the horse Matching, determining the position of each target object, and determining the display position of the related information of each target object on the display screen according to the position information of each target object, and finally respectively, the related information of each target object is in the real image on the display screen. Goals Object overlay display.

 The embodiments of the present invention have been described above with reference to the drawings, but the present invention is not limited to the specific embodiments described above, and the specific embodiments described above are merely illustrative and not restrictive, and those skilled in the art In the light of the present invention, many forms may be made without departing from the spirit and scope of the invention as claimed.

Claims

Claim  [Attach 1] An augmented reality method, comprising the following steps,  S1 divides the real image displayed in the display of the head mounted device into a plurality of display areas; S2 pre-tests and records the instantaneous rotation state of the blinking eye on any display area on the display screen of the wearable device and the specific pixel display position in the area;  S3 forms a mapping relationship between the instantaneous rotation state of the eyeball pre-tested and recorded, and any display area on the display screen and the specific pixel display position in the area;  S4 determines the gaze area of the gaze on the display screen according to the instantaneous rotation state of the acquired eyeball; S5 collects the real image displayed by the gaze area, and uploads it to the cloud database for processing, and obtains association information of each target object in the real image;  S6 determines the location of each target object, and determines the display position of the associated information of each target object on the display screen according to the location information of each target object;  S7 superimposes the associated information of each target object on each screen with each target object in the real image.  [Attachment 2] The augmented reality method of claim 1, wherein the step S5 further comprises:  S51 detects the overall/local features of the acquired real image and identifies each target object.  The augmented reality method of claim 2, wherein the step S51 further comprises:  S511 matches the overall/local features of the real image with the overall/local features of the target objects that may be matched, and retains a reasonable match according to the geometric relative feature relationship of the global/local feature positions, and identifies each target object according to a reasonable match. .  [Attachment 4] The augmented reality method according to claim 1, wherein the association information in step S5 includes product information directly related to the target object attribute.  [Attachment 5] An augmented reality system, comprising a head mounted device, further comprising:  Infrared camera for performing real-time infrared photographing on the eye of the wearer of the head-mounted device, and obtaining a picture of the instantaneous rotation of the eyeball;  An eyeball state determining unit, configured to analyze a picture of a momentary eyeball rotation state, and determine a gaze area of the eyeball; An eyeball recognition unit for determining the wearing according to the gaze area of the eyeball against the key value mapping table Realistic image of the moment of attention;  A screen control unit for controlling display content on the display screen according to a realistic image that the wearer is currently paying attention to.  [Claim 6] The augmented reality system according to claim 5, wherein the key value mapping table is an instantaneous rotation state of an eyeball pre-tested and recorded, and any display area on the display screen and a specific pixel in the area Displays a mapping table formed between locations. The augmented reality system of claim 5, wherein the screen control unit is further configured to upload a real image that is currently of interest to the wearer to the cloud database for processing, to obtain each target object in the real image. Associated information. The augmented reality system of claim 5, wherein the screen control unit is further configured to determine a location of each target object, and determine association information of each target object according to location information of each target object. The display position on the display. The augmented reality system according to claim 5, wherein the screen control unit is further configured to superimpose the associated information of each target object on each of the target objects in the real image on the display screen. .