CN111813214A - Method, device, terminal device and storage medium for processing virtual content - Google Patents

Abstract

The embodiment of the application discloses a method and a device for processing virtual content, terminal equipment and a storage medium, and relates to the technical field of display. The processing method of the virtual content comprises the following steps: determining six-degree-of-freedom (6DoF) information of the interactive device according to the acquired marker image, wherein the marker image comprises a marker arranged on the interactive device; acquiring a target area of a target object based on the 6DoF information, wherein the target area is an area selected by the interactive device; acquiring content data corresponding to a target area; generating virtual content according to the 6DoF information and the content data; displaying the virtual content; and receiving control data sent by the interactive device, generating a corresponding content processing instruction according to the control data, and processing the virtual content according to the content processing instruction. The method can select the virtual content to be processed through the 6DoF information of the interactive device, and directly process the virtual content through the interactive device, thereby being convenient and fast.

Description

Method, device, terminal device and storage medium for processing virtual content

technical field

The present application relates to the field of display technology, and more particularly, to a method, apparatus, terminal device and storage medium for processing virtual content.

Background technique

With the development of science and technology, machine intelligence and information intelligence are becoming more and more popular, and the technology of identifying user images through image acquisition devices such as machine vision or virtual vision to realize human-computer interaction is becoming more and more important. Augmented Reality (AR) uses computer graphics technology and visualization technology to construct virtual content that does not exist in the real environment, and accurately integrates the virtual content into the real environment through image recognition and positioning technology. The real environment is integrated and displayed to the user with a real sensory experience.

The primary technical problem to be solved by augmented reality technology is how to accurately integrate the virtual content into the real world, that is, to make the virtual content appear in the correct position of the real scene with the correct posture, thereby generating a strong sense of visual reality. In traditional technologies, when augmented reality or mixed reality display is performed by superimposing virtual content on a real scene image, the virtual content is usually simply displayed, and the user can only control the display of the virtual content through a controller such as a traditional remote control. state, such as controlling the virtual content to translate, rotate or zoom, etc., but cannot flexibly control the part of the virtual content according to the user. Therefore, the interaction between the user and the displayed virtual content is poor.

SUMMARY OF THE INVENTION

The embodiments of the present application propose a method, apparatus, terminal device and storage medium for processing virtual content, which can improve the interaction between users and virtual content.

In a first aspect, an embodiment of the present application provides a method for processing virtual content, which is applied to a terminal device. The processing method includes: determining six degrees of freedom (6DoF) information of an interaction device according to a collected marker image, and the marker image Including markers set in the interactive device; based on the 6DoF information, obtain the target area of the target object, the target area is the area selected by the interactive device; obtain the content data corresponding to the target area; generate virtual content according to the 6DoF information and content data; display virtual content and receiving control data sent by the interactive device, generating corresponding content processing instructions according to the control data, and processing virtual content according to the content processing instructions.

In a second aspect, an embodiment of the present application provides an apparatus for processing virtual content, the apparatus includes an information determination module, a region determination module, a data acquisition module, a content generation module, a content display module, and a content processing module, wherein the information determination module It is used to determine the six degrees of freedom (6DoF) information of the interactive device according to the collected marker image, and the marker image includes the marker provided in the interactive device; the area determination module is used to obtain the target area of the target object based on the 6DoF information, The target area is an area selected by the interactive device; the data acquisition module is used to obtain content data corresponding to the target area; the content generation module is used to generate virtual content according to the 6DoF information and the content data; the content display module is used to display the virtual content; and the content processing module It is used to receive the control data sent by the interactive device, generate corresponding content processing instructions according to the control data, and process the virtual content according to the content processing instructions.

In a third aspect, embodiments of the present application provide a terminal device, including: one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be One or more processors execute, and one or more programs are configured to execute the method for processing virtual content provided in the first aspect.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where a program code is stored in the computer-readable storage medium, and the program code can be invoked by a processor to execute the processing method for virtual content provided in the first aspect above .

The virtual content processing method, device, terminal device, and storage medium provided by the embodiments of the present application determine the target area of the target object according to the 6DoF information of the interaction device, collect the content data of the target area, and then generate a The corresponding virtual content is displayed, and finally, the control data sent by the interactive device is used, and corresponding content processing instructions are generated according to the control data, and the virtual content is processed according to the content processing instructions. Therefore, the user can select the virtual content to be processed through the 6DoF information of the interactive device, and directly process the virtual content through the interactive device, which is convenient and quick, and can improve the interactivity between the user and the displayed virtual content.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained from these drawings without creative effort.

FIG. 1 is a schematic diagram of a virtual content processing system provided by an embodiment of the present application.

FIG. 2 is a schematic flowchart of a method for processing virtual content provided by an embodiment of the present application.

FIG. 3 is a schematic flowchart of another virtual content processing method provided by an embodiment of the present application.

FIG. 4 is a schematic flowchart of superimposing and displaying virtual objects in the method shown in FIG. 3 .

FIG. 5 is a schematic flowchart of another superimposed display of virtual objects in the method shown in FIG. 3 .

FIG. 6 is a schematic diagram of a process of displaying virtual objects shown in FIG. 5 .

FIG. 7 is a schematic diagram of another process of displaying a virtual object shown in FIG. 5 .

FIG. 8 is a schematic flowchart of determining a target area in the method shown in FIG. 3 .

FIG. 9 is a schematic diagram of determining a target area in the method shown in FIG. 8 .

FIG. 10 is a schematic diagram of another determination of the target area in the method shown in FIG. 3 .

FIG. 11 is a schematic flowchart of determining a target area in the method shown in FIG. 10 .

FIG. 12 is a schematic diagram of dividing the target area in the method shown in FIG. 3

FIG. 13 is a schematic diagram of a content processing process in the method shown in FIG. 3 .

FIG. 14 is a schematic diagram of another content processing process in the method shown in FIG. 3 .

FIG. 15 is a schematic diagram of another content processing process in the method shown in FIG. 3 .

FIG. 16 is a schematic diagram of superimposing a processing content on a target object in the method shown in FIG. 3 .

FIG. 17 is a schematic diagram of another content processing process in the method shown in FIG. 3

FIG. 18 is a structural block diagram of an apparatus for processing virtual content provided by an embodiment of the present application.

FIG. 19 is a structural block diagram of a terminal device provided by an embodiment of the present application.

FIG. 20 is a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application.

Detailed ways

In order to make those skilled in the art better understand the solutions of the present application, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application.

In recent years, with the development of augmented reality technology, AR-related electronic devices have gradually entered people's daily life. Among them, AR is a technology that increases the user's perception of the real world through the information provided by the computer system. It superimposes computer-generated virtual objects, scenes or system prompt information and other content objects into the real scene to enhance or modify the real-world environment or Perception of data representing a real-world environment. However, traditional AR usually only simply displays virtual content, the display method is single, and the user's interaction with the displayed virtual content is poor.

After research, the inventor proposes the virtual content processing method, device, terminal device and storage medium in the embodiments of the present application, which can improve the interaction between the user and the displayed virtual content and improve the immersion.

Referring to FIG. 1 , a virtual content processing system 10 provided by an embodiment of the present application is shown. The virtual content processing system 10 includes: a terminal device 100 , an interaction apparatus 200 , and a target object 300 . The terminal device 100 and the interaction device 200 may be connected by communication methods such as Bluetooth, WIFI (Wireless-Fidelity), ZigBee (Zifeng Technology), or may be connected by wired methods such as data lines. Certainly, the connection manner between the terminal device 100 and the interaction apparatus 200 may not be limited in the embodiment of the present application.

In this embodiment of the present application, the terminal device 100 may be a head-mounted display device, or may be a mobile device such as a mobile phone or a tablet. When the terminal device 100 is a head-mounted display device, the head-mounted display device may be an integrated head-mounted display device. The terminal device 100 can also be a smart terminal such as a mobile phone connected to an external head-mounted display device, that is, the terminal device 100 can be used as a processing and storage device of the head-mounted display device to insert or access the external head-mounted display device to display. virtual content.

In the embodiment of the present application, the interaction device 200 is a tablet-shaped electronic device, which is provided with a marker 210 . The specific morphological structure of the interaction device 200 is not limited, and can be in various shapes, such as square and circle. The number of markers 210 of the interaction device 200 may be one or more. As an embodiment, the marker 210 is provided on the surface of the interaction device 200 . In this case, the interaction device 200 is an electronic device provided with the marker 210 .

When using the interactive device 200, the marker 210 can be located within the field of view of the terminal device 100, so that the terminal device 100 can collect an image containing the marker 210, so as to identify and track the marker 210, thereby realizing the interaction device 200 location tracking. In some embodiments, the interactive device 200 can be held and manipulated by the user, and the marker 210 can be integrated in the interactive device 200 , can also be pasted and attached to the interactive device 200 , and can also be displayed on the display screen of the interactive device 200 .

In some embodiments, marker 210 may include at least one sub-marker having one or more feature points. When the above-mentioned marker 210 is within the field of view of the terminal device 100, the terminal device 100 may use the above-mentioned marker 210 within the field of view as a target marker, and capture an image including the target marker. After the image containing the target marker is collected, the collected image of the target marker can be identified to obtain spatial position information such as the position and posture of the target marker relative to the terminal device 100, as well as the identity information of the target marker, etc. The recognition result is obtained, and then the spatial position information such as the position and attitude of the interaction device 200 relative to the terminal device 100 is obtained, that is, the six degrees of freedom (6DoF) information of the interaction device 200, so as to realize the positioning and tracking of the interaction device 200. .

The 6DoF information of the interactive device 200 means that the interactive device has six degrees of freedom information in space, that is, the movement degrees of freedom along the three rectangular coordinate axes of X, Y, and Z in the space coordinate system and the three coordinates around the three coordinates. Information about the rotational degrees of freedom of the axis. In this embodiment of the present application, the 6DoF information of the interaction apparatus 200 at least includes: the movement direction, movement distance, rotation direction, and rotation angle of the interaction apparatus 200 relative to the terminal device 100 . Further, by acquiring the 6DoF information of the interactive device 200, the interactive device 200 can be positioned and tracked, the absolute position of the interactive device 200 can be monitored, and then the position of the interactive device 200 in the real space can be accurately tracked, so as to accurately perform the operation in the virtual world. map to ensure the accuracy of the interaction process.

Further, the terminal device 100 may display corresponding virtual content based on the relative spatial positional relationship with the interaction device 200 . It can be understood that the specific marker 210 is not limited in the embodiments of the present application, and only needs to be able to be identified and tracked by the terminal device 100 .

In this embodiment of the present application, the target object 300 is a third-party object, that is, the target object 300 may be any physical entity in the real world, or may be any virtual object displayed by the terminal device 100 . When a certain area of the target object 300 is selected through the interaction device 200 , the terminal device 100 can acquire the content data corresponding to the selected area according to the area selected by the interaction device 200 , and then according to the 6DoF information of the interaction device 200 and the acquired content The data generates virtual content, aligns the virtual content with the interactive device 200, and displays the virtual content on the interactive device 200 in an AR manner. In some implementations, at least one manipulation area is provided on the interaction device 200 for the user to perform operations on the manipulation area, so as to process the virtual content. Wherein, the manipulation area may include at least one of buttons and a touch screen. The interaction apparatus 200 may generate a content processing instruction corresponding to the operation action through the operation action detected in the manipulation area, and send the content processing instruction to the terminal device 100 . When the terminal device 100 receives the content processing instruction sent by the interaction apparatus 200, it can process the displayed virtual content according to the content processing instruction, for example, edit and mark the virtual content.

In some embodiments, the interactive device 200 may also be a mobile terminal with a touch screen, such as a smart phone, a tablet computer, etc., with a touch screen that can display images and can be manipulated, and the marker 210 may be provided on the casing of the mobile terminal, It can also be displayed on the touch screen, or inserted into the mobile terminal as an accessory, for example, through a USB interface or an earphone interface, etc., but is not limited to this.

Based on the above processing system, an embodiment of the present application provides a method for processing virtual content, which is applied to a terminal device and an interaction device of the above processing system. The specific processing method of virtual content will be introduced below.

Please refer to FIG. 2. FIG. 2 shows a method for processing virtual content provided by an embodiment of the present application, which can be applied to the above-mentioned terminal device. The method for processing virtual content enables the user to process the displayed virtual content through the 6DoF information of the interactive device, thereby improving the interactivity between the user and the displayed virtual content. Specifically, the virtual content processing method may include steps S110-S160.

Step S110: Determine six degrees of freedom (6DoF) information of the interactive device according to the collected marker image. Wherein, the marker image includes markers provided on the interactive device.

In some embodiments, the image of the marker including the marker can be collected by the image acquisition device of the terminal device, wherein the marker can be integrated into the interactive device, can also be pasted and attached to the interactive device, and can also be displayed on the display of the interactive device on the screen.

In this embodiment of the present application, the image of the marker of the interaction device may be acquired through a visual device (eg, an image sensor, etc.), and the terminal device re-recognizes the marker in the image and obtains the relative value of the interaction device relative to the terminal device according to the recognition result of the marker. The position and attitude information of the interaction device can be positioned and tracked, wherein the attitude information may include the rotation direction and rotation angle of the interaction device relative to the terminal device. In some embodiments, the above identification result includes at least position information, rotation direction and rotation angle of the marker relative to the terminal device, so that the terminal device can obtain the position of the interaction device relative to the terminal device according to the set position of the marker on the interaction device and attitude information, that is, six degrees of freedom (6DoF) information of the interactive device, so as to realize the positioning and tracking of the interactive device.

Step S120: Based on the 6DoF information, obtain a target area of the target object, where the target area is an area selected by the interaction device.

In this embodiment of the present application, the terminal device may acquire the relative positional relationship between the interaction device and the target object according to the acquired 6DoF information of the interaction device, where the target object may be an entity object in the real space, or a terminal device The virtual objects displayed in the real space are superimposed in the way of AR. It should be noted that the relative positional relationship between the interaction device and the target object is the first relative positional relationship, and the first relative positional relationship may be the relative positional relationship between the interaction device and the physical object, or the relative positional relationship between the interaction device and the virtual object. .

Specifically, the second relative positional relationship between the target object and the terminal device can be obtained, and then according to the position and attitude information of the interaction device relative to the terminal device, and the terminal device is used as a reference, the first relative position relationship between the interaction device and the target object can be obtained. The relative position relationship is obtained, and then the area selected by the interactive device, that is, the target area of the target object is obtained. The second relative positional relationship is the relative positional relationship between the target object and the terminal device. Correspondingly, the second relative positional relationship may be the relative positional relationship between the virtual object and the terminal device, or the physical object and the terminal device. relative positional relationship between them. It should be noted that the first relative position relationship may include, but is not limited to, position information, rotation direction, and rotation angle of the interaction device relative to the target object. The second relative positional relationship may include, but is not limited to, the positional information of the target object relative to the terminal device, the rotation direction, that is, the rotation angle, and the like.

Further, the target area may be a part of the area of the target object, or the area where a part of the component structure of the target object is located, or may be one or more components or structures of the target object. Optionally, the target area of the target object may be a partial area of the target object blocked by the interaction device, or may be an area of the target object pointed by the interaction device. That is, the first relative positional relationship between the interaction device and the target object can be used to determine the area where the interaction device blocks or points to the target object.

Step S130: Acquire content data corresponding to the target area.

In the embodiment of the present application, the content data corresponding to the target area refers to the information data corresponding to the target object in the target area, and the information data may be the content of the target object in the target area, such as the shape of the component of the target object in the target area Appearance, internal structure and other information, the information data can also be data corresponding to the content of the target object in the target area, such as component parameters in the target area, etc., but not limited to this.

Further, determining the content data corresponding to the target area can be downloaded from the server by the terminal device according to the corresponding relationship between the target area and the target object, or obtained by the terminal device from other devices according to the corresponding relationship between the target area and the target object, and also It may be obtained by the terminal device from the local storage according to the above-mentioned corresponding relationship.

Step S140: Generate virtual content according to the 6DoF information and content data.

In the embodiment of the present application, the 6DoF information is the 6DoF information of the interaction device acquired by the terminal device, that is, the position and attitude information of the interaction device relative to the terminal device.

In some embodiments, the virtual content may be content such as the appearance, internal structure, etc. corresponding to the target object in the target area, that is, the virtual content may be content generated based on the above-mentioned content data. Further, when virtual content is generated according to the 6DoF information of the interaction device and the acquired content data, the acquired content data may be rendered according to the 6DoF information of the interaction device to generate virtual content. The acquired content data comes from the target area determined according to the 6DoF information of the interactive device, and virtual content is generated by using the acquired content data. Alternatively, the virtual content may be a graphical user interface, a two-dimensional graphic or a three-dimensional model.

Step S150: Display the virtual content.

In this embodiment of the present application, after the terminal device generates the virtual content, the virtual content may be displayed.

In some embodiments, if the interactive device is provided with a display area, the virtual content generated by the terminal device can be directly displayed on the display area of the interactive device. Wherein, the display area of the interaction device may be the area where the display screen of the interaction device is located, that is, the virtual content is displayed through the display screen of the interaction device, and the terminal device may send the generated virtual content to the interaction device, so that the virtual content can be displayed on the interaction device. displayed on the display.

In other embodiments, the interaction device may have a corresponding virtual display area, the virtual display area and the interaction device have a third relative positional relationship, or the virtual display area and the marker of the interaction device may have a third relative positional relationship, the The virtual display area can be used to define an area where the virtual content is superimposed and displayed in the real space, so as to adapt to the interactive device without a display screen, or the display screen of the interactive device is not suitable for displaying the current virtual content. After the terminal device obtains the 6DoF information of the interactive device and the third relative positional relationship between the virtual display area and the interactive device, it can calculate the spatial position of the virtual display area relative to the terminal device according to the 6DoF information of the interactive device and the third relative positional relationship, and The virtual content is generated based on the spatial position of the virtual display area, so that the visual effect of superimposing and displaying the virtual content on the virtual display area can be realized. The third relative positional relationship between the virtual display area and the interactive device can be set according to actual requirements. For example, the plane where the virtual display area is located can be perpendicular to, or overlap with, or be juxtaposed with the touch plane of the interactive device. , or a relatively inclined setting, the third relative positional relationship between the virtual display area and the interactive device can also be adjusted according to the user's viewing habits, so as to facilitate the user's use.

At this time, step S150 may include: determining the virtual display area based on the 6DoF information of the interactive device and a preset third relative positional relationship between the virtual display area and the interactive device, and re-determining the virtual display area according to the operation action of the user on the interactive device The relative spatial positional relationship with the interactive device, and the positional information of the virtual display area is determined based on the relative spatial positional relationship, and then virtual content is generated according to the positional information of the virtual display area, and then the virtual content is superimposed and displayed in the virtual display area, wherein, The user can adjust the angle between the plane where the virtual display area is located and the interactive device by operating the interactive device, adjust the area size of the virtual display area, or adjust the orientation information of the virtual display area relative to the interactive device, etc. It should be noted that the size of the virtual display area may be a preset size, or the size of the virtual display area may be adjusted according to actual application requirements.

Step S160: Receive the control data sent by the interaction device, generate a corresponding content processing instruction according to the control data, and process the virtual content according to the content processing instruction.

In some embodiments, the control data sent by the interactive device is generated based on an operation action (eg, a touch action, a pressing action, etc.) input by the user on the interactive device, and the operational action may include a touch operation, a button pressing operation, etc. and other operation actions, wherein the type of the touch operation includes at least any one or more of click, slide, and multi-touch. The terminal device can determine the corresponding content processing instruction according to the control data sent by the interaction device, wherein the types of the content processing instruction at least include zooming, modifying, marking, moving, rotating, etc., but not limited thereto.

Further, the virtual content is processed according to the content processing instruction, the currently displayed virtual content may be processed, or some objects in the currently displayed virtual content, such as a certain component or structure, may be processed. For example, if the received content processing instruction is a zoom or rotation instruction, the terminal device zooms or rotates the currently displayed virtual content or some objects. If the received content processing instruction is a modification instruction, the terminal device modifies the parameters or data of the currently displayed virtual content or some objects. If the received content processing instruction is a marked instruction, the terminal device performs a text mark or simple mark on the currently displayed virtual content or part of the object. If the received content processing instruction is a moving instruction, the currently displayed virtual content or some objects can be moved.

In the method for processing virtual content provided by the embodiment of the present application, the target area of the target object is determined according to the 6DoF information of the interactive device, and the content data of the target area is collected, and then the corresponding virtual content is generated and displayed according to the 6DoF information and the content data, and finally By receiving the control data sent by the interactive device, and generating corresponding content processing instructions according to the control data, and processing the virtual content according to the content processing instructions. Therefore, the user can select the virtual content to be processed through the 6DoF information of the interactive device, and directly process the virtual content through the interactive device, which is convenient and quick, can improve the interaction between the user and the displayed virtual content, and improve the sense of immersion. .

Another embodiment of the present application provides a method for processing virtual content, which can be applied to the above-mentioned terminal device. The method for processing virtual content enables the user to select the virtual content to be processed through the 6DoF information of the interactive device, directly process the virtual content through the interactive device, and at the same time superimpose and display the processed content to the target area, which can improve the user experience. Interactivity with displayed virtual content.

Specifically, referring to FIG. 3 , the virtual content processing method may include steps S210-S300.

Step S210: Acquire the virtual object to be displayed, and display the virtual object to be displayed to a designated display position.

In this embodiment of the present application, the target object may be a virtual object displayed by the terminal device. When the terminal device displays the virtual object, the terminal device needs to acquire parameter data of the virtual object to be displayed. The parameter data may include model data of the virtual object to be displayed, and the model data is data used to render the virtual object, thereby realizing the display of the virtual object. For example, the model data may include color data, vertex coordinate data, outline data, and the like for establishing the corresponding virtual objects. In addition, the model data of the virtual object to be displayed may be stored in the terminal device, or may be obtained from other electronic devices such as an interaction device and a server.

In this embodiment of the present application, displaying the virtual object to be displayed at a specified display position may be superimposing and displaying the virtual object displayed by the terminal device in the real space (for example, previewing the virtual object to be displayed on the terminal device, and The virtual object to be displayed is superimposed and displayed in a specific area), so that when viewing the virtual object superimposed and displayed in the real space through the terminal device, the structure of the virtual object can be clearly viewed, wherein the display position can be displayed by the virtual object in the virtual space. The spatial coordinate representation in . Specifically, please refer to FIG. 4 , the display step includes steps S410-S430.

Step 410: Determine the superimposed position of the virtual object in the real environment.

The superimposed position is the position superimposed in the real space when the virtual object is displayed, and the position may be a designated position in the real space, and the virtual object superimposed and displayed at the position can be viewed through the terminal device. Referring to FIG. 5 , in this embodiment of the present application, determining the superimposition position further includes steps S411 to S413 . Wherein, steps S411-S413 are another implementation manner of step S410, and optionally, steps S411-S413 may replace step S410.

Step S411: Acquire an image containing a display marker.

In this embodiment, the display marker can be set at any position in the real space. The display marker may include at least one sub-marker, and the sub-marker may be a pattern having a certain shape. In one embodiment, each sub-marker may have one or more feature points, wherein the shape of the feature points is not limited, and may be a rectangle, a circle, a circle, a triangle, or other shapes. As an embodiment, the outline of the display marker is a rectangle, and the shape of the display marker may also be other shapes, which are not limited here. The rectangular area and multiple sub-markers in the area constitute a display marker .

It should be noted that, at this time, the virtual content processing system includes at least two kinds of markers, one is a marker set on the interactive device, which is used to obtain the 6DoF information of the interactive device; the other is set outside the interactive device display marker, which is used to determine the display area. Therefore, it is necessary to distinguish the two markers in order to accurately identify the different markers and realize the corresponding functions of the markers. Specifically, each marker may include at least one sub-marker, and the distribution rules of the sub-markers in different markers are different, for example, each marker may have different identity information. The terminal device can acquire identity information corresponding to the marker by identifying the sub-marker contained in the marker, and the identity information is information such as code that can be used to uniquely identify the marker, but is not limited to this. Wherein, the identity information can represent whether the marker is a marker used to obtain 6DoF information of the interactive device, or a display marker for determining the display area.

As another implementation manner, the two kinds of markers mentioned above may also be objects composed of light spots that can emit light by themselves, or the like. Since the light point marker can emit light of different wavelength bands or different colors, the terminal device obtains the identity information corresponding to the marker by identifying the wavelength band or color of the light emitted by the light point marker, so as to distinguish the two markers. It should be noted that the specific shape, style, size, color, number of feature points, and distribution of the specific marker are not limited in this embodiment, as long as the marker can be identified and tracked by the terminal device.

Further, to acquire the image containing the displayed marker, the image containing the displayed marker can be acquired by the image acquisition device of the terminal device.

Step S412: Identify the display marker in the image, and obtain the relative spatial positional relationship between the display marker and the terminal device.

In the embodiment of the present application, the identification of the display marker in the image is to obtain the spatial position information of the display marker. According to the obtained spatial position information of the display marker, and taking the terminal device as a reference, the relative spatial positional relationship between the display marker and the terminal device can be obtained, and the relative spatial positional relationship may include the position of the display marker relative to the terminal device and attitude, etc.

Step S413: Determine the superposition position based on the relative spatial positional relationship.

In the embodiment of the present application, a virtual space coordinate system is established with the terminal device as the origin, and the superimposed position of the virtual object in the real environment is determined based on the relative spatial position relationship between the displayed marker and the terminal device, so that the superimposed position of the virtual object can be determined according to the superimposed position. Get the display position of the virtual object in the virtual space. The terminal device can obtain the coordinates of the displayed marker in the virtual space coordinate system. The virtual space coordinate system can be a coordinate system established with the world origin in the virtual space, or a virtual camera in the virtual space (used to simulate people in the virtual space. eye position) is the coordinate system established as the origin.

In some embodiments, the superimposed position and the display marker have a certain relative relationship, and the relative relationship may include a relative size ratio, a relative positional relationship, and the like. After acquiring the coordinates of the displayed marker in the virtual space coordinate system, the terminal device can determine the superimposed position in the real environment according to the relative spatial position relationship between the displayed marker and the terminal device, as well as the relative relationship between the superimposed position and the displayed marker. . In some embodiments, the size of the superimposed position may be adjusted according to the size of the virtual object to be displayed, or the size of the model of the virtual object may be adjusted according to the fixed size of the superimposed position, wherein the fixed size of the superimposed position may be It is the set size value, and it can also have a certain proportional relationship with the display marker, such as 2 times, 3 times, 5 times, etc., of the actual physical size of the display marker in the real space, or with the actual size of the display marker. The physical size is the same, which is not limited here.

Step S420: Receive the first display instruction sent by the interaction device.

In this embodiment of the present application, the first display instruction corresponds to an operation action input by the user on the interaction device. The operation action input by the user on the interaction device may be sliding, clicking, and the like. The first display instruction may include at least one of: selecting a virtual object, determining a zoom factor when the virtual object is superimposed and displayed, and determining a dynamic effect when the virtual object is superimposed and displayed.

In some implementation manners, the first display instruction sent by the interaction apparatus may be to superimpose and display the specified virtual object to the superimposed position. Wherein, when there are multiple superimposition positions, the first display instruction sent by the interaction device includes superimposing and displaying the selected virtual object to a designated superposition position. As a specific implementation manner, the terminal device can determine the superimposed position corresponding to the sliding direction according to the sliding direction of the user on the interaction device, and re-render and superimpose the virtual object according to the corresponding superimposed position, so that the user can use the terminal device to See the virtual object overlay displayed on the corresponding overlay position. Referring to FIG. 6 , the virtual objects superimposed and displayed on the interactive device include virtual object A and virtual object B, and the superimposed positions include superimposed position C and superimposed position D. The user selects the virtual object A on the interactive device, and slides the virtual object A out of the interactive device in a direction toward the superimposed position C by sliding. Further, the user's operation action at this time corresponds to an instruction to display the virtual object A to the superimposed position C.

In some implementation manners, according to the first display instruction sent by the interaction device, the multiple virtual objects may be displayed at different superimposed positions respectively. It can be understood that the first display instruction sent by the interactive device is directional, which can display a certain virtual object to a specific overlapping position, so that the user can browse different virtual objects through different overlapping positions, so as to avoid the occurrence of the same overlapping. A situation where multiple virtual objects are displayed in a location and browsing is not good. Referring to FIG. 7 , the virtual objects superimposed and displayed on the interactive device include virtual object A and virtual object B, and the superimposed positions include superimposed position C and superimposed position D. The user first selects virtual object A on the interactive device, slides virtual object A out of the interactive device by sliding left, then selects virtual object B, and slides virtual object B out of the interactive device by sliding right. Further, the operation action of the user at this time corresponds to the instruction to superimpose and display the virtual object A to the superimposition position C, and the instruction to superimpose and display the virtual object B to the superimposition position D.

Step S430: Based on the first display instruction, display the virtual object according to the preset display effect, and the display position of the virtual object corresponds to the superimposed position.

In this embodiment of the present application, the preset superimposed display effect includes the size of the virtual object when it is superimposed and displayed in the display area, that is, the scale of the virtual object to be scaled. The scale at which the virtual object is enlarged may be 1, 2, 3, or 4 times the size of the original virtual object. The size of the original virtual object is the size when the virtual object is displayed on the terminal device for preview.

In this embodiment of the present application, after acquiring the parameter data of the virtual object, the terminal device can generate the virtual object according to the parameter data and the position of the superimposed position. As an embodiment, the virtual content to be created may be constructed according to the parameter data, and the spatial coordinates of the superimposed position in the virtual space may be obtained, and the three-dimensional rendering coordinates of the virtual object in the virtual space may be determined according to the spatial coordinates, so that the rendered three-dimensional virtual object The rendering position in the virtual space is consistent with the superimposing position, so that rendering is performed according to the rendering position to superimpose and display the virtual object at the superimposing position.

In some embodiments, since the terminal device has obtained the superimposed position, a virtual space coordinate system is established with the terminal device as a reference point, and the rendering coordinates of the virtual object in the virtual space can be obtained, that is, the rendering position of the virtual object is obtained. The rendering position includes the rendering coordinates of the virtual object, and is then rendered at the rendering position to display the virtual object. The above-mentioned rendering coordinates may be the three-dimensional space coordinates of the virtual object in the virtual space with the virtual camera of the terminal device as the origin (it may also be regarded as the human eye as the origin).

It can be understood that, after the terminal device determines the rendering coordinates for rendering the virtual object in the virtual space, the terminal device can construct the three-dimensional virtual content according to the acquired parameter data corresponding to the virtual content, and render the virtual object according to the above-mentioned rendering coordinates. . The terminal device may obtain, from the parameter data, the RGB values and corresponding coordinates of each vertex in the three-dimensional virtual object to be displayed.

In some embodiments, the preset display effect also includes an animation effect when the virtual object is displayed from the interactive device to the superimposed position. For example, please refer to FIG. 7 , the animation effect may be that the size of the original virtual object gradually increases The process to the size when displayed in the overlay position. Optionally, in the process of displaying the virtual object from the interactive device to the superimposed position, a transitional animation effect can also be added, for example, the process of the virtual object "flying out" from the interactive device according to a predetermined trajectory and falling into the superimposed position.

In some embodiments, corresponding to the process of displaying the virtual object from the interactive device to the superimposed position, the process further includes a process of recovering the virtual object from the superimposed position. The process may include: receiving a recycling instruction from the interaction device, and recycling the virtual object from the superimposed position, wherein the recycling effect may refer to a preset display effect, which will not be repeated here.

Therefore, when the target object is a virtual object displayed by the terminal device, the above-mentioned display steps can be used to enlarge the target object to generate a virtual object, and then when processing the target object, the target area can be more accurately acquired and processed The content data can expand the scope of application of the processing method of the virtual content.

Step S220: Determine six degrees of freedom (6DoF) information of the interactive device according to the collected marker image, where the marker image contains the marker.

In some embodiments, the number of markers of the interactive device may be multiple. Therefore, as a way, by identifying the position information, rotation direction and rotation angle of each marker relative to the terminal device in the plurality of markers, and according to the position information, rotation direction and rotation angle of each marker relative to the terminal device , to obtain the position and attitude information of the interaction device relative to the terminal device. For example, the markers of the interaction device recognized by the terminal device include a first marker and a second marker, the second marker is different from the first marker, and the terminal device can calculate the first marker and the second marker respectively and the terminal device. The relative positional relationship and rotation relationship between the two can be determined to determine the position and attitude information of the interaction device relative to the terminal device, so that the acquisition of the position and attitude information is more accurate.

In some embodiments, to identify the marker of the interaction device, the terminal device may first collect an image containing the marker through the image acquisition device, and then identify the marker in the image. Before the terminal device collects the image containing the marker, it can adjust the spatial position of the terminal device in the real space, or adjust the spatial position of the interaction device in the real space, so that the marker of the interaction device is in the terminal device. Within the field of view of the image acquisition device, the terminal device can perform image acquisition and image recognition on the marker. Wherein, the field of view of the image acquisition device may be determined by the orientation and size of the field of view angle.

In some embodiments, the interaction device may be provided with multiple markers, and the terminal device may use the marker within the visual field of the interaction device as the target marker. At this time, the terminal device collects an image containing the target marker. All the markers of the device are within the field of view of the image acquisition device, so the target markers in the image collected by the terminal device may be all the markers of the interaction device. If a part of the marker of the interaction device is within the field of view of the image acquisition device, the target marker in the image collected by the terminal device may be a part of the marker of the interaction device.

In other embodiments, when the terminal device recognizes the marker of the interaction device, an image containing the marker may be collected by other sensor devices, and then the terminal device recognizes the marker in the image. Wherein, the sensor device has the function of collecting the image of the marker, and can be a light sensor (eg, an infrared light receiver for receiving the infrared light reflected by the marker capable of reflecting infrared light) and the like. In other embodiments, the spatial position of the terminal device in the real space or the spatial position of the interaction device in the real space can also be adjusted, so that the marker of the interaction device is within the sensing range of the sensor device, thereby The terminal device can perform image acquisition and image recognition on the marker. The sensing range of the sensor device may be determined by the sensitivity. Further, when a plurality of markers are provided on the interaction device, the terminal device may use the marker of the interaction device within the sensing range of the sensor device as the target marker, wherein the target marker may be all markers of the interaction device, It can also be part of the marker of the interactive device.

In other embodiments, the interaction device may further include an inertial measurement sensor, and the inertial measurement sensor includes an inertial measurement unit (Inertial measurement unit, IMU). The IMU can detect the six-degree-of-freedom information of the interactive device, or can only detect the three-degree-of-freedom information of the interactive device. The three-degree-of-freedom information may include the rotational degrees of freedom of the interactive device along three rectangular coordinate axes (X, Y, and Z axes) in space, and the six-degree-of-freedom information may include the freedom of movement of the interactive device along the three rectangular coordinate axes in space. The movement degrees of freedom corresponding to the above three rectangular coordinate axes can constitute the position information of the interactive device, and the corresponding rotational degrees of freedom can constitute the attitude information of the interactive device. Therefore, the terminal device can obtain the attitude information of the interaction device detected by the IMU or the detected position and attitude information by receiving the sensing data of the above-mentioned inertial measurement sensor sent by the interaction device, and then obtain the information between the interaction device and the terminal device. Relative spatial positional relationship.

Further, in order to accurately obtain the position and attitude information of the interactive device, the terminal device can obtain the image including the interactive device and the sensing data of the inertial measurement sensor, so as to obtain the position of the interactive device according to the identification data of the image and the detection data of the IMU. and attitude information (that is, 6DoF information).

Step S230: Based on the 6DoF information, acquire a target area of the target object, where the target area is an area selected by the interaction device.

In this embodiment of the present application, the target object may include at least one of a virtual object displayed by a terminal device and a physical object in the real space. The virtual object displayed by the terminal device may refer to a three-dimensional virtual object projected to the human eye through the lens of the terminal device. At this time, the user can view the virtual object and the real space scene in front of the lens through the lens of the terminal device. Therefore, What the user observes is the virtual objects superimposed on the real space, such as virtual human body, virtual animal, virtual house, etc. Alternatively, the virtual object displayed by the terminal device may be a virtual object displayed by the terminal device using a mixed reality display technology (such as a holographic projection technology). , virtual animals, virtual houses, etc. The physical objects in the real space can be any real physical entities such as vehicles, books, posters, mobile terminals, people, and animals.

In some embodiments, the 6DoF information of the interactive device can be acquired in real time, and the selected target area in the target object can be determined according to the display position of the target object, thereby realizing the selection of the target area according to the 6DoF information of the interactive device. The target area is the area in the target object pointed to by the interactive device when the interactive device is opposite to the target object, which may correspond to the 6DoF information of the interactive device. The target area can be determined according to the user's will, that is, the user can determine the selected target area in the target object by changing the position and posture information of the interactive device. In some embodiments, for the terminal device to acquire the 6DoF information of the interaction device, reference may be made to the above-mentioned method for acquiring the 6DoF information of the interaction device, which will not be repeated here.

It can be understood that the angle between the direction in which the interaction device points and the plane where the interaction device is located is fixed. When the terminal device obtains the 6DoF information of the interaction device, the spatial position information of the interaction device can be obtained. The fixed angle relationship with the interactive device and the spatial position information of the interactive device can obtain the current direction of the interactive device, so that the selected target area in the target object can be obtained according to the display position of the target object and the current direction of the interactive device .

In some embodiments, specifically, referring to FIG. 8 , step S230 may include steps S231a-S232a.

Step S231a: Generate a virtual path according to the 6DoF information and the first spatial position relationship.

Wherein, the first spatial positional relationship is the spatial positional relationship between the virtual path and the interaction device, and the virtual path is used to represent the pointing direction of the interaction device. Therefore, the terminal device can acquire the selected target area in the target object through the virtual path of the interaction device. Specifically, the terminal device can generate and display a virtual path according to the first spatial position relationship and the 6DoF information of the interaction device, and then determine the intersection area of the target object that intersects the virtual path according to the displayed virtual path and the display position of the target object , and use the intersecting area as the selected target area in the target object.

As an embodiment, the first spatial position information may be preset spatial position information, and the preset spatial position information is used to determine the virtual path and the designated position of the interaction device, for example, the position of the interaction device relative to the virtual path. At this time, the direction in which the interaction device points is the guiding direction of the virtual path. In this embodiment of the present application, the virtual path may be a virtual ray emitted from the interaction device, and the virtual ray takes a point on the interaction device as the emission point, and its guiding direction toward the target object follows a preset first spatial position. The emission point may be a point where the virtual ray and the interactive device are superimposed and displayed. In some embodiments, the guiding direction of the virtual path may be parallel to the plane of the interaction device, or may be at a certain angle, such as 45° obliquely upward, which is not limited herein. Optionally, the virtual path can also be a virtual curve, and the guiding direction of the virtual curve can be changed according to the direction and magnitude of the curvature of the curve. In other embodiments, the virtual path may also be a virtual key head, a virtual indicator mark, etc. The form of the virtual path is not limited in the embodiments of the present application, and it is only necessary that the movement of the virtual path corresponds to the change of the 6DoF information of the interactive device. .

In some embodiments, the direction or shape of the virtual path can be adjusted according to the touch command received by the interactive device. For example, the touch command received by the interaction device may be acquired, the first spatial position relationship may be adjusted according to the touch command, and the virtual path may be generated according to the adjusted first spatial position relationship and the 6DoF information. Specifically, the touch command can change the position of the point on which the virtual path is superimposed and displayed on the interactive device, and can also change the guiding direction of the virtual path. For example, referring to FIG. 9, the relative angle between the virtual path and the interactive device can be changed according to the user's operation action, thereby changing the guiding direction of the virtual path, and adjusting the virtual path with the guiding direction A to the virtual path with the guiding direction B path.

In some embodiments, through the 6DoF information of the interactive device, a virtual space coordinate system can be established with the terminal device as a reference point, and the spatial position coordinates of the interactive device in the virtual space coordinate system can be obtained, and then the virtual space can be rendered according to the spatial position coordinates. path. After the terminal device renders the virtual path, the virtual path can be displayed. Through the display lens of the head-mounted display device, the user can see that the virtual path is superimposed and displayed on the interactive device in the real world to realize the effect of augmented reality.

The terminal device can obtain the intersection area of the target object that intersects the virtual path according to the display position of the target object and the displayed virtual path, and use the intersection area as the selected target area in the target object. In some embodiments, the above-mentioned acquisition of the intersection area in the target object intersecting with the virtual path may be the acquisition of the coordinate point area in the target object with the same coordinates as the virtual path, and the coordinate point area may be directly used as the above-mentioned intersection area, or may be The area where the corresponding target object is located is obtained according to the coordinate point area, and the area where the target object is located is taken as the intersecting area, that is, the selected target area in the target object. Wherein, the area where the corresponding target object is obtained according to the coordinate point area may be the area where the closest target object is located according to the coordinate point area. In this way, the user changes the display position of the virtual path, so that the displayed virtual path can intersect with the target area to be selected, so that the target area can be accurately selected in the target object.

Step S232a: Determine the area of the target object pointed to by the virtual path as the target area.

In the embodiment of the present application, the target object is divided into multiple areas, and the division of the area can be divided according to the spatial position, for example, divided into four areas, upper, lower, left and right; it can also be divided according to the structure of the target object itself, for example, please refer to In FIG. 9 , when the virtual object is a car 400 , the area divided by the car 400 may include multiple areas such as the front wheel 430 , the rear wheel 450 , the rearview mirror 470 and the headlight 490 , which are only exemplified here.

Further, the guiding direction of the virtual path points from the interactive device to the target object, and intersects the target object. Wherein, if the intersection of the virtual path and the surface of the target object is a point, it is determined that the area where the point is located is the target area of the target object.

In other embodiments, please refer to FIG. 10 , the target area is the area blocked by the interaction device, that is, according to the 6DoF information, the area where the target object is blocked by the interaction device is determined as the target area. Please refer to FIG. 11. Specifically, step 230 further includes steps S231b-S233b. In this case, steps S231a-S232a may be omitted, that is, steps S231b-S233b may be alternatives to steps S231a-S232a.

Step S231b: Based on the 6DoF information, obtain the first relative positional relationship between the interaction device and the target object.

In the embodiment of the present application, based on the same spatial coordinate system, taking the virtual space coordinate system as an example, the spatial coordinates of the interaction device and the target object can be obtained respectively, and then the interaction device and the target object can be determined according to the spatial coordinates of the interaction device and the target object. the first relative positional relationship. The first relative positional relationship between the interaction device and the target object may be the relative positional relationship between the interaction device and the physical object, or the relative positional relationship between the interaction device and the virtual object.

Specifically, the terminal device can obtain the 6DoF information of the interactive device relative to the terminal device by identifying the marker of the interactive device. Therefore, the terminal device can obtain the spatial position coordinates of the interactive device in the real space, and convert the spatial position coordinates into Space coordinates in the virtual space coordinate system. In some embodiments, the spatial coordinates of the interactive device may be three-dimensional spatial coordinates of the interactive device in a virtual space coordinate system with a specified point (eg, a virtual camera, which can also be regarded as the point where the human eye is located) as the origin. Similarly, the terminal device also obtains the spatial coordinates of the target object in the virtual space coordinate system according to the positional relationship between the target object and the terminal device. It can be understood that the origin of the virtual space coordinate system can also be any point in the virtual space, and a space coordinate system is established based on this point, and the three-dimensional space coordinates of the interactive device and the target object in the same space coordinate system are obtained. The space coordinate system may also be a space coordinate system in a real space, etc., which is not limited here.

In some embodiments, if the target object is a virtual object that the terminal device has displayed in the virtual space coordinate system, the terminal device can directly obtain the second relative positional relationship between the virtual object and the terminal device in the virtual space coordinate system . The terminal device can obtain the spatial position coordinates of the interaction device in the real space according to the position and attitude information of the interaction device relative to the terminal device, and convert the spatial position coordinates into the spatial coordinates in the virtual space coordinate system. The terminal device can obtain the virtual space coordinate system based on the second relative positional relationship between the virtual object and the terminal device in the virtual space coordinate system and the spatial coordinates of the interaction device in the virtual space coordinate system, using the terminal device as a reference. The first relative positional relationship between the interaction device and the virtual object.

In some embodiments, if the target object is an entity object in the real space, the terminal device needs to acquire the spatial position information of the entity object to obtain the first relative position relationship between the interaction device and the entity object. Specifically, the terminal device can obtain the identification result of the entity object by identifying the entity object. The identification result includes at least the shape and size of the entity object, and the spatial position relationship of the entity object relative to the terminal device, so that the terminal device can obtain the entity object and the entity object. The second relative positional relationship of the terminal device. Wherein, the second relative position relationship may include the position, rotation direction, rotation angle, etc. of the entity object relative to the terminal device.

Optionally, when the terminal device acquires the positional relationship of the entity object relative to the terminal device, an image containing the entity object can be captured by an image acquisition device (such as a stereo camera such as a depth camera), and then the entity object in the image can be identified. In one embodiment, the terminal device collects an image containing the entity object, and the image can be uploaded to the server, and after the server recognizes the entity object in the image, the identification result can be returned to the terminal device.

Alternatively, the terminal device obtains the positional relationship of the entity object relative to the terminal device, and can also obtain the positional relationship of the entity object relative to the terminal device by identifying markers set on the entity object (eg, markers pasted or printed on the entity object). Further, after the terminal device recognizes the entity object, it can also obtain the detailed information of the entity object (such as name, category, color, pattern, etc.), that is, after the terminal device recognizes the entity object or recognizes the image containing the entity object, The positional relationship of the entity object relative to the terminal device and the detailed information of the entity object can be obtained.

Since the recognition result obtained by the terminal device recognizing the entity object includes the positional relationship of the entity object relative to the terminal device, the terminal device can obtain the positional relationship of the entity object relative to the terminal device according to the position and attitude information of the interaction device relative to the terminal device and the positional relationship of the entity object relative to the terminal device. The first relative positional relationship between the interactive device and the entity object. Specifically, for example, a spatial coordinate system is established with the terminal device as the origin, the spatial coordinates of the interaction device are obtained based on the position information of the interaction device relative to the terminal device, and the spatial coordinates of the entity object are obtained based on the position information of the entity object relative to the terminal device. Therefore, the first relative positional relationship between the interaction device and the physical object in the virtual space coordinate system can be obtained. The above-mentioned first relative positional relationship may be the relative positional relationship between the interaction device and the physical object in the virtual space coordinate system, or may be the relative position of the interaction device and the physical object in the real world viewed by the user through the head-mounted display device relation.

Step S232b: Based on the first relative positional relationship, determine an occlusion area of the target object occluded by the interaction device.

Since the interactive device and the target object have been placed in the same spatial coordinate system in step S231b, the spatial coordinates of each point of the interactive device are compared with the spatial coordinates of each point of the target object, and it is determined that the target object has at least two coordinates with the interactive device. points with coincident coordinate values, and the spatial coordinates of the points with at least two coincident coordinate values are classified into the occlusion coordinate set. Further, if there is an occlusion coordinate set, it can be determined that there is an occlusion between the interaction device and the target object, and an area corresponding to the occlusion coordinate set is determined as an occlusion area. For example, it is assumed that the space coordinate system established with the terminal device as the origin is the XYZ space coordinate system. If the depth value is represented by the Z axis, it is determined whether there is a point where the two coordinate values of X and Y overlap in the occlusion coordinate set. If there is a point where the two coordinate values of X and Y coincide, it can be determined that the interaction device and the target object are blocked. At this time, the interaction device and the target object have an occlusion area in the X-Y plane. In this case, the direction in which the interaction device points is the direction in which the occlusion area is located.

Further, the specific occlusion relationship between the interaction device and the target object is determined by the depth relationship between the interaction device and the target object by occlusion coordinates, such as determining that the interaction device occludes the target object, the target object occludes the interaction device, and the like. For example, if the Z-axis is used as the depth value, the point where the X and Y values of the interactive device and the target object are equal in the occlusion coordinate set are obtained, and the Z value of the coordinates of the point where the X and Y values of the interactive device and the target object are equal, are compared. The depth relationship between the occlusion area and the target object can be determined. For example, when point A (X1, Y1, Z1) of the interactive device is in the occlusion coordinate set, and point B (X2, Y2, Z2) of the target object is in the occlusion coordinate set, if X1=X2, Y1=Y2, it is considered that Point B and point A have an occlusion relationship in the depth direction of the Z axis. At this time, by comparing the sizes between Z1 and Z2, the specific occlusion relationship between the interactive device and the target object can be determined. If Z2 is greater than Z1, then It is determined that the interactive device blocks the target object, and if Z2 is smaller than Z1, the target object blocks the interactive device and so on.

Further, the terminal device can also obtain the depth value of the interaction device according to the position and attitude information of the interaction device relative to the terminal device, wherein the depth value is the distance value between the object and the terminal device in depth, which can also be understood as the distance of the object. The near and far status of the terminal device. The terminal device can also obtain the depth value of the target object according to the positional relationship between the target object and the terminal device, and then compare the depth value between the interaction device and the target object to obtain the depth relationship between the interaction device and the target object. Therefore, the terminal device can determine the occlusion relationship between the interaction device and the target object according to the depth relationship, and usually a relatively distant object is easily occluded by a relatively close object. In some embodiments, when the depth value of the target object is greater than the depth value of the interaction device, it may be considered that the interaction device blocks the target object. Similarly, if the depth value of the interaction device is greater than the depth value of the target object, it can be considered that the target object blocks the interaction device. Of course, the method for calculating the mutual occlusion relationship between objects may also be other methods, such as cross-checking, depth measurement, etc., which are not limited here.

Step S233b: Determine the occlusion area as the target area.

In this embodiment of the present application, after acquiring the occlusion area of the target object that is occluded by the interaction device, the terminal device may determine the occlusion area as the target area.

It should be noted that, in some embodiments, step S230 may include steps S231a-S232a and steps S231b-S233b at the same time, so that the user can select an appropriate solution to acquire the target area of the target object, and provides a method for acquiring the target area. There are many possible ways to improve the interactivity between the user and the displayed content.

Step S240: Determine content data corresponding to the target area.

In this embodiment, the content data corresponding to the target area may include three-dimensional structure information of the target object corresponding to the target area, wherein the three-dimensional structure information at least includes external structure information and internal structure information of the target object in the target area.

Specifically, the external structure information may be the appearance information of the target object in the target area, such as line, shape, color, size, etc., and the internal structure information may be the internal structure information of the target object in the target area, such as filling, internal structure, etc. For example, if the target object is the car 400 shown in FIG. 12 , the target area is the target area 410 (dotted line frame), and the obtained external structure information is the outer contour of the car 400 in the target area 410 (such as the front wheel , lights, hood, etc.), the internal structure information is the internal structure of the car 400 in the target area 410, such as the engine, transmission, bumper and other mechanical structures.

Step S250: Generate virtual content according to the 6DoF information and content data.

In this embodiment of the present application, the virtual content includes content data (eg, three-dimensional structure information) to form information of different levels. Information of different levels can be layered according to the relative positional relationship between the content data and the interactive device, that is, the layering can be performed according to the corresponding relationship between the acquired content data and the 6DoF information.

Specifically, since the terminal device has acquired the 6DoF information of the interaction device, that is, the position and attitude information of the interaction device relative to the terminal device. Therefore, a virtual space coordinate system is established based on the terminal device, wherein the terminal device can obtain the space position coordinates of the interaction device in the real space, and convert the space position coordinates into the space coordinates in the virtual space coordinate system. In the virtual space coordinate system, with the virtual camera as the origin, according to the positional relationship between each part of the structure (such as the external structure and the internal structure) in the target area and the interactive device, the relative position of each part of the structure in the target area relative to the virtual camera can be obtained. Spatial position, from the spatial position coordinates of each part of the structure in the virtual space coordinate system to be obtained. Finally, according to the spatial position coordinates of the interactive device and the spatial position coordinates of each partial structure in the target area, the relative positional relationship information of the interactive device relative to each partial structure in the target area can be collected.

Further, the content data is layered according to the relative positional relationship (eg depth information) of each part structure in the target area and the interaction device. Among them, the depth information is used to indicate the distance between each part of the structure in the target area relative to the interactive device. For example, according to the depth information, at least the acquired content data of each part of the structure in the target area can be divided into two different levels of information: external structure information and internal structure information, and one level of information is used to represent the external structure information. , another level of information is used to represent the internal structure of the information. In some embodiments, the external structure information and the internal structure information may also be represented by multiple levels of information.

Step S260: Display the virtual content.

Further, if the information of different levels of the virtual content is displayed on the 3D model interface, the information of different levels is mainly presented as disassembling the 3D model, for example, the 3D model is divided into the form of outer contour and inner structure. In other embodiments, if the information of different levels of the virtual content is displayed in a two-dimensional image interface, the information of different levels is mainly presented so that the two-dimensional image can be switched, for example, the target object is displayed in the target area through different image interfaces outer contour and inner structure information.

As an embodiment, if a virtual display area is generated based on the 6DoF information of the interaction device, the virtual content generated by the terminal device is displayed in the virtual display area. Further, the second relative positional relationship between the virtual display area and the interactive device can be adjusted according to the user's viewing habits, so as to facilitate the use of the user. At this time, step S260 may include: generating a virtual display area based on the 6DoF information of the interactive device, determining the relative spatial relationship between the virtual display area and the interactive device according to the user adjustment instruction, presenting the virtual display area and displaying the virtual content in the virtual display area .

In this embodiment of the present application, since the virtual content includes information at different levels, the user can have a clearer understanding of the target object in the target area by browsing the displayed information at different levels. Therefore, in order to present different levels of information, step S261 may also be included after displaying the virtual content.

Step S261: Receive the second display instruction sent by the interaction device, and display the information of the corresponding level according to the second display instruction.

Wherein, the second display instruction corresponds to the user's operation action, and the second display instruction includes but is not limited to an instruction to switch information at a lower level, an instruction for switching information at a previous level, and an instruction for switching information at any level. Taking the instruction to switch information of any level as an example, if the current display interface has an area marked with the number of the information of the current level, the user can input the number of the level of information to be switched, and then send the instruction to switch the information of any level to change the current display interface. Switch to the interface containing the input level information to be switched.

Step S270: Receive an operation instruction sent by the interaction device.

In the embodiment of the present application, the operation instruction sent by the interaction device is based on the operation actions input by the user on the interaction device, such as operation actions such as clicking, sliding, and multi-touch. That is, the interaction device can also detect different operation action parameters (such as a touch position parameter, a touch count parameter) on the interaction device, and send an operation command that fails. The interaction device converts the user's operation action information into an operation instruction, and sends the operation instruction to the terminal device. Step S280: Determine the corresponding content processing instruction based on the operation instruction.

After receiving the operation instruction sent by the interaction device, the terminal device determines the corresponding content processing instruction according to predetermined data analysis and data processing. The types of content processing instructions at least include instructions such as scaling, modifying, marking, moving, and rotating.

Further, in some embodiments, according to different virtual contents, the same operation instruction may correspond to different content processing instructions, then after receiving the operation instruction sent by the interactive device, according to the currently displayed virtual content and the operation instruction, generate The content processing instruction corresponding to the manipulation instruction. For example, it is also an operation instruction of selecting and sliding. When the selected object is a car, the content processing instruction is to drag the car; when the selected object is a car lamp, the content processing instruction is to adjust the brightness of the car lamp.

Step S290: Process the virtual content according to the content processing instruction, and generate corresponding processing content.

Referring to FIG. 13 , in some embodiments, when the terminal device displays virtual content, when the operation instruction input by the user on the interaction device is to press and drag the virtual image with one finger, a content processing instruction for moving the virtual content is generated, For example, the content processing instruction is to control the terminal device to move the currently displayed car to the left or right.

Referring to FIG. 14, in some embodiments, when the terminal device displays virtual content, when the operation instruction input by the user on the interaction device is the relative shrinking and merging of the distance between two fingers, a content processing instruction for reducing the currently displayed virtual content is generated. , the content processing instruction is to control the terminal device to reduce the viewing angle of the currently displayed car relative to the user. If the operation command input by the user on the interaction device is that the distance between the two fingers is relatively widened, a content processing command for enlarging the currently displayed virtual content is generated. enlarge.

Referring to FIG. 15 , in some embodiments, when the terminal device displays virtual content, when the operation instruction input by the user on the interaction device is to double-click or multiple-click on a certain virtual content, an operation instruction for modifying or marking the virtual content is generated. The content processing instruction, for example, the content processing instruction is to control the terminal device to generate a data frame or text frame around the selected virtual content, and further detect the user's operation to modify the parameters of the data frame or input text to the text frame. The data in the data frame may be parameters related to virtual content, such as production date, usage time, manufacturer, and the like.

Step S300: Obtain a second spatial positional relationship of the target area relative to the terminal device, and superimpose and display the processing content on the target area according to the second spatial positional relationship.

Wherein, if the target object is a virtual object, the second spatial position relationship can be directly obtained from the virtual space coordinate system, and the processing content of the virtual content of the target area is superimposed on the target area according to the second spatial position relationship.

If the target object is an entity object, the second spatial positional relationship can be obtained by identifying the entity object by the terminal device. Optionally, the second relative positional relationship between the terminal device and the physical object may be acquired, and the second spatial positional relationship between the target area and the terminal device may be obtained according to the position of the target area on the physical object. Further, the processing content of the virtual content in the target area is superimposed on the target area of the physical object according to the second spatial positional relationship.

In the embodiment of the present application, taking the content processing instruction for marking virtual content as an example, the target object is a virtual object or a physical object, and the user can observe the content marking the virtual content through the terminal device. Correspondingly, the virtual content is processed according to content instructions such as modification, movement, rotation, etc., and the processed content is also superimposed on the target area according to the second spatial positional relationship, and the modification, movement, and modification of the virtual content in the target area can be observed through the terminal device. Rotation, etc. For example, please refer to FIG. 14 and FIG. 16 together, when the content processing instruction is to generate and shrink the currently displayed virtual content, the virtual content displayed on the interactive device is reduced (as shown in FIG. 14 ), and at the same time, the target object in the target area is reduced. It also shrinks accordingly (as shown in Figure 16).

In addition, in the above steps, the virtual content may include a user interface (User Interface, UI), and the content processing instruction may be an instruction to modify UI information, so the terminal device may enhance the relevant data (such as UI data) of the physical object. Reality display to enable interaction with real objects. Through the above-mentioned control method for the virtual content, the terminal device can further operate the virtual content corresponding to the physical object, so as to realize further interaction with the real object.

In some embodiments, when the entity object is a smart home device, the terminal device can also set the state of the smart home device through the interaction device, so that the user can interact with the smart home device through the interaction device. Specifically, the smart home device can be used as the target object, and the terminal device can obtain the virtual content corresponding to the target area of the smart home device, and according to the content processing of the virtual content, generate an execution instruction, and transmit the execution instruction to the smart home device. , the execution instruction is used to instruct the smart home device to perform the setting operation.

In some embodiments, the terminal device can identify the smart home device to display the virtual interactive interface (virtual UI) of the smart home device in the virtual space, and then the user can interact with the virtual interactive device through the input operation action on the interactive device. Different virtual content in the interface can be selected to set the status of smart home devices or control smart home devices. Wherein, the virtual interactive interface can be superimposed and displayed on the interactive device, or superimposed and displayed on the smart home device. In some embodiments, when the terminal device displays the virtual interactive interface of the smart home device, the virtual interactive interface can be operated through the interactive device. The virtual content is a part of the displayed virtual interactive interface. For example, referring to FIG. 17 , the virtual content 900 displayed by the terminal device is the virtual control interface of the smart desk lamp, the target area 910 is the brightness option, and the virtual content corresponding to the target area 910 is the specific brightness value.

In some embodiments, the terminal device can adjust the state of the smart home device according to the processing content. Specifically, the terminal device can generate a corresponding execution instruction according to the processing content corresponding to the target area (eg, the modification content of the UI information), and the execution instruction is used to adjust the state of the smart home device to a state corresponding to the virtual content. For example, when the processing content is 50 brightness, the terminal device can generate an execution instruction to adjust the brightness of the smart desk lamp to 50. In some embodiments, the processing content may correspond to an execution instruction, that is, when the terminal device obtains the processing content of the interaction device, it may generate an execution instruction corresponding to the processing content according to the correspondence between the processing content and the execution instruction. The corresponding relationship between the processing content and the execution instruction may be stored in the terminal device, or may be acquired from the server.

After generating the above execution instruction, the terminal device may transmit the execution instruction to the smart home device, where the execution instruction is used to instruct the smart home device to perform the setting operation. When the smart home device receives the execution instruction, it can perform a setting operation according to the execution instruction, so as to adjust the current state to the state set by the user, that is, the state corresponding to the processing content. Thus, the interaction between the interactive device and the smart home equipment is realized, and the interaction level between the interactive device and the smart home is improved.

In the method for processing virtual content provided by the embodiment of the present application, the target area of the target object is determined according to the 6DoF information of the interactive device, and the content data of the target area is collected, and then corresponding virtual content is generated and displayed according to the 6DoF information and the content data. The received content processing instruction processes the virtual content, and finally obtains the second spatial positional relationship of the target area relative to the terminal device, and superimposes and displays the processed content to the target area according to the second spatial positional relationship. Therefore, the user can select the virtual content to be processed through the 6DoF information of the interactive device, and directly process the virtual content through the interactive device, and at the same time superimpose and display the processed content to the target area, which can improve the user and the displayed virtual content. The interactivity between them increases the sense of immersion.

Please refer to FIG. 18 , which shows a structural block diagram of an apparatus 500 for processing virtual content provided by an embodiment of the present application, which is applied to a terminal device to execute the above-mentioned method for processing virtual content. The virtual content processing apparatus 500 may include: an information determination module 510 , an area determination module 520 , a data acquisition module 530 , a content generation module 540 , a content display module 550 and a content processing module 560 . It can be understood that the above-mentioned modules can be program modules running in a computer-readable storage medium, and the purpose and work of the above-mentioned modules are as follows: The information determination module 510 is used to determine the six aspects of the interaction device according to the collected marker images. Degree of freedom (6DoF) information, the marker image includes markers set in the interactive device; the area determination module 520 is used to obtain the target area of the target object based on the 6DoF information, and the target area is the area selected by the interactive device; the data acquisition module 530 For acquiring the content data corresponding to the target area; the content generation module 540 is used for generating virtual content according to the 6DoF information and the content data; the content display module 550 is used for displaying the virtual content; and the content processing module 560 is used for receiving the control data sent by the interactive device , generate corresponding content processing instructions according to the control data, and process the virtual content according to the content processing instructions.

In some embodiments, the area determination module 520 further includes a virtual path unit 521 and an occlusion determination unit 523 . The virtual path unit 521 is configured to generate a virtual path based on the 6DoF information, the virtual path is a path from the interactive device to the target object, and determine the area of the target object pointed to by the virtual path as the target area. The occlusion judging unit 523 is used to obtain the first relative positional relationship between the interaction device and the target object based on the 6DoF information, and then based on the first relative positional relationship, determine the occlusion area occluded by the target object and the interaction device, and determine the occlusion area as the target area .

In some embodiments, the content processing module 560 further includes an operation instruction receiving unit 561 , a content processing instruction determining unit 563 , a generating and processing content unit 565 and a superimposing processing content unit 567 . The operation instruction receiving unit 561 is configured to receive an operation instruction sent by the interaction device. The content processing instruction determining unit 563 is configured to determine the corresponding content processing instruction based on the operation instruction. The content unit 565 for generating processing unit is configured to process the virtual content according to the content processing instruction, and generate corresponding processing content. The superimposing processing content unit 567 is configured to acquire the second spatial positional relationship of the target area with respect to the terminal device, and superimpose and display the processing content on the target area according to the second spatial positional relationship.

In other embodiments, if the target object is a virtual object, the processing device 500 further includes an object display module 570, and the object display module 570 is configured to display the virtual object to the display area. Specifically, the object display module 570 further includes a display area determination unit 571 , a first display instruction receiving unit 573 and an instruction executing unit 575 . The display area determination unit 571 is used to determine the display area. The display area determination unit 571 may include: acquiring an image containing the display marker; identifying the display marker in the image, and acquiring the relative spatial positional relationship between the display marker and the terminal device; and determining the display area based on the relative spatial positional relationship. The first display instruction receiving unit 573 is configured to receive the first display instruction sent by the interactive device. The instruction execution unit 575 is configured to display the virtual object in the display area according to the preset display effect based on the first display instruction.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the specific working process of the above-described devices and modules, reference may be made to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

In several embodiments provided in this application, the coupling or direct coupling or communication connection between the modules shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or modules may be electrical, mechanical or otherwise.

In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically alone, or two or more modules may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware, and can also be implemented in the form of software function modules.

To sum up, a method and device for processing virtual content provided by the embodiments of the present application are applied to terminal equipment, and the target area of the target object is determined according to the 6DoF information of the interaction device, and the content data of the target area is collected, and then according to the 6DoF information Generate and display the corresponding virtual content with the content data, process the virtual content according to the received content processing instruction, and finally obtain the second spatial positional relationship of the target area relative to the terminal device, and superimpose and display the processed content according to the second spatial positional relationship to the target area. Therefore, the user can process the displayed virtual content through the 6DoF information of the interactive device, and at the same time, the processed content can be superimposed and displayed to the target area, which can improve the interaction between the user and the displayed virtual content and improve the immersion.

Please refer to FIG. 19 , which shows a structural block diagram of a terminal device provided by an embodiment of the present application. The terminal device 100 may be a terminal device capable of running an application program, such as a smart phone, a tablet computer, a head-mounted display device, or the like. The terminal device 100 in this application may include one or more of the following components: a processor 110, a memory 120, an image capture device 130, and one or more application programs, wherein one or more application programs may be stored in the memory 120 and Configured to be executed by one or more processors 110, the one or more programs are configured to perform the method as described in the foregoing method embodiments.

The processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire terminal device 100, and executes by running or executing the instructions, programs, code sets or instruction sets stored in the memory 120, and calling the data stored in the memory 120. Various functions of the terminal device 100 and processing data. Optionally, the processor 110 may employ at least one of a digital signal processing (Digital Signal Processing, DSP), a Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), and a Programmable Logic Array (Programmable Logic Array, PLA). implemented in hardware. The processor 110 may integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU), a modem, and the like. Among them, the CPU mainly handles the operating system, user interface and application programs, etc.; the GPU is used for rendering and drawing of the display content; the modem is used to handle wireless communication. It can be understood that, the above-mentioned modem may also not be integrated into the processor 110, and is implemented by a communication chip alone.

The memory 120 may include random access memory (Random Access Memory, RAM), or may include read-only memory (Read-Only Memory). Memory 120 may be used to store instructions, programs, codes, sets of codes, or sets of instructions. The memory 120 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.) , instructions for implementing the following method embodiments, and the like. The storage data area may also store data and the like created by the terminal device 100 in use.

In this embodiment of the present application, the image acquisition device 130 is configured to acquire an image of a target object and a scene image of a target scene. The image acquisition device 130 may be an infrared camera or a color camera, and the specific camera type is not limited in this embodiment of the present application.

Please refer to FIG. 20 , which shows a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application. The computer-readable storage medium 800 stores program codes, and the program codes can be invoked by the processor to execute the methods described in the above method embodiments. The computer readable storage medium 800 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 800 includes a non-transitory computer-readable storage medium. Computer readable storage medium 800 has storage space for program code 810 to perform any of the method steps in the above-described methods. These program codes can be read from or written to one or more computer program products. Program code 810 may be compressed, for example, in a suitable form.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand: it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or some technical features thereof are equivalently replaced; and these modifications or replacements do not drive the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims (11)

1. A processing method of virtual content is applied to terminal equipment, and is characterized by comprising the following steps:

determining six-degree-of-freedom (6DoF) information of an interactive device according to an acquired marker image, wherein the marker image comprises a marker arranged on the interactive device;

acquiring a target area of a target object based on the 6DoF information, wherein the target area is an area selected by the interaction device;

acquiring content data corresponding to the target area;

generating virtual content according to the 6DoF information and the content data;

displaying the virtual content; and

and receiving control data sent by the interactive device, generating a corresponding content processing instruction according to the control data, and processing the virtual content according to the content processing instruction.

2. The method of claim 1, wherein the obtaining a target region of a target object based on the 6DoF information comprises:

generating a virtual path according to the 6DoF information and a first spatial position relationship, wherein the first spatial position relationship is a spatial position relationship between the virtual path and the interactive device, and the virtual path is used for representing an indication direction of the interactive device;

and determining the area of the target object pointed by the virtual path as a target area.

3. The method according to claim 2, further comprising, after the generating a virtual path according to the 6DoF information and the first spatial position relationship:

acquiring a touch control command received by the interactive device; and

and adjusting the first spatial position relation according to the touch command, and generating the virtual path according to the adjusted first spatial position relation and the 6DoF information.

4. The method of claim 1, wherein the obtaining a target region of a target object based on the 6DoF information comprises:

acquiring the relative position relation between the interaction device and the target object based on the 6DoF information;

determining an occlusion area in which the target object is occluded by the interaction device based on the relative position relation; and

and determining the occlusion area as the target area.

5. The method of claim 4, wherein the obtaining the relative positional relationship of the interaction device and the target object comprises:

respectively acquiring the space coordinates of the interaction device and the target object based on the same space coordinate system; and

determining the relative position relation between the interaction device and the target object according to the space coordinates of the interaction device and the target object;

the determining an occlusion region in which the target object is occluded by the interaction device includes:

acquiring the space coordinates of each point of the interaction device and the space coordinates of each point of the target object in the space coordinate system,

comparing the space coordinates of each point of the interaction device with the space coordinates of each point of the target object, determining a point in the target object, which has at least two coincident coordinate values with the interaction device, and classifying the space coordinates of the point having at least two coincident coordinate values into an occlusion coordinate set; and

and determining the area corresponding to the occlusion coordinate set as an occlusion area.

6. The method of claim 1, wherein the target object is a virtual object displayed by the terminal device, and before the determining the target area of the target object, the method comprises:

determining a superimposed position of the virtual object in a real environment;

receiving a first display instruction sent by the interaction device;

and displaying the virtual object according to a preset display effect based on the first display instruction, wherein the display position of the virtual object corresponds to the superposition position.

7. The method according to any one of claims 1 to 6, wherein the content data includes three-dimensional structure information of the target object corresponding to the target area, the three-dimensional structure information including at least external structure information and internal structure information of the target object in the target area;

generating virtual content according to the 6DoF information and the content data, comprising:

generating virtual content according to the 6DoF information and the three-dimensional structure information, wherein the virtual content comprises information of different layers formed by the three-dimensional structure information;

the displaying the virtual content includes:

and receiving a second display instruction sent by the interaction device, and displaying information of a corresponding layer according to the second display instruction.

8. The method according to any one of claims 1 to 6, wherein the receiving control data sent by the interactive device, generating a corresponding content processing instruction according to the control data, and processing the virtual content according to the content processing instruction comprises:

receiving an operation instruction sent by the interaction device;

determining a corresponding content processing instruction based on the operation instruction;

processing the virtual content according to the content processing instruction and generating corresponding processing content; and

and acquiring a second spatial position relation of the target area relative to the terminal equipment, and displaying the processing content to the target area in an overlapping manner according to the second spatial position relation.

9. An apparatus for processing virtual content, the apparatus comprising:

the information determining module is used for determining six-degree-of-freedom (6DoF) information of the interactive device according to the collected marker image, and the marker image comprises a marker arranged on the interactive device;

a region determining module, configured to obtain a target region of a target object based on the 6DoF information, where the target region is a region selected by the interaction device;

the data determining module is used for acquiring content data corresponding to the target area;

the content generation module is used for generating virtual content according to the 6DoF information and the content data;

a content display module for displaying the virtual content; and

and the content processing module is used for receiving the control data sent by the interactive device, generating a corresponding content processing instruction according to the control data, and processing the virtual content according to the content processing instruction.

10. A terminal device, comprising:

one or more processors;

a memory;

one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to perform the method of any of claims 1-8.

11. A computer-readable storage medium having program code stored therein, the program code being invoked by a processor to perform the method of any one of claims 1 to 8.

Images