CN116124119A - A positioning method, positioning device and system - Google Patents

Abstract

The application provides a positioning method, positioning equipment and a positioning system, wherein the method comprises the following steps: the positioning device tracks the virtual reality device; the positioning device projects the coded image through a projector; the positioning device receives a plurality of light intensity values from the virtual reality device; the positioning device determines the pose of the virtual reality device relative to the positioning device according to the projected coded image, a plurality of light intensity values and the positions of a plurality of light intensity sensors in the virtual reality device; the pose of the virtual reality device relative to the positioning device and the pose of the positioning device in the environment coordinate system are used to determine the pose of the virtual reality device in the environment coordinate system. By implementing the embodiment of the application, the positioning equipment can move along with the virtual reality equipment, and the pose of the virtual reality equipment is determined in the moving process.

Description

A positioning method, positioning device and system

technical field

The embodiments of the present application relate to the field of electronic technologies, and in particular, to a positioning method, positioning device, and system.

Background technique

Virtual Reality (VR) technology is a computer simulation technology that can create and experience virtual scenes. It uses computers to generate a virtual scene and immerses users in the virtual scene.

Virtual reality technology can use data in real life to generate electronic signals through computer technology, and combine it with various virtual reality devices (such as head-mounted display devices and handles, etc.) to convert them into virtual scenes. The device performs information interaction with this virtual scene. In order to realize the interaction between the user and the virtual scene, the virtual reality device needs to recognize its position and posture in the real environment, so as to determine the virtual reality content corresponding to the position and posture of the virtual reality device.

At present, due to the wearing comfort and weight requirements of the head-mounted display device, too many positioning devices cannot be installed, and the virtual reality device is often positioned by a laser scanner or a camera fixed in the space. This method largely limits the range of activities of the user when wearing the virtual reality device, which hinders the improvement of the user's sense of experience.

How to expand the user's range of activities in the virtual scene and enhance the user's immersive experience is the current and future research direction.

Contents of the invention

This application provides a positioning method, a positioning device and a system. The positioning device can track the virtual reality device. The positioning device determines the indoor position of the virtual reality device through its own indoor pose and the relative pose of itself and the virtual display device. posture. The method can expand the range of activities of the user in the virtual scene, and improve the user's sense of immersive experience.

In the first aspect, the embodiment of the present application provides a positioning method, which is applied to a positioning device, where the positioning device includes a projector, and the method includes:

The positioning device tracks the virtual reality device;

The positioning device projects the encoded image through the projector;

The positioning device receives multiple light intensity values from the virtual reality device; the multiple light intensity values include light intensities of encoded images respectively received by multiple light intensity sensors of the virtual reality device;

The positioning device determines the pose of the virtual reality device relative to the positioning device according to the projected coded image, multiple light intensity values and the positions of multiple light intensity sensors in the virtual reality device; the pose and pose of the virtual reality device relative to the positioning device The pose of the positioning device in the environment coordinate system is used to determine the pose of the virtual reality device in the environment coordinate system.

Implementing the embodiment of the present application, the positioning device can track the virtual reality device, and the positioning device determines the indoor pose of the virtual reality device through its own indoor pose and the relative pose between itself and the virtual display device. The positioning device can move along with the virtual reality device, and determine the pose of the virtual reality device during the movement process. This method can expand the user's range of activities and improve the user's sense of experience.

In combination with the first aspect, in a possible implementation manner, the method further includes:

The positioning device determines the pose of the virtual reality device in the environment coordinate system according to the pose of the positioning device in the environment coordinate system and the pose of the virtual reality device relative to the positioning device;

The positioning device sends the pose of the virtual reality device in the environment coordinate system to the virtual reality device.

With reference to the first aspect, in a possible implementation manner, the coded image includes multiple coded images; the positioning device determines the The pose of the virtual reality device relative to the positioning device, including:

The positioning device generates the code of the light intensity sensor based on the light intensity value of each encoded image received by a light intensity sensor;

The positioning device determines the pixel coordinates of the multiple light intensity sensors based on the codes of the multiple light intensity sensors;

The positioning device determines the pose of the virtual reality device relative to the projector based on the pixel coordinates of the multiple light intensity sensors and the positions of the multiple light intensity sensors in the virtual reality device;

Based on the pose of the projector relative to the positioning device, the positioning device converts the pose of the virtual reality device relative to the projector into a pose of the virtual reality device relative to the positioning device.

In combination with the first aspect, in a possible implementation manner, the multiple coded images include M first images and N second images, M is a positive integer, N is a positive integer, and the pattern of the first image is the first direction The second image is a binary image of stripes in the second direction; the positioning device generates the code of the light intensity sensor based on the light intensity value of each coded image received by a light intensity sensor, including :

The positioning device generates a first code of the light intensity sensor based on the light intensity value of the first image received by a light intensity sensor;

The positioning device generates a second code of the light intensity sensor based on the light intensity value of the second image received by the light intensity sensor;

The positioning device determines the pixel coordinates of the multiple light intensity sensors based on the codes of the multiple light intensity sensors, including: the positioning device determines the first coordinates of the light intensity sensors in the first direction based on the first code of the light intensity sensors; the positioning device Based on the second code of the light intensity sensor, the second coordinates of the light intensity sensor in the second direction are determined, and the pixel coordinates of the light intensity sensor include the first coordinates of the light intensity sensor and the second coordinates of the light intensity sensor.

Wherein, the binary coded image is a binary image whose pattern distribution conforms to a coding law, and the coding law of the coded pattern may be a binary code, a Gray code, or the like.

In combination with the first aspect, in a possible implementation manner, the method further includes:

The positioning device determines the pose of the projector relative to the positioning device based on the pose of the projector relative to the camera and the pose of the positioning device relative to the camera.

In combination with the first aspect, in a possible implementation manner, the method further includes:

The positioning device determines the display content based on the pose, three-dimensional map and media resources of the virtual reality device in the environmental coordinate system;

The positioning device sends the display content to the virtual reality device, so that the virtual reality device displays the display content.

With reference to the first aspect, in a possible implementation manner, before the positioning device projects the coded image through a projector, the method includes:

The positioning device sends instruction information to the virtual reality device, where the instruction information is used to instruct the virtual reality device to sample the light intensities received by the multiple light intensity sensors to obtain multiple light intensity values.

In combination with the first aspect, in a possible implementation manner, the positioning device tracks the virtual reality device, including:

The positioning device locates the position of the virtual reality device;

The positioning device moves to a position with a preset distance from the virtual display device;

The positioning device determines the relative position of the user wearing the virtual display device and the positioning device through the captured images;

The pointing device moves to the direction the user's face is facing based on the relative position.

In the second aspect, the embodiment of the present application provides a positioning method, which is applied to a virtual reality device. The virtual reality device includes a plurality of light intensity sensors. The method includes:

Multiple light intensity sensors respectively receive the light intensity of the coded image projected by the positioning device to obtain multiple light intensity values, and the positioning device tracks the virtual reality device;

The virtual reality device sends multiple light intensity values to the positioning device; the multiple light intensity values, coded images, and positions of multiple light intensity sensors in the virtual reality device are used by the positioning device to determine the pose of the virtual reality device relative to the positioning device; The pose of the virtual reality device relative to the positioning device and the pose of the positioning device in the environment coordinate system are used to determine the pose of the virtual reality device in the environment coordinate system.

In combination with the second aspect, in a possible implementation, the method further includes:

The virtual reality device receives from the positioning device the pose of the virtual reality device relative to the positioning device and the pose of the positioning device in the environment coordinate system;

The virtual reality device determines the pose of the virtual reality device in the environment coordinate system based on the pose of the positioning device in the environment coordinate system and the pose of the virtual reality device relative to the positioning device.

In combination with the second aspect, in a possible implementation, the method further includes:

The virtual reality device determines the display content based on the pose, three-dimensional map and media resources of the virtual reality device in the environmental coordinate system;

Virtual reality device showing display content.

In combination with the second aspect, in a possible implementation, the method includes:

When the virtual reality device receives the indication information sent by the positioning device, the virtual reality device samples the light intensities received by the multiple light intensity sensors to obtain multiple light intensity values.

With reference to the second aspect, in a possible implementation manner, sending the physical direction of the second electronic device to the first electronic device by the second electronic device includes:

The second electronic device obtains the physical direction of the second electronic device through the sensor;

When the physical direction changes, the second electronic device sends the physical direction to the first electronic device.

In a third aspect, the present application provides an electronic device. The electronic device can include memory and a processor. Among them, memory can be used to store computer programs. The processor can be used to call a computer program, so that the electronic device executes the first aspect or any possible implementation manner of the first aspect.

In a fourth aspect, the present application provides an electronic device. The electronic device can include memory and a processor. Among them, memory can be used to store computer programs. The processor can be used to call a computer program, so that the electronic device executes the second aspect or any possible implementation manner in the second aspect.

In the fifth aspect, the present application provides a computer program product containing instructions, which is characterized in that, when the above-mentioned computer program product is run on an electronic device, the electronic device is made to execute any possible method according to the first aspect or the first aspect. Method to realize.

In a sixth aspect, the present application provides a computer program product containing instructions, which is characterized in that, when the above-mentioned computer program product is run on an electronic device, the electronic device is made to execute any possible method according to the second aspect or the second aspect. Method to realize.

In the seventh aspect, the present application provides a computer-readable storage medium, including instructions, which is characterized in that, when the above-mentioned instructions are run on the electronic device, the electronic device is made to execute any of the possible steps in the first aspect or the first aspect. Method to realize.

In an eighth aspect, the present application provides a computer-readable storage medium, including instructions, which is characterized in that, when the above-mentioned instructions are run on an electronic device, the electronic device is made to execute any of the possible steps in the second aspect or the second aspect. Method to realize.

In a ninth aspect, an embodiment of the present application provides a positioning system, the positioning system includes a first electronic device and a second electronic device, the first electronic device is the electronic device described in the third aspect, and the second electronic device is the first electronic device Electronic equipment described in four terms.

It can be understood that the electronic devices provided in the third and fourth aspects, the computer program products provided in the fifth and sixth aspects, and the computer-readable storage media provided in the seventh and eighth aspects are all used to implement the present application The method provided by the embodiment. Therefore, the beneficial effects that it can achieve can refer to the beneficial effects in the corresponding method, and will not be repeated here.

Description of drawings

FIG. 1 is a schematic diagram of a pixel coordinate system provided by an embodiment of the present application;

FIG. 2A is a system architecture diagram of a virtual reality system provided by an embodiment of the present application;

FIG. 2B is a system architecture diagram of another virtual reality system provided by the embodiment of the present application;

FIG. 3 is a schematic diagram of a head-mounted display device provided by an embodiment of the present application;

Fig. 4 is a schematic diagram of a robot provided in an embodiment of the present application;

Fig. 5 is a schematic diagram of the relative positional relationship between a head-mounted display device and a robot provided by an embodiment of the present application;

FIG. 6 is a schematic diagram of a scenario provided by an embodiment of the present application;

FIG. 7 is a flow chart of a positioning method provided by an embodiment of the present application;

FIG. 8 is a schematic diagram of a coded image provided by an embodiment of the present application;

FIG. 9 is a schematic diagram of two sets of coded images provided by the embodiment of the present application;

FIG. 10 is a schematic diagram of light intensity values received by a light intensity sensor provided in an embodiment of the present application;

FIG. 11 is a schematic structural diagram of a head-mounted display device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. Among them, in the description of the embodiments of this application, unless otherwise specified, "/" means or means, for example, A/B can mean A or B; "and/or" in the text is only a description of associated objects The association relationship indicates that there may be three kinds of relationships, for example, A and/or B, which may indicate: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiment of the present application , "plurality" means two or more than two.

Hereinafter, the terms "first" and "second" are used for descriptive purposes only, and cannot be understood as implying or implying relative importance or implicitly specifying the quantity of indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of these features. In the description of the embodiments of the present application, unless otherwise specified, the "multiple" The meaning is two or more.

The term "user interface (UI)" in the following embodiments of this application is a medium interface for interaction and information exchange between an application program or an operating system and a user, and it realizes the difference between the internal form of information and the form acceptable to the user. conversion between. The user interface is the source code written in a specific computer language such as java and extensible markup language (XML). The source code of the interface is parsed and rendered on the electronic device, and finally presented as content that can be recognized by the user. The commonly used form of user interface is the graphical user interface (GUI), which refers to the user interface related to computer operation displayed in a graphical way. It may be text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, Widgets, and other visible interface elements displayed on the display screen of the electronic device.

First, the technical terms involved in the embodiments of the present application will be described below.

1. Structured light system

A structured light system refers to a system that uses structured light for measurement. A structured light system can consist of an optical projector, a camera, and a computer processing system. The basic principle is that after the optical projector projects a specific optical signal (specific optical information can also be called structured light) to the surface and background of the object, the camera collects the optical signal projected by the optical projector, and then, according to the optical signal caused by the object The change of the signal calculates information such as the position and depth of the object, and the above position and depth information can be used to restore the entire three-dimensional space. Wherein, the optical projector may be a projector, a laser, and the like.

1. Structured light

A collection of projection rays with known spatial directions is structured light.

Based on the beam mode, structured light can be divided into point structured light, line structured light, multi-line structured light and surface structured light. Furthermore, according to the beam mode of the structured light projected by the optical projector, the structured light system can be divided into a point structured light mode, a line structured light mode, a multi-line structured light mode, and a surface structured light mode.

Among them, the point structured light is a light beam projected onto an object to form a light spot. The point structured light mode means that the beam emitted by the laser is projected onto the object to generate a light spot, which is imaged on the image plane of the camera through the camera lens to form a two-dimensional point. Intersect at each point to form a simple triangular relationship. This triangular geometric constraint relationship can be obtained through a certain calibration, and the spatial position of the light point in a known world coordinate system can be uniquely determined by it. This method needs to scan the object point by point for measurement, and it takes time to take images and image processing, and the time increases sharply as the volume of the measured object increases.

Among them, the line structured light is a light beam projected onto the object to form a light strip, the line structured light mode is to project a light beam to the object, and form a light strip on the surface of the object; the multi-line structured light is to project multiple light strips; the surface structured light The mode projects a two-dimensional structured light image, and the structured light pattern can be a grating stripe.

2. Surface structured light mode

Surface structured light is a beam of light projected onto the surface of an object as a two-dimensional image. The surface structured light mode is an optical projector that projects a two-dimensional structured light pattern onto the surface of an object. This method can realize three-dimensional profile measurement without scanning, and the measurement speed is very fast. The most commonly used method in surface structured light is projection The grating stripes onto the surface of the object.

3. Coded structured light measurement method

Coded structured light is coded structured light, and the pattern formed when the coded structured light is projected onto an object conforms to the coding law corresponding to the coding method. Among them, the encoding methods include Gray Encoding, binary encoding, two-dimensional grid pattern encoding, random pattern encoding, color encoding, grayscale encoding, neighborhood encoding, phase encoding, and hybrid encoding. For example, the pattern on the binary coded image may be stripes, and the arrangement of stripes on the binary coded image conforms to Gray coding.

Determining the correspondence between the surface points of an object and the pixels of a two-dimensional image by coding structured light is called coded structured light measurement. Coded structured light measurement methods can be divided into spatial coding methods and temporal coding methods. Since the coded structured light is projected onto the surface of an object to form an image with a coded pattern, for the convenience of description, the coded structured light is referred to as a coded image below.

Wherein, the time-domain encoding method is that the optical projector projects a plurality of encoded images sequentially in time sequence, wherein the patterns on each encoded image in the plurality of encoded images are different, and the patterns on each encoded image are composed of a plurality of sub-patterns The sub-pattern of each coded image corresponds to a value; when the camera projects a coded image on the optical projector, it takes a projected image to obtain multiple projected images; the computer processing system can identify the sub-patterns on multiple projected images, Obtain multiple sub-patterns corresponding to each pixel on the projection image, and then determine the digital sequence corresponding to each pixel based on the correspondence between the sub-pattern and the value; by identifying the object on each pixel, determine the number on each pixel A sequence of numbers corresponding to an object, which is used to indicate the location of the object.

4. Binary coded image

The binary coded image refers to a binary image (BinaryImage) whose pattern distribution conforms to the coding law corresponding to the coding method. Among them, the binary image means that each pixel on the image has only two possible values or gray levels.

Before introducing the encoding of binary images, we first introduce the pixel coordinate system.

The projector projects the binary coded image onto the object, through which the position of the object in the coordinate system of the projector can be determined, and the position of the object in the coordinate system of the projector can be represented by the pixel coordinate system. The pixel coordinate system is a two-dimensional coordinate system, the unit of the pixel coordinate system is a pixel, and the coordinate origin O is at the upper left corner.

(A) in FIG. 1 exemplarily shows the pixel coordinate system of the first image. As shown in (A) in Figure 1, the rectangle represents the first image, with the upper left corner of the first image as the origin O, the pixel coordinate system as shown in the figure can be established, the abscissa u and the ordinate of the pixel coordinate system The units of v are pixels. It can be understood that the abscissa u and the ordinate v of a pixel in the image respectively indicate the number of columns and the number of rows where the pixel is located.

As shown in (B) in Figure 1, a row of pixels in the image is represented by vertical stripes, the abscissa of the first row of pixels of the image is 1, the abscissa of the second row of pixels is 2, and the abscissa of the third row of pixels The abscissa is 3, and so on. As shown in (C) in Figure 1, a row of pixels in the image is represented by vertical stripes, the abscissa of the first row of pixels in the image is 1, the abscissa of the second row of pixels is 2, and the abscissa of the third row of pixels The coordinate is 3, and so on.

For example, if u=1 and v=2 of a certain pixel point, then the pixel point is located at the pixel position where the first column in (B) in Figure 1 coincides with the second row in (C) in Figure 1 .

The following takes the five-bit Gray code as an example to introduce the encoding and decoding process of the image. Wherein, the five-bit Gray code includes 32 Gray codes, and each Gray code is composed of a five-bit code value.

In the process of encoding a binary coded image, the number of digits of the Gray code and the number of binary coded images can be determined according to the size of the binary coded image. For example, the width of the binary coded image is 32 pixel positions, and the height is less than 32 pixel positions, then the abscissa value of the binary coded image can be coded using five-bit Gray code, and the number of binary coded images is 5. For example, if the five-digit Gray code corresponding to the first pixel position is 00011, then the code of the first binary coded image projected at the first pixel position is 0; the code projected at the first pixel position of the second binary coded image is 0; the code of the third binary coded image projected at the first pixel position is 0; the code of the fourth binary coded image projected at the first pixel position is 1; the fifth binary coded image projected at the first pixel The location is coded as 1. It can be seen that the five-bit Gray code of a pixel can be constituted by five binary coded images.

In the process of encoding the binary encoded image, the two-dimensional encoded image can be encoded according to the abscissa value and the ordinate value of the pixel position. Taking the five-digit Gray code as an example, if the abscissa value of a certain pixel position is 3, since the Gray code corresponding to the binary number 3 is 00011, the pixel position is coded as 00011, then, when determining the Gray code of a certain pixel position When the code is 00011, it can be determined that the abscissa value of the pixel position is 3.

In the embodiment of the present application, a photosensitive sensor is used instead of a camera to cooperate with a projector to acquire pixel coordinates of an object.

2. Pose

1. Definition of pose

The pose of an object refers to the position and attitude of the object in a certain coordinate system, which can be described by the relative attitude of the coordinate system attached to the object.

For example, an object can be represented by a coordinate system B attached to the object, then the posture of the object relative to the coordinate system A is equivalent to the posture of the coordinate system B relative to the coordinate system A. For example, if a fixed point is taken as the origin on the robot and the robot coordinate system F2 is established, the posture of the robot relative to the environment coordinate system F0 is equivalent to the posture of the robot relative to the robot coordinate system F2 relative to the environment coordinate system F0.

则物体相对于坐标系A的姿态可以用

表示。需要说明的是，在坐标系A为环境坐标系F0时，物体的位姿可以用

表示。Among them, the attitude of the coordinate system B relative to the coordinate system A can be expressed by the rotation matrix R and the translation matrix T, and the attitude of the coordinate system B relative to the coordinate system A can be expressed as

Then the attitude of the object relative to the coordinate system A can be used

express. It should be noted that when the coordinate system A is the environment coordinate system F0, the pose of the object can be used

express.

In the embodiment of the present application, if the indoor coordinate system is represented by the environment coordinate system F0, the indoor positioning of the head-mounted display device is the pose of the head-mounted display device in the environment coordinate system F0. The head-mounted display device can be represented by the head-mounted display device coordinate system F3, then the pose of the head-mounted display device in the environment coordinate system F0 is

2. N-point perspective algorithm (Perspective-n-Point, PnP)

The PnP algorithm is a method for solving the motion of 3D to 2D point pairs, and is used to estimate the pose of an object based on n 3D space points and the projected positions of n 3D space points. Wherein, the n 3D space points are points on the object, and n is a positive integer. Wherein, the projected positions of the n 3D space points can be obtained based on the structured light system, and the projected positions of the n 3D space points can be represented by coordinates in a pixel coordinate system.

There are many solutions to the PnP problem, such as P3P, direct linear transformation (DLT), and EPnP, which use three pairs of points to estimate the pose. In addition, non-linear optimization can be used to construct the least squares problem and iteratively solve it.

In the embodiment of the present application, after obtaining the three-dimensional coordinates of the L optical sensors in the coordinate system of the head-mounted display device and the coordinates of the L optical sensors in the pixel coordinate system, the robot can determine the position of the head-mounted display device based on the PnP algorithm. The pose in the projector coordinate system. Wherein, L is a positive integer.

3. Real-time positioning and synchronous map construction (Simultaneous Localization And Mapping, SLAM) algorithm

1. Classification of SLAM

The sensors currently used in SLAM are mainly divided into two categories, one is Lidar-based laser SLAM (Lidar SLAM) and vision-based VSLAM (Visual SLAM). Among them, the laser SLAM is based on the point cloud information returned by the lidar, and the visual SLAM is based on the image information returned by the camera.

In the embodiment of this application, the robot can use laser SLAM to realize map construction and positioning. It should be understood that laser SLAM is mainly applied indoors, and the accuracy of maps constructed by laser SLAM is higher than that of visual SLAM. It should be noted that in other embodiments, visual SLAM may also be used to realize the indoor map building and positioning functions of the robot.

2. Laser SLAM

Laser SLAM can use two-dimensional (2-dimension, 2D) lidar or three-dimensional (3-dimension, 3D) lidar. Among them, two-dimensional lidar is generally used on indoor robots (such as sweeping robots), while two-dimensional lidar is generally used in the field of unmanned driving.

The role of lidar mainly has two aspects: on the one hand, it can provide point cloud data for the composition algorithm. When the composition algorithm obtains enough point cloud data, it can construct a local map with the robot as the center and the radar range as the radius. On the other hand, the predicted pose of the Bayesian filter can be corrected through the system observation model to improve the accuracy of the robot pose estimated by the filter. Under the condition of knowing the best pose, the local map can be updated to the global map.

The map constructed by two-dimensional laser SLAM is a two-dimensional grid map; the map constructed by three-dimensional laser SLAM is generally a three-dimensional point cloud map, and a three-dimensional point cloud map is a map composed of discrete three-dimensional space points. Currently, 2D-SLAM algorithms include gmapping-SLAM algorithm and Hector-SLAM algorithm. For example, the gmapping-SLAM algorithm uses a particle filter method to convert the collected laser ranging data into a grid map. The core idea of particle filtering is to express the spatial distribution by random state particles drawn from the posterior probability (observation equation). In simple terms, the particle filter method refers to the process of approximating the probability density function by finding a group of random samples propagating in the state space, and replacing the state equation with the sample mean value, thereby obtaining the minimum variance distribution of the state. .

In the embodiment of this application, the robot can construct an indoor two-dimensional map through two-dimensional laser SLAM, and combine the point cloud data detected by the structured light system to synthesize an indoor three-dimensional map; the robot can also construct an indoor three-dimensional map through three-dimensional laser SLAM.

In order to introduce the positioning method based on the virtual reality system provided by the embodiment of the present application more clearly and in detail, the virtual reality system provided by the embodiment of the present application is firstly introduced below.

Please refer to FIG. 2A . FIG. 2A is a system architecture diagram of a virtual reality system provided by an embodiment of the present application. As shown in FIG. 2A , the virtual reality system includes a virtual reality device 100 and a positioning device 200 . The virtual reality device 100 and the positioning device 200 can establish a communication connection. in:

The virtual reality device 100 can be a head-mounted display device and a handle, etc. The head-mounted display device can also be called a head-mounted display device, glasses device, VR glasses, etc., and the handle can also be called an electromagnetic handle, a control handle, etc.

The positioning device 200 is used for positioning the virtual reality device 100 . A positioning device is a device with mobility and positioning capabilities, such as a mobile robot or a flying robot.

In some embodiments, the user can wear a head-mounted display device to watch a virtual immersive scene, wherein the head-mounted display device determines the content of the displayed scene according to its position in space. That is to say, the head-mounted display device will change the output display content based on the position change of the head-mounted display device. This involves the positioning of the head-mounted display device. The head-mounted display device needs to obtain its own spatial positioning and determine the content of the displayed scene, so as to realize the user's immersive experience and the obstacle avoidance function of the virtual reality system.

In the embodiment of the present application, the positioning device 200 can locate the virtual reality device 100 to obtain the position information of the virtual display device, and then, the positioning device 200 can send the position information to the virtual reality device 100, so that the virtual reality device 100 can be based on The position information displays a virtual scene picture. For specific implementation, reference may be made to the following embodiments, which are not repeated here.

Wherein, the communication connection established between the virtual reality device 100 and the positioning device 200 may include but not limited to: wireless fidelity direct (wireless fidelity direct, Wi-Fi direct) (also known as wireless fidelity peer-to-peer -peer, Wi-Fi P2P)) communication connection, Bluetooth communication connection, near field communication (near field communication, NFC) connection, etc.

Exemplarily, the embodiment of the present application provides a system architecture diagram of another virtual reality system.

Please refer to FIG. 2B . FIG. 2B is a virtual reality system provided by an embodiment of the present application. As shown in FIG. 2B , the virtual reality system includes a head-mounted display device 300 and a robot 400 . The head-mounted display device 300 communicates with the robot 400 through short-range wireless communication methods such as Bluetooth and WiFi. in:

When the user wears the head-mounted display device 300 and walks indoors, the robot 400 can follow the user to move, and locate the head-mounted display device 300 in real time, and send the indoor pose of the head-mounted display device 300 to the head-mounted display device 400. The display device 300 enables the head-mounted display device 300 to determine display content based on its own pose.

A light intensity sensor may be provided on the head-mounted display device 300, and the light intensity sensor is used to receive light intensity. For example, a projector may be installed on the robot 400, and the projector projects to the head-mounted display device 300, and then the head-mounted display device 300 may send the light intensity value received by the light intensity sensor to the robot 400, so that the robot 400 can use the The aforementioned light intensity values determine the pose of the head-mounted display device 300 .

Wherein, the light intensity sensor is a sensitive device that responds to or converts external light signals or light radiation. Light intensity sensors include photocells, photomultiplier tubes, photoresistors, phototransistors, solar cells, infrared sensors, ultraviolet sensors, fiber optic photoelectric sensors, and color sensors.

Please refer to FIG. 3 , which shows possible distribution of light intensity sensors on the head-mounted display device 300 . As shown in FIG. 3 , black dots represent light intensity sensors 310 , and FIG. 3 shows the distribution of 20 light intensity sensors.

Please refer to FIG. 4 , which is a schematic diagram of a robot 400 provided in an embodiment of the present application. As shown in FIG. 4 , the robot 400 includes a structured light system 410 and a movement system 420 . in:

The structured light system 410 includes a camera 411 and a projector 412 . Wherein, the projector 412 is used for emitting coded structured light, such as a binary coded image; the camera 411 is used for taking images. As shown in FIG. 4 , the dotted line 101 and the dotted line 102 are used to indicate the shooting range of the camera 411, and the solid line 201 and the solid line 202 are used to indicate the projection range of the projector 412. It can be seen that the shooting range of the camera 411 and the projection of the projector 412 The ranges have overlapping areas, so that the camera 411 can capture the area where the projector 412 is projected.

Wherein, the projector 412 may be a wide-angle infrared digital light projector or other projectors, which is not limited here.

The moving system 420 may include several wheels and a driving device, and the driving device is used to drive the several wheels to drive the robot 400 to move. The robot 400 can control the mobile system 420 to maintain a preset distance and angle between itself and the head-mounted display device 300 , so that the head-mounted display device 300 is within the working range of the camera 411 and the projector 412 . For example, the relative positional relationship between the head-mounted display device 300 and the robot 400 can be shown in FIG. Within the distance, that is, when the user moves, the robot 400 moves following the user's movement, so that the head-mounted display device 300 is located within the working range of the camera 411 and the projector 412 .

In one implementation, the robot 400 maintains a preset distance and angle with the head-mounted display device 300, so that the head-mounted display device 300 is located in the working area of the camera 411 and the projector 412; furthermore, the robot 400 can control the projector 412 The coded structured light is emitted to the head-mounted display device 300 ; the robot 400 can take pictures of the head-mounted display device 300 through the camera 411 , and decode the image captured by the camera 411 to determine the pose of the head-mounted display device 300 .

In some embodiments, camera 411 and projector 412 are rotatable. The robot can control the rotation of the camera 411 and the projector 412, and calculate the transformation relationship between the rotated camera 411 and the projector 412 or the transformation relationship between the camera 411 and the robot 400 through rotation parameters.

Optionally, the robot may also include a laser SLAM system. The laser SLAM system is used for the robot 400 to detect the indoor environment and generate an indoor map. Furthermore, the robot 400 may determine its own indoor position based on the map, so as to determine the indoor pose of the head-mounted display device based on its own indoor position. Wherein, the detailed content of determining the indoor pose of the head-mounted display device by the robot 400 may refer to the following embodiments, which will not be repeated here.

Based on the system architecture of FIG. 2B , the head-mounted display device provided in FIG. 3 , and the robot provided in FIG. 4 , the embodiment of the present application provides a scenario as shown in FIG. 6 .

Referring to FIG. 6 , the user wears the head-mounted display device 300 , and the user and the robot 400 are located in the same indoor environment. Meanwhile, the indoor environment also includes several obstacles 500 .

First, introduce several coordinate systems in the room:

1. Environment coordinate system F0

The robot takes a certain point in the room as the coordinate origin, and establishes the environment coordinate system F0 shown in Figure 4. The environmental coordinate system F0 is a three-dimensional coordinate system, which is used to describe the three-dimensional coordinates of objects in the real world. Every position in the room can be represented by a coordinate point in the environmental coordinate system F0. In some embodiments, the environment coordinate system F0 may also be called a world coordinate system.

2. The coordinate system of the structured light system

The coordinate system of the structured light system includes a camera coordinate system F11 and a projector coordinate system F12. Among them, the camera coordinate system F11 is a coordinate system with the optical center of the camera as the origin, and the optical axes coincide with the z-axis; the projector coordinate system F12 is a coordinate system with the optical center of the projector as the origin, and the optical axes coincide with the z-axis.

来表示，其中，R代表旋转矩阵，T代表平移矩阵。其中，标定方法可以通过机器视觉软件(HALCON)，也可以通过其他标定方法，此处不做限定。After the positions of the camera and the projector are determined, the camera and projector can be calibrated to obtain camera parameters, projector parameters, and the transformation relationship between the camera coordinate system F11 and the projector coordinate system F12. Among them, the camera parameters include the focal length of the camera and the coordinates of the center point of the camera; the transformation relationship between the projector coordinate system F12 and the camera coordinate system F11 can be expressed by the matrix

to represent, where R represents the rotation matrix and T represents the translation matrix. Wherein, the calibration method may be through machine vision software (HALCON), or other calibration methods, which are not limited here.

Specifically, the projector will project the coded structured light onto the object, and the camera will capture the object. By decoding the image captured by the camera, the coordinates of the object in the pixel coordinate system can be determined. Wherein, the coordinate origin of the pixel coordinate system is at the upper left corner of the image, and the basic unit is a pixel. It can be understood that the camera coordinate system F11, the projector coordinate system F12 and the position of the object determine the coordinates of the object in the pixel coordinate system, and the coordinates of the object in the pixel coordinate system can reflect the coordinates of the object, the camera coordinate system F11 and the projector The positional relationship of the system F12.

3. Robot coordinate system F2

The robot coordinate system F2 is a coordinate system with a fixed point on the robot as the coordinate origin, and the position of the fixed point is fixed relative to the structured light coordinate system. For example, the origin of the robot coordinate system F2 may be located as shown in FIG. 6 .

表示，

用于指示F11相对于F2的变换关系。需要说明的是，机器人还可以通过其他标定方法确定

此处不做限定。As shown in Figure 6, the camera 411 is installed on the robot 400. After the positions of the camera 411 and the robot 400 are determined, the rotation matrix and translation matrix of the camera coordinate system F11 relative to the robot coordinate system F2 can be determined by hand-eye calibration method, which can be used

express,

It is used to indicate the conversion relationship between F11 and F2. It should be noted that the robot can also be determined by other calibration methods

There is no limit here.

4. Head-mounted display device coordinate system F3

The head-mounted display device coordinate system F3 is a coordinate system with a fixed point on the head-mounted display device as the coordinate origin.

In the embodiment of the present application, multiple light intensity sensors are provided on the head-mounted display device, and each light intensity sensor corresponds to a serial number. After determining the coordinate system F3 of the head-mounted display device, it can be determined that each light intensity sensor Coordinates in the wearable display device coordinate system F3.

表示。In the embodiment of this application, the pose of the head-mounted display device, that is, the pose of the head-mounted display device coordinate system F3 relative to the environment coordinate system F0, can be used

express.

Based on the scenario shown in FIG. 6 , the positioning method provided in the embodiment of the present application will be described in detail below in combination with the head-mounted display device provided in FIG. 3 and the robot provided in FIG. 4 in the embodiment of the present application.

Fig. 7 exemplarily shows the flow of the positioning method provided by the embodiment of the present application. The positioning method may include some or all of the following steps:

S101. The robot establishes a communication connection with the head-mounted display device.

The communication connection established between the head-mounted display device and the robot may include but not limited to: Wi-Fi P2P communication connection, Bluetooth communication connection, NFC connection and so on.

S102. The robot determines the indoor pose of the robot based on the indoor three-dimensional map.

表示，

用于指示机器人坐标系F2相对于环境坐标系F0的变换关系。应理解，机器人坐标系F2的原点位于机器人上，机器人坐标系F2相对于环境坐标系F0的位姿等价于机器人在环境坐标系F0的位姿，因此用

可以表示机器人在室内的定位。Among them, the pose of the robot indoors can be used

express,

It is used to indicate the transformation relationship between the robot coordinate system F2 and the environment coordinate system F0. It should be understood that the origin of the robot coordinate system F2 is located on the robot, and the pose of the robot coordinate system F2 relative to the environment coordinate system F0 is equivalent to the pose of the robot in the environment coordinate system F0, so

It can represent the positioning of the robot indoors.

In some embodiments, the robot can first obtain the indoor three-dimensional map, and then detect the surrounding environment in real time based on the laser SLAM system. Finally, based on the comparison between the surrounding environment and the indoor three-dimensional map, the indoor pose of the robot can be obtained.

Among them, the method for the robot to obtain the indoor three-dimensional map can be generated by robot detection, for example, the robot is equipped with a laser radar, the robot can move slowly in the room, and the indoor two-dimensional map is generated through the laser SLAM system; then the indoor map is obtained based on the structured light system. The point cloud data, so as to generate an indoor three-dimensional map based on the above-mentioned two-dimensional map and the above-mentioned point cloud data. For another example, the robot can obtain an indoor three-dimensional map through a structured light system. Specifically, the robot projects to the target area through a projector, obtains an image through a camera, and then analyzes the image to obtain an indoor three-dimensional map.

It should be noted that the robot can also obtain and save the indoor three-dimensional map from other devices, and the robot can also obtain the indoor three-dimensional map through other methods, which are not limited here. It can be understood that the robot can determine the drivable area in the room based on the indoor three-dimensional map, so as to realize the obstacle avoidance of the robot itself.

S103. The robot projects a binary coded image to the head-mounted display device.

Wherein, the binary coded image is a binary image whose pattern distribution conforms to a coding law, and the coding law of the coded pattern may be a binary code, a Gray code, or the like.

In some embodiments, the robot may project two sets of binary coded images to the head-mounted display device at a preset frequency through a projector in real time or at preset time intervals. Wherein, the patterns on the first group of images are stripes in the first direction, the images on the second group of images are stripes in the second direction, and the first direction and the second direction are vertical directions; each stripe corresponds to a pixel position , taking the first direction as vertical and the second direction as horizontal as an example, the pixel position corresponding to each vertical stripe is each column of pixels; the pixel position corresponding to the first horizontal stripe is each row of pixels. It should be noted that the order of each group of binary coded images is determined based on the coding rules, and the robot projects according to the order of the binary coded images, see the following embodiments for details.

Wherein, the number N of binary coded images is determined by the size and coding method of the binary coded images. For example, the encoding method of the binary coded image is gray coding, the length of the binary coded image is height, and the width is width, then the number of binary coded images in the first direction or the second direction is N>M, and M= max(log ₂ (width), log ₂ (height)), wherein, N is a positive integer. For another example, if the coding method of the binary coded image is sinusoidal, the number of binary coded images is at least three.

Taking the 5-bit Gray code as an example, the 5-bit Gray code can encode 32 (2 ⁵ =32) pixel positions. Please refer to FIG. 8 . FIG. 8 exemplarily shows a group of binary coded images with patterns as stripes and coded as five-digit Gray codes.

Please refer to Figure 8, there are five binary coded images in Figure 8, each binary coded image is divided into multiple vertical bar areas, each vertical bar area represents a pixel position, and the vertical bar area with slashes The code representing the pixel position is 0, and the blank vertical bar area represents the code of the pixel position is 1; each pixel position corresponds to a Gray code, and the five binary coded images correspond to one code value in the five-digit Gray code, respectively. Based on the encoding of the same pixel position in five binary coded images, the five-bit Gray code corresponding to the pixel position can be determined. Taking the third position as an example, the abscissa value of the third position is 3, and the Gray code corresponding to the third position is 00011, then, the third position can be coded as 0 in the first binary coded image, and in the second can be coded as 0 in the first binary coded image, can be coded as 0 in the third binary coded image, can be coded as 1 in the fourth binary coded image, and can be coded as 1 in the fifth binary coded image is 1. It should be noted that only 8 pixel positions are taken as an example in FIG. 1 , and the binary coded image corresponding to the five-bit Gray code may include 32 pixel positions.

Please refer to FIG. 9, which shows two sets of binary coded images provided by the embodiment of the present application. The first group of images is the binary coded image shown in Figure 8, the stripes of the first group of images are vertical stripes; the stripes of the second group of images are horizontal stripes, and the details of the second group of images can be found in Fig. 8, the relevant description will not be repeated here.

In some embodiments, the robot may first send instruction information to the head-mounted display device, and then transmit a binary coded image to the head-mounted display device. Light intensity is sampled. Among them, the way that the robot can first send instruction information to the head-mounted display device may include the following two implementations:

In one implementation, the robot may first project an image with a preset brightness to the head-mounted display device, and the image is used to instruct the head-mounted display device to sample the light intensity received by the light intensity sensor, and then project the image with the preset brightness. The frequency projects a binary coded image to the head-mounted display device. It should be noted that since the preset projection frequency is high, even if the user wears the head-mounted display device and moves, the head-mounted display device can be positioned in real time.

In another implementation, the robot may send a start message to the head-mounted display device through the communication connection established in step S101, where the start message is used to instruct the head-mounted display device to sample the light intensity received by the light intensity sensor, and further, Projecting binary coded images to the head-mounted display device at a preset projection frequency.

In some embodiments, light intensity sensors are arranged around the head-mounted display device. When the robot maintains a preset distance from the head-mounted display device, the head-mounted display device is within the projection range of the robot, and the robot can convert binary The encoded image is projected on the head-mounted display device. For example, the robot can determine the position of the head-mounted display device through Bluetooth positioning, and maintain a preset distance from the head-mounted display device through a tracking algorithm, so that the robot can project binary coded images on the head-mounted display device. For another example, in the initial state, the user wears a head-mounted display device, and the head-mounted display device is within the working range of the projector and camera on the robot; Target distance so that the HMD is within working range of the projector and camera on the robot. It should be noted that the head-mounted display device may be shown in FIG. 3 , as long as the robot is located around the head-mounted display device, the head-mounted display device can be photographed.

In some other embodiments, a part of the head-mounted display device is provided with a light intensity sensor, then the robot can move to a position in front of the part of the head-mounted display device, and maintain a preset distance from the head-mounted display device, So that the head-mounted display device is within the projection range of the robot.

Wherein, the robot may track the head-mounted display device or track the user wearing the head-mounted display device. For example, the robot can identify a user wearing a head-mounted display device, and then track the user to maintain a preset distance from the head-mounted display device, so that the robot can be projected on the head-mounted display device.

Optionally, the robot can also determine the positional relationship between the robot and the human body based on the trained neural network model, and then the robot can move to the front of the user to ensure that the user is within the working range of the projector. For example, a robot can take an image through a camera, input the image into a trained neural network model, and obtain the characteristics of the user. If the robot obtains all the facial features of the user based on the image, it can determine that the robot is in front of the user.

In one implementation, the robot can determine the position of the head-mounted display device through Bluetooth positioning; then, the robot moves to a position that is a preset distance away from the head-mounted display device; A positional relationship of the head-mounted display device; based on the positional relationship, moving relative to a target position of the head-mounted display device, such as the front of the head-mounted display device.

S104. When the robot projects a binary coded image to the head-mounted display device, the head-mounted display device respectively samples the light intensity values received by the light intensity sensors to obtain a sequence of light intensity values corresponding to each light intensity sensor.

Wherein, the head-mounted display device is provided with L light intensity sensors, and L is a positive integer greater than or equal to 4.

In some embodiments, when the robot projects a binary coded image to the head-mounted display device, the head-mounted display device can receive the light intensity signal through the light intensity sensor to obtain the light intensity value received by each light intensity sensor; and then , the head-mounted display device can sample the light intensity received by each light intensity sensor through the data acquisition unit (also called a data acquisition card) at a preset sampling frequency when the projector is determined to start projection, and obtain each light intensity Sequence of light intensity values from the sensor

Wherein, the preset sampling frequency is greater than or equal to twice the value of the preset projection frequency. For example, the robot sends the preset projection frequency to the head-mounted display device through the communication connection established in step S101. Correspondingly, the head-mounted display device receives the preset projection frequency and determines based on the double value of the preset projection frequency as Preset sampling frequency.

For example, a robot projects a binary coded image as shown in Figure 8 to a head-mounted display device, assuming that the first light intensity sensor is located at the position corresponding to the third position, that is to say, the first light intensity sensor is located at the position corresponding to the binary code received The position of the light intensity at the third position in the image, the light intensity signal received by the first light intensity sensor may be as shown in FIG. 10 . Fig. 10 shows the light intensity value of the light intensity signal received by the first light intensity sensor, where the abscissa is time and the ordinate is light intensity value. In FIG. 10 , the time when the first light intensity sensor receives the first image is represented by the origin time 0, and T is the projection period corresponding to the preset projection frequency. The robot projects the first binary coded image to the head-mounted display device in the first cycle, the gray value of the third position in the first binary coded image is 0, and the light intensity value is weak; as shown in Figure 10 , the light intensity value received by the first light intensity sensor in the second cycle and the third cycle is consistent with the light intensity value received in the first cycle; the robot projects the fourth light intensity value to the head-mounted display device in the fourth cycle Binary coded image, the light intensity received by the first light intensity sensor is the third position of the fourth binary coded image, the gray value of the third position is 1, and the light intensity value is stronger, as shown in Figure 10 , the light intensity value received by the first light intensity sensor in the fifth period is consistent with the light intensity value received in the fourth period. It should be noted that due to the influence of unstable factors in the real environment such as other lighting, the light intensity value may change slightly within a cycle, so the light value in a cycle in FIG. 10 can be a curve.

In some embodiments, the head-mounted display device can determine that the projector starts to project through the instruction information sent by the robot to start sampling the light intensity, or it can start sampling when it detects a change in the light intensity, or it can be determined by other methods The projector starts to project, which is not limited here. For example, the indication information can be an image with a preset brightness. When the head-mounted display device receives an image with a preset brightness, it can sample the light intensity received by each light intensity sensor at a preset sampling frequency to obtain each light intensity. The light intensity sequence value of the intensity sensor.

Among them, the head-mounted display device can pre-store or receive parameters such as the number of images or time of a projection process of the robot sent by the robot, and then stop sampling based on the above parameters, and send the sequence of sampled light intensity values to the robot. For example, if the head-mounted display device pre-stores the projection time of one projection of the robot, the head-mounted display device can start sampling the light intensity value based on the instruction information, and end the sampling when the sampling time is the projection time, to obtain the light intensity value sequence, and Sequence of strong values sent to the robot.

S105. The head-mounted display device sends to the robot the sequence of light intensity values corresponding to each light intensity sensor and the three-dimensional coordinates of each light intensity sensor in the coordinate system F3 of the head-mounted display device to the robot based on the above communication connection.

In one implementation, the head-mounted display device generates data corresponding to each of the L light intensity sensors to obtain L sets of data; and sends L sets of data to the robot, wherein any set of data in the L sets of data includes The logo of the light intensity sensor, the sequence of light intensity values corresponding to the light intensity sensor, and the three-dimensional coordinates of the light intensity sensor in the coordinate system F3 of the head-mounted display device; the logo of the light intensity sensor is used to distinguish L light intensity sensors, light The identification of the strong sensor may be a serial number or the like.

For example, the coordinates of each light intensity sensor in the head-mounted display device coordinate system F3 can be represented by {id, x, y, z}, where id is the number of the light intensity sensor, and x, y, z are the light intensity The three-dimensional coordinate value of the sensor in the coordinate system F3 of the head-mounted display device.

S106. The robot determines the coordinates of each light intensity sensor in the pixel coordinate system based on the above sequence of light intensity values.

Specifically, the robot can respectively process the light intensity value sequences corresponding to the L light intensity sensors to generate coordinates of the L light intensity sensors in the pixel coordinate system. Wherein, the coordinates of each optical sensor in the pixel coordinate system can be represented by (u, v), where u is the abscissa of the optical sensor, and v is the ordinate of the optical sensor.

In some embodiments, the robot can divide the light intensity value sequence of each group of light intensity sensors sent by the head-mounted display into two sets of light intensity value subsequences based on the projection period; Two sets of light intensity value subsequences are processed to obtain two codes of each light intensity sensor; based on the two codes of each light intensity sensor, the abscissa and ordinate of each light intensity sensor in the pixel coordinate system are obtained.

For example, the projection period is T, and the robot projects a binary coded image in each projection period. The robot projects the first group of images shown in Figure 9 in the first five periods, and projects the images in Figure 9 in the last five periods. For the second group of images shown, the robot can determine the light intensity values of the first five cycles as the first subsequence, and determine the light intensity values of the last five cycles as the second subsequence for each optical sensor; The first Gray code is obtained based on the first subsequence, and the second Gray code is obtained based on the second subsequence; the binary code corresponding to the first Gray code is determined as the abscissa u of the light intensity sensor, and the binary code corresponding to the second Gray code is determined as Determine the ordinate v of the light intensity sensor to obtain the coordinates (u, v) of the light intensity sensor in the pixel coordinate system.

Wherein, the method for the robot to obtain the first Gray code based on the first subsequence may be that the robot determines a target light intensity value from the subsequence of each cycle, for example, the average value of the light intensity values of the subsequence or any subsequence in the subsequence The light intensity value is determined as the target light intensity value, and five target light intensity values are obtained; each target light intensity value is normalized to obtain five coded values; finally, the five coded values are sorted in time order to obtain First gray code.

Among them, the robot can normalize the target light intensity value through the following formula to obtain the coded value.

为归一化处理后的光强值。如果

头戴式显示设备可以确定

对应的编码值为1；否则，确定

对应的编码值为1。Wherein, I is any light intensity value in five light intensity values, and I _max is the light intensity value with the largest numerical value in five light intensity values, and I _min is the light intensity value with the smallest numerical value in five light intensity values,

is the normalized light intensity value. if

Head-mounted display devices can determine

The corresponding encoding value is 1; otherwise, determine

The corresponding encoding value is 1.

In one implementation, after obtaining the two Gray codes of each light intensity sensor, the robot can convert the two Gray codes into binary codes respectively to obtain the abscissa and ordinate of the light intensity sensor in the pixel coordinate system. For example, the robot obtains the first Gray code as 00011 and the second Gray code as 00111 through the above processing, and the robot can calculate that the binary code corresponding to the first Gray code is 3, and the binary code corresponding to the second Gray code is 6, then the optical sensor The coordinates in the pixel coordinate system are (3, 6).

It should be noted that, in some embodiments, the coordinates of each light intensity sensor can be calculated by the head-mounted display device. After calculating the coordinates of each light intensity sensor, the head-mounted display device calculates the coordinates of each light intensity sensor The coordinates of the sensor are sent to the robot, and the following step S106 is executed.

S107. The robot determines the pose of the head-mounted display device in the projector coordinate system F12 based on the coordinates of each light intensity sensor in the pixel coordinate system and its three-dimensional coordinates in the head-mounted display device coordinate system F3

用于指示头戴式显示设备坐标系F3与投影仪坐标系F12的变换关系。In some embodiments, in step S105, the robot can acquire the three-dimensional coordinates of the L light intensity sensors sent by the head-mounted display device in the coordinate system F3 of the head-mounted display device, and in S106 can acquire the coordinates of the L light intensity sensors in the head-mounted display device coordinate system F3. The coordinates in the pixel coordinate system, therefore, the robot can determine the coordinates of each light intensity sensor in the pixel coordinate system and its three-dimensional coordinates in the head-mounted display device coordinate system F3 based on the identification of the light intensity sensor; The coordinates of a light intensity sensor in the pixel coordinate system and the three-dimensional coordinates in the head-mounted display device coordinate system F3 generate an equation to obtain L equations; solve the L equations to obtain the head-mounted display device in the projector Pose in coordinate system F12

It is used to indicate the transformation relationship between the head-mounted display device coordinate system F3 and the projector coordinate system F12.

其中，PnP算法可以包括光束法平差(BA，BundleAdjustment)算法和直接线性变换(DLT，Direct Linear Transform)算法等，此处不作限制。For example, based on the coordinates of the light intensity sensor in the pixel coordinate system, the internal parameters of the projector, and the three-dimensional coordinates of the light intensity sensor in the head-mounted display device coordinate system F3, the robot can calculate the projection of the head-mounted display device through the PnP algorithm. The pose in instrument coordinate system F12

Wherein, the PnP algorithm may include a bundle adjustment (BA, Bundle Adjustment) algorithm and a direct linear transformation (DLT, Direct Linear Transform) algorithm, etc., which are not limited here.

Among them, the robot generates an equation for each light intensity sensor to obtain a system of equations; by solving the system of equations, the pose of the head-mounted display device in the projector coordinate system F12 can be obtained.

Among other things, the robot can generate equations based on:

Among them, (u, v) are the coordinates of the light intensity sensor in the pixel coordinate system, (x, y, z) are the three-dimensional coordinate values of the light intensity sensor in the coordinate system F3 of the head-mounted display device, and the internal parameters of the projector include The focal length and center point coordinates, the focal length of the projector is f _x and f _x , the center coordinate is (c _x , cy _y ), and n is a proportional value.

转换为头戴式显示设备在相机坐标系F11中的位姿

S108. Based on the transformation relationship between the projector coordinate system F12 and the camera coordinate system F11, the robot calculates the pose of the head-mounted display device in the projector coordinate system F12

Convert to the pose of the head-mounted display device in the camera coordinate system F11

机器人基于以下公式，得到头戴式显示设备在相机坐标系F11中的位姿：In one implementation, the robot stores the transformation relationship of the projector coordinate system F12 relative to the camera coordinate system F11

The robot obtains the pose of the head-mounted display device in the camera coordinate system F11 based on the following formula:

为结构光系统的参数，在相机和投影仪的位置确定后，可以通过机器视觉软件或其他标定方法确定。It should be understood that

The parameters of the structured light system can be determined by machine vision software or other calibration methods after the positions of the camera and projector are determined.

机器人坐标系F2和相机坐标系F11的变换关系

头戴式显示设备在相机坐标系F11中的位姿

计算得到头戴式显示设备在环境坐标系F0中的位姿。S109, the robot is based on the indoor pose of the robot

The transformation relationship between the robot coordinate system F2 and the camera coordinate system F11

The pose of the head-mounted display device in the camera coordinate system F11

Calculate the pose of the head-mounted display device in the environment coordinate system F0.

进而，机器人可以基于以下公式，得到头戴式显示设备在环境坐标系F0中的位姿：In one implementation, the robot can obtain the indoor pose of the robot through step S102

Furthermore, the robot can obtain the pose of the head-mounted display device in the environment coordinate system F0 based on the following formula:

为机器人在环境坐标系F0中的位姿；

为机器人坐标系F2和相机坐标系F11的标定关系；

为头戴式显示设备在相机坐标系F11中的位姿。需要说明的是，机器人坐标系F2和相机坐标系F11的标定关系可以通过手眼标定法等标定方法确定，在结构光系统在机器人上安装固定后，可以标定出机器人坐标系F2和相机坐标系F11的变换关系。in,

is the pose of the robot in the environment coordinate system F0;

is the calibration relationship between the robot coordinate system F2 and the camera coordinate system F11;

is the pose of the head-mounted display device in the camera coordinate system F11. It should be noted that the calibration relationship between the robot coordinate system F2 and the camera coordinate system F11 can be determined by calibration methods such as the hand-eye calibration method. After the structured light system is installed and fixed on the robot, the robot coordinate system F2 and the camera coordinate system F11 can be calibrated transformation relationship.

S110. Based on the above communication connection, the robot sends the pose of the head-mounted display device in the environment coordinate system F0 to the head-mounted display device.

In some embodiments, part of the content from step S107 to step S109 may be executed by a head-mounted display device. For example, the head-mounted display device receives the pose of the head-mounted display device relative to the robot and the pose of the robot in the environment coordinate system sent by the robot; The pose in the environment coordinate system and the pose of the head-mounted display device relative to the robot determine the pose of the head-mounted display device in the environment coordinate system.

S111. The head-mounted display device generates display content based on the pose and the three-dimensional map of the head-mounted display device in the environment coordinate system F0.

In one implementation, the head-mounted display device determines the range of the target field of view in the three-dimensional map based on its pose in the environment coordinate system F0, and combines existing media resources to generate an image of the target field of view. Wherein, the target field of view range is the range displayed when the user wears the head-mounted display device.

S112. The head-mounted display device displays the display content.

The user wears the head-mounted display device and can see the above-mentioned displayed content. For example, based on the indoor pose of the head-mounted display device, the head-mounted display device determines that there are no obstacles on the left side and obstacles on the right side in front of the user wearing the head-mounted display device, then the generated display content can display the The position of the side obstacle is a virtual obstacle, and the position on the left side is displayed as a virtual passage. Then, after seeing the displayed content, the user can avoid obstacles and move in a safe area. Wherein, the virtual obstacle and the virtual channel are media resources stored in the media library of the head-mounted display device.

In other embodiments, the robot generates display content based on the pose of the head-mounted display device in the environment coordinate system F0 and the three-dimensional map, and then the robot sends the display content to the head-mounted display device.

In some embodiments, after step S107, the robot can determine the position of the head-mounted display device in the robot coordinate system based on the pose of the head-mounted display device in the projector coordinate system and the transformation relationship between the robot coordinate system and the projector coordinate system. Then, based on the pose of the head-mounted display device in the robot coordinate system and the pose of the robot in the room, determine the pose of the head-mounted display device in the environment coordinate system F0. Wherein, the transformation relationship between the robot coordinate system and the projector coordinate system can be calibrated in advance. For example, the transformation relationship between the robot coordinate system and the camera coordinate system, the transformation relationship between the camera coordinate system and the projector coordinate system is calibrated by the camera in steps S108 and S109, and then the transformation relationship between the robot coordinate system and the projector coordinate system is determined. It can be understood that after the transformation relationship between the robot coordinate system and the projector coordinate system is determined, in some embodiments, the robot does not need to install a camera.

Referring to FIG. 11 , FIG. 11 shows a schematic structural diagram of a head-mounted display device 300 provided by an embodiment of the present application. As shown in FIG. 11 , the head-mounted display device 300 may include: a processor 301 , a memory 302 , a communication module 303 , a sensor module 304 , a camera 305 , a display device 306 , and an audio device 307 . The above components can be coupled and communicate with each other. Understandably, the structure shown in FIG. 11 does not constitute a specific limitation on the head-mounted display device 300 .

In some other embodiments of the present application, the head-mounted display device 300 may include more or less components than those shown in the illustration. For example, the head-mounted display device 300 may also include physical buttons such as a power button, a volume button, a USB interface, and the like.

The processor 301 may include one or more processing units, for example: the processor 301 may include an AP, a modem processor, a GPU, an ISP, a controller, a video codec, a DSP, a baseband processor, and/or an NPU, etc. . Wherein, different processing units may be independent devices, or may be integrated in one or more processors. The controller can generate operation control signals according to instruction opcodes and timing signals, complete instruction fetching and execution control, and enable each component to perform corresponding functions, such as human-computer interaction, motion tracking/prediction, rendering display, audio processing, etc.

The memory 302 stores executable program codes for executing the interaction method in the virtual reality scene provided by the embodiment of the present application, where the executable program codes include instructions. The memory 302 may include an area for storing programs and an area for storing data. The communication module 303 may include a mobile communication module and a wireless communication module. Among them, the mobile communication module can provide wireless communication solutions including 2G/3G/4G/5G applied on the head-mounted display device 300 . The wireless communication module can provide wireless communication solutions including WLAN, BT, GNSS, FM, and IR applied on the head-mounted display device 300 . The communication module 303 can support the communication between the head-mounted display device 300 and the electronic device. The sensor module 304 is used for collecting motion state data of the user wearing the head-mounted display device 300 . The sensor module 304 may include an accelerometer, a compass, a gyroscope, a magnetometer, or other sensors for detecting motion, among others.

In the embodiment of the present application, the memory 302 may store the three-dimensional coordinate values of each light intensity sensor on the head-mounted display device on the head-mounted display device.

In some embodiments, the sensor module 304 may be an inertial measurement unit (inertial measurement unit, IMU) disposed in the head-mounted display device 300 . The sensor module 304 can be used to acquire the motion data of the user's head, such as head position information, displacement, speed, shaking, rotation and so on. The sensor module 304 may also include an optical sensor for tracking the user's eye position and capturing eye movement data in conjunction with the camera 305 . For example, it can be used to determine the interpupillary distance of the user, the distance between the eyes, the 3D position of each eye relative to the head-mounted display device 300, the magnitude of the twist and rotation (ie, rotation, pitch and shake) of each eye, and the direction of gaze. Camera 305 may be used to capture still images or video. The still image or video may be an externally facing image or video of the user's surroundings or an internally facing image or video.

In the embodiment of the present application, the sensor module 304 may include at least four light intensity sensors.

The camera 305 can track the movement of one or both eyes of the user. The camera 305 includes, but is not limited to, a traditional color camera (RGB camera), a depth camera (RGB depth camera), a dynamic vision sensor (dynamic vision sensor, DVS) camera, and the like.

The head-mounted display device 300 presents or displays VR scenes through the GPU, the display device 306 , and the application processor. The GPU is a microprocessor for image processing, and is connected to the display device 306 and the application processor. Processor 301 may include one or more GPUs that execute program instructions to generate or change display information. The GPU is used to perform mathematical and geometric calculations based on data obtained from electronic devices, and to render 3D virtual scenes using computer graphics technology, computer simulation technology, etc., to provide content for display on the display device 306 . GPUs are also used to add correction or pre-distortion to the rendering process of virtual scenes to compensate or correct distortion caused by optics. The GPU may also adjust the content provided to the display device 306 based on data from the sensor module 304 . For example, the GPU may add depth information to the content provided to the display device 306 based on the 3D position of the user's eyes, pupillary distance, and the like. In some embodiments of the present application, the display device 306 is configured to receive content provided by the GPU of the head-mounted display device 300, and present or display a VR scene according to the content. In other embodiments of the present application, the display device 306 is configured to receive processed data or content from the electronic device (for example, data rendered by the electronic device), and present a VR scene according to the data or content. In some embodiments, the display device 306 may present corresponding images for the user's left and right eyes, respectively, thereby simulating binocular vision.

In the embodiment of the present application, the head-mounted display device 300 may receive display content sent from the robot, and present a VR scene according to the data or content.

In some embodiments, display device 306 may include a display screen and associated optics. Wherein, the display screen may include a display panel, and the display panel may be used to display virtual images, thereby presenting a three-dimensional virtual scene to the user. The display panel can adopt LCD, OLED, AMOLED, FLED, Miniled, MicroLed, Micro-oLed, QLED, etc. The number of display screens can be one or more. Optics may include one or more optical elements, such as Fresnel lenses, convex lenses, concave lenses, filters, and the like. Optics are used to direct light from the display screen to the exit pupil for perception by the user. In some embodiments, one or more optical elements in an optical device may have one or more coatings, such as anti-reflective coatings. Amplification of image light by optics allows displays to be physically smaller, lighter and consume less power. In addition, the magnification of the image light can increase the field of view of what is displayed on the display screen. For example, the optics may allow the field of view of the content displayed on the display screen to be the full field of view of the user. Optics may also be used to correct for one or more optical errors. Examples of optical errors include: barrel distortion, pincushion distortion, longitudinal chromatic aberration, lateral chromatic aberration, spherical aberration, coma aberration, field curvature, astigmatism, and the like. In some embodiments, the content presented to the display screen for display is pre-distorted, and the distortion is corrected by the optics upon receipt of content-based generated image light from the display screen. In other embodiments, the display device 306 may include a projection device for projecting an optical signal (eg, a light beam) directly onto the user's retina. The projection device may be a projector. The projection device can receive the content provided by the GPU, encode the content into an optical signal, and project the encoded optical signal onto the user's retina, so that the user can experience a three-dimensional VR scene. The number of projection devices can be one or more. The audio device 307 is used for collecting and outputting audio. Audio devices 307 may include, but are not limited to: microphones, speakers, earphones, and the like.

In some embodiments, the system shown in FIG. 2B may also include a handheld device. Handheld devices can be connected and communicated with electronic devices wirelessly through BT, NFC, ZigBee and other short-distance transmission technologies, and can also be wired and communicated through USB interfaces, HDMI interfaces or custom interfaces. The implementation forms of the handheld device may include handles, mice, keyboards, stylus pens, bracelets, and the like. Handheld devices can be configured with various sensors, such as acceleration sensors, gyroscope sensors, magnetic sensors, pressure sensors, and the like. The pressure sensor can be arranged under the confirmation button of the handheld device. The confirmation button can be a physical button or a virtual button. The sensors of the handheld device are used to collect corresponding data, for example, the acceleration sensor collects the acceleration of the handheld device, the gyroscope sensor collects the movement speed of the handheld device, and the like. The handheld device can send the data collected by each sensor to the electronic device for analysis. The electronic device can determine the motion and state of the handheld device according to the data collected by various sensors in the handheld device. The movement condition of the handheld device may include but not limited to: whether to move, the direction of movement, the speed of movement, the distance of movement, the trajectory of movement and so on. The status of the handheld device may include: whether the confirmation button of the handheld device is pressed. The electronic device can adjust the image displayed on the head-mounted display device 300 and/or start the corresponding function according to the motion and/or state of the handheld device, such as moving the cursor in the image, and the moving track of the cursor is controlled by the handheld device. Movement is confirmed.

The embodiment of the present application also provides an electronic device, the electronic device includes one or more processors and one or more memories; wherein, the one or more memories are coupled with the one or more processors, and the one or more memories are used for For storing computer program codes, the computer program codes include computer instructions, and when one or more processors execute the computer instructions, the electronic device executes the methods described in the above embodiments.

The embodiment of the present application also provides a computer program product containing instructions, and when the computer program product is run on the electronic device, the electronic device is made to execute the method described in the foregoing embodiments.

The embodiment of the present application also provides a computer-readable storage medium, including instructions, and when the instructions are run on the electronic device, the electronic device is made to execute the method described in the foregoing embodiments.

It can be understood that the various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

In the above embodiments, all or part of them may be implemented by software, hardware, firmware or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions according to the present application will be generated in whole or in part. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server or data center Transmission to another website site, computer, server, or data center by wired (eg, coaxial cable, optical fiber, DSL) or wireless (eg, infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center integrated with one or more available media. The available medium may be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a Solid State Disk).

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments are realized. The processes can be completed by computer programs to instruct related hardware. The programs can be stored in computer-readable storage media. When the programs are executed , may include the processes of the foregoing method embodiments. The aforementioned storage medium includes: ROM or random access memory RAM, magnetic disk or optical disk, and other various media that can store program codes.

In a word, the above description is only an embodiment of the technical solution of the present application, and is not intended to limit the protection scope of the present application. All modifications, equivalent replacements, improvements, etc. based on the disclosure of this application shall be included within the scope of protection of this application.

Claims (16)

1. A positioning method, characterized in that it is applied to a positioning device, the positioning device comprises a projector, and the method comprises: The positioning device tracks a virtual reality device; The positioning device projects an encoded image through the projector; The positioning device receives multiple light intensity values from the virtual reality device; the multiple light intensity values include light intensities of encoded images respectively received by multiple light intensity sensors of the virtual reality device; The positioning device determines the pose of the virtual reality device relative to the positioning device according to the projected coded image, the multiple light intensity values, and the positions of the multiple light intensity sensors in the virtual reality device ; The pose of the virtual reality device relative to the positioning device and the pose of the positioning device in the environment coordinate system are used to determine the pose of the virtual reality device in the environment coordinate system. 2. The method according to claim 1, characterized in that the method further comprises: The positioning device determines the position of the virtual reality device in the environmental coordinate system according to the pose of the positioning device in the environmental coordinate system and the pose of the virtual reality device relative to the positioning device posture; The positioning device sends the pose of the virtual reality device in the environment coordinate system to the virtual reality device. 3. The method according to claim 1 or 2, wherein the coded image comprises a plurality of coded images; and the positioning device according to the projected coded image, the multiple light intensity values and the multiple light intensity values The location of the strong sensor in the virtual reality device determines the pose of the virtual reality device relative to the positioning device, including: The positioning device generates the code of the light intensity sensor based on the light intensity value of each encoded image received by the light intensity sensor; The positioning device determines pixel coordinates of the plurality of light intensity sensors based on codes of the plurality of light intensity sensors; The positioning device determines the pose of the virtual reality device relative to the projector based on the pixel coordinates of the plurality of light intensity sensors and the positions of the plurality of light intensity sensors in the virtual reality device; The positioning device converts the pose of the virtual reality device relative to the projector into a pose of the virtual reality device relative to the positioning device based on the pose of the projector relative to the positioning device . 4. The method according to claim 3, wherein the plurality of encoded images comprises M first images and N second images, wherein M is a positive integer, N is a positive integer, and the The first image is a binary image with a pattern of stripes in a first direction, and the second image is a binary image with a pattern of stripes in a second direction; the positioning device is based on each Encode the light intensity value of the image to generate the code of the light intensity sensor, including: The positioning device generates a first code of the light intensity sensor based on a light intensity value of the first image received by the light intensity sensor; The positioning device generates a second code of the light intensity sensor based on a light intensity value of a second image received by the light intensity sensor; The positioning device determines the pixel coordinates of the multiple light intensity sensors based on the codes of the multiple light intensity sensors, including: the positioning device determines the pixel coordinates of the light intensity sensors based on the first code of the light intensity sensors The first coordinate in the first direction; the positioning device determines the second coordinate of the light sensor in the second direction based on the second code of the light sensor, and the pixel coordinates of the light sensor include The first coordinates of the light intensity sensor and the second coordinates of the light intensity sensor. 5. according to the described method of claim 3 or 4, it is characterized in that, described method also comprises: The positioning device determines the pose of the projector relative to the positioning device based on the pose of the projector relative to the camera and the pose of the positioning device relative to the camera. 6. The method according to any one of claims 1-5, wherein the method further comprises: The positioning device determines display content based on the pose, three-dimensional map and media resources of the virtual reality device in the environment coordinate system; The positioning device sends the display content to the virtual reality device, so that the virtual reality device displays the display content. 7. The method according to claims 1-6, wherein before the positioning device projects the coded image through the projector, the method comprises: The positioning device sends instruction information to the virtual reality device, where the instruction information is used to instruct the virtual reality device to sample the light intensities received by the plurality of light intensity sensors to obtain the plurality of light intensity values. 8. The method according to claims 1-7, wherein the positioning device tracking the virtual reality device comprises: The positioning device locates the position of the virtual reality device; The positioning device moves to a position at a preset distance from the virtual display device; The positioning device determines the relative position of the user wearing the virtual display device and the positioning device through the captured image; The positioning device moves to a direction in which the user's face is facing based on the relative position. 9. A positioning method, characterized in that it is applied to a virtual reality device, the virtual reality device includes a plurality of light intensity sensors, and the method comprises: The plurality of light intensity sensors respectively receive the light intensity of the encoded image projected by the positioning device to obtain multiple light intensity values, and the positioning device tracks the virtual reality device; The virtual reality device sends the multiple light intensity values to the positioning device; the multiple light intensity values, the encoded image and the positions of the multiple light intensity sensors in the virtual reality device are used for The positioning device determines the pose of the virtual reality device relative to the positioning device; the pose of the virtual reality device relative to the positioning device and the pose of the positioning device in the environment coordinate system are used to determine The pose of the virtual reality device in the environment coordinate system. 10. The method according to claim 9, further comprising: The virtual reality device receives from the positioning device the pose of the virtual reality device relative to the positioning device and the pose of the positioning device in the environment coordinate system; The virtual reality device determines the position of the virtual reality device in the environment coordinate system based on the pose of the positioning device in the environment coordinate system and the pose of the virtual reality device relative to the positioning device pose. 11. The method according to claim 9 or 10, further comprising: The virtual reality device determines display content based on the virtual reality device's pose, three-dimensional map, and media resource in the environment coordinate system; The virtual reality device displays the display content. 12. The method according to claims 9-11, characterized in that the method comprises: When the virtual reality device receives the indication information sent by the positioning device, the virtual reality device samples the light intensities received by the plurality of light intensity sensors to obtain the plurality of light intensity values. 13. An electronic device, characterized in that the electronic device comprises one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, the The one or more memories are used to store computer program codes, the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the electronic device is configured to perform the tasks described in claims 1-12. any one of the methods described. 14. A computer program product containing instructions, wherein when the computer program product is run on an electronic device, the electronic device is made to execute the method according to any one of claims 1-12. 15. A computer-readable storage medium, comprising instructions, wherein when the instructions are run on an electronic device, the electronic device is made to execute the method according to any one of claims 1-12. 16. A positioning system, characterized in that the positioning system comprises a first electronic device and a second electronic device, and the first electronic device is configured to execute the method according to any one of claims 1-8, The second electronic device is configured to execute the method according to any one of claims 9-12.

Images