WO2024092890A1 - Local multi-device rapid spatial anchors synchronization method and system for mixed reality - Google Patents

Abstract

Disclosed in the present invention are a local multi-device rapid spatial anchors synchronization method and system for mixed reality. The method comprises the following steps: using any device in a local space as a host to create a local network, and using other devices as guests to join in the local network created by the host; each guest and the host first exchanging respective local timestamps, and then calculating a time offset between the local timestamp of the guest and the local timestamp of the host, so that time synchronization between the guest and the host is realized; and on the basis of the time synchronization, a guest in the local network performing spatial coordinate system synchronization with the host by means of augmented reality image tracking technology, and other guests performing spatial coordinate system synchronization with the host or the synchronized guest, thereby implementing spatial coordinate system synchronization among all devices in the local network. The synchronization method provided by the present invention can use a multi-user mixed reality application in the absence of an external operator network, and implement rapid spatial coordinate system synchronization.

Description

Local multi-device fast spatial anchor synchronization method and system for mixed reality Technical Field

The present invention belongs to the technical field of mixed reality, and in particular relates to a method and system for local multi-device rapid spatial anchor point synchronization for mixed reality.

Background technique

Mixed Reality (MR) and Augmented Reality (AR) are technologies that use HMD (Head Mounted Display), smartphones, or other display devices to render virtual content that can be associated with the real world, allowing users to simultaneously see the real world and the virtual content superimposed on it and interact with these virtual contents. Currently, most applications of mixed reality technology are single-person experiences, that is, the virtual content rendered by the device can only be seen by the user wearing the device. Even if two or more users are in the same real space, the virtual content they see is different, so that more than two users cannot interact with the same virtual content at the same time. Furthermore, since the content rendered by the head display device can only be seen by the wearer, this mixed reality experience can only be observed from a first-person perspective. A third party without a head display cannot see the content that the wearer is playing from a third-person perspective, making the dissemination of the mixed reality experience difficult.

To achieve virtual content interaction between different devices, considering that different platforms and devices have different spatial scanning calculation methods, if the mixed reality devices cannot synchronize space, it will be difficult for them to work together, and the operation of one mixed reality device will be difficult to be reflected in real time in other mixed reality devices. Therefore, a simple and convenient collaborative work algorithm is needed to synchronize the spatial coordinates between different devices and platforms, which involves the synchronization of the spatial coordinate systems of each device.

In order to solve the problem of lack of online interaction between multiple devices, the Chinese invention patent with the authorization announcement number "CN107945116B" discloses a coordinate axis synchronization method for realizing AR indoor positioning between multiple devices. The technical solution includes the following steps: on each device participating in the online connection, the local coordinate system of each device is established using its own position and the angle of the gyroscope; any one device is selected as the main reference device, and the other devices are selected as slave devices, the coordinate system of the main reference device is used as the main reference coordinate system, and the coordinate system of other slave devices is corrected; the slave device obtains the position information of the main reference device and converts it into the coordinates of the slave device based on the main reference coordinate system. The multi-device coordinate axis synchronization method provided by the technical solution can form a unified coordinate system between various devices, and conduct more convenient and unified online interaction, which is helpful to realize the combination of virtual and reality in a large range. However, for mixed reality usage scenarios with high real-time requirements, such as augmented reality multiplayer battle games, due to the delay of the ISP (Internet Service Provider) network or the rapid transformation of the virtual space, the accuracy of coordinate synchronization is insufficient, and the user experience is poor.

In order to improve the accuracy of the synchronization of the spatial coordinate system of the mixed reality device, the Chinese invention patent with the authorization announcement number "CN109766001B" discloses a method for unifying the coordinate systems of different MR devices. The unification method includes: taking any point in the preset space as a reference point, obtaining the relative position and rotation information of each calibration plate in the preset space relative to the reference point; binding each MR device to the corresponding calibration plate; establishing a global coordinate system with the reference point as the origin; unifying the relative position and rotation information of the original coordinate system of each MR device into the global coordinate system. This technical solution sets a calibration plate and a reference point in the preset space, binds the MR device to the calibration plate, determines the position of each MR device in the global coordinate system established with the reference calibration plate as the origin through the relative position and rotation information of the calibration plate and the reference point, and corresponds the MR device to the global coordinate system, thereby realizing the interaction of different MR devices in the same scene. The synchronization process of this technical solution is relatively cumbersome, resulting in a low synchronization speed of the spatial coordinate system.

Summary of the invention

A local multi-device fast spatial anchor synchronization method for mixed reality includes the following steps:

S1: Multiple devices in a local space establish a local network connection to achieve data transmission between the devices. Any one of the multiple devices is used as a host to create a local network, and other devices are used as guests to join the local network created by the host.

S2: Time synchronization is performed between the host and the guest in the local network. The guest and the host first exchange their local timestamps with each other, and then calculate the time offset between the local timestamp of the guest and the local timestamp of the host, and adjust and implement the time synchronization between the guest and the host according to the time offset.

S3: Synchronize the spatial anchor points of the host and a guest in the local network, and use a specific image displayed on the display system of the host as the anchor point. The guest scans the specific image through augmented reality image tracking technology to obtain the second position and rotation of the image anchor point in the spatial coordinate system of the guest, and requests the host for the first position and rotation of the anchor point in the spatial coordinate system of the host at the same timestamp. The guest obtains two position and rotation data as a pair of pose pairs. The guest scans the anchor point multiple times to obtain multiple pose pairs, and calculates the multiple pose pairs through the least squares method to obtain the third position and rotation of the spatial coordinate origin of the host in the spatial coordinate system of the guest. By resetting the origin of the spatial coordinate system of the guest to the third position and rotation, the position and rotation of the origin of the spatial coordinate system of the host and the guest are the same, and the synchronization of the spatial coordinate systems between devices is completed. Other guests can synchronize the spatial coordinate system with the host in the local network, or synchronize the spatial coordinate system with the guest in the local network that has completed the spatial coordinate system synchronization.

Preferably, the step S1 specifically comprises:

S1-1: The host creates a local network and broadcasts on the local network via wireless communication.

S1-2: The guest machine searches for the host's broadcast via the wireless communication method.

S1-3: The guest machine discovers the broadcast and sends a connection request to the host machine using the wireless communication method.

S1-4: The host receives the connection request, agrees to the connection request, and establishes a network connection between the guest and the host.

Preferably, the step S2 is specifically:

S2-1: A guest sends the local timestamp of the guest to the host.

S2-2: The host sends the time point at which the local timestamp 1 is received as the local timestamp 2 to the guest machine.

S2-3: The guest machine receives the time point of the local timestamp 2 as the local timestamp 3, and calculates and saves the timestamp offset between the guest machine and the host machine according to the local timestamp 1, the local timestamp 2 and the local timestamp 3.

S2-4: Repeat steps S2-1 to S2-3 until the number of timestamp offsets reaches a set value, thereby obtaining multiple timestamp offsets.

S2-5: Calculate the standard deviation of the multiple timestamp offsets. If the standard deviation is not less than the preset value, abandon the earliest calculated timestamp offset and repeat steps S2-1 to S2-3 until the standard deviation is less than the preset value. Calculate the average value of the multiple timestamp offsets as the final timestamp offset result.

S2-6: Other guest machines calculate the final timestamp offset result between each other guest machine and the host machine according to steps S2-1 to S2-5.

Preferably, the formula for calculating the timestamp offset in step S2-3 is:

Among them, T0 is local timestamp one, T0′ is local timestamp two, and T1 is local timestamp three.

Preferably, the guest creates a queue to store the multiple timestamp offsets.

Preferably, the step S3 is specifically:

S3-1: A specific picture is displayed on the display system of the host, and the specific picture is an anchor point in the space. The host uses an accelerometer, a gyroscope and an imaging system to calculate the position and rotation of the specific picture in the local coordinate system of the host in real time based on the unchanged offset vector of the specific picture and the host imaging system.

S3-2: A passenger plane in the local network uses an imaging system to scan the specific image through augmented reality image tracking technology. The augmented reality image tracking technology identifies the second position and rotation of the specific image in the scene, sends the second position and rotation to the host, and receives the first position and rotation sent by the host. The first position and rotation and the second position and rotation have the same timestamp.

S3-3: The guest aircraft obtains multiple pairs of first positions and rotations, and second positions and rotations at different timestamps, calculates the best displacement vector and the best rotation by the least squares method, obtains the position and rotation of the origin of the local space coordinate system of the host relative to the origin of the local space coordinate system of the guest aircraft, resets the origin of the local space coordinate system of the guest aircraft to the origin of the local space coordinate system of the host aircraft, and completes the spatial coordinate system synchronization of the host aircraft and the guest aircraft. For other guest aircraft in the local network, the spatial coordinate system of the devices in the local network is synchronized by resetting the origin of its own local space coordinate system to the origin of the local space coordinate system of the host aircraft or the origin of the local space coordinate system of any synchronized guest aircraft.

S3-4: Render a virtual object of the same size as the host device in the mixed reality virtual space of the guest aircraft, and synchronize the position and rotation of the virtual object with the host in real time. If the virtual object does not completely overlap with the host in reality, the guest aircraft will repeat steps S3-1 to S3-3 until the virtual object completely overlaps with the host in reality.

Preferably, the expression of the optimal displacement vector t is:

表示所有锚点在所述主机空间坐标系下的权重中心值，

表示所有锚点在所述一客机空间坐标系下的权重中心值，

是一个沿空间Y轴旋转φ角度的旋转矩阵。 in,

Represents the weight center value of all anchor points in the host space coordinate system,

represents the weight center value of all anchor points in the space coordinate system of the aircraft,

It is a rotation matrix that rotates by an angle φ along the Y axis of the space.

Preferably, the optimal rotation φ is calculated as follows:

Among them, P′ _a,i,x and P′ _a,i,z are the differences between an anchor point in the spatial coordinate system of the host and its weight center, P′ _b,i,x and P′ _b,i,z are the differences between an anchor point in the spatial coordinate system of the guest aircraft and its weight center, R′ _i,1,x , R′ _i,1,z , R′ _i,3,x and R′ _i,3,z are the rotation differences between the spatial coordinate systems of the host and a guest aircraft, and c is an adjustable constant greater than 0. By adjusting the value of c, the formula can be made more biased towards a smaller rotation error or a smaller displacement error.

A local multi-device fast spatial anchor synchronization system for mixed reality, the system comprising a memory and a processor, the memory storing instructions, the instructions enabling the processor to execute: the above-mentioned local multi-device fast spatial anchor synchronization method for mixed reality.

A machine-readable storage medium having instructions stored thereon, wherein the instructions are used to cause a machine to execute: the above-mentioned local multi-device fast spatial anchor synchronization method for mixed reality.

Compared with the prior art, the present invention has the following technical effects:

1. The synchronization method provided by the present invention first establishes a local network in the local space, which helps to overcome the dependence on the external network and reduce the network delay to improve the accuracy and speed of spatial synchronization; then all devices in the local network are time synchronized to further improve the accuracy of spatial synchronization; on the basis of establishing the local network and time synchronization, the spatial anchor point is determined, and the optimal position and optimal rotation are calculated using the least squares method according to the position and rotation of the anchor point in each spatial coordinate system to complete the spatial synchronization between devices, thereby meeting the real-time and accuracy requirements of mixed reality applications for spatial coordinate synchronization.

2. The synchronization method provided by the present invention selects any device in the local space as the host, establishes a local network through a local wireless communication method, and other devices join the local network as guests, thereby overcoming the dependence on the external network environment, so that multiple devices in the local space can use multi-person mixed reality applications even when the external network signal is poor or there is no external network signal.

3. The synchronization method provided by the present invention uses a specific image displayed by the device display system as an anchor point. A QR code image is a good choice because the QR code image is complex enough and has enough feature points, which is conducive to rapid recognition by the scanning device. In addition, the QR code image itself has the characteristic of carrying information, which can send additional information from the scanned device to the scanning device without transmitting it through the network.

4. The synchronization method provided by the present invention can be applied to both handheld mixed reality devices (mobile phones and tablet computers) and mixed reality head displays (HMDs) driven by mobile phones. Any number of the above two devices can establish a network and synchronize spatial anchor points through this method, thereby providing a multi-person mixed reality experience.

5. The synchronization method provided by the present invention uses augmented reality image tracking technology to perform fast spatial coordinate synchronization between two devices by having the guest plane scan a specific image displayed on the host screen as an anchor point, or by having the guest plane scan a specific image displayed on the screen of a synchronized guest plane as an anchor point. This allows synchronization of spatial coordinate systems without the need for manual input of information in advance, and the synchronization speed is fast.

6. The synchronization method provided by the present invention supports any number of mixed reality devices to connect and synchronize spatial coordinates in the local space. When the number of devices exceeds two, the devices that are subsequently added can synchronize spatial coordinates by chain image tracking. Chain image tracking here means that, assuming that host A and guest B have established a network connection and completed synchronization, the newly added guest C can not only synchronize spatial coordinates by scanning the specific picture displayed by host A, but also synchronize spatial coordinates by scanning the picture displayed by guest B. Similarly, when guest D joins, guest D can complete spatial coordinate synchronization with all devices by scanning specific pictures on any one of devices A, B, and C, so as to provide convenient synchronization between spatial coordinate systems.

7. In the spatial synchronization stage, the synchronization method provided by the present invention uses the least squares method to calculate the optimal rotation and optimal displacement values between a pair of spatial coordinate systems, which can easily obtain unknown data and minimize the sum of squares of errors between the obtained data and the actual data, thereby ensuring the accuracy of spatial synchronization between different spatial coordinate systems.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a local multi-device fast spatial anchor synchronization method for mixed reality according to the present invention;

2 is a flow chart of local device networking of the local multi-device fast spatial anchor synchronization method for mixed reality according to the present invention;

3 is a flow chart of time synchronization between local devices of the local multi-device fast spatial anchor synchronization method for mixed reality according to the present invention;

FIG4 is a flow chart of spatial synchronization between local devices of the local multi-device fast spatial anchor synchronization method for mixed reality described in the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in combination with specific embodiments of the present application and with reference to the accompanying drawings.

Please refer to FIG. 1 , which is a flow chart of this embodiment.

The synchronization method described in this embodiment includes three steps: 1. Establishing a local network, a device in the local space acts as a host to create a local network, and other devices act as guests to join the local network; 2. Local timestamp synchronization of multiple devices in the local network, the host and the guest perform time synchronization by calculating the offset of the local timestamp; 3. Rapid spatial anchor point synchronization of multiple devices in the local network, the host and the guest or the guest and the guest share the position information and rotation information of the same image anchor through image tracking technology, and synchronize the spatial coordinate system in combination with the time information.

Spatial Anchors represent physical points that exist in the virtual world. For mixed reality applications, holograms can be attached to spatial anchors. Spatial anchors can be stored and persisted and later queried by the device that created it or any other supported device. This enables backup and sharing of anchors. For example, two people can view a chessboard at the same location (desktop) in the real world on the device side. No matter where you move in physical space, the chessboard will be fixed to a point.

Please refer to FIG. 2 , which is a flowchart of local device networking in this embodiment.

When there are more than one (two or more) devices in the local area, any device can create a local network as a host, and other devices can join the local network established by the host as guests. First, any device A in the local space creates a local network as a host and continuously broadcasts its existence to all nearby devices. Then, another device B in the local space begins to try to join the local network. Device B begins to search for hosts in the nearby space that have established a local network and are broadcasting themselves. After searching for the existence of host A, device B sends a connection request to host A. Finally, after receiving the connection request from device B, host A agrees to the connection request, so that device B joins the local network as guest B. Any number of subsequent devices can join the local network created by host A as guests by repeating the above process.

The local network of this embodiment is based on Bluetooth or wireless LAN (Wi-Fi), and can be directly connected point-to-point without the mobile provider network (4G or 5G) and the router providing the Internet, which can avoid the delay interference of the Internet. The user does not need to do anything when the device is connected. As long as the relevant application software is opened on the mobile phone or tablet, the devices will be automatically connected. This allows multiple devices in the local space to use multi-person mixed reality applications even when the external network signal is poor or there is no external network signal.

Please refer to FIG. 3 , which is a flow chart of time synchronization between local devices according to this embodiment.

After the local network is established and all devices have joined the local network, the devices in the local network will synchronize their time, that is, calculate the local timestamp offset between the two devices. Each device has its own local timestamp, which is used to mark the time when any local operation occurs. There are many ways to implement timestamps in a device, such as calculating the time interval from the device power on to the operation in seconds. Regardless of the timestamp implementation method used, the timestamp remains correct and consistent on the local device, and the units of timestamps between multiple devices are the same, which has no effect on the implementation of this technology. The purpose of calculating the timestamp offset between the two devices is to align the position and rotation of the two devices at the same time in the third step. All subsequent devices connected to the local network can synchronize their timestamps by repeating the same operation.

The specific process of time synchronization of devices in the local network is as follows: at a certain moment, the local timestamp of guest B is T0. Guest B sends this timestamp T0 as a parameter to host A and requests the timestamp of host A. When host A receives the timestamp request from guest B, its local timestamp is T1. Host A sends the received timestamp T0 and its own timestamp T1 back to guest B at the same time. When guest B receives the timestamp sent back by the host, it has three timestamps locally on guest B, namely T0 when guest B sent the timestamp request to the host before, T1 when host A received the timestamp request, and T0' when guest B received the timestamp sent back by the host. The time offset between guest B and host A can be calculated using the following formula:

At this time, there is a time offset data in the guest B. Considering the influence of network delay, it is difficult to obtain an accurate value by calculating the timestamp offset only once. Therefore, in this embodiment, in order to ensure the accuracy and stability of the timestamp offset, the above steps will be repeated, and the timestamp offset obtained each time will be stored in a queue Q created by the guest B. When the number of timestamp offsets stored in the queue Q reaches a preset number value, the standard deviation of the entire queue Q is calculated. If the standard deviation of the queue is too large (greater than the preset standard deviation value), the earliest calculated timestamp offset data in the queue is abandoned (this is the queue dequeue operation), and then the process of obtaining timestamps T0, T1, T0' and calculating the time offset is repeated once, and a new time offset data is obtained and added to the queue. If the queue variance is less than the preset variance value, this means that the timestamp offset at this time has stabilized. The average value of the entire queue Q can be calculated as the final timestamp offset between the guest B and the host A.

Please refer to FIG. 4 , which is a flowchart of spatial synchronization between local devices according to this embodiment.

After the guest B establishes a local network connection with the host A and synchronizes the local timestamp, the host A displays a specific picture as an anchor point on the display system, and starts recording the local pose of the picture over a period of time (pose includes position and rotation) and attaching a timestamp, and stores these poses with timestamps in the queue Q1. Guest B uses augmented reality image tracking technology to scan and try to identify the specific picture displayed on the display system of host A. Every time guest B finds the image anchor point of host A through the augmented reality image tracking algorithm, guest B immediately sends the local pose of this anchor point with a timestamp to host A.

The specific image as the anchor point is displayed on the display system of host A. Host A itself is an augmented reality device. Through its accelerometer, gyroscope and imaging system, it can calculate the position and rotation of the imaging system in the local coordinate system of host A in real time. The difference between the center position of the image and the position of the imaging system is fixed, so host A can calculate the position and rotation of the anchor point of this image in real time. Passenger plane B uses augmented reality image tracking technology to continuously search for the specific image displayed on the host in the images taken by its imaging system (this image is preset in advance, so passenger plane B knows what image is displayed on host A). This augmented reality image tracking technology can accurately identify the position and rotation of a specific image in the scene, so passenger plane B can also know the position and rotation of the anchor point of the image.

The above-mentioned augmented reality picture tracking technology for scanning specific pictures can use Apple's ARKit technology, or other imaging technologies that support the MarkerTracking method.

The above specific images are preset by the program. It is best to use images that are complex enough and have enough feature points. QR code images are a very good choice. In addition to the advantage of being easy to identify, the QR code image itself has the characteristic of carrying information, which can send additional information from host A to guest B without going through the network. In addition to QR code images, any image with enough feature points can also be used as an image anchor.

While the guest B is scanning the specific image displayed on the display system of the host A, the two devices are also sharing their spatial positions. The positions and rotations of the two devices in the local coordinate system are transmitted to each other through the local network. Every time the guest B successfully scans the specific image displayed on the display system of the host A as an anchor point, both devices know the position and rotation of this anchor point in the local spatial coordinate system of each device. Through the local network and timestamp synchronization, the guest B can obtain multiple pairs of positions and rotations of this anchor point in the spatial coordinate systems of the two devices at the same time. Through the least squares method, the position and rotation of the origin of the local spatial coordinate system of the host A relative to the origin of the local spatial coordinate system of the guest B are calculated. At this time, we reset the origin of the local coordinate system of the guest B to the origin of the host A, and the origin and direction of the spatial coordinate systems of the two devices are the same, so as to achieve synchronization of the spatial coordinate systems of the two devices.

The least squares method objectives of this embodiment are as follows:

是一队列图片锚点在主机A本地空间坐标系下的位姿，

是一队列图片锚点在客机B本地空间坐标系下的位姿，队列

与队列

中的锚点在现实世界的时间中一一对应。 Ra _,i is the rotation of an anchor point in the space coordinate system of host A, _Pa,i is the position of an anchor point in the space coordinate system of host A, _Rb,i is the rotation of an anchor point in the space coordinate system of passenger B, Pb _,i is the position of an anchor point in the space coordinate system of passenger B,

is the position of the anchor point of a queue image in the local space coordinate system of host A.

is the position of the anchor point of a queue image in the local space coordinate system of aircraft B.

With queue

The anchor points in the graph correspond one-to-one in real-world time.

Because both mixed reality devices can use the built-in gyroscope to find the same Y-axis orientation based on gravity, we need to find a one-dimensional rotation along the Y-axis of the coordinate system:

and a three-dimensional vector t, to transfer the origin of the coordinate system of aircraft B to the origin of the coordinate system of aircraft A. By solving a rotation angle φ and vector t, the following formula is minimized (minimum error):

F in the above formula can be expressed as:

Wherein, w represents the weight of each pair of anchor points, w _i >0; c represents the weight of the penalty, c >0; c is a constant. In this embodiment, position and rotation data with errors are processed simultaneously. By adjusting the value of c, the formula can be more biased towards a smaller rotation error or a smaller displacement error.

是固定的，可以通过求导t从而求出最佳位移向量，公式如下： Then, calculate the optimal displacement vector, assuming

is fixed, and the optimal displacement vector can be obtained by taking the derivative of t. The formula is as follows:

The expression for the optimal displacement vector t can be derived:

和

分别是 in

and

They are

表示所有锚点在主机A坐标系下的权重中心值，

表示所有锚点在客机B坐标系下的权重中心值。在假设旋转φ不变的情况下，我们可以通过计算A与B的权重中心来计算出最佳位移向量t。 in,

Indicates the weight center value of all anchor points in the host A coordinate system.

represents the weight center value of all anchor points in the aircraft B coordinate system. Assuming that the rotation φ remains unchanged, we can calculate the optimal displacement vector t by calculating the weight center of A and B.

Finally, calculate the rotation:

P′ _a,i is the difference between an anchor point in the spatial coordinate system of host A and its weight center, and P′ _b,i is the difference between an anchor point in the spatial coordinate system of passenger aircraft B and its weight center.

According to the above expression of t when the rotation φ is determined, the optimal displacement vector t is substituted into the target formula F to obtain:

可以在已知一对锚点旋转的情况下计算目标旋转

其中，R′ _i为目标 旋转(两坐标系之间的旋转差值)，

为锚点在客机坐标系下的旋转矩阵的倒置矩阵，R _a,i为锚点在主机坐标系下的旋转)再通过求导φ的方式得到最佳旋转： Since the trace of matrix AB is the same as that of matrix BA, i.e., tr(AB) = tr(BA),

The target rotation can be calculated when a pair of anchor point rotations are known

Where R′ _i is the target rotation (the rotation difference between the two coordinate systems),

is the inverted matrix of the rotation matrix of the anchor point in the passenger aircraft coordinate system, Ra _,i is the rotation of the anchor point in the host coordinate system) and then the optimal rotation is obtained by deriving φ:

The optimal rotation φ obtained from the above calculation is:

Any number of devices that subsequently join the network can complete spatial coordinate synchronization with host A through this method. After successful synchronization, a virtual object with the same size as the host A device is rendered in the mixed reality virtual space of guest B, and the position and rotation of the virtual object are synchronized with host A in real time. If the spatial synchronization accuracy of the two devices is high enough, from the perspective of guest B, the virtual object and the real host A should completely overlap; if the error between the two is too large, it means that the synchronization is unsuccessful, and guest B will repeat the spatial coordinate system synchronization process until it successfully achieves accurate spatial coordinate system synchronization with host A.

In another embodiment of the present invention, the device that subsequently joins the local network completes the synchronization of the spatial coordinate system by scanning the specific picture displayed on the display system of the guest machine that has synchronized the spatial coordinate system in the local network. Guest machine B connects to the local network created by host A and completes the synchronization of the spatial coordinate system with host A by scanning the specific picture on the display system of host A. Subsequently, a new guest machine C joins the network. In addition to scanning the picture on host A, guest machine C can also synchronize the spatial coordinate system with guest machine B by scanning the specific picture on the display system of guest machine B. Since guest machine B has completed the spatial coordinate system synchronization with host A, the three devices A, B, and C have successfully completed the spatial synchronization with each other. Similarly, when a new guest machine D joins, guest machine D can synchronize the spatial coordinate system with all devices in the network by scanning the spatial synchronization of the specific picture with any one of the devices A, B, and C. Chain spatial coordinate synchronization is achieved by the above method, so that any device in the local network can synchronize the spatial coordinate system with other devices nearby.

A local multi-device fast spatial anchor synchronization system for mixed reality, the system comprising a memory and a processor, the memory storing instructions, the instructions enabling the processor to execute: the local multi-device fast spatial anchor synchronization method for mixed reality described in the present invention.

A machine-readable storage medium having instructions stored thereon, wherein the instructions are used to enable a machine to execute: the local multi-device fast spatial anchor point synchronization method for mixed reality described in the present invention.

The above is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several modifications and improvements without departing from the inventive concept of the present invention, which all fall within the protection scope of the present invention.

Claims (10)

A local multi-device fast spatial anchor synchronization method for mixed reality, characterized in that it includes the following steps: S1: Multiple devices in a local space establish a local network connection to achieve data transmission between the devices, any one of the multiple devices is used as a host to create a local network, and other devices are used as guests to join the local network created by the host; S2: performing time synchronization on the host and the guest in the local network, wherein the guest and the host first exchange their local timestamps with each other, then calculate the time offset between the local timestamp of the guest and the local timestamp of the host, and adjust and realize the time synchronization between the guest and the host according to the time offset; S3: Synchronize the spatial anchor points of the host and a guest in the local network, and use a specific image displayed on the display system of the host as the anchor point. The guest scans the specific image through augmented reality image tracking technology to obtain the second position and rotation of the image anchor point in the spatial coordinate system of the guest, and requests the host for the first position and rotation of the anchor point in the spatial coordinate system of the host at the same timestamp. The guest obtains two position and rotation data as a pair of pose pairs. The guest scans the anchor point multiple times to obtain multiple pose pairs, and calculates the multiple pose pairs through the least squares method to obtain the third position and rotation of the host spatial coordinate origin in the spatial coordinate system of the guest. By resetting the spatial coordinate system origin of the guest to the third position and rotation, the position and rotation of the spatial coordinate system origin of the host and the guest are the same, and the synchronization of the spatial coordinate systems between devices is completed. Other guests can synchronize the spatial coordinate system with the host in the local network, or synchronize the spatial coordinate system with the guest in the local network that has completed the spatial coordinate system synchronization. The local multi-device fast spatial anchor synchronization method for mixed reality according to claim 1, characterized in that step S1 specifically comprises: S1-1: The host creates a local network and broadcasts on the local network via wireless communication; S1-2: the guest machine searches for the broadcast of the host machine through the wireless communication method; S1-3: the guest machine discovers the broadcast and sends a connection request to the host machine using the wireless communication method; S1-4: The host receives the connection request, agrees to the connection request, and establishes a network connection between the guest and the host. The local multi-device fast spatial anchor synchronization method for mixed reality according to claim 1, characterized in that step S2 specifically comprises: S2-1: A guest sends a local timestamp of the guest to the host; S2-2: the host sends the time point of receiving the local timestamp 1 as the local timestamp 2 to the guest machine; S2-3: the guest machine receives the time point of the local timestamp 2 as the local timestamp 3, and calculates and saves the timestamp offset between the guest machine and the host machine according to the local timestamp 1, the local timestamp 2 and the local timestamp 3; S2-4: Repeat steps S2-1 to S2-3 until the number of timestamp offsets reaches a set value, thereby obtaining multiple timestamp offsets; S2-5: Calculate the standard deviation of the multiple timestamp offsets. If the standard deviation is not less than a preset value, discard the earliest calculated timestamp offset and repeat steps S2-1 to S2-3 until the standard deviation is less than the preset value. Calculate the average value of the multiple timestamp offsets as the final timestamp offset result. S2-6: Other guest machines calculate the final timestamp offset result between each other guest machine and the host machine according to steps S2-1 to S2-5. The local multi-device fast spatial anchor synchronization method for mixed reality according to claim 3 is characterized in that the formula for calculating the timestamp offset offset in step S2-3 is:

Among them, T0 is local timestamp one, T0′ is local timestamp two, and T1 is local timestamp three. The local multi-device fast spatial anchor synchronization method for mixed reality according to claim 3 is characterized in that the client creates a queue to save the multiple timestamp offsets. The local multi-device fast spatial anchor synchronization method for mixed reality according to claim 1, characterized in that step S3 specifically comprises: S3-1: a specific picture is displayed on the display system of the host, and the specific picture is an anchor point in space. The host uses an accelerometer, a gyroscope and an imaging system to calculate the position and rotation of the specific picture in the local coordinate system of the host in real time according to the invariance of the offset vector of the specific picture and the imaging system of the host; S3-2: A passenger plane in the local network scans the specific image using an imaging system through augmented reality image tracking technology, and the augmented reality image tracking technology identifies a second position and rotation of the specific image in the scene, sends the second position and rotation to the host, and receives the first position and rotation sent by the host, wherein the first position and rotation and the second position and rotation have the same timestamp; S3-3: The guest aircraft obtains multiple pairs of first positions and rotations, and second positions and rotations at different timestamps, calculates the best displacement vector and the best rotation by the least square method, obtains the position and rotation of the origin of the local space coordinate system of the host relative to the origin of the local space coordinate system of the guest aircraft, resets the origin of the local space coordinate system of the guest aircraft to the origin of the local space coordinate system of the host, and completes the spatial coordinate system synchronization of the host and the guest aircraft; for other guest aircraft in the local network, the spatial coordinate systems of the devices in the local network are synchronized by resetting their own local space coordinate system origins to the origin of the local space coordinate system of the host or the origin of the local space coordinate system of any synchronized guest aircraft; S3-4: Render a virtual object of the same size as the host device in the mixed reality virtual space of the guest aircraft, and synchronize the position and rotation of the virtual object with the host in real time. If the virtual object does not completely overlap with the host in reality, the guest aircraft will repeat steps S3-1 to S3-3 until the virtual object completely overlaps with the host in reality. The local multi-device fast spatial anchor point synchronization method for mixed reality according to claim 6, characterized in that the expression of the optimal displacement vector t is:

in,

Represents the weight center value of all anchor points in the host space coordinate system,

represents the weight center value of all anchor points in the space coordinate system of the aircraft,

It is a rotation matrix that rotates by an angle φ along the Y axis of the space. The local multi-device fast spatial anchor synchronization method for mixed reality according to claim 6, characterized in that the calculation method of the optimal rotation φ is:

Among them, P′ _a,i,x and P′ _a,i,z are the differences between an anchor point in the spatial coordinate system of the host and its weight center, P′ _b,i,x and P′ _b,i,z are the differences between an anchor point in the spatial coordinate system of the guest aircraft and its weight center, R′ _i,1,x , R′ _i,1,z , R′ _i,3,x and R′ _i,3,z are the rotation differences between the spatial coordinate systems of the host and a guest aircraft, and c is an adjustable constant greater than 0. By adjusting the value of c, the formula can be made more biased towards a smaller rotation error or a smaller displacement error. A local multi-device fast spatial anchor synchronization system for mixed reality, characterized in that the system includes a memory and a processor, the memory stores instructions, and the instructions enable the processor to execute: a local multi-device fast spatial anchor synchronization method for mixed reality according to any one of claims 1-8. A machine-readable storage medium, characterized in that instructions are stored on the machine-readable storage medium, and the instructions are used to enable a machine to execute: a local multi-device fast spatial anchor synchronization method for mixed reality according to any one of claims 1-8.

Images