CN118354056A - A three-dimensional video communication method with high sensitivity to motion parallax - Google Patents

Abstract

The invention discloses a three-dimensional video communication method with high-sensitivity motion parallax, which belongs to the field of computer technology. The method tracks eye postures through a computer vision algorithm, predicts the motion trajectory of the eyes in space and the future posture, transmits the predicted eye posture information to a three-dimensional reconstruction and rendering module, calculates and obtains a picture image under the future perspective, achieves the effect of compensating for acquisition, transmission, and calculation delays, and realizes three-dimensional video communication with high-sensitivity motion parallax. The method tracks and predicts the eye postures, transmits the information to a three-dimensional reconstruction and rendering module, generates a picture under the future perspective, realizes high-quality three-dimensional video communication, and provides a more immersive and realistic experience.

Description

A three-dimensional video communication method with high sensitivity to motion parallax

Technical Field

The invention belongs to the field of computers, and in particular relates to a three-dimensional video communication method with high-sensitivity motion parallax.

Background technique

With the continuous development of the Internet, AI, and cloud computing, three-dimensional video communication has become the mainstream mode of remote communication in the future. Compared with traditional two-dimensional, three-dimensional video communication enables local users to more accurately perceive the movement, depth, and spatial position of each other by capturing the three-dimensional images and environmental information of remote participants. It can not only make up for the obstacles caused by geographical distance, but also provide a more immersive experience and promote real-time cooperation and communication. In order to achieve the 3D effect of viewing, binocular parallax, motion parallax, and depth parallax need to be considered. Among them, motion parallax is a parallax effect based on the movement of the participants. By capturing the eye position of the local user, the 3D display device displays the picture of the remote participant from this perspective. However, due to the delay of calculations such as data acquisition, data transmission, three-dimensional reconstruction, and rendering, the local user's eyes will move, and there will be a delay in the picture switching of this perspective, that is, there is a hysteresis in motion parallax. To address this problem, the common methods in the industry currently reduce the delay of the picture by adding sensor devices and increasing bandwidth, but these methods cannot completely solve the hysteresis problem of motion parallax. In the current prior art, there is a lack of a three-dimensional video communication method with high sensitivity motion parallax.

Summary of the invention

The technical problem to be solved by the present invention is to provide a high-sensitivity motion parallax three-dimensional video communication method in response to the shortcomings of the background technology. The eye posture is tracked by a computer vision algorithm, and the movement trajectory of the eye in space and the future posture are predicted. The predicted eye posture information is transmitted to a three-dimensional reconstruction and rendering module, and the picture image under the future perspective is calculated, so as to achieve the effect of compensating for the acquisition, transmission, and calculation delays, and realize three-dimensional video communication with high-sensitivity motion parallax.

The present invention adopts the following technical solutions to solve the above technical problems:

A high-sensitivity motion parallax three-dimensional video communication method specifically comprises the following steps:

Step 1: On the local user side, the images collected by the collection device are used for eye tracking, and the position information of the local user's eyes in space is obtained through the eye tracking module;

Step 2: Using the camera on the remote participant's side to collect images for 3D reconstruction and rendering;

Step 3: The spatial position information of the local user's eyes and the image data of the remote participant are transmitted to the cloud host, and the 3D reconstruction rendering module generates a new perspective image according to the eye position and image information;

Step 4: Transmit the new view image to the local user.

As a further preferred solution of the high-sensitivity motion parallax three-dimensional video communication method of the present invention, in step 1, an eye tracking module is used to obtain the position information of the local user's eyes in space through AI estimation or geometric algorithm, which specifically includes the following steps:

Step 1.1, initialize the positions of the two cameras, calculate the relative position relationship between the two cameras, use the camera calibration algorithm to obtain the camera internal and external parameters according to the camera shooting picture, or use the depth sensor to solve in three-dimensional space or use the artificial intelligence algorithm to estimate the relative position;

Step 1.2, perform face recognition and face key point detection on the images taken by the dual cameras, extract the corresponding facial feature points, and obtain the pixel coordinates of each feature point in the image and the corresponding relationship between the feature points in the two cameras;

Step 1.3, according to the relative relationship between the two cameras and the feature points in the picture, determine the unique spatial triangle and use the geometric solution method to calculate the position coordinates of the eye in the three-dimensional space;

Step 1.4, calculate the sight direction of the two eyes by combining the facial features and the head posture to obtain the three-dimensional spatial posture of the eyes.

As a further preferred solution of the high-sensitivity motion parallax three-dimensional video communication method of the present invention, in step 1, an eye posture prediction module is added to compensate for the acquisition, transmission, and calculation delays by predicting the eye trajectory in the future and the eye posture at a certain moment in the future, thereby solving the hysteresis phenomenon of motion parallax.

As a further preferred solution of the high-sensitivity motion parallax 3D video communication method of the present invention, an eye position posture prediction module is added, and the specific steps are as follows:

The local user's camera collects images, and the eye tracking module obtains the local user's eye position information in space;

An eye pose prediction module is added to estimate the trajectory and position of eye movement through a time series prediction algorithm or a state estimation algorithm, and the predicted future timestamp and corresponding pose information are sent to the reconstruction and rendering module.

As a further preferred solution of the high-sensitivity motion parallax 3D video communication method of the present invention, the data transmission protocol of eye tracking and prediction needs to be customized, including timestamp and posture information, and the format is as follows:

{(ts_curr,pos_curr),

(ts_pred1,pos_pred1),

(ts_pred2,pos_pred 2),…}

Among them, ts_curr: current timestamp; pos_curr: current eye pose; ts_pred: future timestamp, where the number indicates multiple predictions; pos_pred: predicted eye pose, where the number indicates multiple predictions.

As a further preferred solution of the high-sensitivity motion parallax 3D video communication method of the present invention, the reconstruction and rendering module receives multiple predicted eye postures, generates multiple new perspective images, and transmits the images back to the local user. The transmission protocol needs to be customized, including timestamp, posture information, frame image information, and the format is as follows:

{(ts_curr,pos_curr,frame_curr),

(ts_pred1,pos_pred1,frame_pred1),

(ts_pred2,pos_pred 2,frame_pred2),…}

Among them, frame represents the picture frame of a new perspective generated according to different eye postures at different times.

As a further preferred solution of the high-sensitivity motion parallax three-dimensional video communication method of the present invention, in step 1, according to the requirements of the eye tracking algorithm, the acquisition device uses a camera, and the sampling camera uses a single camera or multiple cameras.

As a further preferred solution of the high-sensitivity motion parallax 3D video communication method of the present invention, in step 2, according to the requirements of the reconstruction algorithm, the acquisition device uses a camera, and the acquisition camera uses a single camera or multiple cameras.

As a further preferred solution of the high-sensitivity motion parallax three-dimensional video communication method of the present invention, in step 2, the acquisition camera adopts an RGB-D camera.

As a further preferred solution of the high-sensitivity motion parallax three-dimensional video communication method of the present invention, in step 1, the acquisition camera is an RGB camera.

Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

1. A high-sensitivity motion parallax 3D video communication method of the present invention tracks eye postures through a computer vision algorithm, predicts the motion trajectory of the eye in space and the future posture, transmits the predicted eye posture information to a 3D reconstruction and rendering module, calculates and obtains the screen image under the future perspective, achieves the effect of compensating for the acquisition, transmission, and calculation delays, and realizes high-sensitivity motion parallax 3D video communication;

2. This method tracks and predicts the eye posture, transmits the information to the 3D reconstruction rendering module, generates images from the future perspective, realizes high-quality 3D video communication, and provides a more immersive and realistic experience;

3. This method takes into account the acquisition, transmission, and calculation delays during the processing process, and achieves the effect of delay compensation by predicting eye movement, fundamentally solving the hysteresis problem of motion parallax and improving user experience;

4. This method does not require the addition of sensor equipment or bandwidth, has strong portability, is suitable for cloud computing scenarios, and makes full use of cloud computing power. It is a general method to reduce latency.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

FIG1 is a schematic diagram of the process that a local user needs to go through to see the 3D effect of a remote participant in a 3D video communication;

FIG. 2 is a schematic diagram of adding an eye pose prediction module.

Detailed ways

The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings:

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. The present invention is described in detail below based on the drawings and preferred embodiments, and the purpose and effect of the present invention will become clearer. It should be understood that the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

The main purpose of the present invention is to track the eye posture through computer vision algorithms, predict the movement trajectory of the eyes in space and the future posture, transmit the predicted eye posture information to the three-dimensional reconstruction and rendering module, calculate the picture image under the future perspective, achieve the effect of compensating for the acquisition, transmission and calculation delays, and realize three-dimensional video communication with high-sensitivity motion parallax.

With the continuous development of the Internet, AI, and cloud computing, three-dimensional video communication has become the mainstream mode of remote communication in the future. Compared with traditional two-dimensional, three-dimensional video communication enables local users to more accurately perceive the movement, depth, and spatial position of each other by capturing the three-dimensional images and environmental information of remote participants. It can not only make up for the obstacles caused by geographical distance, but also provide a more immersive experience and promote real-time cooperation and communication. In order to achieve the 3D effect of viewing, binocular parallax, motion parallax, and depth parallax need to be considered. Among them, motion parallax is a parallax effect based on the movement of the participants. By capturing the eye position of the local user, the 3D display device displays the picture of the remote participant from that perspective. However, due to the delay of calculations such as data acquisition, data transmission, three-dimensional reconstruction, and rendering, the local user's eyes will move, and there will be a delay in the picture switching of this perspective, that is, there is a hysteresis phenomenon in motion parallax.

In three-dimensional video communication, the local user needs to go through the following steps to see the 3D effect of the remote participant, as shown in Figure 1. First, the picture collected by the camera on the local user side is used for eye tracking. According to the requirements of the eye tracking algorithm, the collection camera may be single or multiple. The eye tracking module obtains the pose information of the local user's eyes in space (pose includes position and direction) through AI estimation or geometric algorithm; this solution mainly tracks the eye pose, uses the eye pose prediction, and compensates for the delay in the process of generating a new perspective. The steps of the eye tracking module here are just an example of a feasible means to achieve this function. Other methods to achieve the function of this module are also possible; this solution uses eye pose prediction to compensate for the delay in the process of generating a new perspective. The hysteresis of motion parallax is more obvious due to the increase of transmission delay in cloud services, but this solution can have an optimization effect even in an end-to-end manner. Here we only take the cloud service case for illustration.

At the same time, the camera on the remote participant's side collects images for 3D reconstruction and rendering. Depending on the requirements of the reconstruction algorithm, the collection camera may be a single or multiple cameras or even an RGB-D camera. The collection device includes but is not limited to cameras, and any sensor that can obtain relevant information is acceptable. It is not limited to RGB-D cameras. Depending on the algorithm requirements for 3D reconstruction or new perspective generation, RGB cameras can also be used, or other sensors that obtain the data required by the algorithm can be used.

Then, since the computing power on the client side may be limited, cloud services can be used to fully utilize the computing power. The spatial pose information of the local user's eyes and the image data of the remote participant need to be transmitted to the cloud host. The 3D reconstruction rendering module generates a new perspective image based on the above eye pose and image information. Finally, the new perspective image is transmitted to the local user.

The eye tracking module can use the following scheme to initialize the position of the two cameras, calculate the relative position relationship between the two cameras, use the camera calibration algorithm to obtain the internal and external parameters of the camera according to the camera shooting picture, or combine the depth sensor to solve in three-dimensional space or use artificial intelligence algorithm to estimate the relative position. Perform face recognition and face key point detection on the pictures taken by the two cameras, extract the corresponding facial feature points, and obtain the pixel coordinates of each feature point in the picture and the corresponding relationship between the feature points in the two cameras. According to the relative relationship between the two cameras and the feature points in the picture, determine the unique spatial triangle and use the geometric solution method to calculate the position coordinates of the eyes in three-dimensional space. Combine facial features and head posture to calculate the line of sight direction of the two eyes and obtain the position and posture of the eyes in three-dimensional space.

However, the inevitable delay in the above solution will affect the real-time motion parallax effect in 3D video communication. When the local user moves at time t1, the local user's camera captures the movement of the local user's eyes. After the delay of data collection, data transmission, reconstruction and rendering calculation, and image return, the image is seen at time t2, that is, the image can only be seen after the eye moves (t2-t1) time. This is the reason for the hysteresis of motion parallax.

In order to solve the problem of motion parallax sensitivity in three-dimensional video communication, an eye posture prediction module is added to compensate for the acquisition, transmission, and calculation delays by predicting the eye trajectory in the future and the eye posture at a certain moment in the future, thereby fundamentally solving the hysteresis phenomenon of motion parallax. The specific steps are shown in Figure 2.

First, as in the previous steps, the local user-side camera captures the image, and the eye tracking module obtains the local user's eye position information in space. Then, an eye position prediction module is added to estimate the trajectory and position of the eye movement through a time series prediction algorithm or a state estimation algorithm, and the predicted future timestamp and corresponding position information are sent to the reconstruction and rendering module. In order to solve the accuracy problem of the prediction algorithm, multiple future eye positions can be predicted. Here, the data transmission protocol for eye tracking and prediction needs to be customized, mainly including timestamps and position information, and the format is as follows:

{(ts_curr,pos_curr),

(ts_pred1,pos_pred1),

(ts_pred2,pos_pred 2),…}

Among them, ts_curr: current timestamp; pos_curr: current eye pose; ts_pred: future timestamp (number indicates multiple predictions); pos_pred: predicted eye pose (number indicates multiple predictions).

In this way, the reconstruction and rendering module will receive multiple predicted eye poses, generate multiple new perspective images, and transmit the images back to the local user. The transmission protocol needs to be customized, mainly including timestamp, pose information, frame image information, and the format is as follows:

{(ts_curr,pos_curr,frame_curr),

(ts_pred1,pos_pred1,frame_pred1),

(ts_pred2,pos_pred 2,frame_pred2),…}

Among them, frame represents the frame of the new perspective generated according to different eye poses at different times. These data are then transmitted back to the local user. At this time, the time has reached time t2. From the above set, the picture with the eye pose at time t2 that is closest to the predicted eye pose is selected for display.

The following describes the whole process of high-sensitivity motion parallax achieved by this scheme, taking the prediction of a moment's posture as an example:

The local user is in motion, and the timing starts at time t1. The moment when the camera takes the picture is recorded as ts_curr. The eye pose corresponding to the picture is calculated by the eye tracking algorithm and recorded as pos_curr. The eye position prediction module predicts the eye pose at the future time ts_pred as pos_pred. The timestamp and eye pose data are transmitted. The reconstruction and rendering module generates a new perspective picture corresponding to pos_pred, and the predicted picture is transmitted to the local user for viewing. At this time, it is time t2. Since the result pos_pred of the eye position prediction module is exactly the picture of the perspective corresponding to time t2, (t2-ts_pred) is approximately 0. Therefore, the acquisition, transmission, and calculation delays are compensated by prediction, and the problem of motion parallax hysteresis is solved.

In summary, the present invention aims to realize three-dimensional video communication with high-sensitivity motion parallax. By tracking eye posture and predicting motion trajectory and future posture, three-dimensional reconstruction and rendering of future perspective images are performed to compensate for acquisition, transmission and calculation delays, so that users can more accurately perceive the stereoscopic effect in three-dimensional communication, providing an immersive viewing experience.

1. This method tracks and predicts the eye posture, transmits the information to the 3D reconstruction rendering module, generates images from the future perspective, realizes high-quality 3D video communication, and provides a more immersive and realistic experience.

2. This method takes into account the acquisition, transmission, and calculation delays during the processing process, and achieves the effect of delay compensation by predicting eye movement, fundamentally solving the hysteresis problem of motion parallax and improving user experience.

3. This method does not require the addition of sensor equipment or bandwidth, has strong portability, is suitable for cloud computing scenarios, and makes full use of cloud computing power. It is a general method to reduce latency.

Those skilled in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention should be included in the protection scope of the invention. All technical features in this embodiment can be freely combined according to actual needs.

Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims (10)

1. A high-sensitivity motion parallax three-dimensional video communication method, characterized in that it specifically comprises the following steps: Step 1: On the local user side, the images collected by the collection device are used for eye tracking, and the position information of the local user's eyes in space is obtained through the eye tracking module; Step 2: Using the camera on the remote participant's side to collect images for 3D reconstruction and rendering; Step 3: The spatial position information of the local user's eyes and the image data of the remote participant are transmitted to the cloud host, and the 3D reconstruction rendering module generates a new perspective image according to the eye position and image information; Step 4: Transmit the new view image to the local user. 2. A high-sensitivity motion parallax 3D video communication method according to claim 1, characterized in that: in step 1, an eye tracking module is used to obtain the position information of the local user's eyes in space through AI estimation or geometric algorithm, specifically comprising the following steps: Step 1.1, initialize the positions of the two cameras, calculate the relative position relationship between the two cameras, use the camera calibration algorithm to obtain the camera internal and external parameters according to the camera shooting picture, or use the depth sensor to solve in three-dimensional space or use the artificial intelligence algorithm to estimate the relative position; Step 1.2, perform face recognition and face key point detection on the images taken by the dual cameras, extract the corresponding facial feature points, and obtain the pixel coordinates of each feature point in the image and the corresponding relationship between the feature points in the two cameras; Step 1.3, according to the relative relationship between the two cameras and the feature points in the picture, determine the unique spatial triangle and use the geometric solution method to calculate the position coordinates of the eye in the three-dimensional space; Step 1.4, calculate the sight direction of the two eyes by combining the facial features and the head posture to obtain the three-dimensional spatial posture of the eyes. 3. According to the high-sensitivity motion parallax three-dimensional video communication method described in claim 1, it is characterized in that: in step 1, an eye posture prediction module is added to compensate for the acquisition, transmission, and calculation delays by predicting the eye trajectory in the future and the eye posture at a certain moment in the future, thereby solving the hysteresis phenomenon of motion parallax. 4. The high-sensitivity motion parallax 3D video communication method according to claim 3, characterized in that: an eye position posture prediction module is added, and the specific steps are as follows: The local user's camera collects images, and the eye tracking module obtains the local user's eye position information in space; An eye pose prediction module is added to estimate the trajectory and position of eye movement through a time series prediction algorithm or a state estimation algorithm, and the predicted future timestamp and corresponding pose information are sent to the reconstruction and rendering module. 5. A high-sensitivity motion parallax 3D video communication method according to claim 4, characterized in that: the data transmission protocol for eye tracking and prediction needs to be customized, including timestamp and posture information, and the format is as follows: {(ts_curr,pos_curr), (ts_pred1,pos_pred1), (ts_pred2,pos_pred 2),…} Among them, ts_curr: current timestamp; pos_curr: current eye pose; ts_pred: future timestamp, where the number indicates multiple predictions; pos_pred: predicted eye pose, where the number indicates multiple predictions. 6. A high-sensitivity motion parallax 3D video communication method according to claim 5, characterized in that: the reconstruction and rendering module receives multiple predicted eye postures, generates multiple new perspective images, and transmits the images back to the local user. The transmission protocol needs to be customized, including timestamp, posture information, frame image information, and the format is as follows: {(ts_curr,pos_curr,frame_curr), (ts_pred1,pos_pred1,frame_pred1), (ts_pred2,pos_pred 2,frame_pred2),…} Among them, frame represents the picture frame of a new perspective generated according to different eye postures at different times. 7. A high-sensitivity motion parallax three-dimensional video communication method according to claim 1, characterized in that: in step 1, according to the requirements of the eye tracking algorithm, the acquisition device uses a camera, and the sampling camera uses a single or multiple cameras. 8. The high-sensitivity motion parallax 3D video communication method according to claim 1 is characterized in that: in step 2, according to the requirements of the reconstruction algorithm, the acquisition device uses a camera, and the acquisition camera uses a single camera or multiple cameras. 9. The high-sensitivity motion parallax three-dimensional video communication method according to claim 7, characterized in that: in step 2, the acquisition camera is an RGB-D camera. 10. The high-sensitivity motion parallax three-dimensional video communication method according to claim 8, characterized in that: in step 1, the acquisition camera is an RGB camera.