WO2025000915A1 - Holographic three-dimensional medical data visualization method and system capable of achieving mid-air gesture interaction - Google Patents

Abstract

The present application relates to the technical field of human-computer interaction, and specifically to a holographic three-dimensional medical data visualization method and system capable of achieving mid-air gesture interaction. The method comprises the following steps: firstly, acquiring user environment information and user motion information, wherein the user environment information is image information of a real scene around a user, and the user wears a mixed reality display device; then, on the basis of the user environment information and the user motion information, generating a virtual image by using a three-dimensional interpolation volume rendering method; next, by using an optical model, projecting the virtual image into the visual field of the user, and merging the virtual image with the real scene to obtain a mixed reality rendered three-dimensional model; and finally, on the basis of the mixed reality rendered three-dimensional model, capturing a gesture of the user by using a gesture recognition method, and performing mid-air gesture interaction. The present application achieves mid-air gesture interaction between a user and a virtual scene by means of efficient three-dimensional object volume rendering and a real-time accurate interaction method design.

Description

Holographic three-dimensional medical data visualization method and system capable of air gesture interaction Technical Field

The embodiments of the present application relate to the field of human-computer interaction technology, and in particular to a holographic three-dimensional medical data visualization method and system capable of air gesture interaction.

Background Art

With the continuous development and innovation of digital medicine, extended reality has gradually become a key technology to assist surgeons in clinical diagnosis and treatment. Thanks to the fusion of disciplines between medicine and computer science, multimodal medical data obtained from medical images such as computed tomography (CT), magnetic resonance imaging (MRI) and ultrasound (US), and models obtained through computer rendering and reconstruction, make three-dimensional visualization of medical data possible, providing a broad space for the development of extended reality (XR) assisted medicine.

The medical interactive visualization system based on XR and equipped with HoloLen2 head-mounted display has demonstrated its potential to present high-quality 3D patient data. Based on 3D visualization, surgical simulation provides doctors with an immersive surgical training environment, allowing doctors to operate 3D objects in the environment, while enhancing doctors' perception and helping them make intuitive diagnoses, and effectively avoiding the risks that real surgery may bring. For holographic XR medical visualization systems, efficient preoperative volume rendering and real-time and accurate intraoperative interactive technology are the key to assisting doctors in surgical diagnosis.

Unlike traditional two-dimensional display technology for medical data, XR can present three-dimensional anatomical structures more realistically and accurately during intraoperative diagnosis and preoperative surgical planning. With the development of GPU technology, volume rendering technology has made great progress in recent years. Volume rendering technology uses direct volume rendering technology to enhance visual and spatial perception from three-dimensional data sets. In the process of volume rendering using GPU, existing methods have never fully achieved the same quality and performance as in medical augmented reality. The proposed methods are limited to rendering polygonal surface data. Although the relevant anatomy and structures can be depicted, these methods are too time-consuming and cannot meet the efficiency requirements of XR-assisted diagnosis systems for volume rendering.

Technical issues

The embodiments of the present application provide a holographic three-dimensional medical data visualization method and system that can interact with gestures in the air. Through efficient three-dimensional object rendering and real-time and accurate interaction method design, users can move, rotate, scale and cut virtual tissues from any angle, manipulate virtual surgical instruments in real time and observe the patient's lesions, thereby realizing gesture interaction between users and virtual scenes in the air.

Technical Solutions

To solve the above technical problems, in the first aspect, an embodiment of the present application provides a holographic three-dimensional medical data visualization method that can interact with gestures in the air, comprising the following steps: first, obtaining user environment information and user motion information; the user environment information is image information of the real scene around the user; the user wears a mixed reality display device; then, based on the user environment information and user motion information, a stereo interpolation volume rendering method is used to generate a virtual image; next, an optical model is used to project the virtual image into the user's field of view, and the virtual image and the real scene are fused to obtain a three-dimensional model rendered by mixed reality; finally, based on the three-dimensional model rendered by mixed reality, the user's gestures are captured by a gesture recognition method and gesture interaction is performed in the air, so as to realize the interaction between the user and the virtual scene.

In some exemplary embodiments, the mixed reality display device is a head-mounted mixed reality display device HoloLens2; and the optical model is an optical model in the head-mounted mixed reality display device HoloLens2.

In some exemplary embodiments, the mixed reality display device has a built-in depth camera, an inertial measurement unit, and multiple sensors to obtain user environment information and user motion information.

In some exemplary embodiments, the mixed reality display device is further provided with a recognition camera and a tracking camera; the recognition camera is used to detect and recognize specific images, objects or scenes in the real scene; the tracking camera is used to track the position and direction of the camera, and place the virtual image in the real world according to the output of the recognition camera to achieve virtual-reality registration.

In some exemplary embodiments, a virtual image and a real scene are fused to obtain a three-dimensional model of mixed reality rendering, including: generating the virtual image into a three-dimensional model of the virtual image through 3D modeling and rendering technology, and adjusting the properties of the virtual image in real time, and fusing the three-dimensional model of the virtual image with the real scene to obtain a three-dimensional model of mixed reality rendering; wherein the properties of the virtual image include the position, size and transparency of the virtual image.

In some exemplary embodiments, the three-dimensional model rendered by the mixed reality includes three-dimensional models at multiple different angles; wherein the three-dimensional models at different angles correspond to the user's observation angle.

In some exemplary embodiments, when performing air gesture interaction, the user uses gestures to move, rotate, scale and cut the virtual tissue in the three-dimensional model rendered by the mixed reality from any angle, manipulate virtual surgical instruments in real time and observe the patient's lesion condition.

In some exemplary embodiments, after obtaining the three-dimensional model of mixed reality rendering, the method further includes: evaluating the three-dimensional model of mixed reality rendering.

In some exemplary embodiments, the three-dimensional model rendered by the mixed reality is evaluated, including: using medical images of three-dimensional CT original data, medical images of three-dimensional CT reconstructed data, and medical images of three-dimensional ultrasound original data as comparison data, and evaluating the efficiency and real-time performance of the three-dimensional model rendered by the mixed reality.

On the second aspect, the embodiments of the present application also provide a holographic three-dimensional medical data visualization system that can interact with gestures through the air, including a mixed reality display device and a gesture recognition system connected to the mixed reality display device for communication; the mixed reality display device is worn on the user's head; the mixed reality display device has a built-in depth camera, an inertial measurement unit and multiple sensors for obtaining user environment information and user motion information; the user environment information is image information of the real scene around the user; the mixed reality display device also generates a virtual image based on the user environment information and user motion information using a stereo interpolation volume rendering method; and uses an optical model to project the virtual image into the user's field of view, and fuses the virtual image and the real scene to obtain a three-dimensional model rendered by mixed reality; the gesture recognition system is used to capture the user's gestures through a gesture recognition method based on the three-dimensional model rendered by mixed reality and perform gesture interaction through the air, so as to realize interaction between the user and the virtual scene.

Beneficial Effects

The technical solution provided by the embodiments of the present application has at least the following advantages:

The embodiment of the present application provides a holographic three-dimensional medical data visualization method and system that can interact with gestures in the air, and the method includes the following steps: first, obtaining user environment information and user motion information; the user environment information is image information of the real scene around the user; the user wears a mixed reality display device; then, based on the user environment information and the user motion information, a stereo interpolation volume rendering method is used to generate a virtual image; next, an optical model is used to project the virtual image into the user's field of view, and the virtual image and the real scene are fused to obtain a three-dimensional model rendered by mixed reality; finally, based on the three-dimensional model rendered by mixed reality, the user's gestures are captured by a gesture recognition method and gesture interaction is performed in the air, so as to realize the interaction between the user and the virtual scene.

In order to enable doctors to better perceive the spatial structure of the patient's target area, this application proposes a holographic three-dimensional medical data intuitive visualization system under air gesture interaction suitable for surgical disciplines, covering preoperative planning and holographic three-dimensional visualization of tissue structure, using augmented reality technology to help surgeons predict tissue structure information and cut tissues and organs under the guidance of virtual information, combined with registration tracking technology and air gesture interaction, to assist surgeons in performing surgical operations naturally, thereby transforming traditional surgery into predictable and accurately guided surgery.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplifications do not constitute limitations on the embodiments. Unless otherwise stated, the pictures in the drawings do not constitute proportional limitations.

FIG1 is a schematic diagram of a flow chart of a holographic three-dimensional medical data visualization method capable of air gesture interaction provided by an embodiment of the present application;

FIG2 is a schematic diagram of image recognition and virtual-real registration based on Vuforia provided in one embodiment of the present application;

FIG3 is a schematic diagram of the holographic gesture interaction principle provided in an embodiment of the present application.

Embodiments of the present invention

As can be seen from the background technology, the volume rendering method in the prior art uses direct volume rendering technology to enhance visual and spatial perception from a three-dimensional data set. In the process of using GPU for volume rendering, the existing methods have never fully achieved the same quality and performance as in medical augmented reality. Therefore, the methods proposed in the prior art have the technical problem of being limited to rendering polygonal surface data and unable to meet the high efficiency requirements of XR auxiliary diagnosis systems for volume rendering.

Unlike traditional two-dimensional display technology for medical data, XR can present three-dimensional anatomical structures more realistically and accurately during intraoperative diagnosis and preoperative surgical planning. With the development of GPU technology, volume rendering technology has made great progress in recent years. Direct volume rendering technology is used to enhance visual and spatial perception from three-dimensional data sets. A related technology proposes ClearView for non-segmented, real-time, focus and background visualization of three-dimensional data. ClearView draws the viewer's attention to the focus area, where important features of the three-dimensional data are embedded. The transparency of the surrounding environment layer is adjusted according to the curvature characteristics of the environment layer or the distance from the focus layer.

Most early interactive technologies were designed from a device-centric perspective, such as using two-dimensional touch screen input and stylus, keyboard and device sensors to interact with virtual objects. Another related technology made a comprehensive analysis of immersive visualization and proposed: immersive interactive interface can use people's stereoscopic vision to display data, shifting the data expression space from the two-dimensional plane to the three-dimensional space around the user; providing mobility, so that the user's physical working environment is no longer limited to a fixed office desk; providing natural interaction methods such as gestures, making the interaction more intuitive, allowing users to use multiple channels and interact with data in parallel.

Since existing methods have never fully achieved the same quality and performance as in medical augmented reality, the proposed methods are limited to rendering polygonal surface data. Although the relevant anatomy and structures can be depicted, these methods are too time-consuming and cannot meet the high efficiency requirements of XR-assisted diagnosis systems for volume rendering. Through the analysis and research of relevant scholars and cases, the medical-assisted diagnosis system should be efficient and the rendering speed should be as fast as possible.

In order to solve the above technical problems, the embodiment of the present application provides a holographic three-dimensional medical data visualization method and system that can interact with gestures in the air, and the method includes the following steps: first, obtaining user environment information and user motion information; the user environment information is the image information of the real scene around the user; the user wears a mixed reality display device; then, based on the user environment information and user motion information, a stereo interpolation volume rendering method is used to generate a virtual image; next, an optical model is used to project the virtual image into the user's field of view, and the virtual image and the real scene are merged to obtain a three-dimensional model rendered by mixed reality; finally, based on the three-dimensional model rendered by mixed reality, the user's gesture is captured by a gesture recognition method and gestures are interacted in the air to realize the interaction between the user and the virtual scene. The embodiment of the present application provides a holographic three-dimensional medical data visualization method and system that can interact with gestures in the air, and through efficient three-dimensional object body rendering and real-time and accurate interaction method design, the user can move, rotate, scale and cut virtual tissues from any angle, manipulate virtual surgical instruments in real time and observe the patient's lesions, so as to realize the interaction between the user and the virtual scene through gestures in the air.

The following will describe the various embodiments of the present application in detail with reference to the accompanying drawings. However, it will be appreciated by those skilled in the art that in the various embodiments of the present application, many technical details are provided in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solution claimed in the present application can be implemented.

Referring to FIG. 1 , an embodiment of the present application provides a holographic three-dimensional medical data visualization method capable of air gesture interaction, comprising:

Step S1, obtaining user environment information and user motion information; the user environment information is image information of the real scene around the user; the user wears a mixed reality display device.

Step S2: Based on the user environment information and the user motion information, a stereo interpolation volume rendering method is used to generate a virtual image.

Step S3: Use an optical model to project the virtual image into the user's field of view, and fuse the virtual image with the real scene to obtain a three-dimensional model rendered by mixed reality.

Step S4: Based on the three-dimensional model rendered by mixed reality, the user's gestures are captured through a gesture recognition method and gesture interaction is performed in the air to enable interaction between the user and the virtual scene.

In some embodiments, the mixed reality display device is a head-mounted mixed reality display device HoloLens2; and the optical model is an optical model in the head-mounted mixed reality display device HoloLens2.

This application proposes an interactive three-dimensional medical data holographic visualization system adapted for use with the HoloLens2 head-mounted display device. Through efficient three-dimensional object volume rendering and real-time and accurate interactive method design, the system enables surgeons to move, rotate, scale and shear virtual tissues from any angle, manipulate virtual surgical instruments in real time and observe the patient's lesions, and diagnose the area of interest intuitively and accurately. Through experiments based on three-dimensional CT raw data, three-dimensional CT reconstruction data and three-dimensional ultrasound raw data on HoloLens2, compared with advanced volume rendering algorithm models, the rendering effect and speed of the method of this application have certain advantages, reaching rendering speeds of 42.00±5.80 FPS, 60.00±1.80 FPS and 42.05±5.55 FPS on HoloLens2, respectively, verifying the feasibility of the system of this application.

In some embodiments, the mixed reality display device has a built-in depth camera, an inertial measurement unit, and multiple sensors to obtain user environment information and user motion information.

Specifically, this application uses Microsoft's new generation of mixed reality head-mounted display device HoloLens2 to realize the mixed reality display of three-dimensional models. Based on the technology of HoloLens2 to integrate virtual images with real scenes, its principle is mainly to use multiple sensors such as depth cameras, inertial measurement units and optical sensors to obtain image information of real scenes and user motion information, and then transmit this information to the computer for processing and analysis, and finally generate virtual images and project them into the user's field of view through the optical device of HoloLens2 to achieve the display effect of mixed reality. In the mixed reality display of HoloLens2, realizing the integration of virtual images and real scenes is the key to realizing mixed reality. In order to achieve this goal, HoloLens2 adopts a variety of technologies, including optical technology, sensor technology and computer graphics technology. Among them, optical technology is the basis of mixed reality display. It uses transparent waveguides to project virtual images into the user's field of view, and realizes the integration of virtual images and real scenes through optical principles such as reflection and refraction.

Sensor technology senses the environment around the user and the user's motion information, and transmits this information to the computer for processing and analysis. HoloLens2 uses a variety of sensors, including depth cameras, inertial measurement units, and optical sensors. These sensors can obtain the user's position, posture, gesture and other information in real time, so as to achieve accurate tracking and positioning of the user, and then adjust the position and size of the virtual image to make it perfectly integrated with the real scene. Computer graphics are used to generate and display virtual images in the user's field of view. In HoloLens2, virtual images are generated into realistic three-dimensional models through 3D modeling and rendering technology, and their position, size, transparency and other properties are adjusted in real time to achieve integration with the real scene.

In some embodiments, the mixed reality display device is also provided with a recognition camera and a tracking camera; the recognition camera is used to detect and recognize specific images, objects or scenes in the real scene; the tracking camera is used to track the position and direction of the camera, and place the virtual image in the real world according to the output of the recognition camera to achieve virtual-reality registration.

The Vuforia engine is used in the holographic three-dimensional medical data visualization method designed by this application that can interact with gestures in the air, as shown in Figure 2. Vuforia can use the built-in cameras (recognition camera and tracking camera) of the mixed reality device to locate specific patterns using image recognition and tracking functions. The basic architecture of Vuforia includes a recognition engine and a tracking engine. The recognition engine is used to detect and identify specific images, objects or scenes in the real world. The tracking engine is responsible for tracking the position and direction of the camera and accurately placing virtual content in the real world based on the output of the recognition engine. Through this method, this application can achieve accurate virtual-real registration.

In some embodiments, the virtual image and the real scene are fused to obtain a three-dimensional model of mixed reality rendering, including: generating the virtual image into a three-dimensional model of the virtual image through 3D modeling and rendering technology, and adjusting the properties of the virtual image in real time, and fusing the three-dimensional model of the virtual image with the real scene to obtain the three-dimensional model of mixed reality rendering; wherein the properties of the virtual image include the position, size and transparency of the virtual image.

In some embodiments, the three-dimensional model rendered by the mixed reality includes three-dimensional models at multiple different angles; wherein the three-dimensional models at different angles correspond to the user's observation angle.

Specifically, in the holographic three-dimensional medical data visualization method capable of air gesture interaction provided in the embodiment of the present application, according to the above-mentioned illumination model, it is rendered by HoloLens 2 for mixed reality. After the model rendering is completed, the model display at different angles is provided based on the viewing angle and position of the wearer of HoloLens 2.

In some embodiments, when performing air gesture interaction, the user uses gestures to move, rotate, scale, and cut the virtual tissue in the three-dimensional model rendered by the mixed reality from any angle, manipulate virtual surgical instruments in real time, and observe the patient's lesion condition.

Specifically, the holographic three-dimensional medical data visualization method capable of air gesture interaction provided by the present application adopts a portable holographic visualization interaction based on gestures for human-computer interaction. The portable holographic visualization interaction technology based on gestures is a human-computer interaction method based on holographic projection and gesture recognition. It mainly realizes a natural and intuitive interaction between the user and the virtual scene by projecting the virtual three-dimensional scene into the real scene, and then uses gesture recognition technology to capture the user's gestures and realize interaction, thereby realizing more sophisticated and flexible operations.

Interaction requires capturing the user's hand movements to identify gestures and converting them into corresponding commands to achieve control of the virtual scene. The accuracy, real-time and stability of gesture recognition can be improved by training and identifying a large amount of gesture data.

The principle of gesture interaction is shown in Figure 2. The gesture recognition system uses devices such as cameras or depth cameras to capture the user's hand movements and convert them into digital signals. These signals are input into the computer and processed by algorithms and models to recognize different gestures. These gestures can represent different instructions, such as sliding, zooming, rotating, etc. The gesture recognition system undergoes a lot of training to recognize different gestures.

Computer vision technology helps gesture recognition systems identify specific patterns and shapes in images, thereby accurately identifying different gestures. For example, it detects information such as the position of fingers, the direction, speed, and size of gestures, thereby helping the system determine the user's gestures. The interactive method of this application supports the movement and rotation of the entire model, and can also scale local key parts.

Additionally, users can cut using hand gestures, which enables surgeons to view various cross-sections of the target area, allowing them to observe and analyze the patient's lesions without blind spots.

In some embodiments, after obtaining the three-dimensional model of mixed reality rendering, the method further includes: evaluating the three-dimensional model of mixed reality rendering.

In some embodiments, the three-dimensional model rendered by mixed reality is evaluated, including: using medical images of three-dimensional CT original data, medical images of three-dimensional CT reconstructed data, and medical images of three-dimensional ultrasound original data as comparison data, and evaluating the efficiency and real-time performance of the three-dimensional model rendered by mixed reality.

This application uses two types of medical data, CT images and ultrasound images, to conduct experiments to test and evaluate the efficiency and real-time performance of the three-dimensional model rendered by mixed reality provided by this application. For each type of medical data, the experiment is conducted according to the following steps:

(1) Obtain 3D CT raw data with a size of (270, 512, 512), 3D CT reconstructed data with a slice resolution of 512×512 and a slice thickness of 0.7-0.8 mm, or 3D ultrasound raw data with a resolution of 183×115×126 FPS.

(2) Adjust the transfer function to emphasize and classify the features of interest in the data to truly restore the material of areas such as bones, blood vessels, or soft tissues.

(3) Surgeons wearing HoloLens2 interact with medical 3D data.

The efficiency and real-time performance of the medical interactive visualization method are tested and evaluated below.

In order to verify the efficiency of the medical interactive visualization method, this application designed an experiment to compare three typical medical images: 3D CT raw data, 3D CT reconstructed data and 3D ultrasound raw data, and the rendering results were obtained using six volume rendering algorithm models. The basic model (no lighting), illumination model, back-to-front direct volume rendering model, back-to-front direct volume rendering + cubic convolution interpolation model, early ray termination model + cubic convolution interpolation model and early ray termination were used. The algorithm of this application has the highest contrast resolution, which can more clearly see the tiny blood vessels and structures on the heart, and can perform appropriate texture analysis in the same tissue with homogeneous features. The algorithm model of this application is more suitable for the evaluation of the myocardium. The sphere centered on the focus can focus the focus area, and the three-dimensional composite display of various tissue compositions highlights their boundaries and contours.

In order to verify the real-time performance of the medical interactive visualization method, this application conducted 20 interactive visualization experiments, recorded and calculated the average number of frames per second, to evaluate the real-time performance of the medical interactive visualization system of this application. Under the premise that the maximum displayable frame rate of HoloLens2 is 60FPS, for the three-dimensional CT raw data, the basic model obtains the highest rendering speed of 59.95±2.25FPS, which is close to the highest resolution that HoloLens 2 can display, followed by the early ray termination model (method of this application) 40.00±5.80FPS, but from the perspective of visual authenticity restoration, the basic model cannot provide an effective spatial reference, and this application is significantly better than the basic model. For three-dimensional ultrasound raw data and three-dimensional CT reconstruction data, the model of this application obtains the highest rendering speed, which is 60.00±1.80FPS and 42.05±5.55FPS, respectively, and in ultrasound images, the rendering speed reaches the highest display frame rate of HoloLens2. Therefore, compared with some other five representative methods, the method of the present application can achieve the best rendering speed and has better real-time performance. The medical interactive visualization system provided by the present application can quickly respond to changes in the posture of three-dimensional data.

In summary, doctors can better perceive the spatial structure of the patient's target area, and develop a holographic 3D medical data visualization system that is interactive with gestures in the air and suitable for surgical medicine. It covers preoperative planning and holographic 3D visualization of tissue structure. Augmented reality technology is used to help surgeons perceive tissue structure information and cut tissue organs under the guidance of virtual information. Combined with registration tracking technology and gesture interaction in the air, it assists surgeons to perform surgical operations naturally and intuitively, thereby transforming traditional surgery into predictable and accurately guided surgery. According to the characteristics of surgical medicine, a suitable medical visualization prototype system was developed, and related interaction methods were formulated. The feasibility of the system was verified using 3D CT raw data, 3D CT reconstruction data, and 3D ultrasound raw data. On this basis, in-depth research and exploration were carried out to determine its practical application value in clinical practice, and continuous improvement and optimization were carried out, laying a solid foundation for its future.

The present invention uses air gesture interaction and three-dimensional medical data visualization technology to assist doctors in achieving intuitive diagnosis. The method includes two strategies: stereo interpolation volume rendering based on the illumination model and holographic mid-air gesture interaction. We conducted multiple experiments for each strategy to verify their effectiveness. We used a variety of illumination models and rendering methods to perform volume rendering on three types of medical images. Compared with some representative methods, our method can achieve the fastest rendering speed while ensuring the visualization effect. At the same time, the interaction algorithm can meet the real-time and accuracy requirements of assisting doctors in diagnosis. Therefore, our method can provide users with a more intuitive three-dimensional medical data visualization effect in a holographic XR environment.

The embodiment of the present application also provides a holographic three-dimensional medical data visualization system that can interact with gestures through the air, including a mixed reality display device and a gesture recognition system connected to the mixed reality display device for communication; the mixed reality display device is worn on the user's head; the mixed reality display device has a built-in depth camera, an inertial measurement unit and multiple sensors for obtaining user environment information and user motion information; the user environment information is image information of the real scene around the user; the mixed reality display device also generates a virtual image based on the user environment information and user motion information using a stereo interpolation volume rendering method; and uses an optical model to project the virtual image into the user's field of view, and fuses the virtual image and the real scene to obtain a three-dimensional model rendered by mixed reality; the gesture recognition system is used to capture the user's gestures through a gesture recognition method based on the three-dimensional model rendered by mixed reality and perform gesture interaction through the air, so as to realize interaction between the user and the virtual scene.

Compared with the prior art, the advantage of the holographic three-dimensional medical data visualization system with air gesture interaction provided by the present application is that: although there are many medical visualization systems at this stage, most of them are based on two-dimensional screen displays, and the holographic display of medical images is still very limited, and multi-perspective interactive surgical operations cannot be achieved. Surgeons use mice to operate on the screen, which greatly limits the doctor's field of vision compared to the mid-air gesture interaction used in our system, and this interaction method is far from the natural method in the real surgical scene, limiting the surgeon's workspace for medical image analysis, and there is a risk of potential diagnostic errors caused by the surgeon's visual impairment of two-dimensional images. In contrast, in our system, surgeons wearing HoloLens2 can move freely, view virtual-physical fusion scenes and perform surgical tasks at the same time. Our system can improve the doctor's comfort during diagnosis and optimize the patient's treatment effect. Surgeons can present medical knowledge to patients intuitively, helping patients better understand and accept treatment, thereby reducing patients' psychological pressure and anxiety.

Based on the above technical scheme, the embodiment of the present application provides a holographic three-dimensional medical data visualization method and system that can interact with gestures in the air, and the method includes the following steps: first, obtaining user environment information and user motion information; the user environment information is image information of the real scene around the user; the user wears a mixed reality display device; then, based on the user environment information and user motion information, a stereo interpolation volume rendering method is used to generate a virtual image; next, an optical model is used to project the virtual image into the user's field of view, and the virtual image and the real scene are fused to obtain a three-dimensional model rendered by mixed reality; finally, based on the three-dimensional model rendered by mixed reality, the user's gestures are captured by a gesture recognition method and gesture interaction is performed in the air, so as to realize the interaction between the user and the virtual scene.

In order to enable doctors to better perceive the spatial structure of the patient's target area, this application proposes a holographic three-dimensional medical data intuitive visualization system under air gesture interaction suitable for surgical disciplines, covering preoperative planning and holographic three-dimensional visualization of tissue structure, using augmented reality technology to help surgeons predict tissue structure information and cut tissues and organs under the guidance of virtual information, combined with registration tracking technology and air gesture interaction, to assist surgeons in performing surgical operations naturally, thereby transforming traditional surgery into predictable and accurately guided surgery.

That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the above-mentioned method of each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

Those skilled in the art can understand that the above-mentioned embodiments are specific examples for implementing the present application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of the present application. Any person skilled in the art can make their own changes and modifications without departing from the spirit and scope of the present application, so the scope of protection of the present application shall be based on the scope defined in the claims.

Claims (10)

A holographic three-dimensional medical data visualization method capable of air gesture interaction, characterized by comprising the following steps: Acquire user environment information and user motion information; the user environment information is image information of a real scene around the user; the user wears a mixed reality display device; Based on the user environment information and the user motion information, a virtual image is generated by using a stereo interpolation volume rendering method; The virtual image is projected into the user's field of vision using an optical model, and the virtual image is fused with the real scene to obtain a three-dimensional model rendered by mixed reality; Based on the three-dimensional model rendered by the mixed reality, the user's gestures are captured through a gesture recognition method and gesture interaction is performed in the air, thereby realizing interaction between the user and the virtual scene. The holographic three-dimensional medical data visualization method capable of air gesture interaction according to claim 1 is characterized in that the mixed reality display device is a head-mounted mixed reality display device HoloLens2; The optical model is an optical model in the head-mounted mixed reality display device HoloLens2. The holographic three-dimensional medical data visualization method capable of air gesture interaction according to claim 1 is characterized in that the mixed reality display device has a built-in depth camera, an inertial measurement unit and multiple sensors to obtain the user environment information and the user motion information. According to the holographic three-dimensional medical data visualization method capable of air gesture interaction according to claim 3, it is characterized in that a recognition camera and a tracking camera are also provided in the mixed reality display device; the recognition camera is used to detect and recognize specific images, objects or scenes in real scenes; the tracking camera is used to track the position and direction of the camera, and place the virtual image in the real world according to the output of the recognition camera to achieve virtual-real registration. The holographic three-dimensional medical data visualization method capable of air gesture interaction according to claim 1 is characterized in that the virtual image and the real scene are fused to obtain a three-dimensional model rendered by mixed reality, comprising: The virtual image is generated into a three-dimensional model of the virtual image through 3D modeling and rendering technology, and the properties of the virtual image are adjusted in real time, and the three-dimensional model of the virtual image is integrated with the real scene to obtain a three-dimensional model of mixed reality rendering; wherein the properties of the virtual image include the position, size and transparency of the virtual image. The holographic three-dimensional medical data visualization method capable of air gesture interaction according to claim 1 is characterized in that the three-dimensional model rendered by mixed reality includes three-dimensional models from multiple different angles; wherein, The three-dimensional models at different angles correspond to the user's viewing angle. According to the holographic three-dimensional medical data visualization method capable of air gesture interaction according to claim 1, it is characterized in that when performing air gesture interaction, the user uses gestures to move, rotate, scale and shear the virtual tissue in the three-dimensional model rendered by the mixed reality from any angle, manipulate virtual surgical instruments in real time and observe the patient's lesion condition. The holographic three-dimensional medical data visualization method capable of air gesture interaction according to claim 1 is characterized in that after obtaining the three-dimensional model rendered by mixed reality, it also includes: The three-dimensional model of the mixed reality rendering is evaluated. The holographic three-dimensional medical data visualization method capable of air gesture interaction according to claim 8 is characterized in that evaluating the three-dimensional model rendered by mixed reality comprises: Using medical images of 3D CT original data, medical images of 3D CT reconstructed data, and medical images of 3D ultrasound original data as comparison data, the efficiency and real-time performance of the 3D model rendered by the mixed reality are evaluated respectively. A holographic three-dimensional medical data visualization system capable of gesture interaction in the air, characterized by comprising a mixed reality display device and a gesture recognition system connected to the mixed reality display device; The mixed reality display device is worn on the user's head; the mixed reality display device has a built-in depth camera, an inertial measurement unit and multiple sensors for obtaining user environment information and user motion information; the user environment information is image information of the real scene around the user; The mixed reality display device also generates a virtual image using a stereo interpolation volume rendering method according to the user environment information and the user motion information; and projects the virtual image into the user's field of view using an optical model, and fuses the virtual image with the real scene to obtain a three-dimensional model rendered by mixed reality; The gesture recognition system is used to capture the user's gestures through a gesture recognition method according to the three-dimensional model rendered by the mixed reality and perform gesture interaction in the air, thereby realizing interaction between the user and the virtual scene.