CN116646052B - Auxiliary acupuncture positioning system and method based on three-dimensional human body model - Google Patents

Abstract

The invention discloses an auxiliary acupuncture positioning system and method based on a three-dimensional human body model. The system consists of head display equipment, a three-dimensional human body model, a human body gesture recognition module, a human body model alignment module, an acupuncture point positioning module and a user interaction module. The head display device captures images of the real environment and three-dimensional structure information thereof. The three-dimensional human body model is a virtual model, and virtual acupuncture points are marked on the three-dimensional human body model. The human body posture recognition module predicts a posture of the patient from the captured image. The mannequin alignment module aligns the predicted patient pose with the virtual mannequin using a numerical optimization algorithm. The acupuncture point positioning module converts the positions of the virtual acupuncture points to the real patient body according to the alignment relation. The user interaction module provides a user interaction mode, so that a user can see the body of a real patient and can see a virtual human body model and acupuncture points. The system enables an acupuncture operator to find acupuncture points more accurately and conveniently, and improves treatment effect.

Description

An auxiliary acupuncture positioning system and method based on a three-dimensional human body model

Technical field

The present application relates to the field of medical technology, and in particular to an auxiliary acupuncture positioning system and method based on a three-dimensional human body model.

Background technique

Acupuncture is an ancient medical technique that mainly stimulates specific acupuncture points on the human body to treat diseases or improve physical conditions. However, acupuncture point positioning is an important skill in acupuncture and requires a lot of time and energy to learn and master. Even experienced acupuncture practitioners may make errors in the positioning of acupuncture points, which affects the effect of acupuncture.

However, the acupuncture auxiliary systems currently on the market are mainly based on two-dimensional images, which have certain limitations on acupoint positioning accuracy and user experience. Although some systems have begun to try to use three-dimensional human body models, most are still in the experimental stage and have high technical requirements for users. In addition, existing acupuncture auxiliary systems based on three-dimensional human models mainly display the three-dimensional model on a computer screen. Users need to operate the patient's body while looking at the screen, which affects the convenience and accuracy of the operation to a certain extent. In addition, the three-dimensional human body models in existing systems are often unified standard models that cannot adapt to the individual differences of different patients.

With the development of new display technologies such as virtual reality (VR), augmented reality (AR) and mixed reality (MR), more and more applications are beginning to try to use these technologies in the medical field to improve the efficiency and quality of medical services. . Especially in areas such as acupuncture that require precise operations, these technologies have great application potential.

Therefore, it is currently necessary to develop a new auxiliary acupuncture positioning system based on a three-dimensional human body model, combined with new display technologies such as virtual reality (VR), augmented reality (AR) and mixed reality (MR), to improve the accuracy of acupuncture positioning. An important direction for the technological development of acupuncture positioning systems.

Contents of the invention

This application provides an auxiliary acupuncture positioning system based on a three-dimensional human body model to improve the positioning accuracy of acupuncture points.

The system includes: a head-mounted display device with a built-in camera and depth sensor for capturing images of the real environment and its three-dimensional structural information;

Three-dimensional human body model, which is a virtual human body model created on a computer and marked with virtual acupuncture points;

Human posture recognition module, used to predict the patient's posture from the image captured by the headset device and its three-dimensional structure information;

The human body model alignment module is used to align the virtual human model to the real patient's body through numerical optimization algorithms based on the predicted patient posture;

An acupuncture point positioning module is used to convert the position of the virtual acupuncture point to the real patient's body based on the alignment relationship between the virtual human body model and the real patient's body;

The user interaction module is used to provide a user interaction method so that the user can see the virtual human body model and the acupuncture points converted to the real patient's body while seeing the real patient's body.

Optionally, the user interaction module allows the user to select and operate acupuncture points through gestures or voice commands, and displays relevant information on the screen after the user selects an acupuncture point.

Optionally, the human posture recognition module includes a deep neural network for predicting the patient's posture from the image of the headset.

Optionally, the human body model alignment module uses a gradient descent algorithm to calculate model parameters that minimize the difference between the virtual human body model and the patient's body.

Optionally, the acupuncture point positioning module calculates the corresponding acupuncture point position on the real patient's body based on the alignment relationship and the position of the virtual acupuncture point, thereby converting the position of the virtual acupuncture point on the real patient's body. .

This application provides an auxiliary acupuncture positioning method based on a three-dimensional human body model. The method includes:

Use the head-mounted display device to capture images of the patient's real environment and obtain its three-dimensional structural information;

Create a three-dimensional human body model and mark acupuncture points on the model;

Predict the patient's posture from images from the headset;

According to the predicted patient posture, the virtual human body model is aligned to the real patient's body through a numerical optimization algorithm;

According to the alignment relationship between the virtual human body model and the real patient's body, the position of the virtual acupuncture point is converted to the real patient's body;

Through user interaction technology, a user interaction method is provided, so that users can see the virtual human body model and the acupuncture points converted to the real patient's body while seeing the real patient's body.

Optionally, the user interaction method also includes allowing the user to select and operate acupuncture points through gestures or voice commands, and display relevant information on the screen after the user selects an acupuncture point.

Optionally, predicting the patient's posture from the image of the headset device includes:

Use a deep neural network to predict patient posture from images from a headset.

Optionally, the virtual human body model is aligned to the real patient's body through a numerical optimization algorithm based on the predicted patient posture, including:

Use a gradient descent algorithm to find model parameters that minimize the difference between the virtual human model and the patient's body;

According to the model parameters, the virtual human body model is aligned to the real patient's body.

Optionally, converting the position of the virtual acupuncture point to the real patient's body based on the alignment relationship between the virtual human body model and the real patient's body includes:

The corresponding acupuncture point position on the real patient's body is calculated based on the alignment relationship and the position of the virtual acupuncture point, thereby converting the position of the virtual acupuncture point onto the real patient's body.

The positioning system provided by this application is based on acupuncture positioning of a three-dimensional human body model. Compared with the traditional acupuncture positioning method based on two-dimensional pictures or naked eye recognition, the use of a three-dimensional human body model can more accurately locate acupuncture points, avoiding the need for individual differences or postures. Positioning errors caused by changes. Using a head-mounted display device to capture the real environment, this method can capture the patient's physical status and environmental information in real time. Compared with static pictures or pre-recorded videos, it can provide richer information and is more conducive to accurate positioning. Combining human posture recognition and model alignment technology, by identifying the patient's actual posture and aligning it with the three-dimensional model, the accuracy of acupuncture point positioning can be further improved. The user interaction module, through virtual reality technology, enables users to observe virtual human models and acupuncture points while observing the real environment, which greatly enhances the user experience.

The beneficial technical effects brought by this application include: the system can achieve precise acupuncture point positioning and improve the effect and efficiency of acupuncture treatment. In addition, through virtual reality technology, acupuncturists can have a more intuitive visual effect when performing acupuncture operations, improving the convenience and accuracy of acupuncture operations. In addition, by using head-mounted devices and depth sensors, patients can be monitored in real time and acupuncture strategies can be adjusted in a timely manner to better meet the needs of individualized treatment.

Description of drawings

Figure 1 is a schematic diagram of an auxiliary acupuncture positioning system based on a three-dimensional human body model provided by the first embodiment of the present application.

Figure 2 is a schematic diagram of a human gesture recognition module according to the first embodiment of the present application.

Figure 3 is a schematic diagram of a posture prediction model related to the first embodiment of the present application.

Figure 4 is a flow chart of an auxiliary acupuncture positioning method based on a three-dimensional human body model provided in the second embodiment of the present application.

Detailed ways

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be implemented in many other ways different from those described here. Those skilled in the art can make similar extensions without violating the connotation of the present application. Therefore, the present application is not limited by the specific implementation disclosed below.

The first embodiment of the present application provides an auxiliary acupuncture positioning system based on a three-dimensional human body model. Please refer to Figure 1, which is a schematic diagram of the first embodiment of the present application. An auxiliary acupuncture positioning system based on a three-dimensional human body model according to the first embodiment of the present application will be described in detail below with reference to FIG. 1 .

The auxiliary acupuncture positioning system 100 includes a head-mounted display device 102 supporting mixed reality, a three-dimensional human body model 104, a human posture recognition module 106, a human body model alignment module 108, an acupuncture point positioning module 110, and a user interaction module 112.

The head-mounted display device 102 supports mixed reality technology and has a built-in camera and depth sensor for capturing images of the real environment and obtaining its three-dimensional structure information.

Mixed Reality (MR) is a new display technology that combines the characteristics of Virtual Reality (VR) and Augmented Reality (AR). It combines the virtual world and the real world through advanced equipment and systems. The worlds merge together to create a new visual experience.

The basic characteristics of MR technology are as follows:

(1) Mixed reality: MR not only embeds virtual objects into the real world, but also allows users to interact with virtual objects, thereby creating an environment that mixes reality and virtuality.

(2) Interaction: MR allows users to interact with the virtual world using natural gestures, voice, etc., providing a more natural and intuitive user experience.

(3) Reality: Through real-time 3D modeling of the real world, MR technology can make virtual objects appear in a more realistic way in the real environment. For example, it can realize the occlusion relationship between virtual objects and objects in the real world and lighting effects. wait.

MR technology requires the support of complex computer vision, machine learning, 3D modeling and other technologies. Its application range is very wide, such as games and entertainment, remote collaboration, education and training, design and manufacturing, medical and health and other fields.

In mixed reality, virtual objects and real objects are displayed in the same field of view, providing users with a rich visual experience. The head-mounted display device 102 provided by this application is designed to achieve this effect.

The head-mounted display device 102 provided by this application has a microdisplay for displaying virtual images in the user's field of view. The device also includes a headband for securing the device to the user's head. The size and shape of the headband can be adjusted to fit the head size and shape of different users.

The camera included in the head-mounted display device 102 is used to capture the real environment in the user's field of view, and the depth sensor is used to obtain the three-dimensional structure information of the real environment. This information is used to generate a corresponding real environment model in the virtual environment, allowing virtual objects to interact with real objects in a realistic way.

When a user wears the head-mounted display device provided in this embodiment, the device's built-in camera and depth sensor will begin to capture and analyze the real environment in the user's field of view. This information is transformed into a model of the virtual environment through an image processing algorithm.

Next, the headset's microdisplay displays the virtual environment and virtual objects in the user's field of view. The virtual targets displayed can be static or dynamic, depending on the needs of the application. Users can interact with virtual targets through the device's input interface, such as a touchpad, buttons, or a voice recognition system. In the above manner, the head-mounted display device 102 provides the user with a mixed reality visual experience.

The three-dimensional human body model 104 provided in this embodiment is a virtual human body model created on a computer, and virtual acupuncture points are marked on the model. The implementation steps of the three-dimensional human body model 104 will be described in detail below.

First, you need to create a basic 3D human body model. This model can be created using various known technologies, including but not limited to: manual modeling using computer graphics technology, obtaining a realistic human body model through laser scanning technology, etc. This model should include the main parts of the human body, such as head, chest, abdomen, legs, etc., to provide a basis for subsequent acupuncture point annotation.

After the human body model is created, virtual acupuncture points need to be marked on the model. First, it is necessary to collect relevant information about acupuncture points, including but not limited to: the names of acupuncture points, positioning methods, corresponding human body parts, etc. This information can be obtained from various TCM literature.

Next, the collected acupuncture point information needs to be converted into three-dimensional coordinates, which can be achieved by using computer graphics technology for coordinate conversion. Then, these three-dimensional coordinates need to be applied to the human body model to mark each acupuncture point to the corresponding position. Each acupuncture point can be represented by a small sphere or other visual icon, and is accompanied by basic information about the acupuncture point to facilitate user identification and understanding. Through the above steps, the acupuncture point annotation based on the three-dimensional human body model is completed.

The human posture recognition module 106 provided in this embodiment is used to predict the patient's posture from the image captured by the headset device and its three-dimensional structure information. This module will be described below in conjunction with Figure 2. This module mainly consists of an image pre-processing part 202, a posture prediction model 204 and a post-processing part 206.

Image preprocessing part 202: Responsible for converting information such as images captured by the camera of the head-mounted display device into a format suitable for inputting the posture prediction model. Specific operations may include cropping, scaling, normalization and other operations.

The image preprocessing part can be implemented through the following steps:

Image receiving S1001: First, use the image and other information captured from the headset device as the original input. These images may be RGB images or depth images, or a combination of the two.

Smart cropping S1002: Next, by using the smart cropping algorithm, the position of the human body in the image is automatically identified and this part of the area is cropped. The smart cropping algorithm uses pre-trained deep learning models, such as convolutional neural networks (CNN). This model is trained on a large number of images with human body location annotations. When cropping, the image is first input into the model, and the model outputs a heat map of the human body's position. Then, based on the maximum value point in the heat map, the approximate center position of the human body is determined. Finally, an area of predefined size is cropped with this central position as the center. This method ensures that the human body is in the center of the cropped image and contains as much information about the human body as possible.

Scale normalization S1003: Since different people have different body sizes, the cropped images need to be scale normalized. The specific method is to uniformly scale the size of the image to a predefined size, such as 256x256 pixels. In this way, the processed image size is consistent regardless of the size of the original human body.

Pixel value normalization S1004: After scale normalization, the pixel values of the image also need to be normalized. The purpose of normalization is to limit the pixel value range of the image to a certain range, such as [0,1] or [-1,1]. This can make the training of the model more stable and prevent certain pixel values from being too large or too small, which will affect the performance of the model. Specific methods of pixel value normalization may include maximum and minimum normalization, Z-Score normalization, etc.

Image enhancement S1005: Finally, in order to improve the generalization ability of the model, you can also perform some enhancement operations on the image, such as random rotation, translation, scaling, etc. There are many methods of image enhancement, and the appropriate method can be selected according to specific application requirements and data characteristics.

Posture prediction model 204: This model is a pre-trained deep learning model that can predict the posture of the human body from the input image. The model may be a convolutional neural network (CNN) or other model suitable for processing image data.

This model is often called a "pose estimation network", which can be a convolutional neural network (CNN) specifically designed for human pose estimation tasks. It will be described below in conjunction with Figure 3. Its basic structure is as follows:

Input layer 302: The input to the model is a preprocessed image, which may be 256x256 pixels in size. The image is usually in RGB format, so the number of input channels is 3.

Convolutional layer 304: Next, there is a series of convolutional layers. Each convolutional layer contains convolution operations, nonlinear activation functions (such as ReLU), and pooling operations (optional). The convolution operation can extract features in the image, and the pooling operation can reduce the spatial size of the features, thereby reducing the complexity of the model.

Key point prediction layer 306: After the convolutional layer, there is a key point prediction layer. The task of this layer is to predict the probability that each pixel in the image is a key point of the human body (such as neck, elbow, knee, etc.). It usually consists of a convolution operation and a Softmax activation function. The number of output channels of the convolution operation is equal to the number of key points, and each channel corresponds to the probability map of a key point. The Softmax function is used to normalize the values of the probability map to between 0-1.

Keypoint offset vector prediction layer 308: This layer is parallel to the keypoint probability prediction layer 306. This layer consists of a convolution operation, and the number of output channels is twice the number of keypoints, since each keypoint has an offset in both x and y directions. The task of this layer is to predict the offset vector of each keypoint, that is, a small change relative to the predicted coordinates.

Decoding layer 310: After the key point prediction layer, there is the decoding layer. The task of the decoding layer is to convert the probability map into specific key point coordinates. The specific method is to find the maximum value point in each probability map, then add the corresponding offset vector, and use its coordinates as the coordinates of the key point.

Output layer 312: Finally, the output of the model is the coordinates of all key points. They form the posture of the human body.

During the implementation process, the structure and parameters of the deep learning model need to be optimized through a large amount of training data. Training data usually includes images with human body key point annotations. Through optimization algorithms such as backpropagation and gradient descent, the parameters of the model can be continuously updated to make the model's prediction results and the real results as close as possible.

Post-processing part 206: The post-processing part is used to convert the output of the attitude prediction model into attitude data that can be used directly. This may include converting continuous numerical values output by the model into discrete pose categories, or converting relative coordinates output by the model into absolute coordinates.

The post-processing part includes the following implementation steps:

Data analysis S2001: First, the key point coordinates and key point offset vectors are parsed from the output of the attitude prediction model. These data represent the location of key points in the form of continuous numerical values.

Adjustment of key point coordinates S2002: Then, use the key point offset vector to correct the key point coordinates to make the prediction result more accurate. The specific method is to add the key point coordinates to the corresponding offset vector to obtain the corrected key point coordinates.

Coordinate conversion S2003: In order to meet the needs of acupuncture positioning, the key point coordinates need to be converted from the coordinate system of the head-mounted device to the coordinate system of the real world. This can be achieved through the following steps: first, map the corrected key point coordinates into the depth image to obtain the pixel coordinates and depth values of the key points; then, use the internal parameters of the head-mounted display device (such as focal length and principal point coordinates) and External parameters, such as the device's position and orientation, convert pixel coordinates and depth values into real-world coordinates.

Coordinate filtering S2004: In order to improve the stability of coordinate prediction, we can use some filtering algorithms, such as Kalman filter, to smooth the continuous prediction results.

Body part identification S2005: Since the positions of acupuncture points are usually defined relative to body parts, it is necessary to identify the body parts corresponding to key points. This can be achieved through some pre-defined rules, for example, if a key point is located between two arm key points, then it can be considered to be a chest key point.

Posture category recognition S2006: Finally, the patient's posture category can be identified based on the relative positional relationship of key points. This can be achieved through some predefined rules, for example, if all hand key points are located below the head key points, then the patient can be considered seated.

After the head-mounted display device 102 captures an image, the image will first be sent to the image preprocessing part. The preprocessed image data will be input into the posture prediction model, and the model will predict the posture of the human body in the image. Finally, the post-processing part converts the predicted attitude data into a format that can be used directly.

Through the above steps, the human posture recognition module can predict the patient's posture from the image of the headset device, providing a basis for subsequent acupuncture point positioning. The human posture recognition module can be implemented in a computer or in a computing unit in a head-mounted display device.

The human body model alignment module 108 provided in this embodiment is used to align the virtual human body model to the real patient's body through a numerical optimization algorithm based on the predicted patient posture.

The human model alignment module 108 is mainly composed of a posture data processing part, an alignment algorithm part and a model adjustment part.

Posture data processing part: This part is responsible for processing the data from the human posture recognition module and converting it into a format suitable for input alignment algorithm. This may include converting pose categories into pose parameters, or converting absolute coordinates into relative coordinates. If the input posture data is in a category form (for example: standing, sitting, etc.), it needs to be converted into specific posture parameters, such as joint angles or joint coordinates, etc. This step can be achieved through lookup tables, rule-based mapping, or machine learning models. If the input posture data is absolute coordinates, it needs to be converted into coordinates relative to the human body model. This can be achieved by subtracting the mean of the model coordinates or other predetermined reference points.

Alignment algorithm part: This part uses a numerical optimization algorithm to align the model with the real human body as much as possible by optimizing the posture parameters of the human body model. The algorithm may be an iterative optimization algorithm such as gradient descent, or another algorithm suitable for this type of problem.

The alignment algorithm part can be implemented through the following steps:

(1) Set the objective function:

Assume that there are predicted posture parameters, represented by vector P (including the positions or angles of all joints), and posture parameters of the virtual human model, represented by vector M. The goal here is to minimize the difference between these two vectors.

Therefore, the objective function can be set as the square of the Euclidean distance between the two vectors, that is, L = ||PM|| ² . The smaller the value of this function, the smaller the difference between the pose of the virtual model and the real pose.

On this basis, in order to improve the accuracy of acupuncture positioning, an optimization item for acupuncture points can be added. Assuming that the positions of the acupuncture points in the predicted posture and model posture are P_a and M_a respectively, the position difference of the acupuncture points can also be included in the objective function, that is, L=||PM|| ² +λ||P_a-M_a|| ² , where λ is a weight parameter used to adjust the relative importance of the two items.

The data (P_a) for predicting the acupuncture point position under the posture comes from the output of the established acupuncture knowledge base and the human posture recognition module. Posture recognition: First, the human posture recognition module predicts the patient's posture parameters P by analyzing the real environment images obtained from the headset device, which may include the positions or angles of all joints, etc. Acupuncture knowledge base: The established acupuncture knowledge base stores the relative position information of acupuncture points relative to each joint. This information is compiled based on ancient classics such as the Huangdi Neijing and modern acupuncture research results. Acupuncture point position prediction: According to the predicted posture parameter P and the acupuncture knowledge base, the acupuncture point position P_a in the predicted posture can be calculated. Specifically, this usually involves some analytical or numerical geometric calculations, such as rotations and translations.

It should be noted that because everyone's body structure has certain differences, the predicted acupuncture point location P_a may not be completely accurate. This is why an optimization term for acupuncture points should be added to the objective function to further improve the accuracy of acupuncture positioning.

(2) Optimization algorithm:

A suitable numerical optimization algorithm, such as the gradient descent algorithm, is used to minimize the objective function. Specifically, the attitude parameter M of the model is first randomly initialized, and then in each iteration, M is adjusted according to the gradient direction of the objective function, so that the value of the objective function gradually decreases.

During the optimization process, the physical constraints of the model, such as the upper and lower limits of joint angles, must be taken into consideration. Specifically, if in a certain iteration, a certain joint angle of the model exceeds its physical upper and lower limits, it needs to be adjusted back to the limit range. This can be achieved by clipping M after each iteration step, that is, M = min (max (M, lower_bound), upper_bound), where lower_bound and upper_bound represent the lower and upper limits of the joint angles respectively.

Through the above steps, the posture of the virtual human body model can be continuously optimized to gradually approach the real posture, while ensuring the physical rationality of the model and improving the accuracy of acupuncture positioning.

Model adjustment part: The model adjustment part adjusts the posture of the virtual human model according to the results of the alignment algorithm to align it with the real human body as much as possible.

After the human posture recognition module predicts the patient's posture, the posture data will first be sent to the posture data processing part for processing. The processed data will be input to the alignment algorithm part, which will optimize the posture parameters of the human model to make the model align to the real human body as much as possible. Finally, the model adjustment part will adjust the posture of the virtual human model based on the optimization results. At the same time, the acupuncture point positions on the virtual human body model are updated according to changes in the model's posture. For example, the position of the acupuncture point can be dynamically adjusted according to changes in the surrounding muscles and bones through linear interpolation or other suitable interpolation methods.

Through the above steps, the posture of the virtual human model can be aligned with the posture of the real patient.

The acupuncture point positioning module 110 provided in this embodiment is used to convert the position of the virtual acupuncture point to the real patient's body based on the alignment relationship between the virtual human body model and the real patient's body.

The acupuncture point positioning module 110 is mainly composed of an alignment data processing part, a positioning algorithm part and a positioning result output part.

Alignment data processing part: This part will first receive data from the human model alignment module, which includes the joint angles, positions and other information of the virtual human model. The work of this part is to convert these posture parameters into an alignment transformation matrix, that is, to convert the model posture parameters into a 4x4 homogeneous transformation matrix, which can describe the rotation, translation, scaling and other transformations of the model. The specific conversion method can use Euler angles or quaternions to describe rotation, and translation and scaling can directly use the corresponding vectors and values. The conversion results are fed into the targeting algorithm part.

Positioning algorithm part: This part receives the transformation matrix from the alignment data processing part and the preset position of the virtual acupuncture point. The position of the virtual acupuncture point can be expressed as a three-dimensional coordinate. The positioning algorithm applies the transformation matrix to the location of the virtual acupuncture point to obtain the corresponding acupuncture point location on the real patient's body. The specific calculation process can use matrix multiplication operation, that is, the new position = transformation matrix * the position of the virtual acupuncture point.

Positioning result output part: Responsible for converting the results of the positioning algorithm into acupuncture point location data that can be used directly. The positioning result may be a three-dimensional coordinate. This part converts the three-dimensional coordinate value into a distance and angle relative to a reference point (such as a certain part of the patient's body or the camera of the head-mounted display device), or converts it into the pixel coordinates of a two-dimensional image. . The specific conversion process can be completed through the corresponding coordinate system conversion formula. The final output format can be set according to actual needs, such as text or visual output.

When the human body model alignment module completes the alignment operation, the alignment data will first be sent to the alignment data processing part for processing. The processed data will be input into the positioning algorithm part, and the positioning algorithm will calculate the corresponding acupuncture point positions on the real patient's body based on the alignment data and the positions of the virtual acupuncture points. Finally, the positioning result output part converts the positioning results into a format that can be used directly.

Through the above steps, the acupuncture point positioning module can convert the position of the virtual acupuncture point to the real patient's body, providing a basis for the acupuncturist's operation.

The user interaction module 112 provided in this embodiment is used to provide a user interaction method, so that the user can not only see the real patient's body, but also see the virtual human body model and the acupuncture points converted to the real patient's body. .

The user interaction module is mainly composed of a virtual reality (VR) display part, an interactive input part and an interactive feedback part.

Virtual reality (VR) display part: This part is responsible for integrating and displaying the virtual human model, virtual acupuncture points, and real environment images captured through the head-mounted display device into the user's field of vision. This involves some image synthesis technologies, such as image fusion, image splicing, etc. In addition, the display part can be upgraded to an augmented reality (AR) display, which can combine virtual human models and acupuncture points with the real environment, and can also dynamically adjust the display content according to the user's perspective. This involves some image fusion and three-dimensional image rendering technology to achieve a seamless blend of virtual and real.

In addition, the system can display virtual acupuncture points in the form of three-dimensional arrows or highlighted areas, which improves the visibility of acupuncture points, provides users with intuitive visual instructions, and helps users better understand and locate acupuncture points. These three-dimensional arrows or highlighted areas can be distinguished by color or size based on the importance of the acupuncture points, body position and other factors, allowing users to quickly distinguish and identify the priority and relative position of each acupuncture point.

In addition, virtual acupuncture points can also be displayed in the form of three-dimensional arrows or highlighted areas to facilitate users to better understand and locate acupuncture points. In order to further improve the user's interactive experience, more types of display methods can be introduced. For example, animation effects can be used, such as making the model skin at the virtual acupuncture point fluctuate or glow, to simulate the scene when the acupuncture point is stimulated. Augmented reality (AR) technology can also be used to add descriptive text labels or graphic symbols to virtual acupuncture points to provide information about the names, functions and applicable diseases of acupuncture points.

In some complex acupuncture operations, virtual reality (VR) technology can also be used to display the three-dimensional dynamic process of the operation, such as needle insertion, needle rotation, needle lifting, etc., so that users can understand the steps of the operation in detail from various angles. and techniques to further enhance the teaching and training functions of the system. In addition, combined with speech recognition technology, users can query or operate models and acupuncture points through voice commands, providing a more convenient and faster interaction method. These innovative display methods not only improve the user experience of the system, but also help improve the accuracy and efficiency of acupuncture positioning.

Interactive input part: This part receives and processes the user's interactive input, such as gesture operations, voice commands, etc. This requires the use of some user input recognition technologies, such as gesture recognition, voice recognition, etc. In addition, a gaze tracking technology can be added so that the system can know which acupuncture point or model area the user is looking at and automatically enlarge or highlight this area. Gaze tracking technology can be implemented through specialized equipment or using the front-facing camera in some existing headsets (such as certain types of VR equipment).

Interactive feedback part: This part adjusts the display of the virtual human model and acupuncture points based on the user's interactive input, and provides corresponding user feedback. This can include changing the location, color, size and other attributes of virtual objects, or providing feedback in the form of vibrations, sounds, etc. In addition, virtual reality technology can be used to display an animation of a virtual needle piercing the selected point after the user selects an acupuncture point to provide visual feedback. In addition, it can also be combined with tactile feedback devices, such as handheld vibration devices or wearable devices. When the virtual needle penetrates the virtual model, the device will vibrate, allowing the user to feel the feeling of the virtual needle penetrating, thus providing more Intuitive and authentic feedback.

The user interaction module can also include an adaptive user assistance system, which can automatically adjust the display mode of the model and acupuncture points, as well as the feedback mode according to the user's operating habits and technical level. For example, for novice users, the system can display more guidance information and auxiliary lines, and for advanced users, it can provide more freedom and customization options. This system can automatically adjust by collecting and analyzing user operation data, or it can allow users to set it manually.

The picture that the user sees through the head-mounted display device is generated by the virtual reality (VR) display part, which includes virtual human models, acupuncture points, and images of the real environment. Users can perform operations through the interactive input section, such as rotating the model, selecting acupuncture points, etc. The interactive input part will recognize the user's input and send the recognition results to the interactive feedback part. The interactive feedback part adjusts the display of virtual objects and provides feedback based on the recognition results, so that the user can perceive that his operation has been received and responded to by the system.

Through the above steps, the user interaction module 112 can provide an intuitive and easy-to-operate interaction method, allowing the user to see and operate the virtual human body model and acupuncture points while seeing the real patient's body.

The second embodiment of the present application provides an auxiliary acupuncture positioning method based on a three-dimensional human body model. Please refer to Figure 4, which is a schematic diagram of the second embodiment of the present application. An auxiliary acupuncture positioning method based on a three-dimensional human body model according to the second embodiment of the present application will be described in detail below with reference to FIG. 4 . Since this embodiment is similar to the first embodiment, the introduction is relatively simple. Please refer to the relevant parts of the first embodiment.

This embodiment provides an auxiliary acupuncture positioning method based on a three-dimensional human body model, including the following steps:

S400: Use the head-mounted display device to capture images of the patient's real environment and obtain its three-dimensional structural information;

S402: Create a three-dimensional human body model and mark acupuncture points on the model;

S404: Predict the patient's posture from the image of the headset device;

S406: According to the predicted patient posture, align the virtual human body model to the real patient's body through a numerical optimization algorithm;

S408 converts the position of the virtual acupuncture point to the real patient's body based on the alignment relationship between the virtual human body model and the real patient's body;

S410: Provide a user interaction method through user interaction technology, allowing users to see the virtual human body model and the acupuncture points converted to the real patient's body while seeing the real patient's body.

In this embodiment, the user interaction method also includes allowing the user to select and operate acupuncture points through gestures or voice commands, and display relevant information on the screen after the user selects an acupuncture point.

In this embodiment, predicting the patient's posture from the image of the headset device includes:

Use a deep neural network to predict patient posture from images from a headset.

In this embodiment, the virtual human body model is aligned to the real patient's body through a numerical optimization algorithm based on the predicted patient posture, including:

Use a gradient descent algorithm to find model parameters that minimize the difference between the virtual human model and the patient's body;

According to the model parameters, the virtual human body model is aligned to the real patient's body.

In this embodiment, converting the position of the virtual acupuncture point to the real patient's body based on the alignment relationship between the virtual human body model and the real patient's body includes:

According to the alignment relationship and the position of the virtual acupuncture point, the corresponding acupuncture point position on the real patient's body is calculated, thereby converting the position of the virtual acupuncture point on the real patient's body.

Although the present application is disclosed as above with preferred embodiments, it is not intended to limit the present application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the present application The scope of protection shall be subject to the scope defined by the claims of this application.

Claims (10)

1. An auxiliary acupuncture positioning system based on a three-dimensional human body model. The system includes: Head-mounted display device, which has a built-in camera and depth sensor to capture images of the real environment and its three-dimensional structure information; Three-dimensional human body model, which is a virtual human body model created on a computer and marked with virtual acupuncture points; Human posture recognition module, used to predict the patient's posture from the image captured by the headset device and its three-dimensional structure information; The human body model alignment module is used to align the virtual human body model to the real patient's body through numerical optimization algorithms based on the predicted patient posture; An acupuncture point positioning module is used to convert the position of the virtual acupuncture point to the real patient's body based on the alignment relationship between the virtual human body model and the real patient's body; The user interaction module is used to provide a user interaction method that allows the user to see the virtual human body model and the acupuncture points converted to the real patient's body while seeing the real patient's body; The human posture recognition module consists of an image pre-processing part, a posture prediction model and a post-processing part; the image pre-processing part is implemented through the following steps: Image reception S1001: First, use the image information captured by the headset device as the original input; Smart Cropping S1002: By using the smart cropping algorithm, it automatically identifies the position of the human body in the image and crops out this area; the smart cropping algorithm uses a pre-trained deep learning model; when cropping, the image is first input To the model, the model will output a heat map of the human body's position; then, based on the maximum point in the heat map, the approximate center position of the human body is determined; finally, an area of predefined size is cut out with this center position as the center; Scale normalization S1003: uniformly scale the size of the image to a predefined size; Pixel value normalization S1004: After scale normalization, normalize the pixel values of the image; Image enhancement S1005: In order to improve the generalization ability of the model, image enhancement operations are performed; Posture prediction model: This model is a pre-trained deep learning model that can predict the posture of the human body from the input image. Its structure is as follows: Input layer: The input of the model is a preprocessed image, the size is 256x256 pixels; the image is in RGB format, and the number of input channels is 3; Convolutional layer: Each convolutional layer contains convolution operations, nonlinear activation functions and pooling operations; Key point prediction layer: After the convolution layer, there is the key point prediction layer; the task of this layer is to predict the probability that each pixel in the image is a human body key point; it consists of a convolution operation and a Softmax activation function; The number of output channels of the convolution operation is equal to the number of key points, and each channel corresponds to the probability map of a key point; the Softmax function is used to normalize the value of the probability map to between 0-1; Keypoint offset vector prediction layer: This layer is parallel to the keypoint probability prediction layer; this layer consists of a convolution operation, and the number of output channels is twice the number of keypoints, since each keypoint has two x and y Directional offset; the task of this layer is to predict the offset vector of each key point, that is, a small change relative to the predicted coordinates; Decoding layer: After the key point prediction layer, there is the decoding layer; the task of the decoding layer is to convert the probability map into specific key point coordinates; the specific method is to find the maximum value point in each probability map, and then add the corresponding The offset vector of , use its coordinates as the coordinates of the key point; Output layer: Finally, the output of the model is the coordinates of all key points; they constitute the posture of the human body; Post-processing part: The post-processing part is used to convert the output of the attitude prediction model into directly used attitude data; this includes converting the continuous values output by the model into discrete attitude categories; The post-processing part includes the following steps: Data analysis S2001: First, the key point coordinates and key point offset vectors are parsed from the output of the attitude prediction model; these data represent the location of the key points in the form of continuous numerical values; Adjustment of key point coordinates S2002: Then, use the key point offset vector to correct the key point coordinates to make the prediction result more accurate; the specific method is to add the key point coordinates to the corresponding offset vector to obtain the corrected key point coordinates ; Coordinate conversion S2003: Convert key point coordinates from the coordinate system of the headset device to the real-world coordinate system; achieved through the following steps: First, map the corrected key point coordinates to the depth image to obtain the pixel coordinates and depth of the key point value; then, use the internal and external parameters of the headset device to convert the pixel coordinates and depth values into real-world coordinates; Coordinate filtering S2004: Use filtering algorithm to smooth the continuous prediction results; Body part recognition S2005: When a key point is located between two arm key points, it is determined to be the key point of the chest; Posture category recognition S2006: Identify the patient's posture category based on the relative position of key points; if all hand key points are located below the head key points, then the patient is considered to be sitting; When the head-mounted display device captures an image, the image will first be sent to the image pre-processing part; the pre-processed image data will be input into the posture prediction model, and the model will predict the posture of the human body in the image; finally, the post-processing part will predict Convert the outgoing attitude data into a directly usable format; The human model alignment module consists of a posture data processing part, an alignment algorithm part and a model adjustment part; Posture data processing part: This part is responsible for processing data from the human posture recognition module, which includes converting posture categories into posture parameters, or converting absolute coordinates into relative coordinates; if the input posture data is in category form, it needs to be converted is the specific posture parameter; this step is implemented through a lookup table, rule-based mapping or machine learning model; if the input posture data is absolute coordinates, it needs to be converted into coordinates relative to the human body model; this is done by subtracting the model coordinates average value or other predetermined reference points; Alignment algorithm part: This part uses a numerical optimization algorithm to align the model with the real human body by optimizing the posture parameters of the human body model; The alignment algorithm is partially implemented through the following steps: (1) Set the objective function: Suppose there are predicted posture parameters, represented by vector P, which includes the positions or angles of all joints, and posture parameters of the virtual human model, represented by vector M; the goal here is to minimize the difference between these two vectors; The objective function is set as the square of the Euclidean distance between the two vectors, that is, L = ||P - M|| ² ; the smaller the value of this function, the smaller the difference between the posture of the virtual model and the real posture; On this basis, add an optimization term for acupuncture points; let the positions of the acupuncture points in the predicted posture and model posture be P_a and M_a respectively, and incorporate the position difference of the acupuncture points into the objective function, that is, L = ||P - M|| ² + λ||P_a - M_a|| ² , where λ is a weight parameter used to adjust the relative importance of the two items; The data P_a for predicting the acupuncture point position under the posture comes from the established acupuncture knowledge base and the output of the human posture recognition module; posture recognition: First, the human posture recognition module predicts the patient's posture by analyzing the real environment image obtained from the headset device. Posture parameter P, acupuncture knowledge base: the established acupuncture knowledge base stores the relative position information of acupuncture points relative to each joint; acupuncture point position prediction: based on the predicted posture parameter P and the acupuncture knowledge base, calculate the predicted posture Acupuncture point location P_a; (2) Optimization algorithm: A numerical optimization algorithm is used to minimize the objective function; specifically, the attitude parameter M of the model is first randomly initialized, and then in each iteration, M is adjusted according to the gradient direction of the objective function, so that the value of the objective function gradually decreases; During the optimization process, the physical constraints of the model, including the upper and lower limits of joint angles, must be taken into consideration; if in a certain iteration, a joint angle of the model exceeds its physical upper and lower limits, it needs to be adjusted back to the limit. Within the range; this is achieved by clipping M after each iteration, that is, M = min(max(M, lower_bound), upper_bound), where lower_bound and upper_bound represent the lower limit and upper limit of the joint angle respectively; Model adjustment part: The model adjustment part adjusts the posture of the virtual human model according to the results of the alignment algorithm to align it with the real human body; When the human posture recognition module predicts the patient's posture, the posture data will first be sent to the posture data processing part for processing; the processed data will be input to the alignment algorithm part, and the alignment algorithm will optimize the posture parameters of the human model to align the model to the real human body; finally, the model adjustment part will adjust the posture of the virtual human body model according to the optimization results; at the same time, the acupuncture point positions on the virtual human body model will be updated according to changes in the model posture; The acupuncture point positioning module consists of an alignment data processing part, a positioning algorithm part and a positioning result output part; Alignment data processing part: This part will first receive data from the human model alignment module, which includes joint angles and position information of the virtual human model; convert these posture parameters into an alignment transformation matrix, and convert the model posture parameters into a 4x4 homogeneous transformation matrix; the transformation result will be sent to the positioning algorithm part; Positioning algorithm part: This part receives the transformation matrix from the alignment data processing part and the preset position of the virtual acupuncture point; the position of the virtual acupuncture point is represented as a three-dimensional coordinate; the positioning algorithm applies the transformation matrix to the position of the virtual acupuncture point , to obtain the corresponding acupuncture point position on the real patient's body; the specific calculation process uses matrix multiplication operation, that is, the new position = transformation matrix * the position of the virtual acupuncture point; Positioning result output part: Responsible for converting the results of the positioning algorithm into directly used acupuncture point position data; the positioning result is a three-dimensional coordinate. This part converts the three-dimensional coordinate value into a distance and angle relative to the reference point, or converts it into Pixel coordinates of the two-dimensional image; the specific conversion process is completed through the corresponding coordinate system conversion formula; When the human body model alignment module completes the alignment operation, the alignment data will first be sent to the alignment data processing part for processing; the processed data will be input to the positioning algorithm part, and the positioning algorithm will calculate the real acupuncture point based on the alignment data and the position of the virtual acupuncture point. The corresponding acupuncture point location on the patient's body; finally, the positioning result output part converts the positioning result into a format for direct use. 2. The auxiliary acupuncture positioning system according to claim 1, characterized in that the user interaction module allows the user to select and operate acupuncture points through gestures or voice commands, and displays it on the screen after the user selects an acupuncture point. out relevant information. 3. The auxiliary acupuncture positioning system according to claim 1, wherein the human posture recognition module includes a deep neural network for predicting the patient's posture from the image of the headset. 4. The auxiliary acupuncture positioning system according to claim 1, wherein the human body model alignment module uses a gradient descent algorithm to calculate model parameters that minimize the difference between the virtual human body model and the patient's body. 5. The auxiliary acupuncture positioning system according to claim 1, wherein the acupuncture point positioning module calculates the corresponding acupuncture point position on the real patient's body based on the alignment relationship and the position of the virtual acupuncture point, thereby The positions of the virtual acupuncture points are converted to the real patient's body. 6. An auxiliary acupuncture positioning method based on a three-dimensional human body model auxiliary acupuncture positioning system as claimed in claim 1, the method comprising: Use the head-mounted display device to capture images of the patient's real environment and obtain its three-dimensional structural information; Create a three-dimensional human body model and mark acupuncture points on the model; Predict the patient's posture from images from the headset; According to the predicted patient posture, the virtual human body model is aligned to the real patient's body through a numerical optimization algorithm; According to the alignment relationship between the virtual human body model and the real patient's body, the position of the virtual acupuncture point is converted to the real patient's body; Through user interaction technology, a user interaction method is provided, so that users can see the virtual human body model and the acupuncture points converted to the real patient's body while seeing the real patient's body. 7. The auxiliary acupuncture positioning method according to claim 6, wherein the user interaction method further includes allowing the user to select and operate acupuncture points through gestures or voice commands, and after the user selects an acupuncture point, the Relevant information is displayed on. 8. The auxiliary acupuncture positioning method according to claim 6, wherein predicting the patient's posture from the image of the headset device includes: Use a deep neural network to predict patient posture from images from a headset. 9. The auxiliary acupuncture positioning method according to claim 6, wherein the virtual human body model is aligned to the real patient's body through a numerical optimization algorithm based on the predicted patient posture, including: Use a gradient descent algorithm to find model parameters that minimize the difference between the virtual human model and the patient's body; According to the model parameters, the virtual human body model is aligned to the real patient's body. 10. The auxiliary acupuncture positioning method according to claim 6, characterized in that, according to the alignment relationship between the virtual human body model and the real patient's body, converting the position of the virtual acupuncture point to the real patient's body includes: According to the alignment relationship and the position of the virtual acupuncture point, the corresponding acupuncture point position on the real patient's body is calculated, thereby converting the position of the virtual acupuncture point on the real patient's body.