CN117727300A - Voice information processing method and device - Google Patents

Abstract

The present disclosure relates to the field of smart medical technology, and in particular, to a method and apparatus for processing voice information. The voice information processing method comprises the following steps: collecting voice information of an operator in the operation process; acquiring state data of a surgical robot and image data of a focus area; extracting the semantics of the voice information according to the state data and the image data to obtain first semantic prompt information; and sending the first semantic prompt information to present the first semantic prompt information to auxiliary medical staff of the operator. According to the embodiment of the specification, the state data of the surgical robot, the image data of the focus area and other intraoperative information can be combined, the semantics of the voice information can be extracted, and therefore semantic prompt information can be presented to auxiliary medical staff. The situations of untimely and inaccurate surgical information transmission and the like caused by the existence of noise on the surgical site are avoided, so that the surgical risk is reduced.

Description

Voice information processing method and device

Technical field

This specification relates to the field of smart medical technology, and in particular to a voice information processing method and device.

Background technique

Medical staff need to communicate with each other through voice to exchange surgical information. In the existing technology, the voice information of medical staff can be collected, the voice information can be converted into text information, and the text information can be presented on the display device. Convey the message expressed by the medical staff through text messages on the display device.

However, various noises often exist at the surgical site, so the collected voice information contains the noise. The text information converted in this way may be inaccurate, affecting the transmission of surgical information between medical staff and causing surgical risks.

Contents of the invention

Embodiments of this specification provide a voice information processing method, device, and computer storage medium, which are used to refine the semantics of medical staff's voice information in combination with surgical scenarios to avoid the presence of noise affecting the transmission of surgical information.

The embodiment of this specification provides a voice information processing method, including:

Collect the surgeon’s voice information during the operation;

Obtain status data of the surgical robot and image data of the lesion area;

Extract the semantics of the speech information based on the status data and image data to obtain the first semantic prompt information;

The first semantic prompt information is sent to present the first semantic prompt information to the surgeon's paramedical staff.

The embodiment of this specification also provides another voice information processing method, including:

Receive first semantic prompt information, which is obtained by refining the surgeon's voice information during the operation;

Collect the response voice information of the surgeon's auxiliary medical staff to the first semantic prompt information;

Obtain image data of the lesion area;

Extract the semantics of the response voice information based on the image data and the first semantic prompt information to obtain the second semantic prompt information;

The second semantic prompt information is fed back to present the second semantic prompt information to the operator.

Embodiments of this specification also provide a voice information processing device, including:

The collection unit is used to collect the operator's voice information during the operation;

An acquisition unit is used to acquire status data of the surgical robot and image data of the lesion area;

The refining unit is used to refine the semantics of the speech information based on the status data and the image data to obtain the first semantic prompt information;

The sending unit is used for sending the first semantic prompt information to present the first semantic prompt information to the surgeon's auxiliary medical staff.

The embodiment of this specification also provides another voice information processing device, including:

A receiving unit configured to receive first semantic prompt information, which is obtained by refining the surgeon's voice information during the operation;

The collection unit is used to collect the response voice information of the surgeon's auxiliary medical staff to the first semantic prompt information;

An acquisition unit is used to acquire image data of the lesion area;

An extraction unit, configured to refine the semantics of the response voice information based on the image data and the first semantic prompt information, to obtain the second semantic prompt information;

The feedback unit is used to feed back the second semantic prompt information to present the second semantic prompt information to the operator.

Embodiments of this specification also provide a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above voice information processing method is implemented.

The voice information processing method of the embodiment of this specification can collect the voice information of the surgeon during the operation; can obtain the status data of the surgical robot and the image data of the lesion area; can refine the semantics of the voice information based on the status data and image data, and obtain First semantic prompt information; the first semantic prompt information may be sent to present the first semantic prompt information to the surgeon's paramedical staff. In this way, intraoperative information such as the status data of the surgical robot and the image data of the lesion area can be combined to refine the semantics of the voice information and present semantic prompt information to the auxiliary medical staff. It avoids untimely and inaccurate transmission of surgical information due to the presence of noise at the surgical site, thereby reducing surgical risks.

Description of the drawings

In order to more clearly explain the embodiments of this specification or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. The drawings in the following description are only for the purpose of this specification. For some embodiments described in , for those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting any creative effort.

Figure 1 is a schematic functional structural diagram of the surgical system in the embodiment of this specification;

Figure 2 is a schematic flow chart of the voice information processing method in the embodiment of this specification;

Figure 3 is a schematic diagram of the functional structure of the storage unit in the manual speed instrument according to the embodiment of this specification;

Figure 4 is a schematic diagram of the functional structure of the joints in the hand speed instrument in the embodiment of this specification;

Figure 5 is a schematic flow chart for refining semantics based on surgical procedures and lesion information in the embodiment of this specification;

Figure 6 is a schematic flowchart of refining semantics according to the working mode in the embodiment of this specification;

Figure 7 is a schematic diagram of the first semantic prompt information and the semantic prompt image in the embodiment of this specification;

Figure 8 is a schematic flow chart of the voice information processing method in the embodiment of this specification;

Figure 9 is a schematic functional structure diagram of the voice information processing device in the embodiment of this specification;

Figure 10 is a schematic functional structure diagram of the voice information processing device in the embodiment of this specification.

Detailed ways

The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this specification. Obviously, the described embodiments are only some of the embodiments of this specification, rather than all of the embodiments. The specific embodiments described here are only used to explain the present disclosure, but not to limit the present disclosure. Based on the described embodiments of the present disclosure, all other embodiments obtained by those of ordinary skill in the art fall within the scope of protection of the present disclosure. In addition, relational terms such as "first" and "second" are merely used to distinguish one entity or operation from another entity or operation and do not necessarily require or imply the existence of any relationship between these entities or operations. This actual relationship or sequence.

Noise at the surgical site can include sounds produced by medical equipment during operation, sounds made by medical staff when communicating unrelated to the surgery, etc. Of course, the noise at the surgical site can also include other types of noise, which will not be listed here.

See Figure 1. Embodiments of this specification provide a surgical system. The surgical system may include a first medical device 1 and a second medical device 2 . The first medical equipment 1 and the second medical equipment 2 are equipment for medical staff. The medical staff includes the surgeon, the surgeon's auxiliary medical staff, etc. The surgeon may include a surgeon, and the auxiliary medical staff may include an assistant doctor, a nurse, etc. The first medical device 1 and the second medical device 2 may include a surgical robot, an imaging trolley, or the like. The surgical robot may include multiple of a control end, an execution end, and a display component. The control end and the execution end have a master-slave control relationship. The control terminal is the main terminal and is used for receiving instructions input by the doctor. The execution end is a slave end, used to execute corresponding actions according to the instructions. The execution end may include a control arm, surgical instrument, etc. The control arm may be a robotic arm, and the surgical instrument may be installed on the control arm. The execution end can control one or more control arms, surgical instruments, etc. to perform corresponding actions according to the instructions input by the doctor. The display component is used for display. The image trolley is used to display images of the doctor's operations during the operation so that other personnel can understand the operation process.

The first medical device 1 and the second medical device 2 may include sound collection components (eg microphones). The first medical equipment 1 and the second medical equipment 2 are equipment for different medical staff. Different medical staff can communicate by voice through the first medical device 1 and the second medical device 2 to exchange surgical information. Specifically, the first medical device 1 can collect the voice information of medical staff; can obtain the status data of the surgical robot and the image data of the lesion area; can extract the semantics of the voice information based on the status data and image data to obtain the first semantic prompt information; The first semantic prompt information may be sent to the second medical device 2 . The second medical device 2 can present the first semantic prompt information. In addition, the second medical device 2 can collect the voice information of the medical staff responding to the first semantic prompt information; can obtain the image data of the lesion area; and can refine the semantics of the response voice information based on the image data and the first semantic prompt information to obtain the second Semantic prompt information; second semantic prompt information can be sent to the first medical device 1 . The first medical device 1 may present second semantic prompt information.

By combining the status data of the surgical robot and the image data of the lesion area and other intraoperative information, the semantics of the voice information of the medical staff are refined to avoid the untimely and inaccurate transmission of surgical information due to the presence of noise at the surgical site, and reduce the risk of Surgical Risks.

In some scene examples, the first medical equipment 1 includes a surgical robot facing the surgeon, and the second medical equipment 2 includes an imaging trolley facing the auxiliary medical staff. The surgical robot can collect the operator's voice information during the operation; it can obtain the status data of the surgical robot and the image data of the lesion area; it can refine the semantics of the voice information based on the status data and image data to obtain the first semantic prompt information; it can provide The image trolley sends the first semantic prompt information. The image cart may present first semantic prompt information. In this way, the operator can exchange information with the auxiliary medical staff during the operation and remind the auxiliary medical staff to perform some auxiliary treatment of the lesion. Auxiliary medical staff can understand the needs of the surgeon in a timely manner, make timely preparations in advance, and effectively help the surgeon complete the surgical operation. Avoid the adverse effects of delayed treatment of patient lesions due to delayed information exchange. In addition, the image trolley can also collect the voice information of the auxiliary medical staff responding to the first semantic prompt information; can obtain the image data of the lesion area; and can refine the semantics of the response voice information based on the image data and the first semantic prompt information, to obtain Second semantic prompt information; the second semantic prompt information can be sent to the surgical robot. The surgical robot can present second semantic prompt information. In this way, the surgeon can know the response of the auxiliary medical staff in a timely manner.

In this scenario example, surgeons, auxiliary medical staff, etc. can communicate by voice through the surgical robot and image trolley to exchange surgical information. Moreover, since the voice information is semantically refined by combining the status data of the surgical robot and the image data of the lesion area and other intraoperative information, it avoids untimely and inaccurate transmission of surgical information due to the presence of noise at the surgical site, and reduces surgical risks. .

The surgical system may be adapted for local surgery. The first medical device 1 and the second medical device 2 may be proximal devices to each other. Of course, the surgical system can also be applied to remote surgery. The first medical device 1 and the second medical device 2 may be remote devices to each other. The first medical device 1 and the second medical device 2 perform long-distance data communication interaction through a 4G (fourth generation mobile communication technology) network, a 5G (fifth generation mobile communication technology) network, a wired network, etc.

See Figure 2. The embodiments of this specification provide a voice information processing method. The voice information processing method can be applied to the aforementioned first medical device 1 . The first medical device 1 may comprise a surgical robot. The voice information processing method may include the following steps.

Step 11: Collect the surgeon’s voice information during the operation.

In some embodiments, the surgeon includes a surgeon. The operator's voice information can be collected through a sound collection component (such as a microphone). For example, during the operation, the surgeon can inform the required information through voice to remind the auxiliary medical staff to assist in the treatment of the lesion. Then the voice information sent by the operator can be collected through the sound collection component. The auxiliary medical personnel include assistant doctors, nurses, etc.

Step 12: Obtain intraoperative information during the surgery.

In some embodiments, through the surgical robot, the surgeon can perform surgical operations under the guidance of medical imaging equipment such as endoscopes or CT. Intraoperative information can be obtained during the surgical procedure. The intraoperative information includes status data of the surgical robot, image data of the lesion area, etc. The status data is used to represent the status of the surgical robot, including but not limited to the instrument type of the surgical instrument, the posture data of the surgical robot, the force information at the end of the surgical instrument, the foot control signal of the surgical robot, and the working mode of the surgical robot. Data etc. The lesion area may be the area targeted by the surgical operation. Image data of the lesion area can be obtained through medical imaging equipment such as endoscopy or CT.

In some embodiments, see Figure 3. The surgical instrument has a storage unit 11 . The storage unit 11 includes ROM, flash memory, etc. The storage unit 11 stores hardware information of the surgical instrument, such as the code of the surgical instrument, the type of the surgical instrument, the duration of use of the surgical instrument, the number of times of use of the surgical instrument, the service life of the surgical instrument, etc. Hardware information is stored in the storage unit 11 when shipped from the factory. During surgery, surgical instruments can be attached to the control arms of the surgical robot. In this way, the instrument type of the surgical instrument can be obtained by reading the hardware information in the storage unit 11 . The types of instruments may include duckbill forceps, rat tooth forceps, powerful duckbill forceps, etc.

The pose data may include position data and/or attitude data. The position data includes position data of surgical instruments and the like. The posture data may include posture data of surgical instruments, posture data of control arms, etc. See Figure 4. The surgical instruments include a pitch joint 12, a yaw joint 13, a rotation joint 14, etc. The joint is used to implement operations such as pitching, rotating, and clamping of the end of the surgical instrument. Sensors are provided on the joints. The sensor is used to detect physical quantities of the joint (such as angle, displacement, etc.). The posture data of surgical instruments can be determined based on physical quantities, such as pitch angle, yaw angle, rotation angle, etc. In addition, the surgical robot includes one or more control arms. The control arm is provided with a plurality of joints, and the plurality of joints may include attitude joints and position joints. The attitude joint is used to control the attitude of the control arm. The position joint is used to control the position of surgical instruments. For example, the control arm may be provided with 7 joints, the 1st joint, the 2nd joint, and the 3rd joint are position joints, and the 4th joint, the 5th joint, the 6th joint, and the 7th joint are attitude joints. Sensors are provided on the joints. The sensor is used to detect physical quantities of the joint (such as angle, displacement, etc.). The attitude data of the control arm and the position data of the surgical instrument can be determined based on physical quantities. The sensors may include Hall sensors, encoders, and other sensors that can measure physical quantities of position.

The force information on the end of the surgical instrument may include whether the end of the surgical instrument is stressed, the magnitude of the force on the end of the surgical instrument, etc. Whether and how much force is exerted on the end of the surgical instrument can help determine whether the surgical instrument is currently performing a certain surgical operation. Specifically, the force information at the end of the surgical instrument can be obtained through the pressure sensor at the end of the surgical instrument. The force information may include clamping force, etc.

The operator can control the energy at the end of the surgical instrument, such as the clamping strength, etc. through the foot pedal. Thus, the foot control signal of the foot pedal can be obtained.

The working mode data is used to represent the working mode of the surgical robot. The working mode may include master-slave control mode, adjustment mode, fault mode, etc. In the master-slave control mode, the operator can control the surgical instruments to perform surgical operations. In the adjustment mode, the operator can adjust the position of the surgical instrument, for example, control the movement of the surgical instrument to the lesion area. In fault mode, the surgical robot becomes abnormal and cannot operate.

Step 13: Extract the semantics of the speech information based on the intraoperative information to obtain the first semantic prompt information.

In some embodiments, semantic information related to the intraoperative information can be extracted from the speech information through refining, and semantic information irrelevant to the intraoperative information can be eliminated; in addition, the extracted semantic information can also be optimized based on the intraoperative information. The optimization includes but is not limited to perfecting and supplementing the context of the extracted semantic information so that the extracted semantic information is easy to understand and avoids ambiguity. For example, the background of the extracted semantic information, the reasons for the extraction of semantic information, etc. are improved and supplemented. The first semantic prompt information can accurately and concisely express the information delivered by the surgeon. Compared with voice information, the first semantic prompt information is concise and concise, making it easier for the operator to understand the needs of the operator. The first semantic prompt information may include text information. Speech can express one or more semantic meanings. One or more kinds of first semantic prompt information can be extracted based on the intraoperative information.

In some embodiments, the text information corresponding to the voice information can be obtained; the semantics of the text information can be refined based on the intraoperative information to obtain the first semantic prompt information. Specifically, the voice information can be converted into text. The converted text can be used as the text information. In this way, the converted text can be semantically refined based on the intraoperative information, and the first semantic prompt information can be obtained. Alternatively, one or more keyword information can also be extracted from the converted text; the keyword information can be used as the text information. In this way, the keyword information can be semantically refined based on the intraoperative information, and the first semantic prompt information can be obtained. Speech information can be converted into corresponding text through speech recognition technology (Speech Recognition). For example, the voice information can be input into the speech conversion model to obtain the corresponding text. The speech conversion model includes but is not limited to RNN-Transducer model, LAS model, etc. Speech information can be input directly into the speech conversion model. Alternatively, the speech information can also be denoised, and the denoised audio data can be input into the speech conversion model.

Words without specific semantics and/or words with vague semantics can be removed from the converted text; the remaining words can be used as keyword information. For example, function words can be removed from the converted text; the remaining words can be used as keyword information. The function words may include honorific words (such as please, etc.), stop words (such as bar, etc.), etc. Specifically, keyword information can be extracted through a preset word set. The word set includes one or more words without specific semantics, words with relatively vague semantics, etc. For example, the converted text can be segmented into words; each segment can be matched in the word set; one or more segmented words that fail to match can be used as keyword information.

In some embodiments, see Figure 5. The working mode of the surgical robot can be identified based on the working mode data; if the surgical robot is in the master-slave control mode, the surgical procedure of the surgical robot can be determined based on the status data and image data; the lesion information of the lesion area can be determined based on the image data ; The semantics of the speech information can be refined based on the surgical procedure and lesion information, and the first semantic prompt information can be obtained. The surgical formula is used to represent the surgical operation performed on the surgical object. The surgical procedures may include nephrectomy, prostate resection, etc. The surgical operations may include suturing, clamping, electrocution, electrocoagulation, etc. The surgical objects may include organ tissues, such as kidneys, liver, arteries, veins, etc. The lesion information includes dynamic change information of the lesion area during surgical operations. The dynamic change information can reflect dynamic changes in the lesion area, such as changes in bleeding volume, completion of resection, coagulation, and smoke changes.

The lesion information of the lesion area can be determined based on an image data, such as whether there is bleeding in the lesion area. For example, the image data can be filtered to remove noise in the image data; the filtered image data can be smoothed to reduce the details and noise of the edge contours; the contour edges of the lesion area in the smoothed image data can be extracted ; Lesion information of the lesion area can be extracted based on the contour edges. For another example, machine learning can also be used to extract lesion information in the lesion area. For example, the image data can be input into the image analysis model to obtain the lesion information of the lesion area. Of course, it is also possible to analyze the dynamic changes of the lesion area within a continuous period of time based on multiple consecutive image data to obtain the lesion information of the lesion area. For example, the image data can be filtered to remove noise in the image data; the filtered image data can be smoothed to reduce the details and noise of the edge contours; the contour edges of the lesion area in the smoothed image data can be extracted; The information within the contour edges in multiple consecutive image data can be compared to obtain the lesion information. For example, if the area of blood in the lesion area expands, it can indicate that the amount of bleeding is increasing. As another example, if the range of blood within the lesion area remains unchanged, it can indicate coagulation. For another example, machine learning can also be used to determine the lesion information of the lesion area. For example, multiple consecutive image data can be input into the image analysis model to analyze the dynamic changes of the lesion area within a continuous period of time to obtain the lesion information.

In the above process of determining the lesion information, the contour edges can be extracted through the morphological edge extraction algorithm. The morphological edge extraction algorithm may include a morphological gradient operator. For example, the morphological gradient operator may include IGrad[A]=c1IGrad1[A]+c2IGrad2[A]. In the formula, c1 and c2 are weight coefficients, and IGrad1 and IGrad2 are the morphological gradients corresponding to the two structural elements. In addition, in the above-mentioned process of determining lesion information, the image analysis model may include a neural network model, etc. Specifically, the image training set can be obtained. The image training set includes one or more image data. Therefore, the image analysis model can be obtained through training with the image training set.

As mentioned before, the status data may include the instrument type of the surgical instrument, the posture data of the surgical robot, the force information on the end of the surgical instrument, the foot control signal of the surgical robot, the working mode data of the surgical robot, etc. Then, at least one operation type that the surgical instrument can perform can be matched according to the instrument type; at least one operation type can be selected according to the posture data; and the operation type of the surgical robot can be determined according to the selected operation type and image data. Surgical procedures. In practical applications, different types of surgical instruments have different functions. Surgical instruments with different functions can perform different types of surgical operations. The functions may include clamping, cutting, etc. The surgical operations may include suturing, clamping, electrocution, electrocoagulation, etc. Sets of operation types can be provided for this purpose. The set of operation types includes at least one subset. Each subset corresponds to an instrument type and includes at least one type of operation that a surgical instrument of that type can perform. The operation type is that of a surgical operation. Each operation type corresponds to pose data. Therefore, a target subset can be selected from the operation type set according to the instrument type of the surgical instrument; one or more operation types can be selected from the target subset according to the posture data.

Specifically, it can be determined whether the surgical instrument is in a clamping state; if it is in a clamping state, the clamping object and the surgical object at the end of the surgical instrument can be determined based on the image data; the movement trajectory of the end of the surgical instrument can be determined based on the pose data. ; The specific surgical procedure of the surgical robot can be determined based on the selected operation type, combined with the clamping object, surgical object and movement trajectory. The clamping objects may include puncture needles, sutures and other clamping objects. The surgical objects may include organ tissues, such as kidneys, liver, arteries, veins, etc.

The energy state of the end of the surgical instrument can be determined based on the pedal control signal; whether the surgical instrument is in a clamping state can be determined based on the energy state of the end of the surgical instrument. If the energy state indicates that there is energy output at the end of the surgical instrument, it can be determined that the surgical instrument is in the clamping state; if the energy state indicates that there is no energy output at the end of the surgical instrument, it can be determined that the surgical instrument is not in the clamping state. Of course, in some cases, such as a malfunction, although the operator can control the energy state of the end of the surgical instrument through the foot pedal so that there is energy output at the end of the surgical instrument, the surgical instrument may still not be in a clamped state. . For this purpose, the force information at the end of the surgical instrument can also be combined to comprehensively determine whether the surgical instrument is in a clamped state. to improve the accuracy of judgment. Specifically, if the energy state indicates that there is energy output at the end of the surgical instrument, and the end of the surgical instrument is in a stressed state, it can be determined that the surgical instrument is in a clamping state; if the energy state indicates that there is no energy output at the end of the surgical instrument, or the surgical instrument is If the end of the instrument is not under force, it can be determined that the surgical instrument is not in a clamping state.

If it is in a clamping state, the clamping object and surgical object at the end of the surgical instrument can also be identified based on the image data. Clamping objects can be identified through machine learning. For example, image data can be input into the image analysis model to obtain clamped object recognition results. The clamping object identification result indicates whether the end of the surgical instrument holds the clamping object, the specific type of the clamping object, etc. Surgical objects can also be identified through machine learning. For example, image data can be input into an image analysis model to obtain surgical object recognition results. The surgical object recognition result represents the organ tissue targeted by the surgical operation. Of course, other methods can also be used to identify the clamping object and the surgical object at the end of the surgical instrument.

The movement trajectory of the end of the surgical instrument can be determined based on the position data of the surgical instrument. For example, the position data of the end of the surgical instrument can be fitted to obtain the movement trajectory of the end of the surgical instrument. Different surgical operations can correspond to different clamping objects and movement trajectories. For example, when a suture and organ tissue is detected at the end of a surgical instrument through image data, and the movement trajectory of the end of the surgical instrument matches the movement trajectory of the suturing operation, it can be determined that a suturing operation is currently being performed. To this end, the selected operation type can be confirmed based on the movement trajectory of the end of the surgical instrument and the clamping object to determine whether the selected operation type matches the movement trajectory of the end of the surgical instrument and the clamping object. The confirmation may include filtering out operation types matching the movement trajectory and the clamped object from the selected operation types. Alternatively, the confirmation may also include determining whether the selected operation type is correct.

According to the confirmed operation type and the surgical object at the end of the surgical instrument, the surgical procedure can be determined.

An example of a scenario for determining the type of abdominal organ suturing is introduced below. Specifically, the instrument type of the surgical instrument on the slave end control arm can be obtained. It can be judged whether the surgical instrument has a clamping function. If it has a clamping function, it can be judged whether the surgical instrument is in a clamped state. For example, it can be determined whether the surgical instrument is in a clamping state by detecting the Hall value of the Hall sensor. If it is in the clamping state, it can be determined based on the image data whether the end of the surgical instrument is clamped with a puncture needle, suture, or other linear clamping object. It can continuously monitor the posture data of joints on surgical instruments. A linear relationship can be fitted to the posture data of the joints on the surgical instrument. The control behavior of surgical instruments can be judged based on the fitting results. Changes in length of sutures or threads can be analyzed based on image data. The organ targeted by the operation of the surgical instrument can be determined based on the image data. Based on control behavior, length changes, and organ synthesis, it can be determined that a certain abdominal organ suture is currently being performed. It can be judged whether the surgical instrument with shearing power has cut the suture. It can be determined based on the image data whether the suture has separated from the organ tissue. Whether the current stitching operation ends can be judged based on the shearing judgment result and the breakaway judgment result.

The text information corresponding to the voice information can be obtained; the semantics of the text information can be refined based on the surgical procedure and lesion information, and the first semantic prompt information can be obtained. Specifically, the text information, surgical procedure and lesion information can be input into the first semantic extraction model to obtain the first semantic prompt information. The first semantic extraction model is suitable for master-slave control mode and is used to extract semantics based on surgical procedures and lesion information. The first semantic extraction model includes but is not limited to hidden Markov model, recurrent neural network model, Gaussian mixture model, etc. Through refining, the semantic information related to the surgical procedure and lesion information can be extracted from the speech information, and the semantic information irrelevant to the surgical procedure and lesion information can be eliminated; in addition, the extracted semantic information can also be optimized based on the surgical procedure and lesion information. . The optimization includes but is not limited to perfecting and supplementing the context of the extracted semantic information so that the extracted semantic information is easy to understand and avoids ambiguity. For example, the background of the extracted semantic information, the reasons for the extraction of semantic information, etc. are improved and supplemented. The first semantic prompt information can accurately and concisely express the information delivered by the surgeon.

For example, the text information may include multiple keyword information. Then the refining the semantics of the text information may include: extracting keyword information related to the surgical procedure and lesion information from the plurality of keyword information; completing and supplementing the semantics expressed by the extracted keyword information to obtain the third 1. Semantic prompt information. Specifically, for example, the semantic background, semantic reasons, etc. can be improved and supplemented.

For example, the surgeon's voice message may include "Start the resection now, please pay attention to the condition of the surrounding tissue." The operator's voice information can be converted into text; keyword information can be extracted from the converted text. The keyword information includes "resection", "note", and "surrounding tissue". The surgical procedure may include liver resection. The lesion information may include incomplete liver resection. Then the keyword information, surgical procedure and lesion information can be input into the first semantic extraction model to obtain the first semantic prompt information "Remove the liver and pay attention to the falling of surrounding tissue".

For example, the surgeon's voice message may include "Please pay attention to the bleeding situation." The operator's voice information can be converted into text; keyword information can be extracted from the converted text. The keyword information may include "attention" and "bleeding". The surgical procedure may include liver resection. The lesion information may include completion of liver resection. Then the keyword information, surgical procedure and lesion information can be input into the first semantic extraction model to obtain the first semantic prompt information. The first semantic prompt information may include "Liver resection is completed, please pay attention to bleeding conditions".

In some embodiments, the working mode of the surgical robot can be identified based on the working mode data; if it is in the adjustment mode, the position status of the surgical instrument can be determined based on the posture data and image data; the surgical instrument can be matched to perform execution according to the instrument type. At least one operation type; the semantics of the voice information can be refined according to the matching operation type and the position state of the surgical instrument to obtain the first semantic prompt information.

The position status is used to represent the position status of the surgical instrument relative to the lesion area, such as whether the surgical instrument moves to the lesion area, the moving direction of the surgical instrument when it needs to move to the lesion area, whether the surgical instrument appears in the field of view of the medical imaging equipment, Whether the surgical instruments are in contact with the organs and tissues in the lesion area, etc. As mentioned before, the pose data may include position data of the surgical instrument. Then the moving direction of the surgical instrument in the patient's body can be determined based on the position data of the surgical instrument; the position status of the surgical instrument can be determined based on the moving direction of the surgical instrument and combined with the image data. For example, the current status of the surgical instrument in the lesion area can be determined based on the image data; the position status of the surgical instrument can be determined based on the movement direction of the surgical instrument and the current status of the surgical instrument in the lesion area. Among them, the current status of the surgical instrument in the lesion area includes: whether the surgical instrument appears in the field of view of the medical imaging equipment, whether the surgical instrument is in contact with the organs and tissues in the lesion area, etc. Specifically, machine learning can be used to determine the current status of surgical instruments in the lesion area based on image data.

Sets of operation types can be provided. The set of operation types includes at least one subset. Each subset corresponds to an instrument type and includes at least one type of operation that a surgical instrument of that type can perform. Therefore, the corresponding target subset can be selected from the operation type set according to the instrument type of the surgical instrument; the operation types in the target subset can be used as the operation types that the surgical instrument can perform.

The text information of the voice information can be obtained; the semantics of the text information can be refined according to the matching operation type and the position status of the surgical instrument, and the first semantic prompt information can be obtained. Specifically, the text information, the matching operation type, and the position status of the surgical instrument can be input into the second semantic extraction model to obtain the first semantic prompt information. The second semantic refining model is suitable for adjusting the mode and is used for refining semantics according to operation type and location status. The second semantic extraction model includes but is not limited to hidden Markov model, recurrent neural network model, Gaussian mixture model, etc. Through refining, the semantic information related to the operation type and position status can be extracted from the speech information, and the semantic information irrelevant to the operation type and position status can be eliminated; in addition, the extracted semantic information can also be optimized according to the operation type and position status. The optimization includes but is not limited to perfecting and supplementing the context of the extracted semantic information so that the extracted semantic information is easy to understand and avoids ambiguity.

In some embodiments, the working mode of the surgical robot can be identified based on the working mode data; if it is in the fault mode, the fault detailed data of the surgical robot can be determined based on the status data and image data; the voice information can be refined based on the fault detailed data. Semantics, get the first semantic prompt information. The fault detail data may include fault location, fault type, fault specific manifestation, fault impact, etc. The fault types may include motor drive failure, robotic arm failure, communication failure, surgical instrument clamping failure, etc.

The surgical robot may have a fault detection component. When the fault detection component detects a fault, the surgical robot enters a fault mode. After entering the fault mode, the detailed fault data of the surgical robot can be determined based on the status data and image data. For example, the first sub-fault data can be extracted from the status data; the second sub-fault data can be extracted from the image data; the first sub-fault data and the second sub-fault data can be combined to obtain the final fault detail data. Specifically, for example, the status data may include pose data. If the position data and/or attitude data in the pose data exceed a preset threshold, the position data and/or the attitude data can be used as the first sub-fault data. The current state of the surgical instrument can be identified from the image data as second sub-fault data. The current state may include a clamping object at the end of the surgical instrument, a surgical object, etc. Of course, other methods can also be used to determine the detailed fault data of the surgical robot based on the status data and image data. For example, status data and image data can be input into a fault analysis model to obtain the fault detailed data.

The text information of the voice information can be obtained; the semantics of the text information can be refined based on the fault details data to obtain the first semantic prompt information. Specifically, the text information and fault details data can be input into the third semantic extraction model to obtain the first semantic prompt information. The third semantic refining model is suitable for fault modes and is used to refine semantics based on fault detailed data. The third semantic extraction model includes but is not limited to hidden Markov model, recurrent neural network model, Gaussian mixture model, etc. Through refining, the semantic information related to the fault details data can be extracted from the voice information, and the semantic information irrelevant to the fault details data can be eliminated; in addition, the extracted semantic information can also be optimized based on the fault details data. The optimization includes but is not limited to perfecting and supplementing the context of the extracted semantic information so that the extracted semantic information is easy to understand and avoids ambiguity.

In some embodiments, see Figure 6. The working mode of the surgical robot can be identified. If the surgical robot is in the master-slave control mode, the surgical procedure of the surgical robot can be determined based on the status data and image data; the lesion information of the lesion area can be determined based on the image data; the voice information can be refined based on the surgical procedure and lesion information. Semantics, get the first semantic prompt information. If the surgical robot is in the adjustment mode, the position status of the surgical instrument can be determined based on the posture data and image data; at least one operation type that the surgical instrument can perform can be matched based on the instrument type; and at least one operation type that the surgical instrument can be performed can be matched based on the matching operation type and surgical instrument. The positional state refines the semantics of the speech information and obtains the first semantic prompt information. If the surgical robot is in a fault mode, the fault detailed data of the surgical robot can be determined based on the status data and image data; the semantics of the voice information can be refined based on the fault detailed data to obtain the first semantic prompt information. In this way, different data can be used to refine the semantics of speech information according to the working mode of the surgical robot. Make the extracted first semantic prompt information more accurate.

Step 14: Send the first semantic prompt information to present the first semantic prompt information to the surgeon's auxiliary medical staff.

In some embodiments, the surgical robot may send the first semantic prompt information to the second medical device 2 . The second medical device 2 can receive and present the first semantic prompt information to the auxiliary medical staff. The presentation includes presentation through display, presentation through voice playback, etc. In this way, the needs of the surgeon can be communicated in a timely manner, allowing the auxiliary medical staff to understand the needs of the surgeon in a timely manner, prepare in advance, and effectively help the surgeon complete the surgical operation. Avoid the adverse effects of delayed treatment of patient lesions due to delayed information exchange.

In some embodiments, language type analysis can also be performed on the first semantic prompt information; based on the analysis results, it can be determined whether to translate the first semantic prompt information; if so, the first semantic prompt information can be translated to obtain the translation result ;Can send the first semantic prompt information and its translation result. Specifically, the first semantic prompt information and its translation result may be sent to the second medical device 2 . The second medical device 2 can simultaneously present the first semantic prompt information and its translation result to the auxiliary medical staff. The synchronous presentation may include synchronously displaying the first semantic prompt information and its translation result, etc. Through translation, the first semantic prompt information can be translated into a designated language to avoid errors in information transmission due to language barriers.

The analysis results can be obtained by analyzing whether the first semantic prompt information contains vocabulary of the set language, the number of occurrences of the set language vocabulary in the first semantic prompt information, the frequency of occurrence of the set language vocabulary in the first semantic prompt information, etc. Among them, the number of occurrences of the set language vocabulary and the total vocabulary of the first semantic prompt information can be counted; the frequency of appearance of the set language vocabulary can be obtained by dividing the number of occurrences of the set language vocabulary by the total vocabulary. The setting language may be different from the designated language. The specified language may include Chinese, English, etc. The setting language may include English, Japanese, Korean, etc.

It can be judged whether the analysis results meet the translation conditions. If satisfied, the first semantic prompt information can be translated.

For example, the translation condition may include that the first semantic prompt information includes vocabulary in the set language. When the first semantic prompt information includes vocabulary in the set language, the first semantic prompt information can be translated to translate the vocabulary in the set language in the first semantic prompt information into vocabulary in another designated language. For another example, the translation condition may include that the number of occurrences of the set language words in the first semantic prompt information is greater than or equal to the first set threshold. Then the number of occurrences of the set language words in the first semantic prompt information can be counted; if the number of occurrences is greater than or equal to the first set threshold, the first semantic prompt information can be translated to change the set language in the first semantic prompt information. of words translated into words in another specified language. For another example, the translation condition may include that the occurrence frequency of the set language vocabulary in the first semantic prompt information is greater than or equal to the second set threshold. Then the occurrence frequency of the set language words in the first semantic prompt information can be counted; if the occurrence frequency is greater than or equal to the second set threshold, the first semantic prompt information can be translated to change the set language in the first semantic prompt information. of words translated into words in another specified language.

In some embodiments, the importance of the first semantic prompt information can be analyzed to obtain importance data; the first display mode of the first semantic prompt information can be determined based on the importance data; the first semantic prompt information and the first semantic prompt information can be sent. Display method. Specifically, the first semantic prompt information and the first display mode may be sent to the second medical device 2 . The second medical device 2 may use a first display mode to present the first semantic prompt information to the auxiliary medical staff. This allows the auxiliary medical staff to intuitively understand the importance of the surgeon's needs.

The first semantic prompt information can be segmented to obtain one or more segmented words; the importance level of the one or more segmented words can be obtained; the first semantic prompt information can be determined based on the importance level of the one or more segmented words. level of importance.

Glossaries are available. The vocabulary set may include one or more vocabulary words. Each word can have a corresponding level of importance. The importance level is used to express the degree of importance. The importance level may be selected from the first level, the second level, the third level, and the fourth level. The importance of the first level, the second level, the third level, and the fourth level decreases in order. Of course, you can also set more or fewer importance levels for selection as needed. The one or more word segments can be matched in the word set to obtain the importance level of the one or more word segments. It should be noted that it is possible that some of the one or more word segments fail to match in the word set, and then the importance level of the failed matching words may be set as a default level. The default level may include the least important level, such as the fourth level.

The number of word segments corresponding to each importance level can be counted; the importance level with the largest number of corresponding word segments can be used as the importance level of the first semantic prompt information. Of course, other methods can also be used to determine the importance level of the first semantic prompt information.

A first display mode set may be provided. The first display mode set may include one or more display modes. Each display mode can correspond to an importance level. The display mode may include a text display mode. The text display method may include the color of the text, the font of the text, the font size of the text, the font shape of the text, whether the text is underlined, etc. The importance level of the first semantic prompt information can be matched in the first display mode set; the matched display mode can be used as the first display mode.

In some embodiments, see Figure 7. The semantic prompt image can be matched according to the first semantic prompt information; if the matching is successful, the first semantic prompt information and the semantic prompt image can be sent. Specifically, the first semantic prompt information and the semantic prompt image may be sent to the second medical device 2 . The second medical device 2 presents the first semantic prompt information and the semantic prompt image to the auxiliary medical staff. In this way, in addition to displaying the first semantic prompt information, the semantics contained in the first semantic prompt information can also be displayed graphically to assist medical staff in intuitively understanding the surgeon's needs.

Image sets are available. The image set may include one or more semantically cueing images. Each semantic cue image can be associated with a semantic label. The semantic tag is used to represent the semantics of the semantic prompt image. The first semantic prompt information may be matched in the image set to select a semantic prompt image whose corresponding semantic label matches the first semantic prompt information. If the matching is successful, it means that the first semantic prompt information can be graphically displayed to assist medical staff in intuitively understanding the surgeon's needs. Wherein, matching the semantic label with the first semantic prompt information may include: the semantics of the semantic label are consistent with the semantics of the first semantic prompt information. For example, the semantic tag includes one or more words. If the first semantic prompt information contains any one or more of the one or more words, it can be considered that the semantic tag matches the first semantic prompt information.

In some embodiments, the importance of the first semantic prompt information can be analyzed to obtain importance data; the first display mode of the first semantic prompt information can be determined based on the importance data; the semantic prompt can be matched according to the first semantic prompt information image; if the match is successful, the second display mode of the semantic prompt image can be determined based on the importance data; the first semantic prompt information, the first display mode, the semantic prompt image and the second display mode can be sent. Specifically, the first semantic prompt information, the first display mode, the semantic prompt image and the second display mode may be sent to the second medical device 2 . The second medical device 2 can use the first display mode to present the first semantic prompt information to the auxiliary medical staff, and can use the second display mode to present the semantic prompt image to the auxiliary medical staff. In this way, the auxiliary medical staff can intuitively understand the importance of the surgeon's needs.

A second set of display modes may be provided. The second display mode set may include one or more display modes. Each display mode can correspond to an importance level. The display mode may include an image display mode. The image display method may include flashing at a certain frequency, displaying at a certain transparency, etc. The importance level of the first semantic prompt information may be matched in the second display mode set, and the matched display mode may be used as the second display mode of the semantic prompt image.

In some embodiments, the response voice information of the surgeon's auxiliary medical staff to the first semantic prompt information can also be collected; the semantics of the response voice information can be refined based on the image data and the first semantic prompt information to obtain the second semantic prompt information; The second semantic prompt information can be fed back to present the second semantic prompt information to the operator.

The voice information processing method of the embodiment of this specification can collect the voice information of the surgeon during the operation; can obtain the status data of the surgical robot and the image data of the lesion area; can refine the semantics of the voice information based on the status data and image data, and obtain First semantic prompt information; the first semantic prompt information may be sent to present the first semantic prompt information to the surgeon's paramedical staff. In this way, the intraoperative information such as the status data of the surgical robot and the image data of the lesion area can be refined into the semantics of the voice information, thereby presenting semantic prompt information to the auxiliary medical staff. This avoids the untimely and inaccurate transmission of surgical information due to the presence of noise at the surgical site, thereby reducing surgical risks.

See Figure 8. The embodiments of this specification provide a voice information processing method accordingly. The voice information processing method can be applied to the aforementioned second medical device 2 . The voice information processing method may include the following steps.

Step 21: Receive the first semantic prompt information.

In some embodiments, the first semantic prompt information is obtained by refining the surgeon's voice information during the operation. For details, reference may be made to the previous embodiments, which will not be described again here.

Step 22: Collect the response voice information of the surgeon's auxiliary medical staff to the first semantic prompt information.

In some embodiments, after obtaining the first semantic prompt information, the auxiliary medical staff can perform auxiliary work and provide response feedback to the surgeon through voice. Therefore, the response voice information of the auxiliary medical staff can be collected through the sound collection component (such as a microphone).

Step 23: Obtain image data of the lesion area.

In some embodiments, the first medical device 1 can obtain image data of the lesion area through medical imaging equipment such as an endoscope or CT; the image data can be sent to the second medical device 2 . The second medical device 2 can receive image data; it can display the image data.

Step 24: Extract the semantics of the response voice information based on the image data and the first semantic prompt information to obtain the second semantic prompt information.

In some embodiments, the text information corresponding to the response voice information can be obtained; the semantics of the text information can be refined based on the image data and the first semantic prompt information to obtain the second semantic prompt information. Specifically, the response voice information can be converted into text. The converted text can be used as the text information. In this way, the converted text can be semantically refined based on the image data and the first semantic prompt information. Alternatively, one or more keyword information can also be extracted from the converted text; the keyword information can be used as the text information. In this way, the keyword information can be semantically refined based on the image data and the first semantic prompt information.

In some embodiments, semantic information related to the lesion information and the first semantic prompt information can be extracted from the response voice information through refining, and semantic information irrelevant to the lesion information and the first semantic prompt information can be eliminated; the extracted Semantic information is optimized.

The lesion information of the lesion area can be determined based on the image data; the semantics of the response voice information can be refined based on the lesion information and the first semantic prompt information to obtain the second semantic prompt information. The process of determining the lesion information may refer to the foregoing embodiments.

The semantics of the text information can be refined through the semantic refining model to obtain the second semantic prompt information. Specifically, the text information corresponding to the lesion information, the first semantic prompt information and the response voice information can be input into the semantic extraction model to obtain the second semantic prompt information. The semantic extraction models include but are not limited to hidden Markov models, recurrent neural network models, Gaussian mixture models, etc.

Step 25: Feed back the second semantic prompt information.

In some embodiments, the second semantic prompt information may be sent to the first medical device 1 . The first medical device 1 can receive the second semantic prompt information; and can present the second semantic prompt information to the surgeon. This enables the paramedic's response to be conveyed.

The voice information processing method of the embodiment of the present specification can receive the first semantic prompt information; can collect the response voice information of the auxiliary medical staff to the first semantic prompt information; can obtain the image data of the lesion area; and can obtain the image data of the lesion area according to the image data and the first semantic prompt information. The prompt information refines the semantics of the response voice information to obtain the second semantic prompt information; the second semantic prompt information can be fed back. In this way, intraoperative information such as image data and the extracted first semantic prompt information can be combined to refine the semantics of the response voice information, thereby presenting the second semantic prompt information to the surgeon. This avoids the untimely and inaccurate transmission of surgical information due to the presence of noise at the surgical site, thereby reducing surgical risks.

See Figure 9. An embodiment of this specification also provides a voice information processing device, including the following units.

The collection unit 31 is used to collect the operator's voice information during the operation;

The acquisition unit 32 is used to acquire the status data of the surgical robot and the image data of the lesion area;

The refining unit 33 is used to refine the semantics of the voice information according to the status data and image data to obtain the first semantic prompt information;

The sending unit 34 is configured to send the first semantic prompt information to present the first semantic prompt information to the surgeon's auxiliary medical staff.

See Figure 10. An embodiment of this specification also provides another voice information processing device, including the following units.

The receiving unit 41 is used to receive the first semantic prompt information;

The collection unit 42 is used to collect the response voice information of the surgeon's auxiliary medical staff to the first semantic prompt information;

The acquisition unit 43 is used to acquire image data of the lesion area;

The refining unit 44 is used to refine the semantics of the response voice information according to the image data and the first semantic prompt information, and obtain the second semantic prompt information;

The feedback unit 45 is used to feed back the second semantic prompt information to present the second semantic prompt information to the operator.

An embodiment of this specification also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above voice information processing method.

Embodiments of this specification also provide a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above voice information processing method is implemented.

Embodiments of this specification also provide a computer program product. The computer program product includes a computer program. When the computer program is executed by a processor, the above voice information processing method is implemented.

Those skilled in the art can understand that this description may be provided as a method, system, or computer program product. The description may thus take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present description may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk memory, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The specification is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the specification. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. The computer may be a personal computer, a laptop, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet, a wearable device, or any of these devices any combination of devices.

Each functional unit in the embodiment of this specification may be integrated into one processing unit, or each functional unit may exist physically alone, or two or more functional units may be integrated into one processing unit.

Those skilled in the art can understand that the description of each embodiment in this specification has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. In addition, it can be understood that after reading this specification document, those skilled in the art can think of any combination of some or all of the embodiments listed in this specification without creative efforts, and these combinations are also within the scope of disclosure and protection of this specification.

Although this specification has been described by way of embodiments, those of ordinary skill in the art will appreciate that the above embodiments are only intended to aid understanding of the core ideas of this specification. Those skilled in the art will appreciate that there are many variations and variations of this description. It is intended that the appended claims cover these modifications and changes without departing from the spirit of this specification.

Claims (10)

1. A voice information processing device, characterized in that it includes: The collection unit is used to collect the operator's voice information during the operation; An acquisition unit, used to acquire status data of the surgical robot and image data of the lesion area; the status data includes the instrument type of the surgical instrument and/or the posture data of the surgical robot; The refining unit is used to refine the semantics of the speech information based on the status data and image data to obtain the first semantic prompt information; The sending unit is used for sending the first semantic prompt information to present the first semantic prompt information to the surgeon's auxiliary medical staff. 2. A voice information processing method, characterized by including: Collect the surgeon’s voice information during the operation; Obtain the status data of the surgical robot and the image data of the lesion area; the status data includes the instrument type of the surgical instrument and/or the posture data of the surgical robot; Extract the semantics of the speech information based on the status data and image data to obtain the first semantic prompt information; The first semantic prompt information is sent to present the first semantic prompt information to the surgeon's paramedical staff. 3. The method according to claim 2, characterized in that the step of refining semantics includes: If the surgical robot is in the master-slave control mode, the surgical procedure of the surgical robot is determined based on the status data and image data; Determine the lesion information of the lesion area based on the image data; The semantics of the speech information are refined based on the surgical procedure and lesion information, and the first semantic prompt information is obtained. 4. The method according to claim 3, characterized in that the step of determining the surgical procedure includes: According to the instrument type, match at least one operation type that the surgical instrument can perform; Selecting from the at least one operation type according to the pose data; According to the selected operation type and image data, the surgical method of the surgical robot is determined. 5. The method according to claim 3 or 4, characterized in that the step of determining the surgical procedure includes: Determine whether the surgical instrument is in a clamped state; If it is in a clamping state, determine the clamping object and surgical object at the end of the surgical instrument based on the image data; Based on the pose data, determine the movement trajectory of the end of the surgical instrument; According to the selected operation type, combined with the clamping object, surgical object and movement trajectory, the surgical procedure of the surgical robot is determined. 6. The method according to claim 2, characterized in that the step of refining semantics includes: If the surgical robot is in adjustment mode, determine the position status of the surgical instrument based on the posture data and image data; According to the instrument type, match at least one operation type that the surgical instrument can perform; The semantics of the speech information are extracted according to the matching operation type and the position status of the surgical instrument, and the first semantic prompt information is obtained. 7. The method according to claim 2, characterized in that the step of refining semantics includes: If the surgical robot is in fault mode, determine the detailed fault data of the surgical robot based on the status data and image data; The semantics of the voice information are refined based on the fault details data to obtain the first semantic prompt information. 8. The method according to claim 2, characterized in that, the method further comprises: Perform language type analysis on the first semantic prompt information; According to the analysis results, determine whether to translate the first semantic prompt information; If so, translate the first semantic prompt information to obtain the translation result; The translation result is sent to present the first semantic prompt information and the corresponding translation result to the surgeon's auxiliary medical staff. 9. The method according to claim 2, characterized in that, the method further comprises: Collect the response voice information of the surgeon's auxiliary medical staff to the first semantic prompt information; Extract the semantics of the response voice information based on the image data and the first semantic prompt information to obtain the second semantic prompt information; The second semantic prompt information is fed back to present the second semantic prompt information to the operator. 10. A computer storage medium, characterized in that the computer storage medium stores computer program instructions, and when the computer program instructions are executed, the steps of the method according to any one of claims 2 to 9 are implemented.