CN116612411A - Method and system for identifying blood vessels, lymph nodes and nerves in operation - Google Patents

Abstract

The invention discloses a method and system for identifying blood vessels, lymph nodes, and nerves during surgery, and relates to the field of computer technology. The method includes S1 acquiring surgical videos, marking and establishing a blood vessel sample database, a lymphatic sample database, and a nerve sample database; S2 constructing an initial identification Model; S3 trains the initial recognition model to obtain the blood vessel recognition model, lymphatic recognition model and nerve recognition model; S4 collects surgical videos, recognizes and outlines the outlines of blood vessels, lymphatics and nerves; S5 visually displays the surgical videos and contours to the surgeon; the system Including acquisition module, central processing unit and display module; AI computer model is used to identify blood vessels, lymph nodes and nerve structures in the field of view of laparoscopic surgery in real time. The guideline guides the operator to identify and remove blood vessels and lymph nodes reasonably, and avoids blood vessels and lymph nodes that do not need to be removed, so as to assist the operation to proceed smoothly.

Description

Method and system for identifying blood vessels, lymph nodes and nerves in surgery

technical field

The invention relates to the technical field of computers, in particular to a method and system for identifying blood vessels, lymph nodes and nerves in operations.

Background technique

During the operation, due to the complexity of the anatomical structure and the subjective factors of the operator, the resection and injury of blood vessels, lymph nodes and nerves are relatively common. Improper injury or wrong resection is very likely to lead to postoperative complications, which is not conducive to the recovery of patients and affects the quality of life of patients.

Contents of the invention

The object of the present invention is to design a method and system for identifying blood vessels, lymph nodes and nerves in surgery in order to solve the above problems.

The present invention achieves the above object through the following technical solutions:

Methods for identifying blood vessels, lymph nodes, and nerves during surgery, including:

S1. Acquire surgical videos of various surgical procedures, mark the surgical stage and blood vessels, lymph and nerves in the field of view, and establish a blood vessel sample database, lymph sample database and nerve sample database;

S2. Construct three initial recognition models, each initial recognition model includes a FPN neural network and a multi-feature extraction neural network, the FPN neural network is a bottom-up, top-down and horizontally connected network structure, and a multi-feature extraction neural network Including four layers of pooling attention layer, the four layers of pooling attention layer are connected to form a bottom-up downsampling structure, the number of bottom-up layers of the FPN neural network is four layers, one layer of pooling attention layer and one layer The bottom-up downsampling layer corresponds to the output of the previous pooling attention layer and the previous downsampling layer as the input of the next downsampling layer;

S3. The blood vessel sample database imports the initial recognition model and trains and optimizes it to obtain the blood vessel recognition model; the lymph sample database imports the initial recognition model and trains and optimizes it to obtain the lymphatic recognition model; the neural sample database imports the initial recognition model and Perform training optimization to obtain a neural recognition model;

S4. Collect surgical video in real time, and import the blood vessel recognition model, lymphatic recognition model and nerve recognition model respectively, obtain the blood vessel recognition result, lymphatic recognition result and nerve recognition result, and outline the blood vessel recognition result, lymphatic recognition result and nerve recognition result contour;

S5. Visually display the operation video and outline to the operator.

Identification systems for blood vessels, lymph nodes and nerves during surgery, including:

Acquisition module for real-time acquisition of surgical video;

Central processing unit; the central processing unit is used to analyze the surgical video to identify blood vessels, lymph and nerves, and mark the obtained blood vessel recognition results, lymphatic recognition results and nerve recognition results in the surgical video;

A display module; the display module is used to display the marked operation video to the operator.

The beneficial effect of the present invention is that the artificial intelligence computer model is used to identify the blood vessels, lymph nodes and nerve structures in the field of view of laparoscopic surgery in real time, and at the same time combined with the visual display function, it provides guidance for the operator to remove and avoid the corresponding blood vessels, lymph nodes and nerves, In this way, the operator can be guided to identify and remove structures such as blood vessels and lymph nodes reasonably, as well as avoid the blood vessels and lymph nodes that do not need to be removed, and assist the operation to proceed smoothly.

Description of drawings

Fig. 1 is a schematic flow chart of the present invention's identification method for blood vessels, lymph nodes and nerves in surgery;

Fig. 2 is a structural representation of the recognition model in the present invention;

Fig. 3 is a structural schematic diagram of a multi-feature extraction neural network in the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, embodiments of the present invention. The components of the embodiments of the invention generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations.

Accordingly, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

It should be noted that like numerals and letters denote similar items in the following figures, therefore, once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

In the description of the present invention, it should be understood that the orientations or positional relationships indicated by the terms "upper", "lower", "inner", "outer", "left", "right" etc. are based on those shown in the accompanying drawings. Orientation or positional relationship, or the orientation or positional relationship that is usually placed when the product of the invention is used, or the orientation or positional relationship that is commonly understood by those skilled in the art, is only for the convenience of describing the present invention and simplifying the description, rather than indicating or It should not be construed as limiting the invention by implying that a referenced device or element must have a particular orientation, be constructed, and operate in a particular orientation.

In addition, the terms "first", "second", etc. are only used for distinguishing descriptions, and should not be construed as indicating or implying relative importance.

In the description of the present invention, it should also be noted that, unless otherwise specified and limited, terms such as "setting" and "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection or a Detachable connection, or integral connection; it can be mechanical connection or electrical connection; it can be direct connection or indirect connection through an intermediary, and it can be the internal communication of two components. Those of ordinary skill in the art can understand the specific meanings of the above terms in the present invention according to specific situations.

The specific implementation manners of the present invention will be described in detail below in conjunction with the accompanying drawings.

As shown in Figure 1, Figure 2, and Figure 3, the identification methods used for blood vessels, lymph nodes and nerves during surgery include:

S1. Acquire surgical videos of various surgical procedures, mark the surgical stage and blood vessels, lymph and nerves in the field of view, and establish a blood vessel sample database, lymph sample database and nerve sample database;

S2. Construct three initial recognition models, each initial recognition model includes a FPN neural network and a multi-feature extraction neural network, the FPN neural network is a bottom-up, top-down and horizontally connected network structure, and a multi-feature extraction neural network Including four layers of pooling attention layer, the four layers of pooling attention layer are connected to form a bottom-up downsampling structure, the number of bottom-up layers of the FPN neural network is four layers, one layer of pooling attention layer and one layer The bottom-up downsampling layer corresponds to the output of the previous pooling attention layer and the previous downsampling layer as the input of the next downsampling layer;

S3. The blood vessel sample database imports the initial recognition model and trains and optimizes it to obtain the blood vessel recognition model; the lymph sample database imports the initial recognition model and trains and optimizes it to obtain the lymphatic recognition model; the neural sample database imports the initial recognition model and Perform training optimization to obtain a neural recognition model;

S4. Collect surgical video in real time, and import the blood vessel recognition model, lymphatic recognition model and nerve recognition model respectively, obtain the blood vessel recognition result, lymphatic recognition result and nerve recognition result, and outline the blood vessel recognition result, lymphatic recognition result and nerve recognition result contour;

S5. Visually display the operation video and outline to the operator.

Each pooling attention layer is equal to the pooling layer and the first self-attention layer. The output of the pooling layer is used as the input of the first self-attention layer. The first self-attention layer down-samples the surgical image through local aggregation and Computes global self-attention.

The multi-feature extraction neural network also includes a four-layer second self-attention layer. The second self-attention layer divides the input into non-overlapping windows and calculates the local self-attention within each window. The output of the second self-attention layer As the input of the FPN neural network.

The multi-feature extraction neural network also includes three layers of mixed window layer. One layer of mixed window layer is used to calculate the local attention in a window. Except for the last block of the mixed window layer, the output of the mixed window layer is used as the output of the FPN neural network. enter.

The pooling attention layer performs a linear projection on any input sequence to obtain three tensors Q, K, and V, expressed as ; Perform pooling processing and attention calculation on the three tensors in turn, and incorporate the relative position information into the attention calculation. The attention calculation is expressed as , decompose the calculation of the distance between elements and along the space-time axis as , where h and w represent the vertical and horizontal directions, respectively.

Identification systems for blood vessels, lymph nodes and nerves during surgery, including:

Acquisition module for real-time acquisition of surgical video;

Central processing unit; the central processing unit is used to analyze the surgical video to identify blood vessels, lymph and nerves, and mark the obtained blood vessel recognition results, lymphatic recognition results and nerve recognition results in the surgical video;

A display module; the display module is used to display the marked operation video to the operator.

The working principle of the identification method and system for blood vessels, lymph nodes and nerves in the operation of the present invention is as follows:

Generally, the diameter of arteries and veins does not exceed 3 cm, the diameter of lymph nodes is usually 0.5-2 cm, and the width of nerves does not exceed 1 cm. These three types of targets that need to be identified and detected are all small individuals. Usually, the pixel area in the lens of the cavity mirror does not exceed 32x32 pixels. The information itself is insufficient, and the discriminative information contained is not enough. In addition, there are The imbalance of the data set and the problem that the reference frame is difficult to match.

In order to better solve these problems, the FPN neural network introduces a bottom-up and top-down network structure. By fusing the features of adjacent layers to achieve the purpose of feature enhancement, there are not enough pixels and features. Rich targets have better results. Creates a strong baseline that improves focused attention along 2 axes, injects location information into pooled attention layers using decomposed location distances, and compensates pooling strides in attention calculations with pooled residual connections , combined with FPN neural network for object detection and instance segmentation.

The key idea of the algorithm is to realize the construction of different stages of high-dimensional and low-dimensional visual modeling by expanding the channel width while reducing the resolution, instead of single-scale blocks. Because it is necessary to extract and fuse features of different scales through up and down sampling, this algorithm proposes pooling attention, as shown in Figure 3. For any input sequence, if it is linearly projected, three tensors of Q (query), K (key), and V (value) will be obtained: ,

Q represents the query vector, K represents the vector of the correlation between the queried information and other information, V represents the vector of the queried information, W is the matrix of linear projection, X is the self-attention weight, and P is the pooling operator.

Q, K, and V are then pooled, mainly to shorten the length of the K, V sequence, and then perform attention calculations on the basis of pooling: ,

where K ^T is the transpose of matrix K, and D is the vector dimension of multi-head pooling self-attention processing.

The first self-attention layer of the multi-feature extraction neural network can be pooled at each step, which can greatly reduce the memory cost and calculation amount during Q-K-V calculation, so that the hardware conditions and computing power required by the system for recognition Conditions will be greatly reduced, enabling more medical institutions to operate at low cost.

Because the absolute position encoding can only provide position information but ignore the translation invariance of features. Therefore, if the absolute position changes, the dependency between them will change, although the relative position of the two regions has not changed. In order to solve this problem, the model incorporates the information of the relative position into the self-attention calculation of the pooled attention layer, which only depends on the relative position distance of K: ,in ,

Where i represents the i-th token token in the time dimension, j represents the j-th token token in the space dimension, R is the relative position code, d is the absolute value of the value range of R, and P represents the space-time position of elements i and j .

In order to simplify the operation and reduce the complexity, the algorithm decomposes the calculation of the distance between elements and along the space-time axis into: ,

Among them, h and w represent the vertical and horizontal directions respectively, because this project uses a single-frame surgical picture instead of a surgical video, and t is the time dimension, so the t dimension is 0, that is .

After the above improvements, the structure of the multi-feature extraction neural network can be integrated into the FPN neural network. Structurally, the multi-feature extraction neural network generates multi-scale feature maps in four stages, so it is naturally integrated into the FPN neural network for object detection tasks, which has a top-down pyramid with lateral connections Semantically powerful feature maps of multi-feature extraction neural networks are constructed at all scales, as shown in Figure 2.

In addition to the pooled attention layer, the algorithm also proposes a second self-attention layer to significantly reduce computational and memory complexity. The pooling attention layer is characterized by downsampling it via local aggregation but maintaining the global self-attention computation, while the second self-attention layer maintains the resolution of the tensor but by dividing the input into non-overlapping windows, then Only local self-attention within each window is computed, thus performing self-attention locally. The intrinsic differences of these two approaches motivate whether they can perform complementary object detection tasks. Meanwhile, a hybrid window layer is proposed to add cross-window connections. The hybrid window layer computes the local attention within a window, except for the last blocks of the last three stages, which are all fed into the FPN neural network, so that the map contains global information.

The two attention mechanisms enable the blood vessel recognition model, lymphatic recognition model, and neural recognition model to accurately identify targets at various scales. No matter how the endoscope moves during the operation, as long as its features are exposed under the lens, they can be accurately recognized. It does not require too much calculation. The number of FLOPs per second of the model is only 42.1G, and the computer function Params is only 56M, which enables the system to segment and mark in real time during the operation, helping medical workers quickly find blood vessels. , lymph nodes and nerves, reducing the risk of surgery.

The artificial intelligence computer model is used to identify the blood vessels, lymph nodes and nerve structures in the field of view of laparoscopic surgery in real time. At the same time, combined with the visual display function, it provides the surgeon with guidance for removing and avoiding the corresponding blood vessels, lymph nodes and nerves, so as to guide the surgeon to identify reasonably With the removal of structures such as blood vessels and lymph nodes, as well as the avoidance of blood vessels and lymph nodes that do not need to be removed, the auxiliary surgery went smoothly.

The technical solution of the present invention is not limited to the limitations of the above-mentioned specific embodiments, and any technical deformation made according to the technical solution of the present invention falls within the protection scope of the present invention.

Claims (6)

1. A method for identifying blood vessels, lymph nodes and nerves in surgery, characterized in that it comprises: S1. Acquire surgical videos of various surgical procedures, mark the surgical stage and blood vessels, lymph and nerves in the field of view, and establish a blood vessel sample database, lymph sample database and nerve sample database; S2. Construct three initial recognition models, each initial recognition model includes a FPN neural network and a multi-feature extraction neural network, the FPN neural network is a bottom-up, top-down and horizontally connected network structure, and a multi-feature extraction neural network Including four layers of pooling attention layer, the four layers of pooling attention layer are connected to form a bottom-up downsampling structure, the number of bottom-up layers of the FPN neural network is four layers, one layer of pooling attention layer and one layer The bottom-up downsampling layer corresponds to the output of the previous pooling attention layer and the previous downsampling layer as the input of the next downsampling layer; S3. The blood vessel sample database imports the initial recognition model and trains and optimizes it to obtain the blood vessel recognition model; the lymph sample database imports the initial recognition model and trains and optimizes it to obtain the lymphatic recognition model; the neural sample database imports the initial recognition model and Perform training optimization to obtain a neural recognition model; S4. Collect surgical video in real time, and import the blood vessel recognition model, lymphatic recognition model and nerve recognition model respectively, obtain the blood vessel recognition result, lymphatic recognition result and nerve recognition result, and outline the blood vessel recognition result, lymphatic recognition result and nerve recognition result contour; S5. Visually display the operation video and outline to the operator. 2. the method for identifying blood vessels, lymph nodes and nerves in the operation according to claim 1, characterized in that, every layer of pooling attention layer is equal pooling layer and the first self-attention layer, the output of pooling layer As the input of the first self-attention layer, the first self-attention layer down-samples the surgical images by local aggregation and computes global self-attention. 3. the method for identifying blood vessels, lymph nodes and nerves in the operation according to claim 2, wherein the multi-feature extraction neural network also includes four layers of the second self-attention layer, and the second self-attention layer passes through The input is divided into non-overlapping windows and the local self-attention within each window is calculated, and the output of the second self-attention layer is used as the input of the FPN neural network. 4. The method for identifying blood vessels, lymph nodes and nerves in surgery according to claim 3, wherein the multi-feature extraction neural network also includes three layers of mixed window layers, and one layer of mixed window layers is used to calculate The local attention of , except the last block of the mixed window layer, the output of the mixed window layer is used as the input of the FPN neural network. 5. The method for identifying blood vessels, lymph nodes and nerves in surgery according to claim 2, wherein the pooling attention layer is linearly projected to any input sequence to obtain Q, K, V three tensors, expressed as , where Q represents the query vector, K represents the vector of the correlation between the queried information and other information, V represents the vector of the queried information, W is the matrix of linear projection, X is the self-attention weight, and P is the pooling operator ; Perform pooling processing and attention calculation on the three tensors in turn, and incorporate the relative position information into the attention calculation. The attention calculation is expressed as /> , decompose the calculation of the distance between elements and along the space-time axis as /> , where h and w represent the vertical and horizontal directions, respectively, /> , where i represents the i-th token token in the time dimension, j represents the j-th token token in the space dimension, R is the relative position code, d is the absolute value of the value range of R, and p represents the space-time of elements i and j Position, t is the time dimension, and the t dimension is 0, ie /> . 6. An identification system for blood vessels, lymph nodes and nerves in surgery, characterized in that it comprises: Acquisition module for real-time acquisition of surgical video; Central processing unit; the central processing unit is used to analyze the surgical video to identify blood vessels, lymph and nerves, and mark the obtained blood vessel recognition results, lymphatic recognition results and nerve recognition results in the surgical video; A display module; the display module is used to display the marked operation video to the operator.