WO2025018587A1 - Mixed reality-based ultrasound image output system and method - Google Patents

Abstract

The present invention includes: a data receiving unit for receiving an ultrasound image and a bio-signal of a subject; a first content generation unit for generating first mixed-reality (MR) content by matching the ultrasound image to a specific object; a second content generation unit for generating second MR content by matching the bio-signal to a predetermined space; and an output device that displays MR content generated by the first content generation unit or the second content generation unit, wherein the first content generation unit recognizes a probe shape of an ultrasound diagnosis device through a camera provided in the output device, detects movement of the probe shape, and changes an output position of the first MR content.

Description

Mixed reality based ultrasound image output system and method

The present invention relates to a mixed reality-based ultrasound image output system and method, and more particularly, to a mixed reality-based ultrasound image output system and method that aligns and outputs an ultrasound image on an examination area using a device that displays in an AR or VR manner.

Ultrasound refers to sound waves with a frequency of over 20,000 Hz, which is higher than the human audible range. Ultrasound imaging (ultrasonography) is the imaging of differences in reception created through diffusion, reflection, absorption, and scattering of sound waves at various boundaries such as internal organs, bones, muscle tissue, and blood in the human body.

Ultrasound-guided biopsy (US-guided biopsy) is a test that obtains tissue samples by inserting a thin needle into a mass or local lesion outside the body under ultrasound guidance. Ultrasound-guided biopsy can help the needle accurately pierce the lesion, which can increase accuracy and reduce false negatives. Accordingly, ultrasound-guided biopsy is widely used in clinical practice.

Ultrasound-guided radiofrequency ablation (US-guided RF ablation) is a method of treating tumors by burning them using the heat generated by inserting a needle-shaped electrode into the tumor and then passing an electric current while looking at the ultrasound examination screen. Ultrasound-guided RF ablation has the advantage of being able to quickly and accurately insert the electrode inside the cancerous tumor and treat it through real-time guidance compared to CT (Computerized Tomography) and MRI (Magnetic Resonance Imaging) images.

Ultrasound-guided biopsy can cause high fatigue because the operator must hold the ultrasound probe for a long time and frequently turn the head to check and compare the ultrasound image with the biopsy site. Ultrasound-guided biopsy is difficult to perform because the ultrasound monitor and biopsy site cannot be viewed at the same time, and it requires high expertise and skills, so only experienced specialists can perform it. In addition, in an actual examination environment, the ultrasound equipment is often placed far away due to the patient's bed, numerous pieces of equipment in the operating room, etc., which can hinder ultrasound-guided biopsy.

Ultrasound-guided radiofrequency ablation therapy may take longer to treat as the size of the tumor to be treated gradually decreases with the early detection of cancerous tumors, and it becomes difficult to identify the location of the tumor with ultrasound imaging. Ultrasound-guided radiofrequency ablation therapy requires a high level of concentration during the treatment, making it difficult to timely monitor the patient's physical risk signs such as blood pressure, heart rate, and oxygen saturation.

Therefore, there is a need for a mixed reality-based ultrasound image output system and method to overcome the limitations of the above ultrasound-guided biopsy and ultrasound-guided radiofrequency thermal therapy.

The purpose of the present invention is to enable a practitioner to simultaneously view a biopsy site and ultrasound through a wearable output device, and to superimpose an ultrasound image and a lesion site on the biopsy site.

In addition, the present invention seeks to display a physical danger signal in the direction of the wearer's gaze through an output device that the operator can wear.

In addition, the present invention seeks to synchronize input signals to ensure the reliability of output information.

In order to achieve the above object, the present invention comprises: a data receiving unit which receives an ultrasound image and a bio-signal of a subject; a first content generating unit which generates a first MR (Mixed-reality) content by aligning the ultrasound image with a specific object; a second content generating unit which generates a second MR content by aligning the bio-signal with a specific space; and an output device which displays the MR content generated by the first content generating unit or the second content generating unit; wherein the first content generating unit is characterized in that it recognizes the probe shape of an ultrasound diagnostic device through a camera provided in the output device, and detects movement of the probe shape to change the output position of the first MR content.

Preferably, the method may further include a lesion detection unit that detects a lesion area in the ultrasound image using artificial intelligence learned from an ultrasound image data set in which the lesion area is annotated.

Preferably, the first content generation unit can change the first MR content so that the lesion area detected by the lesion detection unit is distinguished from other areas.

Preferably, the first content generation unit can change the MR content so that the lesion area is not distinguished from other areas when it is confirmed through the ultrasound image that the needle has contacted the lesion area.

Preferably, the method may further include a synchronization unit that synchronizes the ultrasound image, the bio-signal, and the lesion area so that the ultrasound image and the bio-signal measured at the same time can be output and the lesion area detected at the same time can be displayed.

Preferably, the synchronization unit may include a delay synchronization module that synchronizes based on the most delayed signal among the ultrasound image, the bio-signal, and the lesion area when the lesion area detected by the lesion detection unit is smaller than a preset reference value; and a prediction synchronization module that synchronizes based on the most advanced signal among the ultrasound image, the bio-signal, and the lesion area when the lesion area detected by the lesion detection unit is larger than a preset reference value.

Preferably, the prediction synchronization module can synchronize another signal to the most advanced signal by using extrapolation.

Preferably, the output device includes a first camera for photographing the forward direction of the wearer and a second camera for photographing the face of the wearer, wherein the first camera can recognize the probe shape of the ultrasonic diagnostic device, and the second camera can track the direction of the wearer's gaze.

Preferably, the output device can display the second MR content along the direction of the wearer's gaze tracked by the second camera.

Preferably, the output device can be controlled by the movement of the wearer's eye or pupil captured by the second camera.

In addition, the present invention includes a data receiving step of receiving an ultrasound image and a bio-signal of a subject; a first content generating step of generating a first MR (Mixed-reality) content by aligning the ultrasound image with a specific object; a second content generating step of generating a second MR content by aligning the bio-signal with a specific space; and a step of displaying the MR content generated in the first content generating step or the second content generating step on an output device; and another feature of the first content generating step is that the shape of a probe of an ultrasound diagnostic device is recognized through a camera provided in the output device, and the movement of the probe shape is detected to change the output position of the first MR content.

The present invention has the advantage of reducing operator fatigue by allowing the biopsy site and ultrasound to be viewed simultaneously, and increasing accuracy and stability by allowing the ultrasound image and lesion site to be superimposed on the biopsy site.

In addition, the present invention has an advantage in that it can ensure patient safety by displaying a physical danger signal in the line of sight of the operator through an output device that can be worn by the operator.

In addition, the present invention has the advantage of ensuring the reliability of output information by synchronizing input signals and preventing the needle from being inserted into a different part.

In addition, the present invention can increase the freedom of the operator by providing an environment in which equipment can be controlled without touching.

Figure 1 shows a configuration diagram of a mixed reality-based ultrasound image output system according to an embodiment of the present invention.

Figure 2 shows a configuration of a stand-alone mixed reality-based ultrasound image output system according to an embodiment of the present invention.

FIG. 3 shows a configuration of a mixed reality-based ultrasound image output system using a smartphone tethering method according to an embodiment of the present invention.

FIG. 4 is a drawing for explaining the relationship between the size of MR content output to an output device and the probe angle according to an embodiment of the present invention.

Figure 5 shows a configuration diagram of a lesion detection unit according to an embodiment of the present invention.

Figure 6 shows a process in which a lesion detection unit according to an embodiment of the present invention detects a lesion using artificial intelligence.

Figure 7 shows a configuration diagram of a synchronization unit according to an embodiment of the present invention.

FIG. 8 is a diagram illustrating a synchronization method of a delay synchronization module according to an embodiment of the present invention.

FIG. 9 is a diagram illustrating a synchronization method of a prediction synchronization module according to an embodiment of the present invention.

Figure 10 shows a screen displayed through an output device according to an embodiment of the present invention.

A data receiving unit that receives ultrasound images and bio-signals of a subject;

A first content generation unit that generates first MR (Mixed-reality) content by aligning the above ultrasound image with a specific object;

A second content generation unit that generates second MR content by aligning the above biosignals to a certain space; and

An output device for displaying MR content generated by the first content generation unit or the second content generation unit;

The above first content creation unit,

A mixed reality-based ultrasound image output system characterized by recognizing the probe shape of an ultrasound diagnostic device through a camera equipped in the output device, detecting movement of the probe shape, and changing the output position of the first MR content.

Hereinafter, the present invention will be described in detail with reference to the contents described in the attached drawings. However, the present invention is not limited or restricted by the exemplary embodiments. The same reference numerals presented in each drawing represent components that perform substantially the same function.

The purpose and effect of the present invention can be naturally understood or made clearer by the following description, and the purpose and effect of the present invention are not limited to the following description alone. In addition, when explaining the present invention, if it is judged that a detailed description of a known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description will be omitted.

The terminology used herein is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprises" or "has" and the like are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the description of the invention, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms defined in commonly used dictionaries, such as those defined in common usage, should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein.

In interpreting the components, even if there is no separate explicit description, it is interpreted as including the error range. In the case of a description of a temporal relationship, for example, if the temporal continuity is described as 'after', 'following', 'next to', 'before', etc., it also includes cases where it is not continuous unless 'right away' or 'directly' is used.

Hereinafter, the technical configuration of the present invention will be described in detail with reference to the attached drawings.

FIG. 1 illustrates a configuration diagram of a mixed reality-based ultrasound image output system (100) according to an embodiment of the present invention. Referring to FIG. 1, the mixed reality-based ultrasound image output system (100) may include a data receiving unit (110), a first content generating unit (120), a second content generating unit (130), a lesion detection unit (140), a synchronization unit (150), and an output device (160).

The mixed reality-based ultrasound image output system (100) can reduce operator fatigue by allowing the biopsy site and ultrasound to be viewed simultaneously, and can increase accuracy and stability by allowing the ultrasound image and lesion site to be superimposed on the biopsy site. The mixed reality-based ultrasound image output system (100) can output the subject's bio-signals together with the ultrasound image through an AR/VR/MR device or glasses, etc. The mixed reality-based ultrasound image output system (100) can analyze bio-signals, etc. and display the derived body risk signals in the operator's line of sight through an output device (AR/VR/MR device or glasses, etc.) worn by the operator so that the operator can immediately recognize them, thereby ensuring patient safety.

The mixed reality-based ultrasound image output system (100) has the advantage of ensuring the reliability of the output information by synchronizing the input signals and preventing the needle from being inserted into a different part by outputting signals at different times. The mixed reality-based ultrasound image output system (100) can increase the freedom of the operator by providing an environment in which the equipment can be controlled without touching.

The mixed reality-based ultrasound image output system (100) can be provided in a smartphone tethering method and a stand-alone method.

Fig. 2 shows a configuration of a stand-alone mixed reality-based ultrasound image output system according to an embodiment of the present invention. Referring to Fig. 2, the mixed reality-based ultrasound image output system (100) can be operated independently without a separate user terminal (smart phone, laptop, etc.). The stand-alone method can perform computing operations required for implementing mixed reality in the output device (160) itself. The stand-alone method has the advantage of convenient usability because there is no need to carry a user terminal.

Specifically, the stand-alone method can measure the ultrasound image and bio-signal of a subject using an ultrasound device and a bio-signal device and then transmit them to a medical device data processing server. Here, the medical device data processing server may mean the data receiving unit (110) of the present invention. The medical device data processing server can provide patient bio-signal change information and ultrasound images to an AI analysis server (module) and an output device.

The AI analysis server can output AI analysis results, where the AI analysis may mean lesion detection. That is, the AI analysis server may mean the lesion detection unit (140) of the present invention. The AI analysis server may provide the AI analysis results to an MR HMD (Head Mounted Display) or glasses via the Internet.

MR HMD or glasses can receive patient bio-signal change information and ultrasound images from a medical device data processing server via the Internet, and using the received data, can create first MR (Mixed-reality) content by aligning the ultrasound image with a specific object, and can create second MR content by aligning the bio-signal with a certain space. That is, in the stand-alone method, the first content generation unit (120) and the second content generation unit (130) can be included in the output device (160).

FIG. 3 shows the configuration of a mixed reality-based ultrasound image output system using a smartphone tethering method according to an embodiment of the present invention. Referring to FIG. 3, a mixed reality-based ultrasound image output system (100) can be connected to and operated by a smartphone. The smartphone tethering method can be performed on a smartphone without performing computing operations required for implementing mixed reality in the output device (160) itself. The smartphone tethering method has the advantage of being lightweight and generating less heat because there is no processor or battery for computing inside the output device (160).

Specifically, the smartphone tethering method can transmit patient bio-signal change information, ultrasound images, and AI analysis results to a smartphone via the Internet, and the smartphone can use the transmitted data to align the ultrasound images with a specific object to generate first MR (Mixed-reality) content, and align the bio-signals with a specific space to generate second MR content. That is, the smartphone tethering method may not include the first content generation unit (120) and the second content generation unit (130) within the output device (160).

The data receiving unit (110) can receive ultrasound images and bio-signals of a subject. The data receiving unit (110) can be connected to an ultrasound diagnostic device and a bio-signal measuring device by wire or wirelessly to receive ultrasound images and bio-signals in real time. The data receiving unit (110) can transmit the received ultrasound images and bio-signals to a first content generating unit (120), a second content generating unit (130), or a lesion detecting unit (140). When the data receiving unit (110) is connected wirelessly to another component, it can use a broad mobile communication network such as CDMA (Code Division Multiple Access), WCDMA (Wideband CDMA), LTE (Long Term Evolution), or Wi-Fi. When the data receiving unit (110) is connected wirelessly to another component, it can use a short-range wireless communication such as Bluetooth.

The first content generation unit (120) can generate the first MR (Mixed-reality) content by aligning the ultrasound image with a specific object. Here, the specific object may mean a body part that can be directly superimposed 1:1 on the anatomical structure of the location that the ultrasound image is intended to measure.

The first content generation unit (120) can correct the first MR content by interacting with the ultrasonic diagnostic device, the data receiving unit (110), or the output device (160). That is, the first content generation unit (120) can correct the first MR content so that the ultrasound image can be accurately aligned with a specific object through the shape of the probe of the ultrasonic diagnostic device, the distance between the operator's eyes and the output device (160), etc.

In the past, the tracking pattern (marker, QR code, etc.) attached to the probe of the ultrasound diagnostic device was analyzed to calculate the position where the ultrasound image should be aligned. However, when performing a biopsy or treatment using an actual ultrasound image, the operator continuously moves the probe to accurately obtain the necessary ultrasound image. At this time, the tracking pattern attached to the probe may be deformed or invisible. Technically, even if the tracking pattern is tilted or rotated only 30 degrees, the camera does not recognize the tracking pattern.

There is a limit to calculating the position to be aligned using a tracking pattern in this way. Therefore, the first content generation unit (120) according to the embodiment of the present invention can recognize the probe shape of the ultrasonic diagnostic device through a camera equipped in the output device (160) and detect the movement of the probe shape to change the output position of the first MR content.

The first content generation unit (120) can output the first MR content in the direction in which the front of the probe faces through the recognized probe shape, and preferably, can change the output position of the first MR content so that the ultrasound image directly overlaps 1:1 with the anatomical structure of the location to be measured toward the front of the probe based on the part where the probe and the subject's body come into contact.

Since the ultrasound device outputs ultrasound in a straight direction perpendicular to the probe and visualizes the reflected signal, an ultrasound image is an image of a body part perpendicular to the probe. Therefore, the meaning of outputting the first MR content so that the ultrasound image is directly superimposed on the anatomical structure of the location to be measured is that the output device will implement it so that the image created by ultrasound is actually physically located under the probe.

FIG. 4 is a diagram for explaining the relationship between the size of MR content output to an output device according to an embodiment of the present invention and the probe angle. Referring to FIG. 4 (a), when the probe outputs ultrasound perpendicularly to the body of the subject, the ultrasound image area output within the AR glasses of the practitioner can be confirmed. If the practitioner is wearing AR glasses, the measurement area under the probe is calculated to look like from the position of the eyes of the practitioner wearing the AR glasses and output to the AR glasses. Therefore, if the MR image is output to the practitioner so as to be directly superimposed 1:1 on the anatomical structure of the location to be measured, the three-dimensional location of the actual lesion inside the body can be accurately known, so that a needle can be accurately inserted into the corresponding location. The size of the ultrasound image output to the AR glasses can be determined through the calculation of how the measurement area under the probe looks like to the eyes of the practitioner.

Referring to (b) of Fig. 4, when the probe outputs ultrasound at an obtuse angle with the subject's body, the ultrasound image area displayed within the operator's AR glasses can be confirmed. In this case, the viewing angle for the measurement area under the probe becomes larger than when the probe outputs ultrasound perpendicularly to the subject's body, and accordingly, the size of the ultrasound image area displayed within the AR glasses also becomes larger.

Referring to (c) of Fig. 4, when the probe outputs ultrasound at an acute angle to the subject's body, the ultrasound image area displayed within the operator's AR glasses can be confirmed. In this case, the viewing angle for the measurement area below the probe becomes smaller than when the probe outputs ultrasound perpendicularly to the subject's body, and accordingly, the size of the ultrasound image area displayed within the AR glasses also becomes smaller.

As examined through (a) to (c) of Figs. 4, the size of the ultrasound image displayed on the output device may vary depending on the positional relationship between the output device worn by the practitioner and the probe, such as the angle of the probe, the distance between the practitioner and the probe, etc. However, if the size of the displayed ultrasound image is excessively small, the practitioner may have difficulty performing the procedure while viewing the ultrasound image. To solve this, as shown in (d) of Fig. 4, if the size of the ultrasound image area displayed in the output device is below a certain level, a separate ultrasound image may be displayed in the output device. Accordingly, the practitioner can safely perform the procedure through the separately displayed ultrasound image even if the ultrasound image is displayed excessively small depending on the positional relationship between the output device and the probe. In addition, a separate ultrasound image may be displayed in a specific area of the AR display according to the control of the practitioner.

The first content generation unit (120) determines the location where the ultrasound image will be enhanced not by detecting a tracking pattern, but by determining it through the shape of the probe, so that reliable ultrasound images can be output even when the probe moves during the procedure.

In the case of conventional technologies using tracking patterns, the tracking pattern had to be attached to a probe, etc., in order to enhance the ultrasound image through MR HMD or glasses. The mixed reality-based ultrasound image output system (100) according to an embodiment of the present invention does not use a tracking pattern but uses the shape of a probe, so it can be applied to existing ultrasound diagnostic devices by only having modularized equipment without separate installation.

The first content generation unit (120) can change the first MR content so that the lesion area detected by the lesion detection unit (140) can be distinguished from other areas. Although the operator can distinguish the lesion in the ultrasound image, it is very difficult to distinguish the lesion area from other areas besides the lesion area. Therefore, currently, whether or not to detect the lesion is determined based on the operator's ability. The first content generation unit (120) can emphasize the outer line of the lesion area detected by the lesion detection unit (140) or emphasize it by changing the color of the inside of the lesion area. If the user wants to check the inside of the lesion according to the user's setting, the first content generation unit (120) can emphasize the outer line of the detected lesion area. If the user wants to check the size of the lesion according to the user's setting, the first content generation unit (120) can emphasize the lesion area by changing the color of the inside of the lesion area.

The first content generation unit (120) can change the MR content so that the lesion area is not distinguished from other areas when it is confirmed through the ultrasound image that the needle has contacted the lesion area. Even if the operator has distinguished the lesion area, it is a very difficult technique to accurately insert a needle into the lesion area to acquire lesion tissue or treat the lesion. Therefore, the first content generation unit (120) can visually notify the operator that the lesion area has been contacted by restoring the highlighted lesion area to be identical to other areas when the needle has accurately inserted the lesion area.

The second content generation unit (130) can generate second MR content by aligning a biosignal to a predetermined space. Here, the predetermined space means a space where the operator can check the output biosignal while focusing on the ultrasound image and performing a biopsy or treatment. Preferably, it can mean the gaze direction of the operator's right or left eye. The biosignal can include an electrocardiogram (ECG), an electroencephalogram (EEG), an electromyogram (EMG), a galvanic skin response (GSR), a skin temperature (SKT), a photoplethysmography (PPG), a pulse rate (PR), blood pressure, oxygen saturation, etc.

The second content generation unit (130) can determine that a dangerous situation has occurred for the subject if the bio-signal exceeds the reference value and can provide a warning or notification to the operator. The second content generation unit (130) can change the second MR content so that the screen on which the bio-signal is output blinks or flashes red when an abnormality in the bio-signal occurs.

Fig. 5 shows a configuration diagram of a lesion detection unit (140) according to an embodiment of the present invention. Referring to Fig. 5, the lesion detection unit (140) may include an input unit (141), a preprocessing unit (142), a learning unit (143), a lesion prediction unit (144), and a postprocessing unit (145).

The lesion detection unit (140) can detect a lesion area in an ultrasound image using artificial intelligence learned from an ultrasound image data set in which the lesion area is annotated. Specifically, the lesion detection unit (140) can detect a lesion area in an ultrasound image using a semantic segmentation model. The semantic segmentation model is a process of dividing a digital image into a plurality of pixel sets, and can simplify and convert the expression of the image into something easy to interpret through segmentation. The lesion detection unit (140) can use a CNN-based model as a semantic segmentation model.

The input unit (141) can receive ultrasound images of normal and patient groups for learning, testing, or actual lesion detection. The input unit (141) can receive ultrasound images from the data receiving unit (110). The input unit (141) can transmit the received ultrasound images to the preprocessing unit (142).

The preprocessing unit (142) can preprocess the ultrasound image received from the input unit (141) to be advantageous for learning or detection. The preprocessing unit (142) can perform preprocessing in a manner of normalizing pixel values. The preprocessing unit (142) can perform preprocessing to adjust the size and resolution of the ultrasound image received so that it can be input to the artificial neural network. The ultrasound image used for learning may include the patient's personal information, which may cause security and personal information leakage issues. Therefore, the preprocessing unit (142) according to the present invention can perform preprocessing to de-identify the patient information in the ultrasound image and metadata.

The learning unit (143) can train artificial intelligence using ultrasound images that have been preprocessed and have lesion areas annotated. Here, the artificial intelligence to be trained can be a CNN-based or SOTA architecture-based semantic segmentation model.

A convolutional neural network (CNN) is a type of deep learning model that is inspired by the structure of the visual cortex of animals and is designed to process data with grid patterns such as images. A convolutional neural network may generally include a convolutional layer, a pooling layer, and a fully connected layer. The convolutional layer and the pooling layer may exist repeatedly in the neural network, and the input data may be converted to an output through these layers. The convolutional layer uses a kernel (or mask) for feature extraction, and the element-wise product between each element of the kernel and the input value is calculated at each location and summed to obtain an output value, which is called a feature map. This procedure may be repeated by applying multiple kernels to form an arbitrary number of feature maps. In a convolutional neural network, the convolutional and pooling layers perform feature extraction, while the fully connected layer maps the extracted features to the final output, such as a classification operation.

Convolutional neural networks can be trained in a way that minimizes output errors. Separately from the forward propagation process that extracts values from the input layer to the output layer, backpropagation occurs in the neural network to calculate the error between the input training data and the output value of the neural network for it, and update the weights of the nodes of each layer to reduce this error. The training process in a convolutional neural network can be summarized as the process of finding the kernel that extracts the output value with the least error based on the given training data. The kernel is the only parameter that is automatically learned during the training process of the convolutional layer. On the other hand, the kernel size, the number of kernels, and the padding in the convolutional neural network are hyperparameters that must be set before starting the training process, and therefore, different convolutional neural network models can be distinguished depending on the kernel size, the number of kernels, and the number of convolutional layers and pooling layers.

The lesion prediction unit (144) can predict and detect a lesion area from an ultrasound image of a subject using artificial intelligence learned in the learning unit (143).

The post-processing unit (145) can perform post-processing to distinguish the lesion area detected by the lesion prediction unit (144) from other areas. This is the same function as that performed in the first content generation unit (120) described above.

Figure 6 shows a process in which a lesion detection unit (140) according to an embodiment of the present invention detects a lesion using artificial intelligence. Referring to Figure 6, the lesion detection unit (140) can receive an ultrasound image, input it into a CNN (Convolutional Neural Network), and then input it into a high-performance Semantic segmentation model to obtain a binary mask in which the lesion prediction area and other areas are distinguished. Thereafter, the lesion detection unit (140) can select a lesion area from the lesion prediction area.

Fig. 7 shows a configuration diagram of a synchronization unit (150) according to an embodiment of the present invention. Referring to Fig. 7, the synchronization unit (150) may include a delay synchronization module (151) and a prediction synchronization module (152).

The synchronization unit (150) can synchronize the ultrasound image, the biosignal, and the lesion area so that the ultrasound image and the biosignal measured at the same time can be output, and the lesion area detected at the same time can be displayed. The synchronization unit (150) plays an important role in allowing the operator to perform biopsy and treatment based on information at the same time. If the ultrasound image and the lesion area are measured at different times, the operator expects that there will be a lesion in the displayed lesion area, but in reality, there may be no lesion at that location because it is a lesion area at a different time.

In this way, the reason why synchronization of each signal is necessary is because the time required for the ultrasound image, bio-signal, or lesion area to be composed, detected, or transmitted is different. The synchronization unit (150) can synchronize the ultrasound image, bio-signal, or lesion area displayed to the operator through the output device (160) with those measured at the same time, thereby ensuring stability and reliability.

Fig. 8 is a diagram for explaining a synchronization method of a delay synchronization module according to an embodiment of the present invention. Referring to Fig. 8, the delay synchronization module (151) can synchronize based on the most delayed signal among the ultrasound image, biosignal, and lesion region when the lesion region detected by the lesion detection unit (140) is smaller than a preset reference value.

Specifically, the lesion area, ultrasound image, and bio-signal can be delayed by T1, T2, and T3, respectively, and the signals can be delivered (T1>T2>T3). At this time, if synchronization is performed based on the lesion area with the longest delay time, it is easy to synchronize because the measured values already exist at the synchronization reference time. That is, the ultrasound image can be synchronized with the lesion area by outputting the signal from a time point earlier than Tp2, and the bio-signal can be synchronized by outputting the signal from a time point earlier than Tp3. This synchronization method has the advantage of high reliability and accuracy because the measured values already exist, but has the disadvantage of a long delay time.

FIG. 9 is a diagram illustrating a synchronization method of a prediction synchronization module (152) according to an embodiment of the present invention. Referring to FIG. 9, the prediction synchronization module (152) can synchronize based on the most advanced signal among the ultrasound image, biosignal, and lesion area when the lesion area detected by the lesion detection unit (140) is larger than a preset reference value.

Specifically, if synchronization is performed based on the biosignal with the shortest delay time, there is a problem that the lesion area and the ultrasound image do not have data that has reached the synchronization reference point in time. Therefore, the prediction synchronization module (152) can synchronize another signal to the most advanced signal using extrapolation. Extrapolation refers to a method used when the available data is limited and a value that exceeds the limit is desired to be obtained. In another embodiment, the prediction synchronization module (152) can predict data up to the point in time of the most advanced signal using artificial intelligence.

That is, the prediction synchronization module (152) can synchronize with the biosignal by predicting the signal at a time point ahead by Tf1 and outputting the lesion area, and predicting the signal at a time point ahead by Tf2 and outputting the ultrasound image. This synchronization method has the advantage of a short delay time because it synchronizes at the shortest delay time, but has the disadvantage of low reliability and accuracy because it predicts and outputs values that have not yet been measured.

Therefore, the mixed reality-based ultrasound image output system (100) according to the embodiment of the present invention can use a delay synchronization module (151) with high accuracy when the detected lesion is small, and can use a prediction synchronization module (152) with short delay time when the detected lesion is large. The delay synchronization module (151) and the prediction synchronization module (152) can be selected according to user settings.

Fig. 10 illustrates a screen displayed through an output device (160) according to an embodiment of the present invention. Referring to Fig. 10, the output device (160) can display MR content generated by the first content generation unit (120) or the second content generation unit (130). The output device (160) can be a device that can be worn by a practitioner.

The output device (160) may be an AR/VR/MR device, glasses, or HMD that can display the first MR content and the second MR content in an augmented manner. The output device (160) may include a processor and a battery to perform computing operations required for implementing mixed reality internally. The output device (160) may not include a processor and a battery to perform computing operations in order to reduce weight.

The output device (160) may include a first camera that photographs the front direction of the wearer and a second camera that photographs the face of the wearer. The first camera may recognize the probe shape of the ultrasonic diagnostic device, and the second camera may track the direction of the wearer's gaze. The first camera may be used to recognize the probe shape in the first content generation unit (120) described above.

The output device (160) can display the second MR content along the direction of the wearer's gaze tracked by the second camera. As described above, the second MR content can be composed of biosignals, and the biosignals must be able to identify their status even in a situation where the operator focuses on the ultrasound image. Accordingly, the output device (160) can detect the direction of the wearer's left or right eye gaze using the second camera and display the second MR content near the corresponding direction of gaze.

The output device (160) can be controlled by the movement of the wearer's eyes or pupils captured by the second camera. The operator holds an ultrasound probe in one hand for ultrasound diagnosis and a needle in the other hand for biopsy or treatment. Therefore, in order to control the output device (160), the operator must put down the probe or needle that he or she is holding or request an assistant to do so. However, there is a limitation in controlling the output device (160) if the operator frequently controls the output device (160) or if an emergency situation occurs and the output device (160) must be quickly controlled. Therefore, the output device (160) can provide the operator with a high degree of freedom by being controlled by the blinking of the wearer's eyes or the movement of the pupils. Here, the control may mean changing the UI, turning on/off the first MR content or the second MR content, etc.

A mixed reality-based ultrasound image output method, which is another embodiment of the present invention, may include a data receiving step, a first content generating step, a second content generating step, a lesion detection step, a synchronization step, and a displaying step.

The data receiving step can receive ultrasound images and bio-signals of the subject from the ultrasonic diagnostic device and the bio-signal measuring device. The data receiving step refers to an operation performed in the data receiving unit (110) described above.

The first content generation step can generate first MR (Mixed-reality) content by aligning an ultrasound image with a specific object. The first content generation step can recognize the probe shape of an ultrasound diagnostic device through a camera equipped in an output device, and detect the movement of the probe shape to change the output position of the first MR content. The first content generation step refers to an operation performed in the first content generation unit (120) described above.

The second content generation step can generate second MR content by aligning biosignals to a certain space. The second content generation step refers to an operation performed in the second content generation unit (130) described above.

The lesion detection step can detect lesion areas in ultrasound images using artificial intelligence trained on an ultrasound image data set in which lesion areas are annotated. The lesion detection step refers to an operation performed in the lesion detection unit (140) described above.

The synchronization step can synchronize the ultrasound image, biosignal, and lesion area so that the ultrasound image and biosignal measured at the same time can be output and the lesion area detected at the same time can be displayed. The synchronization step means the operation performed in the synchronization unit (150) described above.

The displaying step can display the MR content generated in the first content generation step or the second content generation step on an output device. The displaying step refers to an operation performed on the aforementioned output device (160).

Although the present invention has been described in detail through representative examples above, those skilled in the art will understand that various modifications can be made to the above-described embodiments without departing from the scope of the present invention. Therefore, the scope of the rights of the present invention should not be limited to the described embodiments, but should be determined by all changes or modifications derived from the claims and equivalent concepts as well as the claims.

[Explanation of symbols]

100: Mixed reality based ultrasound image output system

110: Data receiving unit 120: First content generating unit

130: Second content creation unit 140: Lesion detection unit

141: Input section 142: Preprocessing section

143: Learning section 144: Lesion prediction section

145: Post-processing section 150: Synchronization section

151: Delayed Synchronization Module 152: Predictive Synchronization Module

160: Output device

The present invention aims to provide a method and system for recognizing the context of a digital slide image, to which a contextual image representation method that is invariant to rotational transformation and can reflect the surrounding context is applied.

Claims (11)

A data receiving unit that receives ultrasound images and bio-signals of a subject; A first content generation unit that generates first MR (Mixed-reality) content by aligning the above ultrasound image with a specific object; A second content generation unit that generates second MR content by aligning the above biosignals to a certain space; and An output device for displaying MR content generated by the first content generation unit or the second content generation unit; The above first content creation unit, A mixed reality-based ultrasound image output system characterized by recognizing the probe shape of an ultrasound diagnostic device through a camera equipped in the output device, detecting movement of the probe shape, and changing the output position of the first MR content. In the first paragraph, A mixed reality-based ultrasound image output system further comprising a lesion detection unit that detects a lesion area in the ultrasound image using artificial intelligence learned from an ultrasound image data set in which the lesion area is annotated. In the second paragraph, The above first content creation unit, A mixed reality-based ultrasound image output system characterized in that the first MR content is changed so that the lesion area detected by the above lesion detection unit is distinguished from other areas. In the third paragraph, The above first content creation unit, A mixed reality-based ultrasound image output system characterized in that, when it is confirmed through the ultrasound image that the needle has contacted the lesion area, the MR content is changed so that the lesion area is not distinguished from other areas. In the second paragraph, A mixed reality-based ultrasound image output system characterized by further including a synchronization unit that synchronizes the ultrasound image, the bio-signal, and the lesion area so that the ultrasound image and the bio-signal measured at the same time can be output and the lesion area detected at the same time can be displayed. In paragraph 5, The above synchronization part, A delay synchronization module that synchronizes the most delayed signal among the ultrasound image, the biosignal, and the lesion area when the lesion area detected by the lesion detection unit is smaller than a preset reference value; and A prediction synchronization module that synchronizes based on the most advanced signal among the ultrasound image, the bio-signal, and the lesion area when the lesion area detected by the above lesion detection unit is larger than a preset reference value; A mixed reality-based ultrasound image output system comprising: In Article 6, The above prediction synchronization module, A mixed reality-based ultrasound image output system characterized by synchronizing another signal to the most advanced signal using extrapolation. In the first paragraph, The above output device is, Includes a first camera for photographing the wearer's forward direction and a second camera for photographing the wearer's face, The above first camera, Recognize the probe shape of the ultrasonic diagnostic device, The second camera above, A mixed reality-based ultrasound image output system characterized by tracking the wearer's gaze direction. In Article 8, The above output device is, A mixed reality-based ultrasound image output system characterized by displaying the second MR content along the wearer's gaze direction tracked by the second camera. In Article 8, The above output device is, A mixed reality-based ultrasound image output system characterized in that it is controlled by the movement of the wearer's eyes or pupils captured by the second camera. A data receiving step for receiving ultrasound images and bio-signals of a subject; A first content generation step of generating first MR (Mixed-reality) content by aligning the above ultrasound image to a specific object; A second content generation step of generating second MR content by aligning the above biosignals to a certain space; and A step of displaying MR content generated in the first content generation step or the second content generation step on an output device; The above first content creation step is: A mixed reality-based ultrasound image output method characterized by recognizing the probe shape of an ultrasound diagnostic device through a camera equipped in the output device and detecting movement of the probe shape to change the output position of the first MR content.

Images