WO2025053376A1 - Three-dimensional stomach navigation method and system for capsule endoscope - Google Patents

Abstract

In order to accomplish the objective, the present invention comprises the steps of: obtaining a captured image from a capsule endoscope inserted into a human body; detecting a landmark from the image obtained from the capsule endoscope, and identifying position and pose information of the capsule endoscope; and generating a moving path of the capsule endoscope on the basis of at least one from among the position information and pose information of the capsule endoscope, the landmark, and a three-dimensional gastroscopy map.

Description

3D upper navigation method and system for capsule endoscopy

The present invention was made with the support of the Ministry of Science and ICT, the Ministry of Trade, Industry and Energy, the Ministry of Health and Welfare, and the Ministry of Food and Drug Safety, under the project identification number RS-2021-KD000001, project number 1415184155. The research management specialized institution of the project is the Inter-Ministry Full-Cycle Medical Device Research and Development Project Group, the research project name is "Inter-Ministry Full-Cycle Medical Device Research and Development Project", the research project name is "Development of a medical device for active precision delivery of therapeutic substances based on microcarriers for knee cartilage regeneration", the main institution is the Korea Micro Medical Robot Research Institute, and the research period is from January 1, 2023 to December 31, 2023.

The present invention relates to a three-dimensional gastric navigation method and system for a capsule endoscope, and more particularly, to a three-dimensional gastric navigation method and system for a capsule endoscope that guides a three-dimensional path so that a capsule endoscope positioned inside a human body can explore the inside of the stomach.

Endoscopes generally refer to medical instruments used to examine the inside of the body for medical purposes. Depending on the area being examined, endoscopes are called bronchoscopes, gastroscopes, laparoscopes, colonoscopes, and nasal endoscopes. The most common existing gastrointestinal endoscopy methods are performed using a catheter in the form of a hose or tube with a camera attached, and a catheter with a 2D camera and light is inserted into the esophagus to observe the walls inside the stomach, etc.

This conventional gastroscopy method is diagnosed based on the image of the stomach interior displayed on the screen by the doctor directly manipulating the catheter. The conventional gastroscopy method has the problem that if it is performed without sleep, it is accompanied by vomiting, abdominal distension, pain, and so on, causing the subject to feel distress during the examination process. In addition, if it is performed with sleep, it takes at least 30 minutes to 1 hour for the anesthesia to wear off due to the anesthetic, and there is the problem that an additional recovery period of at least 1 to 2 days is required.

In particular, sleep endoscopy carries additional risks of accidents, such as falling down the floor or stairs due to loss of strength after sleep anesthesia, or accidents while driving or operating equipment. In addition, propofol, a drug used in sleep endoscopy, has no known effective antidote at present, so there is a small chance of death due to failure to wake up. Midazolim, a drug with an antidote, has low sleep depth and analgesic effects, and has the problem of causing hallucinations in the subject.

To make up for the shortcomings of conventional gastrointestinal endoscopes, capsule endoscopes have been developed. Capsule endoscopes are devices that move along the digestive tract by the peristalsis of the digestive tract, photograph the inside of the human digestive tract, and transmit the photographed information to the outside via communication. Medical capsule endoscopes have a small capsule shape, photograph the inside of a living body, and wirelessly transmit the photographed images of the inside of the living body to an external storage device. The images of the inside of the living body captured by the endoscope are stored in the storage device. The images of the inside of the living body stored in the storage device are displayed on a display device, etc. after a conversion process, and the reader observes the image data displayed on the display device to observe the organs inside the living body.

These existing capsule endoscopes have difficulty viewing the entire gastrointestinal tract, are difficult to confirm the current location in real time, and are moved by gastrointestinal peristalsis, making additional control from the outside impossible.

In this regard, Korean Patent Publication No. 10-2020-0114844 discloses an electromagnetic field device for driving a micro robot inside the human body from outside the human body. Although these devices can solve the problem of the existing capsule endoscope having no additional control from the outside, there is a limitation in manually or automatically controlling the capsule endoscope from the outside because the path the capsule endoscope moves cannot be searched in real time.

In addition, existing endoscopes have limitations in image information, such as low visibility because they are filmed with a 2D camera, distortion in the image because the image is obtained through a wide-angle lens with a wide angle of view inside a narrow area, differences in the clarity of the examination and diagnosis depending on the ability of the person operating the capsule endoscope, and inability to recheck specific parts after the examination. Due to these limitations in image information, it is difficult to find the path that the capsule endoscope will move.

In this regard, Korean Patent Publication No. 10-2022-0064464 discloses a magnetic field-controlled pH sensor-assisted navigation capsule endoscope that controls the battery to uniformly acquire normal images from the large intestine. The above-mentioned prior document is characterized in that, in order to efficiently use power, it is determined whether the location of the capsule endoscope is the large intestine through the image captured by the capsule endoscope and the sensed pH, and if it is the large intestine, the battery is turned on, and if it is not the large intestine, the battery is turned off.

The above prior literature aims to consume power efficiently, so it only needs to determine the approximate location of whether it is a colon or not, and since the criteria for determining the location include not only the photographed photograph but also the pH, there is a limitation in that an additional pH sensing device is required, which is disadvantageous in terms of production cost and miniaturization.

That is, the above prior literature does not present a technical feature for searching in real time the path that a capsule endoscope positioned inside an organ will move for diagnosis or treatment, and still does not resolve the problem that it is difficult to search the path that a capsule endoscope will move due to the limitations of image information.

Accordingly, in order to search for the path along which the capsule endoscope will move, a three-dimensional navigation method and system for a capsule endoscope is required, which reconstructs images captured by the capsule endoscope to be advantageous for searching for the path along which it will move, and searches for and guides the path along which the capsule endoscope will move based on the reconstructed images.

The purpose of the present invention is to provide a three-dimensional stomach navigation method and system for a capsule endoscope that guides a three-dimensional path in real time so that a capsule endoscope positioned inside a human body can explore the inside of the stomach.

In addition, the present invention aims to restore capsule endoscope images captured in real time to 3D data so as to enable 3D navigation path search, and to visualize landmarks in the 3D data by aligning the restored 3D data with 3D landmark data.

In addition, the present invention aims to guide the capsule endoscope along a path to the next landmark location to enable the capsule endoscope to sequentially explore the inside of an organ.

In order to achieve the above object, the present invention is characterized by including a step of obtaining an image captured from a capsule endoscope inserted into a human body, a step of detecting a landmark from the image obtained from the capsule endoscope and determining position and pose information of the capsule endoscope, and a step of generating a movement path of the capsule endoscope based on at least one of the position information and pose information of the capsule endoscope, the landmark, or a three-dimensional upper gastrointestinal endoscope map.

Preferably, the step of acquiring the captured image may include a step of aligning the three-dimensionally restored upper data with the generated three-dimensional landmark data to form three-dimensional upper data with the landmarks aligned.

Preferably, the three-dimensionally restored data can be formed in real time by extracting feature points of the current frame and the previous frame of the captured image and then matching the extracted feature points.

Preferably, the step of forming the three-dimensional upper data in which the landmarks are aligned may include moving and rotating the parasitic three-dimensional landmark data so that the Euclidean distance between the points constituting the parasitic three-dimensional landmark data and the three-dimensionally restored upper data is the closest, and overlapping the parasitic three-dimensional landmark data with the three-dimensionally restored upper data.

Preferably, the step of forming the three-dimensional data with the landmarks aligned can visualize the aligned landmark portion in the three-dimensional data with the landmarks aligned so as to be distinguished from other portions.

Preferably, the step of calculating the position and pose information of the capsule endoscope can extract feature points of the current frame and the previous frame of the captured image, and then determine the pose information based on the degree of movement and rotation of the capsule endoscope extracted through matching the extracted feature points.

Preferably, the step of detecting the landmark may include a step of inputting the captured image into an artificial intelligence trained with a two-dimensional landmark data set.

Preferably, the method may include a step of extracting current position information and current pose information of the capsule endoscope from the identified position and pose information of the capsule endoscope when one of the multiple landmarks is detected.

Preferably, the step of generating a movement path of the capsule endoscope may include, if any one of the landmarks is not the first landmark, a step of generating a path for moving to the first landmark based on current location information and current pose information of the capsule endoscope, and if any one of the landmarks is the first landmark, a step of generating a path for moving from the first landmark to a second landmark based on current location information and current pose information of the capsule endoscope.

Preferably, the step of generating the movement path of the capsule endoscope can generate the movement path of the capsule endoscope so that the capsule endoscope moves sequentially according to the order of preset landmarks.

Preferably, the landmark may include an area within the stomach of at least one of the Cardia, Fundus, Greater Curvature, Angulus, or Pylorus.

In addition, the present invention is characterized in that it includes an image receiving unit that acquires an image captured from a capsule endoscope inserted into a human body; a calculation unit that detects a landmark from the image acquired from the capsule endoscope and determines position and pose information of the capsule endoscope from three-dimensionally restored upper data; a search unit that generates a movement path of the capsule endoscope based on at least one of the position information and pose information of the capsule endoscope, the landmark, or a three-dimensional upper endoscope map; and a control unit that controls the capsule endoscope to move along the movement path generated by the search unit.

Preferably, the control unit can control the operation of the capsule endoscope using an external magnetic field.

The present invention has the advantage of enabling convenient and accurate diagnosis and examination by guiding the movement path of a capsule endoscope in real time so as to explore the inside of the stomach.

In addition, the present invention has the advantage of being able to confirm the current position of the capsule endoscope by constructing an image captured by the capsule endoscope into virtual 3D data and providing visually effective path guidance.

Figure 1 shows a flow chart of a three-dimensional upper navigation method of a capsule endoscope according to an embodiment of the present invention.

FIG. 2 illustrates an example of a three-dimensional navigation method of a capsule endoscope according to an embodiment of the present invention.

FIG. 3 illustrates a process in which parasitic 3D landmark data according to an embodiment of the present invention is aligned with the above data restored in 3D.

FIG. 4 is a drawing illustrating an example of determining the current position of a capsule endoscope based on major landmark information inside the stomach according to an embodiment of the present invention.

Figure 5 shows a path search process when the first landmark is detected according to an embodiment of the present invention.

FIG. 6 illustrates a capsule endoscope moving sequentially along the order of preset landmarks according to an embodiment of the present invention.

Figure 7 shows a configuration diagram of a three-dimensional navigation system, which is another embodiment of the present invention.

A step of acquiring images taken from a capsule endoscope inserted into the human body;

A step of detecting a landmark from an image acquired from the capsule endoscope and determining the position and pose information of the capsule endoscope, and

A step of generating a movement path of the capsule endoscope based on at least one of the position information and pose information of the capsule endoscope, the landmark, or a 3D upper gastrointestinal endoscope map.

A three-dimensional upper navigation method of a capsule endoscope including a.

Hereinafter, the present invention will be described in detail with reference to the contents described in the attached drawings. However, the present invention is not limited or restricted by the exemplary embodiments. The same reference numerals presented in each drawing represent components that perform substantially the same function.

The purpose and effect of the present invention can be naturally understood or made clearer by the following description, and the purpose and effect of the present invention are not limited to the following description alone. In addition, when explaining the present invention, if it is judged that a detailed description of a known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description will be omitted.

The terminology used herein is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, the terms "comprises" or "has" and the like are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the description of the invention, but should be understood to not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms defined in commonly used dictionaries, such as those defined in common dictionaries, should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein.

In interpreting the components, even if there is no separate explicit description, it is interpreted as including the error range. In the case of a description of a temporal relationship, for example, if the temporal continuity is described as 'after', 'following', 'next to', 'before', etc., it also includes cases where it is not continuous unless 'right away' or 'directly' is used.

Hereinafter, the technical configuration of the present invention will be described in detail with reference to the attached drawings.

FIG. 1 is a flowchart of a three-dimensional upper navigation method of a capsule endoscope according to an embodiment of the present invention. Referring to FIG. 1, the three-dimensional upper navigation method of a capsule endoscope may include a step of acquiring a photographed image (S101), a step of determining position and pose information of a capsule endoscope (S103), a step of generating a movement path of the capsule endoscope (S105), a step of controlling the capsule endoscope (S107), and a data storage step (S109).

In FIG. 1, each step of the three-dimensional navigation method of the capsule endoscope according to the present embodiment is sequentially performed, but it is not necessarily limited to this, and each step exemplified in FIG. 1 may be performed simultaneously or separately in the process of generating a movement path and driving the capsule endoscope (300) to move along the movement path.

A three-dimensional upper navigation method of a capsule endoscope can provide an environment in which an examiner or doctor can conveniently and accurately perform a diagnosis and examination by guiding a three-dimensional path in real time so that a capsule endoscope (300) positioned inside a human body can explore the inside of the stomach. The three-dimensional upper navigation method of a capsule endoscope can restore a capsule endoscope (300) image captured in real time into three-dimensional data so that three-dimensional path exploration is possible, and the three-dimensionally restored stomach data and the generated three-dimensional landmark data are aligned so that landmarks can be visualized in the three-dimensionally restored stomach data, thereby providing visually effective path guidance.

The three-dimensional upper navigation method of the capsule endoscope can identify the current position of the capsule endoscope (300) and calculate pose information to provide a three-dimensional movement path on the three-dimensionally restored upper data. The three-dimensional navigation path search method can sequentially search multiple landmarks inside the organ according to a set order.

FIG. 2 illustrates an example of a three-dimensional navigation method of a capsule endoscope according to an embodiment of the present invention. Referring to FIG. 2, the three-dimensional navigation method of the capsule endoscope can restore a capsule endoscope image captured in real time into three-dimensional stomach data based on a Visual SLAM algorithm (S201). At the same time, the three-dimensional navigation method of the capsule endoscope can detect a landmark in the captured capsule endoscope image. When a landmark is detected, the three-dimensional navigation method of the capsule endoscope can visualize the landmark by aligning the generated three-dimensional landmark data with the three-dimensionally restored stomach data (S203). The three-dimensional navigation method of the capsule endoscope can grasp the position and pose information of the capsule endoscope (300) in real time during the process of restoring the three-dimensional stomach data, and when a landmark is detected, the current position and current pose information of the capsule endoscope (300) can be extracted (S205). At this time, the extracted current location and current pose information can be used as a basis for generating a movement path.

The three-dimensional upper navigation method of the capsule endoscope can generate a movement path based on the current landmark when a landmark is first detected (S207). The three-dimensional upper navigation method of the capsule endoscope can guide a movement path to move to the location of the second landmark, which is the next search order, if the current landmark is the first landmark (S209). The three-dimensional upper navigation method of the capsule endoscope can guide a movement path to move to the location of the first landmark if the current landmark is not the first landmark (S211). The three-dimensional upper navigation method of the capsule endoscope can end the gastrointestinal endoscope search when the capsule endoscope (300) searches landmarks in a preset order and arrives at the last landmark (S213).

Below, each step of the 3D navigation path search method is described in detail.

The step of acquiring a captured image (S101) can acquire an image captured from a capsule endoscope inserted into the human body.

The step (S101) of acquiring the captured image can restore the image captured by the capsule endoscope (300) into 3D data in real time. The step (S101) of acquiring the captured image can prevent problems such as reduced visibility and image distortion of the capsule endoscope (300) image by guiding the movement path using the 3D restored data.

The step (S101) of acquiring a photographed image can be a preliminary task of three-dimensional stomach data restoration by converting an image captured by a capsule endoscope (300) into a virtually stained image. The step (S101) of acquiring a photographed image can be a step of separating images captured by a capsule endoscope (300) into R, G, and B channels from consecutive frames and then converting them into virtually stained images. The step (S101) of acquiring a photographed image can use a deep learning model trained with a data set consisting of an endoscopic image in which an organ is not stained and an endoscopic image in which an organ is stained. The present embodiment describes that the preliminary task of three-dimensional stomach data restoration is performed in the step (S101) of acquiring a photographed image, but is not limited thereto.

The step (S101) of acquiring the captured image can improve the performance of the subsequent 3D stomach data restoration by converting the image captured by the capsule endoscope (300) into a virtually dyed image. That is, when the 3D stomach data is restored using the virtually dyed image, more feature points can be extracted compared to the undyed image, thereby restoring the 3D stomach data similar to the actual inside of the stomach.

The above data restored in 3D can be formed in real time by extracting feature points of the current frame and the previous frame from a captured image or a virtually dyed image, and then matching the extracted feature points.

In the past, the SfM (Structure from Motion) algorithm was used for feature point extraction and feature point matching. The SfM algorithm recovers three-dimensional data by detecting common feature points and finding correlations based on images acquired for one object at multiple points in time. However, the SfM algorithm is not suitable for the characteristics of the present invention, which recovers images captured in real time by a capsule endoscope (300) into three-dimensional data.

In an embodiment of the present invention, the step (S101) of acquiring a photographed image uses a method of restoring an image photographed and transmitted in real time into three-dimensional data. In one embodiment, the step (S101) of acquiring a photographed image can restore an image photographed in real time into three-dimensional data using a Visual SLAM algorithm. The Visual SLAM algorithm is advantageous in constructing real-time 3D data because it requires less computation than the SfM algorithm.

FIG. 3 illustrates a process in which parasitic 3D landmark data according to an embodiment of the present invention is registered to 3D restored upper data. Referring to FIG. 3, the step (S101) of acquiring a photographed image may include a step of registering the 3D restored upper data with the parasitic 3D landmark data to form 3D upper data with the landmarks registered. Here, image registration refers to a processing technique of transforming different images and representing them in a single coordinate system.

Looking specifically at the alignment process, in the step of forming the 3D upper data with the landmarks aligned, if a landmark is detected from a captured image, the same landmark as the landmark detected from the parasitic 3D landmark data can be extracted and aligned to the 3D restored upper data. In the step of forming the 3D upper data with the landmarks aligned, the parasitic 3D landmark data can be moved and rotated so that the Euclidean distance between the points constituting the parasitic 3D landmark data and the 3D restored upper data is the closest, and the parasitic 3D landmark data can be overlapped with the 3D restored upper data.

As an example, the step of forming 3D data with landmark alignment can perform data alignment using the ICP (Iterative Closest Point) algorithm. The ICP algorithm is an algorithm that calculates the distance between corresponding points when two point clouds are given and finds a transformation matrix that minimizes this distance.

The step of forming landmark-aligned 3D data can visualize the landmark-aligned 3D data so that the landmark portion is distinguished from other portions.

The step of forming 3D landmark-aligned stomach data aligns 3D landmark data to virtual 3D data and visualizes the landmarks, thereby enabling the location of representative landmarks within the 3D-constructed stomach to be identified, and providing a visually effective 3D navigation function.

The step of forming 3D stomach data with aligned landmarks can generate 3D landmark data as a preliminary task of landmark detection. The step of forming 3D stomach data with aligned landmarks can utilize a deep learning model trained with a data set composed of images of several frames corresponding to representative landmark areas inside the stomach photographed with a capsule endoscope (300). The detection step (S103) can generate 3D landmark data by inputting an image corresponding to a representative landmark area inside the stomach into the trained deep learning model.

The step of forming landmark-aligned 3D stomach data is a preliminary task of generating a movement path performed in the step (S105) of generating a movement path of a capsule endoscope to be described later, and can generate a 3D gastroscopy map that serves as a landmark of the movement path. The step of forming landmark-aligned 3D stomach data can convert a general or universal human capsule endoscope image into 3D stomach data, and generate a 3D gastroscopy map by aligning the generated 3D landmark data to the converted 3D stomach data. The present embodiment describes that the preliminary task of landmark detection and movement path generation is performed in the step of forming landmark-aligned 3D stomach data, but is not limited thereto.

FIG. 4 shows an example of identifying the current position of a capsule endoscope based on major landmark information inside the stomach according to an embodiment of the present invention. Referring to FIG. 4, the step (S103) of identifying the position and pose information of the capsule endoscope can detect a landmark from an image acquired from the capsule endoscope (300). Here, a landmark means an area that can identify a specific position or a specific part inside the stomach. For example, a representative landmark of the stomach can include at least one area inside the stomach among the cardia, fundus, greater curvature, angulus, and pylorus.

Referring to the left picture of Fig. 4, four landmarks are set inside the stomach, consisting of Cardia & Fundus (Ex. 1), Greater Curvature (Ex. 2), Angulus (Ex. 3), or Pylorus (Ex. 4). Referring to the right picture of Fig. 4, it can be confirmed that the landmarks on the 3D stomach data with aligned landmarks have better visibility than the landmarks on the 3D restored stomach data.

The step (S103) of identifying the position and pose information of the capsule endoscope may include a step of inputting the captured image into an artificial intelligence learned with a two-dimensional landmark data set. When the image captured by the capsule endoscope (300) is input into an artificial intelligence learned with a two-dimensional landmark data set, landmark detection can be performed in real time. At this time, the artificial intelligence used may be a classification algorithm.

The step (S103) of identifying the position and pose information of the capsule endoscope can identify the position and pose information of the capsule endoscope (300). Here, the position means the approximate position of the capsule endoscope (300) inside the stomach, and the pose information means the position, angle (rotation angle), or movement direction of the camera built into the capsule endoscope (300). The step (S103) of identifying the position and pose information of the capsule endoscope can identify the pose information based on the degree of movement and degree of rotation of the capsule endoscope extracted by matching the extracted feature points after extracting the feature points of the current frame and the previous frame of the captured image. The identified position and pose information can be used for generating a movement path, etc.

The step of extracting the current position information and current pose information of the capsule endoscope can extract the current position information and current pose information of the capsule endoscope from the identified position and pose information of the capsule endoscope when any one landmark among multiple landmarks is detected.

The step of generating a movement path of the capsule endoscope (S105) can generate the movement path of the capsule endoscope based on at least one of position information and pose information of the capsule endoscope, a landmark, or a 3D upper gastrointestinal endoscope map.

The step (S105) of generating a movement path of the capsule endoscope can generate a movement path along which the capsule endoscope (300) can explore the periphery of the detected landmark when a landmark is detected. The step (S105) of generating a movement path of the capsule endoscope can generate a movement path along which the capsule endoscope (300) moves to the location of the next landmark to be explored when a landmark is detected. The step (S105) of generating a movement path of the capsule endoscope can generate a movement path along which the capsule endoscope (300) explores the periphery of the detected landmark when a landmark is detected and then moves to the location of the next landmark to be explored.

The step (S105) of generating a movement path of the capsule endoscope can generate a movement path of the capsule endoscope so that the capsule endoscope moves sequentially according to the order of preset landmarks. In summary, assuming that there are first to third landmarks and that the search is set to proceed in the order of the first landmark to the third landmark, the step (S105) of generating a movement path of the capsule endoscope can generate a movement path so that the capsule endoscope (300) searches around the first landmark and then moves to the location of the second landmark. Subsequently, the step (S105) of generating a movement path of the capsule endoscope can generate a movement path so that, when the capsule endoscope (300) reaches the location of the second landmark, it searches around the second landmark and then moves to the location of the third landmark. The step (S105) of generating a movement path of the capsule endoscope can end the search when the capsule endoscope (300) reaches the location of the third landmark.

In the step (S105) of generating a movement path of the capsule endoscope, the generated movement path can be provided three-dimensionally so that the capsule endoscope (300) can be controlled in the internal space of the stomach. In other words, the movement path does not mean a path of movement on a two-dimensional plane, but can mean a path of movement in space three-dimensionally. The step (S105) of generating a movement path of the capsule endoscope can generate a movement path that takes into account the internal position, angle, and movement direction of the capsule endoscope (300) so that the capsule endoscope (300) can move in the internal space of the stomach.

The step (S105) of generating a movement path of the capsule endoscope can provide a movement path that serves as a guide when an examiner or doctor operates the capsule endoscope (300) from the outside. The step (S105) of generating a movement path of the capsule endoscope can provide a movement path to a control device if there is a separate control device that automatically controls the capsule endoscope (300).

The step (S105) of generating a movement path of the capsule endoscope may include a step of generating a path for moving to the first landmark based on current location information and current pose information of the capsule endoscope if any landmark is not the first landmark, and a step of generating a path for moving from the first landmark to a second landmark based on current location information and current pose information of the capsule endoscope if any landmark is the first landmark.

FIG. 5 illustrates a path search process when a landmark is first detected according to an embodiment of the present invention. Referring to FIG. 5, when a landmark is first detected (S301) in an image captured by a capsule endoscope (300), a step (S105) of generating a movement path of the capsule endoscope can determine whether the detected landmark is a first landmark, which is a starting point of the search (S303). A step (S105) of generating a movement path of the capsule endoscope can generate a movement path to the location of the first landmark if the first detected landmark is not the first landmark, which is a starting point of the path search (S305). At this time, in preparation for a case of moving to a location other than the first landmark, it is determined once again whether the detected landmark is the first landmark after moving to the location of the first landmark.

The step of generating a movement path of a capsule endoscope (S105) can generate a path to move to a location of a second landmark, which is a point following the first landmark, if the landmark is a first landmark, which is a starting point of the path search (S307). The step of generating a movement path of a capsule endoscope (S105) can generate a path to move to a location of a second landmark, which is a point following the first landmark, after generating a path to move to a location of the second landmark (S309). The step of generating a movement path of a capsule endoscope (S105) can end path guidance when the last landmark is reached (S311).

FIG. 6 illustrates a capsule endoscope that moves sequentially according to the order of preset landmarks according to an embodiment of the present invention. Referring to FIG. 6, the step (S105) of generating a movement path of the capsule endoscope may generate a movement path of the capsule endoscope (300) in the order of Cardia & Fundus (Ex. 1), Greater Curvature (Ex. 2), Angulus (Ex. 3), and Pylorus (Ex. 4) according to the order of preset landmarks. At this time, it can be confirmed that each landmark is aligned with the pre-generated 3D landmark data, thereby improving visibility.

The step (S105) of generating a movement path of a capsule endoscope uses the position and pose information of the capsule endoscope (300) extracted when generating the movement path, but when generating a movement path to move to an internal location of the stomach that has not been restored from the three-dimensionally restored stomach data, the movement path can be generated based on a three-dimensional gastrointestinal endoscope map.

The step (S105) of generating a movement path of the capsule endoscope can generate a movement path in a three-dimensional space so that the capsule endoscope (300) can be manipulated in the three-dimensionally restored stomach data. Therefore, the step (S105) of generating a movement path of the capsule endoscope can generate a movement path that takes into account the position, angle, and movement direction of the capsule endoscope (300) inside the stomach so that the capsule endoscope (300) can move in the space inside the stomach by considering the position and pose information of the capsule endoscope (300).

Meanwhile, according to an embodiment of the present invention, the step (S101) of acquiring a photographed image can restore the image photographed by the capsule endoscope (300) into 3D stomach data in real time, so that when the capsule endoscope (300) reaches the first landmark, 3D stomach data for generating a movement path to move to another landmark may not be constructed. In this case, the step (S105) of generating the movement path of the capsule endoscope can generate a movement path to move to the next landmark based on a 3D gastrointestinal endoscope map constructed in advance. Since the shape of the inside of the stomach of most people and the location and shape of landmarks are almost similar, the 3D gastrointestinal endoscope map can be utilized.

In the step (S105) of generating a movement path of a capsule endoscope, a movement path is generated using a three-dimensional upper endoscope map. Then, when the three-dimensional upper data for the movement path is restored, the movement path can be corrected based on the three-dimensionally restored upper data.

The step (S107) of controlling the capsule endoscope can control the capsule endoscope (300) to move along the movement path generated in the step (S105) of generating the movement path of the capsule endoscope. The control step (S107) can control the capsule endoscope (300) by changing the magnetic field of a control device that controls the operation of the capsule endoscope (300) using an external magnetic field.

The storage step (not shown) can store various results generated in performing the present invention, such as 3D stomach data restored in real time, detected landmark images, data in which landmarks and 3D restored stomach data are aligned. The existing gastrointestinal endoscope has a problem in that a specific part cannot be checked again after examination, but the storage step can solve the problem that there may be parts that are missed due to not being able to check depending on the doctor's individual ability, since the images captured by the capsule endoscope (300) and the reconstructed images are stored.

Fig. 7 shows a configuration diagram of a three-dimensional navigation system (100) according to another embodiment of the present invention. Referring to Fig. 7, the three-dimensional navigation system (100) may include an image receiving unit (110), a calculation unit (130), a search unit (150), and a control unit (170).

The image receiving unit (110) can obtain an image captured from a capsule endoscope inserted into the human body. The image receiving unit (110) can receive an image captured from the capsule endoscope (300) via wireless communication. The capsule endoscope (300) can capture several to several tens of still images or moving images per second and transmit data to the image receiving unit (110) in real time. The image receiving unit (110) can perform the operation of the step (S101) of acquiring the captured image described above.

The calculation unit (130) can detect landmarks from images acquired from a capsule endoscope and determine position and pose information of the capsule endoscope from the three-dimensionally restored data. The calculation unit (130) can perform the operation of the step (S103) of determining position and pose information of the capsule endoscope as described above.

The search unit (150) can detect landmarks from images acquired from a capsule endoscope and determine the position and pose information of the capsule endoscope from the three-dimensionally restored data. The search unit (150) can perform the operation of the step of generating the movement path of the capsule endoscope as described above.

The control unit (170) can control the capsule endoscope to move along the movement path generated by the search unit. The control unit (170) can perform the operation of the step (S107) of controlling the capsule endoscope described above.

The storage unit (not shown) can store various results generated in performing the present invention, such as 3D stomach data restored in real time, detected landmark images, and data in which landmarks and 3D restored stomach data are aligned. The storage unit (not shown) can perform the operation of the storage step described above.

Although the present invention has been described in detail through representative examples above, those skilled in the art will understand that various modifications can be made to the above-described embodiments without departing from the scope of the present invention. Therefore, the scope of the rights of the present invention should not be limited to the described embodiments, but should be determined by all changes or modifications derived from the claims and equivalent concepts as well as the claims.

The purpose of the present invention is to provide a three-dimensional stomach navigation method and system for a capsule endoscope that guides a three-dimensional path in real time so that a capsule endoscope positioned inside a human body can explore the inside of the stomach.

In addition, the present invention aims to restore capsule endoscope images captured in real time to 3D data so as to enable 3D navigation path search, and to visualize landmarks in the 3D data by aligning the restored 3D data with 3D landmark data.

In addition, the present invention aims to guide the capsule endoscope along a path to the next landmark location to enable the capsule endoscope to sequentially explore the inside of an organ.

Claims (13)

A step of acquiring images taken from a capsule endoscope inserted into the human body; A step of detecting a landmark from an image acquired from the capsule endoscope and determining the position and pose information of the capsule endoscope, and A step of generating a movement path of the capsule endoscope based on at least one of the position information and pose information of the capsule endoscope, the landmark, or a 3D upper gastrointestinal endoscope map. A three-dimensional upper navigation method of a capsule endoscope including a. In the first paragraph, The steps for obtaining the above-mentioned recorded video are: Step of forming landmark-aligned 3D stomach data by aligning the 3D restored stomach data with the parasitic 3D landmark data. A three-dimensional upper navigation method of a capsule endoscope including a. In the second paragraph, The above three-dimensionally restored data is, A three-dimensional navigation method for a capsule endoscope, characterized in that the method comprises extracting feature points of the current frame and the previous frame of the captured image and then forming the feature points in real time by matching the extracted feature points. In the second paragraph, The step of forming the three-dimensional data above with the above landmarks aligned is: A three-dimensional upper navigation method of a capsule endoscope, characterized in that the parasitic three-dimensional landmark data is moved and rotated so that the Euclidean distance between the points constituting the parasitic three-dimensional landmark data and the three-dimensionally restored upper data is the closest, and the parasitic three-dimensional landmark data is overlapped with the three-dimensionally restored upper data. In the second paragraph, The step of forming the three-dimensional data above with the above landmarks aligned is: A three-dimensional navigation method for a capsule endoscope, characterized in that the three-dimensional data in which the landmarks are aligned are visualized so that the aligned landmark portion is distinguished from other portions. In the first paragraph, The step of calculating the position and pose information of the above capsule endoscope is: A three-dimensional navigation method for a capsule endoscope, characterized in that the pose information is determined based on the degree of movement and rotation of the capsule endoscope by extracting feature points of the current frame and the previous frame of the captured image and then matching the extracted feature points. In the first paragraph, The step of calculating the position and pose information of the above capsule endoscope is: Step of inputting the above-mentioned captured image into the artificial intelligence trained with a 2D landmark data set A three-dimensional upper navigation method of a capsule endoscope, characterized by including a. In the first paragraph, A step of extracting current position information and current pose information of the capsule endoscope from the identified position and pose information of the capsule endoscope when one of several landmarks is detected. A three-dimensional upper navigation method of a capsule endoscope including a. In Article 8, The step of generating the movement path of the above capsule endoscope is: If any of the above landmarks is not the first landmark, a step of generating a path to the first landmark based on the current location information and current pose information of the capsule endoscope, and If any one of the above landmarks is a first landmark, a step of generating a path for moving from the first landmark to the second landmark based on the current location information and current pose information of the capsule endoscope. A three-dimensional upper navigation method of a capsule endoscope including a. In the first paragraph, The step of generating the movement path of the above capsule endoscope is: A three-dimensional navigation method for a capsule endoscope, characterized in that a movement path of the capsule endoscope is generated so that the capsule endoscope moves sequentially according to the order of preset landmarks. In the first paragraph, The above landmarks are, A method for three-dimensional gastric navigation using a capsule endoscope, characterized in that it includes at least one region within the stomach among the cardia, fundus, greater curvature, angulus, and pylorus. An image receiving unit that acquires images captured from a capsule endoscope inserted into the human body; A computational unit that detects landmarks from images acquired from the capsule endoscope and determines position and pose information of the capsule endoscope from the three-dimensionally restored data; A navigation unit that generates a movement path of the capsule endoscope based on at least one of the position information and pose information of the capsule endoscope, the landmark, or a 3D upper gastrointestinal endoscope map; and A three-dimensional navigation system, characterized by including a control unit that controls the capsule endoscope to move along the movement path generated by the search unit. In Article 12, The above control unit, A three-dimensional navigation system characterized by controlling the operation of the capsule endoscope using an external magnetic field.

Images