WO2011071363A2

WO2011071363A2 - System and method for visualizing and learning of human anatomy

Info

Publication number: WO2011071363A2
Application number: PCT/MY2010/000258
Authority: WO
Inventors: Weng Onn Kow; Mohammad Reza Beikzadeh; Dickson Lukose; Pramod G.Bagali; Rajan Bharadwaj; Singh Armandeep; Abdul Qayoom; Sachim Angrish
Original assignee: Mimos Berhad
Priority date: 2009-12-09
Filing date: 2010-11-10
Publication date: 2011-06-16
Also published as: MY148824A; WO2011071363A3

Abstract

A system for semantic images browsing and navigation for visualizing and learning of human anatomy comprises an information database (13) with records of medical knowledge bases; a first display interface (11) which allows user to manipulate an ontology for requested concepts on subject of interest, wherein the concepts and its knowledge bases in the information database (13) having tags which are harmonized and synchronized; and a base server (15) connected to the information database (13) and first display interface (11). The system (10) further comprises a graphical database (14) containing three-dimensional images of human anatomy; a second display interface (12) which allows user to manipulate images of subject of interest, wherein the images having tags which are harmonized and synchronized; and a graphic engine (16) connected to the graphical database (14) and second display interface (12), the graphic engine (16) is linked to the base server (15) for fetching the corresponding concept or image upon user requests.

Description

System and Method for Visualizing and Learning of Human

Anatomy

Field of Invention

The present invention relates generally to a field of medical knowledge, more particularly to a system and method for visualizing and learning of human anatomy. Background of the Invention

Medical knowledge is very complex and scattered in various subject textbooks. Health care students (Medical, Dental, Nursing, ' Physiotherapy) require years of practice and experience to link all the information before they can apply it correctly on patients. One of the main challenges for medical students is the process of linking all the knowledge that they have gained from their studies and applying it to specific cases. Typically, learning in medical college is done in isolated silos of subjects such as Anatomy, Pathology, Microbiology and Pharmacology. Each student has to mentally put all this information together to get from diagnosis of a problem to the best treatment and management. Knowledge of the human anatomy is very important to these health care students, and currently the only way to learn it practically is through the use of life-sized models or working with cadavers. However, models and cadavers are expensive, and the knowledge is left behind when the student leaves the lab or operating room.

Often, they are provided with a series of computed tomography (CT) scans of the human body and these scans are of a two dimensional nature. Then they need to imagine or extrapolate to the corresponding three dimensional structures. They have difficulties achieving a conceptual understanding of 3D anatomy and misconceptions about physiological phenomena are persistent and hard to address. Medical information system with advanced browsing capabilities plays an increasingly important role in medical training, research and diagnostics. Therefore a 3D imaging system must be used in this field to facilitate a better understanding of complex phenomena of the anatomy knowledge.

The objective of the present invention is to provide a system which uses semantics to link all the basic medical information together so that the user able to quickly find the information needed and also gives a better visual understanding on how all the concepts are linked. It is desirable to use medical images to support browsing, searching, visualizing and querying of medical databases. Another objective of the present invention is to provide the system which includes linked of anatomy, physiology and biomedical knowledge to a 3D imaging system that allows them view and manipulate virtual parts of the human body in real time. These two solutions have not been adequately addressed prior to this invention.

Summary of the Invention

In the present invention, a system for semantic images browsing and navigation for visualizing and learning of human anatomy comprises an information database containing records of medical knowledge bases; a first display interface which allows user to manipulate an ontology for requested concepts on subject of interest, wherein the concepts and its knowledge bases in the information database having tags which are harmonized and synchronized; a base server connected to the information database and first display interface, a graphical database containing three-dimensional images of human anatomy; a second display interface which allows user to manipulate images of subject of interest, wherein the images having tags which are harmonized and synchronized, and a graphic engine (16) connected to the graphical database and second display interface) , the graphic engine is linked to the base server for fetching the corresponding concept or image upon user requests . Brief Description of the Drawings

Other objects, features, and advantages of the invention will be apparent from the following description when read with reference to the accompanying drawings. In the drawings, wherein like reference numerals denote corresponding parts throughout the several views:

Figure 1 shows a diagram of a system for semantic images browsing and navigation for visualizing and learning of human anatomy of the present invention where Figure la showing the user using the semantic browser to interact with the system and Figure lb showing the user using the 3D visualizer;

Figure 2 depicts a scalable architecture of the present system;

Figure 3 shows a plurality of individual MSCT images generated by scanning a human head;

Figure 4 shows a generated 3D skull using the present system;

Figure 5 illustrates a screenshot of system with the semantic browser window where on the right shows the skull concept and

) the 3D application on the left showing a rendered image of the skull along with annotations;

Figure 6 is a diagram showing the semantic technology platform for virtual anatomy architecture; and

Figure 7 is a flowchart of a method for semantic images browsing and navigation for visualizing and learning of human anatomy of the present invention.

De-tailed Description of the Preferred Embodiments

In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be understood by those of ordinary skill in the art that the invention may be practiced without these specific details. In other instances, well-known methods, procedures and/or components have not been described in detail so as not to obscure the invention. Reference will now be made in detail to the preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings.

In the preferred embodiment, a system and method for semantic images browsing and navigation for visualizing and learning of human anatomy in relation to physiological functions is disclosed. According to an embodiment of the present disclosure, the system (10) provides the user with a multimodal approach to access medical knowledge. The system (10) includes a 3D imaging program which takes DICOM data from multi slice CT scans which are two dimensional of the entire human body and reconstructs a three dimensional image body structure. These 3D visuals are built from real scans where the characteristics and dimensions of visualized human body and organs are more accurate than drawings in books or molded models. User (20) may interact with the visuals of the system (10) and manipulate the images by zooming in and out, rotating and even slicing the anatomical part being ^' displayed through the use of a graphical semantic browser (11) or a 3D visualizer (12) as shown in Figure la and lb respectively. The image can also be annotated with text and saved to a file for future reference. The system (10) also includes a semantic web of an information database (13) where the necessary medical knowledge bases from various medical topics or subjects are kept therein and a graphical database (14) where the 3D images of the human body and organs are kept therein. The knowledge is linked semantically around a human anatomy to allow user to explore and navigate through the linked knowledge for both learning and reference purposes. The images data is acquired from the MSCT scan of the human body. Figure 3 shows the MSCT images from scanning a human head. The data is generated in a DICOM format which is the international standard for Diagnostic Communication in Medicine. The images will be on a specific plane mostly axial. There are numerous parameters that determine the data acquisition including slice thickness, slice interval, vp, mA, milliseconds of exposure etc. DCM parser will then separate the non co-axial slices from the rest. The images data is then rendered in 3D after preprocessing of this large data, which can be up to approximately 10 to 20 gigabytes of size. The proprietary large data management (LDM) algorithms do an efficient handling of this big data, . which cuts down time to few seconds. This is a real volumetric data, and the user can intuitively interact with it by operations such as zooming, panning and rotating can be done easily on the image rendered. Figure 4 shows the high definition visuals of the 3D skull generated by the system (10) .

The semantic browser (11) is a graphical user interface that takes a semantic web in the form of an RDF file and shows portions of it in the form of nodes and links. The semantic browser (11) includes three main views graphical, HTML and type hierarchy. In graphical view, concepts and relations are shown as nodes and links between nodes. It uses a focus node concept, where the currently selected concept and its links are shown in detail while the other nodes and links are minimized into the background. The depiction of concepts and relations as nodes and links between the nodes is not new. However, other features to make it even easier for the user to understand and relate the knowledge they see in the browser are added into the present system. For example, different coloured nodes and icons are used to represent different classes of concepts. Similarly for the links between the nodes. Self-configuring layouts offer a balance between clarity and depth of knowledge displayed in the browser (11) .

All navigation can be done using the mouse button with the ability to change the graphical display to the user' s liking such as the ability to zoom in and out, and to drag the nodes and screen around. The user also can limit what knowledge he wants to view by filtering out all relations that he is not interested in. He can hide and show relations and nodes at any time.

The concepts shown by the semantic browser (11) need not be just in text form. If the knowledge base contains links to artifacts such as documents, web pages and multimedia files, the browser (11) can launch the relevant application to view these artifacts when they are selected. In HTML view, concepts are represented as a dynamically generated HTML page with relations being shown as header titles and the target concepts listed beneath the headers as links or text.

And in type hierarchy view, the knowledge base is shown as a three dimensional tree. The current selected concept is shown along with its child concepts (if any) and a path through its ancestors to the root of the tree.

The browser (11) includes a keyword search function that allows the user to search for any concept just by typing in part of the concept name. The browser (11) will list all concepts that match the typed characters. For more advanced users, there is the semantic query function (17) that allows the user to type in structured natural language questions such as "What is the common location of Osteoid Osteoma?" and get the answer returned in natural language if it can be found in the knowledge base as shown in Figure 6. The user can then select the anatomy portion mentioned in the answer, and the image of the part is shown along with its semantic we .

The knowledge information in the semantic web (13) and the images in the graphical database (14) are tagged with tags and synchronized to each other so that when the user selects a concept in the browser (11), then the relevant anatomy image (if any) is shown in the visualizer (12). Similarly, selecting a portion of the anatomy in the visualizer (12) leads to semantic information about that concept to be shown in the browser window (11) as sho.wn in Figure 5.

Therefore, when a user manipulates the ontology via the semantic browser (11), the tags of the central concept on display in the semantic browser (11) will be transmitted by a base server (15) which is linked to a graphic engine (16) to fetch the corresponding image from the graphical database (14) based on the tags and displays it on the 3D visualizer (12) as shown in Figure la. On the other hand, when the user manipulates the image via the 3D visualizer (12), then the graphic engine (16) will send the tags associated to the image to the base server (15) which will retrieve the corresponding concept and render it on the semantic browser (11) . For an example, when the user views the concept of "tibia" from the browser, the tibia bone is shown in 3D and similarly, if the user chooses to view the heart in the 3D viewer, the concept of the heart and its related concepts are shown in ₍the browser .

The system (10) also has unlimited scalability. It can work as a stand alone, as well as for large number of concurrent users on a distributed architecture as shown in Figure 2. The present system (10) is using a method for inter-linking and navigating bi-directionally between the 3D reconstructed digital human anatomical and sub-cellular DICOM volumes. The method includes providing a plurality of 3D reconstructed human gross anatomy and sub-cellular histological as well as histopathological volume for medical teaching and training of a structure of interest against physiological functionality or deficits in human which is to be con-currently displayed on a semantic web using a service oriented architecture. The digital anatomical reconstructed 3D structure comprising a plurality of intensities corresponding to a domain of points on a 3-dimensional grid.

Then the structure of interest for each image is tagged and annotated to form a 3D mesh of points. After that, the motion and orientation between tags or annotations and structure of interest within the 3D grid are harmonized and synchronized. User may constantly point at the same structure of interest at all times irrespective of position in the 3D grid axis or motion without the annotations being hide, lost or distorted from viewing. The annotations always orient itself to the region of the body where it was pointing originally against it intended region. The spatial position of annotations is synchronized to its specific region in 3D volume which makes it to point to the original region. Each mesh is then oriented, organized and aligned wherein a registration transformation between each pair of 3D reconstructed volume is calculated. Various measurement parameters as comparison between abnormal ■ structures of interest and normal human anatomical structure within the aligned mesh are then calculated and displayed. The normal and abnormal with pathophysiological findings of 3D reconstructed human body volumes from plurality of images can be retrieved and displayed. A feature vector for each structure of interest in the plurality of the gross and sub-cellular human anatomy and pathophysiological volumes is then calculated. Using boosting to train A classifier is then trained using boosting to categorize key meaning in the medical ontology, comment and annotation mapped against the structure of interest in the plurality of 3D reconstructed gross human bodies into a predefined category based on the meaning, association and complexity of each structure of interest, where the classifier is adapted to segmenting a corresponding structure of interest from a plurality of 3D reconstructed human bodies. Finally a plurality of bi-directional links between anatomical or pathological information in association with the structure of interest; in a semantic web of medical ontology; to its corresponding MPR (Multiplanar reformation) is created and the links which using the service oriented architecture are adapted to facilitate navigation, derivation of information and related medical knowledge through the established bidirectional linkage.

Pixel segmentation, isolation and enhancement for small features and structure or point of interest in a 3D reconstructed gross and sub-cellular human anatomy and pathophysiology volume that needs clarity and greater definition in multi-dimensional data can be provided to define small anatomical feature demarcation that correspond closely to those selected by the user but does so with less complexity. Hounsfield units in a plurality of 3D reconstructed human gross anatomy and sub-cellular structures allowing comparison of pathopysiological between abnormal and normal structures are calculated and compared. The density and volume calculation can be provided to identify the pathological changes.

When a semantic query to view anatomical or pathological information in association with the structure of interest is received, the query to identify one or more ontology or overall meaning in the query is then parsed. The meaning to the query is then mapped and flagged to a corresponding structure in the anatomical structure against spatial information to associate such medical information to the location within a physiological system, where the 3D reconstructed gross human anatomical and pathopysiological and histological structure images are associated with at least one of the links to the corresponding structure in the image. Then at least one link to inter-relate, inter-link bi-directionally is performed and the human anatomical structure of interest mapped against the established semantic web medical ontology is displayed.

When a new 3D reconstructed gross and sub-cellular human anatomy, histology and pathophysiology volume of the structure of interest is requested, a list of highly customized forensic pathology-specific pre-sets within the tool library is used to quickly narrow down to an area or structure of interest. A pre-set and pre-trained classifier is also used to isolate and identify the structure of interest from the volume within a plurality of 3D reconstructed human bodies. Then bi- directional links between the structure of interest to a corresponding semantic web of medical ontology are created and the links which using the service oriented architecture are adapted to facilitate navigation and derivation of information and related medical knowledge through the 3D reconstructed human gross anatomy and pathophysiological structures to the structure of interest. The cause-and-effect medical knowledge specifically within the possible and ideal pharmaceutical intervention, potential microbiological aetiology causing pathophysiological changes grossly and sub-cellularly as well as traumatic events that affects these changes using the semantic ontology model and associating these set of domain knowledge and expert opinions bi-directionally to a specific anatomical or histological structure of interest are also developed. The system (10) allows saving, storing, viewing and replaying chronological video for segmentation, association between the structure of interest against the medical ontology and knowledge base of the structure of interest and the links to the corresponding structure in a metafile. The snapshots capturing both the medical ontology and structure of interest volume are used to be tagged and annotated for future reference or replaying in sequential manner or as a part of a case file to narrate the chronology for sharing and learning.

Figure 7 shows a flowchart of the steps in using the present system (10) . When a user using the system (10) for requesting concepts, the concepts from the ontology will be loaded (21) onto the semantic browser (11) . Then the image of the full human anatomy will be loaded (22) onto the 3D visualizer (12) . User may choose to manipulate (23) the ontology from the semantic browser (11) or manipulate (24) the image from the 3D visualizer (12) . If user uses the semantic browser (11), the base server (15) will identify (25) the central concept displayed on the semantic browser (11) and send the associated tags to the graphic engine (16) and the corresponding image will be displayed on the 3D visualizer (12) . And if the user uses the 3D visualizer (12), the graphic engine will then identify (26) the tags associated to the image displayed on the 3D visualizer (12) and send these tags to the base server, which will use these tags to load the necessary concepts to be displayed on the semantic browser (11) . Based on learning theories underpinning problem based learning, 3D visualizations can be implemented in the existing curricula of the medical, nursing, first aid and physiotherapy programs. This can be used for theory lectures, practical demonstrations and tutorial sessions. This is also ideal for Self-study as students cannot have access to a dissection cadaver for 24 hours a day.

The 3D visualization solution will definitely stimulate the students to understand more and help them to get insights about pathological, microbiological variations and different organs size, space extent and relation to each other. The virtual dissections will give a clearer picture than ordinary dissections and the possibility to turn structures around will be self instructive. Since this is based on authentic, true human scanning data, it will add a new dimension of learning material in anatomy, physiology and probably also pathophysiology .

As will be readily apparent to those skilled in the art, the present invention may easily be produced in other specific forms without departing from its essential characteristics. The present embodiments is, therefore, to be considered as merely illustrative and not restrictive, the scope of the invention being indicated by the claims rather than the foregoing description, and all changes which come within therefore intended to be embraced therein.

Claims

1. A system (10) for semantic images browsing and navigation for visualizing and learning of human anatomy comprising:

an information database (13) containing records of medical knowledge bases;

a first display interface (11) which allows user (20) to manipulate an ontology for requested concepts on subject of interest, wherein said concepts and its knowledge bases in said information database (13) having tags which are harmonized and synchronized;

a base server (15) connected to said information database (13) and first display interface (11);

a graphical database (14) containing three-dimensional images of human anatomy;

a second display interface (12) which allows user to manipulate images of subject of interest, wherein said images having tags which are harmonized and synchronized; and

a graphic engine (16) connected to said graphical database ^» (14) and second display interface (12), said graphic engine (16) is linked to said base server (15) for fetching the corresponding concept or image upon user requests.

2. The system as claimed in claim 1, wherein said first display interface (11) is a semantic browser.

3. The system as claimed in claim 2, wherein said semantic browser (11) includes three main views which are graphical, HTML and hierarchy.

4. The system as claimed in claim 3, wherein said in graphical view of semantic browser (11), concepts are in the form of nodes and links between nodes, wherein said nodes and links are differentiated with colors and icons.

5. The system as claimed in claim 3, wherein said in HTML view of semantic browser (11), concepts are represented as a dynamically generated HTML page with relations being shown as header titles and the target concepts listed beneath the headers as links or text.

6. The system as claimed in claim 3, wherein said in hierarchy view of semantic browser (11), concepts are shown as a three-dimensional tree where the current selected concept is shown along with its child concepts and a path through its ancestors to the root of the tree.

7. The system as claimed in claim 3, wherein said semantic browser (11) includes a keyword search function that allows the user to search for any concept just by typing in part of the concept name.

8. The system as claimed in claim 3, wherein said semantic browser (11) may also includes a semantic query function (17) that allows the user to type in structured natural language questions and gets the answer returned in natural language if it can be found in the knowledge base.

9. The system as claimed in claim 1, wherein said second display interface (11) is a three-dimensional visualizer.

10. A method for interlinking and navigating bi-directionally between a first display interface (11) and second display interface (12), said method comprising the steps of:

providing a plurality of 3D reconstructed human gross anatomy and sub-cellular histological as well as histopathological volume for medical teaching and training of a structure of interest against physiological functionality or deficits in human which is to be con-currently displayed on a semantic web using a service oriented architecture, wherein each said anatomical reconstructed 3D structure comprising a plurality of intensities corresponding to a domain of points on a 3- dimensional grid;

tagging and annotating each image of said structure of interest wherein a 3D mesh of points is formed;

harmonizing and synchronizing motion and orientation between tags and structure within the 3D grid; orienting, organizing and aligning each mesh wherein a registration transformation between each pair of 3D reconstructed volume is calculated;

calculating and displaying various measurement parameters as comparison between abnormal said structures of interest and normal human anatomical structure within said aligned mesh; retrieving and displaying normal and abnormal with pathophysiological findings of 3D reconstructed human body volumes from plurality of images;

calculating a feature vector for each structure of interest in said plurality of the gross and sub-cellular human anatomy and pathophysiological volumes;

training a classifier by using boosting to categorize key meaning in the medical ontology, comment and annotation mapped against said structure of interest in the plurality of 3D reconstructed gross human bodies into a pre-defined category based on the meaning, association and complexity of each said structure of interest, wherein said classifier is adapted to segmenting a corresponding structure of interest from a plurality of 3D reconstructed human bodies; and

creating bi-directional links between anatomical or pathological information in association with the said structure of interest; in a semantic web of medical ontology; to its corresponding multiplanar reformation (MPR) wherein said links using the service oriented architecture are adapted to facilitate navigation, derivation of information and related medical knowledge through the established bidirectional linkage.

11. The method as claimed in claim 10, wherein said method further comprising the step of providing pixel segmentation, isolation and enhancement for small features and structure or point of interest in a 3D reconstructed gross and subcellular human anatomy and pathophysiology volume that needs clarity and greater definition in multi-dimensional data which defines small anatomical feature demarcation that correspond closely to those selected by the user but does so with less complexity.

12. The method as claimed in claim 10, wherein said method further comprising the step of calculating and comparing Hounsfield units in a plurality of 3D reconstructed human gross anatomy and sub-cellular structures allowing comparison of pathopysiological between abnormal and normal structures, density and volume calculation can be provided to identify said pathological changes.

13. The method as claimed in claim 10, wherein said method further comprising the steps of:

pointing annotations and comments to a specific structure of interest onto a 3D reconst ucted human gross anatomical volume; and synchronizing the motion and orientation of the annotations to the specific structure of interest with reference to the 3D grid in any axis to reflect annotations and comments to constantly point at the same structure of interest at all times irrespective of position in the 3D grid axis or motion without the annotations being hide, lost or distorted from viewing, wherein said annotations always orient itself to the region of the body where it was pointing originally against it intended region.

14. The method as claimed in claim 10, wherein said method further comprising the steps of:

receiving a semantic query to view anatomical or pathological information in association with the said structure of interest;

parsing said query to identify one or more ontology or overall meaning in said query; mapping and flagging the meaning to a query to a corresponding structure in said anatomical structure against spatial information to associate such medical information to the location within a physiological system; wherein said 3D reconstructed gross human anatomical and pathopysiological and histological structure images are associated with at least one of said links to the corresponding structure in said image; and

following said at least one link to inter-relate, inter-link bi-directionally and display said human anatomical structure of interest mapped against the established semantic web medical ontology.

15. The method as claimed in claim 10, wherein said method further comprising the steps of:

providing a new 3D reconstructed gross and sub-cellular human anatomy, histology and pathophysiology volume of said structure of interest; creating bi-directional links between said structure of interest to a corresponding medical ontology in the semantic web;

using a list of highly customized forensic pathology-specific pre-sets within the tool library to quickly narrow down to an area or structure of interest;

using a pre-set and pre-trained classifier to isolate and identify said structure of interest from said volume within a plurality of 3D reconstructed human bodies; and

creating bi-directional links between said structure of interest to a corresponding semantic web of medical ontology; wherein said links using the service oriented architecture are adapted to facilitate navigation and derivation of information and related medical knowledge through said 3D reconstructed human gross anatomy and pathophysiological structures to said structure of interest ; Developing cause-and-effect medical knowledge specifically within the possible and ideal pharmaceutical intervention, potential microbiological aetiology causing pathophysiological changes grossly and sub- cellularly as well as traumatic events that affects these changes using the semantic ontology model and associating these set of domain knowledge and expert opinions bi- directionally to a specific anatomical or histological structure of interest.

16. A method for semantic images browsing and navigation for visualizing and learning of human anatomy comprising the steps of: loading (21) the concepts from the ontology onto the semantic browser (11) ;

loading (22) an image of the full human anatomy onto the 3D visualizer (12);

manipulating (23) the ontology from the semantic browser (11); and

identifying (25) the central concept displayed on the semantic browser (11) and send the associated tags to the graphic engine (16) and the corresponding image will be displayed on the 3D visualizer (13) by a base server (15) .

17. A method for semantic images browsing and navigation for visualizing and learning of human anatomy comprising the steps of: loading (21) the concepts from the ontology onto the semantic browser ( 11 ) ;

loading (22) an image of the full human anatomy onto the 3D visualizer (12);

manipulating (24) the image from the 3D visualizer (12); identifying (26) the tags associated to the image displayed on the 3D visualizer (12) by a graphic engine (16) and send these tags to the base server (15), which will use these tags to load the necessary concepts to be displayed on the semantic browser ( 11 ) .