WO2024117555A1 - Method of training classification model for image data and device therefor - Google Patents

Abstract

Disclosed are a method of training, by a computing device, a classification model that outputs classification information on the basis of image data, and a device therefor according to various embodiments. A method and a device therefor are disclosed, the method comprising the steps of: pre-training an encoder on the basis of first training data including first image data and metadata; and training a classification model including the pre-trained encoder on the basis of second training data including label information, wherein the encoder is pre-trained on the basis of first feature vectors extracted from the first image data and third feature vectors obtained by applying the characteristic of meta-vectors related to the metadata to the first feature vectors.

Description

Method for learning a classification model for image data and device for the same

This topic relates to a classification model based on an artificial neural network, a method for learning an encoder included in the classification model, and a device for the same.

Currently, medical images such as CT (computed tomography) are widely used for diagnosis by analyzing lesions. For example, chest CT images are frequently used for interpretation because they can observe abnormalities inside the body, such as the lungs, bronchial tubes, and heart.

Some findings that can be interpreted through chest CT images are not easy to interpret, so even radiologists can distinguish their characteristics and shapes only after many years of training, so they can be easily overlooked by human doctors. In particular, when the level of difficulty in interpreting lung nodules is high, it can become a problem because doctors may miss them even if they pay a high level of attention. In order to assist in the interpretation of images that can be easily overlooked by humans, the need for computer-aided diagnosis (CAD) has emerged, but conventional CAD technology only assists the doctor's judgment in a very limited area. .

In order to solve these problems, methods for performing reading and/or diagnosis using artificial neural network models are being actively studied. For example, the reading and/or diagnosis may be performed through classification of medical images based on an artificial neural network model, object detection, extraction of object boundaries, and registration of different images. Here, the neural network model is a supervised learning algorithm inspired from the structure of neurons in biology. The basic operating principle of a neural network model is to predict the optimal output value for input values by interconnecting multiple neurons. From a statistical perspective, a neural network model can be viewed as a projective trace regression that takes a non-linear function to a linear combination of input variables. In particular, convolution neural networks (CNN) are most widely used to extract features such as the above-mentioned objects from medical images.

The problem to be solved is a method for learning the classification model so as to minimize the impact of the metadata characteristics even when image data with biased metadata characteristics due to limitations in label information are used as learning data; and The goal is to provide a device for this.

The technical problems are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below.

According to one aspect, a method of training a classification model in which a computing device outputs classification information based on image data includes pre-training an encoder based on first training data including first image data and metadata, and a label. A step of training a classification model including the pre-trained encoder based on second training data including information, wherein the encoder includes first feature vectors extracted from the first image data, and the first feature. Vectors may be pre-trained based on third feature vectors that apply characteristics of meta vectors related to the meta data.

Alternatively, the encoder may be pre-trained based on a first loss function for the first feature vectors and the third feature vectors.

Alternatively, the encoder may be pre-trained by further considering second feature vectors extracted from a momentum encoder that moves average the parameters of the encoder.

Alternatively, the momentum encoder may extract the second feature vectors from augmented image data that augments the first image data.

Alternatively, the encoder is characterized in that it is pre-trained based on a second loss function for the first feature vector and the second feature vector and a first loss function for the first feature vector and the third feature vector. .

Alternatively, the computing device obtains the third feature vectors by inputting the metadata and the first feature vector to a metadata fusion module, and the encoder performs momentum metadata fusion to move average the parameters of the metadata fusion module. It is characterized in that it is pre-trained by further considering the fourth feature vectors output by the module.

Alternatively, the encoder is pre-trained based on a first loss function for the first feature vectors and the third feature vectors and a third loss function for the third feature vectors and the fourth feature vectors. It is characterized by

Alternatively, the meta vectors are generated for M channels to correspond to N channels related to the first feature vectors based on the meta data, and the characteristics of the meta vectors are determined by the average and variance of the elements of each meta vector. Characterized by being determined based on at least one.

Alternatively, the computing device obtains the third feature vectors by inputting the metadata and the first feature vector to a metadata fusion module, and the encoder performs momentum metadata fusion to move average the parameters of the metadata fusion module. It is characterized in that it is pre-trained by further considering the fourth feature vectors output by the module.

Alternatively, the encoder is pre-trained based on a first loss function for the first feature vectors and the third feature vectors and a third loss function for the third feature vectors and the fourth feature vectors. It is characterized by

Alternatively, the meta vectors are generated for M channels to correspond to N channels related to the first feature vectors based on the meta data, and the characteristics of the meta vectors are determined by the average and variance of the elements of each meta vector. Characterized by being determined based on at least one.

Alternatively, the third feature vectors are generated through adaptive instance normalization (AdaIN) for the first feature vectors, and the scale factor and bias of the adaptive instance normalization are based on the characteristics of the meta vectors. It is characterized by being determined.

Alternatively, the label information may include a label value for the location of the lung nodule, the type of the lung nodule, or the segmentation of the lung nodule for the second image data included in the second learning data.

Alternatively, the first feature vectors may be feature vectors obtained by applying a multi-layer perceptron (MLP) and additional weights to the extracted feature vectors extracted from the first image data by the encoder.

According to another aspect, a method of a computing device outputting classification information related to lung nodules based on image data using a classification model includes receiving image data, and inputting the image data into the classification model to provide classification information. A step of outputting, wherein the classification model includes an encoder pre-trained based on first training data including first image data and metadata, and the encoder generates a first feature vector extracted from the first image data. and third feature vectors obtained by applying characteristics of meta vectors generated from the meta data to the first feature vectors.

According to another aspect, a computing device that trains a classification model that outputs classification information based on image data includes a communication unit connected to external devices, and a processor connected to the communication unit, wherein the processor receives information obtained through the communication unit. Pre-training an encoder based on first training data including first image data and metadata, and classifying the pre-trained encoder based on second training data including label information acquired through the communication unit. Learning a model, the processor operates the encoder based on first feature vectors extracted from the first image data and third feature vectors obtained by applying characteristics of meta vectors related to the meta data to the first feature vectors. It can be pre-trained.

Various embodiments pre-train the encoder so that the characteristics of the meta data can be taken into account, thereby preventing the influence of the characteristics of the meta data even when it is based on training data for image data in which the characteristics of the meta data are biased due to limitations in label information. The classification model can be trained to minimize damage.

The effects that can be obtained in various embodiments are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

The drawings attached to this specification are intended to provide an understanding of the present invention, show various embodiments of the present invention, and together with the description of the specification, explain the principles of the present invention.

Figure 1 is a diagram to explain the structure of CNN, an artificial neural network.

Figure 2 illustrates a computing device that trains a classification model based on an artificial neural network.

3 and 4 are diagrams for explaining a method by which a computing device pre-trains the encoder.

FIG. 5 is a diagram illustrating a method by which a computing device self-directs pre-training an encoder based on contrastive learning.

Figures 6 and 7 are diagrams for explaining a method by which a computing device applies metadata characteristics to extracted feature vectors using a metadata fusion module.

FIGS. 8 and 9 are diagrams for explaining a method by which a computing device trains a classification model including a pre-trained encoder.

FIG. 10 is a diagram illustrating a method by which a computing device trains an encoder and a classification model including the encoder.

The detailed description of the present invention described below refers to the accompanying drawings, which show by way of example specific embodiments in which the present invention may be practiced to make clear the objectives, technical solutions and advantages of the present invention. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention.

As used throughout the description and claims of this specification, the term “image” or “image data” refers to multidimensional data consisting of discrete image elements (e.g., pixels in two-dimensional images and voxels in three-dimensional images). refers to

For example, “image” may refer to a two-dimensional image corresponding to a slide of a certain tissue observed using a microscope, but “image” is not limited to this and is a (cone-beam) computerized image. It may be a medical image of a subject collected by computed tomography, magnetic resonance imaging (MRI), ultrasound, or any other medical imaging system known in the art. Images may also be provided in non-medical contexts, such as remote sensing systems, electron microscopy, etc.

Throughout the description and claims herein, 'image' is a term that refers to a visible image (e.g., displayed on a screen) or a digital representation of an image.

For convenience of explanation, in the drawings presented, slide image data is shown as an exemplary image modality. However, those skilled in the art will know that the imaging formats used in various embodiments of the present invention include X-ray images, MRI, CT, PET (positron emission tomography), PET-CT, SPECT, SPECT-CT, MR-PET, 3D ultrasound images, etc. It will be understood that the format includes but is not limited to the format listed as an example.

Medical images described throughout the detailed description and claims of this specification may comply with the 'DICOM (Digital Imaging and Communications in Medicine)' standard. The DICOM standard is a general term for several standards used for digital image expression and communication in medical devices. The DICOM standard is announced by a joint committee formed by the American College of Radiology (ACR) and the National Electrical Manufacturers Association (NEMA).

In addition, the medical images described throughout the detailed description and claims of this specification may be stored or transmitted through the 'Picture Archiving and Communication System (PACS)', and the medical image storage and transmission system complies with the DICOM standard. It may be a system that stores, processes, and transmits medical images appropriately. Medical images acquired using digital medical imaging equipment such as .

In addition, throughout the detailed description and claims of this specification, 'learning' or 'learning' is a term referring to performing machine learning through procedural computing, and is intended to refer to mental operations such as human educational activities. This is not true, and training is used in the generally accepted sense of machine learning. For example, 'deep learning' refers to machine learning using deep artificial neural networks. A deep neural network is a machine that automatically learns the characteristics of each data by learning a large amount of data in a structure consisting of a multi-layer artificial neural network, and through this, learns in a way that minimizes the error in the objective/loss function, that is, classification accuracy. It is a learning model and can extract and classify various levels of features, from low-level features such as points, lines, and surfaces to complex and meaningful high-level features.

And throughout the description and claims herein, the word 'comprise' and variations thereof are not intended to exclude other technical features, attachments, components or steps. Additionally, 'one' or 'one' is used to mean more than one, and 'another' is limited to at least the second or more.

Other objects, advantages and features of the invention will appear to those skilled in the art, partly from the specification and partly from practice of the invention. The examples and drawings below are provided by way of example and are not intended to limit the invention. Accordingly, the details disclosed in this specification with respect to specific structures or functions should not be construed in a limiting sense, but are merely representative examples that provide guidance for those skilled in the art to variously practice the present invention with any detailed structures that are practically suitable. It should be interpreted as basic data.

Moreover, the present invention encompasses all possible combinations of the embodiments presented herein. It should be understood that the various embodiments of the present invention are different from one another but are not necessarily mutually exclusive. For example, specific shapes, structures and characteristics described herein with respect to one embodiment may be implemented in other embodiments without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description that follows is not intended to be taken in a limiting sense, and the scope of the invention is limited only by the appended claims, together with all equivalents to what those claims assert, if properly described. Similar reference numbers in the drawings refer to identical or similar functions across various aspects.

In this specification, unless otherwise indicated or clearly contradictory to the context, items referred to in the singular include plural unless the context otherwise requires. Additionally, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description will be omitted.

Hereinafter, in order to enable those skilled in the art to easily practice the present invention, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

Figure 1 is a diagram to explain the structure of CNN, an artificial neural network.

CNN (Convolutional Neural Network) is a type of ANN (Artificial Neural Network) that uses convolution operations. To extract image features, CNN calculates a convolution by traversing the input image (or input data) through a filter, and uses the calculation result of the convolution to generate a feature map or activation map. You can.

The CNN maintains the shape of the input and output data of each layer, effectively recognizes features of adjacent images while maintaining spatial information of the image, extracts and learns image features using multiple filters, and collects and collects features of the extracted images. It has the advantage of being able to optionally include a strengthening pooling layer and using filters as shared parameters, requiring very few learning parameters compared to general artificial neural networks.

Specifically, referring to FIG. 1, the CNN may include a feature extraction area for extracting features from an input image (or input data) and an image classification area for classifying the extracted features. The feature extraction area may include at least one convolution layer that finds features of the image while minimizing the number of shared parameters using a filter. Alternatively, the feature extraction area may further include at least one pooling layer that enhances and collects the features. That is, the pooling layer may be omitted.

The convolution layer is a layer that reflects the activation function after applying a filter to the input image (or input data). One or more filters may be applied to the input image flowing into the convolution layer. The one filter can configure a channel of the feature map. For example, when n filters are applied to the convolutional layer, the feature map or the activation map (or output data) has n channels.

The pooling layer is an optional layer located after the convolution layer. The pooling layer receives the output data of the convolution layer as input and can be used as a layer to reduce the size of the output data or emphasize specific data. Methods for processing the pling layer include max pooling, average pooling, and min pooling. The pooling layer has no parameters to learn, can reduce the size of the matrix, and does not change the number of channels.

The CNN can adjust the shape or size of output data depending on the filter size, stride, whether padding is applied, and the maximum pooling size, and determines the channel through the number of filters. You can.

The FC layer is a layer for recognition and classification operations and is a pre-combined layer used to connect each layer in an existing neural network. The FC layer may perform a one-dimensional flattening operation on the two-dimensional array form of the output data in the feature extraction area. The one-dimensional, flattened output data can be classified through the SoftMAx function.

Such a CNN-based model (hereinafter referred to as a classification model) can be trained to make a predetermined diagnosis or interpretation from medical image data such as MRI or CT. Hereinafter, a method of learning a classification model based on an artificial neural network to perform specific classification and/or diagnosis (e.g., Nodule detection, FPR, Classification, Segmentation) based on medical image data such as MRI, CT, etc. will be described in detail.

Figure 2 illustrates a computing device that trains a classification model based on an artificial neural network.

Referring to FIG. 2, the computing device 20 includes a communication unit 21 and a processor 22, and can communicate directly or indirectly with an external computing device (not shown) through the communication unit 21. Here, the communication unit 21 may correspond to or include a transceiver capable of transmitting and receiving requests and responses to and from other computing devices.

Specifically, computing device 20 is a device that may include typical computer hardware (e.g., a computer processor, memory, storage, input and output devices, and other components of a conventional image processing device; such as a router, switch, etc. electronic communication devices (electronic information storage systems, such as network-attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., devices that enable computing devices to function in certain ways); The desired system performance may be achieved by using a combination of instructions.

This communication unit 21 is capable of transmitting and receiving requests and responses to other image processing devices that are linked to it. As an example, such requests and responses may be made through the same TCP (transmission control protocol) session. It is not limited, and may be transmitted and received as, for example, a UDP (user datagram protocol) datagram. Additionally, in a broad sense, the communication unit 21 may include a pointing device such as a keyboard or mouse, other external input devices, printers, displays, and other external output devices for receiving commands or instructions.

Additionally, the processor 22 of the computing device 20 may include a micro processing unit (MPU), a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a cache memory, and a data bus. It may include hardware configuration such as (data bus). In addition, it may further include an operating system and software configuration of an application that performs a specific purpose. The processor 22 may execute instructions to perform the functions of the neural network described below.

According to one example, the processor 22 may train the artificial neural network-based classification model to perform a predetermined task (detection, classification, and segmentation of lesions). For example, the processor 22 may include medical images and training data including label information corresponding to each medical image in the classification model or the classification model so that the difference between the label information and output information is minimized. The classification model can be learned by adjusting the parameters of the encoder. Here, the classification model may be based on an artificial neural network capable of extracting features or feature vectors from the medical images. For example, the artificial neural network may include Deep Neural Network (DNN), Recurrent Neural Network (RNN), Long Short -It may be at least one of various types of neural networks, such as Term Memory models (LSTM), BRDNN (Bidirectional Recurrent Deep Neural Network), and Convolutional Neural Networks (CNN).

Meanwhile, due to limitations in labeling medical images, training data including label information for learning the classification model may be insufficient. For example, the learning data may include medical images that are biased toward medical images (or image data) related to specific metadata. Here, the metadata includes characteristic information about the patient's unique information, such as the patient's age and gender, and protocol information of the image data acquisition device (e.g., CT vendor, X-ray strength, X-ray beam type, injection method, etc.) It can be. In this case, when trained with the training data, the classification model may be learned without properly distinguishing the characteristics of the medical image by the specific metadata, and the classification model may use metadata different from the specific metadata. Classification performance for medical images may be greatly reduced.

Therefore, even if the classification model is learned based on medical images biased toward the specific metadata, a method is needed to minimize the influence of the specific metadata. As a method for this, the computing device 20 may first perform a process of pre-training the encoder included in the classification model to extract features for the medical image by considering the influence of the metadata. . The dictionary learning can be performed using a large number of images without separate label information.

Hereinafter, a method of self-directed pre-training of the encoder included in the classification model so that the computing device 20 can consider the influence of the metadata will be described in detail.

3 and 4 are diagrams for explaining a method by which a computing device pre-trains the encoder.

The computing device compares the first feature vectors output from the encoder 111 with third feature vectors that reflect the features of metadata for the first feature vectors and trains the encoder 111 to pre-train or self-directed dictionary. It can be learned (self-supervised learning).

Specifically, referring to FIG. 3, the encoder 111 may be self-directed pre-trained based on the first stream (stream, 110) and the second stream (120). A medical image is input into the first stream 110, and first feature vectors (f ₁ (x)) can be extracted through the encoder 111. For example, the encoder 111 extracts N-dimensional feature vectors or first feature vectors (f) from the input medical image or image data (hereinafter referred to as image data) based on the CNN described with reference to FIG. ₁ (x)) can be extracted.

Metadata for the image data may be input into the second stream 120, and extracted feature vectors extracted from the first stream 110 may be input. The meta data and extracted feature vectors are output as third feature vectors (f ₃ (x)) in which the meta data is meaningfully applied or fused to the extracted feature vectors through the meta data fusion module 121. It can be. For example, the meta data fusion module 121 may generate extension meta vectors having channels (N) and corresponding channels (M) of the extracted feature vectors based on the meta data, and the extension meta vectors Based on this, calculate or determine feature values related to metadata to be applied to each of the channels (N) of the extracted feature vectors, and apply the metadata to the extracted feature vectors based on the calculated or determined feature values for each channel. Characteristics related to can be applied. That is, the metadata can be applied to the extracted feature vectors in a feature space (feature space or CNN feature space) for the extracted feature vectors through the metadata fusion module 121. Specific details related to this will be described in detail later based on FIGS. 6 and 7.

Next, the _computing device encoder _{111 and} / Alternatively, the metadata fusion module 121 can be trained. The first loss function (L _ma ) can be defined as Equation 1 below. In this case _, the computing device operates _the encoder ₍ 111) and/or the meta data fusion module 121 may be trained.

[Equation 1]

At this time, the encoder 111 is pre-trained by the metadata fusion module 121 by considering the relationship with the third feature vectors (f ₃ (x)) in which the metadata is meaningfully reflected, so that the metadata The extracted feature vectors can be effectively extracted from the image data by considering the characteristics of the image data according to the influence.

Or, referring to FIG. 4, the extracted feature vectors extracted by the encoder 111 from the first stream 110 are additionally input by the first projector (projector ₁ , 112) and the first predictor (predictor ₁ , 113) The dimensionality can be reduced and additional weights added. Specifically, the first projector 112 applies a multi-layer perceptron network (MLP) to the extracted feature vectors (or CNN features) extracted through the encoder 111 to create a feature space with a smaller dimension. space) or can be converted to feature vectors. The first predictor 113 may apply additional weights to the converted feature vectors and output first feature vectors (f ₁ (x)). In this case, the generalization characteristics of the extracted feature vectors are maintained even if their dimensionality is reduced through the first projector 112, and the first loss function can be calculated more effectively through this reduction in dimensionality. In addition, in the second stream 120, the fusion feature vectors, which are extracted feature vectors to which the metadata is applied or reflected, are additionally input by the second projector (projector ₂ , 122) and the second predictor (predictor ₂ , 123) to increase the dimensionality. It can be reduced, added with additional weights, and converted to third feature vectors (f ₃ (x)).

In this case, the computing device minimizes the first loss function based on the first feature vectors (f ₁ (x)) and the third feature vectors (f ₃ (x)) as described with reference to FIG. 3. Parameters of the encoder 111, the first projector 112, the first predictor 113, the metadata fusion module 121, the second projector 122, and/or the second predictor 123 may be adjusted. Meanwhile, the second projector 122 and the first projector 112 can reduce the dimension to feature vectors having corresponding dimensions.

In this way, the encoder 111 is trained to extract feature vectors by considering the characteristics of metadata in the pre-learning process, so even if metadata for image data input in the inference step for a specific task is not separately input, the input The feature vectors can be extracted by considering the characteristics of metadata in the image data.

Below, a method of self-directed pre-training of the encoder 111 more effectively and stably by additionally using a momentum encoder will be described.

FIG. 5 is a diagram illustrating a method by which a computing device self-directs pre-training an encoder based on contrastive learning.

Referring to FIG. 5, in the first stream 110, the input image data is converted or output into second feature vectors (f ₂ (x)) through the momentum encoder 151 and the first momentum projector 152. Additional processes may be added. Here, the momentum encoder 151 may have its parameters updated through moving averaging or exponential moving averaging for the parameters (or weights, parameter values) of the encoder 111, and the first The momentum projector 152 may be a projector whose parameters are updated through moving averaging of the parameters of the first projector 112.

Specifically, the computing device can (arbitrarily) augment the input image data (for example, using image transformation methods such as image rotation, left/right change, and cropping). In this case, the computing device may input the input image data (or original image) to the encoder 111 and input the augmented augmented image view to the momentum encoder 151. The encoder 111 may extract first extracted feature vectors, which are CNN feature vectors, from the original image. The first extracted feature vectors may be converted to or output as first feature vectors f ₁ (x) through the first projector 112 and the first predictor 113, as described with reference to FIG. 4 . The momentum encoder 151 may extract second extraction feature vectors from the augmented image. The second extracted feature vectors may be input to the first momentum projector 152 and converted to or output as second feature vectors (f ₂ (x)). In this case _, the _computing device encoder 111 ₍ Alternatively, the first projector 112 and/or the first predictor 113 may be pre-trained.

Specifically, the second loss function (L _c ) may be defined as a function of mean squared error as shown in Equation 2.

[Equation 2]

Meanwhile, as described above, the momentum encoder 151 updates the parameters by moving averaging or exponential moving averaging the parameters of the encoder 111 adjusted by learning, and the first momentum projector 152 ), the parameters may be updated through moving averaging for the parameters of the first projector 112 adjusted by the learning. For example, when the encoder 111 sets the first parameter value in the first process and sets the second parameter value in the second process, the momentum encoder 151 sets its parameter value (initial setting value) and the first parameter value. P, which is the value of the parameter, is calculated by weighting the parameters (e.g., more weight can be added to the value of the parameter (e.g., adding 0.99 to the value of the parameter and 0.01 to the value of the first parameter)). It can be updated, and its own parameter value can be updated again by weighting the second parameter value and the updated P value in the second process.

In addition, the second stream 120 is data in which the input metadata is converted or output into fourth feature vectors (f ₄ (x)) through the momentum meta data fusion module 161 and the second momentum projector 162. More flows may be included. In this case, the computing device may input the input metadata to the metadata fusion module 121 and the momentum metadata fusion module 161. The metadata fusion module 121 may generate first fusion feature vectors by applying or fusing characteristics of the metadata to the first extracted feature vectors based on the first extended metavectors generated based on the metadata. there is. The first fusion feature vectors may be converted or output into third feature vectors (f ₃ (x)) through the second projector 122 and the second predictor 123. Additionally, the momentum meta data fusion module 161 generates second extended meta vectors based on the meta data and applies the characteristics of the meta data to the second extracted feature vectors based on the second extended meta vectors. Alternatively, second fusion feature vectors may be generated by fusing. The second fusion feature vectors may be converted or output into fourth dimension-reduced feature vectors (f ₄ (x)) through the second momentum projector 162. In this case, the computing device uses the metadata fusion module 121 based on the third loss function (Lm) for the third feature vectors (f ₃ (x)) and the fourth feature vectors (f ₄ (x)) ) (and/or the second projector 122 and the second predictor 123) can be trained. For example, the third loss function (Lm) can be defined as Equation 3 below.

[Equation 3]

Furthermore, the momentum metadata fusion module 161 and/or the second momentum projector 162 perform moving averaging or exponential moving on the parameters of the metadata fusion module 121 and/or the second projector 122. Parameters can be updated through averaging (exponential moving averaging).

Finally, the computing device may pre-train the encoder 111 based on the fourth loss function (L _ma ) based on Equation 4 below.

[Equation 4]

In this way, the computing device may pre-train the encoder 111 so that the fourth loss function (L _ma ) is also minimized in the learning process based on the first stream 110 and the second stream 120. In this case, the encoder 111 extracts feature vectors from the image feature space (latent space from the CNN encoder), which can be mapped to a multi-dimensional meta-data space. It can be pre-trained so that

Below, we will describe in detail how the metadata fusion module 121 meaningfully applies metadata characteristics to the extracted feature vectors.

Figures 6 and 7 are diagrams for explaining a method by which a computing device applies metadata characteristics to extracted feature vectors using a metadata fusion module.

Referring to FIG. 6, a computing device can input metadata into the metadata fusion module. In this case, the meta data fusion module can generate or configure the meta vector 11 based on the meta data. The meta vector 11 may be composed of a vector of size k (or a vector with k elements) based on the number of patient characteristics and device characteristics included in the meta data. Alternatively, the meta vector 11 may be composed of one channel as a vector of size k (or a vector with k elements) based on the number of patient characteristics and device characteristics included in the meta data.

Next, the meta data fusion module generates extension meta vectors (13) for M or M channels to correspond to the N channels of the extracted feature vectors extracted by the encoder based on the meta vector (11). can do. Specifically, the metadata fusion module extends the metavector 11 into extension metavectors 13 for the M channels, which are the plurality of channels, using at least one full connection layer (FC layer) and a linear layer. You can do it. For example, when the number of channels of extracted feature vectors extracted from the encoder is N, the meta data fusion module can expand and generate meta vectors (11) into extension meta vectors (13) for M channels. Here, M may correspond to N or 2N.

Referring to FIG. 7, the meta data fusion module 121 may calculate or determine a characteristic value (Y) corresponding to or to be applied to each of the N channels based on the extended meta vectors.

Specifically, the meta data fusion module 121 may calculate characteristic values for one channel based on two or more of the extended meta vectors. Specifically, the metadata fusion module 121 may calculate the variance of the elements of one extended metavector and the average of the elements included in the other extended metavector as the characteristic value for one channel (first method ). In this first method, the computing device can calculate and determine characteristic values (Y) for each of the N channels.

Alternatively, the meta data fusion module 121 may calculate a characteristic value for one channel based on one of the extended meta vectors (second method). Specifically, the metadata fusion module can calculate the average and variance (one average and variance set) of the elements of one extended metavector, and determine the average and variance set as the characteristic value for one channel. For example, the meta data fusion module 121 may generate the N extension meta vectors corresponding to extracted feature vectors (i.e., N extracted feature vectors) for N channels based on the meta data, By calculating the average and variance of each element of the N extended meta-vectors, N average and variance sets can be calculated. The one average and variance set can be determined as a characteristic value for one channel.

) 및 분산 (

)에 기반하여 추출 특징 벡터들 (X)을 정규화시킬 수 있다. 이하에서는, 설명의 편의를 위해 상기 정규화된 추출 특징 벡터들을 제1 정규 특징 벡터 (x_norm)들로 정의하고, 특성 값 (Y)에 기반하여 채널 별로 AdaIN된 제1 정규 특징 벡터(x_norm)들을 제2 정규 특징 벡터들로 정의한다.Next, the metadata fusion module 121 performs adaptive instance normalization (AdaIN) on each of the extracted feature vectors (X) based on the characteristic values for each channel. ), the characteristic value (Y) can be applied to each channel. For this purpose, the characteristic value (Y) for each channel can be used as a scale factor and bias for adaptive instance normalization. For example, referring to Equation 5 and/or Equation 6 below, the variance among the characteristic values (Y) for one channel is used as a scale factor for adaptive instance normalization, and the average is used for the adaptive instance normalization. It can be determined by the bias for. Meanwhile, the metadata fusion module 121 may perform normalization (x _norm ) on the extracted feature vectors (X) before applying the feature value (Y) for each channel. For example, the metadata fusion module 121 extracts the average (

) and variance (

), the extracted feature vectors (X) can be normalized based on Hereinafter, for convenience of explanation, the normalized extracted feature vectors are defined as first normal feature vectors (x _norm ), and the first normal feature vector (x _norm ) is AdaIN for each channel based on the feature value (Y). are defined as second normal feature vectors.

When the characteristic value (Y) for one channel is calculated from two or more meta vectors according to the first method described above, the meta data fusion module 121 calculates the first normal value for each channel using Equation 5 below. The second normal feature vectors can be generated by AdaIN the feature vectors (x _norm ).

[Equation 5]

Here, σ(y ₁ ) may be the variance of one meta vector, and μ(y ₂ ) may be the average of another meta vector.

When the characteristic value (Y) for one channel is calculated from one meta vector according to the second method described above, the meta data fusion module 121 calculates the first normal feature for each channel using Equation 6 below. Second normal feature vectors can be generated by AdaIN vectors (x _norm ).

[Equation 6]

Here, σ(y ₁ ) and μ(y ₁ ) can be the variance and mean for one metavector,

The metadata fusion module 121 may generate or output third feature vectors (Z) by linearizing (or activating ReLU) the second regular feature vectors through a ReLU layer. Here, the third feature vectors (Z) are the extracted feature vectors to which the characteristics of the meta data are applied, and may correspond to f ₃ (x) described in FIGS. 2 to 5.

In this way, the metadata fusion module generates extended metavectors that extend metavectors based on the metadata to the same channel as the extracted feature vectors, and effectively combines the characteristics of the extended metavectors through AdaIN into the extracted feature vectors. can be reflected in

FIGS. 8 and 9 are diagrams for explaining a method by which a computing device trains a classification model including a pre-trained encoder.

Referring to FIG. 8, the computing device may pre-train the encoder to consider the characteristics of metadata through first training data including first image data and metadata related thereto, as described above.

Next, the computing device may train a classification model including the pre-trained encoder using second training data including second image data and label information corresponding thereto. At this time, learning the classification model is fine-tuned to perform downstream tasks or specific tasks (lung nodules detection for lung CT images, false positive removal, classification of lung nodules, segmentation of lung nodules). It may be a learning process of (fine-tuning). Meanwhile, the above-described first image data and the second image data may include CT images of the lungs.

Referring to FIG. 9, the computing device may receive second learning data including label information related to the specific task and the second image data. Meanwhile, as described above, the second learning data may be biased toward image data having specific metadata characteristics. The computing device may input the second image data to the pre-trained encoder 211. The pre-trained encoder 211 may extract feature vectors from the second image data. The extracted feature vectors are input to the projector 212 to reduce the dimension in the feature space as described above, and the extracted feature vectors with reduced dimension are input to the predictor 213 to produce classification information or information related to the specific task. It can be output as a classification value. In this case, the computing device may adjust parameters of the classification model (e.g., parameters of the encoder, projector, and/or predictor) such that the loss function (L _ft ) based on the classification information and label information is minimized.

At this time, the pre-trained encoder 211 is pre-trained to extract extracted feature vectors that can be mapped to the multi-dimensional meta data space by considering the characteristics of the meta data, and the predictor 213 uses the input It is possible to output classification information that is not affected by the characteristics of metadata related to image data. In other words, even if the learning data is biased toward image data having the characteristics of specific metadata, the computing device uses the pre-trained encoder 211 to create a classification model 200 to prevent performance degradation due to the biased characteristics of the specific metadata. It can be learned.

The classification model that has completed this learning process can output classification information related to lung nodules, etc. from image data (online inference step). Specifically, when image data is input, the computing device may obtain or output classification information for the image data using the learned classification model. At this time, even if the classification model was learned with second training data that includes image data with specific metadata characteristics in a biased manner, the classification model is based on the metadata that minimizes the impact on the metadata characteristics of the image data. Classification information that is robust to characteristics can be output.

FIG. 10 is a diagram illustrating a method by which a computing device trains an encoder and a classification model including the encoder.

Referring to FIG. 10, the computing device may receive first learning data for dictionary learning of the encoder (S101). Here, the first learning data may include first image data and metadata corresponding to the first image data, as described above. The first training data may include a significant amount of first image data in that label information corresponding to the first image data is not required.

The computing device may self-directly pre-train the encoder based on the first learning data (S103). Specifically, the computing device may pre-train the encoder based on first feature vectors (or extracted feature vectors) and third feature vectors extracted by the encoder. For example, the computing device may pre-train the encoder by adjusting the parameters of the encoder to minimize the first loss function based on Equation 1. Meanwhile, as shown in FIGS. 4 and 5, the first feature vectors have a multi-layer perceptron (MLP) and additional weights applied to the extracted feature vectors extracted from the first image data by the encoder. These may be applied feature vectors. The third feature vectors may also be feature vectors obtained by applying a multi-layer perceptron and additional weights to meta feature vectors, as shown in FIGS. 4 and 5.

Alternatively, the computing device may pre-train the encoder by further reflecting or considering second feature vectors extracted from a momentum encoder that moves averages the parameters of the encoder. In this case, the computing device is based on a second loss function for the first feature vectors and the second feature vectors, and a first loss function for the first feature vectors and the third feature vectors. The encoder can be pre-trained.

Alternatively, the computing device may pre-train the encoder by further considering fourth feature vectors output by a momentum metadata fusion module that moves averages the parameters of the metadata fusion module. In this case, the computing device is based on a first loss function for the first feature vectors and the third feature vectors, and a third loss function for the third feature vectors and the fourth feature vectors. The encoder can be pre-trained.

Alternatively, the computing device may pre-train the encoder by considering all of the first feature vectors, the second feature vectors, the third feature vectors, and the fourth feature vectors. In this case, the computing device may pre-train the encoder based on the fourth loss function defined according to Equation 4.

When pre-training of the encoder based on at least one of the above-described loss functions is completed, the computing device may receive second training data including second image data and label information for the second image data ( S105). As described above, the label information may include a label value for the location of the lung nodule, the type of the lung nodule, or the segmentation of the lung nodule for the second image data included in the second learning data. In this case, the computing device may input the second image data into a classification model including the pre-trained encoder.

Next, the computing device can fine tune or learn the classification model to output classification information for a specific task (S107). The classification model may be trained so that the loss function for the classification information (or classification value) and label information (or label value) output from the classification model is minimized.

When image data (including Query CT, etc.) is input, the classification model learned in this way outputs classification information about the location of the lung nodule, the type of the lung nodule, or the segmentation of the lung nodule based on the image data. (online inference step). Specifically, the computing device can output classification information that is not affected by the characteristics of metadata for input image data using a classification model including a pre-trained encoder.

Based on the description of the above embodiments, those skilled in the art will understand that the method and/or processes of the present invention, and the steps thereof, can be realized with hardware, software, or any combination of hardware and software suitable for a specific application. The point can be clearly understood. The hardware may include general-purpose computers and/or dedicated computing devices or specific computing devices or special features or components of specific computing devices. The processes may be realized by one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors, or other programmable devices, with internal and/or external memory. Additionally, or alternatively, the processes may be configured as an application specific integrated circuit (ASIC), programmable gate array, programmable array logic (PAL), or to process electronic signals. It can be implemented with any other device or combination of devices. Furthermore, the subject matter of the technical solution of the present invention or the parts contributing to the prior art may be implemented in the form of program instructions that can be executed through various computer components and recorded on a machine-viewable recording medium. The machine-viewable recording medium may include program instructions, data files, data structures, etc., singly or in combination. The program instructions recorded on the machine-viewable recording medium may be specially designed and configured for the present invention or may be known and usable by those skilled in the art of computer software. Examples of machine-viewable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs, DVDs, and Blu-rays, and magneto-optical media such as floptical disks. (magneto-optical media), and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include storing and compiling or interpreting them for execution on any one of the foregoing devices, as well as any heterogeneous combination of processors, processor architectures, or combinations of different hardware and software, or any other machine capable of executing the program instructions. machine code, which may be created using a structured programming language such as C, an object-oriented programming language such as C++, or a high- or low-level programming language (assembly language, hardware description languages, and database programming languages and techniques); This includes not only bytecode but also high-level language code that can be executed by a computer using an interpreter.

Accordingly, in one aspect according to the present specification, when the above-described methods and combinations thereof are performed by one or more computing devices, the methods and combinations of methods may be implemented as executable code that performs the respective steps. In another aspect, the method may be practiced as systems that perform the steps above, and the methods may be distributed in several ways across devices or all functions may be integrated into one dedicated, standalone device or other hardware. In another aspect, the means for performing steps associated with the processes described above may include any of the hardware and/or software described above. All such sequential combinations and combinations are intended to be within the scope of this specification.

For example, the hardware device may be configured to operate as one or more software modules to perform processing according to the present disclosure, and vice versa. The hardware device may include a processor such as an MPU, CPU, GPU, or TPU combined with a memory such as ROM/RAM for storing program instructions and configured to execute instructions stored in the memory, and external devices and signals. It may include a communication unit capable of sending and receiving. In addition, the hardware device may include a keyboard, mouse, and other external input devices to receive commands written by developers.

In the above, the present invention has been described with specific details such as specific components and limited embodiments and drawings, but this is only provided to facilitate a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , a person with ordinary knowledge in the technical field to which the present invention pertains can make various modifications and variations from this description.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the patent claims described below as well as all modifications equivalent to or equivalent to the scope of the claims fall within the scope of the spirit of the present invention. They will say they do it.

Such equivalent or equivalent modifications will include, for example, logically equivalent methods that can produce the same results as performing the method according to the present specification, and the spirit and scope of the present invention should not be limited by the foregoing examples, but should be understood in the broadest sense permissible by law.

Embodiments of the present invention as described above can be applied to various medical devices.

Claims (13)

In a method for a computing device to learn a classification model that outputs classification information based on image data, Pre-training an encoder based on first training data including first image data and metadata; and A step of training a classification model including the pre-trained encoder based on second training data including label information, The encoder is pre-trained based on first feature vectors extracted from the first image data and third feature vectors obtained by applying characteristics of meta vectors related to the meta data to the first feature vectors. According to paragraph 1, The method, characterized in that the encoder is pre-trained based on a first loss function for the first feature vectors and the third feature vectors. According to paragraph 1, The method is characterized in that the encoder is pre-trained by further considering second feature vectors extracted from a momentum encoder that moves averages the parameters of the encoder. According to paragraph 3, The method is characterized in that the momentum encoder extracts the second feature vectors from augmented image data that augments the first image data. According to paragraph 3, The encoder is pre-trained based on a second loss function for the first feature vector and the second feature vector and a first loss function for the first feature vector and the third feature vector. . According to paragraph 1, The computing device inputs the metadata and the first feature vector to a metadata fusion module to obtain the third feature vectors, The method is characterized in that the encoder is pre-trained by further considering fourth feature vectors output by a momentum metadata fusion module that moves averages the parameters of the metadata fusion module. According to clause 6, The encoder is pre-trained based on a first loss function for the first feature vectors and the third feature vectors and a third loss function for the third feature vectors and the fourth feature vectors. to do, how to do. According to paragraph 1, The meta vectors are generated for M channels to correspond to N channels associated with the first feature vectors based on the meta data, Characteristics of the meta vectors are determined based on at least one of the mean and variance of the elements of each meta vector. According to paragraph 1, The third feature vectors are generated through adaptive instance normalization (AdaIN) for the first feature vectors, Characterized in that the scale factor and bias of the adaptive instance normalization are determined based on the characteristics of the meta vectors. According to paragraph 1, The label information includes a label value for the location of the lung nodule, the type of the lung nodule, or the segmentation of the lung nodule for the second image data included in the second learning data. According to paragraph 1, The first feature vectors are feature vectors obtained by applying a multi-layer perceptron (MLP) and additional weight to the extracted feature vectors extracted from the first image data by the encoder. In a method where a computing device outputs classification information related to lung nodules based on image data using a classification model, Receiving image data as input; and Including the step of inputting the image data into the classification model and outputting classification information, The classification model includes a pre-trained encoder based on first training data including first image data and metadata, The encoder is pre-trained based on first feature vectors extracted from the first image data and third feature vectors obtained by applying characteristics of meta vectors generated from the meta data to the first feature vectors. In a computing device that trains a classification model that outputs classification information based on image data, A communication unit connected to external devices; and Includes a processor connected to the communication unit, The processor pre-trains the encoder based on first training data including first image data and metadata acquired through the communication unit, and based on second learning data including label information obtained through the communication unit. Train a classification model including the pre-trained encoder, The processor pre-trains the encoder based on first feature vectors extracted from the first image data and third feature vectors obtained by applying characteristics of meta vectors related to the meta data to the first feature vectors. Device.

Images