WO2020159241A1 - Method for processing image, and apparatus therefor - Google Patents

Abstract

The present disclosure relates to an artificial intelligence (AI) system utilizing a machine learning algorithm such as deep learning or the like, and an application thereof. A method by which an electronic apparatus processes an image comprises the operations of: obtaining a plurality of training image sets corresponding to a plurality of types of target objects, wherein feature points forming a pre-set structure are labeled in a training image in the training image sets; generating a first artificial intelligence model for determining a sample structure based on the labeled feature points, by using the training image in the training image sets; identifying a face from the training image modified on the basis of the sample structure; and training a second artificial intelligence model for verifying the first artificial intelligence model on the basis of an image reconstructed from the modified training image and the training image before being modified.

Description

Method and apparatus for processing image

The present disclosure relates to a method and apparatus for processing an image, and more particularly, to a method and apparatus for extracting a face of a target object by converting an image including the target object into a sample structure.

Artificial Intelligence (Artificial Intelligence) system is a system that the machine learns, judges, and becomes smart, unlike the existing Rule-based smart system. As the artificial intelligence system is used, the recognition rate is improved and the user's taste can be more accurately understood, and the existing rule-based smart system is gradually being replaced by a deep learning-based artificial intelligence system.

Artificial intelligence technology is composed of machine learning (deep learning) and elemental technologies utilizing machine learning.

Machine learning is an algorithm technology that classifies/learns the characteristics of input data by itself, and element technology is a technology that utilizes machine learning algorithms such as deep learning, linguistic understanding, visual understanding, reasoning/prediction, knowledge expression, motion control, etc. It consists of technical fields.

The various fields in which artificial intelligence technology is applied are as follows. Linguistic understanding is a technology that recognizes and applies/processes human language/characters, and includes natural language processing, machine translation, conversation system, question and answer, speech recognition/synthesis, etc. Visual understanding is a technology that recognizes and processes objects as human vision, and includes object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, and image improvement. Inference prediction is a technique for logically inferring and predicting information by determining information, and includes knowledge/probability-based reasoning, optimization prediction, preference-based planning, and recommendation. Knowledge expression is a technology that automatically processes human experience information into knowledge data, and includes knowledge building (data generation/classification), knowledge management (data utilization), and so on. Motion control is a technique for controlling autonomous driving of a vehicle and movement of a robot, and includes motion control (navigation, collision, driving), operation control (behavior control), and the like.

Meanwhile, with the development of multimedia technology and network technology, users can acquire images through various devices, and process and use the acquired images. The acquired images can be used for face recognition technology, and the technology for recognizing faces from images can be used in various fields.

In the face recognition technology, the face of the target object included in the image may be learned so that feature points of the target object are extracted from a predetermined number or more of labeled image data. In this case, image data for recognizing the face of the target object needs to be a predetermined number or more. In addition, in order to recognize the face of a different target object from the learned target object, more than a predetermined number of image data including faces of different target objects is required. Accordingly, a technique for recognizing a face of a target object from image data having a predetermined number or less is required to improve the speed of face recognition and increase data efficiency.

Some embodiments aim to provide a method for processing an image to recognize a face of a target object included in the image, and a device accordingly.

In addition, some embodiments may provide a method and apparatus according to which an image is processed to transform images into a common sample structure based on the structure formed by the eyes and nose of the target object.

Also, some embodiments may provide a method and apparatus according to the target image processing method for extracting a face from a target image including a different target object based on a learning image including the target object.

As a technical means for achieving the above-described technical problem, a first aspect of the present disclosure acquires a plurality of sets of learning images corresponding to a plurality of types of target objects, wherein the learning images in the sets of learning images have a predetermined structure. The feature points forming the are labeled; Generating a first artificial intelligence model for determining a sample structure based on the labeled feature points using the training image in the training image sets; Identifying a face from the modified learning image based on the sample structure; And learning a second artificial intelligence model for verifying the first artificial intelligence model based on the reconstructed image from the modified learning image and the before-deformed learning image. It can provide a way to handle.

In addition, a second aspect of the present disclosure, a communication unit; A memory storing at least one instruction; And a processor that controls the electronic device by executing the at least one instruction, wherein the processor acquires a plurality of sets of training images corresponding to a plurality of types of target objects, but learns within the sets of training images. The image is labeled with feature points forming a predetermined structure, and using the training image in the set of training images, creates a first artificial intelligence model for determining a sample structure based on the labeled feature points, and A second artifact for identifying a face from the modified learning image based on a sample structure, and verifying the first artificial intelligence model based on the reconstructed image from the modified learning image and the before-deformed learning image It is possible to provide an electronic device that trains an intelligent model.

In addition, a third aspect of the present disclosure is to acquire a plurality of sets of training images corresponding to a plurality of types of target objects, wherein the training images in the training image sets are labeled with feature points forming a predetermined structure, action; Generating a first artificial intelligence model for determining a sample structure based on the labeled feature points using the training image in the training image sets; Identifying a face from the modified learning image based on the sample structure; And training a second artificial intelligence model for verifying the first artificial intelligence model based on the reconstructed image from the modified learning image and the before-deformed learning image. It is possible to provide a computer program comprising a medium.

1 is a schematic diagram of a system for processing an image according to some embodiments.

2 is a schematic diagram of a method of generating an artificial intelligence model for recognizing a face of a target object from a learning image according to some embodiments.

3 is a flowchart of a method in which an electronic device processes an image to recognize a face of a target object from a learning image, according to some embodiments.

4 is a diagram illustrating a first artificial intelligence model according to some embodiments.

5 is a diagram illustrating a second artificial intelligence model according to some embodiments.

6 is a schematic diagram of a system for extracting a face of a target object included in the target image by applying the target image to the artificial intelligence model according to some embodiments.

7 is a flowchart of a method in which an electronic device processes an image to recognize a face of a target object from a target image according to some embodiments.

8 is a diagram illustrating a method of extracting a face of a target object by applying a target image to an artificial intelligence learning model according to some embodiments.

9 is a block diagram of an electronic device according to some embodiments.

10 is a block diagram of a processor in accordance with some embodiments.

11 is a diagram illustrating an example in which a device and an electronic device learn and recognize data by interworking with each other, according to some embodiments.

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present disclosure pertains can easily implement them. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in order to clearly describe the present disclosure in the drawings, parts irrelevant to the description are omitted, and like reference numerals are assigned to similar parts throughout the specification.

Throughout the specification, when a part is "connected" to another part, this includes not only "directly connected" but also "electrically connected" with other elements in between. . Also, when a part “includes” a certain component, this means that other components may be further included instead of excluding other components, unless otherwise specified.

In the present specification, the target object may refer to various types of objects including faces, such as animals, characters, and emoticons, as well as people. For example, dogs and cats are different target objects, and dog emoticons and cat emoticons may be different target objects.

According to an embodiment of the present disclosure, an artificial intelligence model for extracting a face of a target object from a predetermined image is provided. At this time, an image corresponding to each of a plurality of types of target objects used to generate an artificial intelligence model is referred to as a'learning image'. In addition, in order to extract the faces of different target objects from the target object included in the pre-trained image and the different target object, the image applied to the generated artificial intelligence model is referred to as a'target image'.

In the present specification, the artificial intelligence model may be a learning model trained using at least one of a machine learning, neural network, genetic, deep learning, and classification algorithms as an artificial intelligence algorithm.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

1 is a schematic diagram of a system for processing an image according to some embodiments.

Referring to FIG. 1, the electronic device 1000 may provide a system for processing an image to recognize a face of a target object included in the image. At this time, the electronic device 1000 may be a server, but is not limited thereto.

The electronic device 1000 may generate an artificial intelligence model for recognizing the face of the target object from the learning image, using a plurality of sets of training images corresponding to the plurality of target objects. Specifically, the electronic device 1000 may determine a sample structure based on the labeled feature points based on the feature points forming a predetermined structure labeled on the learning image.

Specifically, the electronic device 1000 includes eyes and noses common to each of the faces of the plurality of types of target objects, and based on the eyes and noses of the target object forming a triangular structure, the sample structure is determined and determined. You can create artificial intelligence models to verify the sample structure. That is, the electronic device 1000 may determine a sample structure irrelevant to the type of the target object, using a learning image corresponding to a plurality of types of target objects. The electronic device 1000 may identify the face of the target object by transforming the learning image based on the sample structure.

Also, the electronic device 1000 may restore the training image from the modified training image, compare the restored training image with the training image before transformation, and verify the generated artificial intelligence model. Specifically, when restoring the modified learning image, the electronic device 1000 may determine whether the sample structure is incorrectly determined based on whether the difference between the restored learning image and the learning image is greater than or equal to a threshold value.

Also, the electronic device 1000 may receive a target image corresponding to a target object of a different type from a plurality of target objects included in the set of learning images used to generate the artificial intelligence model from an external device (not shown). Can. The electronic device 1000 may transform the target image into a sample structure by inputting the target image into the artificial intelligence model, extract faces of different target objects from the modified target image, and verify the extracted face.

2 is a schematic diagram of a system 100 for generating an artificial intelligence model for recognizing a face of a target object from a learning image according to some embodiments.

According to an embodiment of the present disclosure, the electronic device 1000 acquires a plurality of training image sets 110 corresponding to a plurality of types of target objects to generate an artificial intelligence model for recognizing the face of the target object. Can. Referring to FIG. 2, the electronic device 1000 converts a pig image set 111, a dog image set 112, and a cat image set 113 corresponding to pigs, dogs, and cats into learning image sets 110, respectively. Can be obtained. Hereinafter, for convenience of description, a case in which the target object of the acquired learning image set is'pig' will be described. Specifically, it is described based on the first learning image 150 in the pig image set 111, but the pig learning image different from the first learning image 150 in the pig image set 111, as well as the operations described below are subject It can be applied to the entire learning image regardless of the type of object.

The learning image in the training image sets 110 may be labeled with feature points forming a predetermined structure. Specifically, the learning image corresponding to each of a plurality of types of target objects may be labeled with feature points corresponding to the left eye, right eye, and nose of the target object, and the preset structure includes the target object's eye and It may be a triangular structure connecting the nose. For example, the first learning image 150 corresponding to the pig may be labeled with feature points forming a triangular structure corresponding to the left eye, right eye, and nose.

According to an embodiment of the present disclosure, the electronic device 1000 uses the learning image in the plurality of training image sets 110 to determine the first artificial intelligence for determining the sample structure 120 based on the labeled feature points. You can create a model. Specifically, the sample structure 120 may be determined based on labeled feature points and a spatial transform network (STN) function. The STN function may spatially transform the input image based on the transformation parameter θ. According to an embodiment of the present disclosure, the STN function corresponds to [Equation 1], and the electronic device 1000 uses the [Equation 1] to generate a sample structure 120 based on labeled feature points in the learning image. Can decide.

[Equation 1]

학습 이미지 내의 레이블링된 특징점들을 나타낼 수 있다. 예를 들어 입력

는 학습 이미지 내의 레이블링된 특징점들의 좌표 또는 벡터일 수 있으나 이에 제한되는 것은 아니고, 레이블링된 특징점들을 나타낼 수 있는 형태이면 어느 것이든 가능하다. 전자 장치(1000)는 입력

의 평균 값에 기초하여 표본 구조(120)를 결정할 수 있다. 구체적으로, 전자 장치(1000)는 복수의 학습 이미지 세트(110)들 내의 학습 이미지에 포함된 특징점들의 표본 구조(120)를 STN 함수의 출력 값

에 기초하여 결정할 수 있다. Referring to [Equation 1], input

Labeled feature points in the learning image may be indicated. Enter for example

May be a coordinate or vector of labeled feature points in the learning image, but is not limited thereto, and may be any type that can represent the labeled feature points. The electronic device 1000 is input

The sample structure 120 may be determined based on the average value of. Specifically, the electronic device 1000 outputs the sample structure 120 of feature points included in the training image in the plurality of training image sets 110 as an output value of the STN function.

It can be decided based on.

가 입력될 때 출력되는 구조

간 차이가 소정 임계 값 이하가 되도록 표본 구조(120)를 학습할 수 있다. 구체적으로, 전자 장치(1000)는 하기 [수학식 2]를 이용하여, STN 함수에 학습 이미지의 레이블링된 특징점들에 대응되는

가 입력될 때 출력되는 구조

및 표본 구조(120) 및 간 차이

를 계산하고, MAE(Mean Absolute Error) 값이 최소가 되도록 학습(learning)할 수 있다.Meanwhile, the electronic device 1000 corresponds to the sample structure 120 determined by [Equation 1] and the STN function g _STN corresponding to the labeled feature point of the learning image.

Structure that is output when is input

The sample structure 120 may be trained such that the difference between the livers is equal to or less than a predetermined threshold. Specifically, the electronic device 1000 uses the following [Equation 2] to correspond to the feature points labeled in the learning image in the STN function.

Structure that is output when is input

And sample structure 120 and differences between

Calculate and learn to make the mean absolute error (MAE) value to a minimum.

[Equation 2]

According to an embodiment of the present disclosure, the electronic device 1000 may transform the first learning image 150 based on the sample structure 120 determined using the STN function. The electronic device 1000 may identify the face of the pig as a target object from the modified first learning image 151. For convenience of explanation, the modified first learning image 151 is expressed in the form of an image, but what is actually obtained may be a coordinate, vector, or function of the modified first learning image 151.

According to an embodiment of the present disclosure, the electronic device 1000 compares the first learning image 152 reconstructed from the modified first learning image 151 and the first learning image 150 before transformation, and thus, the first A second artificial intelligence model for validating the artificial intelligence model may be trained. Specifically, the electronic device 1000 restores the modified first learning image 151 based on the sample structure 120, and between the restored first learning image 152 and the first learning image 150 before transformation. Based on the difference, the first artificial intelligence model can be verified. At this time, the electronic device 1000 may use Equation 3 below to obtain a difference between the restored first learning image 152 and the first learning image 150.

[Equation 3]

값이 소정 임계 값 이하인지 여부에 기초하여 제1 인공 지능 모델을 검증할 수 있다. 제1 인공 지능 모델을 검증하는 방법에 대해서는 후술하도록 한다.Referring to [Equation 3], the electronic device 1000 may restore the modified first learning image 151 using the reconstruction function g _reg and the STN inverse function g _STN ^-1 . Also, the electronic device 1000 may calculate a difference between the reconstructed first training image 152 and the first training image 150 before transformation using the reconstruction function g _reg and the STN inverse function g _STN ^-1 . have. At this time, the electronic device 1000 is calculated

The first artificial intelligence model may be verified based on whether the value is below a predetermined threshold. The method for verifying the first artificial intelligence model will be described later.

와 비교할 수 있다. After the first learning image 150 is transformed by the STN function g _STN , when the modified first learning image 151 passes the restoration function g _reg , the electronic device 1000 corresponds to the eyes and nose You can infer the coordinates of the feature points. Also, the electronic device 1000 restores the feature points corresponding to the deduced eye and nose using the STN inverse function g _STN ^-1 and then corresponds to the feature points of the labeled learning image.

Can be compared with

3 is a flowchart of a method in which an electronic device processes an image to recognize a face of a target object from a learning image, according to some embodiments.

In operation S310, the electronic device 1000 may obtain a plurality of sets of learning images respectively corresponding to a plurality of types of target objects. The learning image in the training image sets may be labeled with feature points forming a predetermined structure. The predetermined structure may be a triangular structure connecting the eyes and nose of the target object in the learning image.

In operation S320, the electronic device 1000 may generate a first artificial intelligence model for determining a sample structure based on the labeled feature points using the training image in the training image sets. Specifically, the electronic device 1000 may determine the sample structure based on the labeled feature points and the STN function. For example, the electronic device 1000 determines the sample structure based on the average value of the feature points in the learning image, using the above-described [Equation 1] and [Equation 2], and inputs the feature points and the determined You can learn the sample structure so that the difference between the sample structures is minimal.

In operation S330, the electronic device 1000 may identify the face from the modified learning image based on the sample structure. For example, when the learning image is modified based on the sample structure, the electronic device 1000 may identify the faces of the target object by determining points corresponding to the sample structure as eyes and noses of the target object.

In operation S340, the electronic device 1000 may train a second artificial intelligence model for verifying the first artificial intelligence model based on the reconstructed image from the modified learning image and the learning image before being transformed. Specifically, the electronic device 1000 may restore the modified learning image and verify the first artificial intelligence model based on the difference between the restored image and the learning image. Also, the electronic device 1000 may verify the first artificial intelligence model using a second artificial intelligence model, and update the first artificial intelligence model to change the sample structure based on the verification result. For example, the electronic device 1000 may train the first artificial intelligence model so that the difference between the reconstructed image and the learned image is equal to or less than a preset threshold based on the verification result.

4 is a diagram illustrating a first artificial intelligence model according to some embodiments.

The first artificial intelligence model is a learning model for recognizing a face of a target object based on a sample structure formed by labeled feature points in a learning image. Specifically, referring to FIG. 4, when the training image is input to the first artificial intelligence model, the electronic device 1000 may transform the training image into a sample structure based on the STN function. Also, the electronic device 1000 may recognize a face included in the learning image by extracting feature points corresponding to the sample structure from the modified learning image by a detector. For convenience of description, although the structure of the STN and the detector are shown separately, the electronic device 1000 recognizes the face of the target object based on points corresponding to the sample structure of the STN-converted learning image when STN-converting the training image. You may.

5 is a diagram illustrating a second artificial intelligence model according to some embodiments.

The second artificial intelligence model is a learning model for verifying the first artificial intelligence model. Specifically, referring to FIG. 5, when the transformed learning image is input to the second artificial intelligence model, the electronic device 1000 restores the transformed learning image in the regressor, and restores the reconstructed image and the pre-transformed learning image. By comparing, it is possible to verify the first artificial intelligence model. Specifically, the electronic device 1000 may restore the modified learning image based on the reconstruction function and the STN inverse function, and when the difference between the restored image and the learning image is below a predetermined threshold, the first artificial intelligence model is verified You can judge that.

Meanwhile, when the difference between the reconstructed image and the learned image is greater than or equal to a predetermined threshold, the electronic device 1000 may determine that the first artificial intelligence model has not been verified. For example, when it is determined that the first artificial intelligence model has not been verified, there is an error in the sample structure determined by the first artificial intelligence model, or when the determined sample structure has no error but the training image is incorrectly deformed. However, it is not limited thereto. That is, the electronic device 1000 may verify the first artificial intelligence model based on whether the difference between the restored image and the training image is equal to or less than a predetermined threshold, and change or learn the sample structure determined based on the verification result By re-deforming the image based on the sample structure, it is possible to make the difference between the reconstructed image and the training image less than or equal to a predetermined threshold. For example, the electronic device 1000 may verify the first artificial intelligence model using the second artificial intelligence model, and update the first artificial intelligence model to change the sample structure based on the verification result.

6 is a schematic diagram of a system for extracting a face of a target object included in the target image by applying the target image to the artificial intelligence model according to some embodiments.

Referring to FIG. 6, the electronic device 1000 may obtain a target image 250 including target objects of a different type from target objects included in the training image set for generating the artificial intelligence model. Specifically, the electronic device 1000 may apply a few-shot adaptation to the target image 250 by using the sample structure determined by the first artificial intelligence model. Accordingly, when the electronic device 1000 extracts a face from the target image 250 including the target object of a different type from the target objects of the learning image, the number of target images 250 required may be less than or equal to a predetermined number. That is, the electronic device 1000 may extract a face of a target object of a different type from the target objects included in the learning image using the target image 250 having a predetermined number or less. For example, even if the learning image does not include the target object'bear', the electronic device 1000 may target the artificial intelligence model to the artificial intelligence model generated using the learning image including different target objects, such as pig, dog, cat, etc. 250), the face of the target object'bear' can be extracted.

According to an embodiment of the present disclosure, the electronic device 1000 may transform the target image so that feature points in the target image form a determined sample structure by inputting the acquired target image 250 into the first artificial intelligence model. have.

Also, the electronic device 1000 may extract the face of the target object based on the modified target image 251. Specifically, feature points in the modified target image 251 may be inferred from a target object in the target image 250. For example, the electronic device 1000 may extract the face of the target object by determining the feature points corresponding to the sample structure in the modified target image 251 with the eyes and nose of the target object.

According to an embodiment of the present disclosure, the electronic device 1000 faces the face extracted by the first artificial intelligence model by inputting the modified target image 251 and the pre-deformed target image 250 into the second artificial intelligence model. Can be verified. Specifically, the electronic device 1000 is modified in the second artificial intelligence model for verifying the extracted face by comparing the target image 252 reconstructed from the transformed target image 251 and the target image 250 before transformation. The target image 251 may be input.

According to an embodiment of the present disclosure, the electronic device 1000 may restore the modified target image 251 by inputting the modified target image 251 into the second artificial intelligence model, and the restored target image ( The face of the target object may be extracted from 252). In this case, the electronic device 1000 may verify the face extracted from the reconstructed target image 252 by comparing the reconstructed target image 252 and the target image 250. For example, when the difference between the restored target image 252 and the target image 250 is equal to or less than a predetermined threshold, the electronic device 1000 determines a face extracted from the restored target image 252 as a face of the target object. Can.

7 is a flowchart of a method in which an electronic device processes an image to recognize a face of a target object from a target image according to some embodiments.

In operation S710, the electronic device 1000 may obtain a target image including a plurality of types of target objects and different types of target objects. The electronic device 1000 may extract a face of a target object in a target image including a plurality of target objects included in the learning image and a different type of target object.

In operation S720, the electronic device 1000 extracts the faces of different types of target objects from the target image, in which feature points in the target image are modified to form a sample structure, by applying the target image to the generated first artificial intelligence model. Can. Specifically, the electronic device 1000 may determine a face of a different target object based on feature points corresponding to the sample structure among feature points in the modified target image.

In operation S730, the electronic device 1000 may verify the extracted face based on the first artificial intelligence model by inputting the target image before transformation and the modified target image into the second artificial intelligence model. Specifically, the electronic device 1000 restores the deformed target image, extracts faces of different types of target objects from the reconstructed target image, and compares the reconstructed target image and the target image before transformation, thereby generating the first artificial intelligence. The extracted face can be verified based on the model.

8 is a diagram illustrating a method of extracting a face of a target object by applying a target image to an artificial intelligence model according to some embodiments.

Referring to FIG. 8, the electronic device 1000 may input a target image into the parasitic first artificial intelligence model based on the learning image. Specifically, when the target image is input to the first artificial intelligence model, the electronic device 1000 transforms the input target image using the STN function, and the face of the target object included in the target image based on the modified target image Can be extracted by a detector. In the present disclosure, STN and Detector are separately illustrated for convenience of description, but it is needless to say that feature points corresponding to the face of the target object may be extracted when the STN function is applied. Meanwhile, the electronic device 1000 may update the first artificial intelligence model based on feature points extracted from the target image. For example, the electronic device 1000 may update the determined sample structure based on the structure formed by the feature points in the target image.

According to an embodiment of the present disclosure, the electronic device 1000 may input the deformed target image and the pre-deformed target image into the second artificial intelligence model. Specifically, the electronic device 1000 may restore the target image modified by the regressor. The electronic device 1000 may verify the face of the target object extracted from the target image based on the first artificial intelligence model based on whether the difference between the restored target image and the target image before transformation is equal to or less than a predetermined threshold. If the difference between the restored target image and the target image before transformation is greater than or equal to a predetermined threshold, the electronic device 1000 may change the parasitic first artificial intelligence model. For example, the electronic device 1000 may change the determined sample structure based on the feature points in the acquired target image, or may perform transformation of the target image again based on the determined sample structure.

9 is a block diagram of an electronic device according to some embodiments.

As illustrated in FIG. 9, the electronic device 1000 according to some exemplary embodiments includes the electronic device 1000 by executing the communication unit 1500, a memory 1700 storing at least one instruction, and at least one instruction. It may include a processor 1300 for controlling the. However, not all of the components illustrated in FIG. 9 are essential components of the electronic device 1000. The electronic device 1000 may be implemented by more components than those illustrated in FIG. 9, or the electronic device 1000 may be implemented by fewer components than those illustrated in FIG. 9.

The processor 1300 typically controls the overall operation of the electronic device 1000. For example, the processor 1300 may control operations of the electronic device 1000 in the present disclosure by executing programs stored in the memory 1700.

Specifically, the processor 1300 may receive an image from the device 2000 through the communication unit 1500. For example, the processor 1300 may control to process the image received from the device 2000 to extract the face of the target object included in the received image.

According to an embodiment of the present disclosure, the processor 1300 acquires a plurality of sets of learning images corresponding to a plurality of types of target objects, wherein the learning images in the sets of learning images are labeled with feature points forming a predetermined structure. To generate a first artificial intelligence model for determining a sample structure based on labeled feature points, using the learning image in the training image sets, to identify a face from the modified training image based on the sample structure, and to transform A second artificial intelligence model for verifying the first artificial intelligence model may be trained on the basis of the reconstructed image from the learned learning image and the learning image before transformation.

The processor 1300 may determine the sample structure based on the labeled feature points and the STN function. In addition, the processor 1300 may generate a modified learning image by transforming the learning image so that feature points in the learning image form a sample structure.

According to an embodiment of the present disclosure, the processor 1300 acquires a target image including a plurality of types of target objects and different types of target objects, and applies the target image to the generated first artificial intelligence model, thereby targeting The first artificial intelligence model by extracting the faces of different types of target objects from the target image in which the feature points in the image are modified to form a sample structure, and inputting the target image before transformation and the modified target image into the second artificial intelligence model. Based on the extracted face can be verified.

Specifically, the processor 1300 may receive a target image including a plurality of learning image sets corresponding to a plurality of types of target objects and a target object different from a predetermined number through the communication unit 1500. The processor 1300 may transform the target image into a sample structure based on the previously generated first artificial intelligence model and the second artificial intelligence model, and extract faces of different target objects based on the modified target image. That is, the processor 1300 has an advantage in that it is possible to extract a face of a target object included in the target image using only a predetermined number of images or less by using a sample structure, thereby improving data efficiency and face recognition speed.

The communication unit 1500 may include one or more components for communication with an external device (not shown) for receiving a target image or a learning image. Also, the communication unit 1500 may transmit the face recognition result to an external device (not shown). The communication unit 1500 may include a short-range communication unit, a mobile communication unit, and the like, but is not limited thereto.

The memory 1700 may store a program for processing and controlling the processor 1300, and may store data input to or output from the electronic device 1000.

The memory 1700 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory, etc.), RAM (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (ROM, Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk , It may include at least one type of storage medium of the optical disk.

10 is a block diagram of a processor in accordance with some embodiments.

Referring to FIG. 10, the processor 1300 according to some embodiments may include a data learning unit 1310 and a data recognition unit 1320.

The data learning unit 1310 may learn a criterion for determining a sample structure from a learning image. The data learning unit 1310 may learn criteria on which data to use to determine the sample structure and how to transform the learning image using the data. For example, the data learning unit 1310 may use feature points labeled on the learning image. In addition, the data learning unit 1310 determines the sample structure formed by the feature points based on the labeled feature points, and transforms the learning image based on the sample structure, thereby based on the feature points corresponding to the sample structure. You can learn the criteria for identifying your face.

In addition, the data learning unit 1310 may obtain a modified learning image by applying the determined sample structure to the learning image, and verify the sample structure based on the modified learning image. Specifically, the data learning unit 1310 may learn a criterion for verifying the sample structure by comparing the reconstructed learning image and the pre-deformed learning image by restoring the modified learning image.

Specifically, the data learning unit 1310 may acquire a learning image for determining the sample structure. For example, as described above with reference to FIGS. 1 and 2, the data learning unit 1310 may acquire learning image sets corresponding to a plurality of types of target objects, but the above-described implementations in FIGS. 1 and 2. It is not limited to the example.

The data learning unit 1310 may preprocess the acquired data to extract data for face recognition from the acquired learning image. The data learning unit 1310 may process the acquired data in a preset format so that a triangular structure connecting the eyes and nose of the target object in the learning image can be used.

The data learning unit 1310 may select data necessary for learning from pre-processed data. Specifically, the data learning unit 1310 may select data necessary for learning from pre-processed data according to preset criteria.

The data learning unit 1310 may learn criteria for transforming a learning image into a sample structure based on the learning data. In addition, the data learning unit 1310 may also learn the criteria as to which learning data should be used.

In addition, the data learning unit 1310 may train the first artificial intelligence model and the second artificial intelligence model based on the labeling of feature points forming a predetermined structure in the learning image. Hereinafter, for convenience of explanation, the artificial intelligence model includes both the first artificial intelligence model and the second artificial intelligence model.

Alternatively, the data learning unit 1310 may generate an artificial intelligence model and train the generated artificial intelligence model. The artificial intelligence model may be a learning model forew-shot adaptation.

The artificial intelligence model may be constructed in consideration of the application field of the recognition model, the purpose of learning, or the computer performance of the device. The artificial intelligence model may be, for example, a model based on a neural network. For example, a model such as a deep neural network (DNN), a recurrent neural network (RNN), or a bidirectional recurrent deep neural network (BRDNN) may be used as an artificial intelligence model, but is not limited thereto.

According to various embodiments of the present disclosure, when a plurality of pre-built artificial intelligence models exist, the data learning unit 1310 may determine that the relationship between the input learning image and the learning data is an artificial intelligence model for learning the artificial intelligence model.

In addition, the data learning unit 1310 may train an artificial intelligence model using, for example, a learning algorithm including an error back-propagation or a gradient descent method.

Further, the data learning unit 1310 may train the artificial intelligence model, for example, through supervised learning using learning data as an input value. In addition, the data learning unit 1310, for example, through unsupervised learning (unsupervised learning) to discover the criteria for determining the situation by learning the type of data necessary for situation determination without much guidance, for example, artificial intelligence You can train the model. In addition, the data learning unit 1310 may train the artificial intelligence model through, for example, reinforcement learning using feedback on whether a result of situation determination according to learning is correct.

In addition, when the artificial intelligence model is trained, the data learning unit 1310 may store the trained artificial intelligence model. In this case, the data learning unit 1310 may store the learned artificial intelligence model in the memory of the electronic device 1000 including the data recognition unit 1320 to be described later. Alternatively, the data learning unit 1310 may store the learned artificial intelligence model in a memory of a server connected to the electronic device 1000 through a wired or wireless network.

In this case, the memory in which the learned artificial intelligence model is stored may store, for example, instructions or data related to at least one other component of the electronic device 1000. Also, the memory may store software and/or programs. The program may include, for example, a kernel, middleware, application programming interface (API) and/or application program (or "application").

The data learning unit 1310 inputs the reconstructed learning image and the pre-deformation learning image to the second artificial intelligence model, and based on the difference between the reconstructed learning image and the pre-deformation learning image, the difference value does not satisfy a predetermined criterion. In this case, it may be determined that the first artificial intelligence model has not been verified. In this case, the data learning unit 1310 may train the first artificial intelligence model again.

For example, the data learning unit 1310 does not satisfy a predetermined criterion when the number or ratio of the evaluation data in which the recognition result is not accurate among the recognition results of the learned artificial intelligence model for the evaluation data exceeds a preset threshold. It can be evaluated as failed. For example, when a predetermined criterion is defined as a ratio of 2%, when the trained artificial intelligence model outputs an incorrect recognition result for evaluation data exceeding 20 out of a total of 1000 evaluation data, the data learning unit 1310 learns It can be evaluated that the artificial intelligence model is not suitable.

Meanwhile, the data learning unit 1310 may be manufactured in the form of at least one hardware chip and mounted on an electronic device. For example, the data learning unit 1310 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or an existing general-purpose processor (eg, CPU or application processor) or graphics-only processor (eg : GPU) and may be mounted on various electronic devices described above.

Also, the data learning unit 1310 may be implemented as a software module. When the data learning unit 1310 is implemented as a software module (or a program module including an instruction), the software module is a computer-readable, non-transitory computer readable medium. media). Also, in this case, the at least one software module may be provided by an operating system (OS) or may be provided by a predetermined application. Or, some of the at least one software module may be provided by an operating system (OS), and the other may be provided by a predetermined application.

The data recognition unit 1320 may use a first artificial intelligence model to extract faces of different target objects from the target image. The data recognition unit 1320 may modify the target image based on the sample structure by inputting the target image into the first artificial intelligence model. At this time, the data recognition unit 1320 may extract the face of the target object from the modified target image based on the feature points corresponding to the sample structure. The data recognition unit 1320 acquires feature points corresponding to the eye nose of the target object according to a preset criterion by learning, and outputs faces of different target objects extracted from the modified target image based on the acquired feature points It can be provided as data. Also, the data recognition unit 1320 may verify the extracted face based on the difference between the restored target image and the pre-deformed target image by inputting the deformed target image and the pre-deformed target image into the second artificial intelligence model. have.

The data recognition unit 1320 may acquire data related to feature points labeled in the target image, and the data recognition unit 1320 may preprocess the acquired data. The data recognition unit 1320 may process the acquired data in a preset format so that data related to feature points included in the target image can be used. The data recognition unit 1320 may acquire data in FIGS. 6 to 8, for example, but is not limited thereto.

The data recognition unit 1320 may select required data from pre-processed data. The selected data may be provided to the data learning unit 1310. The data recognition unit 1320 may select some or all of the pre-processed data according to a preset criterion. Also, the data recognition unit 1320 may select data according to a preset criterion by learning.

The data recognition unit 1320 may apply the selected data to the artificial intelligence model and output data for extracting the face of the target object included in the target image. The data recognition unit 1320 may apply the selected data to the artificial intelligence model by using the selected data as an input value. Also, the recognition result may be determined by an artificial intelligence model.

At this time, the artificial intelligence model uses the learning image in the training image sets, the first artificial intelligence model for determining the sample structure based on the labeled feature points, and the reconstructed image from the modified training image and the training image before transformation. By comparing, it may include a second artificial intelligence model for verifying the first artificial intelligence model.

The data recognition unit 1320 may cause the first artificial intelligence model to be updated based on the output result of the second artificial intelligence model. For example, the data recognition unit 1320 may provide the data learning unit 1310 with the recognition result provided by the second artificial intelligence model, so that the data learning unit 1310 updates the first artificial intelligence model. have.

Meanwhile, the data recognition unit 1320 may be manufactured in the form of at least one hardware chip and mounted on the electronic device 1000. For example, the data recognition unit 1320 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or an existing general-purpose processor (eg, CPU or application processor) or graphics-only processor (eg : GPU) and may be mounted on various electronic devices described above. Also, the data recognition unit 1320 may be mounted on one electronic device, or may be mounted on separate electronic devices, respectively.

Also, the data recognition unit 1320 may be implemented as a software module. When the data recognition unit 1320 is implemented as a software module (or a program module including an instruction), the software module is a computer-readable non-transitory computer readable medium. media). Also, in this case, the at least one software module may be provided by an operating system (OS) or may be provided by a predetermined application. Or, some of the at least one software module may be provided by an operating system (OS), and the other may be provided by a predetermined application.

At least one of the data learning unit 1310 and the data recognition unit 1320 may be manufactured in the form of at least one hardware chip and mounted on an electronic device. For example, at least one of the data learning unit 1310 and the data recognition unit 1320 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or an existing general-purpose processor (for example, a CPU) Alternatively, it may be manufactured as a part of an application processor or a graphics-only processor (for example, a GPU) and mounted on various electronic devices described above.

In this case, the data learning unit 1310 and the data recognition unit 1320 may be mounted on one electronic device, or may be mounted on separate electronic devices. For example, one of the data learning unit 1310 and the data recognition unit 1320 may be included in the electronic device, and the other may be included in the server. In addition, the data learning unit 1310 and the data recognition unit 1320 may provide the model information constructed by the data learning unit 1310 to the data recognition unit 1320 through wired or wireless communication. Data input to 1320) may be provided to the data learning unit 1310 as additional learning data.

Meanwhile, at least one of the data learning unit 1310 and the data recognition unit 1320 may be implemented as a software module. When at least one of the data learning unit 1310 and the data recognition unit 1320 is implemented as a software module (or a program module including an instruction), the software module is a computer-readable, non-transitory readable It may be stored in a readable media (non-transitory computer readable media). Also, in this case, the at least one software module may be provided by an operating system (OS) or may be provided by a predetermined application. Or, some of the at least one software module may be provided by an operating system (OS), and the other may be provided by a predetermined application.

11 is a diagram illustrating an example in which a device and an electronic device learn and recognize data by interworking with each other, according to some embodiments.

Referring to FIG. 11, the data learning unit 1310 may be included in the electronic device 1000, and the data recognition unit 2300 may be included in the device 2000. At this time, the electronic device 1000 may be a server. However, the data learning unit 1310 and the data recognition unit 2300 are not limited to the above-described examples, and can be implemented in other forms.

The data learning unit 1310 may learn a criterion for transforming a learning image into a sample structure for face recognition based on labeling of feature points forming a predetermined structure in the acquired learning image. Also, when the target image is selected from the device 2000, the data recognition unit 2300 may extract the faces of different target objects in the target image using the artificial intelligence model generated by the data learning unit 1310.

In this case, the data learning unit 1320 of the electronic device 1000 is a reference for recognizing the face of the target object by transforming the learning image into a sample structure based on labeling of feature points forming a predetermined structure in the learning image Can learn. The data learning unit 1320 may acquire data to be used for learning, and generate an artificial intelligence model for identifying a face of a target object included in the learning image based on the acquired data.

Also, the data recognition unit 2300 of the device 2000 may provide the result of applying the data selected as the target image to the artificial intelligence model generated by the electronic device 1000. For example, the data recognition unit 2300 of the device 2000 transmits the target image to the electronic device 1000, and the electronic device 1000 applies the target image to the artificial intelligence model, and the target included in the target image You can request to extract the object's face. Also, the data recognition unit 2300 may receive information related to the face of the extracted target object from the electronic device 1000.

Alternatively, the data recognition unit 2300 of the device 2000 receives the artificial intelligence model generated by the electronic device 1000 from the electronic device 1000 and extracts a face from the target image using the received artificial intelligence model. In order to do this, you can process the target image. On the other hand, in FIG. 11, although the electronic device 1000 and the device 2000 interlock, it has been described as extracting a target object different from a plurality of types of target objects included in a plurality of learning image sets from the target image, but is limited to this. no.

Some embodiments may also be embodied in the form of a recording medium comprising instructions executable by a computer, such as program modules, being executed by a computer. Computer readable media can be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. In addition, computer readable media may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

In addition, in this specification, the “part” may be a hardware component such as a processor or circuit, and/or a software component executed by a hardware component such as a processor.

The foregoing description of the present disclosure is for illustration only, and those skilled in the art to which the present disclosure pertains can understand that the present invention can be easily modified into other specific forms without changing the technical spirit or essential features of the present disclosure. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present disclosure is indicated by the following claims rather than the above detailed description, and it should be construed that all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts are included in the scope of the present disclosure. do.

Claims (15)

A method for an electronic device to process an image, Obtaining a plurality of training image sets corresponding to a plurality of types of target objects, wherein the training image in the training image sets is labeled with feature points forming a predetermined structure; Generating a first artificial intelligence model for determining a sample structure based on the labeled feature points using the training image in the training image sets; Identifying a face from the modified learning image based on the sample structure; And Training a second artificial intelligence model for verifying the first artificial intelligence model based on the reconstructed image from the modified learning image and the before-deformed learning image; How to include. According to claim 1, The second artificial intelligence model, And restoring the modified learning image, and verifying the first artificial intelligence model based on the difference between the restored learning image and the before-deformed learning image. According to claim 1, Verifying the first artificial intelligence model using the second artificial intelligence model; And Updating the first artificial intelligence model to change the sample structure based on the verification result; The method, which further comprises. According to claim 1, The predetermined structure is a triangular structure connecting the eyes and nose of the target object in the learning image. According to claim 1, The sample structure is determined based on the labeled feature points and a spatial transform network (STN) function. According to claim 1, Wherein the modified learning image is generated by modifying the learning image so that the feature points in the learning image form the sample structure. According to claim 1, Obtaining a target image including target objects of different types from the plurality of target objects; Extracting the faces of the different types of target objects from the target image in which feature points in the target image are modified to form the sample structure by applying the target image to the generated first artificial intelligence model; And Verifying the face extracted based on the first artificial intelligence model by inputting a target image before the transformation and the modified target image into the second artificial intelligence model; The method, which further comprises. According to claim 1, The first artificial intelligence model and the second artificial intelligence model are artificial intelligence algorithms, which are learning models learned using at least one of machine learning, neural networks, genes, deep learning, or classification algorithms. An electronic device for processing an image, Communication department; A memory storing at least one instruction; And A processor that controls the electronic device by executing the at least one instruction; It includes, The processor, Acquiring a plurality of sets of learning images corresponding to a plurality of types of target objects, wherein the learning images in the sets of learning images are labeled with feature points forming a predetermined structure, Using the training image in the training image sets, generate a first artificial intelligence model for determining a sample structure based on the labeled feature points, A face is identified from the learning image modified based on the sample structure, An electronic device for training a second artificial intelligence model for verifying the first artificial intelligence model based on the reconstructed image from the modified learning image and the before-deformed learning image. The method of claim 9, The processor, And reconstructing the modified learning image and verifying the first artificial intelligence model based on the difference between the restored learning image and the before-deformed learning image. The method of claim 9, The processor, Verifying the first artificial intelligence model using the second artificial intelligence model, And updating the first artificial intelligence model to change the sample structure based on the verification result. The method of claim 9, The predetermined structure is a triangular structure that connects the eyes and nose of the target object in the learning image. The method of claim 9, The sample structure is determined based on the labeled feature points and a spatial transform network (STN) function. The method of claim 9, The modified learning image is generated by transforming the learning image so that the feature points in the learning image form the sample structure. The method of claim 9, The processor, A target image including a target object of a different type from the plurality of target objects is obtained, By applying the target image to the generated first artificial intelligence model, faces of the different types of target objects are extracted from a target image in which feature points in the target image are modified to form the sample structure, An electronic device for verifying a face extracted based on the first artificial intelligence model by inputting a target image before the transformation and the modified target image in the second artificial intelligence model.

Images