WO2025071024A1 - Electronic device and control method thereof - Google Patents

Abstract

The present electronic device comprises: a memory for storing at least one instruction and a neural network model for improving an image quality; and at least one processor for executing the at least one instruction stored in the memory so as to acquire a data set for training the neural network model and train the neural network model by using the data set, wherein the at least one processor extracts a patch having a preconfigured size for each data set, obtains feature information for the extracted patch, and on the basis of the obtained feature information, determines whether to perform training for the patch in a generative adversarial network (GAN) scheme or in a non-GAN scheme, so as to train the neural network model.

Description

Electronic device and method of controlling the same

Embodiments of the present disclosure relate to an electronic device and a method for controlling the same, and more specifically, to an electronic device capable of performing image quality processing and a method for controlling the same.

Electronic devices perform an operation of generating and displaying images, etc., or providing images to a display device that displays images. These electronic devices can display not only specific content selected by a user as described above, but also various pieces of information.

Images used by electronic devices may have various resolutions, and electronic devices can display images with a lower resolution than the resolution they can display by performing upscaling technology or image quality improvement operations.

According to one embodiment of the present disclosure, an electronic device includes a memory storing at least one instruction and a neural network model for improving the image quality of an image, and at least one processor executing the at least one instruction stored in the memory to obtain a data set for training the neural network model, and training the neural network model using the data set, wherein the at least one processor extracts a patch having a preset size for each of the data sets, calculates feature information for the extracted patch, and determines whether to perform training for the patch in a generative adversarial network (GAN) manner or a non-GAN manner based on the calculated feature information, thereby training the neural network model.

The at least one processor may train the neural network model for the patch in a GAN manner if the generated feature information satisfies a preset condition, and may train the neural network model for the patch in a non-GAN manner if the generated feature information does not satisfy the preset condition.

The above GAN method performs a GAN comparison operation to improve a blur area in an image using GAN loss, and the non-GAN method may be a method in which GAN loss is not applied in the loss function of the GAN method.

The above feature information is information for detecting at least one of complex parts, such as a human face and text, in an image, and the at least one processor may perform learning for the patch in a non-GAN manner if at least one of complex parts, such as a human face or text, is detected in the extracted patch, and may perform learning for the patch in a GAN manner if a complex part, such as a human face or text, is not detected in the extracted patch.

The at least one processor can produce statistical values related to texture features for the extracted patch as feature information.

The at least one processor can generate an entropy map indicating whether each pixel in the extracted patch is similar to surrounding pixels, and can calculate the feature information using the generated entropy map.

The at least one processor can calculate a GLCM (Glay-level co-occurrence matrix) value for each pixel in the extracted patch, and calculate at least one feature information using the calculated GLCM value.

The at least one processor may use the GLCM values to produce at least one of a contrast value indicating a contrast difference within a patch, a correlation value indicating a correlation state between pixels within a patch, a mean square value of GLCM values within a patch, and a homogeneous value indicating a similar state of GLCM values with respect to a diagonal direction within a patch as the feature information.

The at least one processor may output a plurality of feature information, apply a weight corresponding to the feature information to the output feature information to output a feature value, and compare the output feature value with a preset reference value to determine whether to perform learning for the patch in a generative adversarial network (GAN) manner or a non-GAN manner, thereby training the neural network model.

The above neural network model may be a neural network model that generates an output image of a second resolution that is higher quality than the first resolution from an input image of a first resolution.

The at least one processor may, when an image is selected, input the selected image into a learned neural network model to obtain an output image.

Meanwhile, a method for controlling an electronic device according to an embodiment of the present disclosure includes a step of obtaining a data set for training a neural network model that improves image quality, and a step of training the neural network model using the data set, wherein the training step includes a step of extracting a patch having a preset size for each of the data sets, a step of calculating feature information for the extracted patch, and a step of determining whether to perform training for the patch using a generative adversarial network (GAN) method or a non-GAN method based on the calculated feature information, thereby training the neural network model.

The step of calculating the above feature information can calculate statistical values related to texture features for the extracted patch as the feature information.

The step of calculating the above feature information may include generating an entropy map that indicates whether each pixel in the extracted patch is similar to surrounding pixels, and calculating the feature information using the generated entropy map.

The step of calculating the above feature information may calculate a GLCM (Glay-level co-occurrence matrix) value for each pixel in the extracted patch, and calculate at least one feature information using the calculated GLCM value.

The step of calculating the above feature information may use the GLCM values to calculate at least one of a contrast value indicating a contrast difference within a patch, a correlation value indicating a correlation state between pixels within a patch, a mean square value of GLCM values within a patch, and a homogeneous value indicating a similar state of GLCM values with respect to a diagonal direction within a patch as the feature information.

The step of calculating the above feature information calculates a plurality of feature information, and calculates a feature value by applying a weight corresponding to the feature information to the calculated feature information, and the step of training the neural network model compares the calculated feature value with a preset reference value to determine whether to proceed with training for the patch using a generative adversarial network (GAN) method or a non-GAN method, thereby allowing the neural network model to be trained.

The above feature information is information for detecting at least one of complex parts such as a human face or text within an image, and the step of training the neural network model may perform training for the patch in a non-GAN manner if at least one of complex parts such as a human face or text is detected in the extracted patch, and may perform training for the patch in a GAN manner if complex parts such as a human face and text are not detected in the extracted patch.

The present control method may further include a step of inputting the selected image into a learned neural network model to obtain an output image when an image is selected.

A non-transitory computer-readable recording medium storing a program for executing a control method of an electronic device according to an embodiment of the present disclosure, wherein the control method includes a step of obtaining a data set for training a neural network model that improves image quality, and a step of training the neural network model using the data set, wherein the training step includes a step of extracting a patch having a preset size for each of the data sets, a step of calculating feature information for the extracted patches, and a step of determining whether to perform training for the patches in a GAN (generative adversarial network) manner or a non-GAN manner based on the calculated feature information, thereby training the neural network model.

The above and other aspects, features, and advantages of the embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

FIG. 1 is a diagram for explaining an image quality improvement system using a neural network model according to one embodiment of the present disclosure;

FIG. 2 is a diagram illustrating the configuration of an electronic device according to an embodiment of the present disclosure;

FIG. 3 is a diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure;

Figure 4 is a diagram showing an example of image quality degradation due to application of GAN.

FIG. 5 is a diagram for explaining the selective GAN application operation of the present disclosure;

FIG. 6 is a drawing for explaining an example of generating feature information according to one embodiment of the present disclosure;

FIG. 7 is a drawing for explaining an example of generating feature information according to one embodiment of the present disclosure;

FIG. 8 is a drawing for explaining an example of generating feature information according to one embodiment of the present disclosure;

FIG. 9 is a flowchart for explaining a control method according to one embodiment of the present disclosure, and

FIG. 10 is a flowchart illustrating a selective learning method according to one embodiment of the present disclosure.

The present embodiments may have various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope to specific embodiments, but should be understood to include various modifications, equivalents, and/or alternatives of the embodiments of the present disclosure. In connection with the description of the drawings, similar reference numerals may be used for similar components.

In describing the present disclosure, if it is determined that a specific description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, a detailed description thereof will be omitted.

In addition, the following embodiments may be modified in various other forms, and the scope of the technical idea of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to make the present disclosure more faithful and complete, and to fully convey the technical idea of the present disclosure to those skilled in the art.

The terminology used in this disclosure is only used to describe specific embodiments and is not intended to limit the scope of the rights. The singular expression includes the plural expression unless the context clearly indicates otherwise.

In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

In this disclosure, the expressions “A or B,” “at least one of A and/or B,” or “one or more of A or/and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.

The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and/or importance, and are only used to distinguish one component from other components and do not limit the components.

When it is stated that a component (e.g., a first component) is "(operatively or communicatively) coupled with/to" or "connected to" another component (e.g., a second component), it should be understood that said component can be directly coupled to said other component, or can be coupled through another component (e.g., a third component).

On the other hand, when it is said that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component), it can be understood that no other component (e.g., a third component) exists between said component and said other component.

The expression "configured to" as used in this disclosure can be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" does not necessarily mean only "specifically designed to" in terms of hardware.

Instead, in some contexts, the phrase "a device configured to" can mean that the device, in conjunction with other devices or components, is "capable of" doing. For example, the phrase "a processor configured (or set) to perform A, B, and C" can mean a dedicated processor (e.g., an embedded processor) to perform the operations, or a generic-purpose processor (e.g., a CPU or application processor) that can perform the operations by executing one or more software programs stored in a memory device.

In the embodiments, a 'module' or 'part' performs at least one function or operation and may be implemented by hardware or software or a combination of hardware and software. In addition, a plurality of 'modules' or a plurality of 'parts' may be integrated into at least one module and implemented by at least one processor, except for a 'module' or 'part' that needs to be implemented by a specific hardware.

According to various embodiments, operations performed by a module, program or other component may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

Meanwhile, various elements and areas in the drawings are schematically drawn. Therefore, the technical idea of the present invention is not limited by the relative sizes or intervals drawn in the attached drawings.

Meanwhile, an electronic device according to various embodiments of the present disclosure may include, for example, at least one of a smart phone, a tablet PC, a desktop PC, a server, a laptop PC, or a wearable device. The wearable device may include at least one of an accessory-type (e.g., a watch, a ring, a bracelet, an anklet, a necklace, glasses, a contact lens, or a head-mounted device (HMD)), a fabric or clothing-integrated (e.g., an electronic garment), a body-attached (e.g., a skin pad or a tattoo), or a bio-implantable circuit.

In some embodiments, the electronic device may include at least one of, for example, a television, a digital video disk (DVD) player, an audio, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave oven, a washing machine, an air purifier, a set-top box, a home automation control panel, a security control panel, a media box (e.g., Samsung HomeSyncTM, AppleTVTM, or Google TVTM), a game console (e.g., XboxTM, PlayStationTM), an electronic dictionary, an electronic key, a camcorder, or an electronic picture frame. Meanwhile, a device having a display among the above-described electronic devices may be referred to as a display device. Meanwhile, the electronic device of the present disclosure may be a set-top box or a PC that provides an image to a display device even if it does not have a display.

Hereinafter, with reference to the attached drawings, embodiments according to the present disclosure will be described in detail so that a person having ordinary knowledge in the technical field to which the present disclosure pertains can easily implement the present disclosure.

FIG. 1 is a diagram for explaining an image quality improvement system using a neural network model according to one embodiment of the present disclosure.

Referring to FIG. 1, a neural network model (200) is disclosed. The neural network model may be referred to as a neural network, a network model, or a deep learning model.

The neural network model according to the present disclosure can input an image, perform image quality improvement processing on the input image, and generate an output image. For example, the neural network model can input a low-resolution image, and output a high-resolution image. In this case, the neural network model can perform an image quality improvement operation that improves artifacts in the image, in addition to simply upscaling the image to high resolution. In addition, the neural network model can perform only an image quality improvement operation without an upscaling operation.

Meanwhile, in the illustrated example, for the sake of ease of explanation, image quality improvement and upscaling operations are performed using only a neural network model, but an upscaler (not illustrated) may be provided at the front/back end of the neural model (200) or within the model during implementation.

The learning operation for the neural network model is described later in Fig. 5. Fig. 1 only describes the process of using the neural network model generated according to the method described later.

Referring to Figure 1, an input image can be input to a neural network model (200).

The neural network model (200) may include a plurality of convolution layers (rectangular in the illustrated image) and spatial layers (trapezoidal). Each of the plurality of convolutions may include a plurality of nodes (or weight nodes).

Although the illustrated example only shows one channel, the neural network model may consist of multiple channels in which the illustrated multiple convolutional layers are arranged in parallel.

When an input image is input, the neural network model (200) can decompose the input image into multiple patches, and generate an output image for each patch by passing each of the decomposed multiple patches through the neural network model (200). Here, a patch is a processing unit in the neural network model (200) and can have, for example, a w*h pixel size (where w is the number of horizontal pixels and h is the number of height pixels).

And the neural network model (200) can generate an output image (20) by merging output images for each patch.

The neural network model according to the present disclosure is trained using a GAN learning method for improving a blurred area and a non-GAN learning method for a preset area, so that an output image is generated in such a way that blur processing is not applied to areas with high complexity, such as faces/texts, and blur processing is applied to other areas, thereby improving a blurred area, and generating an output image without occurrence of artifacts is possible even in areas that do not require blur processing, such as faces or text areas.

FIG. 2 is a diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 2, the electronic device (100) may include a memory (110) and at least one processor (120).

The memory (110) may be implemented as an internal memory such as a ROM (for example, an electrically erasable programmable read-only memory (EEPROM)), a RAM, etc. included in at least one processor (120), or may be implemented as a separate memory from at least one processor (120). In this case, the memory (110) may be implemented as a memory embedded in the electronic device (100) or as a memory that can be detachably attached to the electronic device (100) depending on the purpose of data storage. For example, data for driving the electronic device (100) may be stored in a memory embedded in the electronic device (100), and data for expanding functions of the electronic device (100) may be stored in a memory that can be detachably attached to the electronic device (100).

The memory (110) may include at least one instruction. The memory (110) may include an instruction for using a neural network model described below (i.e., improving image quality for an image) or an instruction for training a neural network model.

The memory (110) stores a neural network model. The neural network model according to the present disclosure includes a plurality of convolutional layers, and each node in each convolutional layer may be a model learned by selectively applying the GAN method or non-GAN method described below to each patch. The learning operation of the neural network model is described in detail in FIG. 5.

The memory (110) can store a data set for learning a neural network model. And the memory (110) can store an image to be improved in image quality or an output image of a neural network model.

Meanwhile, in the case of memory embedded in the electronic device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD)), and in the case of memory that can be detachably attached to the electronic device (100), it may be implemented in the form of a memory card (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc. there is.

Meanwhile, in the illustrated example, the electronic device (100) is illustrated as consisting of one memory, but when referring to volatile memory and non-volatile memory separately, the electronic device (100) may be referred to as including multiple memories.

At least one processor (120) can perform overall control operations of the electronic device (100). Specifically, at least one processor (120) has a function of controlling overall operations of the electronic device (100).

At least one processor (120) may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON) that processes digital signals. However, the present invention is not limited thereto, and may include one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a graphics-processing unit (GPU), a communication processor (CP), or an ARM processor, or may be defined by the relevant terms. In addition, at least one processor (120) may be implemented as a system on chip (SoC), a large scale integration (LSI), or may be implemented in the form of a field programmable gate array (FPGA) with a built-in processing algorithm. In addition, at least one processor (120) may perform various functions by executing computer executable instructions stored in a memory. Meanwhile, in FIG. 2, only one processor is included in the electronic device (100). Although shown as such, the implementation may include multiple processors (e.g., CPU + GPU, CPU + DSP).

At least one processor (120) can obtain a data set for training a neural network model by executing at least one instruction stored in the memory (110). Such a data set may be pre-stored in the memory (110).

At least one processor (120) can train a neural network model using a data set. For example, at least one processor (120) can extract a patch having a preset size for each data set. For example, for one image, a plurality of patches having a preset size (e.g., w*h size in pixels) at various locations can be extracted.

The preset size here can correspond to a unit size processed by each convolutional layer of the neural network model. And the location where the patch is extracted can be random, and the extraction location can also be based on the learning result of the neural network model.

At least one processor (120) generates feature information for the extracted patch. For example, at least one processor (120) may generate statistical values related to texture features, extract an entropy map, generate GLCM values, and generate feature information using the generated GLCM values. The operation of generating feature information is described later with reference to FIGS. 5 to 8.

At least one processor (120) can train a neural network model by determining whether to proceed with learning for a patch in a GAN (Generative adversarial network) manner or a non-GAN manner based on the generated feature information. For example, at least one processor (120) can train a neural network model for a patch in a GAN manner if the generated feature information satisfies a preset condition, and can train a neural network model for a patch in a non-GAN manner if the generated feature information does not satisfy the preset condition.

To rephrase the above-described operation, the above-described extraction of feature information is intended to determine whether the corresponding patch is a region with high complexity (e.g., a face region, a text region) or a region where blur processing is not preferably applied. Accordingly, if the extracted feature information indicates a region with high complexity or a region where blur processing is not preferably applied, at least one processor (120) can train the neural network model in a non-GAN manner. Conversely, if the current patch is a region where complexity is not high and where blur processing is preferably applied, at least one processor (120) can train the neural network model in a GAN manner.

Here, the GAN method is a learning method that performs the GAN comparison operation to improve the block area within the image using the GAN loss. And the non-GAN method is a learning method in which the GAN loss described above is not applied in the GAN method described above.

Meanwhile, in the above, the learning operation is performed in two ways, either selectively applying the GAN loss or not applying it. However, when implemented, it may be applied by multiplying the GAN loss by a preset feature value. For example, the context value (or feature value) may be applied to the GAN loss, so that the degree of application of the GAN loss may vary. If implemented in this way, the operation of the processor may be expressed as follows.

At least one processor (120) may be said to train a neural network model by changing the ratio of GAN loss based on the produced feature values.

As described above, the electronic device (100) according to the present embodiment selectively performs GAN learning or non-GAN learning according to feature values, or performs learning by varying the GAN loss applied according to feature values, thereby preventing excessive application of blur improvement to image areas with high complexity, such as faces or text, and enabling image quality improvement processing.

Meanwhile, in the above, only a simple configuration constituting the electronic device (100) has been illustrated and described, but various additional configurations may be provided during implementation. This will be described below with reference to FIG. 3.

FIG. 3 is a diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 3, the electronic device (100') may include a memory (110), at least one processor (120), a communication device (130), a display (140), and an operating device (150).

The configuration of the memory (110) and at least one processor (120) has been described above in FIG. 2, and only operations different from those in FIG. 2 will be described below.

The communication device (130) is a configuration that performs communication with various types of external devices according to various types of communication methods. The communication device (130) may include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, etc. Here, each communication module may be implemented in the form of at least one hardware chip.

In addition to the above-described communication method, the wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards, such as zigbee, 3G (3r Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), and 5G (5th Generation).

In addition, the communication device (110) may include at least one of wired communication modules that perform communication using a LAN (Local Area Network) module, an Ethernet module, a pair cable, a coaxial cable, an optical fiber cable, or a UWB (Ultra Wide-Band) module.

The communication device (130) can receive a data set. Specifically, the communication device (130) can receive a data set for training a neural network model from a server (not shown) that provides the data set.

The communication device (130) can input an image to be subjected to image enhancement or upscaling. In addition, the communication device (130) can output an image to which image enhancement or upscaling is applied. This image can be a single still image, or a moving image composed of multiple images. In addition, the image can be received as an image itself, received as an image within a web page, or an image composed from a broadcast signal. For example, if the electronic device according to the present disclosure is a TV, it can receive a low-resolution broadcast signal, generate an image corresponding to the received broadcast signal, and generate an image in which the generated image is upscaled in the manner described above (i.e., applied to a neural network model).

Or, if the electronic device according to the present disclosure is a laptop, smartphone, tablet, or the like that displays a web page, it is possible to receive a web page including a low-resolution image, and, in the process of displaying the received web page, upscale the included low-resolution image to a high-resolution image, and display the web page including the upscaled high-resolution image.

The communication device (130) can transmit and receive the learned neural network model. For example, if the electronic device (100) is a server that learns a neural network model, the communication device (130) can be controlled to transmit the neural network model on which the learning operation has been performed to another device so that the other device can utilize it. Meanwhile, if the electronic device (100) is a TV or the like, the manufacturer can receive a learned neural network model provided by the manufacturer through the communication device (130).

The display (140) may be implemented as a variety of displays, such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, a PDP (Plasma Display Panel), a projector, etc. The display (140) may also include a driving circuit, a backlight unit, etc., which may be implemented as a form of an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. Meanwhile, the display (140) may be implemented as a touch screen combined with a touch sensor, a flexible display, a 3D display, etc.

The display (140) can display various images. For example, the display (140) can display a low-resolution image input through a communication device, or an image for which image quality improvement processing has been performed.

The operating device (150) can receive various settings for the electronic device. Such an operating device (150) can be implemented as a single device (e.g., a touch screen) by being combined with the above-described display (140).

The operating device (150) can set the image quality improvement processing method. For example, the operating device (150) can receive a selection from the user on whether to perform image quality improvement processing on an input image, or if so, the method of image quality improvement. For example, the user can receive input on whether to apply image quality improvement processing on an input image, whether to apply an upscaling method when applying image quality improvement processing, whether to apply blur improvement processing, and, if applying blur improvement processing, whether to not apply blur improvement processing to a feature region.

In addition, the manipulation device (150) can receive inputs such as the method of the learning process of the neural network model. For example, a data set to be applied to the neural network model can be selected, or a learning method for the corresponding neural network model (for example, only the GAN method or a combination of the GAN method and the non-GAN method as in the present disclosure) can be received. In addition, the manipulation device (150) can receive inputs such as what combination of feature values to be described later will be set.

At least one processor (120) can select a data set to be applied when a learning command for a neural network model (200) stored in a memory is input. When a data set is selected, at least one processor (120) can control a communication device (130) to obtain the data set.

When a data set is acquired, at least one processor (120) can perform a learning operation for a neural network model using the data set according to a preset learning method or a learning method selected by the user. When learning for the neural network model is completed, image processing using the learned neural network model can also be performed.

Meanwhile, the electronic device (100) may receive an externally learned neural network model through a communication device (130) and perform image quality processing on an input image using the received neural network model.

Meanwhile, although the electronic device (100) is illustrated in FIG. 3 as including a display, an operating device, etc., if the electronic device (100) is a device that only learns a neural network model, the display, the operating device, etc. may be omitted. In addition, if the electronic device (100) is a device that uses a learned neural network model, it may further include other components not illustrated, such as a speaker, a microphone, etc., in addition to the illustrated components.

Figure 4 is a diagram illustrating an example of image quality degradation due to application of GAN.

Referring to Figure 4, the text area within the output image is shown using a neural network model learned in the GAN method for improving the blur area.

Specifically, the GAN method according to the present disclosure is a learning method applied to improve the blur area during the learning process of a neural network model that performs image quality improvement processing. For example, the GAN method trains two networks adversarially, one of which is a network that generates data similar to real data, and the other is a model that determines the generated result.

In this disclosure, since the GAN method is applied to improve the blurred portion within the image, there is an effect of improving the blurred portion of the entire image. However, as shown in Fig. 4, when the GAN method is applied to improve the blurred portion of an image such as text, the blur improvement may be excessively applied, which may create artifacts. As shown, in the case of text, if the blur improvement is excessively applied, the boundary of the text may deteriorate as shown.

Also, although not in the city, blur enhancement can actually create an effect when applied to areas with high texture, such as a person's face or text.

To improve these points, the present disclosure applies a method to prevent blur enhancement from being applied or from being applied excessively to areas with high texture complexity, such as text areas and face areas.

Meanwhile, for the above-described operation, a method for determining whether the area in question is an area with high texture complexity, such as a text area or a face area, is required, and if it is determined to be an area with high texture complexity, a method for operating differently from the existing GAN method must be considered. Hereinafter, the operation of the present disclosure will be described in more detail with reference to FIGS. 5 to 8.

FIG. 5 is a diagram for explaining the selective GAN application operation of the present disclosure.

Referring to FIG. 5, a neural network model (200) is disclosed. The neural network model may be referred to as a neural network, a network model, or a deep learning model. When an image is input, the neural network model (200) can output an image with improved image quality for the input image. In addition, when a low-resolution image is input, the neural network model can output a high-resolution image with a higher resolution than the input resolution, in addition to improving the image quality.

When simply upscaling a low-quality image, the artwork becomes noticeable due to insufficient pixel information, so recently, upscaling is performed using a neural network model (200). When upscaling is performed using a neural network model, insufficient pixel information can be filled in according to learned information, which has the effect of obtaining a higher quality result.

However, since the neural network model performs an upscaling operation using the learned network values for the insufficient pixel information, there are some parts of the output image that are somewhat blurry. In order to improve these blurry parts, GAN was applied in the learning process of the neural network model in the past, and higher quality results were obtained than when only the neural network model was used.

For example, as illustrated, a judgment model (S510) is used to compare an output image (20) of a neural network model with a high-quality image (30) for an actual input image (10), and an operation is performed to compare whether the output image (20) and the actual high-quality image are the same or not. If it is determined that they are not the same, learning can proceed until they are determined to be the same.

However, apart from being able to obtain high-quality results for the entire image, it was confirmed that there was a problem in which the blur improvement operation was excessively processed in areas such as text areas and face areas due to the application of GAN, resulting in artifacts.

In other words, GAN should be applied to improve blurred areas, but the GAN method should not be applied to areas with complex textures, such as text areas and face areas, or the GAN method should be applied to a lower degree than other areas.

For this application, two issues need to be resolved: how to determine the areas where the GAN method is not applicable or is applied to a low degree, and how to apply it to a low degree and train it even when it is not applicable or is not applied.

In the present disclosure, the first part is solved by extracting feature information (S520) from an image and using the extracted extracted information. Here, the feature information is a feature related to the texture in the image, and in the present disclosure, examples of calculating it through roughly three methods are described below. However, it is not limited to the methods, and if it is related to the feature of the texture of the image, a method not described in the present disclosure or a method of the present disclosure can be used by partially changing it.

For the second part, the feature information described above is numerically calculated and the calculated value is applied as the correction value (or weight) of the GAN loss. In the following, the calculated value is simply multiplied by the GAN loss and used, that is, it is explained assuming that the degree of application is varied depending on the degree of texture. However, when implementing, it can also be implemented in a form where a value of 1 is output so that GAN is applied if the calculated feature value is less than a certain value, and a value of 0 is output so that GAN loss is not applied if the calculated feature value is greater than a certain value.

That is, as in the illustrated step 530, if the output context value (i.e., feature value) is greater than a certain value, GAN is applied, and if the output context value is less than a certain value, GAN is not applied. In addition, the above-described operation can also be implemented in the opposite form depending on the property of the context value. That is, if the context value is proportional to the complexity, it is also possible to not apply GAN if the context value is greater than a preset value, and to apply GAN if the context value is less than a preset value.

In addition, although the illustrated example shows that one preset value and the calculated value are compared, two comparison values may be used during implementation. For example, if the calculated value is lower than the first value, GAN is applied, if it is between the first and second values, GAN is applied by adjusting the GAN loss in proportion to the calculated value, and if it is greater than the second value, GAN is applied.

Meanwhile, in the above, the GAN method is applied to improve the blur area of the output image of the neural network model, and optionally the GAN method is not applied. However, conversely, the application of the GAN method can also be implemented in a form in which the GAN method is applied to improve the sharpness area, and optionally the GAN method is not applied. In addition, although blur/sharpness is mentioned as one factor of image quality improvement in the above, the above-described method can also be used for various image quality improvement processes such as smoothing and brightness correction in addition to the factor.

Below, the learning operation of the neural network model according to the present disclosure is explained based on the basic contents described above.

Neural network models can be trained using multiple data sets. First, we explain the learning operation using the GAN method.

The GAN method inputs a low-resolution image from a data set into a neural network model to generate an output high-resolution image, and a comparator (510) compares the generated high-resolution output image with an actual high-resolution image to determine whether they are the same or not.

If they are not identical, each neuron of the neural network model is trained so that the two images have the same shape, and the neural network model is trained by repeating the above-described process.

The loss function of GAN in this case can be as shown in mathematical expression 1 below.

[Mathematical formula 1]

As explained above, if a neural network model is trained only with the above-described loss function, artifacts may occur in areas with high complexity. Therefore, in order to prevent the GAN loss from being applied to areas with high complexity or to apply a low GAN loss, the following loss function is used in this disclosure.

[Mathematical formula 2]

은 GAN 로스이고, Context는 선택적으로 GAN 방식의 적용을 위한 본 개시의 특징 값이다. Here, Loss is the premise loss function, and L1 is the loss function of the neural network model.

is the GAN loss, and Context is a feature value of the present disclosure for optionally applying the GAN method.

Specifically, in the learning process of an image, multiple patches are extracted, and learning operations are performed in units of multiple patches. The application of the GAN method or non-GAN method of the present disclosure can be performed in units of these patches.

Accordingly, the electronic device of the present disclosure can extract a patch, generate a feature value from the extracted patch, and compare the generated feature value with a preset value to determine whether to perform learning on the patch in a GAN manner or apply it in a non-GAN manner.

Here, the feature value is a value indicating whether blur processing is not applied to text, faces, etc. within the patch, or a value numerically indicating the complexity of the texture, and can be referred to as a context value, texture value, etc.

Below, various methods for calculating feature values from a patch are described with reference to FIGS. 6 to 8.

FIG. 6 is a diagram for explaining an example of generating feature information according to one embodiment of the present disclosure.

Referring to FIG. 6, multiple patches (620, 630) within an image (610) can be extracted. In the illustrated example, only a patch (620) with high complexity and a patch (530) with relatively low complexity are illustrated for explanation, but upon implementation, a large number of patches can be extracted according to a preset method or the operation of a neural network model.

There may be different features for each extracted patch, for example, the first patch (620) may have a lot of texture, and the second patch (630) may have little texture.

As described above, if blur processing is excessively applied to an image area (or patch) (620) with a lot of texture, artifacts may be created. Conversely, blur processing may improve the image quality of an image area (or patch) (630) with little texture.

In this way, the GAN method needs to be applied according to the texture status of the patch. Therefore, in order to judge this texture status, it is necessary to quantify the texture status.

First, statistical values related to texture features can be calculated and utilized. These statistical values can be calculated in the following mathematical expression 3.

[Mathematical Formula 3]

Here, entropy is a statistical value related to texture features.

In the illustrated example, it can be confirmed that an area (620) with a complex texture produces a relatively high entropy value (e.g., 6.9749), and an area (630) with a non-complex texture produces a relatively low entropy value (e.g., 5.4652). In this way, it can be confirmed that the entropy value used in the present disclosure produces a relative value with respect to the complexity of the texture.

Second, values related to texture features can be utilized through image transformation. Specifically, an entropy map indicating whether each pixel in the extracted patch is similar to surrounding pixels can be generated, and feature information can be derived using the generated entropy map. For example, an entropy map can be derived using the brightness difference between each pixel and the nine surrounding pixels adjacent to each pixel in the patch, and the derived entropy map can be utilized.

In the illustrated example, it can be seen that the entropy map (640) for the area (620) with a complex texture has a relatively white color, while the entropy map (650) for the area (630) with a non-complex texture has a dark color. When implemented, the entropy map can be used as feature information as is, or the average brightness value of the entropy map can be calculated and used as a feature value.

In addition, in the implementation, a GLCM matrix may be used in addition to the above-described method. This is described below with reference to Fig. 7.

FIG. 7 is a diagram for explaining an example of generating feature information according to one embodiment of the present disclosure.

Specifically, FIG. 7 is a diagram for explaining an embodiment of generating feature information using GLCM (Glay-level co-occurrence matrix) values.

GLCM can be referred to as intensity co-occurrence matrix, spatial intensity dependence matrix, and is a statistical method for retrieving texture by considering the spatial relationship of pixels. The GLCM function calculates how often a pair of pixels with a specific value in an image occurs in a specific spatial relationship to generate a GLCM, and extracts statistical measures (e.g., contrast, correlation, energy, homogeneity) from this matrix to identify the texture characteristics of the image.

For example, if the gray value is expressed in three steps from 0 to 3, the first value of the GLCM matrix is the number of adjacent two pixels having the value 00, and the second value is the number of adjacent two pixels having the value 01. A GLCM matrix can be created in this way.

And using the GLCM matrix generated in this way, the following various texture-related measurement values can be derived.

Here, the contrast value can measure the local change in the luminance co-occurrence matrix. For example, it represents the contrast difference within a patch and can be calculated through the following mathematical expression 4.

[Mathematical formula 4]

Here, i and j are the matrix coordinates of the GLCM matrix, and this contrast value has a high value when the texture is complex.

The correlation value represents the joint probability of a given pair of pixels, for example, the correlation between pixels within a patch, and can be calculated using the following mathematical expression 5.

[Mathematical Formula 5]

These correlation values can have low values if the texture is complex.

The energy value represents the sum of the squares of the GLCM elements, and can be referred to as the mean square value, uniformity, and ASM value, and can be calculated through the following mathematical expression 6.

[Mathematical Formula 6]

These energy values can have lower values if the texture is complex.

The identity value (or homogeneity) is a value that measures how closely the GLCM elements are distributed along the GLCM diagonal, indicating the similarity of the GLCM values along the diagonal direction within the patch, and can be calculated using the following mathematical expression 7.

[Mathematical formula 7]

These identical values can have lower values if the texture is complex.

Once the patches are extracted in this way, a GLCM matrix for the extracted patches can be calculated, and the four feature values described above (e.g., contrast value, correlation value, energy value, and identity value) can be calculated using the calculated GLCM matrix.

Meanwhile, in the above, only the method of calculating individual feature values for each patch was described, but during implementation, the feature values can be calculated by combining the above-described methods. This is described below with reference to Fig. 8.

FIG. 8 is a diagram for explaining an example of generating feature information according to one embodiment of the present disclosure.

Referring to Fig. 8, it is a drawing showing an example of feature information calculated for two patches having different texture characteristics. Specifically, for each patch, an entropy value is calculated individually, a GLCM matrix is calculated for each patch, and four feature values as in Fig. 7 are calculated, and the calculated entropy value and four features related to the GLCM are combined to finally calculate the feature values.

[Mathematical formula 8]

Here, context value is a feature value, entropy is an entropy value calculated in Fig. 5, contrast is a contrast value calculated in Fig. 7, correlation is a correlation value calculated in Fig. 7, energy is a square root value calculated in Fig. 7, and homogeneity is a homogeneity value calculated in Fig. 7.

As explained above, the correlation value has been processed to change the sign in that the root mean value and the homogeneity value have lower values when the texture is complex, and the entropy value and contrast value have opposite values.

Meanwhile, in the illustrated example, a weight of 1/8 is applied only to entropy, but various types of weights not described can be applied during implementation. In addition, it can be implemented in a form where some of the above-described feature information is omitted.

Referring to Fig. 8, it can be confirmed that a relatively high feature value (0.7021) is produced for a patch with high texture (810), and a relatively low feature value (0.45) is produced for a patch with low texture (840).

In this case, learning can be performed in a non-GAN manner by comparing the preset value (e.g., 0.5) with the calculated feature value, and if the calculated feature value is greater than 0.5, learning can be performed by not applying GAN, and if the calculated feature value is less than 0.5, learning can be performed by applying GAN.

FIG. 9 is a drawing for explaining a control method according to one embodiment of the present disclosure.

Referring to Fig. 9, a data set for training a neural network model that improves image quality is obtained (S910).

And the neural network model is trained using the data set (S920). Specifically, for each data set, a patch having a preset size is extracted, feature information for the extracted patch is calculated, and based on the calculated feature information, whether to proceed with learning for the patch using a GAN (Generative adversarial network) method or a non-GAN method is determined, thereby training the neural network model.

For example, for the extracted patch, statistical values related to texture features can be calculated as feature information, or an entropy map indicating the similarity between each pixel in the extracted patch and surrounding pixels can be generated, and feature information can be calculated using the generated entropy map, or a GLCM (Glay-level co-occurrence matrix) value can be calculated for each pixel in the extracted patch, and at least one feature information can be calculated using the calculated GLCM value.

Meanwhile, during implementation, the above-described feature information can be used in combination. At this time, individual weights can be applied to each feature information to finally calculate the feature value.

Once the feature values are derived in this way, the feature values can be compared with preset reference values to determine whether to proceed with learning for the patch using the GAN (Generative adversarial network) method or the non-GAN method, thereby training the neural network model.

FIG. 10 is a flowchart illustrating a selective learning method according to one embodiment of the present disclosure.

Referring to Fig. 10, multiple patches can be extracted for each input data set (S1010). Each of the multiple patches can have a preset size, and the number and position of the generated patches can be based on a preset algorithm or design method of a neural network model.

In this way, when multiple patches are extracted for one data set, feature information can be calculated for each patch (S1020). Specifically, as described above, an entropy value for a patch, an entropy map, or a GLCM matrix can be calculated, and contrast values, correlation values, energy values, and homogeneity values can be calculated using the calculated GLCM matrix. During implementation, only one of the calculated values can be used, and the feature values can be calculated by combining the above-described values.

When the feature value is calculated, the feature value is compared with the preset value (S1030), and if the preset condition is satisfied, the neural network can be trained in a non-GAN manner (S1040). For example, if the feature value is greater than the preset value, the neural network can be trained in a non-GAN manner. Meanwhile, in this case, the case where the feature value is greater than the preset value is considered to satisfy the preset condition, because the calculated feature value is calculated to be proportional to the complexity of the texture. In other words, if the previously calculated feature value is inversely proportional to the complexity of the texture, the case where the feature value is less than the preset value can be considered to satisfy the preset condition.

Conversely, if the preset condition is not satisfied, the neural network can be trained in the GAN manner (S1050). Meanwhile, in the above, it is illustrated and explained that training is performed selectively in the GAN manner or in the non-GAN manner depending on whether the condition is satisfied. However, when implemented, the GAN loss is always applied, and the calculated feature value is applied as the weight value of the GAN loss, so that training progresses with a low GAN loss when the texture is complex, and training progresses with a high GAN loss when the texture is simple.

In addition, if the above-described process is repeatedly performed for all patches of all data sets, the learning of the neural network model can proceed by repeatedly applying the process so that the overall loss value has a small value.

Meanwhile, methods according to at least some of the various embodiments of the present disclosure described above can be implemented in the form of applications that can be installed on existing electronic devices.

Additionally, the methods according to at least some of the various embodiments of the present disclosure described above can be implemented only with a software upgrade or a hardware upgrade for an existing electronic device.

Additionally, the methods according to at least some of the various embodiments of the present disclosure described above may also be performed via an embedded server provided in the electronic device, or via an external server of at least one of the electronic devices.

Meanwhile, according to an embodiment of the present disclosure, the various embodiments described above can be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call instructions stored from the storage medium and operate according to the called instructions, and may include an electronic device (e.g., the electronic device (A)) according to the disclosed embodiments. When the instruction is executed by the processor, the processor can perform a function corresponding to the instruction directly or by using other components under the control of the processor. The instruction can include code generated or executed by a compiler or an interpreter. The machine-readable storage medium can be provided in the form of a non-transitory storage medium. Here, the ‘non-transitory storage medium’ only means that it is a tangible device and does not include a signal (e.g., electromagnetic wave), and this term does not distinguish between a case where data is stored semi-permanently and a case where it is stored temporarily in the storage medium. For example, a 'non-transitory storage medium' may include a buffer in which data is temporarily stored. According to one embodiment, a method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play StoreTM) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be at least temporarily stored or temporarily generated in a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or an intermediary server.

Various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage media that can be read by a machine (e.g., a computer). The device is a device that can call instructions stored from the storage media and operate according to the called instructions, and may include an electronic device (e.g., an electronic device (100)) according to the disclosed embodiments.

When the above instruction is executed by the processor, the processor may perform a function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or an interpreter.

Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by those skilled in the art without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be individually understood from the technical idea or prospect of the present disclosure.

Claims (15)

In electronic devices, A memory storing at least one instruction and a neural network model that improves the quality of an image; and At least one processor for executing at least one instruction stored in the memory to obtain a data set for training the neural network model and training the neural network model using the data set; At least one processor of the above, An electronic device that trains the neural network model by extracting patches having a preset size for each of the above data sets, calculating feature information for the extracted patches, and determining whether to perform learning for the patches using a GAN (generative adversarial network) method or a non-GAN method based on the calculated feature information. In the first paragraph, At least one processor of the above, An electronic device that trains the neural network model for the patch in a GAN manner if the above-described generated feature information satisfies a preset condition, and trains the neural network model for the patch in a non-GAN manner if the above-described generated feature information does not satisfy the preset condition. In the second paragraph, The above GAN method performs a GAN comparison operation to improve the blur area in the image using GAN loss. The above non-GAN method is an electronic device in which the GAN loss is not applied in the loss function of the GAN method. In the first paragraph, The above feature information is, Information for detecting at least one of a human face and text in an image, At least one processor of the above, An electronic device that performs learning on the patch in a non-GAN manner if at least one of a human face or text is detected in the extracted patch, and performs learning on the patch in a GAN manner if neither a human face nor text is detected in the extracted patch. In the first paragraph, At least one processor of the above, An electronic device that calculates statistical values related to texture features of the extracted patch as feature information. In the first paragraph, At least one processor of the above, An electronic device that generates an entropy map that indicates whether each pixel in the extracted patch is similar to surrounding pixels, and calculates the feature information using the generated entropy map. In the first paragraph, At least one processor of the above, An electronic device that calculates a GLCM (Glay-level co-occurrence matrix) value for each pixel in the extracted patch and calculates at least one feature information using the calculated GLCM value. In Article 7, At least one processor of the above, An electronic device that uses the GLCM values to produce at least one of a contrast value indicating a contrast difference within a patch, a correlation value indicating a correlation state between pixels within a patch, a mean square value of GLCM values within a patch, and a homogeneous value indicating a similar state of GLCM values with respect to a diagonal direction within a patch as the feature information. In Article 8, At least one processor of the above, Generate multiple feature information, and apply a weight corresponding to the feature information to the generated feature information to generate a feature value. An electronic device that trains the neural network model by comparing the above-described generated feature values with preset reference values and determining whether to proceed with learning for the patch using a generative adversarial network (GAN) method or a non-GAN method. In the first paragraph, The above neural network model is, An electronic device, which is a neural network model that generates an output image of a second resolution with higher quality than the first resolution from an input image of a first resolution. In the first paragraph, At least one processor of the above, An electronic device that inputs the selected image into a learned neural network model when an image is selected to obtain an output image. In a method for controlling an electronic device, A step of obtaining a data set for training a neural network model that improves the image quality; and A step of training the neural network model using the above data set; The above learning steps are: A step of extracting a patch having a preset size for each of the above data sets; A step of producing feature information for the extracted patch; and A control method comprising: a step of training the neural network model by determining whether to proceed with learning for the patch in a GAN (generative adversarial network) manner or a non-GAN manner based on the generated feature information; In Article 12, The step of producing the above characteristic information is: A control method for calculating statistical values related to texture features of the above extracted patches as feature information. In Article 12, The step of producing the above characteristic information is: A control method for generating an entropy map indicating whether each pixel in the extracted patch is similar to surrounding pixels, and calculating the feature information using the generated entropy map. In a non-transitory computer-readable recording medium storing a program for executing a method of controlling an electronic device, The above control method is, A step of obtaining a data set for training a neural network model that improves the image quality; and A step of training the neural network model using the above data set; The above learning steps are: A step of extracting a patch having a preset size for each of the above data sets; A step of producing feature information for the extracted patch; and A computer-readable recording medium comprising: a step of training the neural network model by determining whether to proceed with training for the patch using a generative adversarial network (GAN) method or a non-GAN method based on the generated feature information;

Images