JP6236731B1 - Super-resolution processing apparatus, super-resolution processing method, and computer program - Google Patents

Abstract

A high-precision super-resolution image is created at high speed.
A second resolution image is converted into a third resolution image for each of a plurality of learning subsets in a first hierarchy, which is a subset of a learning set including a third resolution image having a higher resolution than the second resolution. A first filter for super-resolution processing, and a first layer filter acquisition unit for acquiring a second filter for super-resolution processing of a first resolution image having a resolution lower than the second resolution into a second resolution image A reduced image creation unit that creates a reduced image of the first resolution from the input image of the second resolution, and a second resolution candidate image by performing super-resolution processing on the reduced image of the first resolution using each second filter The second resolution input image using the first filter corresponding to the second filter that minimizes the difference between the second resolution candidate image and the second resolution input image. resolution And a super-resolution processing section for management.
[Selection] Figure 1

Description

  The present invention relates to a super resolution processing apparatus, a super resolution processing method, and a computer program for performing super resolution processing on an input image.

  In recent years, a super-resolution processing technique for increasing the resolution by performing super-resolution processing on an input image has been put into practical use (see, for example, Patent Document 1).

  As an example of super-resolution processing, input images are input to a machine-learned convolutional neural network (hereinafter referred to as “CNN”), and the input images are subjected to super-resolution processing to increase the resolution. There is known a method for outputting the output image. The CNN is configured by performing machine learning on the weight and bias of the neural network in each layer, using the reduced image as an input and the original image as an output based on the original image and the reduced image of the original image.

Special table 2013-532878 gazette

  There are various types of images such as portraits, landscapes, and illustrations. For example, illustrations include many high frequency components because they include many edge components, while landscape images include all frequency components evenly.

  However, if the characteristics of these images are ignored and the input image is super-resolution processed using one CNN, a satisfactory result may not be obtained. For example, it is conceivable that the reproducibility of the high-frequency component is impaired when the super-resolution processing is performed on an image containing a large amount of high-frequency components by CNN that has been machine-learned using many images that include a large amount of low-frequency components. Further, when an image containing a large amount of low-frequency components is subjected to super-resolution processing by machine learning using a large number of images containing a large amount of high-frequency components, high-frequency noise may be generated.

  In addition, it is generally difficult for the user to select an optimal one from a plurality of CNN presets. For this reason, conventionally, it has been necessary to create a CNN using many types of learning images as described above. When many types of learning images are used, the number of learning images is enormous, so that time is required for machine learning and the configuration of the CNN is complicated.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a super-resolution processing apparatus, a super-resolution processing method, and a computer program that can create a high-precision super-resolution image at high speed. .

  In order to achieve the above object, a super-resolution processing apparatus according to an aspect of the present invention performs super-resolution processing on a second resolution image to create a third resolution image having a higher resolution than the second resolution. A resolution processing apparatus, wherein each of a plurality of learning sub-sets of a first hierarchy, which is a sub-set of a learning set including an image of a third resolution, is machine-learned using the learning subset, Obtaining a first filter for super-resolution processing of an image into a third resolution image and a second filter for super-resolution processing of a first resolution image lower than the second resolution into a second resolution image A first layer filter acquisition unit, a reduced image generation unit that generates a reduced image of the first resolution from an input image of the second resolution, and each of the second filters acquired by the first layer filter acquisition unit. A candidate image creation unit that creates a second resolution candidate image by performing super-resolution processing on the reduced image of the first resolution, and the second resolution candidate image created by the candidate image creation unit and the second resolution Super-resolution processing for generating a super-resolution image of the third resolution by performing super-resolution processing of the input image of the second resolution using the first filter corresponding to the second filter that minimizes the difference from the input image. A part.

  According to this configuration, the second filter that minimizes the difference between the second resolution candidate image and the second resolution input image can be selected from the plurality of second filters. In addition, the input image can be super-resolution processed using the first filter corresponding to the selected second filter. Since the second filter is machine-learned using the same learning subset as the first filter, it has the same properties as the first filter, and the resolution of the input image targeted by the second filter is the first. The resolution is smaller than the resolution of the input image targeted by the filter. For this reason, the second filter having the optimal property for the input image can be selected at high speed, and by executing the super-resolution processing on the input image using the first filter corresponding to the selected second filter, Accurate super-resolution images can be created at high speed.

  Preferably, the above-described super-resolution processing apparatus further includes a plurality of learnings in a second hierarchy that is a subset of the learning subset in the first hierarchy used for machine learning of the second filter having the smallest difference. A second hierarchical filter acquisition unit that acquires a first filter and a second filter machine-learned using the learning subset for each of the subsets, and the candidate image creation unit further includes the second hierarchical filter Using each second filter of the second hierarchy acquired by the acquisition unit, the reduced image of the first resolution is subjected to super resolution processing to create a second resolution candidate image, and the super resolution processing unit further includes: The second image in which the difference between the second resolution candidate image created by the candidate image creation unit using the second filter of the second hierarchy and the second resolution input image is minimized. Using a first filter corresponding to the data, the input image of the second resolution by super-resolution processing, to create a super-resolution image of the third resolution.

  According to this configuration, it is possible to select the second filter in the first hierarchy, and further select the second filter in the second hierarchy based on the selected second filter. In addition, the input image can be super-resolution processed using the first filter corresponding to the selected second filter. Thus, since the first filter and the second filter can be hierarchically structured, the second filter and the first filter corresponding to the second filter can be efficiently selected. Thereby, a highly accurate super-resolution image can be created at high speed.

A super-resolution processing method according to another aspect of the present invention provides a super-resolution for causing a device that creates a third-resolution image having a higher resolution than the second resolution to function by super-resolution processing the second-resolution image. A second resolution image, which is a processing method, machine-learned using each of the plurality of learning subsets in the first hierarchy, which is a subset of the learning set including a third resolution image, using the learning subset. A first filter for super-resolution processing of a first resolution image into a third resolution image, and a second filter for super-resolution processing of a first resolution image lower than the second resolution into a second resolution image A step of creating a reduced image of the first resolution from the input image of the second resolution, and using the acquired second filter, the reduced image of the first resolution is subjected to super-resolution processing and second Using a first filter corresponding to a second filter that minimizes a difference between the created second-resolution candidate image and the second-resolution input image; Creating a third resolution super-resolution image by subjecting the second resolution input image to super-resolution processing.

  This configuration includes steps corresponding to the processing unit included in the super-resolution processing apparatus described above. For this reason, the same operation and effect as the above-described super-resolution processing apparatus can be achieved.

  A computer program according to another aspect of the present invention is a computer program for super-resolution processing a second resolution image to create a third resolution image having a higher resolution than the second resolution. , For each of the plurality of learning subsets of the first hierarchy, which is a subset of the learning set including the third resolution image, the second resolution image machine-learned using the learning subset is converted to the third resolution image. First-layer filter acquisition for acquiring a first filter for performing super-resolution processing on an image and a second filter for performing super-resolution processing on an image having a lower resolution than the second resolution to an image having a second resolution A reduced image creation unit that creates a reduced image of the first resolution from the input image of the second resolution, and each of the second filters acquired by the first hierarchical filter acquisition unit , A candidate image creation unit that creates a second resolution candidate image by performing super-resolution processing on the first resolution reduced image, the second resolution candidate image created by the candidate image creation unit, and the second A super-resolution image of the third resolution is created by performing super-resolution processing on the input image of the second resolution using the first filter corresponding to the second filter that minimizes the difference from the input image of the two resolutions. It functions as a super-resolution processor.

  According to this configuration, the computer can function as the above-described super-resolution processing apparatus. For this reason, a highly accurate super-resolution image can be created at high speed.

  The computer program according to the present invention can be distributed via a computer-readable non-transitory recording medium such as a CD-ROM (Compact Disc-Read Only Memory) or a communication network such as the Internet. Needless to say. In addition, the present invention can be realized as a semiconductor integrated circuit that realizes part or all of the super-resolution processing apparatus, or can be realized as a system including the super-resolution processing apparatus.

  According to the present invention, a highly accurate super-resolution image can be created at high speed.

It is a block diagram which shows the structure of the super-resolution processing apparatus which concerns on embodiment of this invention. It is a figure for demonstrating the size and filter of the image which are handled with embodiment of this invention. It is a figure which shows an example of the learning set and filter which were hierarchically structured. It is a flowchart which shows the process sequence of the filter creation process which the super-resolution processing apparatus which concerns on embodiment of this invention performs. It is a flowchart which shows the process sequence of the super-resolution process of the input image which the super-resolution processing apparatus which concerns on embodiment of this invention performs.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Each of the embodiments described below shows a preferred specific example of the present invention. Numerical values, shapes, components, arrangement positions and connection forms of components, steps, order of steps, and the like shown in the following embodiments are merely examples, and are not intended to limit the present invention. The invention is specified by the claims. Therefore, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims indicating the highest concept of the present invention are not necessarily required to achieve the object of the present invention. It will be described as constituting a preferred form.

  FIG. 1 is a block diagram showing a configuration of a super-resolution processing apparatus according to an embodiment of the present invention. FIG. 2 is a diagram for explaining image sizes and filters handled in the embodiment of the present invention.

  Referring to FIG. 1, a super-resolution processing apparatus 1 is an apparatus that performs super-resolution processing on an input image and creates an image having a higher resolution than the input image. Unit 11, machine learning unit 12, first layer filter acquisition unit 13, candidate image creation unit 14, second layer filter acquisition unit 15, super-resolution processing unit 16, and storage device 17.

  In this embodiment, an example will be described in which super-resolution processing is performed on an input image of 2K size (1920 × 1080 pixels) to create an image of 4K size (3840 × 2160 pixels). However, the image size targeted by the super-resolution processing apparatus 1 is not limited to these.

  The input image acquisition unit 10 acquires an input image to be subjected to super resolution processing. For example, the input image acquisition unit 10 reads out an image selected by the user using a keyboard or the like from the images stored in the storage device 17 from the storage device 17, thereby acquiring the image as an input image. Also good. The input image acquisition unit 10 may acquire the image as an input image by downloading the image via a network or the like.

  As shown in FIG. 2, in the present embodiment, the size (resolution) of the input image is assumed to be 2K size (second resolution).

  Referring to FIG. 1 again, the reduced image creation unit 11 creates a reduced image by reducing the input image acquired by the input image acquisition unit 10.

  As shown in FIG. 2, in the present embodiment, the reduced image creating unit 11 reduces the input image by 1/2 in both the vertical and horizontal directions. The size (resolution) of the reduced image is assumed to be a QHD (Quarter High Definition) size (960 × 540 pixels) (first resolution).

  In addition to the input image, the reduced image creation unit 11 reduces the learning image prepared for machine learning of a filter for super-resolution processing.

  That is, it is assumed that a learning set including a plurality of 4K-size learning images is stored in the storage device 17 in advance. In addition, the learning set includes various types of learning images such as portrait photographs, landscape photographs, and illustrations. The reduced image creation unit 11 reads out the learning set from the storage device 17, reduces the learning images included in the learning set to 1/2 each of the vertical and horizontal sizes, and the vertical and horizontal sizes. A reduced image of QHD size reduced to ¼ is created.

  Hereinafter, the 2K size reduced image and the QHD size reduced image created from the learning image are also referred to as the learning image. In addition, the learning set includes not only a plurality of 4K-size learning images but also a 2K-size learning image and a QHD-size learning image created from them. In this embodiment, the size of the input image is the same as that of the 2K-size learning image included in the learning set, but the size of both is not necessarily the same, and the size of the learning image is not necessarily the same. May be different from the size of the input image.

  The machine learning unit 12 creates a filter for performing super-resolution processing by machine learning using images included in the learning set. The machine learning unit 12 hierarchically structures images included in the learning set, and creates a filter in each layer.

FIG. 3 is a diagram illustrating an example of a learning set and filter having a hierarchical structure.
In other words, the machine learning unit 12 creates learning sets and filters that are hierarchically structured in the following procedure.

  First, the machine learning unit 12 uses a 2K-size learning image and a 4K-size learning image included in the learning set, and performs CNN for super-resolution processing of a 2K-size image into a 4K-size image. Create (Convolutional Neural Network). As shown in FIG. 2, CNN that performs super-resolution processing of a 2K size image into a 4K size image is referred to as a first CNN or a first filter. The machine learning unit 12 creates a first CNN by machine learning of weights and biases in each layer of the CNN using a 2K-size learning image as a CNN input and a 4K-size learning image as a CNN output. The first CNN created using the learning set is referred to as the first CNN0 as shown in FIG.

  In the present embodiment, the machine learning unit 12 will be described using CNN as an example of a filter for super-resolution processing, but the filter is not limited to CNN, and machine learning is performed from a plurality of learning images. The present invention can also be applied to other filters.

  Next, the machine learning unit 12 classifies the learning set into three subsets (learning subsets 1 to 3) using the first CNN0. For example, the machine learning unit 12 performs the above classification based on the difference between the image that has been super-resolution processed by the first CNN0 and the learning image.

  Specifically, the machine learning unit 12 creates a 4K size image by inputting each learning image of 2K size included in the learning set to the first CNN0.

  Next, the machine learning unit 12 calculates a difference between the 4K size image created by the first CNN0 and the 4K size learning image used to create the 2K size learning image. The difference between images can be obtained by, for example, the sum of squares or the sum of absolute values of pixel value differences.

  Next, the machine learning unit 12 classifies the 4K size learning image into one of three subsets based on the calculated difference.

  For example, the machine learning unit 12 classifies learning images having a 4K size whose difference is larger than the first threshold into the learning subset 1. In addition, the machine learning unit 12 sets a learning image having a 4K size less than the first threshold and having a difference larger than a second threshold (where the second threshold is smaller than the first threshold) to the learning subset 2. Classify into: In addition, the machine learning unit 12 classifies learning images having a 4K size whose difference is equal to or smaller than the second threshold into the learning subset 3. The machine learning unit 12 uses the same learning subset as the 4K size learning image for the 2K size learning image and the QHD size learning image created by the reduced image creation unit 11 from the 4K size learning image. Classify. Thereby, as shown in FIG. 3, the learning set is classified into learning subsets 1 to 3.

  Since the difference indicates an error of the image subjected to the super-resolution processing, the learning set is classified into a plurality of learning subsets based on the error.

  The machine learning unit 12 uses each of the learning subsets 1 to 3 to create a first CNN and a CNN (referred to as a second CNN or a second filter) for super-resolution processing of a QHD size image into a 2K size image. .

  For example, the machine learning unit 12 uses a 2K-size learning image included in the learning subset 1 as an input of the CNN and a 4K-size learning image included in the learning subset 1 as an output of the CNN. A first CNN is created by machine learning of weights and biases.

  Further, the machine learning unit 12 uses the QHD size learning image included in the learning subset 1 as an input of the CNN, and uses the 2K size learning image included in the learning subset 1 as an output of the CNN, in each layer of the CNN. A second CNN is created by machine learning of weights and biases.

  The machine learning unit 12 similarly creates the first CNN and the second CNN for the learning subsets 2 and 3 as well.

  The first CNN and the second CNN created from the learning subset i are described as the first CNNi and the second CNNi, respectively (i = 1 to 3).

  The machine learning unit 12 further classifies each of the learning subsets 1 to 3 into three subsets.

  For example, the machine learning unit 12 classifies the learning subset 1 into three subsets (learning subsets 1-1 to 1-3) using the first CNN1. The classification method is the same as that of the learning set except that the classification target is different. Therefore, detailed description thereof will not be repeated here.

  In addition, the machine learning unit 12 creates a first CNN and a second CNN for each of the learning subsets 1-1 to 1-3. The creation method of the CNN is the same as the creation method of the first CNN and the second CNN using the learning subset 1, except that the learning subset to be used is different. Therefore, detailed description thereof will not be repeated here.

  The machine learning unit 12 also performs learning subset classification processing and CNN creation processing using the classified learning subsets for the learning subsets 2 and 3.

  A learning subset obtained by classifying the learning subset i is referred to as a learning subset i-j (i = 1 to 3, j = 1 to 3). Further, the first CNN and the second CNN created from the learning subset i-j are referred to as a first CNNi-j and a second CNNi-j, respectively.

  By the processing executed by the machine learning unit 12, a learning set and a CNN classification tree having a resolution structure as shown in FIG. 3 are created. Assuming that the learning set and the first CNN0 are the zeroth hierarchy, the first hierarchy includes the learning set i, the first CNNi, and the second CNNi. The second hierarchy includes a learning set i-j, a first CNNi-j, and a second CNNi-j.

  The first layer filter acquisition unit 13 to the super resolution processing unit 16 described below are processing units for performing super resolution processing on an input image.

  The first hierarchy filter acquisition unit 13 acquires the first CNN and the second CNN for each of the learning subsets 1 to 3 in the first hierarchy.

  The candidate image creation unit 14 uses the second CNNi acquired by the first hierarchical filter acquisition unit 13 to perform super-resolution processing on the QHD size reduced image obtained by reducing the input image, and generates a 2K size image. The created 2K size image is called a candidate image. In other words, the candidate image creation unit 14 creates three 2K size candidate images by inputting the QHD size reduced image to the second CNNi.

  The second hierarchy filter acquisition unit 15 acquires the first CNNi-j and the second CNNi-j machine-learned using the learning subset i-j for each of the plurality of learning subsets i-j in the second hierarchy. (I = 1-3, j = 1-3).

  Further, the candidate image creation unit 14 uses the second CNNi-j acquired by the second layer filter acquisition unit 15 to perform super-resolution processing on the QHD size reduced image obtained by reducing the input image, thereby obtaining a 2K size image (hereinafter, referred to as “2K size”). "Candidate image"). That is, the candidate image creation unit 14 creates a 2K size candidate image by inputting a QHD size reduced image to the second CNNi-j.

  The super-resolution processing unit 16 calculates the difference between each of the three 2K-size candidate images created from the second CNNi by the candidate image creation unit 14 and the 2K-size input image. The method for calculating the difference between images is as described above.

  The super-resolution processing unit 16 identifies the second CNNm used to create the candidate image with the smallest calculated difference, and identifies the learning subset m used for machine learning of the identified second CNNm.

  The super-resolution processing unit 16 specifies a learning subset mj (j = 1 to 3) that is a subset of the specified learning subset m. For example, when m = 1, learning subsets 1-1 to 1-3 are specified.

  The super-resolution processing unit 16 further compares the difference from the 2K-size input image from among the three candidate images created using the second CNNm-j of the learning subset mj (j = 1 to 3). The candidate image that minimizes is identified.

  The super-resolution processing unit 16 performs super-resolution processing on the 2K-size input image using the first CNNm-n corresponding to the second CNNm-n used to create the identified candidate image. Create a resolution image.

The storage device 17 is a storage device for storing images and various data, and is configured by an HDD (Hard Disk Drive), a nonvolatile memory, a volatile memory, or the like.
Next, a procedure of processing executed by the super resolution processing apparatus 1 will be described.

  FIG. 4 is a flowchart showing a processing procedure of filter creation processing executed by the super-resolution processing apparatus 1 according to the embodiment of the present invention.

  As shown in FIG. 4, the reduced image creating unit 11 reduces the image from each 4K-size learning image included in the learning set stored in the storage device 17 to ½ each vertically and horizontally. The learning image of 2K size and the learning image of QHD size reduced to ¼ each in the vertical and horizontal directions are created (S1).

  The machine learning unit 12 uses a machine learning to convert a 2K size image into a 4K size image based on the 4K size learning image and the 2K size reduced image included in the learning set. (First CNN0) is created (S2).

  The machine learning unit 12 creates a 4K size image by inputting each 2K size learning image included in the learning set to the first CNN0. Next, the machine learning unit 12 calculates a difference between the 4K size image created by the first CNN0 and the 4K size learning image used to create the 2K size learning image. The machine learning unit 12 classifies the 4K-size learning image into any one of the three learning subsets 1 to 3 based on the calculated difference (S3).

  The machine learning unit 12 creates the first CNNi and the second CNNi using the learning subset i (i = 1 to 3) (S4). The machine learning unit 12 stores the created CNN in the storage device 17. The machine learning unit 12 repeatedly executes machine learning while appropriately replacing the learning subset i (i = 1 to 3), so that the error with each learning subset is reduced. 2CNNi may be created.

  For each of the learning subsets 1 to 3, the machine learning unit 12 inputs each learning image of 2K size included in the learning subset i (i = 1 to 3) to the first CNNi, thereby obtaining a 4K size image. Create Next, the machine learning unit 12 calculates a difference between the 4K size image created by the first CNNi and the 4K size learning image used to create the 2K size learning image. Based on the calculated difference, the machine learning unit 12 classifies the 4K-size learning image into any one of the three learning subsets ij (j = 1 to 3) (S5). Thereby, each of the learning subsets 1 to 3 is further classified into three learning subsets.

  The machine learning unit 12 creates the first CNNi-j and the second CNNi-j by machine learning using the learning subset ij (i = 1 to 3, j = 1 to 3) (S6). The machine learning unit 12 stores the created CNN in the storage device 17. Note that the machine learning unit 12 repeatedly performs machine learning while appropriately replacing the learning subsets ij (i = 1 to 3, j = 1 to 3), thereby causing an error from each learning subset. The first CNNi-j and the second CNNi-j may be created to be smaller.

  By the filter creation process shown in FIG. 4, a hierarchically structured CNN as shown in FIG. 3 is created.

  FIG. 5 is a flowchart showing the processing procedure of the super-resolution processing of the input image executed by the super-resolution processing device 1 according to the embodiment of the present invention.

  As illustrated in FIG. 5, the input image acquisition unit 10 acquires a 2K-size input image to be subjected to super-resolution processing (S11).

  The reduced image creation unit 11 creates a QHD size reduced image by reducing the 2K size input image acquired by the input image acquisition unit 10 (S12).

  The first hierarchy filter acquisition unit 13 acquires the first hierarchy filter of the classification tree shown in FIG. 3 (S13). That is, the first hierarchy filter acquisition unit 13 acquires the first CNN 1 to 3 and the second CNN 1 to 3 from the storage device 17.

  The candidate image creation unit 14 creates three 2K size candidate images by inputting the QHD size reduced images obtained by reducing the input images to the second CNNs 1 to 3 (S14).

  The super-resolution processing unit 16 calculates a difference between each of the two 2K-size candidate images created in step S14 and the 2K-size input image (S15).

  The super-resolution processing unit 16 specifies the second CNNm used to create the candidate image that minimizes the difference calculated in step S15 (S16). For example, the second CNN used to create the candidate image with the smallest difference is specified as the second CNN1.

  Note that the input to the second CNN 1 to 3 is not limited to a QHD size reduced image. For example, a part of the image cut out from the input image may be input to the second CNN 1 to 3. For example, the partial image may be created by cutting out a small region at regular intervals from the input image, or the partial image may be created by thinning pixels of the input image at regular intervals. . Further, the partial image may be created by cutting out a part of the partial image from a plurality of images assumed to have the same properties as the input image. For example, when the input image is a part of the moving image, a part of the image may be extracted from the moving image of the same scene as the input image, and a part of the extracted image may be cut out.

  Further, the second CNNm may be specified in the same manner as described above from the difference between the candidate image created by the second CNN 1 to 3 based on the partial image and the partial image. By such processing, the time required to specify the second CNNm can be shortened.

  The second hierarchy filter acquisition unit 15 acquires a filter of the second hierarchy, which is a hierarchy lower than the second CNNm of the first hierarchy specified in step S16 (S17). That is, the second hierarchy filter acquisition unit 15 reads the first CNNm-j and the second CNNm-j in the classification tree shown in FIG. 3 from the storage device 17 (j = 1 to 3). For example, the second hierarchy filter acquisition unit 15 reads the first CNN 1-1 to 1-3 and the second CNN 1-1 to 1-3 from the storage device 17.

  The candidate image creation unit 14 creates three 2K size candidate images by inputting the QHD size reduced images obtained by reducing the input image to the second CNNm-j (j = 1 to 3) (S18). .

  The super-resolution processor 16 calculates a difference between each of the 3K candidate images created in step S18 and the 2K-size input image (S19).

  The super-resolution processing unit 16 identifies the second CNNm-n used to create the candidate image that minimizes the difference calculated in step S19 (S20). For example, the second CNN used to create the candidate image with the smallest difference is specified as the second CNN 1-2.

  Note that the input to the second CNNm-j (j = 1 to 3) is not limited to a QHD size reduced image. For example, a part of the image cut out from the input image may be input to the second CNNm-j (j = 1 to 3). For example, the partial image may be created by cutting out a small region at regular intervals from the input image, or the partial image may be created by thinning pixels of the input image at regular intervals. . Further, the partial image may be created by cutting out a part of the partial image from a plurality of images assumed to have the same properties as the input image. For example, when the input image is a part of the moving image, a part of the image may be extracted from the moving image of the same scene as the input image, and a part of the extracted image may be cut out.

  Further, the second CNNm-n is identified in the same manner as described above from the difference between the candidate image created by the second CNNm-j (j = 1 to 3) based on the partial image and the partial image. Also good. By such processing, the time required for specifying the second CNNm-n can be shortened.

  The super-resolution processing unit 16 performs super-resolution processing on the input image of 2K size using the first CNNm-n corresponding to the second CNNm-n of the second hierarchy specified in step S20, thereby super-resolution of 4K size. An image is created (S21). In the above example, the super-resolution processing is performed using the first CNN 1-2.

  As described above, according to the embodiment of the present invention, the second CNN that minimizes the error when the QHD size reduced image obtained by reducing the 2K size input image is subjected to the super-resolution processing is searched using the classification tree. can do. The second CNN has the same properties as the first CNN, and the resolution of the input image targeted by the second CNN is smaller than the resolution of the input image targeted by the first CNN. For this reason, the second CNN can be searched efficiently. Further, by performing super-resolution processing on the input image using the first CNN corresponding to the searched second CNN, it is possible to create a highly accurate super-resolution image with few errors.

  That is, the first CNN and the second CNN paired with the first CNN are subjected to machine learning using the same learning images having different sizes. For this reason, it can be expected that an image that can be subjected to the super-resolution processing with high accuracy by the second CNN is also accurately processed with the first CNN paired with the second CNN. Accordingly, the first CNN paired with the second NN that minimizes the error is selected, and the input image is super-resolution processed using the selected first CNN, so that the error when the input image is super-resolution processed is reduced. I can expect.

  In addition, since the configuration of the second CNN is made smaller than that of the first CNN and the second CNN is selected and then the first CNN corresponding to the selected second CNN is selected, the processing speed of the selection of the first CNN is improved. be able to. However, the configuration of the second CNN is not necessarily smaller than the configuration of the first CNN.

  In the above-described embodiment, the learning set and the learning subset are each classified into three learning subsets. However, the number of learning subsets to be classified is not limited to three, but two. That is all you need.

  Further, as shown in FIG. 3, the learning subset has two hierarchies, but may have one hierarchy or three or more hierarchies.

  In the above-described embodiment, the learning set and the learning subset are classified based on the difference between the super-resolution processed image and the learning image. However, the classification method is not limited to this. . For example, classification may be performed in consideration of the type of image (illustration, landscape photograph, portrait photograph).

  In the above-described embodiment, the first CNN and the second CNN have been described as receiving the entire area of the image and performing super-resolution processing. However, each CNN has a plurality of block areas obtained by dividing the image, respectively. A configuration may be adopted in which each block area is received as input and merged after super-resolution processing. In addition, each block area may overlap with an adjacent block area, or the size may be different between the block areas. Thus, by configuring the CNN, it is possible to learn the CNN using only a part of the region without inputting the entire region of the learning image.

  Further, the image to be processed by the super-resolution processing apparatus 1 is not limited to a normal image such as an illustration or a photograph. For example, a distribution map of measurement data such as medical scan data such as an MRI (Magnetic Resonance Imaging) image or a CT (Computed Tomography) image may be processed. Further, any processing target that can provide super-resolution processing, such as a vector structure or a three-dimensional structure such as a voxel image, can be used as the input of the super-resolution processing apparatus 1.

  The super-resolution processing apparatus 1 may be specifically configured as a computer system including a microprocessor, ROM, RAM, hard disk drive, display unit, keyboard, mouse, and the like. A computer program is stored in the RAM or hard disk drive. The super-resolution processing device 1 achieves its functions by the microprocessor operating according to the computer program. Here, the computer program is configured by combining a plurality of instruction codes indicating instructions for the computer in order to achieve a predetermined function.

  Further, some or all of the components constituting the super-resolution processing apparatus 1 may be configured by one system LSI (Large Scale Integration). The system LSI is an ultra-multifunctional LSI manufactured by integrating a plurality of components on a single chip, and specifically, a computer system including a microprocessor, ROM, RAM, and the like. . A computer program is stored in the RAM. The system LSI achieves its functions by the microprocessor operating according to the computer program.

  Furthermore, some or all of the constituent elements constituting the super-resolution processing apparatus 1 may be configured as an IC card that can be attached to and detached from the super-resolution processing apparatus 1 or a single module. The IC card or module is a computer system that includes a microprocessor, ROM, RAM, and the like. The IC card or the module may include the super multifunctional LSI described above. The IC card or the module achieves its function by the microprocessor operating according to the computer program. This IC card or this module may have tamper resistance.

  Further, the present invention may be the method described above. Further, the present invention may be a computer program that realizes these methods by a computer, or may be a digital signal composed of the computer program.

  Furthermore, the present invention provides a non-transitory recording medium that can read the computer program or the digital signal, such as a flexible disk, a hard disk drive, a CD-ROM, an MO, a DVD, a DVD-ROM, a DVD-RAM, and a BD. (Blu-ray (registered trademark) Disc), or recorded in a semiconductor memory or the like. Further, the digital signal may be recorded on these non-temporary recording media.

  In the present invention, the computer program or the digital signal may be transmitted via an electric communication line, a wireless or wired communication line, a network represented by the Internet, a data broadcast, or the like.

  The present invention may be a computer system including a microprocessor and a memory, wherein the memory stores the computer program, and the microprocessor operates according to the computer program.

Further, by recording the program or the digital signal on the non-temporary recording medium and transferring it, or transferring the program or the digital signal via the network or the like, another independent computer It may be implemented by the system.
Furthermore, the above embodiment and the above modification examples may be combined.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the meanings described above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

  The present invention is useful when used in a super-resolution processing apparatus or the like that creates an image in which the resolution of an input image is increased.

DESCRIPTION OF SYMBOLS 1 Super-resolution processing apparatus 10 Input image acquisition part 11 Reduced image creation part 12 Machine learning part 13 1st hierarchy filter acquisition part 14 Candidate image creation part 15 2nd hierarchy filter acquisition part 16 Super-resolution process part 17 Storage device

Claims (4)

A super-resolution processing device that performs super-resolution processing on a second-resolution image and creates a third-resolution image that is higher in resolution than the second resolution,
For each of the plurality of learning subsets in the first layer, which is a subset of the learning set including the third resolution image, the second resolution image machine-learned using the learning subset is the third resolution image. A first filter for super-resolution processing, and a first layer filter acquisition unit for acquiring a second filter for super-resolution processing of a first resolution image having a resolution lower than the second resolution into a second resolution image When,
A reduced image creation unit that creates a reduced image of the first resolution from an input image of the second resolution;
A candidate image creation unit that creates a second resolution candidate image by performing super-resolution processing on the reduced image of the first resolution using each of the second filters obtained by the first layer filter acquisition unit;
Using the first filter corresponding to the second filter that minimizes the difference between the second resolution candidate image created by the candidate image creation unit and the second resolution input image, the second resolution input image And a super-resolution processing unit that creates a super-resolution image of the third resolution by performing super-resolution processing. further,
For each of a plurality of learning subsets in the second hierarchy, which is a subset of the learning subset in the first hierarchy used for machine learning of the second filter with the smallest difference, using the learning subset A second hierarchical filter acquisition unit that acquires the first and second machine-learned filters;
The candidate image creation unit further performs super-resolution processing on the reduced image of the first resolution using each of the second filters of the second layer acquired by the second layer filter acquisition unit. Create a candidate image,
The super-resolution processing unit further has a minimum difference between the second-resolution candidate image created by the candidate-image creating unit using the second filter in the second hierarchy and the second-resolution input image. The super-resolution processing apparatus according to claim 1, wherein a super-resolution image of the third resolution is generated by performing super-resolution processing on the input image of the second resolution using a first filter corresponding to the second filter. . A super-resolution processing method for causing a device that performs super-resolution processing of a second resolution image to generate a third resolution image having a higher resolution than the second resolution,
For each of the plurality of learning subsets in the first layer, which is a subset of the learning set including the third resolution image, the second resolution image machine-learned using the learning subset is the third resolution image. Obtaining a first filter for super-resolution processing and a second filter for super-resolution processing of a first resolution image having a lower resolution than the second resolution into a second resolution image;
Creating a reduced image of the first resolution from the input image of the second resolution;
Using each acquired second filter to create a second resolution candidate image by super-resolution processing the reduced image of the first resolution;
Using the first filter corresponding to the second filter that minimizes the difference between the created candidate image of the second resolution and the input image of the second resolution, the input image of the second resolution is subjected to super-resolution processing. Thereby creating a super-resolution image of the third resolution. A computer program for super-resolution processing a second resolution image to create a third resolution image having a higher resolution than the second resolution,
Computer
For each of the plurality of learning subsets in the first layer, which is a subset of the learning set including the third resolution image, the second resolution image machine-learned using the learning subset is the third resolution image. A first filter for super-resolution processing, and a first layer filter acquisition unit for acquiring a second filter for super-resolution processing of a first resolution image having a resolution lower than the second resolution into a second resolution image When,
A reduced image creation unit that creates a reduced image of the first resolution from an input image of the second resolution;
A candidate image creation unit that creates a second resolution candidate image by performing super-resolution processing on the reduced image of the first resolution using each of the second filters obtained by the first layer filter acquisition unit;
Using the first filter corresponding to the second filter that minimizes the difference between the second resolution candidate image created by the candidate image creation unit and the second resolution input image, the second resolution input image A computer program for functioning as a super-resolution processing unit that creates a super-resolution image of the third resolution by performing super-resolution processing on the.

Images