WO2018090355A1 - Method for auto-cropping of images - Google Patents

Abstract

The present invention relates to a method for auto-cropping of images. The method comprises: extracting the aesthetically pleasing diagram and the gradient energy diagram of a to-be-cropped image; densely extracting candidate cropped images from the to-be-cropped image; on the basis of the aesthetically pleasing diagram, screening the candidate cropped images; on the basis of the aesthetically pleasing diagram and the gradient energy diagram, estimating the composition score of the obtained candidate cropped images by means of screening, and determining the candidate cropped image having the highest score as a cropped image. The present invention uses the aesthetically pleasing diagram to explore the aesthetically pleasing region of a picture, and uses the aesthetically pleasing diagram to determine the aesthetically reserved portion, thus reserving to the greatest extent the high aesthetical quality of the cropped image. Moreover, the present invention also uses the gradient energy diagram to analyze a gradient distribution rule, and estimates, on the basis of the aesthetically pleasing diagram and the gradient energy diagram, the composition score of the cropped image. The present invention overcomes the defects of image composition representation, and solves the technical problem of how to improve the robustness and accuracy of automatic cropping of images.

Description

Image automatic cropping method Technical field

The invention relates to the field of pattern recognition, machine learning and computer vision technology, in particular to an image automatic cropping method.

Background technique

With the rapid development of computer technology and digital media technology, people's needs and expectations for computer vision, artificial intelligence, machine perception and other fields are also increasing. Automatic cropping of images as a very important and common task in image automatic editing has also received more and more attention and development. The image auto-cropping technique is designed to remove excess areas and emphasize areas of interest, thereby improving the overall composition and aesthetic quality of the image. An effective and automatic image cropping method not only frees humans from the tedious work, but also provides some professional image editing advice to some non-professionals.

Since image cropping is a very subjective task, it is difficult to consider all the influencing factors in existing rules. Conventional image auto-cropping regions typically use saliency maps to identify major regions or regions of interest in the image, while at the same time formulating rules to calculate energy function minimization or learning classifiers to find crop regions. However, these established rules are not comprehensive enough for the subjective task of image cropping, and the precision is difficult to meet user needs.

In view of this, the present invention has been specifically proposed.

Summary of the invention

In order to solve the above problems in the prior art, an image automatic cropping method is provided in order to solve the technical problem of how to improve the robustness and precision of image automatic cropping.

In order to achieve the above object, the following technical solutions are provided:

An image automatic cropping method, the method comprising:

Extracting a beauty response map and a gradient energy map of the image to be cropped;

Extracting a candidate cropped image intensively to the image to be cropped;

Filtering the candidate cropped image based on the aesthetic response map;

Based on the aesthetic response map and the gradient energy map, the composition score of the selected candidate cropped image is estimated, and the candidate cropped image with the highest score is determined as the cropped image.

Further, the extracting the aesthetic response map and the gradient energy map of the image to be cropped include:

Using the deep convolutional neural network and the category response mapping method, and extracting the aesthetic response map of the image to be cropped by using the following formula:

Wherein, said M(x, y) represents a beauty response value at a spatial position (x, y); said K represents a total number of channels of a feature map of a last layer of the convolutional layer of the deep convolutional neural network; The k represents a kth channel; the f _k (x, y) represents a feature value of the kth channel at the spatial position (x, y); the w _k represents the kth The weight of the feature map of the channel to the high aesthetic category;

Smoothing the image to be cropped, and calculating a gradient value of each pixel to obtain the gradient energy map.

Further, the deep convolutional neural network is trained by:

Forming a convolution layer on the bottom layer of the deep convolutional neural network structure;

After the last convolutional layer of the deep convolutional neural network structure, each feature map is pooled into a point by a global average pooling method;

Connects the full connectivity layer and loss function with the same number of aesthetic quality classification categories.

Further, the filtering the candidate cropped image based on the aesthetic response map includes:

The aesthetic retention score of the candidate cropped image is calculated by the following formula:

Wherein said S _a (C) represented by the candidate image of the cropped aesthetic retention score; C represents the cropped image of the candidate; the (i, j) represents the position of a pixel; I represents the original image The A _{(i, j)} represents an aesthetic response value at the (i, j) position;

Sorting all candidate cropped images according to the aesthetic retention scores from large to small;

Select the candidate crop image with the highest score.

Further, the determining, according to the aesthetic response map and the gradient energy map, the composition score of the selected candidate cropped image, and determining the candidate cropped image with the highest score as the cropped image, specifically including:

Forming a composition model based on the aesthetic response map and the gradient energy map;

A composition score of the selected candidate cropped image is estimated by using the composition model, and the candidate cropped image with the highest score is determined as the cropped image.

Further, the composition model is obtained by:

Establishing a training image set based on the aesthetic response map and the gradient energy map;

Labeling the training image for aesthetic quality categories;

Training a deep convolutional neural network with labeled training images;

Extracting the aesthetic response map and the spatial pyramid feature of the gradient energy map by using the trained deep convolutional neural network for the labeled training image;

Splicing the extracted spatial pyramid features together;

The classifier is used for training, and the composition rules are automatically learned to obtain a composition model.

Embodiments of the present invention provide an image automatic cropping method. The method comprises: extracting a beauty response map and a gradient energy map of the image to be cropped; extracting the candidate cropped image intensively for the cropped image; screening the candidate cropped image based on the aesthetic response map; and estimating the selected candidate based on the aesthetic response map and the gradient energy map The composition score of the image is cropped, and the candidate crop image with the highest score is determined as the crop image. This scheme uses the aesthetic response map to explore the aesthetic influence area of the picture, and uses the aesthetic response map to determine the aesthetic retention part, so as to retain the high aesthetic quality of the cropped image to the greatest extent. At the same time, the scheme also uses the gradient energy map to analyze the gradient distribution rule. And evaluating the composition score of the cropped map based on the aesthetic response map and the gradient energy map. The embodiment of the invention compensates for the defects of image composition expression and solves the technical problem of how to improve the robustness and precision of image automatic cropping. Embodiments of the present invention can be applied to a wide variety of fields involving automatic image cropping, including image editing, photography, and image repositioning.

DRAWINGS

1 is a schematic flow chart of an image auto-cropping method according to an embodiment of the present invention;

2 is a schematic structural diagram of a deep convolutional neural network according to an embodiment of the present invention;

FIG. 3a is a schematic diagram of an image to be cropped according to an embodiment of the present invention; FIG.

Figure 3b is a schematic illustration of a cropped image in accordance with an embodiment of the present invention.

detailed description

The technical problems, the technical solutions, and the technical effects of the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings and specific embodiments. It is apparent that the described embodiments are only a part of the embodiments of the present application, not all of them. Based on the embodiments of the present application, all other equivalent or obvious variations of the embodiments obtained by those skilled in the art without departing from the scope of the present invention fall within the scope of the present invention. The embodiments of the invention may be embodied in many different ways as defined and claimed in the claims.

Deep learning has achieved rapid development and good results in various fields. Embodiments of the present invention contemplate the use of deep learning to automatically learn an area of influence that is important for image cropping to automatically and comprehensively learn the rules so that the high-sensitivity area is preserved as much as possible during cropping.

To this end, an embodiment of the present invention provides an automatic image cropping method. FIG. 1 exemplarily shows the flow of an image automatic cropping method. As shown in FIG. 1, the method can include:

S100: Extract a beauty response map and a gradient energy map of the image to be cropped.

Specifically, this step may include:

S101: Using a deep convolutional neural network and a category response mapping method, and extracting a beauty response map of the image to be cropped by using the following formula:

Where M(x, y) represents the aesthetic response value at the spatial position (x, y); K represents the total number of channels of the feature map f of the last layer of the convolutional layer of the trained deep convolutional neural network; k represents the kth channel; f _k (x, y) represents the eigenvalue of the kth channel at the spatial position (x, y); w _k represents the result of the feature map of the kth channel to the high aesthetic The weight of the category.

The above steps can train the deep convolutional neural network according to actual needs when extracting the aesthetic response map. The training of deep convolutional neural networks can be done in the following ways:

Step 1: Set up the convolution layer at the bottom of the deep convolutional neural network structure.

Step 2: After the last convolutional layer of the deep convolutional neural network structure, each feature map is pooled into a point by a global average pooling method.

Step 3: Connect a fully connected layer and loss function with the same number of aesthetic quality classification categories.

Fig. 2 exemplarily shows a deep convolutional neural network structure.

A deep convolutional neural network model under the aesthetic quality classification task can be trained through steps 1-3. Then, the deep convolutional neural network and the category response mapping method trained for the aesthetic quality classification task are utilized; and the above formula is used to calculate the aesthetic response map M of the image to be cropped under the high aesthetic category.

S102: Smoothing the cropped image, and calculating a gradient value of each pixel, thereby obtaining a gradient energy map.

S110: The candidate cropped image is intensively extracted from the cropped image.

Here, a sliding window of all sizes smaller than the image size may be employed, the candidate cropping window is intensively extracted for the cropped image, and the candidate cropping image is extracted by the candidate cropping window.

S120: Filter the candidate cropped image based on the aesthetic response map.

Specifically, this step may include:

S121: Calculate the aesthetic retention score of the candidate cropped image by the following formula:

Where S _a (C) represents the aesthetic retention score of the candidate cropped image; C represents the candidate cropped image; (i, j) represents the position of the pixel; I represents the original image; A _{(i, j)} represents (i, j) The aesthetic response value.

Through this step, an aesthetic retention model can be constructed. The candidate cropping window is screened by the aesthetic retention model to select candidate windows with higher aesthetic retention scores.

S122: Sort all candidate cropped images according to the aesthetic retention scores from large to small.

S123: Select a part of the candidate cropped image with the highest score.

For example, in a practical application, a candidate cropped image in the first 10,000 candidate cropping windows can be set.

S130: Estimating the composition score of the selected candidate cropped image based on the aesthetic response map and the gradient energy map, and determining the candidate cropped image with the highest score as the cropped image.

Specifically, this step can be implemented by step S131 to step S133.

S131: Establish a composition model based on the aesthetic response map and the gradient energy map.

This step can train the composition model according to the actual situation when building the composition model. In the process of training the composition model, the training data can use the image with better composition as the positive sample, and the image with the composition defect as the negative sample.

The composition model can be trained in the following ways:

Step a: Establish a training image set based on the aesthetic response map and the gradient energy map.

Step b: labeling the training image with an aesthetic quality category.

Step c: Training the deep convolutional neural network with the labeled training images.

For the training process of this step, refer to Step 1 to Step 3 above, and details are not described herein again.

Step d: extracting the spatial pyramid features of the aesthetic response map and the gradient energy map using the trained deep convolutional neural network for the labeled training images.

Step e: stitching the extracted spatial pyramid features together.

Step f: using the classifier for training, automatically learning the composition rules, and obtaining the composition model.

The classifier may be, for example, a support vector machine classifier.

S132: Estimating the composition score of the selected candidate cropped image by using the composition model, and determining the candidate cropped image with the highest score as the cropped image.

Fig. 3a exemplarily shows an image to be cropped; Fig. 3b exemplarily shows a cropped image.

The invention will now be better illustrated by a preferred embodiment.

Step A: The image data set marked with the aesthetic quality category is sent to the deep convolutional neural network for aesthetic quality category model training.

Step B: Input the image data set marked with the composition category into the trained deep convolutional neural network, extract the feature map of the last layer of the convolution layer, calculate the aesthetic response map, calculate the aesthetic gradient map, and then use the support vector machine. The classifier trains the composition model.

Step C: Extract the aesthetic response map and the gradient energy map from the test image.

The extraction method of this step can refer to the method of the training phase.

Step D: Intensively collecting candidate cropping windows of the image to be tested.

For example, on a 1000×1000 image to be tested, a sliding window with an interval of 30 pixels is used for acquisition or extraction.

Step E: Filter the candidate cropping window using the aesthetic retention model.

In this step, the aesthetic retention model is used to calculate the aesthetic retention score of the intensively collected candidate cropping window, and a part of the candidate cropping window with the highest aesthetic classification is selected, for example, 10,000 candidate cropping windows are selected.

Step F: Using the composition model to evaluate the selected candidate cropping window.

In this step, the composition model trained in the training phase is collected to evaluate the composition score of the selected candidate cropping window, and the highest score is taken as the final cropping window, thereby obtaining the cropped image.

In summary, the method provided by the embodiment of the present invention makes good use of the aesthetic response map and the gradient energy map to maximize the aesthetic quality and image composition rules, and obtains more robust and accurate image auto-cropping. Performance, in turn, illustrates the effectiveness of aesthetic response maps and gradient energy maps for automatic image cropping.

Although the methods provided in the embodiments of the present invention are described in the foregoing embodiments, those skilled in the art may understand that, in order to implement the effects of the embodiments, they may also be executed in different orders, such as parallel or reverse order. These simple variations are all within the scope of the present invention.

The above is only the specific embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can understand the alteration or replacement within the scope of the technical scope of the present invention. The scope of the invention should be construed as being included in the scope of the invention.

Claims (6)

An image automatic cropping method, characterized in that the method comprises: Extracting a beauty response map and a gradient energy map of the image to be cropped; Extracting a candidate cropped image intensively to the image to be cropped; Filtering the candidate cropped image based on the aesthetic response map; Based on the aesthetic response map and the gradient energy map, the composition score of the selected candidate cropped image is estimated, and the candidate cropped image with the highest score is determined as the cropped image. The method according to claim 1, wherein the extracting the aesthetic response map and the gradient energy map of the image to be cropped comprises: Using the deep convolutional neural network and the category response mapping method, and extracting the aesthetic response map of the image to be cropped by using the following formula:

Wherein, said M(x, y) represents a beauty response value at a spatial position (x, y); said K represents a total number of channels of a feature map of a last layer of the convolutional layer of the deep convolutional neural network; The k represents a kth channel; the f _k (x, y) represents a feature value of the kth channel at the spatial position (x, y); the w _k represents the kth The weight of the feature map of the channel to the high aesthetic category; Smoothing the image to be cropped, and calculating a gradient value of each pixel to obtain the gradient energy map. The method of claim 2 wherein said deep convolutional neural network is trained in the following manner: Forming a convolution layer on the bottom layer of the deep convolutional neural network structure; After the last convolutional layer of the deep convolutional neural network structure, each feature map is pooled into a point by a global average pooling method; Connects the full connectivity layer and loss function with the same number of aesthetic quality classification categories. The method according to claim 1, wherein the screening the candidate cropped image based on the aesthetic response map comprises: The aesthetic retention score of the candidate cropped image is calculated by the following formula:

Wherein said S _a (C) represented by the candidate image of the cropped aesthetic retention score; C represents the cropped image of the candidate; the (i, j) represents the position of a pixel; I represents the original image The A _{(i, j)} represents an aesthetic response value at the (i, j) position; Sorting all candidate cropped images according to the aesthetic retention scores from large to small; Select the candidate crop image with the highest score. The method according to claim 1, wherein the estimating a composition score of the selected candidate crop image based on the aesthetic response map and the gradient energy map, and determining the candidate crop image with the highest score as the cropping The image specifically includes: Forming a composition model based on the aesthetic response map and the gradient energy map; A composition score of the selected candidate cropped image is estimated by using the composition model, and the candidate cropped image with the highest score is determined as the cropped image. The method according to claim 5, wherein the composition model is obtained by: Establishing a training image set based on the aesthetic response map and the gradient energy map; Labeling the training image for aesthetic quality categories; Training a deep convolutional neural network with labeled training images; Extracting the aesthetic response map and the spatial pyramid feature of the gradient energy map by using the trained deep convolutional neural network for the labeled training image; Splicing the extracted spatial pyramid features together; The classifier is used for training, and the composition rules are automatically learned to obtain a composition model.

Images