KR102584695B1 - Apparatus for inpainting video using reference frame information and method thereof - Google Patents

Abstract

The present invention discloses an image restoration apparatus and method using adjacent frame information. An image restoration device using adjacent frame information of the present invention includes an input unit that receives a video image to be restored; a restoration area setting unit that sets a restoration area in the video image input to the input unit; A matching module that matches the target frame and reference frame to be restored from the video image based on deep learning; A copy module that copies restoration information about the restoration area of the target frame from the reference frame matched in the matching module based on deep learning; A recovery module that restores the information copied from the copy module by pasting it into the restoration area of the target frame; and an image output unit that outputs a video image restored in the recovery module.

Description

Video restoration device and method using adjacent frame information {APPARATUS FOR INPAINTING VIDEO USING REFERENCE FRAME INFORMATION AND METHOD THEREOF}

The present invention relates to an image restoration device and method using adjacent frame information. More specifically, the present invention relates to an image restoration device and method using adjacent frame information. More specifically, the feature information of a target frame in which an obstruction area is set in an input video image and the feature information of reference frames are used based on deep learning. It relates to an image restoration device and method using adjacent frame information that restores the obstruction area of the target frame by copying the area information about the obstruction area of the target frame from the reference frame after matching based on deep learning.

Image restoration technology (Image Inpainting) is a technology that originated from the restoration of damaged works of art in museums in the past, and is currently used for various purposes such as restoring or removing damaged parts of paintings or photos.

This image restoration technology is a technology that interpolates pixels in the background image area adjacent to the border of the target image when there is a target image to be restored, and fills the target image area with the interpolated pixels. Here, the background image refers to the remaining images excluding the target image to be restored from the original image.

However, the resulting image created using this image restoration technology had the problem of severe blurring.

Various technologies have been studied to complement these image restoration technologies, one of which is Object Removal by Exemplar-Based Inpainting (hereinafter referred to as 'original-based restoration technology').

The original-based restoration technology refers to the edges and flat areas in the original image, searches the background image for the image most similar to the background image adjacent to the border of the target image, and fills the target image area with the searched image.

According to this original-based restoration technology, a similar image area with the pixel value most similar to the pixel value of the set restored image area is searched within the background image, and then a similar target corresponding to the target image area within the restored image area is found in the searched similar image area. Only the image is copied and pasted into the target image area of the restored image area, and this operation is repeated until image restoration is complete.

However, there was a problem that when searching for a similar image area with the pixel value most similar to the pixel value of the restored image area, an image completely different from the image to be restored may be searched in the target image area.

The background technology of the present invention is disclosed in Republic of Korea Patent Publication No. 10-1075716 (Notice dated October 21, 2011, Image restoration device and method).

Meanwhile, unlike single images, video images have difficulty in directly referencing pixels due to the movement of cameras, objects, and backgrounds in a video shooting environment. Unlike the restoration of single images, the restoration of video images has the difficult problem of maintaining temporal continuity between images. .

The present invention was created to improve the problems described above, and the purpose of the present invention according to one aspect is to deep learn the feature information of the target frame in which the obstruction area is set and the feature information of the reference frames in the input video image. To provide an image restoration device and method using adjacent frame information that restores the obstruction area of the target frame by copying the area information about the obstruction area of the target frame from the reference frame based on deep learning based on matching.

An image restoration device using adjacent frame information according to an aspect of the present invention includes an input unit that receives a video image to be restored; a restoration area setting unit that sets a restoration area in the video image input to the input unit; A matching module that matches the target frame and reference frame to be restored from the video image based on deep learning; A copy module that copies restoration information about the restoration area of the target frame from the reference frame matched in the matching module based on deep learning; A recovery module that restores the information copied from the copy module by pasting it into the restoration area of the target frame; and an image output unit that outputs a video image restored in the recovery module.

In the present invention, the restoration area setting unit is characterized by setting the area of the obstruction set in the video image as the restoration area.

In the present invention, the restoration area setting unit is characterized in that it recognizes an obstacle object set in a video image and sets it as a restoration area.

In the present invention, the matching module is characterized by extracting features from the target frame and reference frame, extracting an affine transformation matrix, and matching them through a deep learning-based self-supervised learning network.

In the present invention, the copy module is characterized by copying the restoration information for the restoration area by assigning weights based on the similarity between reference frames through context matching in the feature space.

In the present invention, the recovery module is characterized by restoring the target frame and then updating the target frame as a reference frame.

An image restoration method using adjacent frame information according to another aspect of the present invention includes the steps of a restoration area setting unit setting a restoration area on a video image to be restored input from an input unit; A matching module matching a target frame and a reference frame to be restored from a video image based on deep learning; A step of the copy module copying restoration information about the restoration area of the target frame from the reference frame matched by the matching module based on deep learning; and a step of restoration by the restoration module pasting the restoration information copied from the copy module into the restoration area of the target frame.

In the present invention, the step of setting a restoration area is characterized in that the restoration area setting unit sets an obstacle set in the video image as a restoration area.

In the present invention, the step of setting a restoration area is characterized in that the restoration area setting unit recognizes an obstacle object set in a video image and sets it as a restoration area.

The matching step in the present invention is characterized in that the matching module extracts features from the target frame and reference frame, extracts an affine transformation matrix, and performs matching through a deep learning-based self-supervised learning network. .

In the present invention, the copying step is characterized by copying the restored information about the restored area by assigning a weight based on the similarity between reference frames through context matching in the feature space.

In the present invention, the restoring step further includes the step of updating the target frame to a reference frame after the recovery module restores the target frame.

An image restoration apparatus and method using adjacent frame information according to an aspect of the present invention matches the feature information of a target frame in which an obstruction area is set in an input video image and the feature information of reference frames based on deep learning. By copying the area information about the obstruction area of the target frame from the reference frame based on deep learning and restoring the obstruction area of the target frame, not only can a visually natural image be derived while maintaining temporal continuity between frames, but also an optimization-based restoration method. Execution speed can be further improved.

Additionally, according to the present invention, the accuracy of lane detection can be improved based on the restored image even in continuous under-exposed or over-exposed images.

Figure 1 is a block diagram showing an image restoration device using adjacent frame information according to an embodiment of the present invention.
Figure 2 is a diagram schematically illustrating an image restoration device using adjacent frame information according to an embodiment of the present invention.
Figure 3 is a diagram for explaining the context matching process of the copy module in the image restoration device using adjacent frame information according to an embodiment of the present invention.
Figure 4 is an exemplary diagram showing masked softmax in the copy module in the image restoration device using adjacent frame information according to an embodiment of the present invention.
Figure 5 is an exemplary diagram showing a restoration state of an exposed image using an image restoration device using adjacent frame information according to an embodiment of the present invention.
Figure 6 is a flowchart for explaining an image restoration invention using adjacent frame information according to an embodiment of the present invention.

Hereinafter, an image restoration apparatus and method using adjacent frame information according to the present invention will be described with reference to the attached drawings. In this process, the thickness of lines or sizes of components shown in the drawing may be exaggerated for clarity and convenience of explanation. In addition, the terms described below are terms defined in consideration of functions in the present invention, and may vary depending on the intention or custom of the user or operator. Therefore, definitions of these terms should be made based on the content throughout this specification.

Figure 1 is a block diagram showing an image restoration device using adjacent frame information according to an embodiment of the present invention, and Figure 2 is a schematic diagram of an image restoration device using adjacent frame information according to an embodiment of the present invention. It is a diagram, and FIG. 3 is a diagram for explaining the context matching process of the copy module in an image restoration device using adjacent frame information according to an embodiment of the present invention, and FIG. 4 is a diagram showing adjacent frame information according to an embodiment of the present invention. This is an example diagram showing masked softmax in a copy module in an image restoration device using , and Figure 5 is an example diagram showing the restoration state of an exposed image using an image restoration device using adjacent frame information according to an embodiment of the present invention. .

As shown in FIG. 1, an image restoration device using adjacent frame information according to an embodiment of the present invention includes an input unit 10, a restoration area setting unit 20, a matching module 30, a copy module 40, It may include a recovery module 50 and an image output unit 60.

The input unit 10 can receive a video image to be restored.

Additionally, the user's settings can be input through the input unit 10.

The restoration area setting unit 20 can set a restoration area in the video image input from the input unit 10.

Here, the restoration area can be set as an area of the obstruction in the video image that has disappeared or is an incorrect area due to the obstruction, and the area of the obstruction can be set as the restoration area.

At this time, if the user sets an obstacle in the video image, the restoration area setting unit 20 may set the area of the obstacle as the restoration area. If the type of obstacle is set in the video image, the restoration area setting unit 20 may recognize the object of the obstacle and set it as the restoration area. It may be possible.

The matching module 30 can match the target frame to be restored and the reference frame in the video image based on deep learning.

Here, the target frame is a frame from the video image to be restored, and the reference frame is other frames to which information in the restoration area is copied for reference for restoration.

In this embodiment, the matching module 30 extracts features from the target frame and reference frame, extracts an affine transformation matrix, and performs matching through a deep learning-based self-supervised learning network. .

As shown in Figure 2, the alignment module (Alignment network) 30 extracts the features of the target frame and reference frames through an alignment encoder, and then extracts the features of the target frame and the reference frame to be restored to the alignment regressor. An affine transformation matrix for alignment can be extracted by simultaneously matching the frame and the reference frame to be aligned.

Therefore, the reference frame can be matched with the target frame by transforming the reference frame through an affine transformation matrix.

Here, the alignment encoder can extract features by receiving a mask for indicating the restoration area in the target frame and reference frame.

The copy network 40 can copy restoration information about the restoration area of the target frame from the reference frame matched by the matching module 30 based on deep learning.

At this time, if the reference frame matched in the matching module 30 is a frame close to the target frame, the registration is good, but if there is a lot of movement in the image, there are frames for which matching is difficult, so the restoration information that restores the restoration area of the target frame In order to copy, the restoration information for the restoration area can be copied by assigning a weight based on the similarity between the reference frames through context matching in the feature space between the target frame and the matched reference frames. .

Here, context matching can copy the information of the reference frames into one by calculating the similarity of the non-restored area between the target frame and the reference frame and assigning a weight.

For example, as shown in Figure 3, C_match refers to a weight for combining the features of the reference frame, C_mask refers to the area not visible in the reference frames, and C_out refers to the features of the reference frame copied into one. .

Additionally, the visibility map represents the visible area, not the hole, which is the restoration area.

In this way, in order to combine the features of the reference frame based on the similarity between the features of the reference frame and the features of the target frame, the cosine similarity for the area excluding the restored area is calculated between the features of the target frame and the features of the reference frame, respectively. Find and set it as global similarity.

Afterwards, when calculating the weight through softmax, in order to solve the problem of obtaining the weight even for the hole area, which is the restoration area, the weight is obtained through Masked Softmax, which applies a mask to exclude the restoration area. .

As shown in FIG. 4, Masked Softmax can obtain softmax for the remaining areas excluding the hole area based on global similarity. At this time, in the visibility map, 1 means the visible part, and 0 means the invisible hole area.

Therefore, the weight is calculated only for the part excluding the hole area, so the invisible area becomes 0, and C_mask becomes 1.

In this way, the copy module 40 can extract one piece of restored information C_out by copying the restored information according to the weight from the features of the reference frame through context matching, and extract C_mask, which is an area that cannot be seen even in the reference frame.

The recovery module (Paste network) 50 restores the restoration information copied from the copy module 40 by pasting it into the restoration area of the target frame.

That is, the recovery module 50 receives as input C_out, which is copied from the features of the reference frame in the copy module 40, the features of the target frame, and C_mask with information about the invisible area, and can restore the restoration area. .

Meanwhile, if the restoration information for the restoration area cannot be copied from the reference frame, the restoration module 50 may restore it through dilated convolution.

also. The recovery module 50 updates the restored target frame to the reference frame to maintain temporal continuity.

The image output unit 60 may output a video image restored for all frames in the recovery module 50.

When the above-described image restoration device using adjacent frame information according to the present invention is applied to underexposed or overexposed road images, meaningful restoration is achieved for both underexposed and overexposed images, as shown in FIG. 5, and lane recognition is achieved. accuracy can be improved.

As described above, according to the image restoration device using adjacent frame information according to an embodiment of the present invention, the feature information of the target frame in which the obstruction area is set and the feature information of the reference frames in the input video image are deep learning-based. After matching, the area information about the obstruction area of the target frame is copied from the reference frame based on deep learning to restore the obstruction area of the target frame. By doing so, not only can a visually natural image be derived while maintaining temporal continuity between frames. The execution speed can be improved compared to optimization-based restoration methods, and the accuracy of lane detection can be improved based on restored images even in continuous under-exposed or over-exposed images.

Figure 6 is a flowchart for explaining an image restoration invention using adjacent frame information according to an embodiment of the present invention.

As shown in FIG. 6, in the image restoration method using adjacent frame information according to an embodiment of the present invention, first, the restoration area setting unit 20 receives a video image from the input unit 10 (S10).

The restoration area setting unit (200) sets the restoration area on the video image to be restored (S20).

Here, the restoration area can be set as an area of the obstruction in the video image that has disappeared or is an incorrect area due to the obstruction, and the area of the obstruction can be set as the restoration area.

At this time, if the user sets an obstacle in the video image, the restoration area setting unit 20 may set the area of the obstacle as the restoration area. If the type of obstacle is set in the video image, the restoration area setting unit 20 may recognize the object of the obstacle and set it as the restoration area. It may be possible.

After setting the restoration area in step S20, the matching module matches the target frame to be restored and the reference frame in the video image based on deep learning (S30).

Here, the target frame is a frame from the video image to be restored, and the reference frame is other frames to which information in the restoration area is copied for reference for restoration.

In this embodiment, the matching module 30 extracts features from the target frame and reference frame, extracts an affine transformation matrix, and performs matching through a deep learning-based self-supervised learning network. .

After matching the target frame and the reference frame in step S30, the copy module 40 copies restoration information about the restoration area of the target frame from the reference frame matched in the matching module 30 based on deep learning (S40).

At this time, if the reference frame matched in the matching module 30 is a frame close to the target frame, the registration is good, but if there is a lot of movement in the image, there are frames for which matching is difficult, so the restoration information that restores the restoration area of the target frame In order to copy, the restoration information for the restoration area can be copied by assigning a weight based on the similarity between the reference frames through context matching in the feature space between the target frame and the matched reference frames. .

Here, context matching can copy the information of the reference frames into one by calculating the similarity of the non-restored area between the target frame and the reference frame and assigning a weight.

After copying the restoration information in step S40, the recovery module 50 restores the restoration information by pasting it into the restoration area of the target frame (S50).

That is, the restoration module 50 can restore the restoration area based on the restoration information copied from the characteristics of the reference frame in the copy module 40, the characteristics of the target frame, and information about the invisible area.

Meanwhile, if the restoration information for the restoration area cannot be copied from the reference frame, the restoration module 50 may restore it through dilated convolution.

After restoring the target frame in step S50, the recovery module 50 updates the restored target frame to the reference frame to maintain temporal continuity (S60).

As described above, according to the image restoration method using adjacent frame information according to an embodiment of the present invention, the feature information of the target frame in which the obstruction area is set and the feature information of the reference frames in the input video image are deep learning-based. After matching, the area information about the obstruction area of the target frame is copied from the reference frame based on deep learning to restore the obstruction area of the target frame. By doing so, not only can a visually natural image be derived while maintaining temporal continuity between frames. The execution speed can be improved compared to optimization-based restoration methods, and the accuracy of lane detection can be improved based on restored images even in continuous under-exposed or over-exposed images.

Implementations described herein may be implemented, for example, as a method or process, device, software program, data stream, or signal. Although discussed only in the context of a single form of implementation (eg, only as a method), implementations of the features discussed may also be implemented in other forms (eg, devices or programs). The device may be implemented with appropriate hardware, software, firmware, etc. The method may be implemented in a device such as a processor, which generally refers to a processing device including a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices such as computers, cell phones, portable/personal digital assistants (“PDAs”) and other devices that facilitate communication of information between end-users.

The present invention has been described with reference to the embodiments shown in the drawings, but these are merely exemplary, and those skilled in the art will recognize that various modifications and other equivalent embodiments are possible therefrom. You will understand.

Therefore, the true technical protection scope of the present invention should be determined by the claims below.

10: input unit 20: restoration area setting unit
30: Matching module 40: Copying module
50: Recovery module 60: Video output unit

Claims (12)

An input unit that receives a video image to be restored;
a restoration area setting unit that sets a restoration area in the video image input to the input unit;
a matching module that matches a target frame to be restored from the video image and a reference frame based on deep learning;
a copy module that copies restoration information about the restoration area of the target frame from the reference frame matched by the matching module based on deep learning;
a recovery module for restoring the restoration information by pasting the restoration information copied from the copy module into the restoration area of the target frame; and
Including a video output unit that outputs the video image restored in the recovery module,
The copy module is characterized in that it copies the restoration information for the restoration area by assigning a weight obtained through Masked Softmax based on the similarity between the reference frames in feature space. Image restoration device using adjacent frame information.
The image restoration device according to claim 1, wherein the restoration area setting unit sets an area of an obstacle set in the video image as the restoration area.
The image restoration device according to claim 1, wherein the restoration area setting unit recognizes an obstacle object set in the video image and sets it as the restoration area.
The method of claim 1, wherein the matching module extracts features from the target frame and the reference frame, extracts an affine transformation matrix, and performs matching through a deep learning-based self-supervised learning network. An image restoration device using adjacent frame information.
The image restoration using adjacent frame information according to claim 1, wherein the copy module assigns the weight based on similarity between the reference frames through context matching in feature space. Device.
The image restoration apparatus according to claim 1, wherein the recovery module restores the target frame and then updates the target frame with the reference frame.
A restoration area setting unit setting a restoration area on a video image to be restored input from an input unit;
A matching module matching a target frame to be restored and a reference frame from the video image based on deep learning;
A copying module copying restoration information about the restoration area of the target frame from the reference frame matched by the matching module based on deep learning; and
Comprising: a step of restoration by a restoration module pasting the restoration information copied from the copy module into the restoration area of the target frame;
The copying step is characterized in that copying the restoration information for the restoration area by assigning a weight obtained through Masked Softmax based on the similarity between the reference frames in feature space. Image restoration method using adjacent frame information.
The method of claim 7, wherein in the setting of the restoration area, the restoration area setting unit sets an obstacle set in the video image as the restoration area.
The image restoration method according to claim 7, wherein in the step of setting the restoration area, the restoration area setting unit recognizes an obstacle object set in the video image and sets it as the restoration area.
The method of claim 7, wherein in the matching step, the matching module extracts features from the target frame and the reference frame and extracts an affine transformation matrix to create a deep learning-based self-supervised learning network. An image restoration method using adjacent frame information, characterized by matching through.
The image using adjacent frame information according to claim 7, wherein the copying step assigns the weight based on similarity between the reference frames through context matching in feature space. How to restore.
The image restoration method according to claim 7, wherein the restoring step further includes the step of updating the target frame with the reference frame after the recovery module restores the target frame. .

Images