CN116757957A - Two-stage decoupling image defogging method based on zero sample learning - Google Patents

Abstract

The invention discloses a two-stage decoupling image defogging method based on zero sample learning. The main focus of this approach is to recover the haze-free image and the transmission map. Briefly, the method uses an embedded Dark Channel Prior (DCP) in a first stage to obtain a rough estimate of the haze free image and transmission map. In the second stage, the two subnetworks refine the results of the first stage to obtain more accurate haze-free images and transmission maps, while the other subnetwork directly estimates atmospheric light. In particular, we have also proposed a new multi-scale Transformer block and applied it to a sub-network that refines the haze-free image, which performs multi-scale token aggregation in the self-attention part, enabling it to capture features of different scales and effectively recover the potential scene information in the haze-image.

Description

A two-stage decoupled image dehazing method based on zero-shot learning

Technical Field

The present invention relates to the technical field of image defogging, and in particular to a two-stage decoupling image defogging method based on zero-sample learning.

Background Art

Haze is a phenomenon caused by the accumulation and suspension of large amounts of particles in the air. Due to the absorption and scattering of atmospheric light, the light is affected by particles during propagation, which reduces the clarity and contrast of images acquired by various image acquisition devices, seriously degrading the quality of the images, and bringing great difficulties to various subsequent image processing operations. In severe cases, these factors will also distort the color information in the image, making it more difficult to apply various computer vision technologies.

In recent years, with the popularization of computer vision systems, these systems have played an important role in roads, aviation and other fields, but foggy and low-visibility weather has caused great trouble to visual equipment such as aviation and road traffic. Therefore, image dehazing has always been a focus of research in the field of computer vision.

The purpose of image dehazing is to estimate the potential haze-free image from the observed haze image. For the single image dehazing problem, there is a commonly used model to represent the degradation process of haze images:

I＝J(x)t(x)+A(1-t(x))

Where I is the haze image, J is the potential haze-free image, A is the global atmospheric light, t is the transmission map, and x is the pixel position information. The transmission map can be expressed as:

t(x)＝e ^-βd(x) ,

Where β is the scattering coefficient of the atmosphere and d is the scene depth.

From the above, it can be seen that image dehazing is a typical ill-posed problem. Therefore, many image dehazing methods have been proposed, which can be roughly divided into prior-based methods and learning-based methods. Prior-based methods, that is, traditional dehazing methods, tend to use the prior knowledge of the image itself for dehazing. For example: dark channel prior (DCP) is used to detect the haze distribution of the image, and color attenuation prior (CAP) is used to estimate the transmission map. Although these prior-based methods have achieved certain results, the prior knowledge used is easily overturned in practice and has low robustness in complex actual scenes.

Summary of the invention

In view of the shortcomings of the prior art, the purpose of the present invention is to provide a two-stage decoupled image defogging method based on zero-shot learning. The method is based on zero-shot learning and directly inputs the haze image into the network for defogging. There is no need to provide haze-clean image pairs, thereby avoiding large-scale data training. At the same time, in the process of restoring the haze-free image, the method can not only ensure the extraction of multi-scale features, but also ensure the extraction of original scale and neighborhood information.

To achieve the above object, the present invention provides the following technical solution: a two-stage decoupling image defogging method based on zero-sample learning, comprising the following steps:

(1) Obtain two test subsets of the synthetic objective testing set (SOTS) and the hybrid subjective testing set (HSTS) from the real single image dehazing (RESIDE) dataset, and preprocess the original dataset as the test set;

(2) Constructing a dehazing network model: Based on the atmospheric scattering physical model I = J(x)t(x)+A(1-t(x)),

Where x represents the input haze image, I(x) represents the reconstructed haze image, J(x) represents the original scene information, that is, the clear haze-free image, t(x) represents the transmittance, and A represents the atmospheric scattered light.

Based on the idea of decoupling, the network model is divided into three layers: potential fog-free image layer, transmission layer layer and global atmospheric light layer. The potential fog-free image and transmission layer are estimated in two stages. The first stage uses the embedded dark channel prior for rough estimation, and the second stage uses the fog-free image refinement network and the transmission layer refinement network for refined estimation. For the global atmospheric light, an encoder-decoder network (i.e., atmospheric light estimation network) is used for estimation. At the top of the network, the images estimated by the three sub-networks (fog-free image refinement network, transmission layer refinement network and atmospheric light estimation network) are reconstructed according to the atmospheric scattering physical model to obtain the reconstructed haze image.

(3) The haze images in the preprocessed data set are input into the constructed dehazing network model, the loss is calculated according to the designed loss function, the parameters are continuously updated iteratively, and the haze images are directly dehazed.

As a preferred embodiment: Step (2), the rough estimation in the first stage specifically includes the following steps:

(2.1) Calculate the dark channel of the input image x: First, take the minimum value of the three RGB channels to construct a grayscale image, then invert the grayscale image, perform a maximum pooling operation, and invert the result after maximum pooling to obtain I ^dark . The specific formula is: Where I ^c refers to one of the R, G, or B channels of I;

(2.2) Calculate a rough estimate of the transmission map T _DCP (x): Substitute I ^dark into the atmospheric scattering physical model to obtain I ^dark (x) = J ^dark (x)t(x) + A(1-t(x)), where I ^dark (x) and J ^dark (x) are the dark channels of images I and J, respectively; for atmospheric light A, select the top 0.1% of pixels in I ^dark and take their average value as the value of atmospheric light A; and for J ^dark , according to the dark channel prior assumption, the darkness of the non-sky area of the outdoor fog-free image J usually tends to zero, J ^dark (x)→0, so according to The preliminary results of the transmission graph are obtained; since the edge transition of the dark channel image is not smooth, the guided filtering method is used to make the edge of T _DCP (x) smoother. An average pooling operation is used to implement the guided filtering. The kernel size of the average pooling operation is 19*19 and the stride is 1.

(2.3) Calculate a rough estimate of the haze-free image _JDCP (x): According to the physical model of atmospheric scattering, and with the known atmospheric light A and transmission map T _DCP (x), _JDCP (x): is calculated by Eq.

As a preferred embodiment: Step (2), constructing three subnets of the defogging network model, specifically includes the following steps:

(A) The architecture of the haze-free image refinement network is an improved 5-level U-Net architecture, in which the convolutional blocks are replaced by the introduced multi-scale Transformer blocks, and the SK fusion module is used to connect and fuse different layers, and a soft reconstruction layer is used to obtain the global residual;

(B) The transfer graph refinement network is a non-degenerate structured network consisting of five convolutional layers. In the first four layers, each layer only includes a convolutional layer, a batch normalization layer, and a LeakyReLU activation function. The last layer includes a convolutional layer and a sigmoid function that normalizes the output to [0, 1].

(C) The atmospheric light estimation network is a symmetrical encoder-decoder architecture network. The encoder and decoder of the network each have four layers. Each layer of the encoder includes a convolution layer, a ReLU activation function and a maximum pooling layer. Each layer of the decoder includes an upsampling layer, a convolution layer, a batch normalization layer and a ReLU activation function. In the middle of the encoder and the decoder, a reparameterization technique module is used, which converts the output of the encoder into a potential Gaussian distribution, which is then resampled and input into the decoder.

As a preferred embodiment: in step (A), constructing a multi-scale Transformer block specifically includes the following steps:

(A1) The RescaleNorm layer is used as the normalization layer. The specific normalization process of RescaleNorm is expressed as:

where F(·) represents the main part of the multi-scale self-attention in the multi-scale Transformer block, denote the mean and standard deviation respectively, are the learned scaling factor and bias, respectively. and are the two linear layer weights and bias terms used to convert μ and σ respectively. The conversion process is expressed as {γ, β} = {σW _γ +B _γ ,μW _β +B _β }. In order to accelerate convergence, B _γ and B _β are initialized to 1 and 0.

(A2) Multi-scale self-attention is used to calculate self-attention. Multi-scale self-attention (MSSA) has two branches, which aggregates multi-scale feature information within the window while retaining the information of the neighborhood to a certain extent.

As a preferred embodiment: in step (A2), multi-scale self-attention (MSSA) is calculated, including two branches:

(A21) In the branch, the query vector Q, key vector K, and value vector V are processed differently. For Q, the original scale is retained and a normal full connection operation is performed on it. For key K and value V, in the same self-attention layer, scale transformation (ST) is used to obtain K and V of different sizes, which can be specifically expressed as:

Q＝XW ^Q ,

K _i ,V _i =ST _i (X)W _i ^K ,ST _i (X)W _i ^V ,

V _i =V _i +LE _i (V _i ),

Where X represents the input feature sequence, W ^Q , _Wi ^K , _Wi ^V are the linear projection parameters of the i-th scale layer on the same head of the self-attention layer, LE _i (·) is the local enhancement component of the depthwise convolution; and S _Ti (·) is the scale transformation operation of the i-th scale layer, which provides downsampling of X, which can be specifically expressed as:

ST _i (X) = LN (Conv _i (X)),

Among them, LN(·) refers to layer normalization; Conv _i (·) refers to the convolution used by the i-th scale layer. Different scale layers use different kernel sizes and strides, resulting in different scale transformations; therefore, in the same self-attention layer, the keys and values capture features of different scales; then, the self-attention head can be calculated as:

Where Softmax(·) is the activation function, T represents the transposition operation, d is the dimension, and different heads are connected. Multi-head selfattention (MH) in the main branch is calculated by the following method:

MH(Q,K,V)=Concat(head ₀ ,...,head _j )W ^O ,

Where Concat(·) is the concatenation operation, head _i represents the i-th self-attention head, and W ^O is the linear projection parameter;

(A22) Another branch that retains neighborhood information only performs linear projection and convolution operations on the input features in sequence. The multi-scale self-attention MSSA can be expressed as:

MSSA=Concat(MH{Q,K _i ,V _i } _i=1,...,m )+Conv ₀ (XW ⁰ ),

Where Concat(·) is a concatenation operation, MH represents multi-head self-attention, Q, _Ki , _Vi represent the query vector, the key vector of the i-th scale layer, and the value vector of the i-th scale layer, respectively, Conv ₀ (·) refers to the convolution operation, X is the input feature sequence, and W ^O is the linear projection parameter;

Preferably, in step (3), the loss function is composed of five loss functions:

(3.1) Reconstruction loss L _Rec is used to calculate the error between the input image and the reconstructed image to constrain the entire network. L _Rec is defined as:

L _Rec ＝||I(x)-x|| _F ,

Among them, ||·|| _F represents the Frobenius norm of the given matrix, x is the input haze image, and I(x) is the haze image reconstructed by the output of the three subnetworks;

(3.2) The loss function L _J is used to calculate the difference between the brightness and saturation of the haze-free image estimate J _R (x), which is specifically expressed as:

L _J =||V(J _R (x))-S(J _R (x))|| _F ,

where V(·) represents brightness, S(·) represents saturation, and ||·|| _F represents the Frobenius norm of the given matrix.

(3.3) Atmospheric light loss L _H is used to calculate the loss between the estimated atmospheric light A(x) and A _init (x), where A _init (x) is the initial atmospheric light automatically estimated from the input image data. L _H is specifically expressed as:

L _H =||A(x)-A _init (x)|| _F .

(3.4) L _KL is the variational inference loss of the reparameterized module in the atmosphere estimation optical network, making the latent code after resampling in the network Minimize the difference between z and before sampling. Mathematically,

where KL(·) represents the Kullback-Leibler divergence between two distributions, z _i represents the i-th dimension of z, μ _z ,σ _z represent the mean and standard deviation of z respectively. denote the mean and standard deviation of z _i respectively;

(3.5) In order to avoid network overfitting, the regularization term loss L _Reg is set, which is specifically expressed as:

Where N(·) represents the second-order neighborhood, |N(·)| represents the size of the second-order neighborhood, n represents the number of pixels of A(x), and A(x) represents atmospheric light;

The total loss is defined by combining the above five loss functions:

L＝L _Rec +L _J +L _H +L _KL +λL _Reg ,

Among them, the parameter λ before L _Reg is a non-negative parameter used to balance regularization, which is 0.1 in practice.

By adopting the above technical solution, the present invention has the following beneficial effects:

The present invention constructs an image defogging network model: according to the atmospheric scattering physical model, the network model is divided into three layers based on the idea of decoupling, namely, a potential fog-free image layer, a transmission layer layer and a global atmospheric light layer. The potential fog-free image and the transmission layer layer are estimated in two stages. The first stage uses the embedded dark channel prior for a rough estimate, and the second stage uses the fog-free image refinement network and the transmission layer refinement network for refined estimates. For the global atmospheric light, an encoder-decoder network is used for estimation. At the top of the network, the images estimated by the three subnetworks are reconstructed according to the atmospheric scattering physical model to obtain a reconstructed haze image. A new multi-scale Transformer block is introduced in the refined fog-free image subnetwork of the defogging network model, which performs multi-scale token aggregation in the self-attention part, so that it can capture features of different scales and effectively restore the potential scene information in the haze image.

The present invention is based on zero-sample learning and directly inputs the haze image into the network for dehazing without providing haze-clean image pairs, thus avoiding large-scale data training. At the same time, in the process of restoring the haze-free image, the method can not only ensure the extraction of multi-scale features, but also ensure the extraction of original scale and neighborhood information.

The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a defogging network model according to an embodiment of the present invention;

FIG2 is a schematic diagram of a multi-scale Transformer block according to an embodiment of the present invention.

DETAILED DESCRIPTION

Referring to FIG. 1 and FIG. 2 , the present invention discloses a two-stage decoupled image defogging method based on zero-sample learning, comprising the following steps:

(1) Obtain two test subsets of the synthetic objective testing set (SOTS) and the hybrid subjective testing set (HSTS) from the real single image dehazing (RESIDE) dataset, and preprocess the original dataset as the test set;

(2) Constructing a dehazing network model: Based on the atmospheric scattering physical model I = J(x)t(x)+A(1-t(x)),

Where x represents the input haze image, I(x) represents the reconstructed haze image, J(x) represents the original scene information, that is, the clear haze-free image, t(x) represents the transmittance, and A represents the atmospheric scattered light.

Based on the idea of decoupling, the network model is divided into three layers: potential fog-free image layer, transmission layer layer and global atmospheric light layer. The potential fog-free image and transmission layer are estimated in two stages. The first stage uses the embedded dark channel prior for rough estimation, and the second stage uses the fog-free image refinement network and the transmission layer refinement network for refined estimation. For the global atmospheric light, an encoder-decoder network (i.e., atmospheric light estimation network) is used for estimation. At the top of the network, the images estimated by the three sub-networks (fog-free image refinement network, transmission layer refinement network and atmospheric light estimation network) are reconstructed according to the atmospheric scattering physical model to obtain the reconstructed haze image.

(3) The haze images in the preprocessed data set are input into the constructed dehazing network model, the loss is calculated according to the designed loss function, the parameters are continuously updated iteratively, and the haze images are directly dehazed.

As a preferred embodiment: Step (2), the rough estimation in the first stage specifically includes the following steps:

(2.1) Calculate the dark channel of the input image x: First, take the minimum value of the three RGB channels to construct a grayscale image, then invert the grayscale image, perform a maximum pooling operation, and invert the result after maximum pooling to obtain I ^dark . The specific formula is: Where I ^c refers to one of the R, G, or B channels of I;

(2.2) Calculate a rough estimate of the transmission map T _DCP (x): Substitute I ^dark into the atmospheric scattering physical model to obtain I ^dark (x) = J ^dark (x)t(x) + A(1-t(x)), where I ^dark (x) and J ^dark (x) are the dark channels of images I and J, respectively; for atmospheric light A, select the top 0.1% of pixels in I ^dark and take their average value as the value of atmospheric light A; and for J ^dark , according to the dark channel prior assumption, the darkness of the non-sky area of the outdoor fog-free image J usually tends to zero, J ^dark (x)→0, so according to The preliminary results of the transmission graph are obtained; since the edge transition of the dark channel image is not smooth, the guided filtering method is used to make the edge of T _DCP (x) smoother. An average pooling operation is used to implement the guided filtering. The kernel size of the average pooling operation is 19*19 and the stride is 1.

(2.3) Calculate a rough estimate of the haze-free image _JDCP (x): According to the physical model of atmospheric scattering, and with the known atmospheric light A and transmission map T _DCP (x), _JDCP (x): is calculated by equation

As a preferred embodiment: Step (2), constructing three subnets of the defogging network model, specifically includes the following steps:

(A) The architecture of the haze-free image refinement network is an improved 5-level U-Net architecture, in which the convolutional blocks are replaced by the introduced multi-scale Transformer blocks, and the SK fusion module is used to connect and fuse different layers, and a soft reconstruction layer is used to obtain the global residual;

(B) The transfer graph refinement network is a non-degenerate structured network consisting of five convolutional layers. In the first four layers, each layer only includes a convolutional layer, a batch normalization layer, and a LeakyReLU activation function. The last layer includes a convolutional layer and a sigmoid function that normalizes the output to [0, 1].

(C) The atmospheric light estimation network is a symmetrical encoder-decoder architecture network. The encoder and decoder of the network each have four layers. Each layer of the encoder includes a convolution layer, a ReLU activation function and a maximum pooling layer. Each layer of the decoder includes an upsampling layer, a convolution layer, a batch normalization layer and a ReLU activation function. In the middle of the encoder and the decoder, a reparameterization technique module is used, which converts the output of the encoder into a potential Gaussian distribution, which is then resampled and input into the decoder.

As a preferred embodiment: in step (A), constructing a multi-scale Transformer block specifically includes the following steps:

(A1) The RescaleNorm layer is used as the normalization layer. The specific normalization process of RescaleNorm is expressed as:

where F(·) represents the main part of the multi-scale self-attention in the multi-scale Transformer block, denote the mean and standard deviation respectively, are the learned scaling factor and bias, respectively. and are the two linear layer weights and bias terms used to convert μ and σ respectively. The conversion process is expressed as {γ, β} = {σW _γ +B _γ ,μW _β +B _β }. In order to accelerate convergence, B _γ and B _β are initialized to 1 and 0.

(A2) Multi-scale self-attention is used to calculate self-attention. Multi-scale self-attention (MSSA) has two branches, which aggregates multi-scale feature information within the window while retaining the information of the neighborhood to a certain extent.

As a preferred embodiment: in step (A2), multi-scale self-attention (MSSA) is calculated, including two branches:

(A21) In the branch, the query vector Q, key vector K, and value vector V are processed differently. For Q, the original scale is retained and a normal full connection operation is performed on it. For key K and value V, in the same self-attention layer, scale transformation (ST) is used to obtain K and V of different sizes, which can be specifically expressed as:

Q＝XW ^Q ,

K _i ,V _i =ST _i (X)W _i ^K ,ST _i (X)W _i ^V ,

V _i =V _i +LE _i (V _i ),

Where X represents the input feature sequence, W ^Q , _Wi ^K , _Wi ^V are the linear projection parameters of the i-th scale layer on the same head of the self-attention layer, LE _i (·) is the local enhancement component of the depthwise convolution; and S _Ti (·) is the scale transformation operation of the i-th scale layer, which provides downsampling of X, which can be specifically expressed as:

ST _i (X) = LN (Conv _i (X)),

Among them, LN(·) refers to layer normalization; Conv _i (·) refers to the convolution used by the i-th scale layer. Different scale layers use different kernel sizes and strides, resulting in different scale transformations; therefore, in the same self-attention layer, the keys and values capture features of different scales; then, the self-attention head can be calculated as:

Where Softmax(·) is the activation function, T represents the transpose operation, and d is the dimension. Connect different heads and calculate the multi-head self-attention in the main branch by:

MH(Q,K,V)=Concat(head ₀ ,...,head _j )W ^O ,

Where Concat(·) is a concatenation operation, head _i represents the i-th self-attention head, and W ^O is a linear projection parameter; (A22) the other branch that retains neighborhood information only performs linear projection and convolution operations on the input features in sequence. The multi-scale self-attention can be expressed as:

MSSA=Concat(MH{Q,K _i ,V _i } _i=1,...,m )+Conv ₀ (XW ⁰ )..

Where Concat(·) is a concatenation operation, MH represents multi-head self-attention, Q, _Ki , _Vi represent the query vector, the key vector of the i-th scale layer, and the value vector of the i-th scale layer, respectively, Conv ₀ (·) refers to the convolution operation, X is the input feature sequence, and W ^O is the linear projection parameter;

Preferably, in step (3), the loss function is composed of five loss functions:

(3.1) Reconstruction loss L _Rec is used to calculate the error between the input image and the reconstructed image to constrain the entire network. L _Rec is defined as:

L _Rec ＝||I(x)-x|| _F ,

Among them, ||·|| _F represents the Frobenius norm of the given matrix, x is the input haze image, and I(x) is the haze image reconstructed by the output of the three subnetworks;

(3.2) The loss function L _J is used to calculate the difference between the brightness and saturation of the haze-free image estimate J _R (x), which is specifically expressed as:

L _J =||V(J _R (x))-S(J _R (x))|| _F ,

where V(·) represents brightness, S(·) represents saturation, and ||·|| _F represents the Frobenius norm of the given matrix.

(3.3) Atmospheric light loss L _H is used to calculate the loss between the estimated atmospheric light A(x) and A _init (x), where A _init (x) is the initial atmospheric light automatically estimated from the input image data. L _H is specifically expressed as:

L _H =||A(x)-A _init (x)|| _F .

(3.4) L _KL is the variational inference loss of the reparameterized module in the atmosphere estimation optical network, making the latent code after resampling in the network Minimize the difference between z and before sampling. Mathematically,

where KL(·) represents the Kullback-Leibler divergence between two distributions, z _i represents the i-th dimension of z, μ _z ,σ _z represent the mean and standard deviation of z respectively. denote the mean and standard deviation of z _i respectively;

(3.5) In order to avoid network overfitting, the regularization term loss L _Reg is set, which is specifically expressed as:

Where N(·) represents the second-order neighborhood, |N(·)| represents the size of the second-order neighborhood, n represents the number of pixels of A(x), and A(x) represents atmospheric light;

The total loss is defined by combining the above five loss functions:

L＝L _Rec +L _J +L _H +L _KL +λL _Reg ,

Among them, the parameter λ before L _Reg is a non-negative parameter used to balance regularization, which is 0.1 in practice.

In practical application:

The present invention provides a two-stage decoupled image defogging method based on zero-shot learning, which can perform defogging in an end-to-end manner and mainly consists of two stages: the first stage: using the embedded dark channel prior to obtain a rough estimate of the haze-free image and the transmission map; the second stage, using two sub-networks to refine the results of the first stage to obtain a more accurate haze-free image and transmission map, while the other sub-network directly estimates the atmospheric illumination. In particular, the present invention introduces a new multi-scale Transformer block in the sub-network of the refined haze-free image of the defogging network model. After the defogging network model is constructed, the loss is calculated according to the designed loss function, and the parameters are continuously iterated and updated to directly perform image defogging on the haze image.

1. Two-stage decoupling defogging network model

The first stage of building a two-stage decoupled defogging network model includes the following steps:

Step 1: Calculate the dark channel of the input image x. First, take the minimum value of the three RGB channels to construct a grayscale image, then invert the grayscale image, perform the maximum pooling operation, and invert the result after the maximum pooling to get I ^dark . The specific formula is:

Where ^Ic refers to one of the R, G, or B channels of I.

Step 2: Calculate a rough estimate of the transmission diagram, T _DCP (x). Substituting I ^dark into the atmospheric scattering physics model yields:

I ^dark (x)＝J ^dark (x)t(x)+A[1-t(x)], (2)

Where I ^dark (x) and J ^dark (x) are the dark channels of images I and J, respectively. For the atmospheric light A, the pixels with the top 0.1% brightness in I ^dark are selected and their average value is taken as the atmospheric light A. For J ^dark , according to the dark channel prior assumption, the darkness of the non-sky area of the outdoor fog-free image J usually tends to zero, that is,

J ^dark (x)→0. (3)

Combined with equation 2, the preliminary results of the transmission diagram are obtained according to the following formula:

Since the edge transition of the dark channel image is not smooth, guided filtering is used to make the edge of T _DCP (x) smoother. An average pooling operation is used to implement guided filtering, and the kernel size of the average pooling operation is 19*19 and the stride is 1.

Step 3: Calculate a rough estimate of _JDCP (x) for the haze-free image. Based on the physical model of atmospheric scattering, and given the atmospheric light A and the transmission map T _DCP (x), _JDCP (x) can be calculated using the following equation:

The second stage of building a two-stage decoupled defogging network model includes the following three sub-networks:

(1) Fog-free image refinement network: The architecture of the fog-free image refinement network is an improved 5-level U-Net architecture, in which the convolutional blocks are replaced by the introduced multi-scale Transformer blocks, and the SK fusion module is used to connect and fuse different layers. A soft reconstruction layer is used to obtain the global residual.

(2) Transfer Graph Refinement Network: The transfer graph refinement network is a non-degenerate structured network consisting of five convolutional layers. Specifically, each of the first four layers of this subnet only includes a convolutional layer, a batch normalization layer, and a LeakyReLU activation function. The last layer includes a convolutional layer and a sigmoid function that normalizes the output to [0,1].

(3) Atmospheric light estimation network: The atmospheric light estimation network is a symmetrical encoder-decoder network. The encoder and decoder of the network each have four layers. Each layer of the encoder contains a convolution layer, a ReLU activation function, and a maximum pooling layer. Each layer of the decoder contains an upsampling layer, a convolution layer, a batch normalization layer, and a ReLU activation function. In between the encoder and the decoder, a reparameterization technique module is used. This module converts the output of the encoder into a potential Gaussian distribution, which is then resampled and input into the decoder.

2. Multi-scale Transformer Block

A two-stage decoupled defogging network model is constructed to refine the haze-free image, including the RescaleNorm layer, multi-scale self-attention, and MLP layer. The RescaleNorm layer and the multi-scale self-attention layer introduced in this invention are specifically introduced here:

(1) RescaleNorm layer. As a normalization layer, the specific normalization process of RescaleNorm can be expressed as:

where F(·) represents the main part of the multi-scale self-attention in the multi-scale Transformer block, denote the mean and standard deviation respectively, are the learned scaling factor and bias, respectively. and are the weights and bias terms of two linear layers used to convert μ and σ respectively. The conversion process is expressed as {γ, β} = {σW _γ + B _γ , μW _β + B _β }. In order to accelerate convergence, B _γ and B _β are initialized to 1 and 0.

(2) Multi-scale self-attention. Multi-scale self-attention (MSSA) has two branches, which aggregates multi-scale feature information within the window while retaining the information of the neighborhood to a certain extent.

In the main branch, the query vector Q, key vector K, and value vector V are processed differently. For Q, the original scale is retained and a normal full connection operation is performed on it. For key K and value V, in the same self-attention layer, scale transformation (ST) is used to obtain K and V of different sizes, which can be specifically expressed as:

Where X represents the input feature sequence, W ^Q , _Wi ^K , _Wi ^V are the linear projection parameters of the i-th scale layer on the same head of the self-attention layer, and LE _i (·) is the local enhancement component of the deep convolution. And ST _i (·) is the scale transformation operation of the i-th scale layer, which provides downsampling of X, which can be specifically expressed as:

ST _i (X) = LN (Conv _i (X)), (8)

Among them, LN(·) refers to layer normalization. Conv _i (·) refers to the convolution used by the i-th scale layer. Different scale layers use different kernel sizes and strides, resulting in different scale transformations. Therefore, in the same self-attention layer, the keys and values can capture features of different scales. Then, the self-attention head can be calculated as:

Where Softmax(·) is the activation function, T represents the transpose operation, and d is the dimension. Connect different heads and calculate the multi-head self-attention in the main branch by:

MH(Q,K,V)=Concat(head ₀ ,...,head _j )W ^O , (10)

Where Concat(·) is the concatenation operation, head _i represents the i-th self-attention head, and W ^O is the linear projection parameter.

The other branch that retains neighborhood information only performs linear projection and convolution operations on the input features in sequence. In summary, multi-scale self-attention can be expressed as:

MSSA=Concat(MH{Q,K _i ,V _i } _i=1,...,m )+Conv ₀ (XW ⁰ ). (11)

Where Concat(·) is a concatenation operation, MH represents multi-head self-attention, Q, _Ki , _Vi represent the query vector, the key vector of the i-th scale layer, and the value vector of the i-th scale layer, respectively, Conv ₀ (·) refers to the convolution operation, X is the input feature sequence, and W ^O is the linear projection parameter.

3. Loss Function

The loss function is composed of the following five loss functions:

(1) Reconstruction loss L _Rec is used to calculate the error between the input image and the reconstructed image, so as to constrain the entire network. L _Rec is defined as:

L _Rec ＝||I(x)-x|| _F , (12)

Among them, ||·|| _F represents the Frobenius norm of the given matrix, x is the input haze image, and I(x) is the haze image reconstructed by the output of the three subnetworks.

(2) The loss function L _J is used to calculate the difference between the brightness and saturation of the haze-free image estimate J _R (x), which is specifically expressed as:

L _J =||V(J _R (x))-S(J _R (x))|| _F , (13)

Among them, V(·) represents brightness and S(·) represents saturation.

(3) Atmospheric light loss L _H is used to calculate the loss between the estimated atmospheric light A(x) and A _init (x), where A _init (x) is the initial atmospheric light automatically estimated from the input image data. _{L H} is specifically expressed as:

L _H =||A(x)-A _init (x)|| _F . (14)

(4) L _KL is the variational inference loss of the reparameterized module in the atmosphere estimation optical network. Its purpose is to make the latent code after resampling in the network Minimize the difference between z and before sampling. Mathematically,

where KL(·) represents the Kullback-Leibler divergence between two distributions, z _i represents the i-th dimension of z, μ _z ,σ _z represent the mean and standard deviation of z respectively. represent the mean and standard deviation of _zi respectively.

(5) In order to avoid network overfitting, the regularization term loss L _Reg is set, which can be specifically expressed as:

Where N(·) represents the second-order neighborhood, |N(·)| represents the size of the second-order neighborhood, n represents the number of pixels of A(x), and A(x) represents atmospheric light.

In summary, the total loss is defined by combining the above five loss functions:

L＝L _Rec +L _J +L _H +L _KL +λL _Reg , (17)

The parameter λ before L _Reg is a non-negative parameter used to balance regularization, which is 0.1 in practice.

The present invention is based on zero-sample learning and directly inputs the haze image into the network for dehazing without providing haze-clean image pairs, thus avoiding large-scale data training. At the same time, in the process of restoring the haze-free image, the method can not only ensure the extraction of multi-scale features, but also ensure the extraction of original scale and neighborhood information.

According to the physical model of atmospheric scattering, the network model is divided into three layers based on the idea of decoupling, namely the potential haze-free image layer, the transmission layer layer and the global atmospheric light layer. The potential haze-free image and the transmission layer layer are estimated in two stages. The first stage uses the embedded dark channel prior for rough estimation, and the second stage uses the haze-free image refinement network and the transmission layer refinement network for refined estimation. For the global atmospheric light, an encoder-decoder network is used for estimation. At the top of the network, the images estimated by the three subnetworks are reconstructed according to the physical model of atmospheric scattering to obtain the reconstructed haze image. A new multi-scale Transformer block is introduced in the refined haze-free image subnetwork of the dehazing network model, which performs multi-scale token aggregation in the self-attention part, enabling it to capture features of different scales and effectively restore the potential scene information in the haze image.

The specific description of the present invention in the above embodiments is only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Technical engineers in this field may make some non-essential improvements and adjustments to the present invention based on the contents of the above invention, which fall within the scope of protection of the present invention.

Claims (6)

1. A two-stage decoupled image defogging method based on zero-shot learning, characterized in that it includes the following steps: (1) Obtain two test subsets of the synthetic target test set and the mixed subjective test set from the real single image dehazing dataset, and preprocess the original dataset as the test set; (2) Constructing a dehazing network model: According to the atmospheric scattering physical model I(x) = J(x)t(x) + A[1-t(x)], Where x represents the input haze image, I(x) represents the reconstructed haze image, J(x) represents the original scene information, t(x) represents the transmittance, and A represents the atmospheric scattered light; Based on the idea of decoupling, the network model is divided into three layers: potential fog-free image layer, transmission layer layer and global atmospheric light layer. The potential fog-free image and transmission layer are estimated in two stages. The first stage uses the embedded dark channel prior for rough estimation, and the second stage uses the fog-free image refinement network and the transmission layer refinement network for refined estimation. For the global atmospheric light, an encoder-decoder network is used for estimation. At the top of the network, the images estimated by the three sub-networks of the fog-free image refinement network, the transmission layer refinement network and the atmospheric light estimation network are reconstructed according to the atmospheric scattering physical model to obtain the reconstructed haze image. (3) The haze images in the preprocessed data set are input into the constructed dehazing network model, the loss is calculated according to the designed loss function, the parameters are continuously updated iteratively, and the haze images are directly dehazed. 2. According to the two-stage decoupled image defogging method based on zero-shot learning in claim 1, it is characterized in that: step (2), the rough estimation in the first stage specifically comprises the following steps: (2.1) Calculate the dark channel of the input image x: First, take the minimum value of the three RGB channels to construct a grayscale image, then invert the grayscale image, perform a maximum pooling operation, and invert the result after maximum pooling to obtain I ^dark . The specific formula is: Where I ^c refers to one of the R, G, or B channels of I; (2.2) Calculate a rough estimate of the transmission map T _DCP (x): Substitute I ^dark into the atmospheric scattering physical model to obtain I ^dark (x) = J ^dark (x)t(x) + A(1-t(x)), where I ^dark (x) and J ^dark (x) are the dark channels of images I and J, respectively; for atmospheric light A, select the top 0.1% of pixels in I ^dark and take their average value as the value of atmospheric light A; and for J ^dark , according to the dark channel prior assumption, the darkness of the non-sky area of the outdoor fog-free image J usually tends to zero, J ^dark (x)→0, so according to The preliminary results of the transmission graph are obtained; since the edge transition of the dark channel image is not smooth, the guided filtering method is used to make the edge of T _DCP (x) smoother. An average pooling operation is used to implement the guided filtering. The kernel size of the average pooling operation is 19*19 and the stride is 1. (2.3) Calculate a rough estimate of the haze-free image _JDCP (x): According to the physical model of atmospheric scattering, and with the known atmospheric light A and transmission map T _DCP (x), _JDCP (x): is calculated by equation 3. According to the two-stage decoupled image defogging method based on zero-shot learning in claim 2, it is characterized in that: step (2), constructing three subnets of the defogging network model, specifically comprises the following steps: (A) The architecture of the haze-free image refinement network is an improved 5-level U-Net architecture, in which the convolutional blocks are replaced by the introduced multi-scale Transformer blocks, and the SK fusion module is used to connect and fuse different layers, and a soft reconstruction layer is used to obtain the global residual; (B) The transfer graph refinement network is a non-degenerate structured network consisting of five convolutional layers. In the first four layers, each layer only includes a convolutional layer, a batch normalization layer, and a LeakyReLU activation function. The last layer includes a convolutional layer and a sigmoid function that normalizes the output to [0, 1]. (C) The atmospheric light estimation network is a symmetrical encoder-decoder architecture network. The encoder and decoder of the network each have four layers. Each layer of the encoder includes a convolution layer, a ReLU activation function and a maximum pooling layer. Each layer of the decoder includes an upsampling layer, a convolution layer, a batch normalization layer and a ReLU activation function. In the middle of the encoder and the decoder, a reparameterization trick module is used, which converts the output of the encoder into a potential Gaussian distribution, which is then resampled and input into the decoder. 4. The two-stage decoupled image dehazing method based on zero-shot learning according to claim 3 is characterized in that: in step (A), constructing a multi-scale Transformer block specifically comprises the following steps: (A1) The RescaleNorm layer is used as the normalization layer. The specific normalization process of RescaleNorm is expressed as: where F(·) represents the main part of the multi-scale self-attention in the multi-scale Transformer block, denote the mean and standard deviation respectively, are the learned scaling factor and bias, respectively. and are the two linear layer weights and bias terms used to convert μ and σ respectively. The conversion process is expressed as {γ, β} = {σW _γ +B _γ ,μW _β +B _β }. In order to accelerate convergence, B _γ and B _β are initialized to 1 and 0. (A2) Multi-scale self-attention is used to calculate self-attention. Multi-scale self-attention (MSSA) has two branches, which aggregates multi-scale feature information within the window while retaining the information of the neighborhood to a certain extent. 5. The two-stage decoupled image dehazing method based on zero-shot learning according to claim 4, characterized in that: in step (A2), multi-scale self-attention (MSSA) is calculated, including two branches: (A21) In the branch, the query vector Q, key vector K, and value vector V are processed differently. For Q, the original scale is retained and a normal full connection operation is performed on it. For K and V, in the same self-attention layer, scale transformation (ST) is used to obtain K and V of different sizes, which can be specifically expressed as: Q＝XW ^Q , K _i ,V _i =ST _i (X)W _i ^K ,ST _i (X)W _i ^V , V _i =V _i +LE _i (V _i ), Where X represents the input feature sequence, W ^Q , _Wi ^K , _Wi ^V are the linear projection parameters of the i-th scale layer on the same head of the self-attention layer, LE _i (·) is the local enhancement component of the depthwise convolution; and S _Ti (·) is the scale transformation operation of the i-th scale layer, which provides downsampling of X, which can be specifically expressed as: ST _i (X) = LN (Conv _i (X)), Among them, LN(·) refers to layer normalization; Conv _i (·) refers to the convolution used by the i-th scale layer. Different scale layers use different kernel sizes and strides, resulting in different scale transformations; therefore, in the same self-attention layer, the keys and values capture features of different scales; then, the self-attention head can be calculated as: Where Softmax(·) is the activation function, T represents the transposition operation, and d is the dimension. Connect different heads and calculate the multi-head self-attention (MH) in the main branch by the following method: MH(Q,K,V)=Concat(head ₀ ,...,head _j )W ^O , Where Concat(·) is the concatenation operation, head _i represents the i-th self-attention head, and W ^O is the linear projection parameter; (A22) Another branch that retains neighborhood information only performs linear projection and convolution operations on the input features in sequence. The multi-scale self-attention MSSA can be expressed as: MSSA=Concat(MH{Q,K _i ,V _i } _i=1,...,m )+Conv ₀ (XW ⁰ ), Where Concat(·) is a concatenation operation, MH represents multi-head self-attention, Q, _Ki , _Vi represent the query vector, the key vector of the i-th scale layer, and the value vector of the i-th scale layer, respectively, Conv ₀ (·) refers to the convolution operation, X is the input feature sequence, and W ^O is the linear projection parameter. 6. The two-stage decoupling image defogging method based on zero-shot learning according to claim 5, characterized in that: in step (3), the loss function is composed of five loss functions: (3.1) Reconstruction loss L _Rec is used to calculate the error between the input image and the reconstructed image to constrain the entire network. L _Rec is defined as: L _Rec ＝||I(x)-x|| _F , Among them, ||·|| _F represents the Frobenius norm of the given matrix, x is the input haze image, and I(x) is the haze image reconstructed by the output of the three subnetworks; (3.2) The loss function L _J is used to calculate the difference between the brightness and saturation of the haze-free image estimate J _R (x), which is specifically expressed as: L _J =||V(J _R (x))-S(J _R (x))|| _F , where V(·) represents brightness, S(·) represents saturation, and ||·|| _F represents the Frobenius norm of the given matrix. (3.3) Atmospheric light loss L _H is used to calculate the loss between the estimated atmospheric light A(x) and A _init (x), where A _init (x) is the initial atmospheric light automatically estimated from the input image data. L _H is specifically expressed as: L _H =||A(x)-A _init (x)|| _F . (3.4) L _KL is the variational inference loss of the reparameterized skill module in the atmosphere estimation optical network, making the latent code after resampling in the network Minimize the difference between the latent code z before sampling. Mathematically, where KL(·) represents the Kullback-Leibler divergence between two distributions, z _i represents the i-th dimension of z, μ _z ,σ _z represent the mean and standard deviation of z respectively. denote the mean and standard deviation of z _i respectively; (3.5) In order to avoid network overfitting, the regularization term loss L _Reg is set, which is specifically expressed as: Where N(·) represents the second-order neighborhood, |N(·)| represents the size of the second-order neighborhood, n represents the number of pixels of A(x), and A(x) represents atmospheric light; The total loss is defined by combining the above five loss functions: L＝L _Rec +L _J +L _H +L _KL +λL _Reg , Among them, the parameter λ before L _Reg is a non-negative parameter used to balance regularization, which is 0.1 in practice.