WO2024016611A1 - Image processing method and apparatus, electronic device, and computer-readable storage medium - Google Patents

Abstract

The present application relates to the technical field of artificial intelligence, and discloses an image processing method and apparatus, an electronic device, and a computer-readable storage medium. The image processing method comprises: acquiring a first watermark generated for a deepfake model and a second watermark carrying target information; acquiring a target coding model, the target coding model being used for performing watermark embedding on an image; and embedding the first watermark and the second watermark into an initial image by means of the target coding model to obtain a target image, a difference between the initial image and the target image being less than a reference threshold.

Description

Image processing method, device, electronic equipment and computer-readable storage medium

This application claims priority to the Chinese patent application with application number 202210845845.5 and the application title "An active defense method and system against deep forgery" submitted on July 19, 2022, the entire content of which is incorporated into this application by reference. middle.

Technical field

This application belongs to the field of artificial intelligence technology and involves image processing methods, devices, electronic equipment and computer-readable storage media.

Background technique

With the continuous development of artificial intelligence technology, some technologies for tampering with images have gradually emerged, and deepfake technology is one of them. Therefore, the image needs to be processed to prevent it from being tampered with by deepfake technology.

Contents of the invention

This application provides an active defense method and system against deep forgery, and also provides an image processing method, device, electronic equipment and computer-readable storage medium. The technical solution provided by this application includes the following aspects.

On the one hand, an active defense method against deep forgery is provided, the steps of which include:

1) Obtain active defense watermarks: Prepare multiple deep forgery models and already trained deep forgery model parameters. Specifically include:

1-1) Add any original training image and the defensive watermark to the image (if this is the first training, initialize the watermark to random noise), input it into the deep forgery model, and obtain the original image and the watermarked image. of doctored images.

1-2) Pass the loss back on different deep fake models to obtain the gradient sequence on the picture.

1-3) After synthesizing each image and each model gradient sequence, and subjecting them to upper and lower limits, a defensive watermark is obtained.

1-4) During each training, the watermark is updated based on the defensive watermark obtained in the previous training. Specifically, the watermark obtained in this training needs to be multiplied by the coefficient α (usually 0.01) and the last watermark multiplied by the coefficient 1 -α gets new defense watermark.

1-5) Repeat until the upper limit of training times is reached to obtain an active defense watermark that can distort the generation of multiple deepfake models.

2) Training watermark embedding and detection: specifically including:

2-1) Prepare a certain number of face pictures;

2-2) Train a training encoder-decoder. Among them, the encoder embeds the active defense watermark obtained in the previous step into the input image, and uses the loss function to ensure that the embedded information is invisible. Afterwards, the decoder reads the embedded image and decodes the encoded watermark, ensuring the accuracy of the decoded information through the loss function. When training is completed, the corresponding encoder and decoder weights are generated.

3) Deep fake detection: specifically including:

3-1) Prepare face pictures that need to be protected (or videos that need to be protected divided by frames), and deep fake models that need to be defended;

3-2) Use the encoder obtained in the previous step to embed the active defense watermark into the face image, input the face image into the deep forgery model, and obtain the forged image;

3-3) Use the decoder obtained in the previous step to decode the encoded watermark from the forged picture and compare it with the original embedded watermark. When the bit difference between the two is greater than or equal to the set threshold (usually 0.4 ), the image is considered to be deepfake.

On the one hand, an active defense system against deep forgery is provided, which includes:

1) Deep forgery model interface module: includes functions for inputting images to the deep forgery model and obtaining the generated results;

2) Active defense watermark generation module: used to generate defensive watermarks that protect faces from multiple deep forgery models; specifically, this module first completes access to the deep forgery model, calls the basic watermark generation algorithm, and generates the model combined with watermark fusion technology Universal active defense watermark.

3) Active defense watermark embedding module: This module trains the encoder-decoder, and uses the encoder to embed the universal watermark generated by the active defense watermark generation module into the face image.

4) Watermark defense effect evaluation module: used to evaluate the degree to which the watermark distorts the output of the deep forgery model;

5) Deep forgery detection module: Through the decoder provided by the active defense watermark embedding module, pictures with embedded watermarks are detected to determine whether these pictures have been modified by a deep forgery model.

The beneficial effects brought by the technical solutions provided by the embodiments of this application at least include:

By generating a model-general active defense watermark, embedding the watermark into media containing face information can distort the generation of deep forgery models, and use the watermark to detect whether the media content has experienced deep forgery, completely preventing deep forgery. Forgery and tampering. The embodiments of the present application have defense capabilities against a variety of deep forgery models, and can achieve defense effects without the need for structural information of the deep forgery models.

On the one hand, an image processing method is provided, which method includes:

Obtain the first watermark generated for the deep forgery model and the second watermark carrying target information;

Obtain a target coding model, which is used for watermark embedding for the image;

The first watermark and the second watermark are embedded in the initial image through the target encoding model to obtain a target image, and the difference between the initial image and the target image is less than a reference threshold.

In an exemplary embodiment, embedding the first watermark and the second watermark into an initial image through the target encoding model to obtain a target image includes: superimposing the first watermark onto the initial image. , obtain the first image, input the first image and the second watermark into the target coding model, and obtain the target image output by the target coding model; or, combine the second watermark and the initial The image is input into the target encoding model to obtain a second image output by the target encoding model, and the first watermark is superimposed on the second image to obtain the target image.

In an exemplary embodiment, before embedding the first watermark and the second watermark into the initial image through the target encoding model and obtaining the target image, the method further includes: acquiring a first sample image, carrying The third watermark of the first information and the initial coding model; input the first sample image into the initial coding model to obtain the first result image output by the initial coding model; combine the first sample image and the initial coding model The third watermark is input into the initial coding model to obtain a second result image output by the initial coding model, and the second result image is embedded with the third watermark; according to the first result image and the second The result image determines a first loss function, the first loss function is used to indicate the difference between the first result image and the second result image; the initialization is updated by minimizing the first loss function. Encoding model to obtain the target encoding model, wherein the minimized first loss function is smaller than the reference threshold.

In an exemplary embodiment, after embedding the first watermark and the second watermark into the initial image through the target encoding model and obtaining the target image, the method further includes: obtaining the generated data based on the target image. Update the image; input the updated image into the target decoding model to obtain the fourth watermark output by the target decoding model; determine the bit error between the second watermark and the fourth watermark; when the bit error is greater than the error In the case of a threshold, it is determined that the updated image is an image obtained by deep forging the target image using the deep forgery model.

In an exemplary embodiment, before inputting the updated image into the target decoding model, the method further includes: obtaining a second sample image and an initial decoding model, the second sample image is embedded with a third image carrying the second information. Five watermarks; input the second sample image into the initial decoding model to obtain the sixth watermark output by the initial decoding model; determine a second loss function according to the fifth watermark and the sixth watermark; by minimizing The process of the second loss function updates the initial decoding model to obtain the target decoding model.

In an exemplary embodiment, obtaining the first watermark generated for the deep forgery model includes: obtaining a third sample image, a seventh watermark, and the deep forgery model; and inputting the third sample image into the deep forgery model. model to obtain the third result image output by the deep forgery model; superimpose the seventh watermark to the third sample image to obtain a fourth sample image; input the fourth sample image into the deep forgery model, Obtain the fourth result image output by the deep forgery model; determine a third loss function according to the third result image and the fourth result image; determine gradient information according to the third loss function; update according to the gradient information The seventh watermark is the first watermark generated for the deep forgery model.

In an exemplary embodiment, there are multiple deepfake models and multiple gradient information, and multiple deepfake models correspond to multiple gradient information in one-to-one correspondence; the updating of the deepfake model according to the gradient information The seventh watermark is to obtain the first watermark generated for the deep forgery model, including: averaging the multiple gradient information to obtain target gradient information; calculating the target gradient information through a symbolic function to obtain the first calculation Result; generate a second calculation result according to the first calculation result, superimpose the second calculation result to the fourth sample image, and obtain a fifth sample image; apply upper and lower limits constraints on the fifth sample image, and obtain the sixth Sample image; remove the third sample image from the sixth sample image to obtain an eighth watermark; perform a weighted average of the seventh watermark and the eighth watermark to obtain the first watermark.

On the one hand, an image processing device is provided, which device includes:

The acquisition module is used to obtain the first watermark generated for the deep forgery model and the second watermark carrying target information;

The acquisition module is also used to acquire a target coding model, which is used to embed watermarks on images;

An embedding module, configured to embed the first watermark and the second watermark into an initial image through the target encoding model to obtain a target image, where the difference between the initial image and the target image is less than a reference threshold.

In an exemplary embodiment, the embedding module is configured to superimpose the first watermark onto the initial image to obtain a first image, and input the first image and the second watermark into the target encoding. model to obtain the target image output by the target encoding model; or, input the second watermark and the initial image into the target encoding model to obtain the second image output by the target encoding model, and convert the The first watermark is superimposed on the second image to obtain the target image.

In an exemplary embodiment, the acquisition module is also used to acquire a first sample image, a third watermark carrying first information, and an initial encoding model; input the first sample image into the initial encoding model, Obtain the first result image output by the initial encoding model; input the first sample image and the third watermark into the initial encoding model to obtain the second result image output by the initial encoding model, and the third The second result image is embedded with the third watermark; a first loss function is determined according to the first result image and the second result image, and the first loss function is used to indicate the difference between the first result image and the third result image. The difference between the two result images; the initial encoding model is updated through the process of minimizing the first loss function to obtain the target encoding model, wherein the minimized first loss function is smaller than the reference threshold.

In an exemplary embodiment, the acquisition module is also used to acquire an updated image generated based on the target image; input the updated image into the target decoding model to obtain the fourth watermark output by the target decoding model; determine the The bit error between the second watermark and the fourth watermark; when the bit error is greater than the error threshold, it is determined that the updated image is obtained by deep forging the target image through the deep forgery model. Image.

In an exemplary embodiment, the acquisition module is also used to acquire a second sample image and an initial decoding model, the second sample image is embedded with a fifth watermark carrying second information; input the second sample image The initial decoding model obtains the sixth watermark output by the initial decoding model; determines a second loss function according to the fifth watermark and the sixth watermark; and updates the second loss function through the process of minimizing the second loss function. Initial decoding model to obtain the target decoding model.

In an exemplary embodiment, the acquisition module is used to acquire a third sample image, a seventh watermark and the deep forgery model; input the third sample image into the deep forgery model to obtain the deep forgery model. The third result image output; superimpose the seventh watermark to the third sample image to obtain a fourth sample image; input the fourth sample image into the deep forgery model to obtain the output of the deep forgery model. a fourth result image; determine a third loss function according to the third result image and the fourth result image; determine gradient information according to the third loss function; update the seventh watermark according to the gradient information to obtain the Describes the first watermark generated for the deep fake model.

In an exemplary embodiment, the number of the deep forgery models and the number of the gradient information are both multiple, and the multiple deep forgery models correspond to the multiple gradient information one-to-one; the acquisition module is used to obtain the A plurality of gradient information are averaged to obtain target gradient information; the target gradient information is calculated through a symbolic function to obtain a first calculation result; a second calculation result is generated according to the first calculation result, and the second calculation result is superimposed to The fourth sample image is used to obtain a fifth sample image; the upper and lower limits are applied to the fifth sample image to obtain a sixth sample image; the third sample image is removed from the sixth sample image to obtain an eighth sample image. Watermark: perform a weighted average of the seventh watermark and the eighth watermark to obtain the first watermark.

On the one hand, an electronic device is provided. The electronic device includes a memory and a processor; at least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor, so that the electronic device implements the present invention. The active defense method against deep forgery or the image processing method provided by any of the exemplary embodiments of the application.

On the one hand, a computer-readable storage medium is provided. At least one instruction is stored in the computer-readable storage medium. The instruction is loaded and executed by a processor to enable the computer to implement any exemplary implementation of the present application. The active defense method against deep forgery provided by the example, or the image processing method.

On the other hand, a computer program or computer program product is provided. The computer program or computer program product includes: computer instructions. When the computer instructions are executed by a computer, the computer implements any exemplary embodiment of the present application. The embodiment provides an active defense method against deep forgery or an image processing method.

The beneficial effects brought by the technical solutions provided by the embodiments of this application at least include:

Embed the first watermark generated for the deep forgery model and the second watermark carrying target information into the initial image, and use the target encoding model during the embedding process to make the difference between the initial image and the target image obtained after embedding smaller. , thereby making the first watermark and the second watermark invisible in the target image to avoid affecting the content recorded in the target image. Moreover, this kind of target image can not only defend against deep forgery models, but also carry customized target information, ensuring the accuracy of image processing.

Description of drawings

Figure 1 is a schematic diagram of the generation of an active defense watermark provided by an embodiment of the present application;

Figure 2 is a schematic diagram of an active defense watermark embedding and deep forgery detection provided by an embodiment of the present application;

Figure 3 is a schematic diagram of an implementation environment provided by an embodiment of the present application;

Figure 4 is a flow chart of an image processing method provided by an embodiment of the present application;

Figure 5 is a schematic diagram of generating a first watermark provided by an embodiment of the present application;

Figure 6 is a schematic diagram of watermark embedding and deep forgery detection provided by an embodiment of the present application;

Figure 7 is a structural diagram of an image processing device provided by an embodiment of the present application;

Figure 8 is a structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

With the continuous development of deep learning technology, the technology of modifying facial images and videos: deepfake has exploded in popularity on the Internet. Deep forgery technology modifies faces through attribute modification or facial replacement. It can modify hair color, face shape and other appearance features, or replace faces on other videos and images to make characters behave inconsistently with their identities or convey false information. information. For example, StarGAN (Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation, a unified generative adversarial network for multi-domain image-to-image translation) can generate people with different facial features and expressions from an original face picture. Face doctored images. For another example, InterfaceGAN (Interpreting the Latent Space of GANs for Semantic Face Editing, explaining the unified generation of adversarial network latent space for semantic face editing) can be edited through latent variables to generate face images with controllable camera angles.

In social media and short video platforms, audio and video media created using deep fakes can be seen everywhere, and the social problems they cause have gradually attracted attention. For example, audio and video media created by deep fakes may be used to spread false information, causing huge damage to user images.

In response, many short video platforms have begun to take measures to regulate and ban face-changing videos. However, the current measures taken by the platform against deep fakes are mainly passive detection, that is, training detectors to detect videos that have been produced and released to determine whether they are deep fake content. This kind of detection can only passively defend and collect evidence afterwards, and cannot prevent the generation and spread of deep fake content. There is no way to cut off the negative impact of false content; and in the face of the ever-changing deep fake models, detectors need to be continuously trained and updated, which costs a lot of money. The price is very high.

This application designs an active defense system against deep forgery. The system includes five modules: deep forgery model interface, watermark generation, watermark embedding, defense effect evaluation, and deep forgery detection. in:

1) Deep forgery model interface module: includes functions for inputting images to the deep forgery model and obtaining the generated results;

2) Active defense watermark generation module: used to generate defensive watermarks that protect faces from multiple deep forgery models; specifically, this module first completes access to the deep forgery model, calls the basic watermark generation algorithm, and generates the model combined with watermark fusion technology Universal active defense watermark.

3) Active defense watermark embedding module: This module trains the encoder-decoder and uses the encoder to embed the universal watermark generated by the active defense watermark generation module into the face image.

4) Watermark defense effect evaluation module: used to evaluate the degree to which the watermark distorts the output of the deep forgery model;

5) Deep forgery detection module: Through the decoder provided by the active defense watermark embedding module, pictures with embedded watermarks are detected to determine whether these pictures have been modified by a deep forgery model.

In order to further illustrate the present application, the specific implementation is described below through examples, but the scope of application of the method is not limited in any way.

Based on the large-scale face attribute data set CelebA (CelebFaces Attributes Dataset: http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html) and the deep forgery models HiSD and Stargan trained on this data set, AttGAN and Attentiongan are used as attack targets, and the PGD attack algorithm is used as the basic attack algorithm to explain how to generate active defense watermarks, how to embed watermarks, and how to perform deep forgery detection.

Prepare the encapsulated Deepfake model; read the clean CelebA data set, scale it to 256×256 size and perform standardized preprocessing, and divide the CelebaA data set into a training set, a verification set and a test set.

The first step is to obtain the active defense watermark, as shown in Figure 1:

1) Add any batch of original training images and the image with a defensive watermark (if this is the first training, initialize the watermark to random noise), input it into the deep forgery model, and obtain the tampered images of the original image and the watermarked image. picture.

2) Pass the loss back on different deep forgery models to obtain the gradient sequence on the input image. The loss is the loss function of the deep forgery model output obtained from the original image and the watermarked image:

Loss _generation =MSE(G(I),G(I+W))

Among them, I is the original image, W is the watermark, and G is the deep fake model.

3) Fusion of each image and each model gradient sequence, and subjecting them to upper and lower bound constraints, a defensive watermark is obtained. Specifically, when synthesizing the gradient sequence of each picture, after obtaining the gradient on a model for a batch of pictures (8 pictures), the gradient is averaged to obtain g _avg , and the PGD algorithm is used to iteratively update 10 times in the positive direction of the gradient. Obtain the adversarial perturbation P:

When fusing the gradient sequences of each model, the adversarial perturbation P obtained on this model needs to be multiplied by the coefficient α (usually 0.01) and the previous watermark multiplied by 1-α to obtain a new defensive watermark.

W′←(1-α)W+αP

4) Repeat until 128 images are trained, and an active defense watermark that can distort the generation of the deep forgery model is obtained.

The second step is to train watermark embedding and detection:

1) Use the training set of CelebA to train a pair of convolutional neural network encoder-decoder. Among them, the encoder embeds the active defense watermark into the input image, and uses the loss function to constrain the embedded image to be close enough to the original image, that is, to minimize the mean square error and ensure that the embedded information is invisible.

Loss _encoding =MSE(E(I),E(I,N))

Among them, E is the encoder and N is the active defense watermark.

2) After that, the decoder reads the embedded picture and decodes the encoded watermark. The bit error between the decoding result and the original watermark is constrained through the loss function, that is, the BCE error function with logit is minimized.

Lossd _encoding =BCEwithLogitsLoss(W,D(E(I,N)))

Among them, D is the decoder.

3) When training is completed, the corresponding encoder E and decoder weights are generated.

The third step is deep forgery detection, as shown in Figure 2:

1) Select the CelebA test set for input;

2) Use the encoder obtained in the previous step to embed the active defense watermark into the face image, input the face image into each deep forgery model, and obtain the forged image;

Through the decoder obtained in the previous step, the encoded watermark is decoded from the forged picture and compared with the original embedded watermark. When the bit difference between the two is greater than or equal to the set threshold (0.4), the picture is considered Passed deepfake. On the full test set of CelebA, the minimum encoding change rate after encoding by the deep forgery model and without forgery is 41.0%, which can be detected.

In the deepfake model attack test with unknown model structure, the embodiment of the present application achieved a 100% deepfake defense rate.

The embodiment of the present application provides an active defense method against deep forgery and an image processing method. These methods can be applied in the implementation environment as shown in Figure 3. In FIG. 3 , at least one electronic device 31 and a server 32 are included. The electronic device 31 can communicate with the server 32 . Taking the image processing method as an example, the electronic device 31 can obtain the image that needs to be processed from the server 32 . Of course, the electronic device 31 can also acquire images that need to be processed by itself, such as by taking pictures or other methods.

Illustratively, the electronic device 31 includes but is not limited to any electronic product that can perform human-computer interaction with the user through one or more methods such as keyboard, touch pad, touch screen, remote control, voice interaction or handwriting device, such as a PC. (Personal Computer), mobile phones, smartphones, PDAs (Personal Digital Assistant), wearable devices, PPC (Pocket PC), tablets, smart cars, smart TVs, smart speakers, etc.

Optionally, the server 32 may be one server, a server cluster composed of multiple servers, or a cloud server.

Those skilled in the art should understand that the above-mentioned electronic device 31 and server 32 are only examples. If other existing or possible future electronic devices and servers are applicable to this application, they should also be included in the protection scope of this application and be included in the protection scope of this application. This is incorporated herein by reference.

Referring to Figure 4, an embodiment of the present application provides an image processing method, which can be applied to the electronic device shown in Figure 3. As shown in Figure 4, the method includes the following steps 401 to 403.

Step 401: Obtain the first watermark generated for the deep forgery model and the second watermark carrying target information.

Among them, the first watermark generated for the deep forgery model is a universal watermark that can defend against the deep forgery model. For example, being able to defend against a deep forgery model means that after an image embedded with such a first watermark is input into the deep forgery model, the deep forgery model will tamper with the image embedded with such a first watermark and output the tampered image. image. However, the tampered image is distorted, so it can be known that the tampered image has been tampered with by the deep forgery model, forming a defense against the deep forgery model. Optionally, deep fake models include but are not limited to StarGAN, InterfaceGAN, HiSD (Hierarchical Style Disentanglement, hierarchical splitting) and AttGAN (Attention Generative Adversarial Networks, attention generation adversarial networks), etc., without limitation.

For example, the first watermark is a watermark in the form of an image. In other words, the first watermark is composed of multiple pixels. Among them, each pixel can correspond to at least one channel, and each channel corresponds to a value.

In addition, the second watermark carrying target information is a personalized watermark that can be customized according to the actual needs of the user. The target information carried by the second watermark is the information that the user wants to reflect. In some implementations, the second watermark is a watermark in the form of an image, such as a logo. Then the second watermark consists of multiple pixels. Each pixel can correspond to at least one channel, and each channel corresponds to a value. In other implementations, the second watermark is a watermark in the form of a string, such as an identification (Identification, ID) used by the user. The embodiment of the present application does not limit the number of digits in the string, and the number of digits in the string can be determined based on experience or actual needs.

Next, the methods of obtaining the first watermark and the method of obtaining the second watermark will be described respectively.

In an exemplary embodiment, obtaining the first watermark generated for the deepfake model includes the following steps A1 to A3.

Step A1: Obtain the third sample image, the seventh watermark and the deep forgery model, superimpose the seventh watermark to the third sample image, and obtain the fourth sample image; input the third sample image into the deep forgery model, and obtain the output of the deep forgery model. The third result image, and the fourth sample image is input into the deep forgery model to obtain the fourth result image output by the deep forgery model.

The third sample image may be an image that has not yet embedded a watermark, and the third sample image is the clean sample shown in Figure 1 . The third sample image can come from public data sets, including but not limited to CelebA (CelebFaces Attributes Dataset, face attribute data set). The seventh watermark is an initialization watermark used to generate the first watermark. For example, the seventh watermark may be a watermark trained through some training processes, or a randomly generated watermark (also known as random noise). The seventh watermark has the same type as the first watermark. For example, when the first watermark is in the form of an image and the seventh watermark is also in the form of an image, then each channel included in the pixels that make up the seventh watermark corresponds to a value. For example, each channel corresponds to a smaller value. For example, when the value range is 0 to 255, the value corresponding to each channel is between 0 and 1. In addition, the size of the seventh watermark is less than or equal to the size of the third sample image, or in other words, the number of pixels included in the seventh watermark is less than or equal to the number of pixels included in the third sample image.

This seventh watermark can be superimposed on the third sample image to obtain a fourth sample image. The fourth sample image is also the adversarial sample shown in Figure 1. In some implementations, the number of pixels included in the seventh watermark is equal to the number of pixels included in the third sample image, then the pixels included in the seventh watermark correspond to the pixels included in the third sample image one-to-one. Correspondingly, superimposing the seventh watermark onto the third sample image to obtain a fourth sample image may include: adding values corresponding to channels included in corresponding pixels to obtain a fourth sample image. Taking one pixel corresponding to the R (Red, red) channel, G (Green, green) channel and B (Blue, blue) channel as an example, for the two corresponding pixels included in the seventh watermark and the third sample image, these The values of the R channel of the two pixels are added, the values of the G channel are added, and the values of the B channel are also added to obtain the fourth sample image. In other implementations, the number of pixels included in the seventh watermark is less than the number of pixels included in the third sample image, then the embodiment of the present application can intercept a part from the third sample image, and the number of pixels included in this intercepted part is equal to the number of pixels included in the third sample image. The seventh watermark includes the same number of pixels, so that a one-to-one pixel relationship can be formed. Alternatively, the number of pixels of the seventh watermark can be increased through interpolation or other methods, so that the increased number of pixels is the same as the number of pixels included in the third sample image, or a one-to-one pixel relationship can be formed. In short, after forming a one-to-one pixel relationship, the values corresponding to the channels included in the corresponding pixels can be added according to the above description to obtain the fourth sample image, which will not be described again here.

Since the third sample image has been obtained, the third sample image can be input into the deep forgery model, so that the deep forgery model tamperes with the third sample image and outputs the tampered third sample image, that is, the third result image. Among them, the third sample image is represented as I, the deep fake model is represented as G(·), and the third result image is represented as G(I). Since the fourth sample image has been obtained, the fourth sample image can be input into the deep forgery model, causing the deep forgery model to tamper with the fourth sample image and output the tampered fourth sample image, that is, the fourth result image. Among them, the seventh watermark is expressed as W. Since the fourth sample image is obtained by superimposing the seventh watermark W on the third sample image I, the fourth sample image is expressed as I+W, and the deep forgery model is still expressed as G( ·), then the fourth result image is expressed as G(I+W).

Step A2: Determine a third loss function based on the third result image and the fourth result image, and determine gradient information based on the third loss function.

Wherein, the third loss function determined according to the third result image and the fourth result image is used to indicate the difference between the third result image and the fourth result image, which difference also represents the difference between the third sample image and the fourth sample image. difference between. The embodiment of the present application does not limit the determination method of the third loss function. The third loss function is obtained by calculating the third result image and the fourth result image through MSE (Mean-Square Error) as an example. , then the third loss function loss _generation is expressed as the following formula (1):

Loss _generation =MSE(G(I),G(I+W)) (1)

For example, in the process of calculating the third result image and the fourth result image through MSE, the pixels included in the third result image and the pixels included in the fourth result image are in one-to-one correspondence. Therefore, you can first calculate the difference between the values corresponding to the channels included in the corresponding pixels, and perform a square calculation on the difference to obtain the square value corresponding to the pixel. After that, the sum of square values corresponding to different pixels is calculated, and the ratio of the sum of square values to the number of pixels is used as the third loss function.

No matter how the third loss function is determined, the gradient calculation can be performed on the third loss function to obtain gradient information, which indicates the direction in which the value of the third loss function changes fastest. For example, the gradient information may be a vector, and the embodiment of the present application does not limit the form of the gradient information.

Step A3: Update the seventh watermark according to the gradient information to obtain the first watermark generated for the deep forgery model.

According to the above description, it can be seen that the seventh watermark is an initialization watermark used to generate the first watermark. Therefore, after obtaining the gradient information, the embodiment of the present application updates the seventh watermark according to the gradient information, thereby obtaining the first watermark based on the seventh watermark. watermark.

In an exemplary embodiment, the number of deepfake models and the number of gradient information are both multiple. Then for a third sample image, the above-mentioned step A1 and step A2 will be performed for each deepfake model respectively. Therefore, referring to Figure 1 and Figure 5, one gradient information can be obtained for each deep forgery model, so that multiple deep forgery models correspond to multiple gradient information one-to-one.

Correspondingly, step A3 further includes the following steps A31 to A33.

Step A31: average multiple gradient information to obtain target gradient information.

Among them, since the target gradient information is obtained by averaging multiple gradient information, the target gradient information is also the average value of the multiple gradient information, and the target gradient information is expressed as g _avg . Referring to Figures 1 and 5, averaging multiple gradient information is also called gradient fusion.

Step A32: Calculate the target gradient information through the symbolic function to obtain the first calculation result; generate the second calculation result according to the first calculation result, and superimpose the second calculation result to the fourth sample image to obtain the fifth sample image; The five sample images are subject to upper and lower bound constraints to obtain the sixth sample image; the third sample image is removed from the sixth sample image to obtain the eighth watermark.

Among them, the target gradient information is calculated through the symbolic function. The values less than -1 in the target gradient information can be converted to -1, the values that are 0 are kept as 0, and the values greater than 1 are converted to 1, thereby obtaining the first Calculation results. If the sign function is represented as sign(·), then the first calculation result can be correspondingly expressed as sign(g _avg ).

Illustratively, the embodiments of the present application do not limit the method of generating the second calculation result based on the first calculation result, and the generation method can be set according to actual requirements. For example, the generation method is to use the product of the first calculation result and the constant a as the second calculation result. The value of the constant a is, for example, 0.1, which is not limited here. Correspondingly, the second calculation result can be expressed as asign(g _avg ).

则第四样本图像

表示为如下的公式(2)： In the above step A1, the third sample image is represented as I, the seventh watermark is represented as W, and the fourth sample image is represented as I+W. In order to facilitate the representation of the iterative training process here, the seventh watermark is expressed as P ^r and the fourth sample image is expressed as

Then the fourth sample image

Expressed as the following formula (2):

另外，按照如下的公式(3)对第五样本图像进行上下限约束，得到第六样本图像

Since the fourth sample image and the second calculation result have been obtained, the second calculation result can be superimposed on the fourth sample image to obtain the fifth sample image.

In addition, the fifth sample image is subject to upper and lower bound constraints according to the following formula (3) to obtain the sixth sample image

Among them, slip _I,∈ {·} represents the calculation of an upper and lower limit constraint. When the value corresponding to the channel of the pixel included in the fifth sample image is greater than the upper limit, the value is converted into the upper limit. When the fifth sample image includes When the value corresponding to the channel of the pixel is less than the lower limit, the value is converted to the lower limit. Among them, the upper limit and lower limit can be determined according to actual needs and are not limited here.

After obtaining the sixth sample image, the third sample image is removed from the sixth sample image according to the following formula (4) to obtain the eighth watermark P ^r+1 . This eighth watermark is also called adversarial perturbation:

It should be noted that r in the above formulas (2), (3) and (4) is used to represent the number of iterations, and the number of iterations is set according to actual needs. For example, when the eighth watermark is obtained through an iterative process, the value of r is 0 and the eighth watermark is P ¹ . For another example, when the eighth watermark is obtained through 10 iterations, the value of r is 9 and the eighth watermark is P ¹⁰ .

Step A33: perform a weighted average of the seventh watermark and the eighth watermark to obtain the first watermark.

After obtaining the eighth watermark, the seventh watermark and the eighth watermark can be weighted and averaged according to the following formula (5) to obtain the first watermark W′:

W′←(1-α)W+αP (5)

Among them, α is the weight, and the value of α is, for example, 0.01, which can be determined based on experience or actual needs. W is the seventh watermark, which is equivalent to the above-mentioned P ^r , and P is the eighth watermark, which is equivalent to the above-mentioned P ^r+1 .

The above steps A1 to A3 are a generation method based on the PGD (Projected Gradient Descent) algorithm. Moreover, the above steps A1 to A3 are described for one third sample image. The embodiment of the present application can provide multiple sets of images, each set of images including multiple third sample images. In this case, first process the first third sample image in the first group of images according to steps A1 to A3 to obtain the first watermark 1, and then process the second third sample image in the first group of images. The three sample images and the first watermark 1 are substituted into step A1, and steps A1 to A3 are repeatedly executed to obtain the first watermark 2, and so on, until all the third sample images in the first group of images have been used, and we obtain The first watermark K, K is the number of third sample images included in the first group of images. After that, use the second set of images to substitute the first third sample image and the first watermark K in the second set of images into step A1, thereby repeating steps A1 to A3, and so on, until all The last third sample image in the last set of images has also been used, resulting in the first watermark generated for the defense-in-depth model.

In an exemplary embodiment, obtaining the second watermark carrying target information may include: providing a second watermark input interface, obtaining the second watermark in the form of an image uploaded by the user from the second watermark input interface, or obtaining the second watermark uploaded by the user from the second watermark input interface. The second watermark is entered in the form of a string through the second watermark input interface. Of course, in addition to obtaining the second watermark provided by the user, you can also obtain the second watermark stored locally, or obtain the second watermark sent by other electronic devices, servers, etc. The implementation of this application does not limit the acquisition method of the second watermark.

Step 402: Obtain a target coding model, which is used to embed watermarks on the image.

Among them, the target encoding model is the encoder mentioned above. The target encoding model can be a model based on a convolutional neural network or other artificial intelligence models, which are not limited here. In this embodiment of the present application, the target coding model has the ability to embed watermarks on images, and can make the difference between the image before embedding the watermark and the image after embedding the watermark less than a reference threshold. In other words, the embedded watermark is made invisible in the image, thereby enabling hidden embedding of the watermark. Based on this, in the process of training to obtain the target encoding model, it is necessary to combine the reference threshold for training. Therefore, in an exemplary embodiment, the method provided by the embodiment of the present application also includes the following steps B1 to B3.

Step B1, obtain the first sample image, the third watermark carrying the first information and the initial coding model, input the first sample image into the initial coding model, obtain the first result image output by the initial coding model, and convert the first sample The image and the third watermark are input into the initial encoding model to obtain a second result image output by the initial encoding model, and the second result image is embedded with the third watermark.

The first sample image may be an image that has not yet embedded a watermark. The first sample image may be from a public data set such as CelebA. The first sample image may be located in the same data set as the above-mentioned third sample image. The concentrated image, or the first sample image and the above-mentioned third sample image may be images located in different data, which is not limited here. The third watermark carrying the first information is of the same type as the second watermark. For example, the second watermark and the third watermark are both in image form, or the second watermark and the third watermark are both in string form. In addition, the first information carried by the third watermark can be randomly generated, and the first information is not limited in this embodiment of the application. The obtained initial encoding model is an initialization model used to generate the target encoding model.

Since the first sample image has been acquired, the first sample image can be input into the initial encoding model, and the initial encoding model outputs the first result image obtained by encoding. Among them, the first sample image is represented as M, the initial encoding model is represented as E(·), and the first result image is represented as E(M). Since the third watermark is also obtained, both the first sample image and the third watermark can be input into the initial encoding model. The initial encoding model outputs the second result image obtained by encoding. The second result image can be considered to have been embedded. The first sample image of the third watermark. Among them, the third watermark is represented as N, the initial coding model is still represented as E(·), and the second result image is represented as E(M,N).

Step B2: Determine a first loss function based on the first result image and the second result image. The first loss function is used to indicate the difference between the first result image and the second result image.

Among them, the first loss function determined according to the first result image and the second result image is used to indicate the difference between the first result image and the second result image. This difference also represents the difference between the first sample image and the embedded image. The difference between the first sample image of the third watermark. The embodiment of the present application does not limit the method of determining the first loss function. Taking the first loss function obtained by calculating the first result image and the second result image through MSE as an example, the first loss function Loss _encoding is expressed as The following formula (6):

Loss _encoding =MSE(E(M),E(M,N)) (6)

For the process of calculating the first result image and the second result image through MSE, please refer to the process of calculating the third result image and the fourth result image through MSE explained in step 301 above. No further details will be given.

Step B3: Update the initial coding model through the process of minimizing the first loss function to obtain the target coding model, where the minimized first loss function is smaller than the reference threshold.

In the process of minimizing the first loss function, gradient calculation is performed on the first loss function to obtain the first gradient. Gradient backpropagation is performed based on the first gradient to obtain the target encoding when the first loss function is minimized. model parameters. Afterwards, the target encoding model is obtained by updating the initial encoding model parameters in the initial encoding model to the target encoding model parameters.

It should be noted that the minimized first loss function is smaller than the reference threshold, and the reference threshold can be a value infinitely close to 0, which is not limited here. Since the minimized first loss function is smaller than the reference threshold, and the first loss function represents the first sample image (the image before embedding the watermark) and the first sample image that has embedded the third watermark (the image after embedding the watermark) ), therefore, after updating the initial coding model by minimizing the first loss function to obtain the target coding model, the target coding model can make the difference between the image before embedding the watermark and the image after embedding the watermark less than Reference threshold is used to make the embedded watermark invisible in the image, thereby achieving hidden embedding of the watermark.

Step 403: Embed the first watermark and the second watermark into the initial image through the target encoding model to obtain the target image, and the difference between the initial image and the target image is less than the reference threshold.

The initial image may be a photographed image or an image captured from a video, which is not limited here. The embodiment of the present application embeds the obtained first watermark and second watermark into the initial image through the target coding model to obtain the target image. The initial image is the image before the watermark is embedded, and the target image is the image after the watermark is embedded. Since the target encoding model can make the difference between the image before embedding the watermark and the image after embedding the watermark less than the reference threshold, the difference between the initial image and the target image is less than the reference threshold. That is to say, the first watermark and the second watermark are hidden and embedded in the initial image, and the first watermark and the second watermark are not visible in the target image.

In an exemplary embodiment, considering that the first watermark is generated for the deepfake model, the first watermark itself is an invisible watermark, and the embedding of the first watermark may be a direct superposition of the first watermark. The second watermark carries target information, and the second watermark is often a visible watermark, so it is necessary to implement the embedding of the second watermark through the target coding model. Based on these considerations, the first watermark and the second watermark are embedded in the initial image through the target coding model to obtain the target image, which can include the following two embedding methods.

Embedding method one: superimpose the first watermark onto the initial image to obtain the first image, input the first image and the second watermark into the target encoding model, and obtain the target image output by the target encoding model. In the first embedding method, the first watermark is superimposed on the initial image to obtain the first image. For the method of superimposing the first watermark, please refer to the method of superimposing the seventh watermark on the third sample image in step 301, which will not be described again here. After that, the second watermark is embedded into the first image through the target encoding model to obtain the target image.

In the second embedding method, the second watermark and the initial image are input into the target encoding model to obtain the second image output by the target encoding model, and the first watermark is superimposed on the second image to obtain the target image. In the second embedding method, the second watermark is first embedded into the initial image through the target encoding model to obtain the second image. After that, the first watermark is superimposed on the second image to obtain the target image. The method of superimposing the first watermark can also refer to the method of superimposing the seventh watermark on the third sample image in step 301, which will not be described again here.

Through the above description, embodiments of the present application can embed the first watermark and the second watermark into the target image, so that the target image can not only defend against deep forgery models, but also carry target information. In addition, embodiments of the present application can also detect whether such a target image has been tampered with by a deep forgery model. Please refer to the following description for details.

In an exemplary embodiment, after embedding the first watermark and the second watermark into the initial image through the target encoding model to obtain the target image, the method further includes: obtaining an updated image generated based on the target image; inputting the updated image into the target decoding model to obtain The fourth watermark output by the target decoding model; determine the bit error between the second watermark and the fourth watermark; when the bit error is greater than the error threshold, determine that the updated image is a deep forgery of the target image through the deep forgery model The resulting image.

Among them, embodiments of the present application can upload the target image to a network scenario where image tampering may occur, and download the uploaded target image after a period of time to obtain an updated image, so as to detect whether the updated image has been tampered with, or Indicates whether the deepfake model has been deepfaked. Alternatively, in the embodiment of the present application, the target image can also be input into the deep forgery model to obtain an updated image output by the deep forgery model. The embodiment of the present application does not limit the method of generating the updated image.

The target decoding model is the decoder in the above description. The target encoding model can be a model based on a convolutional neural network or other artificial intelligence models, which is not limited here. This target decoding model is used to extract watermarks from images with embedded watermarks. Since the update image is generated based on the target image, and the target image is embedded with the first watermark and the second watermark, the update image is also embedded with the watermark. Therefore, after the updated image is input to the target decoding model, the target decoding model can extract and output the fourth watermark from the updated image.

After that, the bit error between the second watermark and the fourth watermark can be calculated, and the bit error can be compared with an error threshold. The error threshold is, for example, 0.4, which is not limited here. If the bit error is greater than the error threshold, it is determined that the updated image is a tampered image, or in other words, it is determined that the updated image is an image obtained by deep forging the target image through a deep forgery model. The reason is that when the update image has been tampered with, the fourth watermark extracted from the update image is the distorted first watermark and the distorted second watermark. Therefore, the fourth watermark is different from the normal undistorted second watermark. The difference between the watermarks is large, so that the bit error between the second watermark and the fourth watermark is greater than the error threshold. If the bit error is less than or equal to the error threshold, it is determined that the updated image is an image that has not been tampered with, or in other words, it is determined that the updated image is not an image obtained by deep forging the target image through a deep forgery model. The reason is that when the update image has not been tampered with, the fourth watermark extracted from the update image is the normal undistorted first watermark and the second watermark, so that the difference between the second watermark and the fourth watermark is The bit error is less than or equal to the error threshold.

In some implementations, the second watermark and the fourth watermark are watermarks in the form of images. Then determining the bit error between the second watermark and the fourth watermark includes: comparing the image quality index of the second watermark with the image quality index of the fourth watermark, and taking the difference between the image quality indexes as the bit error. For example, image quality indicators include but are not limited to PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity, structural similarity), etc., which are not limited here.

In other implementations, the second watermark and the fourth watermark are watermarks in the form of character strings. Then determining the bit error between the second watermark and the fourth watermark includes: determining the total number of bits included in the second watermark, and the target number of different bits included in the second watermark and the fourth watermark, and setting the target number. The ratio of the quantity to the total quantity is used as the bit error. For example, the second watermark is a string of 4 bits 1100, and the fourth watermark is also a string of 4 bits 0000, then the total number mentioned above is 4, the target number mentioned above is 2, and the bit error is 0.5.

For example, see Figure 6 for an example of the deepfake detection process. The first watermark is superimposed on the initial image, and the obtained first image and the second watermark are input into the target encoding model to obtain the target image output by the target encoding model. Afterwards, the target image is input into the deep forgery model to obtain an updated image output by the deep forgery model, and the updated image is input into the target decoding model to obtain a fourth watermark output from the updated image. Then, the second watermark and the fourth watermark are compared to determine whether the updated image has been tampered with based on the comparison result, thereby completing deep forgery detection.

Of course, before using the target decoding model in this embodiment of the present application, the target decoding model can be trained first. In an exemplary embodiment, before inputting the updated image into the target decoding model, the method further includes the following steps C1 to C3.

Step C1: Obtain the second sample image and the initial decoding model. The second sample image is embedded with the fifth watermark carrying the second information; input the second sample image into the initial decoding model to obtain the sixth watermark output by the initial decoding model.

Among them, the second sample image is an image that has been embedded with the fifth watermark. The embodiment of the present application can directly obtain the second sample image that meets the requirements, or can also produce the second sample image through the process of embedding the watermark. For example, the above step B1 The second result image in is used as the second sample image, then the fifth watermark is also the third watermark in the above step B1. The method of obtaining the second sample image is not limited here. The fifth watermark carrying the second information is of the same type as the second watermark, and the second information can be randomly generated. In addition, the initial decoding model is an initialization model used to generate the target decoding model.

After the second sample image is acquired, the second sample image is input into the initial decoding model, and the initial decoding model outputs the decoded sixth watermark. The fifth watermark and the sixth watermark are of the same type, for example, both are in image form or both are in string form. Among them, taking the second result image as the second sample image and the fifth watermark as the third watermark as an example, the fifth watermark is represented as N and the second sample image is represented as E(M,N). The initial decoding model Expressed as D(·), the sixth watermark is expressed as D(E(M,N)).

Step C2: Determine the second loss function based on the fifth watermark and the sixth watermark.

Wherein, the second loss function determined according to the fifth watermark and the sixth watermark is used to indicate the difference between the fifth watermark and the sixth watermark. This application does not limit the method of determining the second loss function. Taking the second loss function obtained by calculating the fifth and sixth watermarks through BCE (Binary Cross Entropy) with Logit (logistic regression) as an example, the second loss function Loss _decoding represents is the following formula (7):

Loss _decoding =BCEwithLogitsLoss(N,D(E(M,N))) (7)

Among them, BCEwithLogitsLoss is BCE with Logit. During the calculation process, the fifth watermark is normalized to obtain the normalized fifth watermark, and the sixth watermark is normalized to obtain the normalized For the sixth watermark, the Sigmoid (S-shaped growth curve) function can be used for normalization. After that, the normalized fifth watermark and the normalized sixth watermark are calculated through BCE to obtain the second loss function.

Step C3: Update the initial decoding model through the process of minimizing the second loss function to obtain the target decoding model.

In the process of minimizing the second loss function, gradient calculation is performed on the second loss function to obtain the second gradient. Gradient backpropagation is performed based on the second gradient to obtain the target decoding when the second loss function is minimized. model parameters. Afterwards, the target decoding model is obtained by updating the initial decoding model parameters in the initial decoding model to the target decoding model parameters. Since the second loss function is minimized and the second loss function represents the difference between the fifth watermark and the sixth watermark, the difference between the fifth watermark and the sixth watermark can be minimized. Since the fifth watermark is the actual embedded watermark, and the sixth watermark is the extracted watermark, minimizing the difference between the fifth watermark and the sixth watermark can ensure that the fifth watermark and the sixth watermark are close enough to ensure the target The watermark extracted by the decoding model is more accurate.

To sum up, the embodiment of this application embeds the first watermark generated for the deep forgery model and the second watermark carrying target information into the initial image, and uses the target encoding model in the embedding process, so that the initial image is the same as the one obtained after embedding. The difference between the target images is small, so that the first watermark and the second watermark are invisible in the target image to avoid affecting the content recorded in the target image. Moreover, this kind of target image can not only defend against deep forgery models, but also carry customized target information. Therefore, the quality and security of the target image are higher, and the accuracy of image processing is better. The embodiments of the present application can also detect whether the target image has been tampered with by the deep fake model, thereby further ensuring the security of the target image.

An embodiment of the present application provides an image processing device. See Figure 7. The device includes the following modules.

The acquisition module 701 is used to acquire the first watermark generated for the deep forgery model and the second watermark carrying target information;

The acquisition module 701 is also used to acquire the target coding model, which is used to embed watermarks on images;

The embedding module 702 is used to embed the first watermark and the second watermark into the initial image through the target encoding model to obtain the target image, where the difference between the initial image and the target image is less than the reference threshold.

In an exemplary embodiment, the embedding module 702 is used to superimpose the first watermark onto the initial image to obtain the first image, and input the first image and the second watermark into the target encoding model to obtain the target image output by the target encoding model; Alternatively, the second watermark and the initial image are input into the target encoding model to obtain the second image output by the target encoding model, and the first watermark is superimposed on the second image to obtain the target image.

In an exemplary embodiment, the acquisition module 701 is also used to acquire the first sample image, the third watermark carrying the first information, and the initial encoding model; input the first sample image into the initial encoding model to obtain the initial encoding model output The first result image; input the first sample image and the third watermark into the initial coding model to obtain the second result image output by the initial coding model, and the second result image is embedded with the third watermark; according to the first result image and the second The result image determines the first loss function, and the first loss function is used to indicate the difference between the first result image and the second result image; the initial encoding model is updated through the process of minimizing the first loss function to obtain the target encoding model, where, The minimized first loss function is smaller than the reference threshold.

In an exemplary embodiment, the acquisition module 701 is also used to acquire an updated image generated based on the target image; input the updated image into the target decoding model to obtain the fourth watermark output by the target decoding model; determine the relationship between the second watermark and the fourth watermark. bit error between; when the bit error is greater than the error threshold, it is determined that the updated image is an image obtained by deep forging the target image through a deep forgery model.

In an exemplary embodiment, the acquisition module 701 is also used to acquire a second sample image and an initial decoding model. The second sample image is embedded with a fifth watermark carrying second information; input the second sample image into the initial decoding model to obtain The sixth watermark output by the initial decoding model; determine the second loss function based on the fifth watermark and the sixth watermark; update the initial decoding model through the process of minimizing the second loss function to obtain the target decoding model.

In an exemplary embodiment, the acquisition module 701 is used to acquire the third sample image, the seventh watermark and the deep forgery model; input the third sample image into the deep forgery model to obtain the third result image output by the deep forgery model; convert the third sample image to the deep forgery model. The seven watermarks are superimposed on the third sample image to obtain the fourth sample image; the fourth sample image is input into the deep forgery model to obtain the fourth result image output by the deep forgery model; the third loss is determined based on the third result image and the fourth result image. function; determine the gradient information based on the third loss function; update the seventh watermark based on the gradient information to obtain the first watermark generated for the deep forgery model.

In an exemplary embodiment, there are multiple deep fake models and multiple gradient information, and multiple deep fake models correspond to multiple gradient information one-to-one; the acquisition module 701 is used to average multiple gradient information, Obtain the target gradient information; calculate the target gradient information through the symbolic function to obtain the first calculation result; generate the second calculation result based on the first calculation result, and superimpose the second calculation result to the fourth sample image to obtain the fifth sample image; Apply upper and lower bound constraints to the fifth sample image to obtain the sixth sample image; remove the third sample image from the sixth sample image to obtain the eighth watermark; perform a weighted average of the seventh watermark and the eighth watermark to obtain the first watermark.

It should be noted that for the technical effects of the device shown in Figure 7, please refer to the technical effects of the method embodiment shown in Figure 4, which will not be described again here. Moreover, when the device provided in the above embodiment implements its functions, only the division of the above functional modules is used as an example. In practical applications, the above functions can be allocated to different functional modules as needed, that is, the internal structure of the device Divide it into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be described again here.

In an exemplary embodiment, the embodiment of the present application also provides an electronic device. The electronic device includes a memory and a processor; at least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor, so that the electronic device implements active deep forgery provided by any exemplary embodiment of the present application. Defense method, or the image processing method corresponding to Figure 4.

Referring to FIG. 8 , a schematic structural diagram of an electronic device 800 provided by an embodiment of the present application is shown. The electronic device 800 can be a portable mobile electronic device, such as: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Compression Standard Audio Layer 3), MP4 (Moving Picture Experts Group Audio Layer IV , motion picture expert compresses standard audio levels 4) players, laptops or desktop computers. Electronic device 800 may also be referred to as user equipment, portable electronic device, laptop electronic device, desktop electronic device, and other names. Generally, the electronic device 800 includes: a processor 801 and a memory 802.

The processor 801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 801 can adopt at least one of the group consisting of DSP (Digital Signal Processing, digital signal processing), FPGA (Field-Programmable Gate Array, field programmable gate array), and PLA (Programmable Logic Array, programmable logic array). A form of hardware implementation. The processor 801 can also include a main processor and a co-processor. The main processor is a processor used to process data in the wake-up state, also called CPU (Central Processing Unit, central processing unit); the co-processor is A low-power processor used to process data in standby mode. In some embodiments, the processor 801 may be integrated with a GPU (Graphics Processing Unit, image processor), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen 805 . In some embodiments, the processor 801 may also include an AI (Artificial Intelligence, artificial intelligence) processor, which is used to process computing operations related to machine learning.

Memory 802 may include one or more computer-readable storage media, which may be non-transitory. Memory 802 may also include high-speed random access memory, and non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 802 is used to store at least one instruction, and the at least one instruction is used to be executed by the processor 801 to implement the depth-oriented method provided by the method embodiments in this application. The fake active defense method, or the image processing method corresponding to Figure 4.

In some embodiments, the electronic device 800 optionally further includes: a peripheral device interface 803 and at least one peripheral device. The processor 801, the memory 802 and the peripheral device interface 803 may be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 803 through a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of the group consisting of a radio frequency circuit 804, a display screen 805, a camera component 806, an audio circuit 807, a positioning component 808 and a power supply 809.

The peripheral device interface 803 may be used to connect at least one I/O (Input/Output, input/output) related peripheral device to the processor 801 and the memory 802 . In some embodiments, the processor 801, the memory 802, and the peripheral device interface 803 are integrated on the same chip or circuit board; in some other embodiments, any one of the processor 801, the memory 802, and the peripheral device interface 803 or Both of them can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

The radio frequency circuit 804 is used to receive and transmit RF (Radio Frequency, radio frequency) signals, also called electromagnetic signals. Radio frequency circuit 804 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 804 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. Radio frequency circuitry 804 can communicate with other electronic devices through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: metropolitan area network, mobile communication networks of all generations (2G, 3G, 4G and 5G), wireless LAN and/or Wi-Fi (Wireless Fidelity, wireless fidelity) network. In some embodiments, the radio frequency circuit 804 may also include NFC (Near Field Communication) related circuits, which is not limited in this application.

The display screen 805 is used to display UI (User Interface, user interface). The UI can include graphics, text, icons, videos, and any combination thereof. When display screen 805 is a touch display screen, display screen 805 also has the ability to collect touch signals on or above the surface of display screen 805 . The touch signal can be input to the processor 801 as a control signal for processing. At this time, the display screen 805 can also be used to provide virtual buttons and/or virtual keyboards, also called soft buttons and/or soft keyboards. In some embodiments, there may be one display screen 805, which is disposed on the front panel of the electronic device 800; in other embodiments, there may be at least two display screens 805, which are disposed on different surfaces of the electronic device 800 or folded. Design; In other embodiments, the display screen 805 may be a flexible display screen, disposed on a curved or folded surface of the electronic device 800. Even the display screen 805 can be set into a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 805 can be made of LCD (Liquid Crystal Display, liquid crystal display), OLED (Organic Light-Emitting Diode, organic light-emitting diode) and other materials.

The camera assembly 806 is used to capture images or videos. Optionally, the camera assembly 806 includes a front camera and a rear camera. Usually, the front camera is set on the front panel of the electronic device, and the rear camera is set on the back of the electronic device. In some embodiments, there are at least two rear cameras, one of which is a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, so as to realize the integration of the main camera and the depth-of-field camera to realize the background blur function. Integrated with a wide-angle camera to achieve panoramic shooting and VR (Virtual Reality, virtual reality) shooting functions or other integrated shooting functions. In some embodiments, camera assembly 806 may also include a flash. The flash can be a single color temperature flash or a dual color temperature flash. Dual color temperature flash refers to a combination of warm light flash and cold light flash, which can be used for light compensation under different color temperatures.

Audio circuitry 807 may include a microphone and speakers. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input to the processor 801 for processing, or to the radio frequency circuit 804 to implement voice communication. For the purpose of stereo collection or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the electronic device 800 . The microphone can also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert electrical signals from the processor 801 or the radio frequency circuit 804 into sound waves. The loudspeaker can be a traditional membrane loudspeaker or a piezoelectric ceramic loudspeaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves that are audible to humans, but also convert electrical signals into sound waves that are inaudible to humans for purposes such as ranging. In some embodiments, audio circuitry 807 may also include a headphone jack.

The positioning component 808 is used to locate the current geographical location of the electronic device 800 to implement navigation or LBS (Location Based Service). The positioning component 808 may be a positioning component based on the United States' GPS (Global Positioning System), China's Beidou system, Russia's Galileo system, or the European Union's Galileo system.

The power supply 809 is used to power various components in the electronic device 800 . Power source 809 may be AC, DC, disposable batteries, or rechargeable batteries. When the power supply 809 includes a rechargeable battery, the rechargeable battery may support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

In some embodiments, electronic device 800 also includes one or more sensors 810 . The one or more sensors 810 include, but are not limited to: an acceleration sensor 811 , a gyroscope sensor 812 , a pressure sensor 813 , a fingerprint sensor 814 , an optical sensor 815 and a proximity sensor 816 .

The acceleration sensor 811 can detect the acceleration on the three coordinate axes of the coordinate system established by the electronic device 800 . For example, the acceleration sensor 811 can be used to detect the components of gravity acceleration on three coordinate axes. The processor 801 can control the display screen 805 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 811 . The acceleration sensor 811 can also be used to collect game or user motion data.

The gyro sensor 812 can detect the body direction and rotation angle of the electronic device 800 , and the gyro sensor 812 can cooperate with the acceleration sensor 811 to collect the user's 3D movements on the electronic device 800 . Based on the data collected by the gyro sensor 812, the processor 801 can implement the following functions: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

The pressure sensor 813 may be disposed on the side frame of the electronic device 800 and/or on the lower layer of the display screen 805 . When the pressure sensor 813 is disposed on the side frame of the electronic device 800, it can detect the user's holding signal of the electronic device 800, and the processor 801 performs left and right hand identification or quick operation based on the holding signal collected by the pressure sensor 813. When the pressure sensor 813 is provided on the lower layer of the display screen 805, the processor 801 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 805. The operability control includes at least one of the group consisting of a button control, a scroll bar control, an icon control, and a menu control.

The fingerprint sensor 814 is used to collect the user's fingerprint. The processor 801 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 814, or the fingerprint sensor 814 identifies the user's identity based on the collected fingerprint. When the user's identity is recognized as a trusted identity, the processor 801 authorizes the user to perform relevant sensitive operations. The sensitive operations include unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 814 may be disposed on the front, back, or side of the electronic device 800 . When the electronic device 800 is provided with a physical button or a manufacturer's logo, the fingerprint sensor 814 can be integrated with the physical button or the manufacturer's logo.

Optical sensor 815 is used to collect ambient light intensity. In one embodiment, the processor 801 can control the display brightness of the display screen 805 according to the ambient light intensity collected by the optical sensor 815 . Specifically, when the ambient light intensity is high, the display brightness of the display screen 805 is increased; when the ambient light intensity is low, the display brightness of the display screen 808 is decreased. In another embodiment, the processor 801 can also dynamically adjust the shooting parameters of the camera assembly 806 according to the ambient light intensity collected by the optical sensor 815 .

The proximity sensor 816, also called a distance sensor, is usually provided on the front panel of the electronic device 800. The proximity sensor 816 is used to collect the distance between the user and the front of the electronic device 800 . In one embodiment, when the proximity sensor 816 detects that the distance between the user and the front of the electronic device 800 gradually becomes smaller, the processor 801 controls the display screen 805 to switch from the bright screen state to the closed screen state; when the proximity sensor 816 detects When the distance between the user and the front of the electronic device 800 gradually increases, the processor 801 controls the display screen 805 to switch from the screen-off state to the screen-on state.

Those skilled in the art can understand that the structure shown in FIG. 8 does not constitute a limitation on the electronic device 800, and may include more or fewer components than shown, or combine certain components, or adopt different component arrangements.

Embodiments of the present application provide a computer-readable storage medium. At least one instruction is stored in the computer-readable storage medium. The instruction is loaded and executed by a processor, so that the computer can implement any of the exemplary embodiments of the present application. An active defense method against deep forgery, or the image processing method corresponding to Figure 4.

Embodiments of the present application provide a computer program or computer program product. The computer program or computer program product includes: computer instructions. When the computer instructions are executed by the computer, the computer implements the methods provided by any exemplary embodiment of the present application. Active defense method for deep fakes, or the image processing method corresponding to Figure 4.

All the above optional technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described again one by one. Those of ordinary skill in the art can understand that all or part of the steps to implement the above embodiments can be completed by hardware, or can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium. The above-mentioned The storage media mentioned can be read-only memory, magnetic disks or optical disks, etc.

The present application has been described above through detailed implementation examples. Researchers and technicians in the field may make non-substantive changes in form or content based on the above steps without departing from the scope of the essential protection of the present application. Therefore, the present application is not limited to the content disclosed in the above embodiments, and the protection scope of the present application shall be subject to the claims.

Claims (10)

An image processing method, characterized in that the method includes: Obtain the first watermark generated for the deep forgery model and the second watermark carrying target information; Obtain a target coding model, which is used for watermark embedding for images; The first watermark and the second watermark are embedded in the initial image through the target encoding model to obtain a target image, and the difference between the initial image and the target image is less than a reference threshold. The method according to claim 1, characterized in that embedding the first watermark and the second watermark into the initial image through the target encoding model to obtain the target image includes: Superimpose the first watermark onto the initial image to obtain a first image, input the first image and the second watermark into the target encoding model, and obtain the target image output by the target encoding model. ; Or, input the second watermark and the initial image into the target encoding model to obtain the second image output by the target encoding model, and superimpose the first watermark onto the second image to obtain the target image. The method according to claim 1, characterized in that before embedding the first watermark and the second watermark into the initial image through the target encoding model to obtain the target image, the method further includes: Obtain the first sample image, the third watermark carrying the first information, and the initial coding model; Input the first sample image into the initial encoding model to obtain a first result image output by the initial encoding model; Input the first sample image and the third watermark into the initial encoding model to obtain a second result image output by the initial encoding model, and the second result image is embedded with the third watermark; Determine a first loss function based on the first result image and the second result image, the first loss function being used to indicate a difference between the first result image and the second result image; The initial coding model is updated through a process of minimizing the first loss function to obtain the target coding model, wherein the minimized first loss function is smaller than the reference threshold. The method according to any one of claims 1 to 3, characterized in that after embedding the first watermark and the second watermark into the initial image through the target coding model to obtain the target image, the method further include: Obtain an updated image generated based on the target image; Input the updated image into the target decoding model to obtain the fourth watermark output by the target decoding model; determining a bit error between the second watermark and the fourth watermark; When the bit error is greater than the error threshold, it is determined that the updated image is an image obtained by deep forging the target image through the deep forgery model. The method according to claim 4, characterized in that before inputting the updated image into the target decoding model, the method further includes: Obtaining a second sample image and an initial decoding model, where the second sample image is embedded with a fifth watermark carrying second information; Input the second sample image into the initial decoding model to obtain the sixth watermark output by the initial decoding model; Determine a second loss function according to the fifth watermark and the sixth watermark; The initial decoding model is updated through the process of minimizing the second loss function to obtain the target decoding model. The method according to any one of claims 1-3, characterized in that said obtaining the first watermark generated for the deep forgery model includes: Obtain the third sample image, the seventh watermark and the deepfake model; Input the third sample image into the deep forgery model to obtain a third result image output by the deep forgery model; Superimpose the seventh watermark onto the third sample image to obtain a fourth sample image; Input the fourth sample image into the deep forgery model to obtain a fourth result image output by the deep forgery model; determining a third loss function based on the third result image and the fourth result image; Determine gradient information according to the third loss function; The seventh watermark is updated according to the gradient information to obtain the first watermark generated for the deep forgery model. The method according to claim 6, characterized in that the number of deep forgery models and the number of gradient information are both multiple, and multiple deep forgery models correspond to multiple gradient information in one-to-one correspondence; The updating of the seventh watermark according to the gradient information to obtain the first watermark generated for the deep forgery model includes: Average the multiple gradient information to obtain target gradient information; Calculate the target gradient information through a symbolic function to obtain a first calculation result; Generate a second calculation result according to the first calculation result, superimpose the second calculation result to the fourth sample image, and obtain a fifth sample image; Apply upper and lower bound constraints to the fifth sample image to obtain a sixth sample image; Remove the third sample image from the sixth sample image to obtain an eighth watermark; The seventh watermark and the eighth watermark are weighted and averaged to obtain the first watermark. A device for adding watermarks, characterized in that the device includes: The acquisition module is used to acquire the first watermark generated for the deep forgery model and the second watermark carrying target information; The acquisition module is also used to acquire a target coding model, which is used to embed watermarks on images; An embedding module, configured to embed the first watermark and the second watermark into an initial image through the target encoding model to obtain a target image, where the difference between the initial image and the target image is less than a reference threshold. An electronic device, characterized in that the electronic device includes a memory and a processor; at least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to enable the electronic device to implement The image processing method according to any one of claims 1-7. A computer-readable storage medium, characterized in that at least one instruction is stored in the computer-readable storage medium, and the instruction is loaded and executed by a processor, so that the computer implements any one of claims 1-7. The image processing method described above.

Images